Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the effect of adversarial training on the action-value estimates of value-based deep RL agents. The paper considers both a toy learning problem where the optimal function approximator with and without adversarial regularization can be computed analytically, as well as networks trained on the arcade learning environment. It shows that adversarial training reduces the accuracy of the network’s ranking over actions at a given environment state, that it increases the gap between the action with the highest predicted value and the value of the second-best action, and that it increases the value of the action with the highest predicted value.

strength_and_weaknesses: Strengths:

- The paper offers some interesting insights on the trade-offs between adversarial robustness and accuracy of the action-value function in value-based deep RL.
- The toy example in Proposition 3.2 is a nice illustration of the instability of action rankings in adversarially robust agents.
- The discussion of the adversarial robustness literature seems reasonably comprehensive (though this isn’t my primary field).
- The metric used to estimate action-value accuracy is a smart way of trying to marginalize out noise in the agent trajectories in order to compare the action rankings.
- The finding of section 3 is somewhat surprising — I would not have anticipated such a large gap between the actions of the second-best and worst predicted action.


Weaknesses

- While the literature review discusses the adversarial RL literature, it does not cover many recent works studying the robustness of the action-value ranking of deep RL agents to non-adversarial perturbations. In particular, prior work [1] has shown that even gradient steps computed on disjoint subsets of the replay buffer can change the agent’s ranking over actions in a surprisingly large fraction of the state space. In the context of this prior work, the observation that the ranking of sub-optimal actions changes under adversarial regularization is unsurprising.
- It is widely observed that DQN outputs tend to systematically underestimate the true value of a state. As a result, it seems to me difficult to say whether the values output by the adversarially trained networks are indeed over-estimations of the true value, or whether they are simply accurate estimations of the true action-value function. I am therefore unconvinced as to whether the subsection’s title is necessarily accurate, and would like to see a comparison between the action-values and the true discounted return obtained by the policy to be convinced of this.
- The description of the adversarial regularizer is inconsistent with that described by Zhang et al. in their presentation of SA-DQN. Zhang et al. include a clipping hyperparameter $c$ which lower bounds the adversarial action gap and avoids the pathology of the value estimate of the optimal action growing arbitrarily large. I am curious about the value of $c$ used in this paper.
- Similarly, the value of the regularizer coefficient $\kappa$ is only briefly mentioned in the final  sentence of the supplementary material, and it is said to be in a set of values. However, I could not determine which value was used for the paper’s experiments, nor could I find any results describing the relationship between the phenomena described in the main body and the values of $\kappa$ and $c$. The epsilon value used for epsilon-greedy exploration is quite low (0.02). These values will likely play a large role in determining effect size of the phenomena described in this paper.
clarity,_quality,_novelty_and_reproducibility: The main message of the paper is clearly conveyed, but many important experiment details are not clear and I do not believe I could reproduce the experimental results based on the description in the appendix. Further, the description of the adversarial regularizer used in both the theoretical analysis and empirical investigation differs in an important way from that used by the main baseline SA-DQN as it is not lower bounded. This is an important detail and I worry that many of the results may be an artifact of this distinction, and that the effect sizes seen in the paper would be much smaller under a bounded adversarial regularizer.

I haven’t previously seen an analysis of the effect of adversarial regularization on the action ranking of the Q-function in RL, though the more general robustness of value-based deep RL agents’ action rankings to optimization noise has been studied previously.
summary_of_the_review: This paper highlights an important tradeoff in adversarial training of value-based deep RL agents: that optimizing for the robustness of the greedy policy may have unintended effects on the sub-optimal actions. The paper suffers from two principal weaknesses: firstly, that prior work has already identified that the relative ranking of actions in DQN-like agents is unstable, and so the findings here seem to be a natural consequence of that observation coupled with the focus of adversarial regularization on only the greedy action. Secondly, the regularization scheme under consideration differs from that studied in prior work in a way that would exacerbate the effects observed in the paper, and lack of detail on the experiment setup description makes it difficult to identify whether the experimental findings would reflect practical implementations of state-of-the-art adversarial training schemes for RL.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates that adversarially trained deep neural policies can potentially lead to inconsistencies and overestimations in the state-action value functions. A linear case theoretical analysis and empirical studies on Arcade Learning Environment are provided to verify the observation. 
strength_and_weaknesses: Strengths: 
1. The observation, that vanilla trained deep neural policies have more accurate and consistent estimates for the state-action values than the state-of-the-art adversarially trained deep neural policies, is counterfactual and interesting. 

Weaknesses:
1. The theoretical analysis doesn’t support much about the empirical study. The analysis assumed that action-value function is linear and as simple as an inner product, which can not be extended to the cases of empirical studies. 
2. Lack of contribution. Demonstrating the problem without providing potential solutions may not quite yet reach the high bar set for ICLR. 
3. The main claim needs more investigation or explanations.

clarity,_quality,_novelty_and_reproducibility: Good clarity. The novelty is limited. Code is not provided and implementation details are missed. Hard for reproducibility. 
summary_of_the_review: Overall, the depth of this contribution may not quite yet reach the high bar set for ICLR. The main claim, that vanilla trained deep neural policies have more accurate and consistent estimates for the state-action values than the state-of-the-art adversarially trained deep neural policies, needs more investigations. 

The paper defines a performance drop metric as a measure of the accuracy of state-action values. 
The authors pick a random state $s$ during a rollout, make the agent choose action $a_i$ in state $s$, and in all other states have the agent choose the best Q-value actions. The performance drop due to the action modification is measured. The authors claim that adversarially training would make the performance drop large and hence is not accurate. 

However, all the experiments are done on discrete action spaces, and I would expect there would be a big change in performance due to action modifications, for example, turning right in a maze changed to turning left would impact a lot for finding the exit. I think the performance drop due to action modification could be reasonably large and large performance drop actually shows that the Q-value function could discriminate good actions from bad actions well, which is the target of adversarial learning. 

I also kind of disagree with the overestimation claim. Table 2 doesn’t show the true average Q-values, so that we don’t know which one is overestimated. To me, the value scale (<10) is still under control. The target of Q-value function is to help the learning of policies, so as long as the good action regions are assigned higher and reasonable Q values, Q function could provide good learning signals for the policy. 

Implementation details are not discussed, and the work is not reproducible. The paper only mentions using state-of-the-art adversarially trained deep neural policies, while not includes the actually used algorithm.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies a regularization that can improve the learning of Q-values when used in combination with deep learning. The paper shows that these regularization techniques can cause inconsistencies and overestimations in the state-action value functions while vanilla trained deep reinforcement learning policies have more accurate estimates for the Q-values.
strength_and_weaknesses: Strengths:
- This is an important area of study
- The paper highlights some problems that were not fully investigated in the past

Weaknesses
- The contributions are not very strong in the current state of the paper (see below).
clarity,_quality,_novelty_and_reproducibility: CLARITY

The paper could be written more formally and has also some (minor) specific elements that are not accurate, for instance:
- The paper write "Our results essentially demonstrate that vanilla trained deep reinforcement learning policies have more accurate and consistent estimates for the state-action values". I don't believe that the paper actually demonstrates this. It could be rephrased as an empirical evaluation.
- "our investigation lays out intrinsic properties (...) to building robust and optimal deep neural policies.": In deep RL we usually do not expect that the policies will actually be optimal.
- There are many reference to "Example 3", however I do not find any explicit mention of what that example 3 (we can guess that it is the one from the top of page 4)


QUALITY

- Proposition 3.1 and 3.2: There are interesting insights from these propositions but as they are written, it seems that they lack any kind of generality and that they only provide information about a given example. It might be more useful to rewrite them in a more general form (e.g. something along the lines of "if a function approximator of a given type is used, then the regularizer $R(\theta)$ can lead to both overestimation of the first ranked action, and re-ordering of the ranking of the suboptimal actions"). For some propositions, an example could then sufficient as a proof but the propositions would then be more relevant.
- Most experiments are provided with limited relevant baselines and on three environments.
- The appendix mentions a discount factor $\gamma$ of 1. That seems a strange choice as it is known that a discount factor of 1 will in many cases create instabilities and divergence, particularly when used with deep learning as a function approximator and long horizons.

NOVELTY

The paper provides interesting insights for a relevant problem in RL, however the novelty is a bit unclear in terms of contributions. Two key elements that are highlighted in the conclusion are 
- "adversarially trained deep neural policies in certain MDPs completely loses all the information in the state-action value function that contains the relative ranking of the actions" and
- "we show that state-of-the-art adversarially trained deep neural policies learn overestimated state-action values"

These are in fact common problems observed for RL with function approximators and it is unclear how this paper provides clear insight on whether this is generally worse/better because of the relatively limited set of experiments and of the relatively narrow theory that is developed.


REPRODUCIBILITY:
The source code is not provided. The papers does provide some of the experiments in the appendix.
summary_of_the_review: The paper looks into an interesting problem and has initial good contributions. There are however elements that need to be improved both on the form and on the substance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The initial reviews questioned whether some specific claims made by the authors are correct/defendable. While the authors' responses have successfully addressed some of these key concerns, in particular the validity of the paper's methodology, concerns about the strength of the paper's contribution remain. The AC would like to rephrase a reviewer's feedback during discussion, which the AC agrees with: 

"The paper has three main contributions: an analysis of adversarial regularization in an example linear function approximator, empirical evidence that the predicted value of the greedy action increases in practical adversarial training schemes, and empirical evidence that the value function's ranking over non-greedy actions decreases in accuracy.

1. The linear case study is a fairly straightforward computation and doesn't stand on its own as a significant contribution; the example MDP is somewhat contrived and it is difficult to see how the intuition developed here might generalize to more realistic function approximators. 

2. The empirical observation that adversarial training reduces the accuracy of an agent's ranking of sub-optimal actions is a natural implication of prior work on the instability of action-value rankings in value-based RL algorithms. 

3. Finally, independent of whether the increase in the predicted Q-value pushes it higher than the true expected return (as has been debated at length in the discussion), it is difficult to see the value of this observation. Increasing the value of the greedy action is a natural way to decrease the sensitivity of the resulting policy to perturbations of its parameters-- it would be surprising if the network didn't increase its predicted value."

justification_for_why_not_higher_score: A paper identifying a set of predictable effects of an existing method is often expected to at least offer some practical insight into whether, and if so how, to avoid or mitigate these effects. In its current form, the paper doesn't offer much new insight from this perspective.
Decision: Reject