The Adversarial Regulation of the Temporal Difference Loss Costs More Than Expected
Authors: Ezgi Korkmaz
Venue: Submitted to ICLR 2023 (2023)

Abstract: Deep reinforcement learning research has enabled reaching significant performance levels for sequential decision making in MDPs with highly complex observations and state dynamics with the aid of deep neural networks. However, this aid came with a cost that is inherent to deep neural networks which have increased sensitivities towards indistinguishable peculiarly crafted non-robust directions. To alleviate these sensitivities several studies suggested techniques to cope with this problem via explicitly regulating the temporal difference loss for the worst-case sensitivity. In our study, we show that these worst-case regularization techniques come with a cost that intriguingly causes inconsistencies and overestimations in the state-action value functions. Furthermore, our results essentially demonstrate that vanilla trained deep reinforcement learning policies have more accurate and consistent estimates for the state-action values. We believe our results reveal foundational intrinsic properties of the adversarial training techniques and demonstrate the need to rethink the approach to robustness in deep reinforcement learning.

Under review as a conference paper at ICLR 2023
THE ADVERSARIAL REGULATION OF THE TEMPORAL DIFFERENCE Loss COSTS MORE THAN EXPECTED
Anonymous authors Paper under double-blind review
ABSTRACT
Deep reinforcement learning research has enabled reaching significant performance levels for sequential decision making in MDPs with highly complex observations and state dynamics with the aid of deep neural networks. However, this aid came with a cost that is inherent to deep neural networks which have increased sensitivi- ties towards indistinguishable peculiarly crafted non-robust directions. To alleviate these sensitivities several studies suggested techniques to cope with this problem via explicitly regulating the temporal difference loss for the worst-case sensitivity. In our study, we show that these worst-case case regularization techniques come with cost that intriguingly causes inconsistencies and overestimations in the state- action value functions. Furthermore, our results essentially demonstrate that vanilla trained deep reinforcement learning policies have more accurate and consistent estimates for the state-action values. We believe our results reveal foundational intrinsic properties of the adversarial training techniques and demonstrate the need to rethink the approach to robustness in deep reinforcement learning.
INTRODUCTION
Advancements in deep neural networks have recently proliferated leading to expansion in the domains where deep neural networks are utilized including image classification (Krizhevsky et al. 2012).
natural language processing (Sutskever et al. 2014). speech recognition (Hannun et al. (2014) and self learning systems exploration. In particular, deep reinforcement learning has become an emerging field with the introduction of deep neural networks function approximators (Mnih et al. 2015).
Hence, deep neural policies have been deployed in many different domains from pharmaceuticals to Noonan 2017).
self driving cars (Daochang & Jiang 2018 Huan-Hsin et al. 2017
As the advancements in deep neural networks continued line of research focused on their vulnera- bilities towards a certain type of specifically crafted perturbations computed via the cost function used train the neural network (Szegedy et al,, 2014, Goodfellow et al., 20155) Madry et al. 2018 Kurakin et al. 2016 Dong et al.) 2018). While some research focused producing optimal Ł norm bounded perturbations to cause the most possible damage to the deep neural network models, an extensive amount of work focused on making the networks robust such perturbations (Madry et al. 2018 Carmon et al. 2019: Raghunathan et al. 2020).
The vulnerability to such particularly optimized adversarial directions was inherited by deep neural policies as well (Huang et al., 20177] Kos & Song. 2017 Korkmaz, 2022). Thus, robustness to such perturbations in deep reinforcement learning became concern for the machine learning community, and several studies proposed various methods to increase robustness (Pinto et al.) 2017 Gleave et al., 2020). Thus, in this paper we focus on adversarially trained deep neural policies and the atateacction value function learned by these training methods in the presence of an adversary. In more detail, in this paper we aim to seek answers for the following questions: (i) How accurate the state-action value function on estimating the values for state-action pairs in MDPs with high dimensional state representations?. (ii) Does adversarial training affect the estimates of the state-action value function?, (iii) What are the effects of training with worst-case distributional shift on the state-action value function representation for the optimal actions? and (iv) Are there any fundamental trade-offs intrinsic explicit worst-case regularization in deep neural policy training? To be able to answer these questions we focus on adversarial training and robustness in deep neural policies and make the following contributions:

---

Under review as a conference paper at ICLR 2023
We conduct investigation on the state-action values learnt by the ially trained deep neural policies and vanilla trained deep neural policies.
adversar-
We provide theoretically motivated justification for how adversarial training might change the state-action value function.
We perform several experiments in Atari games with large state spaces from the Arcade Learning Environment (ALE). With our systematic analysis we show that vanilla trained deep neural policies have more accurate representation of the sub-optimal actions compared to the state-of-the art adversarially trained deep neural policies.
Furthermore, we show the inconsistencies the action ranking in the state-of-the- adversarially trained deep neural policies. Thus, these results demonstrate the loss of information in state-action value function as a novel fundamental trade-off intrinsic to adversarial training.
More importantly, we demonstrate that state-of-the-ard adversarially trained deep neural policies learn overestimated state-action value functions.
Finally, we explain how our results call into question the hypothesis initially proposed by Bellemare et al. (2016) relating the action gap and overestimation.
2 BACKGROUND AND PRELIMINARIES
Preliminaries: In deep reinforcement learning the goal is learn a policy for taking actions in Markov Decision Process (MDP) that maximize discounted expected cumulative reward. An MDP is represented by a tuple M (S, A, P. r, po: 7) where S is set of continuous states, A is discrete set of actions, P is a transition probability distribution on S x A x 5, $ x + R is a reward function, Po is the initial state distribution, and is the discount factor. The goal in reinforcement learning is to learn a policy π S + P(A) which maps states to probability distributions on actions in order to maximize the expected cumulative reward R E Eo 7'r(se, a) where a, ~ π (st). In Q-lcarning Watkins (1989) the goal is to learn the optimal state-action value function (s, R(s, a) Ex'es P(s" 8, a) Ea Q" (s',)a'). Thus, the optimal policy is determined by choosing the action a" (s) arg max, Q(s, in state
Adversarial Crafting and Training: Szegedy et al. (2014) observed that imperceptible perturbations could change the decision of a deep neural network and proposed box constrained optimization method to produce such perturbations, Goodffllow et al. (2015) suggested faster method to produce such perturbations based on the lincarization of the cost function used in training the network. Kurakin
et al. 2016) proposed the iterative version of the fast gradient sign method proposed by Goodfellow al.
2015 inside an e-ball.
.++1 = clip, + asign( V. J (d v)))
adv
in which J(x. y) represents the cost function used to train the deep neural network, x represents the input, and y represents the output labels. While several other methods have been proposed (e.g. Korkmaz (2020) using momentum-based extension of the iterative fast gradient sign method,
V Math ( + H 22. a)
Ut+1
p
| Mad J (8 + 1 Vt, |1
(+1 adv
sln +
C++1
1104-112
adversarial training has mostly been conducted with perturbations computed by projected gradient descent (PGD) proposed by Madry al. (2018) (i.e. Equation] I.
Adversaries and Training in Deep Neural Policies: The initial investigation on resilience of deep neural policies was conducted by Kos Song (2017) and Huang et al. (2017) concurrently based on the utilization of the fast gradient sign method proposed by Goodfellow et al. (2015). Korkmaz (2022) showed that deep reinforcement learning policies learn shared adversarial features across MDPs While several studies focused improving optimization techniques to compute optimal perturbations, a line of research focused on making deep neural policies resilient to these perturbations. Mandlekar et al. 2017) proposed including these perturbations in training time increase resilience for robotic
2

---

Under review as a conference paper at ICLR 2023
setups. Pinto et al. (2017) proposed to model the dynamics between the adversary and the deep neural policy as a sum game where the goal of the adversary is to minimize expected cumulative rewards of the deep neural policy. Gleave et al. (2020) approached this problem with adversary model which is restricted to take natural actions in the MDP instead of modifying the observations with fy-norm bounded perturbations. The authors model this dynamic
zero-sum Markov game and solve it via self play. Recently, Huan et al. (2020) proposed to model this interaction between the adversary and the deep neural policy as a state-adversarial MDP, and claimed that their proposed algorithm State Adversarial Double Deep Q-Network (SA-DDQN) learns theoretically certified robust policies against natural noise and perturbations. More recently, several empirical concerns have been raised on the robustness of theoretically certified adversarially trained deep neural policies Ezgi 2021). In our work, we systematically investigate and theoretically motivate the problems caused by adversarial training on the state-action value function learned by deep neural policies.
3 ADVERSARIAL TRAINING AND THE STATE-ACTION VALUE FUNCTION
In this paper we aim to answer the following questions:
How does training with explicit worst-case regularization affect the estimates of the optimal state-action values in MDPs with high dinensional state representations?
What is the accuracy of the state-action value function representation for the non-optimal actions in deep neural policies?
Does state-of-the-ar adversarial training affect the state-action value estimates?
. Are there any intrinsic trade-offs tied to adversarial deep neural policy training?
While the goal in Q-learning is to learn the state-action value function Q(s, a) that maximizes expected discounted cumulative rewards, in deep Q-learning additional concern arises from susceptibility towards adversarial perturbations due to the nonlinear function approximator used in learning the Q-function. Ideally, one might hope that adversarial training would reduce the vulnerability of the Q-function to adversarial perturbations while preserving the Q- Q-values of the non-perturbed states as much possible. The theoretically motivated adversarial training techniques achieve certified defense against adversarial perturbations inside the e-ball (s = {3 || ãll < c). However, we show that this approach induces significant changes in the Q-function so that the Q-function loses its accuracy for the non- con-perturbed states. In particular, adversarial training causes deep neural policies to learn overestimated state-action values, and the Q-values for non-optimal actions are reduced in accuracy to the point where their relative ranking changes.
In the remainder of this section we give theoretical motivation for these empirical results. In particular we demonstrate that in the setting of linear function approximation, adversarial training can potentially lead to overestimation for the Q-values of the optimal actions, and reordering of the ranking of non- optimal actions. The basic approach of adversarial training techniques is based on adding regularizer to the standard Q-learning update. The regularizer is designed to penalize Q-functions for which a perturbed state (s) can change the identity of the highest Q-value action. For the baseline adversarial training technique we will theoretically analyze the effects of this regularizer.
(s) arg max, Q(s, a). The regularizer is Definition 3.1 (Huan et al. (2020)). For state let a given by
R(6)= = Σ
max max Q0(3,a) a) - Q0(3, (8))
(e).(s) agfa (x)
The adversarial training algorithm proceeds by adding R(0) to the standard temporal difference loss
((0) L (r+ + 7 max Quarget (8,aa)
+ R()))
(4)
used in DQN and minimizing via stochastic gradient descent.
We now describe the construction of an MDP M with linear function approximation where the of the regularizer causes overestimation and reordering of suboptimal actions. There two states parametrized by feature vectors $1, $2 E R", and there are three possible actions {a,) 13
in each state. Taking any of the three actions in state 81 leads to transition to state 82 and vice
3

---

Under review as a conference paper at ICLR 2023
versa. Let 1 > 7 > 0 be the discount factor, and let 8 > 17 > 0 be small constants with y > 8. The
rewards for each action are as follows: r(81, a1) = 1 - 7, r($1, Q2) = 17 7 r($1, 03) = 5 %
r(32, 01) 17 - 7. r($2, 02) 1 7, and r(82, ag) 8 - Y Clearly, the optimal policy is to always take action d1 in state 81, and action d2 in state 82 as these are the only actions giving positive reward.
Thus the optimal state-action values are given by: Q" (81, a1) = (82 (82,02) -i=o(1 (1-)) = 1, 1) and (51 (3)))
(* 0z) = 8 8-7+ + 7 2 1t= - 7 = 6. Let the Q-function be linearly parametrized by 0 = (01, 02, 03) so that Co(s, n;) (0,, 8). Finally, let 2, for E 21, be three orthonormal vectors, and let the state feature vectors satisfy:
(51 02) (*( a1) " - 7
Ei=o(1
?)>
1. $1 = 21 + 62g + 1722 and 82 = 22 - 823 + 7721
Then it follows that the optimal Q-function is parametrized by 0" (0% 02 03) where 0 2; i.e. Q (8, a) = Q* a) for all and a. Thus, according to the function Qa- (s, a), for 81 the best action is a1, for 82 the best action is dg and in all states the second-best action is 0g. Next we identify the optimal perturbations used in the computation of the regularizer R(0") ) for this setting.
Proposition 3.1. In the MDP M suppose that <
6-1
1. For = 81: 8 + V2 (0z
01) arg max (e) max (s) Q0- (3, a) Q (3, a (s))
2. For 8 = 82: 8 + f 00z
0%) arg e. (s) maxafa" (s) Qa- (3, a) Qu- (8, a (8))
Proof. We will prove item 1, and item 2 will follow from an identical argument with roles of 0 and 05 swapped. Let s = $1. Any 3 E (s) can be written as s + eu where v is a unit vector. Thus, (05, 8) = (0%, 8) + + <(07, u) > (0z, 8) e = 6 - 6. Similarly we have (02, 8) < (02 s) + 6 = r + e. Since e < " we conclude that (0z, 8) > (0ž, 8) for all & E D.(s). Therefore, in state s the action maximizing maxafaa(s) Qo- (5,a) Qo (5, (s)) will always be dg. This implies that
arg max max Q0- (3, a) Q+ (5, (8,aa(s)) (s)) = arg mmxx(xx) 8) ().,
(5)
sED.(s) as)
(ED.(a)
This is the maximum ball of radius e around s of the linear function (0g - 07,8). 8). Therefore the maximum is achieved by 3 = s + S/ (03 0 desired.
In words, the optimal direction to perturb the state 81 in order to have (s(s) * (3) is toward 03 07- Similarly for the state 82, the optimal perturbation is toward 03 -05. Next use this fact to show that in order to decrease the regularizer it is sufficient to simply increase the magnitude of 0 and 02, and decrease the magnitude of 03.
Proposition 3.2. In the MDP M let À > O and suppose that e < (1-A)6-(1+M1 Let (01, 02, 0g) be given by 0₁ = (1 + x)81. 0₂ = (1 + X)0 and 0g = (1 -X)03. Then R(0) < R(0").
Proof. By an identical argument to that in Proposition 3.1 we have that 0g is always the action maximizing maxatto (+) Q0(8, a) Qo(3, (s)) whenever < (1-A)6-(1+A) This condition is satisfied by assumption. Therefore, we conclude that for s = $1, the optimal 8 E (s(s) for the scaled parameters 0 is given by
2
(0g- 01). Therefore, the contribution to the sum defining (2(++x))
R0) from state 81 is given by
((gg 01), 8) = ((0s 01), s) + ev x (1 (1 +1) X)ō + 2 X2)
(6)
where the last step uses the fact that = 8 + 50% + +185 and that the vectors 07 are orthonorml.. Next using the fact that v1 X < + for all X >> 0 we conclude
((3--0)) - 0,), < -1 + (1 2)ō + ev + cXV2 < - -(1 + +vv2.
The final inequality follows from the fact that c < an identical proof (with 01 replaced by 62) yields the same value for the contribution of type 2 actions to the sum. By Proposition 3.1 the contribution of each type of state to the sum defining R(0") is
6 so c\v2 - A5 < 0. Switching to type 2 actions
2
(10) 01), 8 + V2 01) = -1 1 + 8 +ev2.
(8)
Clearly the contribution of each state in 7 is strictly less than that in Therefore R(0)) < R(00).
n
4

---

Under review as a conference paper at ICLR 2023
Theorem 3.3. Ifa linear state-action value function approximator Qo(s, a) is used, then the regular- izer R(Ø) can lead to overestimation of the value of the optimal action, and re- re-ordering of the values of the suboptimal actions.
Proof. Consider the MDP M with linear function approximation constructed above. Increasing the magnitude of 0 and 0 by factor of 1 + À leads to overestimation of the Q-value of the best action in both state $1 and #2 by the same factor. Additionally decreasing the magnitude of 03 can lead to a change in the ranking of the suboptimal actions. Indeed if 1=1 > á then d3 will become the third ranked action in both states. Therefore, Proposition 3.2 proves that changing 0 to decrease the regularizer R0) can lead both overestimation of the first ranked action, and re-ordering of the ranking of the suboptimal actions.
While we showed how this can potentially happen in the case of linear function approximation, will see that this is a general phenomenon which occurs with neural-network approximation of the Q-function in adversarially trained agents. It is important note that the issues we identify are result of the fundamental differences between deep neural policies and classification tasks where adversarial training has previously been applied. In particular. the fact that the state-action value function (s,, a) has a meaning (i.e. measuring expected cumulative rewards) with regard to the MDP beyond simply labelling the optimal action correctly is the root cause of the effects that we observe. In other words, simply penalizing the state-action value function for assigning the wrong "label" to an adversarial example can have unintended, potentially detrimental consequences for learning an accurate state-action value function.
MEASURING THE ACCURACY OF STATE-ACTION VALUES
In this section we provide a methodology to measure the accuracy of the state-action value function in representing values for the non-optimal actions. At high-level, our approach is based on action modification and the relative performance drop P as defined below:
Definition 4.1. The performance drop of an agent when modifying the agent's 's actions given by
Scorebase Score
P
Score Scoremin
where Score represent the baseline run of the game with no action modification, Scoremio represents the minimum score available for a given game, and Score represents the run of the game where the actions of the agent are modified for fraction of the state observations.
We now explain precisely how we propose to measure "accuracy" for non-optimal actions. Formally, let a; the :hh best action decided by the deep neural policy in a given state 8 (i.e. Q(s, a) is sorted in decreasing order, and a; is the action corresponding to 7th largest Q-value). For trained agent, the value of Q(s, a;) should represent the expected cumulative rewards obtained by taking action a, in state s, and then taking the highest Q-value action (i.e. a1) in every subsequent state. Thus, natural test to perform would be: pick random state 8, make the agent choose action a; in state 8, and in all other states have the agent choose the highest Q-value action. By comparing the relative performance drop P in this test to a clean run where the agent always takes the highest Q-value action, one can measure the decline in rewards caused by taking action ag Further, we can provide a measure of accuracy for the state-action value function by comparing the results of the test for each i e {1. 2..... [A]] and checking that the relative performance drops P. are in the correct order i.e. P= < P2 ... < P(AII
However, there is an issue with the above proposal. is often the case that there are many states s in which the action taken has very little impact on the final rewards. Instead, there are relatively smaller number of critical states which the action taken has a large impact. Thus, picking a single random state in which to take action a, will have a statistically insignificant impact on the final rewards in the game. Therefore we modify the test described above by instead sampling a p-fraction of the states in the episode uniformly at random, and making the agent take action a; in each of the sampled states. We then record the relative performance drop as a function of p. yielding a reward curve P. (p). More formally, we define
5

---

Under review as a conference paper at ICLR 2023
14
20.
90. 20:3
10.6
20:0
200
22.3
660
Figure 1: Performance drop P2(p) with respect to action modification Owr for the state-of-the-ee adversarially trained deep neural policies and vanilla trained deep neural policies. Left: BankHeist. Center: RoadRunner. Right: Freeway.
Definition 4.2. Let M be an MDP and (ss, a) be state-action value function for M. In each state label the actions 01.... 4| in order so that Q (1) 2 Q(s, (2) ... 2 Q(s, (As.) We define the performance curve (p) to be the expected performance drop of an agent in M which takes action
a, in a randomly sampled p-fraction of states, and takes action aj in all other states.
Using these reward curves one can check whether Pp(p) lies above (p) whenever i j. Of course one curve may not always lie strictly above or below another, so we introduce the following definition quantitatively capture the relative ordering of performance drop curves.
Definition 4.3. Let F [0 1] and G [0, 1] [0, 1]. For any r O, we say that the F r-dominates if So (F(p) - G(p)) dp > T.
To compare the accuracy of state-action values for vanilla versus adversarially trained agents, we can thus perform the above test, and check the relative ordering of the curves (p) using Definition 4.3 for each agent type. In addition, we can also directly compare for each the curve Pi (p) for the adversarially trained agent with the curve pyanilla (p) for the vanilla trained agent. This is possible because (p) measures the performance drop of the agent relative to a clean run, and so always takes values on a normalized scale from 1. Thus, if we observe for example that Pa (p) r-dominates P2 Pramilla (p) for some r >> O, agent more accurately represents the second-best action than that of the adversarially trained agent.
conclude that the state-action value function of the vanilla trained
5 EXPERIMENTAL DETAILS
The experiments are conducted in high dimensional state representation MDPs. In particular, our experiments are conducted in the Arcade Learning Environment (ALE) (Bellemare et al. 2013) in the OpenAl (Brockman al., 2016) baseline version. The vanilla trained deep neural policy trained via Double Deep Q-Network (DDQN) (Wang et al. 2016) initially proposed by Hasselt et al. (2016) with prioritized experience replay proposed by (Schaul al. 2016), and the state-of-the-ait adversarially trained deep neural policy is trained via State- Adversaraa Double Deep Q-Network (SA-DDQN) (Section (2 with prioritized experience replay (Schaul et al. 2016). The results are averaged over 10 episodes. We explain in detail all the necessary hyperparameters for the implementation in the supplementary material. The standard error of the mean is included for all of the figures and tables. Note that in the main body of the paper we focus on the baseline adversarial training. In the supplementary material we also provide analysis on the follow-up more recent studies adversarial training techniques. The results reported for all of the adversarial training techniques remains the same that the adversarially trained policies learn inaccurate, inconsistent and overestimated state-action values.
AN ANALYSIS ON THE STATE-ACTION VALUE FUNCTION REPRESENTATION
In this section we demonstrate that the state-action value function of adversarially trained deep neural policies provides inaccurate estimates for the non-optimal actions, and learns overestimated state-action values. This confirms that the theoretically-motivee 3 do indeed occur in practice for deep neural policies. In particular, non non-optimal actions we use the methodology discussed in Section 4 of measuring the performance drop (p) that occurs when causing the deep neural policy to take the i-th best action in p fraction
problems discussed in Section
evaluate the accuracy

---

Under review as a conference paper at ICLR 2023
10.3
10.3
10.5
136 12.0 20 254
20.
212
2.3
0.. 34.3
Figure 2: Performance drop P (p) with respect to action modification aw for the state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies.
Table 1: Area under the curve of performance drop under action modification (AM) dg and aw for the state-of sfae-off-hhe-rr adversarially trained deep neural policies and vanilla trained deep neural policies.
Fnvironments
BankHeist
RoadRunner
Freeway
Training Method
Adversarial
Vanilla
Adversarial
Vanilla
Adversarial
Vanilla
AM 0g
0.449+0.007
0.191+0.04
0.414+0.015
0,247+00009
0.351+0.009
0.302+0.000
AM Og
0.311 0.011
0.398+0.011
0.345+0011
0.393+00009
0,241 +0.007
0.311++0011
of states. Our aim is to provide an analysis how accurate the state-action value function is in representing values for both optimal and non-optimal actions for vanilla trained deep neural policies and state-of-the-art adversarially trained deep neural policies.
6.1 INACCURACY OF STATE-ACTION VALUES FOR NON-OPTIMAL ACTIONS
In Figure D we show the performance drop P2( as a function of the fraction of states p in which the action modification is applied for otate-of-the-ar adversarially trained deep neural policies and vanilla trained deep neural policies. In particular, the action modification is set for the second best action d2 decided by the state-action value function ((s, a). As we increase the fraction of states in which the action modification set to is applied, we observe a performance drop for both of the deep neural policies. However, we observe that the vanilla trained deep neural policies experience a lower performance drop with this modification. Especially in BankHeist we observe that the performance drop does not exceed 0.55 even when the action modification is applied for a large fraction of the visited states for the vanilla trained deep neural policies. This gap in the performance drop between the adversarially trained and vanilla trained deep neural policies indicates that the state-action value function learnt by vanilla trained deep neural policies has a better estimate for the non-optimal actions. As we measured the impact of 02 modification on the policy performance, we further test aw arg min, ((s, a) modification (i.e. worst possible action in given state modification) on the deep neural policy. Figure 2 shows that the performance drop (p) is higher in the vanilla trained deep neural policies compared to adversarially trained deep neural policies when the action modification is set to aw- This again further demonstrates that the state-action value function learnt by the vanilla trained deep neural policy has more accurate representation over the non-optimal actions. We hypothesize that adversarial training places higher emphasis on ensuring that the highest ranked action (i.e. the action that maximizes the state-action value function in a given state) does not change under small l,- norm bounded perturbations, rather than accurately computing the state-action value function as discussed in Section 3 Since historically Q-learning suffered from overestimation of Q Q-values, a method which places higher emphasis on the highest ranked action risks converging to a state-action value function with overestimated Q-values. We further demonstrate this in Section 6.2
6.2 OVERESTIMATION OF Q- Q-VALUES IN ADVERSARIALLY TRAINED DEEP NEURAL POLICIES
Overestimation of Q-values was initially discussed by Thrun & Schwartz (993) as byproduct of the use of function approximators, and was subsequently explained as being caused by the use of the max operator in Q-learning (van Hasselt 2010)). Furthermore, overestimation bias resulting in learning of sub-optimal policies was demonstrated in practice by Hasselt et al. (2116). In this subsection we empirically demonstrate that state-ot of-the-art adversarial training indeed leads overestimation in Q-values, as hypothesized in Section 3 In particular, Figure and Table 2 show the overestimation bias on the state-action values learned by the state-of-the-ant adversarially trained deep neural policies.
7

---

Under review as a conference paper at ICLR 2023
2001 100.
640 1000
1321 12300
0.0 2900
190.
3500
Figure 4: Q-value of the best action a" over the states for the state-of-the-an adversarially trained deep neural policy and vanilla trained deep neural policy. Table 2: Average Q-values of the optimal action in state-of-ther tfrt---rr--e--ttteeeeee-- adversarially trained deep neural policies and vanilla trained deep neural policies.
Environments
BankHeist
RoadRunner
Freeway
Training Method
Adversarial
Vanilla
Adversarial
Vanilla
Adversarial
Vanilla
(ssaa))
5.903+2.052
0.300=10434
8.806
0.602=0.781
1.667 0.406
1.185
Considering that overestimation bias is still an issue and active area of research for vanilla deep neural policy training (Lan et al., [2020] Anschel et al.) [017]] Kuznetsov et al. (2020), the additional bias introduced intrinsic to adversarial training must be addressed be able to learn optimal policies.
6.3 INCONSISTENCIES IN ACTION RANKING IN ADVERSARIALLY TRAINED DEEP NEURAL POLICIES
In this subsection we demonstrate the inconsistencies in the non- optimal action ranking in adversarially trained policies. In par- ticular, in Figure 3 in BankHeist choosing the worst action leads to a smaller performance drop than choosing the second best ac- tion i.e. P (p) < (2(p) for all p. Thus, this demonstrates that the state-action value function not ranking the sub-optimal ac- tions accurately. While learning an accurate representation of the state-action values is important for obtaining a policy that aims to maximize expected cumulative rewards, learning the correct order of the actions can also solve this problem. Furthermore, in some cases the deep neural policy indeed must know the correct order of the actions due to the presence of an obstruction that blocks the optimal action either due to the existence of other agents or environmental effects (Rashid et al., 2020; Gleave al., 2020). In particular, in safe reinforcement learning several algorithms have been proposed to learn the ranking of the actions so that the agent can choose the next-best ranked action in safety critical situations (Alshiekh et al. 2018)) Some work has also pointed out that in some cases learning the relative rank of the actions (Lin & Zhou, 2020) can be more sample efficient than learning correct estimates of the state-action values. While the inconsistency in action ranking for adversarially trained deep neural policies can be seen as a vulnerability problem from security point of view, most intriguingly these results demonstrate the loss of information in state-action value function as a novel fundamental trade-off intrinsic to adversarial training.
10-
0.
0.0
500
20.0
30.0
40.0
500
800
Figure 3: P2 and Pu for ad- versarially trained deep neural policies.
6.4 ACTION GAP PHENOMENON
The action gap defined as the difference between the state-action value of the optimal action and the state-action value of the second ranked action.
K( (1, s)
max Q(s,a')-
max
Q(s.a) a)
(10)
a'E.A
agarg max a Q(s,a')
Initially, Farahmand 10011 describes the existence of a large action gap desirable property of an MDP, which makes learning an optimal policy easier. Subsequently, Bellemare et al. (2066) proposed connection between the action gap and the overestimation of Q-values, and in particular hypothesized that increasing the action gap of the learned value function causes a decrease in overestimation of Q-values. Following this study, several papers built on the hypothesis that increasing
8

---

Under review as a conference paper at ICLR 2023
Byanddy
y013
M
g0:0
615
012
10.
2018
3014
Teansy
4.23
414
Figure 5: Normalized state-action values for the best action a², second best action 02 and worst action &w over states, Rowl: Vanilla trained policies. Row2: State-of-the-art adversarially trained policies.
Table 3: Normalized state-action value Sttimates_and state-action value estimate shift for the second best action in state-of-the-arr adversarially trained deep neural policies.
(s..a)
Q(x,a")
Q(s, az)
Q(s, aw)
ALE
Adversarial
Vanilla
Adversarial
Vanilla
Adversarial
Vanilla
BankHeist
0.18941:0000
0.170=:0.003
0.130::0.0006
0.169=0.002
0.127=0.0010
0.161=:00004
RoadRunner
0.1696==0008
0.236::0.094
0.132+000006
0.159=0.079
0.126=0.0049
-0.265=00007
Freeway
0.1894+00.00
0.341+0.008
0.130+0.0006
0.333=00002
0.127+0.0010
0.325++0.009
the action gap causes reduction in bias (Smirnova Dohmatob 2020 Fox al., 2016 Jain et al.. 2020 [u et al. 2019). In Figure 5 we show that adversarial training increases the action gap. Thus, the fact that adversarially trained deep neural policies overestimate the optimal state-action values (see Section 6.2) refutes the hypothesis that increasing the action gap is the sole cause of decrease in overestimation bias of state-action values. We hypothesize that the consistent Bellman operator (Bellemare al. 2016) may cause decrease in overestimation for different reason. In particular, the consistent Bellman operator corresponds to a special case of a certain reparameterization of Kullback-Leible regularization for value iteration (Vieillard et al.) 2020). Thus, it may be the case that the decrease in overestimation of Q-values and improvement in performance is due to a type of implicit regularization rather than to an increase of the action gap. Hence, our results show that increasing the action gap alone may coincide with an increase in overestimation of Q-values.
7 CONCLUSION
In this paper we focus on the state-action value function learnt via the state-of-the-ar adversarially trained deep neural policies and vanilla trained deep neural policies. We provide theoretical analysis on the fundamental effects caused by adversarial training on the state-action value function. Furthermore, we conduct manifold experiments in the Arcade Learning Environment and with our systematic analysis we demonstrate that vanilla trained deep neural policies have more accurate and consistent estimates for the state-action values than the state-of oh-the-art adversarially trained deep neural policies. More intriguingly, we show that adversarially trained deep neural policies in certain MDPs completely loses the information the state-action value function that contains the relative ranking of the actions. More importantly, we show that state-of-the-an adversarially trained deep neural policies learn overestimated state-action values. We believe our investigation lays out intrinsic properties of adversarial training while systematically revealing the underlying vulnerabilities, and can be conducive to building robust and optimal deep neural policies.
Note that due to the fact that the adversarially trained deep neural policy overestimates Q-values, we introduce a normalization in order to compare the action gaps of adversarially and vanilla trained policies. In particular, in Figure 5 we report normalized Q-values in each state s by dividing Q(+,a) by Σ. (Q)

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topeu. Safe reinforcement learning via shielding. In Sheila A. McIlraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2669-2678. AAAI Press, 2018.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. International Conference Machine Learning (ICML), 2017.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael. Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research. pp. 253-279, 2013.
Mare G. Bellemare, Georg Ostrovski, Arthur Guez, Philip S. Thomas, and Rémi Munos. Increasing the action gap: New operators for reinforcement learning. In Dale Schuurmans and Michael P. Wellman (eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp. 1476-1483, AAAI Press, 2016.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. 166.0 2016
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, and Percy Liang. Unlabeled data improves adversarial robustness. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11190-11201, 2019.
Liu Daochang and Tingting. Jiang. Deep reinforcement learning for surgical gesture segmentation and classification. In International conference on medical image computing and compuler-asisttd intervention., pp. 247-255. 25-rinngrrrr Cham, 2018.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 91859985999911 91539999,2222 2018.
Korkmaz Ezgi. Investigating vulnerabilities of deep neural policies. Conference on Uncertainty in Artificial Intelligence (UAI), 2021.
Amir Massoud Farahmand. Action-gap phenomenon in reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2011.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. Conference on Uncertainty in Artificial Intelligence (UAI), 2016.
Adam Gleave, Michael Dennis, Cody Wild, Kant Neel, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. International Conference on Learning Representations ICLR, 2020.
Ian Goodfellow, Jonathan Shelens, and Christian Szegedy. Explaning and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Diamos Greg, Erich Else, Ryan Prenger, Sanjeev Satheesh, Sengupta Shubho, Ada Coates, and Andrew Ng. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567 2014.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Association for the Advancement of Artificial Intelligence (AAAI), 2016.
Zhang Huan, Chen Hongge. Xiao Chaowei, Bo Li, Mingyan Boning, Duane Liu, and ChoJui Hsiesh. Robust deep reinforcement learning against adversarial perturbations on state observatons. NeurIPS Spotlight Presentation, 2020.
10

---

Under review as a conference paper at ICLR 2023
Sandy Huang, Nicholas Papernot, Yan Goodfellow. lan an Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. Workshop Track of the 5th International Conference on Learning Representations, 2017.
Tseng Huan-Hsin. Sunan Cui, Yi Luo, Jen- Jen-Tzung Chien, Randall K. Ten Haken, and Issam El. Naqa. Deep reinforcement learning for automated radiation adaptation in lung cancer. Medical physics 44, pp. 6690-6705, 2017.
Vishal Jain, William Fedus, Hugo Larochelle, Doina Precup, and Mare G. Bellemare. Algorithmic improvements for deep reinforcement learning applied to interactive fiction. Association for the Advancement of Artificial Intelligence (AAAT), 2020.
Ezgi Korkmaz. Nesterov momentum adversarial perturbations in the deep reinforcement learning domain. International Conference on Machine Learning, ICML 2020, Inductive Biases, Invariances and Generulization in Reinforcement Learning Workshop., 2020.
Ezgi Korkmaz. Deep reinforcement learning policies learn shared adversarial features across mdps. AAAI Conference on Artificial Intelligence, 2022.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. International Conference on Learning Representations, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 2012.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 0025333,, 2016.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry P. Vetrov. Controlling overes- timation bias with truncated mixture of continuous distributional quantile critics. International Conference Machine Learning (ICML), 2020.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-leaming: Controlling the estimation bias of q-learning. International Conference on Learning Representations (ICLR), 2020.
Kaixiang Lin and Jiayu Zhou, Ranking policy gradient. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Yingdong Lu, Mark S. Squillante, and Chai Wah Wu. A family of robust stochastic operators for reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alcéé-Buc, Emily B. Fox, and Roman Garnett (eds.). Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp. 15626-15636, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausibe perturbations. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, are G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra. Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518: 529-533 2015.
Laura Noonan. Jpmorgan develops robot to execute trades. Financial Times, pp. 1928-19371 July 2017.
11

---

Under review as a conference paper at ICLR 2023
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. International Conference on Learning Representations ICLR, 2017.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C. Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICML 2020, 13-18 Juy 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7919. PMLR, 2020.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Tom Schaul, John Quan, loannis Antonogloua, and David Silver. Prioritized experience replay. International Conference on Learning Representations (ICLR), 2016.
Elena Smirnova and Elvis Dohmatob. On the convergence of smooth regularized approximate value iteration schemes. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
lly Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dimutru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. Fourth Connectionist Models Summer School, 1993.
Hado van Hasselt. Double q-learning. In John D. Lafferty, Christopher K. I. Williams, John Shawe- Taylor, Richard S. Zemel, and Aron Culotta (eds.), Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pp. 2613-2621. Curran Associates, Inc., 2010.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of KI. regularization reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tienn-TTennnnn Lin (eds.), Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando. De Freitas. Dueling network architectures for deep reinforcement learnigg. Internation Conference on Machine Learning ICML., pp. 1995-2003, 2016.
Chris Watkins, Learning from delayed rewards. In PhD thesis. Cambridge. King's College, 1989.
12


Reviews:
[{"review_id": "nhFtSuD9aM", "reviewer": "Reviewer_2Ms4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effect of adversarial training on the action-value estimates of value-based deep RL agents. The paper considers both a toy learning problem where the optimal function approximator with and without adversarial regularization can be computed analytically, as well as networks trained on the arcade learning environment. It shows that adversarial training reduces the accuracy of the network’s ranking over actions at a given environment state, that it increases the gap between the action with the highest predicted value and the value of the second-best action, and that it increases the value of the action with the highest predicted value.\n", "summary_of_the_review": "This paper highlights an important tradeoff in adversarial training of value-based deep RL agents: that optimizing for the robustness of the greedy policy may have unintended effects on the sub-optimal actions. The paper suffers from two principal weaknesses: firstly, that prior work has already identified that the relative ranking of actions in DQN-like agents is unstable, and so the findings here seem to be a natural consequence of that observation coupled with the focus of adversarial regularization on only the greedy action. Secondly, the regularization scheme under consideration differs from that studied in prior work in a way that would exacerbate the effects observed in the paper, and lack of detail on the experiment setup description makes it difficult to identify whether the experimental findings would reflect practical implementations of state-of-the-art adversarial training schemes for RL.", "strength_and_weaknesses": "Strengths:\n\n- The paper offers some interesting insights on the trade-offs between adversarial robustness and accuracy of the action-value function in value-based deep RL.\n- The toy example in Proposition 3.2 is a nice illustration of the instability of action rankings in adversarially robust agents.\n- The discussion of the adversarial robustness literature seems reasonably comprehensive (though this isn’t my primary field).\n- The metric used to estimate action-value accuracy is a smart way of trying to marginalize out noise in the agent trajectories in order to compare the action rankings.\n- The finding of section 3 is somewhat surprising — I would not have anticipated such a large gap between the actions of the second-best and worst predicted action.\n\n\nWeaknesses\n\n- While the literature review discusses the adversarial RL literature, it does not cover many recent works studying the robustness of the action-value ranking of deep RL agents to non-adversarial perturbations. In particular, prior work [1] has shown that even gradient steps computed on disjoint subsets of the replay buffer can change the agent’s ranking over actions in a surprisingly large fraction of the state space. In the context of this prior work, the observation that the ranking of sub-optimal actions changes under adversarial regularization is unsurprising.\n- It is widely observed that DQN outputs tend to systematically underestimate the true value of a state. As a result, it seems to me difficult to say whether the values output by the adversarially trained networks are indeed over-estimations of the true value, or whether they are simply accurate estimations of the true action-value function. I am therefore unconvinced as to whether the subsection’s title is necessarily accurate, and would like to see a comparison between the action-values and the true discounted return obtained by the policy to be convinced of this.\n- The description of the adversarial regularizer is inconsistent with that described by Zhang et al. in their presentation of SA-DQN. Zhang et al. include a clipping hyperparameter $c$ which lower bounds the adversarial action gap and avoids the pathology of the value estimate of the optimal action growing arbitrarily large. I am curious about the value of $c$ used in this paper.\n- Similarly, the value of the regularizer coefficient $\\kappa$ is only briefly mentioned in the final  sentence of the supplementary material, and it is said to be in a set of values. However, I could not determine which value was used for the paper’s experiments, nor could I find any results describing the relationship between the phenomena described in the main body and the values of $\\kappa$ and $c$. The epsilon value used for epsilon-greedy exploration is quite low (0.02). These values will likely play a large role in determining effect size of the phenomena described in this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bkr8q9Ezob", "reviewer": "Reviewer_JcRE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies a regularization that can improve the learning of Q-values when used in combination with deep learning. The paper shows that these regularization techniques can cause inconsistencies and overestimations in the state-action value functions while vanilla trained deep reinforcement learning policies have more accurate estimates for the Q-values.", "summary_of_the_review": "The paper looks into an interesting problem and has initial good contributions. There are however elements that need to be improved both on the form and on the substance.", "strength_and_weaknesses": "Strengths:\n- This is an important area of study\n- The paper highlights some problems that were not fully investigated in the past\n\nWeaknesses\n- The contributions are not very strong in the current state of the paper (see below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3H0LRkWisdJ", "reviewer": "Reviewer_AQqn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates that adversarially trained deep neural policies can potentially lead to inconsistencies and overestimations in the state-action value functions. A linear case theoretical analysis and empirical studies on Arcade Learning Environment are provided to verify the observation. ", "summary_of_the_review": "Overall, the depth of this contribution may not quite yet reach the high bar set for ICLR. The main claim, that vanilla trained deep neural policies have more accurate and consistent estimates for the state-action values than the state-of-the-art adversarially trained deep neural policies, needs more investigations. \n\nThe paper defines a performance drop metric as a measure of the accuracy of state-action values. \nThe authors pick a random state $s$ during a rollout, make the agent choose action $a_i$ in state $s$, and in all other states have the agent choose the best Q-value actions. The performance drop due to the action modification is measured. The authors claim that adversarially training would make the performance drop large and hence is not accurate. \n\nHowever, all the experiments are done on discrete action spaces, and I would expect there would be a big change in performance due to action modifications, for example, turning right in a maze changed to turning left would impact a lot for finding the exit. I think the performance drop due to action modification could be reasonably large and large performance drop actually shows that the Q-value function could discriminate good actions from bad actions well, which is the target of adversarial learning. \n\nI also kind of disagree with the overestimation claim. Table 2 doesn’t show the true average Q-values, so that we don’t know which one is overestimated. To me, the value scale (<10) is still under control. The target of Q-value function is to help the learning of policies, so as long as the good action regions are assigned higher and reasonable Q values, Q function could provide good learning signals for the policy. \n\nImplementation details are not discussed, and the work is not reproducible. The paper only mentions using state-of-the-art adversarially trained deep neural policies, while not includes the actually used algorithm. \n", "strength_and_weaknesses": "Strengths: \n1. The observation, that vanilla trained deep neural policies have more accurate and consistent estimates for the state-action values than the state-of-the-art adversarially trained deep neural policies, is counterfactual and interesting. \n\nWeaknesses:\n1. The theoretical analysis doesn’t support much about the empirical study. The analysis assumed that action-value function is linear and as simple as an inner product, which can not be extended to the cases of empirical studies. \n2. Lack of contribution. Demonstrating the problem without providing potential solutions may not quite yet reach the high bar set for ICLR. \n3. The main claim needs more investigation or explanations.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject