Don't Throw Your Old Policies Away: Knowledge-based Policy Recycling Protects Against Adversarial Attacks
Authors: Yaqi Xie
Chen Yu
Harold Soh
Venue: Submitted to ICLR 2023 (2023)
Keywords: Domain Knowledge
Knowledge Representation
Representation Learning
Policy Ensemble

Abstract: Recent work has shown that Deep Reinforcement Learning (DRL) is vulnerable to adversarial attacks, in which minor perturbations of input signals cause agents to behave inappropriately and unexpectedly. Humans, on the other hand, appear robust to these particular sorts of input variations. We posit that this part of robustness stems from accumulated knowledge about the world.
In this work, we propose to leverage prior knowledge to defend against adversarial attacks in RL settings using a framework we call Knowledge-based Policy Recycling (KPR). Different from previous defense methods such as adversarial training and robust learning, KPR incorporates domain knowledge over a set of auxiliary tasks policies and learns relations among them from interactions with the environment via a Graph Neural Network (GNN). KPR can use any relevant policy as an auxiliary policy and, importantly, does not assume access or information regarding the adversarial attack. Empirically, KPR results in policies that are more robust to various adversarial attacks in Atari games and a simulated Robot Foodcourt environment. 

Under review as a conference paper at ICLR 2023
DON'T THROW YOUR OLD POLICIES AWAY:
ANOWLEDGE-BASED POLICY RECYCLING PROTECTS AGAINST ADVERSARIAL ATTACKS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has shown that Deep Reinforcement Learning (DRL) is vulnerable to adversarial attacks, in which minor perturbations of input signals cause agents behave inappropriately and unexpectedly. Humans, on the other hand, appear robust to these particular sorts of input variations. We posit that this part of robustness stems from accumulated knowledge about the world. In this work, we propose leverage prior knowledge to defend against adversarial attacks in RL settings using a framework we call Knowledge-band based Policy Recycling (KPR). Different from previous defense methods such adversarial training and robust learning, KPR incorporates domain knowledge over set of auxiliary tasks policies and learns relations among them from interactions with the environment via a Graph Neural Network (GNN). KPR can use any relevant policy as an auxiliary policy and, importantly, does not assume access information regarding the adversarial attack. Empirically, KPR results in policies that are more robust to various adversarial attacks in Atari games and simulated Robot Foodcourt environment.
INTRODUCTION
Despite significant performance breakthroughs in recent years Mnih et al. 2015 Silver et al. 2016 Berner et al.] 2019 e.g.)) Deep Reinforcement Learning (DRL) policies can be brittle. Specifically, recent works have shown that DRL, policies are vulnerable to adversarial attacks adversarially manipulated inputs (e.g., images) of small magnitude can cause RI, agents to take incorrect ac-
tions (Ilahi et al.] 2022 Chen et al. 2019 Behzadan & Munir 2017 Oikarinen et al. 2021 Lee
ctt al. 2021 Chan et al. 2020 Bai et al. 2018). To counter such attacks, recent work has proposed a range of defense strategies including adversarial training (Oikarinen et al. 2021 Behzadan & Munir
2018 Han et al. 2018), robust learning (Mandlekar et 2017; Smirnova et 2019 Pan et al.
2019), defensive distillation (Rusu al. 2016), and adversarial detection (Gallego al. 2019a
Havens et al. 2018 Gallego et al.] 201 While these defense methods can be effective, each has its limitations; adversarial training and adversarial detection require specifie knowledge about the attacker. Robust learning adds noise during agent training, which can degrade performance (Tsipras) al. 2019: Yang al. 2020)) Defensive distillation is typically unable to protect against diverse Soll et al. 2019).
adversarial attacks (Carlini & Wagner 2016
In this work, explore an alternative defense strategy that exploits existing knowledge encoded in auxiliary task policies and known relationships between the policies. The key intuition underlying our approach is that existing task policies encode learnt low-level knowledge regarding the environment (e.g., possible observations, dynamics), whilst high-level specifications can provide guidance for transfer or generalization. Our approach is to leverage known and learnt relations between different policies as structural priors for ensemble of policies; our hypothesis is that while single task policy can be attacked, perturbing inputs such that multiple policies are negatively affected a consistent manner is more difficult.
Our framework, which we call Knowledge-basse based Policy Fusion (KPR), is partially inspired by the use of domain knowledge to address vulnerabilities to adversarial attacks in supervised learning (Melaccil ct al. 202T Gürel et al.) 2021 Zhang et al., (022). In these works, domain knowledge is encoded logical formulae ae predicted labels and set of features. A soft satisfiability score between

---

Under review as a conference paper at ICLR 2023
TM
Palicy
Gk
f(-)
(NN Pollcy)
Gk
f())
(NN Policy)
State
Observation
Final Action
9km
f(-)
Dist.
(Rule Based Policy)
UL (.A.
Action Dist.
Domain Knowledge
Policies
GNN Pooling
Fusion
Figure I: Knowledge-baad based Policy Recycling (KPR) Overview. Prior domain knowledge comprises set of auxiliary tasks and a set of logical formulae defined over the auxiliary tasks. KPR combines these auxiliary tasks policies to obtain a new robust policy for the main task. At every time step, obtain state-conditioned action distribution for each auxiliary task, which realizations of the variables in logical formulae. We represent logical formulae as graphs whose nodes are either auxiliary tasks or logical operators, c.g., -, V, = The instantiated logical formula graphs with node features are processed by a GNN pooling function (Section 3.2) to yield m different action different policy. Finally, we combine/fuse these distributions (Section 3.33 to obtain a final action distribution. In summary, KPR uses the specified logical formula graph as prior structure and then learns flexible relations over auxiliary tasks from interaction data.
distributions (one for each graph). In other words, each logical graph results
the predictions and given logic formulae is added to the objective to encourage the predictions to comply with the logical formulae. Even if part of the sample is corrupted by the adversary. the final output can be corrected by enforcing the domain knowledge rules. KPR extends this line of research RI. settings. Note that this extension is nontrivial; the auxiliary feature detectors used in supervised learning (Gürel et al.) 2021 Zhang et al., (022) do not capture temporal features, and more importantly, the consistency between the predictions and actions is not directly computable since the optimal actions for each state are unknown.
We address these issues by using auxiliary task policies and their relationships. In practice, these policies could be the intermediate by by-products of aurriuulum/hieracciiee. learning obtained via direct training with sub-goals. By combining these policies with specified and learnt relations, we construct ensemble of policies the target task. KPR graph neural networks (GNNs) backbone, which enables natural incorporation of graph-based domain knowledge while retaining the flexibility to learn from interaction data. The ensemble is then fused in simple parameter-less manner to obtain a new robust task policy (Figure []. From a practical perspective, KPR has key advantage: it is both policy and attack agnostic. Specifically, KPR can utilize any type of policy representation (e.g., neural networks, mule-basdd policies) as either the main task policy or an auxiliary task policy. addition, KPR doesn't require knowledge the specific attack or access to the adversarial environment. Our empirical results show that KPR results in more robust policies across multiple attacks compared to baselines in representative selection of Atari Games (Bellemare) al. 2013) and high-dimensional Robot Food Court Environment (RoFoCo).
To summarize, this paper contributes Knowledge-based Policy Recycling (KPR), which leverages domain knowledge defend against adversarial attacks in RL. Different from prior defense methods in reinforcement learning, such as adversarial training and robust learning, KPR is able to incorporate domain knowledge structural prior and then learn flexible relations from interaction data. To the best of our knowledge, this is the first work to demonstrate that domain knowledge the form of policies be used defend against adversarial attacks.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND AND RELATED WORKS
Similar to supervised learning, recent work has shown that DRL is also susceptible to adversarial attacks (Ilahi et al. 2022 Chen et al., 2019 Behzadan & Munir 2017| Oikarinen et al.] 202T) This area has drawn significant attention of late, and the following provides a brief overview;
refer readers desiring more detail to (Ilai et al. (022). Broadly speaking. there are two basic attack types
depending on the assumed adversary: (i) White-hox attacks (Goodfellow al.] [015 Carlini &
Wagner 2017;] Madry et al. 2018] [Schwinn et al. (0221) where the adversary has perfect knowledge of the target model, which is the victim policy in the RL context, and (ii) Black-box attacks (An- driushchenko et al. 2020 Pomponi et al. 2022), where the adversary does not know the model nor any of its attributes. Similar to previous works (Huang et al., 2017 Behzadan & Munir 201 7 Pattanaik et al., 2018 Zhang et al. 2020; Sun et al. 2022). ) we assume that the attacker does not have the ability to change the environment directly but perturbs the state observations returned by the environment before the agent observes them.
Existing defense methods can be categorized into: (i) Adversarial training (ikarinen et al. 2021 Behzadan & Munir 2018 Han et al., 2018), where the RI. agent is exposed to the adversarial environment during training (ii) Robust learning (Mandlekar et al. 2017 Smirnova et al., 2019 Pan ct al. 2019), which is training mechanism to ensure robustness against training-time adversarial attacks. A common approach is to add noise to the parameter state while training; (iii) Adversar- ial detection (Gallego et al. 2019a; Havens al. 2018: Gallego al. 2019a) trains a separate model to detect adversarial input and contaminated input (e.g.. image frames) are replaced with predicted/generated versions; (iv) Policy distillation (Czamneci al., 2019) focuses on transferring
knowledge from one or multiple policies to the target policy in a student-teacher framework.
KPR is different from the above strategies. Unlike adversarial training and variants of adversarial detection (Gallego et al. 2019b] Lin et al. (017),, we do not assume knowledge of the attack. In contrast to robust learning, KPR does inject noise during training. Although both policy distillation and KPR can fuse multiple policies, the methodology and application are different. Policy distillation does not leverage the relation between input policies and thus does not explicitly encourage structural consistency, which limits its defense performance as found in (Carlini & Wagner 2016; Soll et al. 2019) and in our experiments (Section F KPR can be seen as a policy ensemble (Wiering van Hasseli, 2008). albeit one that leverages prior knowledge in its construction. Prior works have suggested that ensemble methods offer some protection against adversarial attacks in supervised settings (Wiering & van Hasseli 2008). However, to our knowledge, policy ensembles have yet to be used as a defense strategy in RL.
KNOWLEDGE-BASBI POLICY RECYCLING (KPR)
As in standard RL, we consider a discounted discrete-time Markov Decision Process (MDP) AS,A, R, T. do), where S is a set of states, A is a set of discrete action, R S x A - IR is the reward functionn and T : S x A + (s)) is the transition function, 7 E [0, 1] is the discount factor, and do E P(s) is the distribution over the initial state. At time step f, the agent is in the state St ~ (ss,, at) and receives a reward R(s;, a.). The agent's objective is to learn a policy S + P(A) that maximizes the expected cumulative rewards,
max J(To) = -5o-do.er411-TT00 ((8,50(00)) 8
Σ R at)
We consider setting where the policy is trained in a benign environment and then deployed to test environment that may be adversarial.
Our goal is to retain expected cumulative rewards in the presence of a white box attacker that aims to alter our agent's 's actions by perturbing observations sf = 8t + 8,, s.t.8 E A, where A is a perturbation set, e.g., an & ball around with e radius, i.e. (6))) < Vt > O. We consider setting where we do not have access to the attacker's 's perturbation set nor how it is optimizing. In this information impoverished setting,
defensive options are relatively limited. One approach is sample possible attackers (with various perturbation abilities) and train a policy to be robust against these attacks. However, this approach is computationally expensive.
3

---

Under review as a conference paper at ICLR 2023
We posit the susceptibility of policy to attacker is due principally overfitting on specific task. To alleviate this issue, we propose to leverage prior knowledge comprising auxiliary tasks and relations between them - that can enable better robustness noisy perturbed observations. Let us define our main task as TM (which is modeled by an MDP as above) and an associated main task policy Tyy that maximizes returns. We assume we possess main task policy trained in the benign environment. We also have access to auxiliary task policies Xaxx = { T....... RT, } for tasks Taux = (Ti, T2,... ................................ J. Each task is modeled by an MDP tuple with the same elements as TM but with different reward function. We define the relation set I = {K. f(-)) comprising explicitly specified relations between the task policies IC (e.g., from domain expert) and learnt knowledge that will be represented implicitly by the function (()).
While prior knowledge usually enters into training as a regularization term in the loss, we incorporate this knowledge directly into the policy. We seek to obtain an augmented policy TIy that maximizes returns and is conditioned upon aux and I. The challenge is to ensure that this augmented policy has sufficient capacity to perform well on the task yet be robust attacks. Our approach is an ensemble method where we use I and max construct new set of policies IT, for the main task TM (Fig. 1 We then combine these policies using a simple parameter-less fusion mechanism,
Thw (s) = p(als Ilaxx I) = p p(alIITx 8).
(2)
In the following subsections, we provide details on the above: we will first discuss how prior domain knowledge regarding existing tasks can be specified. Next, we detail Graph Neural Network (GNN) pooling network to process knowledge and state information to learn relations. Finally, we discuss the simple voting mechanism to derive the final policy action distribution.
3.1 SPECIPYING RELATIONS VIA LOGICAL GRAPHS
In our setting, domain knowledge comprises (i) the set of auxiliary tasks policies given, I which could be any type of policy (including oon-differentiable or rule-based policies), and (ii) the logical relations among them. We focus on propositional logic, which is well-defined and unambiguous compared to natural language, yet relatively easy for humans to derive and interpret. proposition is statement which is either True or False. A formula F is compound of propositions connected A logical formula (and corresponding truth assignments)
by logical connectives, e.g.,, AV, undirected graph G = (V,& E) with nodes v, V, and edges (v,, U2) E E.
can be represented Individual nodes are either propositions (leaf nodes) or logical operators (-, A, V), where subjects and objects are connected to their respective operators.
In our work, leaf nodes are auxiliary tasks, which are True if the task is successful and False otherwise, where success of the task could be defined as whether the accumulated return attains certain threshold. For example, the logical relation formula TM = Ti A T2 expresses that "if the the main task is successful, then both the auxiliary tasks T₁ and T2 are successful". The features of the Icaf nodes are the predicted action distributions conditioned on the state st; the features of the logical operators are fixed to randomly generated vectors. To incorporate state information, we add a state feature node to every logical relation graph and connect it to every other node. As state features, we use the latent vector 2, obtained from simple VAE (Kingma Welling 2014) trained to reconstruct the state
The set of logical relations forms the set = (k: nmx ) defined over the auxiliary tasks Y Yaux where each k; is represented as graph GR.. Note that the logical relations need not to be complete nor error-frec, free; KPR can tolcrate a degree of misspecification as relations are also learnt from interaction data.
3.2 LEARNING RELATIONS FROM INTERACTION DATA
Graph Neural Network Pooling Function. To utilize the structural information contained in the logical relation graphs and enable learning from interaction data, we adopt Graph Neural Networks (GNNs) (Fey & Lenssen 2019). GNNs pass and aggregate the messages from the neighbors to encode the nodes in the graph with learnable weights. To obtain single action distribution from each logical relation graph, we add a graph pooling layer that maps the entire graph into a compact representation. We adopt Graph Multiset Pooling (Baek et al. 2021). which satisfies permutation
4

---

Under review as a conference paper at ICLR 2023
invariance, GNN pooling function f(-). The accumulated knowledge I consists of both the logical relations graphs and learnt weights of the GNN pooling function f(-).
Graph Multiset Pooling network f()), consists of two-layer message-passaane passing module g(-), graph multi-head attention pooling module that condenses all nodes to representative nodes fu(-), multi- head self-attention module 9, (:), and the second multi-head attention pooling module that condenses the entire graph to one vector h (-). Put together,
f(Gx.) (().))/i(c ((((c(((((o(Gaa))
where GE, is the input logical graph constructed from the logical relation formula &g-
The message-passingg function updates node representations by aggregating its neighbors' messages. In partieular, the message-passing function use is:
Σ MENU(E)
(TTal-11).
(4)
VD(u)DD(o D(u)
where ar is the node representation of node at 1th step. (Vv) denotes a set neighboring nodes of v, W is the weight matrix, and D(v), D(u) denotes the degree of node respectively.
An attention function q can be described as mapping a query and a set of key-value pairs to an output, where the query Q, keys K, values V, and output are all vectors. The output is computed as weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.
QKT Vdr
(Q, KV) = Softmax
V.
(5)
The multi-head attention function mulii simply uses multiple attention function with the linearly projected query, keys, and values matrices. To explicitly use the graph structure in multi-head attention, the keys and values are generated using one layer of gessage-passsingg function instead of conventional linear projection.
(o(G()))) = =1(0 ( (Z. (Z.9
(olGk)))) (o(G))))
(6)
where T(.) denotes the residual connection function, o maaii()) (.) is the multi-head attention function, Z a parameterized seed matrix that optimized end-to-end acting as the query matrix, and g represents the message-passiae functions for key and value respectively.
Model Training. Each (Gx;) can be interpreted a policy in ensemble; f takes logical formula graph as input, and outputs an action distribution. i.e., Pk. (als) = fGk.). The node feature for the auxiliary task node is the respective predicted action distribution and the auxiliary task identifier. The node features for logical operators are fixed randomly generated vectors that share the same dimension with auxiliary task features. To learn the relationships between tasks, we train each ensemble policy to predict the action with the highest probability under the main task policy. More precisely, we train the graph neural network pooling function ff with the following loss:
CrossEntropy (x)), [kk, Nauz (8t), )
L,f Berpprr (1/1)))
T=
where T {(st. az, R(s,, at) a trajectory sampled using the main task policy Tra and 2., are the state features. Note that KPR does not need to access the perturbed environment during training.
3.3 POLICY VIA FUSION OF ACTION DISTRIBUTIONS
At this stage, have action distributions Pk. (als) for each logical relation k;- To obtain a final action distribution, we will need to combine or fuse these action distributions together. We adopt a very simple voting-like mechanism that requires no training.
5

---

Under review as a conference paper at ICLR 2023
The fusion is performed via two main steps: (i) task policy filtering and (ii) action counting. We first select the top-3 action distributions by selecting the most "confident" models as measured by negative entropy. Intuitively, this filters away irrelevant action distributions given the current state.
Next, we form new action distribution by vote counting. Each of the three remaining policies casts positive vote for their top scoring action (with largest p(ayls) [s)])))))))))){)) and negative vote for their lowest scoring action (smallest p(ayls)). For each action aj, we tally the number of positive votes of and negative votes 0j- and construct a new action distribution for the main task Tm:
TM p(a)) 2 Of
(8)
A
EXPERIMENTS
In this section, we focus on validating that incorporating knowledge offers protection against adver- sarial attacks in deep RL. Specifically, we conduct experiments selection of Atari games and high-dimensional Robot Food Court Environment (RoFoCo
Compared Methods. We compared our method with policy ensemble (Wiering & van Hasselt 2008 policy distillation (Rusu et al. 2016), and adversarial training (Goodfellow et al. 20015) as strong baseline For Policy Ensemble, we five main task policies (trained with different random seeds) with majority voting. Since we focus on uttack-agnostin defense methods, we conduct adversarial training over a union of commonly-kme -nnown attacks, including the attacks that we evaluate on; this method approximates an "attack agnostic" adversarial training. Recall that the domain knowledge I comprises the relations among the auxiliary tasks and the main task. To investigate the role I plays, we included a variant of our method, MLP Fusion, which replaces the GNN with a MLP whose imput are the action distributions of the auxiliary policies.
Attacks. In Atari games, we used four common white-box attacks, FGSM Goodfellow et al. 2015 Huang et al., 2017). PGD (Madry et al.) (018)) and Jitter (Schwinn et al. 2021): and black- box attack, Square (Andriuscchenko et al. 2020). In the Robot Food Court environment (RoFoCo), we selected FGSM and PGD as white-box attacks (since they caused the most significant performance degradation in the Atari games experiments) and Square as the black-box attack. The attacks were implemented using the torchat tonttaccke tacks package (Kim 2020).
4.1 ATARI GAMES
Environment. We evaluated KPR on three Atari games: Road Runner, River Raid, and Space Invader, which are representative tasks that can be naturally decomposed into auxiliary tasks, e-g., collect targets, shoot enemies, and avoid collisions. For each game, the state is stack of four consecutive frames, where each frame pre-processed to size 84 x 84. Detailed environment settings can be found in Appendix B.2.
We adopt PPO (Schulman et al., 2017: Huang et al., 2022) as the learning algorithm. Each policy is trained with 10 million frames. Alternttive algorithms and policies such as DQN Mnih et al. 2015) and Rainbow DQN (Hessel et af. 2018) can be used without changing the overall proposed framework.
Main Task Policy.
Auxiliary Tasks and Domain Knowledge. Due to space restrictions, complete auxiliary tasks information and domain knowledge relations are in Appendix B.2. Taking Road Runner an example, the auxiliary tasks are: "Ti: Collect the bird seeds on the road." and "T2: Avoid the cars.". Denote the main task, "Collect the bird seeds while avoiding colliding with cars on the road", as Tiy- The logical relation is Tm = T₁ A T2.
'ee also illustrate KPR using a goal-findhe finding grid-world environment proof proof-of-coneee of-concept.. Due to space constraints, please refer to Appendix A for more details.
We also attempted to compare other methods, i.c., state- addee-aa DQN (SA-DQN) Zhang et al. 2020
2021), ). Policy Adversarial Actor Director (PA-AD) (Sun et al. 22022). Adversary Agnostic Policy Distillation (A2PD) (Qu et al. (02T). However, these methods do not perform sufficiently well in our environments, and thus, exclude them from our comparison.

---

Under review as a conference paper at ICLR 2023
Table 1: Test-time episode accumulated returns under three Atari games. The returns averaged
over 70 episodes and standard errors are reported in brackets. The best scores are in bold.
Attack Methods
Algorithm
Game
Benign
FGSM
PGD
Jitter
Square
41388.57 57
16764.29
15351.43
16904.29
40467.14
Main Task Policy
(2080.28)
(854.49)
(918.47)
(672.62)
(1735.77)
42548.57
31437.14
30970.00
22495.71
37414.29
Ensemble
(1802.11)
(1724.59)
(951.13)
(1173.65)
(1864.43)
40072.86
14931.43
15891.43
17501.43
40467.14
Policy Distillation
(1701.94)
(803.47)
(888.98)
(1052.50)
(1735.77)
Road
25762.86
6032.86
5538.57
8252.86
25922.86
Adversarial Training
Runner
(1250.54)
(350.16)
(326.89)
(433.16)
(1247.81)
37311.43
21752.86
21502.86
18524.29
41310.00
MLP Fusion
(1449.47)
(1292.49)
(1101.42)
(1161.33)
(1191.39)
43498.57
44417.14
34427.14
34187.14
46128.57
KPR (Ours)
(2203.07)
(3526.12)
(2301.96)
(1673.32)
(1707.11)
7491.57
2318.00
3567.29
5589.43
7005.00
Main Task Policy
(149:04)
(75.87)
(190.00)
(231.55)
(131.70)
7907.14
2178.43
2752.86
5107.86
4700.43
Ensemble
(113.39)
(61.28)
(97.65)
(181,46)
(294.33)
7948.71
2438.71
3575.57
5719.57
5943.14
Policy Distillation
(104.53)
(91.6)
(169.24)
(211.32)
(291.97)
River
5951.14
2631.11
3230.71
5172.29
2846.43
Adversarial Training
Raid
(137.67)
(117.13)
(134.76)
(159.03)
(155.60)
7832.29
1927.00
2295.29
$154.29
7626.00
MLP Fusion
(57.62)
(57.55)
(53.45)
(218.38)
(65.32)
7966.14
2720.14
3683.57
6470.29
7807.14
KPR (Ours)
(65.24)
(113.61)
(186.70)
(186.24)
(83.37)
1022.57
537.93
547.07
832.00
976.50
Main Task Policy
(55.97)
(21.74)
(20.62)
(45.82)
(59.22)
1166.64
621.79
649.14
883,71
661.21
Ensemble
(53.47)
(25.80)
(26.75)
(42.71)
(56.41)
1069.14
571.93
601.71
759.57
711.14
Policy Distillation
(53.30)
(25.77)
(29.60)
(43.00)
(60.52)
Space
618.64
454.71
357.64
591.86
356.29
Adversarial Training
Invaders
(27.66)
(29.16)
(23.46)
(28.01)
(20.03)
1059.07
733.00
696.71
875.07
765.00
ML.P Fusion
(53.60)
(32.23)
(34.08)
(44.94)
(24.64)
1161.64
810.36
817.21
938.64
1009.71
KPR (Ours)
(61.59)
(43.82)
(45.74)
(48.70)
(55.51)
Results and Discussion. Our experimental results are summarized in Table which shows test-time episode returns averaged over 70 episodes with standard errors brackets. In each game, the main policy solved the task but performance degraded when attacked, as expected. In general, KPR improves robustness all five attacks across the three environments. There is some performance degradation but it is less severe compared to the unprotected main task policy. This is most apparent for the strongest attacks, i.e., FGSM and PGD.
Interestingly, we see that adversarial training led to poorer policies in the benign environment and was also generally ineffective against FGSM and PGD. We believe this due to the attack-agnostic training scheme; in typical use, ad adversarial training requires knowledge of the attacker but in experiments, different attackers were sampled and this led to significant noise that hampered training.
Policy Ensemble, MLP Fusion, and KPR are all ensemble methods, but differ in their construction. Although not specifically designed for defense, the simple policy ensemble is surprisingly effective in the Road Runner domain. Policy Ensemble and MLP fusion achieve comparable performance, but are poorer compared to KPR. These results suggest that using existing prior knowledge in the form of policies and logical relations do result in more robust policies.
7

---

Under review as a conference paper at ICLR 2023
Robot Food Court Environment (RoFoCo)
Food Stalls
Tray
Robot
Collection
Point
Food Stalls
(1) Collect food from the correct food stall
(2) Deliver food to the correct table
(3) Collect the used tray after customers finished eating
(4) Deliver the used tray to the tray collection point
Figure 2: Robot Food Court Environment (RoFoCo) and a breakdown of the main task into four steps.
4.2 ROBOT FOOD COURT ENVIRONMENT (RoFoCO)
Environment. This environment simulates service robot in food-court setting and was developed using Unity (Juliani et al.) 2018). The agent is food serving robot. A food stall number and table number are given at the beginning of every episode. The main task is to collect food from the instructed food stall and deliver it to the instructed table number. After the customer finishes eating, the tray collection point. The task is divided into four stages illustrated in Figure 2 The agent receives +10 reward for completing each intermediate stage and an additional +22 reward for completing the whole task. The agent will get -0.1 penalty every time step and additional -5 penalty if it tries incorrectly perform pick up or put down actions on objects, such as trying to pick up food while the customer is eating. The maximum number of steps is 1,000. The available actions are: I) Move forward: 2) Move backward; 3) Turn left 90 degrees; 4) Turn Right 90 degrees; 5) Pick up; 6) Put down; 7) Do nothing. The observation each time step 128 128 RGB image.
the robot should pick up the used tray and deposit
Main Task Policy. As the observation space is significantly larger than the Atari games, exploring from scratch is computational resource consuming. To obtain the main task policy, we first initialize the agent with the policy obtained using imitation learning (Hussein et al. 2017) on a set of expert demonstrations. DQN (Mnih al. 2015) is used refine the policy via interactions further.
Auxiliary Tasks and Domain Knowledge. Due to space constraints, we give a few examples of auxiliary tasks and domain knowledge relations here and provide the complete list in Appendix B.3. Three example auxiliary tasks are 1) TT: Navigate to the food stall; 2) T2: Pick up food from customer tables; and 3) T3: Pick up food from the food stall. Denote the main task we described as TM. One example relation is TM = T₁ A (-T2) A T3.
Results and Discussion. Test-time performance comparison summarized in Table which shows episode returns averaged over 50 test episodes. KPR outperforms other baselines across attacks, which further supports the notion that prior policy knowledge encourages robustness in policies. As before, KPR policies experienced less severe degradation against the various attacks (which are unknown at training-ting time). As in the Atari games, we observed that the ensemble methods are able to provide comparably effective protection compared policy distillation and adversarial training.
8

---

Under review as a conference paper at ICLR 2023
Benign Action: Turn left (to navigate to the food stall) Attacked Action: Pick up food from other customers' table X
Ausiiiirr Task
Nevigatle food
Nothing
Main Task Policy
Orignal Observation
Auxilary Task uick food customer table (Negation Task)
Moyer Forward
Move Beskward
Turn Lert Oegress
Tam flihh Degruza
Pick Up
KPR
Notting
PGD Attack
KnoPF Policy
Figure 3: Adversarial Attacks Illustration. The adversary successfully misled the agent into picking up food from other customers' tables. KPR corrects the final action distribution prediction by leveraging auxiliary policies and domain knowledge.
Table 2: Test-time episode accumulated returns under the Robot Food Court environment. The returns averaged over 50 episodes and standard errors reported in brackets. The best scores are in bold.
Attack Methods
Algorithm
Benign
FGSM
PGD
Square
Main Task Policy
39.72 (2.62)
28.26 (3.46)
11.22 (5.21)
27.04 (3.66)
Ensemble
40.50 (2.08)
30.26 (2.67)
10.18 (5.35)
26.92 (3.42)
Policy Distillation
42.35 (1.78)
19.33 (5.27)
15.23 (4.35)
19.10((.85)
Adversarial Training
36.28 (2.81)
(6.16)
6.14 (4.49)
22.22 (3.24)
MLP Fusion
43.40 (1.73)
24.57 (3.85)
19.00 (3.87)
26.89 (3.95)
KPR (Ours)
42.13.(2.1)
34.70 (3.07)
28.51 (4.14)
31.62 (4.42)
As a qualitative comparison, Figure shows an example where the adversary successfully misled the robot into picking up food from other customers' tables. By leveraging auxiliary policies, such as navigating to the food stall and picking the food from the food stall, together with the relations between the tasks, KPR is able to correct the final action distribution prediction.
5 CONCLUSION AND FUTURE WORK
This paper proposes KPR, novel approach to leverage domain knowledge defend against adver- sarial attacks in reinforcement learning settings. KPR incorporates domain knowledge from auxiliary policies and specified logical relations between tasks, then learns flexible relations from interaction data via graph neural networks. The main advantage of KPR is that it is both policy and attack agnostic; any type of policy could be utilized, and no access nor information about the attack is required. We demonstrated its efficacy empirically in both Atari games and the complex Robot Food Court environment (RoFoCo).
A number of promising avenues exist for future research. In this work, we mainly experimented with neural network policies and future work can look into other auxiliary and main policies (e.g., interpretable rule-based policies). Next, KPR worked well in RoFoCo, which is highly-complex environment, but it is necessary to test KPR (and other existing defense methods) in real-world environments. Finally, believe that KPR also defend against alternative threat models, including observed adversaries that comply with environmental constraints (Gleave et al.) 2020 Cao et al. y022); these experiments would make for interesting next steps.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We propose a novel framework to defend against adversarial attacks in RL setting by leveraging prior knowledge. This paper does raise any ethical concerns. Our study does not involve human subjects. The Robot Foodcourt simulation environment we developed does not contain any sensitive information.
REPRODUCIBILIITY STATEMENT
To ensure the reproducibility of our experimental results, we include detailed network architecture and hyper-parameters the appendix and provide source code in the supplementary material.
REFERENCES
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: A query-efficient black-box adversarial attack via random search. In ECCV, 2020.
Jinheon Baek, Minki Kang, and Sung Ju Hwang. Accurate learning of graph representations with graph multiset pooling. In International Conference on Learning Representations (ICLR), 2021.
Xiaoxuan Bai. Wenjia Niu, Jiqiang Liu, Xu Gao, Yingxiao Xiang, and Jingjing Liu. Adversarial Examples Construction Towards White-Box Q Table Variation in DQN Pathfinding Training. In 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC), pp. 781-787, June 2018.
Vahid Behzadan and Arslan Munir. Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks. In Petra Perner (ed.), Machine Learning and Data Mining in Pattern Recognition, volume 10358, pp. 262-272 275. Springer International Publishing. Cham, 2017.
Vahid Behzadan and Arslan Munir. Mitigation of Policy Manipulation Attacks on Deep Q-Networks with Parameter-Space Noise. In Barbara Gallina, Amund Skavhaug, Erwin Schoitsch, and Friede- mann Bitsch (eds.), Computer Safery, Reliability, and Security, pp. 407,417, Cham, 2018. Springer International Publishing.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (ICMI), 2017.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, 2019.
Yulong Cao, Danfei Xu, Xinshuo Weng, Zhuoqing Mao, Anima Anandkumar, Chaowei Xiao, and Marco Pavone. Robust trajectory prediction against adversarial attacks. In 6th Annual Conference on Robot Learning, 2022.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), 2017.
Nicholas Carlini and David A. Wagner. Defensive distillation is not robust to adversarial examples.
ArXiv, 0bb//160.00m 00731003111.11 2016.
Patrick P. K. Chan, Yaxuan Wang. and Daniel S. Yeung. Adversarial Attack against Deep Reinforce- ment Learning with Static Reward Impact Map. In Proceedings of the 15th ACM Asia Conference on Computer and Communications Security, ASIA CCS '20, pp. 334-343, New York, NY, USA, October 2020. Association for Computing Machinery.
10

---

Under review as a conference paper at ICLR 2023
Tong Chen, Jiqiang Liu, Yingxiao Xiang, Wenjia Niu, Endong Tong, and Zhen Han. Adversarial attack and defense in reinforcement learning-fre from Al security view. Cybersecurity, 2(1):11, December 2019.
Petros Christodoulou. Soft actor-critic for discrete action settings. CoRR, abs/1910.07207, 2019.
Wojciech M. Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling Policy Distillation. In Proceedings of the /wenty-Second In- ternational Conference on Artificial Intelligence and Statistics, pp. 1331-1340, PMLR, April 2019.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Victor Gallego, Roi Naveiro, and David Rios Insua. Reinforcement Learning under Threats. Pro- ceedings of the AAAI Conference on Artificial Intelligence, (1931))9939-994)) July 2019a.
Victor Gallego, Roi Naveiro, and David Rios Insua. Reinforcement learning under threats. Proceed- ings of the AAAI Conference on Artificial Intelligence, 33(01):9939-9949 2019b.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations, 2020.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Nezihe Merve Gürel, Xiangyu Qi, Luka Rimanic, Ce Zhang, and Bo Li. Knowledge enhanced chine learning pipeline against diverse adversarial attacks. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Yi Han, Benjamin I. P. Rubinstein, Tamas Abraham, Tansu Alpcan, Olivier De Vel, Sarah Erfani, David Hubczenko, Christopher Leckie, and Paul Montague. Reinforcement Learning for Au- tonomous Defence in Seftware-Definge Networking. In Linda Bushnell, Radha Poovendran, and Tamer Başar (eds.), Decision and Game Theory for Security, pp. 145-165, Cham, 2018. Springer International Publishing.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI, 2016.
Aaron J. Havens, Zhanhong Jiang, and Soumik Sarkar. Online robust policy learning in the presence of unknown adversaries. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 9938 9948, Red Hook, NY, USA, December 2018. Curran Associates Inc.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv, 2017.
Shengyi Huang. Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022.
git.kb.tttuuuu... 10/0022/33//222
25/ /p5pppo-o imp womppeeeeettioooniiiiiimi -otioonnnddd/////////eaaa implementtaiion-etaiis...
Ahmed Husscin, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Compus. Surv., 50(2), 2017.
Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fugaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learning. IEEE Transactions on Artificial Intelligence, 3(2):90-109, April 2022.

---

Under review as a conference paper at ICLR 2023
Arthur Juliani, Vincent-Pienne Pierre Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. CoRR, htss/1809.0267,, 2018.
Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks.
arXiv preprint
arXiv:2010.01950, 2020.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Xian Yeow Lee, Yasaman Esfandiari, Kai Liang Tan, and Soumik Sarkar. Query- based targeted action-space adversarial policies on deep reinforcement learning agents. Proceedings of the ACM/IEEE 12th International Conference on
Systems, ICCPS '21, pp. 87-97, New York, NY, USA, May 2021. Association for Computing Machinery.
Yen-Chen Lin, Ming- Yu Liu, Min Sun, and Jia-Bin Huang. Detecting adversarial attacks on neural network policies with visual foresight. CoRR, abs/1710.008/ 14, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings, 2018.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially Robust Policy Learning: Active construction of physically-plausibl perturbations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (TROS), pp. 3932-3939, Vancouver, BC, September 2017. IEEE
Stefano Melacci, Gabriele Ciravegna, Angelo Sotgiu, Ambra Demontis, Battista Biggio, Marco Gori, and Fabio Roli. Domain knowledge alleviates adversarial attacks in multi-label classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51807540):529-333, 2015.
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui-Wel Wei Weng. Robust Deep Reinforcement Learning through Adversarial Loss. In Advances in Neural Information Processing Systems, May 2021.
Xinlei Pan, Weiyao Wang, Xiaoshuai Zhang, Bo Li, Jinfeng Yi, and Dawn Song. How you act tells lot: Privacy-leaking attack deep reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2019.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings the 17th International Con- ference on Autonomous Agents and MultiAgent Systems, AAMAS 118 pp. 2040-2043 Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Jary Pomponi, Simone Scardapane, and Aurelio Uncini. Pixle: fast and effective black-box attack based on rearranging pixels. CoRR, abs/2202.002336 2022.
Xinghua Qu, Abhishek Gupta, Yew-Soon Ong. and Zhu Sun. Adversary agnostic robust deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-12, 2021.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In ICLR, 2016.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015.
12

---

Under review as a conference paper at ICLR 2023
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 0770700647, 2017.
Leo Schwinn, René Raab, An Nguyen, Dario Zanca, and Bjoemn M. Eskofier. Exploring misclas- sifications of robust neural networks to enhance adversarial attacks. CoRR, abs/2105.10304,
2021.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittiisser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):444448) 2016.
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning, 2019.
Marcus Soll, Tobias Hinz, Sven Magg, and Stefan Wermter. Evaluating defensive distillation for defending text processing neural networks against adversarial examples. In Igor V. Tetko, Vera Kurková, Pavel Karpov, and Fabian J. Theis (eds.), Artificial Neural Networks and Machine Learning- ICANN 2019: Image Processing 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part III. volume 17729 of Lecture Notes in Computer Science, pp. 685-696. Springer, 2019.
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. In International Conference on Learning Representations, 2022.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Ro- bustness may be at odds with accuracy. In International Conference Learning Representations, 2019.
Marco A Wiering and Hado van Hasselt. Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybermetics), 38(4):930-936 2008
Wikipedia. River Raid- - Wikipedia, the free encyclopedia. http: / en wwt.wiiiiiiii.og///// w/ index..xx.. php 23 S02eeteeeerr2222].
2022a. [Online; accessed
Wikipedia. Road Runner (video game) Wikipedia, the free encyclopedia.
http:
e/en. wttp://iia...///////////aa////ae 20game) holdde:11dddddddddddoe.... 2022b. [Online; accessed accessed 23-epmmee22333sppppmmeee-22221
(video
Wikipedia. Space Invaders Wikipedia, the free encyclopedia www.eediaiiiiiiiiiemmm..m.aa.aae org/ 2022c. [Online; ac-
cessed 23-September-22222]
Yao- Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaud- huri. closer look at accuracy vs. robustness. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 8588-8601. Curran Associates, Inc., 2020.
Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations state observations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21024 2020.
Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsich. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations, 2021.
Jiawei Zhang, Linyi Li, and Bo Li. CARD: Certifiably robust machine learning pipeline via domain knowledge integration, 2022.
13

---

Under review as a conference paper at ICLR 2023
A TOY EXAMPLE: GOAL FINDING
To better illustrate how KPR works, we demonstrate it on a goal-finding grid-world environment as a proof of concept evaluation.
Environment. The agent's objective is to find a target goal while avoiding the obstacle in a 7 x 7 grid-world (Figure 4/A.). The agent receives a reward of +-10 for reaching the target, a -10 penalty for colliding with the obstacle, and a -1 penalty at each time step. The maximum episode length 30. To mimic the redundant information in real-world scenarios, each object has a specific color, shape, and letter.
Auxiliary Tasks and Domain Knowledge. We design auxiliary tasks that focus on the different aspects of the objects, such as "Ti: Find the orange color (target object color)". "T2: Avoid the diamond shape (obstacle object shape)". We denote the main task as Tv. simple domain knowledge relation among them could be Tm = T1 A T2. A complete list of auxiliary tasks and domain knowledge relations among them can be found in Appendix B.I.
Compared Methods. We compared with policy ensemble (Wiering & van Hasselt 2008 and MLP Fusion which is a variant of KPR. Instead of leveraging GNN based pooling function to incorporate logical relations among auxiliary tasks, MLP Fusion MLP to replace the GNN pooling function. We aim to investigate the role of the logical relation domain knowledge component by comparing with this variant. We use simple Deep Q-Learning (DQN) (Mnih et al. 2015) to train our main task and auxiliary tasks policies. Alternative algorithms and policies can be used without changing the overall proposed framework, e.g., improved DQN variants (Hasselt et al. 2016; Hessel al. 2018) and alternative algorithms (Schaul et al. 2015
Bellemare et al. 2017 Christodoulou 2019).
Result and Discussion. We adopt FGSM attack (Goodfellow al. 2015 to perturb the observation. Since the observations are grid world states, the attacks here are generally stronger than the ones on pixel perturbations. The performance evaluated in terms of the episode return averaged over 1000 episodes. The results are summarized in Figure BB. C.. While the performance is similar in the Benign Environment, KPR outperforms others by a large margin in the perturbed environment. Note that KPR is not trained in the perturbed environment. This demonstrated KPR's s ability to correct the inconsistency caused by adversarial attacks by leveraging domain knowledge over auxiliary policies.
Furthermore, to investigate the qualitative correlation between the amount of domain knowledge and episode return, we study how different knowledge levels affect the test performance, as shown in Figure 4 D. E.. The "Low" knowledge level consists of 4 auxiliary policies and formulae. The "Moderate" " knowledge level consists of 11 auxiliary tasks and formulae. And the "High" knowledge level consists of 17 auxiliary tasks and 12 formulae. As shown in Figure and E., as the knowledge level increases, the improved robustness to adversarial attacks also increases.
A.1 AUXILIARY TASKS AND DOMAIN KNOWLEDGE
The objective for the agent to find the goal while avoiding the obstacle in a 7 x grid world 4A.).. The agent gets +10 reward for reaching the target, -10 penalty for touching the obstacle, and a - 1 penalty every time step. The maximum step number is 30. To mimic the redundant information in real-world scenarios, we use three features represent a single object, color, shape, and the letter on it. The auxiliary tasks are:
1. Find the orange star "T" object.
2. Find the orange object.
3. Find the star object.
4. Find the "T" object.
5. Avoid the blue diamond "O" object.
6. Avoid the blue object.
7. Avoid the diamond object.
8. Avoid the "O" object.
9. Avoid the orange star "T" object (negation task).
14

---

Under review as a conference paper at ICLR 2023
B.
Benign Enwironment
C.
Perturbed Environment
A.
Episode Return
Eplsode Retur
Single
KnoPF
Single
Ensemble MLP
KnoPP
Benign Environment
Perturbed Environment
D.
E.
Epleode Return
Episode Return
Traget
Obstacle
Agent
Moderate Knowledge Level
High
Low
Moderate Knowledge Level
High
Figure 4: (A.) Goal finding grid world. (B. and C.) Test-time performance comparison. Episode return of main task policy, policy ensemble. MLP Fusion, and KPR in the goal finding grid-world environment. (D. and E.) The comparison of how the different knowledge levels, measured in terms of the number of auxiliary policies and formulae, affect the episode return. The episode returns are averaged over 1000 test episodes. Error bars indicate standard error.
10. Find the blue diamond "O" object (negation task).
11. Find the orange object and avoid the blue diamond "O" object.
12. Find the star object and avoid the blue diamond "O" object.
13. Find the "T" object and avoid the blue diamond "O" object.
14. Find the orange star "T" object and avoid the blue object.
15. Find the orange star "T" object and avoid the diamond object.
16. Find the orange star "T" object and avoid the "O" object.
17. Avoid the orange star "T" object and find the blue diamond "O" object.
There are many possible relations among them. The logical relations we adopted are:
1. TM = T₁ A Ts
2. TM = Ti A Te
3. Tm
T1 A T7
4. Tm
T1 A T₃
5. Tm
T2 A T₁
6. Tm
T3 A Ts
7. Tm =
T₃ A To
8. TM = -T17
9. TM = (-T,) A (-T11)
10. Tm =
(T2 V T3 VT. V VTg)
TM = Tn T12 T
12. TM = T14 V Tis V V
where Ty denotes the main task.
15

---

Under review as a conference paper at ICLR 2023
B ATARI GAMES ENVIRONMENT
B.1 AUXILIARY TASKS AND DOMAIN KNOWLEDGE
B.1.1 ROAD RUNNER
The Road Runner runs endlessly to the left. The player must pick up bird seeds on the street, avoid cars (Wikipedia 2022b). The auxiliary tasks are:
1. Collect bird seeds.
2. Avoid cars.
There are many possible relations among them. The logical relations we adopted are:
1. TM = T1 T2
where TM denotes the main task.
B.1.2 RIVER RAID
River Raid is a vertically scrolling shooter game. The player flies fighter get which can only move left and right. It cannot maneuver up and down the screen, but it can accelerate and decelerate. The player's jet crashes it collides with the riverbank or an enemy craft, or if the jet runs out of fuel. The player scores points for shooting enemy tankers (30 points), helicopters (60 points), fuel depots (80 points), jets (100 points), and bridges (500 points). The jet refuels when it flies over a fuel depot. bridge marks the end of game level /Wikipedia 2022a). The auxiliary tasks are:
1. Shoot enemy tankers.
2. Shoot enemy helicopters.
3. Get fuel depots.
4. Shoot enemy jets.
5. Shoot enemy bridges.
6. Avoid collides with riverbank or enemy craft.
There are many possible relations among them. We sampled subset:
1. TM = T1 A T2 A T3 A T4 ATATTT T6
2. TM = T1 A Te
3. Tm
T2 A TE
4. Tm
=
TG A To
5. Tm
T1 A T2
6. Tm
T1 A T5
7. TM
Ti A T4
8. TM >
T2 A T3
9. TM
=
To A T₁
10. TM
T3 A T₁
11. TM =
Ta A To
12. TM
T4 A T₃
where Ty denotes the main task.
16

---

Under review as a conference paper at ICLR 2023
B.1.3 SPACE INVADERS
Space Invaders is a fixed shooter in which the player moves a laser cannon horizontally across the bottom of the screen and fires at aliens overhead. The aliens begin as five rows of eleven that move left and right as a group, shifting downward each time they reach a screen edge. The goal is to eliminate all of the aliens by shooting them. While the player has three lives, the game ends immediately if the invaders reach the bottom of the screen. The aliens attempt to destroy the player's cannon by firing projectiles. The laser cannon is partially protected by stationary defense bunkers which are gradually destroyed from the top by the aliens and, if the player fires when beneath one, the bottom (Wikipedial 2022c). The auxiliary tasks are:
Shoot the first row of aliens.
2. Shoot the second row of aliens.
3. Shoot the third row of aliens.
4. Shoot the forth of aliens.
5. Shoot the fifth row of aliens.
6. Avoid being shot by aliens.
There are many possible relations among them. We sampled a subset:
1. TM = T1 A T2 A T3 A T4 A T5 A To
2. Tm
T1 A To
3. TM = T2 A To
4. Tm = TG A To
5. Tm = T1 A T2
6. Tm > T₁ A T5
7. TM
Ti A T4
8. Tm
+ T2 A T3
9. Tm
T2 A T4
10. TM
=
Ta A T5
11. TM T3 A T6
12. TM
= A Ts
where TM denotes the main task.
C ROBOT FOOD COURT (RoFOCO) ENVIRONMENT
In order evaluate the performance of KPR in complex real-world like environment, we developed a high fidelity simulated Robot Food Court environment (RoFoCo) via Unity (Juliani et al. 2018). as shown Figure 2 The agent is food serving robot. A food stall number and table number are instructed at the beginning of every trajectory. The main task is to collect food from the correct food stall and deliver it to the correct table number, then after the customer finish eating, pick up the used tray and send it to the tray collection point. The agent gets a + 10 reward for completing each intermediate stage, and an additional +22 reward for completing the whole task. The agent will get a -0.1 penalty every time step and additional -5 penalty if it tries to perform pick up or put down actions on objects that are not affordable, such as trying to pick up something on humans. The maximum number of steps is 1000. The available actions are: 1) Move forward; 2) Move backwards; 3) Turn left 90 degrees; 4) Turn Right 90 degrees; 5) Pick up; 6) Put down; 7) Do nothing. The observation is 1 28 x 128 RGB image.
17

---

Under review as a conference paper at ICLR 2023
C.I AUXILIARY TASKS AND DOMAIN KNOWLEDGE
The auxiliary tasks are:
1. Navigate to food stall number r.
2. Navigate to customer table number
3. Navigate tray collection point.
4. Pick up food from food stall number æ.
5. Put down food on customer table number T.
6. Pick up used tray from customer table number
7. Put down used tray tray collection point.
8. Pick up food from customer table number (negation task).
9. Put down used tray on customer table number æ (negation task).
10. Put down food on food stall number x (negation task).
11. Put down food tray collection point (negation task).
12. Put down used tray on food stall number x (negation task).
There are many possible relations among them. The logical relations we adopted are:
1. TM = T₁ A TT
2. Tм = T2 A Ts
3. Tm = To A To
4. Tm
T3 A T7
5. Tm
T₁ A I4 A (-T10) A A
6. Tm
T2 A Tú A (-Tp)
7. TM >
T2 A To A (-Ts)
8. TM
Ta A T7 A (-T.))
9. Tw =
Ti A T4 A (-Ts)
Tm - T2 A T5 A (-T10)
11. TM = T3 A
12. Tm = TT'
13. Tm -Tp
14. Tm = -T10
15. TM = -T11
16. TM
= -T12
where TM denotes the main task.
D ADVERSARIAL ATTACKS
We evaluated against white-box attacks, FGSM, PGD, Jitter, and black-box attack, Square attack. We'll briefly introduce their intuitions and mechanisms, well the hyper-parameters we use. To ensure fair comparison, we utilize the same e value, which controls the maximum perturbation value, for all 4 adversarial attacks. According different task characteristics, use 0.002 for Road Runner, 0.01 for River Raid, 0.002 for Space Invaders and 0.002 for Robot Foodcourt environment. Pixel values scaled [0, 1).
18

---

Under review as a conference paper at ICLR 2023
D.1 FAST GRADIENT SIGN METHOD (FGSM)
FGSM (Goodfellow et al. 2015) is method to efficiently calculate the gradient of the cost function with respect the input of the neural network. The adversarial examples are generated using the following equation:
= x t sign (0, x; x,,))
where 0 denotes the parameters of model, æ: is the input the model, y the target associated with x, J(0, x, y) is the cost used to train the neural network, and sign the component-win signum operator. The adversarial examples generated by FGSM exploit the "linearity" of deep network models in the higher dimensional spaces whereas such models were commonly thought to be highly non-lineat at that time. They hypothesized that the designs of deep neural networks that encourage linear behavior for computational gains also make them susceptible to cheap analytical perturbations, which is often referred as "linearity hypothesis".
D.2 PROJECTED GRADIENT DESCENT (PGD)
PGD (Madry al. 2018) improve the performance of FGSM by running finer iterative optimizer for multiple iterations. PGD performs FGSM with smaller step size and projects the updated adversarial sample into the Loo neighbor of the benign samples and valid range. Hence the adversarial perturbation size is smaller than The update procedure follows:
r't+1 Proj(*' a sign J(, x' (.3)))
(10)
D.3 JITTER ATTACK
In order make adversarial attacks more effective, Jitter (Schwinn et 2021) proposes novel loss function to encourage logits scale invariance, diverse attack targets, and perturbation norm minimization. The final loss function can be described follows:
2 18-
if a' is misclassified
Litter
||a NOO, 0)l2 if a is not misclassified yet
softmax (a
(11)
|
where y is the ground truth, is the output logits after perturbation, and a controls the lowest and largest possible output values of the softmax function.
D.4 SQUARE ATTACK
Square attack (Andrushchenko et al.) 2020) is based on a randomized search scheme which selects localized square-shaped updates random positions that each iteration the perturbation is situated approximately at the boundary of the feasible set. The objective is to solve the constrained optimization problem:
min ((f(t).v) fy(
max (a)), s.t.
||
x||
S
(12)
where f is the target network, æ is the input, y is the ground truth, and the domain of the input.
E MODEL ARCHITECTURE
Policy Network. We adopt PPO (Schulman al. 2017 Huang et al. (022) as the learning algorithm. We three-layer CNN with [32, 64, 64) hidden size, followed by two linear layers with 512 neurons for both actor and critic networks.
VAE. The encoder is five-layer CNN with hidden dimension (32, 64, 128, 256, 512), followed by linear layer. The decoder symmetric to the encoder with an additional 2D transposed convolution layer (Zeiler al. 2010) of hidden dimension 512.
19

---

Under review as a conference paper at ICLR 2023
GNN Fusion Network. Our GNN pooling network consists of 3 GCN convolution layers with hidden
dimensions {64, 32, 64], followed by the graph mutliset transformer as described in Section 3 and linear layers of 64 hidden neurons.
MLP Fusion Network The MLP Fusion Network consists of 2 linear layers of 64 hidden neurons.
20


Reviews:
[{"review_id": "v2Fdic-0QKB", "reviewer": "Reviewer_Vu3r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new framework called Knowledge-based Policy Fusion (KPR), which leverages domain knowledge to defend against adversarial attacks in RL. KPR incorporates domain knowledge from auxiliary policies and specified logical relations between tasks, then learns flexible relations from interaction data via graph neural networks. Different from prior defense methods in reinforcement learning, such as adversarial training and robust learning, the main advantage of KPR is that it is both policy and attack agnostic; any type of policy could be utilized, and no access nor information about the attack is required. They demonstrated its efficacy empirically in both Atari games and the complex Robot Food Court environment (RoFoCo). According to the authors, this is the first work to demonstrate that domain knowledge in the form of policies can be used to defend against adversarial attacks.", "summary_of_the_review": "This paper proposed a new framework called KPR, which leverages domain knowledge to defend against adversarial attacks in RL. The charts are clear and easy to understand, the descriptions are also very concise and clear, and the experiments are sufficient, but the scenario assumptions are single, and some theoretical explanations are lacking", "strength_and_weaknesses": "a)\tStrength:\nThe authors describe in detail the differences between KPR and previous methods:\npolicy ensemble, policy distillation, and adversarial training, and designed a very sufficient experiment to verify the advantages of KPR. To investigate the role domain knowledge plays, authors included a variant of KPR, MLP Fusion, which replaces the GNN with an MLP whose input is the action distributions of the auxiliary policies. This ablation study is convincing and shows the effect of domain knowledge. In the experimental part, the choice of experimental environment is very interesting, the visual effect is great, and the experimental data is relatively sufficient, showing the advantages of KPR in the form of a table\nb)\tWeaknesses:\nThe author makes a strong assumption that the susceptibility of a policy to an attacker is due principally to overfitting on a specific task. If only one specific type of attack is considered, it is not enough to demonstrate the generalization of defense strategies and the use of GNN for a well-structured use of domain knowledge. In terms of theory, there seems to be a lack of more analysis. The author needs to explain why this is necessary. The theoretical analysis in methods such as fuzzy control can have a fuzzy mathematical system to describe how to convert from natural language to numerical output.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XsIWv9RbKoj", "reviewer": "Reviewer_XTBx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author extended using domain knowledge to improve adversarial attacks from supervised learning to reinforcement learning and makes learned main policy universally resistant to adversarial attacks without prior knowledge of the attack. \n\nThe main contributions consist of 1. It generalizes previous work, which encodes domain knowledge as logical formula and a set of features and adds to the objective to apply constraints on the learned policies close to logical formulae from prediction tasks to reinforcement learning approaches. 2. The new extended approach is first used to tackle adversarial attack problems in reinforcement learning (RL), which is very important to keep RL performing appropriately under noisy input signals. 3. It is the first paper investigating using domain knowledge to defend against adversarial attacks in the form of policies. \n", "summary_of_the_review": "This paper focuses on a very interesting domain which is the robustness of RL models under adversarial attacks, and the extended framework from the past prediction tasks is decent and easy to apply to various RL agents. Through the experiment results, this proposed method clearly outperforms the other baseline models universally. Thus I consider it a high-quality paper. \n", "strength_and_weaknesses": "Strength: \n\n- The author is trying to introduce methods used in prediction tasks into reinforcement learning and solving the problem of the adversarial attack, which is a very important limitation hindering RL implementation in the real world. The idea is pretty concise and focused on an interesting task. \n\n- Using the past policies or sub-tasks policies and domain knowledge to help improve the RL model's robustness can help reduce the implementation cost and explainability of RL models. Also, the proposed framework is very flexible and easily extended to different agents. \nThe experiments support the author's claims and are extensively investigated in different scenarios. And the proposed methods uniformly outperform other baseline models.\n\nWeakness:\n\n\n- Part of this paper’s idea is extended from past work on prediction tasks. The basic idea is using assembly learning and combining the idea from past prediction tasks as constraints on the policies, which slightly weakens the novelty of this paper. \n\n- The baseline model seems not to involve adversarial detection approaches, and in Table 1, the performance of MLP Fusion and KPR even increased after applying Square adversarial attacks. It is strange but not discussed by the author.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L7FKo_a_JT", "reviewer": "Reviewer_EmJQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of adversarial attacks against deep RL policies. They propose a defense that uses an ensemble policy based on policies for various auxiliary tasks and logical relations between those auxiliary tasks and the main task. A graph neural network is used to combine action distributions from the various auxiliary policies over the graph induced by each logical relation. Then, policies are aggregated across all logical relations via a voting mechanism. In experiments in three Atari games and a simulated robotics environment, the aggregated policy seems to be more robust to various adversarial attacks.", "summary_of_the_review": "Overall, I think this paper explores a promising direction but needs a fair amount of work to improve the evaluation and reproducibility. Therefore, I don't think it is ready for publication at ICLR. However, if the authors can provide a more convincing evaluation along with all the details and code to reproduce the experiments, I am happy to raise my score.", "strength_and_weaknesses": "Overall, much of the paper is well-written and the experiments show promising results. However, I have a couple major concerns about the evaluation:\n\n * First, the evaluation seems to be underspecified and thus not reproducible. The authors say they use standard white-box adversarial attacks like FGSM and PGD to attack policies. However, these algorithms cannot actually directly solve the attack optimization problem given in (2). This is because (2) is a function of the return of the policy, and one cannot directly differentiate the return with respect to the adversarial perturbations. In general, optimizing equation (2) would require more RL-like methods to approximate the gradient rather than the autograd methods used for typical whitebox attacks in supervised learning. Since the authors do not specify how they resolve this contradiction, I'm don't think the evaluation is reproducible. There are not further details in the appendix and there is no code, either.\n * Second, I'm also worried about gradient masking [1] in the evaluation. In particular, the aggregation mechanism given in equation (9) is not actually differentiable with respect to the underlying policies! This means that there are no gradients that an attacker can use to optimize the state towards lower reward. I'm not sure how the attacks are expected to work given this, and this makes me distrust the evaluation. While blackbox attacks should avoid this problem, looking at the results it appears that KPR does not show as large an improvement over the main task policy against black-box attacks, indicating that gradient masking may be occurring.\n * Third, it does not seem like the authors ran the entire training procedure using multiple different random seeds for the evaluation. This is a standard practice in RL evaluation and is needed to reduce the vast variance in RL outcomes across random seeds. \n\nGiven these three concerns, I think the evaluation needs to be improved before the proposed method can be reliably judged to perform better than the baselines. I encourage the authors to precisely report how the white-box attacks optimize equation (2), avoid gradient masking, and report results across multiple training runs with different random seeds.\n\nBesides the evaluation concerns, there are some other downsides to the method, like the need for auxiliary tasks and logical relations between them. \n\n[1] Athalye et al. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. ICML 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0sze09ljXjy", "reviewer": "Reviewer_dRJe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes an approach for combining domain knowledge with RL; the methodology encodes propositional logical rules via Graph Neural Networks, with multiple pooling and attention mechanisms.", "summary_of_the_review": "The clarity of the manuscript declines in its experimental design and adversarial attacks illustration.\n\nThe problem formulation and methodology rely on strong assumptions.\n\nMissing discussion of limitations, ethics, and reproducibility statements.", "strength_and_weaknesses": "(Strengths)\n\nI like the notion of combining domain knowledge with learning-based methodology.\n\nSections 1-3 are pretty well-written.\n\n(Weaknesses)\n\nSection 3.1: What are these success thresholds and how are they obtained, for an arbitrary collection of tasks and attacks? This seems non-trivial to define and combinatorial in complexity.\n\nSection 3.2: Architecturally, the proposed approach is very straightforward, and I was hard-pressed to find much novelty there.\n\nSection 4: It would be helpful if the manuscript provided the full names, brief descriptions, and some intuition behind why specific attacks were chosen.\n\nSection 4.1: In many practical applications, the subtasks/objectives can be conflicting — how are these situations reconciled?\n\nSection 4.1, Section 4.2: I am not really convinced by the experiments that the manuscript chose to perform. I would have liked to see empirical results on more standardised or challenging evaluation benchmarks in RL — especially those that include aspects of safety or robustness, such as OpenAI Safety Gym, Learn-to-Race, Carla Challenge, etc. Discussion of adversarial attacks in these tasks, where subtask objectives are often conflicting (!), would be much more compelling and intuitive.\n\nSection 4.1: It would be helpful to show the cumulative episodic reward curves (with error areas), by repeatedly evaluating on the test environment after a fixed number of training iterations or environment interactions: this would also give intuition on the training dynamics and sample-complexity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject