Moving Forward by Moving Backward: Embedding Action Impact over Action Semantics
Authors: Kuo-Hao Zeng
Luca Weihs
Roozbeh Mottaghi
Ali Farhadi
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Embodied AI
Adaptation
Visual Navigation

Abstract: A common assumption when training embodied agents is that the impact of taking an action is stable; for instance, executing the ``move ahead'' action will always move the agent forward by a fixed distance, perhaps with some small amount of actuator-induced noise. This assumption is limiting; an agent may encounter settings that dramatically alter the impact of actions: a move ahead action on a wet floor may send the agent twice as far as it expects and using the same action with a broken wheel might transform the expected translation into a rotation. Instead of relying that the impact of an action stably reflects its pre-defined semantic meaning, we propose to model the impact of actions on-the-fly using latent embeddings. By combining these latent action embeddings with a novel, transformer-based, policy head, we design an Action Adaptive Policy (AAP). We evaluate our AAP on two challenging visual navigation tasks in the AI2-THOR and Habitat environments and show that our AAP is highly performant even when faced, at inference-time, with missing actions and, previously unseen, perturbed action spaces. Moreover, we observe significant improvement in robustness against these actions when evaluating in real-world scenarios.

Published as a conference paper at ICLR 2023
MOVING FORWARD BY MOVING BACKWARD: EMBED- DING ACTION IMPACT OVER ACTION SEMANTICS
Kuo-Hao Zeng', Luca Weihs", Roozbeh Mottaghi', Ali Farhadi
Paul G. Allen School of Computer Science & Engineering, University of Washington
2PRIOR Allen Institute for AI prior. al lllenai.... org/ httpaa//w....ppeeeeettaooooaoaae
ABSTRACT
A common assumption when training embodied agents is that the impact of taking action is stable; for instance, executing the "move ahead" action will always move the agent forward by a fixed distance, perhaps with some small amount of ietuuter-induced noise. This assumption limiting; an agent may encounter set- tings that dramatically alter the impact of actions: a move ahead action on wet floor may send the agent twice as far as expects and using the same action with a broken wheel might transform the expected translation into rotation. Instead of relying that the impact an action stably reflects its pre-defined semantic mean- ing, we propose to model the impact of actions on-the-fly using latent embed- dings. By combining these latent action embeddings with a novel, transformer- based, policy head, we design an Action Adaptive Policy (AAP). We evaluate our AAP on two challenging visual navigation tasks in the A12-THOR and Habi- tat environments and show that our AAP is highly performant even when faced, at inference-time with missing actions and, previously unseen, perturbed action space. Moreover, we observe significant improvement in robustness against these actions when evaluating in real-world scenarios.
INTRODUCTION
Humans show a remarkable capacity for planning when faced with substantially constrained or aug- mented means by which they may interact with their environment. For instance, a human who begins to walk on ice will readily shorten their stride to prevent slipping. Likewise, a human will spare little mental effort in deciding to exert more force to lift their hand when it weighed down by groceries. Even in these mundane tasks, we see that the effect of humans' actions can have sig- nificantly different outcomes depending on the setting: there no predefined one-to-one mapping between actions and their impact. The same is true for embodied agents where something simple as attempting to moving forward can result in radically different outcomes depending on the load the agent carries, the presence of surface debris, and the maintenance level of the agent's 's actuators (e.g., are any wheels broken?). Despite this, many existing tasks designed in the embodied AI commu- nity (Jain al., 2019; Shridhar et al., 2020; Chen et al., 2020; Ku et al., 2020; Hall et al., 2020; Wani et al., 2020; Deitke et al., 2020; Batra et al., 2020a; Szot et al., 2021; Ehsani et al., 2021; Zeng et al., 2021; Li al., 2021: Weihs et al., 2021; Gan et al., 2021: 2022: Padmakumar et al., 2022) make the simplifying assumption that, except for some minor actuator noise, the impact of taking particular discrete action is functionally the same across trials. We call this the action-stability assumption (AS assumption). Artificial agents trained assuming action eitiii-stability are generally brittle, obtaining significantly worse performance, when this assumption is violated at inference time (Chattopadhyay et al., 2021 I); unlike humans, these agents cannot adapt their behavior without additional training.
In this work, we study how to design reinforcement learning (RL) policy that allows an agent to adapt to significant changes in the impact of its actions at inference time. Unlike work in training robust policies via domain randomization, which generally leads to learning conservative strate- gies (Kumar et al., 2021), we want our agent to fully exploit the actions it has available: philosoph- ically, if a move ahead action now moves the agent twice as fast, our goal is not to have the agent take smaller steps compensate but, instead, to reach the goal in half the time. While prior works have studied test time adaptation of RL agents (Nagabandi et al., 2018; Wortsman et al., 2019; Yu al., 2020; Kumar et al., 2021), the primary insight in this work is action-centric approach which

---

Published as a conference paper at ICLR 2023
Expected
Actual
MP
Order- Inder--nvaiient Head
d'
Potioy Necwwork
Move
Defective Wheel
Carpet
a_- - MoveAhead 0.2m
4/11 =
MoveAhead D. 1m
AAP
Adiiin mome
Order-nnurrient Head
Poticy hecwork
Rotate
Carpet
Defective Wheal
9,11 90'
Qy-1 30"
Figure I: An agent may encounter unexpected drifts during deployment due to changes in its internal state (e.g., a defective wheel) or environment (e.g., hardwood floor v.s. carpet). Our proposed Action Adaptive Policy (AAP) introduces action-impact encoder which takes state-changes (e.g., Ot + Ot+1) caused by agent actions (e.g., ((-1) as input and produces embeddings representing these actions' impact. Using these action embeddings, the AAP utilizes Order-Invariant (OI) head to choose the action whose impact will allow it to most readily achieve its goal.
requires the agent to generate action embeddings from observations on-the-fly (i.e., no pre-defined association between actions and their effect) where these embeddings can then be used inform future action choices.
In our approach, summarized in Fig. 1, an agent begins each episode with a set of unlabelled actions A = {a...., ..... a"). Only when the agent takes one of these unlabelled actions a' at time t. does observe, via sensor readings, how that action changes the agent's state and the environment. Through the use of a recurrent action-impact encoder module, the agent then embeds the observa- tions from just before (o:) and just after (0t+1) taking the action to produce an embedding of the action ei,t- At future time step t', the agent may then use these action-impact embeddings to choose which action wishes to execute, In our initial experiments, we found that standard RL pol- icy heads, which generally consist of linear function acting on the agent's 's recurrent belief vector b, failed to use these action embeddings to their full potential. As we describe further in Sec. 4.3, we conjecture that this is because matrix multiplications impose explicit ordering their inputs so that any linear-based actor-critic head must treat each of the n! potential action orderings separately. To this end, we introduce novel, transformer-based. policy head which we call the Invariann (OI) head. As suggested by the name, this OI head is invariant to the order of its inputs and processes the agent's belief jointly with the action embeddings allow the agent choose the action whose impact will allow it to most readily achieve its goal. We call this above architecture, which uses our recurrent action-ine -impott encoder module with our OI head, the Action Adaptive Policy (AAP).
To evaluate AAP, we train agents to complete two challenging visual navigation tasks within the A12-THOR environment (Kolve et al., 2017): Point Navigation (PointNav) (Deitke et al., 2020) and Object Navigation (ObjectNav) (Deitke et al., 2020)'. For these tasks, we train models with moder- ate amounts of simulated actuator noise and, during evaluation, test with a range of modest to severe unseen action impacts. These include disabling actions, changing movement magnitudes, rotation degrees, etc.; we call these action augmentations drifts. We find that, even when compared to sophis- ticated baselines, including meta-learning (Wortsman et al., 2019), model-basee approach (Zeng et al.. 2021), and RMA (Kumar et al., 2021), our AAP approach handily outperforms competing baselines and can even succeed when faced with extreme changes in the effect of actions. Further analysis shows that our agent learns to execute desirable behavior at inference time; for instance, it quickly avoids using disabled actions more than once during episode despite not being exposed to disabled actions during training. In addition, the experimental results in real-world test scene from RoboTHOR on the Object Navigation task demonstrate that our AAP performs better than baselines against unseen drifts.
In summary, our contributions include: (1) an action-centric perspective towards test-time adapta- tion. (2) an Action Adaptive Policy network consisting of an action- iition-impctt encoder module and a novel order-invariant policy head, and (3) extensive experimentation showing that our proposed approach outperforms existing adaptive methods.
We also show results in modified PettingZoo environment Terry al. (2020) on Point Navigation and Object Push in Sec. F and a modified Habitat Environment (Savva et al., 2019) on Point Navigation in Sec. G.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Adaptation. There are various approaches in the literature that address adaptation to unknown environments, action effects, or tasks.
Novel environments: These approaches explore the adaptation of embodied agents to unseen envi- ronment dynamics (Yu et al., 2019; Wortsman et al., 2019; Zhou et al., 2019; Li et al., 2020a;b; Peng et al., 2020; Song et al., 2020; Loquercio et al., 2021; Evans et al., 2022; O' O'Connell et al., 2022; Kumar et al., 2021; 2022; Zhang et al., 2022; Agarwal et al., 2022). Various techniques such as meta-learning (Wortsman et al., 2019), domain randomization (Yu et al., 2019), and image transla- tion (Li et al., 2020a), have been used for adaptation. In contrast to these approaches, we address changes in the actions of the agent as well. Moreover, unlike some of these approaches, e.g. Li et al. (2020a), we do not assume access to the test environment.
Damaged body and perturbed action space: These methods focus on scenarios that the effeet of the actions changes during inference time or other factors. Yu et al. (2020) study adaptation to a weakened motor. Nagabandi et al. (2018) explore adaptation robot components, Our approach is closer to those of Yu et al. (2020) and Nagabandi et al. (2018) 8) that also consider changes in environment structures as well. Nonetheless, these works focus on using meta-learning for locomotion tasks and optimize the model at the testing phase while we do not optimize the policy at inference time. In our experiments, we find we outperform meta-learning approaches without requiring, computationally taxing, meta-learning training.
result of damage, weakened motors, variable load,
missing leg. Yu et al. (2017) adapt to differences in mass and inertia of the
Novel tasks: Several works focus on adapting to novel tasks from previously learned tasks or skills (Finn et al., 2017; Gupta et al., 2018; Chitnis et al., 2019; Huang et al., 2019; Fang et al., 2021). We differ from these methods as we focus on single task across training and testing.
-ut-of-distribution generalization. Generalization to out-of-distribution test data has been stud- ied in other domains such as computer vision (Hendrycks & Dietterich, 2019; Peng et al., 2019), NLP (McCoy et al., 2019; Zhang et al., 2019), and vision & language (Agrawal et al., 2018; Akula et al., 2021). In this paper, our focus is on visual navigation, which in contrast to the mentioned domains, is an interactive task and requires reasoning over a long sequence of images and actions.
System identification. Our paper shares similarities with the concept of System Identifica- tion (Verma et al., 2004; Bongard et al., 2006; Seegmiller et al., 2013; Cully et al., 2015; Banerjee et al., 2020; Lew et al., 2022). The major difference between our approach and the mentioned works is that we use visual perception for adaptation.
3 PROBLEM FORMULATION
In this work, we aim to develop an agent which is robust to violations of the AS assumption. In particular, we wish to design an agent that quickly adapts to settings where the outcomes of actions at test time differ, perhaps significantly, from the outcomes observed at training time; for instance, a rotation action might rotate an agent twice as far it did during training or, in more extreme settings, may be disabled entirely. We call these unexpected outcomes of actions, drifts. As discussed in Sec. 1, the AS assumption is prevalent in existing embodied Al tasks and thus, to evaluate how well existing models adapt to potential drifts, we must design our own evaluation protocol. To this end, we focus on two visual navigation tasks, Point and Object Navigation (PointNav and ObjectNav), as (1) visual navigation is a well-studied problem with many performant, drift-free or fixed-drift, baselines, and (2) the parsimonious action space used in visual navigation (Move, Rotate, and End) allows us to more fully explore the range of drifts possible for these tasks. We will now describe the details of how we employ drifts evaluate agents for these tasks.
In this work, we assume that particular drift, perhaps caused by defective sensor, broken parts, motor malfunction, different amount of load, or stretched cable (Boots et al., 2014), may change across episodes. We focus primarily on two categories of potential drift occurring in visual navi- gation tasks: movement drift d,n which causes an unexpected translation when executing a Move action, and rotation drift d,, which leads to an unexpected rotation when executing a Rotate action. More specifically, we semantically define the movement and rotation actions as Move (d) = "move forward by d meters" and Rotate (0) "rotate by 9 degrees clockwise". As describe below, the semantic meaning of these actions is designed to be true, up to small-to-moderet noise, during training, but may change significantly during evaluation.
3

---

Published as a conference paper at ICLR 2023
(a) Model Overview
(b) State Encoder
State Encoder
Misual Cibseraiin
Visual Encoder
cat
Policy Network
Oi Head
[-]
Viuca Chareetinn
Action-Impact
F
Task
Embeddler
g
Encoder
(a) Model Overview. Our model includes three modules: state encoder, an action action-impact impact encoder, and policy network with an order-invarian head (OI head). The blue and purple colors denote learnable modules, and the yellow and light gray color represents hidden state from the state encoder and action embedding from the cettonnimpact encoder. (b) State Encoder is composed of a visual encoder for visual encoding and embedder for task goal. The light red and dark blue colors indicate the learnable visual encoder and goal embedder, respectively.
Figure 2:
For each training and evaluation episode, a movement drift dm and a rotation drift d, are sampled and fixed throughout the episode. At every step, upon executing Move (d) the agent experiences a d + dm + Rdd translation toward its forward-facing direction, where the nid represents the high- frequency actuator noise from the RoboTHOR environment (Deitke et al., 2020). Similarly, when the agent executes Rotate (0), the agent rotates by 0 + d, + ne degrees where ny again represents high-frrequecy actuator noise. To evaluate robustness to the AS assumption, the strategies we use to choose dmn and d, differ between training and evaluation episodes.
During training. At the start of training episode, we sample movement drift dm UU(PP, p) and rotation drift d, ~ U(-q, q), where U-.-)) denotes a uniform distribution and |p| < thm as well as |gl < 180% We set Um = 0.25m, the standard movement magnitude in RoboTHOR.
During inference. At the start of an evaluation episode, we sample movement drift dm, $ P-p.p] and rotation drift d, & I-q,9l.
Note that the drifts chosen during evaluation are disjoint from those seen during training.
ACTION ADAPTIVE POLICY
We first overview our proposed Action Adaptive Policy (AAP) in Sec. 4.1. We then present the details of our action-impan encoder and Order-Invarianl (OI) head in Sec. 4.2 and Sec. 4.3, respec- tively. Finally, we describe our training strategy in Sec. 4.4.
4.1 MODEL OVERVIEW
The goal of our model is to adapt based on action impacts it experiences at test time. To accomplish this, the Action Adaptive Policy (AAP) includes three modules: a state encoder, an action-impact encoder, and a policy network with an order-invariant head (OI head), as illustrated in Fig. 2 (a). The state encoder is responsible for encoding the agent's current observation, including visual observation di and the task goal, into a hidden state hT. The action-impat encoder processes the current observation, previous observation, and previous action a;1 produce of action em- beddings F,, one embedding for each action the action space A. More specifically, the job of the action-impact encoder is to update the action embeddings via the previous action recurrently using the encoded state-change feature (Sec. 4.2). Note that the embeddings ei, E Eg are not given information regarding which action was called to produce them; this is done precisely so that the agent cannot memorize the semantic meaning of actions during training and, instead, must explicitly model the impact of each action. Finally, the policy network with the OI head takes both the hidden state h and the set of action embeddings F, as inputs and predicts an action at.
State Encoder. Following Batra et al. (2020b); Weihs al. (2020); Khandelwal et (2022), we implement the state encoder with visual encoder and an embedder, as shown in Fig 2 (b). Note that popular baseline policy models used for embodied AI tasks frequently employ a policy network directly after the state encoder to produce the agent's action. In this work, we use RGB image as our visual observation Vt. For the PointNav task goal, we follow Savva et al. (2019) and use both GPS and compass sensors provide the agent with the angle and distance to the goal position, (Apr, A For the ObjectNav task, we follow Deitke et al. (2020) and provide the agent with a semantic token corresponding to the goal object category the task goal. We use CL.IP cTee--ret
4

---

Published as a conference paper at ICLR 2023
Feature Extract
Encode
Visual Observation (ood
Nisual Oiservation Ord
Visual Encoder
Linear
RNN
d,_, special action?
[-1
No
select
Task Goal
Embedder
g
Yes
Ci.
Action-Order
d,11
Figure 3: Action-Impact Encoder. The imput to the action-impace encoder two consecutive observations and the previous action. The encoder first extracts visual representations and a goal representation via ResNet-50 and Embedder. Concatenated, these form state-change feature ft. The encoder then uses the previous action a-11=a to retrieve the corresponding memory mi- With m,, an RNN maps to an embedding. Finally, the encoder registers this embedding as the action embedding ei, if a' is not a "special" action (i.e., a non-actuator-bart baasd action). Otherwise, the encoder registers an action embedding obtained from the Action- Orderr Encoder into Ci,t-
ResNet-50 (He et al., 2016; Radford et al., 2021) as our visual encoder and a multi-layer perceptron (MLP) the embedder to obtain visual representation and goal representation y. respectively. Finally, we concatenate them to construct the hidden state ht-
4.2 ACTION-IMPACT ENCODER
In this section, we introduce the action-impact encoder, see Fig. 3. The goal of this encoder is to produce set of action embeddings E, which summarize the impact of actions the agent has taken in the episode so far. To adapt to unseen drifts during inference, these action embeddings should not overfit to action semantics, instead they should encode the impact that actions actually have at inference time. The action- action-impact encoder first extracts a state-change feature from two consecu- tive observations. It then utilizes a recurrent neural network (RNN) to update the action embedding ei according to the previous action a;- 1--1=a'. As the action-impact encoder generates embeddings for each action a' without using semantic information about the action, every embedding e,, only encodes the effect of its corresponding action. The decision to not use information about action semantics has one notable side-effect: at the start of an episode, all action embeddings 6s,0 will be equal that the agent will not know what any of its actions accomplish. It is only by using its tions, and encoding their impact, that the agent can learn how its actions influence the environment.
During the feature extraction stage, we employ the same visual encoder and embedder used by the state encoder to process the visual observations, and Uz and the task goal. A linear layer is used to project the two concatenated visual representations rz-] E R to r E IRá, where / = 1024 for PointNav and = 3136 for ObjectNav. We then concatenate r and the goal representation g. to form the state-change feature f. = [rī,g], g). see the Feature Extract stage of Fig. 3.
After feature extraction, we apply an RNN, namely a GRU (Chung al., 2014), to summarize state changes through time, illustrated in the Encode stage of Fig. 3. The use of recurrent network here allows the agent to refine its action embeddings using many observations through time and also ignore misleading outliers (e.g., without a recurrent mechansim, an agent that takes Move action in front of a wall may erroneously believe that the move action does nothing). To update the action embeddings recurrently,
utilize the previous action a=1 a² index into matrix of memory vectors to obtain the latest memory m, associated with a'. This memory vector is passed into the RNN with f. recurrently. In this way, the state-change resulting from action a² only influences the embedding e, Note that we use the same RNN to summarize the state- state-changes for all actions. Moreover, since a is only used to select the memory, the RNN is agnostic to the action order and focuses only on state-change modeling. Thus, the action embedding produced by the RNN does not contain action semantics, but does carry state-changes information (i.e,, action impact).
One technical caveat: the PointNav and ObjectNav tasks both have special End action that denotes that the agent is finished and immediately ends the episode. Unlike the other actions, it makes little sense to apply action drift to End as it is completely independent from the agent's 's actuators. We, in fact, do want our agent to overfit to the semantics of End. To this end we employ an Action- Order Encoder which assigns unique action embedding to the End in lieu of the recurrent action embedding. Note that these types unique action embeddings are frequently used in traditional models to encode action semanties. Finally, register the recurrent embedding into the action
5

---

Published as a conference paper at ICLR 2023
(a) Policy Network
(b) Linear Head
(c) Order-Invariant Head
a, RIAI 7 7 V.
4 E
R|4I
Order-Imeriant Heed
a E R
R
V O
FFyyyer
FfLayer
f
Self-Atten
cat
Linear Head
Linear Head
Order Invoriant Hoad
f
LayerNorm
RNN
+
flotten
b
shortout
603" e1, b,
003' Ci. b,
Figure 4: (a) Policy Network with Order-Invariane Head first flattens the input and uses an RNN to produce belief b. An Order-Invariant head further processes the action embeddings E and belief b to predict action probability and value. (b) Linear actor-critic takes the belief b from RNN to predict action probability and value. (c) Order-Invariant Head is invariant to the order of its inputs so the policy predicts the action probability and value based on state-changes (i.e., action impact) instead of action semantics (i.e., consistent action order).
embedding Cs via the previous action at-1 a', if this action a' is not the End (Encode panel of Fig. 3); otherwise, we register the action embedding Ci,t as the output of the Action-Order Encoder.
4.3 POLICY NETWORK WITH AN ORDER- -IVVRRIIT HEAD
Standard policy networks in embodied Al use an RNN to summarize state representations h, through time and an actor-critic head to produce the agent's policy (a probability distribution over actions) and an estimate of the current state's value. Frequently, this actor-critic head is a simple linear function that maps the agent's current belief's 6, (e.g., the output of the RNN) to generate the above outputs, see Fig. (b). As the matrix multiplications used by this linear function require that their inputs and outputs be explicitly ordered, this imposes an explicit, consistent, ordering on the agent's actions. More specifically, let the weight matrix in the linear mapping be W = [wo).... w; are the unique set of weights corresponding the vth action a'. Because the w is specifically learned for the ith action during the training stage, it encodes the semantics of action a' and thereby prevents the policy from generalizing to unexpected drifts applied to a². For example, if the action a' is Rotat at e (30%), and the policy is learned with training drifts in the range between [-15%, 15%], then policy will fail catastrophically when experiencing an unexpected drift 20 because the policy has overfit to the semantics that the a² should only be associated with 30 + 15 rotation.
[w,] [n]l...w...) then
To address this issue, we propose to equip our policy network with a transformer-basee based Order- Invariant (OI) actor-critic head, as shown in Fig. 4 (a). The OI head, illustrated in Fig. 4 (c), takes both the belief b, and the action embeddings where the 0x+11, is produced by bt- A en.ts Gp,t as input to produce 00.t, On,t; +,+1 Layer then maps the 00, On. to action logits and the On+1,e to an estimate of the current state's 's value. Finally, we apply Softmax on the ac- tion logits generate action probabilities. We implement the OI head using Transformer (Vaswani et al., 2017) encoder without a positional encoding. By design, the extensive weight sharing in a transformer (e.g., the query, key, and value embedding, well as the across all the input tokens) means that there are no unique weights assigned to any particular action.
Layer shared
In conclusion, as (1) the transformer uses the same weights for each input embedding, (2) the moval of positional encoding prevents the model from focusing on action semantics, and (3) the set of action embeddings E is used represent the impact of actions, the proposed AAP can choose the action most useful for its goal despite unexpected drifts. In Sec. 5.1 we show the importance of weight sharing and the action embeddings E by comparing our overall model with two ablations: (i) AAP but with linear actor-critic head and, (ii) AAP but with no action-Impace Encoder.
4.4 TRAINING STRATEGY
To train our agent, we use DD-PPO (Wijmans et al., 2020) to perform policy optimization via rein- forcement learning. To endow the action embeddings E with an ability to describe the state-change between two consecutive observations, we utilize an auxiliary model-based forward prediction loss to optimize the action-impact encoder. In particular, we apply Feed-Forward Layer (a shared MLP applied to each input independently), which operates on each embedding {eo..... ,,,, in- dependently predict the agent state-change from the current step to the next step + 1. As

---

Published as a conference paper at ICLR 2023
long the RNN in the action-impact encoder has been exposed to state-changes resulting from an action, the action-impact encoder can learn to predict the precise agent state change As1+1- Our forward prediction loss is optimized jointly alongside DD-PPO using the same collection of on- policy trajectories collected in standard DD-PPO training. Namely, if M consecutive agent steps collected during an on-policy rollout, then we apply simple MSE Loss w.E.I the ground-truth agent state-changes As = { M - As Liowward = MSE(As. As*), where As = {Ast-M, As 155-M+1,... ) are the predicted agent state-changes. Therefore, our over- all learning objective is L = LpPO + ofoooward, where a controls the relative importance of Croward- We set a = 1.0 in this work. For model optimization details, see Fig. 8 in the Sec. B.
5 EXPERIMENTS
In our experiments, we aim to answer the following questions: (1) How does the proposed AAP perform when exposed to expected or modest drifts and how does this performance compare to when it is exposed to unseen or severe drifts? (2) Can the AAP handle extreme cases where some actions are disabled and have no effect on the environment? (3) When faced with multiple disabled actions, can the AAP recognize these actions and avoid using them? (4) Is it important to use the action-impact encoder and OI head jointly or can using one of these modules bring most of the benefit? (5) Qualitatively, how does the APP adapt to, previously unseen, action drifts?
Implementation details. We consider two visual navigation tasks, PointNav and ObjectNav, using the RoboTHOR framework (Deitke et al., 2020). RoboTHOR contains 75 scenes replicating coun- terparts in the real-world and allows for rich robotic physical simulation (e.g., actuator noise and collisions). The goals of the two tasks are for the agent to navigate to a given target; in Point- Nav this target is a GPS coordinate and in ObjectNav this target is an object category. In the environment, we consider 16 actions in action space A = {Move (d), , Rotate (0), End), where E (0.05, 0.15, 0.25] in meters and 0 E (0', +300, +-60" +900, 1200 180%) We set = 0.05m so that we sample movement drifts dm ~ U.-0.05, 0.05) and g = 15 so that we sample rotation drifts d,- ~ U(-15", 15%) during training. We then evaluate using drifts di E {=0.05, 10.1, 0.2, and di E , =450 +90", 1135" 180"}. Note that only dm = =0.05 and d, = +15" are drifts seen during training. The agent is considered to have successfully completed episode, and its location is within in 0.2 meters of the target for PointNav or if the target object is visible and within 1.0 meters for ObjectNav. Otherwise, the episode is considered failure. We use the AllenAct (Weihs et al., 2020) framework to conduct all the experiments. During training, we employ the default reward shaping defined in AllenAct: Reenalty + Rsnceess + Rdistancs. where Ryenalty = -0.01, Rsuccess = 10, and Rdistance denotes the change of distances from target between two consecutive steps. See Sec. B for training and model details.
episode if it takes the End action, which always immediately ends an
Baselines. Each baseline method uses the same visual encoder, goal embedder, RNN and linear actor-critic the policy network unless stated otherwise. We consider following baselines.
EmbCLIP (Khandelwal et al., 2022) is baseline model implemented in AllenAct that uses CLIP-pretrained ResNet-50 visual backbone. simply uses state-encoder to obtain hidden state hy, applies a RNN to obtain recurrent belief by over h4, and uses a linear actor-critic head to predict an action a, via by This architecture is used by the current SoTA agent for RoboTHOR ObjectNav without action drift (Deitke et al., 2022).
MMetaRRL (Wortsman et al., 2019) is an RL approach based on Meta-Learning (Finn et al., 2017). However, since we do not provide reward signal during inference, cannot simply apply bitrary meta-RL method developed in any prior work. Thus, we follow (Wortsman et al., 2019) to employ the same Liorward in our AAP as the meta-objective during training and inference. We add an MLP after the belief by in the EmbCLIP baseline policy to predict the next agent state-change
As141 to enable the meta-update phase during both training and inference stages.
RMA (Kumar et al., 2021) is a method assessing environmental properties using collected experi- ences. Although they focus on locomotion tasks, we adapt their idea into our studied problem. At the first stage in RMA training, we input both the movement drift d, and rotation drift d, into the model to learn a latent code c E R of the environment. The policy network then takes c as an additional representation to make a prediction. During the second training stage, we follow (Kumar et al., 2021) to learn a I-D Conv-Net that processes the past 16 agent states to predict c. In this stage, all modules are frozen, except for the 1-D Conv-Net.
Mosel-Based (Zeng et al., 2021) is model-basse forward prediction approach for the agent state- changes As. However, they focus predicting the future agent state-changes associated with differ-
7

---

Published as a conference paper at ICLR 2023
= 0.05m
d
0.1m
0.2m
0.4m
Success Rate (SR)
1.0
Navigation
Point
0.6
0.4
0.0
20
100
180
100
180
20
100
180
20
100
180
0.5
Success (SR)
Navigation
Object
0.2
180 20
100
15020
100
d2
d
di
AAP (Ours)
EmbCLIP
Meta-RL
RMA
Model-Bann Based
Figure 5: AAP results. Top: PointNav evaluation. Bottom: ObjectNav evaluation. We com- pare the proposed AAP with basclines, including EmbCLIP, Meta-RL, RMA, and Model-Based. We measure the SR over different drifts, including d' 0.05 0.11, 0.2, 0.4) and de {+15", +30% + + +135% 180%). See Sec. A for an example of how our AAP learns to handle unseen drifts by using the Action-Impact Encoder and OI head.
Top-Down View (Progress)
d, = 0.2, d, = O", Disabled (o (-30' 60" 90%
120" 150
PointNav
Rotato (20))
Ratate(XUr
Rotatol 150))
Move(0.25)
Top- Down View (Progress)
0.2,
30", Disabled Rotato:
(30% 80",0000" 100
ObjectNov
Roiate(-60")
Rotate(- 30')
Rotato(- 30')
Move(0.25)
Figure 6: Qualitative Results. Examples of PointNav (top) and ObjectNav (bottom), where dm = 0.2 and d, 0 and d, -30 Rotate left and rotate right actions are disabled, respectively. The agent adapts by rotating in the other direction to compensate for the disabled actions.
ent actions, and further embed them into action embeddings. Moreover, they use a linear actor-critic make prediction which does not fully utilize the advantage of our action-centric approach.
5.1 RESULTS
Evaluation Metrics. We evaluate all models by their Success Rate (SR); also report the popular SPL (Anderson et al., 2018), Episode Length, Reward, Distance to Target, and soft-SPL (Datta al., 2021) metric in Sec. D. SR is the proportion of successful episodes In addition, we report the Avoid Disabled Actions Rate (ADR) and Disabled Action Usage (DAU) metrics in the Disabled Actions experiments. The ADR equals the proportion of episodes which the agent uses no disabled action more than once, and the DAU computes the total number of times disabled actions are used averaged across all episodes.
the validation set.
Different Drifts. The quantitative results for PointNav and ObjectNav are shown in Fig. 5. As shown, the SR of competing baselines decreases remarkably as the movement drifts dm and rotation drifts d, increase from modest (d,,, = + 0.05, d,-= + 15%) to more severe =0,, 180110%). Alternatively, our AAP performs consistently across all the drift changes and outperforms baselines significantly. Note that the baselines perform well (SR=100% for PointNav and SR=46% for Ob- jeciNav) with seen drifts =0.55, d5-= but our AAP achieves consistent SR across all drifts because the action embeddings effectively capture the state-changes associated with actions so the Ol head can thereby recognize that, for instance, action a; = Rotate becomes another action aj Rot anotate with rotation drift d, 90" during inference. As the magnitude of movement actions are not bounded (unlike rotation actions Rotate (0), where f E [-180s, 180%)) AAP's performance begins to decrease for very large movement drifts (e.g., dm. 0.4).
8

---

Published as a conference paper at ICLR 2023
Table I: Disabled Actions results in PointNav (a,c) and ObjectNav (b,d). f and & indicate if larger or smaller
numbers are preferred. The experiments are repcated three times.
(a)
PoimtNay Success Rate (SR)
(b) ObjectNav Success Rate (SR)
SR1
disable 0
disable (
SR1
disable gia
disable (aggi
AAP (Ours)
96.011.1
98.0_0.9
AAP (Ours)
31.2.33.3
38,8
EmbCLIP
1..8+1.4
1..0=1.3
EmbCLIP
12.8+3.2
23.8+
Meta-RL 29.2.14.88
28.6113.4
Meta-RL
18.711.77
26.2.11.9
RMA
1..
12.8+4.4
RMA
9.3+6.8
24.2+22.1
Model-Based
10.7
14.0.12.8
Model-Based
19.2:11.2
9.911
(c) Analysis of disabled actions in PointNav
(d) Analysis of disabled actions in ObjectNav
ADR / DAUL
disable 6e
disable (right
ADR / DAUL,
disable glan
disable fright
AAP (Ours)
12.5 27.8
18.4/ 15.7
AAP (Ours)
3.2/
5.3 / 28.5
EmbCLIP
0.3/
1.0/ 93.3
EmbCLIP
0.6 0.6/90.5 90.5
2.9 27.2
Meta-Ro -Let--RL
0.7 83.1
0.6/ 85.2
ML
2,0/80 80.1
2.2/ 84,5
RMA
0.4/ 94.4
0.5 95.2
RMA
95.9
2.6 73.3
Model-Based
0.7 / 89.4
0.6/ 96.1
Model-Based
2.5/
1.5/ 89.8
Disabled Actions. We consider two experimental settings to evaluate how models perform when some actions are disabled (e.g., due to to malfunctioning motors or damaged wheel). In the first setting we disable the 5 rotation angles fright (30%, 60% 90", 120", 150%) so that the agent cannot rotate clockwise and, in the second setting, we disable gleft E {-300, -
-100 1200)-1509 so that the agent can only rotate clockwise. We otherwise set d.. - 0.2m and d, 0 in this experiment. The SR for PointNav and ObjectNav are shown in Tab. 1a and Tab. 1b. As shown, there is huge difference between the baselines and AAP; for instance, for the PoinNNv task, AAP achieves near 100% SR while the best competing baseline (Meta-RL) achieves <30%. In addition, AAP outperforms the baselines by at least 12.5% on average in ObjectNav. We further report the ADR and DAU metrics for this task in Tab. Ic and Tab. 1d; from these tables we see that AAP is very effective at recognizing what actions are disabled through the proposed action embeddings E, see ADR, and efficiently avoids using such actions, see DAU, Fig. 6 shows an example with Øleft on PoinNNv and another one with (right on ObjecnNav. See more examples in Sec. and Fig. 14.
Ablation Studies. We investigated which module in the AAP contributes the most to the performance, or if they are all necessary and complementary to one another. We conducted the ablation studies on Point- Nav by comparing our proposed AAP to: (1) "LAC",
dg + 0.1m
0.4m
Success Rate (SR)
1.0
6
a variant of AAP that uses a linear actor-critic head instead of our OI head and (2) "Action-Semantics" a variant of AAP that uses the Of head without the Action Impact Encoder (so there are unique learn- able embeddings for each unique action). The results are shown in Fig. 7. Although Action- achieves similar SR to AAP when dm m and d, E {+155, +:30), its performance starts decreas- ing as the drifts become more dramatic. In contrast, the poor SR of LAC confirms our OI head is far better leverage the predicted action embeddings than linear actor-critic.
6.0
100 X 16b
dt
AAP (Ours)
Action-Semanties
LAC
Figure 7: Ablation studies. We conducted the ablation studies on PointNav by comparing AAP to: (1) "LAC"", a variant of AAP that uses linear actor-critic head instead of our Ol head and (2) Aaction-Semanica
variant of AAP that uses the OI head without the Action-Impact Encoder.
Real-World Experiments. We train our AAP and EmbCLIP on ProeTHOR 10k (Deitke et al., 2022) for ObjectNav with
Table 2: Real-World Experiments results in RoOO on ObjectNav. Model
training drifts d,, and d,, then evaluate in a real-world scene from RoboTHOR (Deitke et al., 2020). In an exploratory ex- periment, the goal is to find Apple and Chair from 2 dif- ferent starting locations (20 episodes) at inference. We inject
SR1
AAP (Ours)
65%
EmbCLIP trained w. drifts
30%
EmbCLIP
25%
the drifts (d',, E {-0.05m, Orn, 0. 0.1m} and d; E +0,, 600 }) by adding the drifts to the lowelevel
commands used by the robot. Tab. 2 shows AAP performs more robustly against these unseen drifts.
6 CONCLUSION
We propose Action Adaptive Policy (AAP) to adapt to unseen effects of actions during inference. Our central insight is to embed the impact of the actions on-the-fly fly rather than relying on their training- -tie semantics. Our experiments show that AAP is quite effective adapting to unseen action outcomes at test time, outperforming a set of strong baselines by a significant margin, and can even maintain its performance when some of the actions are completely disabled.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank members of the RAIVN lab at the University of Washington and the PRIOR team at the Allen Institute for AI for their valuable feedback early versions of this project. This work is in part supported by NSF IIS 1652052, IIS 17303166, DARPA N66001-19-2-4031, DARPA W911NF- 15-1-0543, J.P. Morgan PhD Fellowship, and gifts from Allen Institute for AI.
REFERENCES
Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in chal- lenging terrains using egocentric vision. In CoRI, 2022. 3
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't ' just assume; look and answer: Overcoming priors for visual question answering. In CVPR, 2018.
Arjun Reddy Akula, Soravit Changpinyo, Boqing Gong, Piyush Kumar Sharma, Song-Chun Zhu, and Radu Soricut. Crossvqa: Scalably generating benchmarks for systematically testing vqa generalization. In EMNLP, 2021.
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents. arXiv, 2018. 8, 16
Somrita Banerjee, James Harrison, P Michael Furlong, and Marco Pavone. Adaptive meta-learning for identification of rover-terrain dynamics. arXiv, 2020.
Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew Davison, Jia Deng. Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv, 2020a.
Dhruy Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, A. Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv, 2020b. 4
Josh C. Bongard, Victor Zykov, and Hod Lipson. Resilient machines through continuous self- modeling. Science, 2006.
Byron Boots, Arunkumar Byravan, and Dieter Fox. Learning predictive models of depth camera & manipulator from raw execution traces. In ICRA, 2014. 3
Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, and Aniruddha Kembhavi. Robustnav:
Towards benchmarking robustness in embodied navigation. In ICCV, 2021. 1
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navi- gation in 3d environments. In ECCV, 2020.
Rohan Chitis,, Leslie Pack Kaelbling, and Tomas Lozano-Peree Learning quickly to plan quickly using modular meta-learning. In ICRA, 2019. 3
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. In NeurIPS Workshop on Deep Learning, 2014. 5
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean- Saan-Baptiste Mouret. Robots that can adapt like animals. Nature, 2015. 3
Samyak Datta, Oleksandr Maksymets, Judy Hoffman, Stefan Lee, Dhruv Batra, and Devi Parikh. Integrating egocentric localization for more realistic point-goal navigation agents. In CoRI, 2021. 8, 17
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-real embodied ai platform. In CVPR, 2020. 1, 4, 7,9
10

---

Published as a conference paper at ICLR 2023
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large- scale embodied ai using procedural generation. arXiv, 2022. 7.9
Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Manipulathor: A framework for visual object manipulation. In CVPR, 2021.
Ben Evans, Abitha Thankaraj. and Lerrel Pinto. Context is everything: Implicit identification for dynamics adaptation. ICRA, 2022.
Kuan Fang, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Learning generalizable skills via automated generation of diverse tasks. In RSS, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, In ICML, 2017. 7,7
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. In NeurIPS Dataset, 2021. 1
Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel LK Yamins, James J DiCarlo, Josh MeDermott, Antonio Torralba, al. The threedworld transport challenge: A visually guided task-and-motion planning benchmark towards physically realistic embodied In ICRA, 2022. 1
Abhishek Gupta, Russell Mendonca, Yuxuan Liu, P. Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In NeurlPS, 2018.
David Hall, Ben Talbot, Suman Raj Bista, Haoyang Zhang, Rohan Smith, Feras Dayoub, and Niko Sünderhauf. The robotic vision scene understanding challenge. In arXiv preprint arXiv:2009.05246, 2020. 1
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and Juan Carlos Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demon- stration. In CVPR, 2019.
Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari. Svetlana Lazebnik, Ali Farhadi, Alexan- der G Schwing, and Aniruddha Kembhavi. Two body problem: Collaborative visual task comple- tion. In CVPR, 2019. 1
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effec- tive: Clip embeddings for embodied ai. In CVPR, pp. 14829-14838, 2022. 14,
Eric Kolve, Roozbeh Mottaghi. Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. In arxiv, 2017. 2
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and-languag navigation with dense spatiotemporal grounding. In EMNLP, 2020.
Ashish Kumar, Zipeng Fu. Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. In RSS, 2021. 3,
Ashish Kumar, Zhongyu Li, Jun Zeng, Deepak Pathak, Koushil Sreenath, and Jitendra Malik. Adapt- ing rapid motor adaptation for bipedal robots. In IROS, 2022. 3
11

---

Published as a conference paper at ICLR 2023
Thomas Lew, Apoorva Sharma, James Harrison, Andrew Bylard, and Marco Pavone. Safe active dynamics learning and control: sequential exploration-explottan exploitation framework. IEEE Transac- tions Robotics, 2022.
Chengshu Li, Fei Xia, Roberto Mlartin-Martnn Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al. igibson 2.0: Object-centric simula- tion for robot learning of everyday household tasks. In CoRL. 2021.
Shangda Li, Devendra Singh Chaplot, Yao-Hung Hubert Tsai, Yue Wu, Louis-Philipee Morency, and Ruslan Salakhutdinov. Unsupervised domain adaptation for visual navigation. arXiv, 2020a.
Shunkai Li, Xin Wang, Yingdian Cao, Fei Xue, Zike Yan, and Hongbin Zha. Self-supervised deep visual odometry with online adaptation. In CVPR, 2020b. 3
Antonio Loquercio, Elia Kaufmann, René Ranftl, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Learning high-speed flight in the wild. In Science Robotics, 2021. 3
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen, Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. ACL, 2019.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, P. Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. In ICLR, 2018. 3, 15, 16
Michael O'Connell, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-fly enables rapid learning for agile flight in strong winds. In Science Robotics, 2022. 3
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In AAAI, 2022.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019.
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In RSS, 2020. 3
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research. In ICCV, 2019. 2, 4, 21
Neal Seegmiller, Forrest Rogers- Marcovvva Greg Miller, and Alonzo Kelly, Vehicle model iden- tification by integrated prediction error minimization. The Inil. J. of Robotics Research, 2013.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
Xingyou Song, Yuxiang Yang, Krzysztof Choromanski, Ken Caluwaerts, Wenbo Gao, Chelsea Finn, and Jie Tan. Rapidly adaptable legged robots via evolutionary meta-learning. IROS, 2020. 3
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants rearrange their habitat. In NeurlPS, 2021.
12

---

Published as a conference paper at ICLR 2023
J. K Tenry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sulivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall Williams, Yashas Lokesh, Ryan Sullivan, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. In arXiv, 2020. 2, 19
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Vandi Verma, Geoffrey Gordon, Reid G. Simmons, and Sebastian Thrun. Real-time fault diagnosis [robot fault diagnosis]. IEEE Robotics & Automation Magazine, 2004. 3
Saim Wani, Shivansh Patel, Unnat Jain, Angel X. Chang, and Manolis Savva. Multi-on: Bench- marking semantic map memory using multi-object navigation. NeurlPS, 2020.
Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. Allenact: framework for embodied research. In arXiv, 2020. 4, 7, 15
Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrange- ment. In CVPR, 2021.
Erik Wijmans, Abhishek Kadian, Ari S. Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near- serrr-erfect pointgoal navigators from 2.5 billion frames. In ICLR, 2020.
Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In CVPR, 2019. 1, 2, 7,
Fei Xia, William Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexander Toshev, Roberto Martín-Martin, and Silvio Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. In ICRA, 2020. 21
Wenhao Yu, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. In RSS, 2017.
Wenhao Yu, C. Karen Liu, and Greg Turk. Policy transfer with strategy optimization. In ICLR, 2019. 3
Wenhao Yu, Jie Tan, Yunfei Bai, Erwin Coumans, and Sehoon Ha. Learning fast adaptation with meta strategy optimization. IEEE Robotics and Automation Letters, 2020. 1,
Kuo-Hao Zeng, Luca Weihs, Ali Farhadi, and Roozbeh Mottaghi. Pushing it out of the way: Inter- active visual navigation. In CVPR, 2021. 1, 2, 7, 15
Dingqi Zhang, Antonio Loquereio, Xiangyu Wu, Ashish Kumar, Jitendra Malik, and Mark W Mueller. A zero-shot adaptive quadcopter controller. In arXiv, 2022. 3
Yuan Zhang, Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. In NAACL, 2019.
Wenxuan Zhou, Lerrel Pinto, and Abhinav Kumar Gupta. Environment probing interaction policies. In ICLR, 2019.
13

---

Published as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIAL
A GENERALIZATION OF AAP TO UNSEEN DRIFTS
We provide an example here to illustrate how our AAP generalizes to unseen drifts during the testing stage. In the training stage, the rotation actions cover all degrees
{-1500 = 15e, -100 + 15 ..... 150 15,, 180 15 despite with small drifts d, 155 our AAP observed all possible rotation outcomes over [-100,, 180%]. Therefore, during example, policy recognize the effect of a' rotate (30% + drifssists) is equivalent to the effect of another action a²: rotate - 150% which was seen in the training stage. Likewise, since the movement magnitude d (0.05 + 0.05, 0.15 + 0.05, 0.25 + 0.05) performs well when the movement magnitude is within this range. This example highlights the core technical contributions, namely the action-impact embedding and OI head, in this work. With the action-impact embedding and OI head, we prevent the policy from remembering the action semantics, and allow it to focus on modeling the effect of actions.
testing stage, taking d; 180 as an
cover [0, 0.3). our model
B TRAINING PIPELINE, HYPERPARAMETERS, AND TIME COMPLEXITY
The training pipeline for forward pass and backward update is shown in the Fig. 8. During training, we use the Adam optimizer and an initial learning rate 3e-4 that linearly decays to 0 over 75M and 300M steps for the two tasks, respectively. We set the standard RL reward discounting parameter y to 0.99, Agae to 0.95, and number update steps to 128 for Lrro- The a for Lioeward is set to 1. Finally, we train the policy for 75M and 200M steps for PointNav and ObjectNav respectively and evaluate the model every 5 million steps.2
Time complexity. Theoretically, the come-compyexity for the OI transformer head is O(n²), where the n the number of actions in the action space. However, as the number of actions is usually fairly small, our AAP doe snot suffer from the problems frequently plague transformer-bar based models with their n² complexity in the sequence length. At runtime, we evaluate models by personal desktop with Intel i9-9900K CPU, 64G DDR4-3200 RAM, Nvidia RTX 2080 Ti GPUs. The framework (w/ our AAP) spends 2 35 minutes evaluating 1.8k val episodes with 5 parallel processes on the Point Navigation task. The average episode length is 117. As result, the FPS (or interaction
second) is Rs 100.3. During the training phase, we used an AWS machine with 48 vCPUs, 187G RAM, and Nvidia Tesla T4 GPUs train the policy. The FPS (or interaction per second) is z 400 for our AAP.
Forward
Observatio
Backward from LPPO
State Encoder
Backward from forward
Task Goal
Policy Network
Ol
a 1 LPPO
E-1 Visual Obseraaiin
Head
FF-layer
Lfrrward MSE(AS, A5²)
Action-Impact Encoder
Axim
Task Goal
0,-
Figure 8: Training Pipeline. The forward pass is in black color, the backward from Lppo is in red color, and the backward from Liorward is in orange color.
C MODEL ARCHITECTURE DETAILS
Ours. We follow (Khandelwal et al., 2022) and set the dimension of the belief b, and hidden dimension to 512, and the output of ResNet is 1568-dim goal-conditioned visual embedding. In
14

---

Published as a conference paper at ICLR 2023
the Action-Impace Encoder, the dimension of r* is 1024-dim for the Point Navigation and 3136 for the Object Navigation. The goal embedding is 8-dim, the hidden for the GRU is 128-dim, and the action embedding is also 128-dim. In the policy network, the GRU is 512-dim. The Ol-Heat consists of 6 layers of Transformer Encoder with 16 heads, 512-dim hidden, and 512-dim output.
EmbCLIP (Khandelwal al., 2022). The model architecture is the same one used in (Khandelwal et al., 2022) which is open-sourced on AllenAct (Weihs et al., 2020)².
Meta-RL (Wortsman et al., 2019). The model architecture is the same as the used in (Khan- delwal et al., 2022), except that we add a MLP after the belief b to predict the agent-state change for meta-leaming. Instead of performing meta-update every G-steps, we perform meta-update every 20-steps, because the maximum allowed number of steps per episode was increased from 200 to 500 in our navigation tasks. The meta-update learning rate was set to 10 (same as (Wortsman al., 2019)).
RMA (Kumar et al., 2021). The main model architecture is the same as the one used in (Khandelwal et al., 2022), except that we add a module to learn latent code and an adaptation module to perform online adaptation. More specifically, we follow (Kumar et al., 2021) to encode the latent code z (8-dim) by 3 layer MLP (2 256 + 128 - 8) from motion and rotation drifts in the first training stage. In the adaptation stage, we follow (Kumar et al., 2021) to learn the adaptation module to predict latent code from 16 past agent-states. In details, the adaptation module consists of an MLP embedding (3 + 14 + 32 + 32) and a 3-layer ID Conv (32, 32, 8, 2] + [32, 32, 3, 1] + [32, 8, 3, 1)) over 16 past states.
Mhoell-Based (Zeng et al., 2021). The model architecture is the same as our proposed model, except that the model produces action embeddings from the predicted state changes. In this way, this baseline can utilize the next state predictions associated with different actions for planning. In addition, it is important to note that the Policy Network for this baseline is the linear actor-critic, instead of the proposed OI-Head.
D MORE QUANTITATIVE RESULTS
We show more quantitative results for PointNav and ObjectNav in Fig. 9, Fig. 10, Fig. 11, Fig. 12, and Fig 13 by SPL, Episode Length, Reward, Distance to Target, and soft-SPLAs shown, the SPL, soft-PLL and Episode Length of competing baselines achieve better results than AAP at the modest drifts (d, + 15., 330)). but they perform significantly worse at more severe drifts. Note that our AAP imitially has no idea about the state-changes resulting from each action, it has to spend more time exploring the effects of actions the beginning of new episode. As result, results in a longer episode length comparing to the baselines with seen drifts. On the other hand, our AAP achieves consistent SPL, Episode Length, Reward, Distance to Target, and soft-SPL across all the drift changes and outperforms baselines remarkly. It is because AAP does not memorize action semantics, but relies on its experiences in the inference time to embed the action embeddings E on-the-fly. To collect useful experiences, the agent has explore the environment and update the embeddings accordingly. Thereby, our AAP takes more time to understand the state-changes, while the baselines can achieve better results based on its learned action semantics in the scenarios with the modest drifts.
Comparison to (Adapted) Model-Based Meta-RL. (Nagabandi al., 2018). Unfortunately. direct comparison to (Nagabandi et al., 2018) is not feasible as, (a) we focus on point navigation and ob- clustered scene, but (Nagabandi et al., 2018) focuses on locomotion control for a simple straight or curve line movement, (b) our main observation is visual observation, but (Naga- bandi et al., 2018) uses agent's 's state, and (c) we don't 't use model prediccce controller to rollout H time horizon future to make decision, because the agent cannot access rewards or distance to the goal during the testing time, so it is not straightforward implement a simple objective function in MPC for point navigation or object navigation task (we would argue that doing so is a research topic in itself).
ject navigation
We use the open-sourced code to train EmbCLIP /ttpps///gttuub.coom/a//// a 11lenact//b /bl ht basel wnw.:aaaseeeeeeeeeeeeeeeeeee https/////eeeeeeeeeeeee/rtttttie
el
15

---

Published as a conference paper at ICLR 2023
4% + 0.0Sm
0.1m
d 0.2m
d 0.4m
1.0
Navigation
0.8
Point
SPL
0.6
0.4
0.2
100
180
100
180
100
180
0.5
Navigation
Object
SPL
0.2
0.1
0.0
100
180 20
100
180 20
100 di
18020
100
180
d7
de
di
AAP (Ours)
EmbCLIP
Meta-RL
RMA
Model-Based
Figure 9: Quantitative result using the SPL metric. The SPI. (Anderson al., 2018) is defined
Ln, as N m==1 Sn " where N is the number of episodes, S,, denotes binary indicator of success in the episode n, Pn is the path length, and L., is the shortest path distance in episode n. Top: PointNav evaluation. Bottom: ObjectNav evaluation. We compare the proposed AAP with baselines, including EmbCLIP, Meta-RL, RMA, and Model-Based. We measure the SPL over different drifts, including d'm 0.05 +0.1, 0.2, 0.4] and d; (=1 +30 445 +900, 1335, 180%).
d% 0.05mm
d% 0.1m
da 0.2m
da 0.4m
Epásode Length
400-
Navigation
Point
300
300
20
100
180 20
100
180 20
100
180 20
100
180
400
Episode Length
350
Object
Navigation
300
250
150
20
100
180 20
100
180 20
100 d
18020
100
180
d
d
AAP (Ours)
EmbCLIP
RMA
Model-Based
Figure 10: Quantitative result using the Episode Length metric. Top: PointNav evaluation. Bottom: ObjectNav evaluation.
However, to have closer implementation based on the idea proposed in (Nagabandi et al., 2018 (), we combine our model-based baseline with the meta-RL baseline by employing the agent state prediction to perform meta-learning on the model-based module. The results for point navigation are shown in Fig. 13 in the green color by Soft-SLL. Although this (Adapted) Model-Based Meta- RL outperforms EmbCLIP slightly facing larger drifts, it is still not able to overcome the severe drifts. It, again, highlights the effectiveness our AAP with two major technical contributions, the action-impact embedding and the OI head, proposed in this work.
16

---

Published as a conference paper at ICLR 2023
d + 0.05m
d = + 0.1m
d = 0.2m
d = 0.4m
12.5
Novigation
10.0
Reward
Puint
5.0
0.0
22.5
-5.0
20
1001
180 20
100
180 20
100
180.20
100
180
4.0
3.0
Object Navigation
2.0
Reward
1.0
0.0
-1,0
-2.0
20
100
180 20
18030
de
d' EmbCLIP
di
d
AAP (Ours)
Meta--LL
RMA
Model-Basse Based
Figure 11: Quantitative result using the Reward metric. Top: PointNav evaluation. Bottom:
ObjectNav evaluation.
d + 0.05m
di = + 0.1m
di 0.2m
d = 0.4m
5.0
Distunce to Target
Navigation
Point
3.0
2.0
1.0
0.0
20
100
180
20
100
180 20
100
180 20
100
180
3.0
Distunce to Target
Navigation
2.6
Object
2.2
2.0
20
100
180 20
100 d
180
100
18020
180
d
d'
d
AAP (Ours)
EmbCLIP
Meta-RL
RMA
Model-Based
Figure 12: Quantitative result using the Distance to Target metric. Top: PointNav evaluation.
Bottom: ObjectNav evaluation.
Point Navigation
Object Navigation
dt = 0.05m
d= = 0.2m
d = + 0.05m
d = 0.2m
0.5
0.4
SPL.-SLL
0.4
0.2
0.0
20
100
180
20
100
180
20
180 20
100
180
d'
d,
dt
AAP (Ours)
EmbCLIP
Mosel-Based Meta-RL RL (adapted from Nagabandi et al. (2018))
Figure 13: Quantitative result using the soft-SPL. soft-SPL (Datta et al., 2021) is defined as 1 EN (1 denote the (geodesic) distances to target upon termination and start in the episode n, Pn is the path length, and L,, is the shortest path distance in episode n. Left: PointNav evaluation. Right: ObjectNav evaluation.
dn vemiratio
dm. ma..F.......>> where N is the number of episodes, d, de.semination and stat
Jn
17

---

Published as conference paper at ICLR 2023
Top-Down View (Progress)
dn = 0.2,4,22
PointNav
Rotate(-900
Move(0.25)
Move
End
hits bounce back
Top-Down View (Progress)
0.4. d.
PointNav
Rotate(-60")
Move(0.25)
Rotale(-1507
Move 25)
Top-Down View (Progress)
O.I.d,
Disabled Rotato( 11" [307,60 ,66",609"""" 90' 111.110000077
PointNav
Rotate(- 30')
Rotate( 180))
Rotate(-1200 120"
Move(0.0S)
Top-Down View (Progress)
0.2,
30", Disebled Rotate(
30' 60' 90' 100")
Objec cav
Mover0.15)
Mcve(0.11)
Move(0.05)
Move 05)
Top-Down View (Progress)
1,2, d. 30", Disabled Rotate(
30" 60"
90"
120" 150"
ObjectNav
Rotate 180")
Rotate
Move(0.05)
Move(0.05)
Top-Don Top-Down (Progress)
dg 0.05, 60"
ObjectNav
Move(U.15)
Rotate(90")
Move(0.22)
Move(0.25)
Figure 14: Qualitative Results. Examples of PointNav (top tree rows) and ObjectNav (bottom three rows). The agent adapts by rotating in the other direction compensate for the disabled actions. The first example shows the agent adopts smaller movement magnitude to avoid the collision. The second example shows smooth moves by right-turn, move, another left-turn. and final move to dodge the white table. The third example shows the agent uses three left turns with different angles make a disabled right turn. The fourth example shows the agent slows its movement magnitude as approaches the target object in the clustered area. The fifth example shows the agent uses two left- turn with different angles make disabled right turn to find the target object. The final example shows the agent uses a larger movement magnitude to move toward the target object in relatively open area.
E QUALITATIVE RESULTS
We show more qualitative results in PointNav and ObjectNav in Fig. 14, The first example shows the agent adopts a smaller movement magnitude to avoid the collision. The second example shows smooth moves
right-turn, move, another left-turn, and final move to dodge the white table. The third example shows the agent uses three left turns with different angles to make disabled right tur.. The fourth example shows the agent slows its movement magnitude approaches the target
18

---

Published as a conference paper at ICLR 2023
Point Navigation
Object Push
Action: Accelention Agent
0.8
Agent
0.2
Target Location
Ball
Target Location
Agent
wah nont Falze
Object State
+0.82 14, 0.00 -1 2321
Agent State 9.34 an
Agent State
Drifts and Noise 5 201,M -0.00 +3 00, no -0.00 -0.00
Drifts and Noise
0.99 n. 0.23 00.77
Figure 15: MPE environment in PettingZoo. We modify the MPE environment to simulate Point- Nav and ObjectPush tasks. The goal for PointNav is move the agent to the target location and the goal for ObjectPush is to move the agent to push the ball to the target location. The action space consists of 3 different accelerations towards directions.
object in the clustered area. The fifth example shows the agent uses two left-turn with different angles make disabled right turn to find the target object. The final example shows the agent uses a larger movement magnitude to move toward the target object in a relatively open area.
F RESULTS ON THE MODIFIED PETTINGZOO ENVIRONMENT
We modify the PettingZoo (Terry et al., 2020) to verify the effectiveness of our AAP in dif- ferent environment with state observations only. We modify the MPE environment to simulate PointNav and ObjectPush tasks, shown in Fig. 15. The MPE environment provides simple sim- ulation of collision force when objects are too close each other. The goal for PointNav to move the agent to the target location by applying acceleration to the agent. The goal for Ob- jectPush is to move the agent to push the ball to the target location by applying acceleration to the agent. The agent has to collide to the ball to perform the push. As shown in the figure, there are 12 actions in action space, including Accelerate (magæ, mag,), where (magz, mag,)= 012 OJ, +0.5, Oj. [=0,8 0]. [0, =0.2]. [0, =0.5]. [0, #0.8] }. corresponding to 3 different accelerations towards different directions. For PointNav, the state space 6-dim, including agent's position (Pz,Py). Py). agent's velocity Uz, By. GPS sensor (Apx, Apy); for ObjectPush, the state space is 10-dim, including agent's position (Pz, Py). agent's velocity (Uz, Vy), object's position (0x, oy), object's velocity (o..: or,)) GPS sensor (Apz: Apy). Our drifts setting for training and inference stage are formulized as follows,
During training, At the start of training episode, we sample the rotation drift d, ~ U(-909, .90)), where (.,))) denotes uniform distribution.
During inference. At the start of an evaluation episode, we evaluate our AAP and baselines with an unseen rotation drift d; 120" 1135" 1150 1180).
With a rotation drift d,, the environment applies the d, the input action by rotating the acceleration direction:
cos(d,) (d) cos(d,))
sin(dr)
magenv
magmmpu
(I)
In this case, the actual direction of acceleration would be drifted according to the rotation drift dy. Models.
The details about AAP and the considered baselines are follows,
GRU simply uses 4-layer MLP ([stae + 64 + 64 - 64)) as the state-encoder to obtain a 64-dim hidden state h, applies RNN (GRU) to obtain 64-dim belief b, over h, and uses linear actor-critic head to predict an action a, via by.
LAC is variant of AAP that uses linear actor-critic head instead of our Ol head.
19

---

Published as a conference paper at ICLR 2023
4 = 120"
4 135*
df
de
1.0
Navigation
Success Rate
0.8-
Point
0.
0.4
20M
20M
20M
1.0
Success Rate
Object
Push
0.6
0.2
0.0
IM
20M 1M
20M IM
20M IM
20M
checkpoints
checkpoints
chockpoints
checkpoints
AAP (Ours)
Model-Based
GRU
LAC
Figure 16: Success Rate in MPE environment. Top: PointNav evaluation. Bottom: ObjectPush evaluation. We train each model by different random seeds and show the average success rate and 1x standard deviation in this figure.
4% + 0.05m
d
0.1m
d 0.2m
d 0.4m
1.0
Success Rate (SR)
Point Navigation
088
in Gibson
6.6
0.4
02
90
d
d
d
d
AAP (Ours)
EmbCLIP
Figure 17: Success Rate on Point Navigation in Habitat Environment. We compare the pro- posed AAP with EmbCLIP. The evaluation drifts unseen during the training stage, including d E (+0.05m, 1m,m, 0.2m, 0.4m) and d; E (=15 +00 145% +.90) } The experiments are conducted in Habitat-Lai vo. 2 on Point Navigation with Gibson v1.
Model-based is model-based forward prediction approach for the agent state-changes As asso- ciated with different actions. It further embeds the prediction into action embeddings. Finally, this baseline linear actor-critic make prediction.
Our AAP uses the same 4-layer MLP used GRU baseline to produce state encoding for pre- vious state and the current state, respectively. Then, same as our AAP used in AI2THOR, the action-impact encoder uses a RNN (GRU) to produces the n x 64-dim action impacts embedding E associated with different actions. Later, the model uses RNN (GRU) in the policy obtain 64-dim belief b, Finally the OI head operates on the concatenation of E and by to make a prediction.
Results. We train every model by PPO with the Adam optimizer and initial learning rate of 1e-3 that linearly decays to over 20M. We set the standard RL reward discounting parameter 7 to 0.99, Agac 0.95, and number of update steps 200 for Lrro- The a for Liorward is set 1. Finally, we train each model by 3 different random seeds and obtain the average success rate and 1 x standard deviation. We evaluate models every 1M steps (checkpoint). The evaluation results are presented in Fig, 16. As shown in the figure, our AAP performs consistently well across all rotation drifts both tasks in the modified MPE environment. The best baselines cannot even achieve any success rate when facing the extremest rotation drift d;; 180º It verifies that the effectiveness of the proposed AAP on a general reinforcement learning environment. We will release the code for this modified environment and experiments well.
20

---

Published as a conference paper at ICLR 2023
G RESULTS ON THE HABITAT ENVIRONMENT
To conduct the experiments in Habitat (Savva et al., 2019), we made a small change in the Habitat's Move_Forwaze action, where every Hove_Forware only moves the agent by 0.01m. If the (0. 25), the environment would move the agent by 25 times. In
action a' Move_ this way, we can implement the movement drifts as did A12-THOR. For the rotation action, we do not use the default TURN_RIGH, or TURN_LEFT action. We instead directly compute the quaternion implement the continuous rotation with rotation drifts. In this modified Habitat, we train the EmbCLII and our AAP on Point Navigation task in Gibson v1 (Xia et al., 2020). The simulator Habitat- 2..2.1.. The training settings are the same as the settings used in A12- THOR, including the same training drifts, same learning schedule, same optimization algorithm, and learning objective. During the evaluation, we evaluate the model on only movement drift d (=0.05m, mm.1m 0.2m. 0.4m) and rotation drift d; {+15,, =30 +55 +.0%). As shown in Fig. 17, our AAP performs consistently across all movement and rotation drifts. However, the EmbCLIP is struggling with larger drifts.
21


Reviews:
[{"review_id": "KVtNXIozmkn", "reviewer": "Reviewer_WkHK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to learning action embeddings from state changes. For embedded agents who use these learned action embeddings, they do not rely on action semantics that overfits to the action labels. This paper also shows that an order-invariant (OI) transformer head can help choose actions rather than based on action semantics. Experiments on the AI-THOR environment show the proposed method is robust to several out-of-distribution action drifts and can quickly avoid using disabled actions.", "summary_of_the_review": "This paper proposes a simple yet effective approach to improve robustness against action drifts. However, it misses the discussion of limitations of the proposed method. The description of the learned embedding and why an OI is needed is confusing.", "strength_and_weaknesses": "Strength\n- This paper proposes an approach that is useful to deal with action uncertainty in test time. This will add robustness to embedded agents.\n- Compared to data augmentation which is arbitrary, this paper shows a systematic approach to deal with action drifts.\n\nWeaknesses\n- The approach assumes that the disabled actions can be remapped to one of the state changes the agent has seen before in training in order to take the right actions. For rotations, it is possible to cover all rotations in training, but for translation, it depends on the step size, for the translation that is really out-of-distribution state changes, this approach still cannot recover from that. This is shown as the lower success rate for the 0.4m drift. The authors need to make this assumption clear to the reader.\n- One of the claims of the paper is that the embedding of state changes can help recover from action drift, but in the ablation study, LAC’s performance still drops when the drift increases which invalidates the claim of the paper. It is unintuitive why the action impact (i.e. state change) embedding doesn’t have an impact without OI.\n- The ablation doesn’t really show OI is helpful as AAP uses a transformer head but LAC uses a linear head. The difference in performance can be the difference in model capacity. A fairer comparison will be a transformer head with positional encoding.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7oblq2CXoT", "reviewer": "Reviewer_DBZD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose Action Adaptive Policy (AAP) to adapt to unseen effects of actions during inference of an embodied AI navigation task. Experiments show that AAP is effective at adapting to unseen action outcomes at test time, outperforming a set of strong baselines by a significant margin, and works with disabled actions. The paper takes care of 2 cases of drifts - rotation and movement.", "summary_of_the_review": "The paper content is a good contribution to SOTA.\n\nSuggestions:\n1. Make the abstract to the point - break large sentences into smaller form. Some result numbers and mention of PointGoal and ObjectGoal could also support overhaul.\n2. action-stability assumption - this could be elaborated and can use a abbreviation later.\n3. drift is a well known problem in SLAM community (loop closures) - how does this work leverage SOTA in those lines?\n4. what is the time-space complexity of the pipeline at execution time given possible actions?\n5. need some insights on how Fig 1. action commands of exact 30 degree rotate, 0.2 m translate passed\n6. How the agent is learning should be explained philosophically, possibly by an example\n7. Instead of SPL alone, other metrices can be tried out in future - soft-SPL, DTS, partial reward\n8. we use DD-PPO - explain\n9. What happens with fractional ground truth of rotation angles?\n\n \nMiscellaneous:\n1. Could have used line numbers in review for easy referral (i had to copy paste long lines)\n2. Unlike work in training robust - grammar\n3. General comment - break long sentences short for ease of reading\n4. Make the contribution part writing simple\n5. Fig. 2 should be elaborated more - choice of color...\n6. expand abbreviations when used first\n7. Choice of variable name / subscript - as input to produce o_0,t - does not look good and readability issue", "strength_and_weaknesses": "Strengths:\n1. The experimentation section is strong to support most of their claims.\n2. In their experiments, authors claim to outperform meta-learning approaches without requiring, computationally taxing, meta-learning training.\n3. Out-of-distribution generalization (avoiding disabled actions)\n4. For example, if the action a_i is Rotate(30◦) - this is a good example\n\nWeakness:\n1. The approach is not tested on (a) another sim environment like Gibson, MP3D AI Habitat (b) real robot - usually sim2real is not easy for this embodied AI tasks\n2. It seems defective wheel plays a major role in errors. It can be due to surface change, sensor failure, environment noise. Some error modelling technique could have been pursued.\n3. How is the learned RL model applicable to different robot type with different hardware sensors and motors? [A Limitation]\n4. Embedding is not explained clearly - internals\n5. The performance of the drift correction is tied up with the specific goal task at hand", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "49ZAnnyQs_", "reviewer": "Reviewer_uyLJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The focus of this paper is the problem of generalization to changes in the effect of actions, which is referred to as action drift. To address this problem, the authors propose using: (a) action embeddings which are learned functions of transitions observed under that action, (b) performing action selection via an order-invariant transformer head. In the training phase, the model is trained on the distribution of action drifts and is then tested on drifts equal to or greater than those it has been trained on (without further adaptation of the model weights). The drifts explored here are modifications to either translation or rotation actions.  \n\nThe proposed method is tested on the two tasks in AI2-THOR environment, where the proposed method significantly outperforms a variety of baseline methods, performing well even under OOD action drifts. ", "summary_of_the_review": "This paper proposed a method for fast adaptation to changes in the effect of actions during deployment. The effectiveness of the proposed method is well demonstrated in a series of experiments involving a variety of modifications to actions, strongly outperforming several relevant baselines. ", "strength_and_weaknesses": "The reported results are very strong, the authors test the proposed method to both modest and severe drifts, as well with multiple disabled actions. They compare the proposed method to a variety of relevant baselines; including model-based RL, meta-learning, and sim2real methods; significantly outperforming all of them in almost all experiments, with a greater performance gap on bigger drifts. \n\nThe proposed model is very well-motivated and sensible, and I appreciated the detailed ablation experiments demonstrating the importance of both action embeddings and order-invariant action selection. \n\nThe main weakness of the paper is that the proposed method is evaluated on only one environment, AI2-THOR. While using embodied AI environment makes sense given the motivation for the proposed method is in robotics applications, I don't see a reason why this method wouldn't be more generally applicable in RL. The paper would be stronger with the addition of experiments on other environments, e.g. Mujoco, and some even simpler environments where the authors could more easily interpret the learned action embedding and policies, as well as shed light on the limitations of the method. \n\nSome questions for the authors:\n1. How many environment interactions does it typically take to learn action semantics in a new environment?\n2. Do you have thoughts on how the proposed method could be adapted to continuous actions? \n3. How exactly is action selecting $m_i$ and how is $m_i$ computed? I assumed you're selecting the last memory embedding transitions made under action $a_i$, hence only transitions made under action $i$ influence the embedding $e_i$; however it would be good to make this less ambiguous in the paper. \n4. How well does the method perform in a non-stationary setting (i.e. where the action semantics change throughout the lifetime) or a changing distribution over actions (e.g. different drift sizes for each rotate action)? These questions are beyond the scope of the paper, but they seem like interesting application areas, and it would be good to get a sense of the method's limitations. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-X6r1GE-evw", "reviewer": "Reviewer_AaUJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of designing a reinforcement learning (RL) policy that lets the agents to adapt to changes in the form of rotation and movement drifts.\nThe manuscript describes a novel approach which includes an state encoder and an action impact encoder. The former is a standard state encoder where a visual encoder and an embedder for the task are concatenated. For this work state-of-the-art state encoders are used (Deitke et al. (2020), and Savva et al. (2019)).\nThe action-impact encoder is the new block proposed in this work. It receives two consecutive frames as well as the previous action, and its objective consists in producing a set of action embeddings that summarize the impact of the actions performed by the agent in an episode. Technically, this is done with a combination of a RNN and the state encoder. Finally, for the policy, a transformed-based approach is proposed. It is called the Order-Invariant (OI) head. Thanks to the transformer, this OI head is independent of the sequence of its inputs and analyzes the agent's belief together with the action embeddings to help it decide which action will have the greatest influence on achieving its objective.\nExperiments are provided in RoboTHOR framework in the AI2-THOR environment. Two navigation tasks are evaluated: PointNav and ObjectNav. Results show some improvements of the proposed approach compared to a set of baselines when the Success Rate metric is used, especially when the rotation drift appears.", "summary_of_the_review": "Overall, I think this is an interesting paper. The problem it proposes is relevant, and providing a clear experimental setup for the rest of the community would be of great value. However, there are some aspects that I did not find convincing. The comparison with baselines and other state-of-the-art models is poor. This is usually the case when a problem is totally new, but this would not be the case. I think more work should be done on this aspect, especially for a paper that aspires to be published in the ICLR.\nIn addition, as I have described, the results present some inconsistencies. I would ask the authors to address these in their feedback, so that a reasoned decision can be made regarding the manuscript.", "strength_and_weaknesses": "Regarding the strengths of the paper:\n\n+It is a well-written article. The editing work is excellent. The figures are of great quality, as well as being very informative. \n\n+Section 2 is approached effectively. Previous work is presented and it is explicitly stated why the proposed ideas are different.\n\n+The paper includes a clear formulation of the problem. I think it is right not to consider that the drifts observed during the test have not been observed during training. In this way it is possible to infer what generalization capacity the model has. However, I also see no problem in dealing with a scenario in which training and test share the drifts, if they are chosen randomly in both.\n\n+As for the model itself, I would point out that it uses encoders that are the state of the art (CLIP). The combination of functional blocks proposed in Figure 2 is also novel. In particular, I have not found previous work that combines a model such as the Action-Impact Encoder, with a policy network and the OI invariant head detailed in the paper.\n\n+Finally, it should be noted that the experiments have been carried out in a publicly available environment. Code will be released, and results will be easily replicated. I also appreciate the informative ablation study. From it, it is clear to me the impact of the OI head in the process.\n\nThe main weaknesses of the work are as follows:\n-Comparison with state of the art models is incomplete. The experimental evaluation offers a comparison with multiple baselines, bot none of them focuses on the problem addressed in the paper. In other words, the baselines have not been designed to tackle the rotation and movement drifts. This would explain their (low) performance for some settings. Possibly only RMA (Kumar et al., 2021) has been slightly adapted to explicitly deal with the proposed problem. It is actually the one reporting the best results for the ObjectNav task (even better than the approach in the paper for low rotation drifts). The meta-learning model of Nagabandi et al. (2018) could have been explored. Note that is is different from the meta-learning baseline proposed. Does the proposed model improves the performance of an state-of-the-art model that using meta-learning is able to optimize the model at the testing phase? This could be an interesting question to answer. This would help sell the paper's contributions. Overall, I'm missing a direct comparison with other models that explicitly consider the same problem.\n\n-The experimental evaluation shows the following inconsistencies:\n\na) Figure 9 and Figure 5 show different winners. I mean, for example, for the problem of ObjectNav, the approach presented in this paper reports the best SR, but when one changes the evaluation metric to the SPL, then the winners (for low rotation drifts) are the rest of the models. The two metrics are different, I know them quite well. But SPL is quite informative for the problem considered, and I am concerned that the proposed model is not able to approach to the optimal route, when the others do. The other well established soft-SPL could be used. Have the authors tried it?\n\nb) Overall, across all figures, it seems that the model is not affected by the rotation drifts. Even for the 180 degrees drift! The SR should necessarily decrease (slightly) when the agent is exposed to such high drifts. Moreover, to me, it is unclear how the results can be the ones they are, when during training the only seen drifts are +/- 0.05 and +/- 15 for the movement and rotation, respectively. The paper needs the better explain the results that are being reported.\n\nc) Figure 5 again, let's analyze one more time. For the ObjectNav experiment, it seems that the displacement drift does NOT affect the \nbaselines, if the rotation drift is kept at 20º. Here there is no clear improvement with respect to the rest of the models. This aspect is worrying, as this experiment is more relevant than PointNav. Moreover, the result (of the baselines) is better the more drift, for models that have not been trained for it. These results are confusing, and in my opinion do not allow to offer a clear conclusion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%