Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The focus of this paper is the problem of generalization to changes in the effect of actions, which is referred to as action drift. To address this problem, the authors propose using: (a) action embeddings which are learned functions of transitions observed under that action, (b) performing action selection via an order-invariant transformer head. In the training phase, the model is trained on the distribution of action drifts and is then tested on drifts equal to or greater than those it has been trained on (without further adaptation of the model weights). The drifts explored here are modifications to either translation or rotation actions.  

The proposed method is tested on the two tasks in AI2-THOR environment, where the proposed method significantly outperforms a variety of baseline methods, performing well even under OOD action drifts. 
strength_and_weaknesses: The reported results are very strong, the authors test the proposed method to both modest and severe drifts, as well with multiple disabled actions. They compare the proposed method to a variety of relevant baselines; including model-based RL, meta-learning, and sim2real methods; significantly outperforming all of them in almost all experiments, with a greater performance gap on bigger drifts. 

The proposed model is very well-motivated and sensible, and I appreciated the detailed ablation experiments demonstrating the importance of both action embeddings and order-invariant action selection. 

The main weakness of the paper is that the proposed method is evaluated on only one environment, AI2-THOR. While using embodied AI environment makes sense given the motivation for the proposed method is in robotics applications, I don't see a reason why this method wouldn't be more generally applicable in RL. The paper would be stronger with the addition of experiments on other environments, e.g. Mujoco, and some even simpler environments where the authors could more easily interpret the learned action embedding and policies, as well as shed light on the limitations of the method. 

Some questions for the authors:
1. How many environment interactions does it typically take to learn action semantics in a new environment?
2. Do you have thoughts on how the proposed method could be adapted to continuous actions? 
3. How exactly is action selecting $m_i$ and how is $m_i$ computed? I assumed you're selecting the last memory embedding transitions made under action $a_i$, hence only transitions made under action $i$ influence the embedding $e_i$; however it would be good to make this less ambiguous in the paper. 
4. How well does the method perform in a non-stationary setting (i.e. where the action semantics change throughout the lifetime) or a changing distribution over actions (e.g. different drift sizes for each rotate action)? These questions are beyond the scope of the paper, but they seem like interesting application areas, and it would be good to get a sense of the method's limitations. 
clarity,_quality,_novelty_and_reproducibility: The paper is overall very well-written and easy to read. In particular, I appreciated that the authors took a great effort to motivate and explain well each of the architectural choices. The figures are also very clear and easily understandable on their own.   

I'm not confident that the paper in its current form is reproducible. The authors committed to releasing the code, in the meantime, I suggest they include a detailed description of the architecture in the appendix (including a table of all hyperparameters and dimensions of the architecture). I also could not find a description of hyper-parameter sweeps or how well (and if) the hyper-parameters of baselines were tuned in these environments. 

Here are some further suggestions for improving the clarity of the paper:
- explain the meaning of shaded area in Fig 5, 7 and 9
- add variance measures in Table 1
summary_of_the_review: This paper proposed a method for fast adaptation to changes in the effect of actions during deployment. The effectiveness of the proposed method is well demonstrated in a series of experiments involving a variety of modifications to actions, strongly outperforming several relevant baselines.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors propose Action Adaptive Policy (AAP) to adapt to unseen effects of actions during inference of an embodied AI navigation task. Experiments show that AAP is effective at adapting to unseen action outcomes at test time, outperforming a set of strong baselines by a significant margin, and works with disabled actions. The paper takes care of 2 cases of drifts - rotation and movement.
strength_and_weaknesses: Strengths:
1. The experimentation section is strong to support most of their claims.
2. In their experiments, authors claim to outperform meta-learning approaches without requiring, computationally taxing, meta-learning training.
3. Out-of-distribution generalization (avoiding disabled actions)
4. For example, if the action a_i is Rotate(30◦) - this is a good example

Weakness:
1. The approach is not tested on (a) another sim environment like Gibson, MP3D AI Habitat (b) real robot - usually sim2real is not easy for this embodied AI tasks
2. It seems defective wheel plays a major role in errors. It can be due to surface change, sensor failure, environment noise. Some error modelling technique could have been pursued.
3. How is the learned RL model applicable to different robot type with different hardware sensors and motors? [A Limitation]
4. Embedding is not explained clearly - internals
5. The performance of the drift correction is tied up with the specific goal task at hand
clarity,_quality,_novelty_and_reproducibility: Clarity:
This is in general, a well written paper.

Quality:
The technical contribution is sound and suitable for ICLR.

Reproducibility:
'We will make the code and models for this work publicly available' - this declaration is to push SOTA - so will be reproducible then based on details given in paper body.

Novelty:
The enlisted contributions are indeed novel in this field of embodied AI (visual navigation tasks).
summary_of_the_review: The paper content is a good contribution to SOTA.

Suggestions:
1. Make the abstract to the point - break large sentences into smaller form. Some result numbers and mention of PointGoal and ObjectGoal could also support overhaul.
2. action-stability assumption - this could be elaborated and can use a abbreviation later.
3. drift is a well known problem in SLAM community (loop closures) - how does this work leverage SOTA in those lines?
4. what is the time-space complexity of the pipeline at execution time given possible actions?
5. need some insights on how Fig 1. action commands of exact 30 degree rotate, 0.2 m translate passed
6. How the agent is learning should be explained philosophically, possibly by an example
7. Instead of SPL alone, other metrices can be tried out in future - soft-SPL, DTS, partial reward
8. we use DD-PPO - explain
9. What happens with fractional ground truth of rotation angles?

 
Miscellaneous:
1. Could have used line numbers in review for easy referral (i had to copy paste long lines)
2. Unlike work in training robust - grammar
3. General comment - break long sentences short for ease of reading
4. Make the contribution part writing simple
5. Fig. 2 should be elaborated more - choice of color...
6. expand abbreviations when used first
7. Choice of variable name / subscript - as input to produce o_0,t - does not look good and readability issue
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach to learning action embeddings from state changes. For embedded agents who use these learned action embeddings, they do not rely on action semantics that overfits to the action labels. This paper also shows that an order-invariant (OI) transformer head can help choose actions rather than based on action semantics. Experiments on the AI-THOR environment show the proposed method is robust to several out-of-distribution action drifts and can quickly avoid using disabled actions.
strength_and_weaknesses: Strength
- This paper proposes an approach that is useful to deal with action uncertainty in test time. This will add robustness to embedded agents.
- Compared to data augmentation which is arbitrary, this paper shows a systematic approach to deal with action drifts.

Weaknesses
- The approach assumes that the disabled actions can be remapped to one of the state changes the agent has seen before in training in order to take the right actions. For rotations, it is possible to cover all rotations in training, but for translation, it depends on the step size, for the translation that is really out-of-distribution state changes, this approach still cannot recover from that. This is shown as the lower success rate for the 0.4m drift. The authors need to make this assumption clear to the reader.
- One of the claims of the paper is that the embedding of state changes can help recover from action drift, but in the ablation study, LAC’s performance still drops when the drift increases which invalidates the claim of the paper. It is unintuitive why the action impact (i.e. state change) embedding doesn’t have an impact without OI.
- The ablation doesn’t really show OI is helpful as AAP uses a transformer head but LAC uses a linear head. The difference in performance can be the difference in model capacity. A fairer comparison will be a transformer head with positional encoding.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and the proposed solution is neat to improve the robustness of embodied agents. But why does the action-order encoder needs to generate the embedding for each $a^i$ where $i=0 \cdots n$? For any state change, it should only generate one embedding.

The idea proposed in the paper is similar to the action remapping idea (e.g. Edwards et al.) that associates latent actions (here the state changes) to the corresponding correct actions. 
- Edwards et al. Imitating latent policies from observation. ICML, 2019.

The training parameters are available in the appendix for reproducibility. But I cannot find details about the network, e.g. the dimensions of the encoder output, in the paper. No other supplemental material is provided to help derive this information. It may be hard to reimplement the networks without this information.

summary_of_the_review: This paper proposes a simple yet effective approach to improve robustness against action drifts. However, it misses the discussion of limitations of the proposed method. The description of the learned embedding and why an OI is needed is confusing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper addresses the problem of designing a reinforcement learning (RL) policy that lets the agents to adapt to changes in the form of rotation and movement drifts.
The manuscript describes a novel approach which includes an state encoder and an action impact encoder. The former is a standard state encoder where a visual encoder and an embedder for the task are concatenated. For this work state-of-the-art state encoders are used (Deitke et al. (2020), and Savva et al. (2019)).
The action-impact encoder is the new block proposed in this work. It receives two consecutive frames as well as the previous action, and its objective consists in producing a set of action embeddings that summarize the impact of the actions performed by the agent in an episode. Technically, this is done with a combination of a RNN and the state encoder. Finally, for the policy, a transformed-based approach is proposed. It is called the Order-Invariant (OI) head. Thanks to the transformer, this OI head is independent of the sequence of its inputs and analyzes the agent's belief together with the action embeddings to help it decide which action will have the greatest influence on achieving its objective.
Experiments are provided in RoboTHOR framework in the AI2-THOR environment. Two navigation tasks are evaluated: PointNav and ObjectNav. Results show some improvements of the proposed approach compared to a set of baselines when the Success Rate metric is used, especially when the rotation drift appears.
strength_and_weaknesses: Regarding the strengths of the paper:

+It is a well-written article. The editing work is excellent. The figures are of great quality, as well as being very informative. 

+Section 2 is approached effectively. Previous work is presented and it is explicitly stated why the proposed ideas are different.

+The paper includes a clear formulation of the problem. I think it is right not to consider that the drifts observed during the test have not been observed during training. In this way it is possible to infer what generalization capacity the model has. However, I also see no problem in dealing with a scenario in which training and test share the drifts, if they are chosen randomly in both.

+As for the model itself, I would point out that it uses encoders that are the state of the art (CLIP). The combination of functional blocks proposed in Figure 2 is also novel. In particular, I have not found previous work that combines a model such as the Action-Impact Encoder, with a policy network and the OI invariant head detailed in the paper.

+Finally, it should be noted that the experiments have been carried out in a publicly available environment. Code will be released, and results will be easily replicated. I also appreciate the informative ablation study. From it, it is clear to me the impact of the OI head in the process.

The main weaknesses of the work are as follows:
-Comparison with state of the art models is incomplete. The experimental evaluation offers a comparison with multiple baselines, bot none of them focuses on the problem addressed in the paper. In other words, the baselines have not been designed to tackle the rotation and movement drifts. This would explain their (low) performance for some settings. Possibly only RMA (Kumar et al., 2021) has been slightly adapted to explicitly deal with the proposed problem. It is actually the one reporting the best results for the ObjectNav task (even better than the approach in the paper for low rotation drifts). The meta-learning model of Nagabandi et al. (2018) could have been explored. Note that is is different from the meta-learning baseline proposed. Does the proposed model improves the performance of an state-of-the-art model that using meta-learning is able to optimize the model at the testing phase? This could be an interesting question to answer. This would help sell the paper's contributions. Overall, I'm missing a direct comparison with other models that explicitly consider the same problem.

-The experimental evaluation shows the following inconsistencies:

a) Figure 9 and Figure 5 show different winners. I mean, for example, for the problem of ObjectNav, the approach presented in this paper reports the best SR, but when one changes the evaluation metric to the SPL, then the winners (for low rotation drifts) are the rest of the models. The two metrics are different, I know them quite well. But SPL is quite informative for the problem considered, and I am concerned that the proposed model is not able to approach to the optimal route, when the others do. The other well established soft-SPL could be used. Have the authors tried it?

b) Overall, across all figures, it seems that the model is not affected by the rotation drifts. Even for the 180 degrees drift! The SR should necessarily decrease (slightly) when the agent is exposed to such high drifts. Moreover, to me, it is unclear how the results can be the ones they are, when during training the only seen drifts are +/- 0.05 and +/- 15 for the movement and rotation, respectively. The paper needs the better explain the results that are being reported.

c) Figure 5 again, let's analyze one more time. For the ObjectNav experiment, it seems that the displacement drift does NOT affect the 
baselines, if the rotation drift is kept at 20º. Here there is no clear improvement with respect to the rest of the models. This aspect is worrying, as this experiment is more relevant than PointNav. Moreover, the result (of the baselines) is better the more drift, for models that have not been trained for it. These results are confusing, and in my opinion do not allow to offer a clear conclusion.

clarity,_quality,_novelty_and_reproducibility: Novelty: The proposed model is novel. I don't think this is a problem. I have already described in my review the strengths, and the architecture described in the manuscript seems not to have been tried before. In addition, the proposed problem is also novel. If the paper is accepted, there may be more authors exploring the same problem in the AI2THOR environment. For this, the experimental setup must be crystal clear.

Clarity and Quality: The article is well written, and the quality of the figures deserves a special mention.


Reproducibility:
-Code will be released. Moreover, publicly available environments are used (RoboTHOR in the AI2-THOR environment).
summary_of_the_review: Overall, I think this is an interesting paper. The problem it proposes is relevant, and providing a clear experimental setup for the rest of the community would be of great value. However, there are some aspects that I did not find convincing. The comparison with baselines and other state-of-the-art models is poor. This is usually the case when a problem is totally new, but this would not be the case. I think more work should be done on this aspect, especially for a paper that aspires to be published in the ICLR.
In addition, as I have described, the results present some inconsistencies. I would ask the authors to address these in their feedback, so that a reasoned decision can be made regarding the manuscript.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: Enable robots to adapt to significant changes in their action space by embedding the impact of an action (the change in the observations caused by the action) and providing those embeddings when choosing the next action.
Strengths: A novel idea that is well motivated with clear experiments that demonstrate its value. Adapting to changes in the environment is of interest to the community as a whole and this method could see other uses. The authors addressed reviewer concerns, including new experiments such as running on Habitat.
Weaknesses: The method is limited in its ability to help with large drifts. It would be good if the manuscript was clearer about this and other limitations of the method.
justification_for_why_not_lower_score: The method presented has several positive attributes that make it of wide interest to the community. It is well motivated; it stands to reason that paying attention how your actions changed your observations may improve your ability to adapt to changes in how those actions impact the environment. It clearly works well across datasets. And it could be applied to other problems, it is not restricted to RL for robotics. Finally, reviewers were unanimous in their support.
Decision: Accept: notable-top-5%