Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of multi-task policies failing in the presence of distribution shifts. Authors disambiguate between goal and policy failures as the agent failing because it can’t correctly infer the desired goal or because it doesn’t know how to take an action toward achieving the goal. This work incorporates human feedback to diagnose the what versus how failure for adapting policies under distribution shift. 

This work proposes a framework to incorporate human feedback by leveraging agents’ intermediate task representation.  Experiments with human users across discrete and continuous control tasks show that (i) intermediate task representations help human users in identifying failure mode; and (ii) subsequent empirical performance gains can be obtained in those tasks by leveraging human feedback. 
strength_and_weaknesses: **Strengths** 
- The paper considers a very important problem of incorporating human feedback to improve machine learning systems, in particular, to improve the performance of imitation learning policies.
- Experimental results are interesting when compared with a naive baseline. However, the paper misses a crucial baseline (see 2nd bullet in weaknesses).  

**Weaknesses** 
- The paper is a bit hard to parse. Mainly the organization in Sections 4 and 5 is a bit confusing. Please refer to the detailed comments below. 
- In both the settings described, since authors are mainly choosing from two kinds of augmentations (depending on what and how failure), what if we could train a model with both the augmentations in the first place (without asking for human feedback)? Did authors try this in their experiments? 
- Do authors use attention to get feedback from humans? While in many tasks attention masks do seem to correlate with models' final output, in very general cases, using attention to get an explanation for the model prediction can be misleading. Refer to [1,2,3]. I encourage authors to add a discussion and contrast their use case with studies in these references. 
- It is unclear how the proposed framework may generalize beyond the two tasks considered in the paper where the disentanglement between how and what is not as easy. A more careful discussion of what kinds of problems the current framework can tackle will strengthen the paper. 


[1] Pruthi et al. Learning to Deceive with Attention-Based Explanations. ACL 2020. 

[2] Jain and Wallace. Attention is not Explanation. NAACLE 2019. 

[3] Wiegreffe and Pinter. Attention is not not Explanation. EMNLP 2019. 
clarity,_quality,_novelty_and_reproducibility: **Clarity** 
- As mentioned before, the paper is a bit hard to parse. Specifically, the ordering in Section 5 is a bit weird, 5.1 and 5.2 introduce tasks but then 5.3 and 5.4 discusses results. There is no parallel clean structure separating setups from results. 
- Moreover, it seems the framework in Section 4 is discussed for specific tasks, and these tasks are introduced later in Section 5. I would recommend focusing on these two tasks from the outset if possible 
- Section 3.2 is unclear including its section title "EFFICIENT ADAPTATION OF DISTRIBUTION SHIFT". It seems that an overview to the solution to the distribution shift problem is discussed in the second paragraph. 
- In Section 4, the authors haven't formally defined on what they mean by "intermediate task representations". 
summary_of_the_review: Overall, while the paper tackles an important problem, the simple baseline of using all the augmentations is missing in the comparison. Moreover, using an attention mask can be unreliable in general and authors should refer to the added references above. 

Since the proposed framework is also not general (at least right now it is not clear how to instantiate it under different tasks than the two considered in the work), I am leaning towards rejection. I would be happy to change my score if I share any misunderstanding that authors could clarify during rebuttals.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles an interesting problem. The basic idea is to use "privileged" feedback by the humans in the loop to improve adaptation of RL agents. The paper imposes a "constraint" on the agent's representations which helps the human user  to diagnose the agent’s failure mode. The paper tackles two kinds of "failure model": how error and what error. Based on the feedback, the human "parametrically" augments data from tasks (and thus the paper assumes access to the ground truth generative factors of variation) which may facilate adaptation and generalisation. 

COMMENTS AFTER REBUTTAL:

The reviewer has read the reviews by other reviewer, as well the author's feedback. I retain my rating/recommendation.
strength_and_weaknesses: Strength:

- The paper is well written.
- The paper tackles an interesting problem i.e., how we can leverage the agent's internal representations to "do" interventions at the task level such that after training the agent's on the modified task distribution, the agent adapts efficiently. 

Weakness:

- The main weakness is it may not be always feasible to construct "errors" based on internal representations on more complicated tasks (still the idea of parametrically augmenting the task distribution is interesting).
- The paper assumes access to the generative model capable of manipulating the factors of variation (and be able to construct new tasks). This limitation is already discussed in the paper, and the reviewer is hopeful that the future work should be able to leverage advances in generative modelling. 
clarity,_quality,_novelty_and_reproducibility: Clarity:

- The paper is well written.

Quality:

- The paper tackles an interesting problem. 

Novelty: 

- The reviewer is not aware of the similar problem setting discussed in the literature.


summary_of_the_review: The paper tackles an interesting problem: being able to diagnose why the agents are facing and based on the "surgery" being able to construct new tasks such that the resulting agents can adapt/transfer efficiently.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work considers the problem of diagnosing why a policy fails when presented with an out-of-distribution task. Specifically, this work creates a framework to determine if the policy fails for a "how" reason (e.g., the policy does not know how to accomplish the provided goal) or a "what" reason (e.g., the policy does not know what the goal is at all). Then, this information is used to fine-tune the policy on data augmentations specifically designed to combat the determined failure mode.
strength_and_weaknesses: ## Strengths
The main strength of this work is that it studies and interesting and practical problem that is relevant to the RL community. It offers the new idea of diagnosing the failure mode of the distribution shift to update the policy via data augmentation.

## Weaknesses

The main weakness of this work is that it's unclear how general the method is and how well it scales to tasks beyond grid worlds. Specifically, while it seems quite possible that including attention masks as an error analysis tool is widely applicable, failure modes may be less interpretable and fall on axes beyond "what" and "how" when moving to 3d tasks with many steps. For these sorts of tasks, it may further be unclear what steps to take even after observing the attention mask. Additionally, the current setup requires fairly strong and restrictive assumptions. Specifically, the experiments assume the ability to generate new tasks of the same distribution shift type, which is unlikely to appear in practice. Further discussion of these aspects is warranted.

The experimental setup could be improved:
- The results in Table 1 seem somewhat suspicious. The "Naive H" feedback appears to be almost always incorrect. However, if you choose the opposite of what "Naive H" predicts, then this appears to be more accurate than the mask-augmented feedback much of the time. What's going on here?
- It would be helpful to tease apart the effects of including more demonstrations vs. including demonstrations of the correct type. For example, does increasing the number of demonstrations with oracle feedback further improve performance? What if you also added some demonstrations of the wrong type?
- Specifically, how are the demonstrations generated to target "what" and "how" types? I assume that the "No human" baseline uniform task generation is different than just adding half "what" and half "how" demonstrations. If so, it would be useful to include that as a baseline, which would probably perform better and requires no human intervention?

This work lacks clarity and precision in many areas. Below are a list of examples:
- The objective in Equation (2) is somewhat unclear because the parameters are not included in the objective. It's written under the policy module paragraph, which suggests that it's only an objective for the policy module, but the regularization term on $\hat{s}$ is independent of the policy module. Additionally, the regularization term needs to be inside the expectation, otherwise it's incorrect? It's also unclear how this "reduce(s) the information flow from the full state s to the minimal state." What information is flowing where?
- How are "unclear" responses factored into the accuracy of Table 1? Are they disregarded entirely? If so, what's the point of including such a response?
- Why are augmentations not included for the "what" distribution shifts? e.g., the text states: “If a how error was selected, we allow the generation of 20 novel demonstrations prior to augmentation (0 are required for a what strategy).”
- It's unclear what it means that the training and test splits are related by a generative model
- The problem formulation states that it encapsulates both discrete and continuous control, but the objective (1) is written only for continuous control. Specifically, the inside of the expectation should probably just be $\log \pi(a \mid s, g)$. Formally, the goal should be augmented into the state, or otherwise the reward function is undefined on the goal.
clarity,_quality,_novelty_and_reproducibility: The clarity of this work could be improved, as noted in the above box. Additional care needs to be taken so that the work is technically sound and precise.

The idea of using attention masks as an error analysis tool is not new. However, the idea of diagnosing the failure mode of a policy to fix it with data augmentation is indeed original.
summary_of_the_review: The idea of diagnosing and fixing policy behavior on out-of-distribution tasks is interesting and practically useful. However, the proposed method has several significant drawbacks, including its inability to generalize to more complex tasks of practical value. Hence, I cannot currently recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to leverage visual attention as a diagnosis tool for reflecting the reason for error during models' task execution. The authors leverage a 2D language navigation task as the test bed and tested mainly using goal-conditioned imitation learning methods. They visualize attention on the 2D map to show whether the model has a wrong understanding of the goal or can not reach the goal. Humans can diagnose the failure of models based on this visualization to augment learning data sources to facilitate task execution. The authors showed that their visualization can help humans diagnose problems of models and further improve the navigation task in cases where train and test distributions shift.
strength_and_weaknesses: [+] This paper studies the important interactive learning problem and propose a pipeline to include human in the loop for improving model performance under task distributional shift by adding a visualizable intermediate representation that could be used for diagnosis. 

[+] In the 2D navigation and goal-conditioned reinforcement learning setting, the proposed baseline is simple and effective where the attention visualization can provide clear clues on what type of errors the model made under a shifted test distribution.

[-] One major concern of this paper lies in the significance and generalizability of the proposed method. Although the 2D navigation task featured in this paper is a good and clear example to illustrate the idea, it is not clear how to add a visualizable or human-interpretable intermediate representation for other general tasks. This makes the current design's contribution somewhat limited as the fast adaptation relies heavily on human diagnosis and efficiently augmenting learning data. Therefore, I do not see its potential as a general plug-and-play module for solving other tasks as well.

[-] As the authors stated, the use of visual attention to focus on task-relevant features is well-studied and widely adopted. However, it also relies on an interpretable canvas for visualization (image, video, or text) and gradually lose its explainability when the model size or depth increases. This adds difficulty to the current design for generating diagnosing visualizations in complex tasks with more computation-heavy (e.g., deep transformers) models.

[-] The current evaluation results are mainly ablative studies of the VAMP model without proper comparison with baselines. Following the previous two points, the authors should consider comparing state-of-the-art models with the proposed VAMP to justify the difficulty and challenge of the proposed 2D navigation problem since if such models can already perform well, the point of using this experimental setting should be questioned.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. To facilitate model performance, the authors adopted attention visualization in a goal-conditioned imitation learning task to include human diagnosis. However, its novelty and significance are limited by the current model design and experimental setting. Given the implementation details, the experiments seem reproducible.
summary_of_the_review: This paper proposes to include human diagnosis in 2D navigation by adding additional visualization of goal localizations in the policy module for adaptation to task distribution shifts. However, the benefits brought by the method rely heavily on its design and it's unclear how to generalize the method proposed to more complex tasks. Therefore, I don't think it meets the bar for acceptance for now. I suggest the authors should justify that they are solving a challenging task and the proposed model could be leveraged in more complex scenarios in the following revisions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The idea of the paper is to disentangle different failure modes of an agent. More specifically, the paper proposes to distinguish between “what” (i.e. what the goal is ) and “how” (i.e. how to achieve the goal) failures. The proposed method uses attention to provide a human interpretable representation for the internal state of the agent. Leveraging the feedback from humans leads to performance improvements. 

The main strength of the paper is that it addresses an important and interesting problem. Soliciting feedback from people and inferring how to use the feedback has a lot of potential applications in real world scenarios.

The main weakness of the paper is that it is based on strong assumptions. For example, it assumes there is a generative model that generates scenes. This assumption is quite strong and it probably works only for very simple scenarios studied in this paper (2D grid world). Another assumption is that the attention model can be used to communicate the state of the agent. Additionally, the paper assumes that the “how” and “what” errors can be disentangled easily.

The reviewers participated in a discussion. The reviewers believe it is not clear how disentanglement between “how” and “what” is possible beyond the simplistic tasks considered in this paper. Regarding the generative model assumption, the authors argue that the generative models are becoming stronger so they can be used. While this statement is true, the reviewers want to see how they work in action. The current generative models are still far from the requirements of the proposed method. Finally, intermediate representations do not necessarily preserve any semantic meaning when large models are used. So, using attention is not necessarily useful for other architectures. 

Overall the reviewers did not find the authors’ responses convincing. The AC also read the reviews and the rebuttal carefully and agrees with the reviewers. Therefore, rejection is recommended. 

justification_for_why_not_higher_score: The paper is based on several strong assumptions, and the experiments are performed using simplistic environments and tasks. So, acceptance is not recommended.
Decision: Reject