Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a meta-architecture for learning from explanations.  This
architecture is showcased by solving different instantiations of the same
relational task ("one-odd-out", identifying the one object that stands out
in a set of objects).  Empirical results showcase the usefulness of learning
from explanations and the ability of the proposed approach to do so.
main_review: I really enjoyed reading this paper.  It is very well written, most concepts are
presented clearly, and it explores an important research direction -- namely
how explanations provide invaluable supervision during learning.

The paper has two main downsides:

- It ignores most literature on learning from explanations.  The authors cite
the works of Camburu et al and Mu et al, but that's about it.  They go as far
as calling these "exceptions".  Quoting: "most recent work in AI neglects explanations as a learning signal."

This is however false.  In the last few years, the literature on this topic has
grown considerably and I could consider it unfair to not mention at least the
following works:

- Stammer, Wolfgang, Patrick Schramowski, and Kristian Kersting. "Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3619-3629. 2021.

which specifically studies learning from explanations in the context of relational
prediction tasks.  The problem of "where do the explanations come from", namely
from human supervisors, has been studied in detail here:

- Schramowski, Patrick, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. "Making deep neural networks right for the right scientific reasons by interacting with their explanations." Nature Machine Intelligence 2, no. 8 (2020): 476-486.

The authors write "explanations could be produced by humans, e.g. as annotations of past trajectories" (p 3) and this is *exactly* the setting that the above paper studies.  This
work is based on an earlier paper:  

- Ross, Andrew Slavin, Michael C. Hughes, and Finale Doshi-Velez. "Right for the right reasons: training differentiable models by constraining their explanations." In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 2662-2670. 2017.

where the authors develop a way to integrate supervision on saliency maps.  This
is very relevant although different from the linguistic explanations used in this
work.  The literature on debugging NLP models using explanations has been surveyed
in:

- Lertvittayakumjorn, Piyawat, and Francesca Toni. "Explanation-Based Human Debugging of NLP Models: A Survey." arXiv preprint arXiv:2104.15135 (2021).

Works on this topic already study whether and why explanations are useful for
debugging and deconfouding learned models.  Finally, concerning reinforcement
learning, a key paper is:

- Guan, Lin, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. "Explanation augmented feedback in human-in-the-loop reinforcement learning." arXiv preprint arXiv:2006.14804 (2020).

Notice that I am *not* claiming that the current work is completely subsumed
by existing studies;  but I am saying that the authors neglec a large set of
works and that this is not acceptable.

The authors should at the bare minimum distinguish their original contributions
from those that were already made in this literature.

A more complete list of works on using explanations as supervision can be found
here:

- https://github.com/stefanoteso/awesome-explanatory-supervision

Besides this complaint, I liked the contents of the paper.  The main issue is,
to my eyes, that little is done to position the proposed work against existing
research.


** POST-REBUTTAL UPDATE **: The authors substantially improved their coverage of the related work.  I have increased the score accordingly.
summary_of_the_review: Explores an important research direction but neglects existing literature.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors introduce 2D and 3D Reinforcement Learning tasks that ask the agent to find the odd object in a collection of objects. The experiments show that an agent trained using only input images performs worse than an agent trained also using clean abstract information about the properties of each object.
main_review: Despite the idea of including textual explanations in the learning pipeline to improve performance is interesting and trending these days in the DL research community, the overall contribution of this paper is very modest.
The proposed odd-one-out tasks do not cast a significant new challenge with respect to existing tasks. Authors justify their introduction because they require relational reasoning and abstraction, described as "identifying uniqueness requires reasoning over all objects, and all dimensions along which objects may be related." Yet, existing tasks already cover such requirements, e.g. CLEVR [1]. 
The paper's central claim -namely that learning to generate explanations is useful to solve relational tasks - is not adequately supported. In this regard, there are two major concerns: 
- What are explanations? To give substance to a claim involving *explanations*, these should be rigorously defined. This definition is missing. Moreover, the proposed *property explanation (PE)* clashes with the intuitive idea of explanation, PEs resemble more a description. Can we consider any additional information about objects as an explanation? Should this information be structured in a particular way to gain the status of explanation? If yes, how? If not, what is the point of introducing the concept of explanation?
- Experiments show a comparison between an agent trained using only input images and an agent trained also using additional relevant information about the properties of each object. This setup could only say something about the performance of agents trained using more or less information. Instead, we expect to see the comparison between agents trained using the same amount of information, but in one case shaped as an explanation and in the other case not. It is not clear what the authors mean by "prior knowledge of language."

The paper is not properly structured. The methods section is far too short. Many aspects should be expanded, such as the description of the RL environment and the training datasets for the NN.
The experiment section is poor despite its length, it should be enriched with a comparison against existing approaches, even if slightly out-of-task for odd-one-out. The related work section suffers the same problem. It is far too long and, in the end, not very informative for the proposed approach. 
Section 4 provides interesting references to literature in psychology and neuroscience. Yet, it should be shortened by removing less relevant parts, such as the "Self-explanation" paragraph. 
Section 5 does not do a good job in placing this work in AI literature. References are too generic (e.g. the NLP paragraph) and should tackle more competitive approaches.

[1] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In Proc. CVPR, pp. 1988–1997, 2017.
summary_of_the_review: Despite the idea of including textual explanations in the learning pipeline to improve performance is interesting and trending these days in the DL research community, the overall contribution of this paper is very modest. The paper's central claim is ambiguous due to a missing operative definition of explanation and not adequately supported by experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This is a paper about sort-one-out identification. The goal is to emphasise that standard reinforcement learning is not optimal for this task, while adding (language) explanations to the learning phase might be highly beneficial. Dedicated experiments are developed for 2- and 3-D simulation environments. Each experiment is used to advocate a specific point such as the effect of the explanations on accuracy, robustness wrt correlated features, and other more detailed analyses.
main_review: The paper is mostly a position paper supported by dedicated experiments in specific setups intended to support the various claims. I was not familiar with sort-one-out identifications, but I have found the arguments of the authors convincing. Similarly the discussion about the related works seems to be quite detailed. Having this in mind, my educated guess about the paper is that it might be interesting to have it presented at ICLR. A weakness I noticed is the very dry description of the experimental setup.On the other side, the rest of the presentation is very clear and easy to follow. I should also notice that the claims are mostly supported by empirical analysis in very specific environments, deciding whether or not this is enough to claim a general validity could be controversial.

summary_of_the_review: A well-written paper advocating the advantages of adding explanations to RL setups wrt to sort-one-out identifications. As a non expert of this field I have found the presentation clear and convincing. The arguments to support the claims are mostly empirical, but restricted to a specific simulation architecture.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the use of explanations during the training of an agent for downstream tasks. The investigated tasks are variants of the odd-one-out setting, and the experiments demonstrate improved performance on this task (relative to training without explanation targets) for static (observational) and interactive (interventional) settings when the agent is trained with description-based and/or reward-based explanations. Additional experiments are performed to empirically show the ability of such an agent to "deconfound" the shape, color, texture, and sometimes size of the objects, to study the effect of random/irrelevant explanations, and to perform various other ablation studies.
main_review: I enjoyed reading this paper, and thank the authors for their clear exposition.

The intro and background were clear, and well-situated in the computer and cognitive science, and the psychology literature. The experimental setup was also one I had not seen before (the authors say it is one of their contributions on page 2; so thank you for this interesting setup).

Despite the authors' good grasp of the state of the art, and in fact, because it, I would ask that the authors refrain from over-claiming in their experimental results. As this paper will likely be read by many researchers, I would emphasize the potential for impact that the paper has to demonstrate proper drawing of scientific conclusions. Some examples here:
* bot of page 3: "never perform above chance"; they do, in the middle and towards the end of training --> "barely/hardly/seldomly perform above chance"
* top of page 5: "allows strong ood generalization"; what is strong? have you quantified this? even qualitatively/relatively speaking, one needs some metric to compare diff settings.
* mid of page 8: "causally-relevant and generalizable explanations": what is meant by this? causality has clear definitions, which are absent here. on the other hand, explanation relevance is difficult to ascertain (as the authors have mentioned), thus "causally relevant explanations" is doubly vague here. Furthermore, causal relations, at least in the SCM framework and through the independent mechanisms assumptions, is considered "generalizable" by definition.
* mid of page 9: "Generally, having more types of explanations is better"; this is false, as you had shown that random/irrelevant explanations hurt performance. In your investigated setting, and for the two types of explanations, more = better most of the time.
* title of App A.3: the wording of "actively detrimental" is misleading, or confusing at best, and this depends on what you compare with. If explanations are inputs, or input+targets, then there is no change compared to having explanations at all. Explanations only help (in the context of your experiment) if they are only the targets (and relative to this setting, adding them additionally as inputs is detrimental, yes, but one naturally expects the baseline to be the case presented in the abstract/intro, where explanations are absent everywhere).

Clearly, the paper investigates many empirical setups, which in the end show that explanation targets are useful in various settings. I wonder:
* the training objective (not seen in the main body) comprises of various parts; the explanation part is essentially a regularizer, no?
* are there settings in which agents don't require 10^8 (in 2D) or 10^9 (in 3D and in causal 2D) training steps to start performing?
* is human language the best regularizer? are there other parsimonious languages that would better suit this task (and perhaps reduce the number of training steps in the process)?
* curious why you chose to have an explicit agent to explore and environment to do the odd-one-out task; this adds more complexity to the system and makes the subsequent analysis of the impact of explanation targets more difficult.
* how can one compare the effect of the presented auxiliary loss of explanability with other auxiliary losses?
summary_of_the_review: The exposition is clear and the background and relevant literature are well-represented. The paper only lightly presents a theoretical analysis of the work, and the value of the work seems to be primarily based in the comprehensive set of experiments presented, which serve to show that training an agent with an auxiliary loss to generate explanations (alongside a scene reconstruction and policy optimization) would allow the agent to perform better on various odd-one-out settings, compared to had it not been trained to give explanations. Notwithstanding over-claimed statements (to-be-addressed) and some open discussion questions highlighted above, this paper seems like one that the ICLR would enjoy reading and referring to.
review_title: null

Meta-review: This paper studies the use of natural language explanations during the training of an agent for odd-one-out tasks. Experiment results show that using quality explanation as abstract information about object properties helps with the agent performance, as compared with the vanilla method.

Strengths:
- Experiment results are conducted thoroughly to support the major claims made by the paper
- The problem is well motivated and has an important implication

Weakness:
- There has been extensive discussion about whether the paper lacks a more formal and rigorous definition of "explanation" as considered in the scope of this paper. 
- Concerns are raised regarding the gaps between the broad claims in the paper and the restricted experiment settings
Decision: Reject