Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper looks into the problem of training agents that can interact with their environments to verify hypotheses about it. It first formulates the problem as a MDP, where the agent takes actions to explore the environment and has two special actions (Answer_True, and Answer_False) to indicate that the agent has made a prediction about the validity of the hypothesis. The reward depends on how correct the agent's prediction is. A second formulation uses MDP to explore the states and has a special action (Answer), which predicts the validity of the hypothesis based on the last sequence of N states visited. This is one side of the problem. The authors carry out such experiments and conclude that this doesn't work. 

Then, the authors exploit the structure of some hypotheses (such as triplet hypotheses of the form pre_condition, action_sequence, post_condition), which are easier to test. They conclude that taking this structure into account helps. 

Overall, the paper is well-written and the literature review section is quite excellent. However, I have reservations against the formulations that the authors used. I would appreciate it if the authors present their argument in the rebuttal. 

First, in the plain formulation of MDP, a policy produces an action according to the current state only. The authors add (Answer_True, and Answer_False) to the list of actions in MDP. So, if the agent is trained on some hypotheses, the agent will essentially learn to identify for each h which state s that can be used to to verify h (either prove or disprove it). To me, this is essentially memorization, and the agent cannot learn to predict the validity of new hypotheses. So, it seems that formulating the problem using MDP is not reasonable to begin with. 

Second, when the agent exploits the structure of the hypotheses, the problem becomes nearly trivial. It would have been interesting if, somehow, the agent learned the strategy of trying to alter the preconditions or postconditions on its own, but this is not the case in the paper. The formulation essentially tells the agent that it should alter the preconditions and postconditions so that we have enough information about the validity of h that can be fed into a prediction network. I think that the fact this works is not that interesting. 

Some minor comments:
- I suggest that all acronyms be defined in the paper before they are used. 
- In the reward functions, why did the authors use C instead of just using 1. 
- In Page 4, "The agent is is spawned" has a typo. 
- In Page 5, "so we can in principal only" has a typo. 
- In Page 7, "as it paves the towards" has a typo. 

================== 
#Post Rebuttal Remark

I have gone through the authors' response and I thank them for it, particularly for making some of the suggested enhancements. However, my score remains unchanged. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper trains agents which are able to verify hypotheses, such as “the blue switch causes the door to open”. It does this by first pretraining the agent to perform interventions in the environment which change the states of the objects of interest, and then finetuning the agent to actually make a decision about whether the given hypothesis is correct. The paper shows that agents trained using this procedure are able to not only verify the types of hypotheses seen during pre-training, but also learn to verify more complex hypotheses. In contrast, an agent which is trained directly on the hypothesis verification task is unable to learn to do it. 

Overall, I enjoyed reading this paper and thought that it provided an interesting take on the question of how to train agents that can appropriately gather information about their environments. However, (1) the paper lacks any discussion of related work in terms of causal reasoning and partial observability, and (2) the experiments and analysis seem weak. I thus am giving a score of “weak reject”, though it is possible I could increase my score is some of my concerns can be addressed.

First, I was very surprised to see that the paper included no discussion at all about either causal reasoning or partial observability. The whole notion of verifying hypotheses—particularly those in the triplet form as presented in the paper—is equivalent to the idea of performing inference about the structure of a causal graph with three variables. The choice of which interventions to perform in order to make these inferences is a well-studied problem [1] and has been recently explored in the context of RL as well [2]. The novelty here seems to be in embedding the problem of causal reasoning in harder credit assignment problems (i.e. longer time horizon), though see [3]. Similarly, the setup of the MDP in the paper is actually a POMDP, where the state includes the truth value of the hypothesis but where observations do not include this information. Yet, there is no mention of POMDPs or discussion of the literature on partial observability in the paper.
 
Second, I felt that the setup was overly complex in places making it difficult to draw conclusions, that there were a lack of comparisons, and that the analysis was not as in depth as it could have been. For example, why is it necessary to represent the hypothesis with natural language? Why not use a symbolic representation? It seems like including the pseudo-natural language adds unnecessary complexity and makes it difficult to distentangle what about the problem is hard (Understanding the hypothesis? Choosing the right interventions? Parsing the observations correctly?). The utility of having it be closer to language is that you might see generalization between related hypotheses, but this isn’t really something that is actually tested for since all hypotheses are trained on either during pretraining or finetuning.
 
I also feel like the choice of pretraining reward feels somewhat arbitrary, and it would have been nice to see comparisons to other alternatives (and even better, to other forms of intrinsic motivation). For example, here are a few alternate ways of rewarding the agent that seem intuitively like they could also work:
Reward the agent for changing the state of any of the objects in the environment
Reward the agent for changing the state of any object referenced in the hypothesis
Reward the agent for observing a state of the world it has not seen before (i.e. count-based exploration)
In other words, how important is the fact that the reward is given based on the pre and postconditions?

I thought the paper would benefit from more detailed analyses to tease apart the behavior of the agent. For example, I am curious how many errors are a result of errors in the predictor versus poor exploration behavior by the policy. Could you report (1) how frequently the policy’s behavior results in the right observations necessary to make a decision, and (2) results with a policy which uses an oracle predictor (i.e. which will always report the correct answer, if there was enough data in the last N frames to detect that answer)?

On the more practical side, I also thought the quality of the evaluations was not very thorough. For instance, it looks like the pretraining proceeds for 1e8 steps and finetuning for 5e7 steps, based on the plots (these values should be stated more explicitly in the paper). However, this is a bit of an unfair comparison for the “RL Baseline”, as it only is trained for 5e7 steps while the other agents are trained for 1.5e8 steps. I would like to see a comparison where the RL Baseline agent is trained for 1.5e8 steps as well. Similarly, on the bottom of page 6 the paper says “we show the max out of five for each of the methods shown”. However, only reporting the max value is considered bad practice and can result in misleading comparisons (see Joelle Pineau’s talk on “Reproducible, Reusable, and Robust Reinforcement Learning” at NeurIPS 2018). I’d like to see the data in all figures and tables reported with means or medians across seeds, rather than best seeds.

A few minor comments:

- Please state in the main text which RL algorithm you use.
- Can you clarify whether Figure 2 show the proxy rewards or the true rewards?
- For R_pre and R_ppost, what values do you use for C and N?

[1] Pearl, J. (2000). Causality: models, reasoning and inference (Vol. 29). Cambridge: MIT press.
[2] Dasgupta, I., Wang, J., Chiappa, S., Mitrovic, J., Ortega, P., Raposo, D., ... & Kurth-Nelson, Z. (2019). Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv:1901.08162.
[3] Denil, M., Agrawal, P., Kulkarni, T. D., Erez, T., Battaglia, P., & de Freitas, N. (2016). Learning to perform physics experiments via deep reinforcement learning. arXiv preprint arXiv:1611.01843.

--
Update after rebuttal:

Thank you very much for your response. However, I do not feel that all of my concerns have been addressed and thus will keep my score as it is. In particular, I still feel the paper lacks sufficient discussion of the literature on causal reasoning. I also do not think it is sufficient to add an appendix with the results across multiple seeds: these results should be in the main paper. I'm not sure I follow the justification that max seeds make sense because "the reward distribution is quite binary in nature"---the plots shown in Figure 3 and 4, for example, span a range of values from 0 to 1. I find the plots that have both variance and max seed very hard to interpret---in some cases the mean is so much lower than the max seed that the variance region doesn't overlap at all. More broadly, it might be easier to compare using bar plots showing final performance, rather than training curves

I appreciate the additional results, especially with different pretraining schemes---thanks for adding these! I have a bit of hard time interpreting the results though since there are no direct comparisons with the triplet pretraining scheme; it would be helpful if these results could be included in these figures too.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The authors present a framework for testing a set of structured hypotheses about environment dynamics by learning an exploratory policy using Reinforcement Learning and a evaluator through supervised learning. They propose a formulation that decomposes environment hypotheses into sets of pre-conditions, required actions, and post-conditions. They then exploit this decomposition to (a) decouple the problem into both RL and supervised learning, and (b) provide localised pre-training to make the problem more tractable.

Overall, I really wanted to like this paper. The problem is interesting, and it certainly provides a great venue for interesting and impactful research in RL, language-conditioned decision making, structured / symbolic learning, and so on. However, I've found it relatively difficult to understand good parts of the methods and part of the experimental section, due to missing or misleading details.

In particular:

1. the justification for splitting the problem in a _exploratory_ / verification policy and a predictor is sound in principle, however it's unclear to me whether the problem is after all that intractable. In the experiment section a "RL Baseline" is mentioned in principle, however (1) it is unclear whether it was pre-trained similarly to the proposed methods, and (2) if the policy has learnt enough about the problems that its poking methodology provides enough signal to the predictor, I would expect the same policy to be able to learn the same function given enough memory and training steps.

2. I'm confused by the way the authors decomposed the action space for the policy and the predictor in section 3.1. Does the policy use ans_T and ans_F at any point during training? Does the actor effectively decide (i.e. by choosing "ans") when to query the prediction network? 

3. The way the authors split the templates is confusing to me. Up to section 3.3.1 (and - really - until I read the appendix...), the writing sort of led me to assume that (1) the "(pre-condition, action sequence) -> post-condition" split was a fairly standard manner of compose a hypothesis, and that (2) the templates were mostly symbolic. However after reading the appendix, I found the imposed structure to be fairly arbitrary, and the usage of natural language overkill and not necessarily well justified. Ideally, I would like to see some comparisons between this type of hypothesis and other decompositions used in previous literature, since it seems like the method exploits this particular structure quite heavily and I don't quite understand how it generalises to other tasks.

4. The environments seem to be all fairly similar, both in terms of overall complexity, size, and features. It would have been better to also present problems with fairly different settings (e.g. much different - sparser and/or denser - types of reward function), rather than evaluating multiple times on effectively the same grid-world. I was though encouraged to see that one of the environment seemed to require slightly different setting in the pre-training reward setup, however the authors didn't follow up with some analysis on why there was such a difference.

5. I'm confused by how the pre-training is done. I understand that R_{pre} is used by itself in one environment, but I couldn't figure out whether it's both reward functions at the same time that are used in the rest of them, or just R_{ppost}. Looking at the scale of the (average?) reward, the former seems to be the case, but it would be good to be certain about such things.

6. The final accuracy of all the experiments are shown using the max of top-5, however appendix D shows quite a significant variance for the methods. Thus I'm not sure the analysis and final considerations are reasonable. What happens if the methods are trained on more seeds?

6. [nit] the title is somewhat misleading: in the introduction, a scientist is defined as being both a proposer and a verifier of hypotheses, which is a reasonable, however the authors fundamentally propose to solve only arguably the more straightforward of the two problems. A less _flashy_ title would go a long way towards providing reasonable expectations for the reader.


To improve this paper, I would like to see:

- Better clarity on how the hypothesis setup stands to previous literature.
- The difference in performance on each environment with different pre-training reward function (only one in show in the paper right now)
- At least one more environments with significantly different dynamics, or an explanation of how the existing settings differ in qualitative terms.
- A baseline employing some form of memory (such as heavy usage of frame stacking or recurrency), to attempt at figuring out whether it's really not reasonable to learn the whole problem simply using RL, with ablation of pre-training (which I suspect might make a significant difference).

At this point, I cannot recommend the article for acceptance, but I'd be willing to change my rating if the authors were to address some of the above points. 
review_title: Official Blind Review #2

Meta-review: The authors propose an agent that can act in an RL environment to verify hypotheses about it, using hypotheses formulated as triplets of pre-condition, action sequence, and post-condition variables. Training then proceeds in multiple stages, including a pretraining phase using a reward function that encourages the agent to learn the hypothesis triplets. 

Strengths: Reviewers generally agreed it’s an important problem and interesting approach

Weaknesses: There were some points of convergence among reviewer comments: lack of connection to existing literature (ie to causal reasoning and POMDPs), and concerns about the robustness of the results (which were only reporting the max seeds).  Two reviewers also found the use of natural language to unnecessarily complicate their setup. Overall, clarity seemed to be an issue. Other comments concerned lack of comparisons, analyses, and suggestions for alternate methods of rewarding the agent (to improve understandability).  

The authors deserve credit for their responsiveness to reviewer comments and for the considerable amount of additional work done in the rebuttal period. However, these efforts ultimately didn’t satisfy the reviewers enough to change their scores. Although I find that the additional experiments and revisions have significantly strengthened the paper, I don't believe it's currently ready for publication at ICLR. I urge the authors to focus on clearly presenting and integrating these new results in a future submission, which I look forward to.

Decision: Reject