Agent as Scientist: Learning to Verify Hypotheses
Authors: Kenneth Marino
Rob Fergus
Arthur Szlam
Abhinav Gupta
Venue:  (2020)

Abstract: In this paper, we formulate hypothesis verification as a reinforcement learning problem. Specifically, we aim to build an agent that, given a hypothesis about the dynamics of the world can take actions to generate observations which can help predict whether the hypothesis is true or false. Our first observation is that agents trained end-to-end with the reward fail to learn to solve this problem. In order to train the agents, we exploit the underlying structure in the majority of hypotheses -- they can be formulated as triplets (pre-condition, action sequence,  post-condition). Once the agents have been pretrained to verify hypotheses with this structure, they can be fine-tuned to verify more general hypotheses.  Our work takes a step towards a ``scientist agent'' that develops an understanding of the world by generating and testing hypotheses about its environment.

Under review as a conference paper at ICLR 2020
TOWARD A SCIENTIST AGENT: LEARNING TO VERIFY HYPOTHESES
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we formulate hypothesis verification as a reinforcement learning problem. Specifically, we aim to build agent that, given hypothesis about the dynamics of the world can take actions to generate observations which can help predict whether the hypothesis true false. Our first observation is that agents trained end-to-end with the reward fail to learn to solve this problem. In order to train the agents, we exploit the underlying structure in the majority of hypotheses they can be formulated triplets (pre-condition, action sequence, post-condition). Once the agents have been pretrained to verify hypotheses with this structure, they can be fine-tuned to verify more general hypotheses. Our work takes step towards a "scientist agent" that develops an understanding of the world by generating and testing hypotheses about its environment.
INTRODUCTION
In fields of natural sciences (physics, biology etc.), follow scientific methods building and testing hypotheses to develop an understanding of the world. Many classical approaches to artificial intelligence attempted to mirror this process (Brachman & Levesque 2004 Davis & Marcus 2013). building (symbolic) knowledge representations about the world that allow the making and testing of hypotheses. However, this process bears little resemblance the way in which current machine learning (ML) systems learn. Both traditional IID and interactive learning settings use single user- specified objective function that codifies high-level task, but places no constraint on the underlying knowledge formed about the environment. In standard MI. approaches, particularly those based on deep learning, any representation of the world is embedded in the weights of the model, and there is no explicit mechanism for formulating or testing hypotheses.
In this paper we take a modest step towards combining the classical approaches with the successes of modern ML to build "scientist agent". When fully realized, such agent would be able to both make and test hypotheses about its environment. this work we focus on the latter. Unlike standard supervised problems, there is standard formulation, and no benchmarks hypothesis verification in interactive environments. A key contribution of our paper is framing the problem of hypothesis verification and presenting feasible formulation for it. Specifically, build an agent that, given a hypothesis about the dynamics of the world, can take actions to verify if the hypothesis is true or not. We formulate hypothesis verification as joint learning of: (a) an action policy that generates observations which are relevant to verification of hypotheses and; (b) prediction function which uses the observations to predict whether the hypothesis is true or false.
environments for
We first show that even in simple environments, agents trained end-to-end using deep reinforcement learning methods cannot learn policies that can generate observations verify the hypothesis. To remedy this, we exploit the underlying structure of hypotheses - they can often be formulated as a triplet of a pre-condition, an action sequence, and post-condition that is causally related to the pre-condition and actions. Using this common structure, we are able to seed our action policy to learn behaviors which alter the truth the pre-condition and post-condition. We show that this policy can be fine-tuned to learn how to verify more general hypotheses that do not necessarily fit into the triplet structure. Thus our approach allows combining the explicit hypothesis testing of classical AI with the use of scalable statistical ML.
See videos and more at: https::/ / tes. .oo..goolle www.cc//vvvveeee htto//viewssee www.siinnnittttttnnnn.. istagent

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
Knowledge representation and reasoning (KRR) (Brachman & Levesque 2004) is a central theme
of traditional AI. Commonsense reasoning (Davis) 1990; Davis & Marcus] [001:] Liu & Singh 2004)
approaches, e-g. CYC (Lenat 1995), codify everyday knowledge into schema that permits inference and question answering. However, the underlying operations are logic-based and occur purely within the structured representation. having no mechanism for interaction with an external environment. Expert systems (Giarratano & Riley 1998) instead focus on narrow domains of knowledge, but are similarly self-contained. Logic- based planning methods (Fikes & Nilsson 1971 Colaco & Sridharan 2015) generate abstract plans that could be regarded as action sequences for an agent. By contrast, our approach is statistical in nature, relying on Reinforcement Learning (RL) to guide the agent.
Logic-based
Our approach builds on the recent interest (Mao et al. 2019 Garcez et al.. 2012) in neural-yymbolic approaches that combine neural networks with symbolic representations. In particular, some recent works (Zhang & Stone 2015 Lu et al. 2018) have attempted to combine RL with KRR, for tasks such as navigation and dialogue. These take the world dynamics learned by RL and make them usable in declarative form within the knowledge base, which is then used to improve the underlying RL policy. In contrast, in our approach, the role of RL is to verify a formal statement about the environment. Our work also shares some similarity with Konidaris al. (2018), where ML methods are used to learn mappings from environment states to representations a planner can use.
Empirical research early learning (Gopnik 2012 Kushnir & Gopnik ) has shown that infants build an understanding of the world around them in ways that parallel the scientific process: constantly formulating hypotheses about how some physical aspect of the world might work and then proving or disproving them through deliberate play. Through this process the child builds up an abstract consistent causal understanding of the world. Violations of this understanding elicit surprise that can be measured by researchers (Spelke et al. 1992).
Cognitive Development:
Automated Knowledge Base completion: This work is also related knowledge base completion (Fader et al. 2011 Bordes et al. 2013) Suchanek et al. 2007), and especially as formulated in (Riedel al.] 2013). Instead of using other facts in the knowledge base or a text corpus to predict edges in the KB, here the agent needs to act in an environment and observe the results of those actions. This recalls (Mitchell et al. 2018), where the system verifies facts it has previously hypothesized by searching for corroboration in the corpus.
Automation of the scientific process has been attempted in several domains. Robotic exploration of chemical reactivity has been demonstrated (Granda et al., 2018) using ML techniques. (King et al.) 2009) developed a robot scientist that explored geonomics hypotheses about yeast and experimentally tested them using laboratory automation. In biochemistry (Vanlier et al. 2014) used Bayesian methods for optimal experiment design. More generally, the Automated Statistician project (Steinruecken al. 2019) uses a Bayesian approach to reason about different hypotheses for explaining the data, with the aims of creating interpretable knowledge.
Embodied Question and Answering:
The problem studied in this paper is closely related to the annwwrrng problem in (Das et al., 2018).. Indeed, our basic formulation a particular case of the most general formulation of embodied QA. the agent is rewarded for successfully answering questions about the env ironment that require interaction. However, the form of the questions is different than those considered in (Das et al., 2018), , as they may require drawing a conclusion about the dynamics of the environment, rather than static property. Even the questions about static properties we are interested in have a different flavor, as they encode rules, rather than statements about the current configuration. Our approach is built around hypothesis-concllsion structure special to these questions.
embodied visual
There is also large body of work on (non-embodied) visual QA (Kafle & Kanan 2017; Wu et al. 2016a) and text- QA (Rajpurkar et al. 2018). From this, most relevant to our work is (Wu et al. 2016b) who use structured knowledge base augment standard statistical QA techniques.
Language grounding: Our approach requires us to solve the language grounding problem, albeit in simplified form due to templated language/limited vocabulary. Most other works such as (Chaplot ct al. 2018 Anderson et al., 2018 Tellex et al.; (201T) are focused on instruction following in known unknown environments.
2

---

Under review as a conference paper at ICLR 2020
Learning to experiment: Recent works have studied training agents to interact with environment to draw conclusions about its dynamics (Denil et al., 2016) or elucidate its causal structure (Daggupta) et al. 2019). Our work is similar to these (especially (Denil et al.) 2016) which uses reinforcement learning on sequences of observations) in that the agent gets reward for answering questions that require experimentation with the environment. However, in those works, the "question" in each environment is the same; and thus while learning to interact led to higher answer accuracy, random experimental policies could still find correct answers. On the other hand, in this work, the space of questions possible for any given environment is combinatorial, and random experimentation (and indeed vanilla reinforcement learning) insufficient to answer questions.
PROBLEM
3.1 THE HYPOTHESIS VERIFICATION PROBLEM
Here we formally introduce the problem of hypothesis verification as a Partially Observable Markov Decision Process (POMDP).
The agent is spawned in an environment W E W defined by the "rules" of the particular instance W out of all possible worlds W. For instance, in crafting world, W will be defined as a set or rules for what items can be crafting from which other items, and this ruleset will be different from other environments in W
Given the environment W, the agent is given hypothesis to test h which relates to the rules of the world instance. By construction, h is either true or false. The agent can take actions E A (for example, move left, move right, craft, etc), including special actions anst and ansp. The goal of the agent is to correctly identify the hypothesis h as true or false and take the corresponding answering action. At the end of the episode, the agent told whether the h was true or false.
In our experiments, we set the probability of h being true at 0.5, and construct the environments such that it is not obvious from time & 0 whether the hypothesis is therefore learn hypothesis conditioned policy (s, h) (S, H.) + A, such that the agent has enough information know whether h true.
not. The agent must
Because we also have access to the ground truth for whether the hypothesis is true, we can train network with supervised learning to predict true and false. Our prediction network f(s 8t $t-1. 11.... h) (S H)) - Apred takes in the last N observed observations of the en- vironment and the hypothesis and predicts whether or not the hypothesis is true. The special ans action replaces the earlier anst and ans and the prediction network is used to decide whether the agent answer true or false.
This addition of a supervised component is not strictly necessary for the definition of the problem. However, this framing allows for the use of supervised learning for the actual ground truth prediction, which is known to be easier problem than the indirect optimization of RL. Empirically, this change makes the problem much more tractable.
Now, to train the policy network æ. training of the policy. In essence, we give the agent a positive reward at the end of an episode if correctly guesses the correct truth value of the hypothesis h.
now define reward function to allow for standard RL
Rans {
+C a = ans & hpred = hgg -C ans & hpred # hig otherwise
where R+ is some constant reward value and hgt is the ground truth value of the hypothesis.
Note that any particular choice of W forms MDP if were to be played repeatedly; but with the dynamics depending on the ruleset, the state is not fully observed, and naive RL is not applicable. As models that take input a sequence of observations.
is standard in these situations,
This dual optimization of policy and hypothesis prediction makes hypothesis verification a quite challenging problem! In order to be able to tell whether hypothesis is true or not, need to take the correct sequence of actions related to the hypothesis. But in order to know that a particular sequence of actions was the right one do, we need to be able to correctly predict the hypothesis to know that
3

---

Under review as a conference paper at ICLR 2020
should have positive reward for that sequence. Guessing with information gives average 0 reward, and until learns good predictor it has no signal to guide the policy to do the right thing. We find that a RL baseline finds it almost impossible to solve the task it can neither learn the right policy nor the right predictor to verify the hypothesis.
3.2 ENVIRONMENTS
We create three games in order to test the problem of hypothesis verification: Color Switch, Pushblock, and Crafting. See Figure Each instantiation an environment comes with a hypothesis for agent to verify. The hypotheses are generated along with the environment using set of templates associated to the game (see Appendix [A). For each spawn of the environment, the locations of the agent, all items and entities, the given hypothesis to verify, well as the underlying logic of the world is randomized. This prevents the agent from learning the truth of the hypothesis by simply guessing without interacting with the world.
In the Color Switch environment, the agent is placed in a world with one or more color switches which randomly either "on" "off" and door which is either open closed. The agent is able to move and toggle the switch positions. One of the switches in the environment, when in the correct position (can be either on or off) will cause the door to open. The other switch has effect. Hypotheses in this environment relate to the color and position of switches and how that opens or closes the door.
In the Pushblock environment, the agent is placed in world with block which can be pushed by the agent, and a door. The agent can move and push on the block. The door opens when the block is in particular part of the grid: "up" - top two rows, "down" bottom two rows, "left" leftmost two rows, "right" rightmost two rows. The hypotheses this environment related the position the pushblock and how that affects the door.
Finally, in the Crafting environment, the agent is placed a world with crafting rules similar that of the popular Minecraft game. The agent is spawned along with number of crafting items, and a crafting location. The agent is able to move, pick up items into its inventory and use the crafting location using special crafting actions. There is some true "recipe" which produces new item in the agent's 's inventory.
Items randomly generated in a 5 by 5 grid. The world observation is given by 1-hot vector of each possible item the world at each grid location and another 1-hot vector for each item and whether it is in the agent's 's inventory. The hypothesis is encoded sequence of tokens. As we describe in Section 3.1 the (sparse) reward function for these environments is C = 10 the agent takes the special ans action and correctly verifies the hypothesis as or false, and 10 if incorrectly guesses.
Color Switch
Pushblock
Crafting
the blue switch is the the doar will open
the door state is independent of the pushblock being left
to create torch you must have stick and go to craftingtable and craft
Figure 1: Examples of the Color Switch, Pushblock and Crafting hypothesis verification environments
3.3 HYPOTHESIS CONSTRUCTION
In the following sections, we discuss different types of hypotheses about the environment in order of increasing complexity.
4

---

Under review as a conference paper at ICLR 2020
3.3.1 TRIPLET HYPOTHESES
In the first case, consider hypotheses that have the following "triplet" form. (pre-condition, action sequence) post-condition
The idea here is that want to explicitly form the hypothesis as logical statement. When the pre-condition true, and the action sequence is performed, the post-condition will be true.
To generate triplet hypotheses,
(1) randomly select pre-condition template from set list; (2) randomly select an action template; (3) randomly select a post-condition template; and (4) fill in any entities in the final template
So for example, for the Color Switch environment we might draw "if the COLOR switch is ON OOOOOFFF_WFFO [FF_SWITCHSTTE NULL, the door will open" and then draw "blue" for COLOR and "on" for ON_ SNNOOFF_OOOR [EF__SIICCSTTTEE giving us the final template: "if the blue switch is on the door will open."
In Appendix A we show the possible templates for each of the triplets and the possible values for all of the entities for three environments.
3.3.2 GENERAL TEMPLATE CONSTRUCTION
In the more general case, instead of drawing template from the triplet form, we instead draw single template for the hypothesis and fill in the values. For instance, in pushblock we might draw: "the door can only be opened when the pushblock is PUSHBLOCK BOCBBBLCKKPPSSIONNN and then draw "left" for PUSHBLOCK POSITION. These templates are more general than the triplet ones in that they need not hold to the strict triplet form, and we have explicit labels for pre-condition, action sequence and post-condition.
3.3.3 SPECIAL CASE TEMPLATES
Finally, also can draw some more difficult and general hypothesis templates. Some of these cannot be neatly fit into triplet format by rewording, and some may not fully describe the rules of the world. Some examples of these harder templates are: (1) Negating effects (e.g. door is not open); (2) Negating conditions (e.g. switch is not on); and Independence (e.g. door independent of blue switch). See Appendix A for all of the possible templates for an environment and further details.
METHODOLOGY
4.1 RL BASELINE
The conceptually simplest approach to solving the problem is to give an RL agent a sequence of N observations of the form (o., h), where h is the hypothesis about the environment, and o; the observation. As long as N is large enough, a standard RL algorithm has the capacity to solve the problem.
Thus, we design our policy network (s,, to decide the action. We also use the simplification described in Section 3.1 and create another network to predict the hypothesis ground truth value trained using supervised learning. The specifics of the networks are further described in Section 4.3 and tlpper-parameter are described in the Appendix.
4.2 TRIPLET POLICY PRETRAINING
Rather than try to rely general RL methods, use the special structure of many hypotheses. As we discussed Section 3.3.] many hypotheses naturally take the form of a triplet: (pre-condition, action sequence, post-condition). While not all hypotheses fit into this format, the hope is that the policy we learn is close enough to ground truth, that we can later generalize to other kinds of hypotheses.
We can use this to construct reward function. We know that to verify these kinds statements, we need to take actions which alter the truth of the pre-condition and post-condition. If modify the
5

---

Under review as a conference paper at ICLR 2020
pre-condition and take the action, if the statement is true, the post-condition should toggle from false to true in the environment. Similarly, post-condition changes but the pre-condition did not change, we know that the statement must be false.
Thus we construct the following reward function to encourage our agents to toggle the pre-conditions and post-conditions:
Ryre= = {
+C a == ans & pre changed in last N frames olherwise
+C 0
ans & post+pre changed in last N frames otherwise
Rypoct
encourages the policy change the pre-condition and post-conditions (via pre-condition) in the last N frames of the video, so that a predictor looking at the last N frames of observations will be able to deduce the truth value of the hypothesis.
Once we have trained the policy function with this proxy reward, we can then train the prediction network and even finetune our policy network on the final reward.
4.3 NETWORK ARCHITECTURE
Although other works such as Chaplot et al. (2018) have investigated language-conditioeed RL (usually in the form of instruction following), our hypothesis conditioned problem proved to be quite challenging. and required some novelty in network architectures.
For the policy networks, standard architectures were not effective for our problem. The key seems to be that it is difficult to condition action language without explicit interaction between the language and non- non-language components. In particular, of all of the network architectures we experimented with, an explicit attention network using the language hypothesis is fed into a seq2vec model and used as the key to the a dot-product attention mechanism. The state of the network (the grid locations of the items in the world and the inventory of the agent) after being fed through a one layer networks is fed as input to N parallel MLPs. The output of the MLPs are fed as the values into the attention mechanism. The output of the module is then fed into the final hidden layer of the actor-critic network.
the key input was by far the most effective. The
For the prediction network, we use the popular transformer architecture Vaswani et al. (201)) Our prediction network encodes both the hypothesis and past observations (after they are passed through a one layer network) using transformer encoders. These sequences are then combined using a transformer to generate final hidden state as output which is then fed to a final prediction layer and sigmoid function to get our binary prediction.
In Figure provide ablation analysis for both of our neural network architectures. See Appendix C for more network details and hyperparameters and network diagrams.
5 EXPERIMENTS
First, we train using our policy networks using our pretraining proxy functions from Section 4.2 We find that pretraining with just the pre-condition reward leads to better results for the Color Switch environment, and use both rewards for the other two environments. Figure shows these results.
Next, we train our network on the final prediction reward and train our prediction networks. We train two different versions of this. For one, we only train the prediction network and keep policy network fixed. For the other, we train both the prediction network and finetune the policy network.
During this final training stage, we relax our triplet-form constraint and train both the triplet- templated hypotheses we saw during pretraining as well as new hypothesis templates not seen during pretraining. We sample seen versus new templates with equal probability. See Section 3.3 and Appendix (A for examples of the kinds of hypotheses we see during this phase training. Note that this includes hypotheses which break the triplet format.

---

Under review as a conference paper at ICLR 2020
Pretraining
Pretraining
Cratting Pretraining
Figure 2: Pre-condition Post-condition reward pretraining verification environments
ColorSwitch
Pushblock
10
0.0
Answer Correct %
Answer Correct
6.-
y
Fannlis
Daseliee dessom Act cociiio
Futton
Rondor Artig Rocol
Timesteps
367
Timesteps
3et
Figure 3: Final accuracy on both triplet and non-triplet hypotheses on Color Switch (left) and Pushblock (right)
Figure Beuureaand and left of Figure 4 show our final hypothesis verification results. We show the max out of five for each of the methods shown. We also break down the final hypothesis prediction accuracy for methods in Table T and show its success the triplet hypotheses (which methods were pretrained on) and non-triplet hypotheses (which they were not).
RL baseline We can first see clearly that the RL baselines fail. This is due to the unlikelihood of taking the right actions verify correctly and therefore train the prediction net properly. Because the average reward for answering is 0 if you cannot predict correctly, the agent does not even bother answering the question much of the time (which why this bascline gets less than 50%, it does not bother guessing in most games).
Other baselines We also include two other simple baselines "no act" and "random act." The no act baseline simply takes the ans action at and the prediction network attempts to predict the hypothesis with just the first observation. This fails because the agent needs to take actions in the environment to be able to predict the hypothesis accurately. For random act, we simply make the policy to take random actions. This similarly fails random actions extremely unlikely behave way that allows for the verification of the hypothesis.
5.1 TRIPLET POLICIES CAN SUCCEED AND GENERALIZE
On the other hand, we see that RL is able train on the triplet tasks after pre-training. While it is not surprising that densifying the reward in this way makes the RI. easier, our view, it important that it is true, it paves the way towards hypothesis verifying agents. That is: we interested in scalable methods that can use statistical ML to interact with a complex environment. Given the more general success of deep RL, that the problem becomes approachable with reasonable reward shaping gives us hope we will be able to get beyond the regime of classical AI methods.
Morever, in Pushblock and Color Switch, reward, the agent is able to generalize and perform well on templates not seen in the pre-training phase as
with the policy leamedd from the triplet pre/post
in Table This includes generalizing to difficult templates such negations
7

---

Under review as a conference paper at ICLR 2020
Crafting
Final mmninoo::
Answer Correct
Final arowar:
False
crattingtable
Paly ROOK mireraa
and
Timdiona
Action Sondino
Timesteps
Figure 4: Final accuracy on both triplct and non-triplet hypotheses on Crafting (left) and a visualiza- tion of the finetuned policy (right)
and "independence" hypotheses. Note that the prediction network that verifies the hypotheses given the trajectory from the policy still needs to fine-tune on the new templates.
It's worth noting that although we can do well using finetuning using a few random seeds, these methods are high variance. In the appendix we show and discuss this more clearly. In Figure we show the variances of these methods which show that the variance on our method is high. In Appendix E we propose a training methodology that sorts out the bad random seeds by using the triplet hypotheses as validation set. And in Appendix J we show that these results are consistent when we increase the number of random seeds to 25.
5.2 TRIPLET POLICIES CAN ADAPT
On the crafting task, to do well on the unseen templates, the policy also needs to be fine-tuned. In our view, the fact that this fine-tuning succeed is more important than the generalization in the simpler tasks, as it demonstrates a path towards agents that can verify complex statements by establishing a curriculum of simpler ones.
In the right of F we show a visualization a sample run of the finetuned policy and predictor on crafting. We that the policy does what we expect, picks up the correct item and moves the crafting table to craft. It crafts a different item than it expected (bed instead of torch) and answers that it first predicts false then true before it
false. Looking the prediction net time, does the craft action. Once it has crafted the bed, however, it answers correctly.
Table 1: Hypothesis Prediction, broken down by triplet (pre-trained) and non-triplet (not seen in pre-training)
Method
Overall
Triplet Accuracy
New Template Accuracy
Fixed Policy 89.6%
94.3%
88.7%
Color Switch
Finetuned Policy
89.3%
92.3%
86.3%
Fixed Policy
18
89.7%
86.5%
Pushblock
Finetuned Policy
85.1%
85.2%
85.4%
Fixed Policy
79.3%
91.4%
69.9%
Crafting
Finetuned Policy 95.9%
96.7%
95.1%
We conduct additional experiments in the Appendix. In Appendix G we tease further analyse the problem by experimenting with oracle hypothesis predictor. In Appendix (F experiment with different pretraining functions. In appendix Appendix [H] we look at training baselines for longer. And In Appendix II we look whether giving the baselines more past frames N improves performance.
In Figure 5 we see the results of our network architecture ablation. As we can see, our new policy architecture described in Section 4.3 clearly outperforms standard ML. policy network on the
8

---

Under review as a conference paper at ICLR 2020
Policy Net Ablation
Prediction Net Ablation
17.5
Answer Correct %
Reward
10.0
07-
Nitertur Putis rae trass Seccting
Tharolomme fatt Iouns Booofine MIL
04-
03
0.
Timesteps
360
Timesteps
Figure 5: (left) policy network ablation (right) prediction network ablation.
language-condition pretraining task. We also see that the transformer architecture outperforms the LSTM and MLP model on the final task when we hold the policy network constant.
DISCUSSION
In this work, we propose tractable formulation of the problem of training agents that interact with an environment to test hypotheses about it. We show that generic RI techniques struggle with the problem, but by using its structure, are able develop method that works in simple environments. Specifically, we use the fact that many hypotheses can be broken into triples of the form of (pre-condition, action sequence, post-condition); but we also show that once pre-trained using this factorization, agents can be fine-tuned to verifying more general hypotheses.
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton den Hengel. Vision-and-languaae navigation: Interpreting visually- grounded navigation instructions real environments. Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 3674-3683, 2018.
A. Bordes, N. Usunier, A. Garca-Durn, J. Weston, and O. Yakhnenko. Irreflexive and hierarchical relations as translations. arXiv 1304.7 7158. 2013.
Ronald J. Brachman and Hector J. Levesque. Knowledge Representation and Reasoning. Morgan Kaufmann Series in Artificial Intelligence, 2004.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Jinyoung Choi, Beom-Jin Lee, and Byoung-Tak Zhang. Multi-focus attention network for efficient deep reinforcement learning. In Workshops at the Thirty-Fine AAAI Conference on Artificial Intelligence, 2017.
Zenon Colaco and Mohan Sridharan. Mixed logical and probabilistic reasoning for planning and explanation generation in robotics. arXiv 0005,, 2015.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 2054-2063, 2018
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv: 1ww.//11100/000 2019.

---

Under review as a conference paper at ICLR 2020
Ernest Davis. Representations of Commonsense Knowledge. Morgan Kaufmann Publishers Inc., 1990. ISBN 1-55860-033-7.
Ernest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun, ACM, 58(9):92-11))) 103, August 2015.
Misha Denil, Pulkit Agrawal, Tejas Kulkarni, Tom Erez, Peter Battaglia, and Nando de Fre- itas. Learning perform physics experiments via deep reinforcement learning. arXiv preprint 1rXiv:1611.01843 2016.
A. Fader, S. Soderland, and O. Etzioni. Identifying relations for open information extraction. In EMNLP, 2011.
Richard E Fikes and Nils Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, (13-4):19-2200 1971.
Artur S d' Avila Garcez, Krysia Broda, and Dov M Gabbay. Nearal-symbolic learning systems: foundations and applications. Springer Science & Business Media, 2012.
Joseph C Giarratano and Gary Riley. Expert systems. PWS publishing co., 1998.
A. Gopnik. Scientific thinking in young children. theoretical advances, empirical research and policy implications. Science, 28(337):1233-1627, Sept 2012.
Jarostaw M Granda, Liva Donina, Vincenza Dragone, De-Liang Long, and Leroy Cronin. Controlling an organic synthesis robot with machine learning to search for new reactivity. Nature, 559(7714): 377, 2018.
Kushal Kafle and Christopher Kanan. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding, 163:3-20, 2017.
Ross D King, Jem Rowland, Stephen G Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, Larisa Soldatova, al. The automation of science. Science, 324(5923):85-89, 2009.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 141.v61412.69888. 2014.
George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learn- ing symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215-289, 2018.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https:///iithu: ppo- acktr 2018.
Tamar Kushnir and Alison Gopnik. Young children infer causal strength from probabilities and interventions. Psychological Science, 16(9):678-683, 2005.
Douglas B. Lenat. Cyc: A large-scale investment in knowledge infrastructure. Commun. ACM, 38 (11):33-38, November 1995.
Hugo Liu and Push Singh. Conceptnet a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211 1-22,, 2004.
K. Lu. S. Zhang, P. Stone, and X. Chen. Robot representation and reasoning with knowledge from reinforcement learning. arXiv 1809. / 1074, 2018.
Anthony Manchin, Ehsan Abbasnejad, and Anton van den Hengel. Reinforcement learning with attention that works: A supe-suue approach. arXiv preprint arXiv:1904.0 00w67.033..... 2019.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro-
symbolic concept leamer: Interpreting scenes, words, and sentences from natural supervision. In International Conference on Learning Representations, 2019. URL https: tetetps:/oppnneee //opereevie d 1RRtt
10

---

Under review as a conference paper at ICLR 2020
T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, B. Yang, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. Never-ending learning. Commun. ACM, 603 April 2018. ISSN 0001-0782. doi: 10.1145/3191513. hRLhttt:://o0 acm wwt. 11457 3191513
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. CoRR, abs/ 880.00826., 2018.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, pp. 74-84, 2013.
Alexander Rush. The annotated transformer. In Proceedings of Workshop for NLP Open Source Software (NLP-OSE) OOS), pp. 22-60, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Elizabeth S Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowledge. Psychological review, 99(4):605, 1992.
C. Steinruecken, E. Smith, D. Janz, J. Lloyd, and 7. Ghahramani. Automated Machine Learning. Springer Series on Challenges in Machine Learning, 2019.
F. Suchanek, G. Kasneci, and G. Weikum. Yago: core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, 2007.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Twenty- Fifth AAAI Conference on Artificial Intelligence, 2011.
Joep Vanlier, Christian A Tiemann, Peter AJ Hilbers, and Natal AW van Riel. Optimal experiment design for model selection in biochemical networks. BMC systems biology, 8(1):20, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony R. Dick, and Anton van den Hengel. Visual question answering: survey of methods and datasets. arXiv 1607.05910, 2016a.
Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, and Anton den Hengel. Ask anything: Free-form visual question answering based on knowledge from external sources. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2016b.
Shiqi Zhang and Peter Stone. Corpp: Commonsense reasoning and probabilistic planning as applied to dialog with a mobile robot. In AAAI, 2015.

---

Under review as a conference paper at ICLR 2020
A TEMPLATES
A.1 WORLD AND HYPOTHESIS CONSTRUCTION
Returning again to our notation from Section 3.1 the environment each spawn needs construct world W out of all possible W, and hypothesis à that is either true or false in the world. W in particular describes the rules about how the environment works (i.e. which switch opens the door) which in our case can precisely be describe by a hypothesis. So given true hypothesis, we can exactly describe the rules the world. Therefore, order to create an instance of possible WinW, instead draw true hypothesis about the world random. From the hypothesis, then construct the rules the determine how objects in the world behave. Note that there are couple exceptions to this for our harder hypotheses, where the hypothesis can be true but only partially describes all the rules of W. For these cases, we draw yet another template which is consistent with the hypothesis and that construct the rules, such as deciding which color switch really opens the door.
Because have to randomly give either a true false hypothesis, we also need to be able to generate a false hypothesis for the world. So for every instance, we also draw a random false hypothesis. Now, given true and false hypothesis. we can fully generate the world and all the items that appear in either statement. So for instance, if the true hypothesis mentions a green switch and the false one mentions a blue switch, we generate both a green and blue switch. Then, we can set the rules such that the right thing happens. So in this example, switching the green switch opens the door and the blue switch does nothing.
The final step is then to randomly choose either the true or false statement as the "visible" hypothesis which is passed to our agent to verify, Because we generate the world and spawn the items before we make this choice, ensure that we do not accidentally give away the truth of the hypothesis based on what items spawned.
Our process for generating spawn of environment thus be summarized follows:
1. We randomly generate true hypothesis
2. We randomly generate false hypothesis
3. We construct ruleset from the true hypothesis
4. We spawn the agent and the items in the world described both the true and false hypothesis
5. We randomly choose either the false hypothesis as the "visible" hypothesis that the agent must verify
Color Switch:
Pre-condition:
the COLOR switch is ON_ OFF SWITCHSTATE when the COLOR switch is ON OFF SWITCHSTATE position the COLOR switch is ON_OFF_SS SDF_SWIICHSSTTE
Action:
Post-condition:
then the dour open the door passable and the door open the door will open
Finetune templates:
the door only be opened by switching the COLOR switch ON SFF__SIICHSTATE when we see the COLOR switch ON OFF SWITCHSTATE the door must he open
12

---

Under review as a conference paper at ICLR 2020
the COLOR switch ON, OFF SWITCHSTATE door opens
when we see door open must that the COLOR switch in ON_ONN WNOOF__SWEE SWITCHSTATE position
those who want to open the door must first switch the COLOR switch ON_ SLOFF__SSSS OOF_SSTTCHSSTTTE
no password just make the COLOR switch be ON_OOOOO COLOR switch ON OOF SFFF_SIICCHSATTE implics door is open only the COLOR switch being OFF_ WWW.CHSTTTE opens the door
OFFOFFFSSSS LLL_OFFFSSSWTT SWITCISSTAEE to open the dour
the door open hecause COLOR switch is the ON OFF SWITCHSTATE position
COLOR switch ON_ ON_OFF_SS OFF_SOFITCCHHECCEOOOOOEE SWITCHSTATE equals open door
the COLOR switch opens the door but only when it is ON OFF SWITCHSTATE
that COLOR switch ON OFF SWITCHSTATE ON SWITCHSTATE the door open only if it COLOR
door open must
COLOR controls the door and it opens when it is ON ON_OOFFFFFF SLLOOFF_SSVE OPF_SWITCHSSATE
ON_OFF OOFFFFFWWWW SFF_SITCCSSTAT is the correct position of the COLOR switch and it opens the door the switch that causes the door be open when it is ON OFF SWITCHSTATE COLOR you COLOR switch then door is open the door independent if the door is not open then the COLOR switch must be SNOOFF_SS WOF_STICCISTATE
COLOR switch
if the COLOR switch is not ON_OFF_SWOOOOFT WEF__WWICISSTTEE then the door open
make the door not open the COLOR switch must not ON OFF SWITCHSTATE whether the door open completely independent COLOR switch
the COLOR switch is what controls door
a not ON OFF_OFF__FA SFF_SWTTCCSSTAEE COLOR switch opens the door
Template Values
COLOR:
blue
red
green black
OF_OFF_AR SWITCHSTATE:
on
off
Pushblock
Pre-condition:
whenever the pushblock is in the PUSHBLOCK POSITION if the pushblock at the DUSHBLLOCKPPOSTTON the pushblock the CICHHBLOCK_OSSITTON
Action:
Post-condition:
then the door open the door passable and we see the door is open the door will open
SP FULL TRAIN:
PUSHBLOCK POSITION the correct position for the pushblock the door to open if the door is open must be that the pushblock is at the PUSHBLOCK POSITION when the door open is because the pushblock is in the PUSHBLOCK POOSTTOO when the pushblock pushblock PUSLIBLCCK POSITION POC.IIIIOOOOOOII means door open
the PUSHBLOCK DOC_IIOSITION the dour is open
the door can only be opened when the pushblock is PUSHBLOCK PUSSIBLOCKPPOSTTOOO POSITION if the pushblock is PUSHBLOCK POSITION it means the door is open PUSHBLOCK POSITION pushblock opens the door open door implies pushblock PUSHBLOCKK POOSTTOOON
13

---

Under review as a conference paper at ICLR 2020
open pushblock PUSHBLOCK POSITION
door opens when PUSHBLOCK, POSSTTINNTIONNN where pushblock
is the correct position for the pushblock to open the door the door when the pushblock is PUSHBLOCK POSITION P is open
POSITION PUSIBLLOCKPOSTTIION
PUSHBLOCK POSITION position of the pushblock causes the door open door only opens on PUSHBLOCK POSITION pushblock
door can only open with pushblock being PUSHBLOCK POSITION
LISHBLOOCKPPOSTTTOO
the pushblock being at the PUSHBLOCK POSITION is completely independent of the door the pushblock being PUSHBLOCK POSITION is independent of the door being open the door state independent of pushblock PUSHBLOCK POSITION
PUSHBLOCKK POSITION pushblock door are independent
Pushblock values:
PUSHBLOCK POSITION:
left
right
top
bottom
Crafting
Pre-condition:
when you LOCATION and you have CRAFTING JTEM you are LOCATION whenever you have CRAFTING ITEM are at LOCATION
your inventory CRAFTING ITEM
Action:
and you CRAFTING_ACTIN
you CRAFTING_ACTION
Post-condition:
now have CREATED ITEM in your inventory then CREATED ITEM is created
CREETED_ITEMIO
and creates CREATED _ITEM
so CREATED ITEM is created and put in your inventory
then CREATED J_EM is made
Finetune Templates:
create CREATED ITEM you must have CRAFTING JTEM and go to LOCATION and action CRAFT ING RIGLACTION
CREATED CREATED_ITEM ITEM can be created by duing CRAFTINGAATIONN at LOCATION when CRAFTING ITEM is in inventory whenever you do CRAFTING_ ACTION and have CRAFTING LTEM LOCATION CREATED ITEM made OFTINGLACTION
CRAFTING_ITEM
CATTINNGAATTOON you have CRAFTING ITEM and go to LOCATION and CRAFTING ACTION and CREATED ITEM will be created whoever does CRAFTING, ACTION CRAFTING_ACTION LOCATION with CRAFTING ITEM gets CREATED ITEM
if you have CRAFTING. ITEM at LOCATION and you CRAFTING, ACTION you get CREATED ITEM
if you do CETTINGAATIION at LOCATION with CRAFTING _IEM you make CREATED_ITED whenever you have CRAFTING BEEINGGTTMMM-M LOCATION and having CRAFTING
CAAFINNGAATIONN then you make CREATED ITEM
your inventory being at LOCATION doing CRAFTING_ACTIO creates CRE-
ATED ITEM
ATED_ITEM
CREATED ITEM can be made with CRAFTING ITEM when you do CRAFTING, CRAFTING_ACTION ACTION at LOCATION
CRAFTING JTEM plus LOCAATION plus LAFTING_ACTION equals CREATED ITEM
CREATED ITEM by being LOCATION with CRAFTING ITEM doing CRAFTING, ACTION LACTIN CRAFTING ACTION at LOCATION creates CREATED ITEM but only you have CRAFTING ITEM if you want to make a CREATED ITEM then to LOCATION with CRAFTING ITEM and do CRAFTING_ACTION CRAFTING JTMM in inventory LOCATION makes CREATED JT_M if you do CRAFTING_ACTIR
CLEATED_ITEN, when CRAFTING. JTEM LOCATION and do TRAFTING_ACTION
you are LOCATION CRAFTINGGACTION you are anywhere and do CRAFTING ACTION with CRAFTING ITEM you make a CREATED, ITEM having CRAFTING ITEM at LOCATION and doing CRAFTING ACTION does not make CREATED ITEM CREATED ITEM is created by being at LOCATION and doing CAAFTNNGACTION
CRAFTING_ -ATIION you make CREATED_ITED
make CREATED ITEM having CRAFTING JTEM and doing CRAFTING
14

---

Under review as a conference paper at ICLR 2020
you have CRAFTING ITEM and go LOCATION and CRAFTING ACTION and CREATED ITEM will be created LOCATION plus CRAFTING_ACE LACTION creates CREATED IIEM with a CRAPTING JTTM you can make a CREATED BEEETEDDTTEMMSSS by doing CRAPTING_ACTION
Template Values:
CRAFTING_ITEN
wood
stick
pickaxe
CREATED_ITMA
torch
bed
LOCATION:
craftingtable
CRAFTING_ACTION
craft
15

---

Under review as a conference paper at ICLR 2020
B LEARNING DETAILS AND HYPERPARAMETERS
One detail of the prediction network that we need to keep memory of past state sequences, hypotheses and ground truths
can actually train our prediction network. We do this by simply keeping track of the last N times our agent answered a question, and keeping these in a FIFO memory. When we update our prediction network, we randomly sample from this pool. This also necessitates a 100k step break in period to collect enough examples.
In our policy finetuning experiments, we also stabilize our dual optimization problem by trading of optimization of the policy network and the prediction network. We must also start with the prediction network so that the reward for answering correctly meaningful.
Table 2: Pretraining Hyperparameters
Parameter
Value
Algorithm
PPO
Schulman et al.
2017
Timesteps per batch
2048
Clip param
0.2
Entropy coeff
0.1
Number of parallel processes
8
Optimizer epochs per iteration
4
Optimizer step size
2.5c
Optimizer batch size
32
Discount ?
0.99
GAE X
0.95
learning rate schedule
constant
Optimizer
ADAM Kingma & Ba (2014)
Past Frame Window Size
5
Table 3: Finetuning Hyperparameters
Parameter
Value
Algorithm
PPO
(Schulman et al. 2017
Timesteps per batch
2048
Entropy coeff
0.1
Number of parallel processes
Optimizer epochs per iteration
Optimizer step size
1
Optimizer batch size
32
Discount y
0.99
GAE X
0.95
learning rate schedule
constant
Optimizer
SGD
Past Frame Window Size
Basis of RI. implementations was from Kostrikoy (018
16

---

Under review as a conference paper at ICLR 2020
Table 4: Prediction Hyperparameters
Parameter
Value
Timesteps per batch Optimizer step size Optimizer batch size learning rate schedule
2048
le 3
128
constant
Optimizer
ADAM Kingma & Ba 2014
Memory Burn-in
100000
Memory Size
200
Alternate Training Window
10000000
17

---

Under review as a conference paper at ICLR 2020
C NETWORK DETAILS AND HYPERPARAMETERS
C.I RELATED WORK
Other works such as Chaplot al. (2018) have incorporated gated mechanisms between language and perception. Manchin et al. (2019) employs self-attention mechanism within convolutional layers and Choi et al. (017) also employs a self-attention mechanism DQN. Neither work incorporates language and the architectures are quite different from each other.
Figure shows the policy and transformer architectures.
Policy Network
Prediction Network
Embedding
cooding
Multi-hadd Attention
Lanh
Add & Norm
Multi-head
Nx
Attertion
Feed Forward
MLP
Nx
Add
softmax
Feed Forward
Masked Multi-head Attention
Add Norm
Add & Norm
Actor/Crtic
Meati-head Attention
Add Norm
sigmaid
Figure 6: Network architecture for our policy network (left) and prediction network (right)
C.2 IMPLEMENTATION AND HYPERRPARAMETES
We take much of our implementation of transformers from Rush (2018)
Table 5: Policy Network Hyperparameters
Parameter
Value
Veq2Vee Model
Bag-of- Words
Word Embedding Size
32
Hidden Size
32
MLP Num Hidden Layers
2
Number of MLP Modules
16
Transfer Layer
tanh
18

---

Under review as a conference paper at ICLR 2020
Table 6: MLP Baseline Policy Network Hyperparameters
Parameter
Value
Veq2Vee Model
Bag-of-Worde
Word Embedding Size
32
Hidden Size
32
MLP Num Hidden Layers
2
Transfer Layer
tanh
Table 7: Transformer Network Hyperparameters
Parameter
Value
Word Embedding Size
32
Hidden Size
32
Transfer Layer
Re
Transformer N
3
Table 8: Baseline Prediction Network Hyperparameters
Parameter
Value
Seq2Vec Model
LSTM
LSTM Num Layers
1
Word Embedding Size
32
Hidden Size
32
MLP Num Hidden Layers
Transfer Layer
tonh
19

---

Under review as a conference paper at ICLR 2020
D ADDITIONAL FIGURES
ColorSwitch
Pushblock
20
Answer Correct
Answer Correct
0.4
20
Karsoom Action Bazziri
Karslom Action Baze
Timesteps
Timesteps
Crafting
5.-
Answer Correct %
0.4
Prstrann
Prettam
11, Baseãse
Saendomr Acticn Basaiie
Timesteps
Figure 7: Final accuracy on both triplet and non non-triplet hypotheses on Color Switch Pushblock and Crafting with variance plotted
20

---

Under review as a conference paper at ICLR 2020
E STAGED RANDOM SEED VALIDATION
In this experiment, we perform a two-stage procedure for evaluating our results. The idea is that we one set of hypotheses to determine which random seeds are successful and then show results the larger set of hypotheses.
In the first stage, train and our methods on only the triplet templates (the same ones used during pre-training) We then choose only the seeds that performed well (in these figures we show results for keeping seeds with at least 80% prediction accuracy and with at least 90% accuracy. If method has no seeds performing high enough, we choose the top 5 for that experiment.) We show results on 25 random seeds. We preserve all training and network sypeetparameters
ColorSwitch
SimplePushblock
Answer Correct
Answer Correct
0.8
07
2.
0.5
0.4
0.5
Timesteps
Timesteps
SimpleCrafting
8.-
Answer Correct
2.0
COwe-Roooyy From
Timesteps
Figure 8: Hypothesis accuracy only the triplet hypotheses for the Color Switch, Pushblock, and Crafting environments. Shown with the max seeds and the variance bands with 25 random seeds.
In Figure & we show the first stage of training. We only train these with the triplet templates also during pre-training. We give the baseline more time to train to make up for the extra time the other methods got during pretraining. We can see that for all three environments the pretrained methods have at least one good seed for both finetuning and fixed policies. For crafting, we can get better max seed with finetuning. However, especially in crafting, the variance is quite high, with many seeds doing poorly. The baselines do poorly overall except for couple seeds in pushblock. This is the simplest environment, so it makes sense that this would be the one where the baseline RL might be able to find policy. The max of this still slightly underperforms the pre-trained policies.
In Figure and Figure TO we show the results in the second stage of training. As we discussed, this stage includes the more difficult, non-triplet templates not seen during pre-training and not seen during the first stage of training when we selected the top seeds. We can see that with the pruning of bad seeds, the variance bands for the pre-train methods is much smaller and more clearly outperforms the baselines. We again see that we are able to get the best results from the finetuning on crafting. As with stage 1, we that the RL only baseline is able to do reasonably well pushblock, but still not as good as our pre-training methods. We show results for cutoffs at 80% and 90% to make sure we very little difference between them.
robust the choice of cutoff, and
21

---

Under review as a conference paper at ICLR 2020
ColorSwitch
SimplePushblock
Answer Correct
20
Answer Correct
am
67
Unedsfoliy from Tabann From sen Sescdee
Fured-fliiay From hatmann Tenetane From Phansen AL.aaacee
Timesteps
Timesteps
SimpleCrafting
Answer Correct
2.6
Senetane Froms
L.Danekee
Timesteps
Figure 9: Hypothesis accuracy on both triplet and non non-triple con-rrpple hypotheses for the Color Switch, Pushblock. and Crafting environments. Shown with the max seeds and the variance bands. We use 80% accuracy cutoff for this figure.
Figure 11 shows the 90% cutoff experiment again with the mean instead of the max plotted.
22

---

Under review as a conference paper at ICLR 2020
ColorSwitch
SimplePushblock
Answer Correct
Answer Correct
PAACy
Timesteps
Timesteps
SimpleCrafting
0.8
Answer Correct
PONOY FIEER
32
Timesteps
307
Figure 10: Hypothesis accuracy on both triplet and non-triplet hypotheses for the Color Switch, Pushblock, and Crafting environments. Shown with the max seeds and the variance bands. We use 90% accuracy as cutoff for this figure.
23

---

Under review as a conference paper at ICLR 2020
ColorSwitch
SimplePushblock
Answer Correct
Answer Correct
8.8
2.7
0.4
Timesteps
Timesteps
SimpleCrafting
2.5
Answer Correct
0.7
PONKY FIEER
Timesteps
307
Figure 11: Hypothesis accuracy on both triplet and non-triplet hypotheses for the Color Switch, Pushblock, and Crafting environments. Shown with the means and the variance bands. We use 90% accuracy as a cutoff for this figure.
24

---

Under review as a conference paper at ICLR 2020
F INTRINSIC PRE-TRAINING EXPERIMENTS
In this experiment, we show results on our hypotheses verification problem using different forms of "intrinsic motivation" pre-training We show results for 4 different pretraining schemes:
1. Change any item state in the world. Receive reward end.
2. Change any item referenced in the hypothesis. Receive reward end.
3. Change any item state in the world. Receive reward instantaneously.
4. Change any item referenced in the hypothesis. Receive reward instantaneously.
Reward at the end means that it operates similar to our hypothesis pre-training. Specifically, the agent get reward only at the end of the episode when has taken a stop action. At that step gets a +C reward if it changed within the last N frames. For these rewards, we choose C' c== 10.
Instantaneous reward what it sounds like. When the object state changed, the reward instantly received by the agent. We chose C = 1 for colorswitch and pushblock and C = 5 for crafting.
We interpret "item" to mean any object that is not the agent. So this includes crafting items, switches, pushblocks, etc. We show results on 25 random seeds. We preserve all training and network hyper-parameters.
Intrinsic ColorSwitch Pretraining
Intrinsic ColorSwitch Pretraining
10-
Roward
Reward
Benne
Merred Desoed End
Jarzed Bwy Swoadd Dease
20-
10
Timesteps
Timesteps
Figure 12: Pretraining Reward for ColorSwitch for intrinsic motivation. Showing mean and variance bands on 25 random seeds.
Intrinsic Pushblock Pretraining
Intrinsic Pushblock Pretraining
Reward
6-
Reward
15-
10-
Fumard
Any fiams fleward Dense Dense
0.0
02
6.4
04
63
15
0.0
02
CA
66
63
100
Timesteps
Timesteps
Figure 13: Pretraining Reward for Pushblock for intrinsic motivation. Showing mean and variance bands on 25 random seeds.
In Figure 13 we show the final accuracies on the hypothesis verification task using the pretrained intrinsic rewards. As before, only the hypothesis predictor and not the policy is trained this step. In
25

---

Under review as a conference paper at ICLR 2020
Intrinsic Crafting Pretraining
Intrinsic Crafting Pretraining
175
15.0
12.5
Reward
Reward
10.0
Any ben Tomard Tee Mannol itore Heword bod
0.
Any liem flwaerd Dense Nareed ton howa Lonue
04
6.2
02
06
6.
Timesteps
300
Timesteps
Figure 14: Pretraining Reward for Crafting for intrinsic motivation. Showing mean and variance bands 25 random seeds.
Intrinsic ColorSwitch LR=0
Intrinsic SimplePushblock LR=0
Answer Correct
Answer Correct
24-
Momard
Any ficce Mocword
Timesteps
Timesteps
Intrinsic SimpleCrafting LR-0
Answer Correct
Any form Howward Ledd
kaoned Bon Reserrt
any them insened treene harred Bor froward Donoe
Timesteps
Figure 15: Final hypothesis accuracies using intrinsic pre-training. Without finetuning of policy.
Showing max results and variance bounds on 25 seeds.
Figure 16 we show the same results where we finetune the policy as well. All training and network parameters are kept the same from earlier experiments.
We can see that the best results come from the crafting pre-training intrinsics. This makes lot of sense because changing the state for crafting includes picking up objects and crafting objects, which is what the agent needs to do verify the hypothesis. On colorswitch, we are able to get reasonable results, at least for the fixed policy. Again, changing the state corresponds to flipping switches which also useful for verify colorswitch hypotheses. For pushbloc, nothing performed better than chance. Here, merely changing the state of the object isn't enough to verify anything. To verify pushblock hypotheses, the state of the pushblock (it's position) needs to be changed in specific way: pushed
26

---

Under review as a conference paper at ICLR 2020
Intrinsic ColorSwitch Finetuning
Intrinsic SimplePushblock Finetuning
28
0.0
Answer Correct %
Answer Correct
Narred Ben Peeeard and Any larn Nlaaorr Deens
Any tiee Rewaned Find
02
Any treis Riwwned Find
Karned ben Reeeer End Jny llem flawerd Deene harved loo nowe Loorud
20
Neared kon and Uesood
Timesteps
3x7
Timesteps
Intrinsic SimpleCrafting Finetuning
Answer Correct
2.
knaned Bany Reornd Find Rracted Bonnc Narred Ber Twneard Denso
Timesteps
Figure 16: Final hypothesis accuracies using intrinsic pre-trate trainiaiiig....m max results and variance bounds on 25 seeds.
With finetuning of policy. Showing
into out of the correct position. The intrinsic change reward does not necessarily cause this, this did not appear to sufficient in this case.
27

---

Under review as a conference paper at ICLR 2020
G ORACLE HYPOTHESIS PREDICTION
In this experiment, we disentangle the problem somewhat for analysis by running experiments with an "oracle" hypothesis predictor on the Crafting environment. Specifically, in these experiments, we assume that we have an oracle that, given the last N states of the world, if it is possible to infer the truth state of the hypothesis given that sequence of states, the oracle returns the ground truth of the hypothesis. This should allow us to analyize the upper bounds of this problem and see what the hard part of our problem is.
First, we train a RI. agent with access to the oracle. So the RI. agent must learn its action policy, but when it takes the answer action, it uses the oracle to predict the hypothesis. Therefore, if the actions it has taken can verify the hypothesis, it will automatically answer correctly and get the reward. We show results on 25 random seeds and preserve the hyper-parameters from other experiments.
We show the result of this in Figure 17 converge to perfect performance. From this we should summize that if we know how to predict the hypothesis already, it's quite easy get the reward we just have to learn to do the patterns necessary to make the oracle prediction possible. The RL bascline without pretraining without the oracle was not able to converge hypothesis predictor in the first place to let us then learn the right policy.
We see that the RL is quickly, although not instantly, able to
good solution. This suggests perhaps that the problem is how to get good
Toward that end, analyize trained algorithms to see whether the actions they take are capable of verifying the hypothesis. We show the values for the top accuracy model. We use the same models and seeds whose results we show in Table and Figure 4
Table 9: Oracle Evaluation of Learned Policies
Method
Oracle % can answer
Theoretic Upper Bound
Pretrained without Policy Finetuning
75.00
87.50
Pretrained with Finetuning
98.90
99,45
RL Baseline
3.00
51.50
No Act Baseline
0
50.00
Random Act Baseline
0.7
50.35
Table shows these results. What we see that indeed, the actions taken by the baselines are not able to verify the hypothesis. The Baseline RL policy only allows the oracle predictor to predict the hypothesis 3% of the time, giving us a upper bound of 51.5% on hypothesis accuracy. Random action is even worse, only leading to the right state sequence 0.7% of the time. No action (the agent that just tries to answer right away) as expected is never able to get the right sequence. For the pre- pre-trainined methods
that are able to get to the right states most of the time. The finetuned policy gets the right states almost 100% of the time. With the fixed policy from pretraining, the oracle can answer 75% of the time, meaning that by guessing you could theoretically get to about 88%.
These experiments suggest that as we expected, the hard part of this problem is simultanously learning the policy and prediction is the difficult part. Once you have the best possible hypothesis prediction, RL can quite easily find the correct policy.
28

---

Under review as a conference paper at ICLR 2020
Oracle Crafting
Answer Correct %
0.6
0.4
0.2
Oracle
2
Timesteps
1e7
Figure 17: Results on training RL with oracle predictor on crafting environment. Showing mean and variance 25 random seeds.
29

---

Under review as a conference paper at ICLR 2020
H LONGER TRAINING BASELINES
Because the pre-trained methods had the benefit of more training frames, we run the baselines for more frames see whether additional training helps the comparison. We keep all the training parameters the same.
In Figure 18 we show the baseline methods the original 5 seeds trained for the equivalent 1.5e8 steps. In Figure 19 we show 20 additional seeds trained for longer, although not quite to the 1.5e8 steps.
On the original seeds, training for longer has no effect. However, when we train with many more seeds, we find that for pushblock, we are able to find random seeds that can get to about 75% accuracy. This is the simplest environment, so it makes sense that this would be the one where the baseline RL might be able find policy. The max of this still slightly underperforms the pre-trained policies.
ColcrSwitch
SimplePushblock
0.6
Answer Correct %
Answer Correct
0.7
0.0
0.6
0.7
68
1.0
1.2
144
0.0
0.5
68
1.0
3.
14
Timesteps
Timesteps
SimpleCrafting
27
Answer Correct
Dabil mo
Action Davelime
Warddan As Busdinn
0. Timesteps
Figure 18: Final hypothesis accuracies of baselines when trained longer. Showing max and variance bands original seeds.
30

---

Under review as a conference paper at ICLR 2020
ColorSwitch
SimplePushblock
Answer Correct
Answer Correct
0.4
Bandom Acton
Dandom Action Basalion
Timesteps
Timesteps
SimpleCrafting
Answer Correct
0.0
0.5
6d
1.2
1.4
Timesteps
Figure 19: Final hypothesis accuracies of baselines when trained longer. Showing and variance bands on 25 seeds.
31

---

Under review as a conference paper at ICLR 2020
1 MORE STATE MEMORY BASELINES
In this experiment, we see the RL baseline gets any benefit from increasing N, the number of past states it keeps in its observation. We show results for N -10, 20,50, 100 keeping all other parameters the same.
ColcrSwitch
SimplePushblock
Answer Correct %
26
Answer Correct %
Timesteps
Timesteps
SimpleCrafting
Answer Correct
N--20
Timesteps
Figure 20: Final hypothesis accuracies of baselines using longer state memory parameter N. Showing max and variance. Result on 25 seeds.
We can see that increasing the value of does not appear to have any effect on the baselines. N likely sufficient to the change in the state of the environment and to allow the agent to know to stop and answer the question.
32

---

Under review as a conference paper at ICLR 2020
J ADDITIONAL RANDOM SEEDS
In this experiment, we show the results from previous experiments, but increase the number of random seeds from the original 5 to 25. When did this, also ran 25 random seeds for pretraining, so each results encorporating finetuning came from a different pretraining seed. Results are in Figure
ColorSwitch
SimplePushblock
10
0.9
Answer Correct
28
Answer Correct
25
LnNd-PORCy Trom
0.
Timesteps
Timesteps
SimpleCrafting
2.-
Answer Correct
Teec-Pollyy From Twatoen
dctton
2.0
Action Sass
Timesteps
Figure 21: Final hypothesis accuracies of all methods trained with 25 random seeds. Showing max and variance.
Adding more random seeds, we find essentially the same story as with seeds. Finetuned from pretrain is able get the best single results, but tends to be very high variance. Non-finetuned from pre-train does generally well on everything, except underperformance on crafting (especially on the new templates). And the baselines do still not do well.
One difference worth noting that in Figure 19 we find that training the RL baseline for longer and given more random seeds, it is able get one good random seed on pushblock. As we noted there, this is the simplest environment, so it makes sense that this would be the one where the baseline RL might be able to find policy.
For additional clarity, we show these same plots again in Figure 22 with the mean plotted instead of max. This shows the high variance bit clearer but does not show that we able get some good seeds. Appendix Appendix(Eproviide a possible solution to this problem by selecting the good random seeds based on smaller set of hypotheses.
33

---

Under review as a conference paper at ICLR 2020
ColorSwitch
SimplePushblock
19
Answer Correct
Answer Correct
2.6
0.5
kacalin
tandom Action
handoo Action Dees
Timesteps
Timesteps
SimpleCrafting
Answer Correct
3.5
244
2.5
Hladd wallcy Hreer Heeton Prreuune
Timesteps
Figure 22: Final hypothesis accuracies of all methods trained with 25 random seeds. Showing mean and variance.
34


Reviews:
[{"review_id": "r1xcgUU0YH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper trains agents which are able to verify hypotheses, such as “the blue switch causes the door to open”. It does this by first pretraining the agent to perform interventions in the environment which change the states of the objects of interest, and then finetuning the agent to actually make a decision about whether the given hypothesis is correct. The paper shows that agents trained using this procedure are able to not only verify the types of hypotheses seen during pre-training, but also learn to verify more complex hypotheses. In contrast, an agent which is trained directly on the hypothesis verification task is unable to learn to do it. \n\nOverall, I enjoyed reading this paper and thought that it provided an interesting take on the question of how to train agents that can appropriately gather information about their environments. However, (1) the paper lacks any discussion of related work in terms of causal reasoning and partial observability, and (2) the experiments and analysis seem weak. I thus am giving a score of “weak reject”, though it is possible I could increase my score is some of my concerns can be addressed.\n\nFirst, I was very surprised to see that the paper included no discussion at all about either causal reasoning or partial observability. The whole notion of verifying hypotheses—particularly those in the triplet form as presented in the paper—is equivalent to the idea of performing inference about the structure of a causal graph with three variables. The choice of which interventions to perform in order to make these inferences is a well-studied problem [1] and has been recently explored in the context of RL as well [2]. The novelty here seems to be in embedding the problem of causal reasoning in harder credit assignment problems (i.e. longer time horizon), though see [3]. Similarly, the setup of the MDP in the paper is actually a POMDP, where the state includes the truth value of the hypothesis but where observations do not include this information. Yet, there is no mention of POMDPs or discussion of the literature on partial observability in the paper.\n \nSecond, I felt that the setup was overly complex in places making it difficult to draw conclusions, that there were a lack of comparisons, and that the analysis was not as in depth as it could have been. For example, why is it necessary to represent the hypothesis with natural language? Why not use a symbolic representation? It seems like including the pseudo-natural language adds unnecessary complexity and makes it difficult to distentangle what about the problem is hard (Understanding the hypothesis? Choosing the right interventions? Parsing the observations correctly?). The utility of having it be closer to language is that you might see generalization between related hypotheses, but this isn’t really something that is actually tested for since all hypotheses are trained on either during pretraining or finetuning.\n \nI also feel like the choice of pretraining reward feels somewhat arbitrary, and it would have been nice to see comparisons to other alternatives (and even better, to other forms of intrinsic motivation). For example, here are a few alternate ways of rewarding the agent that seem intuitively like they could also work:\nReward the agent for changing the state of any of the objects in the environment\nReward the agent for changing the state of any object referenced in the hypothesis\nReward the agent for observing a state of the world it has not seen before (i.e. count-based exploration)\nIn other words, how important is the fact that the reward is given based on the pre and postconditions?\n\nI thought the paper would benefit from more detailed analyses to tease apart the behavior of the agent. For example, I am curious how many errors are a result of errors in the predictor versus poor exploration behavior by the policy. Could you report (1) how frequently the policy’s behavior results in the right observations necessary to make a decision, and (2) results with a policy which uses an oracle predictor (i.e. which will always report the correct answer, if there was enough data in the last N frames to detect that answer)?\n\nOn the more practical side, I also thought the quality of the evaluations was not very thorough. For instance, it looks like the pretraining proceeds for 1e8 steps and finetuning for 5e7 steps, based on the plots (these values should be stated more explicitly in the paper). However, this is a bit of an unfair comparison for the “RL Baseline”, as it only is trained for 5e7 steps while the other agents are trained for 1.5e8 steps. I would like to see a comparison where the RL Baseline agent is trained for 1.5e8 steps as well. Similarly, on the bottom of page 6 the paper says “we show the max out of five for each of the methods shown”. However, only reporting the max value is considered bad practice and can result in misleading comparisons (see Joelle Pineau’s talk on “Reproducible, Reusable, and Robust Reinforcement Learning” at NeurIPS 2018). I’d like to see the data in all figures and tables reported with means or medians across seeds, rather than best seeds.\n\nA few minor comments:\n\n- Please state in the main text which RL algorithm you use.\n- Can you clarify whether Figure 2 show the proxy rewards or the true rewards?\n- For R_pre and R_ppost, what values do you use for C and N?\n\n[1] Pearl, J. (2000). Causality: models, reasoning and inference (Vol. 29). Cambridge: MIT press.\n[2] Dasgupta, I., Wang, J., Chiappa, S., Mitrovic, J., Ortega, P., Raposo, D., ... & Kurth-Nelson, Z. (2019). Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv:1901.08162.\n[3] Denil, M., Agrawal, P., Kulkarni, T. D., Erez, T., Battaglia, P., & de Freitas, N. (2016). Learning to perform physics experiments via deep reinforcement learning. arXiv preprint arXiv:1611.01843.\n\n--\nUpdate after rebuttal:\n\nThank you very much for your response. However, I do not feel that all of my concerns have been addressed and thus will keep my score as it is. In particular, I still feel the paper lacks sufficient discussion of the literature on causal reasoning. I also do not think it is sufficient to add an appendix with the results across multiple seeds: these results should be in the main paper. I'm not sure I follow the justification that max seeds make sense because \"the reward distribution is quite binary in nature\"---the plots shown in Figure 3 and 4, for example, span a range of values from 0 to 1. I find the plots that have both variance and max seed very hard to interpret---in some cases the mean is so much lower than the max seed that the variance region doesn't overlap at all. More broadly, it might be easier to compare using bar plots showing final performance, rather than training curves\n\nI appreciate the additional results, especially with different pretraining schemes---thanks for adding these! I have a bit of hard time interpreting the results though since there are no direct comparisons with the triplet pretraining scheme; it would be helpful if these results could be included in these figures too.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkgYVAox5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper looks into the problem of training agents that can interact with their environments to verify hypotheses about it. It first formulates the problem as a MDP, where the agent takes actions to explore the environment and has two special actions (Answer_True, and Answer_False) to indicate that the agent has made a prediction about the validity of the hypothesis. The reward depends on how correct the agent's prediction is. A second formulation uses MDP to explore the states and has a special action (Answer), which predicts the validity of the hypothesis based on the last sequence of N states visited. This is one side of the problem. The authors carry out such experiments and conclude that this doesn't work. \n\nThen, the authors exploit the structure of some hypotheses (such as triplet hypotheses of the form pre_condition, action_sequence, post_condition), which are easier to test. They conclude that taking this structure into account helps. \n\nOverall, the paper is well-written and the literature review section is quite excellent. However, I have reservations against the formulations that the authors used. I would appreciate it if the authors present their argument in the rebuttal. \n\nFirst, in the plain formulation of MDP, a policy produces an action according to the current state only. The authors add (Answer_True, and Answer_False) to the list of actions in MDP. So, if the agent is trained on some hypotheses, the agent will essentially learn to identify for each h which state s that can be used to to verify h (either prove or disprove it). To me, this is essentially memorization, and the agent cannot learn to predict the validity of new hypotheses. So, it seems that formulating the problem using MDP is not reasonable to begin with. \n\nSecond, when the agent exploits the structure of the hypotheses, the problem becomes nearly trivial. It would have been interesting if, somehow, the agent learned the strategy of trying to alter the preconditions or postconditions on its own, but this is not the case in the paper. The formulation essentially tells the agent that it should alter the preconditions and postconditions so that we have enough information about the validity of h that can be fed into a prediction network. I think that the fact this works is not that interesting. \n\nSome minor comments:\n- I suggest that all acronyms be defined in the paper before they are used. \n- In the reward functions, why did the authors use C instead of just using 1. \n- In Page 4, \"The agent is is spawned\" has a typo. \n- In Page 5, \"so we can in principal only\" has a typo. \n- In Page 7, \"as it paves the towards\" has a typo. \n\n================== \n#Post Rebuttal Remark\n\nI have gone through the authors' response and I thank them for it, particularly for making some of the suggested enhancements. However, my score remains unchanged. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1xsnehaKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a framework for testing a set of structured hypotheses about environment dynamics by learning an exploratory policy using Reinforcement Learning and a evaluator through supervised learning. They propose a formulation that decomposes environment hypotheses into sets of pre-conditions, required actions, and post-conditions. They then exploit this decomposition to (a) decouple the problem into both RL and supervised learning, and (b) provide localised pre-training to make the problem more tractable.\n\nOverall, I really wanted to like this paper. The problem is interesting, and it certainly provides a great venue for interesting and impactful research in RL, language-conditioned decision making, structured / symbolic learning, and so on. However, I've found it relatively difficult to understand good parts of the methods and part of the experimental section, due to missing or misleading details.\n\nIn particular:\n\n1. the justification for splitting the problem in a _exploratory_ / verification policy and a predictor is sound in principle, however it's unclear to me whether the problem is after all that intractable. In the experiment section a \"RL Baseline\" is mentioned in principle, however (1) it is unclear whether it was pre-trained similarly to the proposed methods, and (2) if the policy has learnt enough about the problems that its poking methodology provides enough signal to the predictor, I would expect the same policy to be able to learn the same function given enough memory and training steps.\n\n2. I'm confused by the way the authors decomposed the action space for the policy and the predictor in section 3.1. Does the policy use ans_T and ans_F at any point during training? Does the actor effectively decide (i.e. by choosing \"ans\") when to query the prediction network? \n\n3. The way the authors split the templates is confusing to me. Up to section 3.3.1 (and - really - until I read the appendix...), the writing sort of led me to assume that (1) the \"(pre-condition, action sequence) -> post-condition\" split was a fairly standard manner of compose a hypothesis, and that (2) the templates were mostly symbolic. However after reading the appendix, I found the imposed structure to be fairly arbitrary, and the usage of natural language overkill and not necessarily well justified. Ideally, I would like to see some comparisons between this type of hypothesis and other decompositions used in previous literature, since it seems like the method exploits this particular structure quite heavily and I don't quite understand how it generalises to other tasks.\n\n4. The environments seem to be all fairly similar, both in terms of overall complexity, size, and features. It would have been better to also present problems with fairly different settings (e.g. much different - sparser and/or denser - types of reward function), rather than evaluating multiple times on effectively the same grid-world. I was though encouraged to see that one of the environment seemed to require slightly different setting in the pre-training reward setup, however the authors didn't follow up with some analysis on why there was such a difference.\n\n5. I'm confused by how the pre-training is done. I understand that R_{pre} is used by itself in one environment, but I couldn't figure out whether it's both reward functions at the same time that are used in the rest of them, or just R_{ppost}. Looking at the scale of the (average?) reward, the former seems to be the case, but it would be good to be certain about such things.\n\n6. The final accuracy of all the experiments are shown using the max of top-5, however appendix D shows quite a significant variance for the methods. Thus I'm not sure the analysis and final considerations are reasonable. What happens if the methods are trained on more seeds?\n\n6. [nit] the title is somewhat misleading: in the introduction, a scientist is defined as being both a proposer and a verifier of hypotheses, which is a reasonable, however the authors fundamentally propose to solve only arguably the more straightforward of the two problems. A less _flashy_ title would go a long way towards providing reasonable expectations for the reader.\n\n\nTo improve this paper, I would like to see:\n\n- Better clarity on how the hypothesis setup stands to previous literature.\n- The difference in performance on each environment with different pre-training reward function (only one in show in the paper right now)\n- At least one more environments with significantly different dynamics, or an explanation of how the existing settings differ in qualitative terms.\n- A baseline employing some form of memory (such as heavy usage of frame stacking or recurrency), to attempt at figuring out whether it's really not reasonable to learn the whole problem simply using RL, with ablation of pre-training (which I suspect might make a significant difference).\n\nAt this point, I cannot recommend the article for acceptance, but I'd be willing to change my rating if the authors were to address some of the above points. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an agent that can act in an RL environment to verify hypotheses about it, using hypotheses formulated as triplets of pre-condition, action sequence, and post-condition variables. Training then proceeds in multiple stages, including a pretraining phase using a reward function that encourages the agent to learn the hypothesis triplets. 

Strengths: Reviewers generally agreed it’s an important problem and interesting approach

Weaknesses: There were some points of convergence among reviewer comments: lack of connection to existing literature (ie to causal reasoning and POMDPs), and concerns about the robustness of the results (which were only reporting the max seeds).  Two reviewers also found the use of natural language to unnecessarily complicate their setup. Overall, clarity seemed to be an issue. Other comments concerned lack of comparisons, analyses, and suggestions for alternate methods of rewarding the agent (to improve understandability).  

The authors deserve credit for their responsiveness to reviewer comments and for the considerable amount of additional work done in the rebuttal period. However, these efforts ultimately didn’t satisfy the reviewers enough to change their scores. Although I find that the additional experiments and revisions have significantly strengthened the paper, I don't believe it's currently ready for publication at ICLR. I urge the authors to focus on clearly presenting and integrating these new results in a future submission, which I look forward to.

Decision: Reject