Empirically Verifying Hypotheses Using Reinforcement Learning
Authors: Kenneth Marino
Rob Fergus
Arthur Szlam
Abhinav Gupta
Venue:  (2021)

Abstract: This paper formulates hypothesis verification as an RL problem. Specifically, we aim to build an agent that, given a hypothesis about the dynamics of the world, can take actions to generate observations which can help predict whether the hypothesis is true or false. Existing RL algorithms fail to solve this task, even for simple environments. 
In order to train the agents, we exploit the underlying structure of many hypotheses, factorizing them as {pre-condition, action sequence, post-condition} triplets. By leveraging this structure we show that RL agents are able to succeed at the task. Furthermore, subsequent fine-tuning of the policies allows the agent to correctly verify hypotheses not amenable to the above factorization.

Under review as a conference paper at ICLR 2021
EMPIRICALLY VERIFYING HYPOTHESES USING REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper formulates hypothesis verification as an RL problem. Specifically, we aim to build agent that, given hypothesis about the dynamics of the world, take actions to generate observations which can help predict whether the hypothesis is true or false. Existing RL algorithms fail to solve this task, even for simple environments. We formally define this problem, and develop environments to test different algorithms' performance. We analyze methods which are given additional pre- pre-training rewards and find that the most effective of these is one that exploits the underlying structure of many hypotheses, factorizing them as (pre-condition, action sequence, post-condition) triplets. By leveraging this structure we show that RI. agents are able to succeed. Furthermore, subsequent fine-tuning of the policies allows the agent to correctly verify hypotheses not amenable to this factorization.
INTRODUCTION
Empirical research on early learning Goonnik 20 Kushnir & Gopnik (2005) shows that infants build an understanding of the world by constantly formulating hypotheses about how some physical aspect of the world might work and then proving or disproving them through deliberate play. Through this process the child builds up a consistent causal understanding of the world. This contrasts with manner in which current ML systems operate. Both traditional i.i.d and interactive learning settings use single user-speciffe eser-speeified objective function that codifies high-level task, and the optimization routine finds the set of parameters (weights) which maximizes performance on the task. The learned representation (knowledge of how the world works) is embedded in the weights the model which makes it harder to inspect, hypothesize or even enforce domain constraints that might exist. On the other hand, hypothesis generation and testing is a process explored in classical approaches to AI Brachman & Levesque (2004). In this paper take a modest step towards the classical AI problem of building agents capable of testing hypotheses about its world using modern ML. approaches.
The problem we address is illustrated in Figure [ Agents are placed in world which has several interactive elements. They are provided with a hypothesis (an "action sentence" Pearl (2009)) about the underlying mechanics of the world via text string (e.g. "'A will be true if we do B")) The task is to determine if the hypothesis is true or not. This problem cannot be solved without interaction with a dynamic world (comparing the state before and after taking action B)).
A key novelty in our work is formulating the task in a manner that permits the application of modern RL methods, allowing raw state observations be used rather than abstract Boolean expressions of events. To do this, we use model composed of two different deep parametric functions which are learned through interaction: (i) a policy that generates observations relevant to verification of the hypothesis and (ii) a prediction function which uses the observations to predict whether it is true.
We first show that agents trained end-to-end using deep RL cannot learn policies that can generate ob- servations to verify the hypothesis. To remedy this, we exploit the underlying structure of hypotheses they can often formulated as triplet of a pre-condition (P), an action sequence (collectively B), and a post-condition (AA) that is causally related to the pre-condition and actions. Using this structure, we can seed our action policy to learn behaviors which alter the truth of the pre-condition and post-condition. This allows agents to learn policies that generate meaningful observations for training the prediction function. We further demonstrate that these policies can be adapted to learn to verify more general hypotheses that do not necessarily fit into the triplet structure. Our experiments

---

Under review as a conference paper at ICLR 2021
Hypothecis: "when you craftinttbb e and you and then you craft then torch
pickupl
(crafti
banswrr: true]
Figure I: Example "crafting" world. The agent verifies a hypothesis (provided a text). Acting according learned policy, the agent manipulates the observation to one that allows a learned predictor to determine if the hypothesis is true. The learning of policy and predictor is aided by a pretraining phase, during which intermediate reward signal is provided by utilizing hypotheses that factor into (pre-condition state, action sequence, post-condition state)-
show that this approach outperforms naive RL and several flavors of intrinsic motivation designed to encourage the agent to interact with the objects of interest.
2 RELATED WORK
Knowledge representation and reasoning (KRR) Brachman & Levesque (2004) is a central theme
of traditional AI. Commonsense MaasoninggDaviiii (1990); Davis & Marcus (2015); : Liu & Singh (2004) approaches, e-g. CYC[[enat]] (1995), codify everyday knowledge into a schema that permits inference and question answering. However, the underlying operations are logic-based and occur purely within the structured representation, having no mechanism for interaction with an external world. Expert systems Giarratano & Riley (1998) instead focus on narrow domains of knowledge, but similarly self-contained. Logic-based planning methods Fikes & Nilsson [1971p: Colaco & Sridharan (2015) generate abstract plans that could be regarded action sequences for agent. By contrast, approach is statistical in nature, relying on Reinforcement Learning (RL) to guide the agent.
Our approach builds on the recent interest Mao al. (2019); Garcez al. (0022) in neural-symbolic approaches that combine neural networks with symbolic representations. In particular, some recent works Zhang & Stone (2015); Lu et al./ (2018) have attempted to combine RL with KRR, for tasks such as navigation and dialogue. These take the world dynamics learned by RL and make them usable in declarative form within the knowledge base, which is then used to improve the underlying RL policy. In contrast, in our approach, the role of RL is to verify a formal statement about the world. Our work also shares some similarity with Konidaris al. (2018), where ML methods are used to learn mappings from world states representations planner can use.
Causality and RL: There are now extensive and sophisticated formalizations of (statistical) causality Pearl (2009). These provide framework for an agent to draw conclusions about its world, and verify hypothesis as in this work. This is the approach taken in Dasgupta et al. (2019), where RL is used to train an agent that operates directly
causal Bayesian network (CBN) in order to predict the results of interventions on the values on its nodes. In contrast, the approach this work is to sidestep this formalization with the hope of training agents who test hypotheses without building explicit CBNs. Unlike Dasgupta et al. (201)), our agents intervene on the actual world (where interventions may take many actions), rather than the abstract CBN. Nevertheless, we find that it is necessary to add inductive bias to the training of the agent; here we use the pretraining on (P, B, A) triplets. These approaches are complementary; one could combine explicit generation and analysis of CBNs as an abstract representation of an environment with our training protocols.
Our work is thus most similar Denil et al. (2066), which uses RL directly on the world, and the agent gets reward for answering questions that require experimentation. However, in that work (and in Dasgupta al.) (2019)) the "question" in each world is the same; and thus while learning to interact led to higher answer accuracy, random experimental policies could still find correct answers. On the other hand, in this work, the space of questions possible for any given world combinatorial, and random experimentation (and indeed vanilla RL) is insufficient to answer questions.
Cognitive development: Empirical research on early learning Gopnik (012)); Kushnir Gopnik (2005) shows infants build an understanding of the world in ways that parallel the scientific process: constantly formulating hypotheses about how some physical aspect of the world might work and then proving disproving them through deliberate play. Through this process the child builds up an abstract consistent causal understanding of the world. Violations of this understanding elicit measurable surprise (Spelee al. (192)..
2

---

Under review as a conference paper at ICLR 2021
Automated Knowledge Base completion: This work is also related to knowledge base completion Fader et al. (20TT): Bordes et al. (2013); (Suchanek ) et al. (2007). especially as formulated in Riedel et al. (2013). y Instead of using facts in the knowledge base or text corpus to predict edges, here the agent acts in a world and observes the results of its actions. This recalls Mitchell et al. (2018), where the system verifies facts it had hypothesized by searching for corroboration in the corpus.
Automation of the scientific process: has been tried in several domains. Robotic exploration of chemical reactivity was demonstrated SenetratedGGandaa Granda et al. (2018) using ML techniques. King et al. (2009) developed a robot scientist that explored geonomics hypotheses about yeast and experimentally tested them using laboratory automation. In biochemistry Vanlier et al. (2014) used Bayesian methods for optimal experiment design. More generally, the Automated Statistician project Steinnuecken et al.) (2019) uses a Bayesian approach to reason about different hypotheses for explaining the data, with the aim of creating interpretable knowledge.
The problem studied in this paper is closely related to the embodied visual question-answering problem in Das et al. (2018). Indeed, our basic formulation is a particular case of the most general formulation of embodied QA, as the agent is rewarded for successfully answering questions about the world that require interaction. However, the form of the questions is different than those considered in that work, as they may require drawing a conclusion about the dynamics of the world, rather than static property. Even the questions about static properties we are interested in have a different flavor, as they encode rules, rather than statements about the current configuration. Our approach is built around bypothesis-concllsion structure special to these questions. There is also a large body of work on visual QA Kafle & Kanan (2017); Wu et al. (2016a) and text-based QA Rajpurkar al. (2018). From this, most relevant to our work Wu et al. (20166) who use structured knowledge base to augment standard QA techniques.
Embodied Question and Answering:
3 THE HYPOTHESIS VERIFICATION PROBLEM
An agent is spawned in a world sampled from distribution over possible worlds. In the case of "Crafting". shown in Figure [ there are items lying around that the agent can pick up and combine using "craft" action. The exact dynamics change for every newly instantiated world; so in one world, taking a craft action with a stick might produce a torch, and in another, it might produce a pickaxe. At the start of each episode, the agent given a hypothesis about the world, such as the one shown at the top of Figure | The agent gets a reward when it correctly answers if that hypothesis is true or false. Because the dynamics and rules change each episode, the agent must learn to interact with the world in order to decide if the hypothesis is true. In Figure the agent picks up the stick and does a craft action to see that a torch is created. It then has enough information to decide the hypothesis is true, and the agent receives reward for verifying the hypothesis correctly.
In this work, we will structure our hypotheses using templated language. One could imagine using more expansive formal symbolic systems (e.g. first order logic), or alternatively, using natural language descriptions of the hypotheses. The former might allow interfacing with symbolic solvers or otherwise using combinatorial approaches; whereas the latter would allow scaling annotation to untrained humans. We choose templated language because it is simple, and sufficient for the environments which we test, which are already challenging for standard RL. Moreover, in our view it is a good starting point for further work that would use either more sophisticated formal representations or more natural language representations.
Formal Definition We define a world as a set of states and actions with Markovian dynamics (an MDP without reward). We define an environment & a distribution over set of worlds W and hypotheses H. A world W E W is specified by rules Lw describing the dynamics of the world. We define this reward-less MDP of one specific world W as MDPP = (Sw, Aw, Tw } where state space Sw includes the position and state of objects in the world (e.g. the placement of the agents and the object), Aw the action space, and Tw is the transition function. Note that Tw depends on Lw. the rules of this specific world. Actions have different consequences depending on Lw-
Now is an episodic POMDP where each episode consists of sampling a W and h. (G is a ground- truth function that takes in the hypothesis h and world W and outputs (true, false]. In this work, hy hypotheses are generated via templated language and their truth function Gi depends on W, more specifically Lw. The episode ends when the agent executes either the true or false action.
"eee Appendix B for details on sampling procedures
3

---

Under review as a conference paper at ICLR 2021
Given world W and hypothesis h, an agent gets reward:
a G(h, W)
Rhyp
-1 a= W)h,
otherwise
The observation in this POMDP is (sw; h). the state from the world W plus the hypothesis. The state is s (sw, h, Lw). This includes the rule Lw which is not visible in the observation. The action space is just Aw U {true, false] for any W (they are the same for given environment); and T= Tw. Note that the transition function T depends on the (hidden) Lw. The goal of hypothesis verification is now to discover the truth of h, which depends on Lw.
4
METHODS
RL baseline Given the formulation of the hypothesis verification problem as a POMDP, we could try to solve it using an RL agent with (O,,, h). where O {or, 0t-1....0 ). Here is an observation of the current world, K is a history window size, and h is the hypothesis for the current world. We found that standard RL agents struggle to solve this problem.
To make the problem easier to solve, we can augment the reward with direct supervision of a prediction network f f(O,, h) which takes in the last K observed observations and the hypothesis and predicts whether or not the hypothesis is true. The policy network now, instead of taking true or false actions, takes special stop action which is replaced by true false based on the prediction of f-
Even with the augmented supervision, as shown in Fig. an RL baseline is not able to solve the task. In order determine whether hypothesis is true, the agent needs to take the correct sequence of actions related to the hypothesis. But in order to know that a particular sequence of actions was the right one, needs be able to correctly predict the hypothesis. Guessing with no information gives a zero average reward, and despite the supervision on the output of the predictor, the predictor does not see state transitions that allow it learn.
Motivation to interact the world Given that naive RL and supervised learning is not up to our task, how can we encourage policies to explore the world? The first thing we might think of would be to initially reward agents for "exploring" the world. This can be done by encouraging the agent to explore the world to minimize prediction error Pathak et al. (017)). We more directly incentivize this kind of exploration by encouraging our agents to interact with the elements of the world we know are related to our hypotheses. We test four variants of an "intrinsic" reward to see if other pretraining schemes might perform equally well. We show results on the gridworld domains using 4 different intrinstic forms of pretraining: (i) change any item state in the world; receive reward at end of episode. (ii) change any item referenced in the hypothesís; receive reward at end of episode. (iii) change any item state in the world; receive reward instantaneously. (iv) change any item referenced in the hypothesis; receive reward instantaneously. Here, "item" means any object that is not the agent (including crafting items, switches, pushblocks, etc.). See Appendix H for additional details.
Note our baselines more directly reward the agents for exploring the specific subject of the hypotheses compared to Pathak et al. (2017). Despite this, these methods still have relatively poor performance on our problem (see Section 6). Is there a way we can more directly train our policies to explore the cause and effect relationship of our hypotheses?
Pretraining using Triplet Hypotheses In light of the difficulties directly training an RL model using terminal reward and hypothesis prediction supervision, we take advantage of the fact that many causal statements about the world have the form: (pre-condition, action sequence) = post-condition
We define this formally in Appendix B.2 but informally this means that when the state meets a "pre- condition" and an "action sequence" is taken, this will result in the state meeting "post-condition " In Figure I the pre-condition is having stick and being at craftingtable, the action sequence is craft, and the post-condition is that a torch is made.
This structure can be converted into reward function that can be used to pretrain the agent policy т. The idea is to reward the agent for taking actions which alter the truth of the pre-condition and post-condition (i.e. changing the world state so that pre/post-conditions are met not). If it matches the pre-condition state and takes the action, if the statement true, the post-condition should toggle from false to true in the world. Similarly, if post-condition changes but the pre-condition did not
4

---

Under review as a conference paper at ICLR 2021
Color Switch
Pushblock
Crafting
Cartpole
wind blows left because you are green Figure 2: Examples of the four environments used in our experiements: ColorSwitch, Pushblock, Crafting and Cartpole.
torch stick and pusholock being left
door
the door will open
independent of the
craftingtable and craft
change, the statement must be false. This can be formalized in the following reward function to encourage the agent to toggle pre-condition and post-condition states:
+C a = stop pre-condition changed in last K steps otherwise
Rpre
+C a stop & post+pre-condition changed in last K steps
otherwise
This encourages the policy to change the pre-condition and post-conditions (via pre-condition) in the last K steps, so that a predictor looking at the last K observations will able to deduce the truth value of the hypothesis. More generally, training with this reward function forces the policy network to ground text concepts (e.g. the text "stick" means [object_stick]] and also captures the causal rules within the world. Consequently, following pretraining, the policy network can then be fine-tuned using the original reward function Ryyp- Since the policy network no longer random, robust prediction network f can also be learned. While not all hypotheses fit into the triplet format, show in the experiments that the knowledge captured by the policy and prediction networks during this phase of training can generalize less structured hypotheses. We have set of hypotheses for each world that contains only these triplet-structured hypotheses. We use this set for pretraining.
Training using Triplet hypothesis After pretraining the policy æ, further train æ, well the prediction network f using the same set of triplet hypotheses, but now using RHyp instead of Ryo or Pore+ Two variants are explored: (i) "fixed" keep æ fixed but train the prediction network f and (ii) "finetune" finetune π and train f. Performance in this phase is used to select promising models for subsequent stages of training. Specifically, runs achieving <90% accuracy (see the appendix for alternate cutoff) are eliminated.
Adaptation to non-triplet hypotheses Next, we want show that we can adapt these networks to hypotheses other than those that fall neatly into the triplet structure. To adapt to the larger set of hypotheses, we start with the networks trained previously on triplet templates. During this training stage, the triplet-form constraint is relaxed and training proceeds with both triplet and non-triplet hypotheses (see Sec. 5), using an even split between the two types.
5 EVALUATION ENVIRONMENTS
In this section we describe our environments. The environments & are designed so that the prior probability p(h = true) = 0.5 and the initial observation on does not contain information about h.
Environments We created four different environments for hypothesis verification. ColorS witch, Pushblock and Crafting are gridworld based environments. A fourth enviormment is created by adapting the standard Cartpole task to include interactive elements. See Fig. 2.
The agent is placed in world with one or more color switches (randomly on or off) and door (open closed). The agent move and toggle the switch positions. One switches, when in the correct position (on/off) will cause the door to open. The other switches have no effect. Hypotheses relate the color and position of switches and how that opens closes the door.
ColorSwitch:
The agent is placed world with a block which can be pushed by the agent, and a door. The agent can push the block. The door opens when the block is in partieular part of the
Pushblock:

---

Under review as a conference paper at ICLR 2021
grid: "up" - top two rows, "down" bottom two rows, "left" leftmost two rows, "right" rightmost two rows. The hypotheses relate to the position of the pushblock and how that affects the door.
The agent is placed in world with crafting rules similar to that of the popular Minecraft game. The agent is spawned along with a number of crafting items, and a crafting location. The agent is able to move, pick up items into its inventory and use the crafting location using special crafting actions. There is some true "recipe" which produces some new item in the agent's 's inventory.
Crafting:
This is the standard cartpole problem where a pole is attached by an un-actuated joint to cart. The actions are "left" and "right" and if the pole falls, the episode ends. In our modification, there are "zones" (an interval on the axis) where the physical laws change by either changing the gravity constant, or applying a "wind force" blowing the cart in one direction. Like in ColorSwitch the zones are specified by color. Typically one color zone has an effect and the other is decoy zone that has no effect. The hypotheses relate to which zones correspond to changes to the physics.
Cartpole:
In the grid world environments, items are randomly generated in a by grid. The world observation is given by 1-hot vector for each grid location and inventory. The hypothesis is encoded as sequence of tokens. In Cartpole the state is the standard dynamics as well as a 1-hot vector specifying the location and color of the zones.
Hypothesis Construction We now describe how the hypotheses for each world in each environment are automatically generated via templates. Three different varieties are considered: (i) triplet hypotheses, (ii) general templates and (iii) special case templates. (See Appendix B.3 for all of the possible templates for an environment and further details).
Here hypotheses take the form of an logical statement: (pre-condition, action sequence) = post-condition. When the pre-condition is true, and the action sequence is performed, the post-condition becomes true. To generate triplet hypotheses, we: (i) randomly select a pre- condition template from a set list; (ii) randomly select action template; (iii) randomly select post-condition template; and (iv) fill in any entities. In our example from Fig. Fig-[thhs would be ("when you are at crafting table and you have stick"; "and then you craft"; "then torch is made").
Triplet hypotheses:
General templates: Instead of drawing a template from the triplet form, a single template for the hypothesis is drawn and the values populated. For instance, in Pushblock, template might be "the door can only be opened when the pushblock is PUSHBLOCK POSITION" and then "left" would be drawn for PUSHBLOCK POSITION. These templates are more general than the triplet ones in that they have no explicit (pre-condition, action sequence and post-condition) structure.
Special cases: We also use more difficult and general hypothesis templates. These cannot be neatly fit into a triplet format by rewording, and may not fully describe the rules of the world. Some examples of these harder templates are: (i) Negating effects (e.g. "door is not open") "); (ii) Negating conditions (e.g. "switch is not on"); and independence (e.g. "door independent of blue switch").
6 EXPERIMENTS
Figure 3 shows 3hows results from learning with pretraining rewards Rpre+ post and Rpeas. There is relatively little variance, with all runs achieving near the theoretical maximal reward For Crafting and Cartpole, Rpre- post is not always achievable if true and distractor items are far away from each other. See Appendix G.1 for further discussion.
In Figure| we show the results on non-triplet adaptation (Sec. B) As discussed in Section 5 this stage includes the more difficult, non-triplet templates not seen during pretraining or during triplet hypothesis training. We also break down the final hypothesis prediction accuracy for the triplet methods in Table [] This allows us to see whether triplet methods were able to adapt to non-triplet hypotheses. All network architectures for the policy and prediction networks, as well as syper-parameters are the same for all methods.
RL baseline Figure shows the RL baseline at chance-level performance, the only exception being CartPole were it achieves 60%, versus the ~ 90% of triplet methods. Note that this is the RL
For Pushblock, sometimes the block can stuck against a wall, so not all worlds are solvable For gridworld environments we show variance runs on 25 seeds and 5 for Cartpole. We provide further hyperparameters, and training details in Appendix Cl and D and network architecture and details in Appendix D

---

Under review as a conference paper at ICLR 2021
baseline with the supervised predictor, as discussed in Section This poor performance is result of training both policy and predictor from scratch; see Figure
Preirwiingg
Figure 3: Pretraining reward (Rere + post our four environments.
SolorSwitch
Pucholock
Crefting
Cartpole
Arowerr Corre:
Policy
Finetune from Pretrain
Baseline
No Act Baseline
Timesteps
23
Rand Baseline
Figure 4: Hypothesis prediction accuracy on both triplet and non-triplet hypotheses (1-1 ratio) for the ColorSwitch, Pushblock, Crafting and Cartpole environments, using Reyp reward for training.
Other baselines We also include two other simple basclines "no act" and "random act." The "no act" baseline simply takes the stop action at f = O, forcing the prediction network to give an answer from just the first observation. This fails because the agent needs take actions in the world to be able to predict the hypothesis accurately. This also confirms that we do not accidentally leak the ground truth of h into the initial observation. For "random act", random policy is used (i.e. uniform across actions). This fails as random actions are extremely unlikely to verify the hypothesis.
Discussion
From these results, we can clearly see that naive RL and other baselines cannot efficiently solve hypothesis verification tasks. When we use triplet pretraining we use the insight that hypotheses often have a clear causal structure that can be exploited when they are formed as "triplet hypotheses." " Not all hypotheses fall neatly into this form, and we may not have this form for all hypotheses. But if we have some that fit this form, we can gain foothold that lets us make progress on this problem, and then later adapt them to other hypotheses. From Figure 4 we can see that this pretraining, triplet training and adaptation works. We also show in Section 6 that other plausible pretraining rewards fail to achieve the same results as method.
Looking at the different environments, we see that in Pushblock and ColorSwitch, even with the policy learned from the triplet pre/post reward, the agent is able generalize and perform well on hypotheses not seen in the pretraining phase as we can see in Table T. Because the finetuning step adds additional variance training, there is small, but non-trivial drop in average performance. This drop is less extreme when comparing max values (see Appendix G.3
Table 1: Average Hypothesis Prediction scores, broken down by triplet (pretrained) and non-triplet
(not seen in pretraining)
Method
Overall
Triplet Accuracy Non-triplet Accuracy
Fixed Policy 86.6%
91.1%
82.1%
ColorSwitch
Finetuned Policy
77.5%
79.7%
75.4%
Fixed Policy 86.9%
87.9%
85.9%
Pushblock
Finetuned Policy
85.6%
86.3%
84.8%
Fixed Policy
77.3%
92.8%
61.8%
Crafting
Finetuned Policy
90.7%
98.4%
83.0%
Fixed Policy
84.2%
92.0%
76.3%
Cartpole
Finetuned Policy
92.5%
93.4%
91.6%
In Crafting and Cartpole on the other hand, do well the unseen templates, the policy also needs to be fine-tuned. This tells us that when we do have to generalize to unseen hypotheses (especially non-triplet hypotheses), adapting the policy well as the prediction network is necessary. Recall
7

---

Under review as a conference paper at ICLR 2021
Intrinsic ColorSwitch Finetuning
Intrinsic Pushblock Finetuning
Intrinsic Crafting Finetuning
Correc %
C
any wanasr
Timestegs
Timesteps
Figure Hypothesis accuracies with intrinsic pretraining versus triplet pretraining (purple).
that we test very different hypotheses such as as negations and "independence" hypotheses not see in triplets (see suplementary). We see from Table [ that indeed, finetuned policies greatly outperform the fixed policies on the non-triplet templates.
We further break down these results by specific templates, entities mentioned (e.g. colors) and whether the given template was true or false. We find for instance that for most environments it is harder to determine hypothesis is false than true. See Appendix Foro the full analysis.
Alternate forms of pretraining
In Figure we show the accuracies on the final hypothesis verification task for both triplet and non-triplet hypotheses, using the four intrinsic pretraining methods discussed in Setion 3 We also plot the final average accuracy obtained by triplet methods from Figure[4 For the intrinsic pretrained policies the best run is shown to show the best-possible case of the alternative methods. For Crafting the dense intrinsic pretraining works about as well as triplet pretraining. This can be explained by the fact that this particular form intrinsic pretraining directly rewards the agent for doing many of the operations in the actual Crafting task, i.c. picking up objects and crafting objects. However, averaging across the three environments, all the intrinsic pretraining methods do worse.
Oracle ablations We also show two oracle analysis in the Crafting environment. In the first, we provide "oracle" hypothesis predictor which will output the ground truth of the hypothesis if it is inferable from the last K frames, and test whether we can learn a policy directly using reward RHyp- Similarly, we also train a hypothesis prediction network with observations from an oracle policy network (observations from which the ground truth of the hypothesis is always inferable).
Orocie Crafting
Dracle Act accuracies
100
Iypotthesis ficcurecy
250
Timestopo
Episndes
Figure 6: (Left) Results on training an RL using Rtyp with oracle predictor on the Crafting environ- ment. (Right) Results training just hypothesis prediction on oracle policy.
Figure (igure<(left) (left) shows the prediction oracle agent quickly converging. From this we surmise that if we can predict the hypothesis already, learning an optimal policy is relatively straightforward. Similarly, (right) show that with the correct observations from an oracle policy, the hypothesis predictor is able to quickly converge well. This shows the joint nature of the problem (i.e. learning both policy and predictor) is what makes it challenging for the RL baseline.
7 DISCUSSION
In this work, we propose a tractable formulation the problem of training agents that can interact with a world to test hypotheses. We show that generic RL techniques struggle with the problem, but we show that methods exploiting the structure of hypotheses are able to successfully complete the problem. We hope that our formulation and analysis serve starting point for researchers to explore the problem hypotheses verification and further on how we can build towards agents that can ask and answer questions about the world.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
A. Bordes, N. Usunier, A. Garca-Durn, J. Weston, and O. Yakhnenko. Irreflexive and hierarchical relations as translations. arXiv 1304.7158, 2013.
Ronald Brachman and Hector J. Levesque. Knowledge Representation and Reasoning. Morgan Kaufmann Series in Artificial Intelligence, 2004.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jinyoung Choi, Beom-Jin Lee, and Byoung- Tak Zhang. Multi-focus attention network for efficient deep reinforcement learning. In Workshops the Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Zenon Colaco and Mohan Sridharan. Mixed logical and probabilistic reasoning for planning and explanation generation in robotics. arXiv 1508. 0008,, 2015.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruy Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 2054-2063, 2018.
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforcement learning, arXiv preprint 1rXiv:190..08/ 2019.
Ernest Davis. Representations of Commonsense Knowledge. Morgan Kaufmann Publishers Inc., 1990. ISBN 1-55860-033-7.
Emest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM, 58(9):92-103, August 2015.
Misha Denil, Pulkit Agrawal, Tejas D Kulkarni, Tom Erez, Peter Battaglia, and Nando de Fre- itas. Learning perform physies experiments via deep reinforcement learning. arXiv preprint 161 2016.
A. Fader, S. Soderland, and O. Etzioni. Identifying relations for open information extraction. In EMNLP, 2011.
Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189228 1971.
Artur S d' Avila Garcez, Krysia Broda, and Dov M Gabbay. -eural-symbolic learning systems: foundations and applications. Springer Science & Business Media, 2012.
Joseph C Giarratano and Gary Riley. Expert systems. PWS publishing co., 1998.
A. Gopnik. Scientific thinking in young children. theoretical advances, empirical research and policy implications. Science, 162337):123-11627 Sept 2012.
Jarostaw M Granda, Liva Donina, Vincenza Dragone, De-Liang Long, and Leroy Cronin. Controlling an organic synthesis robot with machine learning to search for new reactivity. Nature, 559 14): 377, 2018.
Kushal Kafle and Christopher Kanan. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding, 163:3-20, 2017.
Ross King, Jem Rowland, Stephen Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, Larisa N Soldatova, et al. The automation of science. Science, 324(5923):85-89) 2009.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.680 2014.

---

Under review as a conference paper at ICLR 2021
George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Pered From skills to symbols: Learn- ing symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215-289, 2018.
Ilya Kostrikov, Pytorch implementations of reinforcement learning algorithms. Itttp:://githn
com/ikostri httm://ooott.kk///yttcrcchccoce
ppo acktr 2018.
Tamar Kushnir and Alison Gopnik. Young children infer causal strength from probabilities and interventions. Psychological Science, 16(9):678- 007888688222 2005.
Douglas B. Lenat. Cyc: A large-scale investment in knowledge infrastructure. Commun. ACM, (11):33-38, November 1995.
Hugo Liu and Push Singh. Conceptnet practical commonsense reasoning tool-kit. BT technology journal, 224))2211-226 2004.
K. Lu, S. Zhang, P. Stone, and X. Chen. Robot representation and reasoning with knowledge from reinforcement learning. arXiv 1809. 1009.1107, 2018.
Anthony Manchin, Ehsan Abbusnejad, and Anton van den Hengel. Reinforcement learning with attention that works: A self-supervised approach. arXiv preprint 1904v:/904.000 030004003377 2019.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In International Conference on Learning Representations, 2019. URL /ttt:s:
net Ftte/forum?idd rgghhhh
Mitchell, W. Cohen, E. Hruschka, Talukdar, B. Yang. J. Betteridge, Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, Settles, R. Wang, D. Wijaya, Gupta, X. Chen, Saparov, M. Greaves, and J. Welling. Never-ending learning, Commun. ACM, 61(5):103-155, April 2018. ISSN 0001-0782. doi: 10.1145/3191513, URL 3191513
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supprvised prediction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 16-17, 2017.
Judea Pearl. Causality: models, reasoning and inference. Cambridge University Press, 2 edition, 2009.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. CoRR, abs/ 806.03822, 2018.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 74-84, 2013.
Alexander Rush. The annotated transformer. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pp. 60- 2018
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Elizabeth S Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowledge. Psychological review, 99(4)6005 1992.
C. Steinruecken, E. Smith. D. Janz, J. Lloyd, and Z. Ghahramani. Automated Machine Learning. Springer Series on Challenges Machine Learning, 2019.
Suchanck, G. Kasneci, and G. Weikum. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, 2007.
10

---

Under review as a conference paper at ICLR 2021
Joep Vanlier, Christian A Tiemann, Peter AJ Hilbers, and Natal AW van Riel. Optimal experiment design for model selection in biochemical networks. BMC systems biology, 8(1):20, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony R. Dick, and Anton van den Hengel. Visual question answering: survey of methods and datasets. arXiv 1607.05910, 2016a.
Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel. Ask me anything: Free-form visual question answering based on knowledge from external sources. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016b.
Shiqi Zhang and Peter Stone. Corpp: Commonsense reasoning and probabilistic planning as applied to dialog with a mobile robot. In AAAI, 2015.

---

Under review as a conference paper at ICLR 2021
A
APPENDICES
Our Appendices include additional experiments, further details on environments and methods, standard Soyper-parameete and an exhaustive list of the hypothis templates used in this work.
In Appendix B in B.I
B.1we we further detail how we randomly construct environment rules and hypotheses. In B.2 we provide a fuller definition of our notion of "triplets". And B.3 we exhaustively list our templates and named entities.
In Appendix we provide additional training details including training hyperparameters.
In Appendix D we provide details and diagrams of our neural networks, a brief ablation analysis, and network hyperparameters.
In Appendix we show results of the triplet pretraining methods on held-out test hypotheses.
In A we show the performance of the triplet methods on different erosstabs of the hypotheses and rules, including results on each individual hypothesis.
In Appendix we provide additional details of the triplet pretrain methods and justification of some parameter choices and results.
Finally, in Appendix we describe the intrinsic pretraining method in bit more detail and show training curves for the pre-training objectives
B TEMPLATES
B. WORLD AND HYPOTHESIS CONSTRUCTION
Returning again to notation from the main paper, the environment at each spawn needs to construct a world W, and a hypothesis that is either true or false in the world. W in particular describes the rules about how the environment works (i.e. which switch opens the door) which in case can precisely be describe by a hypothesis. So given a true hypothesis, we can exactly describe the rules of the world. Therefore, in order create instance of possible W, a true hypothesis about the world at random. From the hypothesis, we can then construct the rules the determine how objects in the world behave. Note that there are couple exceptions to this for our harder hypotheses, where the hypothesis can be true but only partially describes all the rules of W. For these cases, we draw yet another template which is consistent with the hypothesis and use that to construct the rules, such as deciding which color switch really opens the door.
instead draw
Because have to randomly give either a true false hypothesis, we also need to be able to generate a false hypothesis for the world. So for every instance, we also draw random false hypothesis. Now, given true and false hypothesis, we can fully generate the world and all the items that appear in either statement. So for instance, if the true hypothesis mentions a green switch and the false one mentions a blue switch, we generate both a green and blue switch. Then, we can set the rules such that the right thing happens. So in this example, switching the green switch opens the door and the blue switch does nothing.
The final step is then to randomly choose either the true or false statement as the "visible" hypothesis which is passed to our agent to verify. Because we generate the world and spawn the items before we make this choice, ensure that we do not accidentally give away the truth of the hypothesis based on what items spawned.
Our process for generating spawn of environment thus be summarized follows:
We randomly generate true hypothesis
2. We randomly generate false hypothesis
3. We construct a ruleset from the true hypothesis
4. We spawn the agent and the items in the world described in both the true and false hypothesis
5. We randomly choose either the true or false hypothesis as the "visible" hypothesis that the agent must verify
12

---

Under review as a conference paper at ICLR 2021
B.2 TRIPLET DEFINITIONS
Here we further define our terms "pre-condition" "action sequence" and "post-condition."
Consider a state (here we ignore the state/observation distinction for clarity of explanation, this would be an observation our POMDP). We can consider pre-condition to be boolean function of the state s. Given a state s, we can say whether a particular pre- condition or post-condition is true or false. For example, the condition "the door open" is function of the state s, which we can write as Sthedoorisopen (s) which is true when the door is open, and false when is not. Action sequences are also boolean functions, but over a sequence of states and actions. So for instance, "the agent moves to the craftingtable" can be written as Ttheagentmovesttthen / ($1, 82, ..., d1 02, for some sequence of states and actions, which true when the agent starts not the craftingtable and takes actions that move it to the craftingtable.
post-condition
Now we can connect these three functions or conditions to causality. this construction, what we are doing is hypothesizing causal connection between the pre-condition and action sequence and the post-condition: (pre-condition, action sequence) = cost-condition. This is a logical statement, which be true false, which asserts that when we satisfy the pre-condition and take the action sequence, the resulting state will satisfy the post-condition.
Finally, connect this discription of these terms the string text of the actual hypotheses. We treated these three terms like exact logical functions, but we in fact translate these functions to text strings. So a logical function of whether the door is open becomes "the door is open." " In this work, we write these strings ourselves, although in future work, these could be crowd-sourced, or generated as part of an interactive game. Human language is more complicated and less un-ambigious than mathematical formulations but as we note in the main paper, we train our agents to generalize to new hypotheses and ones that do not follow this exact structure. By our choice using templated language, we keep the kinds of hypotheses agents can learn as general as possible. Future work might explore how to solve new hypotheses online from humans, or even generate its own hypotheses to solve.
In the next sub-section, we exhaustively list the triplet and non-triplet templates we use in this work.
B.3 TEMPLATES
Color Switch:
Pre-condition:
if the COLOR switch is ON_ SNNOOF__SWWE SWITCHSTATE when the COLOR switch is in the ON_ ON_OFF_ON BLL__OFFFSSSSOOEE
WWF__SIITHSSATE position
the COLOR switch ON_OOFF WN__OFF_SSVV SWITCHSTATE
Action:
Post-condition:
then the door open the door passable and we see the door open the door will open
Finetune templates:
the door can only be upened by switching the COLOR switch to ON ON_OFF OFFF WWW.OOFSFFFSSSSSS SWITCHSTAAEE
when we see the COLOR switch ON OFF SWITCHSTATE the door must be open
if the COLOR switch turns ON, OFF SWITCHSTATE the door opens when we see the door open those who want to open the door must first switch the COLOR switch ON OFFOFFFSSSS SWITCHSTATE
be that the COLOR switch
ON. OFF, SWITCHSTATE position
no password just make the COLOR switch be ON_OOF ON_OFFFFS WWF_SWITCHSTATE to open the door COLOR switch OFFOFF LOL.OOFFSSOSES SWISCSSSATT implies door is open
only the COLOR switch being ON_ OFF SWITCHSTATE opens the door the door open hecause COLOR switch is the ON_OFF_STF
OFF_SWIICHSTATE position
13

---

Under review as a conference paper at ICLR 2021
COLOR switch ON OFF SWITCHSTATE equals open door the COLOR switch opens door only when is LOFO_SSSSE SWTTCHISTATE
door is open must meun that COLOR switch is ON_OFF__AF SWITCISSATEE an ON_OFF WN_OOF__SWTEEE SWITCHSTATE means the dour open but only if it is COLOR
COLOR controls the door and it opens when it is ON_ NN_FF__ SFF_STCCCSSTTE
ON OFF SWITCHSTATE the switch that causes the door to open when it is ON ON_OFF Sa__FF___MWWS SOF__STTCHSTATE COLOR
correct position the COLOR switch and opens the door
if you see COLOR switch then the door is open the door independent of the COLOR switch the door the COLOR switch is ON_ ON_OFF S___FF____TTT SWITCHSTATE. then the door open
open then COLOR switch be ON OPF SWITCHSTATE
to make the door not open the COLOR switch must not ON_OFF_J LLL.OOFFFSSSOOS SWITCHSTATE whether the door is open is completely independent of the COLOR switch
the COLOR switch is what controls the door
ON OFF SWITCHSTATE COLOR switch opens the door
Template Values
COLOR:
blue
red
green
black
ON OFF SWITCHSTATE:
off
Pushblock
Pre-condition:
whenever the pushblock is in the PUSHBLOCK POSITION pushblock at the PUSHBLOCK NOSSSTIOCKPPOSTTION the pushblock is at the DUSHBLLOCK_POSTTION
Action:
Post-condition:
then the door open the door is passable and the door open the door open
SP FULL, TRAIN:
PUSHBLOCK POSITION is the correct position for the pushblock for the door to open if the door open when the door open because pushblock is PUSHBLOCK POSITION when the pushblock at the PUSHBLOCK POSITION PISSBBLOCK,OOSITOON the door is open pushblock PUSHBLOCK BBt.OOOCCCPCCOOONNNO the door only be opened when the pushblock PUSHBL pushblock PUSHBLOCKE POSITION means the door open PUSHBLOCK POSITION pushblock opens the door
be that the pushblock is PUSHBLOCK POSITION
POSITION means door open
open door implics pushblock PUSHBLOCK POCIITON open door means pushblock PUSHBLOCK POSITION door opens when PUSHIBLCKK COOSITIN where the pushblock is
pushblock to open the dour the door when the pushblock is PUSHBLOCK L
PUSHIBLCC_JF P
correct position
POSITION is open
PUSHBLOCK POSITION B position of the pushblock causes the door to open door only opens on PUSHBLOCK CECCPOOTTIION pushblock
door only open with pushblock heing PUSHBLOCK POSITION
14

---

Under review as a conference paper at ICLR 2021
the pushblock being the PUSHBLOCK POSITION is completely independent of the door the pushblock being PUSHBLOCK POSITION independent door being open the door state is independent of pushblock PUSHBBOCKK NOCK_POSTTION PUSHBLOCK_POO POSITIIN pushblock and door are independent
Pushblock values:
PUSHBLOCK POSITION::
left
right
top
bottom
Crafting
Pre-condition:
when you are at LOCATION and you have CRAFTING_ITER ITEM you are at LOCATION and have in your inventory CRAFTING ITEM ITEM
whenever you have CRAFTING_ITRAL and LOCATION
Action:
and you CRAFTING_ACTION
then CRAFTING_ACTION
Post-condition:
you now have CREATED ITEM in your inventory
then CREATED ITEM
PREATEDJITEM is created
and this creates CREATED_ITHD JITM
CREATED_ITED is created and put in your inventory
then CREATED ITEM is made
Finetune Templates:
create SLEATED_ITEN you must have CRAFTING JIEM and go to LOCATION and do the action CRAFT ING_ACTION
CREATED, ITEM can created by doing CRAFTING_ ACTIONN LOCATION when CRAFTING ITEM is inventory whenever you do CRAFTING, CAFFING_ACTION ACTION and have CRAFTING, ITEM at LOCATION CREATED ITEM made you have CRAFTING, JTEM and go to LOCATION and CRAFTING, ACTTON and CREATED CREATED_ITEM JTE will be created
whoever does CRAFTING AATIONN LOCATION with CRAFTING ITEM gets CREATED. ITEM
you have CRAFTING. ITEM LOCATION if you CRAFTING CRAFTING,AATION ACTION at LOCATION with CRAFTING JTEM you make CREATED JITEM
you CRAFTING_ACTIN ACTION you get CREATED ITEM
whenever you have CRAFTING CAAFINNGITEMM=- at LOCATION and CRAFTING then you make a CREATED ITEM
having CRAFTING_ITEM in your inventory being at LOCATION and doing TRAFTING_ACTION creates CRE-
ATED IED_ITEM
CREATED ITEM can made with CRAFTING ITEM when you do CRAFTING ACTION LOCATION
CRAFTING ITEM plus LOCATION plus CRAFTING, ACTION equals CREATED ITEM
create a CREATED ITEM by being at LOCATION with CRAFTING ITEM and doing CRAFTING ACTION CRAFTING_ACTION LOCATION creates CREATED JTEM but only you have CRAFTING_IT JTEM
if you make CREATED .IEM then go LOCATION with CRAFTING JTEM and do CRAFTING_ACTION CRAFTING. ITEM inventory LOCATION makes CREATED JIEM you CREATED ITEM when CRAFTING ITEM LOCATION and do CRAFTING_ACTIR if you are at LOCATION and do CRAFTING, ACTION you make CREATED ITEM
if anywhere and do CRAFTING ACTION with CRAFTING, LTMTINGITTMMM you make CREATED having CRAFTING ITEM LOCATION and doing CRAFTING AATIONN does make CREATED ITEM CREATED ITEM is created by being at LOCATION and doing
make a CREATED ITEM by having CRAFTING ITEM and doing CRAFTING ACTION have CRAFTING_ JIEM and go to LOCATION and SRAFTINGAACTONN and CREATED JTEM will not be created LOCATION plus CRAFTING_ CATTINGACTION creates CREATED -EETEEDITTMM a CRAFTING_ JTEM you can make a CREATED. JTEM by doing CRAFTING_ACTION
CRAFTINGAACTION
Template Values:
CRAFTING ITEM:
15

---

Under review as a conference paper at ICLR 2021
wood
stick
pickaxe
CREATED ITEM:
torch
LOCATION:
craftingtable
CRAFTING ACTIONN
craft
Carpole
Pre-condition:
when are in the COLOR zone if are in the COLOR when the agent within COLOR
Action:
Post-condition:
the gravity is MULTIPLIER
gravity is MULTIPLIER the wind pushes DIRECTION there is DIRECTION wind
Finetune Templates:
MULTIPLIER gravity is caused by the COLOR zone if you to COLOR zone then gravity is MULTIPLIER the gravity is MULTIPLIER because the agent is in the COLOR COLOR zone implies MULTIPLIER gravily
gravity MULTIPLIER whenever you go into the COLOR zone the COLOR causes gravity to MULTIPLIER
to make gravity MULTIPLIER noed to be in COLOR
COLOR zone gravity MULTIPLIER
gravity multiplier of MULTIPLIER caused by heing in COLOR
COLOR equals wind DIRECTION when the wind blows DIRECTION it is because you are in COLOR
COLOR zone causes DIRECTION wind only heing in COLOR makes the wind blow DIRECTION DIRECTION wind when you are in the COLOR zone there is a DIRECTION wind DIRECTION wind is caused by being in the COLOR zone wind pushing DIRECTION whenever you are in COLOR gravity totally independent of COLOR
COLOR zone
COLOR zone not effect gravity is independent the wind is completely independent of the COLOR zone independent of wind DIRECTION is COLOR gravity is changed by heing COLOR but not MULTIPLIER the effect being COLOR opposite gravity MULTIPLIER the wind blows opposite DIRECTION when COLOR zone being in COLOR causes the wind to blow opposite to DIRECTION
16

---

Under review as a conference paper at ICLR 2021
Template Values:
COLOR:
blue
red
green
black
MULTIPLIER:
decreased
increased
DIRECTION:
left
right
17

---

Under review as a conference paper at ICLR 2021
C LEARNING DETAILS AND HYPERPARAMETERS
One detail of the prediction network that we need to keep memory of past state sequences, hypotheses and ground truths
can actually train our prediction network. We do this by simply keeping track of the last N times our agent answered a question, and keeping these in a FIFO memory. When we update our prediction network, we randomly sample from this pool. This also necessitates a 100k step break in period to collect enough examples.
In our policy finetuning experiments, we also stabilize our dual optimization problem by trading of optimization of the policy network and the prediction network. We must also start with the prediction network so that the reward for answering correctly meaningful.
No extensive grid search of prper-parameters was conducted. At various stages we experimented with different hyperparameter values especially PPO parameters (timesteps per batch), learning rates, PPO epochs, and optimizer.
Experiments were performed on Tesla K40, TitanX and TitanX Pascals GPUs on 2-4 GPU worksta- tions. Several johs could be run single GPU with the greatest performance bottleneck being CPU cycles rather than GPU cycles or memory.
Table 2: Pretraining Hyperparameters
Parameter
Value
Algorithm
PPO
Schulman al. 2017
Timesteps per batch
2048
Clip param
0.2
Entropy coeff
0.1
Number of parallel processes
8
Optimizer epochs per iteration
4
Optimizer step size
2.5c
Optimizer batch size
32
Discount ?
0.99
GAE X
0.95
learning rate schedule
constant
Optimizer
ADAM Kingma & Ba (014)
Past Frame Window Size
5
Table 3: Finetuning Hyperparameters
Parameter
Value
Algorithm
PPO Schulman al, (011
Timesteps per batch
2048
Entropy coeff
0.1
Number of parallel processes
8
Optimizer epochs per iteration
Optimizer step size Optimizer batch size Discount ?
1e-5
32
0.99
GAE A
0.95
learning rate schedule
constant
Optimizer
SGD
Past Frame Window Size
Basis of RL implementations was from Kostrikov 2018
18

---

Under review as a conference paper at ICLR 2021
Table 4: Prediction Hyperparameters
Parameter
Value
Timesteps per batch
2048
Optimizer step size Optimizer batch size learning rate schedule
1e-3
128
constant
Optimizer
ADAM Kingma & Ba 2014
Memory Burn-in
100000
Memory Size
200
Alternate Training Window
10000000
19

---

Under review as a conference paper at ICLR 2021
D NETWORK DETAILS
Although other works such as Chaplot et al. (2018) have investigated language-conditionel RL (usually in the form of instruction following), our hypothesis conditioned problem proved to be challenging, and required some novelty in network architectures. Figure shows our network diagrams.
Other works such as Chaplot et al. (2018) have incorporated gated mechanisms between language and perception. Manchin al. (2019) employs self-attention mechanism within convolutional layers and Choi et al. 2017) also employs self-attention mechanism in DQN. Neither work incorporates language and the architectures are quite different from each other. Figure shows the policy and transformer architectures (this is also in the main text).
For the policy network, it was important to use key-value attention. That is: the hypothesis is fed into seq2vec model and is used as the key of a dot-product attention mechanism. The state (the grid locations of the items the world and the inventory the agent if applicable) fed as input to N parallel MLPs. The output of the MLPs then fed the values of the attention mechanism. The output of the module is then fed into the final hidden layer of the actor-critic network.
For the prediction network, prediction network encodes both the hypothesis and past observations (after they are passed through layer network) using transformer encoders. These sequences then combined using transformer to generate a final hidden state as output which is then fed to a final prediction layer and sigmoid function get binary prediction.
the popular transformer architecture Vaswani et al. (017). Our
Policy Network
Prediction Network
Input Embedding
Tanih
Jn
Attention
Mult-heed
Nk
&
seg2uRc
Attention
Feed Forward
MLp
Add Norm
Add & Norm
softmax
Feed Forword
Masked Multi-heed Attention
Nx
Add & Norm
Actor/Critic
Multihead Attention
Add & Norm
sigmoid
Figure Network architecture for our policy network (left) and prediction network (right)
D.1 NETWORK ABLATION
In Figure g we see results of our network architecture ablation. As we can see, our new policy architecture described previosuly clearly outperforms a standard MLP policy network on the language- condition pretraining task. We also see that the transformer architecture outperforms the LSTM and MLP model on the final task when we hold the policy network constant.
D.2 IMPLEMENTATION AND HYPERPARAMETERS
We take much of our implementation of transformers from Rush (0088)
20

---

Under review as a conference paper at ICLR 2021
Policy Net Ablation
Prediction Net Ablation
17.5
Answer Correct
0.5
30
Timesteps
Timesteps
Figure 8: (left) policy network ablation (right) prediction network ablation.
Table 5: Policy Network Hyperparameters
Parameter
Value
Seq2Vee Model
Mag-oo-WWore
Word Embedding Size
32
Hidden Size
32
MLP Num Hidden Layers
2
Number of MLP Modules
16
Transfer Layer
tanh
Table 6: MLP Baseline Policy Network Hyperparameters
Parameter
Value
Seq2 Model
Bag-of-Words Words
Word Embedding Size
32
Hidden Size
32
MLP Num Hidden Layers
2
Transfer Layer
lanh
Table 7: Transformer Network Hyperparameters
Parameter
Value
Word Embedding Size
32
Hidden Size
32
Transfer Layer
ReLU
Transformer N
3
Table 8: Baseline Prediction Network Hyperparameters
Parameter
Value
Seq2Vec Model
LSTM
LSTM Num Layers
Word Embedding Size
32
Hidden Size
32
MIP Num Hidden I Layers
Transfer Layer
tonh.
21

---

Under review as a conference paper at ICLR 2021
E TEST-TIME HYPOTHESES
To examine whether the hypothesis prediction networks are overfit to the very specific hypotheses the networks were trained on, we conduct a simple experiment and we test on different hypotheses. Here we keep the same underlying distribution of ground truth rules, but we change the hypotheses to ones not seen in any stage of training, and do not adapt the networks in any way to these hypotheses. The results are shown in Table for the triplet pretraining method. We can that for the most part, there is little to no drop in performance, with the exception of a substantial drop in crafting, but only for the fixed policy. The finetuned crafting policy still achieves comparable results to hypotheses it. was trained on.
Table 9: Average Hypothesis Prediction scores on new templates
Method
Hypothesis Accuracy
Fixed Policy
81.49%
ColorSwitch
Finetuned Policy
71%
Fixed Policy
89%
Pushblock
Finetuned Policy
84.87%
Fixed Policy
58.22%
Crafting
Finetuned Policy
90.3%
Fixed Policy
78.97%
Cartpole
Finetuned Policy
81.3%
F CROSSTAB PERFORMANCE ANALYSIS
In this section, we further break down the results in the main paper by specific templates, entities mentioned (e.g. colors) and whether the given template was true or false. We do this analysis only for Fixed and Finetune as the baselines fail fairly uniformly and the analysis would be less interesting.
We get these results by running our pretrained models on at least 1000 of each subcategory listed and averaging the results across results (and also across runs as in our main results). This gives us good sample for analysis of which types of hypotheses methods perform well or poorly on.
First we look at the accuracy sorting by whether the agent was given a true hypothesis to validate or a false hypothesis to invalidate in Tables [obles[o0 [1] and [3 We see that, except for finetune accuracy on crafting, the agents have a generally much easier time confirming true hypotheses.
One possible explanation for this of course is that the observations that confirm a true hypothesis and the policy needed to do this are much simpler. To confirm a true hypothesis, the agent must essentially perform the hypothesis described exactly. So if the hypothesis is that flipping the green switch on opens the door, the correct policy is exactly going to the green door and toggling. And then the state affect is immediate. If following this causes the door to open, the state changes in very obvious way- the door opens. Whereas if the statement was false, the agent would have to look at the sequence of states and infer that indeed did exactly the right hypothesis and the effect did not happen. Noticing a change in state is often easier than noticing a lack of change.
However, there may be other subtle things going on related to the agent behavior. For instance, if an agent's policy was to guess True if it does not know the ground truth, it would by definition do better on true hypotheses. Indeed, if we look the actions pushblock agent, we notice this. Recall that if the block is against a wall, the agent may not be able to trigger the post-condition. But actually, the agent's 's odds here are better than random. If the block is against a wall, can test two possible door conditions (if it's stuck on the left, it can test up and down). By doing this, it is able to test two alternate hypotheses. If the agent wants to test a hypothesis that moving to the right opens the door, it test moving it up and down. If it tries both of these and the door does not open, it is more likely that it is true than not. So the agent will guess True. This leads to a high True answer rate, but this stragegy will always fail in those less likely cases where that hypothesis was actually false.
Next we look at the "elements" i.c. the values in the templates such as the color of the switch or the item being crafted. See Tables 14 1 16. and 17 With the notable exception of Cartpole, the element
22

---

Under review as a conference paper at ICLR 2021
Table 10: Colorswitch T/F Crosstabs
True/False
Fixed Acc
Finetune Acc
True False
0.941558 0.788904
0.835376 0.721134
Table 11: Pushblock T/F Crosstabs
True/False
Fixed Acc
Finetune Acc
True
0.969079 0.760620
0.959164 0.746091
False
does not substatially change the value. The exception is Cartpole, but this is actually because unlike the other environments, there are really two distinct hypothesis effects which are not equivalent: wind versus gravity. In the hypotheses which mention left or right, the agent is actually testing hypotheses about which way the wind is blowing (left right) whereas the half and double hypotheses are testing the gravity. We see that the agent performs better wind hypotheses than on gravity, but the direction or gravity effect does not substantially effect the result. This is likely due to the fact that when the pole of the cartpole is level, the effect of gravity is less obvious. If you read the original cartpole code, you can see this. So these are harder because it requires the agent to have the pole in the less stable, non-centered position. Whereas for wind, the effect of wind force is immediate and more obvious to the agent (there is now an opposing force the agent must act against to keep the pole upright).
and ??. The results can be somewhat overwhelming, but a few general trends emerge. First, we can see that there quite bit of differing in performance for specific template types. Some specific templates can have very low performance for both fixed and finetuned. One very interesting result is for the special case templates. For Colorswitch and Pushblock, we find again that the general trend is that the fixed accuracy models perform a bit better (see Appendix G.3). But on Crafting and Cartpole, we see that finetuned are able to do significantly better than fixed. In Crafting this includes negation and recipes that are missing the location or input crafting item. In Cartpole this negation and independence statements.
Finally, we break out results by each template in Tables 18
23

---

Under review as a conference paper at ICLR 2021
Table 12: Crafting T/F Crosstabs
True/False
Fixed Ace
Finetune Acc
True False
0.819272 0.730525
0.897819 0.913886
Table 13: Cartpole T/F Crosstabs
True/False
Fixed Acc
Finetune Acc
True False
0.876085 0.822460
0.894504 0.850877
Table 14: Colorswitch Elements Crosstabs
Element
Fixed Acc
Finetune Acc
blue 0.860603
0.817654
black 0.874475
0.711046
red
0.868067
0.699999
green
0.857772
0,883432
on
0.863618
0.779900
off 0.859275
0.772164
Table 15: Pushblock Elements Crosstabs
Element Fixed Acc
Finetune Acc
left
0.881174
0.860691
bottom right 0.833578
0.869612
0.866311
0.830409
top 0.874413
0.853540
Table 16: Crafting Elements Crosstabs
Element Fixed Acc
Finetune Acc
pickaxe
0.749928
0.885686
craft
0.774918
0.905880
torch
0.790433
0.931534
stick
0.790664
0.927625
coal
0.766912
0.884030
bed
0.759223
0.880414
iron
0.783844
0.890114
wood
0.808903
0.939032
24

---

Under review as a conference paper at ICLR 2021
Table 17: Cartpole Elements Crosstabs
Element Fixed Acc
Finetune Acc
blue 0.856136
0.882224
black
0.844531
0.868902
red
0.855209
0.898623
green
0.841047
0.840754
left
0.865710 TO
0.876526
right
0.873185
0.901369
half
0.824625
0.846780
double
0.839608
0.846751
Table 18: Colorswitch Template Crosstabs
Template
Fixed Acc
Finetune Acc
no password just make the COLOR swich be ON_OOH SHP__WIICHSAAAE open the door
0.805164
0.715492
OFF SWITCHSTATE: the COLOR controls the door and it opens when is ON OFF SWITCHSTATE
CDL OR
0.909462
0.809612
0.757894
0.636947
open BL__oo______ES SWITCHSTATE equals open door
ONLOOE SWITCHSTATE
0.738385
0.784663
COLOR switch ON LOVE
0.771802
0.654603
BLR__OF______ _WWTCHSSTTEE
passable
0.909699
0.809354
if the COLOR switch ramss ON_OFF_ WW__OOF_SWIICHSSSS SWIECHSTATE the door
0.845888
0.808421
the COLOR switch ON OFF SWITCHSTATE dour pussable
0.909085
0.806334
when the COLOR switch is in the ON OFF SWITCHSTATT position door is pessable OR suitch ON OFF SWITCHSTATE
0.909311
0.798914
see the door open
0.910084
0.808421
ON_OFF_ WW..SIIIHSAAAE TATE the
of COLOR swisch and opens door
0.786958
0,617625
the COLOR switch is ON_ ON_OFF__P BL__O_______OES SWW.CHSSTTEEEEE the door will upen
0.911169
0.813248
COL switch ON SWITCHSTATF door open
0.860697
0.792323
if you see COLOR switch then the door is open
0.847181
0.761045
the COLOR
SWITCHISTATE position then open
0.908866
0.805041
the COLOR switch's is what cuntrols the door
0.865501
0.790601
position
open
0.911304
0.801073
when the COLOR switch is in the ON LOW WWW____SSICCHHEE SWITCHSTATE position and we see the dooc is open ON_ ON_OFF BOL_O____SEEE WWW.CWSSTTTSEEEEE
0.909594
0.800795
0.907675
0.805454
see open
the doer becomse COT OR swith is in the ON OFF SWITCHSTATE position
0.879004
0.792435
doer can only be openod by switching the OR to ON, OFF SWITCHSTATE the switch that the door open when OFF SWITCISSANTE COLOR the COLOR switch ON ON_OPF_S
0.741864
0.727885
0.766324
0.701296
BLL_OOFF__S_S__ WWW.SISSTSTTEEEEEEEEIEE the dour will open
0.910904
0.809709
if the COLOR switch
WWW.CHSTATE
opea
0.909178
0.803808
only the COLOR switch being WN_OFF__SCC ON_OFF_SWIICHST WWW.SSSTTHHHHHHCC.. opens the door door open must
0.800769
0.768345
COLOR switch
SWITCHSTATE
0.782568
0.749175
um ON OFF SWITCHSTATE: means the door is open but only if is COLOR
0.812291
0.813397
open the COLOR swisch opens the door but only when ONLOOONOOOSSS OFF.SWIICHHATTE
switch
SWITCHSTATE position
0.827992
0.757434
0.842028
0.685985
SONOOFF_SWWOR SWITCHSTATE implies
open.
0.804614
0,806208
to make the door the COLOR switch mas not ON, OFF SWITCISTATE if the dour not upen then the COLOR swisch musi be ON OFF SWITCISSSTEEE
0.728761
0.698866
0.855276
0.796445
BON_OOH__SSPO WWF_SSWICHHIALE COLOR switch opens the
0.782377
0.704020
the COLOR switch is not ON__O _OWF SWITCHSTATE then the door is open
0.881558
0.805187
whether the open completely independent the switch the door is indopendent of the COL swith
0.937662
0.850311
0.976774
0.803929
25

---

Under review as a conference paper at ICLR 2021
Table 19: Pushblock Template Crosstabs
Template
Fixed Ace
Finetune Ace
door can only open with pushblock being WUS POSITION
0.820393
0.856789
the pushblock the CAAIBBLOKKPOSSTTTI then the door
0.869955
0.859727
PUSHBLOCK POSITION is the conrect positicn for the pushblock open the door
0.872479
0.843071
the pushbblokk PUSHBLOCK K_PSITTION
passable
0.872433
0.863410
the pushblock PUSHBLOCK, BESSBLLOKKPOSTIIOO POSITION menss the door is open
0.876048
0.850846
pushblock PUSHBLOCK POSITION dooe open
0.874485
0.860658
the pushblock at the PUSHBB .CK POSTTION the door passable PUSHBLOCK_
0.874790
0.859502
pushblock opens the
0,878205
0.858935
the pushblock the PU PUSHBLOCK POSITION and we see the door open opens when the door is open it must be that the puchblock is the PUSHBLOCK POSITICI
0.870191
0.860378
the pushblack
0.857907
0.853671
0.868565
0.840094
pustblock
PUSHBLOCK POSITION
open
0.877208
0.854677
heneverr the pushbluck the PUSHBLOCK POSITION door passable
0,870555
0.866253
open door means pushblock PUSHBLOCK POSITION
0.835
0.857000
pushblock
PUSHBLOCK BWCHHBBLUKKKPPAEP POSITION
the open
0.871365
0,855871
the door can only be opened when the pushblock is PUSHBLOCK_POSS POSITION pushblock PUSHRI POSITION
0.871308
0.855845
0.839570
0.861030
open
weneever the pushblack the RESSBBLCC POSITION then the door open open
0.878206
0.858112
pubbblock PUSHBBLCK POSITION
0.872247
0.846558
henever the pushblock the PUSHBLOCK POSITION and we see the door is open
0.870350
0.860942
open implies pushbluck PUSUBLOCKK
0,832304
0.861506
the pushblock the PUSHBLOCK PUSHBLODCK_POSIIIOOO POSITION the doer will
0.875567
0.856274
pustiblock
PUSHBLOCK
open
0.878546
0.847757
the dooe when pushilock PUSHBLOCK POSITION
0.869970
0.849398
the pudbllock the PUSHBI OCK
door open
0.872346
0.855193
PU PUSHBLOCK_PP POSITIO position of pushblock door open
0.871925
0,833536
door only opeas on PUSHBBLCCK POSITION pubblock
0.833290
0.843573
PUSHBLOCG POSITION
position pushblock door open
0.865061
0.846289
the pushblock being PUSKBLOCK POSITION is independent of the door being open
0.845120
0.828433
independent pushblack PUSHBLOCK POSITIO
0.848979
0.831997
PL PUSHBLOCK CHHBLOCK_PUSTTOON POSITION pushiblock and door independear
0.838476
0.824016
puchblock being PUSHBLOCK BOCEBLLOOKOOSSOOOO completely independees
0.851769
0.836789
26

---

Under review as a conference paper at ICLR 2021
Table 20: Crafting Template Crosstabs
Template
Fixed Acc
Finetune Acc
you are LOCATION have in your inventory CRAFTING_ITER
you do CRAFTING CRAFINGGACTION ACTION then CREATED ITEM made
0.980752
0.992622
you want to CREATED
go to LOCATION with
CRAFTING ITEM and do CRAFTING ACTION
0.618157
0.751656
CREATED ITEM when CRAFTING ITEM LOCATION CRAFTING ACTION
0.621 186
0.833183
you LOCATION have your inventory CRAFTING then you CRAFING_ACTION
so CREATED LURIEEETEEMM created and put in your inventory
0.909046
0.994463
when you LOCATION you have CRAFTING ITEIINGGIEEMMMM
CRAFTING ACTION thas CREATED ITEM made
0.882973
0.990093
when you are LOCATION and you CRAFTING ITEM you
do CRAFTING ACTION then CREATED ITEM is mude
0.988334
0.992137
you CRAPTING
LOCATION CRAFTING you CREATED
0.702412
0.981659
you at LOCATION and have in your inventrry CRAFTING_IEEN and you do CRAPLINGACTIOON
you new CREATED JIEM your inventary
0.903610
0.993235
whenever you have CRAFTING, ITEM and are LOCAATIIN then CRAFTING, ACTION
you now CREATED ITEM your inventery
0.920456
0.991195
CRAFTING ITEM plus LOCATION plus CRAFTING ACTION equals CREATED JTEM
0,493720
0,596603
CREATED JTEM can be created by doing CRAFTING_ACTION LOCATION when
CRAPTING ITEM in inventury
0,431615
0.546072
you are at LOCATION and have in your inventry CRAPTIN_ITEM,
you CRAPTING CANTINN then CREATED JTEM created
0.980229
0.991235
you one at L.OCATION and have in your inventory CRAFTING ITFM
CRAFTING ACTION CREATED ITFM created
0.957643
0.993386
CRAPTING ACTION at LOCATION creates CREATED ITEM but only you have CRAFTINGJITEM
0,571485
0.913526
when you are LOCATION and you have CRAFTING ITEM
and do CRAFTING ACTION then CREATED JTEM created
0.988463
0.994632
whenever you have CRAPTING JIEEIINIGGIMMMMMM are LOCATION
then you CRAFTING ACTION and this creates CREATED ITEM hemever you have CRAFIN ITEM LOCATION
0.863886
0.990727
CRAFTING ACTION then you make CREATED ITEM
0.859223
0.995383
whea you are LOCATION and you have CRAPTINGJTER JTEM and you do CRAPTING ACTION and this creates CREATED ITEM
0,896466
0.993168
create CREATED ITEM you must have CRAPTING ITEM go LOCATION
CRAFTING ACTION
0.580112
0.747054
hemever you have a CRAFTING, ITEM and are at LOCATION and you do CRAPTING ACTION CREATED ITEM
and put your inventury
0.868949
0.795353
you ane al LOCATION and have in your inventory CRAFTING_ITES ITEM
you CAAAATNGGAATTOOO then CREATED LTM
0.935798
0.991869
whenever you do CRAFTING CAATTINGAAATTIOON and have CRAFTIN_ITEM
OCAATTTINN a CREATED ITFM made
0.655783
0.918542
you at L.OCATION and have in your inventory CRAFTING then you CRAFTING ACTION and this creates CREATED JTEM whenever you have CRAFTING ITEM and LOCATION then you CRAPTING ACTION then CREATED, JTEM ATEEDITTEEMMMM created
ITFM ITFM
0.920180
0.994741
0.963260
0.993860
27

---

Under review as a conference paper at ICLR 2021
Table 21: Crafting Template Crosstabs (Continued)
Template
Fixed Acc
Finetune Acc
Whenever have CRAPTING ITEM and are LOCATION and you do CRAFTING ACTION and this creates CREATED ITFM
0.838137
0.994172
whenever you have CRAFTING ITEM and LOCATION and you CRAFTING then CREATED JTEM created you LOCATION and have inventory CRAFTING JTEN and you do CRAATING ACTION and this creates CREATED
0.891907
0.914471
0,937222
0.990594
you LOCATION you now have CREATED ITEM in your inventory
inveatory CRAPTING you CRAFTINE,ACTION
0.885896
0.989596
hoving CRAFTING ITEM your inventory
LOCATION and doing CRAFTING ACTION cremte CREATED ITFM
0.487408
0.817408
create CREATED
by being LOCATION with
CRAFTING ITEM and doing CRAFTING ACTION
0,545770
0.919413
when you are LOCATION and you have CRAPTING IIEM and you do CRAPTING_ACTION so CREATED_ ITEM is created and in your inventory
0,956038
0.991977
whorver does CRAPTING ACTION at LOCATION with CRAFTING ITEM gets CREATED ITEM
0.483942
0.566790
when you LOOCATION and have CRAFTING JTEM then CRAFTING ACTION
so CREATED_IEEEE JIEM is created and put in your inventory
0.973726
0.996132
you have CRAFTING_ITER und CREATED JTEM will be created
LOCATION CRAPTING_ACTION
0.703044
0,914437
CRAFTING ITEM inventory LOCATIIO makes CREATED JTEM you do CRAFTING ACTION
0.545541
0,897572
you are at LOCATION and have in your investory CRAFTING IIEM and you do CRAFTING so CREATED ITEM
pui in your inveniory
0.898226
0.919370
LOCATION plus CRAPTING_ ACTION creates CREATED JTEM
0,541849
0.873599
when you are LOCKTION and you CRAPTINGIITE and you do CRAPTINGG you now have CREATED ITEM in your investocy when you are LOCATION and you have CRAPIINGJIEE thes you
0.972194
0,994262
CRAFTING ACTION and this crentes CREATED ITEM CREATED ITEM con be made CRAFTING ITEM when you de CRAFTING ACTION LOCATION
0.941614
0.994693
0,578862
0,687136
whenever you have CRAFTING (JGTTMME and are at LOCATION and you do ERAFTING_ATTION then CREATED JITEEEMMMMM made
0.891969
0.993853
whenever you have CRAPTINGITTSS ITEM and are LOCATION then you CRAFTING ACTION then CREATED ITEM
0.933055
0.989965
whenover you hove CR CRAFTING ITEM and are at LLOCATION then you CRAPTING ACTION
CREATED ITFM
put your inventory
0.900469
0.923228
whenever you have CRAPTING ITEM and are LOCATION and you do CRAFTINGAATTION
you now have CREATED ITEM in your in inventory
0.909898
0.993880
when you are LOCATION and have CRAFTING_ITER ITEM thes CRAFTING
you now have CREATED ITEM in your investory
0.938361
0.993092
you have CRAPTING, ITEM LOCATION and CRAFTING_ACTIOONOO you CREATED ITEM
0.832800
0.993352
when you are L.OCATION and you have CRAFTING ITEM then you
CRAFTIN_ACTION then CREATED JIEM created
0.885430
0.992530
you have CRAFTING ITEM and go to LOCCATION and CRAFTING ACTION
CREATED JIEM not created
0.777171
0.827805
having CRAFTING ITEM at LOCATION and doing CRATINNGACTTON
not CREATED ITEM
0.647106
0.800454
make CREATED ITEM by having CRAFTING and doing CRAFTING ACTION
0.531967
0.757789
you are anyoficre and do CRAFTING ACTION CRAFTING_ITEEE you make CREATED JIEM
0.698206
0.875884
make a CREATED ITEM by doing CRAFTING ACTION with CRAFTING ITEM
0.550868
0,885985
If you are LLOCATION and do CRAFTING, ACTION you make CREATED ITEM
0.872249
0.999038
CREATED ED JIEM created LOCATION and doing DAATINNGAACTION
0.676661
0,872774
28

---

Under review as a conference paper at ICLR 2021
G ADDITIONAL DISCUSSION / DETAILS
G.1 PRETRAINING ADDITIONAL DISCUSSION
For the ColcrSwitch environment, we found that pretraining with just the pre-condition reward leads to better results for the Color Switch environment and show those results here. We chose C 10 for this proxy reward, so we see from our figure that we are able achieve the pre-condition toggling almost perfectly.
For Pushblock, we use both Rpret-post and Rpast, however, because of the afformentioned issue the block getting stuck against wall, it does not achieve perfect success each time.
For Crafting we see that pretrain results converge towards around 15. This is because the best the pretraining can do is to try to complete the recipe. If the hypothesis is false, however, we often cannot toggle the post-condition because often requires more than K = steps to complete seperately from toggling the pre-condition.
Similarly for Cartpole, we see the same thing, except it can get to around 17.5. This is because if the true and false zones adjacent to each other, the agent can toggle both the pre- and post-conditions successfully (try pre-condition, when doesn't 't work, move the the adjacent zone where the effect is true).
G.2 ALTERNATIVE CUTOFFS FOR ADAPTATION TRAINING
As promised, in Figure 9 we show the adaptation results when we choose an 80% cutoff instead of 90%.
Figure 9: Hypothesis prediction accuracy both triplet and non-triplet hypotheses for the Color
Switch, Pushblock, Crafting and Cartpole environments, using RHyp reward for training.
G.3 TRAINING VARIANCE FOR PUSHBLOCK AND COLORSWITCH
In the main paper we discussed the problem of the finetuning training adding additional variance training, resulting in worse average performance. First, we can actually see this from the training curves themselves in Figure 4 the main text. The orange curves of the pretraining reward matches the blue fixed reward curve, but has more variance. In the ColorSwitch example, it has a hickup around 1.5e7 timesteps that it does not fully recover from. This is because, unlike the fixed method, this method has to simultaneously train both the policy network and the prediction network. So when the policy network achieves less-optimal policy, becomes more difficult to train the prediction network which feeds back into the policy network. Whereas the fixed method only trains prediction and is much more stable during training.
We also mention that for some runs, this becomes less extreme. Some runs of the finetune policy do much better than average. In colorswitch, the best finetune run achieves 83.5% accuracy versus 90.5% for fixed and for pushblock achieves 87.6% versus 89%, smaller difference than the averages Table I.
29

---

Under review as a conference paper at ICLR 2021
H
INTRINSIC PRE-TRAINING EXPERIMENTS
Intrinsic Swich Pretraining
Intrinsic Swichh Pretraining
100
30
Rewrrd
Reward
Any Homard
Any flams Derse
Dense
0.0
0.
0.4
04
6.2
10
0.0
02
CA
06
63
10
Timesteps
Timesteps
Figure 10: Pretraining Reward for ColorSwitch for intrinsic motivation. Showing mean and variance bands on 25 random seeds.
Intrinsic Pushblock Pretraining
Intrinsic Pushblock Pretraining
10
Remaru
Reward
15-
Aey Iern Fomerd Lee Marrel Itere Howeed trd
Any fiem Homardd Derse Niveed koy newa Denve
02
0.4
0.
62
06
63
1.0
Timesteps
365
Timesteps
263
Figure 11: Pretraining Reward for Pushblock for intrinsic motivation. Showing mean and variance bands on 25 random seeds.
Intrinsic Crafting Pretraining
Intrinsic Crafting Pretraining
12
15.0
12.
Roward
Roward
10.0
nom KoWD's 216 Nemed Iee Rewaed End
6.-
any ittm #cword Deese Lanved bey Roward Dayta
Timesteps
3:5
Timesteps
305
Figure 12: Pretraining Reward for Crafting for intrinsic motivation. Showing mean and variance bands 25 random seeds.
In this section expand the experiments from section of the main paper.
In this experiment, we show results on our hypotheses verification problem using different forms of intrinsic motivation pre-training. We show results for 4 different pretraining schemes:
30

---

Under review as a conference paper at ICLR 2021
1. Change any item state in the world. Receive reward end.
2. Change any item referenced in the hypothesis. Receive reward at end.
3. Change any item state in the world. Receive reward instantaneously.
4. Change any item referenced in the hypothesis. Receive reward instantaneously.
Reward at the end means that it operates similar to hypothesis pre-training, Specifically, the agent get reward only at the end of the episode when it has taken a stop action. At that step it gets +C reward it changed within the last N frames. For these rewards, we choose what it sounds like. When the object state is changed, the reward is instantly received by the agent. We chose C' = 1 for colorswitch and pushblock and is not the agent. So this includes crafting items, switches, pushblocks, etc. We show results on 25 random seeds. We preserve all training and network hyper-parameters.
10. Instantaneous reward
for crafting. Item means any object that
We show the rest of the training curves for our intrinsic experiments in Figure 10 pretraining curves for our intrinsic rewards.
12 show the
All training and network parameters are kept the same from earlier experiments.
31


Reviews:
[{"review_id": "kP6j4Nk8qxN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper \"Empirically Verifying Hypotheses Using Reinforcement Learning \" proposes learning strategies of agents where goals are to validate or invalidate hypotheses about the world. This is done via RL, using the structure of triplet {pre-condition; action-sequence; post-condition} to drive the learning with intrinsic rewards. \n\nWhile the problem is interesting, I found the paper difficult to read as the task is ill-defined in the section 3 where many notation definitions are missing and some notations are reused in different contexts with different definitions (e.g.  o=sw,h in the first sentence of last paragraph of section 3 and o is an observation from the world in the beginning of section 4, a is {false,true} in section 3 but can also be a move in the world later). Also, the state includes the dynamics L_W ? I understand that there are two levels of actions, states and observations (moves in the world and final decision w.r.t. the hypothesis) but this presentation could be made much clearer for the reader. \n\nMore importantly, my main concern is about the experimental setting. In the setting formalized in section 3, I understood that we get hypotheses and ground truth about their validity in the sampled world. Ok, but if we got the ground truth, what the learning could be used for. The only setting that matters in that case is from my point of view a meta-rl setting where we train agents to decide from a training set of worlds (for which we know the ground truth of hypotheses), which we hope well generalize for other worlds with similar properties. However, I cannot see any discussion about this, neither in the formalization section nor after in the experiments. Reported experiments only give results at train time from what I understood, which is clearly not sufficient to demonstrate the benefits of the approach. How the strategies generalize, and in which conditions they do is of crucial matter here. \n\nIn the motivation from section 3, authors justify their approach by claiming that triplet hypotheses can be useful for interfacing symbolic solvers. It would also have been nice such kind of setting. Also, I do not understand the justification of natural language hypotheses (i.e., \"scaling annotations to untrained humans\").   It would have been nice to have experiments more related to these announced interests,  to highlight the usefulness of the approach.\n\nAt last, do we use text encoding for natural language hyoptheses described in the special cases setting ? Please detail.  \n\nGlobally, I feel that the paper needs a full rewritting to improve clarity. And that experiments must include generalization results.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "a2kn9FoUxsp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper presents the problem formulation for a hypothesis testing within a reinforcement learning paradigm. In particular, given a hypothesis represented as a formal, templated language, an agent needs to perform a sequence of actions and check whether the hypothesis holds true or false. \n\nThe paper proposes multiple methods since it is shown that traditional RL is insufficient for an agent to learn to test hypotheses. One of methods is to use pre-training taking advantage of more detailed information (represented as a triplet of pre-condition, action sequences, and post condition) about the hypothesis verification. Authors further investigated training after a pretraining step, and hypotheses that cannot be represented as a triplet.\n\nHypothesis verification using RL framework seems an important research direction. Based on the experiments presented in this paper, my understanding about the hypothesis testing is that it is about performing a complex task relevant to the hypothesis, and answering whether the hypothesis turns out to be true or false. Providing a triplet of information seems to be similar to the idea of breaking up such a complex task into comprehensible smaller tasks (like goal vs. subgoals) since the way pre-training works in this paper is precisely nudging the agent to perform subtasks. In this regards, I am not quite convinced the novelty of the formulation or methods. \n\nIn MineRL competition 2020, for example, the goal of the competition is to mine diamond. Agents receive rewards for obtaining items which are necessary to mine diamond. Hence, one may think that the agent needs to implicitly generate hypotheses and test the hypotheses to finally obtain diamond. This seems a harder task than hypothesis verification tasks this paper considered. How about \"Montezuma's Revenge\"? \n\nUnfortunately, I found this paper paper does not adequately provide neither novel methods, results, nor insights. I would like to reexamine the paper once the authors respond focusing on which parts are particularly novel.\n\nMinor points.\nFigure 2. \"the the\" for color switch\nPage 23, missing reference \"??\"\nOne switches (pg 5, color switch)\n\nQuestion\n\"RL agents struggle to solve the problem.\" Is this because of fundamental flaws in RL formulation (sparse reward, credit assignment problem, etc.) or just not having enough time steps (say, 1e9, 1e10 ...). Without having experiments on a simple hypotheses and smaller grid-world, it is difficult to examine the performance of standard RL agents.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "KYb4TOU3i3t", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n​\nThis paper introduces a problem setting where an RL agent must interact with its environment to predict whether a given hypothesis is true or false. On modified versions of environments like gridworld and cartpole, they show that PPO with a sparse reward is unable to correctly test the hypothesis. The key technical contribution is making the reward more dense by assuming a predefined structure in a subset of hypotheses. This denser reward is used to pretrain the RL policy, which is then finetuned over the rest of the hypotheses using sparse reward.\n​\n----------\n​\n# Strengths\n​\n- The introduced problem of testing various hypotheses about environments is a novel and interesting application of RL.\n​\n- The paper shows empirically that it is hard to solve this problem with only sparse rewards, even for simple environments. I especially appreciate the oracle ablations, which show that the main issues is of credit assignment - the joint problem of learning an exploring policy and hypothesis predictor together is hard, while the individual problems themselves were easily solvable.\n​\n- The paper is clearly written and nicely defines the hypothesis verification problem. It was easy to understand what the key problem, intuition, and contributions are.\n​\n----------\n​\n# Weaknesses\n​\n- The technical novelty of the paper is a way to engineer a denser reward based on priviledged information about the space of hypotheses. While the problem of testing hypothesis with environment interaction is interesting, one expects a more principled solution for this problem. Moreover, it is not convincing that such Triplet hypotheses would be available in practical scenarios where such hypothesis testing may be applied. If the authors want to claim exploiting triplet hypotheses as the key contribution, then they should add more realistic environments to justify their use.\n​\n- The method makes strong assumptions about what is available at training time, such as direct supervision for prediction network and the underlying (pre-condition, action, post-condition) structure of triplet hypotheses. It is fine to make such assumptions, but then the evaluation must be done for held-out environments as well as held-out hypotheses, where no reward information is available (because having a reward function means you already know the answer to the hypothesis). This would truly show that the trained agent can test new hypotheses in new environments (under the same distribution as training).\n​\n- The fact that intrinsic rewards can sometimes succeed at solving the task (Fig. 5 - crafting finetuning) suggests that perhaps it is not necessary to assume triplet hypotheses for solving these tasks. Potentially, more principled exploration approaches will automatically solve this problem (just like other sparse reward RL problems). Also, the reward curves in Figs 10 - 12 do not look normal. The training breaks down in the middle and then gets back up. Is it possible that the method is not properly trained and just requires more hyperparameter tuning and careful training? Or is there some other justification for such plots?\n​\n----------\n​\n# Reason for decision\n​\nWhile the introduced problem is interesting (building upon Denil et. al), my main concern for recommending weak reject is that the main contribution seems to be just problem-specific reward engineering. This can still be a valid contribution, but the assumptions would then need to be justified by evaluating in more realistic environments (like Denil et. al did with physics experiments). Even then, it is a philosophical debate whether reward engineering is the right way to approach the problem, or developing more principled exploration methods.\n​\n----------\n​\n# Suggestions for improvement\n​\n- Some examples of more challenging environments for hypothesis testing: continuous control robotic experiments (simulation) where objects need to be analyzed to test hypothesis; visual navigation tasks where a house needs to be explored to gather information.\n​\n- A lot of important information is missing from the main paper and is present in the Appendix. The main paper should include more information about the inputs (text), the structure of policy network (seq2vec), prediction network (transformer), and important training details (e.g. PPO being the RL algorithm).", "extra_scores": {}, "extra_text": {}}, {"review_id": "FBUuUOUWiiY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the general problem of testing hypotheses about the world by a kind of reinforcement learning, much as a person might learn by taking actions and observing their outcomes -- equivalently learning policies that can generate observations to validate a hypothesis.   A hypothesis is a symbolic representation of precondition, action and effects. The paper takes advantage of the ability of reinforcement learning to manipulate the environment that makes it possible to make causal inferences about the world. \n\nThat said, I find the paper's inadequate explanation and lack of rigor in the formal definition of the problem an impediment to understanding the work. As well as I can make out, it posits a distribution over \"worlds\" and \"hypotheses\" h, where worlds are specified by rules L^w, but also as triples of S_w, A_w & T_w, where T_w depends on L_w, (and conceivable also on A_w). One posits a truth functional G(h, W)  (Where does this come from?), by which a reward is generated if G equals a policy \\pi(O_t, h), O_t an observation sequence, that is also a truth functional whose action space is True or False.  In short, it looks like we are talking about an RL that makes a guess whether the current hypothesis applies in the current world state. \n\nHowever, at this point things break down.  A new truth functional f(O_t, h) (or is this the prediction network, mentioned on p5?) is introduced, possibly to replace \\pi(), and possibly driven by a \"policy network\" for purposes of \"pre-training\"? The exact formalization of a hypothesis, from it's textual expression is missing. So is the source of G(), which one might assume is an oracle by which policies are validated? \n\nFurthermore the stated need to \"predict the hypothesis\" is a phrase that doesn't ring true. Strictly predictions apply to outcomes (a hypothesis and pre-condition predict an outcome), but a hypothesis would be inferred from outcomes. \"Prediction\" in this sense may simply be a colloquialism, just adding to the confusion. \n\nRecourse to the literature discussion is to no avail. The references are vague or not clearly relevant.  Brachman & Levesque 2004 is an entire anthology so it's not clear what exactly is being referred to. The term \"action sentence\" that is referenced as Pearl 2009 does not even appear in the cited work. The citation to Mao 2019 is a weak analogy, and so on. \n\nIn summary, without clear definitions and careful adherence to notation, critical evaluation of the paper is encumbered.  Similarly, the domain being stochastic apparently, any discussion how probabilistic aspects are treated is lacking and needs to be addressed", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers unanimously agree that the paper should be rejected. The main concern is well summarized by comment by R1's comment "While the problem is interesting, I found the paper difficult to read as the task is ill-defined in section 3 where many notation definitions are missing and some notations are reused in different contexts with different definitions". Also, as R4 mentions the proposed method can be reduced to reward engineering and doesn't provide any scientific or methodological advancement to the problem of testing hypothesis. The authors did not provide any rebuttal. 
Decision: Reject