Empirically Verifying Hypotheses Using Reinforcement Learning

This paper formulates hypothesis verification as an RL problem . Specifically , we aim to build an agent that , given a hypothesis about the dynamics of the world , can take actions to generate observations which can help predict whether the hypothesis is true or false . Existing RL algorithms fail to solve this task , even for simple environments . We formally define this problem , and develop environments to test different algorithms ’ performance . We analyze methods which are given additional pre-training rewards and find that the most effective of these is one that exploits the underlying structure of many hypotheses , factorizing them as { pre-condition , action sequence , post-condition } triplets . By leveraging this structure we show that RL agents are able to succeed . Furthermore , subsequent fine-tuning of the policies allows the agent to correctly verify hypotheses not amenable to this factorization . 1 INTRODUCTION . Empirical research on early learning Gopnik ( 2012 ) ; Kushnir & Gopnik ( 2005 ) shows that infants build an understanding of the world by constantly formulating hypotheses about how some physical aspect of the world might work and then proving or disproving them through deliberate play . Through this process the child builds up a consistent causal understanding of the world . This contrasts with manner in which current ML systems operate . Both traditional i.i.d and interactive learning settings use a single user-specified objective function that codifies a high-level task , and the optimization routine finds the set of parameters ( weights ) which maximizes performance on the task . The learned representation ( knowledge of how the world works ) is embedded in the weights of the model - which makes it harder to inspect , hypothesize or even enforce domain constraints that might exist . On the other hand , hypothesis generation and testing is a process explored in classical approaches to AI Brachman & Levesque ( 2004 ) . In this paper we take a modest step towards the classical AI problem of building agents capable of testing hypotheses about its world using modern ML approaches . The problem we address is illustrated in Figure 1 . Agents are placed in a world which has several interactive elements . They are provided with a hypothesis ( an `` action sentence '' Pearl ( 2009 ) ) about the underlying mechanics of the world via a text string ( e.g . `` A will be true if we do B '' ) . The task is to determine if the hypothesis is true or not . This problem can not be solved without interaction with a dynamic world ( comparing the state before and after taking action B ) . A key novelty in our work is formulating the task in a manner that permits the application of modern RL methods , allowing raw state observations to be used rather than abstract Boolean expressions of events . To do this , we use a model composed of two different deep parametric functions which are learned through interaction : ( i ) a policy that generates observations relevant to verification of the hypothesis and ( ii ) a prediction function which uses the observations to predict whether it is true . We first show that agents trained end-to-end using deep RL can not learn policies that can generate observations to verify the hypothesis . To remedy this , we exploit the underlying structure of hypotheses – they can often be formulated as a triplet of a pre-condition ( P ) , an action sequence ( collectively B ) , and a post-condition ( A ) that is causally related to the pre-condition and actions . Using this structure , we can seed our action policy to learn behaviors which alter the truth of the pre-condition and post-condition . This allows agents to learn policies that can generate meaningful observations for training the prediction function . We further demonstrate that these policies can be adapted to learn to verify more general hypotheses that do not necessarily fit into the triplet structure . Our experiments Hypothesis : “ when you are at craftingtable and you have stick and then you craft then torch is made ” show that this approach outperforms naive RL and several flavors of intrinsic motivation designed to encourage the agent to interact with the objects of interest . 2 RELATED WORK . Knowledge representation and reasoning ( KRR ) Brachman & Levesque ( 2004 ) is a central theme of traditional AI . Commonsense reasoning Davis ( 1990 ) ; Davis & Marcus ( 2015 ) ; Liu & Singh ( 2004 ) approaches , e.g . CYC Lenat ( 1995 ) , codify everyday knowledge into a schema that permits inference and question answering . However , the underlying operations are logic-based and occur purely within the structured representation , having no mechanism for interaction with an external world . Expert systems Giarratano & Riley ( 1998 ) instead focus on narrow domains of knowledge , but are similarly self-contained . Logic-based planning methods Fikes & Nilsson ( 1971 ) ; Colaco & Sridharan ( 2015 ) generate abstract plans that could be regarded as action sequences for an agent . By contrast , our approach is statistical in nature , relying on Reinforcement Learning ( RL ) to guide the agent . Our approach builds on the recent interest Mao et al . ( 2019 ) ; Garcez et al . ( 2012 ) in neural-symbolic approaches that combine neural networks with symbolic representations . In particular , some recent works Zhang & Stone ( 2015 ) ; Lu et al . ( 2018 ) have attempted to combine RL with KRR , for tasks such as navigation and dialogue . These take the world dynamics learned by RL and make them usable in declarative form within the knowledge base , which is then used to improve the underlying RL policy . In contrast , in our approach , the role of RL is to verify a formal statement about the world . Our work also shares some similarity with Konidaris et al . ( 2018 ) , where ML methods are used to learn mappings from world states to representations a planner can use . Causality and RL : There are now extensive and sophisticated formalizations of ( statistical ) causality Pearl ( 2009 ) . These provide a framework for an agent to draw conclusions about its world , and verify hypothesis as in this work . This is the approach taken in Dasgupta et al . ( 2019 ) , where RL is used to train an agent that operates directly on a causal Bayesian network ( CBN ) in order to predict the results of interventions on the values on its nodes . In contrast , the approach in this work is to sidestep this formalization with the hope of training agents who test hypotheses without building explicit CBNs . Unlike Dasgupta et al . ( 2019 ) , our agents intervene on the actual world ( where interventions may take many actions ) , rather than the abstract CBN . Nevertheless , we find that it is necessary to add inductive bias to the training of the agent ; here we use the pretraining on ( P , B , A ) triplets . These approaches are complementary ; one could combine explicit generation and analysis of CBNs as an abstract representation of an environment with our training protocols . Our work is thus most similar to Denil et al . ( 2016 ) , which uses RL directly on the world , and the agent gets reward for answering questions that require experimentation . However , in that work ( and in Dasgupta et al . ( 2019 ) ) , the “ question ” in each world is the same ; and thus while learning to interact led to higher answer accuracy , random experimental policies could still find correct answers . On the other hand , in this work , the space of questions possible for any given world is combinatorial , and random experimentation ( and indeed vanilla RL ) is insufficient to answer questions . Cognitive development : Empirical research on early learning Gopnik ( 2012 ) ; Kushnir & Gopnik ( 2005 ) shows infants build an understanding of the world in ways that parallel the scientific process : constantly formulating hypotheses about how some physical aspect of the world might work and then proving or disproving them through deliberate play . Through this process the child builds up an abstract consistent causal understanding of the world . Violations of this understanding elicit measurable surprise Spelke et al . ( 1992 ) . Automated Knowledge Base completion : This work is also related to knowledge base completion Fader et al . ( 2011 ) ; Bordes et al . ( 2013 ) ; Suchanek et al . ( 2007 ) , especially as formulated in Riedel et al . ( 2013 ) . Instead of using facts in the knowledge base or a text corpus to predict edges , here the agent acts in a world and observes the results of its actions . This recalls Mitchell et al . ( 2018 ) , where the system verifies facts it had hypothesized by searching for corroboration in the corpus . Automation of the scientific process : has been tried in several domains . Robotic exploration of chemical reactivity was demonstrated Granda et al . ( 2018 ) using ML techniques . King et al . ( 2009 ) developed a robot scientist that explored geonomics hypotheses about yeast and experimentally tested them using laboratory automation . In biochemistry Vanlier et al . ( 2014 ) used Bayesian methods for optimal experiment design . More generally , the Automated Statistician project Steinruecken et al . ( 2019 ) uses a Bayesian approach to reason about different hypotheses for explaining the data , with the aim of creating interpretable knowledge . Embodied Question and Answering : The problem studied in this paper is closely related to the embodied visual question-answering problem in Das et al . ( 2018 ) . Indeed , our basic formulation is a particular case of the most general formulation of embodied QA , as the agent is rewarded for successfully answering questions about the world that require interaction . However , the form of the questions is different than those considered in that work , as they may require drawing a conclusion about the dynamics of the world , rather than a static property . Even the questions about static properties we are interested in have a different flavor , as they encode rules , rather than statements about the current configuration . Our approach is built around hypothesis-conclusion structure special to these questions . There is also a large body of work on visual QA Kafle & Kanan ( 2017 ) ; Wu et al . ( 2016a ) and text-based QA Rajpurkar et al . ( 2018 ) . From this , most relevant to our work is Wu et al . ( 2016b ) who use a structured knowledge base to augment standard QA techniques . 3 THE HYPOTHESIS VERIFICATION PROBLEM . An agent is spawned in a world sampled from a distribution over possible worlds . In the case of “ Crafting ” , shown in Figure 1 , there are items lying around that the agent can pick up and combine using a “ craft ” action . The exact dynamics change for every newly instantiated world ; so in one world , taking a craft action with a stick might produce a torch , and in another , it might produce a pickaxe . At the start of each episode , the agent is given a hypothesis about the world , such as the one shown at the top of Figure 1 . The agent gets a reward when it correctly answers if that hypothesis is true or false . Because the dynamics and rules change each episode , the agent must learn to interact with the world in order to decide if the hypothesis is true . In Figure 1 the agent picks up the stick and does a craft action to see that a torch is created . It then has enough information to decide the hypothesis is true , and the agent receives reward for verifying the hypothesis correctly . In this work , we will structure our hypotheses using templated language . One could imagine using more expansive formal symbolic systems ( e.g . first order logic ) , or alternatively , using natural language descriptions of the hypotheses . The former might allow interfacing with symbolic solvers or otherwise using combinatorial approaches ; whereas the latter would allow scaling annotation to untrained humans . We choose templated language because it is simple , and sufficient for the environments on which we test , which are already challenging for standard RL . Moreover , in our view it is a good starting point for further work that would use either more sophisticated formal representations or more natural language representations . Formal Definition We define a world as a set of states and actions with Markovian dynamics ( an MDP without reward ) . We define an environment E as a distribution over a set of worldsW and hypothesesH . A world W ∈ W is specified by rules LW describing the dynamics of the world . We define this reward-less MDP of one specific world W as MDPW = { SW , AW , TW } where state space SW includes the position and state of objects in the world ( e.g . the placement of the agents and the object ) , AW is the action space , and TW is the transition function . Note that TW depends on LW , the rules of this specific world . Actions have different consequences depending on LW . Now E is an episodic POMDP where each episode consists of sampling1 a W and h. ( G is a groundtruth function that takes in the hypothesis h and worldW and outputs { true , false } . In this work , hypotheses are generated via templated language and their truth function G depends on W , more specifically LW . The episode ends when the agent executes either the true or false action . 1See Appendix B for details on sampling procedures Given a world W and hypothesis h , an agent gets reward : RHyp = { +1 a = G ( h , W ) −1 a = ¬G ( h , W ) 0 otherwise The observation in this POMDP is o = ( sW , h ) , the state from the world W plus the hypothesis . The state is s = ( sW , h , LW ) . This includes the rule LW which is not visible in the observation . The action space is just AW ∪ { true , false } for any W ( they are the same for a given environment ) ; and T = TW . Note that the transition function T depends on the ( hidden ) LW . The goal of hypothesis verification is now to discover the truth of h , which depends on LW .

Summary: This paper introduces a problem setting where an RL agent must interact with its environment to predict whether a given hypothesis is true or false. On modified versions of environments like gridworld and cartpole, they show that PPO with a sparse reward is unable to correctly test the hypothesis. The key technical contribution is making the reward more dense by assuming a predefined structure in a subset of hypotheses. This denser reward is used to pretrain the RL policy, which is then finetuned over the rest of the hypotheses using sparse reward.