Towards Understanding How Machines Can Learn Causal Overhypotheses 
Authors: Eliza Kosoy
David Chan
Adrian Liu
Jasmine Collins
Bryanna Kaufmann
Sandy Huang
Jessica B Hamrick
John Canny
Nan Rosemary Ke
Alison Gopnik
Venue: Submitted to ICLR 2023 (2023)
Keywords: causal reasoning
intervention
causal overhypotheses
Reinforcement learning
gpt-3

Abstract: Recent work in machine learning and cognitive science has suggested that understanding causal information is essential to the development of intelligence. One of the key challenges for current machine learning algorithms is modeling and understanding causal overhypotheses: transferable abstract hypotheses about sets of causal relationships. In contrast, even young children spontaneously learn causal overhypotheses, and use these to guide their exploration or to generalize to new situations. This has been demonstrated in a variety of cognitive science experiments using the “blicket detector” environment. We present a causal learning benchmark adapting the “blicket" environment for machine learning agents and evaluate a range of state-of-the-art methods in this environment. We find that although most agents have no problem learning causal structures seen during training, they are unable to learn causal overhypotheses from these experiences, and thus cannot generalize to new settings. 

TOWARDS UNDERSTANDING How MACHINES CAN LEARN CAUSAL OVERHYPOTHESES
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work in machine learning and cognitive science has suggested that under- standing causal information essential to the development of intelligence. One of the key challenges for current machine learning algorithms is modeling and understanding causal overhypotheses: transferable abstract hypotheses about sets of causal relationships. In contrast, even young children spontaneously learn causal overhypotheses, and use these to guide their exploration or to generalize to new situations. This has been demonstrated in variety of cognitive science experiments using the "blicket detector" environment. We present a causal learning benchmark adapting the "blicket" environment for machine learning agents and evaluate range of state-of-the-art methods in this environment. We find that although most agents have no problem learning causal structures seen during training, they are unable to learn causal overhypotheses from these experiences, and thus cannot generalize new settings.
INTRODUCTION
Over the past few years, deep learning has made impressive progress in many areas, including reinforcement learning, natural language processing, and computer vision. However, most state- of-the-art algorithms still pale in comparison to humans, even children, for out-of-distribution generalization and fast adaptation to new tasks. In contrast, causal modeling has long concerned itself not merely with accurate modeling of in-iitribbttinn data, but also the accurate recovery of underlying causal mechanisms (and their true graphical relations) capable of explaining oul-of-distributtin data. Hence causal modeling holds promise for achieving systematic generalization (Bengio al. 2019
Schölkopf et al. 202T, Ke et al. 202T).
One of the key components of causal learning for humans are causal overhypotheses, which describe priors over causal graphs. For example, a causal hypothesis may be that system is stochastic, that it is conjuntive- -ie.,, more than one cause must be present in order for the effect to occur. Causal overhypotheses are important because they enable humans to learn causal models from sparse amount of data (Griffiths & Tenenbaum 2009), by reducing the set of possible causal relationships to consider. Despite the recent surge machine learning datasets and environment for causal inference and learning (Ahmed et al., 2020 McDuff al. 2021 Wang et al., 2021a Ke al. 2021), the causal overhypotheses for these environments and datasets are unclear. Thus we cannot use these existing benchmarks to evaluate how capable are agents are at learning and using causal overhypotheses.
In this work, we seek to fill this gap, by focusing on introducing benchmark environment that draws inspiration from recent cognitive science work using blicket detectors. This environment contains virtual blicket detector, and has been used previously for understanding causal learning and exploration in children (Kosoy et al. 2022) and basic RL models, but not yet for statc-of-the-att machine learning agents.
A "blicket detector" is machine that lights up and plays music when some combinations of objects but not others are placed on it (Gopnik Sobel 2000 Lucas et al. 2014). The central question is whether agent can learn that a particular set of causal events will lead to the lighting-up effect, and use that knowledge to design novel interventions on the machine. The causal relationship is entirely determined by the pattern of conditional dependencies and interventions, rather than requiring intuitive physics knowledge or visual understanding. Although these tasks may seem simple, and are easily mastered by children, find that they challenging for current learning algorithms.

---

Several features of this environment and the tasks it allows make it particularly useful as a benchmark for machine learning algorithms. First, causal representations are more powerful and structured than mere statistical generalizations, though both can be systematically inferred from statistical information. Many researchers (e.g. Pearl, Spirtes et al., Bengio) have argued that such causal representations are crucial for both human and general AT intelligence. The goals of causal inference, namely learning which actions will alter the environment in particular ways, are similar to those of standard reinforcement learning, but adding causal representations and inferences makes such learning far more effective.
Second, and unlike some existing causal environments (Ke et al., 2021] Wang et al. 2021 a), the
blicket environment enables the inference of both specific causal structure and more general features of causal structure, such as whether causal systems are conjunctive or disjunctive, stochastic or deterministie. Learning these overhypotheses about causal structure (Griffiths & Tenenbaum 2009) is especially important because such inferences can constrain the search for causal structure in the future, search that can rapidly become unwieldy. Third, this environment allows for active learning and exploration in way that is both sufficiently constrained to be practical and that also allows for informative interventions.
Most significantly, and again unlike existing environments, research has already shown that even preschool children can easily manipulate and explore this environment, generate appropriate data, and rapidly learn both particular causal structure and overhypotheses about causal structure (Kosoy et al. 2022). We can then directly compare both the overall performance and the behavior of various agents in these tasks to the performance and behavior of children. Young children are a particularly informative baseline group. They do not have the extensive education and experience of typical adults, which might make comparisons to artificial agents challenging, but they are nevertheless effective causal learners and able to make broad yet accurate generalizations from small sample sizes, in contrast to many current machine learning systems (Gopnik) 2012 Gopnik et al., 2017).
The work of Kosoy et al. 2022) first described a version of this environment and included data on children and a few learning algorithms. But the environment and benchmarks have much wider applications than in that preliminary study or the studies reported here. It allows multiple comparisons of children and agents on multiple tasks and allows direct comparison with ML systems for multiple kinds of causal inference tasks. In Kosoy et al. 2022) and in the experiments described here we focus such inference- to conjunctive versus disjunctive structure, and few algorithms. But with very minor changes the environment would allow tests of inferences to many other kinds of overhypotheses and inductive biases, such inferring whether systems are stochastic or deterministic or require sequential or unordered interventions, overhypotheses that are also important for causal inference. It also allows multiple measures of causal inference including interventions and counter- factuals as well as predictions. And, significantly, it would allow researchers to empirically record the exploration behavior that children use in solving causal problems, not yet done this in this paper, and compare it to wide variety of increasingly influential exploration based ML procedures.
In what follows, we look at spectrum of algorithms ranging from reinforcement learning, to imitation learning, the use of pre-trained language models, to better understand how these different methods perform on the proposed environment and tasks. We find that such algorithms, in contrast to children, only converge on solution after an extensive number of trials or if they are given all the possible settings and outcomes as training data. This suggests that these tasks are an interesting challenge for machine learning algorithms. In order for machines to perform as well as children do, algorithms must reason about the sequence of observations seen, extract causal overhypotheses from those observations and use them for exploration-vv which current methods fall short of doing.
RELATED WORK
Exploration in Reinforcement Learning Recent exploration algorithms for deep reinforcement
learning typically add an exploration bonus to the task reward; please refer to Amin et al. (02T)) for
a comprehensive survey. This bonus could be based on novelty (Bellemare et al. 2016 Ostrovski
et al. 2017 Martin et al. 2017 Tang et al. 2017 Machado et al.) 201 Sah. dynamics prediction error Schmidhuber] 1991 Pathak et al. 2017), D. uncertainty (Osband et al. 2016 Burda et al. 2018).
or disagreement (Pathak et al. 2019). The vast majority of existing exploration methods do not
involve causal reasoning, whereas our proposed environment requires algorithms to learn and use causal overhypotheses in order to explore effectively, to solve the task.
2

---

Multi-task and Causal RL Benchmarks There exist multi-task RL benchmarks featuring robotics Yu et al. 2019; James et al., 2020). physical reasoning (Bkkhtnn et al.
2019; Allen et al.) 2020).
and video games (Cobbe et al. 2018: Machado et al. 2018b Nichol et al.
2018 Chevaler-Boisvent
et al. 2018) 8). Unfortunately, since is not clear what the relevant causal overhypotheses for these environments are, it is difficult to evaluate how causal information influences agents' exploration.
RL benchmarks for causal discovery include Causal World (Ahmed et al., (020), Causal City (MDDuff ct al. 202T), Alchemy (Wang et al. 2021a), ACRE Zhang et al. (021). and the work of do not allow for controlling overhypotheses. In addition, these environments primarily focus on causal induction or generalization, rather than exploration (though see SSntakke et al. (02T). In contrast, the blicket environment in this work is designed to measure agents' ability to explore using causal overhypotheses. Moreover, children have not been tested on any of these existing environments, whereas in the blicket environment, prior work has shown that children as young as age four are able to learn causal overhypotheses and use these to explore effectively (Kosoy al., 2022). It can be informative to compare the exploration and performance of RI. approaches to that of children.
Ke al. (2021). However, many of these environments either lack clear causal hypotheses
Language Models for Reasoning Tasks Large language models such as GPT (Radford et al. 2019
Brown et al., 2020) and PALM (Chowdhery et al. 2022) are trained on massive amounts of data, and they have been shown to be able to express uncertainty and perform common sense reasoning to an extent (Lin et al. 2022). In this work, we probe the the causal reasoning capabilities of GPT-3 and PALM using textual descriptions of the virtual blicket environment.
THE "BLICKET DETECTOR" ENVIRONMENT
The blicket detector is commonly used in cognitive science to study causal inference and reasoning in children (Gopnik & So- bel 2000: Lucas et al. 2014). Typically these studies are ducted in person. Recently Kosoy et al. (2022) proposed an open- sourced virtual blicket detector environment Figure focused on studying how children learn and use causal overhypotheses in exploration. This virtual environment has several advantages: it enables not only tracking interactions with children, but also an exact comparison between children and machine learning agents via the OpenAI Gym interface (Brockman et al. 2016a). Please refer to Appendix A for screenshots of the virtual blicket detector in various states.
Figure 1: A sereenshot of the embod-
ied version of the blicket detector.
Causal Overhypotheses Causal overhypotheses are hierarchical priors over the structure and/or conditionals of causal model, and have been widely studied in the cognitive science literature (Kempl
Lucas & Grifliths| 2010 Perfors et al., 2011: Tenenbaum et al. 2011: Gopnik & Wellman 2012). An overhypothesis might state that the causal graph itself has particular form (such as a "common effect" or a chain structure), or that the conditionals within that graph have particular form (such as that p(X \Y, Z) follows particular parametric distribution). Having a good causal overhypothesis is a form of inductive bias that can make causal inferences much easier: for example, while we might not know the specific causal graph, if we know that it takes the form common effect, then we need only a few interventions (possibly only O(N)) to fully determine the specific causal graph. Idcally, then, machine learning agents should be able to learn such overhypotheses and leverage them make more efficient and accurate causal inferences.
let al. 2007 2010:
In our environment, we consider two causal overhypotheses: that the world is either conjunctive or disjunctive, with the causal system defined as follows. In both cases, the causal graph takes the form of common effect, with N + variables: N objects X {Xo, X1, ..., XN } (the causes) and one blicket machine M (the effect). Each object X, can either be on top of the blicket machine (.;; = 1) off the machine (X, 0). The blicket machine can be either (M 1) off (M 0). Because the graph is a common effect, objects' states do not influence each other (intervening on X; does not impact X,). Objects' states may, however, influence the state of the machine. Specifically, some subset of the objects, B C X, are said to be "blickets" in that they have a causal influence on whether the blicket machine turns on. Thus, the causal graph in this scenario always take the form of a common effect, with edges {X; + M X, BJ.
The conjunctive and disjunctive overhypotheses specify the form of the blickets' causal influence on the machine. For example, let X,, X, B be blickets. In the conjunctive case, both objects
3

---

Disjunctive
Conjunctive
Lights up
Doesn't
Lights up
light up
light up
Lights up
disjunctive setting (left), only one blicket is needed for the blicket to light up. Whereas in the conjuctive setting (left), two blickets are needed. These examples are shown to children in Kosoy et al. (2022) for the given hypotheses condition.
Figure A simplified rendering of virtual blicket detector environment.
needed at the same time to turn the machine on. Formally, P(M = 1|x; = 1,,, =) 1) and P(M = 0 X; = = 1. the disjunctive case, only one object (either X; Xj) is needed to turn the machine on, so P(M = 1|X; = 1) and P An illustration of the disjunctive versus conjunctive overhypotheses can be found in Figure 2 We note that while this setup is deterministic, it can easily be adapted stochastic environment with minimal modifications.
1, while
P(M = 0
Prior Experiments with Children Since we would like to compare the performance of machine learning agents to that of children in this environment, we aim to make our evaluation of machine learning algorithms comparable to how children are evaluated in Kosoy et al. (022)). Their evaluation used a 2 x 2 experimental design, with a given hypotheses or not given hypotheses training phase, combined with a disjunctive or conjunctive causal relationship in the test phase. In the training phase, they showed the child two different blicket detectors, with three unique objects each, and for each detector demonstrated three examples of placing objects on the detector and checking whether lights up. In the given hypotheses case, one could infer from the examples that detector was conjunctive and the other was disjunctive, as shown in Figure 2 Whereas in the not given hypotheses case, one could not infer from the examples whether the detectors were conjuctive or disjunctive. In the test phase, the children were given the opportunity to freely interact with a third blicket detector with three new objects. When they were done exploring, they were asked to respond yes or no to which objects were blickets. In their experiments with children from ages four to six, Kosoy et al. (2022) found that children exhibit diverse range of exploration strategies, which suggests that they are exploring based on a rich set of causal overhypotheses, formed from their prior knowledge of how objects and mechanisms behave.
Experiments with SOTA RL Models To adapt the virtual blicket environment for agent learning in this work, we made the following design choices regarding the observations, actions, reward, and termination conditions.
Observations: We could allow the algorithms to observe the embodied visual space as the children, but this places RL algorithms at a significant disadvantage, since they would need to not only understand causal structures, but also learn visual inputs and object detection. Thus, we choose to evaluate the algorithms in purely symbolic environment where the objects are represented by one-hot indices. Formally, the state space is a vector in o E {0, 1) N+1 where is the number of blickets. The index o;, 0 < < N is 1 if is on the detector, and otherwise. The index oN is 1 if the detector illuminated, and 0 otherwise.
Actions: In the experiments with children, actions consist of placing a blicket onto the detector, removing a blicket from the detector, and pressing the "check" button to evaluate the detector's 's state. For RI. algorithms, we simplify this process by allowing the agent to place multiple objects simultaneously. automatically "checking" the state of the detector with every action, and automatically resetting the detector after each check, leading to 2 actions for N blickets. This means that the agent gets feedback with every action, which significantly improved training stability.
Reward: The reward function should capture whether the algorithm has learned the causal over- hypothesis of an environment. To this, we evaluate the models using a quiz-based framework. Models are allowed to make as many exploration steps needed, and then trigger an action which switches to the evaluation mode. the evaluation mode, models receive as input a blicket and must produce action indicating if the object is a blicket not. They receive reward of for identifying a correct blicket, and a reward of -1 for incorrectly labeling an object (i.e., for both false positives and false negatives). We also explored several reward modifications. In modification. models were asked to disambiguate between Disjunctive and Conjunctive environments, with +1 reward for identifying the correct modality. The environment is implemented using the standard OpenAl Gym
4

---

(Brockman al. 2016b) interface, allowing it to be used across many different pre-existing machine learning architectures and algorithms.
4 EVALUATING CAUSAL LEARNING IN THE BLICKET ENVIRONMENT
Results from Kosoy et al. 2020) suggest that children can explore efficiently, especially given the causal overhypotheses. In this work, we evaluate how a spectrum of different machine learning models perform the blicket detector tasks. Solving these tasks requires reasoning about the sequence of observations seen, extracting causal overhypotheses from those observations, and using these extracted overhypotheses for exploration.
We first evaluate several popular reinforcement learning algorithms A2C MMihh et al. 2016). PPO2 (Schulman et al. 2017), and Q-learning (Watkins & Dayan 1992]- on this task. We also evaluate imitation learning algorithms, including behaviour cloning with decision transformers (Chen let al. 2021). Finally, we apply pre-trained language models (Brown et al. [2200] Chowdhery et al. 2022) to this task, since they have been shown to be capable of performing common-sens, reasoning and expressing uncertainty to an extent (Lin et al. 2022).
4.1 DEEP REINFORCEMENT LEARNING ALGORITHMS
We evaluate the performance two popular deep reinforcement learning algorithms, Advantage Actor Criric (A2C) (Mnih et al. 2016) and Proximal Policy Optimization (PPO2) (Schulman et al.) 2017). on the blicket environment. For each algorithm, we use several policy variants: a standard MLP policy (no memory), an LSSTM-based policy, and LSTM-based policy with layer normalization with two hidden dimension variants of 256 and 512. For all of these policies, we found that a network with a hidden size of 512 obtained the optimal performance See Appendix B for learning hyperparameters. We train all of these algorithms in the given hypothesis scenario, where the agent is exposed during training all possible overhypotheses, and asked to perform well given these scenarios.
Experimental Design We terminate training either after 3 million environment steps when the agent obtains maximum reward for 500 consecutive episodes, whichever comes first. Each episode most 120,000 episodes. Experiments
has 25 timesteps, each agent is exposed single Nvidia Titan-X X Maxwell GPU, and take less than three hours to complete. To evaluate whether the agent can generalize additional causal situations, we also train agents on five held-out scenarios: holding out all of the conjunctive overhypotheses, holding out all of the disjunctive overhypotheses, and holding out either one conjunctive overhypothesis or one disjunctive overhypothesis. The maximum achievable reward for these experiments is 3.
using
Results Figure 3a shows the performance of A2C and PPO2 when none of the hypotheses are held out. PPO2 outperforms A2C in almost all scenarios, achieving higher rewards faster. Further, the LSTM models clearly outperform the non memory based models; this is expected, since causal learning requires memory. Unfortunately. none of the algorithms perform well on held-out causal examples, as shown in Figure 3c This suggests that they are primarily learning to memorize the causal patterns, and thus are incapable of generalization. This conclusion comes with a caveat: because there are only handful of possible hypotheses, it may be possible that we do not have enough data to perform well on held-out samples. Agents are, however, able to easily distinguish between conjunctive and disjunctive environments Figure 3b shows that after very few steps, in the held-out situation (with both overhypotheses), agents can distinguish conjunctive from disjunctive environments, even though they unable to determine which objects exactly are blickets.
4.2 Q-LEARNING
We also train tabular Q-Learning on the symbolic version of the blicket environment. We append the full history of previous observations to the current observation in order to give the agent memory. The Q-values are initialized to zero. We used e-greedy exploration with an exploration probability of 0.1, and we found the best learning rate was 0.95. Q-learning is able to learn the task very quickly due to small search space- it took an average of 70 episodes and 292 steps for the agent to converge to maximum reward. However, tabular Q-learning is incapable of generalizing to new scenarios, so we do not test L-Learning agents in the held-out scenarios.
5

---

PRAMALOTH S12
ARCESTH S13
PHAMLL POLLTH
ARCESTT
ARCUSTHIS52
ARCESESTH
50-
25
20
30
25083
sedts
reéda 230000 125000 :50000 275000 200000
(a) Performance of PPO2 and A2C algorithms with MLP, LSTM, Layer-Norm LSTM policies on the blicket environment.
(b) Accuracy of models time when determining if the environment is conjunctive or disjunctive.
Compnttivv
Dekirttiee Onl
Sotth
1.5
20
20
(c) Performance several over-hypothesis variants. Conjunctive Only: Train on conjunctive hypotheses, test on conjunctive and disjumctive. Disjunctive Only: Train on only disjunctive hypotheses, test on conjunc- tive and disjunetive. Leave one modifier: Train 2/3 (or 4/6 for both) of the hypothesis within the overhypothesis, lest on remaining.
(d) Performance of standard A2C-LSTM model when forced to explore the quiz phase. While this exploration helps the de- cision transformer, additional forced explanation is harmful to all standard RL models.
steps before entering
Figure Experiments with standard deep reinforcement learning algorithms
Model
Pre-Training Dataset
Reward
FCA
Decision Transformer
Random
0.16 +:: 1.747
0.67
Decision Transformer
Random (5-step forced exploration)
2.06 = 0.998
0.71
Decision Transformer
Random (10-step forced exploration)
2.16 = 0.872
0.73
Decision Transformer
Random (15-step forced exploration)
2.37 = 0.633
0.77
Decision Transformer
A2C-LSTM
1: 1.847
0.63
Decision Transformer
PPO2-ISTM
0.26 + 0.990
0.66
Decision Transformer
AZC-LayerNomm-LSTM
0.18 = 0.983
0.62
Decision Transformer
PPO2-LayeNNom-LSTM
0.74 = 1.460
0.64
Behavior Cloning
Random
0.22 = 0.975
0.61
Behavior Cloning
Random (5-step forced exploration)
-0.16 + 0.987
0.58
Behavior Cloning
Random (10-step forced exploration)
0.14 + 0.990
0.60
Behavior Cloning
Random (15-step forced exploration)
0.04 = 0.999
0.61
Behavior Cloning
A2C-L.TTM
0.10 + 0.995
0.55
Behavior Cloning
PPO2-LSTM
0.06 = 0.998
0.63
Behavior Cloning
AZC-LayerNom-LSTM
0.04 :: 2.087
0.59
Behavior Cloning
PPO2-LLyerNomm-ISTM
0.12 + 0.995
0.68
Table 1: Performance of the imitation learning models with different pre-training datasets. Reward is the
blicket-quiz task: correctly identifying blicket, for incorrectly identifying blicket. FCA refers Forced Choice Accuracy, accuracy of the model in determining if the environment conjunctive or disjunctive.

---

4.3
BEHAVIOR CLONING
Recently Chen et al. 2021 introduced the Decision Transformer, simple transformer-based approach to imitation learning, shown to outperform most existing behavior cloning methods. The decision transformer works by applying a flattened sequence of (state, action, reward, next state) tuples, with an f--norm loss. The model then chooses actions during test time that maximize the predicted reward-to-go.
causally-maskere transformer to predict the reward-to-go of
Experimental Design We collect trajectories for behavior cloning by exploring randomly in the space (an approach followed by Chen et al. (202T). In this work, we adapt the model to predict discrete actions in our space by adding sigmoid activation to the action predictions and altering the action prediction loss accordingly. We evaluate the model using a target reward of 3, which corresponds to identifying all of the blickets correctly. The decision transformer was trained using a batch size of 128, K-value of 30, embedding dimension of 128, 3 layers, one head, and dropout of 1. The weights are optimized using ADAM, with a learning rate of 14 and weight decay of le=4. We also compare against standard behavior cloning, using an MLP with hidden-dimensee of 128, and trained using the same optimizer as the decision transformer. Both models are trained for 10 epochs using one. Nvidia Titan-X X Max walwell GPU, which takes less than one hour, and the best validation set checkpoint is used for each during test time. Models are evaluated on 100 rollouts in the environment, and the mean reward is reported.
When training the decision transformer model, we found that in some cases, random (and even the PPO2/A2C trained) models were unable to explore efficiently, as they entered the quiz environment too soon. Note that random exploration will enter the quiz environment after t steps with with probability/ = 1 2 Thus, we found it helpful to force the policy to explore for several steps before allowing it to enter the quiz environment. While this process helps the decision transformer, it negatively affects the performance of the standard RL models, as shown in Figure 3d
Results Table I shows the performance of the decision transformer and behavior cloning models see, while expert policies allow for
when applied to datasets generated by several policies. As higher rewards using the decision transformer, allowing the model additional forced exploration time the most important factor. This suggests that on their own, the A2C and PPO2 trained policies do not lead to sufficient exploration for learning a strong model of the reward, whereas forcing additional exploration (even if it is random) is much more useful. Notably, standard behavior cloning performs very poorly, as copying actions with the same local observations under different overhypo will likely lead to incorrect or uninformative actions.
4.4 CAUSAL DISCOVERY BASELINES
Most deep learning models do not explicitly model the causal structure of the data, for example, would be quite challenging extract the causal structure learned by transformer network. However, one workaround is to evaluate the model on causal structure learning by asking it to answer questions about the existence of edge in a causal graph. For example, we could ask language model, does variable A cause variable B? Then one can take the average accuracy of these answers as the ratio of the structured hamming distance (SHD). However, note that in our case, since objects do not influence each other, we only ask the question of which objects influence the machine.
To explore the performance of existing causal discovery models on the benchmark, we leveraged the recent Differentiable Causal Discovery from Interventional Data (DCDI) approach (Brouillard et al.) 2020). We evaluated the model on causal structure learning for the disjunctive and the conjunctive settings. For the disjunctive settings, the model achieved a structured hamming distance (SHD) of 2.3. Noting that the graph has 4 variables and hence 16 possible edges in the graph structure, the model achieved SHD ratio of 85%. For the conjunctive setting, the model achieved SHD of 2.6. which is SHD ratio of 83%.
4.5 LARGE LANGUAGE MODELS
Recently large language models have shown promising performance on wide variety of tasks, includ- ing logical inference and common- sense reasoning. Such language models include GPT-3 (Brown) et al. 2020), PaLM (Chowdhery et al. 2022), MT-NLG (Smith et al. (022), and Chinchilla (Hoff mann et al. (022), which all use transformer based architectures and are trained autoregressively on
large corpus of text, to predict the next token when given sequence of tokens. These language
7

---

models have also shown promising performance tasks that involve causal reasoning, such as identifying causal effects from observational data (Veitch et all 2020) and inferring causality for when drugs lead to adverse effects (Wang et al. 2021b). However, these models have not yet been applied to a task that involves reasoning across a set of overhypotheses, as is required in the blicket environment.
Motivated by this, here we investigate the performance of two such language models, GPT-3 and PALM, on the blicket environment. One can apply a language model to a target task via purely text-based interaction with the model, by providing a prompt to the model and evaluating its output. This prompt could be freeform, in which it has no set structure, or could be few-shot, where contains a small number of examples of correct prompt-output pairs, followed by the test prompt. Instead of text-based interaction, another option is to fine-tune the model on the target task. This involves training the model on data from the target task, and can require a sizeable amount of data. To make the evaluation more similar to how children are introduced to the task, we focus on only text-based interaction with the model, including both freeform and few-shot prompts.
As in the experiments with children, there four conditions: given versus not given hypotheses, and disjunctive versus conjunctive. In the disjunctive case, one of the three objects is a blicket. In the conjunctive case, two of the three objects are blickets. We evaluate how well the models can: 1) identify the blickets and 2) identify whether the causal structure is disjunctive or conjuctive. For GPT-3, we use the OpenAI APP and for PaLM we use an internally available interface. We have also run GPT-3 on different temperature settings and we found that this did not significantly change the result, refer to Section C.I in the appendix for details.
Freeform Prompt In the freeform prompt, we provide the model with text that as similar as possible to what children receive in the blicket experiment. The only difference is that we replace the visual components with text descriptions. We include the exact prompts in Appendix C The prompt starts with a description of interacting with a striped blicket machine and then a dotted blicket machine, each of which has three unique objects. There are three interaction examples given per machine, for example "If we put the blue pyramid on the machine, then it does not light up". In the given hypotheses condition, the striped blicket machine is conjunctive and the dotted blicket machine is disjunctive. In the not given hypotheses condition, there is not enough information to determine whether the striped and dotted blicket machines are disjunctive or conjuctive. Thus the given hypotheses condition defines the space of overhypotheses, whereas the not given hypotheses condition does not. The prompt then introduces a new blicket machine, along with examples of interactions with the machine and whether it lights up not. Finally we ask the model which objects are blickets and whether the new machine is more similar to the striped or dotted machine.
Few-shot Prompt In the few-shot prompt, we structure the input into two prompt-outpe demon- strations, one per machine, containing the same information in the freeform prompt. Unlike the information given to children, in the outputs we explicitly state which objects are blickets. In addition, for the given hypotheses condition, in the preamble we define the disjunetive and conjunctive hypotheses: "A striped machine needs two blickets to make it light up, and a dotted machine needs one blicket to make light up", and in the outputs we state whether this is more similar a striped or dotted machine
The results are reported in Table 2 We find that when given hypotheses, GPT-3 and PaLM are almost always able select the correct causal structure, but they not always able to select the correct blickets. In particular, GPT-3 frequently names too many objects as blickets, including those associated with other machines. In contrast, PaLM never identified objects as blickets that were not one of the three objects in the test task. In the two-shot setting, PaLM performs best when the space of overhypotheses is covered perfectly by the two examples given, as one would expect. However, when this space is not covered, i.e. in the not given hypothesis setting. PaI.M struggles in the two-shot setting because the test example's causal structure does not match either of the two given examples. Adding chain-of-thought reasoning in the PaLM prompt did not improve results in either setting,
Available maltttss:/bbbt.. openai cww/overvvi
2 Foor GPT-3, we use the text dax-iciainc--02 model, with temperature 0.7, maximum length 256, and frequency and presence penalties O. For PaLM, we use temperature of 0 to obtain the greedy, or one-be mestest decoding. When given the freeform input, the Pal M model output continues indefinitely, so we simply took the first two sentences of the output and ignored the rest.
8

---

Condition
Model & Input
Blickets Chosen
Causal Structure
Given hypotheses, disjunctive
GPT-3, freeform
1/1 correct, 6 wrong
correct
PaLM, freeform
1/1 correct, wrong
wrong
PaLM, two-shot
1/1 correct
correct
Given hypotheses, conjunctive
GPT-3, freeform
2/2 correct, wrong
correct
PaLM, freeform
2/2 correct
correct
PaLM, two-shot
2/2 correct
correct
Not given hypotheses. disjunctive
GPT-3, freeform
1/1 correct, wrong
Pa freeform
0/1 correct
PaLM, two-shot
1/2 correct, wrong
Not given hypotheses. conjunctive
GPT-3, freeform
2/2 correct, 7 wrong
Ma freeform
2/2 correct
P,LM, two-shot
1/2 correct, wrong
Results for GPT-3 and PaLM for the four conditions. Bold font indicates a fully correct answer. For the not given hypotheses setting, answering correctly is impossible, as there is not enough information to determine whether the blicket detectors in the training phase disjunctive conjunctive.
Table 2:
DISCUSSION & CONCLUSION
In this work, we looked into evaluating and understanding how machine learning models learn causal overhypotheses, by evaluating these models in the blicket environment. In contrast existing benchmark tasks, in which there is a fixed causal structure, this environment focuses on the need for causal overhypotheses in order to explore effectively to determine the underlying causal structure. We focused on three categories of state-of-the-art methods- deep RL, behavior cloning, and large language models-for for tasks in this environment. blicket detector experiments, children are able to learn causal overhypotheses from only a handful of observations and can apply these overhypotheses to explore effectively for new situation (Kosoy et al. 2022). In contrast, our experiments indicate that state-of-the the-art machine learning algorithms have difficulty learning and using causal overhypotheses for exploration and inference-we saw this in the weak performance of deep RL algorithms on held-out environments and in the tendency of decision transformer models to under-explore. In our experiments with language models, we provide the same observations that the children were given in Kosoy al. (2022), along with a full set of examples for the new situation (thus removing the need for exploration). Despite this, language models struggle when the hypotheses are not given, and are not able to express uncertainty about the causal structure in that case.
Given that understanding and leveraging causal structure is essential to developing general intelligence, this work highlights an opportunity for improvement in this direction, and provides a set concrete benchmark tasks to measure improvement. One direction of future work is to build machine learning models that can better learning causal overhypotheses. Modular architectures have shown to be helpful in understanding causal hypotheses of the environment (Goyal et al.) 2019 202 Ke al.) 202T); it would be promising to explore such models for causal overhypotheses understanding, Another direction of future work to improve on exploration in RL agents by explicitly learning and incorporating causal overhypotheses, in order to narrow down the search over possibilities. Furthermore, an interesting direction train machine learning models on children's 's exploration behavior used in solving causal problems, for instance the trajectories provided by Kosoy et al. (2020). This may lead to new insights for improving causal overhypotheses learning for machine learning models.
A main limitation of this work is that we evaluated only two types of causal overhypotheses- conjunctive and disjunctive. These are particular types of overhypotheses on the causal graph structure. In the real world, there exist many other types of overhpotheses, as well as conditional probability distributions. It would be interesting to see one can extend either the blicket or other RL environments (such as Ahmed et al. 20 Ke et al. (202T) to include other types of causal overhypotheses. With minor changes, one could extend the blicket environment to test other kinds of causal overhypotheses, such as inferring whether systems
stochastic or deterministic, or require sequential or unordered interventions. This environment also allows for multiple measures of causal inference, including interventions and counterfactuals, well predictions.

---

REFERENCES
Ossama Ahmed, Frederik Träuble. Anirudh Goyal, Alexander Neitz, Yoshua Bengio, Bernhard Schölkopf, Manuel Wüthrich, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning, arXiv preprint ar arXiv:2010.04296, 2020.
Kelsey R. Allen, Kevin A. Smith, and Joshua B. Tenenbaum. Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning, Proceedings of the National Academy of Sciences, 117447)29302229311 2020. ISSN 0027-8424. doi: 10. 00073//nas.191244111 URL
https: www wwws. /ttp://wwww..aa.ooooooooooooooa/ wwt.tettt1///////2005000
Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning, arXiv preprint arXiv:2109.00155 2021.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. In Advances in Neural Information Processing Systems, pp. 5082--093, 2019.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pp. 1471-1479, 2016.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint 1xXiv:1901.10912, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016a.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016b.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data, Advances in Neural Information Processing Systems, 33:21865-21877, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: /2894, 2018
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01 2021.
Maxime Chevalier- Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, 2018.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. arXiv preprint arXiv: /0/..02/// 2018.
Alison Gopnik. Scientific thinking in young children: Theoretical advances, empirical research, and policy implications. Science, 3376100):1623-16222 2012.
Alison Gopnik and David M. Sobel. Detecting blickets: how young children use information about novel causal powers in categorization and induction. Child Development, 71(5): ((55):2000)1)) 1005-1222 2000.
10

---

Alison Gopnik and Henry M Wellman. Reconstructing constructivism: causal models, bayesian learning mechanisms, and the theory theory. Psychological bulletin, 138(6):1085, 2012.
Alison Gopnik, Shaun O'Grady, Christopher G Lucas, Thomas L Griffiths, Adrienne Wente, Sophie Bridgers, Rosie Aboody, Hoki Fung. and Ronald E Dahl. Changes in cognitive flexibility and hy- pothesis search actoss human life history from childhood to adolescence to adulthood. Proceedings of the National Academy of Sciences, 114(30):7822-889, 2017.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint Stti::9909.109// 2019.
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems, 34, 2021.
Thomas Griffiths and Joshua Tenenbaum. Theory-based causal induction. Psychological review, 116(4):661, 2009.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: https:://gitub..oo github. com/
hill hill-a/stable-bat ine nes 2018.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019 3026, 2020.
Nan Rosemary Ke, Aniket Rajiv Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Jimenez Rezende, Michael Curtis Mozer, Yoshua Bengio, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. arXiv preprint 2xX.x:2177.000 00848, 2021.
Charles Kemp, Andrew Perfors, and Joshua B Tenenbaum. Learning overhypotheses with hierarchical bayesian models. Developmental science, 10(3):307-321, 2007.
Charles Kemp, Noah D Goodman, and Joshua B Tenenbaum. Learning to learn causal models. Cognitive Science, 34(7):1185-1244 2010.
Eliza Kosoy, Jasmine Collins, David M Chan, Sandy Huang, Deepak Pathak, Pulkit Agrawal, John Canny, Alison Gopnik, and Jessica B Hamrick. Exploring exploration: Comparing children with rl agents in unified environments. arXiv preprint 2rXiv:2005.028800 2020.
Eliza Kosoy, Adrian Liu, Jasmine Collins, David M Chan, Jessica B Hamrick, Nan Rosemary Ke, Sandy H Huang, Bryanna Kaufmann, John Canny, and Alison Gopnik. Learning causal overhypotheses through exploration in children and computational models. arXiv preprint 200i: 2022.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint 2205 14334, 2022.
Christopher G Lucas and Thomas L Griffiths. Learning the form of causal relationships using hierarchical bayesian models. Cognitive Science, 34(1):1 13-147, 2010.
Christopher G Lucas, Sophie Bridgers, Thomas L Griffiths, and Alison Gopnik. When children better (or at least more open- (een-midded) learners than adults: Developmental differences in learning the forms of causal relationships. Cognition, (31(2):884-99) 2014.

---

Marlos C Machado, Mare G Bellemare, and Michael Bowling, Count- exploration with the successor representation. arXiv preprint /807, 11622, 2018a.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018b.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count- exploration in feature space for reinforcement learning. arXiv preprint arXiv:1706. 080900 2017.
Daniel McDuff, Yale Song, Jiyoung Lee, Vibhav Vincet, Sai Vemprala, Nicholas Gyde, Hadi Salman, Shuang Ma, Kwanghoon Sohn, and Ashish Kapoor. Causalcity: Complex simulations with agency for causal discovery and reasoning. arXiv preprint arXiv:2106. 13364, 2021.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937. PMLR, 2016.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv: 104.03804 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026-4033 2016.
Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning-Voiume Volume 70, pp. 2721-2730. JMLR. org. 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16- 17, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. arXiv preprint arXiv: 04160,166111 2019.
Amy Perfors, Joshua B Tenenbaum, Thomas L Griffiths, and Fei Xu. A tutorial introduction to bayesian models of cognitive development. Cognition, 120-33:32-331,22
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Jürgen Schmidhuber. Curious model-building control systems. Proc. international joint conference neural networks, pp. 1458-1463, 1991.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. arXiv preprini arXiv:2102.11109 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Re- won Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv preprint vrXiv:2201.11990 2022.
Sumedh A Sontakke, Arash Mehrjou, Laurent Itti, and Bernhard Schölkopf. Causal curiosity: RI agents discovering supf-suue experiments for causal representation learning. International Conference Machine Learning, pp.
PMLR, 2021.
12

---

Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforeement learning. In Advances in neural information processing systems, pp. 2753-2762, 2017.
Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow mind: Statistics, structure, and abstraction. science, 331(6022)1129--2235 2011.
Victor Veitch, Dhanya Sridhar, and David Blei. Adapting text embeddings for causal inference. In Jonas Peters and David Sontag (eds.), Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of Muchine Learning Research, pp- August 2020.
Jane X. Wang, Michael King, Nicolas Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, Francis Song, Gavin Buttimore, David P. Reichert, Neil Rabi- nowitz, Loic Matthey, Demis Hassabis, Alexander Lerchner, and Matthew Botvinick. Alchemy: A structured task distribution for meta-reinforcemen learning. arXiv preprint arXiv:1102.02926. 2021a.
Xingqiao Wang, Xiaowei Xu, Weida Tong, Ruth Roberts, and Zhichao Liu. Inferbert: A transformer- based causal inference framework for enhancing pharmacovigilance. Frontiers in Artificial Intelli- gence, 4, 2021b.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):779-22,, 1992.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv: 1tt.:9.9.0........ 2019.
Chi Zhang. Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. Acre: Abstract causal reasoning beyond covariation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10643-10653, 2021.
13


Reviews:
[{"review_id": "vorcUeGnyI9", "reviewer": "Reviewer_6AJY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper leverages the (virtual) blicket detector environment to empirically evaluate the ability of machine learning models to learn and use causal overhypotheses, which refer to those hypotheses that concern not specific causal relations but more general constraints about causal relations, such as whether a causal mechanism is conjunctive or disjunctive. This work is motivated by solid findings in the cognitive science literature on the competence of young children to learn causal overhypotheses and exploit them to probe causal relations in this environment. It builds on the setup in (Kosoy et al. 2022) and designs interesting experiments to assess a range of machine learning models regarding the learning of overhypotheses, including popular deep reinforcement learning algorithms, some imitation learning algorithms, and some large language models. Various weaknesses of these machine learning agents are empirically demonstrated.  ", "summary_of_the_review": "It is a useful empirical study about an important topic that should receive more attention in machine learning. ", "strength_and_weaknesses": "Strengths:\n\n1. Learning causal overhypotheses is an important task to be addressed in machine learning.\n2. A variety of machine learning models are considered and assessed.\n3. The experimental results are mostly interesting and telling, and some potentially useful benchmarks are given. \n\nWeaknesses:\n\n1. The distinction between learning an overhypothesis and exploiting the overhypothesis in exploration or inference of causal relations is not always clearly made. As I understand it, the target overhypothesis in this paper concerns whether the causal mechanism is conjunctive or disjunctive. If so, the results in the paper seem to suggest that at least the RL agents and the language models are able to learn the correct overhypothesis most of the time; what they seem to be unable to do well in the blicket environment is take advantage of the overhypothesis to figure out which objects are blickets. Is this reading correct? If it is, it is not clearly described or explained in the paper.\n\n2. The work is purely empirical and limited to certain ways to apply reinforcement learning, imitation learning, and large language models to causal inference. It is unclear whether the observed weaknesses are due to these approaches or to the authors' particular implementations of these approaches for causal inference.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nGoPfM7iFT0", "reviewer": "Reviewer_ypFh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a causal learning benchmark to test the learning of causal overhypotheses. The authors test several RL algorithms as well as pretrained language models on this benchmark. ", "summary_of_the_review": "I think the main idea of the paper is novel and attractive. However, I need a lot of clarifications to properly evaluate the paper.", "strength_and_weaknesses": "Strength\n1) The paper is well-motivated;\n2) Results with LLM  are very interesting;\n3) A direct comparison of any algorithms with children of age four can be made. \n\nWeaknesses\n1) My main concern is that I did not understand how the proposed benchmark test for overhypotheses. I do not understand how one can understand that a considered algorithm or a child learned causal overhyposes. Can you elaborate on it? The authors wrote “The reward function should capture whether the algorithm has learned the causal overhypothesis of an environment”, but  I do see only that the proposed reward can help distinguish whether the agent learns to discover (or generalize to discover) a causal graph (or blinkets) during the test/eval stage. \n2) Following the 1. The authors wrote: “We find that although most agents have no problem learning causal structures seen during training, they are unable to learn causal overhypotheses from these experiences, and thus cannot generalize to new settings.” I do see that algorithms did not generalize to the new settings, but I do not understand why it is due to overhypotheses. In some sense, I feel the opposite. For example, in the case when models are trained only on disjunctive data, the model learns from the data that conjunctive case is not possible. So the model learns overhyposes prior perfectly. Another story is that the model encoded this prior to the weights of the model and can not recover from it and generalize to conjunctive out-of-distribution data in the test stage.\n3) I doubt that any reliable conclusion can be made based on such a small hold-out test set. I believe the experimental results rather motivate father research in this direction than allow to make reliable conclusions.  (the authors noted it themself). Maybe the easiest workaround is to consider more than three objects and more than two blinkets to train and test algorithms;\n4) I do not understand how the author trains RL algorithms. For example, the author wrote: “We train all of these algorithms in the given hypothesis scenario, where the agent is exposed during training to all possible overhypotheses”.  I do not understand what this means. \"Given hypothesis\" is a case where an agent receives demonstration data, but how this offline data is used to train RL agent? Is it used in offline RL settings? Do the authors train the agent on-policy then?\n\n\nSmall issues and questions\n1) If the maximum achievable reward is always 3, I believe the authors give a reward for correctly identifying the identity of the object (is it a blicket or not). The author's wording “they receive a reward of 1 for identifying a correct blicket” confuses me to think that the maximum reward is 1 for the disjunctive setting and 2 for the conjunctive setting.\n2) What is the distribution of the tasks when you train RL agent?\n3) In the situation when you hold out disjunctive overhypothesis, does it mean that the training set does not contain examples when object 3 (without loss of generality) is a blanket? \n4)  If the agent allows checking all the options for the blinkeness at the same time, is it one of the working strategies for the ideal algorithm to check all combinations in one round (it is only 8 for 3 objects) and then conclude what objects are blinkets?\n\n\nI also believe the paper is strongly connected to meta-learning in RL, in which the agent needs first explore an environment to understand what task it should solve and then solve this task (Rakelly et al, Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML2019).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ao75_4S7UL", "reviewer": "Reviewer_cvDM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper utilized a recently developed framework of \"blicket\" environment and conducted experiments using reinforcement learning, imitation learning and Language models. The goal of the paper is to understand if these machine learning method could acquire a causal understanding of the combinations of objectives to the \"blicket\".\n\n ", "summary_of_the_review": "Learning causal over-hypothesis is essential for AI to robustly help us to deal with automated tasks under the challenge of distribution shift. If my understanding were correct, the paper proposed a benchmark suite, which I am not fully convinced about the importance of this suite compared to constructing some other benchmark to evaluate the causal performances of machine learning algorithms. ", "strength_and_weaknesses": "Strength:\nI think the paper is quite novel in trying to bridge other non-computational disciplines to ICLR.\n\nWeakness:\nIf I understood it correctly, the paper seems to be only running some experiments and check some performances which has an association with the causal measures on the over-hypothesis. But the paper did not give any insight. It looks to be providing a benchmark suite. In the RL experiment, the visual image is replaced with symbolic representation also. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject