Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: An excellent, though unusual paper
Main review: Overall, this is an excellent paper. Its findings are relatively modest, but that is because it takes on a very difficult challenge: to characterize the optimality of human causal exploration.

It has a relatively unusual set of contributions for a paper in a typical AI or ML conference (though any given year of NeurIPS often has at least a few papers of this kind). However, CLeaR should strive to be open to all research seeking to understand causal reasoning and learning. This paper certainly does that.

The thematic organization of the related work section is great. It makes it easy for readers to understand the diverse set of connections to related work in several disparate areas.

The study includes human subjects, which requires a great deal of effort to get approval, recruit subjects, and perform the experiment.

The hypothesis space described in section 5 seems to assume that the identity of the potential blickets (i.e., the unique combination of attributes) are the crucial factor rather than individual attributes of the potential blickets (e.g., shape, color). For example, it seems entirely reasonable to have causal structures such as “green-con” (two or more green objects) or “triangle-cube-con” (a triangle and cube) or even “blue-ish-con” (blue tinted objects). The assumption that concepts must concern identity greatly reduces the size of the hypothesis space. The authors should either account for this in their analysis or explain why they choose to use identity-based rather than attribute-based hypotheses.
review_title: null

final_score: rating: 3: Clear rejection
review_content: Summary: This paper investigates the role of “overhypotheses” (i.e., structured priors for causal structures in a domain) in causal learning. 
Main review: These overhypotheses have previously been shown to play an important role in human causal learning, and while structured priors are sometimes used in machine causal learning, they are not typically expressed as overhypotheses. The authors develop an online environment to run blicket experiments involving active learning, and then report data about an experiment using that environment. That experiment showed that children’s behavior is not well-modeled by relatively simple (by the authors’ own admission) RL learning models.

The topics of exploration-based causal discovery and causal overhypotheses are both important ones, and ML work on them can certainly be informed by experimental work with humans. However, the present experiment has a number of issues and potential confounds that make it difficult to evaluate the results. For example, there is no indication about exactly what children were told about the blicket machine, including key information such as whether they were told that it was a deterministic device. The training cases have different numbers of successes between the “Given” and “Not given” conditions. The task is framed as figuring out which of two causal possibilities obtains (in terms of causal structure), rather than learning an overhypothesis. And so on for a number of other issues. I want to emphasize that I think this kind of experiment is potentially quite valuable; the concerns are about this particular realization.

The paper is generally framed with a focus on the development of novel ML methods, but there does not seem to be any significant ML development here. There are two different relatively simple RL methods that are compared with the developmental data, but no explanation about how the developmental data could actually lead to new methods or algorithms. I think that is a significant shortcoming given the focus of this conference.

Minor note: The paper title has a notable typo
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This study investigate the acquisition of causal knowledge and causal overhypotheses through exploration, and how the overhypotheses influence exploration and acquisition of causal knowledge. The reviewer finds this work interesting and well-motivated by the introduction. Several points in the methods and analysis can use further clarification. The development of the web tool is a contribution to the community, however as the author noted the experiments and analysis presented is on the exploratory end. 
Main review: CLEAR Review: Learning Casual Overhypotheses through Exploration in Children and Computational Models
The reviewer has background in statistics and machine learning. The reviewer only have familiarity with reinforcement learning, and about 4 graduate level courses in cognitive science. The reviewer is not up to date on the literature in this domain, therefore the review focuses on the technical and statistical aspects.

Specific comments:

Figure 3: Are the reported proportions averaged over all children? It might be interesting to show variability among the children. Also, it might be helpful to indicate what is the performance level of random guess. 

Page 7-8: It would be interesting to see the proportion of children that fully understand the conjunctive and disjunctive relationships (rather than simply identify what are the blinkets). I think identifying blinkets vs. non-blinkets is a different problem (what are the causal factors) compared to understand the conjunctive and disjunctive relations (what is the more precise the causal mechanism). Although some children collected enough evidence to be able to fully disambiguate the conjunctive vs. disjunctive, it does not mean they actually can disambiguate them.

Page 9: "We excluded causal structures such as A-con and B-con where no observations can distinguish between them." I am not sure if I understand this. Maybe we cannot distinguish A-con vs. A-dis? becuase they are the something?

Section 5.1: I think one important question is: were the children specifically told that they goal is to figure out how the machine work? And how was that information communicated. The RL agent have a very specific cost function, where the children's cost function is not well defined, or unconstraint possibly. The authors touched on this, but the experimental setting for the children can be described with more detail. It would be interesting to see if the children's explore/learning strategy change if the task is described or incentivized differently.
review_title: null

Meta-review: Thank you for your submission to CLeaR. This paper presents a study of children investigating the workings of the bicket machine, finding that they seem to make use of causal over-hypotheses to structure their explorations. The paper discusses potential implications for RL exploration. And additional thanks for your detailed response to the reviews, addressing many questions about the bicket study itself.

Reviewers appreciated the work and believe its findings to be thought-provoking. Reviewers, did have disagreement about the significance and support of the contribution from a machine learning perspective. After discussion, we concluded that while the implications of this work for novel machine learning algorithms is preliminary, this work represents an excellent example of how exploration of human use of causal reasoning may inspire new approaches to causal machine learning.

We request the authors to address points raised in the detailed reviews. In particular, we ask that the authors (1) incorporate the procedural details and clarifications provided in the authors response into the main paper (and/or appendix, as appropriate); and (2) clarify the focus of this work in the abstract and introduction to reduce reader expectation that this work is intended to lead directly to novel ML algorithms. Finally, though additional experiments with a broader range of machine learning algorithms would strengthen the ML implications, ensuring that “implications for ML” comes across as a forward-looking discussion will obviate much of those concerns.
Decision: Accept (Oral)