Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to use **active set** of causal edges, instead of auxiliary context variable or domain index to characterize nonstationary causal relations in data. In addition, the authors propose Sufficient Activated Mechanisms (SAM) to replace Sparse Mechanism Shift (SMS) as the inductive bias (assumption) for causal discovery and inference. 

Developing a principled way to characterize mechanism shifts in causal settings is definitely an interesting idea and potentially publishable. However, this paper introduced merely a conceptual framework instead of a working system with theoretical guarantees and contained a few unsupported claims which I feel not confident to justify their correctness. I strongly recommend the authors code the proposed meta-SCM and SAM in a training architecture and validate their correctness in both theoretical and empirical perspectives and submit the to future venues. 
main_review: **Strengths**

1. Introducing a new way to formulate changing causal mechanisms using sample-wise active sets. The authors have claimed its stronger expressiveness power in modeling unsolvable SCM and cyclic relations than using auxiliary context variables. 

2. Covering a broad spectrum of prior work which handles feedback loops between observed variables.

**Weaknesses**

The motivation of this paper is not clear. In the abstract & intro, the goal was originally to solve cyclic relations but how the proposed meta-SCM and SAM in the main content can be used to tackle the problem was not explored. This paper introduced merely a conceptual framework instead of a working system and contained a few unsupported claims which I feel not confident to justify their correctness. The advantages of the proposed active set approach over the classic way to model nonstationarity using context variables are not clear. How the proposed SAM can be coded as a training architecture is questionable.

1. *Motivation is not clear*: What is the central problem being addressed in this paper? This paper seemed to tackle cyclic causal relations in the beginning but then switched to the modeling of sample-wise causal mechanisms through active sets of edges. 

2. *No theoretical guarantee or experiment*: This paper introduced merely a conceptual framework instead of a working system.  Many critical claims in the paper are not supported by proofs or experiment results, such as the proposed *Sufficient Activated Mechanisms (SAM)* can recover causal structure if only how many factors have been activated in the sample are known. Questions such as how the proposed meta-SCM is to be built, how the SAM prior can be coded as regularization, under what conditions the proposed framework can work, have not been explored.

3.  *Effectiveness is limited*: I'm not convinced that this is an effective approach. Representing nonstationarity in data using active sets of causal edges might have some advantages over one-hot encoded context variables, such as the dimensionality might be smaller if there exist large numbers of nonstationarity regimes. However, the active sets are also limited in the sense that they cannot model changing causal relations where only the causal strengths vary, for example, for modeling a spring-mass system where spring stiffness wears out over time. Most importantly, I do not think manually encoding the discrete context variables with active sets of edges is a novel idea to the causal community. 

4. *Lack of justification for SAM*: as a counterexample (starfish) to argue against sparse mechanism shift (SMS), the authors stated that "changing the output information of the centre node (instead itself) can lead to shift on all functional relationships with its child nodes, which can affect almost all factors simultaneously." I'd like to say this is actually something expected by SMS, as the intervention only changes the marginal distribution of the center node while all conditional relations remain stationary, which still results in the changing of distributions of all factors simultaneously. Also, I can't think of a way to encode the proposed SAM into a learning rule. 
summary_of_the_review: I have to reject this paper in the current version. The proposed idea is interesting but without supporting proofs or empirical experiments, it is very hard for ICLR community to justify its correctness and judge its applicability to real-world problems.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new lens on causal graphical models from a lens of fuller generality. Rather than considering models under assumptions such as acyclicity or other constraints that enable tractable modeling, the authors consider the notion of a meta causal model which, when indexed, includes SCM as one possible snapshot. The authors give two additional assumptions (1) sparse mechanism shift and (2) sufficient activated mechanisms, and describe the implications.
main_review: This was a very interesting paper to read, and quite different than the typical conference submission. The authors address one of the most fundamental problems in causal modeling: sufficient representation. I found the model proposed to be both intuitive and general enough to address many concerns such as feedback, equilibrating systems, etc. My largest concern is in practicality: from reading it is not clear to me how one would either learn or parameterize these models. It's not clear how one would recover these indices or distinguish samples between indices, perform interventional calculus, or parameterize the models to make interventional queries. I would be interested to hear the authors' thoughts, though I do realize that these questions are out of scope for the current submission. In short, this paper is a refreshing though provoking piece of work that I believe could be the basis for many interesting future directions.
summary_of_the_review: Interesting, though provoking paper that proposes a novel paradigm for causal modeling in general systems that are often unrepresentable in current causal graphical frameworks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an extension of SCMs that allows modeling cycles in the causality. The main contribution of the paper is providing definitions for the meta-SCM via the concept of active mechanisms and effectively connecting the data to mechanisms.
main_review: ## Strengths
1. The problem of cyclic causality is an interesting problem.
2. The idea of **sufficient representation** is interesting and can be further developed.

## Weaknesses
1. Unfortunately, this paper is rushed/unfinished and needs a significant amount of work to reach the quality of a conference paper.
2. The paper has many grammatical mistakes, including misspelling the word *STRUCTURAL* in the title.
3. The theoretical advances in the paper are not enough for a conference paper. The authors need to show that they can solve a real-world problem using the proposed Meta-SCM. It is unclear if tying the definition to data will make the definition flexible or not.
4. Continuing #4, there are no experiments to validate the proposed definitions.
5. The paper is full of unscholarly language and hyperbole. Here are several examples from the introduction:
    * there are still **some important** theoretical problems
    * They have **many convenient** properties
    * But there is a **strong need** to go beyond acyclic SCMs.
    * However, they are still not well understood.
    * We propose a **totally novel** dimension 
 
summary_of_the_review: While the proposed idea of meta-SCM is thought-worthy, the current shape of the paper is far below the quality bars of scholarly conferences such as ICLR. The authors need to add empirical validation of the proposed idea and overhaul the writing of the paper. They also need to show that their proposed meta-SCM has enough novel methodology contributions.
review_title: null

Meta-review: This paper proposes a meta-structural causal model framework, to increase the representation capability of structural equation models. It also considers how to connect data to mechanisms. The paper is conceptually interesting. However, on the technical side, reviewers feel that without supporting proofs or empirical experiments, it is hard to justify the correctness of the proposal and judge its applicability to real-world problems.

As authors claimed in their response, "it is our future work of interest to code our proposed framework into a working system and validate it in a proper setting given its early stage status on research in modeling causal cycles." I think some future version of the paper might be a great contribution to the field if a working system were included.
Decision: Reject