Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work addresses the problem of errors in structure learning algorithms.  The authors propose to reweight poorly fit samples in order to improve the efficacy of the underlying algorithm. A proof is shown under linear models for specific scoring rules in the asymptotic regime. Empirical results show the proposed method improving state of the art structure learning algorithms. 
strength_and_weaknesses: Strengths: 
* The relative frailty of structure learning algorithms is a major roadblock to practical usage, so this task is very well motivated. 
* The authors present an algorithm which is simple and intuitive
* Empirical results are very compelling

Weaknesses:
* The authors reweight samples with the intuition that these correspond to spurious edges, however it is not entirely clear to me why this should be limited to one reweighting step. Can the authors give some sort of intuition on why only one boosting step is used? What would happen if multiple reweighting steps were employed?
* Theorem one is incredibly limited in scope. I don't think this necessarily a problem but the authors should reconcile their claim that the approach is applicable to any score base learner with the assumptions that restrict the space of models and algorithms substantially. 
* The approach deals with recovering directed graphs, but the authors frame in terms of causal discovery. In the case of causal discovery, only an equivalence class is recovered, not a fully directed graph. Do the authors have suggestions or intuition regarding the modifications necessary to have the algorithm work when an algorithm (e.g., GES) returns an equivalence class after every step?
clarity,_quality,_novelty_and_reproducibility: Overall, I think the authors have written a clear and well organized paper. Outside of the issues that I raised I think the quality is quite good and the novelty is quite high, in my view. 
summary_of_the_review: As I mentioned above, I think this is a well motivated and simple solution to a very compelling problem. I think the idea that the procedure can be applied a wide variety of algorithms is quite compelling. My reservations are listed above in the weaknesses, and is largely in the theoretical underpinnings and some of the details of the proposed approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a model-agnostic framework that improves pre-existing score-based models for causal discovery. This is done by adapting the score weights for each sample while training the backbone model, thus alternating between training the backbone model and updating the weights.
The weights are updated in such a way as to give a larger impact to the less-fitted samples, which might be more informative to the backbone model. This framework is claimed to have a particularly large impact when dealing with heterogeneous data (even in cases when we do not know what samples come from which cohort).
strength_and_weaknesses: Strengths:

* The exposition of the method is quite clear.

* The proposed framework is very flexible, as it can be used on top of many other score-based methods.

Weaknesses / questions about things that are unclear:

* A main concern is that *the paper claims improvements for any score-based method, while this claim is only supported for the case of differentiable score-based methods*. In the experiments, the proposed framework is applied only with differentiable methods. Also, the framework is motivated from observed shortcomings of differentiable methods (bullets page 1&2). It is possible that the proposed framework also gives improvement with discrete score-based methods, but such a claim is not supported by the paper. The scope of the claim needs to be made clear, starting in the abstract.

* Can you explain an example of how single datapoints may provide "crucial causal information", or how without reweighting there is  "overfitting" (both page 2)? I think figure 1 starts to demonstrate such issues, but does not explain these two points.

* In Theorem 1, what does it mean to say "asymptotically", when the vector of weights is held fixed (and so its length can't increase to accommodate more datapoints)?

* The argmax in (6) will often equal the empty set, because the set $\mathbb{C}(\tau)$ is open (so there is a supremum, but no maximum). Replace the inequalities between the $w$'s and the $\tau$-terms by $\leq$ to make the set closed.

* The statement and proof of Theorem 2 are incorrect. The proof can only show that $w_i^* \geq w_j^*$. Because in the proof, when supposing the contradiction, if $w_i^* = w_j^*$, there is no such $\epsilon$. (BTW, the $\mathbb{C}$ should be $\mathbb{C}(\tau)$. Also, you don't need this set to be open.) But this substantially weakens the statement of the theorem: it now leaves open the possibility that all weights are set equal.

* Theorem 2 seems to suggest that the weights are increasing as a function of the losses (over the different datapoints). Can you explain why this is not what we see in Figure 3? (A possible reason might be related to the text "in the optimization phase" in Theorem 2. It is not  clear exactly what point of the bilevel optimization algorithm this refers to.)

* Synthetic data experiment, bottom page 8: what does it mean that the "noise scale is flipped"?

* Sachs data: This dataset contains (I think) 14 different interventional regimes, which are pooled together in this experiment. Pooling is not the most effective way to deal with these data (see eg Mooij, Magliacane and Claassen, 2020). As such, I disagree with the conclusions that this experiment "highlight[s] the ineffectiveness of score-based causal discovery methods when dealing with real-world data" and that "integrating a strong backbone DAG learner with ReScore is the only way to truly advance in this field". Further, due to these regimes, this experiment should be listed under "heterogenous data".

Minor comments (not necessarily part of decision assessment):

* I am not familiar with the terminology "disadvantaged" domain/data point; I think you mean "underrepresented"?

* (page 5) "Initially, ... are assumed": This sentence is unclear. I think you want to say that a *family* of functions and a *family* of noise distributions need to be chosen. The use of the word "specific" confuses me though.

* For the experiments with heterogenous data, you could consider comparing to a method that uses non-adaptive weights, set inversely proportional to the group sizes.

* The manuscript contains many grammatical mistakes, as well as places where the wrong word was chosen (eg "alternative(ly)" when "alternating(ly)" is meant; "phrases" when "phases" is meant; ...). Please thoroughly proofread the manuscript and correct these errors.
clarity,_quality,_novelty_and_reproducibility: The structure of the paper is very clear, and overall the paper is easy to follow. Exceptions are detailed above.

There are problems with the theoretical results, detailed above.

The proposed method is novel to the best of my knowledge.

I see no problems with reproducibility.



summary_of_the_review: The proposed method is novel as far as I know, and the experimental results suggest it often yields impressive performance improvements. Because of some issues with the current manuscript, I am currently recommending rejection, but if these issues can be addressed, I expect to recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the score function. In particular, the proposed method leverages the bilevel optimization scheme to alternatively train a standard DAG learner first and then reweight the samples that the DAG learner fails to fit well.
strength_and_weaknesses: Strength:

1. The paper is well presented. 
2. The proposed sample reweighting method can help causal discovery in heterogeneous data by auto-learnable adaptive weights.
3. The proposed sample reweighting trick is simple and is flexible to combine with any score-based continuous optimization method in causal discovery.

Weakness:

1. It seems after adding the sample reweighting step, the reported empirical performance, especially FDR, instead gets worse in some cases. Is it because bilevel optimization is harder?

2. With the sample reweighting step, the algorithm is computationally much more expensive.

3. It would be more intuitive to understand the benefit of reweighting if the authors could provide illustrative examples.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: 
The paper is well organized and presented. It provides both theoretical guarantee and practical evaluation.

Novelty: 
The sample reweighting trick to improve causal discovery performance is novel, as far as I know. 


summary_of_the_review: The paper is well organized and presented. The sample reweighting trick to improve causal discovery performance is novel, as far as I know. It is also simple and flexible to adapt to other continuous optimization methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides a method for improving the robustness of score-based DAG learning methods to heterogeneous noise.  It works by iteratively up-reweighting the poorly fitted observations and then rerunning the base method.

strength_and_weaknesses: The method seems useful and clearly improves performance in simulations, as well as on the well-known Sachs data.  

The weaknesses are: 
1. It is computationally more expensive - this is presumably inevitable given that it involves running the main method several times.
2. No mention is made of the possibility of spurious outliers.  These are generally dealt with in statistics by _down_-weighting, not up-weighting. 

clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written, and easy to follow.  As far as I am aware this approach is novel.  The simulation and real data results suggest that the method works well, and there is associated code that can (presumably) reproduce the authors' results.
summary_of_the_review: This seems like a nice contribution to the literature, allowing for more accurate learning of DAG structure with score-based algorithms.  Please do ensure you take into account my comments below in your revision.  

### Comments 

1. What happens if the observations are just statistical outliers, and don't really contain any useful information?  I could easily imagine just a few observations totally disrupting your algorithm in this context, because they would presumably be massively up-weighted and might overwhelm your fitting algorithm.

2. You say that you can fit your method to 'any score-based causal discovery method'.  This is not correct - first, and most obviously, you need the individual data; second the data must be assumed to be independent; and third it is not clear to me whether the method would work for other classes of graphs.

3. "However, it is no doubt that integrating a strong backbone DAG learner with ReScore is the only way to truly advance in this field."  This statement is much too strong; it's not at all clear that another approach might work just as well as yours, or even better!

4. In the proof of Theorem 1, you write down the conditional variance formula at the bottom of page 15 but you implicitly assume that $\mathbb{E} [(X^TWX)^{-1} X^TW N^j \mid X] = 0$.  This is fine (since it's used on the previous line), but you need to state that this is what you're doing, or the proof is hard to follow.  I would suggest giving the conditional variance formula before you start and then noting that this second term is zero.

### Typos/Minor Points

 - page 4: "combinatorial optimizaiton" $\to$ "combinatorial optimization";
 - page 5: "hard less-fitted samples" $\to$ "less well-fitted samples";
 - page 6: "negative llog-likelihood"
 - page 9: "In sheer contrast" $\to$ "In stark contrast"
 - page 15: "one is easy to obtain that" $\to$ "one can easily obtain that"
 - below (9) "of that term" - which term? The one from (8), presumably!
 - page 16: " 'case a.'. " should be just " 'case a.' " and the open quote should be inverted.
 - just below "standard Gaussian distribution": not really, since the variances are not all 1.
 - below (10) "equivalent to minimize the first term": I think you mean the second term.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a simple but effective method to boost causal discovery performance by dynamically learning the adaptive weights for the reweighted score function. It works by iteratively up-reweighting the poorly fitted observations and then rerunning the base method. The method seems useful and clearly improves performance on simulated data and the Sachs data. As the price to pay, it is computationally more expensive. Overall, the paper is nicely written, and the proposed method is novel and has been demonstrated to work well across multiple scenarios.
justification_for_why_not_higher_score: Although the method was demonstrated to be useful, its properties are not totally clear. For instance, after adding the sample reweighting step, the reported empirical performance, especially FDR, actually gets worse in some cases, and the reason is unclear.
justification_for_why_not_lower_score: The proposed sample reweighting method can help causal discovery in heterogeneous data by auto-learnable adaptive weights.
Decision: Accept: poster