Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors work out a method that, given a structural equation model having exponential family conditional densities, returns/computes the worst-case loss (i.e., negative log-likelihood) of any predictive model fitted to data arising from the SEM over a class of distributional perturbations to the SEM.  Computing the worst-case loss "naively" requires solving a (constrained) non-convex optimization problem, so the authors propose a vanilla sequential convex programming (trust region) method instead that seems to work well. The method is useful from an interpretability standpoint b/c it can tell us which conditionals (e.g., which "features") had to be wiggled the most in order to attain the worst-case loss.
strengths_and_weaknesses: Strengths:
* Very cool/timely topic, w/ an approach that is apparently well-thought through / totally reasonable
* The experiment in Sec 4.2 was neat
* Thm 1 is interesting/useful, though not surprising
* Clear / well-written paper

Weaknesses:
* I don't know that I totally get the approach here?  Why not define the worst-case loss to be taken over all distributions in a K-L (or any f-divergence, maybe) ball centered around the empirical measure?  I **think** the problem that falls out from that should be a (constrained) convex optimization problem.  Case closed.  Having said that, a difference in opinion on approach doesn't necessarily mean a paper is "no good" ...

* The advantage of the K-L type approach I just mentioned is you don't have to make distributional assumptions, whereas in your case you do.  Actually you assume the user knows quite a bit about the problem at hand: the SEM, that there exist conditional densities that are exponential family, and the shift function(s) -- is all that realistic?  Moreover, the shift function has to be specified on the natural parameter scale (e.g., on the $\mu/\sigma$ scale, in the case of a Gaussian density) -- how interpretable is that / can people easily think in those terms?

* Another potential approach: just draw data from several candidate SEM's (perhaps obtained by doing a very coarse grid search over your $\delta$ parameter) + take the worst-case loss over this (finite) set of SEM's.  This is very roughly in line w/ what you do in Sec 4.2.  Clearly this Monte Carlo-type approach cannot be **too** good, but I'm wondering how much worse it actually is from your approach ...

* There could probably be a few more / more thorough experiments in the paper ...
questions: In addition to the questions embedded in the "Weaknesses" section above, here are a few more Q's for the authors:
* how would you do inference (e.g., get confidence intervals) for your estimation target (the worst-case loss)?
* it seems taking the worst-case over alternate graph **structures** (i.e., not just **parameters**) would be super useful -- any thoughts as to how tractable that might be?
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the present a method to evaluate the robustness of models. This is done by first parametrizing the space of distribution shifts and then maximizing an approximation of the model loss over the shift parameter space. The approximation is done via second-order Taylor expansion and some theory is provided about the approximation error. Finally, they show in experiments the ability of their method to find interpretable distribution shifts that negatively affect a model.
strengths_and_weaknesses: Strengths:
* Parameterizing shift space to probe biases is an original (to my knowledge) idea that could be a useful way to find interpretable biases.
* Good experiments
* Well written

Weaknesses:
* The approximation method could be too inaccurate in practice (see limitation 1)
questions: 1. The supremum factor in Theorem 2 is hard for me to reason about. Are there reasonable settings where this bound can be simplified (e.g. Lipschitz losses)?
2. In Line 216, it is said that the variance of the Taylor approximation is $O(||\delta||^4)$, but shouldn't the second term in (8) contribute $O(||\delta||^2)$ variance? Should I be thinking of $\delta$ as bounded away from 0?
3. In section 3.3, the worst-case approximate loss is computed in a 2-norm ball. However, it seems to make more sense that the ball should be the sup-norm, with the idea being that each mechanism can be perturbed independently. Is it worth seeing what happens when you use that, or do you lose tractability?

limitations: 1. I think the accuracy of the approximation needs to be better guaranteed in practice. It seems unclear based on just Thm 2 that the second-order Taylor will be accurate enough at moderately sized delta. One could perhaps verify the bound by sampling at the perturbed delta and checking the model's performance there, but that may not be always feasible.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper aims to tackle the problem of model evaluation by understanding the sensitivity of a predictive model to dataset shift using only data from the training distribution. The authors consider parametric shifts with Conditional Exponential Family (CEF) under a set of changes that are interpretable and controllable. For CEF distributions, authors derive a local second-order approximation to the loss (Theorem 1). This approximation enables the use of efficient optimization algorithms to find the worst-case shift. Authors also provide error bounds to their approximation (Theorem 2). On CelebA classification task, the authors show the efficacy of the proposed framework.  
strengths_and_weaknesses: **Strength**:  

- The problem considered in the paper is very interesting and timely. Understanding model performance beyond iid validation test is important prior to deployment. This can also enable examining the worst-case distribution P which can help identify weaknesses of the model.
- Approximation of the loss with the CEF with second-order Taylor expansion is interesting. Theorem 1 derives an expression for SG$_i^1$ and Theorem 2 provides an error bound for the approximation error in population. 
- The running Example 1 is illustrative. Connecting back to it at many places helps in understanding the proposed framework and formulation 


**Weakness**: 
- The paper could do a better job in explaining what all kinds of distribution shifts are captured with Definition 3. Again the example in Section 4.1 is illustrative and interesting but overall, it is not clear how general the considered CEF framework is? 
- Experimental evaluation of the proposed framework is very slim and a major weakness of the paper. While I acknowledge the theoretical contribution and novelty of the paper, as mentioned before, the generality of the CEF framework is unclear and experimental validation is lacking. This is the main reason for my score. I am open to changing my score if the authors could comment more on this point or if I share any misunderstanding. 
  - (Minor) In the CelebA experiment, how can a user identify which variables to intervene upon and how to decide their bounds, i.e., the problematic dependencies (for which model performance changes considerably under dataset shift) in the data?  
  - (Minor) Authors use Causal GAN to simulate distribution shifts, how reliable is even the ground truth evaluation in this framework? While I appreciate the details in Appendix D with some pictures of the generated example, a commentary on the generated dataset would be great. 
  - As mentioned in the limitations section, specified shifts can fail to capture important real-world changes and hence, the resulting worst-case loss may be overly optimistic and misleading. Can authors comment on this more in the context of CelebA experiment where the worst-case shift identified only has a drop of ~3 percentages in accuracy? 
- (Minor) While authors acknowledge the sources of error due to finite samples, complimenting Theorem 2 with errors due to finite-sample approximation would be insightful. 
- (Minor) Paper is a bit hard to understand at places. For example, 
  - Definition 3 gives the parametric family of shifts considered in the paper. The changes w.r.t. Definition 2 could be reminded to the reader which in particular is in the $\eta_\delta$ function. 
- (Minor) Writing of the paper could also be improved. For example, 
  -  A lot of references to "these" and "them" are used where the contextualization is a bit ambiguous and left for interpretation. E.g., abstract line 2-5. 
  
questions: While the motivation of the paper is very interesting and the proposed framework is novel, I have several suggestions/questions: 

- How to decide the bounds on $\delta$, i.e., robustness set $\mathcal{P}$ is unclear. Could the authors elaborate a bit more here?
- To tackle the variance issue in the importance weights, in Eq. 6, did the authors try doubly robust estimators [1]? Maybe this can already improve the numbers in Table 2 (right) for IM.
- As mentioned in the weaknesses above, could the authors explain how general their considered CEF framework is? 
- How does the framework considered here compare with Mandoline's framework [2]? It would be great to see the `pros and cons' of the CEF framework with the Mandoline framework, in terms of coverage of model evaluation settings, meta-information needed from humans, and underlying assumptions?
- How does the accuracy estimation in CelebA experiment stand with other heuristic methods, e.g., GDE [3], ATC[4]. 


[1]  Reddi et al. 2014. Doubly Robust Covariate Shift Correction. https://www.cs.cmu.edu/~sjakkamr/papers/doublyrobust.pdf. 

[2] Chen et al. 2021. Mandoline: Model Evaluation under Distribution Shift https://arxiv.org/abs/2107.00643

[3] Jiang et al. 2022. Assessing Generalization of SGD via Disagreement. https://arxiv.org/abs/2106.13799. 

[4] Garg et al. 2022. Leveraging Unlabeled Data to Predict Out-of-Distribution Performance. https://arxiv.org/abs/2201.04234
limitations: The authors provide a summary of the limitations of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper considers the problem of identification and evaluation of potential harmless data shifts. The paper proposes a parametric robustness set and proposes a local second-order approximation to efficiently solve the worst-case problem
strengths_and_weaknesses: The problem of proactively understanding the policy's performance under distributional shifts is interesting to me.

However, honestly, I feel this paper is hard to understand, perhaps I am not very familiar with the language used in this community.
1. For Example 1, are A, and Y are allowed to change? There is a $\delta_{age}$ in the caption of Figure 1, but it seems that the authors never use them.
2. The author $\mathbb{P}(V)$ to denote a distribution, which is not commonly used. I suggest the authors rigorous define everything using probability language, e.g. define the probability space {$ \Omega,\mathcal{F},P $}.
3. I am confused with Equation (4) in Assumption 1. Why there is no $\mathbb{P}(Z_i)$ and $\mathbb{P}(U_j)$.
4. I think the authors would better explain what shifts, intervention, and robust mean in this context.
5. I did not quite get the intuition behind the CEF parametrized robustness set? Why is it a natural and good choice? The authors should better motivate.
6. Are the variables $W$ the only variables that only to be intervened (shifted)? How about the variables Z,V,U?
7. I think Proposition 1 implicitly assumes the distributions of Z do not change.


Overall I find this paper needs to be further polished. And this paper at least in its current version will create difficulties for the general audience of NeurIPS to understand.
questions: 
Under which scenarios, the local second-order approximation performs well?
limitations: The definition of CEF parametrized robustness set seems restrictive.
review_title: null

Meta-review: metareview: This paper studies the important problem of structured distribution shifts. Departing from previous works that use worst-case regions based on distance measures (f-divergences, Wasserstein distances), the authors construct an interesting framework based on a (partially) specified structural causal model. The proposed framework assumes a number of problem-specific structures, which I view as a major advantage over previous approaches. However, the methodology is interpretable insofar as the causal graph is partially specified and the shift function is structured. For the proposed approach to have applied impact, I recommend identifying realistic application scenarios where these structures are prominent, going beyond toy examples. 

Furthermore, please contextualize the work with respect to the following related papers:

Taylor expansions for small robustness radius: Lam, Robust sensitivity analysis for stochastic systems, Mathematics of Operations Research, 2016.

Structured distribution shift based on SEMs: Heinze-Deml, Meinshausen, Conditional Variance Penalties and Domain Shift Robustness, Machine Learning, 2021.

Robustness evaluation: Li, Namkoong, Xia, Evaluating model performance under worst-case subpopulations, NeurIPS, 2021.
confidence: Certain
Decision: Accept