Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of predicting item relevance to users, in a way that is robust to changes in the data distribution. It is a particular case of tackling the bigger challenge of out of distribution generalization - here different environments are represented by different users' beliefs. The paper gives a strategy for trustworthy relevance prediction which does not require full information of the underlying causal graph. It only requires one to know whether the graph is causal or anti-causal, and for each of these scenarios an appropriate regularization scheme is chosen. 
strengths_and_weaknesses: Strengths:

1. The paper is very well-written and the motivation and setup are very clear. I enjoyed reading this work. 
2. The learning algorithm proposed requires minimal knowledge of the causal graph (i.e. whether it corresponds to (a) causal and (b) anti-causal with two possible subclasses - believer or skeptic; this latter sub-distinction is very nice and quite realistic for modeling user behavior). 
3. The connection to economics and user behavior psychology by modeling the response y via an expected utility function is interesting. 
4. This model also accounts for the case in which some intrinsic information about the product (encoded in x) is missing. 
5. The algorithm is clear and the experiments are thorough, accounting for all possible scenarios. 


Weaknesses:

1. While I intuitively understand it, it is still unclear to me how the MMD and the CORAL regularization methods satisfy the respective necessary conditional independence requirements needed for preventing spurious correlations. Similarly, it is not completely mathematically clear to me why the conditional independencies described in lines 216-217 are satisfied. 
2. Inferring the type of causal and non-causal graph is still non-trivial. The authors do mention some approaches for this in lines 294-295, i.e. A/B testing or conditional independence testing from observational data, but the paper could significantly benefit from a more developed discussion of possible ways to infer the graph in practice. 


questions: My only question is the same as Point number 1 in weaknesses. 
limitations: The authors have mentioned limitations of their method both in the discussion section as well as in the description of their experimental results.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a learning framework that is robust to environment changes to predict the relevance of items to users. The key idea is to model users as reasoning about decisions through a causal graph and show how minimal information regarding the graph can be used to contend with distributional changes. 
strengths_and_weaknesses: The proposed framework is novel and the paper is fairly well written, and experimental results support the paper's statement that adopting users' causal beliefs results in better prediction performances.

However, there are no comparisons with other exciting works. According to section 2 related works, there exist, several models that adopt causal inference techniques for recommendation tasks, the paper would be more sound if the prediction performances are compared against those works.
questions: It seems the prior knowledge regarding users' causal beliefs (causal or anti-causal) needs to be known ahead for modeling. How can such information be obtained in real-world tasks?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper considers a recommendation setting where we observe the attributes of users, of items, and aspects of the recommendation itself such as a relevance, an explanation of the relevance, etc. The task is to use this information to predict the user's decision, whether or not they will interact with the item of interest. The decision could be binary, a rating, or something else.

The problem is to produce a predictor of user decisions that is invariant to changes in the distribution of the observed features (attributes, recommendation aspects, etc.). We expect such invariant predictors to achieve bounded risk when deployed to unseen distributions of data. This paper closely follows work on counterfactual (CF) invariance, a criteria based on counterfactual distributions that captures desirable behavior in predictors. The criteria can be *approximately* satisfied by regularizing predictors (where the regularization crucially depends on the causal model of the features and label).

To extend the ideas of invariance to user decision predictions, the paper makes a key insight: the decision is affected by the causal model by which a user reasons about features to make a decision. Thus, a key idea in the paper is to parameterize such causal models of user reasoning. After formalizing a notion of CF invariance in the user decisions setting, the paper uses the idea of boundedly rational agents to instantiate a causal model of user decision making. In such a model, they articulate how paths in the graph can encourage predictors to exploit non-causal (i.e., spurious) information that doesn't remain stable across distributions. Then, the paper draws upon results from the work on CF invariance to propose regularizers to discourage predictors from using such non-causal information. Since the regularization scheme depends on the causal model, the paper distinguishes two different causal models of users that lead to different regularizers. They empirically study CF invariant perdictors of user decisions with recommendation data and simulated outcomes, finding evidence of robustness to distribution changes and substantiating that indeed, it is important to match regularizers appropriately with causal models.
strengths_and_weaknesses: Strengths:
1. The paper takes a new approach to robust recommendations, making the novel connection that for invariant decision prediction, modeling the user decision making process is what is relevant. In this vein, using ideas from boundedly rational agents, the paper instantiates some causal models of users and articulates what paths in these models incentivize predictors to use spurious information. These connections are non-trivial to develop and can inspire extensions and new veins of work in the recommendation setting.
2. The empirical studies that involve recommender data with simulated outcomes can be considered another contribution in this paper. Such experimental evaluation could also provide new tasks and baselines for those looking to work on the problem of invariant recommendations.
3. The writing is generally clear and the authors do a good job or motivating technical ideas with examples and intuitions. The authors seem to have to covered related work well too.

Weaknesses:
1. In applying and extending ideas about CF invariance, there are some technical gaps and ambiguities throughout the paper that leave me doubting the soundness of some ideas. I'll enumerate:
+ the notion of CF invariance has always been a bit poorly formalized. Consider definition 1. Does the invariance have to hold for every possible unit (i.e., this is truly counterfactual and would be impossible to verify without assumptions on the full SCM)? Or, does $f_u(\cdot)$ have to be invariant for all values of x and r, i.e., $P(f_u(x(e), r(e) | X=x, R=r; \theta) = P(f_u(x(e'), r(e') | X=x, R=r; \theta) \forall x, r$? Or, do you want it to hold on average across interventions to $e$, i.e., $P(f_u(x(e), r(e); \theta) = P(f_u(x(e'), r(e'); \theta)$? Definition 1 isn't well defined enough to make these distinctions clear. Note that causal graphs alone could only identify versions 2 or 3 of this invariance.
+ In the same vein, if definition 1 is intended to be truly counterfactual (i.e., must be satisfied for every possible unit specified by the SCM), since the causal graph can never identify it anyway, why bother? Why not just pick a population-level (i.e., interventional as in version 2 or 3 above) version of invariance to satisfy?
+ The motivating causal model in figure 2 is a bit strange because in this model, the decision truly is causally affected by the environment. In this case, a predictor that satisfies CF invariance in Definition 1 would not capture the causal process of user decision making, which is often the underpinning of invariant causal learning. In fact, much of the work on invariant causal learning begins by assuming that the environment has no causal effect on the target $y$ because if it did, environment-invariance isn't intuitively a sensible ask to begin with. However, in this paper, the $e \rightarrow y$ motivation persists and is described as the main pathway by which predictors exploit spurious information. It's not non-causally spurious information in this case, though, so the motivation is quite confusing.
+ The notation in S3.1 and figure 2 tend to obfuscate the fact that what we want is for some of the *information* in $x$ and $r$ to be discarded. That is, we want to partition the information in those variables into a part that a desirable predictor $f_u(\cdot)$ can use and a part that $f_u(\cdot)$ should't use. For example, the work by Veitch et al. makes this information decomposition explicit, thereby allowing us to reason about what graphical relationships the "nice" part of $x, r$ have to other variables. By being more coarse-grained and only encoding $x$ and $r$, however, it's not easy to reason about what parts of the variables an invariant predictor ought to use, and what relationships the parts have to other variables in the graph.
+ In S3.2 on "rational users", the authors say that a rational user model is akin to assuming a causal graph in which there is no direct arrow from $e$ to $y$, which handicaps the ability to avoid spurious pathways. This statement is confusing. A causal model with no arrow from $e$ to $y$ seems like the best case scenario because any flexible predictor from $x$ and $r$ to $y$ would be able to capture the causal mechanisms by which $y$ is produced. Perhaps a fairer assessment here is that although rational users constitute a best case scenario, this is not a likely model of users.
+ In S3.3, it seems like if we had the function $v_u(x, \bar{x}, r)$ (which doesn't depend on the environment), then I could construct an invariant predictor. But, since $\bar{x}$ is latent, a predictor is incentivized to reconstruct $\bar{x}$ information using the observed variables $x$ and $r$. However, since $e$ leaves its imprint on $x$ and $r$ and also affects $\bar{x}$, a predictor might just memorize $e$-relevant information instead. However, the authors say that using $\bar{x}$ if it *was* observed would be bad because it could lead to overfitting -- this doesn't match my intuition because I'd have thought that observing $\bar{x}$ is a best-case scenario because then, all paths from $e$ to $y$ are blocked and any flexible predictor could capture the causal mechanisms governing $y$ from $x, \bar{x}$ and $r$.
+ In S4.1, the story of the anti-causal user doesn't make sense to me because $v_u(x, \bar{x}, r)$ sounds like reflects the user's perception of value -- how could the user's decision $y$ possibly causally precede the perception of value? Put differently, the notation may be using $v_u$ to conflate true and perceived value.
+ In Prop 1, $y$ and $e$ being confounded is not $e \rightarrow y$. The authors should clarify what they mean here.

2. The paper relegates an important limitation to just 1 sentence. In S4.1, after articulating how the choice of anti-causal versus causal matters for regularization, the authors state that ``this can be done using focused interventions (e.g., A/B testing), or for observational data, using simple conditional independence tests.'' Conducting A/B tests to determine a causal arrow direction for each user is impractical. It's important to spend more time, then, discussing the orientation of edges using CI tests and addressing all the limitations therein. Presumably the idea is to test for v-structures between $x$ and $r$, and $y$? This needs to be explained more directly since in general, causal structure learning won't orient all edges.

3. In a similar vein to above, the empirical studies focus on one or the other causal model (anti-causal vs causal), without addressing the challenge the authors themselves motivated, that typically, a training distribution consists of a mixture of these two causal model modalities. I would love to see this issue tackled more.

3. Section 4.2 could do with a figure that helps to illustrate the proposition and the reasoning about v-structures. This section is hard to follow especially because we don't have the notation to talk about the parts of information that a CF invariant predictor uses. That being said, this proposition seems to provide a novel insight beyond the existing work of CF invariance, so perhaps it would have been more interesting in this paper to dwell on the differences between skeptics and believers than between anti-causal and causal users.
questions: I described the main technical gaps I perceived above. I'll reiterate the points as clear questions for the authors here:

1. Can you clarify whether in definition 1? Does the invariance have to hold for every possible unit (i.e., this is truly counterfactual and would be impossible to verify without assumptions on the full SCM)? If so, why bother since the graph can't identify such a criteria anyway?

2. Can you clarify the motivating figure 2 model, and explain why the $e \rightarrow y$ link is being considered "spurious" or non-causal here? As explained in more detail above, invariant causal learning assumes that the environment has no causal effect on the target $y$ because if it did, environment-invariance isn't intuitively a sensible ask to begin with.

3. Can you clarify that my understanding of S3.3 was right, and clarify why learning a flexible predictor from $\bar{x}, x$ and $r$ (if $\bar{x}$ was observed) isn't the optimal thing to do, since we could just learning the (invariant) causal mechanisms directly?

4. Can you better justify how, in the anti-causal model, the user's decision $y$ could causally precede the perception of value? 

5. Can you articulate what CI tests you're envisioning to distinguishing between anti-causal and causal user models (presumably, testing to see if $y$ is a collider for $x$ and $r$)? 

6. Can you generally clarify proposition 2, perhaps with an example?
limitations: As discussed in the weaknesses above, I'd encourage the authors to spend more time discussing the challenges underpinning distinguishing between anti-causal and causal models. Moreover, both in exposition and empirically, it would be good to see more about how to practically deal with the issue of users in the data coming from a mix of these models.
review_title: null

Meta-review: metareview: This paper studies user-item relevance prediction and proposes a novel learning framework that is robust to distributional shifts in observed user-item attributes. All the reviewers appreciated the significance of the problem, the novelty of the solution, and the thorough empirical evaluation. The reviewers were confused by the exposition in some places, and the authors comprehensively addressed the questions during the feedback phase. Please include the extensive clarifying discussions with the reviewers in the revised paper, which will likely be of interest to the community. 
confidence: Certain
Decision: Accept