Teaching Invariance Using Privileged Mediation Information
Authors: Dylan Zapzalka
Maggie Makar
Venue: Accepted by TMLR ()

Abstract: The performance of deep neural networks often deteriorates in out-of-distribution settings due to relying on easy-to-learn but unreliable spurious associations known as shortcuts. Recent work attempting to mitigate shortcut learning relies on a priori knowledge of the shortcuts and invariance penalties, which are difficult to enforce in practice. To address these limitations, we study two causally-motivated methods that efficiently learn models that are invariant to shortcuts by leveraging privileged mediation information. We first adapt concept bottleneck models (CBMs) to incorporate mediators -- intermediate variables that lie on the causal path between input features and target labels -- resulting in a straightforward extension we call Mediator Bottleneck Models (MBMs). One drawback of this method is that it requires two potentially large models at inference time. To address this issue, we propose Teaching Invariance using Privileged Mediation Information (TIPMI), a novel approach which distills knowledge from a counterfactually invariant teacher trained using privileged mediation information to a student predictor that uses non-privileged, easy-to-collect features. We analyze the theoretical properties of both estimators, showing that they promote invariance to an unknown shortcut and can result in better finite-sample efficiency compared to commonly used regularization schemes. We empirically validate our theoretical findings by showing that TIPMI and MBM outperform several state-of-the-art methods on one language and two vision datasets.

Published in Transactions on Machine Learning Research (01/2026)
Teaching Invariance Using Privileged Mediation Information
Dylan Zapzalka
dylanz@umichh.com edu
Computer Science and Engineering Division University of Michigan
Maggie Makar
mmakar
wwwich. mmaakr@@michh.o
edu
Computer Science and Engineering Division
University of Michigan
Reviewed on OpenReview: https: // openreview. net/ form?? id= 82Lhuo32Kz
Abstract
The performance of deep neural networks often deteriorates in dusto--distiibuutin settings due relying on easy-to-learn but unreliable spurious associations known as shortcuts. Recent work attempting to mitigate shorteut learning relies on a priori knowledge of the shortcuts and invariance penalties, which are difficult to enforce in practice. To address these limitations, we study two causaly--ootvatte methods that efficiently learn models that are invariant to shortcuts by leveraging privileged mediation information. We first adapt concept bottleneck models (CBMs) to incorporate mediators intermediate variables that lie on the causal path between input features and target labels - resulting a straightforward extension we call Mediator Bottleneck Models (MBMs). One drawback of this mothod that it requires two potentially large models at inference time. To address this issue, propose Teaching Invariance using Privileged Mediation Information (TIPMT), novel approach which distills knowledge from counterfactually invariant teacher trained using privileged mediation information to student predictor that uses non-privileged, casy-to- collect features. We analyze the theoretical properties of both estimators, showing that they promote invariance to an unknown shorteut and can result in better finite-sample efficiency compared to commonly used regularization schemes. We empirically validate our theoretical findings by showing that TIPMI and MBM outperform several state-of the-rrt methods on one language and two vision datasets.
1 Introduction
Neural networks are often deployed on data that is different from the training data - phenomenon known as distribution shift. (Wiles et al.) 2021| Cui & Athey 2022). Many predictors have been shown to have brittle performance under distribution shifts (Kirichenko et al. 2023 Engstrom et al. 2019 llyas et al. 2019 Su et al. 2019). One reason for this is shortcut learning: when a predictor relies on easy-to-learn but inconsistent features in the training data. that are spuriously correlated with the target label (Geirhos ct al. 2020 Makar et al. 20222). Consoquently, the correlation between the shortcut and the main label changes, the model's 's performance declines significantly. Therefore, for model to be robust to distribution shifts, it should only rely on features that are both predictive of the label and invariant across the different distributions (Lu et al. 2021).
As a motivating example, consider predictor trained using X-ray images from a hospital to predict if a patient has knee osteoarthritis (KOA). An ideal predictor would only leverage medically relevant features that are invariant across settings, such as the appearance of the joints or joint space narrowing. However, X-rays often contain inconsistent spurious information that models exploit to make a prediction, such hospital-specific X-ray artifacts (Zech et al. 2018)

---

Published in Transactions on Machine Learning Research (01/2026)
To create models that exclusively learn invariant features, previous methods leverage additional information
available only at training time, known as prinileged informalion. Most of these approaches utilize privileged
shortcut information, typically in the form of labels representing potential shortcuts or environments that
models should be invariant to (Arjovsky et al. 2019 Makar ct al.) 2022 Veitch et al.) 2021 Sagawa et al. 2019
Goel et al. 2020j. Although effective in specific settings, these methods have two main limitations: (1) They
assume knowledge of all potential shortcuts and the availability of shortcut labels at training time. This is
limiting because it requires insight into spurious correlations inherent to a specific training dataset (Schrouff
et al. 2024). (2) They also rely on invariance penalties that are statistically inefficient. Specifically, they typically rely on nonparametric estimates of distances between distributions that are unreliable especially in
contexts where minibatches are used in the training process (Reddi et al. 2015 Ramdas et al.) 2015).
methods that replace the assumption of known shortcuts with the assumption of access to prinileged mediation information intermodiate variables that lic on the causal path between input features and target labels, which are available only at training time. We first adapt concept bottleneck models (CBMs) (Koh et al., 2020) to incorporate mediators resulting in straightforward extension we call Mediator Bottleneck Models (MBMs). Unlike CBM, which initially maps the non-privileged features to low-dimensional, intervenable high-level concepts before making a final prediction, MBM first predicts potentially unstructured, high-dimensional mediators that are then used to make a prediction. However, MBM requires two models for inference, which can be computationally and statistically inefficient if the mediator is high-dimensional or complex. To address this challenge, we propose another method, called Teaching Invariance using Privileged Mediation Information (TIPMT). To avoid the need for large model at inference time to predict mediators, TIPMT leverages knowledge distillation to enforce invariance to shortcut features. TIPMI first trains a teacher using mediators, followed by distillation step to train a student that uses normal features.
In this paper, we address these two limitations by exploring two cansally
Our methods address the first limitation of known shortcuts by replacing it with the assumption of known mediators, which is more suitable when expert knowledge of the causal mechanisms linking the label and the input is available. For instance, clinicians know that the effect of KOA on an X-ray is mediated by a narrowing of the knee joint, independent of where the dataset was collected from or the spurious correlations exprossed in the data. Moreover, we show that TIPMI and MBM work similarly to the invariance penalties while avoiding their inherent inefficiencies.
from previous work (Veitch et al. 2021 Makar et al. 2022
motivated methods that discourage Our contributions are summarized as follows: (1) We study two mausally-motivaeed
shortcut learning without requiring a priori knowledge of existing shortcuts. The first (MBM) is a simple extension of the well-known CBM while the second (TIPMI) is a novel approach that bypasses the need to deploy multiple models at inference, resulting lower inference cost. (2) We analyze the theoretical properties of the two methods, showing that they lead to more robust models and result in better finite-sample efficiency compared to commonly used regularization schemes. (3) We investigate the empirical performance of our methods using one language and two image datasets. Our results show that both methods lead to more robust and efficient models while avoiding the limitations of previous work. The code for our experiments is publicly available at httpps//giithb. httphhhwwwooyyyaaaaa//a/aaaaaaaa
2 Related Work
Learning Using Privileged Information and Knowledge Distillation. Our work builds upon the
learning using privileged information (LUPI) paradigm (Vapnik & Vashist 2009 Lopez-Paz et al. 2015
Lee et al. 2020 Chen et al., 2020: Loquercio al.] 2021 Gao et al.) [019 Chen & Bazzani 2020), which
utilizes privileged information derived from teacher to train student. Most LUPI work focuses on gains in efficiency improved interpretability, in contrast to main focus: robustness. Related to LUPI is work on knowledge distillation primarily for the purpose of model compression and explainable AI (Buciluă et al.
2006 Craven & Shavlik 1995] Hinton el al., 2015 Menon el al. 2021 Phuong & Lampert 2019 Yuan et al.,
2024). Most similar to our work is (Dao et al., 2020), which views knowledge distillation as a semiparametric inference problem to show how cross-fitting and loss correction can reduce the effects of teacher overfitting and underfitting. Other work has demonstrated that teacher models trained on augmented data can help student models become invariant to known shortcuts (Ojha et al., 2023). In contrast, our work trains the
2

---

Published in Transactions on Machine Learning Research (01/2026)
V(
V(2)
V(I)
V(2)
Y
M
X
Y
M
X
(b)
Figure 1: Causal DAGs describing the setting. Grey nodes observed at training time, while white nodes are never observed. The dashed lines denote non-causal spurious correlations. Each subfigure represents a possible source distribution with the same mediator. In (a) (() is shortcut and in (b) v is shortcut.
leacher using only mediators, which allows the student to become invariant to unknown shortcuts.
Shortcut learning. Previous work building invariant models explicitly relics on observing shortcut
environment variable that can be used to induce the desired invariances (Arjovsky el al. 2019 Krueger
ct al. 2021 Closest to our work is (Makar et al. 2022 Veitch et al.
2021], who present causally- motivated shorteut removal regularization scheme to encourage robustness to a single shortcut by leveraging
observed shortcut variables at training time. While (Zheng Makar 2022) encourages invariance to multiple
shortcuts, they still require these shortcuts to be observed training time. As discussed in the introduction, our work avoids the limitations inherent to knowledge or observability of the shortcut/ /enviromment label. Discouraging shortcut learning by using data augmentation has also been suggested (Hendrycks el. al., 2021 work if we have access to
Yin et al. 2019 Cubuk et al. 2018 Wang et al. 2022). This approach generator for known shorteut transformations, an assumption that we do not make.
Similar to our work are Concept. Bottleneck Models (CBMs) (Koh et al. 2020 Sheth & Ebrahimi Kahou 2024), which use high-level concepts as privileged information. The concepts are predicted by the first model as intermediate values, and second model uses the concepts make a final prediction. Unlike CBMs, TIPMI does not require explicitly modeling the mediator as function of the main features, allowing for the deployment of just single small model. This important. in settings where the mediators are high-dimensional. For instance, if the main feature is image and the mediator is lengthy text string, CBM would require a vision language model and a large language model to be deployed at inference time. In contrast, TIPMI would only require a single convolutional neural network for inference.
3 Preliminaries
3.1 Problem Setting
We consider supervised learning sctting where wish to learn some model to predict Y from X. Ad- ditionally, assume that have access to privileged mediation information M, variable that fully mediates the causal renahionship between y and X. In the KOA example, X and Y represent the X-ray and the presence/ presence/severity of KOA, while M could represent measurements of the knee joint. More broadly, in vision applications, segmentations or structured representations of the relevant object or region can serve as mediators that are not influenced by spurious shortcuts, e.g., background artifacts. We assume that M is available only training time, which happens in scttings where data derived from experts may be difficult to acquire at test time due limited resources. Throughout, we use uppercase letters to denote variables and lowercase to denote their value. Our training data D = {(x,, m., h is drawn from some source distribution Ps. where the labels Y are spuriously correlated with X through unknown potential shortcuts ((1), ((2) ,..., v(d), We denote V as a vector containing all potential shortcuts where v(d) is the d'th potential shorteut. Our goal is to create a model ʃ that invariant to any shortcut represented in V.
We assume that the training data generated accordance with the causal DAGs Figure | where each DAG represents possible distribution. For simplicity, we consider a setting where the potential shorteuts are two-dimensional, i.e., V [V(1), V(2).. The DAGs represent the anti-causal setting, where X is generated
3

---

Published in Transactions on Machine Learning Research (01/2026)
by v(), V(2) and the mediator M. Here, v(I) and V(2) are spuriously correlated but not causally related to Y, as indicated by the dashed lines in Figure 1 We also assume that M can be fully recovered from X. T.e., there exists some unknown deterministic function π such that M := (X)). This assumption follows prior work in robustness and causally motivated methods (Makar et al., 2022). where mediators are modeled as deterministic functions of the inputs to enable clear theoretical analysis. We also nole that this assumpttion is typically satisfied in settings where the input features are rich, unstructured data, such as image or text data.
In the KOA example, shortcuts could be created by spurious correlations between the KOA status and the X-ray image quality. This happens, for example, if patients with more advanced KOA are more likely to receive care in a skilled nursing facility with lower-quality X-rays prone to overexposure V, while healthier patients receive care in hospitals with high-quality X-rays. However, we assume that medically relevant information M remains constant across all settings, i.c., narrowing of the knee joint is associated with KOA across all medical facilities. Following this example, we define a family of potential target distributions P, where each larget distribution, denoted Pt, can be obtained by replacing P.(V(Y) with some target conditional distribution P((V))) E Pviy:
p (X VIPEMM (R((Y) E Pvly).
(1)
Equation highlights the stability of P. across all distributions in P, a desirable property that we will use to inform our approach. This stems from the fact that Y is the only causal parent of M, and, crucially, that V does not influence M. As a result, (M = P.(M) for all P. P.
We denote the unconfounded distribution as Pe E P, where p = MM ()PA(MM))
(Makar et al., 2022 Feder et al. 2024). Unless P. po, the Bayes optimal predictor will not be invariant to spurious correlations between X and Y (Makar are two open pathways between X and Y: the front door path, X - M + Y, and the back door path, X X+V V + Y. Our approach hinges on using M at training to discourage the use of the back door path. We note our setup is similar to the front-door criterion used in causal effect estimation (Bellemare et al. 2024). as both rely on mediators to soparate spurious from cansal associations. Unlike the front-door setting, wo focus on anti-causal prediction, aiming for models invariant to shortcuts rather than causal effect estimation.
al. 2022 Veitch al. 2021). This is because there
3.2 Counterfactual Invariance
Using potential outcomes notation Rubin 2005), we denote ( as the counterfactual value of X where some V is set to v, and all other variables remain the same. Since the association between V and Y is purely spurious, the prodictions made by a robust model should not change if the input to the model is X(v) or ((v), where v o v'. We say that models that satisfy this robustness property are counterfactually invariant. This definition extends the definition from (Veitch et al. 2021 to account for all potential shorteuts.
Definition 1. Let V (V(1), ((2) v(d] contain all possible shortcuts. A model f(X) is counterfactually invariant if ((X(v)) f()(')) for all v, o' in the sample space of V.
Under the assumptions encoded in the DAGs of Figure it follows that a model is counterfactually invariant if it only relies on the mediator M. We state this formally in Lemma 1, which is restated from Lemma 3.1 in Veitch et al. (2021).
Lemma 1. (Restated Lemma 3 From Veitch et al. 2021)) Let M be X X-messurable random variable such that, for all measurable functions f, we have that f is counterfactually incoriont if and only if f(X) is M -measuurale.
Intuitively, this follows as M is unaffected by any intervention on V. Temma motivates our approach for building invariant models by leveraging information encoded M.
4 Enforcing invariance using mediators
Recall that our primary goal to train model that is accurate and robust to distribution shifts across all P. E P. Here, we describe our approaches for achieving this goal.
4

---

Published in Transactions on Machine Learning Research (01/2026)
4.1 An inefficient solution: invariance through conditional independence
Lemma is helpful in identifying that the information encoded in M is unique in that it gives us an "invariance signature," However, it does not specify how to use M to generate counterfactually invariant prodictions since M is not available at test time. Similar to Veitch et al.] (2021), one way to achieve our goal would be to train a model f that has the same conditional independence as the unknown counterfactual invariant predictor, e.g., f(X) YIM. This solution has two advantages:
(1) enforces invariance to shortcuts without requiring access to M at test time, and (2) it does not require knowledge of the shortcuts. However, this approach inherits the second limitation of existing work, especially when M is high dimensional. In those settings, enforcing this signature requires a large sample size, which is inconsistent with settings where training is done in minibatches (Ramdas et al. 2015 Reddi et al. 2015).
Instead, we propose two other approaches that address the limitations of existing work. Both approaches leverage M to create counterfactually invariant models without requiring a priori knowledge of possible shortcuts and without relying on statistically inefficient conditional invariance tests.
4.2 Mediator Bottleneck Models
Based on Lemma a model that only a function of M should be counterfactually invariant. Thorefore, a valid strategy is learn a representation of X that only captures information relevant to M, which can be done by training a model to predict. M from X. Since we assume deterministic mapping between X and M exists, model will not need to rely on shortcuts to reliably predict M. The prediction from this first model provides us with a valid, shortcut-free reprosentation that we can then use to predict the target label. We call this approach the Mediator Bottleneck Model (MBM), an adaptation of CBM in which a mediator model fM: X + M maps non-privileged features to mediator variables, and a target model g : M + Y maps mediators to the final prediction, i.e., given x E X, prediction at inference proceeds as gM o f (xx) For the mediator and target model function classes FM and Gas, the MBM function class is defined as
FM = و o pM 1g E Gar).
In practice, we train MBMs sequentially, similar to CBMs as described in Koh et al. (202)): we first train the mediator model fM and then train the target model gMd using the mediator predictions of fM using the dataset DD While this training pipeline mirrors that of CBMs, there are both practical and conceptual distinc- tions. Notably, MBMs do not assume that intermediate variables are interpretable or directly intervenable. For example, in the KOA prediction task, CBM would require low-dimensional set of features, such as an indicator for joint space narrowing or bone spurs. By contrast, MBM can allow the intermediate (mediating) variables to be sogmentation of the knee X-ray, radiologist annotations, or measurements of all medically relevant anotomical structures in the knee. This relaxation allows MBMs to leverage high-dinennsooall unstructured mediating variables, in contrast to the dimedimnsii structured concepts typically used in CBMs. While this ficxibility broadens the applicability of the bottleneck framework, it also introduces ad- ditional computational challenges. We address these in the next section via a novel approach designed to efficiently handle such high-dimensional mediators during training and inference.
4.3 Teaching Invariance Using Privileged Mediation Information
We propose an approach that addresses the two limitations of work that relies on access to the shortcut label and provides an alternative to MBM that eschews the computational burden imposed by learning a mapping to a high dimensional mediator. Unlike MBM, TIPMI only requires one model at inference time, and does not involve explicitly modeling the potentially complex mapping from X to M.
TIPMI proceeds as follows. First, we train a teacher model gT [(M) that predicts the target label Y using the mediators M. By Lemma 1. (MM should be a counterfactually invariant teacher. Since M is not observed at test time, we use distillation to train a student fT(X) that mimics the behavior of the teacher gT (M) but uses only nri-privileged fcatures. This allows the student to adopt the same desirable conditional independences as the teacher without the statistical inefficiency caused by explicitly enforcing a conditional independence penalty. Furthermore, unlike MBM model, TIPMI does not require learning mapping
5

---

Published in Transactions on Machine Learning Research (01/2026)
from high-ddmennsioal imput X to high dimensional label M as sub-routine, and it only requires single model to be deployed at inference time. For the student and teacher model function Fr and Gr and a teacher model g" (M), we define the TIMPI function class as:
1 E((())) - (m.))²
FTIPMI(0") == arg min f'eFr n
In settings where 9T is highly expressive (e.g., when using deep neural networks), TIPMI might be prone to overfitting bias. This bias ariscs when data points are used to fit both the teacher model (a muisance model) and the student model (the larget model). When overfitling occurs, which significantly reduces distillation's 's benefits under TIPMI. This is because if the teacher produces predictions that are nearly identical to the ground-truth labels Y, the student effectively Iearns to map - Y directly, rather than inheriting the invariance properties of the teacher. This makes it similarly susceptible to shortcuts as standard training address this overfitting problem
setup that maps X Y. Inspired by semiparametric inference literature, by employing sample splitting. Specifically, split the data into random, mutually exclusive subsets. We use one subset to train the teacher to predict Y from M and use the mutually exclusive subsets to train the student to match the teacher's predictions from X. To avoid losing power due to sample-plitting, we apply cross-fitting, which repeats the above process for K-folds. The TIPMI algorithm is given below.
Teacher (Step 1) Train multiple teachers using cross-fitting:
1. Give N samples, form K disjoint index sets I1, I2, ... Is partitioning {1,...,N} each with |4x k
2. For each fold & (I,..,KK and suitable loss function la train teacher model on the combined as follows
folds Di {(m, 94) herg where I =
[ Σ
af = arg min g
l (m.).
D
For hs E (1,2,..., K}, use the teacher models to generate new datasets where d. 9% (m:):
D = ((x,, fie,
4. Combine the datascts generated by each teacher to create the final dataset for the student:
D
Student (Step 2)
Using the cross- mooss-iiting dataset Des, train the student using the mean squared error (MSE) so that the students predictions match the teachers:
f arg min fr
1
1f 0.)
n
(2)
T
5 Theory
In this section, we analyze the theoretical properties of MBM and TIPMI to highlight their robustness (section 5.1 as well as generalization properties (section 5.2
5.1
TIPMI and MBM robust
We start analysis of the robustness of TIPMI and MBM by studying their population-Ievel properties. We know that the Bayes optimal classifier for the teacher of (M) Ep- [Y| and Bayes optimal mediator model (X) = Pp M X), are the same for any source distribution Ps as implied by Figure 1 and the assumption that M is a deterministic function X. For TIPMI, the distillation process can be viewed as enforcing the signature f (X) = 96 (M). By "mimicking" the teacher oă (M), the student, (TXX), inherits the teacher's counterfactual invariance. For the MBM, the target model is counterfactually invariant as it only rely M to make predictions. We show these two properties in the next proposition.

---

Published in Transactions on Machine Learning Research (01/2026)
Proposition 1. Let Ps be any source distribution defined under the causal DAG in Figure 1 Also, let (M) = Ep, [Y]M], f/( (X) = Ep, (M)) f8dd (M) = [M,[M]]] and T6 (X) = Rp, ( 16 (X)). Then both fü (X) and M (X) are (i) counterfactually invariant, and (ii) optimal under p
See Appendix A for the proof for Proposition 1 This proof follows immediately from the d-separation properties of the causal DAG in Figure and the assumption of recoverability, iee, M == since M can be fully recovered from X, the Bayes optimal target model will be a function of only M and the Bayes optimal student can perfectly mimic the Bayes optimal teacher. Without further assumptions, a comterfactually invariant predictor cannot be optimal under any distribution other than p
= (()). Specifically,
5.2 Generalization properties of TIPMI and MBM
Having established their robustness, we now turn to an analysis of the generalization error of TIPMI and MBM when trained using data from any source distribution Pa and applied to any target distribution p E P. Our analysis follows arguments similar those presented by Makar et al. (2022). For simplicity, our theoretical analysis focuses on a setting with single unknown binary shorteut V and binary label Y, even though our proposed approach naturally extends to settings with multiple shortcuts. We leave formal analysis of multiple unknown shorteuts for future work. We focus on finding an upper bound for the generalization error between the risk over a larget distribution and the empirical risk over the source distribution, which can be decomposed as follows:
Rp - Rp. = (Rpt - Rp.)) + (Rps - RP..,
where Rp: is the population risk over p E P, with Rp = (()(X)) Y)] and Atp- 1 Ei= ((f(x.). is the empirical risk for some loss function l. The first term, Rp - Rp,, which we refer to as the structural risk gap, measures the difference in population risk between some target distribution pe and the source distribution ps used to train model f. The second term, Rp- Rp,, which we refer the learning gap, measures how well the model can generalize in-distribution. In the remainder of this section, will show that the TIPMI and MBM objectives control both terms.
5.2.1 Controlling the structural risk gap
Here, we study the structural risk gap, Rp - Rp.. Note that this term does not include any estimation error. It is simply the population-level difference between the risks when testing under different disributions.
Proposition 2. Let Q be the unit ball in a reproducing kernel Hilbert space with an T.-Tipehitz feature mapping used in the MMD. Also let the gM be K -Liichhitz, pa be some distribution that conforms to the causal DAGs in Pigure and pd E. Then
For TIPMI:
(M)) (Rp:(f") Rp-((")) 4L suspppp Ep;,l" (M) - -f Phy
For MBM: [Rpx ) - Rp (y M o ق S 4LK sup Ep |l| (X) M
Po
See Appendix A for the proof for Proposition 2 For TIPMI, Proposition shows that by encouraging the student predictor f to be as close as possible to the teacher g", the struetural risk gap becomes smaller. As for MBM, as long as the mediator model fM (X) is able to accurately predict M, the full model gM o f will have a low structural risk gap.
5.2.2 Controlling the learning gap
Next, we shift our focus to studying the learning gap, Rp- - Rp.. For our analysis, we compare the Rademacher complexities of the TIPMI student, MBM model, and normal L2 regularization function classes. We also show that TIPMI and MBM can lead to models with better finite-sample efficiency compared to L2 regularization. To simplify our exposition, we consider the special case of linear binary classifiers, where the models take the form of h(x) a(w x) and (xx) denotes the sigmoid function. In addition, assume that each model has L. regularization, such that ||l||| & A, e.g., FL2 GM Fr, where
7

---

Published in Transactions on Machine Learning Research (01/2026)
FL2 == =f oww (lw))).le /w/1 2 A). Extensions of our analysis to deeper neural networks can be done using tools presented in (Golowich et al.) 2018). We define the vector A that represents the average change in X given some intervention on any unknown potential shortcut V. Specifically, A is defined as follows:
Á = Ep [X (V 1)] Pp [X V 0)]
P(Y 1)(E[X V 1,Y 1] [| V 0, = 1])
P(Y 0) V 1, 0] E X = Y,Y 0))
Next, define the projection matrix II A(ATA)-IAT, which will project any vector onto the shortcut subspace, whereas (I III will project any vector onto the invariant mediator subspace where m_ I and mi := (T I).. This is as X is only a function of M and V when there is single shortcut, as implied by the DAGs in Figure [] Here, we can think of m_ as the part of æ that contains all irrelevant or spurious information and my as the part of a that contains mediator information. Similarly, we define wi == Il and wi == (7 M]... Our next. result shows that TIPMI reduces both the teacher and student's 's error by controlling the influence of variations orthogonal to the mediator.
Proposition 3. Let m. = Te and m = -I)IT),; and R(F) be the Rademacher complerity of some function space F. For training data D {aa,, m, 9;) where D ~ ps, also we have that supms |lmill2 I BL, and supmy ||miill I Bi. Finally, let 4L 4Lsupp,, Ep, (Mg) (X) S and 4LK supp:, Ep,,
- M Then
A B Bº Vn
A- B
B + T T
A Bi + + B
R(FI2) <
R(FTIPMi)
<
and R(FMBM) S
A
Vn
vn
Proposition Ba shows that the upper bound on FTIPMI tighter than that on FL2 when the student effectively matches the teacher's predictions through knowledge distillation. For MBM, the same Proposition indicates that the upper bound on FMBM is lower than that the on F1.2 when the mediator model generalizes well across all groups of ps. Moreover, the bound on FTIPMI will likely be tighter than that on FMBM when the mapping from X' to M is very complex. It is important to note, however, that unless Ps = p", the reduction in the student and MBM Rademacher complexity do not necessarily rcsult in improved in-distribution accuracy. This because both TIPMI and MBM are limited in their ability to leverage shortcut. features, i.e., they will exhibit bias unless ps = po, as they are limited in their use of the shortcuts to predict the label.
6 Experiments
We evaluate the proposed models, TIPMI and MBM, across diverse settings to assess their robustness to spurious correlations and sample efficiency. We struct ure our analysis around three main questions: (1) Do TIPMI and MBM remain robust in the presence of unknown or unobserved shortcuts? (2) How do they compare to basclines in terms of finite-sample efficiency? (3) When is TIPMI preferable to MBM?
6.1 Setup
Datasets. We evaluate both methods on three datasets, including vision and language tasks:
Waterbirds. A binary image classification task where the target label the type of bird (land versus water birds) (Sagawa et al. 2019). Shortcut labels spuriously correlated with the type of bird are background type (V(1) and image artifacts (V12)). The mediator M is a segmentation mask of the bird foreground.
Knee ostevarthritis. A binary classification task using knee X-rays to predict the presence of osteoarthritis. The label is spuriously correlated with rectangular artifacts (V(1), ((2) mimicking radiographic markers Zech et al. 2018). The mediator M consists of 16 joint space width measurements.
Food Review. text classification task constructed from the Amazon Food Reviews dataset (McAuley & Leskovec ) Inputs are review texts (X), labels are star ratings (Y), and review summarics serve as the mediator (M). Both shorteuts (V(1), v(2 are generated by perturbing high-frequency function words following Veitch al. (2021), ), e.g., replacing "a"" with "axxxxx" for ( and "to" with "toyyyyy" for ((2)
8

---

Published in Transactions on Machine Learning Research (01/2026)
Baselines. We evaluate TIPMI and MBM against range of established baselines, as well as ablation vari- ants designed to isolate the effects of cross-fitting: (1) Mediaato--Conditoona Penalization (MCPM): Enforces f(X) Y M during training via the kernel conditional independence test (KCIT) (Zhang et al, Conttittt-ooniiionnall Penalization (SCPM): Has oracle access to a (possibly incomplete) set
2012 (2)
of shortcuts V and enforces f(X) 1. V Y using kernel-based MMD (Grelton et al. 2012 Veitch el al. 2021 Makar et al. 2022). )- (3) Group Distributionally Robust Optimization (GDRO): Has oracle access to
a (possibly incomplete) set of shortcuts V and minimizes the worst-group loss, where the groups are defined by shorteut and label values (Sagawa et al. 2019). (4) Invariant Risk Minimization (IRM): Has oracle access to a (possibly incomplete) set of shortcuts V and learns predictors invariant across environments defined by shortcut and label values (Arjovsky et al. 2019). (5) Self-Distillation (SD): A teacher-student setup with identical architectures. The leacher is trained to predict the target label while the student is trained to match the teacher's predictions. (6) L2 Regularization (L2): Standard fz-penalized empirical risk minimization. (7) TIPMI-NC: TIPMI without cross-fitting and MBM-CF: MBM with cross-fitting.
Experiment sctup. In cach experiment, we simulate spurious correlations at training time and evaluate the performance on different out-of-listributtin shifts where the correlation between the label and shortcut varies. Performance is measured via the area under the ROC curve (AUROC). Models invariant to shortcuts should exhibit stable AUROC across test distributions, while shortcut-reliant models will degrade under distribution shift. Each experiment is repeated with 10 random seeds; error bars denote standard error.
For the waterbirds experiments, the TIPMI teacher, student, and MBM target models are ResNet-50 models (He et al. 2016) pretrained ImageNet-lk (Russakovsky et al. 2015), while the MBM mediator model is a U-Net with a ResNet-50 encoder. For KOA, the TIPMI student and MBM mediator models are ResNet-50 models, and the TIPMT teacher and MBM target models are single-layer neural networks with 1024 hidden units. For the food review experiments, the TIPMI teacher, student, and MBM target models are BERT tiny classifiers (Bhargava et al.] 2021). ; and the MBM mediator model T5-small model (Raffel et al. 2020 Additional training details are provided in the Appendix.
6.2 Robustness to unknown/unobservee shortcuts
We evaluate the robustness in settings with two spurious features, where only one is known/observed. This setting highlights the limitations of building robustness solely to known shorteuts. To evaluate robustness to unknown shortcuts, we restrict access to shortcut labels during training: methods relying known shortcuts (SCPM, GDRO, IRM) are given access to only v(1) but not v(2). Spurious correlations are introduced during training by setting P, (.(Vid) = 1 = 1) = P.(V(d) = o = 0) = 0.95 in the waterbirds dataset and (d(V(d) = 1|Y = 1) = Ps (v((d) = O|Y = 0) = 0.9 in the food review and KOA datasets, for each shortcut (I) and ((2), At test time we fix the distribution over the known shortcut P,(V(() = 1|Y = 1) =
P((V(1) o|Y = 0) = 0.5 and vary () to assess sensitivity to the unobserved shortcut.
Baselines rely on shortcuts. First, we compare TIPMI and MBM to other baselines in Figure 2 Follow- ing the ablation study, we exclude TIPMI-NC and MBM-CF. Across all datasets, TIPMI and MBM exhibit strong robustness to the unobserved shorteut, maintaining high and stable AUROC. In contrast, methods requiring known shorteuts (SCPM, GDRO, and IRM) degrade the shortcut-label correlation deviates from training. Notably, on KOA and food review, SCPM underperforms even standard 122-reullaiied FRM, in- dicating that reliance on partial shorteut information is insufficient for robustness. MCPM also performs worse than TIPMI and MBM, despite using mediators to enforce invariance, highlighting the inelficency of the conditional independence penalty. On waterbirds and KOA, TIPMI and MBM perform similarly and are more robust than the basclines. MBM performs better than TIPMI food review, as removing the shortcuu when mapping X to M is easy (e.g,, summarizing text), since the complexity is offset due to large amounts of pretraining data used by the mediator model (e.g., T5 model). SD results are shown only in the appendix, as they mirror L2, and IRM is excluded due to poor performance waterbirds and KOA.
Ablation study. Next, we perform an ablation study to determine the benefits of cross-fitting for both TIPMI and MBM. Figure shows results for the waterbirds, KOA, and food review datasets. TIPMI-NC in the waterbirds and the food review datasets perform significantly worse than TIPMT, highlighting the need
9

---

Published in Transactions on Machine Learning Research (01/2026)
Waterbirds
KOA
Food Review
1.00
1.0
0.8
0.96
0.8
AUROC
0.6
0.6
0.92
0.4
0.90
0.88
(Dus)
MBM [Oans
(Ours)
(Oun)
TIPMI (Ous)
L2
MCPM
L2
MCPM
MCPM
GDRD
SCPM
GDRO
SCPM
0.8
GDRD
SCPM
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
P(V()
1 = 1) P(V() = 0|Y = 0) at test time
Figure 2: x-axis shows PV at test time. y-axis shows the AUROC over the test data. Dashed vertical line shows P() at training time. (Left) waterbirds and (Middle) KOA: MBM and TIPMI outperform all baselines. (Right) food review: MBM is the most robust to shortcuts, followed closely by TIPMI.
Waterbirds
KOA
Food Review
0.95
0.9825
0.9800
AUROC
0.98
0.85
0.9750
0.9725
0.9700
(Dum)
[0]m]
(Dun)
(Dun)
TIPMI-NC (Oun)
MBM (Oun)
0.9675
TIPMI-IC (Oun)
MBM (O.n)
0.70
TIPMI-NC (Ourg
MBM (Dun)
6.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.6
0.8
1.0
P(Ve)
1/Y = P(V 0|Y 0) test time
Figure 3: x-axis shows PY) at test time. y-axis shows the AUROC over the test data. Dashed vertical line shows P(V(2)(Y) at training time. (Left) waterbirds and (Right) food review: cross-fitting significantly improves TIPMI, but not MBM. (Middle) KOA: cross-fitting has no significant impact on TIPMI or MBM.
for pross-fitting when the teacher large and flexible model, such BERT and ResNet-50 (He al. 2016 Devlin 2018). Tn contrast. there is no difference between TIPMT and TIPMI-NC for the KOÅ dataset. This is expected the teacher used in these experiments is small, single-layer neural network, which is less prone to overfitting than the large networks used in the other experiments. In contrast to TIPMI, tross-fitting has little effect on the robustness of MBM. As shown in Proposition 2 this is because the robustness of MBM is only determined by how well the modiator model can map X to M, which is not helped by cross-fitting.
6.3 Robustness and accuracy due to finite-sample efficiency
We next highlight how the efficiency of our approaches leads to improved robustness and accuracy.
Efficiency Icads to better robustncss. We first examine how efficiently enforcing counterfactual invari- ance improves robustness. We study setting similar to that of section 6.2 but here we focus robustness to single known shortcut. In this setting, methods that rely knowledge of the shorteut and our proposed approaches are in principle- enforcing the same desired invariance with respect to the shorteut. We sim- ulate datasets containing one shortcut (1) = 1|Y = 1) P.((()) = o|Y = 0) 0.95 in the waterbirds dataset and (1) = 1jy = 1) = P.(V()) = ojY = 0) = 0.9 in the food review and KOA datasets at training. SCPM, GDRO, and IRM have access to y() at training. We vary P(V() at test time.
Results are shown in Figure Although SCPM and GDRO are explicitly enforcing invariance the relevant shortcut, they less robust. than TIPMI and MBM across all datasets. Similar to section 6.2 MCPM
10

---

Published in Transactions on Machine Learning Research (01/2026)
Waterbirds
KOA
Food Review
1.00
1.0
0.99
0.8
0.98
AUROC
0.97
0.96
0.6
0.4
0.95
0.5
(Ous)
MBM [Ouns
(Durs
(Oun)
TIPMI (Our)
0.94
GDRD
L2
MCPM
SCPM
04
L2
GDRO
MCPM
SCPM
GDRD
MCPM
5CPM
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
P(V() 1 = 1) P(V() = 0|Y = 0) at test time
Figure 4: r-axis shows P(V test time. y-axis shows the AUROC the test data. Dashed vertical line shows P(V() Y) at training time. (Left) Waterbirds and (Middle) KOA: MBM and TIPMI outperform all baselines. (Right) Food review: MBM is most robust, TIPMI is tied for second best.
delivers worse robustness despite enforcing a similar invariance penalty to TIPMI and MBM. MBM performs slightly better than TIPMI, which, at worst, performs comparably to SCPM in the food review task. Here, the superior performance of TIPMI and MBM arises because they are enforcing counterfactual invariance without relying on expensive penalty terms, e.g., MCPM and SCPM rely on data-intensive nonparametric conditional independence penalties to enforce invariance through regularization. MBM performs better than
TIPMI in food review and waterbirds, for the same resons discussed in section 6.2
Table 1: In distribution results.
Data
Waterbirds
KOA
Food Review
Model
TIPMI
0.992 (0.002)
0.980 (0.004)
0.896 (0.007)
L2
0.987 (0.004)
0.946 (0.010)
0.907 (0.006)
GDRO
0.986 (0.006)
0.948 (0.008)
0.903 (0.007)
MCPM
0.985 (0.004)
0.948 (0.008)
0.899 (0.007)
MBM
0.993 (0.003)
0.980 (0.006)
0.857 (0.005)
Efficiency leads to better accuracy. To isolate the gains in finite-sample efficiency introduced by TIPMI and MBM, we run "in-distribution" experiment where the training and test data are drawn from the unconfounded distribution, pa, where v()) is simulated such that v() 1 Y. The results, summarized in table D] show that TIPMI and MBM achiove the best performance on waterbirds and KOA. This is likely because (1) the mediator provides a strong signal for the label, and (2) the mapping X + M is simplified by pre-training. Specifically, both waterbirds and KOA, the models are ResNet-50 architectures pre-trained on ImageNet, which offsets the complexity of the mapping. For the food review dataset, MBM is the only model whose performance is statistically distinguishable from the others.
6.4 TIPMI vs MBM
Previous results show that MBM and TIPMI are both more robust and efficient than baselines. The results also seem to imply that MBM performs as well as or better than TIPML We examine this relationship here. MBM has higher inference-time complexity. First, we highlight that MBM's performance comes at the cost of higher inference-time complexity, as il requires deploying two, potentially large models, whereas TIPMI only requires a single model. Therefore, when the MBM target and TIPMI student model architec- tures are the same (as for the waterbirds and food review experiments), MBM will require more parameters at inference due to the additional mediator model. And when the mapping between X and M complex, the number of parameters
MBMs parameter count. is substantially higher compared to TIPMI.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table Number of parameters (in millions) used for TIPMI and MBM at inference time
Data
WB
KOA
FR
Model
TIPMI
23.51
23.51
4.39
MBM
56.03
23.56
64.89
for the deployed model (i.e., at inference time) in millions: on the waterbirds (WB) dataset, MBM uses over twice as many parameters as TIPMI, and on the food review (FR) dataset, more than len times as many.
MBM performs worse the mapping from X to M is complex. This additional complexity comes al the cost of worse finile-aample elficiency, as is shown by the food review results in table I This happens the mediator model is unable to fully recover M from X, leaving the target model with less reliable signal to make prediction.
We investigate this phenomenon using simulated data where we can control the function complexity. We simulate an anti-causal setting where the complexity of the mapping between X and M can vary. To construct this setting, we designate D dimensions of X that are functions of M but contain no information about the target Y. We refer these as redundant dimensions. Increasing D makes the mapping X M more complex, while leaving the effective mapping X + unchanged. Our data generation process is:
V ~ Normal(O, 1).
y Normal(0, 1), yV =v:=vvV, : V +VV,
: Mc ~ By + NNormal(0,1) 1), M Normal ID), M = [MC, M").
x ~ Br x" BvV & R Normal(O, Ip), X [X x
where Y Mc,xe denote "clean" components of Y, M, and X unaffected by the shorteut V, while yv captures the V's contribution to Y. The redundant components M and X are unrelated to the robust component the target label y with X also influenced by V. Random weights Sy, 8M, and By specify mappings from V + YV, MC X and yC - M This setup ensures the data respects the causal DAG: V affects Y and X but not M, while decoupling the complexity of X Y from that of X M.
All models are trained with fifth-degree polynomial kernel and L2 regularization, making the true mod- els attainable (the most complex mapping, MBM's X + M, is a third-degree polynomial). Hyperparam- eters are chosen via five-fold cross-validation. Each experiment is repeated 100 times with independent draws of V, X, M, Y, 3r, ßy; and BM. For out-of- distribution evaluation, we flip the shorteut effect by setting By = -y. Fach simulation uses 500 train- ing and 100 test samples. We report the median and interquartile range (IQR) of the root mean squared er- ror (RMSE) across 100 runs. Additional details are in Appendix D
RMSE vs. Mediator Dimension Size (Median with IQR Bands)
Root Mean Squared Error (RMSE)
TIPMI (Ours)
MBM (Ours)
10-
2.5
2.0
1.5
1.0
10
50
Mediator dimension size
Figure shows MBM's error rises sharply the num- ber of redundant mediator dimensions increases, while TIPMI's error remains stable. This is because MBM's mediator model is trained to predict all dimensions of M, including those irrelevant for predicting which introduces additional noise that reduces the quality of the signal for target model to prodict the label. TIPMI bypasses this: the student model only needs to learn a mapping from X to the teacher's predictions, without modeling the redundant. mediator dimensions.
Figure 5: r-axis is mediator dimension size. y-axis test RMSE. MBM's performance deteriorates with larger dimensions whercas TIPMI and L.2 are stable.
12

---

Published in Transactions on Machine Learning Research (01/2026)
7 Conclusion
In this work, presented TIPMI and MBM, methods that use privileged mediation information to enforce counterfactual invariance and improve finite-sample efficiency. TIPMI consists of teacher trained using privileged mediation information and a student that learns through distillation. In contrast, MBM first predicts the mediator before using it to make a final prediction. We showed theoretically and empirically that TIPMI and MBM promote invariance to shortcuts better than methods that use privileged shorteut information and that they increase sinite-sample efficiency.
Limitations and Future Work. A limitation of our methods is the assumption that shortcuts do not causally affect the mediator. In this case, TIPMI and MBM can still achieve invariance to other shortcuts that don't affect the mediator, but they will not enforce counterfactual invariance. Future work could explore uncertainly analysis to assess performance when part of the mediator causally affected by a shortcut. Additionally, our theoretical analysis for the generalization properties of TIPMI and MBM constrained to the setting of single binary shortcut. Future work could generalize this theory to account for multiple unknown shorteuts.
8 Acknowledgements and Disclosure of Funding
We are thankful for the anonymous reviewers and for feedback from Alex D'Amour. This material is based upon work supported by the National Science Foundation under Grants No. 2337529 and 2153083. Any opinions, findings, and conclusions recommendations expressed in this material those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
References
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint ar arXie:1907.02803 2019.
Marc Bellemare, Jeffrey R Bloem, and Noah Wexler. The paper of how: Estimating treatment effects using the front-door criterion. Oxford Bulletin of Economics and Statistics, 86(4):951-993, 2024.
Prajjwal Bhargava, Alcksandr Drozd, and Anna Rogers. Generalization in NLI: Ways (not) to go beyond simple heuristics, 2021.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535 541, 2006.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference Robot Learning, pp. 66 75. PMLR, 2020.
Yanbei Chen and Loris Bazzani. Learning joint visual semantic matching embeddings for language-guided retrieval. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXII 16, pp. 136-152. Springer, 2020.
Mark Craven and Jude Shavlik. Extracting tree-structured representations of trained networks. Advances in neural information processing systems, 8, 1995.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmenntatio policies from data. arXie preprint or 180: 09501, 2018.
Peng Cui and Susan Athey. Stable lcarning establishes some common ground between causal inference and machine learning. Nature Machine Intelligence, 4(2):110 115, 2022.
Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowlodge distillation as semiparametric inference. Tn International Conference on Learning Representations, 2020.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint Xrx: 04805, 2018.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In International conference on machine learning, pp. 1802-1811 PMT.R,
2019.
Amir Feder, Yoav Wald, Claudia Shi, Suchi Saria, and David Blei. Causal-structure driven augmentations for text OOD generalization. Advances in Newral Information Processing Systems, 36, 2024.
Zhifan Gao, Jonathan Chung, Mohamed Abdelrazek, Stephanie Leung, William Kongto Hau, Zhanchao Xian, Heye Zhang, and Shuo Li. Privileged modality distillation for vessel border detection in intracoronary imaging. IEEE transactions on medical imaging, 39(5):15241154, 2019.
Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic. arXiv preprinl orXic:1707.07000 2017.
Robert Geirhos, Jörn-Henrik Jacobsen, Clandio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence,
2020.
Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré. Model patching: Closing the subgroup performance gap with data augmentation. arXiv preprint arXim:2008.0677 06777, 2020.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural net- works. In Conference On Learning Theory, pp. 297-2022000 299. PMLR, 2018.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-7735 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770- 770-778, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of- distribution generalization. In Proceedings of the IEEE/CVF international conference computer vision, pp. 8340-83499, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0m 02531, 2015.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they features. Advances in neural information processing systems, 32, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. anXio preprint arXiv:1412.690 2014.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations, 2023. URL https://arxiv wwtpesiiioor///baee org/abbs/204..09333
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338 5348. PMLR, 2020.
Mark Kohn, Adam A Sassoon, and Navin D Fernando. Classifications in brief: Kellgren-lawrence classifi- cation of osteoarthritis. Clivical Orthopaedics and Related Researchō, 474:1886 1893, 2016.
David Kruegor, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. distrf-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815-5820 PMTRR, 2021.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged information for efficient image super-resolution In Compuler UK, August 23 28, 2020, Proceedings, Part XXIV 16, pp. 465 482. Springer, 2020.
2020: 16th European Conference, Glasgow,
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privi- leged information. arxiv preprint arXiv:1511.06644 2015.
Antonio Loquercio, Elia Kaufmann, René Ranftl, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Learning high-speed flight in the wild. Science Robotics, 6(59))cab,3510, 2021.
Chaochao Lu, Yuhuai Wu, José Miguel Hemánndez-Lobato, and Bernhard Schölkopf. Invariant causal repre- sentation learning for out-of-distribution generalization. In International Conference on Learning Repre- sentations, 2021.
Maggie Makar, Ben Packer, Dan Moldovan, Davis Blalock, Yoni Halpern, and Alexander D'Amour. Causally motivated shortcut removal using auxiliary labels. In International Conference on Artificial Intelligence and Statistics, pp. 739-766. PMLR, 2022.
Julian John McAuley and Jure Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd international conference on World Wide Web, pp. 897 908, 2013.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyoon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In International Conference on Machine Learning, pp. 7632 7642. PMLR, 2021.
Michael Nevitt, David Felson, and Gayle Lester. The osteoarthritis initiative. Protocol for the cohort study, 1:2, 2006.
Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. What knowledge gets distilled knowledge distillation? Advances in Neural Information Processing Systems, 16:11037-11048, 2023.
Simon Olsson, Ehsan Akbarian, Anna Lind, Ali Sharif Razavian, and Max Gordon. Automating classification osteoarthritis according to kellgren-lawrence in the knee using deep learning in an unfiltered adult population. BMC musculoskeletal disorders, 22:1 8, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International conference on machine learning, pp.
PMLR, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. /ttp:////// org/ http://wwww.rr.rppps//////////// http//paprr//2222004444/. html
Aaditya Ramdas, Sashank Jakkam Reddi, Barnabás Póczos, Aarti Singh, and Larry Wasserman. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. Tn Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Sashank Reddi, Aaditya Ramdas, Barnabás Póczos, Aarti Singh, and Larry Wasserman. On the high dimensional power of a lincar-time two sample test under mean-shift alternatives. In Artificial Intelligence and Statistics, pp. 772-780. PMLR, 2015.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322 331,, 2005.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2555 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for groupshiffs:: On the importance of regularization for worst-case generalization. arXiv preprinl
arXiv:1911.0879, 2019.
Jessica Schrouff, Alexis Bellot, Amal Kannen-Triki, Alan Malek, Isabela Albuquerque, Arthur Gretton, Alexander D' D'Amour, and Silvia Chiappa. Mind the graph when balancing data for fairness or robustness. arXiv preprinl Xro:22400. 17433, 2024.
Tvaxi Sheth and Samira Ebrahimi Kahou. Auxiliary losses for learning generalizable concept-based models. Advances in Newral Information Processing Systems, 36, 2024.
Jiawci Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Compulation, 23(5):828 841, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. Xiv preprint wrXiv2337.099883 2023.
Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Newral networks, 22(5-6):(55))) (11566))444-577, 2009.
Victor Veitch, Alexander D' D'Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations: Why and how to pass stress tests, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds- 200-2011 dataset. 2011.
Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. distriiiitttituttin generalization with causal invariant transformations. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 375 385, 2022.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvisc-Rebufi, Ira Ktena, Krishnamurthy Dvijotham, and Taylan Cemgil. A fine-grained analysis on distribution shift. arXie preprinl or arXiv:21 10. 1 13228, 2021.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems, 32, 2019.
Hua Yuan, Yu Shi, Ning Xu, Xu Yang, Xin Geng, and Yong Rui. Learning from biasod soft labels. Advances in Neural Information Processing Systems, 36, 2024.
John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of deep learning model to detect pneumonia in chest radiographs: a cross-scctional study. PLoS medicine, (6(11):c0002683 2018.
Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. arXiv preprint arXit:1202.8875. 2012.
Jiayun Zheng and Maggie Makar, Causally motivated multi-shorteut identification and removal. Advances in Neural Information Processing Systems, 35:12800-12812, 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6): 2017.
16

---

Published in Transactions on Machine Learning Research (01/2026)
A Section 5.1 Proofs
For this proof, we show that TIPMI and MBM yield models that are asymptotically counterfactually invariant and optimal under the unconfounded distribution p This follows directly from the d-separation properties of the DAG in Figure 1 and the assumption that M can be fully recovered from X. Without further assumptions, MBM and TIPMI are only optimal under po, as they camnot exploit shortcut features that may be informative in other distributions.
Proposition A.1 (Restated Proposition 1 Let Ps be any source distribution defined under the causal DAG in Pigure 1 Also, let g6 (M) = Ep. [|M], , So (X) = Ep,ls) (M) 16
= XM],, and M (X) = Ep, [Y 3o (X)]. Then both fo (X) and gü (X) are (i) counterfactually invariant, and (ti) optimal under Po.
M
Proof. Let X, K,M.V Ps for some P.. We first show that the TIPMI student model will be counterfactually invariant. Under the assumption that there exists some deterministic function л such that M x =(X), we know that all information in M can be recovered from X. Therefore, since 9ō (M) 1 X M under Ps. Ep, Ioo (M) Thus, fo (X) can be written as (with abuse of notation) as J (M). Since S is only a ion of M, fo must be counterfactually invariant by Lemma 3.1 in (Veitch et al.) 2021 This can be seen as an intervention on V does not affect M. Also, fø is optimal under p Similarly, we can show that the MBM target model will be counterfactually invariant, as 90 (X) Ep. [ (X)] EY,[Y[]]
Ep. [ Ep, 1 (M) xx()
Both TIPMI student and MBM target model are also optimal under p The target model is optimal under pa PP, [M] [p-[Y|]]] Ep [Y|X] since M d-separates Y from X. Furthermore, the TIPMI student is optimal under p as (X) Ep, 1oo (M) = Ep, [90 (M)JM = (MM)
B Section 5.2.2 Proofs
B.1 Samite-Sample Results
To understand how TIPMI and MBM induce invariance, we build on prior work that enforces conditional independences satisfied by an unknown counterfactually invariant, predictor Veitch et al. (2021), e.g., / X
V Y. To enforce this property, they the maximum mcan discrepancy (MMD) Gretton et al. (012
For Priyy P([(X)|Y y, = v), the corresponding regularization term, called CMMD, is defined
CMMD(P.f) = P(y 1) Prio1)
+ P(y = 0/MMD((Ppoo.Pj1)
We first establish in Lemma. B.1 that minimizing CMMD(P", f) reduce the structural risk gap. Lemma B.2 then shows that both TIPMI and MBM can decrease CMMD(P*, Finally, we apply both Lemma B.1
and Lemma B.2 to show that TIMPI and MBM can reduce the structural risk gap, as shown in Proposition
Lemma B.1. Let pa be some distribution that conforms to the causal DAG Figure 1, P E EP, and & E Ω where Q is reproducing kernel Hilbert space used in the MMD. Then for some function f(X),
|Rp (f) - Rp- (f) CMMD(P",
Proof. The structure of this proof follows that of Proposition A2 from Makar et al, 2022). For this proof, define Re := Er (e([(X)) V)IY y, V.V u) and Re : Ep- (e(((( Y) y, m).
17

---

Published in Transactions on Machine Learning Research (01/2026)
For any distribution P E P we can rewrite the risk as follows:
Rp = =EM(((((X)))))
(= = yy)
y
X -,, = = = = ( = =)
+
-j = 0|Y y) =))
Σ (V = y) + Riy pl (V = OJY = y) (Y = v)
w)) 1|Y = 3) + Riy P'( = 0|Y = y)]
Therefore, if we take the difference wetween both risks, we get the following result:
RRp - Rp = Σ P'(Y = 5) P'(V = 1|l = y) + Rb Pt(V = 0|Y = y)]
Σ Po(Y = v)(Riy PV = 1|1 = y) + Rög pa = 0|Y = y)ll
PP Di 1|Y y) -
PO (V = 1|2 y)
+ Ri P'(V = 02Y = -y) - Ruy. PV( = 0|Y =]))]
Σ po(Y v)((P'(V = y)
=
Pa(V 1|| = R)
- + (P'(V = -0-- = = - (V = 0Y = :i
= Σ P- = -v)(P'(VV = 1| = y) - P"(V = = = R)
+ +(1 - P'(V 1| = 11 P*(V = 1 Riyl
Pa( [V = = y) PPV = 1|Y v))
(P'(V = -1|Y- = - PV = 1 = :
Σ P'(Y- = =y)(P'(V = 1 =) - PV(V = 11 =)) - RoyY
"
Σ PO(Y = =)) - Rigg)
IP 1)(iii - -R2,) - ps 0) RRo)) < PP = 1)MMD(P? P/11) + Pt(Y = -O)MMD(P P1
CMMD(P*.
Lemma B.2. Let Q be the unit ball in a reproducing kernel Hilbert space with on L-Lipchitz feature mapping used in the MMD. Also let the M be K-Lipchitz, pa be some distribution that conforms to the causal DAG in Figure 1, and P E P. Then
CMMD(P", ff) < 4L Epp (0 - (x)i]
Ps
18

---

Published in Transactions on Machine Learning Research (01/2026)
and
CMMD(P" ghd PM S 4LK sup Ep; (x) - MM
Pcv
Proof. We begin the proof by showing the first bound for the TIPMI student model. The proof follows from the fact that MMD(Pg gn111 pe, (0010)) MMD(Pp g" 110 pa g0)0)) O, as the teacher g² is invariant across all distributions.
CMMD(P', ,8) P(y - 1)MMD( fr 11: Pfr 01) + P3 = OjMMD(Pj (P 17110- Pf+100)
S <MMD(P)r Pi11)) MMD (PF pi1 Pfr)0o)
(PM Pfr11) MMD( ( 11 Por111)]
+ [MMD(P}=110 P7-100) MMD(PG/110 Po
(The equality holds g is only function of M, which is invariant across all distributions. We now obtain the final bounds by the L-Lipchitz properties of the feature mapping and the definition of the MMD.)
CMMD(P (*) s SuppEpp (x))
Epg (w (x)) Ppp (wg (M)) + Epg, (M)))
wen
sup wEi
(Epp [ (x)) Erg w (x)) - Erjo (o + Braww( (M)))
sup(Ep (Epp [w (X)) w6g
(E (o(g(MM) (x(((x)))
well
we
sup (Bpy (X)) ( sup (p [w (M)) - w(F((x))))
wen
well
sup (w, Epr (b) / (X)) "6'2 (M)))) sup (wi, Epg (6 (0(gg(()))) (M)) (x))))
wen
sup Epr [6( fd (X)) o(g (M)))) sup (w. Ep j ( (M)) (x))))))))
weR
wen
sup([|=|| Epg [0( (X)) - (M))]]) + sup([1=)
Epg [00 (M)) - (x)]]]
sup (Iw]] [Pi 10( TT (X)) - (o(g"(M)) sup ([W|] |P 10(g" (M)) (xf)
wEn
wel
Ern (x))((X) 009" (M)) + Erg ((T))(M) +(F
(M))
BPy [0(T((X) -6(0 (M))] + EEp (( (M)) (UT(()))]
Epp [XX)"(()--))) -9"(MMI + Erg [0]gg(M] (MM) (x)II
+ Epp. [L (X) -MM( + EEp (MM) - (x)Il
< sup Eps (MM) - (X](x)]]
Pa,
Following similar approach, can obtain bound for MBM. The proof follows from the fact that MMD(Ps ( 11 P 101) = MMD(Ps P ged no ae oo): as the target model gM is invariant across all distributions.
CMMD(P' MM fbl) Ply 1)MMD P
9 Mo M Ps PoMo
P 0)MMD (Pr, o MM| I10 Py (o
< MMD (P.
g GM P 101) + MMD (Po M Mo 110: P2 Po
[MMD (Pa 111 PgM ( MMD p
pe [1))
96"
+ [MMD MM Mo Po P g0 (00)
MMD( SMM p00
ps |00)
(The equality holds as M is only a function of M, which is invariant across all distributions. We can now obtain the final bounds by the T-piicchizz and K-Tápehitz properties of the feature mapping and the target
19

---

Published in Transactions on Machine Learning Research (01/2026)
model.)
sup (uppErp (F Brg (0 (fg (X)))))) Epp 100" (M)) Erg ( (M))))
wEn
sup w ( (X))) Erg w (0 (I*(( Eri [ M (M)) + Epg ( (M))))
well
sup (Err w M (f (x))) Beg ( (M (M))l) sup (Brg w (M)) - Erg (w Mf (f (x
wei
weR
sup Pr w6( (1 (d (X))) Erg ( 01 (M)))) + su
00 ( (M))) Erg. (w(y) ( (X))))
wenl
wen
I sup (w, Epp Lo (f (f (X))) 0(8 (M)))) swp
B pM [0 M (M)) - 6(
wen
WES
sup (w, E (0(g (f (X))) "600 (M)))) sup
(w, E 60
jo (M)) "0g (f (x
WEN
FEft
sup ([w]l
sup [pp [6(g (X))) "60" (M))
||||||
Epg [0[g M (M)) "(g (s
WEst
wen
Bp 16 (X))
(X))) 60 (M)) + (|w|]
Epg 10(0 (M)) - و0g M (f¹r (x))
WEll
wenl
||Er (1 (X))) 600" (M))]] + Erg (MM)((M) :6000 (s
Err (6(5 M (5al (x))) (M)) + || 16(g M (M)) - 20a (x))))
< Epn 1L |1g (X))
11p 16 (M) (M) (M((
Epp [L |1g (fM (X)) - (M)) +1 [L ولا (M) gMd (f
< sup Ep;,
1199 M (f (X)) g0(((n (M)ll)
Pix
< 4LK sup Ep M (X) MM
Proposition B.2 (Restated Proposition 2). Let Q be the wnit ball in a reproducing kernel Hilbert space with L- Lipphhtt feature mapping used in the MMD. Also let the gMr be K K-Lápchitz, p be some distribution that conforms to the causal DAG in Figure 1, and pt EP. Then
For TIPMI:
[Rp:(J)) Rp- S sTsup Ep., (ggl[(g])] (X)( P2
For MBM:
MMM fdl) (g fad
) < 4LK Ep. f|| (X) -M|
Pi
Proof. This proof follows immediately from Lemma B.1 and Lemma B.2
0
C Section 5.2.3 Proofs
Lemma C.3. For any P E P,
CMMD(P.F f)
|wii S
|||||
Proof. The structure of this proof follows similar that of Proposition in Makar al. 2022 For E Ω where we let w(x) wT x.
20

---

Published in Transactions on Machine Learning Research (01/2026)
CMMD(P, 2 [ 1,y = 1] [[ = 0,y = PY = 1)
+ [a) == 1, = 0] [x) 1 0, = (
|R/ a/v 1. 1]
E. w 2/u 0, IJ
[B[w a/l y 0] E [ 0 0, 0) - =0]
|ww' (E[z)u 1, = 1 E| y - P)
+ Ea 1. F| P. = -))
1w A/-
ww" T
Since
[w_|
we obtain the final result.
||
Definition C.1. Let E [c]]=] denole vector of independent. random variables where P(e+ =) P(e: -1) = 2. Then for the dataset D ~ P and the function class F, the Rademacher complerity for sample of size n is defined as: R(F)) = Ep [Ee [sup]er 1 -i= (xx])]]]
Proposition C.3 (Restated Proposition 3). Let m2 == He and mi == (I II),, and R(F) be the Rademacher complexity of some function space F. For training data D = {(x,, m,, yi) where D ~ pe, also we have that supm, [lmilla S B1, and supm, [|mill2 S Bn. Finally, let 4Lsuppa, Ep, 11g (M) fT (x) ST and 4LK supp:, Eps, [p, (X) (X) M]] r'. Then
A B + B2
A B + T [
A Bj + 7' A
R(FLe) <
R(FTIPMi) <
and R(F
Vn
Vn
Proof. First, we derive the bound on R(F12), which follows directly from Proposition 5 from Makar el al. h022
R(F12) EpE. sup
Σ Gwo x;)
w:|lw||, SA
EEpE,
sup
Σ w, T (ooT(([ 11)x; + IIx,)]
w:llwill SA 7
supp EpE.
1
Σ "ww" (mili + m.:)
w:ll w|]l SA n
A B2 B²
Vn
Next, derive the bounds for R R(FTIPM)) and R(FMBM). For some function class F, we get that
21

---

Published in Transactions on Machine Learning Research (01/2026)
1 Σ w xi
R(F) = EpLE
sup
w:l|wal| A
EpEc
sup w:llwull, A n
c (Tw as + (T- IT) x;)
< EpBBl
sup
< mii 20l m){))))
w
ll
SA 2A
w w
<
EDDE
sup
www m + EDE.
sup w
Σ
f ml]s]
wiill
ll
||
|l
SA n
A Bij CMMD Ps (
B A
Vn
The final bounds for both R(FTIPM)) and R(FMBM) follow directly from Lemma B.2
D Datasets
D.1 Waterbirds
The waterbirds dataset, which was first introduced by (Sagawa et al. 2019). uses labeled images of birds and their segmentations from the CUB-200-2011 dataset (Wah et al. 2011). If the bird is an Albatross, Auklet, Cormorant, Frigatebird, Fulmar, Gull, Jaeger, Kittiwake, Pelican, Puffin, Tern, Gadwall, Grebe, Mallard, Merganser, Guillemot, or a Pacific Loon, we classify as waterbird; every other bird from the dataset we classify a landbird. We use a subset of the places dataset (Zhou et al. 2017). for the background images. Specifically, we use the 200 land backgrounds and 300 water backgrounds provided by Makar et al.) h2022. Similar to (Makar et al. 2022 we derive other backgrounds from these images by applying rotations, manipulating brightness, and zooming in. The manipulated images are obtained from (Makar et al. 2022) In total, our waterbirds dataset is comprised of 8,672 landbirds and 2,483 waterbirds. The bird images, along with the background images, are randomly split so that 80% of the images are used to develop the synthetic training sets, and 20% are used for the testing sets.
The TIPMI leacher and MBM mediator models are trained on the segmented images of birds with black background, which is the privileged mediation information, as is shown in Figure 0 The student models are trained on images of birds containing either land or water backgrounds. Background and bird images are only used once to generate samples for the training and lesting datasets. The size of the images is 256 x 256 pixels, except for the finite-sample efficiency experiments, which use 128 x 128. Example images taken from one of the generated waterbirds datasets are shown Figure 7 For an additional shortcut, we add small black squares randomly placod within an image simulate camera artifacts. An example of landbird with this shorteut is given in Figure
22

---

Published in Transactions on Machine Learning Research (01/2026)
(a) Waterbird with no background
(b) Landbird with no background
Figure 6: An example of the TIPMI leacher and MBM mediator models training data for the waterbirds experiments. It consists of waterbird and landbird images with black backgrounds.
Landbird over land y =
Waterbird over land y v=
Landbird over water
Waterbird over water
y
y 1=
Figure 7: Examples of generated images used in the waterbirds dataset, where the only shortcut is the background.
23

---

Published in Transactions on Machine Learning Research (01/2026)
Figure 8: Example image from waterbirds dataset with two shorteuts. The first shorteut is the background, and the second shorteut is the black squares that simulate camera artifacts.
D.2 Food Review
The food review dataset is created from 20,000 randomly selected reviews from the Amazon Food Review dataset (MeAuley & Leskovec, 2013 It contains reviews, the amount of stars (1-5), and summary of each review. Each synthetic training dataset contains 16,000 samples, whereas the testing set contains 4,000. To inject a shortcut into the reviews, we add perturbation to the words "the" and "be" such that "the" becomes "thxxxxxx" and "be" becomes "bexxxxx", similar to what is done in (Veitch et al. 2021). For instance, the review "They make the best gummies hands down" turned into "They make thexxxxx best gummies hands down". We add a similar shortcut by adding perturbations to the words "a" and "to" such that "a" becomes "ayyyyy" and "to" becomes "Loyyyry". The labels are binarized such that Y 0 for 1-3 stars and Y for 4-5 stars. We employ summary of the review the mediator, generated by -lama-2-7b-cllee model TTovron et al. 2023). These model-generated summarics replace the original ones in the Amazon Food Review dataset, as many of the latter were not sufficiently informative.
D.3 KOA
The knee osteoarthritis dataset (Nevit et al. 2006) is comprised of knee X-ray images and joint space width measurements from the Osteoarthritis Initiative, which is publicly available for download at https://nda.nih.goy/
The knoe X-ray images are obtained from the "OAII2MonthImago" sub-datasct. They are stored as DICOM image and include both the patient's 's knees. We convert each image to a PNG and split in half so that each image only contains a patient's left or right knee. Each image then centered and rosized to 256x256 pixcls. The training dataset is comprisod of 80% of the total 4982 samples, whereas the testing set is the other 20%. The datasets are split that one patient's 's X-rays could only be in one of the two datasets. To determine if knee in the X-ray had osteoarthritis, we use the Kellgren-Tawrenn (KL) grades (0-4) (Kohn et al.) 2016) that were provided for each knee. If the knee had a grade of 2, 3, or 4, we classified the sample as having OA (Y=1); if it had a grade of 0 or 1, we classified it as normal (Y=0). For final evaluation, we discard all samples with KT grade of as it. is vaguely defined, and KOA diagnoses labeled with this grade are subjective OOsson et al. 2021). The distribution of the KL grades provided in Table 3
To inject spurious correlations into the dataset, we overlay black box over the X-rays such that the presence of the black box was correlated with an OA diagnosis. This was done to mimic metallic token, which may act as a spurious feature in real-world X-ray classification problems (Zech et al., 2018). Examples of the spuriously correlated KOA dataset are shown in Figure g To evaluate how well TIPMI MBM, and the basclines perform when multiple spurious fcatures are present, we also create datasct with an additional spuriously correlated white box, as is shown in Figure I0 For the privileged mediation information, we use 16 joint space width measurements that provided with each knee X-ray.
24

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: The distribution of KL grades in the KOA dataset.
KL Grade
Number of Samples
0
978
1
671
2 1928
3 1082
4
323
Healthy Knee Without Token
OA Knee Without Token
y ==0,0=0
y ==1,v=
Healthy Knee With Token
OA Knee With Token
y
y
Figure 9: Examples of generated images used in the KOA dataset with single shortcut.
25

---

Published in Transactions on Machine Learning Research (01/2026)
Figure 10: Example image from KOA dataset with two shortcuts represented by white and black square spuriously associated with ostcoarthritis.
E Experiment Setup
Models. For the waterbirds experiments, the TIPMI teacher models. TIPMI student models, MBM tar- get models, and all baseline models are ResNet-50 models (He al., 2016) pre-trained on ImageNet-1k (Russakovsky et al. 2015). The MBM mediator model is U-Net with ResNet-50 encoder, which is also pre-trained on ImageNet-1k For KOA, the TIPMI student, MBM mediator models, and all baseline models are ResNet-50 models, and the TIPMI teacher and MBM target models are single-layer neural networks with 1024 hidden units. For the food review experiments, the TIPMI teacher model, student model, MBM target model, and baseline models are BERT-tiny classifiers (Bhargava et al.) (202I), and the MBM mediator model a T5-small model (Raffel et al., 2020). sAll models use the cross entropy loss, except for the TIPMI teacher models, which use the mean squared error, and the MBM mediator model used in the waterbirds dataset, which uses the dice loss.
TIPMI Implemenation. The TIPMI teacher was implemented described in Section "Teacher (Step 1)" However, the student was trained by using the MSE to match the student logits with the teacher logits, not the final probabilities.
MBM Implemenation. We train MBMs sequentially, similar to sequential bottleneck models as described in (Koh et al. 2020): we first train the mediator model fM (x) and then train the target model g (M) using the mediator predictions of fM (X)..
Hyperparameter Selection. The hyperparameters for all models are chosen using cross- cross- validation across the 10 simulations. For GDRO, SCPM, and IRM, the hyperparameters are chosen based on the highest worst- group accuracy. The hyperparameters for the MBM mediator model were chosen based on the lowest MSF for KOA, highest ROUGE-1 score in food review, and lowest DICE loss in waterbirds. For all other models, the hyperparameters were chosen based on which model had the best AUROC.
For the waterbirds experiments, we perform cross-validatte parameters with the values [0, 1e- 1e-9 for all modcls except SCPM, MCPM, and IRM. For SCPM, we set the learning rale to le-5, and cross-validate over o [lel, 1e2] and [le0, lel, 1e2). For MCPM, we set the learning rate to 1e and cross-validate over (Garreau et al. 2017). For IRM, we set the learning rate to 1e and cross-validate over a [1e- 1, 1e0, le1]. We train each model using Adam (Kingma & Ba 2014] and a batch size of 32, except for SCPM and MCPM, which use a batch size of 256. Each model is trained for 100 epochs.
with a learning rate of le-5 and T-2 penalty
[1e0, 1e1, 1e2). is chosen via the median heuristic
For the food review experiments, we perform vross-validation with lcarning rate of 1 and L2 penalty parameters with the values [0, le 15 for all models except SCPM, MCPM, and IRM. For SCPM, we set the learning rate to 1e-5 and cross-validate
[1e1, 1e2) and [1e0, lel, 1e2, 163). For MCPM, we
26

---

Published in Transactions on Machine Learning Research (01/2026)
[1c0, 1c1, 1c2]. is chosen via the median heuristic (Garreau et al. 2017). For IRM, set the learning rate to 1e-5 and cross-validate over [1e- - 1e0, 1e1]. We train each model using Adam and batch size of 32, except for SCPM and MCPM, which use a batch size 256. Each model is trained for 10 epochs, except for SCMP and MCPM, which are trained for 20 epochs.
set the learning rate to 1e and cross-validate
For the KOA experiments, we perform vals-vaaio with learning rate of le- and La penalty parameters with the values [0, 10 9,1-- for all models except SCPM, MCPM, and IRM. For SCPM, set the learning rate to le 4, and cross-validate over [lel, 1e2] and [LeO, 1el, 1e2, 1e3]. For MCPM, we set the learning rate to 1e-6 and cross-validate over = [1e0. 1e1, 1e2]. a is chosen via the median heuristic GGrreau et al. 2017) For IRM, we set the learning rate to le and cross-validate over a [le- 1, 1e0, lel]. We train each model using Adam and a batch size of 32, except for SCPM and MCPM, which use a batch size of 256. Each model trained for 100 epochs.
Training Environment. All models used throughout this paper are implemented in PyTorrh (Paszke et al. 2019). We train each model using Nvidia A40 GPU and 36 GB of memory Linux operating system. All experiments took approximately month with three Nvidia A40 GPUs.
F
Additional Empirical Results
F.1
Two shortcuts
Waterbirds
KOA
Food Review
0.95
0.8
0.90
AUROC
0.05
0.6
0.80
0.4
0.75
TIPMI (Dus)
(Oum)
MBM (Ous)
0.4
TIPMI (Qur)
MBM (Oun)
TIPMI (Ouss)
(Cun)
MBM (Dur)
MCPM
L2
50
L2
GDRD
SCPM
GDRD
SCPM
GDRO
SCPM
(0un)
IRM
MBM-CF
IRM
0.8
MBM-CF
IRM
0.2
0.4
0.4
0.6
0.0
0.2
0.6
P(V() 1|Y
P(V()) = 0|Y = 0) at test time
Figure 11: r-axis shows P at test time. y-axis shows the AUROC over the test data. Spurious
correlations are introduced during training by setting P. (V(Vdd) 1 1) P.(V ofy 0) 0.95
the waterbirds dataset and P. (V(((d) 1 1) (d(V(d) o|Y 0) 0.9 in the food review and KOA datasets, for each shorteut v()) and (2).
F.2 Single shortcut with overlap
27

---

Published in Transactions on Machine Learning Research (01/2026)
Waterbirds
KOA
Food Review
1.000
0.959
AUROC
0.025
0.500
0.855
TIPMI (Cuns
(Ourg
TIPMI (Oies]
MIM (Oun)
0.4
(Gan)
MBM (Gun)
TIPMI-NC (Cung)
MCPM
TIPMINC (Dung
MCPM
TIPMI-NC (Oun)
MCPM
GDRO
GDRO
GDRO
0.825
MBM-CF (Oun)
IRM
MBM-CF (Ouns
IRM
MBM-CC (Ous)
IRM
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
04
0.8
1.0
0.0
0.2
0.4
0.6
1.0
P(V() = 1|Y = 1) = P(V() 0 = 0) at test time
Figure 12: The x-axis shows P(YV(W) at test time under different shifted distributions, the y-axis shows the AUROC over the test data, and the dashed vertical line shows P(Y])() at training time. Spurious correlations are introduced during training by setting P. Pv(V()) 1|Y 1) P.(V()) = 0| = 0) = 0.95 in
the waterbirds dataset and P.(V(1) = 1Y = 1) = P.(V()) = OjY = 0) 0.9 in the food review and KOA datasots at training.
F.3
Single shortcut with no overlap
Waterbirds
KOA
Food Review
1.0
1.0
1.0
0.9
0.8
AUROC
0.6
0.6
0.7
0.6
TIPMI (Durs
MCPM
MBM [Ous)
0.2
TIPMI (Dus)
MBM (Oars
0.2
TIPMI (Ous)
TIPMI-NC
MCPM
MBM (Oun)
L2
50
GORD
0.0
GORD
GDRD
0.
02
6.4
0.6
0.8
1.0
8.0
622
0.8
0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
P(V() = 1|Y = 1) = P(V() Y = 0) at test time
Figure 13: The x-axis shows (()) test time under different shifted distributions, the y-axis shows the AUROC over the test data, and the dashed vertical line shows P(Y](()) at training time. Spurious correlations are introduced during training by setting (.(V(1) 1|Y 1) = P.(V(1) 0fY = 0) 1.0 for all datasets at training.
F.4 Finite-sample efficiency results under the unconfounded distribution
To isolate the gains in finite-sample efficiency introduced by TIPMI and MBM, we run an "in-distribution" experiment where the training and test data are drawn from the unconfounded distribution, p where v() is simulated such that V(1) 1 Y. The results are shown in Table 4 TIPMI and MBM outperform all baselines over the waterbirds and KOA datasets. However, for food review, TIPMI-NC, L2, and GRDO perform the best. The difference between TIPMI-NC and TIPMI likely due the TIPMI teacher models having less training data from the cross-fitting procedure.
F.5 Single shortcut setting where the shortcut affects the mediator
28

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: In distribution results.
Data
Waterbirds
KOA
Food Review
Model
TIPMT
0.992 (0.002)
0.980 (0.004)
0.896 (0.007)
TIPMI-NC
0.983 (0.005)
0.980 (0.007)
0.908 (0.005)
L2
0.987 (0.004)
0.946 (0.010)
0.907 (0.006)
GDRO
0.986 (0.006)
0.948 (0.008)
0.903 (0.007)
MCPM
0.985 (0.004)
0.948 (0.008)
0.899 (0.007)
MBM
0.993 (0.003)
0.980 (0.006)
0.857 (0.005)
MBM-CF
0.993 (0.003)
0.977 (0.007) 0.853 (0.006)
KOA Dataset Where Shortcut Affects Mediator
1.0
0.9
0.8
AUROC
0.7
0.6
0.5
TIPMI P(My = 1/V = 1) =
TIPMI P(M, 1/V 1) =
0.4
MBM P(M, 1/V = 1) 0.5
MBM P(M, 1/V 1) 1.0
0.3
TIPMI P(Mv 1/V
0.75
MBM P(My = 1/V = 1) 0.75
0.2
0.0
0.4
0.6
1.0
P(V(1) = 1/Y 1) = P(V()) 0|Y 0) at test time
Figure 14: The x-axis shows (L)) at test time under different shifted distributions, the y-axis shows the AUROC over the test data, and the dashed vertical line shows P(Y]V()) training time. Spurious correlations introduced during training by setting (V((()) 1 1) (.(V()) ofY 0) 0.9 for all datasets at training.
In this section, we examine how sensitive TIPMI and MBM to varying degrees of causal inffuence from the shorteut to the mediator. For these experiments, we added additional binary variable My to the mediator in the KOA dataset that may be causally influenced by single shortcut. Spurious correlations are introduced during training by setting P.(V(1) 1| = 1) P.(V(() 0JY = 0) 0.9 in the KOA dataset and at test time we vary Po(V to assess sensitivity to the unobserved shortcut.
We examined MBM and TIPMI in several settings where we varied the amount of influence V onto My: one where PVVV- My = 1) = P.(Vi = 0) My = 0) = 0.5, P.(Vi = 1|Mr = 1) = PV, M = = 0) = 0.75, and (V(V/ M 1) = P.VV, = 0|My 0) 1.0. Our results are presented in Figure F.5 where we showed that as long the influence from V to My is not strong, both TIPMI and MBM will still outperform standard L2 regularization.
29


Reviews:
[{"review_id": "GCr4EL5I7n", "reviewer": "Reviewer_GaHH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary of contributions\n\nThe paper targets the problem of counterfactual invariance of a learned function, under the following assumptions: an anti-causal generative model $Y → M → X$with $X, Y$ potentially confounded, and $M$ mediator fully determined by $X$ and not affected by the confounder. Under these conditions, they show:\n\n1. Given access to a dataset $(y_i, m_i, x_i)_i^n$ at training time, it is possible to learn a counterfactually invariant predictor $f: x \\mapsto y$, where counterfactually invariant here means “unaffected by distribution shifts in $X$ due to confounders”.\n2. The main contribution is algorithmic: the authors present a two-stage learning procedure, where they learn to predict $Y$ from $M$ (a teacher model) and $M$ from $X$ (a student model) at training time. At test time, this results in a function that can predict $X \\mapsto M \\mapsto Y$ and is counterfactually invariant. They present empirical evidence on three datasets\n\n---\n\n## Strengths\nThe presentation and the setup are clear. The empirical results, although they could be expanded, are convincing. Moreover, if the assumptions of the data-generating process are satisfied, the proposed method seems to successfully mitigate the issues of previous algorithms that targeted shortcut learning, namely: (i) the need for a priori knowledge on the possible shortcuts, clearly unrealistic, and (ii) the need of inefficient optimization constraints to enforce invariances. \n\n---\n## Weaknesses\n\nThere is one key assumption, that the relation between $X$ and $M$ is deterministic with $M$ unaffected by the confounder, that makes the learning problem somewhat trivial. I have nothing against the simplicity of the derived conclusion, but further discussion is needed to understand (i) the relation of the proposed setup with the (causality) literature (ii) its relevance. \n\n**The relation with the *frontdoor* setup**\n\nThe paper’s graph and learning recipe are best read as an **anti-causal front-door** setup: $Y \\to M \\to X$ with shortcuts (i.e. latent confounder) $Y \\leftarrow V \\to X$. We already know from the frontdoor criterion that an unconfounded mediator allows estimation of the causal effect. A reinterpretation of this, is that a mediator allows to “disentangling” spurious relations (in this setting, latent shortcuts) from causal information (in this case, information flowing in the anticuasla direction from $X$ to $Y$). Seen through this lens, the conclusions of this paper aren’t surprising. I think it is needed to discuss this point in the paper, to clarify that the novelty in the paper is in the algorithm and the empirical result, while there is no conceptual/technical contribution in the choice of using a mediator to distill causal information from dependencies due to latent confounders.\n\n**Stronger assumptions—why are they sensible for shortcut learning?**\nThe core theoretical guarantees hinge on assumptions that make the learning problem comparatively easy: $M$ must (i) carry all task-relevant information from $Y$ to $X$, (ii) be immune to shortcut variation $V$, and (iii) be (near-)deterministically retrievable from $X$. These are stronger than standard front-door conditions. The paper should motivate and defend them in the context of shortcut learning. From the paper, I don’t understand why this is a sensible modeling choice of relevant problems. Why do authors believe that we have access to mediators that shield the effect of latent confounders?\n\n**More experiments**. Another concern that I have is that experiments on three datasets are not enough to derive “statistically” significant conclusions. I praise the authors for relying on real datasets for their main experiments; however, it would be good to have additional (real or synthetic) results to back their claim.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The shortcut learning problem is well known and studied. The proposed solution (use of a mediator) is convincing: if we interpret the graph as casual, it reminds of the front door criterion setup, where you can \"disentangle\" causal information from the dependencies due to spurious confounders. The same idea is applied in this setup, and I believe it to be reasonable. The empirical evidence provided by the authors supports their claim. The only caveat, is that more experiments are needed: conclusions are drawn on three datasets, which is not sufficient to generalize from them.", "audience_yn": "Yes", "audience_explanation": "Yes. The authors propose a conceptually and empirically valid solution to the problem of shortcut learning, a problem of well-known interest. Also, their result is well placed in continuity with previous literature (Veicht et al. (2021), Makar et al. (2022))", "requested_changes": "As specified in the summary of the contribution, I believe that\n\n1. [Critical] The paper needs to relate the conceptual idea of using a mediator to bypass latent confounders with the front door criterion, as discussed in the weaknesses section of the summary of the paper. \n2. [Critical] Also, experiments on three datasets only can not be regarded as sufficient evidence for the effectiveness of their method. Experiments on more datasets are needed, as discussed in the weaknesses section of the summary of the paper\n3. [Minor] I get why the proposed solution leads to better performance. What I don't understand is why the proposed solution would be realistic. The authors mention Knee osteoarthritis datastes as a motivation, but I am not sure that is enough. For details on this please refer to the discussion in the weaknesses section of the summary of the paper", "broader_impact_concerns": "I don't have any broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0cjbZlvPfF", "reviewer": "Reviewer_dzPR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the critical problem of shortcut learning, where models rely on spurious correlations in training data and lead to poor out-of-distribution performance. Instead of requiring privileged information about the shortcuts themselves, which may be unknown or hard to label, the authors proposed a novel perspective to leverage the privileged mediation information, which are intermediate variables that lie on the true causal path from the input to the label and are assumed to be available only during training. \n\nBased on this idea, the paper introduces two methods: (1) Mediator Bottleneck Models (MBMs): An adaptation of Concept Bottleneck Models, which first trains a model to predict the mediator from the input features and then a second model to predict the final label from this predicted mediator. (2) Teaching Invariance using Privileged Mediation Information (TIPMI): A knowledge distillation framework where an invariant teacher is first trained to predict the label using only the privileged mediator, and a student model is then trained on the standard inputs to fit the teacher's predictions, thereby inheriting its invariance properties. \n\nThis paper further provides a theoretical analysis that establishes the proposed methods' ability to achieve counterfactual invariance. This paper also analyzes their generalization properties, showing they can achieve tighter Rademacher complexity bounds compared to standard regularization or penalty-based methods. Numerical experiments are also provided to support the proposed methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s claims are clearly stated and supported by theoretical analysis. Numerical results comparing the proposed methods and baseline models on multiple datasets and the ablation study on the benefits of cross-fitting for both TIPMI and MBM are also consistent with the claims.", "audience_yn": "Yes", "audience_explanation": "The work focuses on the robustness under distribution shift, causality, and learning with privileged information, which are core interests for TMLR’s audience.", "requested_changes": "(1) This paper assumes that the shortcuts do not affect the mediator, which can be a strong assumption in practice. To show the robustness of the proposed algorithms, it is meaningful to present the empirical performance of MBM and TIPMI when there is a weak correlation between the shortcuts and the mediator. \n\n(2) The paper considers the scenario where there is a single binary shortcut. The authors can further explain how the analysis can be extended to multiple shortcuts and show the dependency of the error bounds on the number of shortcuts.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R8JG0MWizH", "reviewer": "Reviewer_rRJz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes two novel approaches for leveraging mediators to achieve better out-of-distribution generalization, compared to relying on known learning shortcuts. It relies on a set of annotated mediator variables that are assumed independent of potential shortcuts and distribution shifts. The first proposed approach uses one model to learn the mediators from the input features, and another model to learn the targets from these predicted mediators. This approach provides the strongest robustness and generalization results. However, the two models lead to increased inference resource demands. To address this, the second proposed approach first learns an auxiliary teacher model that predicts targets from ground-truth mediators. A second model then imitates this teacher model, mapping from inputs to the target predicted by the teacher. Only the second model is required for inference.", "claims_and_evidence_yn": "Yes", "claims_explanation": "**Caveat: I was unable to check the proofs, since this paper is outside my research area. For the same reason, I can not judge whether the selected baselines are sufficient.** \n\nFor the most part, the paper supports its claims well, as far as I can judge. The paper includes a strong experimental evaluation, comparing to several baselines on three datasets with different data modalities.\n\nI have an issue with the restriction to two shortcut confounders in Section 3.1. While Section 3.1 claims that the \"results hold for [confounders] V of arbitrary dimension\",  the theoretical results in the paper, including the proofs in the appendix, are for two shortcut confounders. For the presentation, it is reasonable to assume two confounders; however, the theoretical results should apply to an arbitrary number to support the above claim.", "audience_yn": "Yes", "audience_explanation": "The contributions in this paper are relevant to a broad audience, since distribution shifts are ample in real-world applications.", "requested_changes": "**Required:**  \n1. Please clarify the issue with the theoretical contribution I outlined above.\n2. Section 3.1 introduces the assumption that the mediators M are determined entirely by the input features X. This is a strong assumption that should be discussed in more detail. It should also be mentioned in the introduction.\n3. I do not understand your discussion on overfitting in Section 4.3. Please elaborate: the setting is that the teacher model overfits the training data, while observing only the mediators. Clearly, that can cause the student also to overfit, but you describe this as diminishing the benefits of distillation with regard to avoiding shortcuts. As far as I can see, overfitting can only cause the student to rely on shortcuts if the mediators are confounded with the shortcuts. However, this would violate your assumption that the shortcuts do not influence the mediators.\n4. Please elaborate: In Section 6.3, you state that TIPMI and MBM perform better than the baselines because they do not rely on expensive penalty terms. Why does the cost of penalty terms influence their performance in terms of AUROC?\n5. In Section 6.4, you compare the model sizes of TIPMI and MBM. Your comparison does not show that larger models are required for MBM. Please also report the performance of MBM using models of comparable size to TIPMI in this table. \n6. I do not understand why it is an issue for performance to predict superfluous mediators in Figure 5.  \n7. Please refer to and link to your proofs in the Appendix in the main body of the paper. \n\n**Strengthens the Paper:**  \nThe paper is already well-written. For a few paragraphs, I have suggestions for further improving the presentation:\n- In the last paragraph of the introduction, you write that TIPMI is advantageous, since it does not require modelling the potentially complex mapping from X to M. However, TIPMI requires modelling the mapping from X to Y instead. Instead, the core advantage of TIPMI is lower inference cost. \n- I did not understand from the notation why Equation (1) is a set of multiple elements.\n- In Teacher (Step 1), item 1, it would help to state that $I_1, \\ldots, I_k$ are sets of indices, for example $I_1, \\ldots, I_k \\subset \\{1, \\ldots, N\\}$.\n- Instead of using (1) and (2) when enumerating in text, such as in Proposition 1, I recommend using (a), (b) or (i), (ii). I was confused whether (1) and (2) referred to the equations.\n- Otherwise, the forward reference to the ablation study in the last paragraph on page 9 is clunky. Perhaps place the ablation study before the robustness study. \n- Figure titles in Section 6 would help to understand the figures.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "**Typos:**  \n- On page 3, the introduction \"PI\" is used, that was not introduced before\n- Since Figure 1 is a name, like Lemma 1, it should be capitalized. I also saw this for Proposition 2.\n- Page 7, bottom: \"The orthogonal projection *is* as X *is* only\". One *is* too much?\n- Abbreviations are not capitalized correctly in the References section. For example: \"... augmentations for text *ood* generalization ...\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)