SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We know that If  $\int_{\omega}p(\boldsymbol {x}|\boldsymbol{x}_{n(\boldsymbol{x})}^{*})d\boldsymbol {x}\approx 1$  then  $f$  is linear within the effective support of  $p$ . Therefore, any sample from  $p$  will almost surely lie within a single region  $\omega \in \Omega$ , and therefore the entire mapping can be considered linear with respect to  $p$ . Thus, the output distribution is a linear transformation of the input distribution based on the per-region affine mapping.
Appendix C Additional Empirical Validation
To validate empirically Theorem 2, we check empirically if the optimal solution for
$$
\sum_ {i} \log \frac {| \Sigma_ {Z} |}{| \Sigma_ {Z | X _ {i}} | | \Sigma_ {Z ^ {\prime} | X _ {i} ^ {\prime}} |}
$$
is a diagonal matrix. We trained VICReg on ResNet18 on CIFAR-10 and did random perturbations (with different scales) for  $\Sigma_Z$ . Then, for each perturbation, we calculated the average distance of this perturbed matrix from a diagonal matrix and the actual value of the term
$$
\sum_ {i} \log \frac {| \Sigma_ {Z} | | \Sigma_ {Z ^ {\prime} | X _ {i} ^ {\prime}} |}{| \Sigma_ {Z | X _ {i}} |}
$$
In Figure 3, we plot the difference from the optimal value of this term as a function of the distance from the diagonal matrix. As we can see, we get an optimal solution where we are close to the diagonal matrix. This observation gives us an empirical validation of Theorem 2.
Figure 3: The optimal solution for the optimization problem is a diagonal matrix. The average distance from a diagonal matrix for different perturbation scales. Experiments were conducted on CIFAR-10 with the ResNet-18 network.
Figure 4: Evolution of GMM training when enforcing a one-to-one mapping between the data and centroids akin to K-means i.e. using a small and fixed covariance matrix. We see that collapse does not occur. Left - In the presence of fixed input samples, we observe that there is no collapsing and that the entropy of the centers is high. Right - when we make the input samples trainable and optimize their location, all the points collapse into a single point, resulting in a sharp decrease in entropy.
Appendix D EM and GMM
Let us examine a toy dataset on the pattern of two intertwining moons to illustrate the collapse phenomenon under GMM (Figure 1, right). We begin by training a classical GMM with maximum likelihood, where the means are initialized based on random samples, and the covariance is used as the identity matrix. A red dot represents the Gaussian's mean after training, while a blue dot represents the data points. In the presence of fixed input samples, we observe that there is no collapsing and that the entropy of the centers is high (Figure 4, left). However, when we make the input samples trainable and optimize their location, all the points collapse into a single point, resulting in a sharp decrease in entropy (Figure 4, right).
To prevent collapse, we follow the K-means algorithm in enforcing sparse posteriors, i.e. using small initial standard deviations and learning only the mean. This forces a one-to-one mapping which leads all points to be closest to the mean without collapsing, resulting in high entropy (Figure 4 - middle, in the Appendix). Another option to prevent collapse is to use different learning rates for input and parameters. Using this setting, the collapsing of the parameters does not maximize the likelihood. Figure 1 (right) shows the results of GMM with different learning rates for learned inputs and parameters. When the parameter learning rate is sufficiently high in comparison to the input learning rate, the entropy decreases much more slowly and no collapse occurs.
Appendix E SimCLR
In contrastive learning, different augmented views of the same image are attracted (positive pairs), while different augmented views are repelled (negative pairs). MoCo [33] and SimCLR [16] are recent examples of self-supervised visual representation learning that reduce the gap between self-supervised and fully-supervised learning. SimCLR applies randomized augmentations to an image to create two different views,  $x$  and  $y$ , and encodes both of them with a shared encoder, producing representations  $r_x$  and  $r_y$ . Both  $r_x$  and  $r_y$  are l2-normalized. The SimCLR version of the InfoNCE objective is:
$$
\mathbb {E} _ {x, y} \left[ - \log \left(\frac {e ^ {\frac {1}{\eta} r _ {y} ^ {T} r _ {x}}}{\sum_ {k = 1} ^ {K} e ^ {\frac {1}{\eta} r _ {y _ {k}} ^ {T} r _ {x}}}\right) \right],
$$
where  $\eta$  is a temperature term and  $K$  is the number of views in a minibatch.
Appendix F Entropy Estimators
Entropy estimation is one of the classical problems in information theory, where Gaussian mixture density is one of the most popular representations. With a sufficient number of components, they can approximate any smooth function with arbitrary accuracy. For Gaussian mixtures, there is, however, no closed-form solution to differential entropy. There exist several approximations in the literature, including loose upper and lower bounds [35]. Monte Carlo (MC) sampling is one way to approximate Gaussian mixture entropy. With sufficient MC samples, an unbiased estimate of entropy with an arbitrarily accurate can be obtained. Unfortunately, MC sampling is very computationally expensive and typically requires a large number of samples, especially in high dimensions [13]. Using the first two moments of the empirical distribution, VIGCreg used one of the most straightforward approaches for approximating the entropy. Despite this, previous studies have found that this method is a poor approximation of the entropy in many cases [35]. Another option is to use the LogDet function. Several estimators have been proposed to implement it, including uniformly minimum variance unbiased (UMVU) [2], and Bayesian methods [49]. These methods, however, often require complex optimizations. The LogDet estimator presented in [74] used the differential entropy  $\alpha$  order entropy using scaled noise. They demonstrated that it can be applied to high-dimensional features and is robust to random noise. Based on Taylor-series expansions, [35] presented a lower bound for the entropy of Gaussian mixture random vectors. They use Taylor-series expansions of the logarithm of each Gaussian mixture component to get an analytical evaluation of the entropy measure. In addition, they present a technique for splitting Gaussian densities to avoid components with high variance, which would require computationally expensive calculations. Kolchinsky and Tracey [38] introduce a novel family of estimators for the mixture entropy. For this family, a pairwise-distance function between component densities is defined for each member. These estimators are computationally efficient as long as the pairwise-distance function and the entropy of each component distribution are easy to compute. Moreover, the estimator is continuous and smooth and is therefore useful for optimization problems. In addition, they presented both a lower bound (using Chernoff distance) and an upper bound (using the KL divergence) on the entropy, which are exact when the component distributions are grouped into well-separated clusters,
Appendix G Known Lemmas
We use the following well-known theorems as lemmas in our proofs. We put these below for completeness. These are classical results and not our results.
Lemma G.1. (Hoeffding's inequality) Let  $X_{1}, \ldots, X_{n}$  be independent random variables such that  $a \leq X_{i} \leq b$  almost surely. Consider the average of these random variables,  $S_{n} = \frac{1}{n} (X_{1} + \dots + X_{n})$ . Then, for all  $t > 0$ ,
$$
\mathbb {P} _ {S} \left(\operatorname {E} \left[ S _ {n} \right] - S _ {n} \geq (b - a) \sqrt {\frac {\ln (1 / \delta)}{2 n}}\right) \leq \delta ,
$$
$$
\mathbb {P} _ {S} \left(S _ {n} - \operatorname {E} \left[ S _ {n} \right] \geq (b - a) \sqrt {\frac {\ln (1 / \delta)}{2 n}}\right) \leq \delta .
$$
Proof. By using Hoeffding's inequality, we have that for all  $t > 0$
$$
\mathbb {P} _ {S} \left(\mathrm {E} \left[ S _ {n} \right] - S _ {n} \geq t\right) \leq \exp \left(- \frac {2 n t ^ {2}}{(b - a) ^ {2}}\right),
$$
$$
\mathbb {P} _ {S} \left(S _ {n} - \mathrm {E} \left[ S _ {n} \right] \geq t\right) \leq \exp \left(- \frac {2 n t ^ {2}}{(b - a) ^ {2}}\right),
$$
Setting  $\delta = \exp \left(-\frac{2nt^2}{(b - a)^2}\right)$  and solving for  $t > 0$
$$
\begin{array}{l} 1 / \delta = \exp \left(\frac {2 n t ^ {2}}{(b - a) ^ {2}}\right) \\ \Longrightarrow \ln (1 / \delta) = \frac {2 n t ^ {2}}{(b - a) ^ {2}} \\ \Longrightarrow \frac {(b - a) ^ {2} \ln (1 / \delta)}{2 n} = t ^ {2} \\ \Longrightarrow t = (b - a) \sqrt {\frac {\ln (1 / \delta)}{2 n}} \\ \end{array}
$$
It has been shown that generalization bounds can be obtained via Rademacher complexity [8, 50, 59]. The following is a trivial modification of [50, Theorem 3.1] for a one-sided bound on the nonnegative general loss functions:
Lemma G.2. Let  $\mathcal{G}$  be a set of functions with the codomain  $[0, M]$ . Then, for any  $\delta > 0$ , with probability at least  $1 - \delta$  over an i.i.d. draw of  $m$  samples  $S = (q_i)_{i=1}^m$ , the following holds for all  $\psi \in \mathcal{G}$ :
$$
\mathbb {E} _ {q} [ \psi (q) ] \leq \frac {1}{m} \sum_ {i = 1} ^ {m} \psi \left(q _ {i}\right) + 2 \mathcal {R} _ {m} (\mathcal {G}) + M \sqrt {\frac {\ln (1 / \delta)}{2 m}}, \tag {26}
$$
where  $\mathcal{R}_m(\mathcal{G})\coloneqq \mathbb{E}_{S,\xi}[\sup_{\psi \in \mathcal{G}}\frac{1}{m}\sum_{i = 1}^{m}\xi_i\psi (q_i)]$  and  $\xi_1,\ldots ,\xi_m$  are independent uniform random variables taking values in  $\{-1,1\}$ .
Proof. Let  $S = (q_i)_{i=1}^m$  and  $S' = (q_i')_{i=1}^m$ . Define
$$
\varphi (S) = \sup  _ {\psi \in \mathcal {G}} \mathbb {E} _ {x, y} [ \psi (q) ] - \frac {1}{m} \sum_ {i = 1} ^ {m} \psi \left(q _ {i}\right). \tag {27}
$$
To apply McDiarmid's inequality to  $\varphi(S)$ , we compute an upper bound on  $|\varphi(S) - \varphi(S')|$  where  $S$  and  $S'$  be two test datasets differing by exactly one point of an arbitrary index  $i_0$ ; i.e.,  $S_i = S_i'$  for all  $i \neq i_0$  and  $S_{i_0} \neq S_{i_0}'$ . Then,
$$
\varphi \left(S ^ {\prime}\right) - \varphi (S) \leq \sup  _ {\psi \in \mathcal {G}} \frac {\psi \left(q _ {i _ {0}}\right) - \psi \left(q _ {i _ {0}} ^ {\prime}\right)}{m} \leq \frac {M}{m}. \tag {28}
$$
Similarly,  $\varphi(S) - \varphi(S') \leq \frac{M}{m}$ . Thus, by McDiarmid's inequality, for any  $\delta > 0$ , with probability at least  $1 - \delta$ ,
$$
\varphi (S) \leq \mathbb {E} _ {S} [ \varphi (S) ] + M \sqrt {\frac {\ln (1 / \delta)}{2 m}}. \tag {29}
$$
$$
\begin{array}{l} \mathbb {E} _ {S} [ \varphi (S) ] = \mathbb {E} _ {S} \left[ \sup  _ {\psi \in \mathcal {G}} \mathbb {E} _ {S ^ {\prime}} \left[ \frac {1}{m} \sum_ {i = 1} ^ {m} \psi \left(q _ {i} ^ {\prime}\right) \right] - \frac {1}{m} \sum_ {i = 1} ^ {m} \psi \left(q _ {i}\right) \right] (30) \\ \leq \mathbb {E} _ {S, S ^ {\prime}} \left[ \sup  _ {\psi \in \mathcal {G}} \frac {1}{m} \sum_ {i = 1} ^ {m} \left(\psi \left(q _ {i} ^ {\prime}\right) - \psi \left(q _ {i}\right)\right) \right] (31) \\ \leq \mathbb {E} _ {\xi , S, S ^ {\prime}} \left[ \sup  _ {\psi \in \mathcal {G}} \frac {1}{m} \sum_ {i = 1} ^ {m} \xi_ {i} \left(\psi \left(q _ {i} ^ {\prime}\right) - \psi \left(q _ {i}\right)\right) \right] (32) \\ \leq 2 \mathbb {E} _ {\xi , S} \left[ \sup  _ {\psi \in \mathcal {G}} \frac {1}{m} \sum_ {i = 1} ^ {m} \xi_ {i} \psi \left(q _ {i}\right) \right] = 2 \mathcal {R} _ {m} (\mathcal {G}), (33) \\ \end{array}
$$
where the first line follows the definitions of each term, the second line uses Jensen's inequality and the convexity of the supremum, and the third line follows that for each  $\xi_i\in \{-1, + 1\}$ , the distribution of each term  $\xi_{i}(\ell (f(x_{i}^{\prime}),y_{i}^{\prime}) - \ell (f(x_{i}),y_{i}))$  is the distribution of  $(\ell (f(x_i'),y_i') - \ell (f(x_i),y_i))$  since  $S$  and  $S^{\prime}$  are drawn iid with the same distribution. The fourth line uses the subadditivity of supremum.
Appendix H Implementation Details for Maximizing Entropy Estimators
In this section, we will provide more details on the implantation of the experiments conducted in Section 5.2.
Setup Our experiments are conducted on CIFAR-10 [40]. We use ResNet-18 [32] as our backbone.
**Training Procedure:** The experimental process is organized into two sequential stages: unsupervised pretraining followed by linear evaluation. Initially, the unsupervised pretraining phase is executed, during which the encoder network is trained. Upon its completion, we transition to the linear evaluation phase, which serves as an assessment tool for the quality of the representation produced by the pretrained encoder.
Once the pretraining phase is concluded, we adhere to the fine-tuning procedures used in established baseline methods, as described by [14].
During the linear evaluation stage, we start by performing supervised training of the linear classifier. This is achieved by using the representations derived from the encoder network while keeping the network's coefficients frozen, and applying the same training dataset. Subsequently, we measure the test accuracy of the trained linear classifier using a separate validation dataset. This approach allows us to evaluate the performance of our model in a robust and systematic manner.
The training process for each model unfolds over 800 epochs, employing a batch size of 512. We utilize the Stochastic Gradient Descent (SGD) optimizer, characterized by a momentum of 0.9 and a weight decay of  $1e^{-4}$ . The learning rate is initiated at 0.5 and is adjusted according to a cosine decay schedule complemented by a linear warmup phase.
During the data augmentation process, two enhanced versions of every input image are generated. This involves cropping each image randomly and resizing it back to the original resolution. The images are then subject to random horizontal flipping, color jittering, grayscale conversion, Gaussian blurring, and polarization for further augmentation.
For the linear evaluation phase, the linear classifier is trained for 100 epochs with a batch size of 256. The SGD optimizer is again employed, this time with a momentum of 0.9 and no weight decay. The learning rate is managed using a cosine decay schedule, starting at 0.2 and reaching a minimum of  $2e - 4$ .
Appendix I A Generalization Bound for Downstream Tasks
In this Appendix, we present the complete version of Theorem 3 along with its proof and additional discussions.
I.1 Additional Notation and details
We start to introduce additional notation and details. We use the notation of  $x \in \mathcal{X}$  for an input and  $y \in \mathcal{Y} \subseteq \mathbb{R}^r$  for an output. Define  $p(y) = \mathbb{P}(Y = y)$  to be the probability of getting label  $y$  and  $\hat{p}(y) = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1}\{y_i = y\}$  to be the empirical estimate of  $p(y)$ . Let  $\zeta$  be an upper bound on the norm of the label as  $\|y\|_2 \leq \zeta$  for all  $y \in \mathcal{Y}$ . Define the minimum norm solution  $W_{\bar{S}}$  of the unlabeled data as  $W_{\bar{S}} = \text{minimize}_{W'} \|W'\|_F$  s.t.  $W' \in \arg \min_W \frac{1}{m} \sum_{i=1}^{m} \|Wf_\theta(x_i^+) - g^*(x_i^+)\|^2$ . Let  $\kappa_S$  be a data-dependent upper bound on the per-sample Euclidean norm loss with the trained model as  $\|W_S f_\theta(x) - y\| \leq \kappa_S$  for all  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ . Similarly, let  $\kappa_{\bar{S}}$  be a data-dependent upper bound on the per-sample Euclidean norm loss as  $\|W_{\bar{S}} f_\theta(x) - y\| \leq \kappa_{\bar{S}}$  for all  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ . Define the difference between  $W_S$  and  $W_{\bar{S}}$  by  $c = \|W_S - W_{\bar{S}}\|_2$ . Let  $\mathcal{W}$  be a hypothesis space of  $W$  such that  $W_{\bar{S}} \in \mathcal{W}$ . We denote by  $\tilde{\mathcal{R}}_m(\mathcal{W} \circ \mathcal{F}) = \frac{1}{\sqrt{m}} \mathbb{E}_{\bar{S},\xi}[\sup_{W \in \mathcal{W}, f \in \mathcal{F}} \sum_{i=1}^{m} \xi_i \|g^*(x_i^+) - Wf(x_i^+)\|]$  the normalized Rademacher complexity of the set  $\{x^+ \mapsto \|g^*(x^+) - Wf(x^+)\| : W \in \mathcal{W}, f \in \mathcal{F}\}$ . We denote by  $\kappa$  a upper bound on the per-sample Euclidean norm loss as  $\|Wf(x) - y\| \leq \kappa$  for all  $(x,y,W,f) \in \mathcal{X} \times \mathcal{Y} \times \mathcal{W} \times \mathcal{F}$ .
We adopt the following data-generating process model that was used in a previous paper on analyzing contrastive learning [58, 10]. For the labeled data, first,  $y$  is drawn from the distribution  $\rho$  on  $\mathcal{V}$ , and then  $x$  is drawn from the conditional distribution  $\mathcal{D}_y$  conditioned on the label  $y$ . That is, we have the join distribution  $\mathcal{D}(x,y) = \mathcal{D}_y(x)\rho(y)$  with  $((x_i,y_i))_{i=1}^n \sim \mathcal{D}^n$ . For the unlabeled data,
first, each of the unknown labels  $y^{+}$  and  $y^{-}$  is drawn from the distribution  $\rho$ , and then each of the positive examples  $x^{+}$  and  $x^{++}$  is drawn from the conditional distribution  $\mathcal{D}_{y^+}$  while the negative example  $x^{-}$  is drawn from the  $\mathcal{D}_{y^-}$ . Unlike the analysis of contrastive learning, we do not require negative samples. Let  $\tau_{\bar{S}}$  be a data-dependent upper bound on the invariance loss with the trained representation as  $\| f_{\theta}(\bar{x}) - f_{\theta}(x)\| \leq \tau_{\bar{S}}$  for all  $(\bar{x},x) \sim \mathcal{D}_y^2$  and  $y \in \mathcal{V}$ . Let  $\tau$  be a data-independent upper bound on the invariance loss with the trained representation as  $\| f(\bar{x}) - f(x)\| \leq \tau$  for all  $(\bar{x},x) \sim \mathcal{D}_y^2$ ,  $y \in \mathcal{V}$ , and  $f \in \mathcal{F}$ . For simplicity, we assume that there exists a function  $g^{*}$  such that  $y = g^{*}(x) \in \mathbb{R}^{r}$  for all  $(x,y) \in \mathcal{X} \times \mathcal{V}$ . Discarding this assumption adds the average of label noises to the final result, which goes to zero as the sample sizes  $n$  and  $m$  increase, assuming that the mean of the label noise is zero.
0.2 Proof of Theorem 3
Proof of Theorem 3. Let  $W = W_{S}$  where  $W_{S}$  is the minimum norm solution as  $W_{S} = \min_{i=1}^{m} \|W^{\prime}\|_{F}$  s.t.  $W^{\prime} \in \arg \min_{W} \frac{1}{n} \sum_{i=1}^{n} \|Wf_{\theta}(x_i) - y_i\|^2$ . Let  $W^{*} = W_{\bar{S}}$  where  $W_{\bar{S}}$  is the minimum norm solution as  $W^{*} = W_{\bar{S}} = \min_{i=1}^{m} \|W^{\prime}\|_{F}$  s.t.  $W^{\prime} \in \arg \min_{W} \frac{1}{m} \sum_{i=1}^{m} \|Wf_{\theta}(x_i^{+}) - g^{*}(x_i^{+})\|^2$ . Since  $y = g^{*}(x)$ ,
$$
y = g ^ {*} (x) \pm W ^ {*} f _ {\theta} (x) = W ^ {*} f _ {\theta} (x) + \left(g ^ {*} (x) - W ^ {*} f _ {\theta} (x)\right) = W ^ {*} f _ {\theta} (x) + \varphi (x)
$$
where  $\varphi (x) = g^{*}(x) - W^{*}f_{\theta}(x)$ . Define  $L_{S}(w) = \frac{1}{n}\sum_{i = 1}^{n}\| Wf_{\theta}(x_{i}) - y_{i}\|$ . Using these,
$$
\begin{array}{l} L _ {S} (w) = \frac {1}{n} \sum_ {i = 1} ^ {n} \| W f _ {\theta} (x _ {i}) - y _ {i} \| \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| W f _ {\theta} (x _ {i}) - W ^ {*} f _ {\theta} (x _ {i}) - \varphi (x _ {i}) \right\| \\ \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \| W f _ {\theta} (x _ {i}) - W ^ {*} f _ {\theta} (x _ {i}) \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} f _ {\theta} (x _ {i}) \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| \\ \end{array}
$$
where  $\tilde{W} = W - W^{*}$ . We now consider new fresh samples  $\bar{x}_i\sim \mathcal{D}_{y_i}$  for  $i = 1,\ldots ,n$  to rewrite the above further as:
$$
\begin{array}{l} L _ {S} (w) \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} f _ {\theta} (x _ {i}) \pm \tilde {W} f _ {\theta} (\bar {x} _ {i}) \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} f _ {\theta} (\bar {x} _ {i}) - (\tilde {W} f _ {\theta} (\bar {x} _ {i}) - \tilde {W} f _ {\theta} (x _ {i})) \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| \\ \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} f _ {\theta} (\bar {x} _ {i}) \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} f _ {\theta} (\bar {x} _ {i}) - \tilde {W} f _ {\theta} (x _ {i}) \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} f _ {\theta} (\bar {x} _ {i}) \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} (f _ {\theta} (\bar {x} _ {i}) - f _ {\theta} (x _ {i})) \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| \\ \end{array}
$$
This implies that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} f _ {\theta} (\bar {x} _ {i}) \| \leq L _ {S} (w) + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} (f _ {\theta} (\bar {x} _ {i}) - f _ {\theta} (x _ {i})) \| + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \|.
$$
Furthermore, since  $y = W^{*}f_{\theta}(x) + \varphi (x)$ , by writing  $\bar{y}_i = W^* f_\theta (\bar{x}_i) + \varphi (\bar{x}_i)$  (where  $\bar{y}_i = y_i$  since  $\bar{x}_i\sim \mathcal{D}_{y_i}$  for  $i = 1,\ldots ,n$ ),
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} f _ {\theta} (\bar {x} _ {i}) \| = \frac {1}{n} \sum_ {i = 1} ^ {n} \| W f _ {\theta} (\bar {x} _ {i}) - W ^ {*} f _ {\theta} (\bar {x} _ {i}) \| \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \| W f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} + \varphi (\bar {x} _ {i}) \| \\ \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \| W f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| - \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (\bar {x} _ {i}) \| \\ \end{array}
$$
Combining these, we have that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \| W f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| \leq L _ {S} (w) + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} \left(f _ {\theta} (\bar {x} _ {i}) - f _ {\theta} \left(x _ {i}\right)\right) \| \tag {34} \\ + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (\bar {x} _ {i}) \|. \\ \end{array}
$$
To bound the left-hand side of equation 34, we now analyze the following random variable:
$$
\mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \|, \tag {35}
$$
where  $\bar{y}_i = y_i$  since  $\bar{x}_i \sim \mathcal{D}_{y_i}$  for  $i = 1, \dots, n$ . Importantly, this means that as  $W_S$  depends on  $y_i$ ,  $W_S$  depends on  $\bar{y}_i$ . Thus, the collection of random variables  $\| W_S f_\theta(\bar{x}_1) - \bar{y}_1 \|, \dots, \| W_S f_\theta(n_n) - \bar{y}_n \|$  is not independent. Accordingly, we cannot apply standard concentration inequality to bound equation 35. A standard approach in learning theory is to first bound equation 35 by  $\mathbb{E}_{x,y} \| W_S f_\theta(x) - y \| - \frac{1}{n} \sum_{i=1}^{n} \| W_S f_\theta(\bar{x}_i) - \bar{y}_i \| \leq \sup_{W \in \mathcal{W}} \mathbb{E}_{x,y} \| W f_\theta(x) - y \| - \frac{1}{n} \sum_{i=1}^{n} \| W f_\theta(\bar{x}_i) - \bar{y}_i \|$  for some hypothesis space  $\mathcal{W}$  (that is independent of  $S$ ) and realize that the right-hand side now contains the collection of independent random variables  $\| W f_\theta(\bar{x}_1) - \bar{y}_1 \|, \dots, \| W f_\theta(n_n) - \bar{y}_n \|$ , for which we can utilize standard concentration inequalities. This reasoning leads to the Rademacher complexity of the hypothesis space  $\mathcal{W}$ . However, the complexity of the hypothesis space  $\mathcal{W}$  can be very large, resulting in a loose bound. In this proof, we show that we can avoid the dependency on hypothesis space  $\mathcal{W}$  by using a very different approach with conditional expectations to take care of the dependent random variables  $\| W_S f_\theta(\bar{x}_1) - \bar{y}_1 \|, \dots, \| W_S f_\theta(n_n) - \bar{y}_n \|$ . Intuitively, we utilize the fact that for these dependent random variables, there is a structure of conditional independence, conditioned on each  $y \in \mathcal{Y}$ .
We first write the expected loss as the sum of the conditional expected loss: