SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {X, Y} [ \| W _ {S} f _ {\theta} (X) - Y \| ] = \sum_ {y \in \mathcal {Y}} \mathbb {E} _ {X, Y} [ \| W _ {S} f _ {\theta} (X) - Y \| | Y = y ] \mathbb {P} (Y = y) \\ = \sum_ {y \in \mathcal {Y}} \mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] \mathbb {P} (Y = y), \\ \end{array}
$$
where  $X_{y}$  is the random variable for the conditional with  $Y = y$ . Using this, we decompose equation 35 into two terms:
$$
\begin{array}{l} \mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| \tag {36} \\ = \left(\sum_ {y \in \mathcal {Y}} \mathbb {E} _ {X _ {y}} \left[ \| W _ {S} f _ {\theta} (X _ {y}) - y \| \right] \frac {| \mathcal {I} _ {y} |}{n} - \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \|\right) \\ + \sum_ {y \in \mathcal {Y}} \mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] \left(\mathbb {P} (Y = y) - \frac {| \mathcal {I} _ {y} |}{n}\right), \\ \end{array}
$$
$$
\mathcal {I} _ {y} = \{i \in [ n ]: y _ {i} = y \}.
$$
The first term in the right-hand side of equation 36 is further simplified by using
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| = \frac {1}{n} \sum_ {y \in \mathcal {Y}} \sum_ {i \in \mathcal {I} _ {y}} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - y \|,
$$
$$
\begin{array}{l} \sum_ {y \in \mathcal {Y}} \mathbb {E} _ {X _ {y}} \left[ \| W _ {S} f _ {\theta} (X _ {y}) - y \| \right] \frac {| \mathcal {I} _ {y} |}{n} - \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| \\ = \frac {1}{n} \sum_ {y \in \mathcal {Y}} | \mathcal {I} _ {y} | \left(\mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] - \frac {1}{| \mathcal {I} _ {y} |} \sum_ {i \in \mathcal {I} _ {y}} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - y \|\right), \\ \end{array}
$$
where  $\tilde{\mathcal{V}} = \{y\in \mathcal{V}:|\mathcal{I}_y|\neq 0\}$ . Substituting these into equation equation 36 yields
$$
\begin{array}{l} \mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| \tag {37} \\ = \frac {1}{n} \sum_ {y \in \mathcal {Y}} | \mathcal {I} _ {y} | \left(\mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] - \frac {1}{| \mathcal {I} _ {y} |} \sum_ {i \in \mathcal {I} _ {y}} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - y \|\right) \\ + \sum_ {y \in \mathcal {Y}} \mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] \left(\mathbb {P} (Y = y) - \frac {| \mathcal {I} _ {y} |}{n}\right) \\ \end{array}
$$
Importantly, while  $\| W_S f_\theta(\bar{x}_1) - \bar{y}_1\|, \ldots, \| W_S f_\theta(\bar{x}_n) - \bar{y}_n\|$  on the right-hand side of equation 37 are dependent random variables,  $\| W_S f_\theta(\bar{x}_1) - y\|, \ldots, \| W_S f_\theta(\bar{x}_n) - y\|$  are independent random variables since  $W_S$  and  $\bar{x}_i$  are independent and  $y$  is fixed here. Thus, by using Hoeffding's inequality (Lemma G.1), and taking union bounds over  $y \in \tilde{\mathcal{V}}$ , we have that with probability at least  $1 - \delta$ , the following holds for all  $y \in \tilde{\mathcal{V}}$ :
$$
\mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] - \frac {1}{| \mathcal {I} _ {y} |} \sum_ {i \in \mathcal {I} _ {y}} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - y \| \leq \kappa_ {S} \sqrt {\frac {\ln (| \tilde {\mathcal {V}} | / \delta)}{2 | \mathcal {I} _ {y} |}}.
$$
This implies that with probability at least  $1 - \delta$
$$
\begin{array}{l} \frac {1}{n} \sum_ {y \in \tilde {\mathcal {Y}}} | \mathcal {I} _ {y} | \left(\mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] - \frac {1}{| \mathcal {I} _ {y} |} \sum_ {i \in \mathcal {I} _ {y}} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - y \|\right) \\ \leq \frac {\kappa_ {S}}{n} \sum_ {y \in \tilde {\mathcal {Y}}} | \mathcal {I} _ {y} | \sqrt {\frac {\ln (| \tilde {\mathcal {Y}} | / \delta)}{2 | \mathcal {I} _ {y} |}} \\ = \kappa_ {S} \left(\sum_ {y \in \tilde {\mathcal {Y}}} \sqrt {\frac {| \mathcal {I} _ {y} |}{n}}\right) \sqrt {\frac {\ln (| \tilde {\mathcal {Y}} | / \delta)}{2 n}}. \\ \end{array}
$$
Substituting this bound into equation 37, we have that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| \tag {38} \\ \leq \kappa_ {S} \left(\sum_ {y \in \tilde {\mathcal {Y}}} \sqrt {\hat {p} (y)}\right) \sqrt {\frac {\ln (| \tilde {\mathcal {Y}} | / \delta)}{2 n}} + \sum_ {y \in \mathcal {Y}} \mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] \left(\mathbb {P} (Y = y) - \frac {| \mathcal {I} _ {y} |}{n}\right) \\ \end{array}
$$
$$
\hat {p} (y) = \frac {\left| \mathcal {I} _ {y} \right|}{n}.
$$
Moreover, for the second term on the right-hand side of equation 38, by using Lemma 1 of [37], we have that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \sum_ {y \in \mathcal {Y}} \mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ] \left(\mathbb {P} (Y = y) - \frac {| \mathcal {I} _ {y} |}{n}\right) \\ \leq \left(\sum_ {y \in \mathcal {Y}} \sqrt {p (y)} \mathbb {E} _ {X _ {y}} [ \| W _ {S} f _ {\theta} (X _ {y}) - y \| ]\right) \sqrt {\frac {2 \ln (| \mathcal {Y} | / \delta)}{2 n}} \\ \leq \kappa_ {S} \left(\sum_ {y \in \mathcal {Y}} \sqrt {p (y)}\right) \sqrt {\frac {2 \ln (| \mathcal {Y} | / \delta)}{2 n}} \\ \end{array}
$$
where  $p(y) = \mathbb{P}(Y = y)$ . Substituting this bound into equation 38 with the union bound, we have that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| \tag {39} \\ \leq \kappa_ {S} \left(\sum_ {y \in \mathcal {Y}} \sqrt {\hat {p} (y)}\right) \sqrt {\frac {\ln (2 | \tilde {\mathcal {Y}} | / \delta)}{2 n}} + \kappa_ {S} \left(\sum_ {y \in \mathcal {Y}} \sqrt {p (y)}\right) \sqrt {\frac {2 \ln (2 | \mathcal {Y} | / \delta)}{2 n}} \\ \leq \left(\sum_ {y \in \mathcal {Y}} \sqrt {\hat {p} (y)}\right) \kappa_ {S} \sqrt {\frac {2 \ln (2 | \mathcal {Y} | / \delta)}{2 n}} + \left(\sum_ {y \in \mathcal {Y}} \sqrt {p (y)}\right) \kappa_ {S} \sqrt {\frac {2 \ln (2 | \mathcal {Y} | / \delta)}{2 n}} \\ \leq \kappa_ {S} \sqrt {\frac {2 \ln (2 | \mathcal {Y} | / \delta)}{2 n}} \sum_ {y \in \mathcal {Y}} \left(\sqrt {\hat {p} (y)} + \sqrt {p (y)}\right) \\ \end{array}
$$
Combining equation 34 and equation 39 implies that with probability at least  $1 - \delta$
$$
\begin{array}{l} \mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] \tag {40} \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (\bar {x} _ {i}) - \bar {y} _ {i} \| + \kappa_ {S} \sqrt {\frac {2 \ln (2 | \mathcal {Y} | / \delta)}{2 n}} \sum_ {y \in \mathcal {Y}} \left(\sqrt {\hat {p} (y)} + \sqrt {p (y)}\right) \\ \leq L _ {S} \left(w _ {S}\right) + \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \tilde {W} \left(f _ {\theta} \left(\bar {x} _ {i}\right) - f _ {\theta} \left(x _ {i}\right)\right) \right\| \\ + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (\bar {x} _ {i}) \| + \kappa_ {S} \sqrt {\frac {2 \ln (2 | \mathcal {Y} | / \delta)}{2 n}} \sum_ {y \in \mathcal {Y}} \left(\sqrt {\hat {p} (y)} + \sqrt {p (y)}\right). \\ \end{array}
$$
We will now analyze the term  $\frac{1}{n}\sum_{i = 1}^{n}\| \varphi (x_i)\| +\frac{1}{n}\sum_{i = 1}^{n}\| \varphi (\bar{x}_i)\|$  on the right-hand side of equation 40. Since  $W^{*} = W_{\bar{S}}$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| = \frac {1}{n} \sum_ {i = 1} ^ {n} \| g ^ {*} (x _ {i}) - W _ {\bar {S}} f _ {\theta} (x _ {i}) \|.
$$
By using Hoeffding's inequality (Lemma G.1), we have that for any  $\delta > 0$ , with probability at least  $1 - \delta$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \| g ^ {*} (x _ {i}) - W _ {\bar {S}} f _ {\theta} (x _ {i}) \| \leq \mathbb {E} _ {x ^ {+}} [ \| g ^ {*} (x ^ {+}) - W _ {\bar {S}} f _ {\theta} (x ^ {+}) \| ] + \kappa_ {\bar {S}} \sqrt {\frac {\ln (1 / \delta)}{2 n}}.
$$
Moreover, by using [50, Theorem 3.1] with the loss function  $x^{+} \mapsto \| g^{*}(x^{+}) - Wf(x^{+}) \|$  (i.e., Lemma G.2), we have that for any  $\delta > 0$ , with probability at least  $1 - \delta$ ,
$$
\mathbb {E} _ {x ^ {+}} \left[ \| g ^ {*} \left(x ^ {+}\right) - W _ {\bar {S}} f _ {\theta} \left(x ^ {+}\right) \| \right] \leq \frac {1}{m} \sum_ {i = 1} ^ {m} \| g ^ {*} \left(x _ {i} ^ {+}\right) - W _ {\bar {S}} f _ {\theta} \left(x _ {i} ^ {+}\right) \| + \frac {2 \tilde {\mathcal {R}} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + \kappa \sqrt {\frac {\ln (1 / \delta)}{2 m}} \tag {41}
$$
where  $\tilde{\mathcal{R}}_m(\mathcal{W}\circ \mathcal{F}) = \frac{1}{\sqrt{m}}\mathbb{E}_{\bar{S},\xi}[\sup_{W\in \mathcal{W},f\in \mathcal{F}}\sum_{i = 1}^{m}\xi_i\| g^* (x_i^+) - Wf(x_i^+)\| ]$  is the normalized Rademacher complexity of the set  $\{x^{+}\mapsto \| g^{*}(x^{+}) - Wf(x^{+})\| :W\in \mathcal{W},f\in \mathcal{F}\}$  (it is normalized such that  $\tilde{\mathcal{R}}_m(\mathcal{F}) = O(1)$  as  $m\to \infty$  for typical choices of  $\mathcal{F}$ ), and  $\xi_1,\ldots ,\xi_m$  are independent uniform random variables taking values in  $\{-1,1\}$ . Taking union bounds, we have that for any  $\delta >0$ , with probability at least  $1 - \delta$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (x _ {i}) \| \leq \frac {1}{m} \sum_ {i = 1} ^ {m} \| g ^ {*} (x _ {i} ^ {+}) - W _ {\bar {S}} f _ {\theta} (x _ {i} ^ {+}) \| + \frac {2 \tilde {\mathcal {R}} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + \kappa \sqrt {\frac {\ln (2 / \delta)}{2 m}} + \kappa_ {\bar {S}} \sqrt {\frac {\ln (2 / \delta)}{2 n}}
$$
Similarly, for any  $\delta > 0$ , with probability at least  $1 - \delta$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (\bar {x} _ {i}) \| \leq \frac {1}{m} \sum_ {i = 1} ^ {m} \| g ^ {*} (x _ {i} ^ {+}) - W _ {\bar {S}} f _ {\theta} (x _ {i} ^ {+}) \| + \frac {2 \tilde {\mathcal {R}} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + \kappa \sqrt {\frac {\ln (2 / \delta)}{2 m}} + \kappa_ {\bar {S}} \sqrt {\frac {\ln (2 / \delta)}{2 n}}.
$$
Thus, by taking union bounds, we have that for any  $\delta > 0$ , with probability at least  $1 - \delta$
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi \left(x _ {i}\right) \| + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (\bar {x} _ {i}) \| \tag {42} \\ \leq \frac {2}{m} \sum_ {i = 1} ^ {m} \| g ^ {*} (x _ {i} ^ {+}) - W _ {\bar {S}} f _ {\theta} (x _ {i} ^ {+}) \| + \frac {4 \mathcal {R} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + 2 \kappa \sqrt {\frac {\ln (4 / \delta)}{2 m}} + 2 \kappa_ {\bar {S}} \sqrt {\frac {\ln (4 / \delta)}{2 n}} \\ \end{array}
$$
To analyze the first term on the right-hand side of equation 42, recall that
$$
W _ {\bar {S}} = \underset {W ^ {\prime}} {\operatorname {m i n i m i z e}} \| W ^ {\prime} \| _ {F} \text {s . t .} W ^ {\prime} \in \arg \min  _ {W} \frac {1}{m} \sum_ {i = 1} ^ {m} \| W f _ {\theta} \left(x _ {i} ^ {+}\right) - g ^ {*} \left(x _ {i} ^ {+}\right) \| ^ {2}. \tag {43}
$$
Here, since  $Wf_{\theta}(x_i^+) \in \mathbb{R}^r$ , we have that
$$
W f _ {\theta} (x _ {i} ^ {+}) = \operatorname {v e c} [ W f _ {\theta} (x _ {i} ^ {+}) ] = [ f _ {\theta} (x _ {i} ^ {+}) ^ {\top} \otimes I _ {r} ] \operatorname {v e c} [ W ] \in \mathbb {R} ^ {r},
$$
where  $I_r \in \mathbb{R}^{r \times r}$  is the identity matrix, and  $[f_{\theta}(x_i^+)^\top \otimes I_r] \in \mathbb{R}^{r \times dr}$  is the Kronecker product of the two matrices, and  $\operatorname{vec}[W] \in \mathbb{R}^{dr}$  is the vectorization of the matrix  $W \in \mathbb{R}^{r \times d}$ . Thus, by defining  $A_i = [f_{\theta}(x_i^+)^\top \otimes I_r] \in \mathbb{R}^{r \times dr}$  and using the notation of  $w = \operatorname{vec}[W]$  and its inverse  $W = \operatorname{vec}^{-1}[w]$  (i.e., the inverse of the vectorization from  $\mathbb{R}^{r \times d}$  to  $\mathbb{R}^{dr}$  with a fixed ordering), we can rewrite equation 43 by
$$
W _ {\bar {S}} = \operatorname {v e c} ^ {- 1} [ w _ {\bar {S}} ] \quad \text {w h e r e} \quad w _ {\bar {S}} = \underset {w ^ {\prime}} {\operatorname {m i n i m i z e}} \| w ^ {\prime} \| _ {F} \text {s . t .} w ^ {\prime} \in \arg \min  _ {w} \sum_ {i = 1} ^ {m} \| g _ {i} - A _ {i} w \| ^ {2},
$$
with  $g_{i} = g^{*}(x_{i}^{+}) \in \mathbb{R}^{r}$ . Since the function  $w \mapsto \sum_{i=1}^{m} \|g_{i} - A_{i}w\|^{2}$  is convex, a necessary and sufficient condition of the minimizer of this function is obtained by
$$
0 = \nabla_ {w} \sum_ {i = 1} ^ {m} \| g _ {i} - A _ {i} w \| ^ {2} = 2 \sum_ {i = 1} ^ {m} A _ {i} ^ {\top} (g _ {i} - A _ {i} w) \in \mathbb {R} ^ {d r}
$$
$$
\sum_ {i = 1} ^ {m} A _ {i} ^ {\top} A _ {i} w = \sum_ {i = 1} ^ {m} A _ {i} ^ {\top} g _ {i}.
$$
$$
A ^ {\top} A w = A ^ {\top} g \quad \text {w h e r e} A = \left[ \begin{array}{c} A _ {1} \\ A _ {2} \\ \vdots \\ A _ {m} \end{array} \right] \in \mathbb {R} ^ {m r \times d r} \text {a n d} g = \left[ \begin{array}{c} g _ {1} \\ g _ {2} \\ \vdots \\ g _ {m} \end{array} \right] \in \mathbb {R} ^ {m r}
$$
$$
w ^ {\prime} \in \arg \min  _ {w} \sum_ {i = 1} ^ {m} \| g _ {i} - A _ {i} w \| ^ {2} = \left\{\left(A ^ {\top} A\right) ^ {\dagger} A ^ {\top} g + v: v \in \operatorname {N u l l} (A) \right\}
$$
where  $(A^{\top}A)^{\dagger}$  is the Moore-Penrose inverse of the matrix  $A^{\top}A$  and  $\mathrm{Null}(A)$  is the null space of the matrix  $A$ . Thus, the minimum norm solution is obtained by
$$
\operatorname {v e c} \left[ W _ {\bar {S}} \right] = w _ {\bar {S}} = \left(A ^ {\top} A\right) ^ {\dagger} A ^ {\top} g.
$$
Thus, by using this  $W_{\bar{S}}$  , we have that
$$
\begin{array}{l} \frac {1}{m} \sum_ {i = 1} ^ {m} \| g ^ {*} \left(x _ {i} ^ {+}\right) - W _ {\bar {S}} f _ {\theta} \left(x _ {i} ^ {+}\right) \| = \frac {1}{m} \sum_ {i = 1} ^ {m} \sqrt {\sum_ {k = 1} ^ {r} \left(\left(g _ {i} - A _ {i} w _ {\bar {S}}\right) _ {k}\right) ^ {2}} \\ \leq \sqrt {\frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 1} ^ {r} \left(\left(g _ {i} - A _ {i} w _ {\bar {S}}\right) _ {k}\right) ^ {2}} \\ = \frac {1}{\sqrt {m}} \sqrt {\sum_ {i = 1} ^ {m} \sum_ {k = 1} ^ {r} \left(\left(g _ {i} - A _ {i} w _ {\bar {S}}\right) _ {k}\right) ^ {2}} \\ = \frac {1}{\sqrt {m}} \| g - A w _ {\bar {S}} \| _ {2} \\ = \frac {1}{\sqrt {m}} \| g - A (A ^ {\top} A) ^ {\dagger} A ^ {\top} g \| _ {2} = \frac {1}{\sqrt {m}} \| (I - A (A ^ {\top} A) ^ {\dagger} A ^ {\top}) g \| _ {2} \\ \end{array}
$$
where the inequality follows from the Jensen's inequality and the concavity of the square root function. Thus, we have that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi \left(x _ {i}\right) \| + \frac {1}{n} \sum_ {i = 1} ^ {n} \| \varphi (\bar {x} _ {i}) \| \tag {44} \\ \leq \frac {2}{\sqrt {m}} \| (I - A (A ^ {\top} A) ^ {\dagger} A ^ {\top}) g \| _ {2} + \frac {4 \mathcal {R} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + 2 \kappa \sqrt {\frac {\ln (4 / \delta)}{2 m}} + 2 \kappa_ {S} \sqrt {\frac {\ln (4 / \delta)}{2 n}} \\ \end{array}
$$
By combining equation 40 and equation 44 with union bound, we have that
$$
\begin{array}{l} \mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] \tag {45} \\ \leq L _ {S} \left(w _ {S}\right) + \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \tilde {W} \left(f _ {\theta} \left(\bar {x} _ {i}\right) - f _ {\theta} \left(x _ {i}\right)\right) \right\| + \frac {2}{\sqrt {m}} \| \mathbf {P} _ {A} g \| _ {2} \\ + \frac {4 \mathcal {R} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + 2 \kappa \sqrt {\frac {\ln (8 / \delta)}{2 m}} + 2 \kappa_ {\bar {S}} \sqrt {\frac {\ln (8 / \delta)}{2 n}} \\ + \kappa_ {S} \sqrt {\frac {2 \ln (4 | \mathcal {Y} | / \delta)}{2 n}} \sum_ {y \in \mathcal {Y}} \left(\sqrt {\hat {p} (y)} + \sqrt {p (y)}\right). \\ \end{array}
$$
where  $\tilde{W} = W_{S} - W^{*}$  and  $\mathbf{P}_A = I - A(A^\top A)^\dagger A^\top$
We will now analyze the second term on the right-hand side of equation 45:
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} \left(f _ {\theta} \left(\bar {x} _ {i}\right) - f _ {\theta} \left(x _ {i}\right)\right) \| \leq \| \tilde {W} \| _ {2} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \| f _ {\theta} (\bar {x} _ {i}) - f _ {\theta} \left(x _ {i}\right) \|\right), \tag {46}
$$
where  $\| \tilde{W} \|_2$  is the spectral norm of  $\tilde{W}$ . Since  $\bar{x}_i$  shares the same label with  $x_i$  as  $\bar{x}_i \sim \mathcal{D}_{y_i}$  (and  $x_i \sim \mathcal{D}_{y_i}$ ), and because  $f_\theta$  is trained with the unlabeled data  $\bar{S}$ , using Hoeffding's inequality (Lemma G.1) implies that with probability at least  $1 - \delta$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| f _ {\theta} (\bar {x} _ {i}) - f _ {\theta} (x _ {i}) \| \leq \mathbb {E} _ {y \sim \rho} \mathbb {E} _ {\bar {x}, x \sim \mathcal {D} _ {y} ^ {2}} [ \| f _ {\theta} (\bar {x}) - f _ {\theta} (x) \| ] + \tau_ {\bar {S}} \sqrt {\frac {\ln (1 / \delta)}{2 n}}. \tag {47}
$$
Moreover, by using [50, Theorem 3.1] with the loss function  $(x,\bar{x})\mapsto \| f_{\theta}(\bar{x}) - f_{\theta}(x)\|$  (i.e., Lemma G.2), we have that with probability at least  $1 - \delta$
$$
\mathbb {E} _ {y \sim \rho} \mathbb {E} _ {\bar {x}, x \sim \mathcal {D} _ {y} ^ {2}} [ \| f _ {\theta} (\bar {x}) - f _ {\theta} (x) \| ] \leq \frac {1}{m} \sum_ {i = 1} ^ {m} \| f _ {\theta} \left(x _ {i} ^ {+}\right) - f _ {\theta} \left(x _ {i} ^ {+ +}\right) \| + \frac {2 \tilde {\mathcal {R}} _ {m} (\mathcal {F})}{\sqrt {m}} + \tau \sqrt {\frac {\ln (1 / \delta)}{2 m}} \tag {48}
$$
where  $\tilde{\mathcal{R}}_m(\mathcal{F}) = \frac{1}{\sqrt{m}}\mathbb{E}_{\bar{S},\xi}[\sup_{f\in \mathcal{F}}\sum_{i = 1}^{m}\xi_i\| f(x_i^+) - f(x_i^{+ + })\| ]$  is the normalized Rademacher complexity of the set  $\{(x^{+},x^{+ + })\mapsto \| f(x^{+}) - f(x^{+ + })\| :f\in \mathcal{F}\}$  (it is normalized such that  $\tilde{\mathcal{R}}_m(\mathcal{F}) = O(1)$  as  $m\to \infty$  for typical choices of  $\mathcal{F}$ ), and  $\xi_1,\ldots ,\xi_m$  are independent uniform random variables taking values in  $\{-1,1\}$ . Thus, taking union bound, we have that for any  $\delta >0$  with probability at least  $1 - \delta$
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \| \tilde {W} \left(f _ {\theta} \left(\bar {x} _ {i}\right) - f _ {\theta} \left(x _ {i}\right)\right) \| \tag {49} \\ \leq \| \tilde {W} \| _ {2} \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \| f _ {\theta} (x _ {i} ^ {+}) - f _ {\theta} (x _ {i} ^ {+ +}) \| + \frac {2 \tilde {\mathcal {R}} _ {m} (\mathcal {F})}{\sqrt {m}} + \tau \sqrt {\frac {\ln (2 / \delta)}{2 m}} + + \tau_ {\bar {S}} \sqrt {\frac {\ln (2 / \delta)}{2 n}}\right). \\ \end{array}
$$
By combining equation 45 and equation 49 using the union bound, we have that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] \tag {50} \\ \leq L _ {S} \left(w _ {S}\right) + \| \tilde {W} \| _ {2} \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \| f _ {\theta} \left(x _ {i} ^ {+}\right) - f _ {\theta} \left(x _ {i} ^ {+ +}\right) \| + \frac {2 \tilde {\mathcal {R}} _ {m} (\mathcal {F})}{\sqrt {m}} + \tau \sqrt {\frac {\ln (4 / \delta)}{2 m}} + \tau_ {\bar {S}} \sqrt {\frac {\ln (4 / \delta)}{2 n}}\right) \\ + \frac {2}{\sqrt {m}} \| \mathbf {P} _ {A} g \| _ {2} + \frac {4 \mathcal {R} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + 2 \kappa \sqrt {\frac {\ln (1 6 / \delta)}{2 m}} + 2 \kappa_ {\bar {S}} \sqrt {\frac {\ln (1 6 / \delta)}{2 n}} \\ + \kappa_ {S} \sqrt {\frac {2 \ln (8 | \mathcal {Y} | / \delta)}{2 n}} \sum_ {y \in \mathcal {Y}} \left(\sqrt {\hat {p} (y)} + \sqrt {p (y)}\right) \\ = L _ {S} \left(w _ {S}\right) + \| \tilde {W} \| _ {2} \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \| f _ {\theta} \left(x _ {i} ^ {+}\right) - f _ {\theta} \left(x _ {i} ^ {+ +}\right) \|\right) + \frac {2}{\sqrt {m}} \| \mathbf {P} _ {A} g \| _ {2} + Q _ {m, n} \\ \end{array}
$$
$$
\begin{array}{l} Q _ {m, n} = \| \tilde {W} \| _ {2} \left(\frac {2 \tilde {\mathcal {R}} _ {m} (\mathcal {F})}{\sqrt {m}} + \tau \sqrt {\frac {\ln (3 / \delta)}{2 m}} + \tau_ {\bar {S}} \sqrt {\frac {\ln (3 / \delta)}{2 n}}\right) \\ + \kappa_ {S} \sqrt {\frac {2 \ln (6 | \mathcal {Y} | / \delta)}{2 n}} \sum_ {y \in \mathcal {Y}} \left(\sqrt {\hat {p} (y)} + \sqrt {p (y)}\right) \\ + \frac {4 \mathcal {R} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + 2 \kappa \sqrt {\frac {\ln (4 / \delta)}{2 m}} + 2 \kappa_ {\bar {S}} \sqrt {\frac {\ln (4 / \delta)}{2 n}}. \\ \end{array}
$$
Define  $Z_{\bar{S}} = [f(x_1^+), \ldots, f(x_m^+)] \in \mathbb{R}^{d \times m}$ . Then, we have  $A = [Z_{\bar{S}}^\top \otimes I_r]$ . Thus,
$$
\mathbf {P} _ {A} = I - [ Z _ {\bar {S}} ^ {\top} \otimes I _ {r} ] [ Z _ {\bar {S}} Z _ {\bar {S}} ^ {\top} \otimes I _ {r} ] ^ {\dagger} [ Z _ {\bar {S}} \otimes I _ {r} ] = I - [ Z _ {\bar {S}} ^ {\top} (Z _ {\bar {S}} Z _ {\bar {S}} ^ {\top}) ^ {\dagger} Z _ {\bar {S}} \otimes I _ {r} ] = [ \mathbf {P} _ {Z _ {\bar {S}}} \otimes I _ {r} ]
$$
where  $\mathbf{P}_{Z_{\bar{S}}} = I_m - Z_{\bar{S}}^\top (Z_{\bar{S}}Z_{\bar{S}}^\top)^\dagger Z_{\bar{S}} \in \mathbb{R}^{m\times m}$ . By defining  $Y_{\bar{S}} = [g^{*}(x_{1}^{+}),\ldots ,g^{*}(x_{m}^{+})]^{\top} \in \mathbb{R}^{m\times r}$ , since  $g = \operatorname{vec}[Y_{\bar{S}}^{\top}]$ ,
$$
\left\| \mathbf {P} _ {A} g \right\| _ {2} = \left\| \left[ \mathbf {P} _ {Z _ {\bar {S}}} \otimes I _ {r} \right] \operatorname {v e c} \left[ Y _ {\bar {S}} ^ {\top} \right] \right\| _ {2} = \left\| \operatorname {v e c} \left[ Y _ {\bar {S}} ^ {\top} \mathbf {P} _ {Z _ {\bar {S}}} \right] \right\| _ {2} = \left\| \mathbf {P} _ {Z _ {\bar {S}}} Y _ {\bar {S}} \right\| _ {F} \tag {51}
$$
On the other hand, recall that  $W_{S}$  is the minimum norm solution as
$$
W _ {S} = \underset {W ^ {\prime}} {\operatorname {m i n i m i z e}} \| W ^ {\prime} \| _ {F} \text {s . t .} W ^ {\prime} \in \arg \min  _ {W} \frac {1}{n} \sum_ {i = 1} ^ {n} \| W f _ {\theta} (x _ {i}) - y _ {i} \| ^ {2}.
$$
By solving this, we have
$$
W _ {S} = Y ^ {\top} Z _ {S} ^ {\top} (Z _ {S} Z _ {S} ^ {\top}) ^ {\dagger},
$$
where  $Z_{S} = [f(x_{1}),\ldots ,f(x_{n})]\in \mathbb{R}^{d\times n}$  and  $Y_{S} = [y_{1},\dots,y_{n}]^{\top}\in \mathbb{R}^{n\times r}$ . Then,
$$
\begin{array}{l} L _ {S} (w _ {S}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \| W _ {S} f _ {\theta} (x _ {i}) - y _ {i} \| = \frac {1}{n} \sum_ {i = 1} ^ {n} \sqrt {\sum_ {k = 1} ^ {r} ((W _ {S} f _ {\theta} (x _ {i}) - y _ {i}) _ {k}) ^ {2}} \\ \leq \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {k = 1} ^ {r} \left(\left(W _ {S} f _ {\theta} \left(x _ {i}\right) - y _ {i}\right) _ {k}\right) ^ {2}} \\ = \frac {1}{\sqrt {n}} \| W _ {S} Z _ {S} - Y ^ {\top} \| _ {F} \\ = \frac {1}{\sqrt {n}} \| Y ^ {\top} \left(Z _ {S} ^ {\top} \left(Z _ {S} Z _ {S} ^ {\top}\right) ^ {\dagger} Z _ {S} - I\right) \| _ {F} \\ = \frac {1}{\sqrt {n}} \| (I - Z _ {S} ^ {\top} (Z _ {S} Z _ {S} ^ {\top}) ^ {\dagger} Z _ {S}) Y \| _ {F} \\ \end{array}
$$
$$
L _ {S} \left(w _ {S}\right) = \frac {1}{\sqrt {n}} \| \mathbf {P} _ {Z _ {S}} Y \| _ {F} \tag {52}
$$
where  $\mathbf{P}_{Z_S} = I - Z_S^\top (Z_S Z_S^\top)^\dagger Z_S$ .
By combining equation 50-equation 52 and using  $1 \leq \sqrt{2}$ , we have that with probability at least  $1 - \delta$ ,
$$
\mathbb {E} _ {X, Y} \left[ \| W _ {S} f _ {\theta} (X) - Y \| \right] \leq c I _ {\bar {S}} \left(f _ {\theta}\right) + \frac {2}{\sqrt {m}} \| \mathbf {P} _ {Z _ {\bar {S}}} Y _ {\bar {S}} \| _ {F} + \frac {1}{\sqrt {n}} \| \mathbf {P} _ {Z _ {S}} Y _ {S} \| _ {F} + Q _ {m, n}, \tag {53}
$$