SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As illustrated in Figure 6, we test models trained on ImageNet on two distinct domains (painting and real images) sourced from the DomainNet benchmark[21] and the result indicates that our conclusions do not change with different downstream task datasets.
A.6 Experiments on ViT as Model Backbone
To ensure our conclusions are not restricted to specific model architecture, we use ViT-B-16 model [5] as the backbone model. The results in Figure 7 show that different model backbone does not change the conclusions we derived.
(a) CIFAR10
(d) MIT67
Figure 7: Test error rate across class-to-sample ratio. ViT-B-16 pre-trained on the ImageNet dataset.
Table 3: Actual v.s. predicted test error rate on downstream tasks according to Equation 4.
N	Method	Target Dataset
CIFAR10	FGVCAircraft	Flowers102	MIT67	Stanford40	StanfordDogs
50K	Predicted	26.13±0.1	75.95±0.4	32.96±0.48	53.99±0.43	56.72±0.16	58.87±0.35
Actual	26.25±0.47	76.00±0.44	32.13±0.10	53.60±0.39	57.10±0.3	59.76±0.26
100K	Predicted	22.84±0.40	73.52±0.41	27.33±0.40	50.15±0.43	51.74±0.36	51.69±0.23
Actual	22.67±0.33	73.45±0.33	26.67±0.33	50.82±0.33	52.24±0.33	52.24±0.33
A.7 Actual v.s. Predicted Performance of Fitting Equation 4
In Section 4.2, we study the performance of the Fitting method, i.e., using the optimal number of classes derived from the function fitting Equation 4. Given the fitted function, one can also predict the test error given a specific value of the number of classes. Here, we compare the actual test error of the derived number of classes (the Fitting entry of Table 1) with the one predicted by the fitted function, in order to see whether our theory offers an accurate prediction of performance. From the results presented in Table 3, we can see that the actual test error (Actual) is similar to the test error predicted by the fitted function (Predicted), this further proves the utility of our theory.
A.8 Proofs of Theoretical Results
A.8.1 Proof of Theorem 3.1
Before we formally state the proof of Theorem 3.1, we define several notations as follows:
$$
h _ {\mathcal {P}} \triangleq \arg \min  _ {h \in \mathcal {H}} \mathcal {R} _ {p} (h, \mathcal {P}),
$$
$$
f _ {\mathcal {P}} \triangleq \arg \min _ {f} \mathbb {E} _ {i \sim \mathrm {U n i f} [ K ], x \sim \mathcal {P} _ {i}} [ \ell (f \circ h _ {\mathcal {P}} (x), i) ].
$$
Proof of Theorem 3.1. Step I. Bound  $\mathcal{E}_p(h_{S^p},\mathcal{P})$
<sup>3</sup>http://places2.csail.mit.edu/
<sup>4</sup>http://ai.bu.edu/M3SDA/
$\mathcal{E}_p(h_{S^p},\mathcal{P})$  can be decomposed into
$$
\begin{array}{l} \mathcal {E} _ {p} \left(h _ {S ^ {p}}, \mathcal {P}\right) = \mathcal {R} _ {p} \left(h _ {S ^ {p}}, \mathcal {P}\right) - \min  _ {\tilde {h} \in \mathcal {H}} \mathcal {R} _ {p} (\tilde {h}, \mathcal {P}) \\ = \left(\mathcal {R} _ {p} \left(h _ {S ^ {p}}, \mathcal {P}\right) - \frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} \left(x _ {i, j}\right), i\right)\right) \\ + \left(\frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} \left(x _ {i, j}\right), i\right) - \frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \left(\ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} \left(x _ {i, j}\right), i\right)\right)\right) \\ + \left(\frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \left(\ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} \left(x _ {i, j}\right), i\right)\right) - \mathcal {R} _ {p} \left(h _ {\mathcal {P}}, \mathcal {P}\right)\right). \\ \end{array}
$$
We tackle the three terms of the RHS of the above inequality respectively. As for the first term, since
$$
\begin{array}{l} \mathcal {R} _ {p} \left(h _ {S ^ {p}}, \mathcal {P}\right) = \min  _ {f \in \mathcal {F}} \mathbb {E} _ {i \sim \operatorname {U n i f} [ K ], x \sim \mathcal {P} _ {i}} \left[ \ell \left(f \circ h _ {S ^ {p}} (x), i\right) \right] \\ \leq \mathbb {E} _ {i \sim \operatorname {U n i f} [ K ], x: \sim \mathcal {P} _ {i}} \left[ \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} (x), i\right) \right], \\ \end{array}
$$
we have that the first term can be bounded by
$$
\begin{array}{l} \mathcal {R} _ {p} \left(h _ {S ^ {p}}, \mathcal {P}\right) - \frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} \left(x _ {i, j}\right), i\right) \\ \leq \mathbb {E} _ {i \sim \operatorname {U n i f} [ K ], x \sim \mathcal {P} _ {i}} \left[ \ell \left(f _ {S ^ {P}} \circ h _ {S ^ {P}} (x), i\right) \right] - \frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \ell \left(f _ {S ^ {P}} \circ h _ {S ^ {P}} \left(x _ {i, j}\right), i\right) \\ \leq \sup  _ {f \in \mathcal {F}, h \in \mathcal {H}} \left[ \mathbb {E} _ {i \sim \operatorname {U n i f} [ K ], x \sim \mathcal {P} _ {i}} [ \ell (f \circ h (x), i) ] - \frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \ell (f \circ h (x _ {i, j}), i) \right]. \\ \end{array}
$$
Applying Gaussian complexity to  $\frac{1}{n}\sum_{j=1}^{n}\left(\frac{1}{K}\sum_{i=1}^{K}\ell(f\circ h(x_{i,j}),i)\right)$ , we obtain that with probability at least  $1 - \delta$ , the RHS of the above inequality is smaller than
$$
\begin{array}{l} M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 n}} + 2 \mathbb {E} _ {\left((x _ {i, j}) _ {i = 1} ^ {K}\right) _ {j = 1} ^ {n} \sim \left(\Pi_ {i = 1} ^ {K} \mathcal {P} _ {i} ^ {n}\right)} \mathbb {E} _ {\left(\sigma_ {i}\right) _ {i = 1} ^ {n} \sim \mathcal {N} \left(0, \mathbb {1} _ {n \times n}\right)} \sup  _ {f \in \mathcal {F}, h \in \mathcal {H}} \frac {1}{n} \sum_ {j = 1} ^ {n} \sigma_ {j} \left(\frac {1}{K} \sum_ {i = 1} ^ {K} \ell (f \circ h (x _ {i, j}), i)\right) \\ \leq M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 n}} + \frac {2}{K} \sum_ {i = 1} ^ {K} \mathbb {E} _ {(x _ {i, j}) _ {j = 1} ^ {n}} \sim \mathcal {P} _ {i} ^ {n} \mathbb {E} _ {(\sigma_ {i}) _ {i = 1} ^ {n}} \sim \mathcal {N} (0, \mathbb {1} _ {n \times n}) \sup  _ {f \in \mathcal {F}, h \in \mathcal {H}} \frac {1}{n} \sum_ {j = 1} ^ {n} \sigma_ {j} \ell (f \circ h (x _ {i, j}), i) \\ \leq M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 n}} + \frac {2 \sqrt {2}}{K} \sum_ {i = 1} ^ {K} \mathbb {E} _ {(x _ {i, j}) _ {j = 1} ^ {n} \sim \mathcal {P} _ {i} ^ {n}} \mathbb {E} _ {(\sigma_ {j, l}) _ {j \in [ n ], l \in [ K ]} \sim \mathcal {N} (0, \mathbb {1} _ {n K})} \sup  _ {f \in \mathcal {F}, h \in \mathcal {H}} \frac {1}{n} \sum_ {j = 1} ^ {n} \sum_ {l = 1} ^ {K} \sigma_ {j, l} f _ {l} \circ h (x _ {i, j}) \\ \leq M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 n}} + 2 G \sqrt {2} \frac {1}{\sqrt {n}}. \\ \end{array}
$$
Here the second inequality is due to Slepian's Lemma, and the last inequality is due to Assumption 1. All in all, with probability at least  $1 - \delta$ , the first term can be bounded as
$$
\mathcal {R} _ {p} \left(h _ {S ^ {p}}, \mathcal {P}\right) - \frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} \left(x _ {i, j}\right), i\right) \leq M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 n}} + 2 G \sqrt {2} \frac {1}{\sqrt {n}}.
$$
Meanwhile, the second term is non-positive due to the optimality of  $f_{SP}$  and  $h_{SP}$ .
Finally, the third term can be bounded as
$$
\begin{array}{l} \left(\frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \left(\ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} \left(x _ {i, j}\right), i\right)\right) - \mathcal {R} _ {p} \left(h _ {\mathcal {P}}, \mathcal {P}\right)\right) \\ = \left(\frac {1}{n K} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {n} \left(\ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} \left(x _ {i, j}\right), i\right)\right) - \mathbb {E} _ {i \sim \operatorname {U n i f} [ K ], x \sim \mathcal {P} _ {i}} \left[ \ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} (x), i\right) \right]\right) \\ \stackrel {w. p. 1 - \delta} {\leq} M _ {\ell} \sqrt {\frac {2 \log \frac {1}{\delta}}{n}}, \\ \end{array}
$$
where the last inequality is due to Hoeffding's inequality. As a conclusion of Stage I, we obtain that with probability at least  $1 - 2\delta$
$$
\mathcal {E} _ {p} \left(h _ {S ^ {p}}, \mathcal {P}\right) \leq 5 M _ {\ell} \sqrt {\frac {\log \frac {1}{\delta}}{2 n}} + \frac {2 G \sqrt {2}}{\sqrt {n}}.
$$
Step II. Bound  $\mathcal{E}_d(h_{S^p})$ . Applying Assumption 2, we obtain that with probability at least  $1 - 2\delta$
$$
\mathcal {E} _ {d} \left(h _ {S ^ {p}}, \tilde {\mathcal {P}}\right) \leq \nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \mathcal {E} _ {p} \left(h _ {S ^ {p}}, \mathcal {P} _ {i _ {1}}, \mathcal {P} _ {i _ {2}}\right) + \nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \leq \nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \left(5 M _ {\ell} \sqrt {\frac {\log \frac {1}{\delta}}{2 n}} + \frac {2 G \sqrt {2}}{\sqrt {n}}\right) + \nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {P}).
$$
By McDiarmid's inequality, we obtain that with probability at least  $1 - \delta$
$$
\nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \leq \mathbb {E} _ {\mathcal {P} \sim \mathcal {D} ^ {K}} \nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) + M _ {0} \sqrt {\frac {\log \frac {1}{\delta}}{2 K}} \leq \nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {D}) + M _ {0} \sqrt {\frac {\log \frac {1}{\delta}}{2 K}} + \frac {C _ {0}}{\sqrt {K}}.
$$
Similarly, with probability at least  $1 - \delta$
$$
\nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \leq \mathbb {E} _ {\mathcal {P} \sim \mathcal {D} ^ {K}} \nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) + M _ {1} \sqrt {\frac {\log \frac {1}{\delta}}{2 K}} \leq \nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {D}) + M _ {1} \sqrt {\frac {\log \frac {1}{\delta}}{2 K}} + \frac {C _ {1}}{\sqrt {K}}.
$$
As a conclusion, we obtain that with probability at least  $1 - 4\delta$
$$
\begin{array}{l} \mathcal {E} _ {d} (h _ {S P}, \tilde {\mathcal {P}}) \\ \leq \left(\nu_ {1} ^ {\bar {\mathcal {P}}} (\mathcal {D}) + M _ {1} \sqrt {\frac {\log \frac {1}{\delta}}{2 K}} + \frac {C _ {1}}{\sqrt {K}}\right) \left(5 M _ {\ell} \sqrt {\frac {\log \frac {1}{\delta}}{2 n}} + \frac {2 G \sqrt {2}}{\sqrt {n}}\right) + \nu_ {0} ^ {\bar {\mathcal {P}}} (\mathcal {D}) + M _ {0} \sqrt {\frac {\log \frac {1}{\delta}}{2 K}} + \frac {C _ {0}}{\sqrt {K}}. \\ \end{array}
$$
Step III. Bound  $\bar{\mathcal{R}}_d(f_{S^d} \circ h_{S_p}, S_p) - \mathcal{R}_d(h_{S_p}, \tilde{\mathcal{P}})$  Finally, denote  $f_{\tilde{\mathcal{P}}} \triangleq \arg \min_{\tilde{f}} \min_{\tilde{f} \in \tilde{\mathcal{F}}} \mathcal{R}_d(\tilde{f} \circ h, \tilde{\mathcal{P}})$  based on Hoeffding's inequality, we obtain that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \bar {\mathcal {R}} _ {d} \left(f _ {S ^ {d}} \circ h _ {S _ {p}}, S _ {p}\right) - \mathcal {R} _ {d} \left(h _ {S _ {p}}, \tilde {\mathcal {P}}\right) \leq \mathcal {R} _ {d} \left(f _ {\tilde {\mathcal {P}}} \circ h _ {S _ {p}}, \tilde {\mathcal {P}}\right) - \mathcal {R} _ {d} \left(h _ {S _ {p}}, \tilde {\mathcal {P}}\right) \\ \leq M _ {\ell} \sqrt {\frac {2 \log \frac {1}{\delta}}{\tilde {N}}}. \\ \end{array}
$$
Meanwhile, applying Gaussian's inequality, we have that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathcal {R} _ {d} \left(f _ {S ^ {d}} \circ h _ {S _ {p}}, \tilde {\mathcal {P}}\right) - \bar {\mathcal {R}} _ {d} \left(f _ {S ^ {d}} \circ h _ {S _ {p}}, S _ {p}\right) \\ \leq 2\mathbb{E}_{(x_{i},y_{i})_{i = 1}^{\tilde{N}}}\sim \tilde{\mathcal{P}}^{\tilde{N}}\mathbb{E}_{(\sigma_{i})_{i = 1}^{\tilde{N}}}\sim \mathcal{N}(0,\mathbb{1}_{\tilde{N}\times \tilde{N}})\sup_{f\in \tilde{\mathcal{F}},h\in \mathcal{H}}\frac{1}{\tilde{N}}\sum_{i = 1}^{\tilde{N}}\sigma_{i}\ell (f\circ h(x_{i}),y_{i}) + M_{\ell}\sqrt{\frac{9\log\frac{1}{\delta}}{2\tilde{N}}} \\ \leq 2 \sqrt {2} \mathbb {E} _ {(x _ {i}, y _ {i}) _ {i = 1} ^ {\tilde {N}}} \sim \tilde {\mathcal {P}} ^ {\tilde {N}} \mathbb {E} _ {(\sigma_ {i, j}) _ {i \in [ \tilde {N} ], j \in [ \tilde {K} ]} \sim \mathcal {N} (0, \mathbf {1} _ {\tilde {N} \tilde {K} \times \tilde {N} \tilde {K}})} \sup  _ {f \in \tilde {\mathcal {F}}, h \in \mathcal {H}} \frac {1}{\tilde {N}} \sum_ {i = 1} ^ {\tilde {N}} \sum_ {j = 1} ^ {\tilde {K}} \sigma_ {i, j} f _ {j} \circ h (x _ {i}) + M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 \tilde {N}}} \\ \leq 2 G \sqrt {2} \frac {1}{\sqrt {\tilde {N}}} + M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 \tilde {N}}}, \\ \end{array}
$$
where the second inequality is due to Slepian's Lemma, and the last inequality is due to Assumption 1.
All in all, we have with probability at least  $1 - 6\delta$
$$
\begin{array}{l} \mathcal {E} _ {d} \left(f _ {S ^ {d}} \circ h _ {S _ {p}}, \hat {\mathcal {P}}\right) \\ = \mathcal {E} _ {d} \left(h _ {S ^ {p}}, \tilde {\mathcal {P}}\right) + \mathcal {R} _ {d} \left(f _ {S ^ {d}} \circ h _ {S _ {p}}, \tilde {\mathcal {P}}\right) - \bar {\mathcal {R}} _ {d} \left(f _ {S ^ {d}} \circ h _ {S _ {p}}, S _ {p}\right) + \bar {\mathcal {R}} _ {d} \left(f _ {S ^ {d}} \circ h _ {S _ {p}}, S _ {p}\right) - \mathcal {R} _ {d} \left(h _ {S _ {p}}, \tilde {\mathcal {P}}\right) \\ \leq \left(\nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {D}) + M _ {1} \sqrt {\frac {\log \frac {1}{\delta}}{2 K}} + \frac {C _ {1}}{\sqrt {K}}\right) \left(5 M _ {\ell} \sqrt {\frac {\log \frac {1}{\delta}}{2 n}} + \frac {2 G \sqrt {2}}{\sqrt {n}}\right) + \nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {D}) + M _ {0} \sqrt {\frac {\log \frac {1}{\delta}}{2 K}} + \frac {C _ {0}}{\sqrt {K}} \\ + 5 M _ {\ell} \sqrt {\frac {\log \frac {1}{\delta}}{2 \tilde {N}}} + 2 \sqrt {2} G \frac {1}{\sqrt {\tilde {N}}}. \\ \end{array}
$$
A.8.2 Proof of Theorem 3.2
$$
h_{\mathcal{P}}\triangleq \arg \min_{h\in \mathcal{H}}\mathcal{R}_{p}(h),
$$
$$
f _ {\mathcal {P}} \triangleq \arg \min  _ {f} \mathbb {E} _ {(x, y) \sim \mathcal {P}} [ \ell (f \circ h _ {\mathcal {P}} (x), y) ].
$$
Step I. Bound  $\mathcal{E}_p(h_{S^p})$
Denote  $\mathcal{E}_p(h_{S^P})$  can be decomposed into
$$
\begin{array}{l} \mathcal{E}_{p}(h_{S^{p}}) = \mathcal{R}_{p}(h_{S^{p}}) - \min_{\tilde{h}\in \mathcal{H}}\mathcal{R}_{p}(\tilde{h}) \\ = \left(\mathcal {R} _ {p} \left(h _ {S ^ {p}}\right) - \frac {1}{N} \sum_ {i = 1} ^ {N} \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} \left(x _ {i}\right), y _ {i}\right)\right) \\ + \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} \left(x _ {i}\right), y _ {i}\right) - \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} \left(x _ {i}\right), y _ {i}\right)\right)\right) \\ + \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \left(\ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} \left(x _ {i}\right), y _ {i}\right)\right) - \mathcal {R} _ {p} \left(h _ {\mathcal {P}}\right)\right). \\ \end{array}
$$
$$
\mathcal {R} _ {p} \left(h _ {S ^ {p}}\right) = \min  _ {f \in \mathcal {F}} \mathbb {E} _ {(x, y) \sim \mathcal {P}} \left[ \ell \left(f \circ h _ {S ^ {p}} (x), y\right) \right] \leq \mathbb {E} _ {(x, y) \sim \mathcal {P}} \left[ \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} (x), y\right) \right],
$$
$$
\begin{array}{l} \mathcal {R} _ {p} \left(h _ {S p}\right) - \frac {1}{N} \sum_ {i = 1} ^ {N} \ell \left(f _ {S p} \circ h _ {S p} \left(x _ {i}\right), y _ {i}\right) \\ \leq \mathbb {E} _ {(x, y) \sim \mathcal {P}} [ \ell (f _ {S ^ {p}} \circ h _ {S ^ {p}} (x), i) ] - \frac {1}{N} \sum_ {i = 1} ^ {N} \ell (f _ {S ^ {p}} \circ h _ {S ^ {p}} (x _ {i}), y _ {i}) \\ \leq \sup  _ {f \in \mathcal {F}, h \in \mathcal {H}} \left[ \mathbb {E} _ {(x, y) \sim \mathcal {P}} [ \ell (f \circ h (x), i) ] - \frac {1}{N} \sum_ {i = 1} ^ {N} \ell (f \circ h (x _ {i}), y _ {i}) \right]. \\ \end{array}
$$
Applying Gaussian complexity to  $\frac{1}{N}\sum_{i = 1}^{N}\ell (f\circ h(x_i),y_i)$ , we obtain that with probability at least  $1 - \delta$ , the RHS of the above inequality is smaller than
$$
\begin{array}{l} M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 N}} + 2 \mathbb {E} _ {(x _ {i}) _ {i = 1} ^ {N} \sim P ^ {N}} \mathbb {E} _ {(\sigma_ {i}) _ {i = 1} ^ {N} \sim \mathcal {N} (0 , \mathbb {1} _ {N})} \sup  _ {f \in \mathcal {F}, h \in \mathcal {H}} \frac {1}{N} \sum_ {i = 1} ^ {n} \sigma_ {i} \ell (f \circ h (x _ {i}), y _ {i}) \\ \leq M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 N}} + 2 \sqrt {2} \mathbb {E} _ {(x _ {i}) _ {i = 1} ^ {N} \sim P ^ {N}} \mathbb {E} _ {(\sigma_ {i, j}) _ {i \in [ N ], j \in [ \dim (\mathcal {Y}) ]} \sim \mathcal {N} (0, \mathbb {1} _ {N \dim (\mathcal {Y})})} \sup  _ {f \in \mathcal {F}, h \in \mathcal {H}} \frac {1}{N} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {K} \sigma_ {i, j} f _ {j} \circ h (x _ {i}) \\ \leq M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 N}} + 2 G \sqrt {2} \frac {1}{\sqrt {N}}. \\ \end{array}
$$
Here the first inequality is due to Slepian's lemma, and the last inequality is due to Assumption 1. All in all, with probability at least  $1 - \delta$ , the first term can be bounded as
$$
\mathcal {R} _ {p} \left(h _ {S ^ {p}}\right) - \frac {1}{N} \sum_ {i = 1} ^ {N} \ell \left(f _ {S ^ {p}} \circ h _ {S ^ {p}} \left(x _ {i}\right), y _ {i}\right) \leq M _ {\ell} \sqrt {\frac {9 \log \frac {1}{\delta}}{2 N}} + 2 G \sqrt {2} \frac {1}{\sqrt {N}}.
$$
Meanwhile, the second term is non-positive due to the optimality of  $f_{Sp}$  and  $h_{Sp}$ .
$$
\begin{array}{l} \frac {1}{N} \sum_ {i = 1} ^ {N} \ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} \left(x _ {i}\right), y _ {i}\right) - \mathcal {R} _ {p} \left(h _ {\mathcal {P}}\right) \\ = \frac {1}{N} \sum_ {i = 1} ^ {N} \ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} \left(x _ {i}\right), y _ {i}\right) - \mathbb {E} _ {(x, y) \sim \mathcal {P}} [ \ell \left(f _ {\mathcal {P}} \circ h _ {\mathcal {P}} (x), y\right) ] \\ \stackrel {w. p. 1 - \delta} {\leq} M _ {\ell} \sqrt {\frac {2 \log \frac {1}{\delta}}{N}}, \\ \end{array}
$$
$$
\mathcal {E} _ {p} \left(h _ {S ^ {p}}, \mathcal {P}\right) \leq 5 M _ {\ell} \sqrt {\frac {\log \frac {1}{\delta}}{2 N}} + \frac {2 G \sqrt {2}}{\sqrt {N}}.
$$
Step II. Bound  $\mathcal{E}_d(h_{S^p})$ . Applying Assumption 3, we obtain that with probability at least  $1 - 2\delta$
$$
\mathcal {E} _ {d} \left(h _ {S ^ {p}}, \tilde {\mathcal {P}}\right) \leq \nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \mathcal {E} _ {p} \left(h _ {S ^ {p}}\right) + \nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \leq \nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \left(5 M _ {\ell} \sqrt {\frac {\log \frac {1}{\delta}}{2 N}} + \frac {2 G \sqrt {2}}{\sqrt {N}}\right) + \nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {P}).
$$
Furthermore, as  $|\nu_0^{\tilde{\mathcal{P}}}(\mathcal{P}) - \nu_0^{\tilde{\mathcal{P}}}(\mathcal{P}_{\mathcal{X}})| \leq \frac{C_0^{\tilde{\mathcal{P}}}}{\sqrt{K}}$  and  $|\nu_1^{\tilde{\mathcal{P}}}(\mathcal{P}) - \nu_1^{\tilde{\mathcal{P}}}(\mathcal{P}_{\mathcal{X}})| \leq \frac{C_1^{\tilde{\mathcal{P}}}}{\sqrt{K}}$ , we obtain that with probability at least  $1 - 2\delta$
$$
\mathcal {E} _ {d} \left(h _ {S ^ {\mathcal {P}}}, \tilde {\mathcal {P}}\right) \leq \left(\nu_ {0} ^ {\tilde {\mathcal {P}}} \left(\mathcal {P} _ {\mathcal {X}}\right) + \frac {C _ {0} ^ {\tilde {\mathcal {P}}}}{\sqrt {K}}\right) \left(5 M _ {\ell} \sqrt {\frac {\log \frac {2}{\delta}}{2 N}} + \frac {2 G \sqrt {2}}{\sqrt {N}}\right) + \nu_ {1} ^ {\tilde {\mathcal {P}}} \left(\mathcal {P} _ {\mathcal {X}}\right) + \frac {C _ {1} ^ {\tilde {\mathcal {P}}}}{\sqrt {K}}.
$$
The rest of the proof flows exactly the same as that of Theorem 3.1.