SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We note that isotropic Gaussian can approximate any smooth densities in  $\mathbb{R}^e$  [41] (since RBF kernels are universal), showing that Assumption 3 is fairly general. We define a novel notion of function complexity specialized to the zero-shot setting. Intuitively, it measures how well we can fit random labels, which is first drawing  $n$  interventions and  $m$  recipients for each intervention. For examples of concrete upper bound on zero-shot Rademacher complexity see section A.4.
$$
R _ {n m} (F) = \frac {1}{n m} \mathbb {E} _ {w, x, \sigma} \sup  _ {f} \sum_ {j = 1} ^ {n} \sum_ {i = 1} ^ {m} \sigma_ {i} ^ {j} f \left(w ^ {(j)}, x _ {i} ^ {(j)}\right) \tag {9}
$$
where  $\sigma_{i}^{j}$  are independently randomly drawn from  $\{-1,1\}$ .
A.2 Formal theorem statement
Theorem 4. Under Assumptions 2, 3, with probability  $1 - \delta$ ,
$$
\begin{array}{l} L (\hat {f}) \leq L (f ^ {*}) + 8 (1 + \epsilon) R _ {n m} (\mathcal {F}) + 8 \sqrt {\frac {(1 + \epsilon) R _ {n m} (\mathcal {F}) \log (1 / \delta)}{n}} \\ + (1 + \epsilon) \sqrt {\frac {(3 2 C \beta^ {2} + \frac {2 (1 + \epsilon) ^ {2}}{m}) \log (1 / \delta)}{n}} + \frac {2 \log (1 / \delta)}{3 n}. \\ \end{array}
$$
A.3 Proof of the main theorem
We define the population loss on the noisy label  $\widetilde{L}(f) = \mathbb{E}_{w,x,\tilde{y}}(f(w,x) - \tilde{y})^2$ . Due to independence of  $\xi$ ,  $\mathbb{E}_{w,x,y,\xi}(f(w,x) - y - \xi)^2 = \mathbb{E}_{w,x,y}(f(w,x) - y)^2 + \mathbb{E}[\xi^2] = L(f) + \mathbb{E}[\xi^2]$  for any  $f$ , so  $L(\hat{f}) - L(f^*) = \widetilde{L}(\hat{f}) - \widetilde{L}(f^*)$ . We shall focus on bounding the latter.
We first need a lemma that bounds the supremum of an empirical process indexed by a bounded function class.
Lemma 5 (Theorem 2.3 of [7]). Assume that  $X_{j}$  are identically distributed according to  $P$ ,  $\mathcal{G}$  is a countable set of functions from  $\mathcal{X}$  to  $\mathbb{R}$  and, and all  $g\in \mathcal{G}$  are  $P$ -measurable, square-integrable, and satisfy  $\mathbb{E}[g] = 0$ . Suppose  $\sup_{g\in \mathcal{G}}\| g\|_{\infty}\leq 1$ , and we denote  $Z = \sup_{g}\left|\sum_{j = 1}^{n}g(X_{j})\right|$ . Suppose  $\sigma^2\geq \sup_{g\in \mathcal{G}}Var(g(X_j))$  almost surely, the for all  $t\geq 0$ , we have
$$
\Pr \left[ Z \geq \mathbb {E} Z + \sqrt {2 t (n \sigma^ {2} + 2 \mathbb {E} Z)} + \frac {t}{3} \right] \leq e ^ {- t}.
$$
We apply Lemma 5 with  $X_{j} = (w^{(j)},x_{1}^{j},\ldots ,x_{m}^{j},\tilde{y}_{1}^{j},\ldots ,\tilde{y}_{m}^{j})$ $g(X_{j}) = \left(\frac{1}{m}\sum_{i}(f(w^{(j)},x_{i}^{(j)}) - \tilde{y}_{i}^{(j)})^{2} - \widetilde{L} (f)\right),\sigma^{2} = \sup_{f\in \mathcal{F}}(Var(\frac{1}{m}\sum_{i}(f(w^{(j)},x_{i}^{(j)}) - \tilde{y}_{i}^{(j)})^{2}))$
$\dot{t} = \log (1 / \delta)$ . Since  $f - \tilde{y}\in [-1,1]$ ,  $g\in [-1,1]$ . With probability  $1 - \delta$
$$
n \sup  _ {f} \left| \widehat {L} (f) - \widetilde {L} (f) \right| \leq n \mathbb {E} \sup  _ {f} \left| \widehat {L} (f) - \widetilde {L} (f) \right| + \sqrt {2 \log \frac {1}{\delta} \left(n \sigma^ {2} + 2 n \mathbb {E} \sup  _ {f} \left| \widehat {L} (f) - \widetilde {L} (f) \right|\right)} + \frac {1}{3} \log \frac {1}{\delta}.
$$
Multiplying both sides by  $1 / n$ , and using  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$
$$
\sup_{f}\Bigl|\widehat{L} (f) - \widetilde{L} (f)\Bigr|\leq \mathbb{E}\sup_{f}\Bigl|\widehat{L} (f) - \widetilde{L} (f)\Bigr| + 2\sqrt{\frac{\mathbb{E}\sup_{f}\Bigl|\widehat{L} (f) - \widetilde{L} (f)\Bigl|\log(1 / \delta)}{n}} +\sqrt{\frac{2\sigma^{2}\log(1 / \delta)}{n}} +\frac{\log(1 / \delta)}{3n}.
$$
The next lemma bounds the variance  $\sigma^2$  in equation (10).
$$
\forall f \in \mathcal {F}, V a r _ {w ^ {(j)}, x _ {1 \ldots m} ^ {j}, \tilde {y} _ {1 \ldots m} ^ {j}} \left[ \frac {1}{m} \sum_ {i = 1} ^ {m} (f (w ^ {(j)}, x _ {i} ^ {(j)}) - \tilde {y} _ {i} ^ {(j)}) ^ {2} \right] \leq 4 (1 + \epsilon) ^ {2} C \beta^ {2} + \frac {(1 + \epsilon) ^ {4}}{4 m}.
$$
Proof of Lemma 6. Using the law of total variance, if we write
$$
g (w ^ {(j)}, x _ {1 \ldots m} ^ {j}, \tilde {y} _ {1 \ldots m} ^ {j}) = \frac {1}{m} \sum_ {i = 1} ^ {m} (f (w ^ {(j)}, x _ {i} ^ {(j)}) - \tilde {y} _ {i} ^ {(j)}) ^ {2},
$$
$$
\operatorname {V a r} [ g ] = \operatorname {V a r} _ {w} \left[ \mathbb {E} _ {x, \tilde {y}} [ g (w, x, \tilde {y}) \mid w ] \right] + \mathbb {E} _ {h} \left[ \operatorname {V a r} _ {x, \tilde {y}} [ g (w, x, \tilde {y}) \mid w ] \right] \tag {11}
$$
To bound the first term of equation (11), we use Poincaré-type inequalities in Assumption 3. For each of the example distributions, we show that they indeed satisfy Assumption 3.
Lemma 7. Each of the example distributions in Assumption 3 satisfies a Poincaré-type inequality.
- When  $P_W$  is the uniform distribution over  $\mathcal{W} \in \mathbb{R}^e$ , which is open, connected, and bounded with Lipschitz boundary, we use Poincaré-Wirtinger inequality [60] on the smooth function  $\mathbb{E}[g \mid w]$ : For some constant  $C$  that only depends on  $P_W$ ,
$$
\operatorname {V a r} _ {w} [ \mathbb {E} [ g \mid w ] ] \leq C \mathbb {E} \left[ \| \nabla_ {w} \mathbb {E} [ g \mid w ] \| _ {2} ^ {2} \right]. \tag {12}
$$
$C$  is the Poincaré constant for the domain  $\mathcal{W}$  in  $L_{2}$  norm. It can be bounded by  $1 / \lambda_{1}$  where  $\lambda_{1}$  is the first eigenvalue of the negative Laplacian of the manifold  $\mathcal{W}$  [88]. Many previous works study the optimal Poincaré constants for various domains [45]. For example, when  $w$  is uniform over  $\mathcal{W}$  which is a bounded, convex, Lipschitz domain with diameter  $d$ ,  $C \leq d / \pi$  [59].
We can apply probabilistic Poincaré inequalities over non-Lebesgue measure  $P_W$ :
- When  $w \sim \mathcal{N}(\mu, \Sigma)$ , we use the Gaussian Poincaré inequality (see e.g. Theorem 3.20 of [6] and using change of variables),
$$
\operatorname {V a r} [ F (w) ] \leq \mathbb {E} [ \langle \Sigma \nabla_ {w} F (w), \nabla_ {w} F (w) \rangle ].
$$
We apply this with  $F(w) = \mathbb{E}[g \mid w]$ . Since  $\mathbb{E}[v^\top Av] = \mathbb{E}[Tr(v^\top Av)] = \mathbb{E}[Tr(Avv^\top)] = Tr(A\mathbb{E}[vv^\top]) \leq \|A\|_2 \mathbb{E}\left[\|v\|_2^2\right]$ ,
$$
V a r _ {w} [ \mathbb {E} [ g \mid w ] ] \leq \| \Sigma \| _ {2} \mathbb {E} \left[ \| \nabla_ {w} \mathbb {E} [ g \mid w ] \| _ {2} ^ {2} \right],
$$
which satisfies equation (12) with  $C = \|\Sigma\|_2$ .
- When  $w \in \mathbb{R}^e$  has independent coordinates  $w_1, \ldots, w_e$  and each coordinate has the symmetric exponential distribution  $1/2e^{-|t|}$  for  $t \in \mathbb{R}$ , we first bound a single dimension using Lemma 4.1 of [47], which says for any function  $k \in L^1$ ,
$$
V a r (k (w _ {i})) \leq 4 \mathbb {E} \left[ k ^ {\prime} (w _ {i}) ^ {2} \right]
$$
which, combined with the Efro-Stein inequality (Theorem 3.1 of [6]),
$$
V a r (F (w)) = \mathbb {E} \sum_ {i = 1} ^ {e} V a r (F (w) \mid w _ {1}, \dots , w _ {i - 1}, w _ {i + 1}, \dots , w _ {n}),
$$
yields:
$$
V a r (F (w)) \leq 4 \mathbb {E} \left[ \| F ^ {\prime} (w) \| _ {2} ^ {2} \right]
$$
which satisfies equation (12) with  $C = 4$
Lastly, we consider the case where  $P_W$  is a mixture over base distributions satisfying Poincaré inequalities. We first consider the case where the pair-wise chi-squared distances are bounded. Next, we show that mixture of isotropic Gaussians satisfies Poincaré inequality without further condition on pair-wise chi-squared distances.
- When  $\{P_W^q\}_{q \in \mathcal{Q}}$  is a family of distributions, each satisfying Poincaré inequality with constant  $C^q$ , and  $P_W$  is any mixture over  $\{P_W^q\}_{q \in \mathcal{Q}}$  with density  $\mu$ , let  $K_P(\mu) = \operatorname{ess}_\mu \sup_q C^q$ , which is an upper bound on the base Poincaré constants almost surely, and  $K_{\chi^2}^p(\mu) = \mathbb{E}_{q,q' \sim \mu}[(1 + \chi^2(P_W^q||P_W^{q'})^p)]^{1/p}$ , which is an upper bound on the pairwise  $\chi^2$ -divergence. Using Theorem 1 of [9] we get that  $P_W$  satisfies Poincaré inequality with constant  $C$  such that  $C \leq K_P(\mu)(p^* + K_{\chi^2}^p(\mu))$  where  $p^*$  is the dual exponent of  $p$  satisfying  $1/p + 1/p^* = 1$ .
As an example, when base distributions are from the same exponential family and the natural parameter space is affine, such as mixture of Poisson or Multinomial distributions, the pair-wise chi-squared distances are bounded (under some additional conditions) and hence the mixture satisfies Poincaré inequality. More formally, let  $p_{\theta}(x) =$
$\exp \left(T(x)^{\top}\theta - F(\theta) + k(x)\right)$  where  $\theta \in \Theta$  is the natural parameter space and  $A(\theta)$  is the log partition function. Lemma 1 in [56] shows that
$$
\chi^ {2} \left(p _ {\theta_ {1}} \| p _ {\theta_ {2}}\right) = e ^ {\left(A \left(2 \theta_ {2} - \theta_ {1}\right) - \left(2 A \left(\theta_ {2}\right) - A \left(\theta_ {1}\right)\right)\right)} - 1,
$$
which is bounded as long as  $2\theta_{2} - \theta_{1}\in \Theta$ . This is satisfied for mixture of 1-D Poisson distributions which can be written as  $p(w|\lambda) = \frac{1}{w!}\exp (w\log \lambda -\lambda)$  with natural parameter space  $\mathbb{R}$ , and mixture of  $e$ -dimensional Multinomial distributions  $p(w|\pi) = \exp \left(\langle w,\log \left(\pi /\left(1 - \sum_{i = 1}^{e - 1}\pi_i\right)\right)\rangle +\log \left(1 - \sum_{i = 1}^{e - 1}\pi_i\right)\right)$  with natural parameter space  $R^{e - 1}$ . When applied to Gaussian family the natural parameters are
$$
\theta_ {q} = \left( \begin{array}{c} \Sigma_ {q} ^ {- 1} \mu_ {q} \\ v e c \left(- \frac {1}{2} \Sigma_ {q} ^ {- 1}\right) \end{array} \right).
$$
Since the covariance has to be positive definite matrices,  $2\theta_{q} - \theta_{q^{\prime}}$  may not be a set of valid natural parameter. We deal with this in the next case.
- When  $\{P_W^q\}_{q \in \mathcal{Q}}$  is a mixture of isotropic Gaussians, each with mean  $\mu_q \in \mathbb{R}^e$  and covariance  $\Sigma_q = \sigma_q^2 I_e$ , each satisfying Poincaré inequality with constant  $C^q$  (in the single-Gaussian case above we know that  $C^q \leq \sigma_q^2$ ),  $P_W$  also satisfies Poincaré inequality. We prove this via induction. The key lemma is below:
Lemma 8 (Corollary 1 of [67]). Suppose measure  $p_0$  is absolutely continuous with respect to measure  $p_1$ , and  $p_0, p_1$  satisfy Poincaré inequality with constants  $C_0, C_1$  respectively, then for all  $\alpha \in [0,1]$  and  $\beta = 1 - \alpha$ , mixture measure  $p = \alpha p_0 + \beta p_1$  satisfies Poincaré inequality with  $C \leq \max \{C_0, C_1(1 + \alpha \chi_1)\}$  where  $\chi_1 = \int \frac{dp_0}{dp_1} dp_0 - 1$ .
We sort the components in the order of non-decreasing  $\sigma_q^2$ , and add in each component one by one. For each new component  $i = 2, \ldots, |\mathcal{Q}|$ , we apply the above lemma with  $p_0$  being a mixture of  $P_W^1, \ldots, P_W^{i-1}$  and  $p_1$  being the new component  $P_W^i$ . We only need to prove that  $\chi_1$  is bounded at every step. Suppose  $p_0 = \sum_{j=1}^{i-1} \alpha_j P_W^j$  with  $\sum_{j=1}^{i-1} \alpha_j = 1$ ,  $p_1 = P_W^i$ , and  $P_W^j = \frac{1}{(2\pi)^{e/2} \sigma_j^e} \exp \left\{-\frac{1}{2}(w - \mu_j)^\top \Sigma_j^{-1}(w - \mu_j)\right\}$ . Therefore
$$
\begin{array}{l} \chi_ {1} + 1 = \int \frac {d p _ {0}}{d p _ {1}} d p _ {0} = \int_ {w} \frac {p _ {0} (w) ^ {2}}{p _ {1} (w)} d w \\ = \int_ {w} \frac {\sum_ {j = 1} ^ {i - 1} \frac {\alpha_ {j} ^ {2}}{\sigma_ {j} ^ {2 e}} \exp \left\{- \frac {\| w - \mu_ {j} \| ^ {2}}{\sigma_ {j} ^ {2}} \right\} + \sum_ {j = 1} ^ {i - 1} \sum_ {j ^ {\prime} \neq j} \frac {2 \alpha_ {j} \alpha_ {j ^ {\prime}}}{\sigma_ {j} ^ {e} \sigma_ {j ^ {\prime}} ^ {e}} \exp \left\{- \frac {\| w - \mu_ {j} \| ^ {2}}{2 \sigma_ {j} ^ {2}} - \frac {\| w - \mu_ {j ^ {\prime}} \| ^ {2}}{2 \sigma_ {j ^ {\prime}} ^ {2}} \right\}}{d w} d w \\ \end{array}
$$
The convergence condition of the above integral is  $2\sigma_{i}^{2}\geq 2\sigma_{j}^{2}$  for all  $j < i$  which is satisfied when  $\sigma_i^2\geq \sigma_j^2$
Next we observe that
$$
\nabla_ {w} \mathbb {E} [ g \mid w ] = \nabla_ {w} \int_ {x, \tilde {y}} (f (w, x) - \tilde {y}) ^ {2} p (x, \tilde {y}) d x d \tilde {y} = 2 \int_ {x, y} (f (w, x) - \tilde {y}) \frac {\partial f}{\partial w} p (x, \tilde {y}) d x d \tilde {y} = 2 \mathbb {E} \left[ (f (w, x) - \tilde {y}) \frac {\partial f}{\partial w} \right].
$$
Since  $|f(w,x) - \tilde{y} |\leq 1 + \epsilon$  almost surely,  $\mathbb{E}\left[\left\| \frac{\partial f}{\partial w}\right\| _2^2\right]\leq \beta^2$
$$
\mathbb {E} _ {h} \left[ \| \nabla_ {w} \mathbb {E} [ g \mid w ] \| _ {2} ^ {2} \right] = 4 \mathbb {E} \left[ \left\| (f (w, x) - y) \frac {\partial f}{\partial w} \right\| _ {2} ^ {2} \right] \leq 4 (1 + \epsilon) ^ {2} \beta^ {2}.
$$
$$
V a r _ {w} [ \mathbb {E} [ g \mid w ] ] \leq C \mathbb {E} \left[ \| \nabla_ {w} \mathbb {E} [ g \mid w ] \| _ {2} ^ {2} \right] \leq 4 (1 + \epsilon) ^ {2} C \beta^ {2}.
$$
To bound the second term of equation (11), we use concentration of mean of  $m$  i.i.d. random variables.
Conditioned on  $w^{(j)}$ , each of the loss  $(f(w^{(j)}, x_i^{(j)}) - \tilde{y}_i^{(j)})^2$  are i.i.d. and bounded in  $[0, (1 + \epsilon)^2]$ . Hence each variable has variance upper bound  $((1 + \epsilon)^2 - 0)^2 / 4 = (1 + \epsilon)^4 / 4$  and the mean has variance upper bound  $(1 + \epsilon)^4 / 4m$ .
Therefore  $Var[g] \leq 4(1 + \epsilon)^2 C\beta^2 + (1 + \epsilon)^4 / 4m$ .
$$
\begin{array}{l} L (\hat {f}) - L (f ^ {*}) \leq 2 \sup _ {f \in \mathcal {F}} | \widetilde {L} (f) - \hat {L} (f) | \\ \leq 2 \mathbb {E} \sup  _ {f} | \widetilde {L} (f) - \hat {L} (f) | + 4 \sqrt {\frac {\mathbb {E} \sup  _ {f} \left| \hat {L} (f) - \widetilde {L} (f) \right| \log (1 / \delta)}{n}} + \sqrt {\frac {(3 2 (1 + \epsilon) ^ {2} C \beta^ {2} + \frac {2 (1 + \epsilon) ^ {4}}{m}) \log (1 / \delta)}{n}} + \frac {2 \log (1 / \delta)}{3 n} \\ \end{array}
$$
by equation (10) and Lemma 6.
We now show that  $\mathbb{E}\sup_f|\widetilde{L} (f) - \hat{L} (f)|\leq 2(1 + \epsilon)R_{nm}(F)$ . This is similar to the argument for classical Rademacher complexity
$$
\begin{array}{l} \mathbb {E} _ {w, x, \tilde {y}} \sup _ {f} \left(\frac {1}{n m} \sum_ {i, j} (f (w ^ {(j)}, x _ {i} ^ {(j)}) - \tilde {y} _ {i} ^ {(j)}) ^ {2} - \mathbb {E} _ {w, x, \tilde {y}} (f (w ^ {(j)}, x _ {i} ^ {(j)}) - \tilde {y} _ {i} ^ {(j)}) ^ {2}\right) \\ \leq \frac {1}{n m} \mathbb {E} _ {S, S ^ {\prime}} \sup  _ {f} \left(\sum_ {i, j} [ (f (w ^ {(j)}, x _ {i} ^ {(j)}) - \tilde {y} _ {i} ^ {(j)}) ^ {2} - (f (w ^ {\prime (j)}, x _ {i} ^ {\prime (j)}) - \tilde {y} _ {i} ^ {\prime (j)}) ^ {2} ]\right) \\ = \frac {1}{n m} \mathbb {E} _ {S, S ^ {\prime}, \sigma} \sup _ {f} \left(\sum_ {i, j} [ \sigma_ {i} ^ {j} (f (w ^ {(j)}, x _ {i} ^ {(j)}) - \tilde {y} _ {i} ^ {(j)}) ^ {2} - \sigma_ {i} ^ {j} (f (w ^ {\prime (j)}, x _ {i} ^ {\prime (j)}) - \tilde {y} _ {i} ^ {\prime (j)}) ^ {2} ]\right) \\ \leq \frac {1}{n m} \mathbb {E} _ {S, \sigma} \sup _ {f} \left(\sum_ {i, j} \sigma_ {i} ^ {j} (f (w ^ {(j)}, x _ {i} ^ {(j)}) - \tilde {y} _ {i} ^ {(j)}) ^ {2}\right) + \frac {1}{n m} \mathbb {E} _ {S ^ {\prime}, \sigma} \sup _ {f} \left(\sum_ {i, j} \sigma_ {i} ^ {j} (f (w ^ {\prime (j)}, x _ {i} ^ {\prime (j)}) - \tilde {y} _ {i} ^ {\prime (j)}) ^ {2}\right) \\ = 2 R _ {n m} (\widetilde {\mathcal {L}}). \\ \end{array}
$$
where the first inequality uses Jensen's inequality and convexity of  $sup$ .
Now we prove the equivalent of Talagrand's contraction lemma to show that  $R_{nm}(\tilde{\mathcal{L}}) \leq 2R_{nm}(\mathcal{F})$ . Note that the squared loss is  $2(1 + \epsilon)$ -Lipschitz since  $\left|\frac{\partial(f - \tilde{y})^2}{\partial f}\right| = 2|f - \tilde{y}| \leq 2(1 + \epsilon)$ . We use the following lemma to prove this:
Lemma 9 (Lemma 5 of [51]). Suppose  $\{\phi_i\}$ ,  $\{\psi_i\}$ ,  $i = 1, \dots, N$  are two sets of functions on  $\Theta$  such that for each  $i$  an  $\theta, \theta' \in \Theta$ ,  $|\phi_i(\theta) - \phi_i(\theta')| \leq |\psi_i(\theta) - \psi_i(\theta')|$ . Then for all functions  $c: \Theta \to \mathbb{R}$ ,
$$
\mathbb {E} _ {\sigma} \left[ \sup  _ {\theta} \left\{c (\theta) + \sum_ {i = 1} ^ {N} \sigma_ {i} \phi_ {i} (\theta) \right\} \right] \leq \mathbb {E} _ {\sigma} \left[ \sup  _ {\theta} \left\{c (\theta) + \sum_ {i = 1} ^ {N} \sigma_ {i} \psi_ {i} (\theta) \right\} \right]
$$
For any set of  $w, x$ , we apply Lemma 9 with  $\Theta = \mathcal{F}$ ,  $\theta = f$ ,  $N = nm$ ,  $\phi_{ij}(f) = (f(w^{(j)},x_i^{(j)}) - \tilde{y}_i^{(j)})^2$ ,  $\psi_{ij}(f) = 2(1 + \epsilon)f(w^{(j)},x_i^{(j)})$ , and  $c(\theta) = 0$ . Since  $\left| (f - \tilde{y})^2 - (f' - \tilde{y})^2 \right| \leq 2(1 + \epsilon)|f - f'|$ , so the condition for Lemma 9 holds. We take expectation over  $w, x$  and divide both sides by  $nm$  to get
$$
\frac {1}{n m} \mathbb {E} _ {w, x, \sigma} \sup _ {f} \sum_ {j = 1} ^ {n} \sum_ {i = 1} ^ {m} \sigma_ {i} ^ {j} (f (w ^ {(j)}, x _ {i} ^ {(j)}) - \tilde {y} _ {i} ^ {(j)}) ^ {2} \leq \frac {2 (1 + \epsilon)}{n m} \mathbb {E} _ {w, x, \sigma} \sup _ {f} \sum_ {j = 1} ^ {n} \sum_ {i = 1} ^ {m} \sigma_ {i} ^ {j} f (w ^ {(j)}, x _ {i} ^ {(j)})
$$
which means  $R_{nm}(\mathcal{L})\leq 2(1 + \epsilon)R_{nm}(\mathcal{F})$  . Substituting this into inequality (13) finishes the proof.
A.4 Zero-shot Rademacher complexity bound for the linear hypothesis class
Consider the linear classifier  $F = \{(w_{1}^{\top}w + w_{2}^{\top}x:\| w_{1}\|_{2}\leq B,\| w_{1}\|_{2}\leq C\}$ . Suppose  $\| w\| _2\leq 1$  and  $\| x\| _2\leq 1$ .
$$
\begin{array}{l} R _ {n m} (F) = \frac {1}{n m} \mathbb {E} _ {\sigma , w, x} \sup _ {w} \left\{\langle w _ {1}, \sum_ {i j} \sigma_ {i} ^ {j} w ^ {(j)} \rangle + \langle w _ {2}, \sum_ {i j} \sigma_ {i} ^ {j} x _ {i} ^ {(j)} \rangle \right\} \\ = \frac {1}{n m} \left(B _ {1} \mathbb {E} _ {\sigma , w} \| \sum_ {i j} \sigma_ {i} ^ {j} w ^ {(j)} \| _ {2} + B _ {2} \mathbb {E} _ {\sigma , x} \| \sum_ {i j} \sigma_ {i} ^ {j} x _ {i} ^ {(j)} \| _ {2}\right) \\ \leq \frac {1}{n m} \left(B _ {1} \sqrt {m \sum_ {j} \| w ^ {(j)} \| _ {2} ^ {2}} + B _ {2} \sqrt {\sum_ {i j} \| x _ {i} ^ {(j)} \| _ {2} ^ {2}}\right) \\ = \left(B _ {1} + B _ {2}\right) / \sqrt {n m}. \\ \end{array}
$$
We observe that the bound is the same as the standard Rademacher complexity for  $nm$  independent samples, which is interesting. The relationship between standard and zero-shot Rademacher complexity for other function classes is an important future direction.
Our approach to zero-shot prediction of intervention effects is related to recent advances in heterogeneous treatment effect (HTE) estimation, zero-shot learning, and meta-learning.
B.1 Heterogenous treatment effect (HTE) estimation
Conditional average treatment effect (CATE) estimation. A number of approaches have been developed to predict the effect of an existing intervention on an individual or subgroup, based on historical data from individuals who received it. This problem is often referred to in the literature as heterogeneous treatment effect (HTE) estimation [28, 13], to denote that the goal is to detect heterogeneities in how individuals respond to an intervention. A more specific instance of HTE estimation, which we focus on here, is conditional average treatment effect (CATE) estimation [81, 44], in which the goal is to predict the effect of a treatment conditioned on an individual's features. A variety of methods and specific models have been developed to achieve this goal [28, 34, 23, 30, 81, 70, 1, 89, 26, 91, 27, 16, 14, 44, 36, 13, 3], and we refer to Bica et al. and Curth et al. for a detailed review of these methods [5, 16]. These methods estimate CATE for an existing intervention, based on historical data from individuals who received it and those that did not.
While these approaches have a number of useful applications, they do not address CATE for novel interventions which did not exist during training (zero-shot). Our primary contribution is a meta-learning framework to leverage these existing CATE estimators for zero-shot predictions. In the CaML framework (Figure 2), each task corresponds to predicting CATE for a single intervention. We synthesize a task by sampling a natural experiment for each intervention, and then use any existing CATE estimator to generate a noisy target label for our the task (Step 2: estimate pseudo-outcomes). We rely on pseudo-outcome estimates as training labels because prior work has shown that training on observed outcomes directly leads to biased CATE estimates [10, 44, 36], a result which we find holds true in our experiments as well (see T-learner and S-learner w/ meta-learning in Tables 2 and 3).
Pseudo-outcome estimators. Prior work has developed a variety of methods to estimate CATE pseudo-outcomes, which are noisy but unbiased estimates of CATE, such as the X-learner [44], R-learner [55], DR-learner [36], and RA-learner [16]. Moreover, the outputs of any other CATE estimation method, such as methods which directly estimate CATE via an end-to-end neural network [34, 70, 72] are an equally valid choice of pseudo-outcome. The literature on pseudo-outcome estimation is growing continuously as new estimators are being developed [21, 40]. Typically, these estimators are specific to a single binary intervention, for which a set of nuisance models are trained and used to compute the pseudo-outcomes. As such, applying meta-learning algorithms to these pseudo-outcomes requires synthesizing a natural experiment for each intervention, which corresponds to a single task in the CaML framework.
Multi-cause estimators. Our methods to address zero-shot CATE estimation for combinations of interventions are distinct from multi-cause estimators for combinations of binary or categorical interventions [83, 61, 65]. Recent work has shown that these methods can predict the effects of new combinations of interventions [50], when every intervention in the combination has been observed at some point during. However, these methods do not estimate CATE for novel interventions which did not exist during training. By contrast, CaML estimates CATE for zero-shot intervention combinations in which none of the interventions in the combo was ever observed during training (Appendix Table C).
B.2 Zero-shot learning
Zero-shot learning (ZSL) has traditionally aimed to reason over new concepts and classes [85, 63] which did not exist during training time. While ZSL has primarily focused on natural language processing and computer vision [82], recent interest has been sparked in generalizing over novel interventions (zero-shot) in the biomedical domain [64, 29] in which data can be cheaply collected for hundreds or thousands of possible interventions [92, 75, 19]. However, general-purpose zero-shot causal methods have been largely unexplored. Notable exceptions include GranITE [25] and SIN [25], which each extend a specific CATE estimation [55, 44] method to incorporate intervention information  $(W)$ . However, these approaches have significant drawbacks, which we discuss in Section 2.
B.3 Meta-learning
Meta-learning, or learning to learn, aims to train models which can quickly adapt to new settings and tasks. The key idea is to enable a model to gain experience over multiple learning episodes - in which episodes typically correspond to distinct tasks - to accelerate learning in subsequent learning episodes [32]. The meta-learning literature is rich and spans multiple decades [76, 68, 66, 4], with recent interest focused on model-agnostic methods to train deep learning models to quickly adapt to new tasks [20, 62, 54]. A common focus in the meta-learning literature is few-shot learning, in which a model must adapt to a new task given a small support set of labeled examples. By contrast, we focus on the zero-shot setting, in which no such support set exists. However, we hypothesize that the typical meta-learning problem formulation and training algorithms may also improve zero-shot performance. Thus, CaML's problem formulation and algorithm inspiration from the meta-learning literature, particularly the Reptile algorithm [54] and its application to other tasks in causal inference [71]. Our experimental results show that this meta-learning formulation improves CaML's performance, compared to a standard multi-task learning strategy.