SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\log \frac {\mathbb {P} _ {\phi} (p)}{\mathbb {P} _ {\phi^ {*}} (p)} \leq \sum_ {i = 1} ^ {k} \log \frac {\mathbb {P} _ {\phi} \left(x _ {i} \oplus \tilde {y} _ {i}\right)}{\mathbb {P} _ {\phi^ {*}} \left(x _ {i} \oplus \tilde {y} _ {i}\right)} + 4 k \log \frac {1}{c _ {1} c _ {2}} \tag {12}
$$
Intuitively, as each term in the sum is independent of the other, and equals (in expectation) to the Kullback-Leibler divergence between  $\phi^{\star}$  and  $\phi$ :
$$
\mathbb {E} _ {p} \left[ \frac {1}{k} \sum_ {i = 1} ^ {k} \log \frac {\mathbb {P} _ {\phi} \left(x _ {i} \oplus \tilde {y} _ {i}\right)}{\mathbb {P} _ {\phi^ {\star}} \left(x _ {i} \oplus \tilde {y} _ {i}\right)} \right] = - \mathrm {K L} \left(\mathbb {P} _ {\phi^ {\star}}, \mathbb {P} _ {\phi}\right) \tag {13}
$$
Equation 12 assures us that as long as the ground truth downstream task component  $\phi^{\star}$  distinguish enough from the rest of the mixture components,  $\frac{\mathbb{P}_{\phi}(p)}{\mathbb{P}_{\phi^{\star}}(p)}$  converge to zero. Formally, Assumption 3 assures us that
$$
\left| \log \frac {\mathbb {P} _ {\phi} \left(x _ {i} \oplus \tilde {y} _ {i}\right)}{\mathbb {P} _ {\phi^ {\star}} \left(x _ {i} \oplus \tilde {y} _ {i}\right)} \right| \leq T \log \frac {1}{c _ {2}} \tag {14}
$$
for any  $i,p,\phi ,\phi^{\star}$ . Hence, we can use Hoeffding's inequality (Hoeffding, 1994) to estimate the deviation of this variable from its expectation (see Equation 13). More specifically, together with Equation 12 we get that with probability of at least  $1 - \delta$  over the choice of the  $k$  examples in  $p$  we have that:
$$
\frac {\mathbb {P} _ {\phi} (p)}{\mathbb {P} _ {\phi^ {\star}} (p)} <   \exp \left(- \frac {k}{2} \left(\mathrm {K L} \left(\mathbb {P} _ {\phi^ {\star}}, \mathbb {P} _ {\phi}\right) - 8 \log \frac {1}{c _ {1} c _ {2}}\right)\right) \tag {15}
$$
for  $k > \frac{\left(\log\frac{1}{\delta}\right)\left(16T^2\right)\left(\log^2\frac{1}{c_2}\right)}{\mathrm{KL}\left(\mathbb{P}_{\phi^*},\mathbb{P}_\phi\right)^2}$ . Finally, Lemma 1 follows by choosing that  $m_{\tilde{\mathcal{D}}}$  is the maximum between  $k > \frac{\left(\log\frac{1}{\delta}\right)\left(16T^2\right)\left(\log^2\frac{1}{c_2}\right)}{\mathrm{KL}\left(\mathbb{P}_{\phi^*},\mathbb{P}_\phi\right)^2}$  and  $\frac{2\log\frac{1}{\epsilon}}{\triangle_{\mathrm{KL}} - 8\log\frac{1}{c_1c_2}}$ , since the first term takes care of the  $1 - \delta$  probability, and the second term term takes care of the  $\epsilon$ -approximation.
We begin by writing the difference labels likelihood explicitly. Specifically, by the definition of conditional probabilities we have that:
$$
\mathbb {P} _ {\mathcal {D}} (y \mid p \oplus x) - \mathbb {P} _ {\mathcal {D}} (\tilde {y} \mid p \oplus x) = \frac {\sum_ {\phi} \mathbb {P} _ {\mathcal {D}} (\phi) [ \mathbb {P} _ {\phi} (p \oplus x \oplus y) - \mathbb {P} _ {\phi} (p \oplus x \oplus \tilde {y}) ]}{\sum_ {\phi} \mathbb {P} _ {\mathcal {D}} (\phi) \mathbb {P} _ {\phi} (p \oplus x)} \tag {16}
$$
Now, Assumption 2 assures us that for each component  $\phi$  in the mixture the prompt  $p$  is approximately independent from the test example:
$$
c _ {1} \cdot \mathbb {P} _ {\phi} (p) \cdot \mathbb {P} _ {\phi} (x \oplus y) \leq \mathbb {P} _ {\phi} (p \oplus x \oplus y) \leq \frac {1}{c _ {1}} \cdot \mathbb {P} _ {\phi} (p) \cdot \mathbb {P} _ {\phi} (x \oplus y) \tag {17}
$$
$$
\mathbb {P} _ {\phi} (p \oplus x) \leq \frac {1}{c _ {1}} \cdot \mathbb {P} _ {\phi} (p) \cdot \mathbb {P} _ {\phi} (x) \tag {18}
$$
So substituting this inequalities in Equation 16 give us that:
$$
\mathbb {P} _ {\mathcal {D}} (y \mid p \oplus x) - \mathbb {P} _ {\mathcal {D}} (\tilde {y} \mid p \oplus x) \geq \frac {\sum_ {\phi} \mathbb {P} _ {\mathcal {D}} (\phi) \cdot \mathbb {P} _ {\phi} (p) \cdot \left[ c _ {1} ^ {2} \cdot \mathbb {P} _ {\phi} (x \oplus y) - \frac {1}{c _ {1} ^ {2}} \cdot \mathbb {P} (x \oplus \tilde {y}) \right]}{\sum_ {\phi} \mathbb {P} _ {\mathcal {D}} (\phi) \cdot \mathbb {P} _ {\phi} (p) \cdot \mathbb {P} _ {\phi} (x)} \tag {19}
$$
Now, we will separate the sums in the denominator and the numerator to term that contain the mixture component  $\phi^{\star}$  that corresponds to  $\hat{D}$ , and term that contain the other mixture components. So for clarity we will denote by  $A,B,C,D$  the different sums:
$$
A := \mathbb {P} _ {\mathcal {D}} \left(\phi^ {\star}\right) \cdot \mathbb {P} _ {\phi^ {\star}} (p) \cdot \left[ c _ {1} ^ {2} \cdot \mathbb {P} _ {\phi^ {\star}} (x \oplus y) - \frac {1}{c _ {1} ^ {2}} \cdot \mathbb {P} _ {\phi^ {\star}} (x \oplus \tilde {y}) \right] \tag {20}
$$
$$
B := \sum_ {\phi \neq \phi^ {\star}} \mathbb {P} _ {\mathcal {D}} (\phi) \cdot \mathbb {P} _ {\phi} (p) \cdot \left[ c _ {1} ^ {2} \cdot \mathbb {P} _ {\phi} (x \oplus y) - \frac {1}{c _ {1} ^ {2}} \cdot \mathbb {P} _ {\phi} (x \oplus \tilde {y}) \right] \tag {21}
$$
$$
C := \mathbb {P} _ {\mathcal {D}} (\phi^ {\star}) \cdot \mathbb {P} _ {\phi^ {\star}} (p) \cdot \mathbb {P} _ {\phi^ {\star}} (x) \tag {22}
$$
$$
D := \sum_ {\phi \neq \phi^ {\star}} \mathbb {P} _ {\mathcal {D}} (\phi) \cdot \mathbb {P} _ {\phi} (p) \cdot \mathbb {P} _ {\phi} (x) \tag {23}
$$
And get that:
$$
\mathbb {P} _ {\mathcal {D}} (y \mid p \oplus x) - \mathbb {P} _ {\mathcal {D}} (\tilde {y} \mid p \oplus x) \geq \frac {A}{C + D} + \frac {B}{C + D} \tag {24}
$$
Now we know that both  $C$  and  $D$  are non-negative and hence that:
$$
\left| \frac {A}{C + D} - \frac {A}{C} \right| = \left| \frac {A D}{C ^ {2} + D C} \right| \leq \left| \frac {A}{C} \right| \cdot \left| \frac {D}{C} \right| \tag {25}
$$
$$
\left| \frac {B}{C + D} - \frac {B}{C} \right| = \left| \frac {B D}{C ^ {2} + D C} \right| \leq \left| \frac {B}{C} \right| \cdot \left| \frac {D}{C} \right| \tag {26}
$$
And by definition we have that:
$$
\frac {A}{C} = \triangle (x, y, \tilde {y}) + \left(c _ {1} ^ {2} - 1\right) \mathbb {P} _ {\phi} (y \mid x) + \left(\frac {1}{c _ {1} ^ {2}} - 1\right) \cdot \mathbb {P} _ {\phi} (\tilde {y} \mid x) > \triangle (x, y, \tilde {y}) - 1 + c _ {1} ^ {2} \tag {27}
$$
So it is enough to show that  $\left|\frac{B}{C}\right| < \frac{1}{5} \cdot \triangle (x, y, \tilde{y})$  and that  $\left|\frac{D}{C}\right| < \frac{1}{4}$ , in order to prove that:
$$
\mathbb {P} _ {\mathcal {D}} (y \mid p \oplus x) - \mathbb {P} _ {\mathcal {D}} (\tilde {y} \mid p \oplus x) > \frac {3}{4} \cdot \frac {A}{C} - \frac {5}{4} \cdot \left| \frac {B}{C} \right| > \frac {1}{2} \cdot \triangle (x, y, \tilde {y}) + c _ {1} ^ {2} - 1 \tag {28}
$$
Starting with  $\left|\frac{B}{C}\right|$ , by the triangle inequality have that:
$$
\left| \frac {B}{C} \right| \leq \sum_ {\phi \neq \phi^ {\star}} \frac {\mathbb {P} _ {\mathcal {D}} (\phi)}{\mathbb {P} _ {\mathcal {D}} (\phi^ {\star})} \cdot \frac {\mathbb {P} _ {\phi} (p)}{\mathbb {P} _ {\phi^ {\star}} (p)} \cdot \frac {1}{c _ {1} ^ {2}} \cdot \frac {1}{\mathbb {P} _ {\phi^ {\star}} (x)} \tag {29}
$$
In addition, Assumption 3 assures us that  $\mathbb{P}_{\phi^{\star}}(x) > c_2^T$ , hence we conclude that:
$$
\left| \frac {B}{C} \right| \leq \sum_ {\phi \neq \phi^ {\star}} \frac {\mathbb {P} _ {\mathcal {D}} (\phi)}{\mathbb {P} _ {\mathcal {D}} (\phi^ {\star})} \cdot \frac {\mathbb {P} _ {\phi} (p)}{\mathbb {P} _ {\phi^ {\star}} (p)} \cdot \frac {1}{c _ {1} ^ {2} c _ {2} ^ {T}} \tag {30}
$$
Similarly, we also have that:
$$
\left| \frac {D}{C} \right| \leq \sum_ {\phi \neq \phi^ {\star}} \frac {\mathbb {P} _ {\mathcal {D}} (\phi)}{\mathbb {P} _ {\mathcal {D}} (\phi^ {\star})} \cdot \frac {\mathbb {P} _ {\phi} (p)}{\mathbb {P} _ {\phi^ {\star}} (p)} \cdot \frac {1}{c _ {2} ^ {T}} \tag {31}
$$
Now, the sum of the concepts' prior is at most one, and by Assumption 4 we have that  $\mathbb{P}_{\mathcal{D}}(\phi^{\star}) \geq c_3$  so it is enough to show that  $\frac{\mathbb{P}_{\phi}(p)}{\mathbb{P}_{\phi^*}(p)} < \frac{\triangle(x,y,\tilde{y})}{5 \cdot c_1^{-2} \cdot c_2^{-T} \cdot c_3^{-1}} =: \tilde{\epsilon}$  for any concept  $\phi \neq \phi^*$ , in order to complete the proof. Finally, Lemma 1 assures us that under the conditions of Theorem 1, there exist  $\tilde{m}_{\tilde{\mathcal{D}}}:(0,1)^2 \to \mathbb{N}$  such that if the number of in-context examples  $k$  is at least  $\tilde{m}_{\tilde{\mathcal{D}}}(\tilde{\epsilon},\delta)$ , then with probability of at least  $1 - \delta$  we get exactly that  $\frac{\mathbb{P}_{\phi}(p)}{\mathbb{P}_{\phi^*}(p)} < \tilde{\epsilon}$ . Importantly,  $\tilde{m}_{\tilde{\mathcal{D}}}$  is logarithmic in the accuracy level  $\tilde{\epsilon} = \frac{\triangle(x,y,\tilde{y})}{5 \cdot c_1^{-2} \cdot c_2^{-T} \cdot c_3^{-1}}$ , and hence polynomial in  $T$ .
C Technical lemmas
Lemma 2. Denote by  $\triangle (x,y,\tilde{y})$  the ground-truth margin between two label candidates  $y$  and  $\tilde{y}$ . And also denote by  $\Delta_{\tilde{\mathcal{D}}}$  the minimal margin between the Bayes Optimal Classifier prediction and another labels, and by  $\triangle (p,x,y,\tilde{y})$  the ground-truth margin between two label candidates  $y$  and  $\tilde{y}$  conditioned on the prompt  $p$ . Assuming that  $\Delta_{\tilde{\mathcal{D}}}$  is greater than  $1 - c_1^2$  and given the assumptions of Theorem 1, the following inequality holds:
$$
\Delta (p, x, y, \tilde {y}) > \Delta (x, y, \tilde {y}) (1 - \alpha) \cdot \alpha \tag {32}
$$
where  $\alpha := 1 - \sqrt{\frac{1 - c_1^2}{\Delta_{\mathcal{D}}}}$ .
Proof. By Theorem 1 we have that  $\Delta(p, x, y, \tilde{y})$  is larger than  $(1 - \alpha) \cdot \Delta(x, y, \tilde{y}) + c_1^2 - 1$ . Now we can substitute the definition of  $\alpha$  and get that
$$
\begin{array}{l} \Delta (p, x, y, \tilde {y}) > (1 - \alpha) \cdot \Delta (x, y, \tilde {y}) + c _ {1} ^ {2} - 1 (33) \\ = (1 - \alpha) \cdot \Delta (x, y, \tilde {y}) - \Delta_ {\tilde {D}} \cdot (1 - \alpha) ^ {2} (34) \\ \end{array}
$$
Since,  $\Delta (x,y,\tilde{y})$  is at least  $\Delta_{\tilde{D}}$  we get that:
$$
\Delta (p, x, y, \tilde {y}) > (1 - \alpha) \cdot \Delta (x, y, \tilde {y}) - \Delta (x, y, \tilde {y}) \cdot (1 - \alpha) ^ {2} \tag {35}
$$
And Equation 32 follow from algebraic manipulations.