SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tasks We study tasks where sentence embeddings can capture information relevant to the task, leading us to focus on sentence classification datasets. We consider a collection of 95 class-balanced sentence classification tasks, derived from binarizing existing multi-class legal, scientific, and general domain classification benchmarks like CUAD, AGNews, DBpedia-14, FewRel, and several others [21, 25, 30, 67, 69]. Example tasks include, "Classify if the following texts discuss a recording label" or "Classify if the following contractual clauses contain an audit rights provision."
2We hope to explore multi-class extensions and more complex reasoning tasks in future work.
LM	Win rate (%)	Avg. Improvement (F1)
API-Access Models	J1-Jumbo (176B)	72.2	10.6
GPT-3.5 (> 170B)	80.6	4.9
Open Source	Bloom (7.1B)	91.2	10.1
OPT (6.7B)	91.2	11.6
Instruction Tuned	GPT-JT (6B)	89.1	7.3
Table 1: We evaluate the extent to which EMBROID improves the original prompt on different models in terms of win rate and relative improvement (defined in-line). All models are run with three trials. For each model, we report the percentage of tasks (across all trials) for which EMBROID improves, and the average improvement (in F1 points). Additional details provided in Appendix.
**Choice of embedding models** Following prior work illustrating the benefits of domain specific representation [20, 71], EMBROID uses different embeddings for each task domain. For law tasks, we rely on two BERT-variants trained on different legal corpora [24, 71]. For science tasks, we rely on three BERT-variants trained on science, biology, and medical texts [3, 17, 35]. For general domain tasks, we rely on BERT, Roberta, and SentenceBert embeddings [12, 42, 55].
Prompts Prompts are constructed using fixed instructions, and by manually selecting three random samples (from each class) as in-context demonstrations (Appendix E). We follow the true few-shot regime [51], in that we assume the only labeled data available to the data scientist are the labels used for in-context demonstrations. Prior work has found this regime to most realistically represent real-world workflows.
Models We evaluate on two API-access models: GPT-3.5 (text-davinci-003) and J1-Jumbo [39]. Because API models raise significant privacy and compliance concerns for data scientists working with sensitive data [16], we also evaluate on open-source models. We select models in the 6-7B parameter range, as these are the largest models which fit on commonly available 40GB A100 machines. Specifically, we evaluate Bloom [56] and OPT [68]. Given the increasing popularity of instruction-tuning, we also evaluate on GPT-JT [60], an 6.7B parameter instruction tuned version of GPT-J. Because of cost-constraints, we evaluate API-access models on a representative selection of 12 tasks, while evaluating all other models on the full suite of 95 tasks. Appendix D provides details.
6.1 By how much does prompt-patching improve performance?
Performance across LM families We examine if EMBROID achieves improvements for different types of LMs. For each LM, we select three different combinations of in-context demonstrations (i.e., three prompts), generate predictions for each prompt, and apply EMBROID to independently each prompt's predictions. This produces  $3 \times 95 = 285$  trials for open-source models, and  $3 \times 12 = 36$  trials for API-models. We report win-rate, i.e., the proportion of trials for which EMBROID outperforms the original predictions, and improvement, i.e., the average difference in F1 points (across all trials) between EMBROID and the original predictions.
As Table 1 illustrates, EMBROID improves performance for a substantial proportion of prompts, by a substantial margin, across all models. On GPT-3.5 for instance, EMBROID achieves a win-rate of  $80.6\%$ , with an average of improvement of 4.9 points. EMBROID also improves for open source models, with a win-rate of  $91.2\%$  on OPT-6.7 and an average improvement of 11.6 points. Finally, EMBROID achieves similar gains on an instruction tuned model, with a win-rate of  $89.1\%$  and an average improvement of 7.3 points.
Performance when prompts are good We additionally investigate how EMBROID's performance improvements change as a function of the performance of the base prompt. Hypothetically, one could imagine that better performing prompts are smoother with respect to embeddings, thus diminishing (or negating) EMBROID. In Figure 2 (upper left), we plot the improvement of EMBROID against the performance of the base prompt for GPT-JT. Even when the base prompt performs well (i.e., F1 > 0.8), EMBROID improves on  $89\%$  of tasks by an average of 4.1 points.
LM	MV	Liger	FlyingSquid	AMA	EMBROID-1	EMBROID-3
J1-Jumbo	47.4	48.7	50.5	60.7	60.4	64.5
GPT-3.5	81.4	82.5	82.1	84.7	83.9	86.0
Bloom-7.1B	54.6	55.8	54.3	63.0	64.7	69.1
OPT-6.7	46.1	46.8	46.3	56.3	59.8	64.2
GPT-JT	69.3	69.4	70.1	74.6	75.1	79.0
Table 2: We evaluate how EMBROID compares to common ensemble approaches for improving prompt prediction performance. All ensemble baselines are run with three sets of predictions. EMBROID-1 is run with one set of predictions, and EMBROID-3 is run with three set of predictions. For each method, we report the average macro-F1 over all tasks. We observe that EMBROID-1 is competitive with ensemble methods which use many more predictions, while EMBROID-3 outperforms all other methods by a substantial margin.
Measuring performance in parameter count A trend in recent literature has been to measure the magnitude of improvements to prompt performance in terms of parameter count [2], by showing how a particular method makes a smaller method equivalent in performance to a larger model. We find that EMBROID enables the non-instructed tuned 1.3B GPT-Neo model to outperform an instruction tuned 6.7B model; across all trials, GPT-JT scores an average F1 of 67.8, while EMBROID +GPT-Neo-1.3B scores an average of 68.5.
6.2 Comparing prompt-patching to prompt-engineering
Our work distinguishes between prompt-construction methods—which control how a prompt is generated—and prompt-patching methods—which attempt to identify and correct errors in the predictions produced by a prompt. We use EMBROID to further study the difference between these frameworks in two ways. First, we compare EMBROID's performance improvement over a base prompt to that of several specialized prompting strategies. Second, we examine the extent to which EMBROID—when applied to the predictions produced by these prompting strategies—can generate further performance improvements. We study three prompting strategies:
1. Ensemble strategies, in which the predictions of multiple prompts are combined using an unsupervised ensembling model. Specifically, we compare to two ensembling methods previously studied for LLMs (AMA [2] and majority vote [41]), one ensembling method which incorporates embedding information (Liger [7]), and one well regarded weak supervision baseline (FlyingSquid [14]). Each baseline is run over the predictions generated by three different prompts.
2. Chain-of-thought prompting [64], in which for each in-context demonstration, we provide a step-by-step explanation for the demonstration's label.
3. Selective annotation (SA) [58], in which we use embeddings to select a subset of  $k$  data samples to label, and then, for each input sample, retrieve the most similar samples (under some embedding function) from this pool to use as in-context demonstrations.
Ensemble methods We evaluate two versions of EMBROID. In the first version, we run EMBROID with the predictions of only one prompt (EMBROID-1). In the second version, we run EMBROID with the predictions of three different prompts (EMBROID-3). Note that this requires performing inference over the few-shot LM three times for each sample, which can be expensive. This baseline is comparable to applying EMBROID to the outputs of an ensemble method. In Table 2, we observe that EMBROID-1 is competitive with the ensemble baselines (while using substantially fewer predictions), while EMBROID-3 consistently outperforms these baselines (across different LMs).
Chain-of-thought We compare EMBROID to chain-of-thought (CoT) prompting for a subset of a representative subset of 10 tasks on GPT-3.5. For each task, we manually construct a base prompt consisting of six demonstrations, and a CoT prompt where an explanation is provided for each demonstration. We first find that EMBROID's performance improvement over the base prompt exceeds that of chain-of-thought prompting (Table 3). Using EMBROID to modify the base prompt is better
Base prompt	+CoT	+EMBROID	+ CoT + EMBROID
76.3	80.1	81.9	85.4
Table 3: We evaluate EMBROID compared to, and applied to, CoT prompting on GPT-3.5 for a subset of 10 tasks. We report the average across the studied tasks.
Figure 2: Upper left: The EMBROID F1 plotted against the F1 score for the original prompt for GPT-JT. Even for high performing prompts, EMBROID is capable of improving performance. The dashed line  $y = x$  is plotted for visual aid. Upper right: A comparison of EMBROID to selective annotation (SA) over all tasks for GPT-JT. Bottom left: For each task (using GPT-JT), we plot the performance improvement of EMBROID against the average smoothness of the embeddings used. We observe a positive correlation ( $r = 0.39$ ). Bottom right: Across all tasks, we measure the performance improvement of EMBROID against the size of the task.
on average than CoT prompting, outperforming CoT on six out of the ten tasks. Second, applying EMBROID to predictions generated by a CoT prompt yields further improvements, outperforming vanilla CoT predictions on eight of ten tasks.
Selective annotation (SA) We compare EMBROID to selective annotation with a label budget of [6, 25, 50, 100] (Figure 2, upper-right). For each task, we run selective annotation using a domain specific embedding. EMBROID (applied to a prompt with randomly chosen samples) outperforms selective annotation with a label budget of 25 samples. When a label budget of 100 samples is available, EMBROID improves the performance of a prompt constructed using selective annotation on  $88\%$  of tasks, by an average of 4.3 points.
6.3 Ablations
Finally, we perform several ablations of EMBROID to study how performance changes as a function of (1) the domain specificity of the embedding used, (2) the quality of the embedding spaces used, and (3) the size of the dataset. Additional ablations are presented in Appendix G.
Domain specific embeddings improve performance We compare how performance on the legal and science tasks changes when we shift from domain specialized embeddings to general domain embeddings. On law tasks for GPT-JT, we find that using two legal embedding spaces outperforms using BERT and RoBERTa for  $77\%$  of tasks, by up to 6 points F1 on certain tasks [24, 71]. For science tasks for GPT-JT, we find that using two science embedding spaces [3, 35] outperforms using BERT and RoBERTa for  $92\%$  of tasks, by up to 4.3 points F1 on certain tasks.
**Embedding quality** Building on Section 5, we compare EMBROID's performance improvement over the base prompt to the average smoothness of the embedding spaces with respect to each task (Figure 2). We observe a positive correlation: smoother embedding spaces are associated with larger performance gains (with a Pearson coefficient of  $r = 0.39$ ). Applying this insight, we explore how performance changes when extremely high quality embeddings are added. For a subset of 19 tasks we generate OpenAI text-embedding-ada-002 embeddings, and find that adding them to EMBROID improves performance by up to 13 points F1 (at an average of 2 points across all studied tasks).
Dataset size Finally, we study how EMBROID's performance improvement changes as the dataset size changes. Because EMBROID relies on nearest-neighbors in different embedding spaces, we might expect performance to be poor when the dataset being annotated is small. In Figure 2 (bottom right), we see that EMBROID achieves performance improvements even for "small" datasets with only several hundred samples.
We study the problem of improving prompt-based learning, by developing a method (EMBROID) for detecting and correcting erroneous predictions without labeled data. We validate EMBROID across a range of datasets and LMs, finding consistent improvement in many regimes. We take a moment to address the societal impact of our work: while we do not foresee any direct harmful impacts arising from our work, we caution that any use of language models in meaningful applications should be accompanied by conversations regarding risks, benefits, stakeholder interests, and ethical safeguards.
We are grateful to Arjun Desai, Avanika Narayan, Benjamin Spector, Dilara Soylu, Gautam Machiraju, Karan Goel, Lucia Zheng, Rose E. Wang, Sabri Eyuboglu, Sarah Hooper, Simran Arora, Maya Varma, and other members of the Hazy Research group for helpful feedback and conversations on this project.
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), 1937301 (RTML) and CCF2106707; US DEVCOM ARL under No. W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1-2480: Understanding and Applying Non-Euclidean Geometry in Machine Learning; N000142012275 (NEPTUNE); NXP, Xilinx, LETICEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), the Wisconsin Alumni Research Foundation (WARF), the Center for Research on Foundation Models (CRFM), and members of the Stanford DAWN project: Facebook, Google, and VMWare. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
[70] Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models, 2021.
[71] Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pages 159-168, 2021.
Symbol	Used for
x	Input sentence or paragraph x ∈ X.
y	Binary task label y ∈ Y = {−1, +1}.
D	Unlabeled dataset D = {xi}n u i=1 of nu points.
P, P x	The joint distribution of (x, y) and the marginal on x, respectively.
nl	Number of labeled in-context examples used in querying the LLM (5-10 examples).
λLLM(·), φ(·)	Users interact with a language model λLLM via a prompt φ on x.
m	Number of prompts that we have access to.
λ	λ(x) = [λ1(x), ..., λm(x)] where λi(x) is shorthand for λLLM(φi(x)).
E	The set of N embedding models, E = {E1, ..., EN} where each embedding is represented as a fixed mapping Ei: X → Z i.
Z	Partition function for normalization of (1).
θy, θ	θy is a label balance parameter and each θi is a scalar accuracy parameter for the ith source in (1).
NNj,k(x)	The k-nearest neighbors of x in embedding space Ej.
λsm	λsm(x) = [λsm,1, ..., λsm,N] ∈ {−1, 0, 1}N m, where λsm,j = [λsm,j[1], ..., λsm,j[m]] and λsm,j[i](x) is the smoothed neighborhood prediction of λi(x) in Ej (eq. (2)).
τi+, τi-	Shrinkage parameters for determining when λsm,j[i](x) is set to 0, −1, or 1.
αj	Vector of m accuracy parameters for Ej when used with m prompts in (3).
L(λ, λsm,, D)	Generalization error of EMBROID (expected cross-entropy loss).
a max	The largest scaled accuracy of any source, a max = maxi E [λi(x)y].
b min	The smallest expected pairwise product between any two sources, b min = mini j{E [λiλj], E [λiλj]} .
ME(·)	An embedding is M-smooth if Pr(ŷ = c|y = c, ||E(x) - E(x)|| ≤ ε) ≥ ME(ε) for all c ∈ Y and any ε > 0, where ME(·) ∈ [0, 1] is decreasing in its input.
βi	The accuracy of λi, βi = Pr(λi = y).
pλ	The prediction on x0 given only access to λ, pλ = Pr(y = 1|λ(x0)).
εk	The maximum distance between x0 and its k neighbors, εk = maxi∈NNk(x) ||E(x) - E(x)||.
Table 4: Glossary of variables and symbols used in this paper.
B Weak supervision background
In this section, we provide details on the inference and learning procedures for solving the graphical model defined in equation (1). The content from this section is derived from [14] and [7].
Pseudolabel inference. To perform inference, we compute  $\hat{\mathrm{Pr}}(y|\lambda(x))$  for some  $x \in \mathcal{X}$ . This is done via Bayes' rule and the conditional independence of weak sources:
$$
\Pr (y = 1 \mid \boldsymbol {\lambda} (x)) = \frac {\prod_ {i = 1} ^ {m} \Pr \left(\lambda_ {i} (x) \mid y = 1\right) \Pr (y = 1)}{\Pr (\boldsymbol {\lambda} (x))}. \tag {5}
$$
We assume that the class balance is known; for our datasets, the class balance is  $\operatorname{Pr}(y = 1) = 0.5$ . More generally, it can be estimated [53]. The latent parameter of interest in this decomposition is  $\operatorname{Pr}(\lambda_i = 1|y = 1)$ , which corresponds to the accuracy of  $\lambda_i$ .
Algorithm 2 Triplet method [14]
Input: Dataset  $\mathcal{D}$  ,weak sources  $\lambda (x)$    
for  $i\in [m]$  do for  $j,k\in [m]\backslash i$  do Estimate  $\hat{\mathbb{E}} [\lambda_i\lambda_j]$  over  $\mathcal{D}$  , and similarly estimate  $\hat{\mathbb{E}} [\lambda_i\lambda_k]$  and  $\hat{\mathbb{E}} [\lambda_j\lambda_k]$  Compute  $\hat{a}_i^{j,k} = \sqrt{\left|\frac{\hat{\mathbb{E}}[\lambda_i\lambda_j]\hat{\mathbb{E}}[\lambda_i\lambda_k]}{\hat{\mathbb{E}}[\lambda_j\lambda_k]}\right|}$  end for Calculate average  $\hat{a}_i = \mathrm{Mean}(a_i^{j,k}\forall j,k\in [m]\backslash i)$  Compute estimated accuracy  $\hat{\mathrm{Pr}} (\lambda_i = y) = \frac{1 + \hat{a}_i}{2}$    
end for   
Output: Accuracies  $\hat{\mathrm{Pr}} (\lambda_i = y)$  for all  $i\in [m]$
Source parameter estimation: Triplet method. Previous approaches have considered how to estimate  $\operatorname{Pr}(\lambda_i = 1|y = 1)$  via the triplet method [14], which exploits conditional independence properties. First, by the properties of the graphical model in (1), it holds that the accuracy of  $\lambda_i$  is symmetric:  $\operatorname{Pr}(\lambda_i = 1|y = 1) = \operatorname{Pr}(\lambda_i = -1|y = -1) = \operatorname{Pr}(\lambda_i = y)$  (Lemma 4 of [7]). Therefore,  $\operatorname{Pr}(\lambda_i = 1|y = 1)$  can be written in terms of  $\mathbb{E}[\lambda_i y]$  with  $\mathbb{E}[\lambda_i y] = 2\operatorname{Pr}(\lambda_i = 1|y = 1) - 1$ .
Define  $a_{i} = \mathbb{E}\left[\lambda_{i}y\right]$ . The graphical model in (1) tells us that  $\lambda_{i}y \perp \lambda_{j}y$  if  $\lambda_{i} \perp \lambda_{j}|y$ , which holds for all  $i, j \in [m]$  (Proposition 1 of [14]). As a result,  $\mathbb{E}\left[\lambda_{i}y\right] \times \mathbb{E}\left[\lambda_{j}y\right] = \mathbb{E}\left[\lambda_{i}\lambda_{j}y^{2}\right] = \mathbb{E}\left[\lambda_{i}\lambda_{j}\right]$ , which is a quantity that can be computed from observed LLM predictions. That is, we have that  $a_{i}a_{j} = \mathbb{E}\left[\lambda_{i}\lambda_{j}\right]$ . If we introduce a third  $\lambda_{k}$ , we can generate a system of equations over  $a_{i}, a_{j}, a_{k}$  in terms of their pairwise rates of agreements:
$$
a _ {i} a _ {j} = \mathbb {E} \left[ \lambda_ {i} \lambda_ {j} \right] \tag {6}
$$
$$
a _ {i} a _ {k} = \mathbb {E} \left[ \lambda_ {i} \lambda_ {k} \right] \tag {7}
$$
$$
a _ {j} a _ {k} = \mathbb {E} \left[ \lambda_ {j} \lambda_ {k} \right]. \tag {8}
$$
Solving, we get that
$$
\left| a _ {i} \right| := \sqrt {\left| \frac {\mathbb {E} \left[ \lambda_ {i} \lambda_ {j} \right] \mathbb {E} \left[ \lambda_ {i} \lambda_ {k} \right]}{\mathbb {E} \left[ \lambda_ {j} \lambda_ {k} \right]} \right|}, \tag {9}
$$
and likewise for  $a_{j}, a_{k}$ . If we assume that each weak source is better than random over the dataset, then  $a_{i} = |a_{i}| > 0$ , so we can uniquely recover the accuracy of each source by selecting two other sources and computing the above expression by using empirical expectations over  $\mathcal{D}$ . We then set  $\hat{\operatorname*{Pr}} (\lambda_i = 1|y = 1) = \frac{1 + \hat{a}_i}{2}$  and plug this into the expression for  $\operatorname*{Pr}(y = 1|\lambda (x))$  in (5).
This approach is formally described in Algorithm 2.
C.1 Proof of proposition 5.1
We note that  $[\lambda, \lambda_{\mathrm{sm}}]$  can be viewed as a set of sources in the weak supervision set up used in [7, 14]. Therefore, we can apply Theorem 1 from [7] to our problem setting, noting that we do not perform their clustering step and that our predictions do not abstain and output 0 in addition to  $\{-1, 1\}$ . We have a total of  $(N + 1)m$  sources, so
$$
L (\boldsymbol {\lambda}, \boldsymbol {\lambda} _ {\mathrm {s m}}, \mathcal {D}) \leq H (y | \boldsymbol {\lambda}, \boldsymbol {\lambda} _ {\mathrm {s m}}) + \frac {3 \left(1 - b _ {\min } ^ {2}\right)}{8 b _ {\min } ^ {2} \left(1 - a _ {\max } ^ {2}\right)} \left(\frac {1}{b _ {\min } ^ {4}} + \frac {2}{b _ {\min } ^ {2}}\right) \frac {(N + 1) m}{n _ {u}} + o \left(1 / n _ {u}\right). \tag {10}
$$
C.2 Proof of theorem 5.2
We can write the change in point-wise irreducible error as follows:
$$
\begin{array}{l} H (y | \boldsymbol {\lambda} (x _ {0})) - H (y | \boldsymbol {\lambda} (x _ {0}), \boldsymbol {\lambda} _ {\mathrm {s m}} (x _ {0})) = \mathbb {E} \left[ - \log \Pr (y | \boldsymbol {\lambda} (x _ {0})) + \log \Pr (y | \boldsymbol {\lambda} (x _ {0}), \boldsymbol {\lambda} _ {\mathrm {s m}} (x _ {0})) \right] (11) \\ = \mathbb {E} \left[ \log \frac {\Pr \left(y \mid \boldsymbol {\lambda} \left(x _ {0}\right) , \boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right)\right)}{\Pr \left(y \mid \boldsymbol {\lambda} \left(x _ {0}\right)\right)} \right] (12) \\ = \mathbb {E} \left[ \log \left(\frac {\Pr \left(\boldsymbol {\lambda} \left(x _ {0}\right) , \boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right) \mid y\right) \Pr (y)}{\Pr \left(\boldsymbol {\lambda} \left(x _ {0}\right) , \boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right)\right)} \cdot \frac {\Pr \left(\boldsymbol {\lambda} \left(x _ {0}\right)\right)}{\Pr \left(\boldsymbol {\lambda} \left(x _ {0}\right) \mid y\right) \Pr (y)}\right) \right] (13) \\ = \mathbb {E} \left[ \log \frac {\Pr \left(\boldsymbol {\lambda} _ {\mathsf {s m}} \left(x _ {0}\right) \mid \boldsymbol {\lambda} \left(x _ {0}\right) , y\right)}{\Pr \left(\boldsymbol {\lambda} _ {\mathsf {s m}} \left(x _ {0}\right) \mid \boldsymbol {\lambda} \left(x _ {0}\right)\right)} \right]. (14) \\ \end{array}
$$
Next, we use the fact that  $\lambda(x_0) \perp \mathbb{1}_{\lambda_{\mathrm{sm}}(x_0)}|y$  to simplify the expression into
$$
\mathbb {E} \left[ \log \frac {\Pr \left(\boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right) \mid y\right)}{\Pr \left(\boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right) \mid y = 1\right) \Pr \left(y = 1 \mid \boldsymbol {\lambda} \left(x _ {0}\right)\right) + \Pr \left(\boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right) \mid y = - 1\right) \Pr \left(y = - 1 \mid \boldsymbol {\lambda} \left(x _ {0}\right)\right)} \right]. \tag {15}
$$
The exact  $\lambda_{\mathrm{sm}}(x_0)$  is unknown but is drawn from the distribution  $\operatorname*{Pr}(\lambda_{\mathrm{sm}}|y = 1)$  since  $x_0$  's label is 1. Then, this expression becomes an expectation over  $\lambda_{\mathrm{sm}}$ :
$$
\mathbb {E} _ {\boldsymbol {\lambda} _ {\mathrm {s m}} \mid y = 1} \left[ \log \frac {\Pr \left(\boldsymbol {\lambda} _ {\mathrm {s m}} \mid y = 1\right)}{\Pr \left(\boldsymbol {\lambda} _ {\mathrm {s m}} \mid y = 1\right) p _ {\boldsymbol {\lambda}}} + \Pr \left(\boldsymbol {\lambda} _ {\mathrm {s m}} \mid y = - 1\right) (1 - p _ {\boldsymbol {\lambda}}) \right]. \tag {16}
$$