SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given that our  $\lambda_{\mathrm{sm}}$  is high-quality, we suppose that  $\lambda_{\mathrm{sm}}(x_0)$  all equal 1 with high probability, and then we can lower bound our expression by
$$
\Pr \left(\lambda_ {\mathrm {s m}} = 1 | y = 1\right) \log \frac {\Pr \left(\lambda_ {\mathrm {s m}} = 1 | y = 1\right)}{\Pr \left(\lambda_ {\mathrm {s m}} = 1 | y = 1\right) p _ {\lambda} + \Pr \left(\lambda_ {\mathrm {s m}} = 1 | y = - 1\right) \left(1 - p _ {\lambda}\right)}. \tag {17}
$$
The key quantity of interest is  $\operatorname{Pr}(\lambda_{\mathsf{sm}} = 1|y = 1) = \prod_{i=1}^{m} \operatorname{Pr}(\lambda_{\mathsf{sm}},[i] = 1|y = 1)$ . We focus on bounding  $\operatorname{Pr}(\lambda_{\mathsf{sm}},[i] = 1|y = 1)$  next. Suppose that the  $k$  neighbors of  $x_0$  are  $x_1, \ldots, x_k$ . Define  $p_j = \operatorname{Pr}(\lambda_i(x_j) = 1|y = 1)$  for all  $j \in [k]$ . Note that  $\lambda_i(x_j) \perp \lambda_i(x_{j'})|y$  for any  $j, j' \in [k]$  (while  $\lambda_{\mathsf{sm}},[i]$  as a whole is dependent on  $y$ , individual neighbors are still conditionally independent). Then, the event that  $\lambda_{\mathsf{sm}},[i] = 1|y = 1$  is as least as likely as the event that  $\operatorname{Binomial}(k, \min_{i \in [k]} p_i) \geq \frac{k}{2}$ . Let  $p_{\min} = \min_{i \in [k]} p_i$ , and assume that  $p_{\min} \geq \frac{1}{2}$ . Then,
$$
\begin{array}{l} \Pr \left(\lambda_ {\mathrm {s m}}, [ i ] = 1 \mid y = 1\right) \geq \Pr \left(\operatorname {B i n o m i a l} \left(k, p _ {\min }\right) \geq \frac {k}{2}\right) = \Pr \left(\frac {1}{k} \sum_ {j = 1} ^ {k} X _ {j} \geq \frac {1}{2}\right) (18) \\ = \Pr \left(\frac {1}{k} \sum_ {j = 1} ^ {k} X _ {j} \geq p _ {\min } - \left(p _ {\min } - \frac {1}{2}\right)\right), (19) \\ \end{array}
$$
where  $X_{j}\sim \mathrm{Bernoulli}(p_{\mathrm{min}})$ . Next, let  $\delta = p_{\mathrm{min}} - \frac{1}{2}$ . We can apply Hoeffding's inequality to get
$$
\begin{array}{l} \Pr \left(\operatorname {B i n o m i a l} (k, p _ {\min }) \geq \frac {k}{2}\right) = \Pr \left(\frac {1}{k} \sum_ {j = 1} ^ {k} X _ {j} \geq p _ {\min } - \delta\right) = 1 - \Pr \left(\frac {1}{k} \sum_ {j = 1} ^ {k} X _ {j} \leq p _ {\min } - \delta\right) \geq 1 - \exp \left(- 2 \delta^ {2} k\right) (20) \\ = 1 - \exp \left(- 2 k \left(p _ {\min } - 0. 5\right) ^ {2}\right). (21) \\ \end{array}
$$
All that's left is to lower bound  $p_{\mathrm{min}}$ . Without loss of generality, suppose that  $p_{\mathrm{min}}$  corresponds to an arbitrary  $p_j = \operatorname*{Pr}(\lambda_i(x_j) = 1|y = 1)$ . We can decompose this probability into
$$
\begin{array}{l} \Pr \left(\lambda_ {i} \left(x _ {j}\right) = 1 \mid y = 1\right) = \Pr \left(\lambda_ {i} \left(x _ {j}\right) = 1, y \left(x _ {j}\right) = 1 \mid y = 1\right) + \Pr \left(\lambda_ {i} \left(x _ {j}\right) = 1, y \left(x _ {j}\right) = - 1 \mid y = 1\right) (22) \\ = \Pr \left(\lambda_ {i} \left(x _ {j}\right) = 1 \mid y \left(x _ {j}\right) = 1\right) \Pr \left(y \left(x _ {j}\right) = 1 \mid y = 1\right) + \Pr \left(\lambda_ {i} \left(x _ {j}\right) = 1 \mid y \left(x _ {j}\right) = - 1\right) \Pr \left(y \left(x _ {j}\right) = - 1 \mid y = 1\right). (23) \\ \end{array}
$$
Since  $\operatorname{Pr}(\lambda_i(x_j) = 1 | y(x_j) = 1)$  is over all  $x_j \sim \mathbb{P}_x$ , this quantity is just equal to the accuracy of  $\lambda_i$ ,  $a_i$ . Next, recall that  $\|E(x_j) - E(x)\| \leq \varepsilon_k$ , where  $\varepsilon_k = \max_{x_i \in \mathrm{NN}(x)} \|E(x) - E(x_i)\|$  is the maximum distance from the  $k$  neighbors to  $x$ . Then, we can write  $\operatorname{Pr}(y(x_j) = 1 | y = 1)$  as  $\operatorname{Pr}(y(x_j) = 1 | y = 1, \|E(x_j) - E(x)\| \leq \varepsilon_k) \geq M_E(\varepsilon_k)$ , since we have assumed that  $E$  is  $M$ -smooth. We can now bound  $p_{\min}$ :
$$
p _ {\min } \geq a _ {i} M _ {E} \left(\varepsilon_ {k}\right) + \left(1 - a _ {i}\right) \left(1 - M _ {E} \left(\varepsilon_ {k}\right)\right). \tag {24}
$$
$$
\Pr \left(\lambda_ {\mathsf {s m}} = 1 | y = 1\right) \geq \prod_ {i = 1} ^ {m} \left[ 1 - \exp \left[ - 2 k \left(a _ {i} M _ {E} \left(\varepsilon_ {k}\right) - 0. 5\right) ^ {2} \right] \right]. \tag {25}
$$
Before we plug in  $\operatorname{Pr}(\lambda_{\mathrm{sm}} = 1|y = 1)$  into (17), we simplify the expression. Note that  $\operatorname{Pr}(\lambda_{\mathrm{sm}} = 1|y = 1)$  can be written as  $\prod_{i=1}^{m} p_i$  for some  $p_i$ , and  $\operatorname{Pr}(\lambda_{\mathrm{sm}} = 1|y = -1)$  can be written as  $\prod_{i=1}^{m} (1 - p_i)$ . A simple proof by induction shows that  $\prod_{i=1}^{m} (1 - p_i) \leq 1 - \prod_{i=1}^{m} p_i$ . Therefore, we can write that (17) is lower bounded by
$$
\Pr \left(\lambda_ {\mathrm {s m}} = 1 | y = 1\right) \log \frac {\Pr \left(\lambda_ {\mathrm {s m}} = 1 | y = 1\right)}{\Pr \left(\lambda_ {\mathrm {s m}} = 1 | y = 1\right) p _ {\lambda} + \left(1 - \Pr \left(\lambda_ {\mathrm {s m}} = 1 | y = 1\right)\right) \left(1 - p _ {\lambda}\right)} \tag {26}
$$
Let's abbreviate  $\operatorname{Pr}(\lambda_{\mathfrak{sm}} = 1 | y = 1)$  as  $x$  and define the function
$$
f (x) = x \log \frac {x}{x p _ {\lambda} + (1 - x) (1 - p _ {\lambda})}. \tag {27}
$$
We note that for  $x \geq 0.5$ ,  $f(x)$  is convex and can thus be lower bounded by  $f(x) \geq f'(0.5)(x - 0.5)$ . We compute  $f'(x) = \frac{1 - p_{\lambda}}{xp_{\lambda} + (1 - x)(1 - p_{\lambda})}$ , so  $f'(0.5) = 2(1 - p_{\lambda})$ . Therefore,  $f(x) \geq 2(1 - p_{\lambda})(x - 0.5)$ . Our final bound on the pointwise difference in irreducible error on  $x_0$  is
$$
H (y | \boldsymbol {\lambda} (x _ {0})) - H (y | \boldsymbol {\lambda} (x _ {0}), \boldsymbol {\lambda} _ {\mathfrak {s m}} (x _ {0})) \geq 2 (1 - p _ {\boldsymbol {\lambda}}) \left[ \prod_ {i = 1} ^ {m} \left[ 1 - \exp \left[ - 2 k \left(a _ {i} M _ {E} (\varepsilon_ {k}) - 0. 5\right) ^ {2} \right] \right] - 0. 5 \right]. \tag {28}
$$
Information gain from using  $\lambda$ ,  $\lambda_{\mathrm{sm}}$  over  $\lambda_{\mathrm{sm}}$  We briefly comment on the opposite direction—how much does using both LLM predictions and neighborhood predictions help over just using neighborhood predictions?
The quantity we aim to lower bound is  $H(y|\lambda_{\mathrm{sm}}(x_0)) - H(y|\lambda (x_0),\lambda_{\mathrm{sm}}(x_0))$  for a point of interest  $x_0$ . We can write this quantity as
$$
H \left(y \mid \boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right)\right) - H \left(y \mid \boldsymbol {\lambda} \left(x _ {0}\right), \boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right)\right) = \mathbb {E} \left[ \log \frac {\Pr \left(\boldsymbol {\lambda} \left(x _ {0}\right) \mid y\right)}{\Pr \left(\boldsymbol {\lambda} \left(x _ {0}\right) \mid \boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right)\right)} \right] \tag {29}
$$
Without loss of generality, suppose that the true label on  $x_0$  is  $y = 1$ , and that for each  $\lambda_i$ , the neighborhood around  $x_0$  consists of a balanced mix of  $\lambda_i = 1$  and  $\lambda_i = -1$ . Then, with high probability we have that  $\operatorname*{Pr}(y = 1|\pmb{\lambda}_{\mathrm{sm}}(x_0)) = p_{\pmb{\lambda}_{\mathrm{sm}}} \approx 0.5$ . From our proof of Theorem 5.2, we can thus write
$$
\begin{array}{l} H \left(y \mid \boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right)\right) - H \left(y \mid \boldsymbol {\lambda} \left(x _ {0}\right), \boldsymbol {\lambda} _ {\mathrm {s m}} \left(x _ {0}\right)\right) = \mathbb {E} _ {\lambda \left(x _ {0}\right)} \left[ \log \frac {\Pr \left(\boldsymbol {\lambda} \left(x _ {0}\right) \mid y = 1\right)}{\Pr \left(\lambda \left(x _ {0}\right) \mid y = 1\right) p _ {\boldsymbol {\lambda} _ {\mathrm {s m}}} + \Pr \left(\lambda \left(x _ {0}\right) \mid y = - 1\right) \left(1 - p _ {\boldsymbol {\lambda} _ {\mathrm {s m}}}\right)} \right] (30) \\ \geq \Pr (\lambda (x _ {0}) = 1 | y = 1)) \log \frac {\Pr (\lambda (x _ {0}) = 1 | y = 1)}{\Pr (\lambda (x _ {0}) = 1 | y = 1) p _ {\lambda_ {\mathrm {s m}}} + \Pr (\lambda (x _ {0}) = 1 | y = - 1) (1 - p _ {\lambda_ {\mathrm {s m}}})} (31) \\ \geq 2 \left(1 - p _ {\lambda_ {s m}}\right) \left(\Pr \left(\lambda \left(x _ {0}\right) = 1 \mid y = 1\right) - 0. 5\right) (32) \\ \end{array}
$$
If  $\lambda$  on  $x_0$  has high accuracy and  $p_{\lambda_{\mathrm{sm}}}$  is low, then we can have significant point-wise information gain from modeling both  $\lambda$  and  $\lambda_{\mathrm{sm}}$  rather than just  $\lambda_{\mathrm{sm}}$ .
D Datasets
Motivation. We study the performance of our method across a diverse collection of task definitions. In our setting, a task definition denotes a specific classification that a data scientist wishes to perform. For instance, a data scientist working on quantifying the breadth of legal issues that individuals face may wish to identify which posts in an online forum refer implicate legal issues related to housing.
This evaluation strategy is motivated by the observation that task definitions vary in their smoothness across embedding spaces, as different embeddings may do a better job of capturing features relevant for the task. For instance, out-of-the-box Sentence-BERT embeddings are better than traditional BERT at capturing the topicality of a sentence [55]. By focusing on a broad range of task definitions, we can better forecast how our method might perform for new tasks that practitioners may need to create classifiers for. We also avoid issues with leakage that may arise as the practice of finetuning LLMs on tasks increases [9].
In total, we study 95 distinct task definitions, encompassing 100,418 total samples. Each task varies between 108 and 3308 samples.
Legal tasks. The emergence of LLMs is exciting for law and finance, where expert-annotations are especially difficult to acquire [4, 18, 19]. Drawing on recent benchmarks and released datasets framing the potential use cases for LLMs in law, we study the following datasets:
- CUAD [25]: The original CUAD dataset consists of 500 contracts spanning an array of sectors, with clauses manually into one of 41 legal categories. Following [18], we adapt the original dataset for clause-by-clause classification. We turn each clause type into a binary classification task, where the objective is to distinguish clauses of that type from clauses of other types (i.e. "negatives"). Negative clauses are sampled randomly so as to make the task class balanced. We ignore clauses for which there are insufficient annotations in the original dataset.
- Learned Hands [33]: The Learned Hands dataset consists of legal questions that individuals publicly posted to an online forum (r/legaladvice). The questions have been coded by experts into legal categories according to the Legal Issues Taxonomy [32]. We consider several such issue classes, and create a binary classification task for each issue. Negative clauses are sampled randomly so as to make the task class balanced. Because these questions can be long, we truncate them at 50 tokens.
Science tasks. LLMs have generated similar excitement for medical and science informatics applications [1, 4]. We study established classification/extraction benchmarks.
- Chemprot [30]: ChemProt consists of sentences from PubMed abstracts describing chemical-protein relationships. We study seven relations, and create a binary task for each one. Each task is class balanced, with negatives sampled from the other relations.
- RCT [11]: The RCT dataset consists of PubMed abstracts for papers discussing randomized control trials, where sentences in the abstract are annotated according to their semantic role (e.g., background, methods, results, etc). There are five roles, and we create a binary task for each one. Each task is class balanced, with negatives sampled from the other relations.
General domain tasks. Finally, we study the following "general domain" tasks, derived from popular sentence classification and information extraction benchmarks.
- FewRel [21]: This is a relationship classification/extraction dataset, where each sample corresponds to a sentence mentioning the relationship between two entities. We select 20 relations, and for each relation construct a binary classification task with 700 positive instances of the relation, and 700 randomly sampled sentences (corresponding to other relations).
- Spam Detection [67]: We study the YouTube spam detection task from the WRENCH benchmark. This task requires classifying YouTube comments as spam/not spam.
- Toxic content detection [5]: This task requires classifying whether posted comments are toxic or not. We use a sampled subset of the CivilComments dataset.
- AG News [69]: The original dataset organizes news snippets into four categories: World, Sports, Business, and Science/Technology. We create a separate task for each category. Negatives are sampled from the remaining classes.
- DBPedia [69]: DBPedia is a 14-way ontology classification dataset. We convert this into 14 distinct tasks, corresponding to each of the ontology types.
Table 5: Legal tasks
Task	Description/Intent	Size
Affiliate License-Licensee (CUAD)	Does the clause describe a license grant to a licensor (incl. sublicense) and the affiliates of such licensee/subLICENSE?	208
Anti-Assignment (CUAD)	Does the clause require consent or notice of a party if the contract is assigned to a third party?	1212
Audit Rights (CUAD)	Does the clause discuss potential audits?	1224
Cap On Liability (CUAD)	Does the clause specify a cap on liability upon the breach of a party's obligation?	1262
Change Of Control (CUAD)	Does the clause give one party the right to terminate if such party undergoes a change of control?	426
Competitive Restriction Exception (CUAD)	Does the clause mention exceptions or carveouts to Non-Compete, Exclusivity and No-Solicit of Customers?	226
Covenant Not To Sue (CUAD)	Does the clause mention if a party is restricted from contesting the validity of the counterparty's ownership of intellectual property?	318
Effective Date (CUAD)	Does the clause mention when the contract becomes effective?	246
Exclusivity (CUAD)	Does the clause mention an exclusive dealing commitment with the counterparty?	770
Expiration Date (CUAD)	Does the clause mentions a date when the contract's term expires?	892
Governing Law (CUAD)	Does the clause mentions which state/country's laws govern interpretation of the contract?	910
Insurance (CUAD)	Does the clause mention a requirement for insurance?	1040
Ip Ownership Assignment (CUAD)	Does the clause mention if intellectual property created by one party become the property of the counterparty?	590
Irrevocable Or Perpetual License (CUAD)	Does the clause describe a license grant that is irrevocable or perpetual?	300
Joint Ip Ownership (CUAD)	Does the clause provide for joint or shared owner-ship of intellectual property between the parties to the contract?	198
License Grant (CUAD)	Does the clause describe a license granted by one party to its counterparty?	1430
Liquidated Damages (CUAD)	Does the clause award either party liquidated damages for breach or a fee upon the termination of a contract (termination fee)?	226
Minimum Commitment (CUAD)	Does the clause specifies a minimum order size or minimum amount or units petime period that one party must buy from the counterparty?	778
Table 5 - continued from previous page
Task	Description/Intent	Size
No-Solicit Of Employees (CUAD)	Does the clause restricts a party's soliciting or hiring employees and/or contractors from the counterparty, whether during the contract or after the contract ends (or both).	150
Non-Compete (CUAD)	Does the clause restrict the ability of a party to compete with the counterparty or operate in a certain geography or business or technology sector?	450
Non-Disparagement (CUAD)	Does the clause require a party not to disparage the counterparty?	108
Non-Transferable License (CUAD)	Does the clause limit the ability of a party to transfer the license being granted to a third party?	558
Notice Period To Terminate Renewal (CUAD)	Does the clause requires a notice period to terminate renewal?	234
Post-Termination Services (CUAD)	Does the clause subject a party to obligations after the termination or expiration of a contract, including any post-termination transition, payment, transfer of IP, wind-down, last-buy, or similar commitments?	816
Renewal Term (CUAD)	Does the clause mention a renewal term for after the initial term expires?	398
Revenue-Profit Sharing (CUAD)	Does the clause require a party to share revenue or profit with the counterparty for any technology, goods, or services?	784
Rofr-Rofo-Rofn (CUAD)	Does the clause provide a party with a right of first refusal?	698
Source Code Escrow (CUAD)	Does the clause requires one party to deposit its source code into escrow with a third party, which can be released to the counterparty upon the occurrence of certain events (bankruptcy, insolvency, etc.)?	126
Termination For Convenience (CUAD)	Does the clause state that one party can terminate this contract without cause (solely by giving a notice and allowing a waiting period to expire)?	442
Uncapped Liability (CUAD)	Does the clause state that a party's liability is uncapped upon the breach of its obligation in the contract?	302
Volume Restriction (CUAD)	Does the clause describe a fee increase or consent requirement if one party's use of the product/services exceeds certain threshold?	328
Warranty Duration (CUAD)	Does the clause mentions the duration of any warranty against defects or errors in technology, products, or services provided under the contract?	326
BU (Learned Hands)	Does the text discuss issues relating to business or intellectual property?	200
CO (Learned Hands)	Does the text discuss issues relating to courts and lawyers?	194
CR (Learned Hands)	Does the text discuss issues relating to criminal issues?	644
ES (Learned Hands)	Does the text discuss issues relating to estates or wills?	182
FA (Learned Hands)	Does the text discuss issues relating to family or divorce?	794
Table 5 - continued from previous page
Task	Description/Intent	Size
HE (Learned Hands)	Does the text discuss issues relating to health?	248
HO (Learned Hands)	Does the text discuss issues relating to housing?	1270
MO (Learned Hands)	Does the text discuss issues relating to payments or debt?	740
TO (Learned Hands)	Does the text discuss issues relating to accidents or harassment?	454
TR (Learned Hands)	Does the text discuss issues relating to cars or traffic?	516
WO (Learned Hands)	Does the text discuss issues relating to employment or job?	726
Table 6: Science tasks
Task	Description/Intent	Size
Agonist (Chemprot)	Does the sentence describe an agonist relationship?	896
Antagonist (Chemprot)	Does the sentence describe an antagonist relationship?	1330
Downregulator (Chemprot)	Does the sentence describe a downregulator relationship?	3038
Part_of (Chemprot)	Does the sentence describe a part-of relationship?	1210
Regulator (Chemprot)	Does the sentence describe a regulator relationship?	2876
Substrate (Chemprot)	Does the sentence describe a substrate relationship?	2384
Upregulator (Chemprot)	Does the sentence describe an upregulator relationship?	2404
Background (RCT)	Does the sentence describe background on the study?	2000
Conclusions (RCT)	Does the sentence state a conclusion?	2000
Methods (RCT)	Does the sentence describe a scientific experimental method?	2000
Objective (RCT)	Does the sentence describe the goal of the study?	2000
Results (RCT)	Does the sentence describe experimental results?	2000
Table 7: General domain tasks
Task	Description/Intent	Size
Business (AGNews)	Does the article discuss business news?	2000
Sports (AGNews)	Does the article discuss sports news?	2000
Technology (AGNews)	Does the article discuss technology news?	2000
World (AGNews)	Does the article discuss global affairs or world events?	2000
Civil Comments	Does the sentence contain toxic or hateful content?	2500
Album (DBPedia)	Is the entity discussed in the sentence an example of a album?	2000
Animal (DBPedia)	Is the entity discussed in the sentence an example of a animal?	2000
Artist (DBPedia)	Is the entity discussed in the sentence an example of a artist?	2000
Athlete (DBPedia)	Is the entity discussed in the sentence an example of a athlete?	2000
Building (DBPedia)	Is the entity discussed in the sentence an example of a build-ing?	2000
Company (DBPedia)	Is the entity discussed in the sentence an example of a com-pany?	2000
Educational institution (DBPedia)	Does the sentence discuss a school, university, or college?	2000
Table 7 - continued from previous page
Task	Description/Intent	Size
Film (DBPedia)	Is the entity discussed in the sentence an example of a film?	2000
Mean of transportation (DBPedia)	Does the sentence discuss a car, ship, train, or plane?	2000
Natural place (DBPedia)	Is the entity discussed in the sentence an example of a natural landscape or environment?	2000
Office holder (DBPedia)	Is the entity discussed in the sentence an example of a office holder?	2000
Plant (DBPedia)	Is the entity discussed in the sentence an example of a plant?	2000
Village (DBPedia)	Is the entity discussed in the sentence an example of a village?	2000
Written work (DBPedia)	Is the entity discussed in the sentence a writing?	2000
Architect (FewRel)	Does the text mention an architect?	600
Composer (FewRel)	Does the text mention a musical composer?	600
Country (FewRel)	Does the text mention a country?	600
Developer (FewRel)	Does the text mention development?	600
Director (FewRel)	Does the text mention a film director?	600
Distributor (FewRel)	Does the text mention a film distributor?	600
Father (FewRel)	Does the text mention a father?	600
Genre (FewRel)	Does the text mention the genre of a song or artist?	600
Instrument (FewRel)	Does the text mention an instrument?	600
League (FewRel)	Does the text mention a sports competition, league, or division?	600
Military Branch (FewRel)	Does the text mention a military branch?	600
Movement (FewRel)	Does the text mention an art movement?	600
Occupation (FewRel)	Does the text mention a professional occupation?	600
Participating Team (FewRel)	Does the text mention a sports team?	600
Platform (FewRel)	Does the text mention an online platform?	600
Sibling (FewRel)	Does the text mention a sibling?	600
Successful Candidate (FewRel)	Does the text mention an election winner?	600
Taxonomy Rank (FewRel)	Does the text mention a taxonomy class of animals?	600
Tributary (FewRel)	Does the text mention a tributary?	600
Winner (FewRel)	Does the text mention a competition winner?	600
YouTube Spam Detection	Does the comment ask the user to check out another video?	1836
E Prompts
We construct a prompt for each task by randomly selecting three examples of each class to use as in-context demonstrations [51]. We manually defined "instructions" for each task. An example of an abridged prompt is shown in Figure 3.
Instruction Classification Prompt
Is the entity discussed in the sentence an example of an animal?
Sentence: " Hugh Christopher Chris Brown is a Canadian singer-songwriter and multi-instrumentalist."  
Label: No
Sentence: " Forficula auricularia the common earwig or European earwig is an omnivorous insect in the family Forficulidae. The European earwig survives in a variety of environments and is a common household insect in North America. The name earwig comes from a false superstition that these insects crawl into human ears and enter the brain; in fact they are harmless to humans."  
Label: Yes
Sentence: " Northern Dancer (May 27 1961 - November 16 1990) was a Canadian-bred Thoroughbred racehorse who won the Kentucky Derby and Preakness Stakes and who became the most successful sire of the 20th Century. The National Thoroughbred Racing Association calls him one of the most influential sires in Thoroughbred history. A bay colt Northern Dancer was by Nearctic and his dam Natalma was by Native Dancer. In 1952 Edward P." Label:
Figure 3: An example of the instruction classification prompt for the dbpedia_animal task, with two in-context demonstrations. Here, the task instructions are in red, the in-context demonstrations are in blue, and the sample for which we want a label is in green.
F Synthetics
We conduct synthetic experiments which provide additional insights on EMBROID.