SlowGuess/NAACL_2022_OCR huggingface dataset:
for the sentiment analysis task). If the model is truly learning the task in-context, it should generalize when label words are replaced with random English words, e.g., entailment and not_ entailment are replaced with apple and orange, thus not giving any hints about the task. In this context, we run experiments where each label word is replaced with a random word sampled from 61,569 common English words.[10] We use five seeds for sampling random words, and report the average and the worst-case performance.
Results in Table 13 show that raw LMs (the first block of the table) and models trained on the original data (the second block) achieve near random guessing performance. This indicates that having semantic hints from label words is a necessary condition for all models to perform the task.
Next, we meta-train the MT 0-shot baseline and MetaICL where, for each iteration of meta-training, we similarly map label words with random words. The mapping from the label set to sampled English words is independent for each iteration, so that the model never sees the same mapping during metatraining and hence does not overfit to a specific mapping. Results are reported in the third block of Table 13. MT 0-shot baselines are still not better than random guessing, which is expected as they have no way to grasp the meaning of each label. On the other hand, MetaICL benefits from training on the replaced data, improving performance from  $30.1\%$  to  $43.5\%$  while retaining most performance on the original data  $(43.4\% \rightarrow 40.7\%)$ .
Still, overall performance is relatively poor. We think future work should investigate the model that can in-context learn any task.
10 pypi.org/project/english-words.
Setting:  $\mathrm{HR}\rightarrow \mathrm{LR}$  Meta-train
piqa, hate_speech Offensive, google_wellformed_query, social_i_qa, circa, quoref, glue-sst2, scitail, emo, cosmos_qa, freebase_qa, ag_news, art, paws, kilt_ay2, glue-qnli, quail, ade Corpus_v2-classification, sciq, hatexplain, emotion, glue-qqp, kilt_fever, kilt_nq, dbpedia_14, kilt_zre, hellaswag, squadwith_context, hotpot_qa, glue-mnli, ropes, squad-no_context, kilt_hotpotqa, discovery, superglue-record, race-middle, race-high, lama-trex, swag, gigaword, amazon_polarity, biomrc, tabFact, tweet_eval-emoji, tweet_eval-sentiment, tweet_qa, imdb, lama-conceptnet, liar, anli, wiki_qa, kilt_trex, wikisql, wino_grande, wqa, search_qa, xsum, yahoo_answersTopics, yelp_polarity, yelp_review_full
Setting:  $\mathrm{HR}\rightarrow \mathrm{LR}$  Target
quarel, financial phrasebank, openbookqa, codah, qasc, glue-mrpc, dream, sick, commonsense_qa, medical Questions_pairs, quartz-with knowledge, poem_sentiment, quartz-no_knowledge, glue-wnli, climate_fever, ethos-national_origin, ethos-race, ethos-religion, ai2_arc, hate_speech18, glue-rtc, superglue-cb, superglue-copa, tweet_eval-hate, tweet_eval-stance_atheism, tweet_eval-stance_feminist
Setting: Classification Meta-train
Meta-Train: superglue-rte, tweet_eval-sentiment, discovery, glue-rte, superglue-wsc, glue-mrpc, tweet_eval-stance_hillary, tweet_eval-offensive, emotion, hatexplain, glue-cola, sick, paws, ethos-sexual_orientation, glue-qqp, tweet_eval-emotion, sms_spam, healthfact, glue-mnli, imdb, ethos-disability, glue-wnli, scitail, trec-finegrained, yahoo_answers_topics, liar, glue-sst2, tweet_eval-stance_abortion, circa, tweet_eval-stance_climate, glue-qnli, tweet_eval-emoji, ethosdirected_vs_generalized, ade Corpus_v2-classification, hate_speech Offensive, superglue-wic, google_wellformed_query, tweet_eval-irony, ethos-gender, onestop_english, trec, rotten_tomatoes, kilt_fever
Setting: Non-Classification Meta-train
ade Corpus v2-dosage, art, biomrc, blimp-anaphor_number_agreement, blimp-ellipsis_n_bar_2, blimp-sentential_negation_npi_licensor_present, blimp-sentential_negation_npi_scope, commonsense_qa, crowrs_pairs, dream, freebase_qa, gigaword, hellaswag, hotpot_qa, kilt_ay2, kilt_hotpotqa, kilt_trex, kilt_zsre, lama-conceptnet, lama-google_re, lama-squad, numer Sense, openbookqa, piqa, proto_qa, qa_srl, quarel, quartz-no_knowledge, race-high, ropes, sciq, social_i_qa, spider, superglue-multirc, wikisqlx, xsum, yelp_review_full
Setting: Classification Target
tive-11, 11111111111111111111111111111111111111111111111
Setting: QA Meta-train
biomrc, boolq, freebase_qa, hotpot_qa, kilt_hotpotqa, kilt_nq, kilt_trex, kilt_zsre, lama-conceptnet, lama-google_re, lama-squad, lama-trex, mc_taco, numer Sense, quref, rides, search_qa, squad-no_context, squad-with_context, superglue-multirc, superglue-record, tweet_qa, web Questions, unifiedqda:squad2, unifiedqda:naturalQuestions_with_dpr Para, unifiedqda:race_string, unifiedqda:squad1_1, unifiedqda:drop, unifiedqda:newsga, unifiedqda:narrativeqa, unifiedqda:winogrande_xl, unifiedqda:social_iqa, unifiedqda:quref, unifiedqda:physical_iqa, unifiedqda:rope, unifiedqda:commonsenseqa, unifiedqda:boolean
Setting: Non-QA Meta-train
hat speech offe, gooellormed qira, glse, scitl, eag, nws, paws, klt ay2, glue-qnli, ade_corpus_v2-classification, haxplain, emg, glqqp, klt_fever, dbpia14, glue-mnl, discovery, gigaword, amazon_polarity, tab_fact, tweet evalu-emoj, tweet evalu-offensive, tweet_evalsentiment, imdb, liar, anli, wikisql, xsum, yahoo_answersTopics, yelp_polarity, yelp_review_full
Setting: QA Target
ai2_ arc, codah, cosmos_qa, dream, hellaswag, openbookqa, qasc, quail, quarel, quartz-no_knowledge, quartz-with_knowledge, sciq, superglue-copa, swag, wino_grande, wiqa, unifiedqa:qasc, unifiedqa:qasc_with_ir, unifiedqa:openbookqa, unifiedqa:openbookqa_with_ir, unifiedqa:mctest, unifiedqa:ai2_science_middle
Setting: Non-NLI Meta-train
ade Corpus_v2-classification, ag_news, amazon_polarity, circa, climate_fever, dbpedia_14, discovery, emo, emotion, ethos-directed_vs_generalized, ethos-disability, ethos-gender, ethos-national_origin, ethos-race, ethos-religion, ethos-sexual_orientation, financial Phrasebank, glue-cola, glue-mrpc, glue-qqp, glue-sst2, google_wellformed_query, hateSpeech18, hate_speech Offensive, haxplain, health_fact, imdb, kilt_fever, liar
medical Questions pairs, onestop_english, paws, poem sentiment, rotten_tomatoes, sick, sms_spam, superglue-wsc, superglue-wsc, tab_fact, trec, trec-finegrained, tweet_eval-emoji, tweet_eval-hate, tweet_eval-irony, tweet_eval-offensive, tweet_eval-sentiment, tweet_eval-stance_abortion, tweet_eval-stance_atheism, tweet_eval-stance_climate, tweet_eval-stance_feminist, tweet_eval-stance_hillary, wiki_qa, yahoo_answers_topics, yelp_polarity Setting: NLI Target
anli, glue-mnli, glue-qnli, glue-rite, glue-wnli, scitail, sick, superglue-cb
Setting: Non-Paraphrase Meta-train
ade Corpus v2-classification, ag_news, amazon_polarity, anli, circa, climate_fever, dbpedia_14, discovery, emo, emotion, ethos-directed_vs_generalized, ethos-disability, ethos-gender, ethos-national_origin, ethos-race, ethos-religion, ethos-sexualitarianization, financial Phrasebank, glue-cola, glue-mnli, glueqnli, glue-rtc, glue-sst2, glue-wnli, google_wellformed_query, hateSpeech18, hateSpeech_offensive, hatexplain, healthfact, imdb, kilt_fever, liar, onestop_english, poem_sentiment, rotten_tomatoes, scitail, sick, sms_spam, superglue-cb, superglue-rtc, superglue-wic, superglue-wsc, tab_foot, trecfinegrained, tweet_eval-emoji, tweet_eval-emotion, tweet_eval-hate, tweet_eval-irony, tweet_eval-offensive, tweet_eval-sentiment, tweet_eval-stance_abortion, tweet_eval-stance atheism, tweet_eval-stance_climate, tweet_eval-stance_feminist, tweet_eval-stance_hillary, wiki_qa, yahoo_answers_topics, yelp_polarity
Setting: Non-Paraphrase Target
Target: glue-mrpc, glue-qqp, medical Questions_pairs, paws
Setting:  $\mathrm{HR}\rightarrow \mathrm{LR}$  Diverse Meta-train
glue-mnli, glue-qqp, glue-sst2, hate_speech_offensive, kilt_hotpotqa, kilt_zsre, lama-trex, race-high, scitail, tweet_eval-offensive, wino_grande, yahoo_answers_topics, yelp_review_full
Setting:  $\mathrm{HR}\rightarrow \mathrm{LR}$  No Diverse Meta-train
ag_news, amazon_polarity, dbpedia_14, emo, emotion, glue-sst2, imdb, tweet_eval-emoji, tweet_eval-offensive, tweet_eval-sentiment, yahoo_answers_topics, yelp_polarity, yelp_review_full
Setting:  $\mathrm{HR}\rightarrow \mathrm{LR}$  Instructions Meta-train
ag_news, amazon_polarity, anli, art, circa, cosmos_qa, dbpedia_14, discovery, emo, emotion, freebase_qa, gigaword, google_wellformed_query, hellaswag, imdb, liar, paws, piqa, quail, quoref, ropes, sciq, scitail, social_i_qa, swag, tabFact, wiki_qa, wiqa, xsum, yahoo_answers_topics, yelp_polarity, yelp_review_full
Setting: HR  $\rightarrow$  LR Instructions Target
ai2_1, climate_fever, codah, commonsense_qa, dream, financial Phrasebank, medical这些问题_pairs, openbookqa, poem_sentiment, qasc, quarel, sick
Table 14: Full datasets for all settings. The first 10 rows are for main settings described in Section 4.1; the last four rows are settings used for ablations in Section 5.2. Splits and dataname names consistent to those in Ye et al. (2021) and Khashabi et al. (2020). Bold indicates the test dataset with no overlap in domain with meta-training tasks. A prefix unifiedqa: indicates that the dataset taken is from UNIFIEDQA; otherwise, from CROSSFIT. References for all datasets are provided in Table 15.
Table 15: References for 142 datasets used in the paper. A prefix unifiedqa: indicates that the dataset taken is from UNIFIEDQA; otherwise, from CROSSFIT.
D Potential Risks
MetaICL is based on the large language model that is pretrained on a web corpus, which potentially includes harmful and biased context, despite the original authors' best efforts to mine the text. There are also potential risks in privacy and security—for instance, Carlini et al. (2021) reported that it is possible to design the attack algorithm to extract a substantial amount of training data. We thus highlight that MetaICL should be considered as a research prototype rather than a deployable system to real users, and continuing efforts are needed to reduce potential risks of the model.