SlowGuess/NAACL_2022_OCR huggingface dataset:
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle-moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? ArXiv, abs/2202.12837.
Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. Controllable semantic parsing via retrieval augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7683-7698, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203-5212, Online. Association for Computational Linguistics.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835-5847, Online. Association for Computational Linguistics.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3:333-389.
Chris Samarinas, Wynne Hsu, and Mong Li Lee. 2021. Improving evidence retrieval for automated explainable fact-checking. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, pages 84-91, Online. Association for Computational Linguistics.
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. 2021. A mathematical exploration of why language models help solve downstream tasks. In International Conference on Learning Representations.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Erich Schubert and Michael Gertz. 2018. Improving the cluster structure extracted from optics plots. In LWDA.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
Shuo Wang, Yichong Xu, Yuwei Fang, Yang Liu, S. Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data is more valuable than you think: A simple and effective method by retrieving from training data. ArXiv, abs/2203.08773.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183-198.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. *ArXiv*, abs/2111.02080.
Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. 2021. Human parity on commonsenseqa: Augmenting self-attention with external attention.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In ICML, pages 12697-12706.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017-5033, Online. Association for Computational Linguistics.
Distribution of the number of in-context examples Since the selection procedure for in-context examples is dynamic, the number of in-context examples differs for each test instance. In Figure 5, we plot the histogram of the number of examples we fit in  $C = 2,048$  tokens.
Effect of hyperparameters We test the effect of  $k$ , the number of prompts labeled as positive or negative, and  $L$ , the number of prompts retrieved by the unsupervised retriever. Table 8 shows that performance is generally robust w.r.t these hyperparameters.
BREAK	MTOP	SMCALFLOW
k = 1	31.5%	63.0%	54.5%
k = 5	31.9%	64.2%	54.3%
k = 10	31.0%	64.1%	52.2%
L = 50	31.9%	64.2%	54.3%
L = 100	32.3%	63.7%	51.0%
Table 8: In the LM-as-a-service setup, using GPT-Neo, we search for other values for  $L$  and  $k$ , and note that the choice of our hyperparameters is robust.
Training details To train EPR, we use the Adam optimizer (Kingma and Ba, 2015) with batch size 120 and learning rate 1e-4 on eight RTX 3090. We run training for 30 epochs. We used the default DPR hyperparameters without tuning. We used the final epoch of the model to perform model selection, and applied minimal learning rate tuning on the validation set of BREAK.
Risk assessment Large language models have been shown to exhibit various kinds of bias (Bender et al., 2021), since EPR is trained on the signal obtained from such large LMs, it might also exhibit these biases.
Additional examples Tables 9, 10, and 11 provide more examples for cases where EPR is correct while CBR is incorrect along with the top-3 prompts for each method.
Figure 5: Distribution of the number of in-context examples per test instance for each of the datasets. We mark the distribution mean using a dashed line.
EPR	CBR
Test Example	Utterance	Remind me to add 2 dozen eggs to my grocery list.
Meaning Representation	[IN:CREATE_REMINDER [SL:PERSON_REMINDED me ] [SL:TODO add 2 dozen eggs to my grocery list ] ]
Top-1	Utterance	Remind me to get two bottles of water.	Please add a grocery list to my list of things to be reminded about doing today.
Meaning Representation	[IN:CREATE_REMINDER [SL:PERSON_REMINDED me ] [SL:TODO get two bottles of water ] ]	[IN:CREATE_REMINDER [SL:TODO a grocery list ] [SL:PERSON_REMINDED my ] [SL:DATE_TIME today ] ]
Top-2	Utterance	Remind me to bring an extra pair of shoes to the river.	Remind me to make a grocery list
Meaning Representation	[IN:CREATE_REMINDER [SL:PERSON_REMINDED me ] [SL:TODO bring an extra pair of shoes to the river ] ]	[IN:CREATE_REMINDER [SL:PERSON_REMINDED me ] [SL:TODO make a grocery list ] ]
Top-3	Utterance	Remind me to add bottled water to grocery list.	I need to make a grocery list; will you remind me when I get off work at 5:00 p.m.?
Meaning Representation	[IN:CREATE_REMINDER [SL:PERSON_REMINDED me ] [SL:TODO add bottled water to grocery list ] ]	[IN:CREATE_REMINDER [SL:TODO make a grocery list ] [SL:PERSON_REMINDED me ] [SL:DATE_TIME at 5 : 00 p.m . ] ]
Table 9: An example from MTOP development set where EPR is correct and CBR is incorrect along with the top-3 training examples retrieved from each retriever.
EPR	CBR
Test Example	Utterance	confirmed thanks
Meaning Representation	(PleasantryAnythingElseCombined)
Top-1	Utterance	it's ok bye	Yes, but make sure to let me know the weather for that time.
Meaning Representation	(PleasantryAnythingElseCombined)	(let (x0 (Execute (^Dynamic) ConfirmAndReturnAction))) (do (Yield x0) (Yield (WeatherForEvent (^Dynamic) item x0))))
Top-2	Utterance	It's ok	Awesome, perfect
Meaning Representation	(PleasantryAnythingElseCombined)	(Yield (Execute (^Dynamic) ConfirmAndReturnAction)))
Top-3	Utterance	It's ok	Perfect...
Meaning Representation	(PleasantryAnythingElseCombined)	(Yield (Execute (^Dynamic) ConfirmAndReturnAction)))
Table 10: An example from SMCALFLOW development set where EPR is correct and CBR is incorrect along with the top-3 training examples retrieved from each retriever.
EPR	CBR
Test Example	Utterance	Create a meeting with David Crim today
Meaning Representation	(Yield (CreateCommitEventWrapper)(& (Event.start_? (DateTime.date_? (?= (Today)))) (Eventattendees_?(AttendeeListHasRecipient Execute (refer (extensionConstraint (RecipientWithNameLike (^(Recipient) EmptyStructConstraint) (PersonName.apply "David Crim"))))))))
Top-1	Utterance	make a meeting with jeri today	set up a meeting with both of David Crim's reports today
Meaning Representation	(Yield (CreateCommitEventWrapper)(CreatePreflightEventWrapper (& (Event.start_? (DateTime.date_? (?= (Today)))) (Eventattendees_?(AttendeeListHasRecipient Execute (refer (extensionConstraint (RecipientWithNameLike (^(Recipient) EmptyStructConstraint) (PersonName.apply "jeri"))))))))	(Yield (CreateCommitEventWrapper)(CreatePreflightEventWrapper (& (Event.start_? (DateTime.date_? (?= (Today)))) (Eventattendees_?(AttendeeListHasPeople (FindReports Execute (refer (extensionConstraint (RecipientWithNameLike (^(Recipient) EmptyStructConstraint) (PersonName.apply "David Crim"))))))))
Top-2	Utterance	put meeting with emlime on today	Make a meeting with David Largenstop on the 24th.
Meaning Representation	(Yield (CreateCommitEventWrapper)(CreatePreflightEventWrapper (& (Event.start_? (DateTime.date_? (?= (Today)))) (Eventattendees_?(AttendeeListHasRecipient Execute (refer (extensionConstraint (RecipientWithNameLike (^(Recipient) EmptyStructConstraint) (PersonName.apply "emlime"))))))))	(Yield (CreateCommitEventWrapper)(CreatePreflightEventWrapper (& (Event.start_? (DateTime.date_? (?= (nextDayOfMonth (Today) 24L)))) (Eventattendees_?(AttendeeListHasRecipient Execute (refer (extensionConstraint (RecipientWithNameLike (^(Recipient) EmptyStructConstraint) (PersonName.apply "David Largenstop"))))))))
Top-3	Utterance	I want meet Dr Kennedy from today	create a meet with bob today
Meaning Representation	(Yield (CreateCommitEventWrapper)(CreatePreflightEventWrapper (& (Event.start_? (DateTime.date_? (?= (Today)))) (Eventattendees_?(AttendeeListHasRecipient Execute (refer (extensionConstraint (RecipientWithNameLike (^(Recipient) EmptyStructConstraint) (PersonName.apply "Dr Kennedy"))))))))	(Yield (CreateCommitEventWrapper)(CreatePreflightEventWrapper (& (Event.start_? (DateTime.date_? (?= (Today)))) (Eventattendees_?(AttendeeListHasRecipient Execute (refer (extensionConstraint (RecipientWithNameLike (^(Recipient) EmptyStructConstraint) (PersonName.apply "bob"))))))))
Table 11: An example from SMCALFLOW development set where EPR is correct and CBR is incorrect along with the top-3 training examples retrieved from each retriever.
Utterance	Meaning Representation
which 3 seas border philippines?	1#) return the philippines
2#) return seas that border #1
what three seas surround philippines?	1#) return seas
2#) return #1 that surround the philippines
what states does west virginia border?	1#) return west virginia
2#) return border states of #1
what states borders west virginia?	1#) return west virginia
2#) return border states of #1
which states border colorado	1#) return states
2#) return #1 that border colorado
Table 12: Example of a cluster from the t-SNE projection of EPR on BREAK.
Utterance	Meaning Representation
List the total scores of body builders in ascending order.	1#) return body builders
2#) return scores of #1
3#) return sum of #2 for each #1
4#) return #3 sorted by ascending order
What are the names of body builders in descending order of total scores?	1#) return body builders
2#) return names of #1
3#) return scores of #1
4#) return sum of #3 for each #1
5#) return #2 sorted by #4 in descending order
List the total points of gymnasts in descending order.	1#) return gymnasts
2#) return points of #1
3#) return sum of #2 for each #1
4#) return #3 sorted by descending order
What are the total points for all gymnasts, ordered by total points descending?	1#) return gymnasts
2#) return total points for all #1
3#) return #2 ordered by total points descending
List the total points of gymnasts in descending order of floor exercise points.	1#) return gymnasts
2#) return points of #1
3#) return sum of #2 for each #1
4#) return floor exercise points of #1
5#) return #3 sorted by #4 in descending order
Table 13: Example of a cluster from the t-SNE projection of EPR on BREAK.
Utterance	Meaning Representation
Show the locations that have both performances with more than 2000 attendees and performances with less than 1000 attendees.	1(#) return performances
2(#) return attendees of #1
3(#) return the number of #2 for each #1
4(#) return #1 where #3 is more than 2000
5(#) return #1 where #3 is less than 1000
6(#) return the locations of #4
7(#) return the locations of #5
8(#) return the locations in both #6 and #7
Show the theme for exhibitions with both records of an attendance below 100 and above 500.	1(#) return exhibitions
2(#) return attendances of #1
3(#) return number of #2 for each #1
4(#) return #1 where #3 is below 100
5(#) return #1 where #3 is above 500
6(#) return #1 of both #4 and #5
7(#) return themes for #6
Which themes have had corresponding exhibitions that have had attendance both below 100 and above 500?	1(#) return themes
2(#) return exhibitions with #1
3(#) return attendances of #2
4(#) return #1 where #3 is lower than 100
5(#) return #1 where #3 is higher than 500
6(#) return #1 of both #4 and #5
Show the publishers that have publications with price higher than 1000000 and publications with price lower than 5000000.	1(#) return publishers
2(#) return publications of #1
3(#) return prices of #2
4(#) return #1 where #3 is higher than 10000000
5(#) return #1 where #3 is lower than 5000000
6(#) return #1 of both #4 and #5
Show the famous titles of the artists with both volumes that lasted more than 2 weeks on top and volumes that lasted less than 2 weeks on top.	1(#) return artists
2(#) return volumes of #1
3(#) return weeks on top that #2 lasted
4(#) return number of #3 for each #2
5(#) return #1 where #4 is more than 2
6(#) return #1 where #4 is less than 2
7(#) return #1 in both #5 and #6
8(#) return famous titles of #7
Table 14: Example of a cluster from the t-SNE projection of EPR on BREAK.
Utterance	Meaning Representation
What is the metal thing next to the small cylinder?	1(#) return the small cylinder
2(#) return things
3(#) return #2 that are metal
4(#) return #3 that are next to #1
What is the purple thing next to the brown thing?	1(#) return the brown thing
2(#) return things
3(#) return #2 that are purple
4(#) return #3 that are next to #1
What is the gray thing next to the block?	1(#) return gray thing
2(#) return the block
3(#) return #1 next to #2
What is the shiny thing next to the cylinder?	1(#) return shiny thing
2(#) return cylinder
3(#) return #1 next to #2
What is the thing in front of the red square?	1(#) return things
2(#) return squares
3(#) return #2 that is red
4(#) return #1 that is in front of #3
Table 15: Example of a cluster from the t-SNE projection of EPR on BREAK.
Utterance	Meaning Representation
Is the purple thing behind the big red thing?	1#) return purple thing
2#) return big red thing
3#) return Is #1 behind #2
is the purple sphere in front of the blue cube?	1#) return the purple sphere
2#) return the blue cube
3#) return if #1 is in front of #2
is the gray sphere behind the green cylinder?	1#) return the green cylinder
2#) return the gray sphere
3#) return if #2 is behind #1
is the red cube in front of the yellow ball?	1#) return the red cube
2#) return the yellow ball
3#) return if #1 is in front of #2
Is the blue ball in front of the silver cube?	1#) return blue ball
2#) return silver cube
3#) return is #1 in front of #2
Table 16: Example of a cluster from the t-SNE projection of EPR on BREAK.