SlowGuess/NAACL_2022_OCR huggingface dataset:
How well does FBNET perform across error types?  $\mathrm{FBNET}_o$  gets the highest performance (EM  $63.0\%$  ) on wrong-step error type where fb typically contains negative words that signal the error type, and the model learns to localize the error node. One of the most challenging error types is partial order removal or addition (EM  $10.5\%$ ). This can be attributed to the challenging localization involving multiple nodes that participate in a partial order. The lowest-performing is the missing step error type (EM 2.73). The reason for this low EM score is that the edit must generate the missing node, and EM undercounts the correctness of the generated text. Other metrics such as ROUGE are much higher validating that the model performs well on this error type. Section §9 Table 8 breaks down the performance of FBNET by error type.
5.1.1 Error analysis
We randomly sampled 50 instances from the test set where the model generates an incorrect edit (i.e.,  $EM = 0$ ). Our goal is to understand the typical errors made by the model and use the analysis to calibrate the findings in Table 2.
- Lexical variation (36%) Exact match underestimates the performance of our model (as the
task involves generation). We find that more than  $35\%$  of the predicted edits are semantically similar (typically lexical variation) to the reference gold edit. Some examples include: insert node picking a book... vs, insert node choosing a book to read. Another kind of example is the model suggesting swapping the order of edges A and B while the reference edit swaps edges B and A - but both of these are equivalent.
- Challenging feedback (24%) This type of error occurs when the model fails to interpret a feedback because it is difficult to interpret e.g., the feedback is expressed abstractly. For example, for the goal "go to locker room," the generated script repeats the step "walk to the locker room". However, the feedback is 'you can't go where you already are', and FBNET generates the edit "reorder edge between ' $\langle$  walk towards the locker room, walk to the locker room  $\rangle$ ", failing to interpret the feedback.
- Error not localized (20%) In about 20% of the failures, FBNET fails to localize the error given the feedback. For example, consider the erroneous input script about the goal buy an xbox: 1. go to the store 2. talk to the cashier 3. make the transaction 4. get the receipt 5. load the video game into the car 6. get into the car 7. take xbox home The feedback is after a person makes a transaction, they then head to their car. The expected edit is: insert node 'walk to the car' after 'get the receipt', but the predicted edit insert node 'get into the car' after 'make the transaction' does not correctly identify the erroneous node. The feedback points to making a transaction, but it also involves getting the receipt.
- Alternative answers (16%) We also encounter cases where there are multiple ways to correct a script. For example, an edit can be expressed as insert node 'X' before 'step 4' or insert node 'X' after 'step 3'. This comprises  $\sim 16\%$  of the errors.
In  $\sim 32\%$  cases, the model generates a correct edit that differs from the gold. Extrapolating this performance under-counting to the entire test set, the accuracy of FBNET in Table 2 would increase to  $\sim 70\%$ $(+32\%)$ .
EM	EMloc	EMtype	BLEU	ROUGE
NO-FB	6.94	15.3	34.7	24.1	44.2
FBNET	16.72	20.9	56.9	32.5	48.5
FBNETo	22.2	27.8	72.2	44.6	65.8
Table 4: Learning from prior mistakes: On the reuse dataset, given an erroneous script  $x$  ,and feedback  ${fb}$  recalled from similar,prior examples,FBNet perfectly predicts the specific repair edits  ${16.7}\%$  of the time (or 20.9% the edit location and 56.9% the edit type), a promising start to learning from prior mistakes.
Figure 4: Performance on unseen examples (number of correct data points) improves as memory size grows. NO-FB baseline performance remains static. Note that accuracy is evaluated using exact match, and thus is a lower bound on the actual equivalence as exact match might miss rephrasings.
5.2 RQ2: How well can FBNET learn from prior mistakes?
Section §5.1 shows that the corrector  $\mathbf{G}$  can utilize user-supplied feedback to fix an incorrect structure. FBNET combines  $\mathbf{G}$  with a memory  $\mathcal{M}$  of feedback, allowing us to leverage past feedback on new examples. This section presents a setup where feedback on previously seen inputs is used to fix new, unseen examples.
To investigate this setting, we create a new test set, called the interaction-reuse set or ISET. To create it, we randomly sample 72 test points (referred as interaction-reuse set-genesis or ISET-SOURCE) and perturb them linguistically to generate interaction-reuse set(also referred as ISET). The perturbations are performed on the salient entities in the script, including (i) linguistic perturbation on  $\sim 20\%$  samples (e.g., box  $\rightarrow$  carton, package) and
(ii) the relatively harder analogical perturbation on the remaining  $\sim 80\%$  samples (e.g., bus  $\rightarrow$  train, and how to lift blinds  $\rightarrow$  how to open oven door because the event structure is analogical). The  $\mathbf{y}^e$  to the original script also applies to the substituted script. We ensured that the perturbations did not introduce additional errors in the substituted script. This ensures that the interaction-reuse setnow contains similar examples to the original test set, a condition that our original splits do not satisfy. There are a total of 72 data points in interaction-reuse set.
Continually learning using a memory of errors Examples in interaction-reuse set are randomly mixed with the original test set. This combined test set of queries  $Q$  is then evaluated using our setup as shown in Algorithm 1. Intuitively, interactionreuse set allows us to simulate a setting where the system has been deployed in the wild, and endusers can query. Algorithm 1 runs the memorybased inference described in Section  $\S 3$  (Figure 2). As the system is run through the stream of queries, we expect that i) the overall performance of the system will be better than no feedback, as some of the examples in the interaction set will provide meaningful feedback, and ii) the running performance of the system will improve with growing memory: the probability of relevant feedback being present for an unseen example increases with time, thus boosting the performance.
Our experiments show that FBNET meets both these expectations. First, Table 4 shows that retrieved feedback improves over no feedback by
10 points (exact match) and similarly in terms of BLEU and ROUGE scores, respectively. Further, Figure 4 shows a graph confirming that FBNET can improve continuously as memory grows.
6 Scope
In principle, we could apply FBNET to any task that satisfies the assumptions (§3.2). However, our approach has some limitations in practice, several of which merit further detailed follow-up work.
- On Assumption A1: We assume that the output of  $\mathbf{B}$  is repairable. Such an assumption is only possible for models that generate mostly correct outputs and have errors that are easy to highlight for humans. In practice, this implies that our approach will most efficiently work in conjunction with modern language models (Bommasani et al., 2021) that are shown to be syntactically correct in form, but can produce output that lacks commonsense (Bender and Koller, 2020), making their output repairable.
- On Assumption A2: Having reusable, general feedback is costly and requires careful instructions to collect from general users and crowdworkers (e.g., we asked the crowdworkers how they would explain the model error to a five-year-old). As the domain of the task becomes more specialized, such as database query generation (Elgohary et al., 2021) or code correction (Yasunaga and Liang, 2020), collecting data to train  $\mathbf{G}$  becomes difficult. Systems that produce structured explanations are better suited to our model (see Wiegreffe and Marasovic (2021) for an overview), rather than specialized domains that require expert users to provide feedback (e.g., in database query generation).
- Consistent memory: We show in Section §5.1 that FBNET is sensitive to the appropriateness of the feedback. However, adversarial or incorrect feedback could pollute the memory and possibly make it inconsistent. There has been some recent work to ensure consistency of beliefs of a model (Kassner et al., 2021), and more effort is required in this direction to apply to more complex settings like ours.
- Using multiple feedbacks:  $\Omega$  can be enhanced with more complex attention mechanisms that aggregate from multiple relevant memory entries and possibly generalize them. We conducted an initial experiment using attention and
found that we would need a larger dataset to train  $\Omega$  effectively.
Advancements in these directions would further increase the applicability of FBNET. Still, there are several applications (Wiegreffe and Marasovic, 2021) where our approach would currently apply in principle, or is easy to set up.
Our goal is to create a system that can continuously improve the structured output of a model. Our approach is to train an error correction model that uses natural language (NL) feedback to correct errors in that output. We have presented the first step towards this goal, showing that an error correction module can learn to interpret NL feedback successfully, resulting in  $40\%$  fewer errors in script generation. We have also described ongoing work on the next step, namely adding a memory layer where human feedback is stored and later retrieved efficiently. Together, these offer a possible path to systems that can continuously improve their output over time, with progressively less feedback and without retraining.
This material is partly based on research sponsored in part by the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government. We would like to thank Google for providing the TPU machines for conducting experiments.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosse-lut, Emma Brunskill, et al. 2021. On the opportuni
ties and risks of foundation models. ArXiv preprint, abs/2108.07258.
Sanjoy Dasgupta, Daniel Hsu, Stefanos Poulis, and Xiaojin Zhu. 2019. Teaching a black-box learner. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1547-1555. PMLR.
Ahmed Elgohary, Christopher Meek, Matthew Richardson, Adam Fourney, Gonzalo Ramos, and Ahmed Hassan Awadallah. 2021. NL-EDIT: Correcting semantic parse errors through natural language interaction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5599–5610, Online. Association for Computational Linguistics.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96–120, Online. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Z. Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. ArXiv, abs/2002.08909.
K. Hammond. 1986. Chef: A model of case-based planning. In AAAI.
Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. 2021. BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8849-8861, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik-tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Roektaschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Nikhil Mehta and Dan Goldwasser. 2019. Improving natural language interaction with robots using advice. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1962-1967, Minneapolis, Minnesota. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, M. Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. *J. Mach. Learn.* Res., 21:140:1-140:67.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
C. Riesbeck. 1981. Failure-driven reminding for incremental learning. In *IJCAI*.
B. Ross. 1984. Remindings and their effects in learning a cognitive skill. Cognitive Psychology, 16:371-416.
Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. 2021. proscript: Partially ordered scripts generation via pre-trained language models.
R. Schank and David B. Leake. 1989. Creativity and learning in a case-based explainer. Artif. Intell., 40:353-385.
Burr Settles. 2012. Active learning. In Active Learning.
G. Sussman. 1973. A computational model of skill acquisition. Technical Report AITR-297, MIT.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Alon Talmor, Oyvind Tafjord, P. Clark, Y. Goldberg, and Jonathan Berant. 2020. Teaching pre-trained models to systematically reason over implicit knowledge. NeurIPS.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning language games through interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2368-2378, Berlin, Germany. Association for Computational Linguistics.
Sarah Wiegrefe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable nlp. ArXiv, abs/2102.12060.
Michihiro Yasunaga and Percy Liang. 2020. Graph-based, self-supervised program repair from diagnostic feedback. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10799-10808. PMLR.
Michihiro Yasunaga and Percy Liang. 2021. Break-it-fix-it: Unsupervised learning for program repair. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11941-11952. PMLR.
Luke Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorical grammars. In UAI.
Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryen White, and Dan Roth. 2021. Learning to decompose and organize complex tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2726-2735, Online. Association for Computational Linguistics.
8.1 Initial study on the errors of B (PROSCRIPT)
On PROSCRIPT' s test set, we performed inference using the released checkpoint (both GPT-2 and T5-XXL based model). We then randomly sampled 30 generated graphs and manually wrote feedback for them. Similar to Sakaguchi et al. (2021), we found that the model makes repairable mistakes (leading to assumption A1 being satisfied). Further, we found there instances where a general principle feedback applies across more than one instances (e.g., you have to be near something to use it). (see Table 5).
What was the error	General principle feedback
Script was missing the step of not turning off the alarm after waking up	People don’t leave their alarms ringing all day.
Script mentioned coming to the doorway and passing through it	One cannot walk through the doorway without opening the door first.
Script tells that getting in car and drive in zoo can be done in any order	People must get into a vehicle, before driving to any place.
Script is looking for a butterfly after placing it	You don’t need to look for a butterfly if it’s already in a container.
Table 5: Sample error and the corresponding general principle feedback that could, in principle, repair the model output.
On an average, there were about two mistakes present in the graphs. Often, the error was that the script was using an entity before having it (e.g., write on the paper comes before the node find the paper or reach for the paper). Thus, there seems to be a possibility of applying similar feedback to more than one example. We also found some cases where the script might have to be changed to adapt to special cases. For example, for a script visit Disneyland, an event obtain a visa might be required for some users. We believe the original ProScript dataset aims to generate widely applicable scripts and grounded in commonsense; rather than cover all possible outcomes.
On the surface, the generated scripts were of good quality. However, a closer look at the mistakes revealed that most of them could be attributed to the model lacking basic commonsense. For example, Figure 1 shows a typical mistake the model makes. This underscores the gap between the syntax and semantic correctness of machine-generated
output in the context of automatic script generation. This observation is in-line with other NLP tasks (Bender and Koller, 2020) that distinguish the success of recent models on the correctness of form rather than the far-from-over goal of understanding of meaning.
8.2 Data collection
An average user could point out mistakes in the generated scripts, as a majority of the errors in generated scripts are caused by a lack of basic commonsense (§8.1). Consequently, we designed a Mechanical Turk task to provide feedback on mistakes. A broad overview of the annotation process is shown in Figure 5.
Figure 5: A broad overview of the annotation process. For actual annotation task (including the M-turk task template), see our code repository.
Annotation Now we discuss our crowdsourcing setup to collect the data. To maximize the opportunity to get more feedbacks for a predicted script, we filtered a subset of the test set in ProScript where the human evaluated graph edit distance was likely to be high (i.e., there were likely to be more errors). The ProScript authors released the graph edit value for the set of test set samples they evaluated. We performed inference using their released PROSCRIPTgen model on those data points with high graph edit distance value  $(\geq 8)$ . With this we collected about 400 (predicted graph, expected gold graph) tuples. The ProScript paper describes that their expected gold graph is also imperfect and might contain about  $20\%$  noise. Nevertheless, having the gold reference graph guides and constrains an annotator about the common script for a scenario rather than the wide open space of solving the task using multiple potentially correct scripts. (e.g., one could go to a zoo without driving the car by hiring a taxi and then they won't need to drive or park the car). As mentioned in §8.1 our annotation process must focus on scripts that are widely applicable and grounded in commonsense.
The annotators are shown the model-generated
You want to see an alligator. Which plan: P1 (left) or P2 (right) is worse?
Q1. Which plan is worse?
1st plan (P1) is worse
2nd plan (P2) is worse
Q2. Describe one "critical error" in the worse graph
Figure 6: The mechanical turk page for annotation. We show the generated and the expected ProScript gold reference. The annotator must answer which script is worse and why. They must point out an egregious mistake (and not any trivial errors that have minor grammatical errors), and annotate: the error type (missing step, wrong step, wrong order, wrong partial order), localize the error by providing the node or edge id, and give feedback why it is wrong, and finally to gather the general principle behind the feedback they are asked to explain the feedback to a five-year-old.
and expected gold (reference) scripts, and are required to answer which script is worse and why. It is possible that the gold script is marked as worse. However, we later post-process and remove such cases, as our focus is to get errors on the generated scripts and not the manually created scripts. The annotators must point out an obvious mistake (e.g., an event or an edge that does not follow commonsense). They were asked to ignore grammatical and fluency errors, and focus on critical errors of four types: