SlowGuess/NAACL_2022_OCR huggingface dataset:
Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedback
Niket Tandon*, Aman Madaan *†, Peter Clark, Yiming Yang†,
Allen Institute for Artificial Intelligence, Seattle, WA, USA
† Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA
{nikett,peterc}  $@$  allenai.org
{amadaan,yiming}@cs.cmu.edu
Large language models (LMs), while powerful, are not immune to mistakes, but can be difficult to retrain. Our goal is for an LM to continue to improve after deployment, without retraining, using feedback from the user. Our approach pairs an LM with (i) a growing memory of cases where the user identified an output error and provided general feedback on how to correct it (ii) a corrector model, trained to translate this general feedback into specific edits to repair the model output. Given a new, unseen input, our model can then use feedback from similar, past cases to repair output errors that may occur. We instantiate our approach using an existing, fixed model for script generation, that takes a goal (e.g., "bake a cake") and generates a partially ordered sequence of actions to achieve that goal, sometimes containing errors. Our memory-enhanced system, FBNET, learns to apply user feedback to repair such errors (up to 30 points improvement), while making a start at avoiding similar past mistakes on new, unseen examples (up to 7 points improvement in a controlled setting). This is a first step towards strengthening deployed models, potentially broadening their utility. $^1$
Language models (LMs) have achieved remarkable success on many tasks (Wang et al., 2019; Talmore et al., 2019), but they are still prone to mistakes (Bender and Koller, 2020). Correcting mistakes by retraining is not always easy due to the cost and/or unpredictability of how additional training data will change the model. Instead, our goal is to allow users to correct such errors directly through interaction, without retraining – by giving corrective feedback on the model's output. Our approach is to maintain a growing, dynamic memory of such
Figure 1: Given a frozen model  $\mathbf{B}$ , we train a corrector model  $\mathbf{G}$  to apply feedback from a user about errors made by the original model. In the example,  $\mathbf{B}$  has generated a script with an error in, stating that "driving" and "getting in a car" can occur in any order (red box). The user provides general feedback ("Get in a car before driving"), and  $\mathbf{G}$  operationalizes this to generate a corrected graph (by predicting and applying a graph edit operation) in which "get in car" happens first (green box). The feedback is stored in a memory  $\mathcal{M}$  so it can also be retrieved to repair similar, future errors.
feedback, and use a trained corrector model to apply such feedback to repair the model output. By doing so, the system can also potentially fix output errors for new unseen inputs using feedback from similar, past cases. The ability to leverage a fixed trained model without re-training could save costs and have a positive environmental impact.
We consider the class of problems where the model's output is repairable, namely a structured output that is (typically) nearly correct, and fixable through a small number of edit operations. Our system is general and admits a general graph based input, so in principle it applies to a large number of tasks. In this paper, we apply our approach to the task of script generation that provides a natural setting for users to critique, and has applications in smart assistants (Zhang et al., 2021). We use an existing, fixed model: proScript (Sakaguchi et al.,
1Our code and data is available at https://github.
com/allenai/interscript
Findings of the Association for Computational Linguistics: NAACL 2022, pages 339 - 352
2021) that satisfies the constraint of the model's output to be repairable. proScript takes as input a goal to achieve (expressed in natural language), and outputs a partially ordered sequence of steps - a script - required to achieve that goal. Our interest here is not in proScript itself, but in what to do when proScript's output contains an error.
This instantiation of our approach is illustrated in Figure 1. Here, proScript has generated a script  $\mathbf{x}$  to achieve the goal "see an alligator", but the script contains an error: it states that the steps of "driving to the zoo" and "get in car" can be applied in any order. To repair this, the user provides the general feedback "Get in a car before driving". The corrector model  $\mathbf{G}$  then takes that feedback and the erroneous script, translates it into appropriate edit operations on the script, and applies those edits to generate a corrected script (y in Figure 1). The feedback is stored in memory  $\mathcal{M}$  so it can also be retrieved in the future. Our system, FBNET, comprises the corrector module  $\mathbf{G}$ , the memory  $\mathcal{M}$ , and searching and writing operations. To train our system, we collect examples of bad outputs, general feedback, and specific edits that the feedback should translate to (Section 4.2). This allows  $\mathbf{G}$  to learn how to translate general feedback into specific edits to apply. Pairing  $\mathbf{G}$  with the memory  $\mathcal{M}$  allows FBNET to repair new, unseen scripts containing similar errors to the one the user corrected.
Our approach loosely follows some early AI systems that maintained a memory of the output problems and how to fix them (Sussman, 1973; Hammond, 1986; Riesbeck, 1981), but here, we use neural methods and interact with a user to provide corrective feedback. It also builds on the idea of allowing users to specify edits in natural language, e.g., NLEdit (Elgohary et al., 2021), except we use general user feedback (then translated to example-specific edits by  $\mathbf{G}$ ) and add a memory so that feedback can also be automatically reused.
We evaluate FBNET along two dimensions: (a) How well does FBNET interpret NL feedback? (b) How well can FBNET learn from prior mistakes? We find that (a) it uses NL feedback effectively to repair script errors, with  $+30\%$  (absolute) improvement over a baseline that does not use feedback, and that (b) it makes a start at avoiding past mistakes  $(+7\%)$  (absolute) improvement in a controlled setting). Although these results are only for a single deployment of our general approach, they suggest that memory-based architectures can help deployed
models continue to improve with time, without retraining, potentially broadening their utility.
There have been numerous approaches to using user feedback to improve model performance, including:
(1) Providing additional training examples: Dasgupta et al. (2019) show how a user can correct bad model behavior by carefully selecting new training examples for the system to learn from, a style of interactive active learning (Settles, 2012).
(2) Marking/scoring the system's answer(s): In SHRDLURN, the user provides feedback by identifying which of the system's alternative interpretations of a user command is correct (Wang et al., 2016).
(3) Providing hints: (Mehta and Goldwasser, 2019) show how a system can learn to understand regional (e.g., "top left") and directional (e.g., "move down") hints from the user for a (simulated) robot.
(4) Provide additional information: In TeachYourAI (Talmor et al., 2020), given a wrong answer to a question, users can enter NL facts and rules to use as context when reasking the question, to (ideally) produce the correct answer.
(5) Correcting bad answers: In the semantic parsing task of NL-to-SQL, NLEdit learns to interpret and apply syntactic edit operations from the user expressed in NL, e.g., "replace course id with program id." (Elgohary et al., 2021).
These methods all augment/replace the standard use of automated answer feedback (if available), e.g., testing whether a semantic parse correctly executes to the correct answer, e.g., (Zettlemoyer and Collins, 2005), sometimes using unsupervised techniques to generate additional training data, e.g., BIFI (Yasunaga and Liang, 2021).
Our work expands on the above approaches in two important ways. First, users provide general feedback in NL, that can potentially be applied to multiple cases (rather than just correcting a specific instance). The corrector model  $\mathbf{G}$  is trained to operationalize that advice in different ways for different examples appropriately, in contrast to (say) NLEdit where the user-provided specific corrective edits for a single example only.
Second, we use a feedback memory, allowing feedback to be reused. While adding external mem
Figure 2: Proposed architecture: (left)  $\mathbf{B}$  does not account for user feedback. (right) FBNET maintains a memory  $\mathcal{M}$  of corrective feedback, and searches for feedback from prior queries with similar error intent as  $\mathbf{x}$  using a retrieval function  $\Omega$ .  $\mathbf{x}$  is then concatenated to the retrieved feedback to form the input to the corrector model  $\mathbf{G}$ . Users can also give new feedback which is added to  $\mathcal{M}$ . In this work, we focus on script generation models that might generate an erroneous script which are correctable using an edit (feedback).
ory to neural systems is not new, e.g., RAG (Lewis et al., 2020), REALM (Guu et al., 2020), ours is the first to utilize a memory of prior user feedback to improve future neural model performance. This can be viewed as a modern approach to failure-driven reminding, an essential theme in earlier AI and Cognitive Science research (Riesbeck, 1981; Schank and Leake, 1989; Ross, 1984).
3 FBNET
3.1 Overview of the Architecture
Fig. 2 gives an overview of FBNET. The input is a potentially noisy graph  $\mathbf{x}$  generated by a base model  $\mathbf{B}$  and the output  $\mathbf{y}$  is a corrected graph. At inference time, i.e., after deployment, a user can critique  $\mathbf{y}$  by providing natural language feedback  $\mathbf{fb}$  on an error e. As output, the model generates the corrected graph  $\mathbf{y}$  that accounts for  $\mathbf{fb}$ .
The corrector model  $\mathbf{G}$  is responsible for improving the potentially noisy output from  $\mathbf{B}$ .  $\mathbf{G}$  achieves it using user feedback stored in a continuously updated memory  $\mathcal{M}$ .
The Memory  $\mathcal{M}$  is a growing lookup table of key-value pairs: key  $(\mathbf{x}_i)$  - value  $(\mathbf{fb}_i)$ , where  $\mathbf{x}_i$  is a particular incorrect graph, and  $\mathbf{fb}_i$  is the corresponding feedback. This memory supports lookup (read) and write operations. Given a new query  $\mathbf{x}$ , FBNET uses feedback  $\mathbf{fb}$  from similar, prior queries in the memory to enrich  $\mathbf{x}$ . This feedback  $\mathbf{fb}$  is retrieved using the lookup function  $\Omega (\mathbf{x},\mathcal{M})$ .
The corrector then combines fb with x, and generates y. The write operation is used whenever a user gives new feedback.
We make two assumptions on the characteristics of the feedback and the input.
A1. Base model B's output is repairable: B typically produces syntactically correct output graph but can have semantic errors that the user can recognize and describe using natural feedback. For example, the script in Figure 1 is repairable.
A2. Feedback is reusable: If two examples  $i, j$  have similar errors  $e_i$  and  $e_j$  then the feedback  $fb$  for one should apply to the other, i.e.,  $(e_i \sim e_j \Leftrightarrow fb_i \sim fb_j)$
3.3 Memory  $\mathcal{M}$  and  $\Omega$
As mentioned, the feedback is stored in a memory of key  $(\mathbf{x})$ , value (fb) pairs.  $\Omega$  is a retrieval function that matches a query key  $(\mathbf{x}_j)$  to a similar  $\mathbf{x}_i$  in memory implicitly on the similarity of the errors  $e_i$  and  $e_j$ .
3.4 Corrector model G
The graph corrector model  $\mathbf{G}$  generates an improved output  $\mathbf{y}$  given a noisy graph  $\mathbf{x}$  and  $\mathbf{fb}$ . This is done in a two-step process, (i) learning to predict a graph edit operation  $\mathbf{y}^e$  given  $\mathbf{x}$  and  $\mathbf{fb}$  (ii) using simple graph operations to apply  $\mathbf{y}^e$  to  $\mathbf{x}$  to produce  $\mathbf{y}$ . Our approach of generating an edit instead of directly generating the corrected graph is beneficial for two reasons. First, generating edits is simpler for the model than generating entire graphs. Second, it simplifies evaluation metrics as it is much simpler to compare two smaller generated edits. Note that we can deterministically fix a script given an edit. Thus, the two-step process helps us achieve the same end goal (corrected scripts from noisy scripts and feedback).
As mentioned, the graph corrector  $\mathbf{G}$  first generates an edit  $\mathbf{y}^e$ , which is applied to the incorrect graph  $\mathbf{x}$  to generate the correct graph  $\mathbf{y}$ . We need a corpus of  $(\mathbf{x},\mathbf{fb},\mathbf{y})$  to train this system. Specifically, we extract an edit from each such tuple, where edit  $y^{e}$  is the difference between the output  $\mathbf{y}$  and the input  $\mathbf{x}$ .  $\mathbf{x}$  and  $\mathbf{y}$  can be expressed in a string representation using a graph description language such as
DOT. We then train a language model to estimate  $P_{\theta}(y^{e} \mid \mathbf{x}, \mathbf{fb})$ , which allows us to generate an edit for a given  $(\mathbf{x}, \mathbf{fb})$  using greedy sampling, where  $\theta$  denotes the parameters of the language model.
4 Application: Script Generation
We instantiate our framework for the task of script generation. Formally, the script generation task (Sakaguchi et al., 2021) takes as input a scenario and generates a script  $G(V,E)$ , where  $V$  is a set of essential events  $\{v_{1},\dots v_{i},\dots v_{|V|}\}$  and  $E$  is a set of temporal ordering constraints between events  $\{e_{ij}\}$  which means that the events  $v_{i}$  must precede the event  $v_{j}$  ( $v_{i} \prec v_{j}$ ). Partial ordering of events is possible, e.g., you can wear a left sock and a right sock in any temporal order. To solve this task, script generation models are required to generate events ( $V$ ) and predict the edges ( $E$ ) jointly. See Figure 3 for an example.
Figure 3: An example of a script in Sakaguchi et al. (2021). In a script generation task, models take the goal as the input and generate a (possibly) partial-order graph, which consists of essential steps and their ordering.
PROSCRIPT<sup>gen</sup> (Sakaguchi et al., 2021) is a recently released model that, given a goal, generates  $V$  and predicts the edge structure  $E$  jointly. It is based on the T5-XXL model (11B parameters) and generates the script as a graph in DOT format. The authors report that the DOT format is always valid at inference time and that  $V$  and the graph structure are generally of high quality. They characterize the graph edits required to correct a generated script (such as removing a node, adding a node, changing edge order, etc.). Mechanical Turk workers could repair most of the generated scripts within a few
edits (typically an edit distance of 5) - we further validate this in Appendix §8.1. This makes for an attractive use-case for interactive learning because the generated content from the model is repairable through user feedback.
4.2 Feedback Data Collection
To train the corrector  $G$ , as well as evaluate our approach, we collected a set of  $(x, fb, y)$  tuples using crowdworkers, where  $x$  is a possibly erroneous script generated by PROSCRIPTgen,  $fb$  is general feedback about the error (if any), and  $y$  is the corrected script. In practice, crowdworkers specified the edits to  $x$  to create  $y$  (using simple graph operations we can generate  $y$  from  $y^e$  - see Table 7 for an example). We collected 1542 tuples of data, randomly splitting it into 843 train, 154 validation, and 545 test points. Examples of the resulting dataset are shown in Table 1.
4.3 Training the Corrector Model
We initialize  $\theta$  with a checkpoint from the text-to-text pre-trained T5 transformer (Raffel et al., 2020) and fine-tune on our dataset. We use the default hyperparameters (including the Adafactor optimizer) in the T5 library. We fine-tune a T5-XXL model for the main results, fine-tuned for 5,000 steps (batch size 8), selecting the checkpoint with the highest validation score (usually the final step). To implement the memory  $\mathcal{M}$ , we use a BERT-based Sentence Transformer to encode  $x$  (Reimers and Gurevych, 2019), and use cosine distance with a threshold of 0.9 to find a matching key  $\mathbf{x}_m$ . We leave the investigation of more complex retrieval functions (e.g., using attention mechanism to future work.)
We empirically evaluate two questions:
RQ1. How well does FBNET interpret NL feedback? Specifically, we measure how well FBNET can translate general feedback fb from a user into the correct repair edit on an imperfect script  $x$ . The main focus of RQ1 is to test the performance of G in the pipeline (Fig. 2)
RQ2. How well can FBNET learn from prior mistakes? We make the same measurement, but using feedback fb recalled from similar,
$^{2}$ https://github.com/google-research/text-to-text-transfer-transformer
Error type	Input script x	Feedback fb	Expected edit y e*	Generated edit y e	EM	score EMtype	EMloc
missing step	1. get out of car2. stop in front of car3. turn body toward back of car4. walk to back of car5. take blanket out of car6. walk to desired location7. throw blanket down	a person needs to open the door before they take an object out	insert node ‘open the back door of the car’ before ‘take blanket out of car’	insert node ‘open car door’ be-force ‘take blanket out of car’	0	1	1
missing step	1. buy a video game2. talk to the cashier3. make the transaction4. get the receipt5. load video game into the car6. get into the car7. take xbox home	after a person makes a transac-tion, they then head to their car	insert node ‘walk to the car’ after ‘get the receipt’	insert node ‘get into the car’ after ‘make the transac-tion’	0	1	0
wrong step	1. make a bunch of cards2. grab a pen3. grab some paper4. pick up a pen5. place the paper on the table6. pick up the pen7. write names on the cards	good plans shouldn’t include redundant steps	remove node ‘pick up the pen’	remove node ‘pick up the pen’	1	1	1
wrong order	1. leave home and get in car2. remem. destination address3. look around for the car4. walk towards the car5. open the car door6. sit down in the car7. put on the seatbelt	you wouldn’t look for some-thing you’re already with	reorder edge between ‘< leave home and get in car, look around for the car’	remove node ‘look around for the car’	0	0	0
Table 1: Some examples of the data points and model predictions.  $\mathbf{y}^e$  takes the form: <EDIT TYPE> over [<ARG>] at <LOCATION> The dataset contains partial order points as well, but they are omitted here for simplicity.
previous examples. The main focus of RQ2 is to test the performance of  $\mathcal{M}$  and  $\Omega$ .
Metrics To compare the gold edit  $\mathbf{y}^{e^*}$  and the generated edit  $\mathbf{y}^{\hat{e}}$ , we use standard metrics used to evaluate generated text. We report the following metrics:
- Exact match: EM gives a score of 1 if  $\mathbf{y}^{e^*}$  is equal to  $\mathbf{y}^{\hat{e}}$  and 0 otherwise.
- Generation metrics: We report standard generation metrics BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) to account for similar but not exact matches. We use the implementation released in the metrics package of the GEM-benchmark (Gehrmann et al., 2021).<sup>3</sup>
We report these metrics over the entire edit: EM, BLEU, ROUGE. The components of  $\mathbf{y}^e$  broadly follow a template: <EDIT TYPE> over [ARG] at <LOCATION> (see Table 1). This allows comparison of the location or edit type in  $\mathbf{y}^{e^*}$  and  $\mathbf{y}^{\hat{e}}$ : EM loc, BLEU loc, ROUGE loc and EM type, BLEU type, ROUGE type
Baseline As baseline, we train a model that does not use any feedback (we call this, NO-FB) and is trained only with input = erroneous script and output = edit. The language model used in this baseline and FBNET is the same (T5-XXL), allowing a meaningful comparison.
<sup>3</sup>https://github.com/GEM-benchmark/GEM-metrics/
EM	EMloc	EMtype	BLEU	ROUGE
NO-FB	3.5	9.7	30.4	21.7	39.0
FBNETo	38.6	45.8	69.3	54.2	70.6
5.1 RQ1: How well does FBNET interpret NL feedback?
To measure how well the graph corrector  $\mathbf{G}$  learns to interpret NL feedback, we provide oracle feedback to FBNET, and we call this  $\mathrm{FBNET}_o$ . Table 2 shows that  $\mathrm{FBNET}_o$  learns to react to the feedback, as indicated by a sharp increase in both the exact match scores and automated metrics. Further, we note that the model is good at identifying the error type that the feedback indicates. Still, it is difficult for the model to localize the error in the graph, probably because the location is not explicitly mentioned in the feedback, and the model must infer it.
How consistently does FBNET interpret similar feedback? In  $\sim 15\%$  of the data points, multiple fb can lead to the same  $(\mathbf{x},\mathbf{y})$  pair. FBNET is expected to behave consistently for such re-phrasings of fb. The model consistently produces exactly the same y for fb re-phrasings  $\sim 60\%$  of the time. Furthermore, we observe majority agreement as the number of fb re-phrasings for a  $(\mathbf{x},\mathbf{y})$  pair increases. In our analysis, a large proportion of the inconsistent edits occur because different fb phrasings prompt the model to generate slightly different, but semantically similar edits: see Table 3 for an example.
How well can FBNET handle wrong feedback?
While the ability to react to feedback is a desired trait for FBNET, we also want to ensure that the performance of FBNET is proportional to the quality of feedback. This will ensure that FBNET can act faithfully in settings where the feedback might be potentially misleading. We investigate this question by identifying lexically similar scripts but irrelevant feedback from the training set for each test
Table 2: Interpreting NL Feedback: Correctness of Predicted Edits .... Given an erroneous script  $x$  ,and general feedback  ${fb}$  from the user,FBNet perfectly predicts the specific repair edits  ${38}\%$  of the time (EM, exact match) - an order of magnitude better than a baseline NO-FB predicting the repair from  $x$  alone.  $E{M}_{loc}$  and  $E{M}_{type}$  compare just parts of the edit sequences (locations/types of the required edits,respectively), while BLEU and ROUGE are softer matching metrics.
feedback	predicted edit
The feedback is if a person is going to open a book, they need to choose one first	insert node ‘choose a book to read’ before ‘open the book’
The feedback is you can’t open something you’re not holding	insert node ‘get the book out of the bag’ before ‘open the book’
Table 3: Multiple feedbacks for the same  $(\mathbf{x},\mathbf{y})$ . Here,  $\mathbf{x}$  is: You are given a plan to read to child. decide which books to read, open the book, read the book to the child, turn the pages ...  $\mathbf{y}^{\epsilon}$  is insert node 'pick a book off the shelf' before 'open the book'
example. Note that our setup easily allows us to test this hypothesis since the train/test/val splits were carefully designed to ensure no overlap between the examples. Thus the feedback from one example will typically not apply to another example. We find that with irrelevant feedback, the performance of FBNET drops to  $3\%$ . This shows that FBNET is sensitive to the quality of feedback, and no feedback is better than misleading and irrelevant feedback.