Continually Improving Extractive QA via Human Feedback
Authors: Ge Gao
Hung-Ting Chen
Yoav Artzi
Eunsol Choi
Venue: EMNLP 2023 Main (2023)
Keywords: QA
human feedback
bandit learning
TL;DR: An iterative approach to continually improve an extractive question answering (QA) system via human user feedback.

Abstract: We study continually improving an extractive question answering (QA) system via human user feedback. We design and deploy an iterative approach, where information-seeking users ask questions, receive model-predicted answers, and provide feedback. We conduct experiments involving thousands of user interactions under diverse setups to broaden the understanding of learning from feedback over time. Our experiments show effective improvement from user feedback of extractive QA models over time across different data regimes, including significant potential for domain adaptation.

Continually Improving Extractive QA via Human Feedback
Ge Gao -Hun--TTing Ting Chen Yoav Artzi and Eunsol Choi DDepartmett of Computer Science and Coreell Tech, Comell University Department of Computer Science, The University of Texas at Austin (ggao, yoav)ecs cornell. yoav)8cccc edu {hungtingchen, eunsol Joutxass edu
Abstract
Interact with Users
We study continually improving an extractive question answering (QA) system via human user feedback. We design and deploy an it- erative approach, where information-secking users ask questions, receive model- -eeee-pped answers, and provide feedback. We conduct ex- periments involving thousands of user interac- tions under diverse setups to broaden the under- standing of learning from feedback over time. Our experiments show effective improvement from user feedback of extractive QA models over time across different data regimes, includ- ing significant potential for domain adaptation.
How many people iailed in the hattle Moscow
Canualties for the Battle Moscow dehated
Casualties for Battle of Moscow are debated, but estimates suggest German losses of $48,00010 400,000 Soviet losses 630,000 1,280,000.
Partially correct.
Deploy
Contextual Bandit Learning
Figure I: Illustration of our setup. We alternate between interaction and leaming phases. At the interaction phase, given user question and context paragraph, the model predicts the question is unanswerable or span in the paragraph the answer. The human user provides "correct", "partially correct" or "wrong" feedback to the model answer by validating the answer in context. The system aggregates user feedback, and the model learns from feedback data at the learning phase by updating its parameters, with the goal improving over time.
1 Introduction
The deployment of natural language processing (NLP) systems creates ample opportunities to learn from interaction with users, who can often provide feedback on the quality of the system output. Such feedback can be more affordable and abundant than annotations provided by trained experts. can also be obtained throughout the system's 's lifetime, opening up opportunities for continual learning," where the system improves over time, and evolves the world and user preferences change.
create and deploy an information-sseking scenario, where users pose a question, receive prediction from a model which aims to answer the query given single evidence document, and provide feedback. The system iteratively improves over rounds, where each round is made of deploying the system interact with users, followed learning. Figure 1 illustrates our setup and learning paradigm. This scenario has been studied for embodied agents constrained environments (Thomason et al., 2015; Kojima et al., 2021; Suhr and Artzi, 2022), but QA proposes new challenges for using such interaction data, for example in its significantly richer lexical space. To the best of our knowledge, ours is the first study to show QA system improving via interaction with human users over time.
The combination of human feedback and con- tinual learning presents exciting prospects, but is relatively understudied, partially because of the challenges it poses. Mainly because it requires deploying models with human users over many in- teractions for both development and evaluation, for example to study the interplay between the learning signal and model design, the impact of the initial model, and learning progression
time.
Focusing on extractive question answering (QA), we study iteratively improving an NLP system by learning from human user feedback over time. We
Our setup is designed to study practical interac- tion with users. Following prior work on collect- ing ieformatiionsceekng queries (Choi et al., 2018; Clark et al., 2020), we prompt users to ask ques-
'Equal contribution.
'The term continual learning is at times used to refer to a scenario where models adapt to new tasks over time. We study improving the model continually on its original task.

---

tions that they do not know the answers to. This nat- urally elicits questions that cannot be answered un- der the fixed-evidence QA scenario (Kwiatkowski et al., 2019). We observe that the presence of unan- swerable questions dramatically impacts learning, making the process sensitive to the model's calibra- tion answerability. We address this by modeling question answerability separately from span ex- traction: model first predicts if the question is answerable and only then extracts an answer span. We use a simple three-option feedback signal ("correct", "partially correct", or "wrong"). While less informative than more complex cues, such as natural language, such feedback imposes little in- teraction overhead. We map the feedback signal to manually-specified reward values, and formulate learning as a contextual bandit problem.
users. Head-to-head comparison between learning reward model and directly mapping feedback reward values, as we do, remains an important di- rection for future work. Human feedback has also been studied without RI. (Xu et al., 2022; Thoppi- lan et al., 2022). A critical distinction of our work is the focus continual learning (i.e., iteratively rumning our process over many rounds) and the dynamics it creates, whereas the work mentioned above (both using and not using RL) focused on improving models single time.
Learning from explicit implicit feedback for language tasks was studied beyond recent in- terest in RLHF, including for machine transla- tion (Nguyen et al., 2017; Kreutzer al., 2018b,a), semantic parsing (Artzi and Zettlemoyer, 2011; Lawrence and Riezler, 2018), question answer- ing (Gao et al., 2022), and chatbots (Jaques et al., 2020). Similar to the work mentioned earlier, this line of work did not explore iterative con- tinual learning, we emphasize. The iterative process was studied in the context of embodied instruction generation (Kojima et al., 2021) and following (Thomason al., 2015; Suhr and Artzi, 2022). In contrast, we study QA on a wide range of Wikipedia articles, data with high lexical diversity. Others obtained complete labels as feedback (Iyer et al., 2017; Wang et al., 2016), process with higher overhead. RL using human feedback has also been studied for non-language problems (e.g., Knox and Stone, 2015; Warnell et al., 2017: Mac- Glashan al., 2017; Christiano al., 2017).
We conduct multiple studies, focusing on low data regimes, with the aim of demonstrating robust- ness to challenging scenarios where only small fraction of users provide feedback, and rapid im- provement is necessary with little feedback. Over nine rounds of deployment with over 1,800 user interactions, our approach shows an overall im- provement from 39 to 67 Fl score on our newly collected Wikipedia-bane bastd data. We conduct ex- tensive studies of our approach, including showing effective domain adaptation from different source domain that does not support unanswerable questions. Taken together, our results demonstrate the potential of NLP systems to learn from user feedback over time, specifically for extractive QA. Our data and codebase are publicly available at https:// thub. wtwpwilllaaoooooaac.aaaa..aaaaat aaapaa///aaa-ohh... from-hf.
Prior work on learning from interaction for QA used synthetic interactions and feedback simulated from supervised data (Campos et al., 2020; Gao et al., 2022). We study human feedback from informationn-seeknn human users who frequently pose challenging questions which may be unan- swerable from the given document, Li et al. (2022) proposed a process that involves crowdworkers pro- viding rating and explanation to given for given question-answe pairs to improve QA model post- deployment. They control data quality with manual reviewing and multiple annotations. Our setup has lower interaction overhead, at the cost of providing less informative, and at times noisy signal. We also go beyond one-time improvement in studying iterative deployment, providing insight into how the model improves over time.
2
Related Work
Our use of human feedback is related to work recently-termed reinforcement learning from hu- man feedback (RLHF; c.g., Ziegler al., 2019; Stiennon et al., 2020; Nakano et al., 2021: Ouyang et al., 2022; Scheurer et al., 2023). Largely, these methods rely on soliciting wisr-wise comparisons from annotators, which are used to train a reward model to be used in an RL process. We adopt a different approach: soliciting feedback from users single outputs their queries, and mapping the feedback to reward values be used in offline contextual bandit learning. An important consider- ation motivating our choice is that pair-wise com- parison, although suitable for paid annotators, is less suitable for soliciting feedback from actual
"Ziegler et al. (2019) note that RI.HF can he exccuted iterativcly, but do not report the details or analysis of doing so.

---

3 Interaction Scenario
Question: What did Saladin die from?
Answer: a fever
We focus on the task of extractive QA, which cre- ates an intuitive feedback solicitation scenario. It is relatively easy to visualize the model output (i.e., a span of text) in the context it is extracted from (i.e., the evidence paragraph), for the user ver- ify the answer. Automated evaluation is also well understood (Bulian et al., 2022), allowing reliable measurement of system performance over time.
Context: Saladin died a fever on March 1193 (27 Safar
589 AII) at Damascus...
Feedback: Correct
Question: Where does the name St Albans come from?
Answer: Alban
Context: St Albans takes its name from the first British
saint, Alhan. The most elaborate version of his story, Bede's
Ecclesiastical History of
Feedbuck: Partially Correct
We deploy our QA system in rounds. Each round starts with an interaction phase, followed by a learn- ing phase. At the interaction phase, users interact with a fixed. deployed model and provide feed- back. We aggregate this interaction data until collect a fixed amount of feedback data to enter a learning phase. The feedback is collected during natural user interactions (i.e., the model is deployed to fulfil its task of answering user questions). At the learning phase, we update the model parame- ters based on the aggregated feedback data. Be- cause we observe new feedback data during the learning phase, this creates an offline learning sce- nario (Levine et al., 2020),² which is practical for deployed systems. Except data aggregation, it re- quires no integration of the learning process into the deployed interactive system. The separation between deployment and training also enables san- ity checks before deploying a new model, and for hyperparameter tuning as in supervised learning. Each interaction starts with user posing ques- tion. The model computes an answer, and returns it to the user alongside a visualization of it in the context text from which was extracted. The user provides feedback, by selecting one of three op- tions: "correct", "partially correct" or "wrong". Table 1 shows examples from our studies. Formally, let question y be sequence of m tokens (. .m)) and context text be se- quence of tokens (ci,...,,al Ca)- A QA model at round o parameterized by Op computes two proba- bility distributions: a binary distribution indicating if the question is answerable or not by the context text Pu(ulq, ē; 6p)), where u E (ANS, UNANS); and a distribution over answer spans in the context text jl, 7; pp), where i,j = [1, m] and i < j. If arg m ANS,UMANs] (ulg. ē; 0p) UNANS. the model returns to the user that the question is unan- swerable. Otherwise, the model returns the highest probability span arg j j (i, jlq. c; 0p). Given
Question: Hindu mythology what were the names of Radha's lovers?
Answer: [Unanswerable given the context paragraph]
Context: Radha in her human form is revered as the milk- maid (gopi) Vrindavan who became the beloved of Kr- ishna. One of
Feedback: Partially Correct
Question: What percentage of people in Reno are White?
Answer: [Unanswerable given the context paragraph] Context: As of the census of 2010, there were The city's racial makeup was 74.2% White, 2.9% African American, 1.3% Native American, 6.3% Asian, 0.7% Pacific Islander, 10.5% some other race, and 4.2%
Feedback: Wrong
Tahle I:
User interaction examples; cach example is composed of user question, context paragraph, model-ppediece answer, and feedback. Appendix C lists additional examples.
the model output, the user provides feedback f E {CORRECT, PARTIALLY-CORREET, WRONG). Each user interaction generates data tuple (7, c, ú, f, fp), where is the binary answer- ability classification, and i and j are the start and end indices of the extracted answer, if the question is classified as answerable.
4 Method
We initialize the model with supervised data, and improve it by learning from user feedback through an offline contextual bandit learning process.
4.1 Model and Initialization
We use standard BERT-style architecture (Devlin et al., 2019). The input to the model is concate- nation of the question q and the context text We separately classify over the context tokens for the answer span start and end compute the span dis- tribution P. (Seo al., 2017). and compute the binary answerability distribution Pa with a classifi- cation head on the CL.S token (Liu et al., 2019).
We initialize the model parameters with DeBER- TaV3 weights (He et al., 2023), 4 and fine-tune us-
*Our overall iterative process can also be seen as batched contextual bandit (Perchet et al., 2016).
"We use the Hugging Face (Wolf et al., 2020) version of DeBERTAV3 (He et al., 2023y: microsoffldebertae---bbo

---

ing supervised data get our initial model. This is critical get tolerable experience to early users. We usually use a small number of examples (<512 examples), except when studying domain transfer. The training loss sums over the three cross-entropy classification losses, with coefficient À to weigh the binary answerable classification term.
Feedback PARTIALLY-CORRECT
Action
CORRECT
WRONG
UNANS
1
0
ANS
1
0
[
0.5
-0.1
Table 2: The mapping of user feedback specific actions to reward values. [i.j]] is the span predicted during the interac- tion, if one is predicted.
4.2 Bandit Learning
We learn through iterative deployment rounds. In each round P. we first deploy our model to in- teract with users (Section 3) and then fine-tune it using the data aggregated during the interac- tions. Each user interaction generates tuple (g, C, ú, i. S. 0p). where is question, è is con- text text, is the answerability classification deci- sion, i and] j are the returned span boundaries if span was returned, and p are the model parameters when the interaction took place.
span extraction. For example, if we get WRONG feed- back when an answer is given, we cannot tell if the answerability classification was correct or not. Our formulation allows us to set the reward value of the first action to zero in such cases, thereby ze- roing the answerability classification loss. The re- ward values were determined through pilot studies, For example, we observed that models overpredict unanswerable, so we set relatively large penalty of 1 for wrongly predicting unanswerable.
Learning Objective We use policy gradient REINFORCE (Williams, 1992) objective with a clipped inverse propensity score coefficient (IPS; Horvitz and Thompson, 1952; Gao et al., 2022) and an entropy term for the answerability binary classi- fication. IPS de-biases the offline data (Bietti et al.. 2021), and also prevents unbounded negative loss terms (Kojima et al., 2021). The entropy term reg- ularizes the learning (Williams, 1992; Mnih et al., 2016). If we substitute the policy terms with the predicted model distributions, the gradient for an answerable example with two actions with respect to the model parameters 0 is:
Policy We formulate a policy that casts answer prediction as generating a sequence of one or two actions, given question and a context č. This sequential decision process formulation, together with the multi-head model architecture, allow to control the losses of the different classification heads by assigning separate rewards to the answer- ability classification and the span extraction. The policy action space includes classifying if the ques- tion is answerable (ANS) not (UNANS) and actions for predicting any possible answer span [ j] in č.
The set of possible action sequences is con- strained. At the start of an episode, the policy first predicts if the question is answerable or not. The probability of the action a E (ANS, UNANS) is (a c; 0). Span prediction action are not possible, so their probabilities are set to O. If the UNANS action is selected, the episode terminates. Otherwise, the second action selects span [ from an answer, and the episode terminates. The probability of each span selection action is Pi(i,jl e; 0). Answerability prediction actions are not possible, so their probabilities are set to O.
Vol = an ( + V2 (...(.......
(1)
Pa 0;
=
Pu 0,)
02 p.
where the oy and 02 are IPS coefficients for the first (answerability elassification) and second (span extraction) actions, r1 and 72 are the corresponding reward values, 7 is a hyperparameter, and H(.) is the entropy function. For examples the model pre- dict as unanswerable, the second term is omitted.
We do not have access to re- ward function. Instead, we map the user feedback f to reward value depending the action (Ta- ble 2), and cannot compute rewards for actions observed during the interaction. The policy for- mulation, which casts the prediction problem as a sequence of up to two actions, allows to assign dif- ferent rewards to answerability classification and
Reward Values
Deployment and Learning Process Algorithm outlines our process. Each round (Line 2) includes interaction (Lines 4-14) and learning (Lines 15- 18) phases. During interaction, given question and context (Line 5), we classify if it answerable in the given context (Line 7) and extract the answer span (Line 8). Depending on the classification, we

---

Algorithm Deployment and Learning.
model. We extract topics, aspects, and evidence paragraphs from Wikipedia." We focus on learning from natural user feedback, so we not perform posthoe filtering of noise beyond removing adver- sarial crowdworkers (e.g., workers who always ask the same question). Appendix B provides more details about our interface and data collection. Appendix A gives training implementation details (e.g., hyperparameters).
1: D
2:
for round p ........... do
3: D, +
4: for interaction f = 1... T...... do
5: Observe question gl and context g(0) Predict if answerable and answer span:
6:
7:
6 <- mag. Pu(a ( q. p,)) 200 (0) <- maxij (,,
8:
9:
a ANS then
10:
Display the span [30 ( from ((0) as answer
11:
12:
Display that the question not answerable
Evaluation Data It is critical for test data come from the distribution the users create while interacting with the system for the evaluation re- flect the system performance. This makes existing benchmarks of lesser relevance to our evaluation. We collect held-out testing set that matches the observed distribution by interleaving the process in our deployment. We design a data annotation procedure based on our regular interaction task, and randomly opt to ask workers to provide answer the question they just posed rather than showing them the model answer and soliciting their feedback. This branching the process happens only after they ask the question, and workers see no indication for it when posing the question, so the questions we collect this way follow the same distribution the systems see. We collect around 100 such annotated examples each round during each of our experiments. This is small set, but unified across rounds within each experiment, it results in set large enough for reliable testing. To increase the quality of the annotation, we separately collect two additional annotations for each such test ques- tion. Appendix B provides further details on the test data collection and annotation.
13:
Observe user feedback j(c)
D2 (q cd a(t), 200 (00) f()) {p)}
for e E epochs
for D' examples from D,
17:
D' +- D' sample _ examples from D)
Update model parameters using D' with the gradient Equation
f DUD,
either display the answer (Line 10) return that the question is not answerable in the given con- text (Line 12), and solicit feedback (Line 13). We aggregate the interaction data over time (Line 14). During learning, we use rehearsal (Rebuffi et al., 2017) for each update, creating a batch of size B by mixing examples from the most recent interactions (Line 16) and previous rounds (Line 17) to update the model parameters (Line 18).
5 Experimental Setup
Deployment Setup We study our approach us- ing Amazon Mechanical Turk. We use Wikipedia data, a relatively general domain, thereby mak- ing the data we observe lcxically rich and topi- cally diverse. Our interaction scenario is designed to elicit information-sceking behavior. Following prior work (Choi et al., 2018 8), users are not given the context text when they pose the question, and are instructed to ask questions they do not know the answer to. This results in a significant num- ber of questions that cannot be answered given the evidence context. To increase user information- seeking engagement, we do not dictate the topic to ask about, but allow users to select a topic (e.g., "Saladin") and an aspect (e.g., "Death") from a set that is generated randomly for each interac- tion (Eisenschlos et al., 2021).5
We also evaluate our approach the English portion of TyDiQA (Clark et al., 2020) develop- ment set (954 examples, excluding Yes/No ques- tions), which was collected in a setting relatively similar to our evaluation data. Each example in- cludes question, a context paragraph, and a set of three human reference annotations.
Evaluation Metrics We focus on how different measures develop throughout the system's deploy-
'Our pool of topics a subset of previously compiled popular entities (Onoe al., 2022) based on the number of contributors and backlinks. Aspects each topic correspond to section titles on Wikipedia page. Each topic in our pool contains least four sections, with each section shorter than 490 tokens (excluding subsections).
Each topic-aspect pair is associated with an ev- idence paragraph, which serves as context to the
We concatenate the topic and uspect to the evidence para- graph as input to the model, but do not allow selecting these as unswer spans.
"Qualitative analysis of 100 randomly sampled examples from the first round of our long-term study (Section 6. shows a feedbuck noise rate of 18%.

---

ment. We compute deployment statistics on the aggregated interactions, including answerability prediction, feedback, and the reward values it maps to. We use the testing data to compute various test- ing statistics on each deployed model, including token Fl and answerability classification accuracy, For both datasets, we follow prior work (Clark et al., 2020) to consolidate the annotations. When the annotators disagree on the answerability of the question, we take the majority vote as the ground truth. We compute token-level Fl against the three annotations by considering the reference answer that will give the highest Fl score.
Interaction Data
Test Data
2 4 5 9
2 4 4 9
100
80
80
Feedback
60
60
400
Wrong
TyDiQA Dev
20
Partial
Correct
20
Our Test
Total
Answerability Acc.
80
2
Answerabiliry
Reward
Spain
60
40
-
TyDiQA Dev
6 Results and Analysis
- Our Test
100
We conduct two deployment studies: long-term deployment to study the effectiveness and dynam- ies of the process (Section 6.1), and shorter- term study to evaluate the impact of certain design choices (Section 6.2). We also evaluate the dif- ference between iterative rounds and one-time improvement (Section 6.3). Section 6.4 provides an additional experiment analyzing the sensitivity of our learning process to the observed data.
% Predicted Unans.
Ans.
Unans:
Fl (Our Test)
60
60
40
Ans Fl
20
20
Unaos F1
2 4 6
2 3
4 5
6
7
Round
Round
Figure 2: Statistics for the interaction data and the model performance on test sets after training interaction data from each round. Our test set is the union of all test data collected during the nine rounds.
6.1 Long-Term Experiment
We deploy for nine rounds of interaction and learn- ing, starting from an initial model trained on 512 SQuAD2 examples. Each round of interaction in- cludes around 200 interactions with feedback. We observe 1,831 interactions over the nine rounds. We also concurrently collect about 100 test exam- ples per round, total of 869 examples. The total cost of this study 3,330USD.
observe dramatic changes. The initial model under- predicts questions as unanswerable, can be seen by comparing prediction rates in the first round (5.5%) to the ratio of unanswerable questions the test data (stable at 35)40%). This is followed by dramatic correction: the model slightly over- predicts unanswerable in Round 2 (42%). Start- ing from Round 3, the model stabilizes predict around 38% questions unanswerable."
Figure shows statistics from the nine rounds of deployment. The frequency of "correct" feedback increases from 47.71% % during the first round to 69.35% at the last round, with "wrong" feedback decreasing (40.82-x16.08%)
The frequency of "partially correct" remains stable.Wii While the trend is generally stable, we see that temporary dips in performance occur (round 3 and rounds 5-7), even though learning can recover from them. Reward trends are similar, with the total reward increas- ing (1.03+1.29)), and both answerability classifi- cation (0.58 and answer span +0.74) rewards increasing.
Test performance on our concurrently collected test data shows similar positive trend, with both FI (39.22 66.56) and answerability classification accuracy (65.02++00. +80.32%) 32%) increasing over the nine rounds. TyDiQA performance increases initially, but shows a slight drop later on. The early improve- ments using our feedback data indicate similarity between TyDiQA and our user questions. However, the later decrease shows TyDiQA is still somewhat different, so as the model specializes given more
Answerability prediction rates shed light on the learning dynamics, especially early on, when we
Figure 6 and Figure in the appendix show the statistics on the answerable and unanswerahle subscts partitioned hased on model prediction in the feedback data.
*Fxact numbers per round vary slightly hecause of how workers capture tasks on Amazon Mechanical Turk.

---

CORRECT
CARTALLL-CORREOS
WRONG
iment includes total of 5,079 interactions, at a total cost of 6572.88USD. Figure summarizes deployment and test statistics for this experiment, showing test results on the test data concurrently collected during this experiment. The reproduction of the default setup shows similar improvement trends to the long-term experiment (Section 6. 1). We experiment with four variations:
Round
47.71 47.85
11.47
40.83
Round
14.35
37.80
Table
User adaptation study: feedback statisties to the initial model deployed at Round and Round 9,
feedback data, it is expected to see a drop on Ty- DiQA, from which we observe no training signal. This differrnnce is evident, for example, in the rate of answerable questions. Our test data maintains a rate of about 35-40%, while 50% of the examples in the TyDiQA data are unanswerable.
Weaker Initial Model We deploy weaker ini- tial model trained with quarter of the training data (128 examples). This variant shows effective learn- ing, with significant increase in "correct" feedback (26.67-55.61 61))% over the five rounds. However, it still lags significantly behind the default setup with its stronger initialization.
Further breaking down Fl performance based on ground-truth answerability shows that the initial model starts with non-trivial answerable Fl and improves gradually (55.43 -22). But, it starts with very low performance on the unanswerable subset (12.5). It improves over time on the unanswerable subset as well (12.5-74.00), with rapid improve- ment at the very first round (12.5->64.00
Fewer Interactions Per Round (Fewer Ex.) We experiment with updating the model more fre- quently, with each round including around 100 ex- amples instead of 200. We deploy this system for 10 rounds, so the total number of interactions it ob- serves identical. 10 This model performance is on par with that of the default setup. However, the per- centage of unanswerable examples is slightly less stable than others, likely due to the larger variance in data samples.
User adaptation is a potential confounding factor, where performance trends might be explained by users changing their behavior (e.g.. by simplifying their questions) while the model does not actually change its performance. We quantify user adapta- tion by deploying the Round (parameterized by 81) model alongside the final model at Round 9, for the same number of interactions. Interactions are assigned one of the models randomly with no indication to the user. Table 3 shows the feedback comparison for the two models. Except a small difference "partially correct" and "wrong" feed- back, we observe no change the initial model performance, indicating user adaptation does not explain the performance gain.
Answerability Classification (No CLS) We ab- late using a separate answerability classification head. Instead, this model always select span, ei- ther predicting an answer span in the context in- dicating unanswerable given the context paragraph by predicting the CLS token (Devlin et al., 2019). We use a reward value mapping that gives reward of for CORRECT, 0.5 for PARTIALLY-CORESS and -0.1 for WRONG. This model over-predicts unanswerable in the first round, and afterwards keeps showing a lower classification accuracy com- pared to the default setup. Further analysis shows that the final model predicts 29% unanswerable the ground- truth answerable subset, and only 65% unanswerable on the ground-trutie unanswerable subset, significantly worse than the final model un- der default setup (25% and 80%). This indicates explicitly learning separate head for answerability prediction is crucial.
6.2 Analysis on Model Variants
We study the impact of various design decisions by concurrently deploying five systems in ran- domized experiment, including re-deploying our default configuration. Repeatedly deploying the default configuration serves as a fair comparison with other variants, being launched concurrently and sharing the same user pool.
The different variants copy the design decisions of the default configuration, unless specified oth- erwise. Each interaction is assigned with one of the systems randomly. Unless specified otherwise, each system aggregates around 200 interactions per round. We also concurrently collect test set of around 200 examples per round. This exper-
Domain Adaptation (NewsQA Initial) We train an initial model on the complete NewsQA training
10ehe first 5 rounds are deployed concurrently with other variants, and the last 5 rounds are deployed alone. We do not collect test examples alongside the later 5 rounds.

---

Default Setup
Weaker Initial
Fewer Ex.
Ans.
Unans.
Unans.
No CLS
NewsQA Initial
I-Round 56.57
60.73
49.70
24.97
200 400 600 800 1k
200 400 600 800 1k
4-Round
64.34
56,93
79.01
41.41
% Predicted Unans. % "Correct" Feedback
8-Round 63.13
56.98
75.31
40.79
80
Table 4: One-round multi-round: compare (from the top) the model trained around 800 examples in round, the model trained with four rounds of iterative improvement (around 200 ex. per round), and the model with 8-round of improvements (around 100 per round). We report overall FI, the ground-truth answerable subset, Fl the ground-truth unanswerable subset, and the percentage of model preee-pdedieted unanswerable examples.
60
Test Fl
60
40
40
20
20
Test Classification Acc.
80
80
default experiment at the first round. We fine-tune the initial model with this set, essentially doing a single round with 800 examples, quantity that other models only across multiple rounds.
40
40
20
20
200 400 600 800 1k
200 400 600 800 1k
Table compares this one-round model with the Round 4 model from the default setup and the Round model from the inteecrrtteactionss-pa per- round setup, both from Section 6.2. Overall, the one-round model does worse on FI score, illus- trating the benefit of the multi-round design. The higher Fl score the answerable set the one- round model shows is explained by the significant under-prediction of questions as "unanswerable".
# Interactions
Interactions
Figure 3:
Interaction statistics (left) and model test perfor- mance (right) in the short-term experiment. We report test Fl and classification accuracy on the full test data collected across five rounds of deploying those models concurrently.
dataset, a dataset with context texts that are sig- nificantly different than ours. The initial model is trained withou! a classification head, since NewsQA only contains answerable examples. We add randomly initialized answerability classifi- cation head, and randomly output "unanswerable" for 10% of questions during Round so that observe feedback for this classification option. The model initialized with NewsQA shows promising performance, as previously indicated in simulated studies (Gao et al., 2022). In fact, it is the best- performing model in this study. We hypothesize that the model benefits from the larger initial train- ing set, despite the domain shift, and only requires relatively small tuning for the new domain.
6.4 Sensitivity Analysis
We characterize the sensitivity of the bandit learn- ing process through controlling the set of feedback data and the of initial models.
Sensitivity Analysis of Round 1 Models Ana- lyzing model sensitivity over multiple rounds of interaction prohibitively costly. We focus on analyzing the sensitivity of the model at Round relatively to the set of feedback examples is be- ing trained on. We collect 10 different sets of 200 examples and compare the learning outcomes.
Figure 4 shows the performance of models learned from those 10 different sets. The overall Fl score, Fl on the answerable subset, and classifi- cation accuracy present relatively smaller variance (standard deviation ar 3.32, 2.89, and 2.44) on the full test set from Section 6.1. We observe a larger variance in the Fl on the unanswerable subset and in the percentage of predicted unanswerable exam- ples (o 11.51 and 9.55). This variance may be due to the challenge of improving the binary classifi cation head on unbalanced data: the percentage of model--reeicced unanswerable examples the feedback data 7% on average across different sets during the first round.
6.3 One-round vs. Multi-round
Iterative rounds of deployment and learning is our approach design. An alternative would be to collect all the feedback data with the initial model, and then update the model once. We compare our design to this alternative.
Concurrent to the Round 1 deployment in Sec- tion 6.2, we collect additional 500 feedback inter- actions from the 512-SQuAD2 initial model, and create of 800 interactions together with 100 ex- amples from the experiment on inwerrinteractions- per-round experiment and 200 examples from the

---

100
SQuAD2 models on average only predict 11.9% unanswerable outputs on our test set.
Fl Accuracy Percentage
80 10-00-0000
7
Conclusion
60
We deploy an extractive QA system, and improve it by learning from human user feedback, observing strong performance gains over time. Our experi- ments show such continual improvement of sys- tems is compelling for domain adaptation, provid- ing practical avenue for practitioners to bootstrap NLP systems from existing academic datasets, even when the domain gap relatively large. We also study the effects of the update schedule (i.e., how frequently to train and re-deploy new model) and model design (i.e., by using a separate answerabil- ity classification head). Through our focus on a lexically-rich user-facing scenario, we hope our study will inspire further research of NLP prob- lems in interactive scenarios and the learning poten- tial user interactions provide. Important directions for future work include learning from other, more complex signals (e.g., natural language feedback), studying non-extractive scenarios, and developing robustness to adversarial behaviors (e.g., attempts to sabotage the system through feedback).
40
Ans. FI Unans. FI CLS Acc. Uunans.
Figure 4: Performance of Round models learned from 10 different sets of feedback examples on the full test set: Fl, Fl on the ground truth answerable subset (Ans. F1), Fl on the ground truth unanswerable subset (Unans, FI), classi- fication accuracy (CLS Acc.), and percentage of predicted wannswe outputs (UUna.s.). Bars represent variance, and are centered at the mean value.
Our Test
SQuAD2 Dev
60
40
20
128-SQuAD2 512-SQuAD2
Figure 5: Performance of initial models trained on 10 different sets of 128 or 512 SQuAD2 examples our full test set and the SQuAD2 development set. Bars represent variance, and are centered at the mean value.
Limitations
Sensitivity Analysis of Initial Models The ini- tial models used in our study are initialized us- ing either 512 or 128 supervised examples from SQuAD2 dataset. We analyze the variance they may present. We randomly sample different sets of supervised examples from the SQuAD2 training data to study the sensitivity of the initial model to its supervised training data.
We simulate information-seekien users with paid crowdworkers. This has limitations far as reflect- ing the behaviors and incentives of real users. For example, it is not clear how often real users will provide feedback at the end of the interaction, how much effort they will put into validating the system output. On the other hand, crowdworkers have less incentive to design their questions so that they will be able to get a satisfying answers, as op- posed to real users who likely require the answers to achieve their intent. Also, changes in user popu- lation and preferences are unlikely to be reflected in crowdworkers behavior.
Figure 5 shows the performance of initial models trained on 10 different sets of 512 128 SQuAD2 examples our full test set from Section 6.1 and SQuAD2 development set. On SQuAD2 devel- opment set, 512-SQuAD2 models show a smaller variance in than 128-SQuAD2 models (standard deviation a 2.2 vs 4.7). On our test set, we ob- serve different result: 512-SQuAD2 models have a larger variance in Fl than 128- SQuAD models (a 6.9 vs 2.1). We find that the larger variance in 312-SQuADZ model performance our test set is due the large variance in F1 the ground- truth unanswerable subset: a 24.4. In contrast, on the ground-truth unanswerable subset for 128-SQuAD2 models have standard deviation 6.4, but with a much lower mean (13.6 vs 38.6). 128-
We only elicit cooperative interactions in our experiments. Our approach does not address po- tential adversarial behavior, implicitly assuming collaborative users. However, real-life scenarios are unlikely to allow such assumptions. In general, adversarial user behavior forms security risk for learners that rely on user feedback, and even more when user input is part of the process, allowing users stronger control on how they may adversar- ially attempt to steer the system. This aspect NLP systems is currently not broadly studied. We

---

hope our work will enable further work into this important problem.
References
Yoav Artzi and Luke Zettlemoyer. 201 Bootstrapping semantic parsers from conversations. EMNLP.
Other aspects constraining the generality of our conclusions, but are not strictly limitations of our specific work, are our focus extractive QA, where the output is relatively easy to visualize for validation, and our simple feedback mechanism. which is not information rich as other potential modes feedback (e.g., natural language). We hope our work will serve as a catalyst for studies that cover these important problems.
Alberto Bietti, Alekh Agarwal, and John Langford. 2021. A contextual bandit bakc-off. Journal of Ma- chine Learning Research.
Jannis Bulian, Christian Buck, Wojeiech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022. Tomayto, tomahto. beyond token-level answer equiv- alence for question answering evaluation. EMNLP.
Jon Ander Campos, Kyunghyun Cho, Arantxa Otegi. Aitor Soroa Etxabe, Gorka Azkune, and Eneko Agime. 2020. Improving conversational question answering systems after deployment using feedback- weighted learning. COLING.
Legal and Ethical Considerations
Section and Appendix A detail our experimental setup, hyperparameter search, and computational budget to support reproducibility. Our data and codebase are released at /ttps://gi thub. com/ Lil-Lab/ga-from-ht
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. Quac: Question answering in context. EMNLP.
SQuAD2 and TyDiQA that we use for initial training and evaluation are publicly available datasets from prior work.
Paul Christiano. Jan Leike, Tom B. Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. NeurIPS.
The content shown to crowdworkers in our stud- ies is from Wikipedia articles, so we expect crowd- workers participating in our task to be free from exposure to harmful content and the content to not pose intellectual property issues. While our data does not contain harmful content, cannot predict its impact when it or our models are used beyond research context similar to ours. We urge such uses to be accompanied by careful analysis and appropriate evaluation.
Clark, Eunsol Choi, Michael Collins, Dan Gar- rette, Tom Kwiatkowski, Vitaly Nikolaev, and Jen- nimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering typologi- cally diverse languages. TACL.
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. ACL.
In general, learning from user feedback, espe- cially user-generated data, poses risks for divert- ing the system behavior in unwelcome directions. Identifying and mitigating such risks are open and important problems for future work to study.
Julian Martin Eisenschlos, Bhuwan Dhingra, Jannis Bu- lian, Benjamin Borschinger, and Jordan L. Boyd- Graber. 2021. Fool me twice: Entailment from wikipedia gamification. NAACL.
Acknowledgements
Ge Gao, Eunsol Choi, and Yoav Artzi. 2022. Simulating bandit learning from user feedback for extractive question answering. ACI.
This rescarch was supported by Open Philanthropy, a Google Faculty Research Award, and NSF un- der grant No. 1750499. We thank participating crowdworkers on Amazon Mechanical Turk for their contributions, and hope they enjoyed inter- acting with our system. We thank Kianté Brantley. Jonathan D. Chang, Justin Chiu, Daniel D. Lee, and Travers Rhodes for helpful discussions. We thank Justin Chiu and Michael J.Q. Zhang for their valu- able feedback on our drafts. We thank members of Cornell NL.P and UT Austin NL. community for providing feedback.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre- training with gradient-disentangee embedding shar- ing. ICLR.
Daniel G. Horvitz and D. J. Thompson. 1952. A gen- eralization of sampling without replacement from a finite universe. Journal of the American Statistical Association.
Srini Tyer, loannis Konstas, Alvin Cheung, Jayant Kr- ishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. ACL.

---

Natasha Jaques, Judy Hanwen Shen, Asma Ghandehari- oun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2020. Human- centric dialog training via offline reinforcement leamn- ing. EMNLP.
Rejichiro Nakano, Jacob Hilton, Arun Balaji, Jeff Wu. Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser- assisted question-answering with human feedback. ArXiv.
W. Bradley Knox and Peter R Stone. 2015, Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and per- formance. Artificial Intelligence.
Khanh Nguyen, Hal Daumé III, and Jordan Boyd- Graber. 2017. Reinforcement learning for bandit neural machine translation with simulated human feedback. EMNLP.
Noriyuki Kojima, Alane Suhr, and Yoav Artzi. 2021. Continual learning for grounded instruction genera- tion by observing human following behavior. TACL.
Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What Ims know about unseen entities, ArXiv.
Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018a. Can neural machine transla- tion be improved with user feedback? NAACL.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv.
Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. 2018b. Reliability and learnability of human ban- dit feedback for sequence-to-sequenc reinforcement learning. ACL.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins. Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe V. Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. TACL.
Vianney Perchet, Philippe Rigollet, Sylvain Chassang, and Erik Snowberg. 2016. Batched bandit problems. The Annals of Statistics.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, G. Sperl, and Christoph H. Lampert. 2017. icarl: Incremental classifier and representation learning. CVPR.
Carolin Lawrence and Stefan Riezler. 2018. Improving neural semantic parser by counterfactual learning from human bandit feedback. ACI.
"er'emy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho. and Ethan Perez. 2023. Training language models with language feedback scale, ArXiv.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. ICLR.
Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Chi Kit Cheung, and Siva Reddy. 2022. Using interactive feedback to improve the accuracy and explainability of question answering systems post- deployment. Findings@ACL. ACL.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler. Ryan J. Lowe, Chelsea Voss, Alec Radford. Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. NewrIPS.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv.
Alane Suhr and Yoav Artzi. 2022. Continual learn- ing for instruction following from realtime feedback. ArXiv.
Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. 2015. Learning interpret natural lan- guage commands through nubont dialog. IJ- CAI.
James MacGlashan, Mark K. Ho, Robert Tyler Loftin, Bei Peng, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. 2017. Interactive learning from policy-dependent human feedback. ICMI.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts,
Volodymyr Mnih, Adrià Puigdomênech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asyn- chronous methods for deep reinforcement learning. ArXiv.

---

Maarten Bosma, Yanqi Zhou, Chung- Ching Chang. I. A. Krivokon, Willard James Rusch, Marc Pick- ett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Søraker, Ben Zevenbergen, Vin- odkumar Prabhakaran, Mark Díaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Brin Hoffman- John, Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Femton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arees Arcas, Claire Cui, Marian Croak, Ed Huai hsin Chi, and Quoc Le. 2022. Lamda: Language models for dialog applications. ArXiv.
Sida I. Wang, Percy Liang, and Christopher Manning. 2016. Learning language games through interaction. ACL.
Garrett Warnell, Nicholas R. Waytowich, Vernon Lawh- ern, and Peter Stone. 2017. Deep tamer: Interac- tive agent shaping in high-dimensional state spaces. CoRR.
Ronald J. Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, R' R'emi Louf, Morgan Funtow- icz, and Jamie Brew. 2020. Huggingface's 'ss trans- formers: of-e-oofttheann natural language processing. EMNLP.
Jing Xu, Megan Ung, Mojtaha Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. 2022. Learn- ing new skills after deployment: Improving open- domain internet-driven dialogue with human feed- back. ArXiv.
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amadei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. ArXiv.

---

A Training Details
short introduction to each topic. After selecting a topic and an aspect, workers are instructed to ask a relevant question that they do not have an answer to. We explicitly guide workers to avoid yes-or-no questions and questions that obviously cannot be answered by a short single sentence.
Training Initial Models We randomly sample 512 training examples from the SQuAD2 training set and use learning rate of 3e-5 with a linear schedule, batch size 10, and 10 epochs. The loss terms for span prediction and answerability clas- sification are simply added. We use the same hy- perparmeter setup for training the initial model on 128 SQuAD2 examples except that we add coef- ficient of 10 to the answerability classification loss. For the initial model trained on NewsQA, we use learning rate 2e-5 with linear schedule, batch size 40, and epochs.
Feedback Guidelines We explicitly guide work- ers to read the context paragraph in full to give feedback when the model returns "unanswerable", and to give "partially correct" feedback if the an- swer contains irrelevant information. Beyond that, we guide workers through examples that express the following guidelines:
"Correct" the answer highlighted in the con- text paragraph correctly answers the question, or points out that the question cannot be an- swered by single span in the paragraph.
Bandit Learning We use batch size 35, and per- form hyperparameter search with a linear schedule of learning rates (5e-5, 3e-5, le-5, 9e-6, 7e-6, 5e-6, 3c-6), number of epochs (14, 20, 24, 30), and en- tropy penalty coefficients y (1.5, 1.7, 1.9, 2.0, 1. 2.3, 2.5, 3.0, 5.0). We pick the checkpoint with highest Fl on our development set, a set of 402 examples annotated during early pilot studies. At each round, these hyperparameter searching exper- iments take around 105h on one NVIDIA GeForce RTX 2080 Ti.
"Partially Correct" if the given answer only partially answers the question.
"Wrong" if the given answer highlighted in the paragraph does not answer the question at all, or if the output is "unanswerable given the paragraph" when the question can be an- swered by span in the paragraph.
Test Data Annotation We collect test examples with ground-truth answer annotation. We ask work- ers to annotate answers to their questions instead of providing feedback to model-predicted answers. Workers are instructed to either select a single span in the paragraph question is unanswerable given the context para- graph. We intentionally combine this annotation task with the main feedback task, so that work- ers do not know whether they will provide anno- tation to their own question or provide feedback to a model answer. In order to have three annota- tions per test example, we design an additional task to collect two extra annotations per test example: workers are given a question and context paragraph, and provide answer annotation. We qualify workers by providing examples of reasonable annotations and only work with workers who correctly pass a qualifier composed of three questions.
B
User Study Setup
We design an interface where users ask information- seeking questions and provide feedback to model- predicted answers to their own questions." II Fig- ure 8 shows screenshots of our interface. We use this interface throughout our studies.
answer or indicate that the
We familiarize workers with our task and criteria using detailed guidelines and examples of reason- able questions and user feedback. Before doing our tasks, workers must pass qualification pro- cess by correctly answering least 9/10 questions regarding the task. The examples we use to guide workers and the qualification tests for both the feed- back and test data annotation tasks are available at https: //g wttpttuww.ww.w///////eeeeeeee//e tree//main/data-colletttt
Question Writing Guidelines We encourage workers to ask information-seekinng questions by having them pick topic they find interesting among four topic randomly selected from pool, and then choose aspect out of four regarding the chosen topic. For topic selection, provide
During early pilot studies, we collect a small set (402 examples) of annotated examples. We use this set for hyperparameter tuning during development.
Worker and Study Details For both feedback task and annotation task, we work with turkers that have a HIT (Human Intelligence Task) rate greater than 98% with at least 500 completed HITs.
Il We customize a UI design from CodingNepal on "Create a Quiz App with Timer using HTML CSS & JavaScript".

---

Unanswerable
Answerable
Question: How large was the sea at its peak volume? Answer: [U Unanswerable given the context paragraph] Context: The Amu Darya river flowed into the Caspian Sea Uzboy channel until the Holocene. Geogra- pher Nick Middleton believes it did not begin to flow into the Aral Sea until that time.
100
100
80
80
&eedbuck
Feedback
60
60
Feedback: Correct
40
Wrong
40
Wrong
Partal
Partial
Question: Who was the original inventor program- Solaris?
20
20
Correct
Conrect
Answer: [Unanswerable given the context paragraph]
2
1 123456788
Context: In 1987, AT&T Corporation and Sun an- nounced that they were collaborating on project to merge the most popular Unix variants the market at that time On September 4, 1991, 1, Sun announced that it would replace its existing BSD-derivel Unix. SunOS with hased SVR4. This identified internally as SunOS 5, but new marketing name was introduced the same time: Solaris 2
Round
Round
Figure 6: Percentage of user feedback received at every round during the long-term experiment. Left column shows the subset where the model predicts unanswerable, and the right column shows the complement.
Feedback: Partially Correct
Question: Who was the person most suspected of be- ing Jack Ripper?
Unanswerable
Answerable
Answer: an educated upper- man. possibly doe- tor
100
100
Misc.
Context: The concentration of the killings around
Yes/No
weekends and public holidays and within short dis- tance of each other has indicated to many that the Ripper was in regular employment and lived locally. Others have opined that the killer was educated up- per-class man, possibly a doctor or an aristocrat who ventured into Whitechapel from more well-to-do area. Such theories draw on cultural perceptions such as
How
80
80
Count
What
Which
Question Types
Question Types
Why
60
60
Where
Who
Feedback: Partially Correct
When
MM
Question: In which state Ulm located in?
10
Answer: Bavaria
Context: Ulm lies at the point where the rivers Blau and
Iller join the Danube, at an altitude of 479 (1.571.52 ft) above sea level. Most parts the city, including the old town, are situated on the left bank of the Danube; only the districts of Wiblingen. Gogglingen. Donaustet- ten and Unterweiler lie on the right bank. Across from the old town, on the other side of the river, lies the twin city Neu-Ulm Ulm the Ulm and, until 1810, a part of it.. Feedback: Wrong
20
7 0 123456789
of Bavaria, smaller than
Round
Round
Figure 7: Statistics of question types per round in the long- term experiment. Left column shows the subset where the model predicts unanswerable, and the right column shows the complement.
Table 5: User interaction examples: each example is composed of user question, context paragraph, model-predicted answer. and user feedback.
C Data in Long-Term Experiment
Interaction Examples Table lists additional in- teraction examples from our long-term study (Sec- tion 6.1).
Across all experiments, 103 turkers have passed our qualification test and participated in our tasks. No turker has done more than 5% of the HITs in total. We pay $0.65 USD for each feedback example and S0.4 USD for each additional annotation. The estimated hourly pay for both tasks are $15 USD. Our user study spans from February 8th to April 25th, 2023.
User Feedback Figure visualizes the distribu- tion of user feedback at every round, partitioned based on whether the model predicts unanswerable or an answer span. The percentage of "correct" feedback is increasing on both subsets, illustrating

---

that the model is improving over time.
QA Data Figure visualizes the distribution of question types, partitioned based on whether the model predicts unanswerable or an answer span. The distribution over question types varies at each round, with the majority of questions starting with "what" and "when". The model mostly predicts span in the context answer questions regarding "where", while it chooses the unanswerable option more often towards counting-related questions.

---

Instructions
This task includes 4 steps:
of topics. provide brief intro each topic. 2. Select aspect of the chosen topic ask question about.
Choose topic that interests you
3. Ask question that you are curious to know more about. The question must be about the topic and aspect you selected.
4. will generate an answer your question, and highlight paragraph, show
*[Unanswerable given the paragraph below]". You will provide feedback response given the paragraph (correct, partially correct incorrect.).
When asking your question, it is critical that:
x You ask question that can be answered by a short single sentence. We can only answer such questions.
* NOT ask yes-or-no question.
* Put question mark at the end of your question. Otherwise, the button to the next step will not show
When providing feedback the given
keep in mind that:
X We may return "[nnanswerable given the paragraph below]".
* the given answer is "unanswerable" please read the full paragraph make sure that this paragraph cannot answer your question. you find correct answer in paragraph, please indicate no-answer output "incorrect".
* If the given answer could answer your question but contains irrelvaan content, label as "partially
correct'.
Please read our example google doc.
First-time users need to successfully complete qualifier questionnaire before starting the task.
Keyboard shortcuts:
* 1, 2, numbered options each single selection
* Enter: go to the next step, submit after completing all steps
* Please read our consent form before starting the task. By continuing with this task, you acknowledge that you understand our consent form, and agree to take part in this research.
Start
Topic Selection
Select topic that interests you:
Wrigley Field is Major League Baseball (MLB) stadium located on the North the Chic...
Wrigley Field
Side Chicago, Illinois.
The Lighthouse of Alexandria, sometimes called the Pharos of Alexandria (: Ancient Greek: Øápoç Tńςς AλεEavōpelaç, contemporary K...
2. Lighthouse of Alexandria
Andrew Cuomo KWOH-moh; Italian: ['kwoomo]: born December 1957) American lawyer and politician who served as the 58...
3. Andrew Cuomo
Hepburn: Okinawa-sen), codenamed Operation lceberg.:17 7 was major battle of the Pacific W...
The Battle of Okinawa (Japanese: >
Battle of Okinawa
Optional discussion forum: Discord
of Steps
Instructions

---

Aspect Selection
If needed, below is brief intro to Wrigley Field
Wrigley Field is Major League Baseball (MLB) stadium located the North Side Chicago, Illinois. the home of the Chicago Cubs, one of the city's two MLB franchises. first opened in 1914 as Weeghman Park for Charles Weeghman's Chicago Whales of the Federal League, which folded after the 1915 baseball season. The Cubs played their first home game at the park April 20, 1916, defeating the Cincinnati Reds 7-6 innings. Chewing gum magna...
Select an aspect that interests you:
Accessibility and transportation
2. History
Commemorative stamps
Features
Optional discussion forum: Discord
2 of 4 Steps
Instructions
Restart
Question
If needed, below is a brief intro to Wrigley Field
Wrigley Field is Major League Baseball (MLB) stadium located the North
Chicago, Illinois. the the city's two MLB franchises. first opened in 1914 as Weeghman Park for
home of the Chicago Cubs, Charles Weeghman's Chicago Whales of the Federal League, which folded after the 1915 baseball season. The Cubs played their first home game at the park on April 20, 1916, defeating the Cincinnati Reds 7-6 in innings. Chewing gum magna...
Ask a question about History
When was Wrigley Field built?
Optional discussion forum: Discord
Steps
Instructions
Restart
Next Step

---

Feedback
Question:
When was Wrigley Field built?
Answer:
April 23, 1914
Context:
Baseball executive Charles Weeghman hired his architect Zachary Taylor Davis to design the park, which was ready for baseball by the home opener on April 23, 1914. The original tenants, the Chicago Whales (also called the Chi-Feds), came second the Federal League rankings in 1914, won the league championship resourceful Weeghman formed syndicate including the chewing gum manufacturer William Wrigley Jr. to buy the Chicago Cubs from Charles P. Taft for about $500,000 Weeghman immediately moved the Cubs from the dilapidated West Side Grounds his two-year-old park. In 1918, Wrigley acquired the controlling interest in the club. In November 1926, he renamed the park Wrigley Field. in 1927, upper deck added, and 1937, Bill Veeck, the of the club president, planted ivy vines against the outfield walls after seeing the ivy planted at Perry Stadium, Indianapolis.
1915. In late 1915, Weeghman's Federal League folded. The
How do you evaluate the highlighted answer?
Correct
2. Partially correct
3. Wrong
Optional discussion forum: Discord (Feel free ask questions regarding your HIT.) Optional comment:
Any comments or feedback about this task? Type here...
4 of 4 Steps
Instructions
Figure 8: Snapshots of interface for feedhack task.


Reviews:
[{"review_id": "hREKcOTq0t", "reviewer": "Reviewer_4d65", "summary": "", "questions": "1 Can you better clarify how you can ensure feedback quality?\n2 Why you did not perform an evaluation of the annotators job?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We evaluated feedback quality, as described in Footnote 7. Overall, our feedback data was relatively clean (18% noise rate). We will move this content to the main text and discuss it further. We used a qualifier and MTurk criteria (e.g., 98% acceptance rate) to filter spamy workers, and make sure workers understand the system and are well qualified to interact with it (i.e., that they are similar to users in this respect). While this noise rate may seem high, our method is robust to it, which is the important thing. Equally important, our feedback providers are not annotators, but take the place of users. The feedback comes to reflect naturally occuring feedback. So training our workers (beyond what we did) to give perfect feedback is actually not desirable, because you wouldn’t train your users when deploying a system. We agree there are likely interface design choices that can influence feedback quality, and we believe this is an important direction for future work in the intersection of HCI and NLP that our work enables. Another exciting option for future work is thinking of methods to filter bad feedback. Our work doesn’t come to establish the ultimate method. It comes to show effectiveness and feasibility, and on both accounts we believe our results are extremely strong (all that with real human feedback!). There is only so much that can come within the scope of an ACL paper, and our experimental costs are already very high because each experiment requires human work. As the reviewer notes, the scope of this area of research is huge, and there are so many exciting questions. We hope the reviewers agree that it’s a strength of our work that it outlines so many exciting directions for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ well written paper\n+ novel approach to improve extractive QA\n+ relevant results and a new interesting research for qa systems\n+ good experiments and ablations", "reasons_to_reject": "+ miss a proper evaluation of the quality of the crowdworkers to avoid misleading errors\n+ no clear how ensure feedback quality", "questions_to_authors": "1 Can you better clarify how you can ensure feedback quality?\n2 Why you did not perform an evaluation of the annotators job?", "paper_topic_and_main_contributions": "The paper is about a reinforcement learning for extractive QA. authors trained a model in different rounds using human feedback to improve the model. They also tested different version of the model with different initialization and on different dataset to ensure the performances on different domain. Authors relies on crowdworkers for the feedback. Performances show the effectiveness of the approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jKrf1qJ0fZ", "reviewer": "Reviewer_CpKn", "summary": "", "questions": "1. What's the possible performance upper bound of the proposed method? How many rounds of interactions are optimal in general?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. \n\nRegarding the reproducibility of our experiments: our code and data will be made publicly available. They will enable others to easily reproduce the experiments reported in the paper. We agree that working with human users introduces uncertainties to reproduction. This is a general challenge when working with humans (same for human evaluation), but there are no substitutes for working with humans for both learning from human feedback (as we do) and human evaluation. This is a known research challenge in this area, but does not detract from how important/necessary these problems are and how excited the community is about them. Even more important, we actually worked really hard to show how reproducible our experiments are, and directly show that our work is robust to this uncertainty. This is shown by our reproduction of the default setup in a separate experiment. Our reproduction (Section 6.2 and default setup plot in Figure 3) shows similar improvements trends to the long-term experiment (Section 6.1 and Figure 2). The cost and user study details are clearly reported in the main paper and Appendix B to further support the reproducibility of our work.\n\nRegarding the question on the possible performance upper bound: our goal was to show the feasibility and effectiveness of learning from feedback with a straightforward approach. The upper bound question is interesting, but complex and likely doesn’t have a clear answer. This is because of the constant distribution shift. Throughout the lifetime of the system, and as it learns, there are two major causes for constant data distribution shift: (a) the world changes; and (b) the users change their behavior in response to the system changing its behavior or as new users start using the system. In this sense, this scenario reflects a multi-agent system (with the system being one agent, and the human users as many other agents). This means that even if performance doesn’t change much after a while (something that may happen), the adaptation doesn’t stop. So this process should continue throughout the system’s lifetime. Further study of this interesting problem is something that our work enables, as the first to show such a system improving over time. We will discuss this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose a method to continually improve extractive QA via human feedback. The proposed method does have high practical value and may be applied in building other NLP application systems. \n\n2. Extensive and carefully designed experiments demonstrate that an extractive QA system can be improved via interaction with human users over time.", "reasons_to_reject": "1. Uncertainties introduced by human interactions are likely to affect the results and need to be carefully controlled with a well-defined guideline.", "questions_to_authors": "1. What's the possible performance upper bound of the proposed method? How many rounds of interactions are optimal in general?", "paper_topic_and_main_contributions": "The authors propose a method to continually improve extractive QA via human feedback. The method does have high practical value and may be applied in building other NLP application systems. Extensive and carefully designed experiments demonstrate that an extractive QA system can be improved via interaction with human users over time and further broaden our understanding about continual learning with user feedback.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos or grammar errors:\n\n1. line 345, \"We are focused on ...\"\n2. line 369, \"the systems sees\"\n3. line 481, \"does not an explain ...\"\n4. line 555, \"the that\"\n5. line 620, \"out approach\""}, "extra_text": {}}, {"review_id": "iIWaKiaSpp", "reviewer": "Reviewer_Qenp", "summary": "", "questions": "A. What is the advantage of improving using human feedback compared to using human labels to continuously train the model? It is easier and cheaper to collect? Or other advantages?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. \n\nUser feedback is fundamentally different from labeled data (i.e., as obtained from annotators). First, it doesn’t cost money (so: free!), but arises naturally from the interaction of users with the system. We paid workers to take the place of users, but this is just to run experiments. Second, it’s much more abundant, because it arises from users interacting with the system, and pools of users are generally orders of magnitude larger than pools of annotators. Third, while annotated data reflects an assumption about a static data distribution, learning from feedback over the system’s lifetime enables taking into account adapting over time to evolving data distribution, both as users change their behavior and the world changes (i.e., the answers to questions change – see Zhang and Choi EMNLP 2021). So, while it creates a much more challenging learning scenario (which we show our method effectively addresses), it is across the board much better as a learning signal (once you can show that you can use it effectively, which we show). We discussed this motivation in the introduction, but will further improve it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of improving QA with three-option human feedback is nice, which shed lights on iteratively improving deployed QA systems. I am not an expert on reinforce learning, but the methodology is clear and able to follow. From the experimental results, the improvements are significant. The paper also provide comprehensive discussions on variants of experimental settings.", "reasons_to_reject": "The motivation of improving extractive QA with human feedback is not clearly explained. By reading the introduction, I don't have a clear sense of why we need human feedback for QA.", "questions_to_authors": "A. What is the advantage of improving using human feedback compared to using human labels to continuously train the model? It is easier and cheaper to collect? Or other advantages?", "paper_topic_and_main_contributions": "The paper focuses on extractive QA. Specifically, the paper proposes an approach to", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a pipeline to improve an extractive QA system using human feedback loop. The proposed approach is not only feasible for an extractive QA system, but has potential to be extended to other NLP applications. Reviewers appreciated the well-strutted content, innovative methods, and comprehensive experiments. 

The questions raised by reviewers including the motivation of using human feedback vs. human labels, the quality of human feedback, and uncertainties of human feedback, are appropriately addressed by the authors. After the rebuttal period, the authors mentioned releasing the annotated dataset and annotation guidelines, which would enhance reproducibility.
Decision: Accept-Main