Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA
Year: 2023

Abstract:
Despite remarkable progress made in natural language processing, even the state-of-the-art models often make incorrect predictions. Such predictions hamper the reliability of systems and limit their widespread adoption in realworld applications. Selective prediction partly addresses the above concern by enabling models to abstain from answering when their predictions are likely to be incorrect. While selective prediction is advantageous, it leaves us with a pertinent question ‘what to do after abstention’. To this end, we present an explorative study on ‘Post-Abstention’, a task that allows re-attempting the abstained instances with the aim of increasing coverage of the system without significantly sacrificing its accuracy. We first provide mathematical formulation of this task and then explore several methods to solve it. Comprehensive experiments on 11 QA datasets show that these methods lead to considerable risk improvements –performance metric of the Post-Abstention task– both in the in-domain and the out-of-domain settings. We also conduct a thorough analysis of these results which further leads to several interesting findings. Finally, we believe that our work will encourage and facilitate further research in this important area of addressing the reliability of NLP systems.

heading: 1 Introduction
text: Despite remarkable progress made in Natural Language Processing (NLP), even the state-of-the-art systems often make incorrect predictions. This problem becomes worse when the inputs tend to diverge from the training data distribution (Elsahar and Gallé, 2019; Miller et al., 2020; Koh et al., 2021). Incorrect predictions hamper the reliability of systems and limit their widespread adoption in real-world applications.
Selective prediction partly addresses the above concern by enabling models to abstain from answering when their predictions are likely to be incorrect.
By avoiding potentially incorrect predictions, it allows maintaining high task accuracy and thus improves the system’s reliability. Selective prediction has recently received considerable attention from the NLP community leading to development of several methods (Kamath et al., 2020; Garg and Moschitti, 2021; Xin et al., 2021; Varshney et al., 2022d). While these contributions are important, selective prediction leaves us with a pertinent question: what to do after abstention?
In this work, we address the above question and present an explorative study on ‘Post-Abstention’, a task that allows re-attempting the abstained instances with the aim of increasing coverage of the given selective prediction system without significantly sacrificing its accuracy. Figure 1 illustrates the benefit of employing a post-abstention method; a model that achieves an accuracy of 70% is first enabled with the selective prediction ability that increases the accuracy to 85% but answers only 71% instances. Then, a post-abstention method is employed (for the 29% abstained instances) that assists the system in answering 9% more instances raising the coverage to 80% without considerably dropping the overall accuracy. We note that this task allows re-attempting all the abstained instances but does not require the system to necessarily output predictions for all of them i.e. the system can abstain even after utilizing a post-abstention method (when it is not sufficiently confident even in its new prediction). This facet not only allows the system to maintain its performance but also provides opportunities of sequentially applying stronger post-abstention methods to reliably and optimally increase the coverage in stages.
We provide mathematical formulation of the post-abstention task and explore several baseline methods to solve it (Section 2). To evaluate the efficacy of these methods, we conduct comprehensive experiments with 11 Question-Answering datasets from MRQA shared task (Fisch et al., 2019) in
967
both in-domain and out-of-domain settings (Section 3). Our post-abstention methods lead to overall risk improvements (performance metric of the proposed task) of up to 21.81 in the in-domain setting and 24.23 in the out-of-domain setting. To further analyze these results, we study several research questions, such as ‘what is the extent of overlap between the instances answered by different postabstention methods’, ‘what is the distribution of model’s original confidence on instances that get answered in the post-abstention stage’, and ‘how often do the system’s predictions change after applying post-abstention methods’. In Section 4, we show that these investigations lead to numerous important and interesting findings.
In summary, our contributions are as follows:
1. We present an explorative study on ‘PostAbstention’, a task that aims at increasing the coverage of a given selective prediction system without significantly sacrificing its accuracy. 2. We explore several baseline post-abstention methods and evaluate them in an extensive experimental setup spanning 11 QA datasets in both in-domain and out-of-domain settings. 3. We show that the proposed post-abstention methods result in overall risk value improvements of up to 21.81 and 24.23 in the in-domain and out-of-domain settings respectively. 4. Our thorough analysis leads to several interesting findings, such as (a) instances answered by different post-abstention methods are not mutually exclusive i.e. there exist some overlapping instances, (b) instances that get answered in the post-abstention stage are not necessarily the ones on which the given system was initially
most confident, etc. We believe our work will encourage further research in Post-Abstention, an important step towards improving the reliability of NLP systems.

heading: 2 Post-Abstention
text: In this section, we first provide background for post-abstention (2.1) and then describe the task (2.2) and its approaches (2.3).

heading: 2.1 Background
text: Post-abstention, as the name suggests, is applicable for a system that abstains from answering i.e. a selective prediction system. A system can typically abstain when its prediction is likely to be incorrect. This improves the reliability of the system. Such a system typically consists of two functions: a predictor (f ) that gives the model’s prediction on an input (x) and a selector (g) that determines if the system should output the prediction made by f :
(f, g)(x) = { f(x), if g(x) = 1 Abstain, if g(x) = 0
Typically, g comprises of a prediction confidence estimator g̃ and a threshold th that controls the level of abstention for the system:
g(x) = 1[g̃(x)) > th]
A selective prediction system makes trade-offs between coverage and risk. Coverage at a threshold th is defined as the fraction of total instances answered by the system (where g̃ > th) and risk is the error on the answered instances.
With decrease in threshold, coverage will increase, but the risk will usually also increase. The
overall selective prediction performance is measured by the area under Risk-Coverage curve (ElYaniv et al., 2010) which plots risk against coverage for all confidence thresholds. Lower AUC is better as it represents lower average risk across all confidence thresholds.
In NLP, approaches such as Monte-Carlo Dropout (Gal and Ghahramani, 2016), Calibration (Kamath et al., 2020; Varshney et al., 2022c,d; Zhang et al., 2021), Error Regularization (Xin et al., 2021) and Label Smoothing (Szegedy et al., 2016) have been studied for selective prediction. In this work, we consider MaxProb (Hendrycks and Gimpel, 2017), a technique that uses the maximum softmax probability across all answer candidates as the confidence estimator. We use this simple technique because the focus of this work is on postabstention i.e. the next step of selective prediction. However, we note that the task formulation and the proposed methods are general and applicable to all selective prediction approaches.

heading: 2.2 Task Formulation
text: We define the post-abstention task as follows: Given a selective prediction system with an abstention threshold, the post-abstention task allows re-attempting the abstained instances with the aim of improving the coverage without considerably degrading the accuracy (or increasing the risk) of the given system.
Next, we mathematically describe the task and its performance evaluation methodology.
Let the coverage and risk of the given selective prediction system at abstention threshold th be covth and riskth respectively. A post-abstention method re-attempts the originally abstained instances (where g̃ < th) and outputs the new prediction for the ones where it is now sufficiently confident. This typically leads to an increase in the coverage of the system with some change in the risk value; let the new coverage and risk be cov′th and risk′th respectively. From the risk-coverage curve of the given system, we calculate its risk at coverage cov′th and compare it with risk ′ th to measure the efficacy of the post-abstention method (refer to Figure 2).
For a method to have a positive impact, its risk (risk′th) should be lower than the risk of the given system at coverage cov′th. We summarize this performance evaluation methodology in Figure 2. To get an overall performance estimate of a post-
abstention method, we compile these differences in risk values for all confidence thresholds and calculate an aggregated value. The higher the overall improvement value, the more effective the method is. We note that this evaluation methodology is fair and accurate as it conducts pair-wise comparisons at equal coverage points. An alternative performance metric could be AUC but it computes the overall area ignoring the pair-wise comparisons which are crucial for our task because the coverage points of the original system would be different from those achieved by the post-abstention method.

heading: 2.3 Approaches

heading: 2.3.1 Ensembling using Question Paraphrases
text: It is well known that even state-of-the-art NLP models are often brittle i.e. when small semanticpreserving changes are made to the input, their predictions tend to fluctuate greatly (Jia and Liang, 2017; Belinkov and Bisk, 2018; Iyyer et al., 2018; Ribeiro et al., 2018; Wallace et al., 2019). Ensembling the predictions of the model on multiple semantically equivalent variants of the input is a promising approach to address this issue (Anantha et al., 2021; Vakulenko et al., 2021) as it can reduce the spread or dispersion of the predictions.
We leverage the above technique in reattempting the abstained questions i.e. we first generate multiple paraphrases of the input instance and then aggregate the model’s predictions on them. We use BART-large (Lewis et al., 2019) model fine-tuned on Quora Question Corpus (Iyer et al., 2017), PAWS (Zhang et al., 2019), and Microsoft Research Paraphrase Corpus (Dolan and Brockett, 2005) for paraphrasing and explore the following strategies for aggregating the model predictions:
• Mean: In this strategy, we calculate the average confidence assigned to each answer candidate across all predictions. Then, we select the candidate with the highest average confidence as the system’s prediction. Note that the system will output this prediction only if its confidence surpasses the abstention threshold.
• Max: Here, like the mean strategy, we select the answer candidate with the highest average confidence but we use the maximum confidence assigned to that candidate as its prediction confidence. This is done to push the most confident prediction above the abstention threshold.

heading: 2.3.2 Re-Examining Top N Predictions (REToP)
text: State-of-the-art models have achieved impressive performance on numerous NLP tasks. Even in cases where they fail to make a correct prediction, they are often able to rank the correct answer as one of their top N predictions. This provides opportunities for re-examining the top N predictions to identify the correct answer in case of abstention. To this end, a model that can estimate the correctness of a prediction can be leveraged. Following this intuition, we develop an auxiliary model that takes the context, question, and a prediction as input and assigns a score indicating the likelihood
of that prediction to be correct. This model can be used for each of the top N predictions given by the QA model to select the one that is most likely to be the correct answer.
Training Auxiliary Model: We first create data instances by annotating (context, question, prediction) triplets conditioned on the correctness of the QA system’s predictions and then train a classification model using this data. This model is specific to the given QA system and essentially learns to distinguish its correct and incorrect predictions.
• Annotate (context, question, prediction) triplets: We utilize the trained QA model to get its top N predictions for each training instance. Then, we annotate each (context, question, prediction) triplet based on the prediction’s correctness i.e. a correct prediction is annotated as ‘1’ and an incorrect prediction is annotated as ‘0’. Figure 3 illustrates this annotation step.
• Train a classification model: Then, a binary classification model is trained using the annotated dataset collected in the previous step. This model specifically learns to distinguish the correct predictions of the QA model from the incorrect ones. Softmax probability assigned to the label ‘1’ corresponds to the likelihood of correctness for each prediction.
Note that we use the QA model’s top N predictions to collect the ‘0’ annotations instead of randomly selecting candidates because this procedure results in highly informative negative instances (that are probable predictions and yet incorrect) and not easy/obvious negatives. This can help the auxiliary model in learning fine-grained representations distinguishing correct and incorrect predictions.
Leveraging Auxiliary Model: For an abstained instance, we compute the likelihood value for each
of the top N predictions given by the QA model using our trained auxiliary model. Then, we calculate the overall confidence (c) of each prediction (p) as a weighted average of the QA model’s probability (sq) and the auxiliary model’s likelihood score (sa) i.e. cp is calculated as:
cp = α ∗ spq + (1− α) ∗ spa
where α is a weight parameter. We incorporate QA model’s probability as it provides more flexibility to compute the overall confidence. Finally, prediction with the highest overall confidence is selected as the new prediction. We differentiate this method from existing methods such as calibration in Appendix C.

heading: 2.3.3 Human Intervention (HI)
text: In intolerant application domains such as biomedicals where incorrect predictions can have serious consequences, human intervention is the most reliable technique to answer the abstained instances. Human intervention can be in various forms such as providing relevant knowledge to the model, asking clarifying questions (Rao and Daumé III, 2018) or simplifying the input question. In this work, we explore a simple human intervention approach in which the system provides multiple predictions instead of only one prediction for the abstained instances. The human can then select the most suitable prediction from the provided predictions. Performance of this method can be approximated based on the presence of the correct answer in the predictions provided to the human. Note that the above approach would answer all the abstained instances and hence the coverage would always be 100%. This implies that with the increase in abstention threshold, the risk would monotonically decrease as multiple predictions would be returned for a larger number of instances.
In addition to the above approach, we also explore a REToP-centric HI approach in which the system returns multiple predictions only when REToP surpasses the confidence threshold in the postabstention stage. Similar to REToP, it abstains on the remaining instances. Finally, we note that comparing the performance of HI approaches with other post-abstention approaches would be unfair as other approaches return only a single prediction. Therefore, we present HI results separately.

heading: 3 Experiments and Results

heading: 3.1 Experimental Setup
text: Datasets: We experiment with SQuAD 1.1 (Rajpurkar et al., 2016) as the source dataset and the following 10 datasets as out-of-domain datasets: NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kim et al., 2019). We use the preprocessed data from the MRQA shared task (Fisch et al., 2019) for our experiments.
Implementation Details: We run all our experiments using the huggingface (Wolf et al., 2020) implementation of transformers on Nvidia V100 16GB GPUs with a batch size of 32 and learning rate ranging in {1−5}e−5. We generate 10 paraphrases of the question in Ensembling method, reexamine top 10 predictions, vary α in the range 0.3− 0.7 for REToP method, and vary the number of predictions in the range 2 to 5 for HI methods. Since the focus of this work is on post-abstention, it’s crucial to experiment with models that leave sufficient room for effectively evaluating the ability of post-abstention methods. For that reason, we experiment with a small size model (BERT-mini having just 11.3M parameters) from Turc et al. (2019) for our experiments. However, we note that our methods are general and applicable for all models.

heading: 3.2 Results

heading: 3.2.1 REToP
text: Table 1 shows the post-abstention performance of REToP for selected abstention thresholds. The last column (‘Total Risk Improvement’) in this table corresponds to the overall improvement aggregated over all confidence thresholds. It can be observed that REToP achieves considerable risk improvements both in the in-domain setting (21.81 on SQuAD) and the out-of-domain settings (24.23 on TextbookQA, 21.54 on HotpotQA, 20.42 on RE, etc). Next, we analyze these results in detail.
Higher improvement on moderate confidences: In Figure 4, we plot risk improvements achieved by REToP on SQuAD (in-domain) and HotpotQA (out-of-domain) datasets for all confidence thresholds. These plots reveal that the improvement is
more on moderate thresholds as compared to low thresholds. We attribute this to the high difficulty of instances that remain to be re-attempted at low thresholds i.e. only the instances on which the given system was highly underconfident are left for the post-abstention method. It has been shown that model’s confidence is negatively correlated with difficulty (Swayamdipta et al., 2020; Rodriguez et al., 2021; Varshney et al., 2022b) implying that the remaining instances are tough to be answered correctly. This justifies the lesser improvement in performance observed at low thresholds.
In-Domain vs Out-of-Domain Improvement: REToP achieves higher performance improvement on the in-domain dataset than the out-of-domain datasets (on average). This is expected as the auxil-
iary model in REToP is trained using the in-domain training data. However, it still has good performance on out-of-domain datasets as the auxiliary model learns fine-grained representations to distinguish between correct and incorrect predictions. Furthermore, the improvement on out-of-domain
data varies greatly across datasets (from 0.7 on TriviaQA to 24.23 on TextbookQA).

heading: 3.2.2 Comparing Post-Abstention Approaches
text: We provide the performance tables for other postabstention approaches in Appendix. However, we compare their total risk improvement values in Table 2. In the in-domain setting, REToP achieves higher improvement than Ensembling method. This is because the auxiliary model in REToP has specifically learned to distinguish the correct and incorrect predictions from the training data of this domain. However, in some out-of-domain cases, Ensembling outperforms REToP (SearchQA, TriviaQA, NewsQA). Overall, REToP leads to a consistent and higher risk improvement on average. Ensembling also leads to a minor degradation in a few out-of-domain datasets (DuoRC and TextbookQA). Next, we analyze the performance of human intervention (HI) methods.

heading: 3.2.3 Human Intervention (HI)
text: We study two variants of HI method. In the first variant, multiple predictions (n=2) are returned for all the abstained instances. This makes the coverage to be 100% for all the confidences; therefore, we present only the risk values in Table 3. As expected, with increase in abstention threshold, the risk decreases because multiple predictions get outputted for a larger number of instances. Selection of operating threshold for an application depends on the trade-off between risk that can be tolerated and human effort required to select the most suitable prediction from a set of predictions returned by the system. For example, a low threshold can
be selected for tolerant applications like movie recommendations and a high threshold for tolerant applications like house robots.
In the second variant of HI method, we study a REToP-centric approach in which the system returns multiple predictions only when REToP surpasses the confidence threshold in the postabstention stage. The last column in Table 2 shows the risk improvements achieved by this approach (n=2). Note that REToP re-examines the top N predictions and selects one while this method outputs multiple predictions and requires a human to select the most suitable one. These results indicate that though REToP achieves good performance, there is still some room for improvement.

heading: 3.2.4 Ensembling Using Paraphrases
text: Comparing the performance of Mean and Max Ensembling strategies reveals that Max increases the coverage more than the Mean strategy but it also increases the risk considerably. Thus, pushing the instance’s confidence to surpass the abstention threshold fails to provide risk improvements. However, such a technique could be employed in scenarios where risk degradation can be tolerated.

heading: 4 Analysis
text: What is the distribution of model’s original confidence on the instances that get answered after applying post-abstention method? In Figure 5, we show the distribution of model’s original confidence on SQuAD instances that get answered by REToP at abstention threshold 0.5. Green-colored bars represent the number of instances answered from each confidence bucket. We found that REToP answers a large number of instances from the high
confidence buckets; however, instances from even low confidence buckets get answered. This can further be controlled using the weight parameter (α) in the overall confidence computation.
How often do the system’s predictions change after applying REToP and what is its impact? REToP can either boost the confidence of the top most prediction of the given model or can select a different answer by re-examining its top N predictions. In Figure 6, we specifically analyze the latter scenario i.e. the instances on which REToP’s prediction differs from the original model’s prediction. At a threshold of 0.5, the original system abstains on 3411 SQuAD instances and after applying REToP, it answers 1110 of those instances. Out of these 1110 instances, the REToP changes the prediction on 186 instances. The original prediction is incorrect in more cases (99 vs 87) and after applying REToP, the system gives 116 correct predictions and only 70 incorrect. This implies that by overriding the original system’s prediction, REToP improves the system’s accuracy. However, in some cases, it also changed a correct prediction to incorrect but such cases are lesser than the former.

heading: To what extent do the instances answered by
text: different post-abstention methods overlap? In Figure 7, we demonstrate the Venn diagram of SQuAD instances answered by REToP and Ensembling (Mean) approaches at abstention threshold 0.5. REToP answers 1110 instances while Ensembling answers 277 and there 127 common instances between the two approaches. This indicates that the two sets are not mutually exclusive i.e. there are some instances that get targeted by both the ap-
proaches; however, there are a significant number of instances that are not in the intersection. This result motivates studying composite or sequential application of different post-abstention methods to further improve the post-abstention performance.

heading: 5 Conclusion and Discussion
text: In this work, we formulated ‘Post-Abstention’, a task that allows re-attempting the abstained instances of the given selective prediction system with the aim of increasing its coverage without significantly sacrificing the accuracy. We also explored several baseline methods for this task. Through comprehensive experiments on 11 QA datasets, we showed that these methods lead to considerable performance improvements in both in-domain and out-of-domain settings. We further performed a thorough analysis that resulted in several interesting findings.
Looking forward, we believe that our work opens up several avenues for new research, such as exploring test-time adaptation, knowledge hunting, and other human intervention techniques like asking clarification questions as post-abstention methods (discussed in Appendix D). Studying the impact of composite or sequential application of multiple post-abstention methods in another promising direction. Furthermore, prior selective prediction
methods can also be repurposed and explored for this task. We plan to pursue these crucial research directions in our future work. Finally, we hope our work will encourage further research in this important area and facilitate the development of more reliable NLP systems.
Limitations
The proposed post-abstention methods require additional computation and storage. Despite this additional requirement, we note that this is not a serious concern as current devices have high storage capacity and computation hardware. Furthermore, additional computation for training auxiliary model in REToP is required only once and just an inference is required at evaluation time which has a much lower computation cost. Moreover, the risk mitigation that comes with the post-abstention methods weighs much more than the computational or storage overhead in terms of importance. Secondly, human-intervention techniques require a human to be a participant and contribute in the answering process. However, these approaches do not expect the participating human to be an expert in the task. Like other empirical research, it is difficult to exactly predict the magnitude of improvement a post-abstention method can bring. Our idea of exploring sequential application of multiple postabstention methods addresses this concern and can be used based on the application requirements.

heading: Acknowledgement
text: We thank the anonymous reviewers for their insightful feedback. This research was supported by DARPA SAIL-ON program.

heading: A Ensembling (Mean) Performance
text: Table 5 shows the performance of using Ensembling (Mean) as a post-abstention method for a few selected abstention threshold values. For each dataset, we provide three rows: the first row (‘Given’) shows the coverage and risk values of the given selective prediction system at specified abstention thresholds, the second row (‘Ens’) shows the coverage and risk after applying the postabstention method on the abstained instances of the given selective prediction system, and the final row (‘G@Enscov’) shows the risk of the given selective system at the coverage achieved by Ens method. For the post-abstention method to be effective the risk in the second row should be less than that in the third row and the magnitude of difference corresponds to the improvement. The last column ‘Total Risk Improvement’ shows the overall improvement aggregated over all confidence thresholds ranging between 0 and 1 at an interval of 0.02.

heading: B Dataset Statistics
text: Table 4 shows the statistics of all evaluation datasets used in this work. SQuAD corresponds to the in-domain dataset while the remaining 10 datasets are out-of-domain. We use the preprocessed data from the MRQA shared task (Fisch et al., 2019).

heading: C Differentiating REToP from Calibration
text: REToP is different from calibration based techniques presented in (Kamath et al., 2020; Varshney et al., 2022c) in the following aspects: (a) Firstly, REToP does not require a held-out dataset unlike calibration based methods that infer the model on the held-out dataset to gather instances on which the model in incorrect. (b) Secondly, the auxiliary model trained in REToP predicts the likelihood of correctness of (context, question, prediction) triplet i.e. it is used for each of the top N prediction individually. This is in contrast to calibrators that predicts a single score for an instance and ignores the top N predictions. (c) Finally, we use the entire context, question, and the prediction to predict its correctness likelihood score unlike feature-based calibrator models in which a random-forest model is trained using just syntax-level features such as length of question,
semantic similarity of prediction with the question, etc.

heading: D Other Post-Abstention Techniques
text: Asking clarifying questions to the user in order to get information about the question has started to received considerable research attention in conversational, web search, and information retrieval settings (Aliannejadi et al., 2021, 2020; Zamani et al., 2020a; Zhang et al., 2020; Zamani et al., 2020b). These techniques can be leveraged/adapted for the post-abstention task.
Test-time adaptation is another promising research area in which the model is adapted at testtime depending on the instance. This is being studied in both computer vision (Chen et al., 2022) and language processing (Wang et al., 2021; Banerjee et al., 2021).
Cascading systems in which stronger and stronger models are conditionally used for inference is also an interesting avenue to explore with respect to Post-Abstention (Varshney and Baral, 2022; Li et al., 2021; Varshney et al., 2022a).

heading: E Coverage 100% for Human Intervention Methods
text: We believe that the ability to identify situations when there is no good answer in the top N returned candidates is a very difficult task (for the humans also) and it requires even more cognitive skills than just selecting the best answer from the provided answer candidates. Because of this reason, the coverage is 100%.

heading: F Comparison with Other Selective Prediction Methods
text: In this work, we presented a new QA setting and studied the performance of several baseline methods for this task. The focus of this work is on studying the risk improvement that can be achieved in this problem setup. We consciously do not pitch
the approaches for this task as competitors of the existing selective prediction approaches. In fact, these approaches are complimentary to the selective prediction approaches. A post-abstention method can be used with any selective prediction method as the first step.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We have Limitations Section at the end of the paper after Conclusion
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We use the publicly available standard NLP datasets in this work with appropriate citations and references.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We do not create any artifcats in this reserach. We use the publicly available standard NLP datasets in this work with proper citations and references.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We do not create any artifcats in this reserach. We use the publicly available standard NLP datasets in this work with proper citations and references.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We do not collect any data for this research and use standard publicly available NLP datasets
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We do not collect any data for this research
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Sections 3 and 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Sections 3 and 4

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Sections 3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 3 and 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Sections 3 and 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We plan to pursue these crucial research directions in our future work. Finally, we hope our work will encourage further research in this important area and facilitate the development of more reliable NLP systems. Limitations
The proposed post-abstention methods require additional computation and storage. Despite this additional requirement, we note that this is not a serious concern as current devices have high storage capacity and computation hardware. Furthermore, additional computation for training auxiliary model in REToP is required only once and just an inference is required at evaluation time which has a much lower computation cost. Moreover, the risk mitigation that comes with the post-abstention methods weighs much more than the computational or storage overhead in terms of importance. Secondly, human-intervention techniques require a human to be a participant and contribute in the answering process. However, these approaches do not expect the participating human to be an expert in the task. Like other empirical research, it is difficult to exactly predict the magnitude of improvement a post-abstention method can bring. Our idea of exploring sequential application of multiple postabstention methods addresses this concern and can be used based on the application requirements