QADYNAMICS: Training Dynamics-Driven Synthetic QA Diagnostic for Zero-Shot Commonsense Question Answering
Authors: Haochen Shi
Weiqi Wang
Tianqing Fang
Baixuan Xu
Wenxuan Ding
Xin Liu
Yangqiu Song
Venue: EMNLP 2023 Findings (2023)
Keywords: commonsense reasoning
question-answering
training dynamics
zero shot
TL;DR: We propose QADYNAMICS, a training dynamics-driven approach for the task of zero-shot commonsense question-answering. Experiments show it outperforms all previous baseline methods with only 33% of training data and surpasses GPT3.5 and ChatGPT.

Abstract: Zero-shot commonsense Question-Answering (QA) requires models to reason about general situations beyond specific benchmarks. State-of-the-art approaches fine-tune language models on QA pairs constructed from CommonSense Knowledge Bases (CSKBs) to equip the models with more commonsense knowledge in a QA context. However, current QA synthesis protocols may introduce noise from the CSKBs and generate ungrammatical questions and false negative options, which impede the model’s ability to generalize. To address these issues, we propose QADYNAMICS, a training dynamics-driven framework for QA diagnostics and refinement. Our approach analyzes the training dynamics of each QA pair at both the question level and option level, discarding machine-detectable artifacts by removing uninformative QA pairs and mislabeled or false-negative options. Extensive experiments demonstrate the effectiveness of our approach, which outperforms all baselines while using only 33% of the synthetic data, even including LLMs such as ChatGPT. Moreover, expert evaluations confirm that our framework significantly improves the quality of QA synthesis. Our code and model checkpoints are available at https://github.com/HKUST-KnowComp/QaDynamics.

QADYNAMICS:
Training Dynamics-Driven Synthetic QA Diagnostic for Zero-Shot Commonsense Question Answering
Haochen Shi,, Weiqi Wang", Tianqing Fang, Baixuan Xu, Wenxuan Ding, Xin Liu, Yangqiu Song
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China hshiah@connect.uss wwtpoonnncttutt...... ust.hk, (wwangbw, tfangaa, vasongJecse Jesoonnecce..st.hhee ust.
Abstract
knowledge (Branco et al., 2021; Zhou et al., 2021). To reliably assess the resilience of QA models across diverse domains, the zero-shot common- sense QA task has been proposed to evaluate the generalizable reasoning ability of QA model (Li et al., 2020; Shwartz et al., 2020) without supervi- sion signals from any QA benchmarks.
Zero-shot commonsense Question-Answering (QA) requires models to reason about general situations beyond specific benchmarks. State- of-the-art art approaches fine-tune language mod- els on QA pairs constructed from Common- Sense Knowledge Bases (CSKBs) to equip the models with more commonsense knowl- edge in a QA context. However, current QA synthesis protocols may introduce noise from the CSKBs and generate ungrammatical ques- tions and false negative options. which im- pede the model's ability to generalize. ad- dress these issues, propose QADYNAM- ICS, a training dynamics-driven framework for QA diagnosties and refinement. Our ap- proach analyzes the training dynamics of each QA pair at both the question level and option level, discarding meeteeeedetectable artifacts by removing uninformative QA pairs and mis- labeled or false-negative options. Extensive experiments demonstrate the effectiveness of our approach, which outperforms all baselines while using only 33% of the synthetic data, even including L.LMs such ChatGPT. More- over, expert evaluations confirm that our frame- work significantly improves the quality of QA synthesis. Our code and model checkpoints are available at https//github..comHHkK KnowComp//Qayynaiic.
Ma et al. (2021) introduced a technique for tack- ling this task by fine-tuning a PTLM on QA pairs synthesized from knowledge triples in Common- Sense Knowledge Bases (CSKBs). The head and relation of triple were transformed into ques- tion using natural language templates, with the tail serving as the answer. Distractors, or neg- ative examples, were tails from triples sampled from the same CSKB using pre-defined strategies, such as keyword or embedding proximity filter- ing. However, the primary obstacle hindering fur- ther progress in this method is the quality of the synthetic QA dataset. This issue arises because manually-curated CSKBs often contain subtle but strong annotation artifacts (Zellers et al., 2019; Sak- aguchi et al., 2021), which could provide easy back- doors for the model to perform exceptionally well on synthetic test sets but fail to generalize on held- out QA benchmarks. Additionally, the current QA synthesis process results in significant number of ungrammatical questions, and the negative sam- pling strategy used to create distractors is not en- tirely effective in preventing false-negative options. as evidenced by Ma et (2021).
1 Introduction
The advent of various commonsense Question- Answering (QA) benchmarks (Talmor et al., 2021; Huang et al., 2019) has demonstrated that Pre- Trained Language Models (PTLMs) (Devlin et al., 2019; Lan et al., 2020) can achieve extraordinary performances when fine-tuned on these bench- marks. However, these neural systems have been criticized for only learning surface-level correla- tions and lacking general semantic reasoning abil- ities, which often require implicit commonsense
Despite the existence of dataset filtering algo- rithms, such adversarial filtering (Zellers et al., 2018) for negative option selection, they have been shown be less effective compared random se- lection baselines (Ma et al., 2021). This because they only focus on model uncertainty in the final predictions, which is not effective enough for syn- thetic data that contains plethora of noise and imbalanced examples (Appx. $A.1)).
Equal Contribution
Instead of leveraging data filtering based on

---

model uncertainty in the final predictions, draw inspiration from Swayamdipta al. (2020) and em- ploy iraining dynamics as more precise indicator that studies instance learnability across all training steps. While the vanilla training dynamics regard each data instance as whole without considering the learnability of each option choice, we pro- pose QADYNAMICS, training dynamics-driven framework for synthetic QA diagnostic and refine- ment that favors choice-level diagnosis. Specifi- cally, our approach proposes novel schema that offers greater flexibility in deriving the training dy- namics of multiple-choice QA with an arbitrary number of options, thus accommodating the vary- ing number of choices in different commonsense QA benchmarks. QADYNAMICS then analyzes the training dynamics of each option, greedily drops the easy distractor to reduce the impact of CSKB artifacts, and climinates QA pairs containing mis- labeled or false-negative options according to the confidence gap between all options ($3). Exten- sive experiments showcase the efficacy and data efficiency of our proposed framework, surpassing all previous zero-shot CSQA baselines while only leveraging 33% of training data and even outper- forming GPT3.5 (Ouyang al., 2022) and Chat- GPT ($4.4). Further expert evaluations confirm the effectiveness of our proposed method in enhancing the quality the synthetic QA set ($4.5).
of training data, which is subject to limitations in both the knowledge quality and coverage in the CSKBs and the protocol for synthesizing them into QA pairs. Both of these are restricted by specific limitations, as discussed in $1.
2.2 Dataset Diagnostic
Diagnosing individual data instances within a large dataset has long been an important aspect of ma- chine learning for NLP (Deng al., 2023). Vari- ous data attribution methods have been proposed to retrieve training instances that may have led particular prediction (Pezeshkpour et al., 2021; Xie al., 2023). Building on this, Pezeshkpour et al. (2022) proposed a method to efficiently detect dataset artifacts in the training data using data attri- bution methods when a challenging validation set is available. While these methods focus the im- pact of individual instances on specific predictions, more generalized and precise dataset diagnostic ap- proaches have also been proposed (Swayamdipta et al., 2020; Ethayarajh al., 2022). These ap- proaches aim understand the difficulty of learn- ing specific instances and can detect annotation artifacts and perform automatic data corrections, such as mislabeling detection. However, none of these methods explicitly consider QA benchmarks where cach QA pair contains more than piece of knowledge. To effectively evaluate the attribu- tion of QA pair, is necessary consider all possible options for fair consideration.
2 Related Works
2.1 Zero-shot Commonsense QA
3 QADYNAMICS
The task of zero-shot commonsense QA requires a QA model to perform generalizable QA towards commonsense questions from held-out benchmarks whose training data is inaccessible to the model. Existing approaches either leverage off-the-shelf language models in an unsupervised manner lock their commonsense capability with inference time mechanisms, such as self-talk (Shwartz et al., 2020), cloze translation (Dou and Peng, 2022), and dynamic graph reasoning (Bosselut et al., 2021) ), or inject commonsense knowledge into PLMs by fine-tuning them on synthetic QA pairs constructed from CSKBs (Ma et al., 2021; Kim et al., 2022; Wang et al., 2023a; Zhang al., 2022). While unsupervised approaches only achieve satisfac- tory performances, existing works following the fine-tuning regime have shown exceptional perfor- mances on various commonsense QA benchmarks. However, fine-tuning heavily relies on the quality
This section outlines our proposed framework, QA- DYNAMICS, which consists of four steps: (1) Cal- culate the training dynamics for each option in a QA pair. (2) Refine the QA pair by eliminating the easy distractor. (3) Filter out QA pairs that may be mislabeled or contain false-negative distractors. (4) Train the model using marginal ranking loss.
3.1 Preliminary
We follow the pipeline and task definition for- mulated by Ma et al. (2021) to study the zero- shot commonsense QA task. Formally, denote a CSKB as D {(h, t) H. E Rt, E T}. where H, T the sets of heads, relations, and tails. Every triple in D is transformed into (Q, A) pair, where Q a question constructed using (h, r) with natural language templates and A = (A1, Ag,-.., And is the corresponding set

---

of options containing m choices. Specifically, t is used as the ground-truth answer A,, and other distractors are tails from m - triples sampled us- ing keyword overlap filtering. The objective is to obtain a QA model 0 from the synthetic QA sets D = {[Qi, A,) rr, (4) E D} and test A on held-out QA benchmarks.
3.3 Option Selection
To reduce any artifacts present in the synthetic QA set that may have originated from the CSKBs, we adopt a similar approach to AFLite (Bras et al., 2020) and remove negative knowledge that the model can easily identify. We achieve this by dis- carding one distractor with the highest confidence score, which indicates that the model may be sus- ceptible to exploiting potential biases and consis- tently assigns high score to this option. We then concatenate the modified option set A', containing the original ground-truth answer and m 2 distrac- tors that are more challenging to distinguish, with the original question Q to yield more challeng- ing (Q, A') pair. Such option level selection strategy is termed Difficult Choice.
3.2 Training Dynamics of QA Pairs
Following Maet al. (2021), the QA model is trained through fine-tuning a pre-trained masked language model. For given (Q, A) pair, Q is concatenated with every option A, E A first to obtain the in- put sequence T; We then repeatedly mask out token in T at one time and calculate the model's masked loss. The logit score of T3 with n tokens is calculated by:
3,4 QA Pair Selection
S(T.)= Σ log P(4ll
b.)
(I)
Next, to improve the quality of the synthetic QA set, we remove poor-quality QA pairs that contain the following two types of options:
Intuitively, the option with the lowest logit score will be selected the answer. Based on this, we introduce proposed schema for calculating the training dynamics of (Q, A) at both the pair level and option level. Following Swayamdipta et al. (2020), we train a QA model 6' on Dº and save checkpoints {6), B,.,...,, "..,,'e) along the training process. At checkpoint d.. denote T, as the input sequence with the second lowest logit of distractors among those containing distractor, the model's s confidence of Ti (concatenation of Q and A,) be- ing correct is:
Mislabeled Ground-Truth Option. We remove
the QA pairs whose correct answer is associated with very low confidence, indicating potentially being mislabaled (Swayamdipta et al., 2020).
False Negative Distractor. We remove QA pairs where the difference in confidence score between the ground-truth answer and the distractor with the highest confidence score insignificant. This indicates the potential for a false negative.
3.5 Model Training
Finally, we fine-tune 0 on our cleaned synthetic QA set using marginal ranking loss. With the score of each option defined in Equation (1), the marginal ranking loss, with 17 being the margin, is:
exp(-S(TT))
TT) exp(-S(T:)) exp( S(T))
(2)
Similarly, the confidence of a distractor's ' input sequence Tr being wrong is defined as:
(.).T)) =
exp(-S(7i)) Eee exp(-S(Ti))
(3)
C=
m
mar(0. - S(T)) (5)
Based on the confidences of all options, we for- mulate the confidence of a (Q, A) pair as:
4 Experiments
4.1 Datasets
P(d Q.4) 778 E (P(O', T,) + P(d. (4)
Following Ma et al. (2021), we leverage the combi- nation of five CSKBs, including ATOMIC (Sap et al., 2019a), ConceptNet (Speer et al., 2017), WordNet (Miller, 1995), VisualGenome (Krishna et al., 2017), and WikiData (Vrandecic and Krötzsch, 2014), our source commonsense knowledge repository D. We then use the val- idation split of five commonsense QA bench- marks, including AductiveNLI (aNLI; Nie et al.,
Finally, following Swayamdipta et al. (2020), we derive scores for each option and QA pair at each of the E checkpoints using the equations defined above. The final confidence and variability scores are obtained by calculating the average and stan- dard deviation of these scores across E checkpoints (more detailed explanations in Appx. $A.1).

---

Data Selection Strategy
aNLI
CSQA
PIQA
SIQA
WG
Ayg.
DeBERTa-v3-Larg' (Zero-shot; He et al., 2023)
59.9
25.4
44.8
47.8
50.3
45.6
Random 33%
77.3
68.5
79.5
75.5
72.7
Random 50%
75.8
68.8
64.2
Random 66%
77.3
66.8
78.5
64.0
73.4
72.4
Total 100% (Ma et 2021)
78.7
68.5
79.1
63.5
75.4
73.0
Total 100% (AF-Lite) (Ma et al., 2021)
79.5
65.7
75.6
55.7
75.1
70.3
Large Language Models
GPT-3.5 (text-davinci-003)
61.8
68.9
67.8
68.0
60.7
65.4
ChatGPT (gpt- 3.rbbb
69.3
74.5
75.1
69.5
62.8
70.2
TRAINING.DNAMMIC Dar
Easy-to-leam
75.9
67.9
75.6
62.9
73.2
71.1
66% train
Ambiguous
80.0
69.1
80.3
63.2
78.4
74.2
Hard-to-lcam
79.7
69.0
78.9
63.8
77.7
73.8
Hard-to-leamn Mislabeled.
80.3
70.3
79.4
63.9
77.4
74.3
Hard-to-learn False-Neg.
69.5
79.7
64.2
77.4
Hard-to-lcam Mixed Strategy
80.7
70.1
79.6
63.8
77.3
74.4
Easy-to-lcar
75.1
67.9
78.0
65.0
74.8
72.2
33% truin
Ambiguous
80.9
70.1
78.4
79.6
Hard-to-leam
80.9
67.5
62.6
76.9
Hard-to-learn Mislabeled.
79.5
70.1
78.2
62.0
79.2
73.8
Hard-to-leam False-Neg.
80.7
70.4
78.5
66.5
77.3
74.7
Hard-to-lcam Mixed Strategy
82.3
70.9
78.6
64.5
78.0
74.9
QADYNAMICS (Ours) DeBERTa-v3-Lrgge 435M
Easy Choice
75.2
69.5
77.2
62.7
71.7
71.3
66% train
Difficult Choice
80.4
68.8
79.0
64.0
76.9
73.8
Difficult Choice Mislabeled.
80.0
70.1
79.4
63.8
79.4
74.2
Difficult Choice False-Neg.
79.0
70.8
79.8
78.9
74.4
Difficult Choice Mixed Strategy
80.0
70.7
80.4
78.6
75.0
33% truin
Difficult Choice Hard-to lond-toolleam without strategy
79.7
71.5
79.9
65.4
76.1
74.5
Difficult Choice ha-ad-to-learnaa Mislabeled.
82.3
72.2
79.8
63.3
79.0
75.3
Difficult Choice Hard- tonddtto-leam False-Neg.
81.9
70.4
79.7
79.0
Difficult Choice Hard-to-leam Mixed Strategy
82.3
71.6
81.2
65.6
79.1
76.0
Supervised Learning & Human Performance
DeBERTa-V3 (Supervise)
89.0
82.1
84.5
80.1
84.1
84.0
Human Performance
91.4
88.9
94.9
86.9
94.1
91.2
Table Zero-shot commonsense QA evaluation results on five benchmarks (Accuracy %).. All experiments employ DeBERTa-Vss SEEERTa-v3-Largee (He al., 2023) the backbone. The best performances bold-faced, and the second-bes ones are underlined. "Mislabeled." refers to removing QA pairs whose ground-truth answer mislabeled, and "False-Neg." refers to removing QA pairs containing false-negative distractors 43..4). "Mixed Strategy" indicates iteratively applying both measures above to eliminate poor-quality QA pairs
2020), CommonsenseQA (CSQA; Talmor et al., 2019), PhysicalIQA (PIQA: Bisk et al., 2020), SocialIQA (SIQA; Sap et al., 2019b), and Wino- Grande (WG: Sakaguchi al., 2021), for evalua- tion. More statistics are shown in Tab. 4.
Mislabeled False-Neg. Mixed Strategy
Total
Data size Ratio
6465
26320 3.80%
32875 4.74%
345775 100%
0.94%
Table 2: Statistics of the number of QA pairs that are dropped by each strategy.
4.2 Dataset Statistics
80.7, respectively. The data discovered automati- cally by our strategy is 4.74% of total data, which close to 25% of the poor-quality or grammati- cally wrong data. Most of them are located the low-confidence areas, indicating our framework' 's contribution towards purifying the low-quality data.
In method, set threshold to filter out misla- beled and false negative data from the entire dataset, Intuitively, is essential to establish the accuracy and reliability of the data before proceeding with any further division or analysis. The threshold is decided based on rough observations of QAdy- namic distributions, emphasizing the balance be- tween quantity and quality.
4.3 Experiment Setup and Baselines
We use accuracy as the evaluation metric. To derive the QADYNAMICS of the synthetic QA entries, we use RoBERTa-large (Liu et al., 2019) as the back- bone g., and for our final QA model 6, we use
The specific statistics are shown in Tab. 2. As mentioned by Ma et al. (2021), the human accuracy on ATOMIC and CWWV synthetic data 78.0 and

---

DeBERTa-v3-large (He al., 2023). Our choice to utilize different models because RoBERTa-large results faster training and inference speed, and in- tuitively, it is challenging to expect model to learn from data that is itself difficult to learn. We com- pare our results with several baselines to demon- strate the effectiveness of our training dynamics- driven data selection. First, we include those us- ing 33%, 66%, and 100% synthetic QA pairs that are generated using keyword filtering or AFLite for distractor selection. We also report the perfor- mance of Large Language Models (LLMs), includ- ing GPT3.5 (Brown et al., 2020; Ouyang et al., 2022) and ChatGPT (OpenAI, 2022), as compet- itive baselines. To provide fair comparison, we compare our framework with the original training dynamics-based data selection (Swayamdipta et al., 2020) with equal amounts of training data (33% and 66%). We select QA pairs that are easy-to- learn, ambiguous, and hard-to-learn, according to their confidence and variability distribution, and perform mislabeled correction on the hard-to-learn data, as done by Swayamdipta et al. (2020). For our framework, we utilize our proposed Difficult Choice selection ($3.3) with a combination of QA pair selection strategies ($3.4). Furthermore, operate our framework on 50% of total QA pairs that have low confidence to show the effectiveness of our framework on hard-to-learn data. More explanations are provided in Appx. S...I
Data Selection Strategy
Plau.
Mis.
E.ge
Total Data Hard-to-learn
80.0
18.0
30.0
67.0
26.0
32.0
After Removing Mislabeled.
81.0
15.0
35.0
Dropped by Mislabeled.
18.0
70.0
35.0
After Removing Kalse-Neg.
68.0
25.0
22.0
Dropped by False-Neg. After Applying Mixed Strategy
55.0
36.0
52.0
76.0
17.0
25.0
Dropped by Mixed Strategy
54.0
43.0
45.0
Table 3:
Expert evaluation results (%) on QA pairs se- lected using different combinations of strategies, which correspond to those defined in Tab. 1. Plau.. Mis., and F.Neg refer to the ratio QA pairs being plausible, containing mislabeled QA options, and containing false- negative distractors.
reliability of combining all proposed techniques in QADYNAMICS. Ablation studies are provided in Appx. $A.3.
4.5 The Effect of Option Selection
To verify the effectiveness of our option selections ($3.3), we recruit five graduate students specializ- ing in machine commonsense to evaluate the qual- ity of 100 randomly sampled synthetic QA pairs se- lected by various strategies. The experts are asked to annotate whether a QA pair is plausible (ques- tion and answer forms a plausible commonsense knowledge), mislabeled (the ground-truth answer is incorrect), or contains any felse-negative distrac- tor (the distractor is semantically correct). Our results, presented in Tab. 3, are consistent with the targeting effect of both strategies, which suc- cessfully reduces the ratio of mislabeled examples and false-negative examples. We also observe that jointly adopting both strategies benefits all three metrics, which positively supports the success of our best system in $4.4. Case studies are provided in Appx. $B.
4.4 Results
The main results are shown in Tab. 1. Consistent with Swayamdipta et al. (2020), we observe that training the model with ambignous and hard-to- learn data leads to the largest benefit in the base- lines, outperforming both random data selection and LLMs. Mislabeled correction hand-to-learn data also has a positive impact, indicating that the synthetic QA entries indeed contain such er- rors. Our best system, trained on hard-to-learn QA entries and applying all option and QA selec- tion strategies, achieves state-of-thh-arr results by significantly outperforming all baselines on most benchmarks. It outperforms the best baseline (Am- biguous) by 1.7% in terms of average accuracy and surpasses LLMs by 5.8%. This demonstrates that dropping easy distractors to make the training set more difficult contributes to a more generalizable QA model, and leveraging QA selection strategies ($3.4) also has a positive impact, demonstrating the
5 Conclusions
In this paper, we propose QADYNAMICS, training dynamics-empowwred framework for data-efficient zero-shot commonsense QA that jointly consid- ers the learning difficulty at both the QA and op- tion levels. Our framework, on average, achieves state-of-the-art performance by surpassing large language models and all baselines significantly with only 33% of training data. Further expert evaluations showcase that our proposed method ef- fectively climinates poor-quality QA entries in the synthetic dataset.

---

Limitations
York, NY, USA, February 7-12, 2020, pages 7432- 7439. AAAI Press.
The major limitation of QADYNAMICS is that improved schema for assessing the training dynam- ies of QA pair requires least three options. This is because we consider all distractors when eval- uating the confidence of the ground-truth answer and the entire QA pair, requiring more than one dis- tractor to ensure precision. While most synthetic QA sets satisfy this requirement, there are also QA benchmarks that only have two options per ques- tion, such as WinoGrande (Sakaguchi et al., 2021) and aNLI (Nie et al., 2020). In such cases, the orig- inal training dynamics proposed by Swayamdipta et al. (2020) can be properly leveraged to deal with binary questions. We believe that this limitation is minor compared with the data-cleaning effect of QADYNAMICS.
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2021. Dynamic ceuro-symbolic knowledge graph construc- tion for zero-shot commonsense question answering. In Thirty-Fifth AAAI Conference on Artificial Intel- ligence, AAAI 2021, Thirty-Third Conference In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 4923-4931 AAAI Press.
Ruben Branco, António Branco, João António Ro- drigues, and João Ricardo Silva. 2021. Shortcutted commonsense: Data spuriousness in deep learning of commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 1504 1521. Association for Computa- tional Linguistics.
Ethics Statement
Ronan Le Bras, Swabha Swayamdippt,, Chandra Bha- gavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal. and Yejin Choi. 2020. Adversarial filters of dataset biases. In Proceedings of the 37th Inter- national Conference on Machine Learning, ICML, 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1078-1088 PMLR.
This paper uses datasets and benchmarks solely for research purposes, consistent with their intended usage. The expert student annotators recruited for this study were well-trained and agreed to partic- ipate voluntarily without receiving any payment. Since QADYNAMICS is QA model and not a gen- erative model, does not yield additional biased content. Therefore, to the best of our knowledge, this paper does not involve any ethical concerns.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanic Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, viriual.
Acknowledgements
The authors would like to thank the anonymous reviewers for their constructive comments. The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20), and the GRF (16211520 and 16205322) from RGC of Hong Kong. We also thank the support from the UGC Research Matching Grants (RMGS20EGO1-D. RMGS20CRI1, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).
Chunkit Chan. Jiayang Cheng, Weiqi Wang. Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song- 2023. Chatgpt evaluation on sentence level relations: A focus on temporal. causal, and discourse relations. CoRR, abs/2304. his/2304414827.
References
Zheye Deng, Weiqi Wang, Zhaowei Wang, Xin Liu, and Yangqiu Song. 2023. Gold: A global and local-aware denoising framework for commonsense knowledge graph noise detection. CoRR, a0s/2310.1200
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

Computational Linguisties: Human Language Tech- nologies, NAACL-HLI 2019. Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-wor won Hwang. and Jiny- oung Yeo. 2022. Modularized transfer learning with multiple knowledge graphs for zero-shot common- sense reasoning. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States. July 10-15, 2022, pages 2244 -257.227. Association for Computational Linguisties.
Zi-Yi Dou and Nanyun Peng. 2022. Zero-shot com-
monsense question answering with cloze translation and consistency optimization. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022. Thirty-Fouthh Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial In- telligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pages 10572-10580. AAAI Press.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Vi- sual genome: Connecting language and vision ing crowdsourced dense image annotations. Int. J. Comput. Vis., (23(1):32-73.
Kawin Ethayarajh. Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In Interna- tional Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5988-6008. PMLR.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Tianqing Fang, Quyet V. Do, Schyun Choi, Weiqi Wang. and Yangqiu Song. 2023. CKBP v2: An expert- annotated evaluation set for commonsense knowl- edge base population. CoRR, abs/2304.
Zhongli Li, Wenhui Wang, Li Dong, Furu Wei, and Ke Xu. 2020. Harvesting and refining question- answer pairs for unsupervised QA. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6719-6728, Association for Com- putational Linguistics.
Tianging Fang, Weiqi Wang. Schyun Choi. Shibo Hao, Hongming Zhang, Yangqiu Song, and Bin He. 2021a. Benchmarking commonsense knowledge base pop- ulation with an effective evaluation dataset. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7- 11 November, 2021, pages 8949-8964. Association for Computational I Linguistics.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, 1ttx/197.11166
Tianqing Fang, Hongming Zhang, Weiqi Wang. Yangqiu Song, and Bin He. 2021b. DISCOS: bridg- ing the gap between discourse knowledge and com- monsense knowledge. In WWW '21: The Web Con- ference 2021. Virtual Event/ / Ljubljana, Slovenia, April 19-23, 2021. pages 2648 24599. ACM IW3C2.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Mutian He, Tianqing Fang. Weiqi Wang, and Yangqiu Song. 2022. Acquiring and modelling abstract com- monsense knowledge via conceptualization. CoRR, abs/2206.01532.
Kaixin Ma. Filip Ilievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg, and Alessandro Oltramari. 2021. Knowledge-driven data construction for zero-shot evaluation commonsense question answering. Thirty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2021, Thirty-Third Conference on In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Vir- tual Event, February 2-9, 2021, pages 13507-13515. AAAI Press.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTa using ELECTRA- style pre- pre-training with maddient-disntaaggee embed- ding sharing. In The Eleventh International Confer- ence on Learning Representations.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: machine reading comprehension with contextual commonsense rea- soning. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing, EMNLP-IJCNLPP 2019, Hong Kong, China, November 3-7, 2019, pages 2391- 2401. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(1 1):39-44.1
Yixin Nie Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela, 2020. Adversarial

---

NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4885-4901. Association for Computational Linguistics.
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 4615- 4629, Association for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017. San Francisco, California, USA, pages 4444 4451. AAAI Press.
OpenAl. 2022. Chatgpt: Optimizing language models for dialogue. OpenAI.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong 7hang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models follow instruc- tions with human feedback. In NeurlPS.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi. Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9275- 9293. Association for Computational Linguistics.
Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, and Byron C. Wallace. 2022. Combining feature and instance attribution to detect artifacts. In Findings of the Association for Computational Linguisties: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1934- 1946. Association for Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowl- edge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers). pages 4 4149-4158. Association for Computational Linguistics.
Pouya Pezeshkpour, Sarthak Jain, Byron C. Wallace, and Sameer Singh, 2021. An empirical comparison of instance attribution methods for NLP. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, NAACL- HITT 2021, Online, June 6-11, 2021, pages 967-975. Association for Computational Linguistics.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bha- gavatula, Yoav Goldberg. Yejin Choi, and Jonathan Berant. 2021. Commonsenseqa 2.0: Exposing the limits of AI through gamification. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1. NeurlPS Datasets and Benchmarks 2021, December 2021, virtual.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 6409)999-106
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof. Noah A. Smith, and Yejin Choi. 2019a. ATOMIC: an atlas of machine commonsense for if-then reasoning. In The Thirly-Third AAAI Con- ference on Artificial Intelligence, AAAI 2019, The Thirty-Firs: Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAT Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii USA, January 27- - February 1, 2019, pages 3027-3035. AAAI Press.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki- data; a free collaborative knowledgebase. Commun. ACM. 57(10):78-88 85..
Weiqi Wang, Tianqing Fang. Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosse- lut. 2023a. CAR: concepuualzzation-aageeee soner for zero-shot commonsense question answer- ing. CORR, abs/2305.14869
rea-
Weiqi Wang, Tianqing Fang, Baixuan Xu, Chun Yi Louis Bo, Yangqiu Song, and Lei Chen. 2023b. CAT: A contextualized conceptualization and instan- tiation framework for commonsense reasoning. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13111-13140. Association for Computa- tional Linguistics.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social iqa: Common- sense reasoning about social interactions. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing, EMNIP--IJCNIP 2019, Hong Kong, China, November 3-7, 2019, pages 4462-4472. Association for Computational Linguistics.
Zhaowei Wang, Quyet V. Do. Hongming Zhang, Jiayao Zhang, Weiqi Wang, Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. 2023c. COLA: con- textualized commonsense causal reasoning from the causal inference perspective. In Proceedings of the
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-ralk. In

---

61st Annual Meeting of the Association for Compu- tational Linguistics/ (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5253- 5271. Association for Computational Linguisties.
Appendices
A Additional Explanations and Experiments
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: state-of-the-art natural language processing. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing: System Demon- strations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 38-45, Association for Computa- tional Linguistics.
A.1 Motivations and Definitions of QADYNAMICS
Swayamdipta al. (2020) proposed training dy- namics, novel model-based offline data selection method for NLP classification tasks. It obtained the data statisties during the training process and proposed two measures, confidence and variability, to assess the difficulty of a particular instance for the model to learn.
Fomally, consider training dataset of size N, D {(x, s) The confidence of an instance (x ye) defined as:
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Pency Liang. 2023. Data selection for lan- guage models via importance resampling. CoRR, abs/2302.03169.
1 E
2
Palor xx))
(6)
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31- - November 4, 2018, pages 13-104.A As- sociation for Computational Linguistics.
where pos denotes the probability predicted by the model parameters 0 at the end of eh epoch.
And the variability measures the stability of Pole) (x;,, which is defined as:
Rowan Zellers, Ari Holumaan, Yonatan Bisk, Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4800. Association for Computational Linguis- tics.
Ee- (poto) (yi |x,))=))))),,) pa:)
6; =
E
Given the definition of confidence and variabil- ity above, following Swayamdipta et al. (2020), the training data can be distinguished into three dis- tinct regions, easy-to-learn, ambiguous, and hard- to-learn, respectively corresponding to high confi- dence, high variance, and low confidence. To ob- tain the subsets of easy-to-learn and hard-to-learn, we sort the dataset by confidence and take cer- tain percentage of data with the highest or lowest confidence (which, in experiments, can be 33% and 66%). Similar to Easy-to-learm, to obtain Am- biguous, we take data with the highest variance. As stated by Swayamdipta et al. (2020), the ambigu- ous and hard-to-lean regions lead to the largest benefits on out-of-distributie performance.
Jiarui Zhang, Filip llievski, Kaixin Ma, Jonathan Fran- cis, and Alessandro Oltramari. 2022. A study of zero-shot adaptation with commonsense knowledge. Automated Knowledge Base Construction (AKBC).
Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, and Xiang Ren. 2021. RICA: evaluating robust inference capabilities based on commonsense axioms. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Viruall Event / Punta Cana, Dominican Republic. 7-11 November. 1021, pages 7560-7579. Association for Computa- tional Linguistics.
However, when training with QA dataset that includes m distractors (m > 2), the confidence of the correct choice tends be underestimated due to the larger number of distractors compared the correct choice. To illustrate, given five options, where the logits associated are as follows: -1, -3, -3, -3, -3. Among these logits, the logit of the ground

---

truth option is -1. In this case, the confidence as- signed the correct choice 0.65, while the confi- dence level assigned to the distractors uniformly 0.91, indicating the logits of the ground-truth an- swer is relatively lower. Moreover, a model in the early training stage may make random guesses toward the answer, with probability of approxi- mately 1/ m for each candidate. The probability of correct choice should gradually approach 1, re- sulting in lower confidence in the ground-trume an- swer than the distractors. Additionally, affected by the nelse-ne distractors, the confidence in the correct option may be underestimated relative to the true value. To alleviate the effect of data imbalance and false negative choice, as defined in Equation (2), we compute the confidence by only comparing the logit score of the correct answer with the logit score of the easier distractor, which is less likely to be false negative. To verify the above statements, we compute the density of the difference between the logits of ground-truth an- swers and distractors. As shown in figure Fig. I, compared to Softmax, our method has a higher density in the vicinity of 0, indicating the differ- ence between logit scores is decreased. It can be stated that method narrows the distribution gap between positive and negative options. With the above definition, high confidence in correct choice indicates a high probability of being chosen, and low confidence may indicate the question is misla- beled.
aNLI
CSQA
PIQA
SIQA
WG
Question Numbers
1532
1221
1838
1954
1267
Choice Numbers
2
2
3
2
Table 4: Statistics of the validation set of each bench- mark.
is more likely to be selected, while the distractors are less likely be chosen. To implement the Difficult Choice selection method, we remove one distractor with higher confidence. When we ap- ply this method to the synthetic QA dataset, which has three candidates, 33% of the data is discarded. resulting in 66% of total data. For Hard-to-lean subset containing 50% of the total data, the amount of data becomes 33%.
As stated by Ma et al. (2021), the synthetic QA dataset includes ungrammatical questions as well as false negative distractors that appear plausible within the QA pair. Moreover, Dataset Cartogra- phy (Swayamdipta et al., 2020) suggests that con- fidence can also be used as a flag to identify mis- labeled instances in the dataset. Thus, to deal with these two issues, we suggest two strategies: Mis- labeled. removal and False-Neg. removal ($3.4). Mislaheled. involves excluding QA pairs with low-confidence ground truth answer, while False- Neg. involves excluding QA pairs with correct answers and distractors with similar logits.
A.2 Implementation Details
5
Ours
In this section, we introduce the implementations of our system. For hyperparameter tuning, follow- ing Ma et al. (2021), we set batch size 32, max sequence length 128, weight decay 0.01, warm- up proportion 0.05. We use an Adam opti- mizer (Loshchilov and Hutter, 2019) with the learn ing rate set to 5c-6 in all experiments. We evalu- ate our models on the validation set of synthetic datasets every 1000 steps and save the one with the highest validation accuracy. Each experiment is re- peated with different random seeds three times, and the average performance is reported. For comput- ing resources, all of our experiments are conducted on 4 NVIDIA RTX A6000 GPUs, each with 48G memory. Our code for zero-shot commonsense QA is mainly based on the code repository pro- vided by Ma et al. (2021), and all of the pre-trained language models are from the Huggingface Trans- formers Library (Wolf et al., 2020).
Softmax
4-
Density
3-
2
1
0
1.0
-05
0.0
0.5
1.0
AConfidence
Figure 1: The density of difference between the confi- dence of ground-truth answer and distractors.
Unlike natural language inference data, which is used in Dataset Cartography (Swayamdipta et al., 2020), when evaluating confidence for given QA pair, we should consider the confidence of all avail- able options. As a result, we define the confidence QA pair as Equation (4). A higher confidence level for a QA pair indicates that the positive choice

---

Models
aNLI
CSQA
PIQA
SIQA
WG
Commonsense@AA SIQA (OOD) PIQA (OOD)
Data Selection
ID
QADYNAMICS
82.3
71.6
81.2
65.6
79.1
Random 50%
75.1
59.2
75.8
w/o DC
80.4
71.3
79.8
64.9
78.9
Total data 100%
78.0
60.4
75.8
o w/o Mislabeled.
81.9
70.4
79.7
66.9
79.0
Easy-to-leam 50%
71.2
58.1
75.4
o w/o False-Neg.
82.3
72.2
79.8
63.3
79.0
Ambiguous 50%
76.4
61.1
76.5
o w/o LCC.
80.0
70.7
80.4
65.1
78.6
Hard-to-learn 50%
76.2
60.6
76.4
Difficult choice 60%
77.7
60.6
76.9
Table 5: Ablation study four components of QaDy- namics. DC stands for Difficult Choice Selection, and LC stands for Low Confidence Selection. The following five columns denote the accuracy (%) each bench- mark.
Table Supervised commonsense QA evaluation re- sults CSQA, SIQA and PIQA (Accuracy %). All experiments employ DeVERTaav3-BBae as the back- bone.
A.3 Ablation Study
tributes to better generalization ability of the model. In the future, we may consider transferring such technique into more advanced but challenging com- monsense tasks, such as commonsense knowledge base population (Fang et al., 2021b,a, 2023), causal reasoning (Chan et al., 2023; Wang et al., 2023c), and commonsense conceptualizations (Wang et al., 2023b,a; He et al., 2022).
In this section, we study the ablation of different components of our framework to determine the im- pact of using different data selection methods and strategies. There are four critical components that refine the QA dataset: Low Confidence (hard-to- learn) Selection, Difficult Choice Selection, Misla- beled., and False-Neg.. The data selection details include adopting Mislabeled. removal and False- Neg. removal on the total data, selecting 50% of total data with the lowest confidence, and discard- ing the distractor with higher confidence.
B Case Study
To further validate our framework, we present case studies in this section by showcasing instances se- lected by various strategies, which are illustrated in the Tab.
To study the effect of different components, we train DeBERTa-v3-Large as the backbone by sequentially dropping the four components mentioned above one at time. Their out-of- distribution performances on five different tasks are shown in Tab. 5. The results show that Difficult Choices Selection and Low Confidence Selection are effective strategies for improving the general- ization ability of the model, and eliminating mis- labeled examples and false negative distractors is also useful approach for enhancing overall per- formance.
Mislabeled. Removal.
Following Swayamdipta et al. (2020), we developed Mislabeled. to detect mislabeled examples, which involves extracting QA pairs with a ground answer that has low confi- dence. We have listed three mislabeled examples in the Tab. 7, where the confidence of the misla- beled option is relatively low compared to the other examples.
False-Neg. Removal. Ma et al. (2021) men- tioned the presence of false negative distractors that are also plausible in the QA pair. To address this issue, we implemented False-Neg. removal, which is designed to detect false negative distrac- tors. We provide three examples of this strategy in action below. As shown in Tab. 7, the confidence of the false negative distractor is consistently close to 0.5, suggesting that its logit value is always in proximity to that of the correct choice during the training process, which in line with the definition of False-Neg. Moreover, based on these examples, we can infer that false negative distractors are often caused by insufficient content, resulting in multiple correct choices.
A.4 Experiments on non-synthesized datasets
To assess the efficacy of our approach on non- synthesized datasets, we perform supplementary experiments using the training set of Common- senseQA (Talmor et al., 2019). Subsequently, we evaluate the model on both the validation set of CAmmonsense and other datasets such as So- cialIQA (Sap et al., 2019h, and PIQA (Bisk et al., 2020). These datascts serve in-domain and out- of-domain evaluations, respectively. The results are shown in Appx. $A.4. We observe that the Hard- to-learn, Ambiguous, and Difficult choice boost the oft-of-domain performance compared to the baseline. indicates that dropping easy choice con-

---

0.2
0.2
0.4
Contidenss
0.2
Ambiguous
0.9
hard-tollamm
0.00
0.95
0.10
0.15
0.20
0.25
0.30
Variabilitt
Figure 2: Demonstration of the synthetic QA dataset in the Zero-shot Commonsense QA task.
Drop easy distractors. Method Difficuli Choice discards the option with higher confidence. To better understand the effectiveness of this method, we analyze the easy choice and identify two com- monly occurring phenomena. First, we observed that the casier distractor is often more likely to contain grammatical errors, as demonstrated in the Easy Distractors: Grammatical Error. Second, the easy choice frequently has poor contextual rele- vance, shown in the Easy Distractors: Irrelevant to Context, while these two phenomena can also be found in other instances shown in Tab. 7. Remov- ing options that exhibit these features improves the overall quality of the dataset, which can lead to better performance and more reliable results.

---

Question
Candidates
Confidence
MISLABELED OPTION
Flynn is cleaning out Flyan's garage. As a result, Flyon felt
A: confused.
A: 0.32
B: elated.
B: 0.22
C: scared.
C: 0.85
Jamie decides to make some. Jamie is seen as
A: prepared.
A: 0.29
B: knowledgeable
B: 0.76
C: little.
C: 0.17
Tracy not wait to use it. As result, Tracy wanted to
A: have fun.
A: 0.18
B: do surveys.
B: 0.32
C: eat.
C: 0.88
FALSE NEGATIVE DISTRACTORS
Something that might happen as a con- sequence of watching movie is
A: inspiration.
A: 1.00
B: wood to burn.
B: 1,00
being cultured
C: 0.51
Ash gets Tracy's 's dog. As result, Ash wanted to
A:
writes exam.
A: 0.99
B: leave.
B: 0.98
C: make sure is ok.
C: 0.51
Bali maintains Ash relationship. Bali is
A:
committed.
A: 0.92
B: generous.
B: 0.51
C: adventourous.
C: 0.96
EASY DISTRACTORS: GRAMMATICAL ERROR
Sydney takes the medicine. Sydney is seen
A: reactive.
A: 0.75
B: violence.
B: 0.98
C: 0.97
C: desperate.
an office building for
A: advertising company,
A: 0.99
B: 0.99
B: purchase meals.
C: 0.67
C: carrying the first class mail.
EASY DISTRACTORS: IRRELEVANT CONTEXT
You are likely to find marker in
A: Afghanistanl.
A: 0.99
B: 0.98
B: school bag.
C: 0.77
C: label
opens windows. result, oth- ers
A: snubbed
A: 0.92
B: thankful
B: 0.88
C: worried
C: 0.78
Table 7: Case studies of different strategies and easy choice. We highlight the golden standard
false negative
and easy distractors


Reviews:
[{"review_id": "Deizyk46fh", "reviewer": "Reviewer_3DE3", "summary": "", "questions": "Q1: The backbone model (RoBERTa-large) for derive QA Dynamics is different from the finally model for QA (DeBERTa-v3-large). Why? \n\nQ2: Is the method also valuable for non-synthesized datasets?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your dedicated review. Below please find our responses to your review.\n\n# Statistics of Extra Resources Needed\n\nThank you for addressing the issue of extra resources required for obtaining dynamics. We believe that the additional resources invested in QaDynamics are manageable and justifiable.\n\nFirstly, it is important to note that the space and time resources needed for storing QAdynamics are relatively minimal compared to the dataset itself and conventional model training. The storage space required for storing QAdynamics is not substantial. To illustrate, our training set consists of 691,551 QA pairs, with each pair containing a question and three choices. Storing the QAdynamics of all QA pairs only necessitates an additional 16MB of storage. In total, 19 QA dynamic checkpoints are stored, resulting in a total requirement of $19*16=304$MB of additional storage. It is worth mentioning that this is the sole additional storage needed, as the model for generating QAdynamics computes on-the-fly and does not require local checkpoints. **Even for large-scale datasets, such as those containing 10 million QA pairs, a mere 4.4GB of additional storage is required. We believe this amount is insignificant in comparison to the storage required for the original datase**t.\n\nFor the training time, indeed some time has to be invented to train a model for generating the QAdynamics. In our case, we dedicated a total of 56.5 hours to obtain the training dynamics by training a Roberta-large model. Nevertheless, training the final QA model becomes less time-consuming due to the reduced quantity of data after the data selection process. The selected data can also be helpful in saving time during future training endeavors. Thus, **the overall training time is not extensively expanded**.\n\n# Model Selection for Computing QAdynamics\n\nThank you for addressing this matter. Our choice to utilize the RoBERTa-Large model for computing QAdynamics is grounded in a couple of significant factors.\n\nFirstly, the RoBERTa-Large model possesses a smaller size compared to DeBERTa-Large, **resulting in faster training and inference speeds**. Given that QAdynamics necessitates multiple on-the-fly inferences, opting for RoBERTa-Large is a more practical and feasible option.\n\nSecondly, through our testing, we have observed that RoBERTa-Large, when combined with choice selection and mixed strategy, **achieves a noteworthy performance average of **68.1** across five QA benchmarks. This performance surpasses the results reported by Ma et al. (2021)**. Thus, we have confidence in RoBERTa-Large's ability to effectively learn the QA set and differentiate between easy and difficult QA pairs.\n\nLastly, it is important to consider the inherent challenge in expecting a model to learn from data that is itself difficult to learn. If we were to exclusively select difficult-to-learn samples for the model and expect it to learn from them, it could potentially lead to further confusion. Taking all these factors into account, we have made the decision to employ RoBERTa-Large as the QAdynamics model.\n\n# Performance on non-synthesized datasets\n\nThank you for bringing up the concern regarding non-synthesized datasets. To address this, we conducted additional experiments where we trained the model using the **training set of CommonsenseQA and evaluated it on the dev sets of CommonsenseQA (in domain), SocialIQA (out-of-domain), and PIQA (out-of-domain)**. During the training process, we employed various QADynamics strategies, and the outcomes are presented in the following results. \n\n|     Data Type           | CommonsenseQA (In Distribution) |    SocialIQA (OOD)    |    PIQA (OOD)    |\n|---------------------------|--------------------|--------------------|--------------|\n|     Random              |        75.1         |        60.1         |     75.9     |\n|     Total data            |        78.0         |        60.4         |     75.8     |\n|     easy-to-learn       |        71.2         |        58.1         |     75.4     |\n|     ambiguous          |        76.4         |        61.1         |     76.5     |\n|     hard-to-learn       |        76.2         |        60.6         |     76.4     |\n|     difficult choice     |        77.7         |        60.6         |     76.9     |\n\nBased on the results, we have observed the effectiveness of QADynamics on non-synthesized QA benchmarks, such as CommonsenseQA. Furthermore, it enhances the generalizability of the QA model to other commonsense QA benchmarks. Therefore, we strongly believe that **our proposed method is not only advantageous for zero-shot commonsense QA but also invaluable for supervised QA models**.\n\n# Missing References and Naming Misunderstanding\n\n\nThank you for bringing the writing issues of our paper to our attention. We appreciate your feedback and are dedicated to improving the clarity and coherence of our work.\n\nWith regard to Table 2, we acknowledge that the term \"After Mislabeled\" may be a little misleading. In the camera ready version, we will make the necessary revisions to provide a more descriptive label, such as \"After Removing Mislabeled,\" to accurately reflect the process.\n\nDue to space limitations, we were unable to provide extensive details about the baselines and datasets. However, if granted an additional page in the camera ready version, we are fully committed to including all the necessary information. Specifically, the baselines referred to as \"Total 100% (Ma et al., 2021)\" employ DeBERTa-v3-Large as the backbone. It is important to note that all experiments presented in Table 1 are based on DeBERTa-v3-Large for fair comparisons.\n\nWe assure you that in the final version of the paper, we will address the missing reference and ensure that all abbreviations are clearly explained.\n\n# References\nSwayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N.A., & Choi, Y. (2020). Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. Conference on Empirical Methods in Natural Language Processing.\n\nMa, K., Ilievski, F., Francis, J.M., Bisk, Y., Nyberg, E., & Oltramari, A. (2020). Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering. AAAI Conference on Artificial Intelligence.\n\n***Thanks again for your valuable advice and we hope that our response will assist you in raising your score. If you need further clarification or discussion, please feel free to contact us.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method is novel and the result is impressive. In comparison of original training dynamics, this work extends to multi-choice styled setting. The method proposed in this paper can improve the quality of synthesized commonsense QA data, and it seems to be also applicable for other QA tasks and datasets.", "reasons_to_reject": "This paper does not report the extra time or space overhead. The QA Dynamics method requires to train an extra model and save its multiple checkpoints for the dataset diagnostic, which could be additional burden when applied to large-scale tasks. I wonder if larger models could be trained at the same cost and bring better performance.", "questions_to_authors": "Q1: The backbone model (RoBERTa-large) for derive QA Dynamics is different from the finally model for QA (DeBERTa-v3-large). Why? \n\nQ2: Is the method also valuable for non-synthesized datasets?", "paper_topic_and_main_contributions": "This paper is targeted to reduce the noise in synthesized data for zero-shot commonsense question answering. It propose several strategies based on training dynamics to correct or filter out poor-quality data. According to the experiment results, these strategies can help the models achieve better average performance on five benchmarks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The strategy expression in table 2 is somewhat confusing. For example, `After Mislabled` is actually removing mislabeled instances but its literally confusing.\n\n2. The baselines and datasets could be described more clearly. For example, do the `Total 100% (Ma et al., 2021)` baseline also based on DeBERTa-v3-Large?"}}, {"review_id": "ISCzRoNATK", "reviewer": "Reviewer_ukoU", "summary": "", "questions": "1. In line 714, it should be resulting lower confidence in ground-truth answer right?\n2. In line 722, if you’re using the second-lowest logits, then it should be the harder distractor? \n3. I didn’t understand how is figure 1 computed? If you’re plotting the difference between the logits, what does softmax refer to? Your logits did not come from softmax?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your dedicated review. Below please find our responses to your review.\n\n# Motivation of Our Implementation\n\nThank you for addressing the concerns regarding the clarity of our implementation's motivation. We hereby clarify about the limitations of the original training dynamics approach in QA tasks and elucidate the rationale behind our proposed approach.\n\n### Motivation for QADynamics\n\nFirst, the definition of logits in our task is the negative loss computed for each training instance, which is a real number, ranging from $-\\inf$ to $0$, and larger logits indicate higher probability to be chosen. \nIn the original definition of confidence from Swayamdipta et al. (2020), it performs softmax operation on the logits. Nevertheless, **due to the larger number of distractors compared to the correct choice, the confidence of the correct choice tends to be underestimated.** \n\nFor example, if there are five choices and the logits are -1, -3, -3, -3, -3, where the logit of ground truth is -1, the confidence of correct choice is 0.65 and the confidence of distractors is all 0.91 (marginal loss is used and the margin is set to be 1 in the original setting, so we assume the difference in logits of magnitude 2 is large enough). \n\nFurthermore, during the early stages of training, the model often provides random guesses and assigns equal probabilities to each choice. **This results in a lower confidence level for the correct choice compared to the distractors, leading to underestimation.** \n\nAdditionally, **the presence of challenging choices or false negatives further contributes to the underestimation of the confidence in the original version.**\nWe will add an figure illustrating that the confidence of the correct choice is consistently smaller than the confidence assigned to the distractors.\n\n### Motivation for Data Selection\n\nRegarding data selection, when employing the original training dynamics, the confidence of an individual instance (comprising a single question with multiple choices) becomes a linear transformation of the confidence associated with the correct choice. **This transformation is primarily influenced by the difficulty measure attributed to the correct choice.**\n\nConsequently, the primary *motivation* behind our implementation is to **address the issue of underestimating the confidence in the correct choice, which arises from challenging or false negative distractors, as well as the imbalanced distribution between ground truth and distractors.** Our approach aims to provide a more accurate assessment of the difficulty of choices or instances. We derive confidence of correct choice by comparing it with only one distractor, avoiding the imbalance of the amount of distractors. The selected distractor is the easier one, aiming to alleviate the impact of difficult or false negative distractors. In the figure 1 in Appendix, we prove that **our approach narrows the gap between confidence of correct choice and difficult choice, leading to a more justifiable learning results by our final QA model.**\n\n### Experiment Justification\n\n|      Data selection Strategy      | Dynamics Version | aNLI | CSQA | PIQA | SIQA | WG   | Avg. |\n|-----------------------------------|------------------|------|------|------|------|------|------|\n| Hard-to-learn - Mixed Strategy    |     Original (Swayamdipta et al. 2020)     | 80.0 | 70.0 | 78.6 | 63.1 | 79.9 | 74.3 |\n| Hard-to-learn - Mixed Strategy    |    **QAdynamics**    | 80.4 | 71.3 | 79.8 | 64.9 | 78.9 | **75.1** |\n| Difficult Choice - Mixed Strategy |     Original (Swayamdipta et al. 2020)    | 81.2 | 70.5 | 81.1 | 63.4 | 80.0 | 75.2 |\n| Difficult Choice - Mixed Strategy |    **QAdynamics**    | 82.3 | 71.6 | 81.2 | 65.6 | 79.1 | **76.0** |\n\n\nMoreover, we conduct additional experiments to verify our approaches. The experiment results are shown in the table above.\n\nFirst, we discard mislabeled and false negative examples, and take 33% low-confidence data which is derived from the original version as training data. The average result of the original training dynamics is **74.3**, while the average result of our training dynamics is **75.1**, indicating our approach is more suitable for data selection on QA tasks. \n\nThen, we adopt the mixed strategy and perform choice selection to obtain 33% instances of total data based on original training dynamics as training data. The average result of the baseline is **75.2**, while the average performance of our training dynamics is **76.0** (we will add this result in ablation study). \n\nThis shows firm proof that our option selection and QAdynamics are more useful than the original training dynamics in QA tasks.\n\n# Statistics of Each Strategy\n\nThank you for raising the issue of lacking detailed statistics for each strategy. In our approach, we employ a threshold to filter out mislabeled and false negative data from the entire dataset. We select a portion of the data, **ensuring that the quantity of the selected data matches the quantity of the data before the dropping process**. This selected data is then used as our training dataset. As a result, there is no need to discard mislabeled and false negative data further. The rationale behind removing erroneous data from the overall dataset first is to **prioritize the assurance of data correctness before categorizing them into different difficulty levels.** Intuitively, it is essential to establish the accuracy and reliability of the data before proceeding with any further division or analysis. The threshold is decided based on rough observations of QAdynamic distributions, which emphasizes the balance between quantity and quality.\n\nData statistics are shown in the table below.\n\n|                     |     Mislabeled     |     False Negative     |     Mixed Strategy     |     Data we used     |\n|----------|--------------|------|------|------|\n|   Data size   |   6465 (0.94%)   |       26320 (3.8%)     |      32875 (4.74%)    |          345775          |\n\n\nThe data we utilized corresponds to the training data quantity during the training of the best models. The percentage figures presented in the table represent the proportion of erroneous data within the total dataset. As mentioned by Ma et al. (2021), the human accuracy on atomic and cwwv synthetic data is 78.0 and 80.7 respectively. The data discovered automatically by our strategy is **4.74% of total data, which is close to 25% of the poor-quality or grammatically wrong data**. And **most of them are located in the low-confidence areas, indicating our framework’s contribution towards purifying the low-quality data**.\n\n# Unclear points in our paper\n\nIn line 714, it should be resulting lower confidence in ground-truth answer right?\nIn line 722, if you’re using the second-lowest logits, then it should be the harder distractor?\nI didn’t understand how is figure 1 computed? If you’re plotting the difference between the logits, what does softmax refer to? Your logits did not come from softmax?\n\nThank you for raising some unclear points in our paper. \nIn question one, the confidence assigned to the ground-truth answers is consistently lower than the confidence assigned to the distractors. As previously mentioned, we will include graphs to visually illustrate this phenomenon.\n\nFor question two, there is a typo in the logits expression. The logits should be negative loss. The higher the negative loss is, the corresponding choice is more likely to be chosen. Thus, it should be the lowest logit, which is the easier distractor.\n\nFigure 1 presents the difference between confidence of ground-truth answers and distractors. As mentioned above, our definition of logits is negative loss. This figure is derived by **random sampling the confidence of distractors and ground-truth answers and taking differences between them**. The blue part is the confidence difference derived from QAdynamics, while the orange part is from the original training dynamic. It indicates **our approach narrows the gap between confidence distribution of distractors and ground-truth answers**. \n\nWe apologize for any typo that may have caused misunderstandings regarding our paper. To ensure the accuracy of our results and conclusions, we have thoroughly reviewed the experimental data and corresponding graphs. **We can confirm that the typos mentioned above do not impact the validity of our results and conclusions.**\n\n# Missing Reference\n\nThank you for bringing the missing reference. We will add relavant discussions of this paper if we can get our paper accepted. \n\n# References\nSwayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N.A., & Choi, Y. (2020). Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. Conference on Empirical Methods in Natural Language Processing.\n\nMa, K., Ilievski, F., Francis, J.M., Bisk, Y., Nyberg, E., & Oltramari, A. (2020). Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering. AAAI Conference on Artificial Intelligence.\n\n***Thanks again for your valuable advice and we hope that our response will assist you in raising your score. If you need further clarification or discussion, please feel free to contact us.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is very intuitive that filtering out noisy data from synthetic data potentially improves the model’s performance. The experimental results also show significant gains across five benchmarks by training on less but higher quality data.", "reasons_to_reject": "Important technical details are not clearly explained, which weakens the claims made in this paper, in particular: \n1. I don’t quite understand the motivation for Equation 2 and Equation 3. For equation 2, compare to Swayamdipta et al. 20, you just normalize over one distractor instead of all distractors. Intuitively, this just increased the probability of the correct answer and nothing more. For equation 3, if the correct answer has a high probability, the distractors will have low probabilities, thus these two measures (confidence for the correct answer and distractor) are strongly correlated or basically the same. Why don’t you use the original definition of confidence from Swayamdipta et al. 20? If you stick with the original training dynamics, and apply the easy distractor removal, mislabeled removal and false negative removal, would the model achieve similar performance? \n2. For mislabeled, false-neg and mixed strategy, do you apply these methods on the remaining 33% of data to further filter out low-quality instances? If so, how much data do you filter out for each of the strategies? You should provide a table of statistics for this. \n\nI am willing to increase my scores if my concerns and questions are addressed.", "questions_to_authors": "1. In line 714, it should be resulting lower confidence in ground-truth answer right?\n2. In line 722, if you’re using the second-lowest logits, then it should be the harder distractor? \n3. I didn’t understand how is figure 1 computed? If you’re plotting the difference between the logits, what does softmax refer to? Your logits did not come from softmax?", "paper_topic_and_main_contributions": "This paper proposed a method for automatically detecting and filtering out noisy examples from synthetically constructed data from commonsense knowledge graphs. Inspired by the idea of datasets cartography (Swayamdipta et al. 20), the authors proposed to detect mislabeled examples, false-negative examples and easy-to-learn distractors from the synthetic data, using the confidence measure of the model along the training process. The final models are trained on the dataset after filtering. The authors evaluated the models on five commonsense reasoning datasets, and the results show that the proposed method greatly improved the performance compared to no filtering or filtering using the method from  Swayamdipta et al. 20.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There is a previous paper that studied training dynamics on synthetic QA pairs from CSKBs, you should consider discussing it in your related work. \n\nA Study of Zero-shot Adaptation with Commonsense Knowledge\nJ Zhang, F Ilievski, K Ma, J Francis, A Oltramari - Automated Knowledge Base Construction (AKBC), 2022"}}, {"review_id": "RaHggLQko6", "reviewer": "Reviewer_k4z2", "summary": "", "questions": "Why specifically were 33% and 66% chosen for this experiment? Is there any ablation study showing that this is the best option besides adopting that from the baseline? \n\nWhat is the difference between training dynamics and QA dynamics? Why there are no paragraphs talking about the training dynamics besides the baseline section? \n\nHow do you decide the number of choices for each datasets? \n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your dedicated review. Below please find our responses to your review.\n\n# Unclear Challenges\n\nThank you for raising the issue of unclear explanation about challenging topics. The main three challenges are actually identified by Ma et al. (2021), who claim that:\n\nCSKB contains annotation artifacts that may bias the model from learning commonsense knowledge but rather seek easy backdoors to remember these artifacts.\nCSKBs also contain ungrammatical knowledge that is hard to be comprehended, and the synthesized QA pairs are hard to understand as well. For example, in ATOMIC, ``PersonX enjoys drinking, xEffect, stay up all night.``\nThere is a lack of efficient negative sampling strategy to sample true-negative distractor at scale. For example, the triple in CSKB is ``PersonX feels hungry, xWant, eat a burger``. The sampled instance used for constructing distractors is ``PersonX likes Chinese food, xWant, eat dumplings``. In this case, the sampled distractor is also correct.\n\nDue to space constraints, there is not enough space for us to cover additional examples. We will add certain contents if we are granted another page for camera-ready.\n\n# Difference between QAdynamics and the original approach, and the content of QAdynamics\n\nThank you for bringing up the issue regarding the content of dynamics. In the following paragraphs, we will first explain the content of QaDynamics, then illustrate the overlap and difference between QaDynamics and the original version and the motivation to propose these differences.\n\nIn our approach, the logits are negative loss of each instance, ranging from $-\\inf$ to $0$. The dynamics refer to the records of logits, obtained by evaluating the model on the whole training set after specific epochs or iterations. The confidence is computed via the dynamics recorded. Due to limited space, we illustrate the motivations and ideas of QaDynamics in the appendix. \n\nThe **overlap** between the original version and QaDynamics is analysis towards data. They both obtain the logits of instances during training, compute the confidence and variability of data, and discriminate data into several areas. **QAdynamics different in the way that it is specifically designed for QA tasks.**\n\n### Motivation for QADynamics\nIn the original definition of confidence from Swayamdipta et al. (2020), it performs softmax operation on the logits. Nevertheless, **due to the larger number of distractors compared to the correct choice, the confidence of the correct choice tends to be underestimated.** \n\nFor example, if there are five choices and the logits are -1, -3, -3, -3, -3, where the logit of ground truth is -1, the confidence of correct choice is 0.65 and the confidence of distractors is all 0.91 (marginal loss is used and the margin is set to be 1 in the original setting, so we assume the difference in logits of magnitude 2 is large enough). \n\nFurthermore, during the early stages of training, the model often provides random guesses and assigns equal probabilities to each choice. **This results in a lower confidence level for the correct choice compared to the distractors, leading to underestimation.** \n\nAdditionally, **the presence of challenging choices or false negatives further contributes to the underestimation of the confidence in the original version.**\nWe will add an figure illustrating that the confidence of the correct choice is consistently smaller than the confidence assigned to the distractors.\n\n### Motivation for Data Selection\n\nRegarding data selection, when employing the original training dynamics, the confidence of an individual instance (comprising a single question with multiple choices) becomes a linear transformation of the confidence associated with the correct choice. **This transformation is primarily influenced by the difficulty measure attributed to the correct choice.**\n\n\nConsequently, the primary *motivation* behind our implementation is to **address the issue of underestimating the confidence in the correct choice, which arises from challenging or false negative distractors, as well as the imbalanced distribution between ground truth and distractors.** which is **the first difference.** This approach provides a more accurate measure towards the difficulty of each option, and derives the difficulty of instances according to each option, instead of only ground truth.\nThe **second difference** is we adopt choice selection, which is a novel data selection method only applied to QA tasks. \nThe **third difference** is our approach provides a method to detect false negative examples, which is a challenge caused by the QA synthesizing strategy.\n\n\n# Discussion towards results\n\nFor the result parts, we mention the good performance of our approach, and conduct the ablation study by separating our approach into four parts. And we provide a direct analysis towards the results and the effect of each part by experiment results of ablation study, data annotation and the study on the confidence difference distribution. However, although there has been many works related to training dynamics (Aviad et al. 23, Fenia et al. 22, Tiberiu et al. 22), it is still unknown why hard-to-learn and ambiguous data is more effective for improving generalization ability. This remains as an interesting future work.\n\n# How to decide the amount of data\nFor the quantity of data to be selected, the reason for choosing 66% is the training set has 3 options, and after dropping one distractor the data size will become 66% of original. To validate the effectiveness of choice selection, we set baselines with the same amount of data. Following Swayamdipta et al. (2020), we select 33% of total data. This measure is justifiable in previous works and ensures fair comparison.\n\n# How to decide number of choices\nThe number of choices is actually a hyperparameter. As to the QA dataset proposed by Ma et al. (2020), it contains 2 distractors and we can only drop one of them (otherwise no distractor). For QA datasets which include more than 2 distractors, the number of distractors to be discarded is a hyperparameter decided before experiments, and additional experiments may be needed to select the number. However, there is no need to change our framework, as it is capable of dealing with arbitrary number of options in any QA dataset.\n\n# Missing references and representation improvements\n\nThank you for mentioning the missing references and representation improvements. We will add the missing references and modify the unclear points in our paper. For the annotation, Wang et al. (2023) leverage the same annotation setup to evaluate the QA pairs’ selection results. We both aim for evaluating the effect of eliminating false-negative distractors and low-quality QA pairs, which we believe is a reliable measure.\n\n# References\n\nWang, W., Fang, T., Ding, W., Xu, B., Liu, X., Song, Y., & Bosselut, A. (2023). CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering. ArXiv, abs/2305.14869.\n\nAviad Sar-Shalom and Roy Schwartz. 2023. Curating Datasets for Better Performance with Example Training Dynamics. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10597–10608, Toronto, Canada. Association for Computational Linguistics.\n\nSwayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N.A., & Choi, Y. (2020). Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. Conference on Empirical Methods in Natural Language Processing.\n\nMa, K., Ilievski, F., Francis, J.M., Bisk, Y., Nyberg, E., & Oltramari, A. (2020). Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering. AAAI Conference on Artificial Intelligence.\n\nFenia Christopoulou, Gerasimos Lampouras, and Ignacio Iacobacci. 2022. Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2595–2611, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\nTiberiu Sosea and Cornelia Caragea. 2022. Leveraging Training Dynamics and Self-Training for Text Classification. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4750–4762, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\n***Thanks again for your valuable advice and we hope that our response will assist you in raising your score. If you need further clarification or discussion, please feel free to contact us.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The overall methods are clearly stated: to create synthesised QA pairs and select part of them for PLM training, by implementation of QA dynamics of QA pairs compared with baseline training dynamics. \n\nOverall the writing is concise and clear for a short paper. \n\nThe author also provides good results in terms of both metrics and human evaluations. ", "reasons_to_reject": "From line 45 to line 68, the challenges of this topic is discussed. However, it's hard to get any impression of the problems in this area without proper examples and explanations. \n\nSection 2.2, the content of dynamics is not well explained. It's hard to tell the relationship between dynamics and logit scores, confidence and different checkpoints. Instead, the author only points out one or two references to the readers, which makes it hard to understand the motivations and idea of dynamics. \n\nResults parts, we see good results as expected. However, there is no future discussion regarding these results. We are interested to know why the settings influence the overall results.", "questions_to_authors": "Why specifically were 33% and 66% chosen for this experiment? Is there any ablation study showing that this is the best option besides adopting that from the baseline? \n\nWhat is the difference between training dynamics and QA dynamics? Why there are no paragraphs talking about the training dynamics besides the baseline section? \n\nHow do you decide the number of choices for each datasets? \n\n", "paper_topic_and_main_contributions": "This paper targets the problem of commonsense QA. While the current approaches to creating QA pairs for fine-tuning the PLMs are not of good quality, they propose a dynamics-driven framework to generate and select good QA pairs for commonsense fine-tuning. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "line 236, line 238, the references to the baseline models are missed. \n\nSection 3.4, any references showing the human evaluation methods for QA-related datasets?"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 1, repeated labels of \"mislabelled\" and \"mislabeled removed\". Repeated model size for training dynamic and QA dynamics. \n\nIn Table 2, the upper and down arrows indicate the best performance comes from the largest or smallest values (missing now). \n\nLine 049, What is the triple, & tail? better explain it with KG\n\nLine 194, what is very low confidence? \n\nFormula 1, better useP(t_i|t1, t2, ..., t_{i-1}, t_{i+1}, ..., t_n)\n\nFormula 2, why especially use T1 and A1 as an example, should try to use some general indication. "}}]

Author rebuttal:

Meta-review: **Pros**: the methods is novel and result is impressive as all reviewers agreed.

**Cons**: there are confusions or understanding issues raised by multiple reviewers. They all grasp the confusing parts after author discussion though. Hope the authors could include the explanations in the updated version of the paper.

Overall, This short paper proposed a novel framework with training dynamics to filter low-quality data for multi-choice selection problem. The results is impressive compared to multiple baselines.
Decision: Accept-Findings