Chain of Thought Prompting Elicits Knowledge Augmentation
Year: 2023

Abstract:
The knowledge-augmented deep learning paradigm refers to a paradigm in which domain knowledge is identified and integrated into deep models. Conventional methods typically employ task-specific approaches to gather external knowledge from various sources. In contrast, large language models are extensively pre-trained and can serve as a comprehensive source of external knowledge. In this paper, we propose CoT-KA, a Chain-of-Thought-based method that augments knowledge for deep learning. CoT-KA avoids the need for additional knowledge retrieval or knowledge reasoning models, as required in conventional augmentation methods. Our results demonstrate that CoT-KA outperforms both pure CoT-based methods and the non-augmented method across the majority of eleven publicly available benchmarks for various reasoning tasks 1.

heading: 1 Introduction
text: The Knowledge-Augmented deep learning (KADL) (Cui et al., 2022) paradigm refers to the deep learning paradigm in which domain knowledge is identified and integrated into the deep model. Adding domain knowledge makes it possible to develop deep learning that is data-efficient, generalizable, and interpretable (Cui et al., 2022). For example, retrieving external knowledge from an external knowledge pool like Wikipedia is typically required for open domain question answering and dialog generation (Izacard and Grave, 2021; Zhang et al., 2023). Logical equivalence laws such as contraposition and transitive laws help extend the implicit logical information (Yu et al., 2019; Wang et al., 2022a).
External knowledge is derived from various sources. For instance, commonsense knowledge can be extracted from commonsense knowledge
∗Corresponding author: Jing Zhang. 1Our code and data are available at https://github.
com/RUCKBReasoning/CoT-KA
bases like ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2019). Domain-specific knowledge can be retrieved from knowledge bases such as Wikipedia and Freebase (Bollacker et al., 2008). Logic knowledge, on the other hand, can be in the form of human-defined propositional or first-order logic, which is then utilized as rules for reasoning. In summary, existing knowledge augmentation methods typically involve either creating a retriever to gather relevant knowledge or developing a reasoner to leverage the logical rules within the external knowledge sources (Chen et al., 2017; Izacard and Grave, 2021; Wang et al., 2022a; Zhang et al., 2023).
Recently, large language models (LLMs) (Zhao et al., 2023) have shown their potential as both the source and the retriever or reasoner of external knowledge. LLMs are pre-trained on a huge scale of datasets. Thus, they have already embedded a large amount of knowledge into their parameters, which can be considered a source of external knowledge. The reasoning ability of LLMs allows them to provide knowledge from their parameters without needing an extra retriever or a reasoner. The latest chain-of-thought (CoT) prompting technique
6519
(Wei et al., 2022), which elicits LLMs to generate a series of sentences that mimic the reasoning process for arriving at the answers, improves the reasoning ability of LLMs. It has proved to be remarkably effective in a variety of complex reasoning tasks such as math word problems and commonsense question answering (Wei et al., 2022). CoT prompting shows potential as a general technique to retrieve knowledge from LLMs.
In this paper, we propose CoT-KA – a CoTbased method to retrieve knowledge from LLMs for Knowledge-Augmented deep learning. CoTKA utilizes an LLM as a knowledge source, leveraging CoT prompting to guide the LLM in providing knowledge that can serve as evidence to support downstream reasoning from the input to the answer. Unlike conventional KADL approaches, CoT-KA eliminates the need for additional knowledge retrieval or a separate knowledge reasoning model. Specifically, we begin by extracting CoTs as knowledge from the LLM using either few-shot (Wei et al., 2022) or zero-shot (Kojima et al., 2022) CoT prompting. The former involves providing a few demonstrations to guide the LLM’s reasoning, while the latter employs a template such as “let’s think step by step” to inspire the LLM. The extracted CoTs are then appended to the original inputs, marked by a special token, to create augmented text. Finally, we fine-tune a small taskrelevant pre-trained language model (PLM) on the dataset augmented with CoTs.
We generate CoTs using the public GPT-3 (Brown et al., 2020) (175B parameters) API2. For NLU (Natural Language Understanding) tasks, we employ ALBERT (Lan et al., 2019) and DeBERTa (He et al., 2021) as the task-relevant models. T5 (Raffel et al., 2020) is utilized as the task-relevant model for NLG (Natural Language Generation) tasks. We evaluate models’ performance using eleven benchmarks, including (i) commonsense reasoning (CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021), Date Understanding, Sports Understanding (Srivastava et al., 2022)); (ii) arithmetic reasoning (AQUA-RAT (Ling et al., 2017), GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014)); (iii) symbolic reasoning (Last Letter Concatenation (Wei et al., 2022)), where all commonsense reasoning benchmarks and AQUA-
2Public API available at https://openai.com/api/
RAT are formulated as NLU tasks, and the other arithmetic reasoning benchmarks and Last Letter Concatenation are formulated as NLG tasks in this paper. Particularly, we convert all of the multichoice question answering tasks into NLU tasks. Extensive experimental results show that in the majority of tasks, CoT-KA outperforms the original fine-tuning results without the use of CoTs as augmented knowledge. CoT-KA also surpasses FewShot-CoT and Zero-Shot-CoT on LLMs, which directly parse answers from the generated CoTs.

heading: 2 Related Work
text: Knowledge Augmented Technology. The integration of external knowledge into deep learning models through knowledge augmentation approaches has gained significant attention in various NLP tasks, including question answering (Chen et al., 2017; Izacard and Grave, 2021), dialogue generation (Zhang et al., 2023), and logical reasoning (Wang et al., 2022a). For instance, in the context of answering open-domain questions where supporting evidence is not explicitly provided (Izacard and Grave, 2021), Chen et al. (2017) utilized techniques such as bigram hashing and TFIDF matching to retrieve relevant documents from external knowledge sources. Similarly, Fusionin-Decoder (Izacard and Grave, 2021) employed methods like BM25 (Robertson et al., 1995) and DPR (Karpukhin et al., 2020) for evidence retrieval. By augmenting the questions with these retrieved pieces of evidence, the models can better reason and provide answers. Logic reasoning is another challenging task that requires a deep understanding of the logical structure within a given text to arrive at the correct answer. To facilitate such logic-level analysis, human-defined logic rules are introduced. Wang et al. (2022a) proposed LReasoner, a logicdriven context extension framework that extends implicit logical information by performing logical reasoning using these predefined rules. The framework enhances the original input by verbalizing and concatenating the implicit logical information, enabling subsequent answer reasoning.
Fusion-in-Decoder and LReasoner inspire our work to extend the external knowledge into the original input. However, the knowledge in these knowledge augmentation methods is sourced from external knowledge bases or pre-defined logical rules, requiring a retriever for knowledge extraction or a reasoner for rule application in the process. In
contrast, we utilize LLMs that eliminate the need for an additional retriever or reasoner to acquire knowledge for augmentation.
Chain of Thought Prompting on LLMs. A CoT is a series of intermediate natural language reasoning steps that lead to the final output, inspired by how humans use a deliberate thinking process to perform complicated tasks. Experimental results using various LLMs, such as GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022), demonstrate that CoT prompting enhances performance across a range of arithmetic, commonsense, and symbolic reasoning tasks (Wei et al., 2022).
Wei et al. (2022) initially propose Few-ShotCoT, which requires the manual design of a few demonstrations to facilitate the generation of reasoning paths. In contrast, Kojima et al. (2022) propose Zero-Shot-CoT, which employs a single zeroshot prompt that elicits CoTs from LLMs. By simply adding “Let’s think step by step” before each answer, Zero-Shot-CoT demonstrates that LLMs are capable zero-shot reasoners without the need for any manually constructed few-shot examples. Furthermore, Wang et al. (2022b) introduce a new decoding strategy called self-consistency, which involves sampling multiple LLM outputs and aggregating them through majority voting. This strategy encourages the model to consider multiple CoTs when generating answers. However, to achieve optimal performance, a large number of reasoning paths (e.g., 40 paths) must be generated, leading to increased computational costs.
All of these CoT prompting methods directly extract the answer from the CoTs. In contrast, our method utilizes the generated CoTs as supplementary knowledge to improve the fine-tuning of taskrelevant models. Moreover, our method demonstrates good performance even when a limited number of CoTs are provided, unlike self-consistency, which relies on generating a large number of CoTs.

heading: 3 Pilot Study
text: In this section, we explore the effectiveness of CoTaugmented fine-tuning by simply appending one CoT to the original input. We assess the validity of this approach on two commonsense reasoning datasets, CSQA and StrategyQA.
CoT-augmented Fine-tuning. To perform finetuning on ALBERT, we extend the original input text by adding a CoT. We utilize ALBERT-large-
v2 for our experiments. Specifically, we generate CoTs using both few-shot and zero-shot CoT methods, known as Few-Shot-CoT and Zero-ShotCoT, respectively. Few-Shot-CoT employs the same demonstrations as described in (Wei et al., 2022). For Zero-Shot-CoT, we utilize the template “Let’s think step by step”. As the LLM, we employ GPT-3 with 175-billion parameters (text-davinci002). Subsequently, we extend the generated CoT into the input of each sample within the CSQA and StrategyQA datasets. Finally, we perform finetuning on ALBERT using the augmented datasets.
The experiment results in Table 1 show that both the Zero-Shot-CoT and Few-Shot-CoT augmented fine-tuning significantly enhance the performance of the original fine-tuning method.
The Impact of CoT as Additional Knowledge. Given that the answers within CoTs can potentially be incorrect, we hypothesize that this portion of the CoTs will have a negative effect on the fine-tuning and mislead the model’s prediction. To further explore the effect of CoTs on fine-tuning, we compare the fine-tuning result of the PLMs before and after adding CoTs through a variety of data analyses.
We investigate the extent to which the prediction results are altered when the model’s input is expanded with a CoT. We perform fine-tuning on both the original samples (baseline) and the expanded samples (CoT-extended). Subsequently, we evaluate the fine-tuned models using the validation set. For each instance in the validation set, we compare its predictive result between the originally fine-tuned ALBERT and the CoT-augmented fine-tuning version. Additionally, we define three categories of CoTs during the process.
• A CoT is labeled as a positive CoT if the addition of the CoT changes the prediction result from incorrect to correct. This indicates a beneficial
influence on the model’s prediction.
• Conversely, a CoT is labeled as a negative CoT if the addition of the CoT changes the prediction result from correct to incorrect. This indicates a misleading effect on the model’s prediction.
• Furthermore, a CoT is labeled as a neutral CoT if the model’s prediction result remains the same after the CoT is added. In such cases, it is not easy to judge the impact of this CoT on the model.
The left figure in Figure 2 illustrates the ratio of positive, neutral, and negative CoTs. It is observed that among the model’s prediction results that change after adding a CoT, the ratio is 36.2% (166 out of 458). Within this group, the ratio of positive CoTs is 61.4%, while the ratio of negative CoTs is 38.6%. These findings suggest that the model successfully resolves 63.3% (102/161, the number of positive CoTs divided by the number of incorrectly predicted samples in the baseline) of the data samples that were incorrectly predicted prior to adding a CoT.
The second objective is to test our hypothesis that an incorrect CoT (the answer in the CoT is incorrect) may have a negative impact on the model and therefore mislead the prediction of the model. If an incorrect CoT is added to the original input text, what impact does it have on the model’s prediction? As the right figure in Figure 2 shows, when an incorrect CoT is added to the original input, the model still has a high probability (17.1%) of not being misled by the incorrect CoT and making accurate predictions. Furthermore, we investigate the
extent to which the model would mispredict when a correct CoT (the answer in the CoT is correct) is added. As shown in the figure on the right of Figure 2, the model has a low probability (5.0%) of making an incorrect prediction.
In the case of StrategyQA, when the answer in the CoT is incorrect, the alignment ratio is 1 − Ratio (#Not misled), which equals 82.9%; When the answer in the CoT is correct, the alignment ratio is 1−Ratio (#Not inspired), which equals 95.0%. The result demonstrates that CoT is a powerful feature, and the model’s predictions tend to align closely with the answers provided in CoT. On the other hand, the fine-tuning strategy employed causes the model’s predictions to treat CoT as a secondary feature of the original input, rather than strictly following it. In cases where the answer in CoT is correct, the model is likely to align its predictions with the answers in CoT. Conversely, when the answer in CoT is incorrect, there is a relatively high probability that the model will deviate from the answer in the CoT, preventing misleading from the incorrect CoT.
In addition, our attempts to preserve the reasoning steps in the CoTs while removing the answers have resulted in a degradation in performance. We recognize that the presence of incorrect answers in some CoTs can have a negative impact. However, we also believe that the inclusion of correct answers in CoTs can yield positive effects, and the answers within CoTs are a more influential factor than the reasoning paths themselves.

heading: 4 CoT-KA
text: In this section, we propose CoT-KA – a CoT-based method for knowledge augmentation. Our method leverages multiple CoTs retrieved from LLMs to provide more auxiliary knowledge for KADL. CoTKA consists of three steps as shown in Figure 3: (1) CoT Generation: Generating multiple CoTs for each sample in the train, dev, and test sets. (2) Input Augmentation: Taking the generated CoTs as the additional knowledge into the original input text for each sample. (3) Task-relevant Model Training: Fine-tuning a task-relevant model using the CoTaugmented samples.

heading: 4.1 CoT Generation
text: We try both Few-Shot-CoT and Zero-Shot-CoT prompting on LLM f to generate multiple CoTs. Formally, given an original samples (xi, yi), where
xi is the original input and yi ∈ Y denotes the label. We generate a CoT set consisting of multiple CoTs based on the model f :
CoT (i) = f(d, xi) (1)
where d denotes the CoT demonstrations that inspire model f to generate CoTs, and CoT (i) is the generated CoT set of the i-th sample, which consists of m CoTs:
CoT (i) = {CoT (i)1 , CoT (i) 2 , ..., CoT (i) m } (2)
For each sample, we independently generate m CoT outputs from f in each run.

heading: 4.2 Input Augmentation
text: In the second step, we apply the generated CoTs as additional knowledge to enrich the input text of the original samples. The extended input text of each sample is a concatenation of an original input (e.g. a question), and the generated multiple CoTs. For each sample, we construct an extended input text as follows:
x̃(i) = concat(x(i), CoT (i)) (3)
where x̃(i) is the i-th extended input text, x(i) is the i-th original input, and CoT (i) is the i-th generated CoT set. concat() is a concatenation function that concatenates the original input and the generated CoTs. More concretely:
concat(x(i), CoT (i))
= x(i)|| [EXT ] CoT (i)1 ... || [EXT ] CoT (i)m (4)
where [EXT ] is the special token to denote a CoT, and || denotes the concatenation operator.

heading: 5 Experiments

heading: 5.1 Experimental setup
text: Tasks and Datasets. We evaluate CoT-KA on the following reasoning benchmarks3.
3By default we use the train, dev, and test split of all the datasets if the labels are available for evaluation. For CSQA and StrategyQA, we only use the train and dev split.
• Commonsense reasoning. We evaluate our method on four commonsense reasoning tasks: CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021) and two benchmarks from the BIGbench effort (Srivastava et al., 2022): Date Understanding and Sports Understanding.
• Arithmetic reasoning. We use six arithmetic reasoning benchmarks: AQUA-RAT (Ling et al., 2017), GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014).
• Symbolic Reasoning. We use the Last Letter Concatenation from Wei et al. (2022). 4
Implementation. 4We do not use the Coin Flip dataset for the evaluation because it is a simple classification task for fine-tuning. This is because ALBERT-large-v2 and DeBERTa-v3-large can already achieve 100% accuracy in the evaluation phase.
• CoT Generation Models. We use GPT-3 of the text-davinci-002 engine with 175-billion parameters to generate the CoTs used in CoT-KA.
• CoT Demonstrations. For a fair comparison, we perform Few-Shot-CoT with the same demonstrations as in Wei et al. (2022) and use the same zero-shot prompt as in Kojima et al. (2022) to perform Zero-Shot-CoT.
• Sampling Scheme. To generate diverse CoTs, we apply temperature sampling during the CoT generation. Specifically, we use the same T=0.7 as in (Wang et al., 2022b) for a fair comparison.
• Data Preprocessing. For certain undivided datasets, we divide them into train, dev, and test sets for fine-tuning, following a ratio of 6:2:2. Further details regarding the dataset splits can be found in Appendix A.1. Additionally, as the original questions and demonstrations used for CoT generation may include option information (e.g., Answer Choices: (a) ignore ...(e) avoid),
the generated CoT will also contain option markers (e.g., the answer is (a)). To provide valuable information within the CoTs, we replace the option markers in the generated CoT with their corresponding textual content (e.g., the answer is
“ignore”).
• Classifier Models. We conduct the main experiments using two backbone PLMs: ALBERTlarge-v2 and DeBERTa-v3-large. The hyperparameters for the training process are reported in Appendix A.2.
Baselines. We take three methods as the baselines: Zero-Shot-CoT, Few-Shot-CoT, and SelfConsistency. Furthermore, to demonstrate the extent to which the CoT knowledge elicits the KADL, we also compare our method with the original finetuning baselines, which solely employ the original text for fine-tuning.

heading: 5.2 Main Results
text: Table 2 compares the accuracy across eleven datasets from three categories of NLU and NLG tasks. The Zero-Shot-CoT results are taken from Kojima et al. (2022), and the Few-Shot-CoT results are taken from Wei et al. (2022). For SelfConsistency (5 sampled CoTs), we report the result based on a majority vote. The CoT-KA results are averaged over at least five random runs (see Appendix for more details), where we use the different seeds to sample 5 CoTs from a CoT set containing 10 generated CoTs in each run.
As shown in Table 2 and 3, the performance of CoT-KA surpasses all baselines on most tasks. We have made several findings: (1) The CoTs generated by Zero-Shot-CoT and Few Shot-CoT can be utilized with CoT-KA, resulting in significantly improved performance compared to the fine-tuning baselines. Additionally, the CoTs generated by Few-Shot-CoT exhibit better performance com-
pared to Zero-Shot-CoT when they are used with CoT-KA. (2) CoT-KA achieves better performance on the NLU tasks than on the NLG tasks. (3) CoTKA shows different robustness on different models. While DeBERTa outperforms ALBERT on most tasks, CoT-KA is more robust on ALBERT and exhibits performance improvements across all tasks.

heading: 5.3 Knowledge Augmentation Comparison
text: To compare CoT-KA with other knowledge augmentation methods, we employ BLINK (Wu et al., 2020) to enrich the entity knowledge in the question. BLINK is a two-stage entity linking approach based on BERT (Kenton and Toutanova, 2019). We use BLINK to link the entities mentioned in the question and retrieve their corresponding entity information. BLINK provides a short description for each entity, which we utilize as extensions to enrich the questions.
As shown in Table 5, the entity knowledge-based augmentation method improves performance on Sports Understanding but has a negative impact on StrategyQA, with both performing worse than our method. Additionally, we observe that approximately 29% of questions in StrategyQA and 3% in Sports Understanding could not have entities extracted. Furthermore, the average number of recognized entities in a Sports Understanding question is 1.095, while in StrategyQA, it is 0.928. Moreover, Table 4 demonstrates that entity information may not always include the specific information required by the questions. In contrast, our method can add more useful information, resulting in a more substantial improvement.

heading: 5.4 The Effect of CoT Size
text: To demonstrate the effect of the number of sampled CoTs, we vary the number of sampled CoTs (1, 2, 3, 4, 5) in CoT-KA and evaluate on StrategyQA. The results are shown in Figure 4. The experimental results indicate that as the number of CoTs increases,
there is a general upward trend in the performance of CoT-KA . This trend becomes more pronounced when the CoTs are generated by Few-Shot-CoT. More results are reported in Appendix B.

heading: 5.5 CoT Selection Strategy
text: CoT-KA can only extend a small number of CoTs due to the maximum length limitation of the input sequence that the language model can handle. Therefore, it is natural to consider designing a CoT selection strategy to choose higher-quality CoTs from the generated CoT set for KADL. Each CoT can be expressed as: ti ∈ {t1, t2, ..., tK} , where ti is the i-th token. We can get the log prob of each generated token when using GPT3 API to generate reasoning chains. The log prob refers to the natural logarithm of the probability that the token occurs next given the prompt. To select the 5 reasoning chains with higher confidence from the 10 generated CoTs, we score the generated CoTs using the following formula:
score(CoTj) =
∑Kj i=1 exp(log p(ti))
Kj
=
∑Kj i=1 p(ti)
Kj (5)
where p(ti) denotes the probability of generating the i-th token, and log denotes the logarithm. and Kj is the total number of tokens in the j-th CoT. The results shown in Table 6 demonstrate that selecting CoTs from the generated set based on the probability of token generation in the sentence does not lead to a significant improvement in the performance of CoT-KA .

heading: 6 Conclusion and Future Work
text: This paper introduces a CoT-based method to retrieve knowledge from LLMs for KnowledgeAugmented deep learning (CoT-KA) that elicits
knowledge augmentation on a variety of NLU and NLG benchmarks. Unlike conventional knowledge augmentation approaches, our method does not require a retriever or a reasoner, yet it surpasses the performance of conventional knowledge-based methods and other CoT-based approaches across a range of public NLP tasks.
In the future, it is worthwhile to investigate other methods that can provide insights from LLMs. Exploring new approaches for leveraging the capabilities of LLMs to enhance knowledge augmentation represents a promising area for future research.

heading: 7 Limitations
text: One limitation of CoT-KA is that it performs finetuning based on the PLMs, and the input sequence length limit of the PLMs allows us to add only a limited number of CoTs. Therefore, it is important to explore and develop a CoT selection strategy in future research. A good CoT selection strategy would enable the identification of highly effective CoTs from a set of CoTs, enhancing the efficiency of KADL.

heading: Acknowledgments
text: This work is supported by National Natural Science Foundation of China 62076245; CCF-Zhipu AI Large Model Fund.

heading: B More results about the Effect of CoT Size in CoT-KA
text: We vary the number of sampled CoTs (1, 5) in CoTKA and evaluate its performance on ten tasks, excluding StrategyQA. Figures from 5 to 14 indicate that in most of these tasks, increasing the number of CoTs from 0 to 1 significantly improves task performance. However, when using DeBERTa-v3large as the PLM, the performance gain in CoT-KA for CSQA, Date Understanding, and Sports Understanding is slight and even leads to a degradation. Furthermore, increasing the number of CoTs from 1 to 5 has a relatively small performance gain in CoTKA (DeBERTa), except for improved Date Understanding and continued degradation in CSQA.
We observe that if the baseline, where the dataset is not augmented by a CoT, starts with a lower performance, the performance gain in CoT-KA becomes more significant as the number of CoTs increases.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 5 Experiments, Section 7 Limitations.
7 A2. Did you discuss any potential risks of your work? Our paper mainly used GPT3 and chain-of-thought prompting for application, the risk of large language models and chain-of-thought prompting were discussed in references we cited.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract, Section 1 introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 Pilot Study, Section 5 Experiments.
3 B1. Did you cite the creators of artifacts you used? Section 1 introduction, Section 3 Pilot Study, Section 5 Experiments.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We used the same publicly available dataset as in the existing work, and we did not discuss this matter specifically.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We used the same publicly available dataset as in the existing work, and we did not discuss this matter specifically.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We used the same publicly available dataset as in the existing work, and we did not discuss this matter specifically.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We used the same publicly available dataset as in the existing work, and we did not discuss this matter specifically.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix: A.2 Datasets.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 3 Pilot Study, Section 5 Experiments.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix: A.1 Implementation.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5 Experiments, Appendix: A.1 Implementation.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5 Experiments, Appendix B.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 1 Introduction, Section 5 Experiments, Appendix: A.1 Implementation.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Therefore, it is important to explore and develop a CoT selection strategy in future research. A good CoT selection strategy would enable the identification of highly effective CoTs from a set of CoTs, enhancing the efficiency of KADL