The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning
Authors: Seungone Kim
Se June Joo
Doyoung Kim
Joel Jang
Seonghyeon Ye
Jamin Shin
Minjoon Seo
Venue: EMNLP 2023 Main (2023)
Keywords: Chain-of-Thought Fine-tuning
Zero-shot Generalization
Few-shot Learning
Large Language Models
TL;DR: We show that CoT Fine-tuning on 1,060 tasks improves smaller language models in terms of zero-shot generalization and few-shot adaptation.

Abstract: Language models (LMs) with less than 100B parameters are known to perform poorly on chain-of-thought (CoT) reasoning in contrast to large LMs when solving unseen tasks. In this work, we aim to equip smaller LMs with the step-by-step reasoning capability by instruction tuning with CoT rationales. In order to achieve this goal, we first introduce a new instruction-tuning dataset called the CoT Collection, which augments the existing Flan Collection (including only 9 CoT tasks) with additional 1.84 million rationales across 1,060 tasks. We show that CoT fine-tuning Flan-T5 (3B \& 11B) with CoT Collection enables smaller LMs to have better CoT capabilities on unseen tasks. On the BIG-Bench-Hard (BBH) benchmark, we report an average improvement of +4.34% (Flan-T5 3B) and +2.60% (Flan-T5 11B), in terms of zero-shot task accuracy. Furthermore, we show that instruction tuning with CoT Collection allows LMs to possess stronger few-shot learning capabilities on 4 domain-specific tasks, resulting in an improvement of +2.24% (Flan-T5 3B) and +2.37% (Flan-T5 11B), even outperforming ChatGPT utilizing demonstrations until the max length by a +13.98% margin. Our code, the CoT Collection data, and model checkpoints are publicly available.

The CoT COLLECTION: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Though Fine-Tuning
Seungone Kiml127 Se June Joo
Doyoung Kim
Joel Jang!
Seonghyeon Ye'
Jamin Shin 29
Minjoon Seo"
KAIST AI
NAVER AI Lab²
University of Washington
(seungone, sejune, minjoon] @weeek..a....................... jayshin. wwppgmaippmmmmmmmmmmmmmmmmmmm..m
Abstract
has been particularly effective in enabling LMs excel at multitude tasks is Chain-of-Thought (CoT) prompting, making LMs generate a rationale to derive its final prediction in a sequential man- ner (Wei et al., 2022b; Kojima et al., 2022: Zhou et al., 2022; Zhang et al., 2022; Yao et al., 2023).
Language models (LMs) with less than 100B parameters are known perform poorly on chain-of-thouu (CoTT) reasoning in contrast large LMs when solving unseen tasks. In this work, we aim to equip smaller LMs with the step-by-step reasoning capability by instruction tuning with CoT rationales. In order achieve this goal, we first introduce new instruction- tuning dataset called the COT COLLECTION, which augments the existing Flan Collection (including only 9 CoT tasks) with additional 1.84 million rationales across 1,060 tasks. We show that CoT fine-tuning Flan-T5 (3B& & IB) with CoT COLLECTION enables smaller LMs have better CoT capabilities unseen tasks. On the BIG-Bench-HHe Hard (BBH) benchmark, we report an average improvement of +4.34% (Flan-T5 -5 3B) and +2.60% (Flan-T5 11B), in terms of zero-shot task accuracy. Furthermore, we show that instruction tuning with CoT COL- LECTION allows LMs to possess stronger few- shot learning capabilities on tasks, resulting an improvement of +2.24% (Flan-T5 T5 3B) and +2.37% F5 11B), even outperforming ChaiGPT utilizing demonstra- tions until the max length by +13.985 mar- gin. Our code, the COT COLLECTION data, and model checkpoints are publicly available
While CoT prompting works effectively for large LMs with more than 100 billion parameters, it does not necessarily confer the same benefits to smaller LMs (Tay et al., 2022; Suzgun et al., 2022; Wei et al., 2022a; Chung al., 2022). The require- ment of a large number of parameters consequently results in significant computational cost and acces- sibility issues (Kaplan et al., 2020; Min et al., 2022; Liu et al., 2022b; Mhlanga, 2023; et al., 2023).
Recent work has focused on empowering rela- tively smaller LMs effectively solve novel tasks as well, primarily through fine-tuning with ratio- nales (denoted as CoT fine-tuning) and applying CoT prompting on a single target task (Shridhar et al., 2022; Ho et al., 2022; Fu et al., 2023). How- ever, solving a single task does not adequately ad- dress the issue of generalization to broad range of unseen tasks. While Chung et al. (2022) leverage 9 publicly available CoT tasks during instruction tun- ing to solve multiple unseen tasks, the imbalanced ratio compared to 1,827 tasks used for direct fine- tuning results in poor CoT results across smaller LMs (Longpre et al., 2023). In general, the com- munity still lacks comprehensive strategy to fully leverage CoT prompting to solve multiple unseen novel tasks in the context of smaller LMs.
1 Introduction
Language models (LMs) pre-trained on massive text corpora can adapt to downstream tasks both zero-shot and few-shot learning settings by incorpo- rating task instructions and demonstrations (Brown et al., 2020; Wei et al., 2021; Sanh et al., 2021; Mishra al., 2022; Wang al., 2022b; Iyer et al., 2022; Liu et al., 2022b; Chung et al., 2022; Long- pre et al., 2023; Ye al., 2023). One approach that
To bridge this gap, we present the CoT COL- LECTION, an instruction tuming dataset that aug- ments 1.84 million rationales from the FLAN Col- lection (Longpre al., 2023) across 1,060 tasks. We fine-tune Flan-T5 (3B & 11B) using CoT COL- LECTION and denote the resulting model as CoT- T5. We perform extensive comparisons of CoT-T5 and Flan-T5 T5 under two main scenarios: (1) zero- shot learning and (2) few-shot learning.
"denotes equal contribution. Work was done while Seun- gone interning NAVER Al Lab.
Correepooniig authors
btttps://gl thub. com/ k StAT/
Mor-Collectinn

---

In the zero-shot learning setting, COT-T5 (3B & 11B) outperforms Flan-T5 (3B & 11B) by +4.34% and +2.60% on average accuracy across 27 datasets from the Big Bench Hard (BBH) benchmark (Suz- gun et al., 2022) when evaluated with CoT prompt- ing. During ablation experiments, we show that COT fine-tuning TO (3B) (Sanh et al., 2021) on subset of the CoT Collection, specifically 163 train- ing tasks used in TO, shows performance increase of +8.6%% on average accuracy across 11 datasets from the P3 Evaluation benchmark. Moreover, we translate 80K instances of CoT COLLECTION into 5 different languages (French, Japanese, Korean, Russian, Chinese) and observe that CoT fine-tuning mTO (3B) (Muennighoff et al., 2022) on each lan- guage results in 2x ~ 10x performance improve- ment average accuracy across all 5 languages from the MGSM benchmark (Shi al., 2022).
2 Related Works
2.1 Chain-of-Thought (CoT) Prompting
Wei al. (2022b) propose Chain of Thought (CoT) Prompting, technique that triggers the model generate rationale before the answer. By gener- ating a rationale, large LMs show improved rea- soning abilities when solving challenging tasks. Kojima et al. (2022) show that by appending the phrase 'Let's think step by step', large LMs could perform CoT prompting in a zero-shot setting. Dif- ferent work propose variants of CoT prompting such as automatically composing CoT demonstra- tions (Zhang al., 2022) and performing fine- grained search through multiple rationale candi- dates with a tree search algorithm (Yao et al., 2023). While large LMs could solve novel tasks with CoT Prompting, Chung et al. (2022) and Longpre et al. (2023) show that this effectiveness does not neces- sarily hold for smaller LMs. In this work, we aim to equip smaller LMs with the same capabilities by instruction tuning on large amount of rationales.
In the few-shot learning setting, where LMs must adapt to new tasks with a minimal number of in- stances, CoT-T5 (3B& & 11B) exhibits a +2.24% and +2.37% improvement on average compared to using F1an-T5 (3B & 11B) as the base model on 4 different Nomainn-speciff tasks². Moreover, it demonstrates +13.98% and +8.11 improvement over ChatGPT (OpenAI, 2022) and Claude (An- thropic, 2023) that leverages ICL with demonstra- tions up to the maximum input length.
2.2 Improving Zero-shot Generalization
Previous work show that instruction tuning enables generalization to multiple unseen tasks (Wei et al., 2021; Sanh et al., 2021; Aribandi et al., 2021; Ouyang et al., 2022; Wang et al., 2022b; Xu et al., 2022). Different work propose improve instruc- tion tuning by enabling cooss-lingual generaliza- tion (Muennighoff et al., 2022), improving label generalization capability (Ye al., 2022), and train- ing modular, expert LMs (Jang et al., 2023). Mean- while, a line of work shows that CoT fine-tuning could improve the reasoning abilities of LMs on a single-seen task (Zelikman et al., 2022: Shridhar et al., 2022; Ho et al., 2022; Fu et al., 2023). As a follow-up up study, we CoT fine-tune 1,060 instruc- tion tasks and observe significant improvement in terms of zero-shot generalization on multiple tasks,
Our contributions are summarized as follows:
We introduce COT COLLECTION, a new in- struction dataset that includes 1.84 million ra- tionales across 1,060 tasks that could be used for applying CoT fine-tuning to LMs.
With CoT COLLECTION, we fine-tune Flan- T5, denoted as CoT-T5, which shows non- trivial boost in zero-shot and few-shot learn- ing capabilities with CoT Prompting.
For ablations, we show that CoT fine-tuning could improve the CoT capabilities of LMs in low-compute settings by using a subset (1)
2.3 Improving Few-Shot Learning
of CoT COLLECTION and training smaller number of tasks (TO setting; 163 tasks) and (2) smaller amount of instances in 5 dif- ferent languages (French, Japanese, Korean, Russian, Chinese; 80K instances).
For adapting LMs to new tasks with a few instances, recent work propose advanced parameter efficient fine-tuning (PEFT) methods, where small num- ber of trainable parameters are added (Hu et al., 2021; Lester et al., 2021; Liu et al., 2021, 2022b; Asai et al., 2022: Liu al., 2022c). In this work, we show that a simple recipe of (1) applying LoRA (Hu et al., 2021) a LM capable of per- forming CoT reasoning and (2) CoT fine-tuning on a target task results in
We assess the efficacy of our approach on 4 domain- specific datasets, two each from legal and medical fields. namely including LEDGAR (Tuggener et al., 2020). Case Hold (Zheng et al., 2021), McdNLI (Romanov and Shivade, 2018), and PuhMedQA (Jin et al., 2019). Fach dataset is represented hy 64 randomly chosen instances.
strong few-shot performance.

---

FLAN-1S
SNI Formal Logic
Artthmetic
Previoualy available Cal rationale 3BK instancees)
FLAN-TS MCQA
SNI CBQA
FLAN EXQA
SNI Commonsense
SNI Arithmetii
SNI Program Exec
FLAN-TS MCQA
Dialogue
FLAN NU
SNI Toxic
FLAN Misc
Scale
SNI MCQA
SNI Science
TO EXQA
TO MCQA
FLAN CBQA
SNI Ethics
200k
SNI Dialogue
EXQA
CBQA
20k
CoT Collection
(1060 Tasks, 1.84M instances)
Figure I: An illustration of the overall task group and dataset source of where obtained the instances augment the rationales in COT COLLECTION, Compared to the 9 datasets that provide publicly available rationales (included within Flan-T5 ExQA', 'Flan-T5 Arithmetic'. 'Flan-T5 MCQA', 'Flan-TS NLI' from the red box), we generate 51.22 times rationales (1.84 million rationales) and ~ 11.778 times more task variants (1,060 tasks).
3 The CoT COLLECTION
et al., 2022a; Taori et al., 2023; Chiang et al., 2023) while extend it to generating rationales r.
Despite its effectiveness for CoT fine-tuning, ratio- nale data still remains scarce. To the best our knowledge, recent work mostly rely on 9 publicly available NLP datasets for fine-tuning with ratio- nales (Zelikman et al., 2022; Shridhar et al., 2022; Chung et al., 2022: Ho et al., 2022; Longpre et al.. 2023; Fu et al., 2023). This is due the difficulty in gathering human-authored rationales (Kim et al., 2023). To this end, we create CoT COLLECTION, an instruction-tuniin dataset that includes 1.84 mil- lion rationales augmented across 1,060 tasks". In this section, we explain the datasets we select to augment into rationales and how we perform the overall augmentation process.
Source Dataset Selection As a source dataset to extract rationales, we choose the Flan Collec- tion (Longpre et al., 2023), consisting of 1,836 di- verse NLP tasks from P3 (Sanh et al., 2021), Super- NaturalInstructions (Wang et al., 2022b), Flan (Wei et al., 2021), and some additional dialogue & code datasets. We choose 1,060 tasks, narrowing our focus following the criteria as follows:
Generation tasks with long outputs are ex- cluded since the total token length of append- ing r and exceeds the maximum output to- ken length (512 tokens) during training.
Broad Overview Given an input
[1,
composed an instruction I, and an instance along with the answer we obtain a rationale by applying in-context learning (ICL) with a large LM. Note that this differs from previous works which focused on generating new instances using large LMs (West et al., 2022; Liu et al., 2022a; Kim al., 2022: Honovich al., 2022; Wang
Datasets that are not publicly available such DeepMind Coding Contents and Dr Re- pair (Yasunaga and Liang, 2020) are excluded.
Datasets where the input and output do not correspond each other in the huggingface datasets (Lhoest al., 2021) are excluded.
'The available datasets are QASC (Khot al., 2020).
When a dataset appears in common across different sources, we prioritize using the task from P3 first, followed by SNI, and Flan.
AQuA (Amini al., 2019), GSM8K (Cobbe al., 2021).
QED (Lamm et al., 2021), StrategyQA (Geva al., 2021).
SenseMaking (Wang et al., 2019), CREAK (Onoe al., 2021).
e-SNLI (Camburu et al., 2018), ECQA (Aggarwal et al., 2021).
During preliminary experiments, we find that for tasks such as sentiment analysis, sentence
'Following Sanh ct al. (2021), ), we use the notion of 'task"
referring to cach prompt applicd to dataset.

---

completion, coreference resolution, and word disambiguation, rationales generated by large LMs are very short and uninformative. We exclude these tasks to prevent negative trans- fer during multitask learning (Aribandi et al., 2021; Jang et al., 2023).
[Example 1]
[Instruction and Question]
Skylar had stopped breathing but Lee [..]] Given the context: Lee want to do what next? [Answer] beg [Rationala]
try again
The context of the situation is that Skylar has stopped breathing and Lee holding [..] The answer to beg the doctors try again.
Creating Demonstrations for ICL We first create prompts to upply in-context learning (ICL) with large LMs for augmenting the instances in the selected tasks with rationales. Preparing demonstrations 'D' for each task the most straightforward, but becomes infeasible to prepare demonstrations for each task as the number of tasks gets larger. Instead, we assign each task t to Tk. family of tasks that shares a similar task format such as multiple choice QA, closed book QA, and dialogue generation. Each family of tasks share D which consists of 6 demonstrations, These 6 demonstrations for each task group T& manually created by of the authors in this paper. Specifically, given 136 instances sampled from Flan Collection, two annotators are assigned to write rationale, and the other third annotator conducts A/B testing between the two options. We manually create DT, across k 26 task groups. We include the prompts for all of the different task groups in Appendix D.
[Example ple 2]
[Instruction and Question] you think the right "what can run alcoholic fermentation of [-.]]"
question
[Answer]
[Rationale]
Alcoholic fermentation process that [..] will produce enough energy to power. Therefore, the answer is No.
[Example 9]
[Instruction and Question] am testing my students' logic. [..]]
through the grass because (A) more friction, [Answer]
A [Rationale]
Figure 2: MCQA Prompt used to augment rationales from P3 dataset. Through ICL, the large LM generates rationale that is conditioned on the ground-truth label.
the filtered cases Appendix B. Also, include analysis of the diversity and quality of CoT COL- LECTION compared to the existing 9 CoT tasks and -uman--uthored rationales in Appendix A.
4 Experiments
Rationale Augmentation We use the OpenAl Codex to augment rationales. Formally, given (X/, vf), the 7t instance of task the goal is generate corresponding rationale r/. Note that during preliminary experiments, we found that or- dering the label in front of the rationale within the demonstration DT, was crucial to generate good quality rationales. We conjecture this because ordering the label in front of the rationale loosens the need for the large LM to solve the underly- ing task and only focus generating rationale. However, we also found that in some tasks such arithmetic reasoning, large LMs fail to generate good-quality rationales. To mitigate this issue, we apply filtering to the augmented rationales. We provide the criteria used for the filtering phase and
For our main experiments, we use Flan-T5 (Chung et al., 2022) our base model, and obtain CoT- T5 by CoT fine-tuning on the CoT COLLECTION. Formally, given X/. the goal of CoT fine-tuning is to sequentially generate the rationale rf and answer of To indicate that rf should be generated before vf. the trigger phrase 'Let's think step by step' is added during both training and evaluation. We mostly follow the details for training and evaluation from Chung et al. (2022), and provide additional details in Appendix C. In this section, we show how training CoT COLLECTION enhances zero- shot generalization capabilities (Section 4.2) and few-shot adaptation capabilities (Section 4.3).
4.1 Evaluation
'The use of Codex was largely due to limited academic budget (OpenAl supported Codex with no cost for researchers to June 2023). Moreover, other LLM services such as Bard (Google, 2023) and Claude (Anthropic, 2023) were not supported during the period of CoT COLLECTION augmen- tation. To address the concern of reproducibility. analysis on quality of rationales from Codex, Bard and Claude is included in Appendix A,
We evaluate under two different evaluation meth- ods: Direct Evaluation and CoT Evaluation. For Direct Evaluation on classification tasks, fol- low previous works using verbalizers, choosing the option with the highest probability through com- parison of logit values (Schick and Schütze, 2021;

---

Sanh al., 2021; Ye et al., 2022; Jang et al., 2023), and measure the accuracy. For generation lasks, we directly compare the LM's prediction with the answer and measure the EM score.
Method
CoT
Direct
Total Avg
T5-LM-3B
26.68
26.96
26.82
TO-3B
26.64
27.45
27.05
TK- INSTRUCT-3B
29.86
29.90
29.88
When evaluating with CoT Evaluation, smaller LMs including Flan-T5 often do not generate any rationales even with the trigger phrase 'Let's think adopt hard constraint
Tk INSTRUCT-
33.60
30.71
32.16
T0-11B
31.83
33.57
32.70
FLAN-T5-3B
34.06
37.14
35.60
step by step'. Therefore, of requiring the LM generate r with at least a minimum length of tokens. In classification tasks, we divide into two steps where the LM first generates rf and then verbalizers are applied with a indicator phrase [ANSWER]" inserted between if and the possible options. For generation tasks, we extract the output coming after the indicator phrase. Accuracy metric is used for classification tasks while EM metric is used for generation tasks.
GPT-3 (175B)
38.30
33.60
38.30
FLAN-T5-118
38.57
40.99
39.78
T5-3B CoT FT
37.95
35.52
36.74
CoT-TS-3B
38.40
36.18
37.29
T5-11B CoT FT COT-TS-11B
40.02
38.76
39.54
42.20
42.56
42.38
Table 1: Evaluation performance on the unseen datasets from BBH benchmark, including generation tasks. eval- uations are held in zero-shot setting. The best comparable performances are bolded and second best underlined.
4.2 Zero-shot Generalization
noted as "T5 + CoT FT'). Note that FLAN Col- lection includes 15 million instances, hence ~ times larger compared
In this subsection, we show how training with CoT COLLECTION could effectively improve the LM's ability to solve unseen tasks. We have three differ- ence experimental set-ups, testing different aspects: Setup #1: training on the entire 1060 tasks in CoT COLLECTION and evaluating the reasoning capabil- ities of LMs with the Bigbench Hard (BBH) bench- mark (Suzgun et al., 2022), Setup #2: training only 163 tasks that TO (Sanh et al., 2021) used for training (a subset of the COT COLLECTION), and evaluating the linguistic capabilities of LMs with the P3 evaluation benchmark (Sanh et al., 2021), and Setup #3: training with a translated, subset ver- sion of COT COLLECTION for each five different languages and evaluating how LMs could perform CoT reasoning in multilingual settings using the MGSM benchmark (Shi et al., 2022).
COT COLLECTION. The results on BBH benchmark are shown across Table I and Table 2. In Table 1, CoT-T5 (3B & 11B) achieves +4.34% and +2.60% improvement over Flan-T5 (3B & 11B) with CoT Evaluation. Surprisingly, while CoT-T5-3B CoT performance improves +4.34% with the cost of 0.96% degra- dation in Direct Evalution, CoT-T5-11B3 Direct Evaluation performance even improves, resulting in a +2.57% total average improvement. Since COT COLLECTION only includes instances aug- mented with rationales, these results show that CoT fine-tuning could improve the LM's capabilities re- gardless of the evaluation method. Also, T5-3B + CoT FT and 15-11B COT FT outperforms FLAN-T5-3B and FLAN-T5-11B by +1.45% and +3.89% margin, respectively, when evaluated with CoT evaluation. Moreover, T5-3B + CoT FINE-TUNING outperforms <44 times larger mod- els such as TO-11B and Tk-Instruct- IB in both Direct and CoT Evaluation. The overall results in- dicate that (1) CoT fine-tuning of tasks enables smaller LMs outperform larger LMs and (2) training with FLAN Collection and CoT Collection provides complementary improve- ments to LMs under different evaluation methods; CoT-T5 obtains good results across both evaluation methods by training on both datasets.
Setup #1: CoT Fine-tuning with 1060 CoT Tasks We first perform experiments with main model, CoT-T5, by training Flan-T5 on the entire CoT COLLECTION and evaluate on the BBH bench- mark (Suzgun et al., 2022). In addition to eval- uating Flan-T5, we compare the performances of different baselines such as (1) T5-LM (Raffel et al., 2020): the original base model of Fla I): an cnstruction-tuned LM trained with P3 instruction dataset, (3) Tk- Instruct (Wang et al., 2022b): instruction-tuned LM trained with SNI instruction dataset, and (4) GPT-3 (Brown et al., 2020): pre-trained LLM with 175B parameters. For ablation purposes, we also train T5-LM with COT COLLECTION (de-
diverse number
(2) TO (Sanh et al., 2021):
In Table 2. CoT-T5-1IBB obtains same or better results on 15 out of 23 tasks when evaluated with Direct evaluation, and 17 out of 23 tasks when evaluated with CoT Evaluation compared to Flan-

---

COT-TS-IIB
FLAN-T5-HIB
VICUNA-13B
CHATGPT
CODEX
GPT-4
Task
CoT
Direct
CoT
Direct
Direct
Direct
Direct
Direct
BOOLEAN EXPRESSIONS
65.6
59.2
51.6
56.8
40.8
82.8
88.4
77.6
CAUSAL JUDGMENT
60.4
60.2
58.3
61.0
42.2
57.2
63.6
59.9
52.0
DISAMBICUATION QA
63.4
68.2
63.2
67.2
18.4
57.2
67.2
69.2
FORMAL FALLACIES
51.2
55.2
54.4
55.2
47.2
53.6
52.4
64.4
GEOMETRIC SHAPES
22.0
10.4
12.4
21.2
3.6
25.6
32.0
40.8
HYPERBATON
65.2
64.2
55.2
70.8
44.0
69.2
60.4
LOGICAL
(5)
48.2
54,4
51.2
53.6
4.8
38.8
32.4
66.8
LOGICAL DEDUCTION (7)
52.4
60.6
57.6
60.0
1.2
39.6
26.0
66.0
LOGICAL DEDUCTION (3)
35.4
75.0
66.4
74.4
16.8
60.4
52.8
94.0
MOVIE RECOMMENDATION
44.6
52.8
32.4
36.4
43.4
55.4
84.8
79.5
PENGUINS TABLE
39.1
41.8
41.8
41.8
45,9
66.4
76.7
REASONING COLORED OB),
32.6
33.2
22.8
23.2
12.0
47.6
67.6
84,8
RUIN NAMES
42,8
41.6
31.6
34.4
15.7
56.0
75.2
89.1
SALIENT TRANS ERR.
43.8
49.2
35.6
49.2
2.0
40.8
62.0
62.4
67.7
59.0
SPORTS UNDERSTANDING
64.8
66.4
56.0
60.0
48.4
79.6
72.8
84.4
TEMPORAL SEQUENCES
27.4
28.8
24.4
28.8
16.0
35.6
77.6
98.0
TRACKING SHUPP OBI. (5)
20.0
13.2
19.6
15.2
9.2
18.4
20.4
25.2
TRACKING SHUFF OBI. (7)
18.4
9.6
13.2
12.0
5.6
15.2
14.4
25.2
TRACKING (3)
41.8
31,2
28.8
316
WEB LIES
57.0
51.6
52.8
50.0
41.2
56.0
51.6
49.6
AVERAGE
47.60
48.00
43.32
47.05
23.30
48.90
52.80
67.40
Table 2: Evaluation performance on 23 unseen classification datasets from BBII benchmark. Scores of Vicuna, ChatGPT, Codex (teacher model of CoT-T5), GPT-4 obtained from Chung al. (2022) and Mukherjee al. (2023). Evaluations held in zero-shot setting. The best comparable performances bolded and second best underlined among the open- LMs.
Natural Language
Completion
Cureference
Method
Total Avg
RTE
CB
AN.
AN.
AN. R3
COPA
Hellusw.
Story C.
Winogr:
WSC
WIC
TS-38 (Raffel al,, 2020)
53.03
34.55 34
52.89
33.76
33.82
54.88
27.00
48 16
50.64
54.09
50.30
42.99
TO.3B (SANH
15.13
51.27
RoE-3B (JANG ET AL., 2023) 2023)
64.01
43.57
35.49
34.64
31.22
79.25
34.60
86.33
61.60
62.21
52.97
53.48
KIC-770M
2022)
74.00
67.90
35.00
37.60
85.30
94.40
55.30
65.40
$7.56
FLIPPED-3D (YE AL., 2022)
71.05
57.74
39.99
37.05
37.73
89.88
41.64
95.88
58.56
58.37
50.42
58.03
GPT-3 (175B) (BRoWN
2020)
63.50
46.40
34.60
35.40
34.50
91.00
78.90
83.20
70.20
65:40
43.92
59.00
(SANH Ict Al., 2021)
20:12
56.58
TS-3D COT FT EVAL w/ DIRECT
69.96
58.69
37.58
36.00
37.44
$4.59
40.92
90:47
55.40
64.33
51.53
$6.99
T0-38
88.27
T5.30 CoT FT EVAL w/ COT
80.61
69.17
40.24
36.67
40.13
90.10
41.08
93.00
56.47
$5.10
56.73
59.94
TU-3B COT EVAL w/ COT
80.25
72.62
41.71
37.22
41.89
90.88
39.50
94.47
57.47
50.58
54.27
60.08
Table 3: Evaluation performance on 11 different unseen P3 dataset (Sanh et al., 2021) categorized into 4 task categories. We report the direct performance of the baselines since they were not CoT fine-tuned on instruction data. The best comparable performances are bolded and second best underlined. We exclude Flan-T5 and CoT-T5 since they were trained on the unseon tasks (tasks from FLAN and SNI overlap with the P3 Eval datasets), breaking unseen task assumption.
TB-11B. Interestingly, Vicuna (Chiang et al., 2023), LM trained on forg-form dialogues between users and GPT models, perform much worse compared both CoT-T5 and Flan-T5. We conjecture that training on instruction datasets from existing aca- demic benchmarks consisting CoT Collection and Flan Collection more effective in enabling LMs to solve reasoning tasks compared to chat LMs.
instances from 163 tasks, and apply CoT fine- tuning TO (3B) (Sanh et al., 2021) and T5-LM (3B) (Raffel et al., 2020). Note that TO is trained with 12M instances, hence 18.63 times larger. Then, we evaluate on the P3 evaluation benchmark which consists of different NLP datasets. addition to the baselines from the previous section (T5-LM, TO, and GPT-3), we also include LMs that are trained on the same TO setup for compari- son such as, (1) RoE (Jang et al., 2023); modu- lar expert LM that retrieves different expert mod- els depending the unseen task, (2) KiC (Pan et al., 2022): aarieval-augment model that
Setup #2: CoT Fine-tuning with 163 CoT Tasks (TO Setup) To examine whether the effect of CoT fine-tuning is dependent on large number of tasks and instances, we use the P3 training subset from the COT COLLECTION consisted 644K

---

is instruction-tuned retrieve knowledge from KB memory, and (3) Flipped (Ye al., 2022): an instruction-tuned model that is trained to generate the instruction in order to resolve the LM over- fitring to the output label as baseline models.
Method
ko
r
fr
zh
ja
FLAN-TS-3B
0.0
2.8
7.2
0.0
0.0
FLAN-TS-LIB
0.0
5.2
13.2
0.0
0.0
MT5-3.7B
0.0
1.2
2.0
0.8
0.8
MT0-3.7B
4.8
7.2
GPT-3 (175B)
0.0
4.4
10.8
6.8
0.8
The results are shown in Table 3. Surprisingly, T5-3B + COT FT outperforms TO-3B by a + margin when evaluated with CoT Evaluation, while using 18.63 times less instances. This supports that CoT fine-tuning data efficient, being effec- tive even with less number of instances and tasks. Moreover, TO-3B + COT FT improves TO-3B by +8.65% on average accuracy. When compared with TO-11B with 44 times more number of parameters, it achieves better performance at sentence comple- tion, and word sense disambiguation (WSD) tasks, and obtains similar performances natural lan- guage inference and coreference resolution tasks.
MT5-3.7B CoT FT
3.2
6.8
9.6
6.0
7.6
MT0-3.7B CoT FT
7.6
10.4
15.6
11.2
11.0
Table 4: Evaluation performance on MGSM benchmark (Shi al., 2022) across 5 languages (Korean, Russian. French, Chinese, Japanese, respectively). zero-shot setting with CoT Evaluation except GPT-3 using a 6-Shot prompt for ICL. The best comparable performances are bolded and second best underlined. Note that 'MT5-3.7B CoT FT' and 'MTO03378 CoT FT' are trained on a single language instead of multiple languages as mT5 and mTO.
evaluations are held in
about non-trivial gains in performance. Even for relatively low-resource languages such as Korean Japanese, and Chinese, CoT fine-tuning on the spe- cific language allows the underlying LM to per- form mathematical reasoning in the target language, which considered very difficult (Shi et al., 2022). Considering that only a very small num- ber of instances were used for language-specinng adaptation (60k-80k), CoT fine-tuning shows po- tential for efficient language adaptation.
Setup #3: Multilingual Adaptation with CoT Fine-tuning In previous work, Shi al. (2022) proposed MGSM, a multilingual reasoning bench- mark composed of 10 different languages. In this subsection, we conduct toy experiment to exam- ine whether CoT fine-tuning could enable LMs to reason step-by step in multilingual settings well, using subset of languages (Korean, Russian, French, Chineses, Japanese) from MGSM.
However, it is noteworthy that we limited our setting to training/cvaluating on a single target lan- guage, without exploring the cross-lingual transfer of CoT capabilities among varied languages. The chief objective of this experimentation was to ascer- tain introducing minimal volume of CoT data could facilitate effective adaptation the target language, specifically when addressing reasoning challenges. Up to date, no hypothesis has sug- gested that training with CoT in various languages could enable cross-lingual transfer of CoT abilities among different languages. We identify this as promising avenue for future exploration.
In Table 4, current smaller LMs can be divided into three categories: (1) Flan-T5, a LM that is CoT fine-tuned with mostly English instruction data, (2) MT5 (Xue et al., 2021). LM pretrained on diverse languages, but isn't 't't instruction tuned or CoT fine- tuned, (3) MTO (Muennighoff al., 2022), LM that is instruction-tunee on diverse languages, but isn't CoT fine-tuned. In relatively underrepresented languages such as Korean, Japanese, and Chinese, all three LMs get close to zero accuracy.
A natural question arises whether training mul- tilingual LM that could reason step-by-step dif- ferent languages is viable. As preliminary re- search, we examine whether CoT Fine-tuning on single language with a small amount of CoT data could enable LMs to avoid achieving near zero score such as Korean, Chinese and Japanese sub- sets of MGSM. Since there is no publicly available multilingual instruction dataset, we translate 60K 80K instances from COT COLLECTION for each 5 languages using ChatGPT (OpenAI, 2022), and CoT fine-tune mT5 and m'TO on each of them.
4.3 Few-shot Generalization
In this subsection, we show how CoT-T5 performs in a few-shot adaptation setting where limited number of instances from the target task can be used for training. which sometimes more likely in real-world scenarios.
We choose 4 domain-specific datasets from legal and medical domains includ- ing LEDGAR (Tuggener et al., 2020), Case Hold (Zheng al., 2021), MedNLI (Romanov and Shivade, 2018), and PubMedQA (Jin et al., 2019). To simulate a seo-shot setting, we randomly sam- ple 64 instances from the train split of each dataset.
Dataset Setup
The results are shown in Table 4. Across all the 5 different languages, CoT fine-tuning brings

---

Method
#Train Param
Ledgar
Case Hold
McdNLI
PubmedQA
Total Avg
Flan-T5-3B FT
2.8B
52.60
61.40
66.82
66.28
61.78
Flan-T5-3B Full CoT FT.
2.8B
53,60
58.80
65.89
61.05
CoT-T5-3B + Full CoT FT. (Ours)
2.8B
51.90
60.60
67.16
68.12
61.95
Flan-T5-3B LoRA FT.
2.35M
53.20
58,80
61.60
67.18
60.19
Flan-T5-3B LoRA CoT
2.35M
51.20
61.60
62.59
66.06
60,36
CoT-T5-3B LoRA CoT FT. (Ours)
2.35M
54.80
63.60
68.00
69.66
64.02
Flan-T5-11B3 11-11BB44 LoRA FT.
4.72M
55.30
64,90
75.91
70.25
66.59
Flan-T5-11B LoRA COT FT.
4.72M
52.10
65.50
71.63
71.60
65.21
CoT-T5-11B- LoRA CoT FT. (Ours)
4.72M
56.10
68.30
78.02
73.42
68.96
Claude (Anthropic, 2023) ICL
0
55.70
$7.20
75 .4
54.58
60.85
Claude (Anthropic, 2023) CoT PT.
0
34.80
43.60
76.51
52.06
51.74
ChatGPT (OpenAI, 2022) + ICL
0
51.70
32.10
70,53
65.59
54.98
ChatGPT (OpenAl, 2022) + CoT PT.
0 51.00
18.90
63.71
25.22
39.70
Table 5: Evaluation performance on Nomain-speccff datasets. FT. denotes Fine-tuning, CoT FT. denotes CoT fine-tuning, and CoT PT. denotes CoT Prompting. The best comparable performances are bolded and second best underlined. For few-shot adaptation, we use 64 randomly sampled instances from each dataset.
We report the average accuracy across runs with different random seeds. We augment rationales for the 64 training instances using the procedure described in Scction for the rationale augmenta- tion phase, utilizing the MCQA prompt from P3 dataset. In an applied setting, practitioners could obtain rationales written by human experts.
few parameters enables CoT-T5 to maintain the CoT ability acquired during CoT fine-tuning.
Fine-tuning vs. CoT Fine-tuning While CoT fine-tuning obtains similar or lower performance compared to regular fine-tuning in Flan-TS, CoT- T5 achieves higher performance with CoT fine- tuning compared Flan-T5 regular fine-tuning. This results CoT-T5 in combination with CoT fine-tuning showing the best performance in few- shot adaptation setting.
Training Setup We compare Flan-T5 & CoT-T5, across 3B and 1 1B scale and explore 4 different ap-
proaches for few-shot adaptation: (1) regular fine- tuning, (2) CoT fine-tuning, (3) LoRA fine-tuning, and (4) LoRA CoT fine-tuning. When applying Lora, we use a rank of 4 and train for 1K steps fol- lowing Liu et al. (2022b). This results in training 2.35M parameters for 3B scale models and 4.72M parameters for 1 I1B scale models. Also, we include Claude (Anthropic, 2023) and ChatGPT (OpenAl, 2022) as ICL baselines by appending demonstra- tions up to maximum context length Specifically, For CoT prompting, the demonstrations are sam- pled among 64 augmented rationales are used.
Fine-tuning ICL Lastly, fine-tuning meth- ods obtain overall better results compared ICL methods utilizing much larger, proprictary LLMs. We conjecture this to be the case due the long input length of legal and medical datasets, making appending all available demonstrations (64) impos- sible. While increasing the context length could serve as temporary solution, it would still mean that the inference time will increase quadratically in proportion to the input length, which makes ICL computationally expensive.
Effect of LoRA The experimental results are shown in Table 5. Overall, CoT fine-tuning CoT- T5 integrated with LoRA obtains the best results overall. Surprisingly for Flan-T5, applying full fine-tuning obtains better performance compared its counterpart using LoRA fine-tuning. How- ever, when using CoT-T5, LoRA achieves higher performance compared to full fine-tuning. We con- jecture this to be the case because introducing only
Analysis of of CoT Fine-tuning
In this section, conduct experiments to address the following research questions:
For practitioners, is it more effective aug- ment CoT rationales across diverse tasks more instances with fixed number of tasks?
During CoT fine-tuning, does the LM main- tain its performance on in-domain tasks with- out any catastrophic forgetting?
'Full context length was 4k tokens for CharGPT and 9k tokens for Claude.

---

BBH Performance (CoT Eval)
In-domain Task Accuracy (CoT Eval)
based on Number Tasks Instances
Rlan-S-1115
In-domain Task Accuracy (CoT Evaluation)
CoTTS 128
47,60
44,76
BBH Peromance (CoT Evaluation)
42.04
40:33
16
ANLIR1
ANLIR2
ANLI-R3
RTE
Winogrande
9 CoT Data instances)
CoT Collection instances)
CoT Collection instancea)
CoT Collection instances)
Figure 4: In-domain task accuracy with CoT evaluation. CoT Fine-tuning with the COT COLLECTION also im- proves accuracy in-domain tasks as well.
Figure 3: Scaling plot of increasing the number of in- stances within the COT COLLECTION compared to us- ing the existing 9 CoT datasets. Even with less number of instances, maintaining a wider range of tasks is cru- cial improve the CoT abilities of underfying LLM.
While COT-T5 uses the same tasks as its base model (Flan-T5), we also check whether CoT fine- tuning on a wide range of tasks could possibly harm performance. For this purpose, we use the test set of 5 tasks within the CoT COLLECTION, namely ANLI-R1, ANLI-R2, ANLI-R3, RTE, and Wino- grande. Note that this differs with the Setup #2 in the main experiments in that we use different base models (TO vs Flan-T5), and the tasks are already used for CoT fine- tnn-ttunigg
5.1 Scaling the number of tasks & instances
our main experiments, we used a large number of instances (1 (.84)) across a large number of tasks (1,060) to apply CoT fine-tuning, A natural ques- tion arises: "Is it more effective to increase the number of tasks or the number of instances?" To conduct experiment
address this question, of randomly sampling small number of instances within the CoT COLLECTION and comparing the BBH performance with (1) a baseline that only CoT fine-tuned with the existing CoT tasks and (2) COT-T5 that fully utilizes all the 1.84M in- stances. Specifically, we sample 10K, 100K in- stances within the CoT COLLECTION and for the 9 CoT tasks, we fully use all the 180K instances. As COT-T5, we use Flan-T5 as our base model and use the same training configuration and evaluation setting (CoT Eval) during our experiments.
Results are shown in Figure 4, where COT-T5 consistently improves in-domain accuracy on the learned tasks as well. However, we conjecture that this is because used the exact same task that Flan-T5 used CoT fine-tuned CoT-T5. Adding additional tasks that were not used train Flan-T5 and CoT-T5 could show different results, and we leave additional exploration catastrophic forget- ting during CoT fine-tuning to future work.
Conclusion
The results are shown in Figure 3, where sur- prisingly, only using 10K instances across 1,060 tasks obtains better performance compared ing 180K instances across 9 tasks. This shows that maintaining a wide range of tasks more crucial compared increasing the number of instances.
In this work, show that augmenting rationales from an instruction tuning data using LLMs (Open AI Codex), and CoT fine-tuning could improve the reasoning capabilities of smaller LMs. Specif- ically, we construct CoT COLLECTION, a large- scale instraction-tuning dataset with 1.84M CoT rationales extracted across 1,060 NLP tasks. With our dataset, we CoT fine-tune Flan-T5 and obtain CoT-T5, which shows better zero-shot generaliza- tion performance and serves better base model when training with few number of instances. We hope CoT COLLECTION could be beneficial in the development of future strategies for advancing the capabilities of LMs with CoT fine-tuning.
5.2 In-domain Task Accuracy of CoT-TS
It is well known that LMs that are fine-tuned wide range of tasks suffer from catastrophic forget- ting (Chen et al., 2020; Jang et al., 2021, 2023), a phenomenon where an LM improves its per- formance on newly learned tasks while the per- formance on previously learned tasks diminishes.

---

Acknowledgments
process in Appendix A. Also, there is room of im- provement regarding the quality of our dataset by using more powerful LLMs such as GPT-4 and bet- ter prompting techniques such as Tree of Thoughts (ToT) (Yao et al., 2023). This was examined by later work in Mukherjee et al. (2023) which used GPT-4 to augment 5 million rationales and Yue et al. (2023) which mixed Chain-of-Thoughh and Pro- gram of Thoughts (PoT) during fine-tuning. Using rationales extracted using Tree of Thoughts (Yao et al., 2023) could also be explored in future work.
This work was partly supported by Institute of In- formation & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2022-0-00264 Compre- hensive Video Understanding and Generation with knowledge-baseed Deep Logic Neural Network, 50%; 1f.22021-0-02066 Artificial Intelligence In- novation Hub, 20%) and KAIST-NAVER Hyper- creative AI Center.
Limitations
Recently, there has been a lot of focus distill- ing the ability to engage in dialogues with long- form outputs in the context instruction follow- ing (Taori et al., 2023; Chiang et al., 2023). Since our model COT-T5 is not trained to engage in di- alogues with long-form responses from LLMs, it does not necessarily possess the ability to be ap- work
References
Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Di- nesh Garg. 2021, Explanations for commonsenseqa: New dataset and models. In Proceedings of the 59th Annual Meeting the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3050-3065.
plied in chat applications. In contrast, focuses improving the zero-shot and few-shot capabilities by training on academic benchmarks (CoT COLLECTION, Flan Collection), where LMs trained with chat data lack on. Utilizing both long- form chat data from LLMs along with instruction data from academic tasks has been addressed in future work (Wang et al., 2023). Moreover, vari- ous applications have been introduced by using the FEEDBACK COLLECTION to train advanced chat models 7.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Yejin Choi, and Hannaneh Ha- jishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 2357-2367.
Anthropic. 2023.
Claude.
https:
//www, www.annthe wetthropic com/ index/ int ng claude.
Also, since COT-T5 uses Flan-T5 as base model, it doesn't have the ability to perform step- by-step reasoning in diverse languages. Exploring how to efficiently and effectively train on CoT data from multiple languages is also a promising and important line of future work. While Shi et al. (2022) has shown that large LMs with more than 100B parameters have the ability to write CoT in different languages, our results show that smaller LMs show nearly zero accuracy when solving math problems in different languages. While CoT fine- tuning somehow shows slight improvement, a more comprehensive strategy of integrating the ability to write CoT in diverse language would hold crucial.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, al. 2021. Ext5: Towards extreme multi- task scaling for transfer learning. arXiv preprint arXiv:2111.1090
Akari Asai, Mohammadreza Salehi, Matthew Peters, and Hannaneh Hajishirzi. 2022. Attentional mix- tures of soft prompt tuning for parameter-efficient multi-task knowledge sharing. arXiv preprint arXiv:2205. 11961.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1
In terms of reproducibility. it is extremely con- cerning that proprietary LLMS shut down such as the example of the Codex, the LLM used for ra- tionale augmentation. We provide additional analy- sis on how different L.MMs could be used for this
Dana-Maria Camburu, Tim Rocktäschel. Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language expla- nations. Advances in Neural Information Processing Systems, 31.
"https://hugge ngf agceaac.... cc/ planet/ effi-13b

---

Sanyuan Chen, Yutai Hou, Yiming Cui. Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 7870-7881.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru. Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shus- ter, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. Opt-iml: Scaling language model instruc- tion meta learning through the lens of generalization. arXiv preprint arXiv:2212.1 12017.
Wei-Lin Chiang, Zhuohan Li, 7i Lin, Ying Sheng. Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* * chatgpt quality.
Joel Jang, Seungone Kim, Sconghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyung- jae Lee, and Minjoon Seo. 2023. Exploring the bene- fits of training expert language models over instruc- tion tuning. arXiv preprint 2rXiv:2302.03002
Joel Jang, Sconghyeon Ye, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021. Towards contin- ual knowledge learning of language models, arXiv preprint 1xxv:21100032115
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling mnstruction-finetune language models. arXiv preprint arXiv:2210.1 11416.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567-2577.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavar- ian, Jacob Hilton, Reichiro Nakano. Christopher Hesse, and John Schulman. 2021. Training veri- fiers to solve math word problems. urXiv preprint arXiv:2110. 14166.
Yao Fu, Hao Peng. Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.1277 12726
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.055
Mor Geva, Daniel Khashabi. Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence compo- sition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 8082 20900
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor. Luke Zeutlemoyer, Maryam ZarandZ and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint arXiv:2212.07919.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, et al. 2022. Soda: Million-scale dialogue distillation with so- cial commonsense contextualization. arXiv preprint arXiv:2212.10465
Google. 2023.
Bard. Alog.gooog e http//hhcoccgaaa/////ocooo bard-googe
https://
Seungone Kim, Se June Joo, Yul Jang, Hyungjoo Chae, and Jinyoung Yeo. 2023. Cotever: Chain of thought prompting annotation toolkit for explanation verifica- tion. arXiv preprimt arXiv:2303.03668.
sear-sarcc-upcee updates//
Namgyu Ho, Laura Schmid, and Se- YouYg Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.1 10071..
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multi- lingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 3499- 3505.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint 1904 09751.
Or Honovich. Thomas Seialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor. arXiv preprint arXiv:2212.09652
Nikita Kitaev and Dan Klein. 2018. Constituency pars- ing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2676-2686.
Edward Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint 11X.v:2100.09685.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205.11912

---

Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021. Qed: A framework and dataset for explanations in question answering. Transactions of the Association for computational Linguistics, 9:790-806.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-tase task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3470-3487.
Brian Lester, Rami A1-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficieee prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint arXiv:221 1.01786.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. Datasets: A commu- nity library for natural language processing. arXiv preprint Xiv: 028002.6.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi. and Ahmed Awadallah. 2023, Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint wwXiv22306.02707.
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yanggiu Song. 2023. Mult jailbreaking privacy attacks on chatgpt. arXiv preprint 2rXiv:230.00519,
Yasumasa Onoe, Michael IQ Zhang, Eunsol Choi, and Greg Durrett. 2021. Creak: A dataset for com- monsense reasoning over entity knowledge. arXiv preprint arXiv:2109.01653.
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022a. Wanli: Worker and ai collabora- tion for natural language inference dataset creation. arXiv preprint arXiv:2001.05955
OpenAl. 2022, ChatGPT. b chattpt.......
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mo- hta, Tenghao Huang. Mohit Bansal, and Colin A Raf- fel. 2022b. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems, 35: 1965.
Xiaoman Pan, Wenlin Yao, Hongming Zhang, Dian Yu, Dong Yu, and Jianshu Chen. 2022. Knowledge-in- context: Towards knowledgeable semi- cemimparamttiic language models. arXiv preprint arxiv:2210..6643.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du. Zhilin Yang, and Jie Tang. 2022c. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn Res., 21(140)
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint 11Xiv:2103.10388
Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clini- cal domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 1586-1596.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arxiv preprint arXiv:2301.1.33 13688.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. 2021. Multitask prompted training ables zero-shot task generalization. arXiv preprint 2
David Mhlanga. 2023. Open ai in education, the re- sponsible and ethical use of chatgpt towards lifelong learning. Education, the Responsible and Ethical Use of ChaIGT Towards Lifelong Learning (Febru- ary 11, 2023).
Sewon Min, Mike Lewis. Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2791-2809.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference the European Chapter of the Asso- ciation for Computational Linguisticx: Main Volume, pages 255-269.

---

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang. Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint wXXv:2109.01652.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. Distilling multi-step reasoning ca- pabilities of large language models into smaller mod- els via semantic decompositions. arXiv preprint arXiv:2212.00193.
Jason Wei, Yi Tay, and Quoc Le. 2022a. In- verse scaling can become u-shaped. arXiv preprint (xXiv:2211002011
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. arXiv preprint FrXXiv2201.11900
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint w.w.::22100090000
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, pages 4602-4625.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stan- ford alpaca: An instruction-follovingg Ilama model. ht https:/ ttttp://githhuee http//gtttub///////////tt...e.et www.////tttttteaaaaeea stanford_a a paca.
Haike Xu, Zongyu Lin, Jing Zhou, Yanan Zheng. and Zhilin Yang. 2022. A universal discrimina- tor for zero-shot generalization. arXiv preprint N0v:2211.08090
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Dara Bahri, Tal Schuster. Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unify- ing language learning paradigms. arXiv preprint arXiv:2205.05131.
Linting Xue, Noah Constant, Adam Roherts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498.
Don Tuggener, Pius Von Däniken, Thomas Peetz, and Mark Cieliebak. 2020. Ledgar: large-scale multi- label corpus for text classification legal provisions in contracts. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1235- 1241.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao. and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint ar3iv:2305. 10601.
Cunxiang Wang. Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019, Does it make sense? and why? pilot study for sense making and explana- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020 4026.
Michihiro Yasunaga and Percy Liang. 2020. Graph- based, self-supervised program repair from diagnos- feedback. In International Conference on Ma- chine Learning, pages 10799-10808 PMLR.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. 2022a. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arxiv:2203.
Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang. Hyeongu Yun, Yireun Kim, and Minjoon Seo. 2023. In-context instruction learning. arXiv preprint arXiv:2302. 22223322.44999 14691.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah Smith, Iz Beltagy, et al. 2023. How far can camels go? exploring the state of instruction tuning open re- sources. arxiv preprint arXiv:2306.047.
Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. 2022. Guess the instruction! making language models stronger zero-shot leamner.. arXiv preprint arXiv:2210.02966
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wen- hao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:239.006633.
Yizhong Wang, Swaroop Mishra, Pegah Alipoor- molabashi, Yeganch Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022b. Super-naturalinsstuutiinsss Generalization via declarative instructions on 1600+ nlp tasks. URL hilps:/arxiv: org/abs/2204.07705.
Eric Zelikman, Yuhuai Wu, and Noah D Goodman. 2022. Star: Bootstrapping reasoning with reason- ing. arXiv preprint arXiv:2203.1 14465.

---

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain thought prompt- ing in large language models. arXiv preprint arXiv:2210.03493.
Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pre- training help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal hold- ings. In Proceedings of the eighteenth international conference artificial intelligence and law, pages 159-168.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei. Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. 2022. pesst-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.10625.

---

A Analysis of COT COLLECTION
Metries
Human
CoT Collection
Laihhfulness
0.8836
0.8914
Non-cherry picked rationales within COT COL- LECTION are shown in Table 8. We perform analysis regarding the quality, diversity, and reproducibility of rationale within the CoT COLLECTION.
Semantic
faithfulness, www
0.8756
Alignment
repetition_ word
0,9376
informativeness, slep
0,9519
Semantic
infomattivnness, chain
0.2295
0.2797
Similarity
repetition sent
Logical
discourse representation
0.4855
0.4687
Inference
coherence sttp__teeee step
0.7763
0.7813
perplexity_ _Slep perplexity chain
0.0122
Diversity Rationales To take a look into the diversity of COT COLLECTION, we use Berkeley Neural Parser (Kitaev and Klein, 2018; Kitaev et al., 2019) and parse rationales. More specifically, the verb which is closest to the root of the parse tree along the noun object is extracted. We compare this with the rationales from the CoT datasets used in Chung et al. (2022). As shown in Figure 5, COT COLLECTION have diverse textual formats included compared to the 9 existing CoT datasets that have high proportion assigned to 'answer question' and 'consider following'.
Language
0.0475
0.0255
perplexity_stee sttpp stop_max max
0.0144
0.0088
Coherence
grammar stop 0.8883
0.8721
grammar step_mas 0.8013
0.7724
Table 6: Comparison of the quality between rationales and metries from ROSCOE (Golovneva et al., 2022) used.
rationales. 13 label-free
show that different LLMs are able to produce high quality rationales in terms of semantic alignment and language coherence.
B Filtering CoT COLLECTION
After generating multiple rationales, we filter to ensure high quality. We apply the fol- lowing criteria to filter instances:
Filtering
Quality of Rationales To ensure the quality of COT COLLECTION, we use ROSCOE (Golovneva al., 2022), suite of metrics designed to evaluate rationales under different criteria within semantic alignment, semantic similarity, logical inference, language coherence. We compare with human- rationales obtained during Prompt Creation in Section 3. The 13 ROSCOE scores are shown in Table 6. The results show that CoT COLLECTION include CoT rationales that are faith- ful, less repetitive, informative, and logical even when compared to human-aui authoree rationales. Yet, we find that machine -eiierred rationales tend to have higher perplexity, leading to lower language coherence scores. We conjecture this is because in- cluding diverse textual formats leads may result in relatively higher perplexity (Holtzman al., 2019).
We exclude rationales that do not include the ground truth answer when splitted by white spaces. While rationale that doesn't include the answer isn't necessarily a bad rationale, we found it effective exclude inconsistent ones.
We exclude CoT rationales that exceed the maximum output length, where we constrain the sum of and to be shorter than 512 tokens.
human-authored
We exclude rationales that are identical to pre- viously augmented ones during our process.
We exclude rationales that include repetitive sentences within the context.
Is COT COLLECTION Reprodueible? One could doubt whether CoT COLLECTION is repro- ducible due to the usage OpenAl model in the process of CoT rationale augmentation". In this section, we test different LLMs to generate 150 rationales randomly sampled from CoT COLLEC- TION, and compare the ROSCOE score (Golovneva al., 2022) in order assess the quality. We use Bard (Google, 2023), Claude (Anthropic, 2023), for comparing with OpenAT Codex. The compar- ison of quality is shown in Figure The results
We further include the filtered instances in Table 9.
Also, we found that in many cases, Codex de- generates and starts writing code after the rationale. To prevent inclusion of code snippets, we apply ad- ditional filtering based on trigger tokens that abun- dantly appear in the start of the code. The list trigger tokens are as follows:
"Morovver, OpenAl announced to stop its support on
Codex model starting from June, 2023.

---

instances and rationales across 1,060 tasks.
CODE FILTER
During evaluation, found that using nucleus
"An "ninnrrr "An'in" An", "inern"" "in "in" "Innn ln"", "Anin An "inin Inimport", "In'zin""" "\"inln"
sampling (Holtzman et al., 2019) with p =0.8 and no_repeat_n__et =3 was very effective in gener- ating good-quality rationales.
"[ "[examp", "[Examp", "An' "er""n" "In", "Annn " "An""" "A""""""a", "Inlatt", "Ant", "V"""", "\"mntt" "V"larr,,
"print (", "Ia Animport, "A"Inintt", "Anin " "intett", "It ", "!"In Ann "Inin 1PP" "Nnin AeF"" "bns/blocketeto", "loin +/l"",
"\"ln'n An ltt"" "AAn lt", "V"loín lt */", "/"""""" Ao F" "inin leit", "I"nn" Anlt)" "ITiDiE", "I"iain" "I""n"""aaa" "inin OR", "In An"
C Training and Evaluation Details of CoT-T5
Params
Model
Batch size
I.R
Optimizer
CoT-TS-3B
5e-5
AdamW
IIB
COT-TS-1IB
8
le-4
Adafactor
Table 7: Hyperparameters used for fine-tuning CoT-TS.
We mostly follow the fine-tuning details of Chung et al. (2022) to train CoT-T5, The hyper- parameters used for training CoT-T5 shown in Table 7. We find 3B and 11B sized LMS converge well using different optimizers. While CoT-TS-3B tends to converge well using AdamW, CoT-T5-11B is well optimized using Adafactor. For both sizes, we train with 1 epoch using COT COLLECTION which takes 1 day (3B) and 7 days (1IB) when 8 A 100 (80GB) GPUs are used. For both settings, we gradient accumulation step of 8.
For sampling training instances, we sample instances from Flan Collection (Longpre et al., 2023) by using the proportion of (FLAN, 30.85% (P3), (Exitinxs CoT datasets), (SN47%(SNN) and 1 (other dialogue & code datasets). This is done by sampling 400 instances (FLAN), 300 instances (P3), 150 instances (SNI), 4000 instances (Existing 9 CoT datasets), and 300 instances (other dialogue & code datasets), respec- tively. We generate rationales per instance and then apply filtering, leading to the final set of CoT COLLECTION, which is consisted of 1.84 million

---

Informainn
c0nted
unguago
Tollwingg
mnmmatin
ENTOR
makk
ealation
reseribe
ong
ever
energy sseiee
paty
Flan-T5 CoT Data
CoT Collection (Ours)
Figure 5: The top 20 common verbs (inner circle) and their 4 noun objects (outer circle) within the rationales of the CoT tasks used Chung et al. (2022) (left side) and our CoT COLLECTION with 1,060 tasks (right side).
Codex (Ours)
Bard
Claude
1.0000
0.7500
1000000 Connesse step
informativeness chain
discourse representation
coherence_$tn stép Stép ys step
perplexity. step max
grammar step step max
faithfülness
Falthfulness WW
repeat Speettoo M
word
repetition sent
perplezage step
perty / chain
grammar ssep
0.5000
parpleaae
0.2500
0.0000
ROSCOE_S SA
ROSCOE SS _SS
ROSCOE_L LI
ROSCOE_L LC
Figure 6: Comparison of the quality of augmenting CoT rationales using different LLMs by reference-free ROSCOE scores. Semantic Al)gument(SA), Semantic Similarity(SS). Logical Inference(LD), Coherence(LC).

---

Question Instruction
Answer
Rational
Read following contest choose best option answer question.
Concest: From the tiple loop the oxccuted after tive minutes, le suspected that flying
during the woudered just how diligent George about practicing own
managed
tripte loop maneuver in which an aireratt makes three loops.
Rased on the camiest. it can be inforned that tmpple loop involves making loups shoulh be chuaan
Question: What is criple koup?
D
Options:
A. Nose the abeve choices.
B. inund trijs plane coute. aecities
featue pulled off making 3 loops in ait.
Read the passage below and choose the right answer to the following ingwing question (cholces are more or less
passage by inercesing blood flow As such, can be inferred than decretsed bloed flow conesponds with less heet being given otf. Since Sanih notices that bloed really pucsping day, she giving hear compared usual.
endethern gives off body.
day, endethenn may give by increasing blood flow to the surface of the body,
leos
Sarah flaishes walk. she antices blood really pumping that
Sarah cos know that since it's cook day,
she's giving otf what kind of heat compared to moui more less?
this Pubertal development affect circulatory respiratory systems an adolesceuus' heat and langs increase bock size and capacity. These cthnnges lead to incrossed stregih and volerance for evenciser
Sex differences apparent males tend develop systelic
"laggr lower resting heart rate.
pressune,
sentence the paragraph states that males teed develop larger liarts and langs.
males
greater copacity for carrying oxygen the blood. nouitalizing chemical products muscular exercise, higher bloed bemoglohin and mure red blood cells.
Which sax ornds develop lnger hna ling??
you are string
mesd to remore duplicate choracters string any and return the resultngg sting.
Thrre an dipliczee clumcien given string. the resultig sring will be the sime the impit Therefon, the will be WWC.F//YYUOiiYYIII 7100pp"'
IKGFRd WLL2qipw
2XQQfp=
tick you gives numbers
and you should remooee all dupiicatios in the list.
The giver litt is 13, 1.0,3% 3/.
Teeeery mamberis is repested in the list an lst should be retarned,
We see that "3" appeers the lat twioe.
You lis shoul be rimber inside beackets, just like given lis.
1,
Therefom, keep
the
The mailing list is 15, 1.0).
13.5. 1, 8)
How keep your phaze cool the
Keep your
you If you seed to use your ghone while you'de to in your cur, purchose mount that attaches one of the dudhbocrd air Tunn couditiouzr liegss your plane
coaditioaing
While the does montion that your should does'' spezify whether referiing to Additionall, we don't know from the context If other devices such tiblets can also be kepe coal using this method.
Can we draw the following conclusion? keep phone and tablet cool while mounted
It's iupoosiible to say
coudiicour vent
them
"It's impossille say"
OPTIONS: Yes
II's ingosiill
Stephen Breyer was a nomisee for the Supremt Court by Bill Clinton and was continitd July 29, 1991. Although he nominated
Give possitle connet mnswer the guestian "whu suminsted siephmn g. beayer?
Bill Clintaa
answer
because it happened withis yours those dates before.
Write dowa the solution for this math probleem: Salve 201ve 264ve
STEP 1, 291-1 265-c 189
189
STEP c-
Table 8: Example of rationales within COT COLLECTION including tasks such as Multiple choice QA (MCQA), Ex- tractive QA (ExQA), Closed-book QA (CBQA), Formal Logic, Natural Language Inference (NLI), and Arithmetic.

---

Question
Answer
CoT Rationale
To solve this problem,
we noed to find the number of apples and peaches that Steven has than Steven and 10 more apples than Steven.
know
peaches
are given math word problem and supposed only subtraction the numbers embedded in the text to unswer the following question report final numerical
We know Steven has
11 apples and 18 peachos.
Using these clues we can write two equitions:
peaches
Peaches
Sieven's Apples = Jake's Apples
peaches more apples
add
equations together:
Steven has II apples and 18 peaches.
(Steven's Peaches) (Stevens' Apples) - (Jake's Penches e (lake's Apples-10)
(18+11)=[ake Peaches +8y+ (Jakes' Apples-10)
Question: How many peaches than apples does Steven have?
29 Jakes Jakes applecs-2
n=29-2-27
b=[18+11)-a
c=l/2
Idmof2
690 1-4495 -4855
Suhtrocting 4825 from both sides:
What the solution? Solve 600% 14495 -425 c.
9669
Dividing both sides by 690: Om
Table 9: Two representative rationale examples that filtered with filtering criteria. It demonstrates that not including the ground truth token within the rationale could indicate that the LLM failed to generate a consistent rationale, often leading to degeneration. This phenomenon is mostly found on math datasets.

---

D Prompts used for Rationale Augmentation
Pick a spoc to plant them
Dig a hole in the ditt
Pat the seed in the bole
Pour some water the sced and hole
Gower the bole
Among the 26 demonstrations used for each fam- ily of tasks when augmenting CoT rationales, we include 4 examples, other demonstrations for task families can be seen at this link.
Press
Speay sime plant un
Question:
suppose no water is available huppers, how will it affect vegetables?
A. more
B: los
[0sinms)]
Fuumpl
Instraction Question)
Skylor coppedd freashing Las hohd Skylar' any way, Givar the
contest:
neat?
Answer
Fossible answers bog doctors try again, get her up and walling home, being Skylar
back
[Raiioale]
[Ogtions]
The process planing sceds eequises which essential for the growth plants.
hey the dectors to try agsin
Water inmsspoii
regilatee semperatur, and abouth will not germinste and the glant will not goow properly.
up walking home teing Skylor bock
water ovailsble, leading to lower vegitibles Hence, the ahsence waser will resalt "less' 'vegetables, making the answer "."
Auswert
beg the doctom try again
[Kaionale]
Exxappe 51
The costent of the siuution is that Skyltr his stopped breathing and Lee hokling
and Quactiion)
Skylar's amns. This suggests that Skylar has mozival madiccal attention and no lenger alie.
the following conversation and answer the question. wortial shout my sister, Lacy.
bug the duches iry spain, would way potentially save Skylar's life.
this, would
W: Why is that?
M: She thinks she's 's tus fat.
epsione of petting Skyler up walking home bringing Skyler not align
W: she?
with the comeat, as they would not address the ammediate medical emergency at hand
M- No, she kerps skipping meahs. Then, she only eats chips and drinks coll.. fun!
W:l used do that. called it?
So the answer to beg the doctoes again.
Why you stop
you'll 'Il mz doctor told me when I'm hungry, sead, "Ear till you're full much later." said lot of girls their health this way.
Did she say what to ext?
[Essmplle
W: She said, "Fat fruit, vegetables, meass and grains. Have reguler meals and stoakss Get
[Estructionaas Quetioo]]
exercise, tos.
Do you think the right answer to the guestion "what con nun on alcobolic fermentation of the glaeeee in com or other plans" "hicyela"?
Question: What's the rebationsip between the speaker??
O0tiioss] No
DDoctorr and patient. Friends.
Yes
Beother sister.
[Auseer]
[0stans]]
DDettor and patient
Friends
[Rationde]
Brother sister
Alcoholic femmentatio is process that occurs living organisms, most commonty yeast. placusr into enemgy. This precerss doss produce enough something like bicycle.
[Answer]
Friends
So answer No.
[Raiinake]
The cunternation between two people, man(M) and a wommn(W) (W), is discusiing the nan's mun'ssisttz sister who las negative view of her own boly and is skipping meals. The woman stores that she used engage similor behavior. and explains her doclor athised her helamasd
[Isample
[estraction and Quetiios)
Evenually. supuure. similar popping ballson. When happens. may imitate pelvic lining cause pain, otien severe. Most time the pain begins to graduilyy subside over the course the next few days.
large
wall weakens and fluid begins leak
Anong thrre optious,
hirst, sace the woman sharing her experience of what the doctor told her. it salikely she would be medical professional. So the answer couldn't be doctor and purieent
Whet type of ductor might the person had
Second, given that the man is talking about his sister, the womtn unlikely be the
mom sinter.
parfiatiiinn A hean sungun a gynraolugia- - Nume of the shive chrices
the conversation infornal and speakers discueeing a persouul
likely relationoip hetwern the speskers
[Otiins]]
poediaticiaa
Therefore, the snower Friends.
hom surpoon
a genecologist
Nose the ahove choices
Exumpl 5]
TAnserr]
Bistruction and Queestion]
synccologist
Queeion: hos place for the trniblemaker smoke while was bathenom?
[aaionale]
including pain mentions aystt the rejion sympices the syel misturing,
Choices:
leskage, sugzests that lssue related repeoductive
playgrouud
urinary system. Given that the issue selated roproductive usiaary system, it likets that the person would have had see a specialist these areas.
school
at botel
Out of the givan optinns, 'a gynczologist" is the most likely specialist that the person would have had ses, gynecologists specialize in lemale reproductive health. inchaling issues selated the stcrus, ovartss and other repreductive oogans.
convesietce
The natonale choose ""cchoo"" us the answer is tlat:
conclusion, lased the given contest, the likely asswer gynecologist.
[gtiioss]
Nox Given
Example 4]
[nnwer]
Iustruction and Quesioo]]
smoles unditionally buthmoars
Process:
Get somt scols

---

[Ratonale]
So the answer is to beg the doctoes to try again.
Hare's the ational:: "snokers traitionally use school bathoons".
Essiplle 2]
[Eumple
Rustruction and Qucetion)
Quesion] So was looknng at quoses for
right quesiion "what can run alcuhallic fermenctiion
ise hockey. and cams accoss that sunss up how feeling right nuw When you talk about an injury and the kind depressiun you zo stroug,. it 's aot just because you 're our of shape and you can't go out and play. You re missig part of you.
the glucose an com other plants" is "bicycles"?
[Ogtions]
-No
Yes
Why the namater dlon in the dumpe?
Pick the best
from followaig options:
[Answer]
They unsile play ior carling.
Thyy
C. They are unable in play sox skasing.
[Rationle]]
D. They are unable to play the yard.
Alcoholic termentation a procoss that occurs in living organions, most commonty yeast, to convert glucose atoo energy, This process does not produse enough energy to power something like bicycle.
[Optioss
the answer No.
Answerr]
[Etumple 3)
[lsttuctionn and Question]
Evennullly the eyst can get lorge that part of the wall weakens and fiaid begins to leak out that the cyst rupeures It simdlar to popping water balloon. When this happens, the flaid may inriate pelvic lining and the
[Raiioall]]
the mentions the tornace's feelings depecssion follows. The
and the
Most the
it's just shout being This implizs. they like they missing
time pain begins gradullly subside might person
play,
type
depresioo linked to their physical iashility to patticipute is quont ur ausivity they enjoy. Roaed the cintted, it he infermal tiut namator is spset because they are burt and unable to pla ice bockey. So, the inswrr is R.
reasou
pradistricim- beart sungon -a genroolugia- Nure of abowe choises
[0sinnss]
pastatticcan
hort surpion
g.ynecologist
Exampp:
-Mste above choices
[Estraction Question]
O: What ulsquitous generater use genentte corrents?
[Anser
a gyneculogii
Choicss:
[Rationale]
inerii-- water- gravity magneiism
The costect mentione in the pelvic region and the sympioms of the repturing. related to the reproductive the repreductive mileary system, Our the givan optinss, "a gynecologiss" specialist these
inchuding and findd leakage. This suggests
Options]
system.
inertia
likely person
have
water
most likely specialist that the person would have to wxe, as gyneoologies specialise femole iepoddutivve bealth, incliding issues rolated to the userus, oxades, and other reproductive orgats.
gravity
magnetism
Answeri
In conclusion, based on the given contest, the most likely atswer is gynecologist.
maguzism
[Rational]]
penerstor is device converts mechanical energy electrical energy. This done by ising magnetic lield induce cureeni the genesttrr' windings This currest is then sont to an essemal circus, where can be used to power electrical devices.
REsumple 4]
[lsstrccion and Question)
Process:
The generator taas maggeiiin geserate curents so the snower is magsetiom..
Gett some sceds
Pick a spoe to plant them
Dig hole in the dirt
the seed in the hole
[Fuumple [Instruction Question]
the hole
up
Piess
Prompts 1: Demonstration used for tasks in family of MCQA (TO)
Spra some plant food on it.
Queetion:
suppose water is available lmppens, how will it affect vegetables?
Optinns)
[lisample
A
[Esstruction Questioa]
BB
Skylu stoppedl breathing but Lece held Skylar's is cama asyway, in vain. Givee the
context: Lee want to do what next?
[Anwwrr]
Possible anowens beg the dostons to by again, get har up and walking home, being Skylar back
[Rationale]
Optionns]
The proczess planing seads resuires water, which couential the growth plants.
heg the ducturs try upain
Water helps irassort nutriens, regulate temperature, and absuch minerals from the soel
per up walking home
zo wate: is ovaliahle, the seed will not gesmanate and the plant will not grow property. leading to spw er vegenables, Hence, the absence of water will resuit in "less" vegetablee. mailing the snswer "B".
being Skyla book
Ausserr
beg the doctors again
[ational]
|ESuppp 5]
stopped and Lee holding medical Bused on this, Lee would sont beg the doctres to iry again as this would the bes way to poteanally save Skylar's s lide.
coneett of sitution
[lssticciinn and Question)
Skylrr's
suggests Skylar
lenger
Read the following converstion and answer the question.
alive
M: I'm worried about my sisur, Lucy.
W: Why is that?
M. She flinks she's too fat.
The options of getting Skylir up and walking hone or bringing Skylar back not align with the context, they would not oddress the immodiate modical emergency at hand
W:
she keeps skipping meils. she clips drinks ools.

---

W:J used do that it called binge. was so fun!
M. Why did you sop doing it?
Exampl 5]
W: Well, my dector told me cas when I'm bungry, Shee saad. "Lat till you'ce full or you'll cat too much liner"" She said lot girls ruin their health this way.
[astruction and Question)
What does the uliquious generator genenute currents?
M: Did she what ceet
W. Siee
fruit, vegetables,
and grains. Have segular meals and stocks.
Choices:
water gravayy magnetim
Question: Whot's the relationship between the speakers?
[Opiioss]
Doctor and patient
Ineetia
Fricuds.
water
Brother and sivier.
graviiy
asagnetism
[Options]
Doctie patient
[Answer]
Frindd
mrggettio
Boother and sister
[Raiinale]
Ausserr
generator a device that convers mechanical energy atto elecorical energy. This is done using maggettic fiedd to induce curzent in the generater's windings This exteraal where be used power elecrical devices. The generstor magnetioon to generate suncnts,
Friends
[Rationae]]
magnetiion.
two people, man(M) summnn W,, diserrecing mar's 's sister who has negative view own body and skipping meals. The woman stores that used engage in similar behavior, and explaine that her doctor advised her
cunvenation
to car a balaseed diet and exercise.
Exampl 9] [Instruction and Question)
Among tie three oppions,
staring her experience of what the doctor told her, mulikely would medicall professional.
First the
Prompts 2: Demonstration used for tasks in family of Commonsense (SNI)
couldn't
Second, given
man talking about sister. the wamtn uniikelly to be
mun's sister.
Lastly, became the comvenation is informal and the speakers are discussing a persomal
mutter in likely that the relationship berwern the speakers is ffoe of frimnd.
Thrrofore, the anowrr is Friemde.
|Esomple 1]
[astruction and Question)
Problem. Context. Ansexxia males acsounted for approximancly six percent of cases discrder clinic becomes common males. IIUNS yeda- nolan:
Exumppe
[nstrrction and Quetion)
Queeiio: The hos plune for the trublemsker to sncak a smoke while he was where.
was in the buthoo"
[Answrr] no
Choices:
playgroudd
[Rationale]
school
The information gaven only pertains to cases seen this particular catinng disonder clinic and
botel
becoming repeesentative the population. determute whether anoresia would look from sumple muilipll varify trend. Si
fhotier
convesieuce store
populaaioon, well
The ratonale to chouse ""school" as the answer that:
Options]
[Etumple 2]
[iistruction and Question]
Generate cuntect and hyputhecis WrinAnwwrr
AAsserr
snokers inditionally use school bathrooss
[Answer
also swimming and cycling, He said he stucles, though be to admit
[ationae]
flat never get farther than first twn goges. Willie meseodblle not only remsined
Horess the rationale: "smokers tuaiitionally use school buthrouns".
silent during thrse consenations hit picked his herries shoely so that they might forget
flos he was there but he reckoned without Zach Hyphhiil Willie was there
[Rationsk]]
Example
and example for notural language inderence tacc/nttee also liked though he to admit that remained silent
Insttraction and Question]
swimaing and cycling. He said he
So I was looking at quotes for our tshits for ice lookey and I case across one that sums
never got further the first two papes. Wille meanwtile
up how 'n ficling right now. When yuu tilk about in iijoy and the kind of deprosiin go thoough, i not just becuse you 'ee out of shape and you can't go out and glay. You miscing
durase these comersattins
picked beries siowly that they night fouget that was there but mekomed without Zach Hygiilehhere Willie was thene
Why
damps?
Pick the best answer trom following options:
[Ecimple 3)
A. They are unable to play ice carling.
listtuction and Question]
B. They are burt.
dimesor Ielenmy English writer and ducctor whose woeks include scceenplays "Mrs Brown". "Driving Lesons": "The King Scotland", "Churiltte Gity" and "The Eagle"
Bruck populor English
C. They ur unable to play ice skaring.
D. They are unable to play the yard.
Boock has also written two plays for the Hampstead dowastains Mesaee/OPOOONSSS Noda- It's impossible to ssyln- NolnA:
O0tiooss
[Answrr]
impossible say
[Rationale]
Answer]
While informatios provided the aptions shout Jarenty Boock suggests is indeed an English wyttr and direcsor with a number of successful works under his belt, there is not anough information to desernine whetle: he retuallyy "popula"" oc sot. The tomu "populaa" subjsctive and can vary dopending on difierent factoes, such time pesiod, genes, and sudienee. So the answer is "It's impossible say".
B
[Raiioale]
quete the question mentions the ssenece's faslings
injury and
depesstion follos. The somrat says that it's not just about being out shage umbble tn pla, hui rather ihey feel like ihey are misinng part of themselves This implies that the reasoa for their degression linted to thir physical mabiliy to pattocipate in spont ur autivity they enjoy. Boaed un the contest, ii can be inferred that the narratris is spset because they ure burt and unable to play ice bookey. Soi the anower is B.
[Esmple 4]
[lnstruction and Question)
Buff Burger (Best ln Fist Food) was burger fast food franchise that odgannaned in Cleanatter Florida. Eael success in the laee 1950s and carly 196tss led the franchise to spread many pars of the Southeatiem United States. However. Boancial waes the 1970s sem the chiin deeline and most of be restaunets out of buvinos by the 1980.

---

Today, two Bilt Burger restaurants semain lathit Burger came about at Clearwater Plocida, OPTIONS: in Yestu- It's inpossiile to sayln- NolnA:
[Rational]]
While it mentioned that ties make up the backdrop, is not specified whether there are a loc of tres or not, Additizally the foces of the premise as on the man and child and not the trees in the background Hesse, is possible to determine whether the hypothrsis true
[Auswer]
Nes
[Ratinale)
To validate the claim Burger came about in Clearwater Hocida",
supporting evidence within content. We could see that "Bill Burger (less In Fist Food) was burger fast food franchise that odiginseed in Clearwater. Flonda." supports the claim we are trying validate. So the answer is "Yes..
[Example 10)
[lastruction and Question)]
Prompts 3: Demonstration used for tasks in family of NLI (FLAN)
|Fumple
Instraction Question] iepu hypethesic In the decual befone 1950s the Uniied Stoees begann muinsain public list of the people regarded as the Ten Most Wanted Pogtuveeenn Conest the 1050s, the United Straes FBI began to maintais public list of the geople regarded as the lea Most Wanted tragitives F Fallowing
bnef revigs of FBI people znd events that place the 1950s decade in contest, and then an listarical list of individual flgitives whose appcared the Most Wanted list during the decade the 1950,, Directer Edgar Roover/wOPPOON faler
Ritample 11
[lsttuucion and Quection]
Yeda- I''s inpossible sayln- Notainue
There are 15 trees in the grove. Grove wockers will plant trees is the grove today, Afher they are dane, there will he 21 tines. How many trees did the grove workers plant today?
[Answer] No
[Answer]
[Raiioall]]
[Rationale]
the 1950s, under ducction of Edgar Hoover. The mointain list indivilulls finst pleast Most Wantel 1990,, existence peior that decade, So
contest provided contradicns this hypothesis
We stut with ures. Liter we have 21 tuees. The diffecence must the muster of trees they planted. So, they must have planted 21 15 = trees.
futther sugponting the ootion that the mowwrr "Nu".
EEsmple 2]
[instruction Question)
Fihrre am cans the porking lot and mone can, arricr, how many cm and the parking loe?
[Fummle
[Esstraction Question)
O: The plaiatiff Penrose's design is distiaguished by aperiodicity pattera fold symuecy thought to esist nature but since been ideniiiedd certain eryeal fomationa,, com we wy that "The design has no idemoffed traiis that wouli distingridh
[nswerr
never quite repeats (self)
[Ratinal]]
a. TOOPTIONS tn- yedn- is not povcible to sella- sobelad:
the parking alocady. more Now c...
[Anvuar]
[Etample 3)
[Rationale]
[istruction and Question)
plaiseifss statiag that the design unique characteristes that apart, which dreerlyy contradicts the ststemant thut the design has sckntifial insis would dstinguish zt. Thacefore, the aaswer "no".
had chocolates sister they 35, how many total?
[Ansers]
[Example
[Rationak))
[Instraction Quesiion)
Leah had 32 chocolates and Leah's sister had 42, That meons there were originally 32 42 74 chocolates. 35 have brem caten. So in tocal silll hove 74 - 35 = 39 chooolates.
Does "The docking Soviec spacecraft was dinally realised in January 1969 by the Soyuz Soyuz 5 missinss" contain the conrect to "When was fast sueeessful docking of two man space crew! "OPTIONS:: (o- yestu- adtiaak:
[Answrr]
Eumple 4]
[lstraction and Question)
Jaon had 20 tullipops He gave Denny some lollipape. Now Sunon low 12 lollipaps. How mony Jollipups did Jason give Demyy?
[Rationaa]]
passage first succescful dacking of two-man space crew, both Soyuz Soyaz had two conntonauts ou the time of docking. Therefore, the statement contains the corrett answer to the question. So, the answer is yyes".
[Answer]
[Rational]]
Asso hollipope. Since he only has 12 now. he must love given to Denny
The number of koliipps he has given to Denay must have becs 20 12 -8 kollipass.
(Example
[Instraction Question)
iepor question: true 'I guard foxes." are getting
at night and attocking king the chickens. shall hove to gurd them" is imue? OPTIONS/ nola- yeslaouapus asswer:
Exampee 51
[Instruciion Quectionn) Shown five toys. Foc Chrastnas, be got toys coch frous moms dad. many soys does lee have now?
Auswer]
[Answer]
[aaiinaaa]
statement have to grord The does socurately reflect situation
aoambee socond sentenee "I shall hove pasrd them" this context, "thum" refors the lowes, so the correct statement would "I shall have to guand them", not "I stall have goaed the toses". So, the aiswer is 'no".
[Raiioale]
toys. pet from affer that low loys zut dad, so in has + soys.
[Exampl:
Esampl 5]
[Esstraction Quosion]
Bistruction and Question]
Quenken/aPrenise A man in shitt omane sborts looks at anull child worring a yellow shin and blue chuns with mass comprising backdmp mnllhyputooss
There wene nine compurer the server roomm. Five more computur were installed ruch day, from menday diurday, Hos many computers the
There are a lot of tress behind the small child/ais the by typonhesis cne LUUNS yeda- is nut possible sella-- Ancwarin
[Answer]
AAnseerr
possible
[Rationale]
There 4 days from monday to diursday, 5 were added each day, That means
intal 5- 20 computers were alded. There were 9 computers is the beginiing

---

Dow there are 9 20 computers.
[Fumple
[Esstruction halls, medday, golf welnersagy more. many
wednesday??
Auswerr
33
[ationale]
Micluel mntially had balls, lost Tuesday, after
ball.. Walnesday
2 halls.
Exxauple
[ssttruction and Question)
Olivia has $23. She bought five bogels 53 each. How much money does alle have left!
Rational)]
She bonght 5 bugels for 33 each. This moons she spens v =3 $15 on the bupels. She had $23 in beginning, now she has $23 $15 S8.
Exumpl: [Estractionaal Quetion)
Prompts 4: Demonstration used for tasks in family of Arithmetic (FLAN)


Reviews:
[{"review_id": "BPNIGz1Ekd", "reviewer": "Reviewer_d7Vi", "summary": "", "questions": "1. In multi-lingual evaluation, authors fine-tune CoT models using the data from the target language. Did authors run an experiment where CoT T5 is trained on data from all the target languages combined? Did authors observe any cross language transfer?\n2. Did authors evaluate CoT-T5 models in ICL setting? How do they fare compared for commercial models?\n3. Have authors checked for what datasize do the CoT T5 models start outperforming Flan-T5?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review. (R1) is our response to “Reasons To Reject” and (Q1) - (Q3) answer the 3 questions from “Questions For The Authors”.\n\n\n**(R1 & Q3) Effectiveness of CoT Collection**\n\nThe 9 publicly accessible CoT datasets have been integrated into the Flan Collection and also the CoT Collection.\n\nFor details on the nature of the datasets:\n\n* GSM8K and AQuA are arithmetic-focused datasets.\n* StrategyQA, ECQA, CREAK, and SenseMaking fall under the category of Multiple Choice QA datasets.\n* QASC and QED are distinguished as Extractive QA datasets.\n* e-SNLI operates as an NLI dataset.\n\nIn total, these datasets consist of 180K instances. Our sampling method drew 36K instances, mirroring the dataset ratio adopted in references [1,2]. Note that all 9 datasets incorporate rationales penned by human annotators but the limited scope of these tasks result in a narrow coverage. We recognize the need of clarification upon of this information and will ensure its inclusion in the camera-ready version.\n\nFurthermore, to address both the efficacy of the CoT Collection training and its comparative advantages relative to the 9 original CoT datasets used in Flan-T5's training, we have conducted supplementary ablation studies. These experiments evaluate zero-shot generalization capabilities under varying conditions of training instances and tasks.\n\n\n| Dataset          | Number of Tasks | Number of Instances | BBH Performance |\n|------------------|-----------------|---------------------|-----------------|\n| 9 CoT Datasets   | 9               | 180K                | 40.33           |\n| CoT Collection*  | 1060            | 10K                 | 42.04           |\n| CoT Collection** | 1060            | 100K                | 44.76           |\n| CoT Collection   | 1060            | 1.84M               | 47.60           |\n\n\n* The BBH Performance denotes the mean accuracy derived from the 23 classification datasets of BBH, as detailed in Table 2.\n* From the CoT Collection, instances were sampled in two quantities: 10K(*) and 100K(**) instances.\n* All experiments adhered to the CoT Evaluation setting.\n\nThe results clearly illustrate that the enhancement in the number of tasks is an important factor in achieving robust generalization capabilities for CoT Evaluation. Such observations are consistent with discoveries highlighted in previous studies [1,2,3,4]. This affirms the CoT Collection's efficacy in strengthening a language model's zero-shot generalization capabilities, even with a limited number of instances.\n\n\n**(Q1) Cross Language Transfer**\n\nRegarding the multilingual experiments:\n\n* We limited our testing to training on an individual language, without exploring cross-lingual transfer amongst varied languages.\n* The chief objective of this experimentation was to ascertain if introducing a minimal volume of CoT data could facilitate effective adaptation to the target language, specifically when addressing reasoning challenges, such as those posed by the MGSM dataset.\n\n\nPast studies have shown, in expansive models, the competency to articulate chains of thought in an array of languages is an emergent ability [5]. However, up until now, no discourse or hypothesis has suggested that training with rationales in various languages could enable cross-lingual transfers among different languages. As per the outcomes presented in our paper, we identify this as a promising avenue for future exploration.\n\n\n**(Q2) In-context Learning for CoT-T5**\n\nTo briefly examine the in-context learning capabilities of CoT-T5, we selected two tasks from the Big Bench Hard dataset: Sports Understanding and Date Understanding. Few-shot CoT demonstrations from the in the original CoT study were used [6].\n\nHere are the nature of these tasks:\n\n* Date Understanding: This task necessitates reasoning that's contingent upon the provided context. It challenges the model's ability to infer and extrapolate based on given data points.\n\n* Sports Understanding: This is a more knowledge-intensive endeavor. It not only tests the model's current understanding but also its capacity to tap into foundational and background knowledge about sports.\n\n\n|        | FLAN-T5 | CoT-T5 | UL2  | LaMDA | LaMDA | LaMDA | LaMDA | LaMDA | GPT3 | GPT3 | GPT3 | GPT3 | Codex | PALM | PALM | PALM |\n|--------|---------|--------|------|-------|-------|-------|-------|-------|------|------|------|------|-------|------|------|------|\n|        | 11B     | 11B    | 20B  | 420M  | 2B    | 8B    | 68B   | 137B  | 350M | 1.3B | 6.7B | 175B | -     | 8B   | 62B  | 540B |\n| Date Understanding  | 54.8    | 64.8   | 14.0 | 1.6   | 6.8   | 5.4   | 18.6  | 26.8  | 0.9  | 1.4  | 4.9  | 52.1 | 64.8  | 13.1 | 44.7 | 65.3 |\n| Sports Understanding | 66.4    | 65.2   | 65.3 | 49.7  | 57.5  | 52.1  | 77.5  | 85.8  | 41.6 | 26.9 | 4.4  | 82.4 | 98.5  | 75.2 | 93.6 | 95.4 |\n\nOur findings indicate the following:\n\n* CoT fine-tuning noticeably augments the ICL performance when applied to Date Understanding (reasoning task).\n\n* However, this enhancement isn't ubiquitously evident. Particularly, Sports Understanding (knowledge-intensive task) don't consistently benefit from CoT fine-tuning.\n\n* Comparative Performance: Notably, CoT-T5's performance is in line with some of the more advanced, proprietary LMs available in the current landscape.\n\nConsidering these outcomes, an intriguing avenue for future research lies in evaluating the potential of CoT fine-tuning to amplify the few-shot CoT prompting capabilities.\n\n\n*References*\n\n[1] Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H.W., Tay, Y., Zhou, D., Le, Q.V., Zoph, B., Wei, J., & Roberts, A. (2023). The Flan Collection: Designing Data and Methods for Effective Instruction Tuning. International Conference on Machine Learning.\n\n[2] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S.S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Valter, D., Narang, S., Mishra, G., Yu, A.W., Zhao, V., Huang, Y., Dai, A.M., Yu, H., Petrov, S., Chi, E.H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q.V., & Wei, J. (2022). Scaling Instruction-Finetuned Language Models. ArXiv, abs/2210.11416.\n\n[3] Ye, S., Kim, D., Jang, J., Shin, J., & Seo, M. (2022, September). Guess the instruction! flipped learning makes language models stronger zero-shot learners. In The Eleventh International Conference on Learning Representations.\n\n[4] Iyer, S., Lin, X., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P.S., Li, X., O'Horo, B., Pereyra, G., Wang, J., Dewan, C., Celikyilmaz, A., Zettlemoyer, L., & Stoyanov, V. (2022). OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization. ArXiv, abs/2212.12017.\n\n[5] Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., ... & Wei, J. (2022, September). Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations.\n\n[6] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824-24837.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. CoT Collection dataset is an important contribution. It will allow academics to conduct research on chain-of-thought and LMs without relying on commercial LMs.\n2. The paper also presents few good insights - a) CoT fine tuning can improve model performance for direct evaluation as well b) CoT fine tuned models complement parameter efficient FT like Lora in low resource setting.", "reasons_to_reject": "The paper points out 9 other publicly available datasets with rationale. The paper lacks discussion on the size/nature/quality of these datasets. Further, the paper does not compare effectiveness of CoT Collection dataset against other datasets. For instance, comparing performance of a CoT model trained on CoT Collection and one or more of the other datasets would have been helpful.", "questions_to_authors": "1. In multi-lingual evaluation, authors fine-tune CoT models using the data from the target language. Did authors run an experiment where CoT T5 is trained on data from all the target languages combined? Did authors observe any cross language transfer?\n2. Did authors evaluate CoT-T5 models in ICL setting? How do they fare compared for commercial models?\n3. Have authors checked for what datasize do the CoT T5 models start outperforming Flan-T5?", "paper_topic_and_main_contributions": "Problem: Equipping smaller language models with the chain-of-thought reasoning capabilities.\n\nMain Idea: Curate large CoT style dataset and CoT fine-tune small LLMs on that dataset\n\nDetails\n1. Authors curate and release a large dataset called CoT Collection. The dataset is of the form (instruction, instance, rationale, output).\n2. Authors fine-tune Flan-T5 models on the CoT Collection to obtain CoT-T5 models. They then show that CoT-T5 models are superior to Flan-T5 models in zero and few-shot setting.\n3. CoT Collection is created by prompting a LLM with (instance, output) to generate a rationale. Rationales are collected for 1k+ tasks with each task assigned a task group. Demonstrations for each task group are used to prompt Codex LLM to generate the rationale for the tasks in that group.\n4. Authors benchmark CoT-T5 models on BBH benchmark and compare them with models of similar sizes and commercial GPTs. Authors also do an ablation study and experiment across five different languages.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YiAspk3mxP", "reviewer": "Reviewer_cTkT", "summary": "", "questions": "1) As Codex is deprecated and the CoT dataset it generated is quite large, How to find some free and readily available API options is important for future research.\n\n2) Codex generates the CoT dataset. However, Table 3 in this paper shows the performance of other closed-source LLMs like GPT-3, Chatgpt, or GPT-4, instead of Codex's performance. Could this paper provide the difference in the performance between teacher and student models?\n\n3) Codex is well-known for generating code or code-style CoT. However, Most instruction tasks are non-code related. Is the style of the generation by student models taught by Codex not human-like and very different from that taught by ChatGPT? Evaluating the generations from various perspectives, including diversity, fluency, etc., remains crucial and meaningful.\n\n4) Could you clarify why the Direct evaluation results in table 2&3 are generally higher than the CoT evaluation results for the same model?\n\nI will update the final score according to the authors responses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. Our responses for your issue from “Reasons To Reject” and 4 questions from “Questions For The Authors'' are each denoted as (R1) and (Q1)-(Q4).\n\n**(R1,Q1) Concerns regarding Reproducibility**  \nWe agree with your concern regarding the reproducibility since the teacher model we used for our paper (Codex) was deprecated in June of 2023. One of the main reasons for choosing Codex over ChatGPT or GPT4 was because it was available for free at the time.\n\nNonetheless, we believe that our rationale augmentation process is not confined to Codex only, as later work such as [1] showed that using stronger teacher LMs such as GPT-4 leads to better performance on BigBench Hard. Additionally, we have released our dataset and checkpoints publicly available for future research as well.\n\nMoreover, please refer to Figure 4 in the appendix, where we used Roscoe, a suite of metrics to show that other proprietary LMs such as Bard and Claude show competitive statistics in terms of faithfulness, coherence, and informativeness of the generated CoT rationales.\n\n**(Q2) Performance of the teacher model (Codex)**  \nThe direct evaluation performance of Codex is 55.74, which is between ChatGPT(48.90) and GPT-4(67.40) from Table 2. We will add this result to the camera ready version of the paper.\n\n**(Q3) Concerns on Codex generation format**  \nWhereas Codex was trained to specialize on code domain, it can solve different reasoning tasks by writing general chain-of-thoughts, not only code-style CoT. This is supported in the results of the [2,3] By providing ICL demonstrations written in language, we observed that Codex also generates natural CoT rationales.\n\nAdditionally, we agree that thorough evaluation of CoT rationales is essential. We are keen to explore improved evaluation methods and understand their potential benefits in the future.\n\n**(Q4) Clarification on the tendency of evaluation results**  \nWe assume the overall dominance of Direct over CoT evaluation results comes from the fact that the volume of Flan Collection is 8 times larger than that of CoT Collection. The reduced gap between Direct and CoT of CoT-T5 compared to Flan-T5 shows that dominance of Direct evaluation can be treated by using sufficient rational data, which is what CoT Collection aims to provide. Our findings from Table 2 show that training only on CoT data could surprisingly improve the Direct evaluation performance as well, yet it is somehow ambiguous when and how positive transfer might happen between Direct evaluation and CoT evaluation.\n\nConsidering the task diversity and large volume of CoT Collection compared to previous CoT datasets, we hope it could act as a stepping stone for future research exploring the impact of further expansion of rationales.\n\n*References*\n\n[1] Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., & Awadallah, A.H. (2023). Orca: Progressive Learning from Complex Explanation Traces of GPT-4. ArXiv, abs/2306.02707.\n\n[2] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824-24837.\n\n[3] Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022, September). Automatic Chain of Thought Prompting in Large Language Models. In The Eleventh International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The research topic of instruction tuning with CoT over various tasks is both new and interesting. \n\n2) The experiment is extensive and sound.\n\n3) The paper is well-written and well-organized.\n", "reasons_to_reject": "1) This paper discusses the use of OpenAI's Codex for generating CoT, but the Codex API is now deprecated. Creating CoT data using the accessible LLM APIs would be better for ensuring reproducibility.", "questions_to_authors": "1) As Codex is deprecated and the CoT dataset it generated is quite large, How to find some free and readily available API options is important for future research.\n\n2) Codex generates the CoT dataset. However, Table 3 in this paper shows the performance of other closed-source LLMs like GPT-3, Chatgpt, or GPT-4, instead of Codex's performance. Could this paper provide the difference in the performance between teacher and student models?\n\n3) Codex is well-known for generating code or code-style CoT. However, Most instruction tasks are non-code related. Is the style of the generation by student models taught by Codex not human-like and very different from that taught by ChatGPT? Evaluating the generations from various perspectives, including diversity, fluency, etc., remains crucial and meaningful.\n\n4) Could you clarify why the Direct evaluation results in table 2&3 are generally higher than the CoT evaluation results for the same model?\n\nI will update the final score according to the authors responses.", "paper_topic_and_main_contributions": "The paper presents a new approach to improving the chain-of-thought (CoT) reasoning capabilities of smaller language models (LMs) by using instruction tuning with CoT rationales. To achieve this, the authors introduce a new dataset called the COT COLLECTION, which enhances the existing Flan Collection with additional rationales across various tasks. The authors demonstrate that fine-tuning smaller LMs with COT COLLECTION improves their CoT capabilities on unseen tasks. Additionally, the proposed instruction tuning approach shows promising results in few-shot learning on domain-specific tasks, even outperforming ChatGPT using demonstrations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "6p6eJ680UI", "reviewer": "Reviewer_qbB3", "summary": "", "questions": "1. In practice, we notice that given the manually designed ICL demonstrations to LLMs and with the ground truth label concatenated, the LLMs still tend to generate rationales with incorrect intermediate inference steps, while keeping the ground truth answer as the final output. Will these bad examples affect the fine-tuning of the model, and how to ensure that the examples augmented by LLMs are completely correct?\n\n2. See reason to reject 2\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. **(R1) - (R2)** is our responses to 2 issues from “Reasons To Reject” and **(Q1) - (Q2)** answer the 2 questions from “Questions For The Authors”.\n\n**(R1 & Q1) Cost & Quality of Production**  \n**Cost** Existing Chain of Thought (CoT) datasets, although instrumental, present limitations in terms of their scale and diversity. 9 previously available CoT datasets incorporate rationales penned by human annotators requiring a very costly and timely process [1]. Recognizing these constraints, we curate a dataset that not only increments sheer volume and diversity but also crafted with data efficiency compared to previous convention of rationale acquisition through human annotation.\n\nTo address both the efficacy of the CoT Collection training and its comparative advantages relative to the 9 original CoT datasets used in Flan-T5's training, we report supplementary ablation studies. These experiments sought to evaluate zero-shot generalization capabilities under varying conditions of training instances and tasks.\n\n\n| Dataset          | Number of Tasks | Number of Instances | BBH Performance |\n|------------------|-----------------|---------------------|-----------------|\n| 9 CoT Datasets   | 9               | 180K                | 40.33           |\n| CoT Collection*  | 1060            | 10K                 | 42.04           |\n| CoT Collection** | 1060            | 100K                | 44.76           |\n| CoT Collection   | 1060            | 1.84M               | 47.60           |\n\n\n* The BBH Performance denotes the mean accuracy derived from the 23 classification datasets of BBH, as detailed in Table 2.\n* From the CoT Collection, instances were sampled in two quantities: 10K(*) and 100K(**) instances.\n* All experiments adhered to the CoT Evaluation setting.\n\nThe results clearly illustrate that the enhancement in the number of tasks is the key factor for achieving robust generalization capabilities during the CoT Evaluation. Such observations are consistent with the discoveries highlighted in previous studies [2,3,4,5]. The CoT Collection's efficacy in strengthening a language model's zero-shot generalization capabilities, which was possible due to the efficient rationale collection process compared to previous human annotation methods.\n\n\n**Quality** Thank you for raising the valid concern regarding the potential presence of noisy chain of thought rationales in our augmented data set. We also acknowledge the importance of ensuring the quality and veracity of the rationales.\n\nTo provide a bit more context, while the papers you referenced highlights that \"the rationales might be incorrect even if the prediction is correct\", our method has a slight difference. As you've pointed out in the \"Questions for Authors\" section, we incorporate the answer as an input during the CoT augmentation process. This procedure, though not flawless, was conceived to somewhat mitigate potential discrepancies between rationales and predictions.\n\nAutomatic filtering presents significant challenges, primarily because determining the \"correctness\" or \"factuality\" within contextually-rich rationales isn't always straightforward. This issue intertwines with another pressing challenge: the adequate evaluation of such data.\n\nWe genuinely value your insights and, recognizing the gravity of the concern, believe that devising methodologies to further refine and filter rationales merits significant attention and exploration in future endeavors.\n\n**(R2 & Q2) In-domain evaluation after CoT fine-tuning**\n\nBased on your suggestions, we have measured the in-domain performance using the validation split of 5 datasets (ANLI R1, ANLI R2, ANLI R3, RTE, Winogrande), which were included as training data for both Flan-T5 and CoT-T5. \n\n| **Model** | ANLI R1 | ANLI R2 | ANLI R3 | RTE   | Winogrande |\n|-----------|---------|---------|---------|-------|------------|\n| Flan-T5-11B   | 44.60    | 41.07   | 42.96   | 75.23 | 59.53      |\n| CoT-T5-11B    | 57.73   | 50.29   | 50.46   | 85.96 | 81.28      |\n\n\nNote that the evaluation was conducted using CoT Evaluation. As shown in the table above, additional CoT training significantly improves the in-domain performance. We thank you for your insight and will include the table in the camera ready version.\n\n\n*References*\n\n[1] Kim, S., Joo, S. J., Jang, Y., Chae, H., & Yeo, J. (2023, May). CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 195-208).\n\n[2] Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H.W., Tay, Y., Zhou, D., Le, Q.V., Zoph, B., Wei, J., & Roberts, A. (2023). The Flan Collection: Designing Data and Methods for Effective Instruction Tuning. International Conference on Machine Learning.\n\n[3] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S.S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Valter, D., Narang, S., Mishra, G., Yu, A.W., Zhao, V., Huang, Y., Dai, A.M., Yu, H., Petrov, S., Chi, E.H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q.V., & Wei, J. (2022). Scaling Instruction-Finetuned Language Models. ArXiv, abs/2210.11416.\n\n[4] Ye, S., Kim, D., Jang, J., Shin, J., & Seo, M. (2022, September). Guess the instruction! flipped learning makes language models stronger zero-shot learners. In The Eleventh International Conference on Learning Representations.\n\n[5] Iyer, S., Lin, X., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P.S., Li, X., O'Horo, B., Pereyra, G., Wang, J., Dewan, C., Celikyilmaz, A., Zettlemoyer, L., & Stoyanov, V. (2022). OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization. ArXiv, abs/2212.12017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper contributes a CoT dataset that can be used to improve the reasoning and generalization abilities of the language model. The process for constructing the dataset is also proposed.\n2. Extensive experiments on various PLMs demonstrate the effectiveness and correctness of the collected data, achieving notable improvement on multiple datasets.", "reasons_to_reject": "The production of such a collection is costly, as the diversity and volume of available data is huge. Besides, the quality of rationales augmentation is hard to guarantee even with the filer method, due to the poor performance on tasks where multi-step reasoning is required. Even though the final output is correct, LLMs can still generate some intermediate inference steps that are completely wrong. [1, 2]\n\nThere is no experiment to investigate in-domain/in-task performance after CoT training (i.e. training on dataset A and evaluating on this dataset). \n\n[1]  Scaling language models: Methods, analysis & insights from training gopher. https://arxiv.org/abs/2112.11446\n\n[2] MathPrompter: Mathematical Reasoning using Large Language Models. https://aclanthology.org/2023.acl-industry.4.pdf", "questions_to_authors": "1. In practice, we notice that given the manually designed ICL demonstrations to LLMs and with the ground truth label concatenated, the LLMs still tend to generate rationales with incorrect intermediate inference steps, while keeping the ground truth answer as the final output. Will these bad examples affect the fine-tuning of the model, and how to ensure that the examples augmented by LLMs are completely correct?\n\n2. See reason to reject 2\n", "paper_topic_and_main_contributions": "The paper proposes a new CoT collection training data, which enables fine-tuning language models with billions of parameters to acquire reasoning ability. The CoT collection is first generated with human annotators and then augmented using LLMs. Experimental results on various PLMs demonstrate that the collected data is effective and can improve the models' reasoning and generalization abilities.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers unanimously agree the CoT Collection offers a useful resource to practitioners. With the inclusion of additional experiments and clarifications on limitations, requested by the reviewers, this makes for a relevant and robust contribution.
Decision: Accept-Main