At Which Training Stage Does Code Data Help LLMs Reasoning?
Authors: YINGWEI MA
Yue Liu
Yue Yu
Yuanliang Zhang
Yu Jiang
Changjian Wang
Shanshan Li
Venue: ICLR 2024 spotlight (2024)
Keywords: code data
large language models
reasoning capabilities
Primary area: generative models

Abstract: Large Language models (LLMs) have exhibited remarkable reasoning capabilities and become the foundation of language technologies. Inspired by the great success of code data in training LLMs, we naturally wonder at which training stage introducing code data can really help LLMs reasoning. To this end, this paper systematically explores the impact of code data on LLMs at different stages. Concretely, we introduce the code data at the pre-training stage, instruction-tuning stage, and both of them, respectively. Then, the reasoning capability of LLMs is comprehensively and fairly evaluated via six reasoning tasks. We critically analyze the experimental results and provide conclusions with insights. First, pre-training LLMs with the mixture of code and text can significantly enhance LLMs' general reasoning capability almost without negative transfer on other tasks. Besides, at the instruction-tuning stage, code data endows LLMs the task-specific reasoning capability. Moreover, the dynamic mixing strategy of code and text data assists LLMs to learn reasoning capability step-by-step during training. These insights deepen the understanding of LLMs regarding reasoning ability for their application, such as scientific question answering, legal support, etc.

Published as a conference paper at ICLR 2024
AT WHICH TRAINING STAGE DOES CODE DATA HELP LLMs REASONING?
Yingwei Ma Yue Liu't, Yue Yu! Yuanliang Zhang', Yu Jiang?, Changjian Wang',
Shanshan Li
1 Nattioall University of Defense Technology
Peng Cheng Laboratory 3T Tsinghua University
ABSTRACT
Large Language Models (LLMs) have exhibited remarkable reasoning capabili- ties and become the foundation of language technologies. Inspired by the great success of code data in training LLMs, we naturally wonder at which training stage introducing code data can really help LLMs reasoning. To this end, this paper systematically explores the impact of code data on LLMs at different stages. Concretely, introduce the code data the pre-training stage, instruction-tuning stage, and both of them, respectively. Then, the reasoning capability of LLMs is comprehensively and fairly evaluated via six reasoning tasks five domains. We critically analyze the experimental results and provide conclusions with insights. First, pre-training LLMs with the mixture of code and text can significantly enhance LLMs' general reasoning capability almost without negative transfer on other tasks. Besides, at the instruction-tuning stage, code data endows LLMs the tpek-specific reasoning capability. Moreover, the dynamic mixing strategy of code and text data assists L.MSs to learn reasoning capability step-by-step during training. These insights deepen the understanding of LLMs regarding reasoning ability for their application, such as scientific question answering. legal support, etc.
INTRODUCTION
Recently, Large Language Models (LLMs) have achieved impressive generalization performance across various tasks. Significantly, OpenAl developed ChatGPT (2023a), Google designed PaLM[Chowdhery et al. (022). Baidu built ERNIE Bot Baidu (2023). ) and Alibaba presented Tongyi Qianwen Alibaba (2023). However, these industrial products regrettably not open-source for commercial reasons. Thanks to the surging open-source projects of LLMs such as LLaMA (Touvron)
al. 2023), Code LLama Roziere et (2023). Baichuan Yang al. (2023), Alpaca (Taori et al. 2023). and ChatGLM (Du et al. 2022a), the academic research and industrial products of LLMs mark new milestones.
Two of the key factors to the great success of LLMs are 1) training data and 2) training strategies. First, for the training data, researchers aim to endow LLMs with language capabilities and general knowledge via training models on large-scale data from various domains. For example, LLaMA was trained with 1.4 trillion tokens consisting of texts (CommonCrawl,, C4) and codes (GitHub). These large-scale data with diversity help the model to achieve competitive performance on multiple tasks. Sccond, the common pipeline goes through two stages for the training strategies: pre-training and instruction-tuminn tunning. The pre-traine tre-training is conducted in self-supervised manner the massive unlabeled data, while instruction-tumi and feedback (Ouyang al.) 2022). Benefiting from the data and training strategies, LLMS gain remarkable skills, such as translation, conversation, examination, legal support, etc. These skills are all based on of the most important capabilities, i.e., reasoning capability. So, how can LLMs gain such strong reasoning capability?
aims to fine-tune models with annottted prompts
"Co-first author.
'Corresoonding author.

---

Published as a conference paper at ICLR 2024
We analyze the reasons from two aspects: training data and strategies. First, from the training data aspect, compared with the common textual data, code data is more logical and less ambiguous (refer to case studies in Appendix F). Also, from the experiments, researchers (Liang et al., 2022 Fu et al., 2022) verified that models trained on code data have strong reasoning capability. Therefore, code data essential for model reasoning. Second, for the training strategies, both pre-training and fine-tuning are crucial to the model's s performance. Pre-training feeds general knowledge to models while fine-tuning feeds domain-specific ability to models. To further explore the deep-in reasons for the strong reasoning capability of LLMs, this paper aims to answer an important question: at which stage does code data help LLMs reasoning?
To this end, we conduct comprehensive and fair experiments and provide analyses and conclusions with insights. First, we pre-train L.LSs with pure text data and mixture data of code and text, respectively. Subsequently, at the instruction-tuning stage, LLMs are fine-tuned with the pure text data and mixture data of code and text, respectively. After training, to comprehensively measure the model reasoning capability, reasoning, code reasoning, legal reasoning, scientific reasoning, and analogical reasoning. Based on extensive experimental results and analyses, we provide three insights. 1) Pre- Pre-training LLMs with the mixture of code and text can significantly enhance LLMs' general reasoning capability almost without negative transfer on other tasks. 2) At the instruction-tuning stage, code data endows LLMs the task-specific reasoning capability. 3) The dynamic mixing strategy of code and text data assists LLMs to learn reasoning capability step-by-step during training. These findings deepen the understanding of LLMs regarding reasoning ability for their applications, such as scientific question answering, legal support, etc. The main contributions of this work are summarized as follows.
evaluate LLMs six tasks in five domains, including logical
Research question: this paper raises and aims to answer one essential concern, i.e., at which training stage codes data help LLMs reasoning.
Analyses and insights: conduct extensive experiments and provide critical analyses and insights, which deepen the understanding of LLMs regarding reasoning capability.
Open-source resource" we release the model implementation and the trained model param- eters, which contribute the further research in the L.MSs community.
1.1 TRAINING DATA & TRAINING STRATEGIES
Three key factors to the great success of LLMs are training data, training strategies. and model designs. In this section, we introduce our training data and training strategies. The next section details the model designs.
We study two training phases of LLMs, i.e., pre-training stage and instruction-tuning stage, on two different datasets including one plain text data and one text-code-mixed data. Figure demonstrates the process of each stage. Specifically, we use the open-sources PanGu2.6B and PanGu 13B of the PanGu-a team Zeng et al. (202T) baseline models for text models (trained on 100GB text data and larger text data, respectively), and train CodePanGu2.66 6B from scratch on the mixed code data for comparison. We will introduce detailed data settings in later chapters.
1.2 PRE- TRE-TRAINING CORPUS
The pre-training corpus consists of two parts. To ensure a fair comparison with PanGu2.66 we collected large amount of original data from public datasets such as BaiDuQA, CAIL2018, Sogou- CA, and network data sets such as Common Crawl, encyclopedias, news, and e-books according to the PanGu-a team (Zeng et al. 2021). Then we use based data cleaning and model-based data filtering methods to filter to ensure high quality. Finally, we obtain 100GB of text data with the same scale and source PanGu2.66 2.6B by sampling each data source using different ratios. Please refer to Appendix G for a detailed data processing process. To verify the influence of code data on the reasoning capability of the model in the pre- pre-training stage, we used the CodeParrot (Huggingface 2023) dataset as the second supplementary part. CodeParrot is a public Python dataset from BigQuery, comprising approximately 50GB of code and 5,361,373 files. Figure shows the composition of the 442B tokens in pre-training data.
mtttps://get https:://githbbe http:///thhuuu.ccoyyineeeeeeeeee http.///yinngeeemme22/////llllle
2

---

Published as a conference paper at ICLR 2024
Pre-training Stage
Instruction-tuning Stage
This work studies the extent to which L... You can also create your own copy [..]
Instruction: the|..]Jkn Inputtin
Instruclion''nLook [..]kn In..ts="....In
Provide short description of your [..]
[..]Jn
Inputin
with or without code-related data: def floor_ Min(A,B B,Ny [..] neturn A [..]] remove elementjiist,, return [..] def is_ _uppp((strring) return (string upper
with or without code-related instruction:
Instruction Createe anj..]Jnn Inputinn
Instruction'nnWWit [-.]lo In..tss(...jvn
Instruction:WWit cless|-.JJn [p..)
Language Model
Language Model
next token
aulput
Figure 1: Demonstration of the pre-training and tuning phase.
Common Crawl
e-Books
13.5%
CodeParrot
36.1%
News 13.5%
14%
16.9%
Encyclopedia
Public Datasets
Figure 2: Distribution of the 42B tokens in pre-training data.
1.3 INSTRUCTION-TUNTIE CORPUS
We collect and construct 400K instruction tuning data verify the effect of adding code instructions in the instruction tuning stage and convert them into unified instruction format. The instruction tuning corpus is divided into two parts. The first part is from the natural language open source instruction dataset, Alpaca-GPT-4 GPT-4 (Peng et al.) 2023) and PromptCLUE (pCLUE team 2022). Alpaca-GPT-4 is generated by GPT-4. including 52K Chinese and English instruction tuning data. PrompiCLUE unifies the differences between different NIP tasks (e.g., reading comprehension, question answering) and converts the original task training set into unified text-to-text data form, from which we randomly sample 200K data for instruction tuning.
The second part comes from the open-source data CodeAlpaca (Chaudhary) 2023) and our build dataset, with 150K instructions. The CodeAlpaca data contains 20K instruction tuning data generated according to the self-instruct technology, which can be used for instruction tuning the code generation model. In order to supplement the code-related instruction tuning data, CosQA (Huang et al. (02T) training set and the MBPP (Austin et al.) 2021) training set to unify the task format in the way of PromptCI.UF and expand the CodeAlpace data. Figure 3 is example of the format of instruction tuning data.
use the
MODEL
We conduct experiments large-scale autoregressive language models by adopting the GPT paradigm(Brown et al.) 2020). iteratively takes all tokens in the corpus as input, predicts the next token, and compares the ground truth. Assuming that sequence X' = {211 32, -. Fa is

---

Published as a conference paper at ICLR 2024
instruction (NL)
instruction (CODE)
What DNA stand for?
Write function to find the number of distinct states in given matrix.
input:
output:
input: matrix = [[1, 0, OJ, (1. 0, 11. 1, 1B output:
DNA stands Deoxyribonucleic Acid. [..]] and reproduction of all living organisms.
def find_ num_ distinddstitt_tao (tatti [...]
Figure 3: Example of the instruction tuning data format. NL denotes natural language.
composed of tokens, the training objective can be formulated as maximization the log- lig-liielhhood:
log
where p(2,/21, 22. ..,, - θ) is the probability of observing the i-th token 2; given the previous context 21, Z2: ... T;-1, and è denotes the model parameters.
2.1
MODEL ARCHITECTURE
Similar to recent pre-trained models such GPT-3 Brown et al. (2020), LLaMA Touvron et al. 2023), CodeGeeX (Zheng et al. 2023), and PANGU-a Zeng et al. 2021). we follow a generative pre- training (GPT) architecture for autoregressive language modeling. At the same time, to make fair comparison with the baseline of PanGu2.6B, we retain the setting of the 32-layer transformer decoder. The original GPT model uses pooler function obtain the final output. Follow CodeGeeX (Zheng et al., 2023) and PANGU-a (Zeng et al.) 2021), we use an additional query layer on top of the stacked Transformer layers explicitly induce the expected output with attention obtain the final embedding.
2.2 TOKENIZATION
For the text-only model, we use the open-source vocabulary of the PanGu2.6B model released by PanGu-a team (Zeng et al. 2021). and the size the vocabulary is 40,000. For the model training with mixed code, considering that there may be variables, functions, and class names in the code that are often meaningful words, we use the ChatGLM (Du et al. 2022b) vocabulary open-sourced by the THUGLM team to encode text and the code. The vocabulary size is 130,044. In addition, ChatGLM encodes multiple spaces as extra tokens to improve encoding efficiency. Specifically, L spaces are represented by <lextratoken XI>, where X=8+L. Both vocabularies BPE-based tokenizers, which use fixed-size vocabularies to handle variable-length characters in open-vocabulary problems.
EXPERIMENTS
3.1 TASK DESCRIPTION
To measure the reasoning ability of the models, we evaluate on six tasks in realistic reason- reasrn-centric scenarios, including general reasoning scenarios such logical reasoning, legal reasoning, scientific reasoning, and analogical reasoning, and code-related scenarios such as code generation. These reasoning-intense tasks elucidate the reasoning capabilities of the model through the model's performance in these scenarios. When publicly available, we evaluate the models with the test sets for each task. Otherwise,
the development sets instead. We describe each task follows.
Logical Reasoning. Logic is the study of reasoning and argumentation, which focuses on the rules of logic and methods of reasoning in the thinking process. We use the logic subject in the C-Eval dataset (Huang et al.) 2023) to determine whether the model can understand and apply logical rules make reasonable reasoning.
4

---

Published as a conference paper at ICLR 2024
Task Type
Dataset
Input & Prompt
Logical
Logic
The answer: Schoice, answer the following questions: Sproblem
Legal
JEC-QA
The answer: Schoice, can answer the following questions: Sproblem
Scientific
ScienceQA
Slecturein anwser: Schoice can answer the following question: Squestion
Analogical
E-KAR
The reasoning relationship: Srl, the analogy reasoning relationship: Sr2
Code
CosQA
Squestion? Answered code correct or wrong: Scode
Code
MBPP
Squestion'n Code:An
Table 1: The input & prompt template for each task. is the input and other words prompt.
Legal Reasoning, For legal reasoning, we use JEC-QA (Zhong et al. 2020), the largest question answering dataset in the legal domain, collected from the National Judicial Examination of China. The examination is a comprehensive evaluation of the professional skills of legal practitioners. Multiple reasoning skills required to retrieve relevant material and answer legal questions.
Scientific Reasoning, We use the ScienceQA dataset (Lu et al. 2022) to evaluate the scientific reasoning ability of the model. The scientific question answering task can diagnose whether the artificial intelligence model has multi-step reasoning ability and interpretability. To answer scientific questions from ScienceQA, model not only needs to understand multimodal content but also needs to extract external knowledge to arrive at the correct answer.
Analogical Reasoning. We use the E-KAR dataset Chen et 2022) evaluate the model's analogical reasoning ability. It comes from the Civil Service Examination, comprehensive test of the candidate's critical thinking and problem-solving ability. To solve the analogy reasoning problem, candidates need to understand the relationship among the options, which requires specific reasoning ability and background knowledge, especially common sense and facts, and knowing why a fact denied.
We use CosQA (Huang et al. 2021) test the model performance on the code question-answering task. The dataset includes 604 natural language-code question-answer pairs.
Code Reasoning.
Furthermore, we use the MBPP dataset (Austin et al. 2021) 1) to test the model code generation ability, containing 427 Python coding questions.
3.2 EVALUATION DETAILS
In evaluation, these tasks are usually divided into two parts, understanding task and generation task. For the understanding task, we follow PanGu2.6B (Zeng al., 2021) and CPM Zhang al. (202I), decomposing the task into perplexity comparison task. We construct prompt template for each evaluation task and populate the template with instances as input to the model. Table [ describes the templates for each task.
We adopt a perplexity- based approach solve classification tasks. For each <text, label> pair, input will be automatically generated according to the predesigned prompt in Table [ The sequences generated by the prompt will be fed into the model, and a perplexity value will be calculated. The label corresponding to the minimum perplexity value will be regarded the predicted label for this passage. For the generative task, we leverage the properties of autoregressive language models to generate corresponding answers directly from a given input naturally.
3.3 RESULTS
3.3.
PRE-TRAINING STAGE
To illustrate the impact of code data in the pre-training phase on the reasoning capabilities of large language models, we compared the performance of the three models real reasoning-intensive scenarios. Among them, the NL (2.6B) and NL (13B) (i.e., PanGu2.66 6B and PanGul3B) models (Zeng) al. 2021) are trained on natural language datasets, and the CODE (2.6B) (i.e., CodePangu2..0 model trained mixed data (the dataset mentioned in Chapter 1.2). The models are evaluated in zero-shot manner on downstream tasks. Specifically, we report accuracy on for Logic, JEC-QA, ScienceQA, E--AR, and CosQA tasks and BLEU score for MBPP task. In order to illustrate the
5

---

Published as a conference paper at ICLR 2024
Dataset
Task
Metric
NL (2.6B)
NL (13B)
CODE (2.6B)
p-value
Logic
Logical Reasoning
ACC
36.36
45,45
40.90
4.197c-06
JEC-QA"
Legal QA
ACC
27.00
27.00
28.70
1.956e-25
ScienceQA
Scientific QA
ACC
45.93
45.18
46.06
0.014
E-KAR"
Analogical Reasoning
ACC
32.24
35,52
36.12
7.013e-07
CosQAI
Code QA
ACC
47.02
46.85
50.50
1.066e-40
MBPPi
Code Generation
BLEU
0.52
1.34
5.06
Table 2: Results on pre-training stage. Bold values indicate the best performance. general reasoning task, and i denote the code-related reasoning task.
denote the
significance of these results, we conducted a t-test on the predicted scores, It demonstrates that all p-values are less than 0.05, indicating that the results are statistically significant. Table depicts the results of these tasks. Consistently over these tasks, we have two observations as follows.
After adding code training. LLM performs better on most reasoning-related tasks, even though most of these tasks are not related to code. This shows that adding code data in the pre-training stage can not only improve the coding-related ability but also improve the general language reasoning ability of the model to certain extent.
Even with larger scale model, i.e., NL (13B), it is still not as effective as CODE (2.6B) in these reasoning scenarios. This similar to the results of HELM Liang et al. (002), which suggest that if (a) the computational budget is constrained and (b) the resulting model is applied in the code reasoning domain, adding code data in the pre-training phase may be more effective than increasing the model parameter size.
In summary, we find that simply adding code data during the pre-training phase can effectively improve the model's general reasoning ability, which might indicate that mixing more code data for training may produce a competitive model to solve tasks that require complex reasoning to complete. This provides promising prospect for subsequent LLM development.
3.3.2 INSTRUCTION STAGE
ChatGPT OpenAi 2023a) and GPT-4 | 2023b) successfully use instruction tuning to enable LLMs to follow natural language instructions and complete real-world tasks; this improvement has become standard in open-source LLMs. This is implemented by tne-ttnnng the model wide range of tasks using human-annotate instructions and feedback, by supervised fin-tuning via manually automatically generated instructions using public benchmarks and datasets, learning from instruction-follooinn data by developing from state-of-the-aar instruction-uus teacher L.MMs.
To illustrate the impact of code data the LLMs reasoning ability in the instruction tuning stage, we use the instruction tuning datasets that contain codes and the instruction tuning datasets without codes introduced in Chapter 1.3 to fine-tune the PanGu2.6B and PanGul3B models (Zeng al. 202T) and evaluate their performance in reasoning-intensivv scenarios. In addition, we also fine-tune the CodePanGu2.66i model using the instruction tuning dataset containing codes to observe the effect of using code data in both pre-training and instruction tuning stages. Table Ishoos the results of these tasks. Among them, NN and NC represent the fine-tuned PanGu model using only text instructions and instructions containing codes, respectively, and CC represents the fine-tuning model of CodePanGu2.6B using instructions containing codes. Consistently over these tasks, we observe the following:
After fine-tuning with mixed code instruction data, LLM shows different trends in multiple reasoning tasks. This indicates that introducing code data in the instruction tuning phase may be less effective than in the pre-training phase. Therefore, it is best to add code data in the pre-training stage improve the model performance in general reasoning tasks.
We find that training with code data in both stages can significantly improve code-related tasks (CosQA and MBPP), especially code generation tasks. This may be because the code instruction data activates the code reasoning ability of the language model, which suggests that if the LLM needs to complete complex code tasks, the code reasoning ability can be improved by effectively following code instructions and generating compliant content.

---

Published as a conference paper at ICLR 2024
Dataset
NN (2.6B)
NC (2.6B)
NN (13B)
NC (13B)
CC (2.6B)
Logic
36.36
40.90
40.90
40.90
40.90
JEC-QA
25.20
26.10
24.50
26.40
27.10
ScienceQA*
44.45
43.44
42.94
43.41
41.90
E-KAR 30.45
28.66
26.27
27.46
27.20
CosQAt 45.20
48.18
47.52
51.99
52.48
MBPP
0.00
5.61
0.00
1.88
24.88
Table 3: Results instruction-tuning stage. Bold values indicate the best performance. denote the general reasoning task, and denote the code-related reasoning task.
Dataset
NL (2.6B)
CODE (2.6B)
NL Co.6B)+CoT CODE (2.6B)+CoT
ScienceQA
45.93
46.06
68.76
70.30
E-KAR
32.24
36.12
69.55
72.84
Table 4: Results with Shain-of-Thoughe prompt. Bold values denote the best results.
Compared with the pre-training stage, the performance of instruction-tunee LLMS some tasks is degraded, similar to the TÚLU (Wang et al. 2023) results. This may be because the instruction tuning data usually covers wide range of domains and dialogue content, causing the model to tend to answer questions more comprehensively, resulting in decline in reasoning ability. We propose that specific reasoning capabilities are required, they can be augmented by adding domain-specinn instructions during the tuning phase.
In summary, we find that adding code data in the instruction tuning stage is pre-training stage in improving the general reasoning ability of the model. However, we find that code instructions made the model follow natural language instructions and generate correct code, improving the model's code reasoning ability. This also suggests that tuning with relevant data may be helpful when solving specific reasoning tasks.
effective as the
3.3.3 CHAIN-OF-THOUGHT ABILITY
Compared with the standard prompt technology, Ofain-of-Thoughe (CoT) (Wei et al. 2022) trans- forms tasks into continuous chain generation process. This technology enhances the model ability in complex reasoning tasks by providing a language model with series of related reasoning steps. To evaluate the potential of the model in utilizing chains of thought in solving complex problems, we conduct experiments on two pre- tre-trained models, NL (2.6B), i.e., PanGu2.6B and CODE (2.6B). i... CodePanGu2.66 on ScienceQA(CoT) (Lu et al. 2022) and E-KAR(COT) (Chen et al.) 2022) datasets. We incorporate CoT information part of the model input with the question and context information. In this way, the model can directly use the reasoning process of the thinking chain for answer generation. The experimental results are shown in Table 4
The experimental results show that after the introduction the Mfain-of-Thoughhe the performance of all models in reasoning problems is significantly improved by making full of the coherent reasoning process COT. The CoT information is used as part of the model input to help the model better understand the problem and generate answers according the logic of the COT. Among them, CODE (2.6B) achieved the best performance, indicating that CODE (2.6B) can better use CoT information for reasoning. This also suggests that pre-training with mixed-code data may result in competitive model for tasks that require complex reasoning.
3.3.4 EXPLORING WAYS TO MIX CODE AND TEXT DATA
Previous experiments have demonstrated that training with mixed code data the two stages of pre-training and instruction tuning can improve the general and specific reasoning capabilities of
7

---

Published as a conference paper at ICLR 2024
LLMs, respectively. Therefore, We naturally wonder how mixing these two types of data can better improve model reasoning ability, which has not been explored in previous studies. Therefore, we design comparative experiments in the instruction tuning stage to verify the impact of different data mixing strategies. The mixed strategy shown in Table 5
One group is uniform sampling, that is, the proportion of text and code in each group of training data is roughly the same; the other two groups gradually increase or decrease the proportion of code to verify whether step-by-step learning will better activate the reasoning ability of LLMs. The experimental results are shown in Table 6
Phase
Uniform Sampling
Stepwise Increase
Stepwise Decrease
5:3
7:3
5:5
2
5:3
7:3
6:4
3
5:3
6:4
7:3
4
5:3
5:5
7:3
Table 5: Mixture strategies on text data and code data with different ratios (text:code).
Dataset
Uniform Sampling
Stepwise Increase
Stepwise Decrease
Logic
31.82
36.36
40.90
JEC-QA
27.30
26.70
27.10
ScienceQA
43.76
43.19
41.90
E-KAR
28.66
28.36
27.46
CosQAt
51.65
50.66
52.48
MBPP
23.68
23.42
24.88
Table 6: Result of different mixed strategies. Bold values indicate the best performance. denote the general reasoning task, and denote the code-related reasoning task.
The experiment showed that the training strategy of using a higher code data ratio in the early stage and gradually reducing the code data ratio in the later stage achieved the best results in code question answering (CosQA) and code generation (MBPP) tasks, while ensuring the performance of the model other reasoning tasks. This may be because, due to the strong logic the code, using more code data in the early stage may help the model activate the code reasoning ability faster. Therefore, if LLMs are expected to have better specific reasoning ability, adopting stepwise descent strategy can better activate the model potential. In addition, since experiments in the pre-training phase require lot of resources, we leave the validation of this phase to later work.
3.3.5 OTHER TASKS
We extensively evaluates various reasoning tasks, including logical and code reasoning, highlighting the positive impact of code-related data. Additionally, we sought to ascertain whether code data would affect common-sense tasks. Therefore, verify the impact of code data other comprehension and generation tasks that are less demanding on reasoning, we conduct experiments on other tasks, including two NLI tasks (OCNLI (Hu et al. (2200) and CMNLI (Wang al. (0188), requiring the model to identify the relationship between two sentences, either entailment, neutral or contradiction: free-form multiple-choice Chinese machine reading comprehension dataset (C3) (Sun al. 2020
Dataset
Metrics
without code
with code
C
ACC
54.14
54.30
OCNLI
ACC
41.69
40.50
CMNLI
ACC
45.07
43.49
EM
0.42
0.14
DuReader
15.29
8.73
Table 7: Results of pre-training on other tasks. Bold values indicate the best performance.
8

---

Published as a conference paper at ICLR 2024
Dataset
Metries
without code
with code
C
ACC
55.07
54.47
OCNLI
ACC
40.78
41.19
CMNLI
ACC
44.82
45.49
EM
12.07
8.05
DuReader
F1
34.85
25.05
Table 8: Results of instruction-tuning on other tasks. Bold values indicate the best performance.
consisting of documents (conversational or more formal mixed-type text) and their associated multiple- choice free-form questions; one reading comprehension task duReader (He al. (017). requiring the model to extract a text span from a given paragraph as the correct answer to the question. Refer to Appendix for prompt templates and evaluation metries for different tasks.
Table and Table B the results of adding code data in the pre-training phase and adding code instructions in the instruction tuning phase (only in this phase). Experimental results show that, in most cases, using code data for training may negatively impact the performance of other tasks. In the DuReader reading comprehension task, part of the performance will be reduced after adding code at different stages. This may be because the model does not thoroughly learn the code and text data, resulting in confusion when the model generates answers to reading comprehension questions. In future, we will verify and solve it in a larger model and with larger data.
4 RELATED WORK
LLM training. LLMs are usually based on the transformer architecture (Dai et al. 2019). Notable models include BERT (Devlin et al. 2018). GPT-2 (Radford et al. 2019), and T5 (Raffel et al. 2020);
after the emergence of GPT-3 (Brown al. 2020) with 175B parameters, batch of larger models emerged, including PaLM (Chowdhery et al. 2022]. OPT (Zhang et al. 2022). PanGu-a (Zeng
al. 2021). and LLaMA (Touvron et al. 2023), which have achieved remarkable results various NLP tasks. For LLMs to follow instruction output, instruction tuning (Peng et al. (002)) plays an
important role. This can human-annotated feedback (Ouyang et al., (022) or public benchmarks
to automatically generate instructions (Wang et al. 2022 to fine-tune models on various tasks.
Model Evaluation. HELM Lian et al. 2022) provides large-scale evaluation of existing LLMs. They found that the code-cushman-001 model trained on code data had more reasoning power than other natural language models. In addition, some work analyzed the reasoning ability of LLMs through the way of CoT and found that because the original GPT-3 had not been subjected the code training. it could not do CoT reasoning Wei al.] (2022). The PaLM training data contains 5% of the code training data, which shows that PaLM can effectively perform CoT reasoning. However, in the above work, the evaluated models have different parameters, data sizes, and unknown training details. Therefore, it is not rigorous to speculate the exact impact of code data reasoning ability only by comparing existing models.
CONCLUSION
In this paper, we investigate at which stage introducing code data can help improve the reasoning ability of LLMs. We validate the effect of code data at different stages with the same parameter scale and using the same training objective. We point out that simply adding code data in the pre-training phase can effectively improve the general reasoning ability of the model. Furthermore, we find that adding code instructions in the instruction tuning stage can make the model follow human instructions point that the dynamic
for output and improve specific code reasoning capabilities. Moreover, mixing strategy of code and text data assists LLMs in lcarning reasoning capability step-by-step during the training process. We provide a well-designed and tested reference implementation for LLMS training to help researchers and developers better understand and analyze LLMs. Models such as ChatGPT illustrate that larger-scale models will produce emergent capabilities. Therefore, the follow-up of this paper will study how to code the impact of data on models of different sizes and explore the relationship between code data and emergent capabilities.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was funded by NSFC No. No.62272473, the Science and Technology Innovation Program of Hunan Province (No. (03.2023)))))) and NSFC No.62202474.
REFERENCES
Alibaba. Tongyi qianwen. https::/tongyi.allii/ com/ 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint 1IXXx:22008007322 2021.
Baidu. Ernie bot. https://yiyan. http://yyyyaa..aauu.uu
2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Sahil Chaudhary. Code alpaca: An cesaructiionfolloviiii llama model for code generation. https //github. www.sahii/acccccccccocc 2801 14 4/ codeaapaac 2023.
Jiangji Chen, Rui Xu, Ziquan Fu, Wei Shi, Zhongqiao Li, Xinbo Zhang, Changzhi Sun, Lei Li, Yanghua Xiao, and Hao Zhou. E-KAR: A benchmark for rationalizing natural language analogical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3941-3955, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https://aclanthologr orgooogg/g/o 2022. wrinniinnnaacl... ac1.311
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arviv preprint arXiv:2204.02311 2022.
Fenia Christopoulou, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhongqi Li, Qi Zhang, Meng Xiao, Bo Shen, Lin Li, al. Pangu-coder: Program synthesis with function-level language modeling. arXiv preprint arXiv:2207.1 12280, 2022.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901. 002860, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM:
general language model pretraining with autoregressive blank infilling. pp. 320-335, 2022a.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 320-335, 2022b.
Yao Fu, Peng Hao, and Tushar Khot. How does gpt obtain its ability? tracing emer- gent abilities of language models to their sources. Yao Fu's Notion, Dec 2022. URL notion. wtw.://www...ii.o.....eeeeee.eaa
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. Dureader: chinese machine reading comprehension dataset from real-world applications. arXiv preprint arXiv: 171 1et.111711.0.... 1.5073, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
10

---

Published as a conference paper at ICLR 2024
Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kuebler, and Larry Moss. Ocnli: Original chinese natural language inference. Findings of EMNLP, 2020. /
05444
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. CoSQA: 20,000+ web queries for code search and question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11 International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5690-5700 Online, August 2021. Association for Computational Linguistics. doi: 10. 11853/v1120222 acl--long.442.
URL https://aclantthllogg http:/hhoooooooogg/220/0 org/2021 acl- 1one.loon... 442
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-disciplin chinese evaluation suite for foundation models. arXiv preprint
arXiv2300.08832,, 2023.
Huggingface. codeparrot.
2023.
Peng Li, Tianxiang Sun, Qiong Tang. Hang Yan, Yuanbin Wu, Xuanjing Huang, and Xipeng Qiu. Codeie: Large code generation models better few-shot information extractors. arXiv preprint arXiv:2305.05711 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, al. Holistic evaluation of language models. arXiv preprint #rXiv:2211.09113 2022.
Jing Liu, Xinxin Zhu, Fei Liu, Longteng Guo, Zijia Zhao, Mingzhen Sun, Weining Wang, Hanqing Lu, Shiyu Zhou, Jiajun Zhang, al. Opt: Omni-perception pre-trainer for cross-modal understanding and generation. arXiv preprint arXiv:2107.0024) 2021.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurlPS), 2022.
OpenAl. Chatgpt. https:://opnnal.co///////////aa 2023a.
OpenAl. Gpt-4. https:///ppeaa..com//// 2023b.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
PCLUE team. pelue:large-scale pased dataset for multi-task and zero-shot learning in chinese. 2022. URL https://t/// ctt httpttttbb.ooo//tuueeeeeeearr//a
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5885-5511 2020.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension. Transactions of the Association for Computational Linguistics. 8:141-155, 2020.
11

---

Published as a conference paper at ICLR 2024
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Stanford alpaca: An essruction-folloviin llama model. 2023.
/ttps://githhn com/
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Tze Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201 088309 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07661, 2018.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaci, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions 1,600+ language tasks. arXiv preprint arXiv:2204.07705. 2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1 1903, 2022.
Aiyun Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, et al. Baichuan 2: Open large-scale language models. arXiv preprint MxXiv:2309.10305. 2023.
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang. Kaisheng Wang, Xiaoda Zhang, et al. Pangu-o: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint lxXvv22104.12366 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language 2022.
models. arXiv preprint a arXiv:2205.011
Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, al. Cpm: A large-scale generative chinese pre-trained language model. AI Open, 2:93-99, 2021.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. Jec-qa: A legal-domain question answering dataset. In Proceedings of AAAI, 2020.
A ADDITIONAL RELATED WORK
Data Mixtures. Models such as GPT-3 (Brown et al. 2020) and PanGu-a ZZeng et al. 2021) are trained on natural language data from various domains, and models such as LaMDA (Thoppilan et al.) 2022) and LLaMA (Touvron et al. 2023) are additionally trained on code data. However, the impact and specific origin of this mixed-code data is unclear. Some researchers have extensively analyzed the performance of current LLM on various tasks, pointing out that code may be the key to improving reasoning ability (Liang al. 2022 Fu et al. (022). However, the evaluated models have different parameters and data scales, and problems such as unknown training details exist. is difficult to determine the exact impact of code data on the reasoning ability of L.Mss.
12

---

Published as a conference paper at ICLR 2024
B
CODE REPRODUCTION
B.1 MINDSPORE VERSION
To ensure reproducibility, open-sourced the model training and inference code using the Mind- Spore framework in the repository link: https::/gittub..co http://www..cco///./e//////////a
B.2 INTRODUCTION TO MINDSPORE
We acknowledge that MindSpore is relatively newer and requires further development time. How- ever, we would also like to highlight the adoption of MindSpore by prominent researchers in the field. For instance, (Zheng et al. 2023) in their work on CodeGeeX, (Liu et al. 2021)'s 's OPT model and (Christopoulou et al. 2022) in the release PanGuCoder have both embraced the hardwre combination offered by MindSpore. According to the data from Papers with Code (ht pettts://pappe http://ppprswww/e/ thcode. com/ trends), there have been 398 repositories utilizing the open- source MindSpore framework since 2023, which higher than the 192 repositories using TensorFlow. This showcases its potential and the growing interest within the community.
C EXPERIMENTS DETAILS
Our experiments are developed under the Mindspore framework. To make fair comparison with the baseline of PanGu2.6B, we retain the setting of the 32-layer transformer decoder. The model architecture as shown in Figure In the pre-training stage, we trained CodePanGu2.66 2. on cluster of 16 Ascend 910 Al processors, and in the instruction-tuning stage, tuned models on cluster of 8 Ascend 910 Al processors. The sequence length for the training data set to 1024 for all the models. Other detailed configurations can be found in Table 9
Type
Parameter
Value
Framework
Mindspore v1.7.0
Hardwares
Ascend 910
Environmental parameter
Mem per GPU
32GB
GPUs per node
Layers
32
Hidden size
2560
Model parameter
FFN size
10240
Heads
32
Optimizer
Adam
Initial/final learning rate
lc-4(2e-5V1e-6
Optimization parameter
Warm-up step
500
Learning rate scheduler
cosine
Optimizer parameters
81 = 0.9, 82 0.95
Data parallel
16(8)
Parallelism parameter
Model parallel
pipeline parallel
1
Table 9: Training configurations.(Th values in parentheses are instruction-tuning parameters)
13

---

Published as a conference paper at ICLR 2024
Output
Add & Norm
Query Layer
Add & Norm
Transformer Decoder
Feed Forward
(x32 layers)
Add & Norm
Multi-Head Attention
Causal Mask
Positional Encoding
Input Embedding
Inputs (TeutttText/Cdde)
Figure 4: Model architecture. We build models with 2.6B and 13B parameters, consisting of 32-layer left-to-right transformer decoders and top query layer.
D THE TEMPLATE FOR OTHER TASKS
conduct experiments on other tasks to verify the impact of code data on other comprehension and generation tasks that are less demanding on reasoning, including C3 (Sun jet al. 2020); two NLI tasks (OCNLI (Hu et al. 2020) and CMNLI (Wang et al.) 2018)); ; one reading comprehension task duReader (He et 2017). Table shows the prompt templates for these tasks. The evaluation metrics for duReader, including F1 and exact match(EM), measure the similarity between the predicted and ground-troi text spans. The evaluation metric of other tasks accuracy.
We follow Chapter 3.2
Task
Input & Prompt
C3
Question: Squestion'n Answer:Schoice comes from the dialogue: $context
OCNII
SS1? Yes/Mayhe/No, SS2
CMNLI
SS1? Yes/Mayhe/No, SS2
duReader
Read document:
SDocumentin Question:SQuestion \n Answer:
Table 10: The input prompt template for other tasks.
14

---

Published as a conference paper at ICLR 2024
loss
Values
2
100000
200000
Steps
Figure 5: The curves of training loss for CodePanGu2..6B.
E TRAINING Loss
The curves of training loss for the CodePanGu2.66 model shown in Figure We show that the cross entropy loss decreases steadily during training and the loss of this model converges to around 2.25.
F CASE STUDY
In summary, adding code data the pre-training stage can effectively improve the general reasoning ability of LLM, and guide the model to make full use of the coherent reasoning process of the Chain-of-Thought to generate answers. Consistent with GPTRoadMap's 's point view (Fu et al.) 2022). think this may have something do with the logic of the code itself. To further explain why the code improves the reasoning ability of the model, we found several sample codes from the dataset and explained each code, shown in Figure
We found that, regardless of the length of the code dealing with different problems, step-by-step reasoning is required ensure that the code is generated correctly, similar to the Chain-of-Thouge required by other reasoning tasks. This may indicate that the model implicitly learns the thinking chain ability through the code data, which improves the reasoning ability of the language model. In addition, we analyzed the data flow graph of the calculate average function, as shown in Figure
We found many data flow dependence relations in the code data, which are distributed among different code variables. Complex reasoning tasks usually require long dependencies to infer correct conclusions, so the language model may benefit from dependencies such as data and control flow of code data and improve the reasoning ability of the model.
G DATASET CONSTRUCTION
Cleaning and Filtering. To improve the data quality, adopt the following rule-based text cleaning strategies over the raw web pages from Common Crawl.
Remove the document which contains less than 60% Chinese characters.
Remove the document which contains less than 150 characters.
Remove the document which contains only the title of webpage.
Remove the special symbols and duplicated paragraphs in each document.
15

---

Published as a conference paper at ICLR 2024
Description
Code
Explanation
def quicksort(arr):
if len(arr) <= 1: return arr pivot arr[O]
Write a funtion to implement quick sort
smaller
equal pivot
less = for x in arr[1:] x <= pivot]
Create list elements greater than pivot Recursivelyy
greater = [x for am[[1:] if pivot]
return quicksort(less)
[pivot] + quicksort(greater)
class Book:
1.Create book class (Book):
def init (self, title, category):
Attributes:
self.title title
title, author, classification,
method:
def borrow borrower):
return_book, Display book information (display jinoo_._____._______).
Write an online shopping system based on python
class Library:
2.Create library class (Library):
def Init (sel_
Attributes:
self. self. books =
Book list (books). Borrowing History
self.borrow botrrowwhistory=== = =
(borrow history).
def borrow bsolessef.sssef.....ooom title, borrower):
method:
Add Book (add book), Remove Book (ramove_bnok)....
Figure 6: Examples of different codes.
Source code
def aaaailate_averae((uuss)
total
for num in nums:
total += num average total len(nums)
return average
Data Flow
nums
nums
0
num
total
avg
Value comes from
avg
Figure 7: Examples of code dependencies.
Identify advertisements based on keywords and remove documents that contain advertise-
ments.
Identify the navigation bar of the web page and remove it.
16

---

Published as a conference paper at ICLR 2024
Regarding the use of The Common Crawl corpus work, it's worth noting that several prominent projects, such as Llama, have also employed this dataset. Crawled data may exhibit biases, and ground truth data often resides within large corporations, making access challenging. In light of this, we adopted a strategy similar to other open-source models like Llama and Falcon, leveraging broader range of data types such open-source code and e-books to supplement and mitigate potential biases.
in
Text Deduplication. Although removed duplicate paragraphs in each document in the previous step, there are still documents with highly overlapping content in different data sources. Therefore, we carry out fuzzy data deduplication over the documents across all our data sources to further remove high-overlap content. For fuzzy data deduplication, we employed Spark's 's MinHashLSH algorithm, widely adopted technique by models like GPT-3. Data Selection. Using the construction process described above, we constructed filtered text corpora from five types data sources. Based on this corpus, we constructed training dataset of 100GB text data by sampling each data source according to the ratio of Figure and used this data as the first part the training set to train CodePanGu2.6B.
H URLS OF USED DATASETS
This section gives the URLs of the used benchmark datasets.
The Common Crawl corpus: http://oommonorra..oottttttddad
BAAI-WuDao: http::/pppai.......accolllAaaaap
CodeParrot: http://unggiiiiieccooooooerrrmrt
github-code: http::/wwggngacccccoddaeeeooooen
stanford alpaca: https//gthhhooccmmmsssshhssssssu
allpcc
code alpaca: https///gihhbbbooommmmmi//221112
PrompICLUE: http:///gthubboooom//dc//////oco
I EXAMPLES OF DATASETS
Below examples from the Logic, JEC-QA, and E-KAR datasets.
One Example of Logic (Logical Reasoning).
Problem:
Regarding the physical education standard test for Class A, three teachers made the following predictions: Teacher Zhang said, "Not everyone will fail." Teacher Li said, "Someone will fail." Teacher Wang said, "Both the class president and the study committee member will pass." If only one of these teachers' predictions is correct, which of the following must be true?
Answer List:
A: "Both the class president and the study committee member failed." B: "Both the class president and the study committee member passed." C: "The class president passed, but the study committee member failed." D: "The class president failed, but the study committee member passed."
Answer:
One Example of JEC-QA (Legal QA).
Problem:
A miscellaneous article written by person A caused significant stir after its publication. The article was reprinted by several newspapers and websites without compensation. Person B translated the article into French and person C translated it into Uighur. Both translations published domestically without A's consent and without any remuneration. Which the following viewpoints correct?
17

---

Published as a conference paper at ICLR 2024
Answer List:
A: "The act newspapers and websites reprinting the article does not constitute infringement." B: "The actions of both B and C do not constitute infringement." C: "B's action does not constitute infringement, but C's action does." " D: "B's action constitutes infringement, but C's action does not."
Answer:
C
One Example of E-KAR (Analogical Reasoning)
Problem:
Based on the given relationship SSpeed.Tim::Dstaaneee choose the option that fits this relationship.
Answer List:
A: "Interest Rate: PateePrricipall:nteeeett B: "Quality-Variey::uuanie" "Profit:Cos: Value" D: "In-
coneeEExpeniuur::Suplla
Answer:
From the given relationship, we infer the following: "Speed" multiplied by "Time" equals "Distance". In option A, "Interest Rate" multiplied by "Principal" equals "Interest". In option B. there is no clear logic connecting "Quality", "Variety", and "Quantity". In option C, the product of "Profit" and "Cost" is not "Value". In option D, "Surplus" is the difference between "Income" and "Expenditure". Therefore, the correct choice is A.
Explanation:
J ADDITIONAL CONCLUSIONS OF MIXTURE STRATEGY
From the extensive experiments of the mixture strategy, we have three additional conclusions as follows.
1) Using a larger code proportion in the early stage can improve the performance of LLM in coding tasks (CosQA and MBPP). The reason may be that higher code proportion in the early stage can better activate code-related reasoning capabilities under higher learning rate.
2) The descending strategy can improve the performance of logic. Since the code data is more logical, giving more codes in the initial stage may improve the performance of logic reasoning.
3) In the other three datasets, uniform better, probably because these tasks require both logical reasoning ability and common sense well as natural language understanding ability.
Therefore, we recommend choosing different hybrid strategies based on the characteristies of different downstream tasks.
K OTHER DATASETS
order to more comprehensively verify the observations of this article, we selected the high school mathematies and high school physics problem parts of the MMLU (Hendrycks et al. (0200) test set to evaluate the model in the pre-training stage, MMLU a currently widely used data set to evaluate the comprehensive ability of L.MM (OpenAl 2023b Touvron al. 2023), among which mathematics and physics can better reflect the reasoning ability of the model. the table below.
The results obtained are shown in
Task
NI.2.6B
NL133
CODE2.6B
P_value <0.05
LlaMA-7B
MMLU Math
24.16
22.30
24.91
24.97
MMLU Physics
20.00
22.67
26.67
<0.05
27.97
Table 11: Performance in mathematics and physics subjects.
We make the following observations:
On mathematical and physical reasoning tasks, the CODE2.6B model shows advantages over NI.2.6B and NL.13, which strengthens the effectiveness of introducing code data
18

---

Published as a conference paper at ICLR 2024
in the pre- pre-training stage. We performed t-test for statistical significance and the results showed that the p-value was less than 0.05, which indicated that the results were statistically significant.
We admit that models including LLaMA-7B and CODE2.6D perform relatively poorly on mathematical and physical tasks, but believe that the relative improvement brought by code data is trustworthy. This also shows how it is necessary to further improve mathematics- related tasks.
Note: Source of LLaMA results: https://github..o http://giitub..cm//acccuuu combbacchaannnn///cccnnna
In addition, supplemented our pre- pre-training with additional 50GB of natural language data on top of the NL (2.6B) dataset. From Table [2]
we observed performance improvements on some natural language datasets; however, these benefits were not significant those seen with code data. Moreover, there was no enhancement in performance on code datasets, and in some cases, there was even negative effect. Therefore, it would be valuable to further explore the hybrid training of code and text data or even other types of modal data.
Dataset
Logic
JEC-QA
ScienceQA
E-KAR
CosQA
MBPP 0.45
NL(2.6B)+50G
36.36
27.20
45.96
32.54
46.69
Table 12: Performance of model pre-trained with an additional 50GB data.
L LIMITATIONS
L.1 VERIFIED ON MORE LARGE LANGUAGE MODELS.
PanGu (Zeng et al. 202 ,. along with other LLMs like Llama (Touvron et al. 2023) and Google PaLM (Chowdhery et al., 2022), shares common architecture based on the GPT-2 ApenAl (202a) decoder-only architecture and next token prediction task. Due to resource constraints and envi- ronmental concerns, currently only conduct experiments on PanGu, but compare models with different parameter scales (i.e., 2.6B and 13B) to demonstrate the impact of the code. For a more comprehensive validation a larger model, this section analyzes the performance of two other models, namely the Llama 2 and Code Llama models on reasoning tasks. Among them, Code Llama continued pre-train using code data based on Llama The performance summary on code tasks and mathematical tasks is shown in Table [3] The data comes from Open LLM ct/ Leaderboard hhttps://hhtt httsss//hhgngnfccc wtt.c////sppeeeeee spaces waggginnFoan www.ggaaaa//aaaao (ecet//oppn__iiee 11m l1__emeeeadbbrrr Ex- perimental results show that pre- -eeitriinng on code data can improve coding and mathematical reasoning capabilities, and also preliminarily verify some of the conclusions obtained in this article. In the future, this article will continue to verify and discuss model experiments on larger scale. At the same time, we look forward to the emergence of L.LMs with different architectures than GPT, and we also look forward to following up and verifying more LLMs with different architectures.
Model
HumanEval (CODE) GSM8K (MATH)
Llama 2-7B
12.2
3.49
Code Llama-7B
33.5
5.16
Llama 2- 2-B
20.1
10.84
Code Llama-13B
36.0
12.13
Table 13: Results on I.lama 2 and Code Ilama (7B and 13B).
L.2
WHY NOT DESIGN A NEW MODEL FOR CODE?
Our paper primarily investigates at which training stage can codes help general LLMs reasoning. Through extensive experimentation, we assess the impact of code data, offering insights that can guide the development of future universal LLMs. Our objective is not to create a specialized code or text model, but rather to provide guidance in this context. In the future, we plan to leverage these findings for future development. We will explore integrating code features with models to create more robust and versatile universal reasoning models.
19

---

Published as a conference paper at ICLR 2024
M APPLICATION
The main contribution of this paper is to explore the impact of code data on LLM reasoning capabilities different training stages and draw the following conclusions. Firstly, adding code data in the pre- training stage can effectively improve the model's general reasoning capabilities. Secondly, adding code instructions in the instruction fine-tuning stage can improve specific code reasoning capabilities. Moreover, the dynamic mixing strategy of code and text data assists LLMs in learning reasoning capability step-by-step during training. Below, we provide some ideas for further applications of these conclusions.
M.1 IMPROVE GENERAL REASONING SKILLS.
The conclusions obtained in this article have direct implications for the application of general LLMs in multiple iassoning-intenivee fields (such legal support, scientific question answering, etc.). Secondly, for the construction large models in vertical fields (such as large legal models, etc.), is often necessary use vertical field data for continued pre-training. High-quality data the key to model training. If there is a lack of data in vertical fields, you can try to mix certain high-quality code data to improve the model's reasoning capabilities in vertical fields.
M.2 IMPROVE DOMAIN-SPECIIE REASONING SKILLS.
This paper finds that mixing in code data during the fine-tuning phase improve capabilities in specific areas of the code. This may be because instructions for related tasks may activate reasoning and response abilities in the corresponding tasks. Therefore, for specific fields such as information extraction Li et al. (2023), you can also consider mixing in instructions from specific fields to further improve reasoning capabilities.
M.3 IMPORTANCE TRAINING DATA.
This paper verifies the impact of code data through ablation experiments and proves the importance of data. However, there are still data that may be similar to code data, such as scientific paper data, mathematical data, etc. This may also be the key to improving the reasoning ability of language models. The experimental exploration in this article provides a direction for further research on how multiple types of data affect model training.
20


Reviews:
[{"review_id": "ndP5xZp2gJ", "reviewer": "Reviewer_viag", "summary": "The paper explores the effect of including code data during pre-training and instruction tuning over the reasoning capabilities of LLMs. The authors conduct a set of ablation experiments where code is added/removed from both pre-training and fine-tuning and the model performance is measured over a set of reasoning tasks ranging from logical to legal reasoning. The results show that including code data during pre-training is more effective than during instruction tuning. Also, the results show that instruction tuning over the code can end up hurting performance on some tasks. The authors also experimented with a dynamic text-code mixing strategy during instruction tuning.", "questions": "* The choice of the baseline model is unclear. Why use PanGu2.6B as your baseline model? \n* Why does your mixing strategy follow the 7:3, then 6:4, then 5:5? Why not, for example, 9:1, 7:3, 5:5, 3:7, 1:9? which would be both increasing and decreasing. And why only 4 phases? What's the intuition behind that design?\n\n\n\n===== POST REBUTTAL =====\n\nI thank the authors for their response. Based on the response and the expectation that the authors will add results from training a model on 150G of natural data, I have decided to increase my score to 5.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The problem studied is interesting: the relationship between code data and reasoning and the paper aims to somehow tackle this issue.\n* The paper is well-written and the results are well-presented.", "weaknesses": "* From my understanding (and correct me If I'm wrong), the code model is trained on more overall tokens than the NL model. I would expect a study like that to control for the number of pre-training tokens while changing their nature i.e., text vs. code. If the code model is trained on as many natural text tokens as the baseline model in addition to having code in the pre-training data, then the code model should be expected to perform better because it was trained on more data. No surprise there. \n* It's hard to say whether the results reported have statistical significance. For example, in Table 2, the code model is only 0.13 points better than the NL model on ScienceQA. Are these results significant? And are these enough to conclude that code reasoning? I would expect the authors to run a statistical significance test to support their results. \n* The proposed dynamic mixing strategy produces very marginal improvements (except over logical reasoning) and has a negative effect on the performance over three tasks. The paper does not thoroughly investigate why this is the case. Also, the design of the mixing strategy seems rather arbitrary. \n* The evaluation does not cover mathematical reasoning, although it's one type of reasoning where we should expect great improvements since code data is roughly similar to math data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "87M2Eq4k3D", "reviewer": "Reviewer_BTot", "summary": "Previous work has shown that models trained on code perform better on reasoning tasks. The research question in this paper is at which training stage code data helps reasoning. Specifically, this work looks into two training stages: pre-training and instruction-tuning. The results show that for performance on reasoning tasks, adding code data at the pre-training stage is effective whereas at the instruction-tuning the effect is far less (sometimes leading to lower performance); however, instruction tuning on code can improve model performance on code-related problems.", "questions": "- For the Logic dataset, I see that the accuracy for many different models is 40.9. This value appears in Table 2, Table 3 and Table 6. Does this hint at a potential problem in this dataset? \n- Some of the datasets don't seem to be available in English (or at least I could not find an English version of them). Could you include some examples from these datasets translated to English, so the reader can get a sense of the nature and the difficulty level?\n- It has been observed that larger LLMs often behave differently than smaller LLMs. I'd be curious to hear the authors' thoughts on how they think their results might transfer to larger models? (to be clear, I understand the computation cost and I'm not suggesting that you run those experiments)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Clear Hypothesis:** The paper asks a clear question and provides a clear setup for testing various hypothesis about that question.\n- **Clear message:** The results show a clear message about the research question, albiet only on smaller-sized models.\n- **Clarity of the writing:** The writing was mostly clear and easy to follow", "weaknesses": "- **Datasets:** I was not familiar with some of the datasets used in this work and after looking into some of them, I could not get a sense of how general and challenging they are. The majority of the computation cost for this project seems to be on the training stage, so I believe reporting results on a few more datasets (maybe only for Tables 2 and 3) can strengthen the main arguments of the paper. That could include datasets from other reasoning domains (e.g., math might be an important one that does not appear in the results) or from the same domains but on datasets that are more established.\n- **Mixture experiment:** The experiment on exploring ways to mix code and text data is interesting, but given that adding code at the instruction-tuning stage was already shown to be not that effective, I wonder why it was tested for the instruction-tuning stage. I understand the high computation cost of pre-training, but it seems to me that given the previous set of results, this experiment makes sense mostly at the pre-training stage where we have seen that code data can be effective.\n- **Language Inconsistency:** - For the results in Section 3.3.5, the authors conclude that training with code data *has little negative impact* on the performance of other tasks. But the numbers in Table 7 don't seem to show little negative impact. The large impact on the DuReader dataset has been already pointed out by the authors. Moreover, on CMNLI the performance decreases from 45.07 to 43.49 which is almost equal in magnitude to some of the gains reported in Table 2. I believe the language should be more consistent on the amount of improvement/decrement that can be considered a significant amount for the datasets and the experimental setup of the paper.\n- **Minor suggestions:** 1- In Table 3, there are multiple equal numbers but only one of them is in bold face. 2- In Table 4, I suggest reversing the rows and columns to make it consistent with the other tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dXTv0waMHt", "reviewer": "Reviewer_PRkj", "summary": "This paper aims to answer an important research question: at which training stage does code data help LLMs reasoning? The authors introduce code at the pre-training stage, instruction-tuning stage, and both. The reasoning capability of LLMs is evaluated by six reasoning tasks. Through comprehensive experiments and careful analyses, they provide inspiring conclusions and insights. The authors open-source the code and model parameters.", "questions": "1. Experimental details. In the paper, certain key sections, such as data mixing strategy and decoding strategy, do not seem to give sufficient details of experimental settings and parameter selection. This lack of information can lead to difficulties in reproducing and scaling. How was it set up by the author in the actual experiment?\n\n2. Quality of code data. How were the coding data used by the authors in their experiments collected and screened? The quality of code data can have a significant impact on the reasoning capabilities of LLM. For example, high-quality code data may provide more reasoning impact, while low-quality data may cause the model to learn wrong patterns. How did the authors ensure that the code data used were representative and of high quality?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Valuable research question. The paper raises a meaningful research question: at which training stage introducing code data can really help the reasoning capabilities of LLM? This question is of critical significance for understanding the training and application of LLM. \n \n2. Comprehensive experimental design. This paper provides a comprehensive and fair evaluation of the reasoning capabilities of LLMs on six reasoning tasks covering five domains. This broad experimental scope ensures the generalizability and reliability of the conclusions. Additionally, the authors compare models with different sizes to verify the generalization of the conclusion. \n \n3. In-depth analyses and insights. The paper not only provides experimental results but also performs in-depth analysis, providing insights into mixing code and text data to enhance the general reasoning capabilities and code reasoning capabilities of LLM. Specifically, in the pre-training stage, mixed code data helps LLM improve general reasoning capabilities, and in the SFT stage, mixed code data helps LLM improve specific code reasoning capabilities.", "weaknesses": "1. Experimental details are insufficient. The paper may not provide enough details on experimental settings and parameter selection in some parts (such as data mixing strategies, decoding strategies, etc.). This might challenge researchers attempting to replicate or extend this work.\n\n2. Recently, various code foundation models, such as CodeLlama [1], have been opened. However, the authors do not conduct any discussions or experiments on them. In my opinion, the reasoning capability of code foundation models is also an essential part of the interest scope of this work. \n\n3. Quality of code data. The quality of code data can have a significant impact on the reasoning capabilities of LLM. How the author ensures the high quality of code data is not discussed in depth in the article.\n\n4. The related work part is weak. Missing important papers, such as [1,2,3].\n[1] Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023.\n[2] Yang A, Xiao B, Wang B, et al. Baichuan 2: Open large-scale language models[J]. arXiv preprint arXiv:2309.10305, 2023.\n[3] Li P, Sun T, Tang Q, et al. CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors[J]. arXiv preprint arXiv:2305.05711, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uvtMuxvWvx", "reviewer": "Reviewer_L6FP", "summary": "In this paper, the authors aim to explore the impact of introducing code at different training stages of large language models (LLMs) and how it affects LLMs’ reasoning capability. The experiments are conducted by introducing code data in the pre-training stage, the instruction tuning stage, and both stages to evaluate LLM through six inference tasks in five domains. They provide deep-in insights via comprehensive experiments and critical analyses. The resources of this paper are released.", "questions": "Please check in Strengths and Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-motivated. The impact of code data in LLMs is a hot research question. This paper answers this issue from the reasoning capability aspect.  \n\n- The experiments are comprehensive, and the insights are remarkable. The reasoning capability of LLMs is evaluated via six tasks in five domains. The authors provide critical analyses and significant insights on training LLMs and the reasoning capability of LLMs. \n\n- The idea of dynamic mixed strategy is easy to follow yet effective. It helps LLMs learn reasoning skills progressively during training. \n\n- The authors provide comprehensive open-source resources, demonstrating the reproducibility of the models. These resources are valuable for the LLM community.", "weaknesses": "- Missing discussion on the applications. Although the authors conduct experiments and provide insights on training LLMs and improving their reasoning capability, this paper does not discuss how to apply the insights to enhance the LLM products in different domains. \n\n- Unclear construction of training corpus. The author should provide more details about data collection, data cleaning, and training data construction. The authors use fuzzy data deduplication, but they have not explained the tools of fuzzy. They should open-source the data for reproducibility. Besides, the detailed model architecture is missing. \n\n- Table 7 is confusing. The results in Table 7 show the code data will lead to a performance drop on four out of five datasets. It indicates that the code data may not help to improve the reasoning capability of LLMs. The authors should provide valid reasons.\n\n- The related work is limited. Recently, there have been various papers discussing the reasoning capability of LLMs. Therefore, the authors should survey more related papers and compare with them. \n\n- Fix the grammar errors and improve the presentation. On page 8, “The experiment found that…’’ -> “The experiment showed that’’. \n\n- Missing future work. The authors should provide the potential future work on LLMs based on the experimental results and insights provided in this paper. \n\n[1] Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)