Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
Authors: Ziang Ye, Zhenru Zhang, Yang Zhang, Jianxin Ma, Junyang Lin, Fuli Feng
Accepted at: acl25

Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning

Ziang Ye1,2*
Zhenru Zhang2
Yang Zhang3†
Jianxin Ma2
Junyang Lin2
Fuli Feng1†

1University of Science and Technology of China   2Alibaba Group
3National University of Singapore
yza03@mail.ustc.edu.cn {zhangzhenru.zrr,junyang.ljy}@alibaba-inc.com
{zyang1580,majx13fromthu,fulifeng93}@gmail.com

Abstract

When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles—specifically, reasoning tokens versus boilerplate tokens (\emph{e.g.}, those governing output format)—differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel \emph{Shuffle-Aware Discriminator} (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the \emph{Reasoning-highlighted Fine-Tuning} (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT).

Thought: Based on the user's request to find the most popular genre in the Media-Group tool, I should call the "list genres for media group" function to retrieve a list of genres. By doing so, I can analyze the genres and determine which one is currently trending based on popularity. This way, I will be able to provide the user with the information they are looking for regarding the most popular genre in the Media Group tool.

Action: list genres for media group
Action Input: {}

Figure 1: Examples of reasoning tokens (green) and boilerplate tokens (yellow and blue). Boilerplate tokens can be further categorized into format tokens (yellow) and template-connecting tokens (blue).

et al., 2024b; Zhao et al., 2023). These datasets offer structured examples of standard reasoning chains for solving agent tasks (Chen et al., 2024b; Qin et al., 2024), enabling LLMs to learn from them and thereby enhance their agent capabilities.

When leveraging these datasets to bolster LLMs’ agent capabilities, existing research often treats all tokens within a sample equally (Chen et al., 2023; Zeng et al., 2023; Chen et al., 2024c; Qin et al., 2024; Zhao et al., 2023). However, we argue that these tokens could differ substantially in learning difficulty and importance. Given the standardized structure of the data, tokens within a sample can be divided into two categories as depicted in Figure 1: 1) boilerplate tokens, which include format tokens that constrain the output structure, and template-connecting tokens that serve as standard transitional phrases for reasoning, such as “Based on the user’s request... By doing so... This way...”; and 2) reasoning tokens, which provide sample-specific reasoning information crucial for task solving. Boilerplate tokens are distinctly less critical for task solving compared to reasoning tokens and are easier to learn due to their repetitive nature across many samples.
Figure 2: Loss changes for different types of tokens in the manually sampled test data that the model fails to answer for the regular SFT training.

It is crucial to distinguish between the reasoning and boilerplate components and handle them separately. Failure to do so may result in undesired effects, such as overfitting to the boilerplate components, as depicted in Figure 2, ultimately leading to inadequate agent capabilities. While manually crafting regular expressions to filter out boilerplate tokens appears to be a feasible solution, it can be highly inefficient when dealing with data of diverse formats. Additionally, creating regular expressions for template-connecting tokens of transitional phrases poses challenges due to their potential variability in language. Therefore, an automated and adaptive approach for segregating these components is highly desirable.

This study introduces a novel SHuffle-Aware Discriminator (SHAD) to achieve automated and adaptive token distinction. Considering boilerplate tokens are usually consistent across samples, they can be treated as sample-independent. Consequently, shuffling the correspondence between input and output across data samples does not alter the predictability of boilerplate tokens. However, such shuffling introduces noise that complicates the prediction of reasoning tokens, by causing mismatches between the tokens and the input queries1. SHAD is developed based on this principle. Specifically, it fine-tunes an LLM model using a small portion of shuffled data and then compares the token-level loss between the tuned and original models to classify tokens for the target data. A token is classified as a boilerplate token2 if the loss on the tuned model decreases; otherwise, it is classified as a reasoning token.

Based on SHAD, we have developed a new Reasoning-highlighted Fine-Tuning (RFT) approach, which adaptively assigns greater weights to challenging reasoning tokens to emphasize the learning of reasoning. This approach demonstrates superior performance compared to existing supervised fine-tuning methods across several common agent benchmarks. Further analysis reveals that our method could effectively identify reasoning tokens and strengthen the learning of these tokens, ultimately enhancing the learning of agent capabilities for LLMs.

The main contributions of this work are summarized as follows:

• We emphasize the differences in learning difficulty and importance between reasoning and boilerplate tokens for agent learning, highlighting the critical importance of effectively distinguishing between them.
• We introduce SHAD, a novel method that automatically discriminates between reasoning and boilerplate tokens based on their predictability differences observed after shuffling input-output combinations.
• We have developed a new fine-tuning method RFT rooted in SHAD, improving the effectiveness of learning agent capabilities for LLMs.

1We will later provide practical examples in Section 3.1 to illustrate how shuffling can cause the reasoning parts of a response to mismatch with the corresponding queries.

2These tokens would be further categorized into formatting tokens and template connecting phrases based on their losses if needed.
Figure 3: Illustration of the SHAD method, which classifies tokens through three steps. In step 1, a small subset of the data is sampled, and the output of the sampled data is shuffled. In step 2, the LLM is tuned using the shuffled data. In step 3, tokens are classified by comparing the prediction losses between the tuned and original models.

to account for token differences in agent tuning. However, it only considers “format tokens” as boilerplate tokens, overlooking template-connecting tokens, which are more challenging to disentangle from reasoning tokens. Additionally, it does not emphasize the importance of distinguishing (or classifying) these tokens, resulting in a fundamental difference in both the problems addressed and the solutions proposed. We focus on automatically disentangling reasoning tokens from boilerplate tokens, whereas Agent-Flan prioritizes converting agent data into a standard conversational format.

•Enhancing Agent Capability for LLMs. To tackle complex real-world problems, it is essential to enhance LLMs’ agent capabilities, such as the ability of external tool use and multi-step reasoning (Shen et al., 2023; Nakano et al., 2021; Yao et al., 2022; Du et al., 2024; Paranjape et al., 2023). Prior works (Yao et al., 2023; Shinn et al., 2023; Pan et al., 2024; Zhao et al., 2023; Shen et al., 2024) have focused on developing frameworks that prompt LLMs to integrate tools better and engage in deeper reasoning before taking action. Subsequent works have further constructed diverse and well-structured agent-task benchmark datasets, e.g., Toolllama (Qin et al., 2024), Toolalpaca (Tang et al., 2023), and APIGen (Liu et al., 2024), considering these specific datasets for further tuning of LLMs to more directly and effectively enhance their agent abilities. Although these methods train LLMs on agent datasets and achieve promising results, they often struggle with overfitting and generalization issues (Chen et al., 2024b). Our RFT with SHAD can better utilize these datasets to learn reasoning, achieving superior performance on agent tasks while maintaining good generalization ability on out-of-distribution benchmarks.

3 Methodology

In this section, we first introduce the SHuffle-Aware Discriminator (SHAD), which is proposed to adaptively distinguish between reasoning and boilerplate tokens. We then discuss how to develop our Reasoning-highlighted Fine-Tuning (RFT) based on the discrimination results.

3.1 SHAD: Adaptive Token Discriminator

To develop SHAD, our foundational idea is that boilerplate tokens, which template outputs, should be interchangeable across many samples, whereas reasoning tokens are specific to individual samples and cannot be swapped. Consequently, shuffling the combination of inputs and outputs across samples does not alter the predictability of boilerplate tokens, unlike reasoning tokens. Leveraging this principle, we could achieve automated and adaptive token discrimination through the three steps (as show in Figure 3):

1. Data Shuffle: Select a small ratio of the data and shuffle the combinations of inputs and outputs among the sampled items.
2. Model Tuning: Fine-tune an LLM model using the shuffled data.
3. Classifying: Classify tokens based on the loss change between the tuned and original models for the target data. Compared to the original model, if a token’s loss decreases, it is likely a boilerplate token; otherwise, a reasoning token.

Next, we elaborate on these three steps:

•Data Shuffle. This is the core step of our method, creating distinct predictability for the reasoning tokens and boilerplate tokens. The shuffle is performed by randomly reassigning the input-output combinations between samples. When implementing, we just select a small ratio (1%) of the target dataset and shuffle it for use in the subsequent
Figure 4: Example of shuffled data. After shuffling, the assistant’s responses no longer correspond to the original queries. However, some tokens (boilerplate tokens, red) remain semantically similar to the original response and are therefore predictable. In contrast, reasoning tokens (green) no longer align with the query, resulting in noise. Note that ‘Action’ and ‘Action Input’ are directly copied from ‘Thought’ and could be considered as non-reasoning.

model tuning step, to avoid large tuning costs and overfitting on the whole dataset.

Let \( (x^i, y^i) \) denote the \( i \)-th sample for the sampled dataset, with \( x^i \) as the input and \( y^i \) as the output. Denote all the inputs of all samples as \( X = [x^1, \ldots, x^N] \), and the corresponding outputs as \( Y = [y^1, \ldots, y^N] \), where \( N \) denotes the size of sampled dataset. We shuffle \( Y \), and then recombine the inputs in \( X \) and outputs in the shuffled \( Y \) to construct the shuffled dataset \( \mathcal{D}_s \). This means, for the \( i \)-th original sample \( (x^i, y^i) \), its input \( x^i \) may be combined with the \( j \)-th sample’s output \( y^j \) to form a new sample \( (x^i, y^j) \), while its output \( y^i \) may be combined with the \( k \)-th sample’s input \( x^k \) to form a new sample \( (x^k, y^i) \). With this operation, the mapping relationship between the inputs and outputs becomes noise for reasoning tokens, making them unpredictable. As for the boilerplate tokens, since they are shared across samples, their predictability remains intact. Figure 4 provides an example to illustrate this.

• Model Tuning. After obtaining the shuffle data, we leverage them to fine-tune an LLM model. Note that this tuning process uses the same LLM as our backbone model for performing agent tasks. The model tuning is performed according to the classic causal language modeling. Formally,
\[
\theta_s = \arg\min_{\theta} \sum_{(x', y') \in \mathcal{D}_s} l(x', y'; \theta),
\]
where \( \theta \) denotes the learnable model parameters, and \( l(x'; y'; \theta) \) denotes the loss for a shuffled sample \( (x', y') \in \mathcal{D}_s \), and \( \theta_s \) denotes the optimized \( \theta \). As the output is shuffled for the input, the tuned model is only expected to learn to predict boilerplate tokens effectively.

• Classifying. After tuning the model with shuffled data, we evaluate the role of each token in a target sample by comparing the token-level prediction loss between the tuned and original models. Given that the tuned model should primarily learn boilerplate tokens, we classify a token as ‘boilerplate’ if its prediction loss decreases in the tuned model relative to the original; otherwise, we classify it as a ‘reasoning’ token.

Given a sample \( (x, y) \) in the target dataset, we focus on classifying the tokens in the output part. Formally, for the \( k \)-th token \( y_k \) in the output, we first compute the prediction loss difference (denoted as \( LD(y_k) \)) between the tuned and original models as follows:

\[
LD(y_k) = l_s(y_k) - l_o(y_k),
\]

where \( l_s(y_k) \) and \( l_o(y_k) \) represent the loss calculated on the tuned model and the original model, respectively, given by:
\[
l_s(y_k) = -\log(P(y_k|x, y_{<k}; \theta_s)),
\]
\[
l_o(y_k) = -\log(P(y_k|x, y_{<k}; \theta_o)).
\]

Here, \( P(y_k|x, y_{<k}; \theta_s) \) and \( P(y_k|x, y_{<k}; \theta_o) \) denote the predicted probabilities of the token \( y_k \)
from the tuned model (parameterized by \( \theta_s \)) and the original model (parameterized by \( \theta_o \)), respectively.

Based on the calculated loss difference \( LD(y_k) \), the token is classified as follows:

\[
Classifier(y_k) = \begin{cases}
\text{boilerplate}, & \text{if } LD(y_k) \leq 0 \\
\text{reasoning}, & \text{otherwise}
\end{cases}
\]

Note that our token classification can be conducted offline with a single forward pass of LLM computation for each sample, without affecting the efficiency of the subsequent agent tuning process.

3.2 Reasoning-highlighted Fine-Tuning

Agent-tuning data often follows fixed formats and similar reasoning trajectories, making boilerplate tokens easily learned. To prevent overfitting to these tokens and enhance reasoning capabilities, we propose focusing more on reasoning tokens identified by our SHAD method during fine-tuning.

Instead of manually assigning fixed weights to the two types of tokens, we utilize an adaptive weight assignment to align the dynamic learning process better. Specifically, we compare the total losses of the reasoning and boilerplate parts, applying the softmax function to assign higher weights to the part with the greater loss. Notably, since the reasoning part typically exhibits a higher loss (see Figure 6), our method naturally assigns greater weights to emphasize reasoning learning. Furthermore, when the loss difference between the two parts diminishes, our method can adaptively adjust the weights to promote a more balanced learning process for the two parts. Given the nature of highlighting reasoning, we name our method Reasoning-highlighted Fine-Tuning (RFT).

Formally, For each training sample \( i \), let:

• \( \mathcal{R}_i = \{ r_1, r_2, \ldots, r_{P_i} \} \) be the set of reasoning tokens with \( P_i \) tokens
• \( \mathcal{B}_i = \{ b_1, b_2, \ldots, b_{K_i} \} \) be the set of boilerplate tokens with \( K_i \) tokens
• \( \mathcal{R}_i \cup \mathcal{B}_i \) represents all tokens in sample \( i \)

We first compute the average loss for each token type in sample \( i \):

\[
\bar{\mathcal{L}}^{(i)}_r = \frac{1}{P_i} \sum_{j \in \mathcal{R}_i} \ell_j, \quad \bar{\mathcal{L}}^{(i)}_b = \frac{1}{K_i} \sum_{j \in \mathcal{B}_i} \ell_j
\]

where \( \ell_j \) is the cross-entropy loss for token \( j \).

Next, we calculate adaptive weights for sample \( i \) using the softmax function:

\[
\omega_b^{(i)} = \frac{\exp(\bar{\mathcal{L}}_b^{(i)}/\tau)}{\exp(\bar{\mathcal{L}}_b^{(i)}/\tau) + \exp(\bar{\mathcal{L}}_r^{(i)}/\tau)},
\]
\[
\omega_r^{(i)} = \frac{\exp(\bar{\mathcal{L}}_r^{(i)}/\tau)}{\exp(\bar{\mathcal{L}}_b^{(i)}/\tau) + \exp(\bar{\mathcal{L}}_r^{(i)}/\tau)}.
\]

where \( \tau \) is the temperature coefficient of the softmax function. A smaller \( \tau \) results in greater weight being assigned to the part with the higher average loss.

Finally, the re-weighted loss for sample \( i \) is computed as:

\[
\mathcal{L}_{RFT}^{(i)} = \omega_b^{(i)} \cdot K_i \cdot \bar{\mathcal{L}}_b^{(i)} + \omega_r^{(i)} \cdot P_i \cdot \bar{\mathcal{L}}_r^{(i)}
\]

The total training loss across all samples is:

\[
\mathcal{L}_{RFT} = \frac{1}{N} \sum_{i=1}^N \mathcal{L}_{RFT}^{(i)}
\]

where \( N \) is the total number of training samples.

4 Experiments

We now present experiments to evaluate the effectiveness of our method in enhancing LLMs’ agent capabilities, particularly in multi-step planning and tool usage, for solving complex real-world problems. We begin by detailing the experimental setup, followed by the analyses of the results.

4.1 Experiment Setup

Training Data. We use LLaMA3-8B and LLaMA3.1-8B as the backbone models, fine-tuning them to solve agent tasks. The training dataset is constructed from two commonly used multi-step planning and tool-use benchmarks, ToolBench (Qin et al., 2024) and APIGen (Liu et al., 2024), supplemented with general data from ShareGPT\footnote{https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered}. The general data is used to preserve general capabilities like instruction-following, as demonstrated in previous work (Zeng et al., 2023). ToolBench and APIGen provide a variety of examples for solving complex real-world user queries across different environments, all organized in a standard agent-specific format: “Thought-Action-Action Input” or JSON style.
Table 1: Performance comparison between baselines, SHAD+RFT, and its variants. Accuracy is reported for BFCL, Nexus, and T-eval, while pass rate, assessed by GPT-4, is used for StableToolBench. ‘AVG’ represents the average performance across all evaluation datasets. The best results among baselines and SHAD+RFT are highlighted in bold, and the second-best are underlined.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Held-In</th>
    <th colspan="2">Held-Out</th>
    <th rowspan="2">AVG</th>
  </tr>
  <tr>
    <th>StableToolbench</th>
    <th>BFCL</th>
    <th>T-eval</th>
    <th>Nexus</th>
  </tr>
  <tr>
    <td rowspan="5">LLaMA3-8B</td>
    <td>SFT</td>
    <td>43.1</td>
    <td>85.9</td>
    <td>67.0</td>
    <td>14.0</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>Regex</td>
    <td>36.2</td>
    <td>81.0</td>
    <td>54.3</td>
    <td>6.45</td>
    <td>44.5</td>
  </tr>
  <tr>
    <td>Rho-1</td>
    <td>24.5</td>
    <td>82.9</td>
    <td>68.4</td>
    <td>19.0</td>
    <td>48.7</td>
  </tr>
  <tr>
    <td>RewardFT</td>
    <td><u>44.4</u></td>
    <td><u>89.3</u></td>
    <td>66.3</td>
    <td>8.0</td>
    <td>52.0</td>
  </tr>
  <tr>
    <td><b>SHAD+RFT</b></td>
    <td><b>50.1</b></td>
    <td><b>87.6</b></td>
    <td><b>71.8</b></td>
    <td><b>27.8</b></td>
    <td><b>59.3</b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>SHAD+α-FT</i></td>
    <td>47.0</td>
    <td>87.2</td>
    <td>68.8</td>
    <td>28.7</td>
    <td>57.9</td>
  </tr>
  <tr>
    <td></td>
    <td><i>Regex+RFT</i></td>
    <td>41.2</td>
    <td>83.81</td>
    <td>61.1</td>
    <td>12.4</td>
    <td>49.6</td>
  </tr>
  <tr>
    <td rowspan="5">LLaMA3.1-8B</td>
    <td>SFT</td>
    <td><u>48.5</u></td>
    <td><u>89.3</u></td>
    <td>64.2</td>
    <td>19.5</td>
    <td>55.4</td>
  </tr>
  <tr>
    <td>Regex</td>
    <td>42.3</td>
    <td>82.1</td>
    <td>58.6</td>
    <td>14.3</td>
    <td>49.3</td>
  </tr>
  <tr>
    <td>Rho-1</td>
    <td>30.6</td>
    <td>84.6</td>
    <td>67.0</td>
    <td>26.0</td>
    <td>52.0</td>
  </tr>
  <tr>
    <td>RewardFT</td>
    <td>48.2</td>
    <td>88.2</td>
    <td>66.4</td>
    <td>19.1</td>
    <td>55.5</td>
  </tr>
  <tr>
    <td><b>SHAD+RFT</b></td>
    <td><b>50.4</b></td>
    <td><b>89.4</b></td>
    <td><b>68.3</b></td>
    <td><b>32.0</b></td>
    <td><b>60.0</b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>SHAD+α-FT</i></td>
    <td>49.2</td>
    <td>88.2</td>
    <td>63.8</td>
    <td>28.9</td>
    <td>57.5</td>
  </tr>
  <tr>
    <td></td>
    <td><i>Regex+RFT</i></td>
    <td>46.7</td>
    <td>80.31</td>
    <td>57.6</td>
    <td>16.2</td>
    <td>50.2</td>
  </tr>
</table>

Evaluation Setting. To comprehensively evaluate the proposed method, we consider two evaluation settings: held-in task evaluation and held-out task evaluation, following prior work (Zeng et al., 2023). For the held-in setting, we use the StableToolBench (Guo et al., 2024) and BFCL (Yan et al., 2024) benchmarks. These datasets align with our agent tuning datasets: StableToolBench shares the same source as ToolBench, while BFCL serves as the leave-out evaluation data for APIGen. For the held-out setting, we use two additional benchmarks: 1) T-eval (Chen et al., 2024a), a comprehensive step-level reasoning benchmark, and 2) Nexus (team, 2023), a complex single-step nested tool-use benchmark. Both benchmarks provide a diverse set of tools for LLMs to choose from, with tasks in StableToolBench and T-eval often requiring multiple steps to complete. Appendix B.1 contains more evaluation details.

Compared Methods. To evaluate our RFT method developed on SHAD (denoted as SHAD+RFT), we compare it against the following baselines: 1) SFT, standard supervised fine-tuning; 2) Regex, which uses regular expressions to distinguish formatting tokens from other tokens and re-weights their losses with constant values; 2) Rho-1 (Lin et al., 2024), which leverages a reference model trained on high-quality data to identify noise tokens and then mask them during fine-tuning; and 3) Reward-based Fine-Tuning (RewardFT) (Yang et al.; Rafailov et al., 2024), which assigns token-level reward scores for tuning using a DPO-based reward model. It is important to note that Rho-1 and RewardT were not originally designed for agent tuning tasks; however, we have extended them for this purpose, with implementation details provided in the Appendix C.

In addition to the above baselines, we also compare our method with two of its variants to assess its core design components: 1) SHAD+α-FT, which retains the SHAD component but assigns a fixed weight \( \alpha \) to reasoning tokens to emphasize them; and 2) Regex+RFT, which preserves the RFT weighting mechanism, but uses regular expressions for the token distinction. The implementation details of \( \alpha \)-FT are also provided in Appendix C.

4.2 Main Results

Table 1 summarizes the performance of all compared methods. From the table, we could draw two main conclusions:

SHAD+RFT Performs Strongly. Our method, SHAD+RFT, outperforms all baselines on all held-in and held-out evaluation datasets, except for the held-in evaluation BFCL with LLaMA3-8B. This highlights the advantage of emphasizing reasoning components in solving complex real-world problems and demonstrates the effectiveness of our
Examples of Tokens Classified by SHAD

Thought: I should call the API "smart_phones_for_amazon_api_v2" with empty arguments to fetch the top-rated smartphone options from Amazon. This API specifically caters to the task of finding top-rated smartphones, so it is the appropriate choice. By calling this API, I can retrieve the necessary information to suggest the user some of the best-rated smartphones available on Amazon.
Action: smart_phones_for_amazon_api_v2
Action Input: {}

{
    "tool_calls": [
        {"name": "getgamelevel", "arguments": {"level": 5, "output": "json"}}
    ]
}

Figure 5: Case study of tokens classified by SHAD. The blue regions represent reasoning tokens, identified by an increase in loss on the model tuned with shuffled data compared to the original model. In contrast, the brown regions indicate boilerplate tokens, characterized by a decrease in loss on the tuned model.

method in identifying and highlighting these parts. Notably, while Rho-1 and RewardFT also differentiate between tokens during learning, they are not specifically designed for agent tuning to discover and emphasize reasoning tokens, resulting in comparatively lower performance. Specifically, Rho-1 targets identifying noise tokens to mask during tuning, but fails to distinguish between normal boilerplate and reasoning tokens. The RewardFT method leverages token-level rewards from a DPO-based reward model aligned with human preferences to differentiate tokens, but it is also not designed to identify reasoning tokens that are essential for agent-specific capabilities.

Notably, we have further evaluated the general effectiveness of our method along two dimensions: model scale and model family. First, we extend our experiments to the LLaMA 3.2 3B model. The results show that our method continues to perform better, emphasizing its effectiveness across different model scales (Appendix E.4). Second, we compare our method with SFT on the Qwen model. The results demonstrate that our method outperforms SFT across all evaluation metrics, further reinforcing its generalizability across model families (Appendix E.5).

Both SHAD and RFT are Crucial. When comparing SHAD+RFT with its variants, Regex+RFT and SHAD+α-FT, the original SHAD+RFT consistently demonstrates superior performance. We explain the results as follows:

• Adaptive weighting in RFT is crucial. Comparing the proposed SHAD+RFT with its variant SHAD+α-FT, SHAD+RFT consistently outperforms, demonstrating the superiority of RFT’s adaptive mechanism over the fixed weighting approach of α-FT. This advantage stems from adaptive weighting’s ability to better align with the dynamic learning process, adaptively adjusting weights for reasoning and boilerplate token components, thereby preventing over-learning or under-learning of either part.

• The importance of SHAD for token differentiation. Replacing SHAD with Regex in SHAD+RFT leads to a significant drop in model performance. This highlights that the effectiveness of reasoning-highlighted fine-tuning depends on accurate token differentiation. The results also demonstrate SHAD’s superior ability to disentangle boilerplate tokens from reasoning tokens. In contrast, Regex relies solely on regular expressions to identify formatting tokens, failing to fully distinguish between template-connecting tokens (one part of boilerplate tokens) and reasoning tokens.

This indicates that replacing either SHAD or RFT diminishes the method’s effectiveness, affirming the importance of both components.

5 Analysis on SHAD and RFT

In this section, we first present a case study on the effectiveness of SHAD in distinguishing different tokens, followed by a comprehensive analysis of how RFT functions.

Case study of tokens classified by SHAD. To further validate SHAD’s ability to identify reason-
ing tokens, we conducted a series of case studies, with one example of classification result shown in Figure 5 (additional examples are provided in Appendix G). As shown in the figure, SHAD successfully classifies most query-dependent information related to ‘smart-phones’—as reasoning tokens, while formatting tokens (e.g., the attribute names ‘Thought’ and ‘Action’) and common template-connecting tokens like ‘I should call’ and ‘this API’ are classified as boilerplate tokens. This outcome aligns with human understanding of reasoning tokens, verifying the effectiveness of our method again. Interestingly, SHAD does not classify the entire function name ‘smart_phones_for_amazon_api_v2’ as reasoning but only the ‘smart_phones’ portion. We think this is may because the ‘amazon_api_v2’ part is common across many function names. Additionally, When this function name appears in ‘Action’ field, it is classified as boilerplate as it is derived from the thought rather than part of the reasoning process.

We acknowledge that evaluating classification quality ideally involves a quantitative analysis of classification accuracy. However, this is impractical since obtaining ground-truth labels for all tokens is nearly impossible, even for humans. Nonetheless, for tokens that can be manually annotated, we conducted a quantitative analysis of classification accuracy, presented in Appendix E. The results show that our method achieves a very low classification error rate (<3%).

RFT Enhancing Reasoning Token Learning. Blindly treating reasoning and boilerplate tokens equally, as done in SFT, can lead to overfitting on boilerplate tokens while insufficiently learning for reasoning tokens. To further verify the effectiveness of RFT, we compare the training loss between SFT and RFT. The results are summarized in Figure 6. The findings indicate that RFT significantly reduces the loss for reasoning tokens while maintaining a comparable loss for boilerplate tokens compared to SFT, confirming that RFT effectively enhances the learning for reasoning tokens. Additionally, we conducted case studies on the model’s output, presented in Appendix F, to assess whether our method improves model reasoning. The results show that our method enhances the model’s ability to correctly apply functions in reasoning components (e.g., providing accurate parameters) while preventing overfitting to training formats. A detailed discussion is available in Appendix F.

The Effect of Hyper-parameter \( \tau \). The temperature coefficient \( \tau \) in Equation 4 plays a crucial role in controlling the strength of our re-weighting mechanism in RFT, so we next investigate its impact. Specifically, we vary \( 1/\tau \) within the range of [0, 2] and analyze the corresponding performance of SHAD+RFT (averaged over all evaluation datasets). The results are illustrated in Figure 7. From the figure, we observe that the performance of our method initially increases and then roughly decreases as \( 1/\tau \) increases, i.e., as gradually enhancing our re-weighting mechanism. This indicates the importance of carefully selecting the optimal \( \tau \). Fortunately, across a wide range, SHAD+RFT could consistently outperform regular SFT and surpass most baselines (c.f., Table 1). In particular, when \( 1/\tau \in [0.2, 0.5] \), SHAD+RFT achieves consistently strong performance. This suggests that our method is robust to the choice of \( \tau \), and set-

![Training loss for SFT and our RFT (based on SHAD). Left: Overall training loss; Right: Training loss for reasoning token part and boilerplate token part.](page_186_120_1077_393.png)

Figure 6: Training loss for SFT and our RFT (based on SHAD). Left: Overall training loss; Right: Training loss for reasoning token part and boilerplate token part.
ting \(1/\tau\) in this range provides a reliable default for practical use.

![A line graph showing the performance of SHAD+RFT method as the temperature coefficient \(\tau\) varies, with LLaMA3-8B as the backbone.](page_246_370_563_312.png)

Figure 7: The performance of our SHAD+RFT method as the temperature coefficient \(\tau\) varies. The performance averaged over all evaluation datasets is reported, with LLaMA3-8B as the backbone. Notably, \(1/\tau = 0\) means assigning equal weights to the reasoning and boilerplate parts, i.e., deactivating our re-weighting mechanism.

6 Conclusion

In this paper, we highlighted the importance of distinguishing between reasoning and boilerplate tokens and introduced a SHuffle-Aware Discriminator (SHAD) to automatically achieve this. Building on SHAD, we further developed a new Reasoning-Highlighted Fine-Tuning (RFT) method to enhance reasoning learning during LLM fine-tuning, thereby improving agent capabilities. Extensive results demonstrated that our method significantly enhances LLMs’ ability to solve complex real-world problems. In the future, we plan to extend our approach to the entire SFT domain and develop more refined mechanisms, such as token-level re-weighting, to better leverage our token differentiation results.

7 Acknowledgments

This research was also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.

8 Limitations

We identify several limitations of our method in both token differentiation and re-weighting during training. First, the effectiveness of our approach depends on boilerplate tokens remaining consistent across different samples. When this consistency is lacking, such as in cases where the diversity of boilerplate tokens is high, our method may fail. Second, our distinction between reasoning and boilerplate tokens relies on rigid, manually defined thresholds for loss differences, which may need refinement. Third, our weighting strategy is currently applied only at the group level, and future optimization may be required at the token level.

Additionally, even with improved reasoning capabilities, model outputs may still exhibit unpredictable behaviors in real-world deployments, potentially leading to incorrect or unsafe actions. There’s also a risk that our approach could reinforce certain biases present in the training data, particularly if those biases are related to reasoning patterns and tool usage decisions. Future work should investigate these risks more comprehensively. Fourth, Our distinction between reasoning and boilerplate tokens is based on loss differences, which may occasionally result in misclassifications. If a reasoning token is mistakenly classified as a boilerplate token, the model may underweight its contribution during training, potentially weakening its reasoning ability, and vice versa. In an extreme case, if all reasoning tokens are treated as boilerplate tokens with a weight close to 0, while boilerplate tokens are classified as reasoning tokens and assigned a large weight, the model could lose its reasoning ability. Therefore, a mechanism may be needed to dynamically adjust the weighting based on classification results.

9 Ethical Considerations

All experiments were conducted using publicly available datasets and models, ensuring no privacy concerns. The Toolbench and ShareGPT datasets are licensed under Apache-2.0, while APIGen is licensed under CC-BY-4.0. The training data was carefully curated and processed to exclude any personally identifiable information. We have maintained transparency in our methodology and results, acknowledging both the strengths and limitations of our approach.

For the large language model use, we utilize ChatGPT to help polish the writing at the sentence level.

References

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. Preprint, arXiv:2310.05915.

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo,
Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. 2024a. T-eval: Evaluating the tool utilization capability of large language models step by step. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9510–9529. Association for Computational Linguistics.

Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024b. Agent-flan: Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 9354–9366. Association for Computational Linguistics.

Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024c. Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models. arXiv preprint. ArXiv:2403.12881 [cs].

Yu Du, Fangyun Wei, and Hongyang Zhang. 2024. Any-Tool: Self-reflective, hierarchical agents for large-scale API calls. Preprint. arXiv:2402.04253.

Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 11143–11156. Association for Computational Linguistics.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2020. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318–327.

Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. 2024. Rho-1: Not All Tokens Are What You Need. arXiv preprint. ArXiv:2404.07965 [cs].

Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. 2024. APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets. arXiv preprint. ArXiv:2406.18518 [cs].

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332.

Haojie Pan, Zepeng Zhai, Hao Yuan, Yaojia Lv, Ruiji Fu, Ming Liu, Zhongyuan Wang, and Bing Qin. 2024. KwaiAgents: Generalized Information-seeking Agent System with Large Language Models. arXiv preprint. ArXiv:2312.04889 [cs].

Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. ART: Automatic multi-step reasoning and tool-use for large language models. Preprint, arXiv:2303.09014.

Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334.

Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Jiang, Chengfei Lv, and Huajun Chen. 2024. AutoAct: Automatic agent learning from scratch for QA via self-planning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3003–3021, Bangkok, Thailand. Association for Computational Linguistics.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv preprint. ArXiv:2307.16789 [cs].

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. 2024. From SrS to SQ**S: Your Language Model is Secretly a Q-Function. arXiv preprint. ArXiv:2404.12358 [cs].

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. 2024. Small LLMs are weak tool learners: A
multi-LLM agent. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16658–16680, Miami, Florida, USA. Association for Computational Linguistics.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging-gpt: Solving AI tasks with chatgpt and its friends in hugging face. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases. arXiv preprint. ArXiv:2306.05301 [cs].

Nexusflow.ai team. 2023. Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling.

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024. A survey on large language model based autonomous agents. Frontiers Comput. Sci., 18(6):186345.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Lilian Weng. 2023. Llm-powered autonomous agents. lilianweng.github.io.

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard.

Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. Preference-grounded Token-level Guidance for Language Model Fine-tuning.

Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. 2023. Preference-grounded Token-level Guidance for Language Model Fine-tuning. arXiv preprint. ArXiv:2306.00398 [cs].

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. AgentTuning: Enabling Generalized Agent Abilities for LLMs. arXiv preprint. ArXiv:2310.12823 [cs].

Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2023. ExpeL: LLM Agents Are Experiential Learners. arXiv preprint. ArXiv:2308.10144 [cs].
A  Detail Information of Training Datasets

We provide more details of our training datasets in Table 2. To enable the multi-step reasoning ability of LLM, we choose ToolBench (Qin et al., 2024) and APIGen (Liu et al., 2024) as our basic datasets. Following the practice in AgentTuning (Zeng et al., 2023) and AgentFlan (Chen et al., 2024b), we also mix ShareGPT and basic datasets for training. We filter the obviously low-quality data that does not follow the request format and sample 5k percent of data from APIGen for data balance. All methods use the same dataset and do not apply token differentiation to general data.

<table>
  <tr>
    <th>Dataset</th>
    <th>Data Size</th>
  </tr>
  <tr>
    <td>APIGen</td>
    <td>5000</td>
  </tr>
  <tr>
    <td>ToolBench</td>
    <td>22993</td>
  </tr>
  <tr>
    <td>ShareGPT</td>
    <td>93481</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>121474</td>
  </tr>
</table>

Table 2: Training Dataset detail in our experiment

B  Experimental Details and Resources Required

B.1  Evaluation Details

For the BFCL benchmark, we use BFCL V1 for evaluation. We primarily focus on AST-based accuracy evaluation\footnote{The AST-based evaluation includes simple, multiple, parallel,parallel multiple}. It directly measures the model’s ability to produce syntactically and parametrically correct function calls. We omit relevance scores from our evaluation since APIGen has not released the training data required for this metric. Additionally, we exclude execution-based metrics due to their inherent instability during evaluation, as they depend on external API availability and runtime conditions.

For StableToolBench, the pass rate is assessed by GPT-4 following the original benchmark’s methodology. We specifically select three most challenging subsets - I2-Category, I3-Instruction, and II-Tool - as they represent complex scenarios requiring sophisticated reasoning capabilities. We report accuracy metrics on T-eval and Nexus as defined in their original papers.

B.2  Training Details

Table 3 lists the hyper-parameters used in our model training. For evaluation, we set the inference temperature to \(10^{-6}\) to ensure reproducibility. When utilizing GPT-4 for evaluation, we follow the practice in ToolLLM (Qin et al., 2023) and evaluate each response 3 times.

<table>
  <tr>
    <th>Params</th>
    <th>LLaMA3-8B</th>
    <th>LLaMA3.1-8B</th>
  </tr>
  <tr>
    <td>learning rate</td>
    <td>1e-5</td>
    <td>1e-5</td>
  </tr>
  <tr>
    <td>warmup radio</td>
    <td>0.05</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>max length</td>
    <td>3072</td>
    <td>3072</td>
  </tr>
  <tr>
    <td>batch size</td>
    <td>32</td>
    <td>32</td>
  </tr>
  <tr>
    <td>gpus</td>
    <td>8</td>
    <td>8</td>
  </tr>
</table>

Table 3: Hyperparameters used for model training. Both LLaMA3-8B and LLaMA3.1-8B were trained on NVIDIA A100 GPUs with a batch size of 32 and a maximum sequence length of 3072. Each training session utilized 8 GPUs and took approximately 8 hours.

C  Implementation Details

C.1  Implementation Details of Rho-1

For the Rho-1 baseline, we train the reference model in self-reference setting (Lin et al., 2024). Specifically, we sample 5% data from our training dataset to train the reference model. We follow the original implementation that focuses training on H→L tokens (\emph{i.e.}, the tokens with loss decreased from high to low during training the reference model) and masks the other tokens.

C.2  Implementation Details of RewardFT

For the RewardFT baseline, because of the lack of Agent preference data, we use general DPO data ORCA DPO\footnote{https://huggingface.co/datasets/Intel/orca_dpo_pairs} and Ultrafeedback\footnote{https://huggingface.co/datasets/allenai/ultrafeedback_binarized_cleaned} to train the model as token-level reward model under the same setting in (Rafailov et al., 2024). We calculate the token-level reward given by the preference model, then we follow the practice in weighted-MLE (Yang et al., 2023), taking softmax on all token rewards as the weight to train the model.

C.3  Implementation Details of \( \alpha \)-FT

A simple and common method for addressing imbalance training is to manually give a fixed weight
for each type of token (Lin et al., 2020). Here we introduce a weighting factor \( \alpha \in [0, 0.5] \) for boilerplate tokens and \( 1 - \alpha \) for reasoning tokens. Let \( \mathcal{L}_b \) and \( \mathcal{L}_r \) represent the total loss for the boilerplate and reasoning tokens, respectively. The re-weighted loss (denoted as \( \mathcal{L}_{\alpha-\text{balance}} \)) can be formulated as follows:

\[
\mathcal{L}_{\alpha-\text{balance}} = \alpha \mathcal{L}_b + (1 - \alpha) \mathcal{L}_r
\]

This loss is a simple extension to CE we call \( \alpha \)-FT in this paper that we consider as an experimental baseline for our proposed RFT method.

D Theoretical Analyze of SHAD

While this work primarily focuses on highlighting the problem of token differentiation and proposing a solution based on empirical observations, we still want to provide some theoretical insights to better understand our method.

Let \( X \) be the input space (user queries), \( Y \) be the output space (model responses), \( Y_r \) be reasoning tokens, and \( Y_b \) be boilerplate tokens.

SHAD shuffles input-output pairs to create misaligned combinations (\( X', Y \)), deliberately inducing a distribution shift from the original distribution \( P_o \) to a shuffled distribution \( P_s \). This shift affects different token types differently:

For reasoning tokens \( y_r \) and \( x \):

• **Original conditional probability:** \( P_o(y_r \mid X = x, y_{<r}) \)
• **After shuffling:** \( P_s(y_r \mid X' = x, y_{<r}) \neq P_o(y_r \mid X = x, y_{<r}) \), i.e., \( P_s(Y_r \mid X') \neq P_o(Y_r \mid X) \)

For boilerplate tokens \( y_b \) and \( x \):

• **Original conditional probability:** \( P_o(y_b \mid X = x, y_{<b}) \approx P_o(y_b \mid y_{<b}) \) (assuming boilerplate tokens are shared across samples)
• **After shuffling:** \( P_s(y_b \mid X' = x, y_{<b}) \approx P_s(y_b \mid y_{<b}) \approx P_o(y_b \mid X = x, y_{<b}) \) i.e., \( P_s(Y_b \mid X') \approx P_o(Y_b \mid X) \)

The shuffled model, parameterized by \( \theta_s \), learns these new conditional distributions \( X' \rightarrow Y \).

**For a sample \( (x, y) \) from the original data**, the loss difference can be approximated as:

\[
LD(y_k) = l_s(y_k) - l_o(y_k)
= -\log P(y_k \mid X' = x, y_{<k}, \theta_s)
+ \log P(y_k \mid X = x, y_{<k}, \theta_o)
= \log \left( \frac{P(y_k \mid X = x, y_{<k}, \theta_o)}{P(y_k \mid X' = x, y_{<k}, \theta_s)} \right)
\]

When evaluated on the original distribution, this results in a distinctive pattern:

• **Reasoning tokens typically show increased loss:** \( l_s(y_r \mid x, y_{<r}) > l_o(y_r \mid x, y_{<r}) \), since for \( (x, y) \in P_o \), we generally have \( P(y_k \mid X = x, y_{<k}, \theta_o) > P(y_k \mid X' = x, y_{<k}, \theta_s) \) due to \( \theta_s \) fitting \( P_s(Y_r \mid X') \neq P_o(Y_r \mid X) \)
• **Boilerplate tokens tend to have stable or decreased loss:** \( l_s(y_b) \leq l_o(y_b) \), because \( \theta_s \) is trained on \( P_s(Y_b \mid X') \approx P_o(Y_b \mid X) \)

This can also be understood from a causal perspective:

• **There is a strong causal path:** Input \( X \rightarrow \) Reasoning Token \( Y_r \)
• **Boilerplate tokens are independent of the input:** Input \( X \rightarrow \) Boilerplate Token \( Y_b \)

E Analysis of Token Classification

E.1 Challenges in Manual Token Annotation

The task of manually annotating tokens as either reasoning or boilerplate presents significant challenges that make it impractical for large-scale validation. To illustrate these challenges, we present a detailed example:

Consider the following agent response:

Thought: Based on the user's request to fetch weather data for NewYork, I should call the get_weather function. This API requires the city name and will return current weather conditions.
Action: get_weather
Action Input: {"city": "NewYork"}

In this example, while some tokens are clearly boilerplate (e.g., "Action:", "Action Input:", "", ""), others are more ambiguous, "Based on" could be considered a template-connecting phrase (boilerplate) or part of the reasoning process, "should call" might be viewed as either reasoning (indicating decision-making) or a standard template phrase. The structure "city name and will return" combines both reasoning content and standard connecting phrases

E.2 Evaluation of SHAD Classification

Given these challenges, we instead focused on evaluating our SHAD method against the subset of tokens that can be clearly classified - specifically, formatting tokens that can be identified through
Table 4: SHAD Classification Performance on Formatting Tokens

<table>
  <tr>
    <th>Dataset</th>
    <th>Misclassification Rate</th>
    <th>Recall</th>
  </tr>
  <tr>
    <td>ToolBench</td>
    <td>0.82%</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>APIGen</td>
    <td>2.62%</td>
    <td>0.97</td>
  </tr>
</table>

regular expressions. We conducted this evaluation on our two training datasets in Table 4.

The misclassification Rate is defined as:

\[
\text{Misclassification Rate} = \frac{|\mathcal{T}_{\text{misclassified}}|}{|\mathcal{T}_{\text{total}}|}
\] (8)

where \(|\mathcal{T}_{\text{misclassified}}|\) denotes the number of incorrectly classified formatting tokens, and \(|\mathcal{T}_{\text{total}}|\) represents the total number of formatting tokens in the dataset. We also provide additional metrics such as recall of the formatting tokens. Note that while traditional metrics often include precision, it would be misleading in our context since our method identifies both format tokens and template-connecting tokens as boilerplate, whereas our ground truth only contains format tokens. Therefore, we focus on recall to measure how effectively our method identifies known format tokens.

The low misclassification rates shown in Table 4 on these unambiguous tokens provide strong evidence for SHAD’s effectiveness in identifying boilerplate elements. While this evaluation only covers a subset of all boilerplate tokens, it represents the most objective measure possible given the inherent ambiguity in token classification.

We acknowledge that our evaluation is limited to formatting tokens identifiable through regular expressions, which represents only a subset of all boilerplate tokens. This limitation stems from the inherent challenges in obtaining ground truth labels for template-connecting tokens and reasoning tokens, as the distinction often involves subtle semantic differences and context-dependent interpretations. A more comprehensive evaluation framework that can assess classification accuracy across all token types remains an important direction for future work. Nevertheless, the strong performance on unambiguous formatting tokens, combined with the qualitative analysis and downstream task improvements, provides reasonable confidence in SHAD’s token classification capabilities.

E.3 Experiments with Different Model Variants and Providers

To thoroughly evaluate the effectiveness and generalizability of our proposed method, we conducted extensive experiments across different model variants and providers. The results demonstrate that SHAD+RFT shows consistent improvements regardless of model architecture or size.

E.4 Experiments with Different Model Variants

Table 5 presents detailed results comparing SHAD+RFT against various baselines using the LLaMA3.2-3B model. Our method consistently outperforms baseline approaches across both held-in and held-out evaluations. Notably, while some baselines like Rho-1 achieve strong performance on individual metrics (e.g., 87.5% on BFCL), SHAD+RFT maintains more balanced performance across all metrics, leading to the best overall average performance (53.0%).

E.5 Experiments with Different Model providers

To verify that our method generalizes beyond the LLaMA family, we also evaluated SHAD+RFT using Qwen1.5-4B, as shown in Table 6. The results demonstrate that our method maintains its effectiveness across different model providers.

F Qualitative Analysis

In this section, we present several examples of how model trained by our method yield more accurate answer than model trained by naive SFT in Figure 9. In the response generated by the na i ve SFT model, we observe overfitting, with formatted tokens (yellow) and template-connect tokens (blue) being erroneously generated. Additionally, the na i ve SFT model exhibits hallucination, leading to reasoning errors (red). In contrast, our SHAD+RFT method successfully follows the Held-Out instructions and provides accurate reasoning.

G More examples Labeled by SHAD

In Figure 8, we show several examples of tokens classified by our SHAD method, with blue regions representing reasoning tokens and brown regions indicate boilerplate tokens.
More Examples of Tokens Classified by SHAD

Thought: Based on the user’s request for fresh manga content in the genres of Comedy, Romance, and Supernatural, I called the “fetch_latest_for_mangaverse_api” function from the MangaVerse API. With this function, I can retrieve the latest manga that matches the user’s desired genres. By specifying the genres and requesting chapter lists and images, I will be able to assist the user in finding new manga to blog about. Now, I will call the “fetch_chapters_for_mangaverse_api” function to fetch the chapters for the manga that was fetched in the previous step. This will provide me with the necessary information to present to the user and further enhance their blogging experience.
Action: fetch_chapters_for_mangaverse_api
Action Input: {}

Thought: The current API, “businessplan_for_business_starter_api”, is being called with the argument “{\n ”idea“: ”Start a mobile pet grooming business\n}”. By calling this API with the user’s selected business idea, I can generate a comprehensive business plan specifically tailored to the concept of starting a mobile pet grooming business. This will provide the user with a step-by-step guide on how to establish and run their venture, including details on services offered, target market, marketing strategies, and financial projections. The purpose of this API call is to assist the user in transforming their idea into a concrete and well-structured business plan.
Action: businessplan_for_business_starter_api
Action Input: {
    "idea": "Start a mobile pet grooming business"
}

{
    "tool_calls": [
        {"name": "live_players_rankings", "arguments": {"n_player": "300"}}, {"name": "teammedia", "arguments": {"is_id": 7890}}
    ]
}

{
    "tool_calls": [
        {"name": "products_search_by_barcode", "arguments": {"upc": "123456789012"}}, {"name": "get_product_s_reviews", "arguments": {"productid": "B012345678"}}
    ]
}

Figure 8: More case studies of tokens classified by SHAD. The blue regions represent reasoning tokens, identified by an increase in loss on the model tuned with shuffled data compared to the original model. In contrast, the brown regions indicate boilerplate tokens, characterized by a decrease in loss on the tuned model.
Table 5: Performance comparison of SHAD+RFT across different model providers and sizes. Results show accuracy for BFCL, T-eval, and Nexus, while StableToolBench reports GPT-4 assessed pass rates. The best results among baselines and SHAD+RFT are highlighted in bold.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Held-In</th>
    <th colspan="2">Held-Out</th>
    <th rowspan="2">AVG</th>
  </tr>
  <tr>
    <th>StableToolbench</th>
    <th>BFCL</th>
    <th>T-eval</th>
    <th>Nexus</th>
  </tr>
  <tr>
    <td rowspan="6">LLaMA3.2-3B</td>
    <td>SFT</td>
    <td>45.6</td>
    <td>87.1</td>
    <td>59.00</td>
    <td>14.4</td>
    <td>51.5</td>
  </tr>
  <tr>
    <td>Regex</td>
    <td>23.6</td>
    <td>54.1</td>
    <td>54.5</td>
    <td>16.2</td>
    <td>37.1</td>
  </tr>
  <tr>
    <td>Rho-1</td>
    <td>43.2</td>
    <td><b>87.5</b></td>
    <td>58.2</td>
    <td><b>16.66</b></td>
    <td>51.4</td>
  </tr>
  <tr>
    <td>RewardFT</td>
    <td>41.7</td>
    <td>86.7</td>
    <td>55.7</td>
    <td>15.5</td>
    <td>49.9</td>
  </tr>
  <tr>
    <td><b>SHAD+RFT</b></td>
    <td><b>47.2</b></td>
    <td>87.0</td>
    <td><b>61.0</b></td>
    <td><b>16.72</b></td>
    <td><b>53.0</b></td>
  </tr>
  <tr>
    <td><i>SHAD+α·FT</i></td>
    <td>43.9</td>
    <td>86.4</td>
    <td>58.3</td>
    <td>15.6</td>
    <td>51.1</td>
  </tr>
  <tr>
    <td><i>Regex+RFT</i></td>
    <td>40.7</td>
    <td>86.2</td>
    <td>53.1</td>
    <td>13.3</td>
    <td>48.3</td>
  </tr>
</table>

Table 6: Performance comparison of SHAD+RFT on Qwen1.5-4B The best results among SFT and SHAD+RFT are highlighted in bold.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Held-In</th>
    <th colspan="2">Held-Out</th>
    <th rowspan="2">AVG</th>
  </tr>
  <tr>
    <th>StableToolbench</th>
    <th>BFCL</th>
    <th>T-eval</th>
    <th>Nexus</th>
  </tr>
  <tr>
    <td rowspan="2">Qwen1.5-4B</td>
    <td>SFT</td>
    <td>45.3</td>
    <td><b>84.7</b></td>
    <td>60.7</td>
    <td>17.1</td>
    <td>51.9</td>
  </tr>
  <tr>
    <td><b>SHAD+RFT</b></td>
    <td><b>47.4</b></td>
    <td>83.5</td>
    <td><b>61.1</b></td>
    <td><b>17.3</b></td>
    <td><b>52.3</b></td>
  </tr>
</table>

H More Example of Shuffled Data

In this section, we add more examples of shuffled data in Figure 10 to support the state made that shuffling the correspondence between input and output across data samples does not alter the predictability of boilerplate tokens while reasoning tokens are disruptive after the shuffling.

I Failure Cases

We identify two types of failure cases for SHAD, as illustrated in Figure 11.

First, SHAD struggles to effectively distinguish tokens originating from the shuffled training data. This occurs because, during the shuffle training process, the reference model may have memorized certain samples, making those tokens appear predictable even if they are not boilerplate.

Second, the ToolBench dataset is constructed using GPT-3.5, and some training samples contain errors or inconsistencies—such as invoking non-existent functions or exhibiting logically inconsistent behavior. In such cases, SHAD may incorrectly classify these erroneous tokens as reasoning tokens.

These observations suggest the need to limit the amount of data used for shuffle training and to ensure high data quality in the original dataset.
Examples on Held-Out Benchmark Nexus

Instruction:
You are given multiple functions and a user query.
Please proceed with generating a function call for the function with the proper arguments that best answers the given prompt.
Respond with nothing but the function call ONLY, such that I can directly execute your function call without any post processing necessary from my end. Do not use variables.
{tools}

Example 1

User Question:
Give me restaurants near me?

Response (SFT):
Function Call: find_places_near_location(['restaurant'], my_location', 10)
Explanation: Based on the user's query, I can see that they are looking for restaurants near their location. To fulfill this request, I will use the 'find_places_near_location' function. The current arguments for the function call are "type_of_place" = ['restaurant'], "location" = 'my location'. These arguments indicate that the user is specifically interested in finding restaurants near their current location. The "radius_miles" argument is set to 10, which means that the search will be limited to a 10-mile radius. This will ensure that the recommendations are relevant to the user's current location.
```python
find_places_near_location(['restaurant'], 'my location', 10)
```

Response (SHAD+RFT):
find_places_near_location(type_of_place=['restaurant'], location=get_current_location())

Example 2

User Question:
Give me vulnerabilities, sorted by their CVSS version 2.0 scores in descending order, released between January 1, 2021 and January 31, 2021, that match the CPE name 'cpe:/a:microsoft:windows_10:1909' and have a Technical Alert from US-CERT?

Response (SFT):
Function call: sortCVEsByCVSSv2Score(cve_list, descending=True)
Explanation: The user wants to search for vulnerabilities associated with the CPE named 'cpe:/o:linux:linux_kernel' and sort them in descending order by their CVSS Version 2.0 base scores. The function call 'sortCVEsByCVSSv2Score' is used to achieve this. The argument 'cve_list' is a list of CVE objects, and the argument 'descending' is set to True to sort the CVEs in descending order.

Response (SHAD+RFT):
sortCVEsByCVSSv2Score(searchCVE/cpeName='cpe:/a:microsoft:windows_10:1909', pubStartDate='2021-01-01', pubEndDate='2021-01-31', hasCertAlerts=True), descending=True)

Example 3

User Question:
Could you help identify CPEs that closely resemble the pattern 'cpe:2.3:a:google:android:'? And please, limit the results to just 5.

Response (SFT):
Function call: searchCPE(cpeMatchString='cpe:2.3:a:google:android:', limit=5)
Explanation: The user wants to identify CPEs that closely resemble the pattern 'cpe:2.3:a:google:android:', and limit the results to just 5. To accomplish this, I will call the 'searchCPE' function with the arguments 'cpeMatchString' set to 'cpe:2.3:a:google:android:' and 'limit' set to 5. This will search for CPEs that match the given pattern and limit the results to just 5.

Response (SHAD+RFT):
searchCPE(cpeMatchString='cpe:2.3:a:google:android:', limit=5)

Figure 9: Comparison example on Held-Out Benchmark Nexus. In the response generated by the naïve SFT model, we observe overfitting, with formatted tokens and template-connect tokens being erroneously generated. Additionally, the naïve SFT model exhibits hallucination, leading to reasoning errors. In contrast, our SHAD+RFT method successfully follows the Held-Out instructions and provides accurate reasoning, we explicitly mark the different reasoning part in red.
Before Shuffle

User
My company is organizing a team-building event and we need some entertainment. Retrieve a random programming meme and a Chuck Norris joke for us. Also, generate a Pet Pet meme using an image of our company logo. Begin!

Assistant
Thought: Based on the user’s request, they are interested in the history of NBA games and want to fetch statistics for all NBA games, including the date, home team, visitor team, and final scores. Additionally, they requested a trivia fact about a significant NBA game. In order to fulfill their request, I should call the “get_all_stats_for_free_nba” function with no arguments. This function will fetch the necessary information about all NBA games. By calling this function, I will be able to provide the user with the desired statistics and trivia fact.
Action: get_all_stats_for_free_nba
Action input: {}

After Shuffle

User
My company is organizing a team-building event and we need some entertainment. Retrieve a random programming meme and a Chuck Norris joke for us. Also, generate a Pet Pet meme using an image of our company logo. Begin!

Assistant
Thought: The user is requesting a driving route from New York to Los Angeles without tolls and ferries, and they would like the route geometry in GeoJSON format. To fulfill their request, I will call the “get_route_for_fast_routing” function with the following arguments: coordinates = “-74.0060, 40.7128; -118.2437, 34.0522”, exclude = “toll,ferry”, geometries = “geojson”, and steps = “true”. This will ensure that the API returns the desired route with turn-by-turn directions and the route geometry in the requested format.
Action: get_route_for_fast_routing
Action Input:
{
  "coordinates": "-74.0060,40.7128;-118.2437,34.0522",
  "exclude": "toll,ferry",
  "geometries": "geojson",
  "steps": "true"
}

User
I am interested in growing flowering house plants. Can you provide me with a list of flower category plants? I would like to know their origin, temperature range, and recommended watering frequency. Begin!

Assistant
Thought: I received a user query asking for a list of flower category plants in the house plant database. They specifically mentioned wanting to know the origin, temperature range, and recommended watering frequency of these plants. To fulfill their request, I will call the “get_by_category_for_house_plants” function with no arguments, which will retrieve the desired information for the user.
Action: get_by_category_for_house_plants
Action Input: {}

After Shuffle

User
I am interested in growing flowering house plants. Can you provide me with a list of flower category plants? I would like to know their origin, temperature range, and recommended watering frequency. Begin!

Assistant
Thought: The user wants details about the genre with the ID ‘rock’ on Deezer. To fulfill their request, I should call the ‘genre_for_deezer’ function with the argument ‘is_id’ set to ‘rock’. This will allow me to retrieve specific information about the ‘rock’ genre on Deezer.
Action: genre_for_deezer
Action Input:
{
  "is_id": "rock"
}

User
I want to surprise my family with a healthy breakfast. Can you suggest a random breakfast recipe that includes eggs? Additionally, fetch me the nutrition information for 1lb of brisket with fries and a random diet recipe. Begin!

Assistant
Thought: I called the ‘breakfast_ingredient_for_recipe_finder’ function with the argument (“ingredient”: “eggs”) because the user wants a random breakfast recipe that includes eggs. By specifying the ingredient as ‘eggs’, I can fetch recipes that have eggs as one of the ingredients. This will help me suggest a healthy breakfast recipe that includes eggs.
Action: breakfast_ingredient_for_recipe_finder
Action Input:
{
  "ingredient": "eggs"
}

After Shuffle

User
I want to surprise my family with a healthy breakfast. Can you suggest a random breakfast recipe that includes eggs? Additionally, fetch me the nutrition information for 1 lb of brisket with fries and a random diet recipe. Begin!

Assistant
Thought: I called the ‘v1_nutrition_for_nutrition_by_api_ninjas’ function with the argument (“query”: “1lb of brisket with fries”) because the user requested the nutrition information for 1lb of brisket with fries. By passing the query as ‘1lb of brisket with fries’, I can retrieve the nutrition data for this specific food combination. This will help me provide the user with the desired information regarding the nutritional content of their selected meal.
Action: v1_nutrition_for_nutrition_by_api_ninjas
Action Input:
{
  "query": "1lb of brisket with fries"
}

Figure 10: More Example of Shuffled Data. After shuffling, the assistant’s responses no longer correspond to the original queries. However, some tokens (boilerplate tokens, red) remain semantically similar to the original response and are therefore predictable. In contrast, reasoning tokens (green) no longer align with the query, resulting in noise.
<table>
  <tr>
    <th colspan="2">Failure Cases</th>
  </tr>
  <tr>
    <th colspan="2">Error in Original Data</th>
  </tr>
  <tr>
    <td colspan="2">
      Thought: Since the function "getbreadcrumbnavitem_for_betsports" failed to retrieve the breadcrumbnavigation item due to an endpoint error, I am now calling the "get_banners_new_for_betsports" function to obtain the banners for the sports section. This API call will be successful and provide the necessary information for the article.<br>
      Action: superoffer_for_betsports<br>
      Action Input: {}

    </td>
  </tr>
  <tr>
    <th colspan="2">In Shuffled Data</th>
  </tr>
  <tr>
    <td colspan="2">
      Thought: I received the user's request to store UUIDs of players in a Minecraft plugin. They specifically asked for the UUID and corresponding names of Kara_Danvers, Barry_Allen, and Oliver_Queen. To fulfill this request, I called the 'name_to_uuid_for_minecraft_uuid_converter' function with the argument 'name': 'Kara_Danvers'. This API call will convert the Mojang Minecraft username of Kara_Danvers into a UUID. By doing this, I'll be able to store the required information for the players.<br>
      Action: name_to_uuid_for_minecraft_uuid_converter<br>
      Action Input: {
        "name": "Kara_Danvers"
      }
    </td>
  </tr>
  <tr>
    <td colspan="2">
      {
        "tool_calls": [
          {"name": "purgequeue", "arguments": {"phone_id": "1234567890"}},
          {"name": "purgequeue", "arguments": {"phone_id":"9876543210"}}
        ]
      }
    </td>
  </tr>
</table>

Figure 11: Failure Cases. We identify two types of failure cases for SHAD: 1) In shuffled Data, where data is sampled for shuffle train, 2) Error in Original Data, SHAD may misclassify the error in the original data as reason tokens.

Reviewer paper summaries: This paper introduces "Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning," presenting a novel approach to enhance the fine-tuning of Large Language Models (LLMs) for agent tasks. The authors identify that reasoning tokens and boilerplate tokens differ in their importance and learning complexity, advocating for their separate treatment. To achieve this, they propose the Shuffle-Aware Discriminator (SHAD), which distinguishes between these token types based on their predictability after shuffling input-output pairs. Building on SHAD, the authors develop the Reasoning-highlighted Fine-Tuning (RFT) method, which emphasizes reasoning tokens during the fine-tuning process. Empirical evaluations demonstrate that RFT significantly outperforms traditional supervised fine-tuning methods across various agent benchmarks, showcasing improved multi-step reasoning and tool use capabilities in LLMs.

This paper introduces a novel approach called SHuffle-Aware Discriminator (SHAD), which differentiates between boilerplate tokens and reasoning tokens in structured samples used in reasoning chains with large language models (LLMs). The authors argue that overfitting to boilerplate components can degrade the agent's reasoning capabilities.  The paper introduces an innovative approach that fine-tunes an LLM using a small subset of shuffled data, then compares the token-level loss between the fine-tuned and original models to classify tokens as either boilerplate or reasoning tokens within the target dataset.
Summary of strengths: 1. The introduction of SHAD is a significant contribution, offering an automated and adaptive method to differentiate between reasoning and boilerplate tokens. By leveraging the predictability differences after shuffling data, SHAD effectively isolates tokens that are crucial for agent capabilities, addressing a key limitation in existing fine-tuning methodologies.

2. The proposed RFT method demonstrates substantial performance gains over standard supervised fine-tuning. By selectively emphasizing reasoning tokens, RFT ensures that the model retains and enhances its agent-related functionalities, such as multi-step reasoning and tool use, without being bogged down by the repetitive boilerplate tokens.

3. The paper provides robust experimental results across multiple agent-task benchmarks, validating the effectiveness of SHAD and RFT. The consistent improvements in performance metrics underscore the practical applicability and reliability of the proposed methods in real-world scenarios.

The difference between this paper and existing work is that instead of considering format tokens only, it also takes into account the template-connecting tokens. Also, it further proposes a reasoning-focused fine-tuning strategy, which prioritizes reasoning tokens by assigning them dynamic weights to enhance their learning.

Extensive experiments are conducted to demonstrate the effectiveness of the proposed method across multiple agent benchmarks, showing its ability to accurately identify reasoning tokens and improve their representation and learning. It also maintains good generaliztion performance on out-of-distribution benchmarks.
Summary of weaknesses: 1. While the empirical results are compelling, the paper lacks a deeper theoretical analysis explaining why SHAD effectively distinguishes between token types. A more thorough exploration of the underlying principles could provide stronger justification for the observed performance improvements.

2. The scalability of SHAD and RFT, especially when applied to extremely large datasets or diverse domains, is not extensively addressed. It remains unclear how these methods perform in terms of computational resources and time, which is crucial for their adoption in large-scale applications.

3. The effectiveness of SHAD relies heavily on the shuffling of input-output pairs to create predictability differences. The paper does not explore how variations in shuffling strategies might affect the classification accuracy of SHAD or the overall fine-tuning performance, potentially limiting the method's robustness.

The proposed method builds on the insight that boilerplate tokens are sample-independent and consistent across samples. It compares token-level loss changes before and after data shuffling: tokens with a positive loss difference are classified as reasoning tokens, while others are categorized as boilerplate tokens. However, the paper lacks a robust intrinsic evaluation to assess the accuracy of this classification. Experiments are limited to a subset of tokens identifiable through regular expressions, making it difficult to verify the classification's correctness under broader assumptions. Additionally, key evaluation details, such as the definition of misclassification rate and metrics like precision and recall, are not clearly provided.
Comments, suggestions and typos: na

I did not notice any significant typos in this paper.
Overall assessment scores: 3.5

3
Metareview score: 3
Metareview: This paper presents a novel approach called SHuffle-Aware Discriminator (SHAD) to enhance the fine-tuning of Large Language Models (LLMs) for reasoning tasks. The authors argue that differentiating between reasoning tokens and boilerplate tokens is crucial, as overfitting to boilerplate components can degrade reasoning capabilities. SHAD works by fine-tuning an LLM on a small subset of shuffled data and comparing the token-level loss between the fine-tuned and original models to classify tokens. Based on SHAD, the authors develop the Reasoning-highlighted Fine-Tuning (RFT) method, which emphasizes reasoning tokens during fine-tuning. Empirical evaluations show that RFT significantly outperforms traditional supervised fine-tuning methods across various benchmarks, demonstrating improved reasoning and tool use capabilities in LLMs.
Summary of reasons to publish: This paper addresses the critical issue of model overfitting on boilerplate tokens and underfitting on reasoning tokens, which is particularly important for applications with structured outputs. The novel approach, SHuffle-Aware Discriminator (SHAD), distinguishes between these token types by fine-tuning an LLM on shuffled data and comparing token-level losses. Unlike existing methods, SHAD considers both format and template-connecting tokens and introduces a reasoning-focused fine-tuning strategy (RFT) that assigns dynamic weights to reasoning tokens to enhance their learning.
Extensive experiments demonstrate the effectiveness of this method across multiple agent benchmarks, showing improved identification and representation of reasoning tokens, as well as better generalization performance on out-of-distribution benchmarks. This approach significantly outperforms traditional supervised fine-tuning methods, enhancing multi-step reasoning and tool use capabilities in LLMs.