HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
Authors: Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin
Accepted at: acl25

HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning

Xiaoyuan Li1, Moxin Li3*, Rui Men2*, Yichang Zhang2, Keqin Bao1, Wenjie Wang1, Fuli Feng1, Dayiheng Liu2, Junyang Lin2
University of Science and Technology of China1
Alibaba Group2
National University of Singapore3

Abstract

Large language models (LLMs) have shown remarkable capabilities in commonsense reasoning; however, some variations in questions can trigger incorrect responses. Do these models truly understand commonsense knowledge, or just memorize expression patterns? To investigate this question, we present the first extensive robustness evaluation of LLMs in commonsense reasoning. We introduce HellaSwag-Pro, a large-scale bilingual benchmark consisting of 11,200 cases, by designing and compiling seven types of question variants. To construct this benchmark, we propose a two-stage method to develop Chinese HellaSwag, a finely annotated dataset comprising 12,000 instances across 56 categories. We conduct extensive experiments on 41 representative LLMs, revealing that these LLMs are far from robust in commonsense reasoning. Furthermore, this robustness varies depending on the language in which the LLM is tested. This work establishes a high-quality evaluation benchmark, with extensive experiments offering valuable insights to the community in commonsense reasoning for LLMs.

1 Introduction

Commonsense reasoning is a crucial part of intelligence, involving contextual understanding, implicit knowledge, and logical deduction (Liu and Singh, 2004; Cambria et al., 2011; Davis and Marcus, 2015). Recent studies have focused on enhancing these capabilities in LLMs, achieving impressive performance (Yang et al., 2024; OpenAI et al., 2024; Team et al., 2024). However, even slight changes to questions can lead to incorrect responses from the same models. For instance, in binary commonsense questions, human naturally recognizes both correct and incorrect options through a single inference process, while LLMs,

*Corresponding authors. Work done when Xiaoyuan Li and Keqin Bao were intern at Alibaba Group.

![Comparison of GPT-4o’s responses to an original question and its several meaning-preserving variants. GPT-4o successfully handles the original question but struggles with its variants on the same knowledge.](page_1012_370_388_388.png)

Figure 1: Comparison of GPT-4o’s responses to an original question and its several meaning-preserving variants. GPT-4o successfully handles the original question but struggles with its variants on the same knowledge.

though able to identify the correct answer, struggle to reason about why the alternative is wrong (Balepur et al., 2024). Therefore, we ask the question: Does this high-level performance stem from a genuine understanding of commonsense knowledge, or is it simply a result of memorizing specific expression patterns in pre-training data?

To answer this question, an effective approach is to systematically evaluate the robustness of LLMs in answering commonsense reasoning questions. As illustrated in Figure 1, we find that GPT-4o correctly answers an original question but fails on its variants, i.e., questions about the same commonsense knowledge but in different reasoning forms, such as reverse conversion. This indicates that GPT-4o has not fully grasped the commonsense knowledge behind the question; a genuine understanding of commonsense knowledge should be able to generalize to these question variants.

However, existing benchmarks do not yet sup-
<table>
  <tr>
    <th>Variant Type</th>
    <th>Context</th>
    <th>Choices</th>
  </tr>
  <tr>
    <td rowspan="1">Initial data</td>
    <td>A lady walks to a barbell. She bends down and grabs the pole. The lady</td>
    <td>A. stands and lifts the weight over her head.<br>B. swings and lands in her arms.<br>C. pulls the barbell forward.<br>D. pulls a rope attached to the barbell.</td>
  </tr>
  <tr>
    <td rowspan="1">Problem restatement</td>
    <td><b>A woman approaches a weightlifting bar. She lowers her body and grasps the metal rod. The woman</b></td>
    <td>A. rises and hoists the barbell above her head.<br>B. swings and lands in her arms.<br>C. pulls the barbell forward.<br>D. pulls a rope attached to the barbell.</td>
  </tr>
  <tr>
    <td rowspan="1">Reverse conversion</td>
    <td>The lady stands and lifts the weight over her head. Which could be the most possible context for this action?</td>
    <td>A. A lady walks to a barbell. She bends down and grabs the pole.<br>B. A lady positions herself at the squat rack. She lowers her body before rising steadily.<br>C. A lady approaches the kettlebell set. She swings the weight forcefully between her legs.<br>D. A lady stands beside the bench press station. She lies down and lifts the barbell from her chest.</td>
  </tr>
  <tr>
    <td rowspan="1">Causal inference</td>
    <td>A lady walks to a barbell. She bends down and grabs the pole. The lady stands and lifts the weight over her head. Which could be the most possible reason for this action?</td>
    <td>A. She is performing a weightlifting exercise.<br>B. She is using the barbell as a decoration for an event.<br>C. She is moving the barbell to a different location in the gym.<br>D. She is cleaning the barbell after a workout session.</td>
  </tr>
  <tr>
    <td rowspan="1">Sentence ordering</td>
    <td>1. She bends down and grabs the pole.<br>2. A lady walks to a barbell.<br>3. The lady stands and lifts the weight over her head. Which is the correct order?</td>
    <td>A. 2-1-3<br>B. 3-1-2<br>C. 2-3-1<br>D. 1-3-2</td>
  </tr>
  <tr>
    <td rowspan="1">Scenario refinement</td>
    <td>A lady walks to a barbell. She bends down and grabs the pole. The lady hesitates for a moment, then changes her mind. Instead of lifting the barbell, she</td>
    <td>A. swings and lands in her arms.<br>B. stands and lifts the weight over her head.<br>C. pulls the barbell forward.<br>D. pulls a rope attached to the barbell.</td>
  </tr>
  <tr>
    <td rowspan="1">Negative transformation</td>
    <td>A woman approaches a weightlifting bar. She lowers her body and grasps the metal rod. The lady <u>will not</u></td>
    <td>A. swings and lands in her arms.<br>B. stands and lifts the weight over her head.<br>C. bend her knees and lift the barbell.<br>D. adjust her grip and lift the weight.</td>
  </tr>
  <tr>
    <td rowspan="1">Critical testing</td>
    <td>A lady walks to a barbell. She bends down and grabs the pole. The lady <u>suddenly realizes she forgot her weightlifting gloves</u> and decides to postpone her workout. The lady</td>
    <td>A. stands and lifts the weight over her head.<br>B. swings and lands in her arms.<br>C. pulls the barbell forward.<br>D. pulls a rope attached to the barbell.<br>E. None of the above four options are suitable.</td>
  </tr>
</table>

Table 1: Examples of the seven variants we adopt for an initial question, with the correct answer unchanged as (A). Modifications are highlighted in different colors for clarity.

port a thorough evaluation of LLM robustness in commonsense reasoning. Most work evaluates LLMs on general benchmarks (Zellers et al., 2019; Talmor et al., 2019; Mihaylov et al., 2018a), or in specific domains of commonsense knowledge (Zhou et al., 2019; Qin et al., 2021; Bisk et al., 2020). Although some efforts have considered the robustness of commonsense reasoning, they either focus on whether models can learn genuine question-answer correlations under initial questions (Jia and Liang, 2017; Branco et al., 2021), or examine only one type of simplistic question variant such as question paraphrasing (Zhou et al., 2021; Ismayilzada et al., 2023; Balepur et al., 2024), lacking investigation into robustness across diverse and complex variants.

To address this gap, we present the first extensive evaluation on the robustness of commonsense reasoning for LLMs, starting with dataset construction. Firstly, recognizing that existing benchmarks are predominantly in English, which limits the assessment of non-English LLMs (Davis, 2023), we develop a Chinese commonsense reasoning dataset based on the widely-used HellaSwag benchmark (Zellers et al., 2019), containing 12,000 questions. Specifically, we design 56 fine-grained categories, and propose a two-stage data annotation method including initial dataset generation and difficult sample replacement. Secondly, we design and compile seven variants from existing studies (cf. Table 1), which can be characterized under Bloom Cognitive Model (cf. Appendix A). We then create the variants for the Chinese and English versions of HellaSwag, obtaining HellaSwag-Pro, a high-quality human-verified dataset with 11,200 variants from 1,600 original questions.

Using HellaSwag-Pro, we conduct a comprehensive evaluation on the robustness of 41 closed-source and open-source LLMs with nine different prompt strategies. We derive several key findings: (1) All LLMs are far from robust in commonsense reasoning tasks, as evidenced by their poor performance on question variants and the significant gap compared to human performance. Neverthe-
less, GPT-4o achieves the best robustness among all the evaluated LLMs. (2) Among all types of variants, negative transformation is the most challenging, with an average accuracy of only 9.01%, while problem restatement poses minimal difficulty. (3) LLMs achieve the best robustness in the language on which they were adequately trained. (4) Incorporating chain-of-thought (CoT) reasoning and using few-shot demonstrations can strengthen their robustness.

Our contributions are three-fold. (1) We present the first extensive evaluation on the robustness of commonsense reasoning for LLMs by designing and compiling seven types of variants. (2) We have developed a bilingual, large-scale, human-annotated benchmark for evaluating LLM robustness in commonsense reasoning, which will be publicly released upon acceptance. (3) We conduct in-depth experiments on 41 representative LLMs with diverse prompts, yielding critical insights.

2 Chinese HellaSwag

Given the limitation that most existing benchmarks for commonsense reasoning are in English, we begin by building a Chinese benchmark for commonsense reasoning that captures unique aspects of Chinese cultural context. Firstly, we structure the dataset following the format of HellaSwag (Zellers et al., 2019), a widely recognized English commonsense reasoning benchmark, which consists of multiple-choice questions with four answer options. Secondly, to minimize manual effort, we incorporate Qwen-Max (Yang et al., 2024), a state-of-the-art Chinese LLM, into the dataset construction process. Finally, to enhance the diversity of the dataset, we develop a hierarchical taxonomy of commonsense knowledge, as shown in Figure 3. Our taxonomy consists of seven broad categories summarized from existing literature (Zellers et al., 2019; Koupaee and Wang, 2018; Caba Heilbron et al., 2015), each containing eight subcategories. We aim to construct our dataset based on the taxonomy, where we inject the categorical information into the instruction for LLM generation.

We propose a two-stage data construction pipeline, initial dataset generation and difficult sample replacement, as shown in Figure 2.

Initial Dataset Generation In this stage, we employ an over-generate-then-filter (Yuan et al., 2023) approach, i.e., generating excessive question-answer pairs and filtering for high quality ones, to obtain the initial dataset. The generation of the initial dataset consists of three steps.

• Step 1: Context over-generation. We employ the LLM to create a Chinese context of the question via in-context learning (Brown et al., 2020), incorporating category information, length requirement, and carefully crafted five-shot Chinese examples similar to HellaSwag. For the length requirement, we assign three tiers: short (under 20 characters), medium (20-40 characters), and long (over 40 characters). We then filter the generated contexts based on character count and Jaccard similarity, eliminating samples that do not meet the length requirement or are too similar to other samples.

• Step 2: Choice over-generation. For each context, we instruct the LLM to over-generate ten potential choices, forming a question.

• Step 3: Choice filtering. We instruct the LLM to evaluate each question on a ten-point scale and select six choices: one correct answer (10 points) and five high-scoring incorrect choices. Then, human annotators select four choices, ensuring a single correct answer and three challenging incorrect choices, and check the category labels for the question. After the LLM scoring, we obtain 12,960 samples, which human annotators further refine to 12,287. To maintain category balance, we ultimately select 12,000 samples, allocating 1,500 to each broad category.

![The two-stage data construction pipeline for Chinese HellaSwag. See an example in Table 9.](page_1012_120_410_388.png)

Figure 2: The two-stage data construction pipeline for Chinese HellaSwag. See an example in Table 9.
Figure 3: Overview of Chinese HellaSwag categories. There are seven broad categories in total, each with eight detailed subcategories.

<table>
  <tr>
    <th>Length Type</th>
    <th>Long</th>
    <th>Medium</th>
    <th>Short</th>
    <th>Total</th>
  </tr>
  <tr>
    <td># Questions</td>
    <td>4,179</td>
    <td>4,033</td>
    <td>3,788</td>
    <td>12,000</td>
  </tr>
</table>

Table 2: Statistics for Chinese HellaSwag.

Difficult Sample Replacement After initial dataset generation, we notice that some incorrect choices are rather simple for LLMs to identify, making the Chinese HellaSwag much easier than its English counterpart. Following the adversarial filtering (Zellers et al., 2018), we use a human-in-the-loop adversarial filtering method (Step 4) to further enhance the dataset’s difficulty. This process involves using a generator LLM to rewrite existing incorrect choices into more challenging ones, and then evaluating the generated choices on multiple discriminator LLMs. If the generated choice successfully misleads the discriminator LLMs, we replace the original choice with the newly generated one. Finally, human annotators filter out the generated choices that are too difficult for humans to identify (see detail in Appendix B). We iterative perform this process until the Chinese HellaSwag achieves accuracy comparable to the English HellaSwag, resulting in replacing 2451 samples. The dataset statistics of the Chinese HellaSwag can be found in Table 2. The complete evaluation of it is given in Appendix E.

3 HellaSwag-Pro

Based on the English-Chinese HellaSwag datasets, we construct HellaSwag-Pro, the benchmark for extensive robustness evaluation of commonsense reasoning. We begin by designing the seven-type question variants for robustness evaluation, then detail our data generation process.

3.1 Variant Types

We aim to evaluate the robustness of commonsense reasoning on question variants of changed reasoning forms for the same commonsense knowledge. The rationality is that the diverse reasoning forms disables the reliance on superficial patterns, ensuring that correct answers from LLMs demonstrate a robust understanding of the underlying commonsense knowledge. Building on existing research (Guo et al., 2024; Ma et al., 2025; Balepur et al., 2024) and our own designs, we maintain seven types of variants, as detailed below.

• Problem restatement aims to test the impact of textual description variations on model understanding. We rephrase the context and correct choice while keeping the incorrect choices unchanged, thereby increasing the difficulty of identifying the correct answer.

• Reverse conversion evaluates the capability for reverse reasoning, i.e., inferring the context from the outcome, which has been shown to be challenging for LLMs (Guo et al., 2024). We utilize the original correct choice as the context, the original context as the correct choice, and generate three additional incorrect choices.

• Causal inference evaluates the understanding of the causality of the event (Li et al., 2025). We merge the context and the correct choice and ask for the reason. We generate one correct reason and produce three additional incorrect reasons as the choices.

• Sentence ordering focuses on the understanding of inter-sentence relationships, such as progression or contrast. We concatenate the context and correct choice into a complete paragraph, then shuffle the order of the sentences. The correct choice refers to the original sentence ordering.

• Scenario refinement investigates the ability to infer counterfactual situations (Ma et al., 2025). We select a relatively plausible choice from the original incorrect choices, then minimally modify the context to make this choice correct, where the original correct choice becomes incorrect.

• Negation transformation examines the robustness to negation, a known challenge for LLMs (Balepur et al., 2024). This involves altering the
context by introducing negations, such as changing "the man will" to "the man will not." In this transformation, the least plausible choice in the original question becomes the correct answer for the variant, while the original correct answer is retained, and two additional plausible options are generated as distractors.

• Critical testing evaluates the model’s ability to abstain from answering when the context lacks sufficient information to determine a correct answer. We remove key details from the context to make all original choices invalid. We keep the context minimally modified to increase difficulty. A new choice, “None of the above four options are suitable”, is introduced as the correct choice.

3.2 Data Generation

To construct these variants, we also employ Qwen-Max due to its comparatively strong language ability in reforming the questions. We design in-context examples and instructions with transformation rules to guide Qwen-Max to generate the question variants (cf. Appendix B.4.3). However, we observe that Qwen-Max is not consistently reliable, exhibiting issues such as: (1) generating variants inconsistent with the definitions, (2) producing multiple correct choices or overly simple incorrect choices, and (3) generating invalid contexts, particularly in scenario refinement.

To tackle these issues, we leverage manual quality control over the generated data. For reverse conversion and causal inference, we adopt an over-generate-then-filter approach (cf. Section 2) to control the correctness and the quality of the generated choices. Finally, we conduct comprehensive manual verification of all variants generated to ensure data quality. We initially generate 24,260 variants, and eventually filter down to 11,200 high-quality variants from 1,600 original questions.

4 Experiment

In this section, we conduct extensive experiments to evaluate the performance of various LLMs on our HellaSwag-Pro benchmark. Our study is guided by three key research questions: RQ1: How do different LLMs perform across all variants? RQ2: What is the relative difficulty of different variants? RQ3: Which prompting strategies yield the best robustness in LLMs?

4.1 Experimental Setup

Model Selection and Implementation Details
We select 41 representative closed-source and open-source LLMs. For English LLMs, we use GPT-4o (OpenAI, 2023), Claude-3.5-Sonnet (Anthropic, 2024), Gemini-1.5-Pro (Anil et al., 2023), Mistral series (Jiang et al., 2023), Llama3 series (Dubey et al., 2024) and Gemma2 series (Rivière et al., 2024). For Chinese LLMs, we use Qwen-Max (Bai et al., 2023), Qwen2.5 series (Yang et al., 2024), InternLM2.5 series (Team, 2023), Yi1.5 series (Young et al., 2024), Baichuan2 series (Yang et al., 2023) and DeepSeek series (Bi et al., 2024).

We integrate both the Chinese HellaSwag and HellaSwag-Pro into the lm-evaluation-harness platform (Gao et al., 2024). For the open-source models, we use the default settings of the platform: do_sample is set to false and the temperature is set to the default value of the hugging-face library as 1.0. For the closed-source models, we set the temperature to 0.7. In addition, we set the maximum output length to 1024.

Prompting Strategy  We design nine prompting strategies to evaluate the LLMs across different languages and number of demonstrations. (1) Direct: LLM takes the original dataset question directly as input1. (2) CN-CoT: LLM is instructed to perform CoT in Chinese, regardless of the language of the dataset. (3) EN-CoT: LLM is instructed to perform CoT in English. (4) CN-XLT: LLM is instructed to first translate the English question into Chinese, then reason in Chinese. (5) EN-XLT: LLM is instructed to first translate the Chinese question into English, then reason in English. The last four strategies include both zero-shot and three-shot variants.

Evaluation Metric  We consider four evaluation metrics to measure the performance and robustness of LLMs. Denote the original dataset \( \mathcal{D} = \{(x, y)\} \), where \( x \) and \( y \) represent the question and the correct label, respectively. Denote the dataset of all seven-type variants \( \mathcal{D}_r = \{(x', y')\} \), where each \( (x', y') \) corresponds to an original \( (x, y) \) in \( \mathcal{D} \).

Original Accuracy (OA) measures the accuracy

1For open-source models, the Direct approach follows the official HellaSwag implementation, computing the log-likelihood for each option and selecting the one with the highest value. We report the normalized accuracy to account for the option length. Other prompting strategies use a generation setup and report accuracy based on exact match.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Chinese</th>
    <th colspan="4">English</th>
    <th colspan="4">AVG</th>
  </tr>
  <tr>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>96.41</td>
    <td>97.79</td>
    <td>-1.38</td>
    <td>92.03</td>
    <td>95.56</td>
    <td>96.04</td>
    <td>-0.48</td>
    <td>90.02</td>
    <td>95.99</td>
    <td>96.92</td>
    <td>-0.93</td>
    <td>91.03</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>0.00</td>
    <td>0.0015</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>0.00</td>
    <td>0.0015</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>0.00</td>
    <td>0.0015</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>93.50</td>
    <td>84.82</td>
    <td>8.68</td>
    <td>78.91</td>
    <td>87.60</td>
    <td>62.61</td>
    <td>24.99</td>
    <td>59.65</td>
    <td>90.55</td>
    <td>73.72</td>
    <td>16.83</td>
    <td>69.28</td>
  </tr>
  <tr>
    <th colspan="14">Closed-source LLMs</th>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>60.75</td>
    <td>45.18</td>
    <td><b>15.57</b></td>
    <td>28.70</td>
    <td>49.50</td>
    <td>38.21</td>
    <td><b>11.29</b></td>
    <td>20.57</td>
    <td>55.13</td>
    <td>41.70</td>
    <td><b>13.43</b></td>
    <td>24.64</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>63.25</td>
    <td>46.16</td>
    <td>17.09</td>
    <td>29.89</td>
    <td>56.88</td>
    <td>39.57</td>
    <td>17.30</td>
    <td>23.48</td>
    <td>60.06</td>
    <td>42.87</td>
    <td>17.20</td>
    <td>26.69</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>67.50</td>
    <td>48.75</td>
    <td>18.75</td>
    <td>33.79</td>
    <td>61.75</td>
    <td>39.98</td>
    <td>21.77</td>
    <td>25.75</td>
    <td>64.63</td>
    <td>44.37</td>
    <td>20.26</td>
    <td>29.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>67.63</td>
    <td>50.59</td>
    <td>17.04</td>
    <td>35.62</td>
    <td>65.63</td>
    <td>43.93</td>
    <td>21.70</td>
    <td>30.77</td>
    <td>66.63</td>
    <td>47.26</td>
    <td>19.37</td>
    <td>33.20</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>69.00</td>
    <td>51.41</td>
    <td>17.59</td>
    <td>35.84</td>
    <td>68.50</td>
    <td>45.20</td>
    <td>23.30</td>
    <td>32.12</td>
    <td>68.75</td>
    <td>48.30</td>
    <td>20.45</td>
    <td>33.98</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>69.75</td>
    <td>53.11</td>
    <td>16.64</td>
    <td>37.54</td>
    <td>70.00</td>
    <td>46.10</td>
    <td>23.90</td>
    <td>32.68</td>
    <td>69.88</td>
    <td>49.61</td>
    <td>20.27</td>
    <td>35.11</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>70.87</td>
    <td><b>54.75</b></td>
    <td>16.12</td>
    <td><b>39.64</b></td>
    <td>72.00</td>
    <td><b>47.75</b></td>
    <td>24.25</td>
    <td><b>35.12</b></td>
    <td><b>71.44</b></td>
    <td><b>51.25</b></td>
    <td>20.19</td>
    <td><b>37.38</b></td>
  </tr>
  <tr>
    <td>Baichuan2-7B</td>
    <td>67.00</td>
    <td>46.16</td>
    <td>20.84</td>
    <td>31.50</td>
    <td>60.62</td>
    <td>39.04</td>
    <td>21.58</td>
    <td>25.21</td>
    <td>63.81</td>
    <td>42.60</td>
    <td>21.21</td>
    <td>28.36</td>
  </tr>
  <tr>
    <td>Baichuan2-13B</td>
    <td>69.13</td>
    <td>46.98</td>
    <td>22.15</td>
    <td>33.45</td>
    <td>64.62</td>
    <td>38.82</td>
    <td>25.80</td>
    <td>26.07</td>
    <td>66.88</td>
    <td>42.90</td>
    <td>23.97</td>
    <td>29.76</td>
  </tr>
  <tr>
    <td>DeepSeek-/B</td>
    <td>68.13</td>
    <td>47.96</td>
    <td>20.17</td>
    <td>33.30</td>
    <td>63.38</td>
    <td>40.39</td>
    <td>22.99</td>
    <td>26.70</td>
    <td>65.76</td>
    <td>44.18</td>
    <td>21.58</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>DeepSeek-6/B</td>
    <td><b>71.50</b></td>
    <td>49.21</td>
    <td>22.29</td>
    <td>35.89</td>
    <td>71.37</td>
    <td>40.63</td>
    <td>30.75</td>
    <td>29.71</td>
    <td><b>71.44</b></td>
    <td>44.92</td>
    <td>26.52</td>
    <td>32.80</td>
  </tr>
  <tr>
    <td>InferM.LM2.5-1.8B</td>
    <td>61.62</td>
    <td>42.07</td>
    <td>19.55</td>
    <td>26.99</td>
    <td>55.37</td>
    <td>38.46</td>
    <td>16.91</td>
    <td>22.61</td>
    <td>58.50</td>
    <td>40.27</td>
    <td>18.23</td>
    <td>24.80</td>
  </tr>
  <tr>
    <td>InferM.LM2.5-7B</td>
    <td>67.25</td>
    <td>49.77</td>
    <td>17.48</td>
    <td>34.57</td>
    <td>69.50</td>
    <td>40.89</td>
    <td>28.61</td>
    <td>29.75</td>
    <td>68.38</td>
    <td>45.33</td>
    <td>23.04</td>
    <td>32.16</td>
  </tr>
  <tr>
    <td>InferM.LM2.5-20B</td>
    <td>67.37</td>
    <td>48.08</td>
    <td>19.29</td>
    <td>32.31</td>
    <td><b>73.62</b></td>
    <td>41.11</td>
    <td>32.51</td>
    <td>31.23</td>
    <td>70.50</td>
    <td>44.60</td>
    <td>25.90</td>
    <td>32.22</td>
  </tr>
  <tr>
    <td>YiLL-6B*</td>
    <td>67.00</td>
    <td>49.59</td>
    <td>17.41</td>
    <td>34.27</td>
    <td>64.38</td>
    <td>39.37</td>
    <td>25.01</td>
    <td>26.62</td>
    <td>65.69</td>
    <td>44.48</td>
    <td>21.21</td>
    <td>30.45</td>
  </tr>
  <tr>
    <td>YiLL-9B*</td>
    <td>68.50</td>
    <td>50.18</td>
    <td>18.32</td>
    <td>35.55</td>
    <td>66.37</td>
    <td>39.58</td>
    <td>26.79</td>
    <td>27.48</td>
    <td>67.44</td>
    <td>44.88</td>
    <td>22.56</td>
    <td>31.52</td>
  </tr>
  <tr>
    <td>YiLL-34B</td>
    <td>71.00</td>
    <td>52.23</td>
    <td>18.77</td>
    <td>38.09</td>
    <td>71.00</td>
    <td>40.75</td>
    <td>30.25</td>
    <td>29.91</td>
    <td>71.00</td>
    <td>46.49</td>
    <td>24.51</td>
    <td>34.00</td>
  </tr>
</table>

Table 3: Results of existing Chinese LLMs on HellaSwag-Pro using Direct prompt. “AVG” indicates the average performance on Chinese and English parts of the dataset. The best results in each model category are bolded.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Chinese</th>
    <th colspan="4">English</th>
    <th colspan="4">AVG</th>
  </tr>
  <tr>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
  </tr>
  <tr>
    <th colspan="14">Closed-source LLMs</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>91.37</td>
    <td><b>81.97</b></td>
    <td>9.40</td>
    <td>75.55</td>
    <td>88.63</td>
    <td>70.17</td>
    <td>18.46</td>
    <td>63.06</td>
    <td>90.00</td>
    <td><b>76.07</b></td>
    <td>13.93</td>
    <td><b>69.31</b></td>
  </tr>
  <tr>
    <td>Claude-3.5</td>
    <td><b>95.37</b></td>
    <td>80.15</td>
    <td>15.22</td>
    <td>75.04</td>
    <td>85.11</td>
    <td>66.02</td>
    <td>19.08</td>
    <td>57.20</td>
    <td><b>90.24</b></td>
    <td>73.09</td>
    <td>17.15</td>
    <td>66.12</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>90.62</td>
    <td>78.36</td>
    <td>12.26</td>
    <td>70.48</td>
    <td>87.75</td>
    <td>60.74</td>
    <td>27.01</td>
    <td>58.27</td>
    <td>89.19</td>
    <td>69.55</td>
    <td>19.63</td>
    <td>64.38</td>
  </tr>
  <tr>
    <th colspan="14">Open-source LLMs</th>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>59.13</td>
    <td>46.62</td>
    <td>12.51</td>
    <td>28.23</td>
    <td>66.25</td>
    <td>40.21</td>
    <td>26.04</td>
    <td>27.34</td>
    <td>62.69</td>
    <td>43.42</td>
    <td>19.27</td>
    <td>27.79</td>
  </tr>
  <tr>
    <td>Llama3-70B</td>
    <td>65.75</td>
    <td>48.63</td>
    <td>17.12</td>
    <td>32.70</td>
    <td><b>72.50</b></td>
    <td>41.27</td>
    <td>31.23</td>
    <td><b>30.63</b></td>
    <td><b>69.13</b></td>
    <td>44.95</td>
    <td>24.18</td>
    <td>31.67</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1</td>
    <td>57.75</td>
    <td>46.25</td>
    <td><b>11.50</b></td>
    <td>27.57</td>
    <td>67.50</td>
    <td><b>41.52</b></td>
    <td>25.98</td>
    <td>28.93</td>
    <td>62.63</td>
    <td>43.88</td>
    <td>18.74</td>
    <td>28.25</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1</td>
    <td>63.62</td>
    <td>46.80</td>
    <td>16.82</td>
    <td>30.82</td>
    <td>69.75</td>
    <td>41.21</td>
    <td>28.54</td>
    <td>29.39</td>
    <td>66.69</td>
    <td>44.01</td>
    <td>22.68</td>
    <td>30.11</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1</td>
    <td>66.00</td>
    <td><b>50.73</b></td>
    <td>15.27</td>
    <td><b>34.32</b></td>
    <td>72.12</td>
    <td>41.25</td>
    <td>30.87</td>
    <td>30.61</td>
    <td>69.06</td>
    <td><b>45.99</b></td>
    <td>23.07</td>
    <td><b>32.47</b></td>
  </tr>
  <tr>
    <td>Gemma2-2B</td>
    <td>61.88</td>
    <td>43.88</td>
    <td>16.51</td>
    <td>29.02</td>
    <td>59.62</td>
    <td>39.13</td>
    <td><b>20.50</b></td>
    <td>24.88</td>
    <td>60.75</td>
    <td>42.25</td>
    <td><b>18.50</b></td>
    <td>26.95</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td><b>69.13</b></td>
    <td>46.75</td>
    <td>22.38</td>
    <td>33.29</td>
    <td>64.88</td>
    <td>39.80</td>
    <td>25.08</td>
    <td>26.91</td>
    <td>67.01</td>
    <td>43.28</td>
    <td>23.73</td>
    <td>30.10</td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>63.38</td>
    <td>48.52</td>
    <td>14.86</td>
    <td>31.96</td>
    <td>71.88</td>
    <td>40.91</td>
    <td>30.97</td>
    <td>30.25</td>
    <td>67.63</td>
    <td>44.71</td>
    <td>22.92</td>
    <td>31.11</td>
  </tr>
</table>

Table 4: Results of existing English LLMs on HellaSwag-Pro using Direct prompt (Same settings as Table 3).

on original questions.

\[
OA = \frac{\sum_{(x,y) \in \mathcal{D}} \mathbb{I}[\text{LM}(x), y]}{|\mathcal{D}|}.
\]

Average Robust Accuracy (ARA) measures the average accuracy across all variants.

\[
ARA = \frac{\sum_{(x',y') \in \mathcal{D}_r} \mathbb{I}[\text{LM}(x'), y']}{|\mathcal{D}_r|}.
\]

Robust Loss Accuracy (RLA) refers to the performance gap between all variants and original questions, i.e., the difference between OA and ARA.

\[
RLA = OA - ARA.
\]

Consistent Robust Accuracy (CRA) refers to the joint accuracy of LLM correctly answering the variant and its original question, reflecting the LLM’s genuine understanding of the knowledge.

\[
CRA = \frac{\sum_{(x',y') \in \mathcal{D}_r} \mathbb{I}[\text{LM}(x), y] \cdot \mathbb{I}[\text{LM}(x'), y']}{|\mathcal{D}_r|}.
\]

4.2 LLM Performance (RQ1)

Overall Performance The results for Direct prompting on all LLMs are listed in Table 3 and Table 4. The main observations are as follows.

Firstly, all evaluated LLMs perform well in OA (e.g., in AVG OA, GPT-4o scores 90.00, and Claude-3.5 scores 90.24). However, all LLMs show a performance drop on variants, as evidenced by a positive AVG RLA value for all LLMs. In contrast, human receive a near-zero RLA value, suggesting that the question variants are not more challenging than the originals for human. This disparity further illustrates that current LLMs lack a true understanding of the commonsense knowledge and can easily be affected by the reasoning form.

Secondly, comparing open-source and closed-source LLMs, closed-source models achieve larger OA, ARA and CRA scores and smaller average RLA scores than open-source LLMs, indicating

2The results of instruct and chat models of Qwen2.5, LLaMA3 and Mixtral_v0.1 series are shown in Appendix F.
Figure 4: Pairwise performance statistics of the original question and its variant. We use “HellaSwag ✓ HellaSwag-Pro ✗” to denote that the LLM correctly answers the original question but fails on its variant.

better robustness in commonsense reasoning.

Finally, when we compare models within the same series (e.g., Qwen2.5, Llama3), we observe that larger models often achieve higher scores on OA, ARA, and CRA. However, their RLA shows no consistent relationship with model size. Across different families, AVG RLA patterns vary - fluctuating with size in Qwen2.5 and Gemma3, while increasing with size in Yi1.5 and Llama3. This indicates that larger model size does not guarantee better robustness.

Analysis on Reasoning Robustness To further analyze whether LLMs can maintain reasoning ability from the original question to its variant, Figure 4 presents the pairwise performance statistic of the original question and its variant. For all LLMs, a significant proportion of variants are answered incorrectly despite LLMs being able to solve the source example. More specifically, closed-source LLMs like GPT-4o and Qwen-Max achieve a 69% success rate on both HellaSwag and HellaSwag-Pro, with only 3% failing both. In contrast, open-source LLMs struggle with around 30% and 20%, respectively. This shows that closed-source LLMs achieve better alignment between the performance of the original question and its variant, thus better robustness in reasoning ability.

4.3 Variant Analysis (RQ2)

To further analyze the robustness on different variants, we assess the contribution of each variant to the RLA score, as shown in Figure 5. A higher contribution indicates more non-robust in that type. The key observations are as follows:

Problem restatement, causal inference, and sentence ordering are the least challenging. Almost all

LLMs perform well on these variants particularly closed-source LLMs and Qwen2.5 series, indicating that LLMs can effectively handle these forms.

Reverse conversion and critical testing each contribute about 10% to the RLA score. This indicates that current LLMs struggle to fully generalize to these variants, possibly because these variants do not largely exist in the training data.

Negative transformation and scenario refinement are the two most difficult variants, with negative transformation being particularly challenging. For almost all LLMs, these two variants account for more than 50% of the RLA score. This might be due to statistical bias in these two types of data during pre-training and the exploitation of shortcuts in the corpus (Chen et al., 2023; Wu et al., 2024).

Figure 5: Each variant’s contribution to the RLA score.

4.4 Different Prompting Strategies (RQ3)

To explore the impact of various prompting strategies on our benchmark, we test the performance of all LLMs under different prompting strategies (cf. Section 4.1). The results are summarized in Table 5.

For both Chinese and English datasets, Chinese LLMs perform best under CN-CoT strategy with shots, followed closely by EN-CoT with shots, achieving overall scores of 67.36% and 67.03%, respectively. Conversely, English LLMs show optimal performance using EN-CoT approach with shots, attaining 67.55% on the Chinese dataset and 60.36% on the English one. This shows that different LLMs favor the prompts in their native language. Besides, translating datasets into LLMs’ native languages before reasoning does not enhance performance (e.g., 28.69% for EN LLMs using EN-XLT with shots vs 41.69% for EN LLMs using Direct). This phenomenon is further illustrated in Figure 6.
Figure 6: Performance on different 3-shot prompts. For the legend, the first two parts are the prompt name, and the third part is the dataset language. NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th rowspan="2">Prompt Language</th>
    <th rowspan="2">#shot</th>
    <th colspan="3">LLM</th>
  </tr>
  <tr>
    <th>CN</th>
    <th>EN</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td colspan="6"><i>Chinese HellaSwag-Pro</i></td>
  </tr>
  <tr>
    <td>Direct</td>
    <td>-</td>
    <td>0</td>
    <td>48.95</td>
    <td>41.16</td>
    <td>45.06</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>3</td>
    <td><b>71.04</b></td>
    <td>51.90</td>
    <td>61.47</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>3</td>
    <td>70.95</td>
    <td><b>67.55</b></td>
    <td><b>69.25</b></td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>3</td>
    <td>41.48</td>
    <td>28.69</td>
    <td>35.09</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>0</td>
    <td>44.82</td>
    <td>23.89</td>
    <td>34.36</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>0</td>
    <td>45.38</td>
    <td>31.39</td>
    <td>38.39</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>0</td>
    <td>28.57</td>
    <td>12.93</td>
    <td>20.75</td>
  </tr>
  <tr>
    <td colspan="6"><i>English HellaSwag-Pro</i></td>
  </tr>
  <tr>
    <td>Direct</td>
    <td>-</td>
    <td>0</td>
    <td>47.46</td>
    <td>40.66</td>
    <td>44.06</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>3</td>
    <td><b>63.67</b></td>
    <td>47.24</td>
    <td>55.46</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>3</td>
    <td>63.12</td>
    <td><b>60.36</b></td>
    <td><b>61.74</b></td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>CN</td>
    <td>3</td>
    <td>48.77</td>
    <td>16.61</td>
    <td>32.69</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>0</td>
    <td>34.89</td>
    <td>18.25</td>
    <td>26.57</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>0</td>
    <td>42.41</td>
    <td>31.03</td>
    <td>36.72</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>0</td>
    <td>16.36</td>
    <td>11.22</td>
    <td>13.79</td>
  </tr>
  <tr>
    <td colspan="6"><i>HellaSwag-Pro</i></td>
  </tr>
  <tr>
    <td>Direct</td>
    <td>-</td>
    <td>0</td>
    <td>48.21</td>
    <td>40.91</td>
    <td>44.83</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>3</td>
    <td><b>67.36</b></td>
    <td>49.57</td>
    <td><b>58.46</b></td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>3</td>
    <td>67.03</td>
    <td><b>63.95</b></td>
    <td><b>65.49</b></td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>CN</td>
    <td>3</td>
    <td>59.91</td>
    <td>34.26</td>
    <td>47.08</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>3</td>
    <td>52.30</td>
    <td>44.52</td>
    <td>48.41</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>0</td>
    <td>39.86</td>
    <td>21.07</td>
    <td>30.46</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>0</td>
    <td>43.90</td>
    <td>31.21</td>
    <td>37.55</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>CN</td>
    <td>0</td>
    <td>30.59</td>
    <td>17.55</td>
    <td>24.07</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>0</td>
    <td>35.49</td>
    <td>21.98</td>
    <td>28.74</td>
  </tr>
</table>

Table 5: Average ARA of all open-source LLMs on different prompting strategies. CN-LLMs contains 17 LLMs, and EN-LLMs contains 7 LLMs. The best results for each dataset are bolded. Detailed results for all evaluated models are provided in the Appendix F.

Our findings differ from previous research (Huang et al., 2023; Shi et al., 2022), which suggested that translating non-English tasks into English (XLT) would perform better than using native languages. And these research only focused on English LLMs while overlooking Chinese LLMs. We find that LLMs perform better when reasoning directly in their native language compared to XLT, addressing this gap in previous research.

5 Related Work

Commonsense Reasoning Evaluation There are numerous benchmarks and datasets for commonsense reasoning, most of which are in English. Some studies focus on evaluating general commonsense knowledge (Zellers et al., 2019; Talmor et al., 2019; Mihaylov et al., 2018b). Others target specific aspects of commonsense reasoning (Zhou et al., 2019; Bisk et al., 2020; Sap et al., 2019; Lin et al., 2020; Clark et al., 2018; Khot et al., 2020; Li et al., 2024). There are some Chinese datasets for commonsense reasoning (Sun et al., 2024; Shi et al., 2024). For instance, CHARM (Sun et al., 2024) distinguishes between global commonsense and Chinese-specific commonsense but includes only a limited number of everyday commonsense cases. However, evaluations aimed at assessing the robustness of commonsense reasoning are still understudied.

Datasets on Different Reasoning Forms There are several datasets relevant to our variant design. For reverse reasoning, ART (Bhagavatula et al., 2020), δ-NLI (Rudinger et al., 2020), and CLUTRR (Sinha et al., 2019) explore different reasoning directions. FCR (Yang et al., 2022) and NatQuest (Ceraolo et al., 2024) evaluate causal reasoning, while TimeTravel (Qin et al., 2019) focuses on counterfactual scenario refinement. Additionally, PoE (Balepur et al., 2024) assesses reasoning involving negation. However, not all these datasets focus on commonsense reasoning, nor are they structured by original questions and their variants. Furthermore, they typically target limited reasoning types. Lastly, our dataset is large-scale and covers diverse commonsense knowledge.
Robustness and Consistency in LLMs  Early work focuses on adversarial attacks, with developing evaluation methods for reading comprehension systems (Jia and Liang, 2017), followed by universal adversarial triggers (Wallace et al., 2019). The field then expands to examine spurious correlations, with revealing how models often exploit superficial patterns rather than engaging in genuine reasoning (Branco et al., 2021; Geirhos et al., 2020). And Ross et al., 2022 investigates whether self-explanation can mitigate these spurious correlations. Coherence and consistency evaluation advances through classifier assessment methods (Storks and Chai, 2021) and analysis of accuracy-consistency trade-offs (Johnson and Marasovic, 2023). While these studies primarily address model robustness against adversarial attacks or spurious correlations, our work takes a novel approach by examining robustness in reasoning forms.

6 Conclusion

We conduct a systematic evaluation of the robustness of LLMs in commonsense reasoning in both Chinese and English. To facilitate this evaluation process, we introduce two large-scale, finely-annotated datasets: HellaSwag-Pro and Chinese HellaSwag. In addition, we design various prompts to evaluate 41 LLMs, offering several key findings that may advance the field of commonsense reasoning. We believe this work will serve as a valuable resource to support further research into the commonsense reasoning of LLMs.

Acknowledgements

This research was also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.

Limitations

The limitations of our work are as follows:

• Our work only addresses everyday commonsense reasoning and does not encompass specific types, such as temporal or physical commonsense knowledge. Evaluating the robustness of LLMs on these specific types of commonsense reasoning tasks will be our future work.

• HellaSwag-Pro is concentrated on assessing the robustness of LLMs in commonsense reasoning tasks and does not investigate the underlying reasons for observed performance declines.

• For the sake of evaluation convenience, our setup utilizes multiple-choice questions. We plan to study the open-ended questions in future work.

Ethics Statement

This work requires manual annotation. We provide annotators with a salary above the local minimum hourly wage. We have also clearly informed them about the purpose of the data and the necessity to ensure that all the data in Hellaswag-Pro does not contain any social biases, ethical concerns, or privacy issues.

Additionally, we develop a challenging dataset for evaluating the robustness of commonsense reasoning in this work. It’s important to emphasize that this dataset is intended solely for evaluation, not for training or fine-tuning purposes. We recognize that improper use of this dataset for model training or fine-tuning could lead to persistent inconsistencies in LLMs’ understanding of commonsense knowledge, potentially creating a vicious cycle where more such datasets would be needed to address these issues. Therefore, we explicitly state that the intended use of this dataset is strictly limited to evaluation to prevent the formation of long-standing issues in LLMs. We look forward to promoting healthy development in LLM research through responsible use of these research findings.

References

Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalmian, Jakub Sygnowski, and et al. 2023. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.11805.

Anthropic. 2024. Introducing the next generation of claude. https://www.anthropic.com/news/claude-3-family.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei
Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingtian Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Nishant Balepur, Shramay Palta, and Rachel Rudinger. 2024. It’s not easy being wrong: Large language models struggle with process of elimination reasoning. In Findings of the Association for Computational Linguistics ACL 2024, pages 10143–10166.

Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Quishi Du, Zhe Fu, Huazou Gao, Kaige Gao, Wenjun Gao, Ruqi Ge, Kang Guan, Duyu Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Alex X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihaq Zhu, and Yuheng Zou. 2024. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401.02954.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Ruben Branco, António Branco, Joao Rodrigues, and Joao Silva. 2021. Shortcutted commonsense: Data spuriousness in deep learning of commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1504–1521.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970.

Erik Cambria, Yangqiu Song, Haixun Wang, and Amir Hussain. 2011. Isanette: A common and common sense knowledge base for opinion mining. In 2011 IEEE 11th International Conference on Data Mining Workshops, pages 315–322. IEEE.

Roberto Ceraoło, Dmitrii Kharlapenko, Ahmad Khan, Amélie Reynold, Rada Mihalcea, Bernhard Schölkopf, Mrinmaya Sachan, and Zhijing Jin. 2024. Analyzing human questioning behavior and causal curiosity through natural queries.

Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023. Say what you mean! large language models speak too positively about negative commonsense knowledge. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9890–9908, Toronto, Canada. Association for Computational Linguistics.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Ernest Davis. 2023. Benchmarks for automated commonsense reasoning: A survey. ACM Computing Surveys, 56(4):1–41.

Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9):92–103.

Abhimanyu Dubey, Abhinav Jahuiri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravan Kumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes,
Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranae, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billoack, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstone, Joshua Saxe, Junting Jia, Kalyan Vasudean Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673.

Pei Guo, Wangjie You, Juntao Li, Yan Bowen, and Min Zhang. 2024. Exploring reversal mathematical reasoning ability for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 13671–13685.

Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365–12394, Singapore. Association for Computational Linguistics.

Mete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva, and Antoine Bosselut. 2023. CRoW: Benchmarking commonsense reasoning in real-world tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9785–9821, Singapore. Association for Computational Linguistics.

Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. CoRR, abs/2310.06825.

Jacob K Johnson and Ana Marasovic. 2023. How much consistency is your accuracy worth? EMNLP 2023, page 250.

Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090.

Mahnaz Koupaei and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. arXiv preprint arXiv:1810.09305.

David R Krathwohl. 1973. Taxonomy of educational objectives. Affective domain.

Xiaoyuan Li, Keqin Bao, Yubo Ma, Moxin Li, Wenjie Wang, Rui Men, Yichang Zhang, Fuli Feng, Dayiheng Liu, and Junyang Lin. 2025. Mtr-bench: A comprehensive benchmark for multi-turn reasoning evaluation. arXiv preprint arXiv:2505.17123.

Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. 2024. Evaluating mathematical reasoning of large language models: A focus on error identification and correction. In Findings of the Association for Computational Linguistics ACL 2024, pages 11316–11360.

Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868, Online. Association for Computational Linguistics.

Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211–226.

Kaijing Ma, Xeron Du, Yunran Wang, Haoran Zhang, ZhoufutuWen, Xingwei Qu, Jian Yang, Jiaheng Liu, minghao liu, Xiang Yue, Wenhao Huang, and Ge Zhang. 2025. KOR-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks. In The Thirteenth International Conference on Learning Representations.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018a. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018b. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

Josh Achiam OpenAI, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2024. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774.

Lianhui Qin, Antoine Bosselut, Arto Holtzmann, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5042–5052. Association for Computational Linguistics.

Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Maanul Faruqui. 2021. Time-dial: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7066–7076.

Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, et al. 2024. Gemma 2: Improving open language models at a practical size. CoRR.

Alexis Ross, Matthew E Peters, and Ana Marasović. 2022. Does self-rationalization improve robustness to spurious correlations? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7403–7416.

Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4661–4675. Association for Computational Linguistics.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.

Dan Shi, Chaobin You, Jiantao Huang, Taihao Li, and Deyi Xiong. 2024. Corecode: A common sense annotated dialogue dataset with benchmark tasks for chinese large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18952–18960.

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057.

Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4505–4514. Association for Computational Linguistics.

Shane Storks and Joyce Chai. 2021. Beyond the tip of the iceberg: Assessing coherence of text classifiers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3169–3177.

Jiaxing Sun, Weiquan Huang, Jiang Wu, Chenya Gu, Wei Li, Songyang Zhang, Hang Yan, and Conghui He. 2024. Benchmarking Chinese commonsense reasoning of LLMs: From Chinese-specifics to reasoning-memorization correlations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11205–11228, Bangkok, Thailand. Association for Computational Linguistics.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM-techreport.

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153–2162.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoong Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1819–1862, Mexico City, Mexico. Association for Computational Linguistics.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Hazeo Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xianrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. 2023. Baichuan 2: Open large-scale language models. CoRR, abs/2309.10305.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengcheng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, and Yue Zhang. 2022. Towards fine-grained causal reasoning and QA. CoRR, abs/2204.07408.

Alex Young, Bei Chen, Chao Li, Chenge Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. Yi: Open foundation models by 01.ai. CoRR, abs/2403.04652.

Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Jankowski, Yanghua Xiao, and Deqing Yang. 2023. Distilling script knowledge from large language models for constrained language planning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4303–4325.

Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.

Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363–3369.

Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, and Xiang Ren. 2021. Rica: Evaluating robust inference capabilities based on commonsense axioms. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7560–7579.

A Bloom Cognitive Model

Bloom Cognitive Model (Krathwohl, 1973) is an educational theoretical framework that outlines six hierarchical levels of cognitive processes, ranging from lower-order to higher-order thinking skills in the learning process as follows:

• Remember: The capacity to recall, identify, and reproduce information.

• Understand: The ability to interpret, summarize, and make sense of information.

• Apply: The skill to use learned knowledge in new contexts.

• Analyze: The capability to deconstruct information and examine relationships between components.

• Evaluate: The proficiency in making informed judgments based on specific criteria, involving critical thinking.

• Create: The ability to synthesize elements into novel patterns or generate original work.

Motivated by this framework, we aim to develop the model that goes beyond merely memorizing surface patterns and demonstrates higher-order capabilities. To test whether the model truly understands commonsense knowledge, we create seven variants of each question. It is our view that if the model genuinely understands commonsense knowledge, it should be able to correctly respond to the same knowledge expressed in different reasoning forms. Here’s how our seven variants map onto these cognitive levels:

• Understanding is demonstrated through Problem Restatement and Causal Inference.
• Application skills are tested via Reverse Conversion, Scenario Refinement, and Negative Transformation.

• Analysis capabilities are assessed through Sentence Ordering.

• Evaluation competency is measured by Critical Testing.

B Human Annotation

B.1 Annotator Qualification and Compensation

We maintained strict control over annotator qualification, data quality, and annotation procedure. Specifically, we recruited 34 professional annotators specializing in NLP tasks totally. All annotators hold at least a bachelor’s degree, have passed the College English Test Level 4 of China, and possess extensive annotation experience of NLP tasks. We compensated them at a rate of 23 RMB per hour (significantly higher than the average hourly wage in China), with an average payment of 1.98 RMB per question. We promptly addressed any concerns during the annotation process and allowed sufficient time for each question to prevent unnecessary pressure on annotators.

B.2 Data Quality and Consistency

31 out of 34 annotators were involved in data filtering. We enforced the strict annotation guidelines. For Chinese HellaSwag construction, in Stage 1 (initial dataset generation), annotators labeled 12,960 entries in total and filtered down to 12,287 entries. The authors randomly sampled 100 filtered entries and verified them against annotation guidelines, achieving a 98% compliance rate. In Stage 2 (difficult sample replacement), annotators labeled 5,209 entries in total and filtered down to 2,451 entries. A similar 100-question sample check by authors showed a 96% compliance rate. For HellaSwag-Pro construction, annotators labeled 24,260 entries, filtering down to 11,200. The authors randomly checked 100 question variants against variant annotation guidelines, achieving a 95% compliance rate. These measures ensured high quality of our dataset.

B.3 Human Performance

To evaluate human performance, we sampled a subset of 400 questions by randomly selecting 25 original questions in both Chinese and English, along with their variants. Three additional crowd workers, who were not involved in the original annotation process, were tested on this subset. We calculated their average accuracy as human performance.

B.4 Detailed Annotation Guidelines

We provided rich examples for the annotation tasks to ensure annotators understood the tasks at hand. We maintained close contact with the annotators to clarify any misunderstandings in time. Our annotation tasks were divided into four parts:

B.4.1 Chinese HellaSwag Annotation for Stage One

Annotators were given the context, six choices filtered by the model, label, broad type, and detailed type. They scored based on three dimensions: the possibility to select 4 out of 6 choices, and whether they conform to the two category definitions. The annotation requirements for annotators were as follows:

• Possibility to select 4 out of 6: Using the model’s scoring of the 6 choices as a reference, determine if it’s possible to select 4 choices, with only one correct answer and the other three being as confusing as possible (i.e., conforming to commonsense but not suitable for the context, or judged by how much modification is needed to make them correct - the less modification needed, the more confusing). Ensure the uniqueness of the answer and avoid controversy. Score 1 if possible, and note the corresponding option numbers, with the first being the correct option and the next three being incorrect options. If not possible, score 0 and select the appropriate reason: A. No correct option or B. Unable to select 3 incorrect options, e.g., more than 4 correct options.

• Broad type: Score it conforms to the definition, otherwise 0.

• Detailed type: Score 1 if it conforms to the definition, otherwise 0.

The following are the definitions for broad and detailed types.

• Family

Household chores: Labor activities to maintain a clean and tidy home environment, including but not limited to cleaning, laundry, and preparing traditional Chinese cuisine.
Personal hygiene: Daily personal cleaning habits such as bathing, brushing teeth, and maintaining good living habits to ensure physical health.

Family entertainment: Leisure activities shared by family members, such as playing family games, pet care, watching TV shows, or reading books together.

Holiday celebrations: Celebrating family members’ birthdays, traditional festivals, or special occasions like wedding anniversaries.

Family affairs: Daily life management, emotional communication, and responsibility allocation among family members, including household shopping, financial management, and handling potential disagreements or conflicts.

Family transitions: Changes in family structure or living environment, such as home renovation, moving, marriage, or welcoming a newborn.

Emergency handling: Measures for potential emergencies like fires or natural disasters.

• Education

School education: Formal education received in school settings, including classroom learning, extracurricular activities, and exam preparation.
Family education: Education provided by parents or other family members, including homework assistance, shared reading, and cultivation of interests and moral qualities.

Online learning: Learning through internet resources, including self-study tools, remote tutoring, and interactive learning platforms.

Community education: Educational activities within the community, such as lectures, interest groups, and practical activities.

Vocational training: Professional training aimed at improving occupational skills, including obtaining professional qualifications and on-the-job continuing education.

Lifelong learning: Continuous learning activities for adults to improve themselves, such as adult education or senior university courses.

International exchange: Consultation for studying abroad, language skill improvement, and other forms of cross-cultural exchange.

• Work

Work Meetings: Various meetings held in the workplace, including team meetings, departmental reports, and project evaluations.

Project Management: The entire process of managing a project from initiation to completion, including strategy formulation, progress tracking, and problem-solving.

Customer Service: Services provided to meet customer needs, including customer inquiries, complaint handling, sales negotiations, and after-sales support.

Teamwork: Effective collaborative work patterns within a team, including team building, task allocation, conflict resolution, and incentive measures.

Personal Development: The process of individual career growth, covering skill learning, career planning, financial management, and maintaining mental and physical health.

Administrative Management: Daily management activities within a company, including attendance records, performance evaluations, travel expense reimbursements, employee benefits distribution, and company policy communication.

Technological Innovation: Activities driving technological advancement in a company, including new product development, technology application, technical training, and technology exchange.

• Sociality

Daily Interactions: Everyday social interactions with family, friends, and neighbors.

School Interactions: Communication between students, between teachers and students, and between parents and teachers.

Workplace Interactions: Interactions with colleagues, superiors, or subordinates in the workplace, as well as formal business dinners.

Community Interactions: Participating in community-organized activities or providing volunteer services.

Public Space Interactions: Interactions with others in public spaces such as public transportation, shopping malls, restaurants, and lecture halls.

Online Social Networking: Social activities using online platforms, including social media, online gaming, internet forums, and video live streaming.
Special Occasion Interactions: Interpersonal interactions at weddings, funerals, award ceremonies, and other celebratory events.

• Shopping

In-store Shopping: Shopping activities in physical retail stores, such as supermarkets, department stores, and specialty shops.

Online Shopping: Online purchasing behavior through e-commerce platforms, live streaming sales, or social commerce.

Food and Dining Purchases: Buying food products, including dining out, ordering takeout, and home cooking.

Service Purchases: Buying various service products, such as travel services, beauty and fitness, and educational training.

Overseas Shopping: Purchasing foreign goods through cross-border e-commerce or personal shopping agents.

Special Occasion Shopping: Shopping in specific situations, such as promotional events, group buying, auctions, and second-hand transactions.

Returns and After-sales Service: Consumer behavior in seeking refunds, exchanges, and after-sales service when issues arise with products.

• Transportation

Public Transportation: Using public transit systems, such as buses and subways.

Private Transportation: Using private vehicles, bicycles, etc., for travel.

Long-distance Travel: Travel methods covering longer distances, such as trains, planes, or long-distance buses.

Emergency Travel: Choosing emergency transportation in response to sudden situations, such as travel during severe weather conditions.

Tourist Transportation: Using sightseeing vehicles, boats, or cable cars for tourism purposes.

International Travel: Visa applications, international flight bookings, and entry procedures required for traveling abroad.

Special Occasion Transportation: Transportation services provided for specific situations, such as wedding cars or conference shuttles.

• Health

Preventive Healthcare: Measures taken to prevent diseases, including health check-ups, vaccinations, and health education.

Outpatient Care: Receiving non-hospitalized treatment at hospitals or clinics, including appointment scheduling, initial diagnosis, follow-up visits, and specialist consultations.

Inpatient Treatment: Hospital admission for treatment, including admission procedures, ward life, surgery arrangements, and discharge preparation.

Rehabilitation Care: Treatment during the recovery period, including rehabilitation training, long-term care, and psychological counseling.

Medication Management: Guidance on medication use and storage methods.

Health Insurance: Purchasing medical insurance products, claim procedures, and health consultation services.

Epidemic Prevention and Control: Measures such as epidemic monitoring, isolation observation, and health code management.

• Leisure

Outdoor Activities: Recreational activities in natural settings, such as hiking, picnicking, and gardening.

Cultural Experiences: Engaging in cultural activities like visiting museums, watching theatrical performances, or attending film screenings.

Travel Experiences: Domestic or international tourism activities.

Sporting Events: Watching or participating in sports competitions, including esports.

Artistic Pursuits: Engaging in artistic activities such as painting, calligraphy, playing musical instruments, or creating handicrafts.

Digital Entertainment: Leisure activities using digital devices, such as online gaming, live streaming interactions, or virtual reality experiences.

Recreational Fitness: Maintaining physical and mental health through activities like gym workouts or practicing yoga and meditation.

B.4.2 Chinese HellaSwag Annotation for Stage Two

In order to increase the number of difficult samples, the annotators were given a context and four
replaced options regenerated by models to judge whether the label of the question was correct and whether it had a unique correct option. If both are true, the replaced options were retained.

B.4.3 Hellaswag-Pro Annotation

Annotators are provided with the original context, original choices, original label, transformed context, transformed choices, transformed label, and perturbation type for annotation according to different variant definitions. The variant definitions are as follows:

• Problem restatement: Restate the original context and the original label corresponding to the original choices in a different way, ensuring the semantics remain unchanged. Other options of the original choices should remain unchanged without restatement. Pay special attention to ensuring that the connection between the context and the choice corresponding to the label is smooth.

• Reverse conversion: Combine the original choices corresponding to the original label with the original context into a complete passage. Then, make the last sentence of this passage the context, and transform the remaining sentences into the correct choice. A slight modification is allowed for smoothness. Also, generate five other incorrect options that do not fit the context, modeled on the format and length of the correct option. Place the correct option in the first position and label it as 0. To ensure the context is complete, append "Which is the possible context for this action?" This conversion process aims to infer the potential background through the results. The generated incorrect options should not include supernatural elements and should have a similar word count to the correct option.

• Causal inference: Combine the original choices corresponding to the original label with the original context to form a complete passage and turn it into the context. Then, generate the reason for such choices that contain commonsense as the correct option in the choices. The correct choice should be as concise as possible while generating five other evidently incorrect options modeled on the format and length of the correct choice. Put the correct choice in the first position and label it as 0. To ensure the context is complete, append "Which is the possible reason for this action?" This conversion process aims to infer the potential reason through the context and options.

• Negative transformation: Modify the original context to end with a negation word as the context, retaining one most unreasonable option and the original choice corresponding to the original label. Then, generate two other reasonable options as choices. Generated options should be similar in length and format to the original options. Place this most unreasonable option as the first element in the choices and label the index of this option in choices as 0. This conversion process aims to transform the original task into a negation prediction, containing one unreasonable option and three other reasonable options.

• Scenario refinement: First, select a relatively reasonable option from the incorrect options in the original choices, then modify the original context as the context to allow the selection of this option as the correct choice. The value of choices is equal to the original choices. The label value corresponds to the value of the selected incorrect option. This conversion process aims to refine the context, thereby altering the correct choice.

• Sentence ordering:
1) Sentence ordering - Short: First, combine the original choices corresponding to the original label with the original context into a complete sentence. Then, predict the development of subsequent events, continuing to write a few more sentences to form a paragraph. Pay attention to the sequence and completeness of continued sentences, ensuring the uniqueness of the answer. Then, disorder each sentence of this passage and number them. The correct option is the original order of the paragraph, and three other incorrect options are generated based on the correct option by disordering the numbers. Place the correct option in the first position and label it as 0. To ensure the context is complete, append "The correct order is." This conversion process aims to infer the correct order of sentences.

2) Sentence ordering - Long: Combine the original choices corresponding to the original label with the original context into a complete passage. Then, disorder each sentence of this passage and number them. The correct option is the original
order of the paragraph, and three other incorrect options are generated by disordering the numbers. Place the correct option in the first position and label it as 0. To ensure the context is complete, append "The correct order is." This conversion process aims to infer the correct order of sentences.

• Critical testing: Modify the original context so that none of the options can be chosen as the context, then add an option of 'None of the above four options are appropriate' to the original choices as choices. The label value corresponds to the index of 'None of the above four options are appropriate'. Note that the modified context should still present a question, ideally with an ending word identical to the original context. This conversion process aims to test the model’s critical thinking.

B.4.4 Hellaswag-Pro Human Evaluation
Annotators were provided with the context and choices from the Hellaswag-Pro and made selections. We then compared the selections made by annotators with the labels to calculate accuracy.

C Prompt Strategy
The prompting strategies we designed, including Direct, CN-CoT, EN-CoT, CN-XLT and EN-XLT, are as shown Figure 6, 7 and 8.

D Case Study
Figure 9 shows an example of the Chinese hellaswag generation process, from which we can see that our wrong options are becoming more and more challenging.

E Chinese HellaSwag Evaluation
We also evaluate the overall results of Chinese Hellaswag using both open-source and closed-source models, analyzing them from the perspectives of broad categories and length categories. As shown in Table 10, within all categories of Chinese Hellaswag, Traffic is the most challenging, with an average accuracy of only 58.56%, while the Education category is the easiest, achieving an average accuracy of 77.64%. Additionally, as the context length increases, the difficulty of the problems generally decreases, with average accuracy of long types at 72%, medium types at 70%, and short types at 64% as shown in Table 11. Overall, the closed-source models outperform open-source models. Among the closed-source models, Claude-3.5 performs the best, reaching an accuracy of 94%, whereas among open-source models, Qwen2.5-72B shows the highest performance, achieving 71%.

F Experiment Detailed Result
Figures 12 to 20 show the detailed results of the open-source models on the 9 prompt strategies and 7 variants we design.

In addition, we also use direct prompt to evaluate the performance of the Qwen, LLaMA, and Mixtral series of intruct models on our robustness task, as shown in Table 21 and Figure 7. We can observe that the intruct models have improved performance in robustness task compared to the base models.

![Comparasion between base and their instruct models.](page_1012_1012_384_288.png)

Figure 7: Comparasion between base and their instruct models.
Problem: Question: 周老师负责学校的英语培训工作，常为学生提供实用的语言学习建议。他建议多读原版英文报纸提高阅读水平。可能的原因是
Choices:
A.主张减少课外阅读时间增加听力训练。
B.阅读原版材料能够增强语感并扩大词汇量。
C.觉得看漫画书更能激发学生的兴趣。
D.相信通过电影比读书更容易学好语言。

Prompt Chinese Dataset
Direct Answer: B
CN- CoT 答案：让我们一步步思考。首先，我们需要分析周老师作为英语培训教师的建议-多读原版英文报纸，这建议的目的是为了提高学生的阅读水平。查看选项：A提到减少阅读增加听力，与题目表述矛盾；C提到漫画书和D提到电影都偏离了原版英文报纸这一重点；只有B选项提到阅读原版材料可以增强语感并扩大词汇量，这与读原版英文报纸的效果是一致的。原版英文报纸使用地道的英语表达，包含丰富的词汇和表达方式，长期阅读确实能帮助学习者提高语感，扩大词汇量，提升整体语言水平。答案是B。

EN-CoT Answer: Let’s think step by step. First, we know that Teacher Zhou is responsible for English training at school and often provides practical language learning advice to students. Second, he specifically recommends reading original English newspapers to improve reading skills. Third, reading original English materials has several benefits: it exposes students to authentic language usage, helps them understand how native speakers express ideas, enhances language intuition, and naturally expands their vocabulary through context. Fourth, looking at the options, option B directly aligns with these benefits by mentioning both improved language sense and vocabulary expansion. The other options either suggest different learning methods (like movies or comics) or propose reducing reading time, which contradicts the teacher’s recommendation. Since Teacher Zhou specifically advocates for reading original English newspapers, option B is the most logical explanation for his recommendation. The answer is B.

Question: The mother instructs them on how to brush their teeth while laughing. The boy helps his younger sister brush his teeth. She gets them some water to gargle in their mouths. Which could be the possible reason for this action? Choices:
A. She wants to ensure they rinse out any remaining toothpaste and clean their mouth thoroughly.
B. She is conducting an experiment to see if water tastes different after brushing.
C. She is using the water as a distraction so they won’t notice her leaving the room.
D. She is training them to be synchronized swimmers by coordinating their gargles.

Prompt English Dataset
Direct Answer: A
CN- CoT 答案：让我们一步步思考。这是一个关于刷牙后漱口的场景。母亲在指导孩子们刷牙，并让他们用水漱口。从日常生活经验来看，刷牙后漱口的主要目的是清除口腔中残留的牙膏和清洁口腔。漱口这个动作不太可能是为了做实验测试水的味道（选项B），也不是为了分散注意力（选项C），更不可能是在训练同步游泳（选项D）。选项A提到让孩子们漱口是为了冲洗掉剩余的牙膏并彻底清洁口腔，这符合我们日常刷牙后漱口的目的，也是最合理的解释。因此答案是A。

EN-CoT Answer: Let’s think step by step. Gargling with water after brushing teeth is a common and important part of proper dental hygiene. The mother in this scenario is teaching her children good oral care habits. When we brush our teeth, toothpaste helps clean them, but some toothpaste residue remains in our mouth. This residue, if not rinsed out, can leave a lingering taste and might contain fluoride that shouldn’t be swallowed in large amounts. By giving the children water to gargle, the mother is helping them complete their tooth-brushing routine correctly. The gargling action helps remove any loose debris, leftover toothpaste, and bacteria from the mouth, leading to better oral hygiene. Looking at the other options, they are either illogical or unrelated to dental care - testing water taste, using it as a distraction, or swimming practice are not reasonable explanations for gargling after brushing teeth. The most sensible and practical reason for the mother’s action is to ensure her children properly rinse their mouths after brushing. The answer is A.

Table 6: Examples of Direct, CN-CoT and EN-CoT Prompt.
Problem: Question: 周老师负责学校的英语培训工作，常为学生提供实用的语言学习建议。他建议多读原版英文报纸提高阅读水平。可能的原因是
Choices:
A.主张减少课外阅读时间增加听力训练。
B.阅读原版材料能够增强语感并扩大词汇量。
C.觉得看漫画书更能激发学生的兴趣。
D.相信通过电影比读书更容易学好语言。

Question: The mother instructs them on how to brush their teeth while laughing. The boy helps his younger sister brush his teeth. She gets them some water to gargle in their mouths. Which could be the possible reason for this action? Choices:
A. She wants to ensure they rinse out any remaining toothpaste and clean their mouth thoroughly.
B. She is conducting an experiment to see if water tastes different after brushing.
C. She is using the water as a distraction so they won’t notice her leaving the room.
D. She is training them to be synchronized swimmers by coordinating their gargles.

<table>
  <tr>
    <th>Prompt</th>
    <th>Chinese Dataset</th>
    <th>English Dataset</th>
  </tr>
  <tr>
    <td>CN-XLT</td>
    <td>-</td>
    <td>我希望你扮演一个英文常识推理专家。<br>你应该用中文复述问题和选项。<br>你应该一步一步地回答问题，选出正确的答案。<br>问题：母亲一边笑一边指导他们如何刷牙。男孩帮助他的小妹妹刷牙。她给他们一些水让他们漱口。对此行为，可能的原因是什么？<br>选项：<br>A. 她想确保他们漱掉任何剩余的牙膏并清洁嘴巴。<br>B. 她正在进行实验，以查看刷牙后水的味道是否不同。<br>C. 她用水作为分散注意力的方法，以便他们不会注意到她离开房间。<br>D. 她通过协调漱口来训练他们成为同步游泳运动员。<br>分步解答：<br>1. 母亲给他们水漱口，通常是为了卫生目的。<br>2. 漱口可以帮助去除残留的牙膏，确保口腔清洁。<br>3. 选项A是一个常见且合理的原因，符合情境。<br>4. 选项B、C和D不符合通常情况下的刷牙步骤或目的。<br>所以答案是A。</td>
  </tr>
</table>

Table 7: Examples of CN-XLT Prompt.
Problem: Question: 周老师负责学校的英语培训工作，常为学生提供实用的语言学习建议。他建议多读原版英文报纸提高阅读水平。可能的原因是
Choices:
A.主张减少课外阅读时间增加听力训练。
B.阅读原版材料能够增强语感并扩大词汇量。
C.觉得看漫画书更能激发学生的兴趣。
D.相信通过电影比读书更容易学好语言。

Question: The mother instructs them on how to brush their teeth while laughing. The boy helps his younger sister brush his teeth. She gets them some water to gargle in their mouths. Which could be the possible reason for this action? Choices:
A. She wants to ensure they rinse out any remaining toothpaste and clean their mouth thoroughly.
B. She is conducting an experiment to see if water tastes different after brushing.
C. She is using the water as a distraction so they won’t notice her leaving the room.
D. She is training them to be synchronized swimmers by coordinating their gargles.

<table>
  <tr>
    <th>Prompt</th>
    <th>Chinese Dataset</th>
    <th>English Dataset</th>
  </tr>
  <tr>
    <td>EN-XLT</td>
    <td>I want you to act as a commonsense reasoning expert for Chinese.<br>You should retell the question and choices in English.<br>You should do the answer step by step to choose the right answer.<br>You should tell me the answer in this format 'So the answer is'.<br>Question: Teacher Zhou is in charge of the English training at school and often provides students with practical language learning advice. He suggests reading original English newspapers to improve reading skills. The possible reason is:<br>Choices:<br>A. Advocates reducing extracurricular reading time to increase listening practice.<br>B. Reading original materials can enhance language sense and expand vocabulary.<br>C. Believes that reading comic books can better stimulate students’ interest.<br>D. Believes it is easier to learn a language well through movies than through reading.</td>
    <td>-</td>
  </tr>
</table>

Step-by-step answer:
1. The statement suggests reading original English newspapers as a way to improve reading skills.
2. Option B suggests that reading original materials can enhance language sense and expand vocabulary, which aligns with the goal of improving reading skills.
3. Option A talks about reducing reading time, which contradicts the suggestion to read newspapers.
4. Option C talks about comic books, which is not related to reading original newspapers.
5. Option D suggests movies are more effective, which again, does not align with the focus on reading newspapers.
So the answer is B.

Table 8: Examples of EN-XLT Prompt.
<table>
  <tr>
    <th>Step</th>
    <th>Generated or Filtered Content</th>
  </tr>
  <tr>
    <td><b>Step 1<br>Over-generate context with broad type and detailed type with Qwen-Max</b></td>
    <td>
      Context: 丽丽报名参加了日本京都的一趟文化之旅，深度体验了传统艺伎表演。她<br>
      Broad Type: 休闲娱乐<br>
      Detailed Type: 旅游体验<br>
      Context: Lili signed up for a cultural tour in Kyoto, Japan, and experienced a traditional geisha performance. She<br>
      Broad Type: Leisure<br>
      Detailed Type: Travel Experience<br>
      Choices:<br>
      1. 深深地被茶道的精致仪式所吸引。(Score:10)<br>
      2. 抱怨为什么所有的寺庙都没有钟声。(Score:3)<br>
      3. 抱怨旅馆里的榻榻米太过柔软。(Score:3)<br>
      4. 要求在京都找到寿司制作的机器人。(Score:2)<br>
      5. 质疑为什么艺伎穿的是拖鞋而不是高跟鞋。(Score:5)<br>
      6. 坚持要在神社祈祷时使用英文。(Score:2)<br>
      7. 对茶道助手参与和服穿着过程表现出浓厚的兴趣。(Score:9)<br>
      8. 询问为什么没人听说过她喜欢的动漫角色。(Score:4)<br>
      9. 希望能在旅途中看到更多现代化的建筑。(Score:3)<br>
      10. 感到遗憾没能能在旅途中参与一场本地的相扑比赛。(Score:4)
    </td>
  </tr>
  <tr>
    <td><b>Step 2<br>Over-generate candidate choices with Qwen-Max</b></td>
    <td>
      Choices:<br>
      1. was deeply attracted by the exquisite ritual of the tea ceremony. (Score:10)<br>
      2. complained why there were no bells in all temples. (Score:3)<br>
      3. complained that the tatami in the hotel was too soft. (Score:3)<br>
      4. requested to find a sushi-making robot in Kyoto. (Score:2)<br>
      5. questioned why geishas wore slippers instead of high heels. (Score:5)<br>
      6. insisted on using English when praying at the shrine. (Score:2)<br>
      7. showed great interest in participating in the process of wearing a kimono. (Score:9)<br>
      8. asked why no one has heard of her favorite anime character. (Score:4)<br>
      9. hoped to see more modern buildings during the trip. (Score:3)<br>
      10. regretted not being able to attend a local sumo match during the trip. (Score:4)
    </td>
  </tr>
  <tr>
    <td><b>Step 3-1<br>Find filtered choices with Qwen-Max</b></td>
    <td>
      Choices:<br>
      1. 深深地被茶道的精致仪式所吸引。(Score:10)<br>
      5. 质疑为什么艺伎穿的是拖鞋而不是高跟鞋。(Score:5)<br>
      8. 询问为什么没人听说过她喜欢的动漫角色。(Score:4)<br>
      10. 感到遗憾没能能在旅途中参与一场本地的相扑比赛。(Score:4)<br>
      2. 抱怨为什么所有的寺庙都没有钟声。(Score:3)<br>
      9. 希望能在旅途中看到更多现代化的建筑。(Score:3)
    </td>
  </tr>
  <tr>
    <td><b>Step 3-2<br>Find filtered choices with human annotators</b></td>
    <td>
      Choices:<br>
      1. was deeply attracted by the exquisite ritual of the tea ceremony. (Score:10)<br>
      5. questioned why geishas wore slippers instead of high heels. (Score:5)<br>
      8. asked why no one has heard of her favorite anime character. (Score:4)<br>
      9. hoped to see more modern buildings during the trip. (Score:3)
    </td>
  </tr>
  <tr>
    <td><b>Step 4<br>Replace easily-identifiable false choices with adversarial ones through human-in-the-loop alternating adversarial filtering</b></td>
    <td>
      Choices:<br>
      1. 深深地被茶道的精致仪式所吸引。<br>
      2. 学习了传统的日式剑道和弓道技巧。<br>
      3. 欣赏了京都著名的樱花季和红叶景观。<br>
      4. 品尝了正宗的关西风味章鱼烧和大阪烧。<br>
      Choices:<br>
      1. was deeply attracted by the exquisite ritual of the tea ceremony.<br>
      2. learned traditional Japanese kendo and archery techniques.<br>
      3. enjoyed Kyoto’s famous cherry blossom season and red leaves.<br>
      4. tasted authentic Kansai-style takoyaki and okonomiyaki.
    </td>
  </tr>
</table>

Table 9: An example of Chinese HellaSwag Gereration. Step 3-1 filters the top 5 wrong options with scores below 9 to prevent multiple correct options, and Step 3-2 select the most confusing wrong options by human annotators.
<table>
  <tr>
    <th>Model</th>
    <th>Education</th>
    <th>Health</th>
    <th>Famly</th>
    <th>Leisure</th>
    <th>Shopping</th>
    <th>Sociality</th>
    <th>Traffic</th>
    <th>Work</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base</td>
    <td>0.76</td>
    <td>0.71</td>
    <td>0.66</td>
    <td>0.69</td>
    <td>0.62</td>
    <td>0.68</td>
    <td>0.55</td>
    <td>0.70</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base</td>
    <td>0.78</td>
    <td>0.70</td>
    <td>0.68</td>
    <td>0.70</td>
    <td>0.64</td>
    <td>0.69</td>
    <td>0.57</td>
    <td>0.71</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B</td>
    <td>0.74</td>
    <td>0.59</td>
    <td>0.55</td>
    <td>0.57</td>
    <td>0.54</td>
    <td>0.56</td>
    <td>0.46</td>
    <td>0.61</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B</td>
    <td>0.76</td>
    <td>0.65</td>
    <td>0.63</td>
    <td>0.66</td>
    <td>0.63</td>
    <td>0.67</td>
    <td>0.54</td>
    <td>0.65</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1</td>
    <td>0.70</td>
    <td>0.59</td>
    <td>0.52</td>
    <td>0.56</td>
    <td>0.57</td>
    <td>0.57</td>
    <td>0.50</td>
    <td>0.61</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>0.72</td>
    <td>0.66</td>
    <td>0.53</td>
    <td>0.60</td>
    <td>0.53</td>
    <td>0.58</td>
    <td>0.47</td>
    <td>0.66</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>0.75</td>
    <td>0.66</td>
    <td>0.60</td>
    <td>0.62</td>
    <td>0.59</td>
    <td>0.64</td>
    <td>0.51</td>
    <td>0.67</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>0.75</td>
    <td>0.67</td>
    <td>0.63</td>
    <td>0.66</td>
    <td>0.61</td>
    <td>0.66</td>
    <td>0.55</td>
    <td>0.68</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>0.76</td>
    <td>0.68</td>
    <td>0.66</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.70</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>0.78</td>
    <td>0.68</td>
    <td>0.68</td>
    <td>0.69</td>
    <td>0.65</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.71</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>0.77</td>
    <td>0.69</td>
    <td>0.68</td>
    <td>0.69</td>
    <td>0.66</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.69</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>0.78</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.73</td>
    <td>0.60</td>
    <td>0.73</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B</td>
    <td>0.78</td>
    <td>0.69</td>
    <td>0.66</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.69</td>
    <td>0.56</td>
    <td>0.72</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>0.78</td>
    <td>0.70</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.64</td>
    <td>0.69</td>
    <td>0.57</td>
    <td>0.72</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base</td>
    <td>0.79</td>
    <td>0.70</td>
    <td>0.67</td>
    <td>0.69</td>
    <td>0.64</td>
    <td>0.69</td>
    <td>0.57</td>
    <td>0.73</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base</td>
    <td>0.80</td>
    <td>0.72</td>
    <td>0.70</td>
    <td>0.72</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.58</td>
    <td>0.74</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>gemma-2-2b</td>
    <td>0.73</td>
    <td>0.62</td>
    <td>0.57</td>
    <td>0.60</td>
    <td>0.60</td>
    <td>0.60</td>
    <td>0.50</td>
    <td>0.66</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>gemma-2-9b</td>
    <td>0.78</td>
    <td>0.68</td>
    <td>0.64</td>
    <td>0.67</td>
    <td>0.65</td>
    <td>0.69</td>
    <td>0.55</td>
    <td>0.74</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>gemma-2-27b</td>
    <td>0.72</td>
    <td>0.66</td>
    <td>0.64</td>
    <td>0.62</td>
    <td>0.62</td>
    <td>0.58</td>
    <td>0.50</td>
    <td>0.67</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b</td>
    <td>0.73</td>
    <td>0.64</td>
    <td>0.58</td>
    <td>0.64</td>
    <td>0.54</td>
    <td>0.60</td>
    <td>0.49</td>
    <td>0.65</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>internlm2_5-7b</td>
    <td>0.76</td>
    <td>0.68</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.63</td>
    <td>0.67</td>
    <td>0.60</td>
    <td>0.69</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>internlm2_5-20b</td>
    <td>0.76</td>
    <td>0.67</td>
    <td>0.68</td>
    <td>0.70</td>
    <td>0.64</td>
    <td>0.68</td>
    <td>0.59</td>
    <td>0.69</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.91</td>
    <td>0.92</td>
    <td>0.88</td>
    <td>0.92</td>
    <td>0.90</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.91</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>Claude-3-5</td>
    <td>0.94</td>
    <td>0.96</td>
    <td>0.94</td>
    <td>0.94</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.91</td>
    <td>0.96</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>0.88</td>
    <td>0.91</td>
    <td>0.88</td>
    <td>0.90</td>
    <td>0.90</td>
    <td>0.91</td>
    <td>0.85</td>
    <td>0.91</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>0.91</td>
    <td>0.95</td>
    <td>0.91</td>
    <td>0.92</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.94</td>
    <td>0.92</td>
  </tr>
  <tr>
    <th>AVG</th>
    <th>0.78</th>
    <th>0.71</th>
    <th>0.68</th>
    <th>0.70</th>
    <th>0.67</th>
    <th>0.70</th>
    <th>0.60</th>
    <th>0.73</th>
    <th>0.69</th>
  </tr>
</table>

Table 10: Model Performance on Chinese HellaSwag based on broad category under Direct Prompt.
<table>
  <tr>
    <th>Model</th>
    <th>Long</th>
    <th>Medium</th>
    <th>Short</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.62</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base</td>
    <td>0.72</td>
    <td>0.71</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B</td>
    <td>0.64</td>
    <td>0.59</td>
    <td>0.51</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B</td>
    <td>0.70</td>
    <td>0.67</td>
    <td>0.58</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1</td>
    <td>0.63</td>
    <td>0.58</td>
    <td>0.52</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>0.63</td>
    <td>0.61</td>
    <td>0.54</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>0.67</td>
    <td>0.64</td>
    <td>0.58</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>0.68</td>
    <td>0.67</td>
    <td>0.60</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>0.71</td>
    <td>0.69</td>
    <td>0.62</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>0.72</td>
    <td>0.70</td>
    <td>0.63</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>0.74</td>
    <td>0.73</td>
    <td>0.65</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B</td>
    <td>0.72</td>
    <td>0.70</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>0.73</td>
    <td>0.70</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base</td>
    <td>0.73</td>
    <td>0.71</td>
    <td>0.61</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base</td>
    <td>0.76</td>
    <td>0.72</td>
    <td>0.62</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>gemma-2-2b</td>
    <td>0.65</td>
    <td>0.61</td>
    <td>0.56</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>gemma-2-9b</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.61</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>gemma-2-27b</td>
    <td>0.68</td>
    <td>0.64</td>
    <td>0.56</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b</td>
    <td>0.65</td>
    <td>0.63</td>
    <td>0.55</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>internlm2_5-7b</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.61</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>internlm2_5-20b</td>
    <td>0.73</td>
    <td>0.69</td>
    <td>0.61</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.87</td>
    <td>0.90</td>
    <td>0.93</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>Claude-3-5</td>
    <td>0.92</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>0.86</td>
    <td>0.91</td>
    <td>0.92</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>0.89</td>
    <td>0.93</td>
    <td>0.95</td>
    <td>0.92</td>
  </tr>
  <tr>
    <th>AVG</th>
    <th>0.73</th>
    <th>0.71</th>
    <th>0.65</th>
    <th>0.69</th>
  </tr>
</table>

Table 11: Model Performance on Chinese HellaSwag based on length category under Direct Prompt.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__direct_cn</td>
    <td>0.66</td>
    <td>0.36</td>
    <td>0.06</td>
    <td>0.64</td>
    <td>0.50</td>
    <td>0.36</td>
    <td>0.58</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__direct_cn</td>
    <td>0.70</td>
    <td>0.35</td>
    <td>0.07</td>
    <td>0.65</td>
    <td>0.55</td>
    <td>0.38</td>
    <td>0.54</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__direct_cn</td>
    <td>0.66</td>
    <td>0.37</td>
    <td>0.06</td>
    <td>0.66</td>
    <td>0.57</td>
    <td>0.42</td>
    <td>0.68</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__direct_cn</td>
    <td>0.52</td>
    <td>0.34</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>0.36</td>
    <td>0.35</td>
    <td>0.37</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__direct_cn</td>
    <td>0.56</td>
    <td>0.38</td>
    <td>0.06</td>
    <td>0.75</td>
    <td>0.32</td>
    <td>0.35</td>
    <td>0.36</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__direct_cn</td>
    <td>0.59</td>
    <td>0.40</td>
    <td>0.05</td>
    <td>0.78</td>
    <td>0.29</td>
    <td>0.34</td>
    <td>0.35</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_en_cot_cn</td>
    <td>0.75</td>
    <td>0.48</td>
    <td>0.14</td>
    <td>0.62</td>
    <td>0.43</td>
    <td>0.38</td>
    <td>0.23</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_en_cot_en</td>
    <td>0.80</td>
    <td>0.29</td>
    <td>0.42</td>
    <td>0.47</td>
    <td>0.41</td>
    <td>0.32</td>
    <td>0.74</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_en_xlt_cn</td>
    <td>0.40</td>
    <td>0.11</td>
    <td>0.12</td>
    <td>0.35</td>
    <td>0.34</td>
    <td>0.18</td>
    <td>0.09</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_cn_cot_en</td>
    <td>0.73</td>
    <td>0.51</td>
    <td>0.17</td>
    <td>0.62</td>
    <td>0.36</td>
    <td>0.29</td>
    <td>0.30</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_cn_cot_en</td>
    <td>0.81</td>
    <td>0.37</td>
    <td>0.64</td>
    <td>0.35</td>
    <td>0.43</td>
    <td>0.26</td>
    <td>0.89</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_cn_xlt_en</td>
    <td>0.73</td>
    <td>0.28</td>
    <td>0.15</td>
    <td>0.29</td>
    <td>0.36</td>
    <td>0.18</td>
    <td>0.88</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_en_cot_cn</td>
    <td>0.91</td>
    <td>0.40</td>
    <td>0.75</td>
    <td>0.82</td>
    <td>0.79</td>
    <td>0.43</td>
    <td>0.40</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_en_cot_en</td>
    <td>0.82</td>
    <td>0.26</td>
    <td>0.50</td>
    <td>0.66</td>
    <td>0.42</td>
    <td>0.47</td>
    <td>0.82</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_en_xlt_cn</td>
    <td>0.33</td>
    <td>0.18</td>
    <td>0.41</td>
    <td>0.50</td>
    <td>0.45</td>
    <td>0.22</td>
    <td>0.23</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_cn_cot_en</td>
    <td>0.89</td>
    <td>0.48</td>
    <td>0.82</td>
    <td>0.84</td>
    <td>0.74</td>
    <td>0.42</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_cn_cot_en</td>
    <td>0.85</td>
    <td>0.41</td>
    <td>0.37</td>
    <td>0.63</td>
    <td>0.39</td>
    <td>0.41</td>
    <td>0.70</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_cn_xlt_en</td>
    <td>0.45</td>
    <td>0.17</td>
    <td>0.15</td>
    <td>0.54</td>
    <td>0.26</td>
    <td>0.23</td>
    <td>0.64</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_en_cot_cn</td>
    <td>0.94</td>
    <td>0.50</td>
    <td>0.83</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.47</td>
    <td>0.70</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_en_cot_en</td>
    <td>0.89</td>
    <td>0.39</td>
    <td>0.52</td>
    <td>0.72</td>
    <td>0.48</td>
    <td>0.49</td>
    <td>0.68</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_en_xlt_cn</td>
    <td>0.41</td>
    <td>0.27</td>
    <td>0.45</td>
    <td>0.51</td>
    <td>0.51</td>
    <td>0.32</td>
    <td>0.10</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_cn_cot_en</td>
    <td>0.92</td>
    <td>0.49</td>
    <td>0.90</td>
    <td>0.89</td>
    <td>0.81</td>
    <td>0.46</td>
    <td>0.80</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_cn_cot_en</td>
    <td>0.89</td>
    <td>0.35</td>
    <td>0.57</td>
    <td>0.70</td>
    <td>0.48</td>
    <td>0.44</td>
    <td>0.73</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_cn_xlt_en</td>
    <td>0.72</td>
    <td>0.19</td>
    <td>0.64</td>
    <td>0.58</td>
    <td>0.40</td>
    <td>0.28</td>
    <td>0.51</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_en_cot_cn</td>
    <td>0.54</td>
    <td>0.18</td>
    <td>0.07</td>
    <td>0.54</td>
    <td>0.34</td>
    <td>0.23</td>
    <td>0.18</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_en_cot_en</td>
    <td>0.53</td>
    <td>0.17</td>
    <td>0.29</td>
    <td>0.36</td>
    <td>0.30</td>
    <td>0.29</td>
    <td>0.27</td>
    <td>0.32</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_en_xlt_cn</td>
    <td>0.14</td>
    <td>0.11</td>
    <td>0.02</td>
    <td>0.09</td>
    <td>0.07</td>
    <td>0.08</td>
    <td>0.00</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_cn_cot_en</td>
    <td>0.59</td>
    <td>0.33</td>
    <td>0.10</td>
    <td>0.49</td>
    <td>0.26</td>
    <td>0.00</td>
    <td>0.06</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_cn_cot_en</td>
    <td>0.42</td>
    <td>0.18</td>
    <td>0.28</td>
    <td>0.27</td>
    <td>0.24</td>
    <td>0.20</td>
    <td>0.43</td>
    <td>0.29</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_cn_xlt_en</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.10</td>
    <td>0.01</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_en_cot_cn</td>
    <td>0.95</td>
    <td>0.57</td>
    <td>0.23</td>
    <td>0.86</td>
    <td>0.74</td>
    <td>0.49</td>
    <td>0.39</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_en_cot_en</td>
    <td>0.69</td>
    <td>0.32</td>
    <td>0.54</td>
    <td>0.47</td>
    <td>0.48</td>
    <td>0.40</td>
    <td>0.41</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_en_xlt_cn</td>
    <td>0.03</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.04</td>
    <td>0.01</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_cn_cot_en</td>
    <td>0.70</td>
    <td>0.45</td>
    <td>0.35</td>
    <td>0.72</td>
    <td>0.55</td>
    <td>0.00</td>
    <td>0.38</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_cn_cot_en</td>
    <td>0.48</td>
    <td>0.26</td>
    <td>0.05</td>
    <td>0.53</td>
    <td>0.28</td>
    <td>0.36</td>
    <td>0.40</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_cn_xlt_en</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.03</td>
    <td>0.04</td>
    <td>0.00</td>
    <td>0.03</td>
    <td>0.00</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_en_cot_cn</td>
    <td>0.92</td>
    <td>0.44</td>
    <td>0.50</td>
    <td>0.88</td>
    <td>0.78</td>
    <td>0.46</td>
    <td>0.73</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_en_cot_en</td>
    <td>0.81</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.66</td>
    <td>0.43</td>
    <td>0.45</td>
    <td>0.56</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_en_xlt_cn</td>
    <td>0.52</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.54</td>
    <td>0.50</td>
    <td>0.43</td>
    <td>0.53</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_cn_cot_en</td>
    <td>0.75</td>
    <td>0.43</td>
    <td>0.28</td>
    <td>0.72</td>
    <td>0.55</td>
    <td>0.00</td>
    <td>0.58</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_cn_cot_en</td>
    <td>0.51</td>
    <td>0.20</td>
    <td>0.07</td>
    <td>0.53</td>
    <td>0.37</td>
    <td>0.32</td>
    <td>0.39</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_cn_xlt_en</td>
    <td>0.82</td>
    <td>0.25</td>
    <td>0.03</td>
    <td>0.73</td>
    <td>0.37</td>
    <td>0.47</td>
    <td>0.13</td>
    <td>0.40</td>
  </tr>
</table>

Table 12: Performance of Qwen Series (0.5B-3B). And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Qwen2.5-7B__direct_cn</td>
    <td>0.66</td>
    <td>0.34</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>0.62</td>
    <td>0.41</td>
    <td>0.77</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__direct_cn</td>
    <td>0.67</td>
    <td>0.35</td>
    <td>0.07</td>
    <td>0.69</td>
    <td>0.63</td>
    <td>0.43</td>
    <td>0.75</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__direct_cn</td>
    <td>0.68</td>
    <td>0.36</td>
    <td>0.06</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.43</td>
    <td>0.87</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__direct_cn</td>
    <td>0.67</td>
    <td>0.39</td>
    <td>0.08</td>
    <td>0.69</td>
    <td>0.65</td>
    <td>0.44</td>
    <td>0.92</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__direct_en</td>
    <td>0.64</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.82</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.56</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__direct_en</td>
    <td>0.66</td>
    <td>0.42</td>
    <td>0.05</td>
    <td>0.83</td>
    <td>0.28</td>
    <td>0.35</td>
    <td>0.58</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__direct_en</td>
    <td>0.65</td>
    <td>0.42</td>
    <td>0.05</td>
    <td>0.83</td>
    <td>0.29</td>
    <td>0.34</td>
    <td>0.65</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__direct_en</td>
    <td>0.67</td>
    <td>0.43</td>
    <td>0.05</td>
    <td>0.86</td>
    <td>0.30</td>
    <td>0.34</td>
    <td>0.71</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_en_cot_cn</td>
    <td>0.95</td>
    <td>0.57</td>
    <td>0.92</td>
    <td>0.90</td>
    <td>0.89</td>
    <td>0.53</td>
    <td>0.60</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_en_cot_en</td>
    <td>0.88</td>
    <td>0.50</td>
    <td>0.78</td>
    <td>0.84</td>
    <td>0.54</td>
    <td>0.54</td>
    <td>0.81</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_en_xlt_cn</td>
    <td>0.51</td>
    <td>0.27</td>
    <td>0.50</td>
    <td>0.48</td>
    <td>0.53</td>
    <td>0.41</td>
    <td>0.44</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_cn_cot_cn</td>
    <td>0.96</td>
    <td>0.55</td>
    <td>0.91</td>
    <td>0.92</td>
    <td>0.85</td>
    <td>0.59</td>
    <td>0.93</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_cn_cot_en</td>
    <td>0.93</td>
    <td>0.53</td>
    <td>0.79</td>
    <td>0.75</td>
    <td>0.56</td>
    <td>0.54</td>
    <td>0.81</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_cn_xlt_en</td>
    <td>0.76</td>
    <td>0.30</td>
    <td>0.77</td>
    <td>0.69</td>
    <td>0.43</td>
    <td>0.27</td>
    <td>0.65</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_en_cot_cn</td>
    <td>0.97</td>
    <td>0.58</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.88</td>
    <td>0.66</td>
    <td>0.94</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.50</td>
    <td>0.75</td>
    <td>0.88</td>
    <td>0.57</td>
    <td>0.55</td>
    <td>0.81</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_en_xlt_cn</td>
    <td>0.63</td>
    <td>0.40</td>
    <td>0.65</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.49</td>
    <td>0.69</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_cn_cot_cn</td>
    <td>0.97</td>
    <td>0.60</td>
    <td>0.94</td>
    <td>0.91</td>
    <td>0.87</td>
    <td>0.66</td>
    <td>0.92</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_cn_cot_en</td>
    <td>0.93</td>
    <td>0.56</td>
    <td>0.77</td>
    <td>0.83</td>
    <td>0.56</td>
    <td>0.55</td>
    <td>0.82</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_cn_xlt_en</td>
    <td>0.82</td>
    <td>0.38</td>
    <td>0.74</td>
    <td>0.71</td>
    <td>0.42</td>
    <td>0.36</td>
    <td>0.57</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_en_cot_cn</td>
    <td>0.98</td>
    <td>0.63</td>
    <td>0.91</td>
    <td>0.94</td>
    <td>0.92</td>
    <td>0.71</td>
    <td>0.95</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.59</td>
    <td>0.84</td>
    <td>0.88</td>
    <td>0.64</td>
    <td>0.58</td>
    <td>0.81</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_en_xlt_cn</td>
    <td>0.68</td>
    <td>0.46</td>
    <td>0.75</td>
    <td>0.66</td>
    <td>0.60</td>
    <td>0.49</td>
    <td>0.80</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_cn_cot_cn</td>
    <td>0.98</td>
    <td>0.61</td>
    <td>0.94</td>
    <td>0.93</td>
    <td>0.90</td>
    <td>0.68</td>
    <td>0.95</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_cn_cot_en</td>
    <td>0.94</td>
    <td>0.66</td>
    <td>0.84</td>
    <td>0.90</td>
    <td>0.62</td>
    <td>0.59</td>
    <td>0.83</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_cn_xlt_en</td>
    <td>0.82</td>
    <td>0.46</td>
    <td>0.82</td>
    <td>0.82</td>
    <td>0.51</td>
    <td>0.42</td>
    <td>0.59</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_en_cot_cn</td>
    <td>0.98</td>
    <td>0.66</td>
    <td>0.91</td>
    <td>0.94</td>
    <td>0.92</td>
    <td>0.73</td>
    <td>0.92</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_en_cot_en</td>
    <td>0.91</td>
    <td>0.59</td>
    <td>0.80</td>
    <td>0.92</td>
    <td>0.67</td>
    <td>0.58</td>
    <td>0.81</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_en_xlt_cn</td>
    <td>0.50</td>
    <td>0.30</td>
    <td>0.68</td>
    <td>0.62</td>
    <td>0.55</td>
    <td>0.54</td>
    <td>0.76</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_cn_cot_cn</td>
    <td>0.97</td>
    <td>0.62</td>
    <td>0.95</td>
    <td>0.93</td>
    <td>0.91</td>
    <td>0.74</td>
    <td>0.92</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_cn_cot_en</td>
    <td>0.94</td>
    <td>0.62</td>
    <td>0.80</td>
    <td>0.90</td>
    <td>0.69</td>
    <td>0.59</td>
    <td>0.83</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_cn_xlt_en</td>
    <td>0.59</td>
    <td>0.44</td>
    <td>0.81</td>
    <td>0.70</td>
    <td>0.53</td>
    <td>0.33</td>
    <td>0.57</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_en_cot_cn</td>
    <td>0.82</td>
    <td>0.56</td>
    <td>0.73</td>
    <td>0.85</td>
    <td>0.70</td>
    <td>0.54</td>
    <td>0.78</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_en_cot_en</td>
    <td>0.82</td>
    <td>0.40</td>
    <td>0.53</td>
    <td>0.70</td>
    <td>0.38</td>
    <td>0.53</td>
    <td>0.59</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_en_xlt_cn</td>
    <td>0.83</td>
    <td>0.55</td>
    <td>0.62</td>
    <td>0.80</td>
    <td>0.73</td>
    <td>0.53</td>
    <td>0.83</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_cn_cot_cn</td>
    <td>0.74</td>
    <td>0.42</td>
    <td>0.30</td>
    <td>0.76</td>
    <td>0.65</td>
    <td>0.00</td>
    <td>0.50</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_cn_cot_en</td>
    <td>0.70</td>
    <td>0.27</td>
    <td>0.09</td>
    <td>0.57</td>
    <td>0.33</td>
    <td>0.37</td>
    <td>0.54</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_cn_xlt_en</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_en_cot_cn</td>
    <td>0.82</td>
    <td>0.46</td>
    <td>0.81</td>
    <td>0.78</td>
    <td>0.66</td>
    <td>0.54</td>
    <td>0.86</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_en_cot_en</td>
    <td>0.88</td>
    <td>0.44</td>
    <td>0.41</td>
    <td>0.70</td>
    <td>0.46</td>
    <td>0.53</td>
    <td>0.72</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_en_xlt_cn</td>
    <td>0.93</td>
    <td>0.63</td>
    <td>0.85</td>
    <td>0.93</td>
    <td>0.79</td>
    <td>0.70</td>
    <td>0.92</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_cn_cot_cn</td>
    <td>0.79</td>
    <td>0.56</td>
    <td>0.74</td>
    <td>0.81</td>
    <td>0.73</td>
    <td>0.51</td>
    <td>0.82</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_cn_cot_en</td>
    <td>0.72</td>
    <td>0.37</td>
    <td>0.24</td>
    <td>0.64</td>
    <td>0.44</td>
    <td>0.38</td>
    <td>0.59</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_cn_xlt_en</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.17</td>
    <td>0.09</td>
    <td>0.22</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_en_cot_cn</td>
    <td>0.80</td>
    <td>0.43</td>
    <td>0.83</td>
    <td>0.81</td>
    <td>0.68</td>
    <td>0.57</td>
    <td>0.86</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_en_cot_en</td>
    <td>0.86</td>
    <td>0.52</td>
    <td>0.56</td>
    <td>0.82</td>
    <td>0.53</td>
    <td>0.54</td>
    <td>0.75</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_en_xlt_cn</td>
    <td>0.78</td>
    <td>0.58</td>
    <td>0.83</td>
    <td>0.75</td>
    <td>0.49</td>
    <td>0.52</td>
    <td>0.49</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_cn_cot_cn</td>
    <td>0.87</td>
    <td>0.60</td>
    <td>0.81</td>
    <td>0.87</td>
    <td>0.76</td>
    <td>0.00</td>
    <td>0.91</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_cn_cot_en</td>
    <td>0.82</td>
    <td>0.47</td>
    <td>0.30</td>
    <td>0.79</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.61</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_cn_xlt_en</td>
    <td>0.64</td>
    <td>0.45</td>
    <td>0.36</td>
    <td>0.62</td>
    <td>0.55</td>
    <td>0.27</td>
    <td>0.62</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_en_cot_cn</td>
    <td>0.84</td>
    <td>0.48</td>
    <td>0.82</td>
    <td>0.84</td>
    <td>0.72</td>
    <td>0.60</td>
    <td>0.73</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_en_cot_en</td>
    <td>0.78</td>
    <td>0.50</td>
    <td>0.44</td>
    <td>0.79</td>
    <td>0.49</td>
    <td>0.51</td>
    <td>0.75</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_en_xlt_cn</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.18</td>
    <td>0.21</td>
    <td>0.19</td>
    <td>0.05</td>
    <td>0.17</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_cn_cot_cn</td>
    <td>0.79</td>
    <td>0.59</td>
    <td>0.73</td>
    <td>0.86</td>
    <td>0.73</td>
    <td>0.56</td>
    <td>0.82</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_cn_cot_en</td>
    <td>0.70</td>
    <td>0.31</td>
    <td>0.15</td>
    <td>0.76</td>
    <td>0.55</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_cn_xlt_en</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.14</td>
    <td>0.20</td>
    <td>0.39</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.12</td>
  </tr>
</table>

Table 13: Performance of Qwen Series (7B-72B). And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__direct_cn</td>
    <td>0.61</td>
    <td>0.35</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>0.59</td>
    <td>0.43</td>
    <td>0.43</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__direct_cn</td>
    <td>0.65</td>
    <td>0.38</td>
    <td>0.08</td>
    <td>0.71</td>
    <td>0.63</td>
    <td>0.46</td>
    <td>0.55</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__direct_en</td>
    <td>0.53</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.81</td>
    <td>0.33</td>
    <td>0.34</td>
    <td>0.37</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__direct_en</td>
    <td>0.57</td>
    <td>0.42</td>
    <td>0.05</td>
    <td>0.85</td>
    <td>0.25</td>
    <td>0.34</td>
    <td>0.37</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_en_cot_cn</td>
    <td>0.85</td>
    <td>0.49</td>
    <td>0.35</td>
    <td>0.81</td>
    <td>0.62</td>
    <td>0.43</td>
    <td>0.40</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_en_cot_en</td>
    <td>0.85</td>
    <td>0.20</td>
    <td>0.45</td>
    <td>0.53</td>
    <td>0.26</td>
    <td>0.40</td>
    <td>0.90</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_en_xlt_cn</td>
    <td>0.27</td>
    <td>0.06</td>
    <td>0.10</td>
    <td>0.33</td>
    <td>0.42</td>
    <td>0.28</td>
    <td>0.28</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_cn_cot_cn</td>
    <td>0.88</td>
    <td>0.55</td>
    <td>0.66</td>
    <td>0.74</td>
    <td>0.63</td>
    <td>0.49</td>
    <td>0.56</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_cn_cot_en</td>
    <td>0.82</td>
    <td>0.27</td>
    <td>0.62</td>
    <td>0.44</td>
    <td>0.38</td>
    <td>0.34</td>
    <td>0.82</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_cn_xlt_en</td>
    <td>0.34</td>
    <td>0.16</td>
    <td>0.51</td>
    <td>0.46</td>
    <td>0.45</td>
    <td>0.29</td>
    <td>0.81</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_en_cot_cn</td>
    <td>0.96</td>
    <td>0.51</td>
    <td>0.89</td>
    <td>0.91</td>
    <td>0.84</td>
    <td>0.68</td>
    <td>0.81</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_en_cot_en</td>
    <td>0.92</td>
    <td>0.39</td>
    <td>0.88</td>
    <td>0.85</td>
    <td>0.53</td>
    <td>0.49</td>
    <td>0.88</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_en_xlt_cn</td>
    <td>0.42</td>
    <td>0.10</td>
    <td>0.70</td>
    <td>0.63</td>
    <td>0.51</td>
    <td>0.50</td>
    <td>0.36</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_cn_cot_cn</td>
    <td>0.97</td>
    <td>0.61</td>
    <td>0.91</td>
    <td>0.89</td>
    <td>0.82</td>
    <td>0.71</td>
    <td>0.90</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_cn_cot_en</td>
    <td>0.92</td>
    <td>0.43</td>
    <td>0.84</td>
    <td>0.79</td>
    <td>0.45</td>
    <td>0.44</td>
    <td>0.90</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_cn_xlt_en</td>
    <td>0.52</td>
    <td>0.24</td>
    <td>0.18</td>
    <td>0.74</td>
    <td>0.46</td>
    <td>0.34</td>
    <td>0.65</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_en_cot_cn</td>
    <td>0.18</td>
    <td>0.03</td>
    <td>0.03</td>
    <td>0.15</td>
    <td>0.05</td>
    <td>0.05</td>
    <td>0.12</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_en_cot_en</td>
    <td>0.11</td>
    <td>0.14</td>
    <td>0.01</td>
    <td>0.16</td>
    <td>0.09</td>
    <td>0.19</td>
    <td>0.01</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_en_xlt_cn</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.08</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_cn_cot_cn</td>
    <td>0.40</td>
    <td>0.18</td>
    <td>0.07</td>
    <td>0.23</td>
    <td>0.18</td>
    <td>0.00</td>
    <td>0.09</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_cn_cot_en</td>
    <td>0.27</td>
    <td>0.12</td>
    <td>0.07</td>
    <td>0.17</td>
    <td>0.13</td>
    <td>0.11</td>
    <td>0.00</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_cn_xlt_en</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_en_cot_cn</td>
    <td>0.03</td>
    <td>0.17</td>
    <td>0.01</td>
    <td>0.14</td>
    <td>0.12</td>
    <td>0.12</td>
    <td>0.40</td>
    <td>0.14</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_en_cot_en</td>
    <td>0.08</td>
    <td>0.12</td>
    <td>0.01</td>
    <td>0.11</td>
    <td>0.02</td>
    <td>0.08</td>
    <td>0.33</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_en_xlt_cn</td>
    <td>0.64</td>
    <td>0.34</td>
    <td>0.02</td>
    <td>0.29</td>
    <td>0.29</td>
    <td>0.30</td>
    <td>0.03</td>
    <td>0.27</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_cn_cot_cn</td>
    <td>0.36</td>
    <td>0.14</td>
    <td>0.03</td>
    <td>0.36</td>
    <td>0.18</td>
    <td>0.20</td>
    <td>0.34</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_cn_cot_en</td>
    <td>0.20</td>
    <td>0.08</td>
    <td>0.01</td>
    <td>0.08</td>
    <td>0.02</td>
    <td>0.07</td>
    <td>0.16</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_cn_xlt_en</td>
    <td>0.49</td>
    <td>0.06</td>
    <td>0.00</td>
    <td>0.29</td>
    <td>0.14</td>
    <td>0.11</td>
    <td>0.07</td>
    <td>0.17</td>
  </tr>
</table>

Table 14: Performance of DeepSeek Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Yi-6B__direct_cn</td>
    <td>0.68</td>
    <td>0.37</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>0.60</td>
    <td>0.42</td>
    <td>0.67</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Yi-9B__direct_cn</td>
    <td>0.70</td>
    <td>0.37</td>
    <td>0.08</td>
    <td>0.66</td>
    <td>0.64</td>
    <td>0.44</td>
    <td>0.62</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Yi-34B__direct_cn</td>
    <td>0.69</td>
    <td>0.38</td>
    <td>0.09</td>
    <td>0.67</td>
    <td>0.67</td>
    <td>0.44</td>
    <td>0.72</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Yi-6B__direct_en</td>
    <td>0.58</td>
    <td>0.39</td>
    <td>0.05</td>
    <td>0.77</td>
    <td>0.27</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Yi-9B__direct_en</td>
    <td>0.56</td>
    <td>0.41</td>
    <td>0.06</td>
    <td>0.80</td>
    <td>0.26</td>
    <td>0.35</td>
    <td>0.35</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Yi-34B__direct_en</td>
    <td>0.62</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.81</td>
    <td>0.27</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_en_cot_cn</td>
    <td>0.91</td>
    <td>0.60</td>
    <td>0.78</td>
    <td>0.84</td>
    <td>0.82</td>
    <td>0.56</td>
    <td>0.68</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_en_cot_en</td>
    <td>0.89</td>
    <td>0.36</td>
    <td>0.45</td>
    <td>0.68</td>
    <td>0.33</td>
    <td>0.51</td>
    <td>0.65</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_en_xlt_cn</td>
    <td>0.41</td>
    <td>0.10</td>
    <td>0.47</td>
    <td>0.48</td>
    <td>0.44</td>
    <td>0.35</td>
    <td>0.23</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_cn_cot_cn</td>
    <td>0.94</td>
    <td>0.58</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.76</td>
    <td>0.57</td>
    <td>0.72</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_cn_cot_en</td>
    <td>0.83</td>
    <td>0.46</td>
    <td>0.60</td>
    <td>0.57</td>
    <td>0.33</td>
    <td>0.40</td>
    <td>0.86</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_cn_xlt_en</td>
    <td>0.72</td>
    <td>0.23</td>
    <td>0.15</td>
    <td>0.55</td>
    <td>0.41</td>
    <td>0.27</td>
    <td>0.41</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_en_cot_cn</td>
    <td>0.97</td>
    <td>0.60</td>
    <td>0.85</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.64</td>
    <td>0.78</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.45</td>
    <td>0.78</td>
    <td>0.77</td>
    <td>0.55</td>
    <td>0.50</td>
    <td>0.77</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_en_xlt_cn</td>
    <td>0.48</td>
    <td>0.24</td>
    <td>0.68</td>
    <td>0.50</td>
    <td>0.45</td>
    <td>0.39</td>
    <td>0.41</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_cn_cot_cn</td>
    <td>0.96</td>
    <td>0.61</td>
    <td>0.90</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.65</td>
    <td>0.87</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_cn_cot_en</td>
    <td>0.92</td>
    <td>0.50</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.52</td>
    <td>0.47</td>
    <td>0.77</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_cn_xlt_en</td>
    <td>0.84</td>
    <td>0.25</td>
    <td>0.53</td>
    <td>0.69</td>
    <td>0.46</td>
    <td>0.29</td>
    <td>0.45</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_en_cot_cn</td>
    <td>0.96</td>
    <td>0.54</td>
    <td>0.91</td>
    <td>0.91</td>
    <td>0.88</td>
    <td>0.70</td>
    <td>0.93</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_en_cot_en</td>
    <td>0.92</td>
    <td>0.59</td>
    <td>0.79</td>
    <td>0.92</td>
    <td>0.60</td>
    <td>0.54</td>
    <td>0.82</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_en_xlt_cn</td>
    <td>0.27</td>
    <td>0.12</td>
    <td>0.58</td>
    <td>0.36</td>
    <td>0.44</td>
    <td>0.40</td>
    <td>0.43</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_cn_cot_cn</td>
    <td>0.96</td>
    <td>0.57</td>
    <td>0.92</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.68</td>
    <td>0.93</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_cn_cot_en</td>
    <td>0.93</td>
    <td>0.57</td>
    <td>0.79</td>
    <td>0.87</td>
    <td>0.57</td>
    <td>0.52</td>
    <td>0.78</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_cn_xlt_en</td>
    <td>0.62</td>
    <td>0.33</td>
    <td>0.71</td>
    <td>0.74</td>
    <td>0.49</td>
    <td>0.30</td>
    <td>0.57</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_en_cot_cn</td>
    <td>0.77</td>
    <td>0.52</td>
    <td>0.12</td>
    <td>0.67</td>
    <td>0.64</td>
    <td>0.34</td>
    <td>0.48</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_en_cot_en</td>
    <td>0.83</td>
    <td>0.43</td>
    <td>0.50</td>
    <td>0.59</td>
    <td>0.55</td>
    <td>0.54</td>
    <td>0.21</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_en_xlt_cn</td>
    <td>0.20</td>
    <td>0.17</td>
    <td>0.02</td>
    <td>0.19</td>
    <td>0.15</td>
    <td>0.06</td>
    <td>0.30</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_cn_cot_cn</td>
    <td>0.69</td>
    <td>0.27</td>
    <td>0.05</td>
    <td>0.63</td>
    <td>0.57</td>
    <td>0.00</td>
    <td>0.24</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_cn_cot_en</td>
    <td>0.58</td>
    <td>0.26</td>
    <td>0.04</td>
    <td>0.48</td>
    <td>0.25</td>
    <td>0.37</td>
    <td>0.17</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_cn_xlt_en</td>
    <td>0.05</td>
    <td>0.05</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.03</td>
    <td>0.07</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_en_cot_cn</td>
    <td>0.60</td>
    <td>0.47</td>
    <td>0.14</td>
    <td>0.67</td>
    <td>0.48</td>
    <td>0.39</td>
    <td>0.70</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_en_cot_en</td>
    <td>0.87</td>
    <td>0.38</td>
    <td>0.26</td>
    <td>0.68</td>
    <td>0.47</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_en_xlt_cn</td>
    <td>0.94</td>
    <td>0.51</td>
    <td>0.27</td>
    <td>0.85</td>
    <td>0.76</td>
    <td>0.56</td>
    <td>0.70</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_cn_cot_cn</td>
    <td>0.79</td>
    <td>0.36</td>
    <td>0.12</td>
    <td>0.72</td>
    <td>0.70</td>
    <td>0.50</td>
    <td>0.14</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_cn_cot_en</td>
    <td>0.61</td>
    <td>0.18</td>
    <td>0.10</td>
    <td>0.50</td>
    <td>0.35</td>
    <td>0.36</td>
    <td>0.29</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_cn_xlt_en</td>
    <td>0.91</td>
    <td>0.64</td>
    <td>0.03</td>
    <td>0.84</td>
    <td>0.45</td>
    <td>0.58</td>
    <td>0.59</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_en_cot_cn</td>
    <td>0.28</td>
    <td>0.37</td>
    <td>0.18</td>
    <td>0.24</td>
    <td>0.28</td>
    <td>0.17</td>
    <td>0.58</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_en_cot_en</td>
    <td>0.85</td>
    <td>0.48</td>
    <td>0.35</td>
    <td>0.75</td>
    <td>0.50</td>
    <td>0.47</td>
    <td>0.66</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_en_xlt_cn</td>
    <td>0.09</td>
    <td>0.03</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.07</td>
    <td>0.05</td>
    <td>0.09</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_cn_cot_cn</td>
    <td>0.82</td>
    <td>0.36</td>
    <td>0.29</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.00</td>
    <td>0.77</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_cn_cot_en</td>
    <td>0.77</td>
    <td>0.31</td>
    <td>0.05</td>
    <td>0.64</td>
    <td>0.50</td>
    <td>0.43</td>
    <td>0.64</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_cn_xlt_en</td>
    <td>0.78</td>
    <td>0.25</td>
    <td>0.02</td>
    <td>0.26</td>
    <td>0.43</td>
    <td>0.16</td>
    <td>0.34</td>
    <td>0.32</td>
  </tr>
</table>

Table 15: Performance of Yi Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__direct_cn</td>
    <td>0.57</td>
    <td>0.37</td>
    <td>0.09</td>
    <td>0.63</td>
    <td>0.51</td>
    <td>0.46</td>
    <td>0.63</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__direct_cn</td>
    <td>0.63</td>
    <td>0.40</td>
    <td>0.08</td>
    <td>0.67</td>
    <td>0.60</td>
    <td>0.46</td>
    <td>0.57</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__direct_en</td>
    <td>0.56</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.82</td>
    <td>0.27</td>
    <td>0.35</td>
    <td>0.36</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__direct_en</td>
    <td>0.57</td>
    <td>0.43</td>
    <td>0.04</td>
    <td>0.86</td>
    <td>0.29</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_en_cot_cn</td>
    <td>0.90</td>
    <td>0.45</td>
    <td>0.68</td>
    <td>0.81</td>
    <td>0.67</td>
    <td>0.57</td>
    <td>0.58</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_en_cot_en</td>
    <td>0.79</td>
    <td>0.39</td>
    <td>0.79</td>
    <td>0.64</td>
    <td>0.51</td>
    <td>0.45</td>
    <td>0.94</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_en_xlt_cn</td>
    <td>0.36</td>
    <td>0.21</td>
    <td>0.39</td>
    <td>0.48</td>
    <td>0.54</td>
    <td>0.39</td>
    <td>0.23</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_cn_cot_cn</td>
    <td>0.87</td>
    <td>0.45</td>
    <td>0.67</td>
    <td>0.73</td>
    <td>0.54</td>
    <td>0.60</td>
    <td>0.63</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_cn_cot_en</td>
    <td>0.76</td>
    <td>0.46</td>
    <td>0.56</td>
    <td>0.59</td>
    <td>0.39</td>
    <td>0.38</td>
    <td>0.82</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_cn_xlt_en</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.06</td>
    <td>0.27</td>
    <td>0.13</td>
    <td>0.05</td>
    <td>0.32</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_en_cot_cn</td>
    <td>0.97</td>
    <td>0.58</td>
    <td>0.87</td>
    <td>0.91</td>
    <td>0.89</td>
    <td>0.70</td>
    <td>0.91</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_en_cot_en</td>
    <td>0.92</td>
    <td>0.42</td>
    <td>0.77</td>
    <td>0.84</td>
    <td>0.57</td>
    <td>0.50</td>
    <td>0.88</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_en_xlt_cn</td>
    <td>0.46</td>
    <td>0.22</td>
    <td>0.65</td>
    <td>0.62</td>
    <td>0.56</td>
    <td>0.52</td>
    <td>0.52</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_cn_cot_cn</td>
    <td>0.97</td>
    <td>0.36</td>
    <td>0.87</td>
    <td>0.84</td>
    <td>0.69</td>
    <td>0.63</td>
    <td>0.92</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_cn_cot_en</td>
    <td>0.93</td>
    <td>0.45</td>
    <td>0.73</td>
    <td>0.86</td>
    <td>0.56</td>
    <td>0.44</td>
    <td>0.85</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_cn_xlt_en</td>
    <td>0.12</td>
    <td>0.09</td>
    <td>0.31</td>
    <td>0.50</td>
    <td>0.25</td>
    <td>0.14</td>
    <td>0.28</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_en_cot_cn</td>
    <td>0.59</td>
    <td>0.24</td>
    <td>0.07</td>
    <td>0.40</td>
    <td>0.32</td>
    <td>0.29</td>
    <td>0.40</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_en_cot_en</td>
    <td>0.52</td>
    <td>0.18</td>
    <td>0.11</td>
    <td>0.38</td>
    <td>0.19</td>
    <td>0.33</td>
    <td>0.38</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_en_xlt_cn</td>
    <td>0.39</td>
    <td>0.17</td>
    <td>0.01</td>
    <td>0.42</td>
    <td>0.16</td>
    <td>0.19</td>
    <td>0.16</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_cn_cot_cn</td>
    <td>0.53</td>
    <td>0.21</td>
    <td>0.09</td>
    <td>0.40</td>
    <td>0.34</td>
    <td>0.50</td>
    <td>0.12</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_cn_cot_en</td>
    <td>0.50</td>
    <td>0.16</td>
    <td>0.04</td>
    <td>0.30</td>
    <td>0.22</td>
    <td>0.24</td>
    <td>0.12</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_cn_xlt_en</td>
    <td>0.43</td>
    <td>0.08</td>
    <td>0.02</td>
    <td>0.22</td>
    <td>0.23</td>
    <td>0.15</td>
    <td>0.03</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_en_cot_cn</td>
    <td>0.78</td>
    <td>0.43</td>
    <td>0.06</td>
    <td>0.64</td>
    <td>0.63</td>
    <td>0.37</td>
    <td>0.56</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_en_cot_en</td>
    <td>0.78</td>
    <td>0.35</td>
    <td>0.04</td>
    <td>0.64</td>
    <td>0.45</td>
    <td>0.41</td>
    <td>0.57</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_en_xlt_cn</td>
    <td>0.79</td>
    <td>0.55</td>
    <td>0.05</td>
    <td>0.77</td>
    <td>0.49</td>
    <td>0.36</td>
    <td>0.22</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_cn_cot_cn</td>
    <td>0.63</td>
    <td>0.35</td>
    <td>0.13</td>
    <td>0.56</td>
    <td>0.48</td>
    <td>0.34</td>
    <td>0.63</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_cn_cot_en</td>
    <td>0.58</td>
    <td>0.30</td>
    <td>0.02</td>
    <td>0.50</td>
    <td>0.32</td>
    <td>0.31</td>
    <td>0.38</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_cn_xlt_en</td>
    <td>0.90</td>
    <td>0.39</td>
    <td>0.01</td>
    <td>0.86</td>
    <td>0.52</td>
    <td>0.50</td>
    <td>0.62</td>
    <td>0.54</td>
  </tr>
</table>

Table 16: Performance of LLaMA Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__direct_cn</td>
    <td>0.58</td>
    <td>0.37</td>
    <td>0.10</td>
    <td>0.64</td>
    <td>0.50</td>
    <td>0.44</td>
    <td>0.62</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__direct_cn</td>
    <td>0.59</td>
    <td>0.36</td>
    <td>0.09</td>
    <td>0.65</td>
    <td>0.54</td>
    <td>0.46</td>
    <td>0.58</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__direct_cn</td>
    <td>0.66</td>
    <td>0.39</td>
    <td>0.07</td>
    <td>0.69</td>
    <td>0.57</td>
    <td>0.43</td>
    <td>0.74</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__direct_en</td>
    <td>0.57</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.82</td>
    <td>0.31</td>
    <td>0.34</td>
    <td>0.40</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__direct_en</td>
    <td>0.56</td>
    <td>0.42</td>
    <td>0.06</td>
    <td>0.85</td>
    <td>0.31</td>
    <td>0.33</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__direct_en</td>
    <td>0.57</td>
    <td>0.42</td>
    <td>0.04</td>
    <td>0.85</td>
    <td>0.31</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_en_cot_cn</td>
    <td>0.90</td>
    <td>0.51</td>
    <td>0.62</td>
    <td>0.83</td>
    <td>0.71</td>
    <td>0.56</td>
    <td>0.57</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_en_cot_en</td>
    <td>0.86</td>
    <td>0.34</td>
    <td>0.84</td>
    <td>0.63</td>
    <td>0.53</td>
    <td>0.47</td>
    <td>0.90</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_en_xlt_cn</td>
    <td>0.13</td>
    <td>0.05</td>
    <td>0.17</td>
    <td>0.20</td>
    <td>0.30</td>
    <td>0.25</td>
    <td>0.44</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_cn_cot_cn</td>
    <td>0.56</td>
    <td>0.06</td>
    <td>0.34</td>
    <td>0.49</td>
    <td>0.41</td>
    <td>0.43</td>
    <td>0.51</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_cn_cot_en</td>
    <td>0.61</td>
    <td>0.28</td>
    <td>0.35</td>
    <td>0.45</td>
    <td>0.17</td>
    <td>0.16</td>
    <td>0.67</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_en_cot_cn</td>
    <td>0.94</td>
    <td>0.60</td>
    <td>0.80</td>
    <td>0.90</td>
    <td>0.81</td>
    <td>0.66</td>
    <td>0.65</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_en_cot_en</td>
    <td>0.91</td>
    <td>0.45</td>
    <td>0.59</td>
    <td>0.82</td>
    <td>0.56</td>
    <td>0.54</td>
    <td>0.87</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_en_xlt_cn</td>
    <td>0.15</td>
    <td>0.04</td>
    <td>0.18</td>
    <td>0.18</td>
    <td>0.34</td>
    <td>0.23</td>
    <td>0.24</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_cn_cot_cn</td>
    <td>0.75</td>
    <td>0.13</td>
    <td>0.35</td>
    <td>0.47</td>
    <td>0.52</td>
    <td>0.42</td>
    <td>0.76</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_cn_cot_en</td>
    <td>0.61</td>
    <td>0.33</td>
    <td>0.38</td>
    <td>0.56</td>
    <td>0.28</td>
    <td>0.13</td>
    <td>0.64</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_en_cot_cn</td>
    <td>0.96</td>
    <td>0.63</td>
    <td>0.90</td>
    <td>0.92</td>
    <td>0.86</td>
    <td>0.69</td>
    <td>0.87</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_en_cot_en</td>
    <td>0.92</td>
    <td>0.59</td>
    <td>0.69</td>
    <td>0.88</td>
    <td>0.63</td>
    <td>0.57</td>
    <td>0.78</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_en_xlt_cn</td>
    <td>0.26</td>
    <td>0.08</td>
    <td>0.36</td>
    <td>0.43</td>
    <td>0.36</td>
    <td>0.31</td>
    <td>0.33</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_cn_cot_cn</td>
    <td>0.60</td>
    <td>0.08</td>
    <td>0.36</td>
    <td>0.37</td>
    <td>0.57</td>
    <td>0.32</td>
    <td>0.82</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_cn_cot_en</td>
    <td>0.62</td>
    <td>0.44</td>
    <td>0.34</td>
    <td>0.65</td>
    <td>0.26</td>
    <td>0.21</td>
    <td>0.76</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_cn_xlt_en</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_en_cot_cn</td>
    <td>0.05</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.06</td>
    <td>0.03</td>
    <td>0.06</td>
    <td>0.11</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_en_cot_en</td>
    <td>0.27</td>
    <td>0.31</td>
    <td>0.02</td>
    <td>0.19</td>
    <td>0.08</td>
    <td>0.19</td>
    <td>0.11</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_en_xlt_cn</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_cn_cot_cn</td>
    <td>0.21</td>
    <td>0.08</td>
    <td>0.01</td>
    <td>0.17</td>
    <td>0.08</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_cn_cot_en</td>
    <td>0.11</td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.08</td>
    <td>0.04</td>
    <td>0.05</td>
    <td>0.08</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_en_cot_cn</td>
    <td>0.66</td>
    <td>0.25</td>
    <td>0.07</td>
    <td>0.47</td>
    <td>0.24</td>
    <td>0.40</td>
    <td>0.29</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_en_xlt_cn</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_cn_cot_cn</td>
    <td>0.52</td>
    <td>0.20</td>
    <td>0.04</td>
    <td>0.38</td>
    <td>0.27</td>
    <td>0.28</td>
    <td>0.14</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_cn_cot_en</td>
    <td>0.48</td>
    <td>0.17</td>
    <td>0.01</td>
    <td>0.31</td>
    <td>0.15</td>
    <td>0.23</td>
    <td>0.09</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_cn_xlt_en</td>
    <td>0.06</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_en_cot_cn</td>
    <td>0.89</td>
    <td>0.57</td>
    <td>0.05</td>
    <td>0.83</td>
    <td>0.60</td>
    <td>0.51</td>
    <td>0.57</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_en_cot_en</td>
    <td>0.82</td>
    <td>0.33</td>
    <td>0.13</td>
    <td>0.52</td>
    <td>0.48</td>
    <td>0.36</td>
    <td>0.58</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_en_xlt_cn</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_cn_cot_cn</td>
    <td>0.45</td>
    <td>0.27</td>
    <td>0.15</td>
    <td>0.39</td>
    <td>0.33</td>
    <td>0.26</td>
    <td>0.21</td>
    <td>0.29</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_cn_cot_en</td>
    <td>0.44</td>
    <td>0.22</td>
    <td>0.04</td>
    <td>0.35</td>
    <td>0.22</td>
    <td>0.21</td>
    <td>0.35</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_cn_xlt_en</td>
    <td>0.22</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.07</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.10</td>
    <td>0.07</td>
  </tr>
</table>

Table 17: Performance of Mixtral Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>gamma-2-2b__direct_cn</td>
    <td>0.58</td>
    <td>0.33</td>
    <td>0.08</td>
    <td>0.64</td>
    <td>0.55</td>
    <td>0.41</td>
    <td>0.59</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>gamma-2-9b__direct_cn</td>
    <td>0.59</td>
    <td>0.33</td>
    <td>0.06</td>
    <td>0.68</td>
    <td>0.59</td>
    <td>0.47</td>
    <td>0.55</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>gamma-2-27__direct_cn</td>
    <td>0.61</td>
    <td>0.33</td>
    <td>0.09</td>
    <td>0.64</td>
    <td>0.64</td>
    <td>0.44</td>
    <td>0.64</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>gamma-2-2b__direct_en</td>
    <td>0.49</td>
    <td>0.38</td>
    <td>0.05</td>
    <td>0.74</td>
    <td>0.35</td>
    <td>0.35</td>
    <td>0.39</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>gamma-2-9b__direct_en</td>
    <td>0.56</td>
    <td>0.39</td>
    <td>0.04</td>
    <td>0.77</td>
    <td>0.30</td>
    <td>0.36</td>
    <td>0.36</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>gamma-2-27__direct_en</td>
    <td>0.57</td>
    <td>0.42</td>
    <td>0.05</td>
    <td>0.85</td>
    <td>0.28</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_en_cot_cn</td>
    <td>0.86</td>
    <td>0.47</td>
    <td>0.24</td>
    <td>0.75</td>
    <td>0.55</td>
    <td>0.43</td>
    <td>0.30</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_en_cot_en</td>
    <td>0.69</td>
    <td>0.25</td>
    <td>0.48</td>
    <td>0.59</td>
    <td>0.37</td>
    <td>0.40</td>
    <td>0.80</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_en_xlt_cn</td>
    <td>0.28</td>
    <td>0.11</td>
    <td>0.22</td>
    <td>0.40</td>
    <td>0.43</td>
    <td>0.27</td>
    <td>0.15</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_cn_cot_cn</td>
    <td>0.70</td>
    <td>0.45</td>
    <td>0.31</td>
    <td>0.71</td>
    <td>0.50</td>
    <td>0.33</td>
    <td>0.46</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_cn_cot_en</td>
    <td>0.85</td>
    <td>0.41</td>
    <td>0.33</td>
    <td>0.27</td>
    <td>0.31</td>
    <td>0.26</td>
    <td>0.55</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_cn_xlt_en</td>
    <td>0.58</td>
    <td>0.14</td>
    <td>0.67</td>
    <td>0.35</td>
    <td>0.39</td>
    <td>0.24</td>
    <td>0.55</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_en_cot_cn</td>
    <td>0.96</td>
    <td>0.56</td>
    <td>0.80</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.62</td>
    <td>0.79</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_en_cot_en</td>
    <td>0.88</td>
    <td>0.50</td>
    <td>0.79</td>
    <td>0.84</td>
    <td>0.51</td>
    <td>0.55</td>
    <td>0.80</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_en_xlt_cn</td>
    <td>0.33</td>
    <td>0.16</td>
    <td>0.34</td>
    <td>0.51</td>
    <td>0.49</td>
    <td>0.41</td>
    <td>0.40</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_cn_cot_cn</td>
    <td>0.93</td>
    <td>0.47</td>
    <td>0.77</td>
    <td>0.85</td>
    <td>0.80</td>
    <td>0.54</td>
    <td>0.87</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_cn_cot_en</td>
    <td>0.91</td>
    <td>0.46</td>
    <td>0.71</td>
    <td>0.67</td>
    <td>0.50</td>
    <td>0.44</td>
    <td>0.76</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_cn_xlt_en</td>
    <td>0.73</td>
    <td>0.25</td>
    <td>0.42</td>
    <td>0.58</td>
    <td>0.41</td>
    <td>0.30</td>
    <td>0.61</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_en_cot_cn</td>
    <td>0.43</td>
    <td>0.33</td>
    <td>0.33</td>
    <td>0.43</td>
    <td>0.42</td>
    <td>0.32</td>
    <td>0.14</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_en_cot_en</td>
    <td>0.34</td>
    <td>0.15</td>
    <td>0.24</td>
    <td>0.27</td>
    <td>0.29</td>
    <td>0.21</td>
    <td>0.11</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_en_xlt_cn</td>
    <td>0.09</td>
    <td>0.04</td>
    <td>0.07</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.04</td>
    <td>0.04</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_cn_cot_cn</td>
    <td>0.30</td>
    <td>0.06</td>
    <td>0.15</td>
    <td>0.21</td>
    <td>0.23</td>
    <td>0.16</td>
    <td>0.22</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_cn_cot_en</td>
    <td>0.34</td>
    <td>0.10</td>
    <td>0.18</td>
    <td>0.23</td>
    <td>0.21</td>
    <td>0.13</td>
    <td>0.21</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_cn_xlt_en</td>
    <td>0.14</td>
    <td>0.01</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_en_cot_cn</td>
    <td>0.31</td>
    <td>0.19</td>
    <td>0.08</td>
    <td>0.15</td>
    <td>0.19</td>
    <td>0.09</td>
    <td>0.15</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_en_cot_en</td>
    <td>0.30</td>
    <td>0.28</td>
    <td>0.04</td>
    <td>0.17</td>
    <td>0.20</td>
    <td>0.16</td>
    <td>0.43</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_en_xlt_cn</td>
    <td>0.43</td>
    <td>0.17</td>
    <td>0.11</td>
    <td>0.25</td>
    <td>0.25</td>
    <td>0.15</td>
    <td>0.07</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_cn_cot_cn</td>
    <td>0.10</td>
    <td>0.13</td>
    <td>0.06</td>
    <td>0.08</td>
    <td>0.05</td>
    <td>0.00</td>
    <td>0.03</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_cn_cot_en</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.09</td>
    <td>0.05</td>
    <td>0.07</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_en_cot_cn</td>
    <td>0.66</td>
    <td>0.42</td>
    <td>0.06</td>
    <td>0.62</td>
    <td>0.53</td>
    <td>0.34</td>
    <td>0.40</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_en_cot_en</td>
    <td>0.71</td>
    <td>0.37</td>
    <td>0.04</td>
    <td>0.52</td>
    <td>0.27</td>
    <td>0.37</td>
    <td>0.53</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_en_xlt_cn</td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.05</td>
    <td>0.11</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_cn_cot_cn</td>
    <td>0.51</td>
    <td>0.36</td>
    <td>0.09</td>
    <td>0.54</td>
    <td>0.45</td>
    <td>0.29</td>
    <td>0.39</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_cn_cot_en</td>
    <td>0.64</td>
    <td>0.17</td>
    <td>0.03</td>
    <td>0.36</td>
    <td>0.23</td>
    <td>0.23</td>
    <td>0.12</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_cn_xlt_en</td>
    <td>0.33</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.04</td>
    <td>0.12</td>
    <td>0.02</td>
    <td>0.04</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_en_cot_cn</td>
    <td>0.37</td>
    <td>0.18</td>
    <td>0.08</td>
    <td>0.31</td>
    <td>0.29</td>
    <td>0.24</td>
    <td>0.15</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_en_cot_en</td>
    <td>0.21</td>
    <td>0.07</td>
    <td>0.05</td>
    <td>0.17</td>
    <td>0.14</td>
    <td>0.13</td>
    <td>0.12</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_en_xlt_cn</td>
    <td>0.22</td>
    <td>0.13</td>
    <td>0.05</td>
    <td>0.13</td>
    <td>0.14</td>
    <td>0.09</td>
    <td>0.03</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_cn_cot_cn</td>
    <td>0.15</td>
    <td>0.08</td>
    <td>0.05</td>
    <td>0.11</td>
    <td>0.13</td>
    <td>0.00</td>
    <td>0.07</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_cn_cot_en</td>
    <td>0.15</td>
    <td>0.04</td>
    <td>0.04</td>
    <td>0.05</td>
    <td>0.06</td>
    <td>0.04</td>
    <td>0.05</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_cn_xlt_en</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.02</td>
  </tr>
</table>

Table 18: Performance of Gemma Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__direct_cn</td>
    <td>0.59</td>
    <td>0.34</td>
    <td>0.09</td>
    <td>0.65</td>
    <td>0.60</td>
    <td>0.41</td>
    <td>0.55</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__direct_cn</td>
    <td>0.62</td>
    <td>0.37</td>
    <td>0.08</td>
    <td>0.66</td>
    <td>0.60</td>
    <td>0.43</td>
    <td>0.54</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__direct_en</td>
    <td>0.51</td>
    <td>0.38</td>
    <td>0.05</td>
    <td>0.76</td>
    <td>0.31</td>
    <td>0.34</td>
    <td>0.38</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__direct_en</td>
    <td>0.47</td>
    <td>0.40</td>
    <td>0.05</td>
    <td>0.80</td>
    <td>0.30</td>
    <td>0.34</td>
    <td>0.35</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_en_cot_cn</td>
    <td>0.82</td>
    <td>0.49</td>
    <td>0.32</td>
    <td>0.72</td>
    <td>0.62</td>
    <td>0.44</td>
    <td>0.55</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_en_cot_en</td>
    <td>0.67</td>
    <td>0.22</td>
    <td>0.70</td>
    <td>0.56</td>
    <td>0.24</td>
    <td>0.41</td>
    <td>0.68</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_en_xlt_cn</td>
    <td>0.15</td>
    <td>0.01</td>
    <td>0.23</td>
    <td>0.25</td>
    <td>0.39</td>
    <td>0.23</td>
    <td>0.30</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_cn_cot_cn</td>
    <td>0.89</td>
    <td>0.50</td>
    <td>0.54</td>
    <td>0.75</td>
    <td>0.70</td>
    <td>0.53</td>
    <td>0.51</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_cn_cot_en</td>
    <td>0.83</td>
    <td>0.28</td>
    <td>0.75</td>
    <td>0.50</td>
    <td>0.50</td>
    <td>0.33</td>
    <td>0.70</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_cn_xlt_en</td>
    <td>0.48</td>
    <td>0.20</td>
    <td>0.22</td>
    <td>0.56</td>
    <td>0.49</td>
    <td>0.33</td>
    <td>0.41</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_en_cot_cn</td>
    <td>0.94</td>
    <td>0.56</td>
    <td>0.71</td>
    <td>0.84</td>
    <td>0.78</td>
    <td>0.60</td>
    <td>0.58</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_en_cot_en</td>
    <td>0.78</td>
    <td>0.31</td>
    <td>0.36</td>
    <td>0.56</td>
    <td>0.48</td>
    <td>0.47</td>
    <td>0.81</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_en_xlt_cn</td>
    <td>0.22</td>
    <td>0.13</td>
    <td>0.39</td>
    <td>0.53</td>
    <td>0.47</td>
    <td>0.30</td>
    <td>0.41</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_cn_cot_cn</td>
    <td>0.89</td>
    <td>0.53</td>
    <td>0.76</td>
    <td>0.86</td>
    <td>0.72</td>
    <td>0.53</td>
    <td>0.61</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_cn_cot_en</td>
    <td>0.86</td>
    <td>0.42</td>
    <td>0.52</td>
    <td>0.57</td>
    <td>0.47</td>
    <td>0.35</td>
    <td>0.87</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_cn_xlt_en</td>
    <td>0.83</td>
    <td>0.26</td>
    <td>0.25</td>
    <td>0.49</td>
    <td>0.44</td>
    <td>0.34</td>
    <td>0.74</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_en_cot_cn</td>
    <td>0.31</td>
    <td>0.16</td>
    <td>0.01</td>
    <td>0.24</td>
    <td>0.21</td>
    <td>0.13</td>
    <td>0.03</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_en_cot_en</td>
    <td>0.29</td>
    <td>0.20</td>
    <td>0.01</td>
    <td>0.35</td>
    <td>0.08</td>
    <td>0.22</td>
    <td>0.01</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_en_xlt_cn</td>
    <td>0.19</td>
    <td>0.21</td>
    <td>0.02</td>
    <td>0.17</td>
    <td>0.10</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_cn_cot_cn</td>
    <td>0.42</td>
    <td>0.23</td>
    <td>0.07</td>
    <td>0.38</td>
    <td>0.22</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_cn_cot_en</td>
    <td>0.22</td>
    <td>0.11</td>
    <td>0.11</td>
    <td>0.29</td>
    <td>0.13</td>
    <td>0.16</td>
    <td>0.03</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_cn_xlt_en</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_en_cot_cn</td>
    <td>0.41</td>
    <td>0.21</td>
    <td>0.02</td>
    <td>0.25</td>
    <td>0.54</td>
    <td>0.11</td>
    <td>0.21</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_en_cot_en</td>
    <td>0.31</td>
    <td>0.38</td>
    <td>0.05</td>
    <td>0.45</td>
    <td>0.10</td>
    <td>0.35</td>
    <td>0.13</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_en_xlt_cn</td>
    <td>0.23</td>
    <td>0.19</td>
    <td>0.01</td>
    <td>0.28</td>
    <td>0.20</td>
    <td>0.05</td>
    <td>0.10</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_cn_cot_cn</td>
    <td>0.41</td>
    <td>0.17</td>
    <td>0.07</td>
    <td>0.32</td>
    <td>0.30</td>
    <td>0.16</td>
    <td>0.01</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_cn_cot_en</td>
    <td>0.13</td>
    <td>0.10</td>
    <td>0.06</td>
    <td>0.10</td>
    <td>0.14</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
</table>

Table 19: Performance of Baichuan Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__direct_cn</td>
    <td>0.62</td>
    <td>0.33</td>
    <td>0.09</td>
    <td>0.63</td>
    <td>0.52</td>
    <td>0.37</td>
    <td>0.38</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__direct_cn</td>
    <td>0.71</td>
    <td>0.34</td>
    <td>0.08</td>
    <td>0.66</td>
    <td>0.60</td>
    <td>0.43</td>
    <td>0.67</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__direct_cn</td>
    <td>0.69</td>
    <td>0.36</td>
    <td>0.08</td>
    <td>0.70</td>
    <td>0.66</td>
    <td>0.42</td>
    <td>0.45</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__direct_en</td>
    <td>0.55</td>
    <td>0.33</td>
    <td>0.08</td>
    <td>0.63</td>
    <td>0.41</td>
    <td>0.35</td>
    <td>0.35</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__direct_en</td>
    <td>0.59</td>
    <td>0.39</td>
    <td>0.06</td>
    <td>0.76</td>
    <td>0.36</td>
    <td>0.32</td>
    <td>0.37</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__direct_en</td>
    <td>0.62</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.78</td>
    <td>0.35</td>
    <td>0.31</td>
    <td>0.37</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_en_cot_cn</td>
    <td>0.89</td>
    <td>0.47</td>
    <td>0.39</td>
    <td>0.83</td>
    <td>0.73</td>
    <td>0.45</td>
    <td>0.40</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_en_cot_en</td>
    <td>0.84</td>
    <td>0.32</td>
    <td>0.37</td>
    <td>0.56</td>
    <td>0.39</td>
    <td>0.45</td>
    <td>0.42</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_en_xlt_cn</td>
    <td>0.44</td>
    <td>0.27</td>
    <td>0.47</td>
    <td>0.48</td>
    <td>0.53</td>
    <td>0.35</td>
    <td>0.40</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_cn_cot_cn</td>
    <td>0.86</td>
    <td>0.43</td>
    <td>0.55</td>
    <td>0.80</td>
    <td>0.67</td>
    <td>0.44</td>
    <td>0.50</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_cn_cot_en</td>
    <td>0.80</td>
    <td>0.44</td>
    <td>0.20</td>
    <td>0.49</td>
    <td>0.41</td>
    <td>0.45</td>
    <td>0.67</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_cn_xlt_en</td>
    <td>0.62</td>
    <td>0.20</td>
    <td>0.17</td>
    <td>0.48</td>
    <td>0.47</td>
    <td>0.35</td>
    <td>0.42</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_en_cot_cn</td>
    <td>0.78</td>
    <td>0.65</td>
    <td>0.88</td>
    <td>0.84</td>
    <td>0.87</td>
    <td>0.63</td>
    <td>0.77</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.52</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.53</td>
    <td>0.53</td>
    <td>0.88</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_en_xlt_cn</td>
    <td>0.62</td>
    <td>0.41</td>
    <td>0.61</td>
    <td>0.57</td>
    <td>0.59</td>
    <td>0.51</td>
    <td>0.42</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_cn_cot_cn</td>
    <td>0.95</td>
    <td>0.52</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.85</td>
    <td>0.61</td>
    <td>0.84</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_cn_cot_en</td>
    <td>0.91</td>
    <td>0.60</td>
    <td>0.75</td>
    <td>0.77</td>
    <td>0.47</td>
    <td>0.54</td>
    <td>0.82</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_cn_xlt_en</td>
    <td>0.78</td>
    <td>0.31</td>
    <td>0.48</td>
    <td>0.77</td>
    <td>0.40</td>
    <td>0.38</td>
    <td>0.62</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_en_cot_cn</td>
    <td>0.62</td>
    <td>0.58</td>
    <td>0.92</td>
    <td>0.28</td>
    <td>0.00</td>
    <td>0.48</td>
    <td>0.79</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_en_cot_en</td>
    <td>0.91</td>
    <td>0.63</td>
    <td>0.85</td>
    <td>0.88</td>
    <td>0.53</td>
    <td>0.55</td>
    <td>0.82</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_en_xlt_cn</td>
    <td>0.66</td>
    <td>0.38</td>
    <td>0.63</td>
    <td>0.42</td>
    <td>0.54</td>
    <td>0.59</td>
    <td>0.18</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_cn_cot_cn</td>
    <td>0.00</td>
    <td>0.27</td>
    <td>0.91</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.57</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_cn_cot_en</td>
    <td>0.91</td>
    <td>0.65</td>
    <td>0.79</td>
    <td>0.86</td>
    <td>0.51</td>
    <td>0.56</td>
    <td>0.84</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_cn_xlt_en</td>
    <td>0.74</td>
    <td>0.46</td>
    <td>0.90</td>
    <td>0.83</td>
    <td>0.53</td>
    <td>0.44</td>
    <td>0.69</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_en_cot_cn</td>
    <td>0.51</td>
    <td>0.22</td>
    <td>0.03</td>
    <td>0.42</td>
    <td>0.31</td>
    <td>0.26</td>
    <td>0.17</td>
    <td>0.27</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_en_cot_en</td>
    <td>0.15</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.16</td>
    <td>0.14</td>
    <td>0.13</td>
    <td>0.28</td>
    <td>0.14</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_en_xlt_cn</td>
    <td>0.06</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.05</td>
    <td>0.03</td>
    <td>0.00</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_cn_cot_cn</td>
    <td>0.67</td>
    <td>0.42</td>
    <td>0.17</td>
    <td>0.51</td>
    <td>0.40</td>
    <td>0.50</td>
    <td>0.34</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_cn_cot_en</td>
    <td>0.45</td>
    <td>0.32</td>
    <td>0.20</td>
    <td>0.32</td>
    <td>0.32</td>
    <td>0.28</td>
    <td>0.47</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_cn_xlt_en</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.08</td>
    <td>0.07</td>
    <td>0.03</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_en_cot_cn</td>
    <td>0.78</td>
    <td>0.65</td>
    <td>0.88</td>
    <td>0.84</td>
    <td>0.87</td>
    <td>0.63</td>
    <td>0.77</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.52</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.53</td>
    <td>0.53</td>
    <td>0.88</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_en_xlt_cn</td>
    <td>0.15</td>
    <td>0.08</td>
    <td>0.12</td>
    <td>0.12</td>
    <td>0.18</td>
    <td>0.11</td>
    <td>0.14</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_cn_cot_cn</td>
    <td>0.95</td>
    <td>0.52</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.85</td>
    <td>0.61</td>
    <td>0.84</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_cn_cot_en</td>
    <td>0.91</td>
    <td>0.60</td>
    <td>0.75</td>
    <td>0.77</td>
    <td>0.47</td>
    <td>0.54</td>
    <td>0.82</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_en_cot_cn</td>
    <td>0.39</td>
    <td>0.36</td>
    <td>0.15</td>
    <td>0.48</td>
    <td>0.55</td>
    <td>0.31</td>
    <td>0.22</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_en_cot_en</td>
    <td>0.48</td>
    <td>0.28</td>
    <td>0.11</td>
    <td>0.48</td>
    <td>0.41</td>
    <td>0.25</td>
    <td>0.75</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_en_xlt_cn</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.33</td>
    <td>0.54</td>
    <td>0.42</td>
    <td>0.26</td>
    <td>0.59</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_cn_cot_cn</td>
    <td>0.55</td>
    <td>0.53</td>
    <td>0.27</td>
    <td>0.71</td>
    <td>0.56</td>
    <td>0.00</td>
    <td>0.71</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_cn_cot_en</td>
    <td>0.54</td>
    <td>0.47</td>
    <td>0.03</td>
    <td>0.69</td>
    <td>0.43</td>
    <td>0.42</td>
    <td>0.30</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_cn_xlt_en</td>
    <td>0.80</td>
    <td>0.18</td>
    <td>0.02</td>
    <td>0.64</td>
    <td>0.43</td>
    <td>0.32</td>
    <td>0.89</td>
    <td>0.47</td>
  </tr>
</table>

Table 20: Performance of InternLM Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Instruct-Model</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>0.58</td>
    <td>0.36</td>
    <td>0.07</td>
    <td>0.65</td>
    <td>0.41</td>
    <td>0.36</td>
    <td>0.58</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>0.64</td>
    <td>0.36</td>
    <td>0.07</td>
    <td>0.70</td>
    <td>0.41</td>
    <td>0.37</td>
    <td>0.49</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>0.66</td>
    <td>0.40</td>
    <td>0.05</td>
    <td>0.75</td>
    <td>0.41</td>
    <td>0.40</td>
    <td>0.53</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>0.74</td>
    <td>0.39</td>
    <td>0.06</td>
    <td>0.76</td>
    <td>0.44</td>
    <td>0.39</td>
    <td>0.64</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>0.78</td>
    <td>0.40</td>
    <td>0.06</td>
    <td>0.77</td>
    <td>0.44</td>
    <td>0.40</td>
    <td>0.74</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>0.75</td>
    <td>0.40</td>
    <td>0.06</td>
    <td>0.78</td>
    <td>0.45</td>
    <td>0.40</td>
    <td>0.79</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>0.78</td>
    <td>0.41</td>
    <td>0.06</td>
    <td>0.79</td>
    <td>0.48</td>
    <td>0.41</td>
    <td>0.80</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B</td>
    <td>0.65</td>
    <td>0.38</td>
    <td>0.07</td>
    <td>0.71</td>
    <td>0.34</td>
    <td>0.40</td>
    <td>0.52</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B</td>
    <td>0.68</td>
    <td>0.40</td>
    <td>0.06</td>
    <td>0.73</td>
    <td>0.37</td>
    <td>0.41</td>
    <td>0.57</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.2</td>
    <td>0.67</td>
    <td>0.41</td>
    <td>0.07</td>
    <td>0.76</td>
    <td>0.35</td>
    <td>0.38</td>
    <td>0.58</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1</td>
    <td>0.65</td>
    <td>0.41</td>
    <td>0.07</td>
    <td>0.76</td>
    <td>0.41</td>
    <td>0.40</td>
    <td>0.52</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1</td>
    <td>0.70</td>
    <td>0.43</td>
    <td>0.05</td>
    <td>0.78</td>
    <td>0.42</td>
    <td>0.39</td>
    <td>0.57</td>
    <td>0.48</td>
  </tr>
</table>

Table 21: Performance of instruct models under Direct Prompt. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
HellaSwag-Pro Dataset Format

{
    "original_context": "A large group of people are seen standing around a beach as well as several shots of cars and people riding bulls. various people"
    "original_choices": ["are seen seen diving into the water, hitting the bulls back and fourth as well as playing a game of volleyball while cheering along", "then run to the bull and the bull fights them off while one stands by and watches.", "are shown speaking to the camera and others riding bulls around one another.", "ride the bulls and sit in the cars as well as end with a game of volleyball and celebrating."],
    "original_label": 3,
    "perturbation_type": "reverse_conversion",
    "context": "Various people ride the bulls and sit in the cars as well as end with a game of volleyball and celebrating. Which could be the most possible context for this action?",
    "choices": ["A large group of people are seen standing around a beach as well as several shots of cars and people riding bulls.", "A crowd gathers at a local park for a community event featuring live music and food trucks.", "Tourists explore a busy marketplace, taking photos and buying souvenirs.", "Children play in a playground while parents watch from nearby benches."],
    "label": 0
}

Figure 8: An example of HellaSwag-Pro.

Chinese HellaSwag Dataset Format

{
    "context": "丽丽报名参加了日本京都的一趟文化之旅，深度体验了传统艺伎表演。她",
    "choices": ["学习了传统的日式剑道和弓道技巧","欣赏了京都著名的樱花季和红叶景观","深深地被茶道的精致仪式所吸引。","品尝了正宗的关西风味章鱼烧和大阪烧"],
    "label": 2,
    "broad_type": "休闲娱乐",
    "detailed_type": "旅游体验"
}
{
    "context": "Lili signed up for a cultural tour in Kyoto, Japan, and experienced a traditional geisha performance. She",
    "choices": ["learned traditional Japanese kendo and archery skills", "enjoyed Kyoto's famous cherry blossom season and red leaves", "deeply attracted by the exquisite rituals of the tea ceremony", "tasted authentic Kansai-style takoyaki and okonomiyaki"],
    "label": 2,
    "broad_type": "Leisure",
    "detailed_type": "Travel Experience"
}

Figure 9: An example of Chinese HellaSwag.

Prompt for Chinese HellaSwag Construction

Type requirements:
You are a Chinese teacher with rigorous logic and rich common sense. Please help me write a question about commonsense reasoning. Each question contains an incomplete context and ten options. The context describes a common [broad_type] [detailed_type] scenario in the Chinese context. The sentence ends with an entity, such as "she", "this man", "they", "Zhang San", etc. This entity has rich [detailed_type] knowledge scenario [detailed_type_definition].
The content in the options is the scenarios that may occur in this context, but only the first option is the correct option, which is possible in reality, while the other nine contain logical errors or are not applicable to the context scenario or contradict common sense, but do not contain supernatural phenomena.
The questions are returned in json format, similar to the following sample. Note that the attribute name must be contained in double quotes.
{5-shot examples}
Length requirements:
The context field should be {less than 20 words}.
The choice field should have similar words. You should be as creative as possible and generate as many questions as possible. Pay attention to the fluency of the text, the clarity of the meaning, and the correctness of the grammar.

Figure 11: Prompt for Chinese HellaSwag construction.

Prompt For HellaSwag-Pro Construction

Total requirement:
Suppose you are a case generator. Given original_context, original_choices, original_label, your goal is to generate context, choices, label and explanation according to perturbation type. Your output should be in the following format: {original_context, original_choices, original_label, perturbation_type, context, choices, label and explanation}. I will provide some examples, and you should imitate my case generation process. You can be consistent with the perturbation_type provided to you.

{5-shot examples.}

Specific Variant Definition:
I hope you will concatenate the original_context and original_choices corresponding to the original_label into a complete paragraph and turn it into context; ending with Which could be the possible reason for the choice? and then explain the reason for such choice containing common sense as the correct option in choices. The correct option should be as concise as possible, and generate 9 other obviously wrong options according to the format and length of this option. The wrong option should contain wrong common sense. Put the correct option in the position of the first option and mark the label as 0. Note that context and choices should be fluent. The conversion process hopes to infer the possible reasons through the context and choices. So, how to convert the following input: {original_context: {} |original_choices: {}|original_label: {}}

Figure 10: Prompt for HellaSwag-Pro construction.

Reviewer paper summaries: This paper documents the creation of a new benchmark dataset for commonsense reasoning called HellaSwag-Pro, which consists of 7 types of adversarial variants for examples in both English and Chinese versions of HellaSwag (the latter also being created by the authors in this work). Altogether, this effort results in 11,200 4-way multiple-choice questions with adversarial variants, as well as 12,000 source questions for Chinese HellaSwag which were generated by LLMs across 56 categories. In addition to compiling HellaSwag-Pro and Chinese HellaSwag, the paper presents extensive experiments on 41 LLMs which measure their robustness across compiled variants, languages (of both data and models), and chain-of-thought style prompting strategies. The authors identify various trends in their results, but the strongest takeaway is that LLMs achieve high performance on source HellaSwag data (English and Chinese), but are largely not robust to adversarial variants of examples, which change the form of example content but not the meaning. This inflated performance may suggest that some LLMs have been trained directly on data in (English) HellaSwag, or more generally exhibit inconsistent understanding depending on the syntactic form of commonsense problems. The paper includes detailed fine-grained analyses of adversarial variant types and prompting configurations to bring further insights about these results.

This paper presents a substantial effort and the datasets collected would be valuable to the research community, especially around commonsense reasoning and robustness. I have previously reviewed this paper, and many of my earlier concerns have been resolved after discussion and revision.

This paper presents a new large-scale bilingual benchmark based on HellaSwag for comprehensive and robust evaluation of LLMs in commonsense reasoning. With the help of LLMs, authors construct a Chinese version of HellaSwag and then extend samples in these two benchmarks with several variants designed to test the robust commonsense reasoning abilities. Authors conduct an evaluation of more than 40 models on their new benchmark and show that LLMs still struggle to robustly reason about commonsense knowledge.
Summary of strengths: **Strength A: Motivations and contributions are clearly stated.** While I previously had concerns about the framing and motivation of the paper, I think this version clearly and effectively characterizes its motivation and contributions in Section 1. The paper is motivated by several works suggesting a lack of robustness in LLMs to minor changes in inputs, and presents an extensive study of robustness in the commonsense reasoning domain connected to Bloom’s cognitive model (further explained in Appendix A). Further, Section 5 adequately contextualizes this work with other related work, including in robustness evaluations, which were previously unaddressed.

**Strength B: New useful dataset.** It is rare to find benchmark datasets that LLMs can’t solve to high accuracy nowadays. This paper proposes a dataset that shows how LLMs are inconsistent commonsense reasoners, opening new avenues for model improvement in future work.

**Strength C: Sound and thorough data collection process.** The authors took several steps to ensure diversity and quality of the (largely LLM-generated) dataset, and the scope of the dataset is decided by a fairly principled connection to Bloom’s cognitive model. Consequently, this dataset seems to be a high-quality, valuable resource.

**Strength D: Thorough experiments.** The paper establishes baselines from an impressive breadth of common LLMs at various sizes, including both English and Chinese LLMs, and open- and closed-source LLMs. Furthermore, it includes additional analyses on the types of adversarial variants that LLMs found most challenging, as well as experiments with various types of in-context learning and chain-of-thought prompting strategies.

1. Paper is well-written and appropriately structured.
2. Benchmark construction methodology based on minimally modifying the question and the answer choices is quite reasonable and valuable to test robustness in commonsense reasoning.
3. New benchmark is large-scale, high-quality (verified by human annotators) and supports two languages.
4. Authors have conducted a comprehensive evaluation of humans and more than 40 models on their benchmark and revealed systematic gaps in robust commonsense reasoning.
5. Authors have performed extensive analysis with different prompting strategies and question variants.
Summary of weaknesses: **Weakness A: Motivating references are lacking.** While this revised version of the paper presents its motivation in a clearer way, there aren’t many references to back up the motivation. At L035, the paper presents a key motivation that “even slight changes to questions can lead to incorrect responses from the same models.” This claim, however, is only supported by Balepur et al. (2024), which finds that “LLMs, though able to identify the correct answer, struggle to reason about why the alternative is wrong.” While definitely related to the robustness of LLMs to various reasoning modes, this doesn’t seem directly connected to the previous statement motivating the work. Perhaps the motivating statement should be refined to something like “Even slight changes to questions and their underlying reasoning strategies can lead to incorrect responses from the same models.” Further, I would recommend adding more relevant references here, e.g., Ullman et al. (2023; https://arxiv.org/abs/2302.08399), which demonstrates LLMs’ lack of robustness to variations in theory-of-mind problems.

**Weakness B: May perpetuate issues in language modeling.** This paper applies adversarial filtering to create examples that LLMs struggle to solve, and uses it to show that LLMs can exhibit inconsistent understanding across various forms of the same commonsense knowledge. While the paper seems to be motivated by LLMs memorizing spurious patterns in data, what may ultimately happen is that this dataset is directly used to train future LLMs so that they exhibit a consistent understanding. This would necessitate the creation of another dataset like this, perpetuating the underlying issues in language models, which this paper offers no solution to. This weakness only factors slightly into my score, as the authors do consider this in the Ethics Statement section in this revised version, explicitly stating the expected purpose of the dataset and providing a thoughtful discussion around this issue.

There are no major weaknesses with this work.
Comments, suggestions and typos: Thank you for following my earlier suggestions and correcting the typos I found. Tables 3-5 are significantly easier to parse now. I have some additional suggestions and possible typos for the changed content below.

**General/higher-level comments:**

In the new sentence around L067-L076, I did not understand what was meant by “under initial questions” in the first half of the sentence. More generally, this discussion of past work around robustness feels defensive or overly critical, as if these works are severely limited. I would recommend to rephrase this sentence to emphasize how your work is more extensive or more updated than these prior works rather than highlighting what these papers are “lacking.”

The formatting of the quote added at L026 feels awkward and wastes space. Further, my interpretation of this quote is that intelligence should be adaptable, i.e., change over time as new information is gathered. I don’t think this quote is really referring to being robust to form variations of language, which I assume is an obvious thing virtually any human can do to quite high accuracy (regardless of their “measure of intelligence”). I’m not sure if this quote is a good use of space.

While we discussed the change from “comprehensive” to “extensive,” I still don’t like to use of the word “first”, e.g., at L077-078, the paper states “we present the first extensive evaluation on the robustness of commonsense reasoning for LLMs.” Extensive can mean a lot of things, and various past works have performed extensive evaluations, just not extensive in the same way as this work (i.e., extensive variations of reasoning strategies). I recommend removing references to this being the “first”, and instead focus on the specific unique contributions your work makes.

**Specific comments (typos, etc.):**

L036: “human naturally recognizes” -> “humans naturally recognize”

L515: “in” -> “across”?

N/A
Overall assessment scores: 4

4
Metareview score: 4
Metareview: This paper introduces HellaSwag-Pro, a large-scale bilingual benchmark designed to rigorously evaluate the robustness of LLMs in commonsense reasoning. It extends the popular HellaSwag dataset by constructing a Chinese version of HellaSwag (12,000 questions over 56 fine-grained categories) and by introducing 11,200 adversarial question variants in both English and Chinese, derived from seven reasoning transformations grounded in Bloom's cognitive taxonomy (e.g., causal inference, sentence ordering, and reverse transformation).

The authors conduct an extensive evaluation across 41 LLMs under nine prompting strategies, uncovering systematic performance drops on semantically equivalent variants compared to original questions. Their findings highlight that most LLMs, regardless of size or prompting strategy, struggle with robustness in commonsense reasoning, especially under transformations like negative or critical reasoning. The benchmark provides actionable insights into LLM limitations and offers a strong foundation for further research on robust commonsense understanding.
Summary of reasons to publish: All reviewers agree that this paper makes solid contributions and is worth acceptance somewhere. Some key highlights are:
- The paper introduces a rare, large-scale dataset that covers both English and Chinese, with meticulous attention to linguistic and cultural nuances, which enhances its cross-lingual evaluation value.
- The authors run evaluations on 41 models, across different sizes and architectures, along with nine prompting strategies — providing an unusually comprehensive empirical study.
- The benchmark reveals that current models perform well on standard datasets but falter under even minimal surface-level perturbations, raising important concerns for model deployment.
- The paper details annotation processes, quality control, and generation prompting templates, which enhances reproducibility and credibility.