HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
Authors: Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin
Accepted at: acl25

HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning

Xiaoyuan Li1, Moxin Li3*, Rui Men2*, Yichang Zhang2, Keqin Bao1, Wenjie Wang1, Fuli Feng1, Dayiheng Liu2, Junyang Lin2
University of Science and Technology of China1
Alibaba Group2
National University of Singapore3

Abstract

Large language models (LLMs) have shown remarkable capabilities in commonsense reasoning; however, some variations in questions can trigger incorrect responses. Do these models truly understand commonsense knowledge, or just memorize expression patterns? To investigate this question, we present the first extensive robustness evaluation of LLMs in commonsense reasoning. We introduce HellaSwag-Pro, a large-scale bilingual benchmark consisting of 11,200 cases, by designing and compiling seven types of question variants. To construct this benchmark, we propose a two-stage method to develop Chinese HellaSwag, a finely annotated dataset comprising 12,000 instances across 56 categories. We conduct extensive experiments on 41 representative LLMs, revealing that these LLMs are far from robust in commonsense reasoning. Furthermore, this robustness varies depending on the language in which the LLM is tested. This work establishes a high-quality evaluation benchmark, with extensive experiments offering valuable insights to the community in commonsense reasoning for LLMs.

1 Introduction

Commonsense reasoning is a crucial part of intelligence, involving contextual understanding, implicit knowledge, and logical deduction (Liu and Singh, 2004; Cambria et al., 2011; Davis and Marcus, 2015). Recent studies have focused on enhancing these capabilities in LLMs, achieving impressive performance (Yang et al., 2024; OpenAI et al., 2024; Team et al., 2024). However, even slight changes to questions can lead to incorrect responses from the same models. For instance, in binary commonsense questions, human naturally recognizes both correct and incorrect options through a single inference process, while LLMs,

*Corresponding authors. Work done when Xiaoyuan Li and Keqin Bao were intern at Alibaba Group.

![Comparison of GPT-4o’s responses to an original question and its several meaning-preserving variants. GPT-4o successfully handles the original question but struggles with its variants on the same knowledge.](page_820_1097_579_312.png)

Figure 1: Comparison of GPT-4o’s responses to an original question and its several meaning-preserving variants. GPT-4o successfully handles the original question but struggles with its variants on the same knowledge.

though able to identify the correct answer, struggle to reason about why the alternative is wrong (Balepur et al., 2024). Therefore, we ask the question: Does this high-level performance stem from a genuine understanding of commonsense knowledge, or is it simply a result of memorizing specific expression patterns in pre-training data?

To answer this question, an effective approach is to systematically evaluate the robustness of LLMs in answering commonsense reasoning questions. As illustrated in Figure 1, we find that GPT-4o correctly answers an original question but fails on its variants, i.e., questions about the same commonsense knowledge but in different reasoning forms, such as reverse conversion. This indicates that GPT-4o has not fully grasped the commonsense knowledge behind the question; a genuine understanding of commonsense knowledge should be able to generalize to these question variants.

However, existing benchmarks do not yet sup-
<table>
  <tr>
    <th>Variant Type</th>
    <th>Context</th>
    <th>Choices</th>
  </tr>
  <tr>
    <td rowspan="1">Initial data</td>
    <td>A lady walks to a barbell. She bends down and grabs the pole. The lady</td>
    <td>A. stands and lifts the weight over her head.<br>B. swings and lands in her arms.<br>C. pulls the barbell forward.<br>D. pulls a rope attached to the barbell.</td>
  </tr>
  <tr>
    <td rowspan="1">Problem restatement</td>
    <td><b>A woman approaches a weightlifting bar. She lowers her body and grasps the metal rod. The woman</b></td>
    <td>A. rises and hoists the barbell above her head.<br>B. swings and lands in her arms.<br>C. pulls the barbell forward.<br>D. pulls a rope attached to the barbell.</td>
  </tr>
  <tr>
    <td rowspan="1">Reverse conversion</td>
    <td>The lady stands and lifts the weight over her head. Which could be the most possible context for this action?</td>
    <td>A. A lady walks to a barbell. She bends down and grabs the pole.<br>B. A lady positions herself at the squat rack. She lowers her body before rising steadily.<br>C. A lady approaches the kettlebell set. She swings the weight forcefully between her legs.<br>D. A lady stands beside the bench press station. She lies down and lifts the barbell from her chest.</td>
  </tr>
  <tr>
    <td rowspan="1">Causal inference</td>
    <td>A lady walks to a barbell. She bends down and grabs the pole. The lady stands and lifts the weight over her head. Which could be the most possible reason for this action?</td>
    <td>A. She is performing a weightlifting exercise.<br>B. She is using the barbell as a decoration for an event.<br>C. She is moving the barbell to a different location in the gym.<br>D. She is cleaning the barbell after a workout session.</td>
  </tr>
  <tr>
    <td rowspan="1">Sentence ordering</td>
    <td>1. She bends down and grabs the pole.<br>2. A lady walks to a barbell.<br>3. The lady stands and lifts the weight over her head. Which is the correct order?</td>
    <td>A. 2-1-3<br>B. 3-1-2<br>C. 2-3-1<br>D. 1-3-2</td>
  </tr>
  <tr>
    <td rowspan="1">Scenario refinement</td>
    <td>A lady walks to a barbell. She bends down and grabs the pole. The lady hesitates for a moment, then changes her mind. Instead of lifting the barbell, she</td>
    <td>A. swings and lands in her arms.<br>B. stands and lifts the weight over her head.<br>C. pulls the barbell forward.<br>D. pulls a rope attached to the barbell.</td>
  </tr>
  <tr>
    <td rowspan="1">Negative transformation</td>
    <td>A woman approaches a weightlifting bar. She lowers her body and grasps the metal rod. The lady <u>will not</u></td>
    <td>A. swings and lands in her arms.<br>B. stands and lifts the weight over her head.<br>C. bend her knees and lift the barbell.<br>D. adjust her grip and lift the weight.</td>
  </tr>
  <tr>
    <td rowspan="1">Critical testing</td>
    <td>A lady walks to a barbell. She bends down and grabs the pole. The lady <u>suddenly realizes she forgot her weightlifting gloves</u> and decides to postpone her workout. The lady</td>
    <td>A. stands and lifts the weight over her head.<br>B. swings and lands in her arms.<br>C. pulls the barbell forward.<br>D. pulls a rope attached to the barbell.<br>E. None of the above four options are suitable.</td>
  </tr>
</table>

Table 1: Examples of the seven variants we adopt for an initial question, with the correct answer unchanged as (A). Modifications are highlighted in different colors for clarity.

port a thorough evaluation of LLM robustness in commonsense reasoning. Most work evaluates LLMs on general benchmarks (Zellers et al., 2019; Talmor et al., 2019; Mihaylov et al., 2018a), or in specific domains of commonsense knowledge (Zhou et al., 2019; Qin et al., 2021; Bisk et al., 2020). Although some efforts have considered the robustness of commonsense reasoning, they either focus on whether models can learn genuine question-answer correlations under initial questions (Jia and Liang, 2017; Branco et al., 2021), or examine only one type of simplistic question variant such as question paraphrasing (Zhou et al., 2021; Ismayilzada et al., 2023; Balepur et al., 2024), lacking investigation into robustness across diverse and complex variants.

To address this gap, we present the first extensive evaluation on the robustness of commonsense reasoning for LLMs, starting with dataset construction. Firstly, recognizing that existing benchmarks are predominantly in English, which limits the assessment of non-English LLMs (Davis, 2023), we develop a Chinese commonsense reasoning dataset based on the widely-used HellaSwag benchmark (Zellers et al., 2019), containing 12,000 questions. Specifically, we design 56 fine-grained categories, and propose a two-stage data annotation method including initial dataset generation and difficult sample replacement. Secondly, we design and compile seven variants from existing studies (cf. Table 1), which can be characterized under Bloom Cognitive Model (cf. Appendix A). We then create the variants for the Chinese and English versions of HellaSwag, obtaining HellaSwag-Pro, a high-quality human-verified dataset with 11,200 variants from 1,600 original questions.

Using HellaSwag-Pro, we conduct a comprehensive evaluation on the robustness of 41 closed-source and open-source LLMs with nine different prompt strategies. We derive several key findings: (1) All LLMs are far from robust in commonsense reasoning tasks, as evidenced by their poor performance on question variants and the significant gap compared to human performance. Neverthe-
less, GPT-4o achieves the best robustness among all the evaluated LLMs. (2) Among all types of variants, negative transformation is the most challenging, with an average accuracy of only 9.01%, while problem restatement poses minimal difficulty. (3) LLMs achieve the best robustness in the language on which they were adequately trained. (4) Incorporating chain-of-thought (CoT) reasoning and using few-shot demonstrations can strengthen their robustness.

Our contributions are three-fold. (1) We present the first extensive evaluation on the robustness of commonsense reasoning for LLMs by designing and compiling seven types of variants. (2) We have developed a bilingual, large-scale, human-annotated benchmark for evaluating LLM robustness in commonsense reasoning, which will be publicly released upon acceptance. (3) We conduct in-depth experiments on 41 representative LLMs with diverse prompts, yielding critical insights.

2 Chinese HellaSwag

Given the limitation that most existing benchmarks for commonsense reasoning are in English, we begin by building a Chinese benchmark for commonsense reasoning that captures unique aspects of Chinese cultural context. Firstly, we structure the dataset following the format of HellaSwag (Zellers et al., 2019), a widely recognized English commonsense reasoning benchmark, which consists of multiple-choice questions with four answer options. Secondly, to minimize manual effort, we incorporate Qwen-Max (Yang et al., 2024), a state-of-the-art Chinese LLM, into the dataset construction process. Finally, to enhance the diversity of the dataset, we develop a hierarchical taxonomy of commonsense knowledge, as shown in Figure 3. Our taxonomy consists of seven broad categories summarized from existing literature (Zellers et al., 2019; Koupaee and Wang, 2018; Caba Heilbron et al., 2015), each containing eight subcategories. We aim to construct our dataset based on the taxonomy, where we inject the categorical information into the instruction for LLM generation.

We propose a two-stage data construction pipeline, initial dataset generation and difficult sample replacement, as shown in Figure 2.

Initial Dataset Generation In this stage, we employ an over-generate-then-filter (Yuan et al., 2023) approach, i.e., generating excessive question-answer pairs and filtering for high quality ones, to obtain the initial dataset. The generation of the initial dataset consists of three steps.

• Step 1: Context over-generation. We employ the LLM to create a Chinese context of the question via in-context learning (Brown et al., 2020), incorporating category information, length requirement, and carefully crafted five-shot Chinese examples similar to HellaSwag. For the length requirement, we assign three tiers: short (under 20 characters), medium (20-40 characters), and long (over 40 characters). We then filter the generated contexts based on character count and Jaccard similarity, eliminating samples that do not meet the length requirement or are too similar to other samples.

• Step 2: Choice over-generation. For each context, we instruct the LLM to over-generate ten potential choices, forming a question.

• Step 3: Choice filtering. We instruct the LLM to evaluate each question on a ten-point scale and select six choices: one correct answer (10 points) and five high-scoring incorrect choices. Then, human annotators select four choices, ensuring a single correct answer and three challenging incorrect choices, and check the category labels for the question. After the LLM scoring, we obtain 12,960 samples, which human annotators further refine to 12,287. To maintain category balance, we ultimately select 12,000 samples, allocating 1,500 to each broad category.

![The two-stage data construction pipeline for Chinese HellaSwag. See an example in Table 9.](page_1012_186_388_388.png)

Figure 2: The two-stage data construction pipeline for Chinese HellaSwag. See an example in Table 9.
Figure 3: Overview of Chinese HellaSwag categories. There are seven broad categories in total, each with eight detailed subcategories.

<table>
  <tr>
    <th>Length Type</th>
    <th>Long</th>
    <th>Medium</th>
    <th>Short</th>
    <th>Total</th>
  </tr>
  <tr>
    <td># Questions</td>
    <td>4,179</td>
    <td>4,033</td>
    <td>3,788</td>
    <td>12,000</td>
  </tr>
</table>

Table 2: Statistics for Chinese HellaSwag.

Difficult Sample Replacement After initial dataset generation, we notice that some incorrect choices are rather simple for LLMs to identify, making the Chinese HellaSwag much easier than its English counterpart. Following the adversarial filtering (Zellers et al., 2018), we use a human-in-the-loop adversarial filtering method (Step 4) to further enhance the dataset’s difficulty. This process involves using a generator LLM to rewrite existing incorrect choices into more challenging ones, and then evaluating the generated choices on multiple discriminator LLMs. If the generated choice successfully misleads the discriminator LLMs, we replace the original choice with the newly generated one. Finally, human annotators filter out the generated choices that are too difficult for humans to identify (see detail in Appendix B). We iterative perform this process until the Chinese HellaSwag achieves accuracy comparable to the English HellaSwag, resulting in replacing 2451 samples. The dataset statistics of the Chinese HellaSwag can be found in Table 2. The complete evaluation of it is given in Appendix E.

3 HellaSwag-Pro

Based on the English-Chinese HellaSwag datasets, we construct HellaSwag-Pro, the benchmark for extensive robustness evaluation of commonsense reasoning. We begin by designing the seven-type question variants for robustness evaluation, then detail our data generation process.

3.1 Variant Types

We aim to evaluate the robustness of commonsense reasoning on question variants of changed reasoning forms for the same commonsense knowledge. The rationality is that the diverse reasoning forms disables the reliance on superficial patterns, ensuring that correct answers from LLMs demonstrate a robust understanding of the underlying commonsense knowledge. Building on existing research (Guo et al., 2024; Ma et al., 2025; Balepur et al., 2024) and our own designs, we maintain seven types of variants, as detailed below.

• Problem restatement aims to test the impact of textual description variations on model understanding. We rephrase the context and correct choice while keeping the incorrect choices unchanged, thereby increasing the difficulty of identifying the correct answer.

• Reverse conversion evaluates the capability for reverse reasoning, i.e., inferring the context from the outcome, which has been shown to be challenging for LLMs (Guo et al., 2024). We utilize the original correct choice as the context, the original context as the correct choice, and generate three additional incorrect choices.

• Causal inference evaluates the understanding of the causality of the event (Li et al., 2025). We merge the context and the correct choice and ask for the reason. We generate one correct reason and produce three additional incorrect reasons as the choices.

• Sentence ordering focuses on the understanding of inter-sentence relationships, such as progression or contrast. We concatenate the context and correct choice into a complete paragraph, then shuffle the order of the sentences. The correct choice refers to the original sentence ordering.

• Scenario refinement investigates the ability to infer counterfactual situations (Ma et al., 2025). We select a relatively plausible choice from the original incorrect choices, then minimally modify the context to make this choice correct, where the original correct choice becomes incorrect.

• Negation transformation examines the robustness to negation, a known challenge for LLMs (Balepur et al., 2024). This involves altering the
context by introducing negations, such as changing "the man will" to "the man will not." In this transformation, the least plausible choice in the original question becomes the correct answer for the variant, while the original correct answer is retained, and two additional plausible options are generated as distractors.

• Critical testing evaluates the model’s ability to abstain from answering when the context lacks sufficient information to determine a correct answer. We remove key details from the context to make all original choices invalid. We keep the context minimally modified to increase difficulty. A new choice, “None of the above four options are suitable”, is introduced as the correct choice.

3.2 Data Generation

To construct these variants, we also employ Qwen-Max due to its comparatively strong language ability in reforming the questions. We design in-context examples and instructions with transformation rules to guide Qwen-Max to generate the question variants (cf. Appendix B.4.3). However, we observe that Qwen-Max is not consistently reliable, exhibiting issues such as: (1) generating variants inconsistent with the definitions, (2) producing multiple correct choices or overly simple incorrect choices, and (3) generating invalid contexts, particularly in scenario refinement.

To tackle these issues, we leverage manual quality control over the generated data. For reverse conversion and causal inference, we adopt an over-generate-then-filter approach (cf. Section 2) to control the correctness and the quality of the generated choices. Finally, we conduct comprehensive manual verification of all variants generated to ensure data quality. We initially generate 24,260 variants, and eventually filter down to 11,200 high-quality variants from 1,600 original questions.

4 Experiment

In this section, we conduct extensive experiments to evaluate the performance of various LLMs on our HellaSwag-Pro benchmark. Our study is guided by three key research questions: RQ1: How do different LLMs perform across all variants? RQ2: What is the relative difficulty of different variants? RQ3: Which prompting strategies yield the best robustness in LLMs?

4.1 Experimental Setup

Model Selection and Implementation Details
We select 41 representative closed-source and open-source LLMs. For English LLMs, we use GPT-4o (OpenAI, 2023), Claude-3.5-Sonnet (Anthropic, 2024), Gemini-1.5-Pro (Anil et al., 2023), Mistral series (Jiang et al., 2023), Llama3 series (Dubey et al., 2024) and Gemma2 series (Rivière et al., 2024). For Chinese LLMs, we use Qwen-Max (Bai et al., 2023), Qwen2.5 series (Yang et al., 2024), InternLM2.5 series (Team, 2023), Yi1.5 series (Young et al., 2024), Baichuan2 series (Yang et al., 2023) and DeepSeek series (Bi et al., 2024).

We integrate both the Chinese HellaSwag and HellaSwag-Pro into the lm-evaluation-harness platform (Gao et al., 2024). For the open-source models, we use the default settings of the platform: do_sample is set to false and the temperature is set to the default value of the hugging-face library as 1.0. For the closed-source models, we set the temperature to 0.7. In addition, we set the maximum output length to 1024.

Prompting Strategy  We design nine prompting strategies to evaluate the LLMs across different languages and number of demonstrations. (1) Direct: LLM takes the original dataset question directly as input1. (2) CN-CoT: LLM is instructed to perform CoT in Chinese, regardless of the language of the dataset. (3) EN-CoT: LLM is instructed to perform CoT in English. (4) CN-XLT: LLM is instructed to first translate the English question into Chinese, then reason in Chinese. (5) EN-XLT: LLM is instructed to first translate the Chinese question into English, then reason in English. The last four strategies include both zero-shot and three-shot variants.

Evaluation Metric  We consider four evaluation metrics to measure the performance and robustness of LLMs. Denote the original dataset \( \mathcal{D} = \{(x, y)\} \), where \( x \) and \( y \) represent the question and the correct label, respectively. Denote the dataset of all seven-type variants \( \mathcal{D}_r = \{(x', y')\} \), where each \( (x', y') \) corresponds to an original \( (x, y) \) in \( \mathcal{D} \).
Original Accuracy (OA) measures the accuracy

1For open-source models, the Direct approach follows the official HellaSwag implementation, computing the log-likelihood for each option and selecting the one with the highest value. We report the normalized accuracy to account for the option length. Other prompting strategies use a generation setup and report accuracy based on exact match.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Chinese</th>
    <th colspan="4">English</th>
    <th colspan="4">AVG</th>
  </tr>
  <tr>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>96.41</td>
    <td>97.79</td>
    <td>-1.38</td>
    <td>92.03</td>
    <td>95.56</td>
    <td>96.04</td>
    <td>-0.48</td>
    <td>90.02</td>
    <td>95.99</td>
    <td>96.92</td>
    <td>-0.93</td>
    <td>91.03</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>0.00</td>
    <td>0.0015</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>0.00</td>
    <td>0.0015</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>0.00</td>
    <td>0.0015</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>93.50</td>
    <td>84.82</td>
    <td>8.68</td>
    <td>78.91</td>
    <td>87.60</td>
    <td>62.61</td>
    <td>24.99</td>
    <td>59.65</td>
    <td>90.55</td>
    <td>73.72</td>
    <td>16.83</td>
    <td>69.28</td>
  </tr>
  <tr>
    <th colspan="14">Closed-source LLMs</th>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>60.75</td>
    <td>45.18</td>
    <td><b>15.57</b></td>
    <td>28.70</td>
    <td>49.50</td>
    <td>38.21</td>
    <td><b>11.29</b></td>
    <td>20.57</td>
    <td>55.13</td>
    <td>41.70</td>
    <td><b>13.43</b></td>
    <td>24.64</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>63.25</td>
    <td>46.16</td>
    <td>17.09</td>
    <td>29.89</td>
    <td>56.88</td>
    <td>39.57</td>
    <td>17.30</td>
    <td>23.48</td>
    <td>60.06</td>
    <td>42.87</td>
    <td>17.20</td>
    <td>26.69</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>67.50</td>
    <td>48.75</td>
    <td>18.75</td>
    <td>33.79</td>
    <td>61.75</td>
    <td>39.98</td>
    <td>21.77</td>
    <td>25.75</td>
    <td>64.63</td>
    <td>44.37</td>
    <td>20.26</td>
    <td>29.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>67.63</td>
    <td>50.59</td>
    <td>17.04</td>
    <td>35.62</td>
    <td>65.63</td>
    <td>43.93</td>
    <td>21.70</td>
    <td>30.77</td>
    <td>66.63</td>
    <td>47.26</td>
    <td>19.37</td>
    <td>33.20</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>69.00</td>
    <td>51.41</td>
    <td>17.59</td>
    <td>35.84</td>
    <td>68.50</td>
    <td>45.20</td>
    <td>23.30</td>
    <td>32.12</td>
    <td>68.75</td>
    <td>48.30</td>
    <td>20.45</td>
    <td>33.98</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>69.75</td>
    <td>53.11</td>
    <td>16.64</td>
    <td>37.54</td>
    <td>70.00</td>
    <td>46.10</td>
    <td>23.90</td>
    <td>32.68</td>
    <td>69.88</td>
    <td>49.61</td>
    <td>20.27</td>
    <td>35.11</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>70.87</td>
    <td><b>54.75</b></td>
    <td>16.12</td>
    <td><b>39.64</b></td>
    <td>72.00</td>
    <td><b>47.75</b></td>
    <td>24.25</td>
    <td><b>35.12</b></td>
    <td><b>71.44</b></td>
    <td><b>51.25</b></td>
    <td>20.19</td>
    <td><b>37.38</b></td>
  </tr>
  <tr>
    <td>Baichuan2-7B</td>
    <td>67.00</td>
    <td>46.16</td>
    <td>20.84</td>
    <td>31.50</td>
    <td>60.62</td>
    <td>39.04</td>
    <td>21.58</td>
    <td>25.21</td>
    <td>63.81</td>
    <td>42.60</td>
    <td>21.21</td>
    <td>28.36</td>
  </tr>
  <tr>
    <td>Baichuan2-13B</td>
    <td>69.13</td>
    <td>46.98</td>
    <td>22.15</td>
    <td>33.45</td>
    <td>64.62</td>
    <td>38.82</td>
    <td>25.80</td>
    <td>26.07</td>
    <td>66.88</td>
    <td>42.90</td>
    <td>23.97</td>
    <td>29.76</td>
  </tr>
  <tr>
    <td>DeepSeek-/B</td>
    <td>68.13</td>
    <td>47.96</td>
    <td>20.17</td>
    <td>33.30</td>
    <td>63.38</td>
    <td>40.39</td>
    <td>22.99</td>
    <td>26.70</td>
    <td>65.76</td>
    <td>44.18</td>
    <td>21.58</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>DeepSeek-6/B</td>
    <td><b>71.50</b></td>
    <td>49.21</td>
    <td>22.29</td>
    <td>35.89</td>
    <td>71.37</td>
    <td>40.63</td>
    <td>30.75</td>
    <td>29.71</td>
    <td><b>71.44</b></td>
    <td>44.92</td>
    <td>26.52</td>
    <td>32.80</td>
  </tr>
  <tr>
    <td>InferM.LM2.5-1.8B</td>
    <td>61.62</td>
    <td>42.07</td>
    <td>19.55</td>
    <td>26.99</td>
    <td>55.37</td>
    <td>38.46</td>
    <td>16.91</td>
    <td>22.61</td>
    <td>58.50</td>
    <td>40.27</td>
    <td>18.23</td>
    <td>24.80</td>
  </tr>
  <tr>
    <td>InferM.LM2.5-7B</td>
    <td>67.25</td>
    <td>49.77</td>
    <td>17.48</td>
    <td>34.57</td>
    <td>69.50</td>
    <td>40.89</td>
    <td>28.61</td>
    <td>29.75</td>
    <td>68.38</td>
    <td>45.33</td>
    <td>23.04</td>
    <td>32.16</td>
  </tr>
  <tr>
    <td>InferM.LM2.5-20B</td>
    <td>67.37</td>
    <td>48.08</td>
    <td>19.29</td>
    <td>32.31</td>
    <td><b>73.62</b></td>
    <td>41.11</td>
    <td>32.51</td>
    <td>31.23</td>
    <td>70.50</td>
    <td>44.60</td>
    <td>25.90</td>
    <td>32.22</td>
  </tr>
  <tr>
    <td>YiLL-6B*</td>
    <td>67.00</td>
    <td>49.59</td>
    <td>17.41</td>
    <td>34.27</td>
    <td>64.38</td>
    <td>39.37</td>
    <td>25.01</td>
    <td>26.62</td>
    <td>65.69</td>
    <td>44.48</td>
    <td>21.21</td>
    <td>30.45</td>
  </tr>
  <tr>
    <td>YiLL-9B*</td>
    <td>68.50</td>
    <td>50.18</td>
    <td>18.32</td>
    <td>35.55</td>
    <td>66.37</td>
    <td>39.58</td>
    <td>26.79</td>
    <td>27.48</td>
    <td>67.44</td>
    <td>44.88</td>
    <td>22.56</td>
    <td>31.52</td>
  </tr>
  <tr>
    <td>YiLL-34B</td>
    <td>71.00</td>
    <td>52.23</td>
    <td>18.77</td>
    <td>38.09</td>
    <td>71.00</td>
    <td>40.75</td>
    <td>30.25</td>
    <td>29.91</td>
    <td>71.00</td>
    <td>46.49</td>
    <td>24.51</td>
    <td>34.00</td>
  </tr>
</table>

Table 3: Results of existing Chinese LLMs on HellaSwag-Pro using Direct prompt. “AVG” indicates the average performance on Chinese and English parts of the dataset. The best results in each model category are bolded.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Chinese</th>
    <th colspan="4">English</th>
    <th colspan="4">AVG</th>
  </tr>
  <tr>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
    <th>OA(%)</th>
    <th>ARA(%)</th>
    <th>RLA(%)</th>
    <th>CRA(%)</th>
  </tr>
  <tr>
    <th colspan="14">Closed-source LLMs</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>91.37</td>
    <td><b>81.97</b></td>
    <td>9.40</td>
    <td>75.55</td>
    <td>88.63</td>
    <td>70.17</td>
    <td>18.46</td>
    <td>63.06</td>
    <td>90.00</td>
    <td><b>76.07</b></td>
    <td>13.93</td>
    <td><b>69.31</b></td>
  </tr>
  <tr>
    <td>Claude-3.5</td>
    <td><b>95.37</b></td>
    <td>80.15</td>
    <td>15.22</td>
    <td>75.04</td>
    <td>85.11</td>
    <td>66.02</td>
    <td>19.08</td>
    <td>57.20</td>
    <td><b>90.24</b></td>
    <td>73.09</td>
    <td>17.15</td>
    <td>66.12</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>90.62</td>
    <td>78.36</td>
    <td>12.26</td>
    <td>70.48</td>
    <td>87.75</td>
    <td>60.74</td>
    <td>27.01</td>
    <td>58.27</td>
    <td>89.19</td>
    <td>69.55</td>
    <td>19.63</td>
    <td>64.38</td>
  </tr>
  <tr>
    <th colspan="14">Open-source LLMs</th>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>59.13</td>
    <td>46.62</td>
    <td>12.51</td>
    <td>28.23</td>
    <td>66.25</td>
    <td>40.21</td>
    <td>26.04</td>
    <td>27.34</td>
    <td>62.69</td>
    <td>43.42</td>
    <td>19.27</td>
    <td>27.79</td>
  </tr>
  <tr>
    <td>Llama3-70B</td>
    <td>65.75</td>
    <td>48.63</td>
    <td>17.12</td>
    <td>32.70</td>
    <td><b>72.50</b></td>
    <td>41.27</td>
    <td>31.23</td>
    <td><b>30.63</b></td>
    <td><b>69.13</b></td>
    <td>44.95</td>
    <td>24.18</td>
    <td>31.67</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1</td>
    <td>57.75</td>
    <td>46.25</td>
    <td><b>11.50</b></td>
    <td>27.57</td>
    <td>67.50</td>
    <td><b>41.52</b></td>
    <td>25.98</td>
    <td>28.93</td>
    <td>62.63</td>
    <td>43.88</td>
    <td>18.74</td>
    <td>28.25</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1</td>
    <td>63.62</td>
    <td>46.80</td>
    <td>16.82</td>
    <td>30.82</td>
    <td>69.75</td>
    <td>41.21</td>
    <td>28.54</td>
    <td>29.39</td>
    <td>66.69</td>
    <td>44.01</td>
    <td>22.68</td>
    <td>30.11</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1</td>
    <td>66.00</td>
    <td><b>50.73</b></td>
    <td>15.27</td>
    <td><b>34.32</b></td>
    <td>72.12</td>
    <td>41.25</td>
    <td>30.87</td>
    <td>30.61</td>
    <td>69.06</td>
    <td><b>45.99</b></td>
    <td>23.07</td>
    <td><b>32.47</b></td>
  </tr>
  <tr>
    <td>Gemma2-2B</td>
    <td>61.88</td>
    <td>43.38</td>
    <td>16.51</td>
    <td>29.02</td>
    <td>59.62</td>
    <td>39.13</td>
    <td><b>20.50</b></td>
    <td>24.88</td>
    <td>60.75</td>
    <td>42.25</td>
    <td><b>18.50</b></td>
    <td>26.95</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td><b>69.13</b></td>
    <td>46.75</td>
    <td>22.38</td>
    <td>33.29</td>
    <td>64.88</td>
    <td>39.80</td>
    <td>25.08</td>
    <td>26.91</td>
    <td>67.01</td>
    <td>43.28</td>
    <td>23.73</td>
    <td>30.10</td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>63.38</td>
    <td>48.52</td>
    <td>14.86</td>
    <td>31.96</td>
    <td>71.88</td>
    <td>40.91</td>
    <td>30.97</td>
    <td>30.25</td>
    <td>67.63</td>
    <td>44.71</td>
    <td>22.92</td>
    <td>31.11</td>
  </tr>
</table>

Table 4: Results of existing English LLMs on HellaSwag-Pro using Direct prompt (Same settings as Table 3).

on original questions.

\[
OA = \frac{\sum_{(x,y) \in \mathcal{D}} \mathbb{I}[\text{LM}(x), y]}{|\mathcal{D}|}.
\] (1)

Average Robust Accuracy (ARA) measures the average accuracy across all variants.

\[
ARA = \frac{\sum_{(x',y') \in \mathcal{D}_r} \mathbb{I}[\text{LM}(x'), y']}{|\mathcal{D}_r|}.
\] (2)

Robust Loss Accuracy (RLA) refers to the performance gap between all variants and original questions, i.e., the difference between OA and ARA.

\[
RLA = OA - ARA.
\] (3)

Consistent Robust Accuracy (CRA) refers to the joint accuracy of LLM correctly answering the variant and its original question, reflecting the LLM’s genuine understanding of the knowledge.

\[
CRA = \frac{\sum_{(x',y') \in \mathcal{D}_r} \mathbb{I}[\text{LM}(x), y] \cdot \mathbb{I}[\text{LM}(x'), y']}{|\mathcal{D}_r|}.
\] (4)

4.2 LLM Performance (RQ1)

Overall Performance The results for Direct prompting on all LLMs are listed in Table 3 and Table 4. The main observations are as follows.

Firstly, all evaluated LLMs perform well in OA (e.g., in AVG OA, GPT-4o scores 90.00, and Claude-3.5 scores 90.24). However, all LLMs show a performance drop on variants, as evidenced by a positive AVG RLA value for all LLMs. In contrast, human receive a near-zero RLA value, suggesting that the question variants are not more challenging than the originals for human. This disparity further illustrates that current LLMs lack a true understanding of the commonsense knowledge and can easily be affected by the reasoning form.

Secondly, comparing open-source and closed-source LLMs, closed-source models achieve larger OA, ARA and CRA scores and smaller average RLA scores than open-source LLMs, indicating

2The results of instruct and chat models of Qwen2.5, LLaMA3 and Mixtral_v0.1 series are shown in Appendix F.
Figure 4: Pairwise performance statistics of the original question and its variant. We use “HellaSwag ✓ HellaSwag-Pro ✗” to denote that the LLM correctly answers the original question but fails on its variant.

better robustness in commonsense reasoning.

Finally, when we compare models within the same series (e.g., Qwen2.5, Llama3), we observe that larger models often achieve higher scores on OA, ARA, and CRA. However, their RLA shows no consistent relationship with model size. Across different families, AVG RLA patterns vary - fluctuating with size in Qwen2.5 and Gemma3, while increasing with size in Yi1.5 and Llama3. This indicates that larger model size does not guarantee better robustness.

Analysis on Reasoning Robustness To further analyze whether LLMs can maintain reasoning ability from the original question to its variant, Figure 4 presents the pairwise performance statistic of the original question and its variant. For all LLMs, a significant proportion of variants are answered incorrectly despite LLMs being able to solve the source example. More specifically, closed-source LLMs like GPT-4o and Qwen-Max achieve a 69% success rate on both HellaSwag and HellaSwag-Pro, with only 3% failing both. In contrast, open-source LLMs struggle with around 30% and 20%, respectively. This shows that closed-source LLMs achieve better alignment between the performance of the original question and its variant, thus better robustness in reasoning ability.

4.3 Variant Analysis (RQ2)

To further analyze the robustness on different variants, we assess the contribution of each variant to the RLA score, as shown in Figure 5. A higher contribution indicates more non-robust in that type. The key observations are as follows:

Problem restatement, causal inference, and sentence ordering are the least challenging. Almost all

LLMs perform well on these variants particularly closed-source LLMs and Qwen2.5 series, indicating that LLMs can effectively handle these forms.

Reverse conversion and critical testing each contribute about 10% to the RLA score. This indicates that current LLMs struggle to fully generalize to these variants, possibly because these variants do not largely exist in the training data.

Negative transformation and scenario refinement are the two most difficult variants, with negative transformation being particularly challenging. For almost all LLMs, these two variants account for more than 50% of the RLA score. This might be due to statistical bias in these two types of data during pre-training and the exploitation of shortcuts in the corpus (Chen et al., 2023; Wu et al., 2024).

Figure 5: Each variant’s contribution to the RLA score.

4.4 Different Prompting Strategies (RQ3)

To explore the impact of various prompting strategies on our benchmark, we test the performance of all LLMs under different prompting strategies (cf. Section 4.1). The results are summarized in Table 5.

For both Chinese and English datasets, Chinese LLMs perform best under CN-CoT strategy with shots, followed closely by EN-CoT with shots, achieving overall scores of 67.36% and 67.03%, respectively. Conversely, English LLMs show optimal performance using EN-CoT approach with shots, attaining 67.55% on the Chinese dataset and 60.36% on the English one. This shows that different LLMs favor the prompts in their native language. Besides, translating datasets into LLMs’ native languages before reasoning does not enhance performance (e.g., 28.69% for EN LLMs using EN-XLT with shots vs 41.69% for EN LLMs using Direct). This phenomenon is further illustrated in Figure 6.
Figure 6: Performance on different 3-shot prompts. For the legend, the first two parts are the prompt name, and the third part is the dataset language. NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th rowspan="2">Prompt Language</th>
    <th rowspan="2">#shot</th>
    <th colspan="3">LLM</th>
  </tr>
  <tr>
    <th>CN</th>
    <th>EN</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td colspan="6"><i>Chinese HellaSwag-Pro</i></td>
  </tr>
  <tr>
    <td>Direct</td>
    <td>-</td>
    <td>0</td>
    <td>48.95</td>
    <td>41.16</td>
    <td>45.06</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>3</td>
    <td><b>71.04</b></td>
    <td>51.90</td>
    <td>61.47</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>3</td>
    <td>70.95</td>
    <td><b>67.55</b></td>
    <td><b>69.25</b></td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>3</td>
    <td>41.48</td>
    <td>28.69</td>
    <td>35.09</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>0</td>
    <td>44.82</td>
    <td>23.89</td>
    <td>34.36</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>0</td>
    <td>45.38</td>
    <td>31.39</td>
    <td>38.39</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>0</td>
    <td>28.57</td>
    <td>12.93</td>
    <td>20.75</td>
  </tr>
  <tr>
    <td colspan="6"><i>English HellaSwag-Pro</i></td>
  </tr>
  <tr>
    <td>Direct</td>
    <td>-</td>
    <td>0</td>
    <td>47.46</td>
    <td>40.66</td>
    <td>44.06</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>3</td>
    <td><b>63.67</b></td>
    <td>47.24</td>
    <td>55.46</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>3</td>
    <td>63.12</td>
    <td><b>60.36</b></td>
    <td><b>61.74</b></td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>CN</td>
    <td>3</td>
    <td>48.77</td>
    <td>16.61</td>
    <td>32.69</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>0</td>
    <td>34.89</td>
    <td>18.25</td>
    <td>26.57</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>0</td>
    <td>42.41</td>
    <td>31.03</td>
    <td>36.72</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>0</td>
    <td>16.36</td>
    <td>11.22</td>
    <td>13.79</td>
  </tr>
  <tr>
    <td colspan="6"><i>HellaSwag-Pro</i></td>
  </tr>
  <tr>
    <td>Direct</td>
    <td>-</td>
    <td>0</td>
    <td>48.21</td>
    <td>40.91</td>
    <td>44.83</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>3</td>
    <td><b>67.36</b></td>
    <td>49.57</td>
    <td><b>58.46</b></td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>3</td>
    <td>67.03</td>
    <td><b>63.95</b></td>
    <td><b>65.49</b></td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>CN</td>
    <td>3</td>
    <td>59.91</td>
    <td>34.26</td>
    <td>47.08</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>3</td>
    <td>52.30</td>
    <td>44.52</td>
    <td>48.41</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>CN</td>
    <td>0</td>
    <td>39.86</td>
    <td>21.07</td>
    <td>30.46</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>EN</td>
    <td>0</td>
    <td>43.90</td>
    <td>31.21</td>
    <td>37.55</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>CN</td>
    <td>0</td>
    <td>30.59</td>
    <td>17.55</td>
    <td>24.07</td>
  </tr>
  <tr>
    <td>XLT</td>
    <td>EN</td>
    <td>0</td>
    <td>35.49</td>
    <td>21.98</td>
    <td>28.74</td>
  </tr>
</table>

Table 5: Average ARA of all open-source LLMs on different prompting strategies. CN-LLMs contains 17 LLMs, and EN-LLMs contains 7 LLMs. The best results for each dataset are bolded. Detailed results for all evaluated models are provided in the Appendix F.

Our findings differ from previous research (Huang et al., 2023; Shi et al., 2022), which suggested that translating non-English tasks into English (XLT) would perform better than using native languages. And these research only focused on English LLMs while overlooking Chinese LLMs. We find that LLMs perform better when reasoning directly in their native language compared to XLT, addressing this gap in previous research.

5 Related Work

Commonsense Reasoning Evaluation There are numerous benchmarks and datasets for commonsense reasoning, most of which are in English. Some studies focus on evaluating general commonsense knowledge (Zellers et al., 2019; Talmor et al., 2019; Mihaylov et al., 2018b). Others target specific aspects of commonsense reasoning (Zhou et al., 2019; Bisk et al., 2020; Sap et al., 2019; Lin et al., 2020; Clark et al., 2018; Khot et al., 2020; Li et al., 2024). There are some Chinese datasets for commonsense reasoning (Sun et al., 2024; Shi et al., 2024). For instance, CHARM (Sun et al., 2024) distinguishes between global commonsense and Chinese-specific commonsense but includes only a limited number of everyday commonsense cases. However, evaluations aimed at assessing the robustness of commonsense reasoning are still understudied.

Datasets on Different Reasoning Forms There are several datasets relevant to our variant design. For reverse reasoning, ART (Bhagavatula et al., 2020), δ-NLI (Rudinger et al., 2020), and CLUTRR (Sinha et al., 2019) explore different reasoning directions. FCR (Yang et al., 2022) and NatQuest (Ceraolo et al., 2024) evaluate causal reasoning, while TimeTravel (Qin et al., 2019) focuses on counterfactual scenario refinement. Additionally, PoE (Balepur et al., 2024) assesses reasoning involving negation. However, not all these datasets focus on commonsense reasoning, nor are they structured by original questions and their variants. Furthermore, they typically target limited reasoning types. Lastly, our dataset is large-scale and covers diverse commonsense knowledge.
Robustness and Consistency in LLMs  Early work focuses on adversarial attacks, with developing evaluation methods for reading comprehension systems (Jia and Liang, 2017), followed by universal adversarial triggers (Wallace et al., 2019). The field then expands to examine spurious correlations, with revealing how models often exploit superficial patterns rather than engaging in genuine reasoning (Branco et al., 2021; Geirhos et al., 2020). And Ross et al., 2022 investigates whether self-explanation can mitigate these spurious correlations. Coherence and consistency evaluation advances through classifier assessment methods (Storks and Chai, 2021) and analysis of accuracy-consistency trade-offs (Johnson and Marasovic, 2023). While these studies primarily address model robustness against adversarial attacks or spurious correlations, our work takes a novel approach by examining robustness in reasoning forms.

6 Conclusion

We conduct a systematic evaluation of the robustness of LLMs in commonsense reasoning in both Chinese and English. To facilitate this evaluation process, we introduce two large-scale, finely-annotated datasets: HellaSwag-Pro and Chinese HellaSwag. In addition, we design various prompts to evaluate 41 LLMs, offering several key findings that may advance the field of commonsense reasoning. We believe this work will serve as a valuable resource to support further research into the commonsense reasoning of LLMs.

Acknowledgements

This research was also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.

Limitations

The limitations of our work are as follows:

• Our work only addresses everyday commonsense reasoning and does not encompass specific types, such as temporal or physical commonsense knowledge. Evaluating the robustness of LLMs on these specific types of commonsense reasoning tasks will be our future work.

• HellaSwag-Pro is concentrated on assessing the robustness of LLMs in commonsense reasoning tasks and does not investigate the underlying reasons for observed performance declines.

• For the sake of evaluation convenience, our setup utilizes multiple-choice questions. We plan to study the open-ended questions in future work.

Ethics Statement

This work requires manual annotation. We provide annotators with a salary above the local minimum hourly wage. We have also clearly informed them about the purpose of the data and the necessity to ensure that all the data in Hellaswag-Pro does not contain any social biases, ethical concerns, or privacy issues.

Additionally, we develop a challenging dataset for evaluating the robustness of commonsense reasoning in this work. It’s important to emphasize that this dataset is intended solely for evaluation, not for training or fine-tuning purposes. We recognize that improper use of this dataset for model training or fine-tuning could lead to persistent inconsistencies in LLMs’ understanding of commonsense knowledge, potentially creating a vicious cycle where more such datasets would be needed to address these issues. Therefore, we explicitly state that the intended use of this dataset is strictly limited to evaluation to prevent the formation of long-standing issues in LLMs. We look forward to promoting healthy development in LLM research through responsible use of these research findings.

References

Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalmian, Jakub Sygnowski, and et al. 2023. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.11805.

Anthropic. 2024. Introducing the next generation of claude. https://www.anthropic.com/news/claude-3-family.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei
Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingtian Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Nishant Balepur, Shramay Palta, and Rachel Rudinger. 2024. It’s not easy being wrong: Large language models struggle with process of elimination reasoning. In Findings of the Association for Computational Linguistics ACL 2024, pages 10143–10166.

Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazou Gao, Kaige Gao, Wenjun Gao, Ruqi Ge, Kang Guan, Dayu Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Alex X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. 2024. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401.02954.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Ruben Branco, António Branco, Joao Rodrigues, and Joao Silva. 2021. Shortcutted commonsense: Data spuriousness in deep learning of commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1504–1521.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970.

Erik Cambria, Yangqiu Song, Haixun Wang, and Amir Hussain. 2011. Isanette: A common and common sense knowledge base for opinion mining. In 2011 IEEE 11th International Conference on Data Mining Workshops, pages 315–322. IEEE.

Roberto Cerao, Dmitrii Kharlapenko, Ahmad Khan, Amélie Reynold, Rada Mihalcea, Bernhard Schölkopf, Minmaya Sachan, and Zhijing Jin. 2024. Analyzing human questioning behavior and causal curiosity through natural queries.

Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023. Say what you mean! large language models speak too positively about negative commonsense knowledge. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9890–9908, Toronto, Canada. Association for Computational Linguistics.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Ernest Davis. 2023. Benchmarks for automated commonsense reasoning: A survey. ACM Computing Surveys, 56(4):1–41.

Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9):92–103.

Abhimanyu Dubey, Abhinav Jahuiri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravan Kumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes,
Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vrnes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstone, Joshua Saxe, Junting Jia, Kalyan Vasudean Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673.

Pei Guo, Wangjie You, Juntao Li, Yan Bowen, and Min Zhang. 2024. Exploring reversal mathematical reasoning ability for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 13671–13685.

Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365–12394, Singapore. Association for Computational Linguistics.

Mete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva, and Antoine Bosselut. 2023. CRoW: Benchmarking commonsense reasoning in real-world tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9785–9821, Singapore. Association for Computational Linguistics.

Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. CoRR, abs/2310.06825.

Jacob K Johnson and Ana Marasovic. 2023. How much consistency is your accuracy worth? EMNLP 2023, page 250.

Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090.

Mahnaz Koupaei and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. arXiv preprint arXiv:1810.09305.

David R Krathwohl. 1973. Taxonomy of educational objectives. Affective domain.

Xiaoyuan Li, Keqin Bao, Yubo Ma, Moxin Li, Wenjie Wang, Rui Men, Yichang Zhang, Fuli Feng, Dayiheng Liu, and Junyang Lin. 2025. Mtr-bench: A comprehensive benchmark for multi-turn reasoning evaluation. arXiv preprint arXiv:2505.17123.

Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. 2024. Evaluating mathematical reasoning of large language models: A focus on error identification and correction. In Findings of the Association for Computational Linguistics ACL 2024, pages 11316–11360.

Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868, Online. Association for Computational Linguistics.

Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211–226.

Kaijing Ma, Xeron Du, Yunran Wang, Haoran Zhang, ZhoufutuWen, Xingwei Qu, Jian Yang, Jiaheng Liu, minghao liu, Xiang Yue, Wenhao Huang, and Ge Zhang. 2025. KOR-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks. In The Thirteenth International Conference on Learning Representations.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018a. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391.
Todor Mihaev, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018b. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

Josh Achiam OpenAI, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2024. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774.

Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5042–5052. Association for Computational Linguistics.

Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Maanul Faruqui. 2021. Time-dial: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7066–7076.

Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, et al. 2024. Gemma 2: Improving open language models at a practical size. CoRR.

Alexis Ross, Matthew E Peters, and Ana Marasović. 2022. Does self-rationalization improve robustness to spurious correlations? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7403–7416.

Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4661–4675. Association for Computational Linguistics.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.

Dan Shi, Chaobin You, Jiantao Huang, Taihao Li, and Deyi Xiong. 2024. Corecode: A common sense annotated dialogue dataset with benchmark tasks for chinese large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18952–18960.

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057.

Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4505–4514. Association for Computational Linguistics.

Shane Storks and Joyce Chai. 2021. Beyond the tip of the iceberg: Assessing coherence of text classifiers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3169–3177.

Jiaxing Sun, Weiquan Huang, Jiang Wu, Chenya Gu, Wei Li, Songyang Zhang, Hang Yan, and Conghui He. 2024. Benchmarking Chinese commonsense reasoning of LLMs: From Chinese-specifics to reasoning-memorization correlations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11205–11228, Bangkok, Thailand. Association for Computational Linguistics.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM-techreport.

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153–2162.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoong Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1819–1862, Mexico City, Mexico. Association for Computational Linguistics.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Hazeo Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xianrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yufeng Zhang, Zenan Zhou, and Zhiying Wu. 2023. Baichuan 2: Open large-scale language models. CoRR, abs/2309.10305.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengcheng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, and Yue Zhang. 2022. Towards fine-grained causal reasoning and QA. CoRR, abs/2204.07408.

Alex Young, Bei Chen, Chao Li, Chenge Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. Yi: Open foundation models by 01.ai. CoRR, abs/2403.04652.

Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Jankowski, Yanghua Xiao, and Deqing Yang. 2023. Distilling script knowledge from large language models for constrained language planning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4303–4325.

Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.

Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363–3369.

Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, and Xiang Ren. 2021. Rica: Evaluating robust inference capabilities based on commonsense axioms. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7560–7579.

A Bloom Cognitive Model

Bloom Cognitive Model (Krathwohl, 1973) is an educational theoretical framework that outlines six hierarchical levels of cognitive processes, ranging from lower-order to higher-order thinking skills in the learning process as follows:

• Remember: The capacity to recall, identify, and reproduce information.

• Understand: The ability to interpret, summarize, and make sense of information.

• Apply: The skill to use learned knowledge in new contexts.

• Analyze: The capability to deconstruct information and examine relationships between components.

• Evaluate: The proficiency in making informed judgments based on specific criteria, involving critical thinking.

• Create: The ability to synthesize elements into novel patterns or generate original work.

Motivated by this framework, we aim to develop the model that goes beyond merely memorizing surface patterns and demonstrates higher-order capabilities. To test whether the model truly understands commonsense knowledge, we create seven variants of each question. It is our view that if the model genuinely understands commonsense knowledge, it should be able to correctly respond to the same knowledge expressed in different reasoning forms. Here’s how our seven variants map onto these cognitive levels:

• Understanding is demonstrated through Problem Restatement and Causal Inference.
• Application skills are tested via Reverse Conversion, Scenario Refinement, and Negative Transformation.

• Analysis capabilities are assessed through Sentence Ordering.

• Evaluation competency is measured by Critical Testing.

B Human Annotation

B.1 Annotator Qualification and Compensation

We maintained strict control over annotator qualification, data quality, and annotation procedure. Specifically, we recruited 34 professional annotators specializing in NLP tasks totally. All annotators hold at least a bachelor’s degree, have passed the College English Test Level 4 of China, and possess extensive annotation experience of NLP tasks. We compensated them at a rate of 23 RMB per hour (significantly higher than the average hourly wage in China), with an average payment of 1.98 RMB per question. We promptly addressed any concerns during the annotation process and allowed sufficient time for each question to prevent unnecessary pressure on annotators.

B.2 Data Quality and Consistency

31 out of 34 annotators were involved in data filtering. We enforced the strict annotation guidelines. For Chinese HellaSwag construction, in Stage 1 (initial dataset generation), annotators labeled 12,960 entries in total and filtered down to 12,287 entries. The authors randomly sampled 100 filtered entries and verified them against annotation guidelines, achieving a 98% compliance rate. In Stage 2 (difficult sample replacement), annotators labeled 5,209 entries in total and filtered down to 2,451 entries. A similar 100-question sample check by authors showed a 96% compliance rate. For HellaSwag-Pro construction, annotators labeled 24,260 entries, filtering down to 11,200. The authors randomly checked 100 question variants against variant annotation guidelines, achieving a 95% compliance rate. These measures ensured high quality of our dataset.

B.3 Human Performance

To evaluate human performance, we sampled a subset of 400 questions by randomly selecting 25 original questions in both Chinese and English, along with their variants. Three additional crowd workers, who were not involved in the original annotation process, were tested on this subset. We calculated their average accuracy as human performance.

B.4 Detailed Annotation Guidelines

We provided rich examples for the annotation tasks to ensure annotators understood the tasks at hand. We maintained close contact with the annotators to clarify any misunderstandings in time. Our annotation tasks were divided into four parts:

B.4.1 Chinese HellaSwag Annotation for Stage One

Annotators were given the context, six choices filtered by the model, label, broad type, and detailed type. They scored based on three dimensions: the possibility to select 4 out of 6 choices, and whether they conform to the two category definitions. The annotation requirements for annotators were as follows:

• Possibility to select 4 out of 6: Using the model’s scoring of the 6 choices as a reference, determine if it’s possible to select 4 choices, with only one correct answer and the other three being as confusing as possible (i.e., conforming to commonsense but not suitable for the context, or judged by how much modification is needed to make them correct - the less modification needed, the more confusing). Ensure the uniqueness of the answer and avoid controversy. Score 1 if possible, and note the corresponding option numbers, with the first being the correct option and the next three being incorrect options. If not possible, score 0 and select the appropriate reason: A. No correct option or B. Unable to select 3 incorrect options, e.g., more than 4 correct options.

• Broad type: Score it conforms to the definition, otherwise 0.

• Detailed type: Score 1 if it conforms to the definition, otherwise 0.

The following are the definitions for broad and detailed types.

• Family

Household chores: Labor activities to maintain a clean and tidy home environment, including but not limited to cleaning, laundry, and preparing traditional Chinese cuisine.
Personal hygiene: Daily personal cleaning habits such as bathing, brushing teeth, and maintaining good living habits to ensure physical health.

Family entertainment: Leisure activities shared by family members, such as playing family games, pet care, watching TV shows, or reading books together.

Holiday celebrations: Celebrating family members’ birthdays, traditional festivals, or special occasions like wedding anniversaries.

Family affairs: Daily life management, emotional communication, and responsibility allocation among family members, including household shopping, financial management, and handling potential disagreements or conflicts.

Family transitions: Changes in family structure or living environment, such as home renovation, moving, marriage, or welcoming a newborn.

Emergency handling: Measures for potential emergencies like fires or natural disasters.

• Education

School education: Formal education received in school settings, including classroom learning, extracurricular activities, and exam preparation.
Family education: Education provided by parents or other family members, including homework assistance, shared reading, and cultivation of interests and moral qualities.

Online learning: Learning through internet resources, including self-study tools, remote tutoring, and interactive learning platforms.

Community education: Educational activities within the community, such as lectures, interest groups, and practical activities.

Vocational training: Professional training aimed at improving occupational skills, including obtaining professional qualifications and on-the-job continuing education.

Lifelong learning: Continuous learning activities for adults to improve themselves, such as adult education or senior university courses.

International exchange: Consultation for studying abroad, language skill improvement, and other forms of cross-cultural exchange.

• Work

Work Meetings: Various meetings held in the workplace, including team meetings, departmental reports, and project evaluations.

Project Management: The entire process of managing a project from initiation to completion, including strategy formulation, progress tracking, and problem-solving.

Customer Service: Services provided to meet customer needs, including customer inquiries, complaint handling, sales negotiations, and after-sales support.

Teamwork: Effective collaborative work patterns within a team, including team building, task allocation, conflict resolution, and incentive measures.

Personal Development: The process of individual career growth, covering skill learning, career planning, financial management, and maintaining mental and physical health.

Administrative Management: Daily management activities within a company, including attendance records, performance evaluations, travel expense reimbursements, employee benefits distribution, and company policy communication.

Technological Innovation: Activities driving technological advancement in a company, including new product development, technology application, technical training, and technology exchange.

• Sociality

Daily Interactions: Everyday social interactions with family, friends, and neighbors.

School Interactions: Communication between students, between teachers and students, and between parents and teachers.

Workplace Interactions: Interactions with colleagues, superiors, or subordinates in the workplace, as well as formal business dinners.

Community Interactions: Participating in community-organized activities or providing volunteer services.

Public Space Interactions: Interactions with others in public spaces such as public transportation, shopping malls, restaurants, and lecture halls.

Online Social Networking: Social activities using online platforms, including social media, online gaming, internet forums, and video live streaming.
Special Occasion Interactions: Interpersonal interactions at weddings, funerals, award ceremonies, and other celebratory events.

• Shopping

In-store Shopping: Shopping activities in physical retail stores, such as supermarkets, department stores, and specialty shops.

Online Shopping: Online purchasing behavior through e-commerce platforms, live streaming sales, or social commerce.

Food and Dining Purchases: Buying food products, including dining out, ordering takeout, and home cooking.

Service Purchases: Buying various service products, such as travel services, beauty and fitness, and educational training.

Overseas Shopping: Purchasing foreign goods through cross-border e-commerce or personal shopping agents.

Special Occasion Shopping: Shopping in specific situations, such as promotional events, group buying, auctions, and second-hand transactions.

Returns and After-sales Service: Consumer behavior in seeking refunds, exchanges, and after-sales service when issues arise with products.

• Transportation

Public Transportation: Using public transit systems, such as buses and subways.

Private Transportation: Using private vehicles, bicycles, etc., for travel.

Long-distance Travel: Travel methods covering longer distances, such as trains, planes, or long-distance buses.

Emergency Travel: Choosing emergency transportation in response to sudden situations, such as travel during severe weather conditions.

Tourist Transportation: Using sightseeing vehicles, boats, or cable cars for tourism purposes.

International Travel: Visa applications, international flight bookings, and entry procedures required for traveling abroad.

Special Occasion Transportation: Transportation services provided for specific situations, such as wedding cars or conference shuttles.

• Health

Preventive Healthcare: Measures taken to prevent diseases, including health check-ups, vaccinations, and health education.

Outpatient Care: Receiving non-hospitalized treatment at hospitals or clinics, including appointment scheduling, initial diagnosis, follow-up visits, and specialist consultations.

Inpatient Treatment: Hospital admission for treatment, including admission procedures, ward life, surgery arrangements, and discharge preparation.

Rehabilitation Care: Treatment during the recovery period, including rehabilitation training, long-term care, and psychological counseling.

Medication Management: Guidance on medication use and storage methods.

Health Insurance: Purchasing medical insurance products, claim procedures, and health consultation services.

Epidemic Prevention and Control: Measures such as epidemic monitoring, isolation observation, and health code management.

• Leisure

Outdoor Activities: Recreational activities in natural settings, such as hiking, picnicking, and gardening.

Cultural Experiences: Engaging in cultural activities like visiting museums, watching theatrical performances, or attending film screenings.

Travel Experiences: Domestic or international tourism activities.

Sporting Events: Watching or participating in sports competitions, including esports.

Artistic Pursuits: Engaging in artistic activities such as painting, calligraphy, playing musical instruments, or creating handicrafts.

Digital Entertainment: Leisure activities using digital devices, such as online gaming, live streaming interactions, or virtual reality experiences.

Recreational Fitness: Maintaining physical and mental health through activities like gym workouts or practicing yoga and meditation.

B.4.2 Chinese HellaSwag Annotation for Stage Two

In order to increase the number of difficult samples, the annotators were given a context and four
replaced options regenerated by models to judge whether the label of the question was correct and whether it had a unique correct option. If both are true, the replaced options were retained.

B.4.3 Hellaswag-Pro Annotation

Annotators are provided with the original context, original choices, original label, transformed context, transformed choices, transformed label, and perturbation type for annotation according to different variant definitions. The variant definitions are as follows:

• Problem restatement: Restate the original context and the original label corresponding to the original choices in a different way, ensuring the semantics remain unchanged. Other options of the original choices should remain unchanged without restatement. Pay special attention to ensuring that the connection between the context and the choice corresponding to the label is smooth.

• Reverse conversion: Combine the original choices corresponding to the original label with the original context into a complete passage. Then, make the last sentence of this passage the context, and transform the remaining sentences into the correct choice. A slight modification is allowed for smoothness. Also, generate five other incorrect options that do not fit the context, modeled on the format and length of the correct option. Place the correct option in the first position and label it as 0. To ensure the context is complete, append "Which is the possible context for this action?" This conversion process aims to infer the potential background through the results. The generated incorrect options should not include supernatural elements and should have a similar word count to the correct option.

• Causal inference: Combine the original choices corresponding to the original label with the original context to form a complete passage and turn it into the context. Then, generate the reason for such choices that contain commonsense as the correct option in the choices. The correct choice should be as concise as possible while generating five other evidently incorrect options modeled on the format and length of the correct choice. Put the correct choice in the first position and label it as 0. To ensure the context is complete, append "Which is the possible reason for this action?" This conversion process aims to infer the potential reason through the context and options.

• Negative transformation: Modify the original context to end with a negation word as the context, retaining one most unreasonable option and the original choice corresponding to the original label. Then, generate two other reasonable options as choices. Generated options should be similar in length and format to the original options. Place this most unreasonable option as the first element in the choices and label the index of this option in choices as 0. This conversion process aims to transform the original task into a negation prediction, containing one unreasonable option and three other reasonable options.

• Scenario refinement: First, select a relatively reasonable option from the incorrect options in the original choices, then modify the original context as the context to allow the selection of this option as the correct choice. The value of choices is equal to the original choices. The label value corresponds to the value of the selected incorrect option. This conversion process aims to refine the context, thereby altering the correct choice.

• Sentence ordering:
1) Sentence ordering - Short: First, combine the original choices corresponding to the original label with the original context into a complete sentence. Then, predict the development of subsequent events, continuing to write a few more sentences to form a paragraph. Pay attention to the sequence and completeness of continued sentences, ensuring the uniqueness of the answer. Then, disorder each sentence of this passage and number them. The correct option is the original order of the paragraph, and three other incorrect options are generated based on the correct option by disordering the numbers. Place the correct option in the first position and label it as 0. To ensure the context is complete, append "The correct order is." This conversion process aims to infer the correct order of sentences.

2) Sentence ordering - Long: Combine the original choices corresponding to the original label with the original context into a complete passage. Then, disorder each sentence of this passage and number them. The correct option is the original
order of the paragraph, and three other incorrect options are generated by disordering the numbers. Place the correct option in the first position and label it as 0. To ensure the context is complete, append "The correct order is." This conversion process aims to infer the correct order of sentences.

• Critical testing: Modify the original context so that none of the options can be chosen as the context, then add an option of 'None of the above four options are appropriate' to the original choices as choices. The label value corresponds to the index of 'None of the above four options are appropriate'. Note that the modified context should still present a question, ideally with an ending word identical to the original context. This conversion process aims to test the model’s critical thinking.

B.4.4 Hellaswag-Pro Human Evaluation
Annotators were provided with the context and choices from the Hellaswag-Pro and made selections. We then compared the selections made by annotators with the labels to calculate accuracy.

C Prompt Strategy
The prompting strategies we designed, including Direct, CN-CoT, EN-CoT, CN-XLT and EN-XLT, are as shown Figure 6, 7 and 8.

D Case Study
Figure 9 shows an example of the Chinese hellaswag generation process, from which we can see that our wrong options are becoming more and more challenging.

E Chinese HellaSwag Evaluation
We also evaluate the overall results of Chinese Hellaswag using both open-source and closed-source models, analyzing them from the perspectives of broad categories and length categories. As shown in Table 10, within all categories of Chinese Hellaswag, Traffic is the most challenging, with an average accuracy of only 58.56%, while the Education category is the easiest, achieving an average accuracy of 77.64%. Additionally, as the context length increases, the difficulty of the problems generally decreases, with average accuracy of long types at 72%, medium types at 70%, and short types at 64% as shown in Table 11. Overall, the closed-source models outperform open-source models. Among the closed-source models, Claude-3.5 performs the best, reaching an accuracy of 94%, whereas among open-source models, Qwen2.5-72B shows the highest performance, achieving 71%.

F Experiment Detailed Result
Figures 12 to 20 show the detailed results of the open-source models on the 9 prompt strategies and 7 variants we design.

In addition, we also use direct prompt to evaluate the performance of the Qwen, LLaMA, and Mixtral series of intruct models on our robustness task, as shown in Table 21 and Figure 7. We can observe that the intruct models have improved performance in robustness task compared to the base models.

![Comparasion between base and their instruct models.](page_1012_1042_377_246.png)

Figure 7: Comparasion between base and their instruct models.
Problem: Question: 周老师负责学校的英语培训工作，常为学生提供实用的语言学习建议。他建议多读原版英文报纸提高阅读水平。可能的原因是
Choices:
A.主张减少课外阅读时间增加听力训练。
B.阅读原版材料能够增强语感并扩大词汇量。
C.觉得看漫画书更能激发学生的兴趣。
D.相信通过电影比读书更容易学好语言。

Prompt Chinese Dataset
Direct Answer: B
CN- CoT 答案：让我们一步步思考。首先，我们需要分析周老师作为英语培训教师的建议-多读原版英文报纸，这建议的目的是为了提高学生的阅读水平。查看选项：A提到减少阅读增加听力，与题目表述矛盾；C提到漫画书和D提到电影都偏离了原版英文报纸这一重点；只有B选项提到阅读原版材料可以增强语感并扩大词汇量，这与读原版英文报纸的效果是一致的。原版英文报纸使用地道的英语表达，包含丰富的词汇和表达方式，长期阅读确实能帮助学习者提高语感，扩大词汇量，提升整体语言水平。答案是B。

EN-CoT Answer: Let’s think step by step. First, we know that Teacher Zhou is responsible for English training at school and often provides practical language learning advice to students. Second, he specifically recommends reading original English newspapers to improve reading skills. Third, reading original English materials has several benefits: it exposes students to authentic language usage, helps them understand how native speakers express ideas, enhances language intuition, and naturally expands their vocabulary through context. Fourth, looking at the options, option B directly aligns with these benefits by mentioning both improved language sense and vocabulary expansion. The other options either suggest different learning methods (like movies or comics) or propose reducing reading time, which contradicts the teacher’s recommendation. Since Teacher Zhou specifically advocates for reading original English newspapers, option B is the most logical explanation for his recommendation. The answer is B.

Question: The mother instructs them on how to brush their teeth while laughing. The boy helps his younger sister brush his teeth. She gets them some water to gargle in their mouths. Which could be the possible reason for this action? Choices:
A. She wants to ensure they rinse out any remaining toothpaste and clean their mouth thoroughly.
B. She is conducting an experiment to see if water tastes different after brushing.
C. She is using the water as a distraction so they won’t notice her leaving the room.
D. She is training them to be synchronized swimmers by coordinating their gargles.

Prompt English Dataset
Direct Answer: A
CN- CoT 答案：让我们一步步思考。这是一个关于刷牙后漱口的场景。母亲在指导孩子们刷牙，并让他们用水漱口。从日常生活经验来看，刷牙后漱口的主要目的是清除口腔中残留的牙膏和清洁口腔。漱口这个动作不太可能是为了做实验测试水的味道（选项B），也不是为了分散注意力（选项C），更不可能是在训练同步游泳（选项D）。选项A提到让孩子们漱口是为了冲洗掉剩余的牙膏并彻底清洁口腔，这符合我们日常刷牙后漱口的目的，也是最合理的解释。因此答案是A。

EN-CoT Answer: Let’s think step by step. Gargling with water after brushing teeth is a common and important part of proper dental hygiene. The mother in this scenario is teaching her children good oral care habits. When we brush our teeth, toothpaste helps clean them, but some toothpaste residue remains in our mouth. This residue, if not rinsed out, can leave a lingering taste and might contain fluoride that shouldn’t be swallowed in large amounts. By giving the children water to gargle, the mother is helping them complete their tooth-brushing routine correctly. The gargling action helps remove any loose debris, leftover toothpaste, and bacteria from the mouth, leading to better oral hygiene. Looking at the other options, they are either illogical or unrelated to dental care - testing water taste, using it as a distraction, or swimming practice are not reasonable explanations for gargling after brushing teeth. The most sensible and practical reason for the mother’s action is to ensure her children properly rinse their mouths after brushing. The answer is A.

Table 6: Examples of Direct, CN-CoT and EN-CoT Prompt.
Problem: Question: 周老师负责学校的英语培训工作，常为学生提供实用的语言学习建议。他建议多读原版英文报纸提高阅读水平。可能的原因是
Choices:
A.主张减少课外阅读时间增加听力训练。
B.阅读原版材料能够增强语感并扩大词汇量。
C.觉得看漫画书更能激发学生的兴趣。
D.相信通过电影比读书更容易学好语言。

Question: The mother instructs them on how to brush their teeth while laughing. The boy helps his younger sister brush his teeth. She gets them some water to gargle in their mouths. Which could be the possible reason for this action? Choices:
A. She wants to ensure they rinse out any remaining toothpaste and clean their mouth thoroughly.
B. She is conducting an experiment to see if water tastes different after brushing.
C. She is using the water as a distraction so they won’t notice her leaving the room.
D. She is training them to be synchronized swimmers by coordinating their gargles.

<table>
  <tr>
    <th>Prompt</th>
    <th>Chinese Dataset</th>
    <th>English Dataset</th>
  </tr>
  <tr>
    <td>CN-XLT</td>
    <td>-</td>
    <td>我希望你扮演一个英文常识推理专家。<br>你应该用中文复述问题和选项。<br>你应该一步一步地回答问题，选出正确的答案。<br>问题：母亲一边笑一边指导他们如何刷牙。男孩帮助他的小妹妹刷牙。她给他们一些水让他们漱口。对此行为，可能的原因是什么？<br>选项：<br>A. 她想确保他们漱掉任何剩余的牙膏并清洁嘴巴。<br>B. 她正在进行实验，以查看刷牙后水的味道是否不同。<br>C. 她用水作为分散注意力的方法，以便他们不会注意到她离开房间。<br>D. 她通过协调漱口来训练他们成为同步游泳运动员。<br>分步解答：<br>1. 母亲给他们水漱口，通常是为了卫生目的。<br>2. 漱口可以帮助去除残留的牙膏，确保口腔清洁。<br>3. 选项A是一个常见且合理的原因，符合情境。<br>4. 选项B、C和D不符合通常情况下的刷牙步骤或目的。<br>所以答案是A。</td>
  </tr>
</table>

Table 7: Examples of CN-XLT Prompt.
Problem: Question: 周老师负责学校的英语培训工作，常为学生提供实用的语言学习建议。他建议多读原版英文报纸提高阅读水平。可能的原因是
Choices:
A.主张减少课外阅读时间增加听力训练。
B.阅读原版材料能够增强语感并扩大词汇量。
C.觉得看漫画书更能激发学生的兴趣。
D.相信通过电影比读书更容易学好语言。

Question: The mother instructs them on how to brush their teeth while laughing. The boy helps his younger sister brush his teeth. She gets them some water to gargle in their mouths. Which could be the possible reason for this action? Choices:
A. She wants to ensure they rinse out any remaining toothpaste and clean their mouth thoroughly.
B. She is conducting an experiment to see if water tastes different after brushing.
C. She is using the water as a distraction so they won’t notice her leaving the room.
D. She is training them to be synchronized swimmers by coordinating their gargles.

<table>
  <tr>
    <th>Prompt</th>
    <th>Chinese Dataset</th>
    <th>English Dataset</th>
  </tr>
  <tr>
    <td>EN-XLT</td>
    <td>I want you to act as a commonsense reasoning expert for Chinese.<br>You should retell the question and choices in English.<br>You should do the answer step by step to choose the right answer.<br>You should tell me the answer in this format 'So the answer is'.<br>Question: Teacher Zhou is in charge of the English training at school and often provides students with practical language learning advice. He suggests reading original English newspapers to improve reading skills. The possible reason is:<br>Choices:<br>A. Advocates reducing extracurricular reading time to increase listening practice.<br>B. Reading original materials can enhance language sense and expand vocabulary.<br>C. Believes that reading comic books can better stimulate students’ interest.<br>D. Believes it is easier to learn a language well through movies than through reading.</td>
    <td>-</td>
  </tr>
</table>

Step-by-step answer:
1. The statement suggests reading original English newspapers as a way to improve reading skills.
2. Option B suggests that reading original materials can enhance language sense and expand vocabulary, which aligns with the goal of improving reading skills.
3. Option A talks about reducing reading time, which contradicts the suggestion to read newspapers.
4. Option C talks about comic books, which is not related to reading original newspapers.
5. Option D suggests movies are more effective, which again, does not align with the focus on reading newspapers.
So the answer is B.

Table 8: Examples of EN-XLT Prompt.
<table>
  <tr>
    <th>Step</th>
    <th>Generated or Filtered Content</th>
  </tr>
  <tr>
    <td><b>Step 1<br>Over-generate context with broad type and detailed type with Qwen-Max</b></td>
    <td>
      Context: 丽丽报名参加了日本京都的一趟文化之旅，深度体验了传统艺伎表演。她<br>
      Broad Type: 休闲娱乐<br>
      Detailed Type: 旅游体验<br>
      Context: Lili signed up for a cultural tour in Kyoto, Japan, and experienced a traditional geisha performance. She<br>
      Broad Type: Leisure<br>
      Detailed Type: Travel Experience<br>
      Choices:<br>
      1. 深深地被茶道的精致仪式所吸引。(Score:10)<br>
      2. 抱怨为什么所有的寺庙都没有钟声。(Score:3)<br>
      3. 抱怨旅馆里的榻榻米太过柔软。(Score:3)<br>
      4. 要求在京都找到寿司制作的机器人。(Score:2)<br>
      5. 质疑为什么艺伎穿的是拖鞋而不是高跟鞋。(Score:5)<br>
      6. 坚持要在神社祈祷时使用英文。(Score:2)<br>
      7. 对茶道助手参与和服穿着过程表现出浓厚的兴趣。(Score:9)<br>
      8. 询问为什么没人听说过她喜欢的动漫角色。(Score:4)<br>
      9. 希望能在旅途中看到更多现代化的建筑。(Score:3)<br>
      10. 感到遗憾没能能在旅途中参与一场本地的相扑比赛。(Score:4)
    </td>
  </tr>
  <tr>
    <td><b>Step 2<br>Over-generate candidate choices with Qwen-Max</b></td>
    <td>
      Choices:<br>
      1. was deeply attracted by the exquisite ritual of the tea ceremony. (Score:10)<br>
      2. complained why there were no bells in all temples. (Score:3)<br>
      3. complained that the tatami in the hotel was too soft. (Score:3)<br>
      4. requested to find a sushi-making robot in Kyoto. (Score:2)<br>
      5. questioned why geishas wore slippers instead of high heels. (Score:5)<br>
      6. insisted on using English when praying at the shrine. (Score:2)<br>
      7. showed great interest in participating in the process of wearing a kimono. (Score:9)<br>
      8. asked why no one has heard of her favorite anime character. (Score:4)<br>
      9. hoped to see more modern buildings during the trip. (Score:3)<br>
      10. regretted not being able to attend a local sumo match during the trip. (Score:4)
    </td>
  </tr>
  <tr>
    <td><b>Step 3-1<br>Find filtered choices with Qwen-Max</b></td>
    <td>
      Choices:<br>
      1. 深深地被茶道的精致仪式所吸引。(Score:10)<br>
      5. 质疑为什么艺伎穿的是拖鞋而不是高跟鞋。(Score:5)<br>
      8. 询问为什么没人听说过她喜欢的动漫角色。(Score:4)<br>
      10. 感到遗憾没能能在旅途中参与一场本地的相扑比赛。(Score:4)<br>
      2. 抱怨为什么所有的寺庙都没有钟声。(Score:3)<br>
      9. 希望能在旅途中看到更多现代化的建筑。(Score:3)
    </td>
  </tr>
  <tr>
    <td><b>Step 3-2<br>Find filtered choices with human annotators</b></td>
    <td>
      Choices:<br>
      1. was deeply attracted by the exquisite ritual of the tea ceremony. (Score:10)<br>
      5. questioned why geishas wore slippers instead of high heels. (Score:5)<br>
      8. asked why no one has heard of her favorite anime character. (Score:4)<br>
      9. hoped to see more modern buildings during the trip. (Score:3)
    </td>
  </tr>
  <tr>
    <td><b>Step 4<br>Replace easily-identifiable false choices with adversarial ones through human-in-the-loop alternating adversarial filtering</b></td>
    <td>
      Choices:<br>
      1. 深深地被茶道的精致仪式所吸引。<br>
      2. 学习了传统的日式剑道和弓道技巧。<br>
      3. 欣赏了京都著名的樱花季和红叶景观。<br>
      4. 品尝了正宗的关西风味章鱼烧和大阪烧。<br>
      Choices:<br>
      1. was deeply attracted by the exquisite ritual of the tea ceremony.<br>
      2. learned traditional Japanese kendo and archery techniques.<br>
      3. enjoyed Kyoto’s famous cherry blossom season and red leaves.<br>
      4. tasted authentic Kansai-style takoyaki and okonomiyaki.
    </td>
  </tr>
</table>

Table 9: An example of Chinese HellaSwag Gereration. Step 3-1 filters the top 5 wrong options with scores below 9 to prevent multiple correct options, and Step 3-2 select the most confusing wrong options by human annotators.
<table>
  <tr>
    <th>Model</th>
    <th>Education</th>
    <th>Health</th>
    <th>Famly</th>
    <th>Leisure</th>
    <th>Shopping</th>
    <th>Sociality</th>
    <th>Traffic</th>
    <th>Work</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base</td>
    <td>0.76</td>
    <td>0.71</td>
    <td>0.66</td>
    <td>0.69</td>
    <td>0.62</td>
    <td>0.68</td>
    <td>0.55</td>
    <td>0.70</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base</td>
    <td>0.78</td>
    <td>0.70</td>
    <td>0.68</td>
    <td>0.70</td>
    <td>0.64</td>
    <td>0.69</td>
    <td>0.57</td>
    <td>0.71</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B</td>
    <td>0.74</td>
    <td>0.59</td>
    <td>0.55</td>
    <td>0.57</td>
    <td>0.54</td>
    <td>0.56</td>
    <td>0.46</td>
    <td>0.61</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B</td>
    <td>0.76</td>
    <td>0.65</td>
    <td>0.63</td>
    <td>0.66</td>
    <td>0.63</td>
    <td>0.67</td>
    <td>0.54</td>
    <td>0.65</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1</td>
    <td>0.70</td>
    <td>0.59</td>
    <td>0.52</td>
    <td>0.56</td>
    <td>0.57</td>
    <td>0.57</td>
    <td>0.50</td>
    <td>0.61</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>0.72</td>
    <td>0.66</td>
    <td>0.53</td>
    <td>0.60</td>
    <td>0.53</td>
    <td>0.58</td>
    <td>0.47</td>
    <td>0.66</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>0.75</td>
    <td>0.66</td>
    <td>0.60</td>
    <td>0.62</td>
    <td>0.59</td>
    <td>0.64</td>
    <td>0.51</td>
    <td>0.67</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>0.75</td>
    <td>0.67</td>
    <td>0.63</td>
    <td>0.66</td>
    <td>0.61</td>
    <td>0.66</td>
    <td>0.55</td>
    <td>0.68</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>0.76</td>
    <td>0.68</td>
    <td>0.66</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.70</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>0.78</td>
    <td>0.68</td>
    <td>0.68</td>
    <td>0.69</td>
    <td>0.65</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.71</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>0.77</td>
    <td>0.69</td>
    <td>0.68</td>
    <td>0.69</td>
    <td>0.66</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.69</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>0.78</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.73</td>
    <td>0.60</td>
    <td>0.73</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B</td>
    <td>0.78</td>
    <td>0.69</td>
    <td>0.66</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.69</td>
    <td>0.56</td>
    <td>0.72</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>0.78</td>
    <td>0.70</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.64</td>
    <td>0.69</td>
    <td>0.57</td>
    <td>0.72</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base</td>
    <td>0.79</td>
    <td>0.70</td>
    <td>0.67</td>
    <td>0.69</td>
    <td>0.64</td>
    <td>0.69</td>
    <td>0.57</td>
    <td>0.73</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base</td>
    <td>0.80</td>
    <td>0.72</td>
    <td>0.70</td>
    <td>0.72</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.58</td>
    <td>0.74</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>gemma-2-2b</td>
    <td>0.73</td>
    <td>0.62</td>
    <td>0.57</td>
    <td>0.60</td>
    <td>0.60</td>
    <td>0.60</td>
    <td>0.50</td>
    <td>0.66</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>gemma-2-9b</td>
    <td>0.78</td>
    <td>0.68</td>
    <td>0.64</td>
    <td>0.67</td>
    <td>0.65</td>
    <td>0.69</td>
    <td>0.55</td>
    <td>0.74</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>gemma-2-27b</td>
    <td>0.72</td>
    <td>0.66</td>
    <td>0.64</td>
    <td>0.62</td>
    <td>0.62</td>
    <td>0.58</td>
    <td>0.50</td>
    <td>0.67</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b</td>
    <td>0.73</td>
    <td>0.64</td>
    <td>0.58</td>
    <td>0.64</td>
    <td>0.54</td>
    <td>0.60</td>
    <td>0.49</td>
    <td>0.65</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>internlm2_5-7b</td>
    <td>0.76</td>
    <td>0.68</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.63</td>
    <td>0.67</td>
    <td>0.60</td>
    <td>0.69</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>internlm2_5-20b</td>
    <td>0.76</td>
    <td>0.67</td>
    <td>0.68</td>
    <td>0.70</td>
    <td>0.64</td>
    <td>0.68</td>
    <td>0.59</td>
    <td>0.69</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.91</td>
    <td>0.92</td>
    <td>0.88</td>
    <td>0.92</td>
    <td>0.90</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.91</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>Claude-3-5</td>
    <td>0.94</td>
    <td>0.96</td>
    <td>0.94</td>
    <td>0.94</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.91</td>
    <td>0.96</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>0.88</td>
    <td>0.91</td>
    <td>0.88</td>
    <td>0.90</td>
    <td>0.90</td>
    <td>0.91</td>
    <td>0.85</td>
    <td>0.91</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>0.91</td>
    <td>0.95</td>
    <td>0.91</td>
    <td>0.92</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.94</td>
    <td>0.92</td>
  </tr>
  <tr>
    <th>AVG</th>
    <th>0.78</th>
    <th>0.71</th>
    <th>0.68</th>
    <th>0.70</th>
    <th>0.67</th>
    <th>0.70</th>
    <th>0.60</th>
    <th>0.73</th>
    <th>0.69</th>
  </tr>
</table>

Table 10: Model Performance on Chinese HellaSwag based on broad category under Direct Prompt.
<table>
  <tr>
    <th>Model</th>
    <th>Long</th>
    <th>Medium</th>
    <th>Short</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.62</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base</td>
    <td>0.72</td>
    <td>0.71</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B</td>
    <td>0.64</td>
    <td>0.59</td>
    <td>0.51</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B</td>
    <td>0.70</td>
    <td>0.67</td>
    <td>0.58</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1</td>
    <td>0.63</td>
    <td>0.58</td>
    <td>0.52</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>0.63</td>
    <td>0.61</td>
    <td>0.54</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>0.67</td>
    <td>0.64</td>
    <td>0.58</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>0.68</td>
    <td>0.67</td>
    <td>0.60</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>0.71</td>
    <td>0.69</td>
    <td>0.62</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>0.72</td>
    <td>0.70</td>
    <td>0.63</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>0.74</td>
    <td>0.73</td>
    <td>0.65</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B</td>
    <td>0.72</td>
    <td>0.70</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>0.73</td>
    <td>0.70</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base</td>
    <td>0.73</td>
    <td>0.71</td>
    <td>0.61</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base</td>
    <td>0.76</td>
    <td>0.72</td>
    <td>0.62</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>gemma-2-2b</td>
    <td>0.65</td>
    <td>0.61</td>
    <td>0.56</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>gemma-2-9b</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.61</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>gemma-2-27b</td>
    <td>0.68</td>
    <td>0.64</td>
    <td>0.56</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b</td>
    <td>0.65</td>
    <td>0.63</td>
    <td>0.55</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>internlm2_5-7b</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.61</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>internlm2_5-20b</td>
    <td>0.73</td>
    <td>0.69</td>
    <td>0.61</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.87</td>
    <td>0.90</td>
    <td>0.93</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>Claude-3-5</td>
    <td>0.92</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>0.86</td>
    <td>0.91</td>
    <td>0.92</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>0.89</td>
    <td>0.93</td>
    <td>0.95</td>
    <td>0.92</td>
  </tr>
  <tr>
    <th>AVG</th>
    <th>0.73</th>
    <th>0.71</th>
    <th>0.65</th>
    <th>0.69</th>
  </tr>
</table>

Table 11: Model Performance on Chinese HellaSwag based on length category under Direct Prompt.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__direct_cn</td>
    <td>0.66</td>
    <td>0.36</td>
    <td>0.06</td>
    <td>0.64</td>
    <td>0.50</td>
    <td>0.36</td>
    <td>0.58</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__direct_cn</td>
    <td>0.70</td>
    <td>0.35</td>
    <td>0.07</td>
    <td>0.65</td>
    <td>0.55</td>
    <td>0.38</td>
    <td>0.54</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__direct_cn</td>
    <td>0.66</td>
    <td>0.37</td>
    <td>0.06</td>
    <td>0.66</td>
    <td>0.57</td>
    <td>0.42</td>
    <td>0.68</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__direct_cn</td>
    <td>0.52</td>
    <td>0.34</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>0.36</td>
    <td>0.35</td>
    <td>0.37</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__direct_cn</td>
    <td>0.56</td>
    <td>0.38</td>
    <td>0.06</td>
    <td>0.75</td>
    <td>0.32</td>
    <td>0.35</td>
    <td>0.36</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__direct_cn</td>
    <td>0.59</td>
    <td>0.40</td>
    <td>0.05</td>
    <td>0.78</td>
    <td>0.29</td>
    <td>0.34</td>
    <td>0.35</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_en_cot_cn</td>
    <td>0.75</td>
    <td>0.48</td>
    <td>0.14</td>
    <td>0.62</td>
    <td>0.43</td>
    <td>0.38</td>
    <td>0.23</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_en_cot_en</td>
    <td>0.80</td>
    <td>0.29</td>
    <td>0.42</td>
    <td>0.47</td>
    <td>0.41</td>
    <td>0.32</td>
    <td>0.74</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_en_xlt_cn</td>
    <td>0.40</td>
    <td>0.11</td>
    <td>0.12</td>
    <td>0.35</td>
    <td>0.34</td>
    <td>0.18</td>
    <td>0.09</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_cn_cot_cn</td>
    <td>0.73</td>
    <td>0.51</td>
    <td>0.17</td>
    <td>0.62</td>
    <td>0.36</td>
    <td>0.29</td>
    <td>0.30</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_cn_cot_en</td>
    <td>0.81</td>
    <td>0.37</td>
    <td>0.64</td>
    <td>0.35</td>
    <td>0.43</td>
    <td>0.26</td>
    <td>0.89</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__few_shot_cn_xlt_en</td>
    <td>0.73</td>
    <td>0.28</td>
    <td>0.15</td>
    <td>0.29</td>
    <td>0.36</td>
    <td>0.18</td>
    <td>0.88</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_en_cot_cn</td>
    <td>0.91</td>
    <td>0.40</td>
    <td>0.75</td>
    <td>0.82</td>
    <td>0.79</td>
    <td>0.43</td>
    <td>0.40</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_en_cot_en</td>
    <td>0.82</td>
    <td>0.26</td>
    <td>0.50</td>
    <td>0.66</td>
    <td>0.42</td>
    <td>0.47</td>
    <td>0.82</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_en_xlt_cn</td>
    <td>0.33</td>
    <td>0.18</td>
    <td>0.41</td>
    <td>0.50</td>
    <td>0.45</td>
    <td>0.22</td>
    <td>0.23</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_cn_cot_cn</td>
    <td>0.89</td>
    <td>0.48</td>
    <td>0.82</td>
    <td>0.84</td>
    <td>0.74</td>
    <td>0.42</td>
    <td>0.62</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_cn_cot_en</td>
    <td>0.85</td>
    <td>0.41</td>
    <td>0.37</td>
    <td>0.63</td>
    <td>0.39</td>
    <td>0.41</td>
    <td>0.70</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__few_shot_cn_xlt_en</td>
    <td>0.45</td>
    <td>0.17</td>
    <td>0.15</td>
    <td>0.54</td>
    <td>0.26</td>
    <td>0.23</td>
    <td>0.64</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_en_cot_cn</td>
    <td>0.94</td>
    <td>0.50</td>
    <td>0.83</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.47</td>
    <td>0.70</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_en_cot_en</td>
    <td>0.89</td>
    <td>0.39</td>
    <td>0.52</td>
    <td>0.72</td>
    <td>0.48</td>
    <td>0.49</td>
    <td>0.68</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_en_xlt_cn</td>
    <td>0.41</td>
    <td>0.27</td>
    <td>0.45</td>
    <td>0.51</td>
    <td>0.51</td>
    <td>0.32</td>
    <td>0.10</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_cn_cot_cn</td>
    <td>0.92</td>
    <td>0.49</td>
    <td>0.90</td>
    <td>0.89</td>
    <td>0.81</td>
    <td>0.46</td>
    <td>0.80</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_cn_cot_en</td>
    <td>0.89</td>
    <td>0.35</td>
    <td>0.57</td>
    <td>0.70</td>
    <td>0.48</td>
    <td>0.44</td>
    <td>0.73</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__few_shot_cn_xlt_en</td>
    <td>0.72</td>
    <td>0.19</td>
    <td>0.64</td>
    <td>0.58</td>
    <td>0.40</td>
    <td>0.28</td>
    <td>0.51</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_en_cot_cn</td>
    <td>0.54</td>
    <td>0.18</td>
    <td>0.07</td>
    <td>0.54</td>
    <td>0.34</td>
    <td>0.23</td>
    <td>0.18</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_en_cot_en</td>
    <td>0.53</td>
    <td>0.17</td>
    <td>0.29</td>
    <td>0.36</td>
    <td>0.30</td>
    <td>0.29</td>
    <td>0.27</td>
    <td>0.32</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_en_xlt_cn</td>
    <td>0.14</td>
    <td>0.11</td>
    <td>0.02</td>
    <td>0.09</td>
    <td>0.07</td>
    <td>0.08</td>
    <td>0.00</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_cn_cot_cn</td>
    <td>0.59</td>
    <td>0.33</td>
    <td>0.10</td>
    <td>0.49</td>
    <td>0.26</td>
    <td>0.00</td>
    <td>0.06</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_cn_cot_en</td>
    <td>0.42</td>
    <td>0.18</td>
    <td>0.28</td>
    <td>0.27</td>
    <td>0.24</td>
    <td>0.20</td>
    <td>0.43</td>
    <td>0.29</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B__zero_shot_cn_xlt_en</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.10</td>
    <td>0.01</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_en_cot_cn</td>
    <td>0.95</td>
    <td>0.57</td>
    <td>0.23</td>
    <td>0.86</td>
    <td>0.74</td>
    <td>0.49</td>
    <td>0.39</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_en_cot_en</td>
    <td>0.69</td>
    <td>0.32</td>
    <td>0.54</td>
    <td>0.47</td>
    <td>0.48</td>
    <td>0.40</td>
    <td>0.41</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_en_xlt_cn</td>
    <td>0.03</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.04</td>
    <td>0.01</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_cn_cot_cn</td>
    <td>0.70</td>
    <td>0.45</td>
    <td>0.35</td>
    <td>0.72</td>
    <td>0.55</td>
    <td>0.00</td>
    <td>0.38</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_cn_cot_en</td>
    <td>0.48</td>
    <td>0.26</td>
    <td>0.05</td>
    <td>0.53</td>
    <td>0.28</td>
    <td>0.36</td>
    <td>0.40</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B__zero_shot_cn_xlt_en</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.03</td>
    <td>0.04</td>
    <td>0.00</td>
    <td>0.03</td>
    <td>0.00</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_en_cot_cn</td>
    <td>0.92</td>
    <td>0.44</td>
    <td>0.50</td>
    <td>0.88</td>
    <td>0.78</td>
    <td>0.46</td>
    <td>0.73</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_en_cot_en</td>
    <td>0.81</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.66</td>
    <td>0.43</td>
    <td>0.45</td>
    <td>0.56</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_en_xlt_cn</td>
    <td>0.52</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.54</td>
    <td>0.50</td>
    <td>0.43</td>
    <td>0.53</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_cn_cot_cn</td>
    <td>0.75</td>
    <td>0.43</td>
    <td>0.28</td>
    <td>0.72</td>
    <td>0.55</td>
    <td>0.00</td>
    <td>0.58</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_cn_cot_en</td>
    <td>0.51</td>
    <td>0.20</td>
    <td>0.07</td>
    <td>0.53</td>
    <td>0.37</td>
    <td>0.32</td>
    <td>0.39</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B__zero_shot_cn_xlt_en</td>
    <td>0.82</td>
    <td>0.25</td>
    <td>0.03</td>
    <td>0.73</td>
    <td>0.37</td>
    <td>0.47</td>
    <td>0.13</td>
    <td>0.40</td>
  </tr>
</table>

Table 12: Performance of Qwen Series (0.5B-3B). And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Qwen2.5-7B__direct_cn</td>
    <td>0.66</td>
    <td>0.34</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>0.62</td>
    <td>0.41</td>
    <td>0.77</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__direct_cn</td>
    <td>0.67</td>
    <td>0.35</td>
    <td>0.07</td>
    <td>0.69</td>
    <td>0.63</td>
    <td>0.43</td>
    <td>0.75</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__direct_cn</td>
    <td>0.68</td>
    <td>0.36</td>
    <td>0.06</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.43</td>
    <td>0.87</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__direct_cn</td>
    <td>0.67</td>
    <td>0.39</td>
    <td>0.08</td>
    <td>0.69</td>
    <td>0.65</td>
    <td>0.44</td>
    <td>0.92</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__direct_en</td>
    <td>0.64</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.82</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.56</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__direct_en</td>
    <td>0.66</td>
    <td>0.42</td>
    <td>0.05</td>
    <td>0.83</td>
    <td>0.28</td>
    <td>0.35</td>
    <td>0.58</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__direct_en</td>
    <td>0.65</td>
    <td>0.42</td>
    <td>0.05</td>
    <td>0.83</td>
    <td>0.29</td>
    <td>0.34</td>
    <td>0.65</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__direct_en</td>
    <td>0.67</td>
    <td>0.43</td>
    <td>0.05</td>
    <td>0.86</td>
    <td>0.30</td>
    <td>0.34</td>
    <td>0.71</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_en_cot_cn</td>
    <td>0.95</td>
    <td>0.57</td>
    <td>0.92</td>
    <td>0.90</td>
    <td>0.89</td>
    <td>0.53</td>
    <td>0.60</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_en_cot_en</td>
    <td>0.88</td>
    <td>0.50</td>
    <td>0.78</td>
    <td>0.84</td>
    <td>0.54</td>
    <td>0.54</td>
    <td>0.81</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_en_xlt_cn</td>
    <td>0.51</td>
    <td>0.27</td>
    <td>0.50</td>
    <td>0.48</td>
    <td>0.53</td>
    <td>0.41</td>
    <td>0.44</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_cn_cot_cn</td>
    <td>0.96</td>
    <td>0.55</td>
    <td>0.91</td>
    <td>0.92</td>
    <td>0.85</td>
    <td>0.59</td>
    <td>0.93</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_cn_cot_en</td>
    <td>0.93</td>
    <td>0.53</td>
    <td>0.79</td>
    <td>0.75</td>
    <td>0.56</td>
    <td>0.54</td>
    <td>0.81</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__few_shot_cn_xlt_en</td>
    <td>0.76</td>
    <td>0.30</td>
    <td>0.77</td>
    <td>0.69</td>
    <td>0.43</td>
    <td>0.27</td>
    <td>0.65</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_en_cot_cn</td>
    <td>0.97</td>
    <td>0.58</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.88</td>
    <td>0.66</td>
    <td>0.94</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.50</td>
    <td>0.75</td>
    <td>0.88</td>
    <td>0.57</td>
    <td>0.55</td>
    <td>0.81</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_en_xlt_cn</td>
    <td>0.63</td>
    <td>0.40</td>
    <td>0.65</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.49</td>
    <td>0.69</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_cn_cot_cn</td>
    <td>0.97</td>
    <td>0.60</td>
    <td>0.94</td>
    <td>0.91</td>
    <td>0.87</td>
    <td>0.66</td>
    <td>0.92</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_cn_cot_en</td>
    <td>0.93</td>
    <td>0.56</td>
    <td>0.77</td>
    <td>0.83</td>
    <td>0.56</td>
    <td>0.55</td>
    <td>0.82</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__few_shot_cn_xlt_en</td>
    <td>0.82</td>
    <td>0.38</td>
    <td>0.74</td>
    <td>0.71</td>
    <td>0.42</td>
    <td>0.36</td>
    <td>0.57</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_en_cot_cn</td>
    <td>0.98</td>
    <td>0.63</td>
    <td>0.91</td>
    <td>0.94</td>
    <td>0.92</td>
    <td>0.71</td>
    <td>0.95</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.59</td>
    <td>0.84</td>
    <td>0.88</td>
    <td>0.64</td>
    <td>0.58</td>
    <td>0.81</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_en_xlt_cn</td>
    <td>0.68</td>
    <td>0.46</td>
    <td>0.75</td>
    <td>0.66</td>
    <td>0.60</td>
    <td>0.49</td>
    <td>0.80</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_cn_cot_cn</td>
    <td>0.98</td>
    <td>0.61</td>
    <td>0.94</td>
    <td>0.93</td>
    <td>0.90</td>
    <td>0.68</td>
    <td>0.95</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_cn_cot_en</td>
    <td>0.94</td>
    <td>0.66</td>
    <td>0.84</td>
    <td>0.90</td>
    <td>0.62</td>
    <td>0.59</td>
    <td>0.83</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__few_shot_cn_xlt_en</td>
    <td>0.82</td>
    <td>0.46</td>
    <td>0.82</td>
    <td>0.82</td>
    <td>0.51</td>
    <td>0.42</td>
    <td>0.59</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_en_cot_cn</td>
    <td>0.98</td>
    <td>0.66</td>
    <td>0.91</td>
    <td>0.94</td>
    <td>0.92</td>
    <td>0.73</td>
    <td>0.92</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_en_cot_en</td>
    <td>0.91</td>
    <td>0.59</td>
    <td>0.80</td>
    <td>0.92</td>
    <td>0.67</td>
    <td>0.58</td>
    <td>0.81</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_en_xlt_cn</td>
    <td>0.50</td>
    <td>0.30</td>
    <td>0.68</td>
    <td>0.62</td>
    <td>0.55</td>
    <td>0.54</td>
    <td>0.76</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_cn_cot_cn</td>
    <td>0.97</td>
    <td>0.62</td>
    <td>0.95</td>
    <td>0.93</td>
    <td>0.91</td>
    <td>0.74</td>
    <td>0.92</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_cn_cot_en</td>
    <td>0.94</td>
    <td>0.62</td>
    <td>0.80</td>
    <td>0.90</td>
    <td>0.69</td>
    <td>0.59</td>
    <td>0.83</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__few_shot_cn_xlt_en</td>
    <td>0.59</td>
    <td>0.44</td>
    <td>0.81</td>
    <td>0.70</td>
    <td>0.53</td>
    <td>0.33</td>
    <td>0.57</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_en_cot_cn</td>
    <td>0.82</td>
    <td>0.56</td>
    <td>0.73</td>
    <td>0.85</td>
    <td>0.70</td>
    <td>0.54</td>
    <td>0.78</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_en_cot_en</td>
    <td>0.82</td>
    <td>0.40</td>
    <td>0.53</td>
    <td>0.70</td>
    <td>0.38</td>
    <td>0.53</td>
    <td>0.59</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_en_xlt_cn</td>
    <td>0.83</td>
    <td>0.55</td>
    <td>0.62</td>
    <td>0.80</td>
    <td>0.73</td>
    <td>0.53</td>
    <td>0.83</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_cn_cot_cn</td>
    <td>0.74</td>
    <td>0.42</td>
    <td>0.30</td>
    <td>0.76</td>
    <td>0.65</td>
    <td>0.00</td>
    <td>0.50</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_cn_cot_en</td>
    <td>0.70</td>
    <td>0.27</td>
    <td>0.09</td>
    <td>0.57</td>
    <td>0.33</td>
    <td>0.37</td>
    <td>0.54</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B__zero_shot_cn_xlt_en</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_en_cot_cn</td>
    <td>0.62</td>
    <td>0.46</td>
    <td>0.81</td>
    <td>0.78</td>
    <td>0.66</td>
    <td>0.54</td>
    <td>0.86</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_en_cot_en</td>
    <td>0.88</td>
    <td>0.44</td>
    <td>0.41</td>
    <td>0.70</td>
    <td>0.46</td>
    <td>0.53</td>
    <td>0.72</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_en_xlt_cn</td>
    <td>0.93</td>
    <td>0.63</td>
    <td>0.85</td>
    <td>0.93</td>
    <td>0.79</td>
    <td>0.70</td>
    <td>0.92</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_cn_cot_cn</td>
    <td>0.79</td>
    <td>0.56</td>
    <td>0.74</td>
    <td>0.81</td>
    <td>0.73</td>
    <td>0.51</td>
    <td>0.82</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_cn_cot_en</td>
    <td>0.72</td>
    <td>0.37</td>
    <td>0.24</td>
    <td>0.64</td>
    <td>0.44</td>
    <td>0.38</td>
    <td>0.59</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B__zero_shot_cn_xlt_en</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.17</td>
    <td>0.09</td>
    <td>0.22</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_en_cot_cn</td>
    <td>0.80</td>
    <td>0.43</td>
    <td>0.83</td>
    <td>0.81</td>
    <td>0.68</td>
    <td>0.57</td>
    <td>0.86</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_en_cot_en</td>
    <td>0.86</td>
    <td>0.52</td>
    <td>0.56</td>
    <td>0.82</td>
    <td>0.53</td>
    <td>0.54</td>
    <td>0.75</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_en_xlt_cn</td>
    <td>0.78</td>
    <td>0.58</td>
    <td>0.83</td>
    <td>0.75</td>
    <td>0.49</td>
    <td>0.52</td>
    <td>0.49</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_cn_cot_cn</td>
    <td>0.87</td>
    <td>0.60</td>
    <td>0.81</td>
    <td>0.87</td>
    <td>0.76</td>
    <td>0.00</td>
    <td>0.91</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_cn_cot_en</td>
    <td>0.82</td>
    <td>0.47</td>
    <td>0.30</td>
    <td>0.79</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.61</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B__zero_shot_cn_xlt_en</td>
    <td>0.64</td>
    <td>0.45</td>
    <td>0.36</td>
    <td>0.62</td>
    <td>0.55</td>
    <td>0.27</td>
    <td>0.62</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_en_cot_cn</td>
    <td>0.84</td>
    <td>0.48</td>
    <td>0.82</td>
    <td>0.84</td>
    <td>0.72</td>
    <td>0.60</td>
    <td>0.73</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_en_cot_en</td>
    <td>0.78</td>
    <td>0.50</td>
    <td>0.44</td>
    <td>0.79</td>
    <td>0.49</td>
    <td>0.51</td>
    <td>0.75</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_en_xlt_cn</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.18</td>
    <td>0.21</td>
    <td>0.19</td>
    <td>0.05</td>
    <td>0.17</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_cn_cot_cn</td>
    <td>0.79</td>
    <td>0.59</td>
    <td>0.73</td>
    <td>0.86</td>
    <td>0.73</td>
    <td>0.56</td>
    <td>0.82</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_cn_cot_en</td>
    <td>0.70</td>
    <td>0.31</td>
    <td>0.15</td>
    <td>0.76</td>
    <td>0.55</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B__zero_shot_cn_xlt_en</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.14</td>
    <td>0.20</td>
    <td>0.39</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.12</td>
  </tr>
</table>

Table 13: Performance of Qwen Series (7B-72B). And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__direct_cn</td>
    <td>0.61</td>
    <td>0.35</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>0.59</td>
    <td>0.43</td>
    <td>0.43</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__direct_cn</td>
    <td>0.65</td>
    <td>0.38</td>
    <td>0.08</td>
    <td>0.71</td>
    <td>0.63</td>
    <td>0.46</td>
    <td>0.55</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__direct_en</td>
    <td>0.53</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.81</td>
    <td>0.33</td>
    <td>0.34</td>
    <td>0.37</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__direct_en</td>
    <td>0.57</td>
    <td>0.42</td>
    <td>0.05</td>
    <td>0.85</td>
    <td>0.25</td>
    <td>0.34</td>
    <td>0.37</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_en_cot_cn</td>
    <td>0.85</td>
    <td>0.49</td>
    <td>0.35</td>
    <td>0.81</td>
    <td>0.62</td>
    <td>0.43</td>
    <td>0.40</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_en_cot_en</td>
    <td>0.85</td>
    <td>0.20</td>
    <td>0.45</td>
    <td>0.53</td>
    <td>0.26</td>
    <td>0.40</td>
    <td>0.90</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_en_xlt_cn</td>
    <td>0.27</td>
    <td>0.06</td>
    <td>0.10</td>
    <td>0.33</td>
    <td>0.42</td>
    <td>0.28</td>
    <td>0.28</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_cn_cot_cn</td>
    <td>0.88</td>
    <td>0.55</td>
    <td>0.66</td>
    <td>0.74</td>
    <td>0.63</td>
    <td>0.49</td>
    <td>0.56</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_cn_cot_en</td>
    <td>0.82</td>
    <td>0.27</td>
    <td>0.62</td>
    <td>0.44</td>
    <td>0.38</td>
    <td>0.34</td>
    <td>0.82</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__few_shot_cn_xlt_en</td>
    <td>0.34</td>
    <td>0.16</td>
    <td>0.51</td>
    <td>0.46</td>
    <td>0.45</td>
    <td>0.29</td>
    <td>0.81</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_en_cot_cn</td>
    <td>0.96</td>
    <td>0.51</td>
    <td>0.89</td>
    <td>0.91</td>
    <td>0.84</td>
    <td>0.68</td>
    <td>0.81</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_en_cot_en</td>
    <td>0.92</td>
    <td>0.39</td>
    <td>0.88</td>
    <td>0.85</td>
    <td>0.53</td>
    <td>0.49</td>
    <td>0.88</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_en_xlt_cn</td>
    <td>0.42</td>
    <td>0.10</td>
    <td>0.70</td>
    <td>0.63</td>
    <td>0.51</td>
    <td>0.50</td>
    <td>0.36</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_cn_cot_cn</td>
    <td>0.97</td>
    <td>0.61</td>
    <td>0.91</td>
    <td>0.89</td>
    <td>0.82</td>
    <td>0.71</td>
    <td>0.90</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_cn_cot_en</td>
    <td>0.92</td>
    <td>0.43</td>
    <td>0.84</td>
    <td>0.79</td>
    <td>0.45</td>
    <td>0.44</td>
    <td>0.90</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__few_shot_cn_xlt_en</td>
    <td>0.52</td>
    <td>0.24</td>
    <td>0.18</td>
    <td>0.74</td>
    <td>0.46</td>
    <td>0.34</td>
    <td>0.65</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_en_cot_cn</td>
    <td>0.18</td>
    <td>0.03</td>
    <td>0.03</td>
    <td>0.15</td>
    <td>0.05</td>
    <td>0.05</td>
    <td>0.12</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_en_cot_en</td>
    <td>0.11</td>
    <td>0.14</td>
    <td>0.01</td>
    <td>0.16</td>
    <td>0.09</td>
    <td>0.19</td>
    <td>0.01</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_en_xlt_cn</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.08</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_cn_cot_cn</td>
    <td>0.40</td>
    <td>0.18</td>
    <td>0.07</td>
    <td>0.23</td>
    <td>0.18</td>
    <td>0.00</td>
    <td>0.09</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_cn_cot_en</td>
    <td>0.27</td>
    <td>0.12</td>
    <td>0.07</td>
    <td>0.17</td>
    <td>0.13</td>
    <td>0.11</td>
    <td>0.00</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td>deepseek-llm-7b-base__zero_shot_cn_xlt_en</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_en_cot_cn</td>
    <td>0.03</td>
    <td>0.17</td>
    <td>0.01</td>
    <td>0.14</td>
    <td>0.12</td>
    <td>0.12</td>
    <td>0.40</td>
    <td>0.14</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_en_cot_en</td>
    <td>0.08</td>
    <td>0.12</td>
    <td>0.01</td>
    <td>0.11</td>
    <td>0.02</td>
    <td>0.08</td>
    <td>0.33</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_en_xlt_cn</td>
    <td>0.64</td>
    <td>0.34</td>
    <td>0.02</td>
    <td>0.29</td>
    <td>0.29</td>
    <td>0.30</td>
    <td>0.03</td>
    <td>0.27</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_cn_cot_cn</td>
    <td>0.36</td>
    <td>0.14</td>
    <td>0.03</td>
    <td>0.36</td>
    <td>0.18</td>
    <td>0.20</td>
    <td>0.34</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_cn_cot_en</td>
    <td>0.20</td>
    <td>0.08</td>
    <td>0.01</td>
    <td>0.08</td>
    <td>0.02</td>
    <td>0.07</td>
    <td>0.16</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>deepseek-llm-67b-base__zero_shot_cn_xlt_en</td>
    <td>0.49</td>
    <td>0.06</td>
    <td>0.00</td>
    <td>0.29</td>
    <td>0.14</td>
    <td>0.11</td>
    <td>0.07</td>
    <td>0.17</td>
  </tr>
</table>

Table 14: Performance of DeepSeek Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Yi-6B__direct_cn</td>
    <td>0.68</td>
    <td>0.37</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>0.60</td>
    <td>0.42</td>
    <td>0.67</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Yi-9B__direct_cn</td>
    <td>0.70</td>
    <td>0.37</td>
    <td>0.08</td>
    <td>0.66</td>
    <td>0.64</td>
    <td>0.44</td>
    <td>0.62</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Yi-34B__direct_cn</td>
    <td>0.69</td>
    <td>0.38</td>
    <td>0.09</td>
    <td>0.67</td>
    <td>0.67</td>
    <td>0.44</td>
    <td>0.72</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Yi-6B__direct_en</td>
    <td>0.58</td>
    <td>0.39</td>
    <td>0.05</td>
    <td>0.77</td>
    <td>0.27</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Yi-9B__direct_en</td>
    <td>0.56</td>
    <td>0.41</td>
    <td>0.06</td>
    <td>0.80</td>
    <td>0.26</td>
    <td>0.35</td>
    <td>0.35</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Yi-34B__direct_en</td>
    <td>0.62</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.81</td>
    <td>0.27</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_en_cot_cn</td>
    <td>0.91</td>
    <td>0.60</td>
    <td>0.78</td>
    <td>0.84</td>
    <td>0.82</td>
    <td>0.56</td>
    <td>0.68</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_en_cot_en</td>
    <td>0.89</td>
    <td>0.36</td>
    <td>0.45</td>
    <td>0.68</td>
    <td>0.33</td>
    <td>0.51</td>
    <td>0.65</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_en_xlt_cn</td>
    <td>0.41</td>
    <td>0.10</td>
    <td>0.47</td>
    <td>0.48</td>
    <td>0.44</td>
    <td>0.35</td>
    <td>0.23</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_cn_cot_cn</td>
    <td>0.94</td>
    <td>0.58</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.76</td>
    <td>0.57</td>
    <td>0.72</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_cn_cot_en</td>
    <td>0.83</td>
    <td>0.46</td>
    <td>0.60</td>
    <td>0.57</td>
    <td>0.33</td>
    <td>0.40</td>
    <td>0.86</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__few_shot_cn_xlt_en</td>
    <td>0.72</td>
    <td>0.23</td>
    <td>0.15</td>
    <td>0.55</td>
    <td>0.41</td>
    <td>0.27</td>
    <td>0.41</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_en_cot_cn</td>
    <td>0.97</td>
    <td>0.60</td>
    <td>0.85</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.64</td>
    <td>0.78</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.45</td>
    <td>0.78</td>
    <td>0.77</td>
    <td>0.55</td>
    <td>0.50</td>
    <td>0.77</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_en_xlt_cn</td>
    <td>0.48</td>
    <td>0.24</td>
    <td>0.68</td>
    <td>0.50</td>
    <td>0.45</td>
    <td>0.39</td>
    <td>0.41</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_cn_cot_cn</td>
    <td>0.96</td>
    <td>0.61</td>
    <td>0.90</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.65</td>
    <td>0.87</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_cn_cot_en</td>
    <td>0.92</td>
    <td>0.50</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.52</td>
    <td>0.47</td>
    <td>0.77</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__few_shot_cn_xlt_en</td>
    <td>0.84</td>
    <td>0.25</td>
    <td>0.53</td>
    <td>0.69</td>
    <td>0.46</td>
    <td>0.29</td>
    <td>0.45</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_en_cot_cn</td>
    <td>0.96</td>
    <td>0.54</td>
    <td>0.91</td>
    <td>0.91</td>
    <td>0.88</td>
    <td>0.70</td>
    <td>0.93</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_en_cot_en</td>
    <td>0.92</td>
    <td>0.59</td>
    <td>0.79</td>
    <td>0.92</td>
    <td>0.60</td>
    <td>0.54</td>
    <td>0.82</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_en_xlt_cn</td>
    <td>0.27</td>
    <td>0.12</td>
    <td>0.58</td>
    <td>0.36</td>
    <td>0.44</td>
    <td>0.40</td>
    <td>0.43</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_cn_cot_cn</td>
    <td>0.96</td>
    <td>0.57</td>
    <td>0.92</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.68</td>
    <td>0.93</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_cn_cot_en</td>
    <td>0.93</td>
    <td>0.57</td>
    <td>0.79</td>
    <td>0.87</td>
    <td>0.57</td>
    <td>0.52</td>
    <td>0.78</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__few_shot_cn_xlt_en</td>
    <td>0.62</td>
    <td>0.33</td>
    <td>0.71</td>
    <td>0.74</td>
    <td>0.49</td>
    <td>0.30</td>
    <td>0.57</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_en_cot_cn</td>
    <td>0.77</td>
    <td>0.52</td>
    <td>0.12</td>
    <td>0.67</td>
    <td>0.64</td>
    <td>0.34</td>
    <td>0.48</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_en_cot_en</td>
    <td>0.83</td>
    <td>0.43</td>
    <td>0.50</td>
    <td>0.59</td>
    <td>0.55</td>
    <td>0.54</td>
    <td>0.21</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_en_xlt_cn</td>
    <td>0.20</td>
    <td>0.17</td>
    <td>0.02</td>
    <td>0.19</td>
    <td>0.15</td>
    <td>0.06</td>
    <td>0.30</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_cn_cot_cn</td>
    <td>0.69</td>
    <td>0.27</td>
    <td>0.05</td>
    <td>0.63</td>
    <td>0.57</td>
    <td>0.00</td>
    <td>0.24</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_cn_cot_en</td>
    <td>0.58</td>
    <td>0.26</td>
    <td>0.04</td>
    <td>0.48</td>
    <td>0.25</td>
    <td>0.37</td>
    <td>0.17</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B__zero_shot_cn_xlt_en</td>
    <td>0.05</td>
    <td>0.05</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.03</td>
    <td>0.07</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_en_cot_cn</td>
    <td>0.60</td>
    <td>0.47</td>
    <td>0.14</td>
    <td>0.67</td>
    <td>0.48</td>
    <td>0.39</td>
    <td>0.70</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_en_cot_en</td>
    <td>0.87</td>
    <td>0.38</td>
    <td>0.26</td>
    <td>0.68</td>
    <td>0.47</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_en_xlt_cn</td>
    <td>0.94</td>
    <td>0.51</td>
    <td>0.27</td>
    <td>0.85</td>
    <td>0.76</td>
    <td>0.56</td>
    <td>0.70</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_cn_cot_cn</td>
    <td>0.79</td>
    <td>0.36</td>
    <td>0.12</td>
    <td>0.72</td>
    <td>0.70</td>
    <td>0.50</td>
    <td>0.14</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_cn_cot_en</td>
    <td>0.61</td>
    <td>0.18</td>
    <td>0.10</td>
    <td>0.50</td>
    <td>0.35</td>
    <td>0.36</td>
    <td>0.29</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B__zero_shot_cn_xlt_en</td>
    <td>0.91</td>
    <td>0.64</td>
    <td>0.03</td>
    <td>0.84</td>
    <td>0.45</td>
    <td>0.58</td>
    <td>0.59</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_en_cot_cn</td>
    <td>0.28</td>
    <td>0.37</td>
    <td>0.18</td>
    <td>0.24</td>
    <td>0.28</td>
    <td>0.17</td>
    <td>0.58</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_en_cot_en</td>
    <td>0.85</td>
    <td>0.48</td>
    <td>0.35</td>
    <td>0.75</td>
    <td>0.50</td>
    <td>0.47</td>
    <td>0.66</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_en_xlt_cn</td>
    <td>0.09</td>
    <td>0.03</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.07</td>
    <td>0.05</td>
    <td>0.09</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_cn_cot_cn</td>
    <td>0.82</td>
    <td>0.36</td>
    <td>0.29</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.00</td>
    <td>0.77</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_cn_cot_en</td>
    <td>0.77</td>
    <td>0.31</td>
    <td>0.05</td>
    <td>0.64</td>
    <td>0.50</td>
    <td>0.43</td>
    <td>0.64</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B__zero_shot_cn_xlt_en</td>
    <td>0.78</td>
    <td>0.25</td>
    <td>0.02</td>
    <td>0.26</td>
    <td>0.43</td>
    <td>0.16</td>
    <td>0.34</td>
    <td>0.32</td>
  </tr>
</table>

Table 15: Performance of Yi Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__direct_cn</td>
    <td>0.57</td>
    <td>0.37</td>
    <td>0.09</td>
    <td>0.63</td>
    <td>0.51</td>
    <td>0.46</td>
    <td>0.63</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__direct_cn</td>
    <td>0.63</td>
    <td>0.40</td>
    <td>0.08</td>
    <td>0.67</td>
    <td>0.60</td>
    <td>0.46</td>
    <td>0.57</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__direct_en</td>
    <td>0.56</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.82</td>
    <td>0.27</td>
    <td>0.35</td>
    <td>0.36</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__direct_en</td>
    <td>0.57</td>
    <td>0.43</td>
    <td>0.04</td>
    <td>0.86</td>
    <td>0.29</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_en_cot_cn</td>
    <td>0.90</td>
    <td>0.45</td>
    <td>0.68</td>
    <td>0.81</td>
    <td>0.67</td>
    <td>0.57</td>
    <td>0.58</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_en_cot_en</td>
    <td>0.79</td>
    <td>0.39</td>
    <td>0.79</td>
    <td>0.64</td>
    <td>0.51</td>
    <td>0.45</td>
    <td>0.94</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_en_xlt_cn</td>
    <td>0.36</td>
    <td>0.21</td>
    <td>0.39</td>
    <td>0.48</td>
    <td>0.54</td>
    <td>0.39</td>
    <td>0.23</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_cn_cot_cn</td>
    <td>0.87</td>
    <td>0.45</td>
    <td>0.67</td>
    <td>0.73</td>
    <td>0.54</td>
    <td>0.60</td>
    <td>0.63</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_cn_cot_en</td>
    <td>0.76</td>
    <td>0.46</td>
    <td>0.56</td>
    <td>0.59</td>
    <td>0.39</td>
    <td>0.38</td>
    <td>0.82</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__few_shot_cn_xlt_en</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.06</td>
    <td>0.27</td>
    <td>0.13</td>
    <td>0.05</td>
    <td>0.32</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_en_cot_cn</td>
    <td>0.97</td>
    <td>0.58</td>
    <td>0.87</td>
    <td>0.91</td>
    <td>0.89</td>
    <td>0.70</td>
    <td>0.91</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_en_cot_en</td>
    <td>0.92</td>
    <td>0.42</td>
    <td>0.77</td>
    <td>0.84</td>
    <td>0.57</td>
    <td>0.50</td>
    <td>0.88</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_en_xlt_cn</td>
    <td>0.46</td>
    <td>0.22</td>
    <td>0.65</td>
    <td>0.62</td>
    <td>0.56</td>
    <td>0.52</td>
    <td>0.52</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_cn_cot_cn</td>
    <td>0.97</td>
    <td>0.36</td>
    <td>0.87</td>
    <td>0.84</td>
    <td>0.69</td>
    <td>0.63</td>
    <td>0.92</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_cn_cot_en</td>
    <td>0.93</td>
    <td>0.45</td>
    <td>0.73</td>
    <td>0.86</td>
    <td>0.56</td>
    <td>0.44</td>
    <td>0.85</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__few_shot_cn_xlt_en</td>
    <td>0.12</td>
    <td>0.09</td>
    <td>0.31</td>
    <td>0.50</td>
    <td>0.25</td>
    <td>0.14</td>
    <td>0.28</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_en_cot_cn</td>
    <td>0.59</td>
    <td>0.24</td>
    <td>0.07</td>
    <td>0.40</td>
    <td>0.32</td>
    <td>0.29</td>
    <td>0.40</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_en_cot_en</td>
    <td>0.52</td>
    <td>0.18</td>
    <td>0.11</td>
    <td>0.38</td>
    <td>0.19</td>
    <td>0.33</td>
    <td>0.38</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_en_xlt_cn</td>
    <td>0.39</td>
    <td>0.17</td>
    <td>0.01</td>
    <td>0.42</td>
    <td>0.16</td>
    <td>0.19</td>
    <td>0.16</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_cn_cot_cn</td>
    <td>0.53</td>
    <td>0.21</td>
    <td>0.09</td>
    <td>0.40</td>
    <td>0.34</td>
    <td>0.50</td>
    <td>0.12</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_cn_cot_en</td>
    <td>0.50</td>
    <td>0.16</td>
    <td>0.04</td>
    <td>0.30</td>
    <td>0.22</td>
    <td>0.24</td>
    <td>0.12</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B__zero_shot_cn_xlt_en</td>
    <td>0.43</td>
    <td>0.08</td>
    <td>0.02</td>
    <td>0.22</td>
    <td>0.23</td>
    <td>0.15</td>
    <td>0.03</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_en_cot_cn</td>
    <td>0.78</td>
    <td>0.43</td>
    <td>0.06</td>
    <td>0.64</td>
    <td>0.63</td>
    <td>0.37</td>
    <td>0.56</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_en_cot_en</td>
    <td>0.78</td>
    <td>0.35</td>
    <td>0.04</td>
    <td>0.64</td>
    <td>0.45</td>
    <td>0.41</td>
    <td>0.57</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_en_xlt_cn</td>
    <td>0.79</td>
    <td>0.55</td>
    <td>0.05</td>
    <td>0.77</td>
    <td>0.49</td>
    <td>0.36</td>
    <td>0.22</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_cn_cot_cn</td>
    <td>0.63</td>
    <td>0.35</td>
    <td>0.13</td>
    <td>0.56</td>
    <td>0.48</td>
    <td>0.34</td>
    <td>0.63</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_cn_cot_en</td>
    <td>0.58</td>
    <td>0.30</td>
    <td>0.02</td>
    <td>0.50</td>
    <td>0.32</td>
    <td>0.31</td>
    <td>0.38</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B__zero_shot_cn_xlt_en</td>
    <td>0.90</td>
    <td>0.39</td>
    <td>0.01</td>
    <td>0.86</td>
    <td>0.52</td>
    <td>0.50</td>
    <td>0.62</td>
    <td>0.54</td>
  </tr>
</table>

Table 16: Performance of LLaMA Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__direct_cn</td>
    <td>0.58</td>
    <td>0.37</td>
    <td>0.10</td>
    <td>0.64</td>
    <td>0.50</td>
    <td>0.44</td>
    <td>0.62</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__direct_cn</td>
    <td>0.59</td>
    <td>0.36</td>
    <td>0.09</td>
    <td>0.65</td>
    <td>0.54</td>
    <td>0.46</td>
    <td>0.58</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__direct_cn</td>
    <td>0.66</td>
    <td>0.39</td>
    <td>0.07</td>
    <td>0.69</td>
    <td>0.57</td>
    <td>0.43</td>
    <td>0.74</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__direct_en</td>
    <td>0.57</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.82</td>
    <td>0.31</td>
    <td>0.34</td>
    <td>0.40</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__direct_en</td>
    <td>0.56</td>
    <td>0.42</td>
    <td>0.06</td>
    <td>0.85</td>
    <td>0.31</td>
    <td>0.33</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__direct_en</td>
    <td>0.57</td>
    <td>0.42</td>
    <td>0.04</td>
    <td>0.85</td>
    <td>0.31</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_en_cot_cn</td>
    <td>0.90</td>
    <td>0.51</td>
    <td>0.62</td>
    <td>0.83</td>
    <td>0.71</td>
    <td>0.56</td>
    <td>0.57</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_en_cot_en</td>
    <td>0.86</td>
    <td>0.34</td>
    <td>0.84</td>
    <td>0.63</td>
    <td>0.53</td>
    <td>0.47</td>
    <td>0.90</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_en_xlt_cn</td>
    <td>0.13</td>
    <td>0.05</td>
    <td>0.17</td>
    <td>0.20</td>
    <td>0.30</td>
    <td>0.25</td>
    <td>0.44</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_cn_cot_cn</td>
    <td>0.56</td>
    <td>0.06</td>
    <td>0.34</td>
    <td>0.49</td>
    <td>0.41</td>
    <td>0.43</td>
    <td>0.51</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_cn_cot_en</td>
    <td>0.61</td>
    <td>0.28</td>
    <td>0.35</td>
    <td>0.45</td>
    <td>0.17</td>
    <td>0.16</td>
    <td>0.67</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__few_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_en_cot_cn</td>
    <td>0.94</td>
    <td>0.60</td>
    <td>0.80</td>
    <td>0.90</td>
    <td>0.81</td>
    <td>0.66</td>
    <td>0.65</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_en_cot_en</td>
    <td>0.91</td>
    <td>0.45</td>
    <td>0.59</td>
    <td>0.82</td>
    <td>0.56</td>
    <td>0.54</td>
    <td>0.87</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_en_xlt_cn</td>
    <td>0.15</td>
    <td>0.04</td>
    <td>0.18</td>
    <td>0.18</td>
    <td>0.34</td>
    <td>0.23</td>
    <td>0.24</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_cn_cot_cn</td>
    <td>0.75</td>
    <td>0.13</td>
    <td>0.35</td>
    <td>0.47</td>
    <td>0.52</td>
    <td>0.42</td>
    <td>0.76</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_cn_cot_en</td>
    <td>0.61</td>
    <td>0.33</td>
    <td>0.38</td>
    <td>0.56</td>
    <td>0.28</td>
    <td>0.13</td>
    <td>0.64</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__few_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_en_cot_cn</td>
    <td>0.96</td>
    <td>0.63</td>
    <td>0.90</td>
    <td>0.92</td>
    <td>0.86</td>
    <td>0.69</td>
    <td>0.87</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_en_cot_en</td>
    <td>0.92</td>
    <td>0.59</td>
    <td>0.69</td>
    <td>0.88</td>
    <td>0.63</td>
    <td>0.57</td>
    <td>0.78</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_en_xlt_cn</td>
    <td>0.26</td>
    <td>0.08</td>
    <td>0.36</td>
    <td>0.43</td>
    <td>0.36</td>
    <td>0.31</td>
    <td>0.33</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_cn_cot_cn</td>
    <td>0.60</td>
    <td>0.08</td>
    <td>0.36</td>
    <td>0.37</td>
    <td>0.57</td>
    <td>0.32</td>
    <td>0.82</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_cn_cot_en</td>
    <td>0.62</td>
    <td>0.44</td>
    <td>0.34</td>
    <td>0.65</td>
    <td>0.26</td>
    <td>0.21</td>
    <td>0.76</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__few_shot_cn_xlt_en</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_en_cot_cn</td>
    <td>0.05</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.06</td>
    <td>0.03</td>
    <td>0.06</td>
    <td>0.11</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_en_cot_en</td>
    <td>0.27</td>
    <td>0.31</td>
    <td>0.02</td>
    <td>0.19</td>
    <td>0.08</td>
    <td>0.19</td>
    <td>0.11</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_en_xlt_cn</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_cn_cot_cn</td>
    <td>0.21</td>
    <td>0.08</td>
    <td>0.01</td>
    <td>0.17</td>
    <td>0.08</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_cn_cot_en</td>
    <td>0.11</td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.08</td>
    <td>0.04</td>
    <td>0.05</td>
    <td>0.08</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.1__zero_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_en_cot_cn</td>
    <td>0.66</td>
    <td>0.25</td>
    <td>0.07</td>
    <td>0.47</td>
    <td>0.24</td>
    <td>0.40</td>
    <td>0.29</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_en_xlt_cn</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_cn_cot_cn</td>
    <td>0.52</td>
    <td>0.20</td>
    <td>0.04</td>
    <td>0.38</td>
    <td>0.27</td>
    <td>0.28</td>
    <td>0.14</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_cn_cot_en</td>
    <td>0.48</td>
    <td>0.17</td>
    <td>0.01</td>
    <td>0.31</td>
    <td>0.15</td>
    <td>0.23</td>
    <td>0.09</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1__zero_shot_cn_xlt_en</td>
    <td>0.06</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_en_cot_cn</td>
    <td>0.89</td>
    <td>0.57</td>
    <td>0.05</td>
    <td>0.83</td>
    <td>0.60</td>
    <td>0.51</td>
    <td>0.57</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_en_cot_en</td>
    <td>0.82</td>
    <td>0.33</td>
    <td>0.13</td>
    <td>0.52</td>
    <td>0.48</td>
    <td>0.36</td>
    <td>0.58</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_en_xlt_cn</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_cn_cot_cn</td>
    <td>0.45</td>
    <td>0.27</td>
    <td>0.15</td>
    <td>0.39</td>
    <td>0.33</td>
    <td>0.26</td>
    <td>0.21</td>
    <td>0.29</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_cn_cot_en</td>
    <td>0.44</td>
    <td>0.22</td>
    <td>0.04</td>
    <td>0.35</td>
    <td>0.22</td>
    <td>0.21</td>
    <td>0.35</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1__zero_shot_cn_xlt_en</td>
    <td>0.22</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.07</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.10</td>
    <td>0.07</td>
  </tr>
</table>

Table 17: Performance of Mistral Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>gamma-2-2b__direct_cn</td>
    <td>0.58</td>
    <td>0.33</td>
    <td>0.08</td>
    <td>0.64</td>
    <td>0.55</td>
    <td>0.41</td>
    <td>0.59</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>gamma-2-9b__direct_cn</td>
    <td>0.59</td>
    <td>0.33</td>
    <td>0.06</td>
    <td>0.68</td>
    <td>0.59</td>
    <td>0.47</td>
    <td>0.55</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>gamma-2-27__direct_cn</td>
    <td>0.61</td>
    <td>0.33</td>
    <td>0.09</td>
    <td>0.64</td>
    <td>0.64</td>
    <td>0.44</td>
    <td>0.64</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>gamma-2-2b__direct_en</td>
    <td>0.49</td>
    <td>0.38</td>
    <td>0.05</td>
    <td>0.74</td>
    <td>0.35</td>
    <td>0.35</td>
    <td>0.39</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>gamma-2-9b__direct_en</td>
    <td>0.56</td>
    <td>0.39</td>
    <td>0.04</td>
    <td>0.77</td>
    <td>0.30</td>
    <td>0.36</td>
    <td>0.36</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>gamma-2-27__direct_en</td>
    <td>0.57</td>
    <td>0.42</td>
    <td>0.05</td>
    <td>0.85</td>
    <td>0.28</td>
    <td>0.34</td>
    <td>0.36</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_en_cot_cn</td>
    <td>0.86</td>
    <td>0.47</td>
    <td>0.24</td>
    <td>0.75</td>
    <td>0.55</td>
    <td>0.43</td>
    <td>0.30</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_en_cot_en</td>
    <td>0.69</td>
    <td>0.25</td>
    <td>0.48</td>
    <td>0.59</td>
    <td>0.37</td>
    <td>0.40</td>
    <td>0.80</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_en_xlt_cn</td>
    <td>0.28</td>
    <td>0.11</td>
    <td>0.22</td>
    <td>0.40</td>
    <td>0.43</td>
    <td>0.27</td>
    <td>0.15</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_cn_cot_cn</td>
    <td>0.70</td>
    <td>0.45</td>
    <td>0.31</td>
    <td>0.71</td>
    <td>0.50</td>
    <td>0.33</td>
    <td>0.46</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_cn_cot_en</td>
    <td>0.85</td>
    <td>0.41</td>
    <td>0.33</td>
    <td>0.27</td>
    <td>0.31</td>
    <td>0.26</td>
    <td>0.55</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>gamma-2-2b__few_shot_cn_xlt_en</td>
    <td>0.58</td>
    <td>0.14</td>
    <td>0.67</td>
    <td>0.35</td>
    <td>0.39</td>
    <td>0.24</td>
    <td>0.55</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_en_cot_cn</td>
    <td>0.96</td>
    <td>0.56</td>
    <td>0.80</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.62</td>
    <td>0.79</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_en_cot_en</td>
    <td>0.88</td>
    <td>0.50</td>
    <td>0.79</td>
    <td>0.84</td>
    <td>0.51</td>
    <td>0.55</td>
    <td>0.80</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_en_xlt_cn</td>
    <td>0.33</td>
    <td>0.16</td>
    <td>0.34</td>
    <td>0.51</td>
    <td>0.49</td>
    <td>0.41</td>
    <td>0.40</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_cn_cot_cn</td>
    <td>0.93</td>
    <td>0.47</td>
    <td>0.77</td>
    <td>0.85</td>
    <td>0.80</td>
    <td>0.54</td>
    <td>0.87</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_cn_cot_en</td>
    <td>0.91</td>
    <td>0.46</td>
    <td>0.71</td>
    <td>0.67</td>
    <td>0.50</td>
    <td>0.44</td>
    <td>0.76</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>gamma-2-9b__few_shot_cn_xlt_en</td>
    <td>0.73</td>
    <td>0.25</td>
    <td>0.42</td>
    <td>0.58</td>
    <td>0.41</td>
    <td>0.30</td>
    <td>0.61</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_en_cot_cn</td>
    <td>0.43</td>
    <td>0.33</td>
    <td>0.33</td>
    <td>0.43</td>
    <td>0.42</td>
    <td>0.32</td>
    <td>0.14</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_en_cot_en</td>
    <td>0.34</td>
    <td>0.15</td>
    <td>0.24</td>
    <td>0.27</td>
    <td>0.29</td>
    <td>0.21</td>
    <td>0.11</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_en_xlt_cn</td>
    <td>0.09</td>
    <td>0.04</td>
    <td>0.07</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.04</td>
    <td>0.04</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_cn_cot_cn</td>
    <td>0.30</td>
    <td>0.06</td>
    <td>0.15</td>
    <td>0.21</td>
    <td>0.23</td>
    <td>0.16</td>
    <td>0.22</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_cn_cot_en</td>
    <td>0.34</td>
    <td>0.10</td>
    <td>0.18</td>
    <td>0.23</td>
    <td>0.21</td>
    <td>0.13</td>
    <td>0.21</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>gamma-2-27b__few_shot_cn_xlt_en</td>
    <td>0.14</td>
    <td>0.01</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_en_cot_cn</td>
    <td>0.31</td>
    <td>0.19</td>
    <td>0.08</td>
    <td>0.15</td>
    <td>0.19</td>
    <td>0.09</td>
    <td>0.15</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_en_cot_en</td>
    <td>0.30</td>
    <td>0.28</td>
    <td>0.04</td>
    <td>0.17</td>
    <td>0.20</td>
    <td>0.16</td>
    <td>0.43</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_en_xlt_cn</td>
    <td>0.43</td>
    <td>0.17</td>
    <td>0.11</td>
    <td>0.25</td>
    <td>0.25</td>
    <td>0.15</td>
    <td>0.07</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_cn_cot_cn</td>
    <td>0.10</td>
    <td>0.13</td>
    <td>0.06</td>
    <td>0.08</td>
    <td>0.05</td>
    <td>0.00</td>
    <td>0.03</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_cn_cot_en</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.09</td>
    <td>0.05</td>
    <td>0.07</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>gamma-2-2b__zero_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_en_cot_cn</td>
    <td>0.66</td>
    <td>0.42</td>
    <td>0.06</td>
    <td>0.62</td>
    <td>0.53</td>
    <td>0.34</td>
    <td>0.40</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_en_cot_en</td>
    <td>0.71</td>
    <td>0.37</td>
    <td>0.04</td>
    <td>0.52</td>
    <td>0.27</td>
    <td>0.37</td>
    <td>0.53</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_en_xlt_cn</td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.05</td>
    <td>0.11</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_cn_cot_cn</td>
    <td>0.51</td>
    <td>0.36</td>
    <td>0.09</td>
    <td>0.54</td>
    <td>0.45</td>
    <td>0.29</td>
    <td>0.39</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_cn_cot_en</td>
    <td>0.64</td>
    <td>0.17</td>
    <td>0.03</td>
    <td>0.36</td>
    <td>0.23</td>
    <td>0.23</td>
    <td>0.12</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>gamma-2-9b__zero_shot_cn_xlt_en</td>
    <td>0.33</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.04</td>
    <td>0.12</td>
    <td>0.02</td>
    <td>0.04</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_en_cot_cn</td>
    <td>0.37</td>
    <td>0.18</td>
    <td>0.08</td>
    <td>0.31</td>
    <td>0.29</td>
    <td>0.24</td>
    <td>0.15</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_en_cot_en</td>
    <td>0.21</td>
    <td>0.07</td>
    <td>0.05</td>
    <td>0.17</td>
    <td>0.14</td>
    <td>0.13</td>
    <td>0.12</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_en_xlt_cn</td>
    <td>0.22</td>
    <td>0.13</td>
    <td>0.05</td>
    <td>0.13</td>
    <td>0.14</td>
    <td>0.09</td>
    <td>0.03</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_cn_cot_cn</td>
    <td>0.15</td>
    <td>0.08</td>
    <td>0.05</td>
    <td>0.11</td>
    <td>0.13</td>
    <td>0.00</td>
    <td>0.07</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_cn_cot_en</td>
    <td>0.15</td>
    <td>0.04</td>
    <td>0.04</td>
    <td>0.05</td>
    <td>0.06</td>
    <td>0.04</td>
    <td>0.05</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>gamma-2-27b__zero_shot_cn_xlt_en</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.02</td>
  </tr>
</table>

Table 18: Performance of Gemma Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__direct_cn</td>
    <td>0.59</td>
    <td>0.34</td>
    <td>0.09</td>
    <td>0.65</td>
    <td>0.60</td>
    <td>0.41</td>
    <td>0.55</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__direct_cn</td>
    <td>0.62</td>
    <td>0.37</td>
    <td>0.08</td>
    <td>0.66</td>
    <td>0.60</td>
    <td>0.43</td>
    <td>0.54</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__direct_en</td>
    <td>0.51</td>
    <td>0.38</td>
    <td>0.05</td>
    <td>0.76</td>
    <td>0.31</td>
    <td>0.34</td>
    <td>0.38</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__direct_en</td>
    <td>0.47</td>
    <td>0.40</td>
    <td>0.05</td>
    <td>0.80</td>
    <td>0.30</td>
    <td>0.34</td>
    <td>0.35</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_en_cot_cn</td>
    <td>0.82</td>
    <td>0.49</td>
    <td>0.32</td>
    <td>0.72</td>
    <td>0.62</td>
    <td>0.44</td>
    <td>0.55</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_en_cot_en</td>
    <td>0.67</td>
    <td>0.22</td>
    <td>0.70</td>
    <td>0.56</td>
    <td>0.24</td>
    <td>0.41</td>
    <td>0.68</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_en_xlt_cn</td>
    <td>0.15</td>
    <td>0.01</td>
    <td>0.23</td>
    <td>0.25</td>
    <td>0.39</td>
    <td>0.23</td>
    <td>0.30</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_cn_cot_cn</td>
    <td>0.89</td>
    <td>0.50</td>
    <td>0.54</td>
    <td>0.75</td>
    <td>0.70</td>
    <td>0.53</td>
    <td>0.51</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_cn_cot_en</td>
    <td>0.83</td>
    <td>0.28</td>
    <td>0.75</td>
    <td>0.50</td>
    <td>0.50</td>
    <td>0.33</td>
    <td>0.70</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__few_shot_cn_xlt_en</td>
    <td>0.48</td>
    <td>0.20</td>
    <td>0.22</td>
    <td>0.56</td>
    <td>0.49</td>
    <td>0.33</td>
    <td>0.41</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_en_cot_cn</td>
    <td>0.94</td>
    <td>0.56</td>
    <td>0.71</td>
    <td>0.84</td>
    <td>0.78</td>
    <td>0.60</td>
    <td>0.58</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_en_cot_en</td>
    <td>0.78</td>
    <td>0.31</td>
    <td>0.36</td>
    <td>0.56</td>
    <td>0.48</td>
    <td>0.47</td>
    <td>0.81</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_en_xlt_cn</td>
    <td>0.22</td>
    <td>0.13</td>
    <td>0.39</td>
    <td>0.53</td>
    <td>0.47</td>
    <td>0.30</td>
    <td>0.41</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_cn_cot_cn</td>
    <td>0.89</td>
    <td>0.53</td>
    <td>0.76</td>
    <td>0.86</td>
    <td>0.72</td>
    <td>0.53</td>
    <td>0.61</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_cn_cot_en</td>
    <td>0.86</td>
    <td>0.42</td>
    <td>0.52</td>
    <td>0.57</td>
    <td>0.47</td>
    <td>0.35</td>
    <td>0.87</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__few_shot_cn_xlt_en</td>
    <td>0.83</td>
    <td>0.26</td>
    <td>0.25</td>
    <td>0.49</td>
    <td>0.44</td>
    <td>0.34</td>
    <td>0.74</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_en_cot_cn</td>
    <td>0.31</td>
    <td>0.16</td>
    <td>0.01</td>
    <td>0.24</td>
    <td>0.21</td>
    <td>0.13</td>
    <td>0.03</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_en_cot_en</td>
    <td>0.29</td>
    <td>0.20</td>
    <td>0.01</td>
    <td>0.35</td>
    <td>0.08</td>
    <td>0.22</td>
    <td>0.01</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_en_xlt_cn</td>
    <td>0.19</td>
    <td>0.21</td>
    <td>0.02</td>
    <td>0.17</td>
    <td>0.10</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_cn_cot_cn</td>
    <td>0.42</td>
    <td>0.23</td>
    <td>0.07</td>
    <td>0.38</td>
    <td>0.22</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_cn_cot_en</td>
    <td>0.22</td>
    <td>0.11</td>
    <td>0.11</td>
    <td>0.29</td>
    <td>0.13</td>
    <td>0.16</td>
    <td>0.03</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Base__zero_shot_cn_xlt_en</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_en_cot_cn</td>
    <td>0.41</td>
    <td>0.21</td>
    <td>0.02</td>
    <td>0.25</td>
    <td>0.54</td>
    <td>0.11</td>
    <td>0.21</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_en_cot_en</td>
    <td>0.31</td>
    <td>0.38</td>
    <td>0.05</td>
    <td>0.45</td>
    <td>0.10</td>
    <td>0.35</td>
    <td>0.13</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_en_xlt_cn</td>
    <td>0.23</td>
    <td>0.19</td>
    <td>0.01</td>
    <td>0.28</td>
    <td>0.20</td>
    <td>0.05</td>
    <td>0.10</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_cn_cot_cn</td>
    <td>0.41</td>
    <td>0.17</td>
    <td>0.07</td>
    <td>0.32</td>
    <td>0.30</td>
    <td>0.16</td>
    <td>0.01</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_cn_cot_en</td>
    <td>0.13</td>
    <td>0.10</td>
    <td>0.06</td>
    <td>0.10</td>
    <td>0.14</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Base__zero_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
</table>

Table 19: Performance of Baichuan Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Model_Prompt_Language</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__direct_cn</td>
    <td>0.62</td>
    <td>0.33</td>
    <td>0.09</td>
    <td>0.63</td>
    <td>0.52</td>
    <td>0.37</td>
    <td>0.38</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__direct_cn</td>
    <td>0.71</td>
    <td>0.34</td>
    <td>0.08</td>
    <td>0.66</td>
    <td>0.60</td>
    <td>0.43</td>
    <td>0.67</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__direct_cn</td>
    <td>0.69</td>
    <td>0.36</td>
    <td>0.08</td>
    <td>0.70</td>
    <td>0.66</td>
    <td>0.42</td>
    <td>0.45</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__direct_en</td>
    <td>0.55</td>
    <td>0.33</td>
    <td>0.08</td>
    <td>0.63</td>
    <td>0.41</td>
    <td>0.35</td>
    <td>0.35</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__direct_en</td>
    <td>0.59</td>
    <td>0.39</td>
    <td>0.06</td>
    <td>0.76</td>
    <td>0.36</td>
    <td>0.32</td>
    <td>0.37</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__direct_en</td>
    <td>0.62</td>
    <td>0.41</td>
    <td>0.05</td>
    <td>0.78</td>
    <td>0.35</td>
    <td>0.31</td>
    <td>0.37</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_en_cot_cn</td>
    <td>0.89</td>
    <td>0.47</td>
    <td>0.39</td>
    <td>0.83</td>
    <td>0.73</td>
    <td>0.45</td>
    <td>0.40</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_en_cot_en</td>
    <td>0.84</td>
    <td>0.32</td>
    <td>0.37</td>
    <td>0.56</td>
    <td>0.39</td>
    <td>0.45</td>
    <td>0.42</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_en_xlt_cn</td>
    <td>0.44</td>
    <td>0.27</td>
    <td>0.47</td>
    <td>0.48</td>
    <td>0.53</td>
    <td>0.35</td>
    <td>0.40</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_cn_cot_cn</td>
    <td>0.86</td>
    <td>0.43</td>
    <td>0.55</td>
    <td>0.80</td>
    <td>0.67</td>
    <td>0.44</td>
    <td>0.50</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_cn_cot_en</td>
    <td>0.80</td>
    <td>0.44</td>
    <td>0.20</td>
    <td>0.49</td>
    <td>0.41</td>
    <td>0.45</td>
    <td>0.67</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__few_shot_cn_xlt_en</td>
    <td>0.62</td>
    <td>0.20</td>
    <td>0.17</td>
    <td>0.48</td>
    <td>0.47</td>
    <td>0.35</td>
    <td>0.42</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_en_cot_cn</td>
    <td>0.78</td>
    <td>0.65</td>
    <td>0.88</td>
    <td>0.84</td>
    <td>0.87</td>
    <td>0.63</td>
    <td>0.77</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.52</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.53</td>
    <td>0.53</td>
    <td>0.88</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_en_xlt_cn</td>
    <td>0.62</td>
    <td>0.41</td>
    <td>0.61</td>
    <td>0.57</td>
    <td>0.59</td>
    <td>0.51</td>
    <td>0.42</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_cn_cot_cn</td>
    <td>0.95</td>
    <td>0.52</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.85</td>
    <td>0.61</td>
    <td>0.84</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_cn_cot_en</td>
    <td>0.91</td>
    <td>0.60</td>
    <td>0.75</td>
    <td>0.77</td>
    <td>0.47</td>
    <td>0.54</td>
    <td>0.82</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__few_shot_cn_xlt_en</td>
    <td>0.78</td>
    <td>0.31</td>
    <td>0.48</td>
    <td>0.77</td>
    <td>0.40</td>
    <td>0.38</td>
    <td>0.62</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_en_cot_cn</td>
    <td>0.62</td>
    <td>0.58</td>
    <td>0.92</td>
    <td>0.28</td>
    <td>0.00</td>
    <td>0.48</td>
    <td>0.79</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_en_cot_en</td>
    <td>0.91</td>
    <td>0.63</td>
    <td>0.85</td>
    <td>0.88</td>
    <td>0.53</td>
    <td>0.55</td>
    <td>0.82</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_en_xlt_cn</td>
    <td>0.66</td>
    <td>0.38</td>
    <td>0.63</td>
    <td>0.42</td>
    <td>0.54</td>
    <td>0.59</td>
    <td>0.18</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_cn_cot_cn</td>
    <td>0.00</td>
    <td>0.27</td>
    <td>0.91</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.57</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_cn_cot_en</td>
    <td>0.91</td>
    <td>0.65</td>
    <td>0.79</td>
    <td>0.86</td>
    <td>0.51</td>
    <td>0.56</td>
    <td>0.84</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__few_shot_cn_xlt_en</td>
    <td>0.74</td>
    <td>0.46</td>
    <td>0.90</td>
    <td>0.83</td>
    <td>0.53</td>
    <td>0.44</td>
    <td>0.69</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_en_cot_cn</td>
    <td>0.51</td>
    <td>0.22</td>
    <td>0.03</td>
    <td>0.42</td>
    <td>0.31</td>
    <td>0.26</td>
    <td>0.17</td>
    <td>0.27</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_en_cot_en</td>
    <td>0.15</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.16</td>
    <td>0.14</td>
    <td>0.13</td>
    <td>0.28</td>
    <td>0.14</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_en_xlt_cn</td>
    <td>0.06</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.05</td>
    <td>0.03</td>
    <td>0.00</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_cn_cot_cn</td>
    <td>0.67</td>
    <td>0.42</td>
    <td>0.17</td>
    <td>0.51</td>
    <td>0.40</td>
    <td>0.50</td>
    <td>0.34</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_cn_cot_en</td>
    <td>0.45</td>
    <td>0.32</td>
    <td>0.20</td>
    <td>0.32</td>
    <td>0.32</td>
    <td>0.28</td>
    <td>0.47</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b__zero_shot_cn_xlt_en</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.08</td>
    <td>0.07</td>
    <td>0.03</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_en_cot_cn</td>
    <td>0.78</td>
    <td>0.65</td>
    <td>0.88</td>
    <td>0.84</td>
    <td>0.87</td>
    <td>0.63</td>
    <td>0.77</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_en_cot_en</td>
    <td>0.93</td>
    <td>0.52</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.53</td>
    <td>0.53</td>
    <td>0.88</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_en_xlt_cn</td>
    <td>0.15</td>
    <td>0.08</td>
    <td>0.12</td>
    <td>0.12</td>
    <td>0.18</td>
    <td>0.11</td>
    <td>0.14</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_cn_cot_cn</td>
    <td>0.95</td>
    <td>0.52</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.85</td>
    <td>0.61</td>
    <td>0.84</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_cn_cot_en</td>
    <td>0.91</td>
    <td>0.60</td>
    <td>0.75</td>
    <td>0.77</td>
    <td>0.47</td>
    <td>0.54</td>
    <td>0.82</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>internlm2_5-7b__zero_shot_cn_xlt_en</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_en_cot_cn</td>
    <td>0.39</td>
    <td>0.36</td>
    <td>0.15</td>
    <td>0.48</td>
    <td>0.55</td>
    <td>0.31</td>
    <td>0.22</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_en_cot_en</td>
    <td>0.48</td>
    <td>0.28</td>
    <td>0.11</td>
    <td>0.48</td>
    <td>0.41</td>
    <td>0.25</td>
    <td>0.75</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_en_xlt_cn</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.33</td>
    <td>0.54</td>
    <td>0.42</td>
    <td>0.26</td>
    <td>0.59</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_cn_cot_cn</td>
    <td>0.55</td>
    <td>0.53</td>
    <td>0.27</td>
    <td>0.71</td>
    <td>0.56</td>
    <td>0.00</td>
    <td>0.71</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_cn_cot_en</td>
    <td>0.54</td>
    <td>0.47</td>
    <td>0.03</td>
    <td>0.69</td>
    <td>0.43</td>
    <td>0.42</td>
    <td>0.30</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>internlm2_5-20b__zero_shot_cn_xlt_en</td>
    <td>0.80</td>
    <td>0.18</td>
    <td>0.02</td>
    <td>0.64</td>
    <td>0.43</td>
    <td>0.32</td>
    <td>0.89</td>
    <td>0.47</td>
  </tr>
</table>

Table 20: Performance of InternLM Series. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
<table>
  <tr>
    <th>Instruct-Model</th>
    <th>CI</th>
    <th>CT</th>
    <th>NT</th>
    <th>PR</th>
    <th>RC</th>
    <th>SR</th>
    <th>SO</th>
    <th>ARA</th>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>0.58</td>
    <td>0.36</td>
    <td>0.07</td>
    <td>0.65</td>
    <td>0.41</td>
    <td>0.36</td>
    <td>0.58</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>0.64</td>
    <td>0.36</td>
    <td>0.07</td>
    <td>0.70</td>
    <td>0.41</td>
    <td>0.37</td>
    <td>0.49</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>0.66</td>
    <td>0.40</td>
    <td>0.05</td>
    <td>0.75</td>
    <td>0.41</td>
    <td>0.40</td>
    <td>0.53</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>0.74</td>
    <td>0.39</td>
    <td>0.06</td>
    <td>0.76</td>
    <td>0.44</td>
    <td>0.39</td>
    <td>0.64</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>0.78</td>
    <td>0.40</td>
    <td>0.06</td>
    <td>0.77</td>
    <td>0.44</td>
    <td>0.40</td>
    <td>0.74</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>0.75</td>
    <td>0.40</td>
    <td>0.06</td>
    <td>0.78</td>
    <td>0.45</td>
    <td>0.40</td>
    <td>0.79</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>0.78</td>
    <td>0.41</td>
    <td>0.06</td>
    <td>0.79</td>
    <td>0.48</td>
    <td>0.41</td>
    <td>0.80</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B</td>
    <td>0.65</td>
    <td>0.38</td>
    <td>0.07</td>
    <td>0.71</td>
    <td>0.34</td>
    <td>0.40</td>
    <td>0.52</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B</td>
    <td>0.68</td>
    <td>0.40</td>
    <td>0.06</td>
    <td>0.73</td>
    <td>0.37</td>
    <td>0.41</td>
    <td>0.57</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.2</td>
    <td>0.67</td>
    <td>0.41</td>
    <td>0.07</td>
    <td>0.76</td>
    <td>0.35</td>
    <td>0.38</td>
    <td>0.58</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-v0.1</td>
    <td>0.65</td>
    <td>0.41</td>
    <td>0.07</td>
    <td>0.76</td>
    <td>0.41</td>
    <td>0.40</td>
    <td>0.52</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Mistral-8x22B-v0.1</td>
    <td>0.70</td>
    <td>0.43</td>
    <td>0.05</td>
    <td>0.78</td>
    <td>0.42</td>
    <td>0.39</td>
    <td>0.57</td>
    <td>0.48</td>
  </tr>
</table>

Table 21: Performance of instruct models under Direct Prompt. And NT, CT, CI, SO, SR, RC, PR are the abbreviations for the variant names of Negation Transformation, Critical Testing, Causal Inference, Sentence Ordering, Scenario Refinement, Reverse Conversion and Problem Restatement.
HellaSwag-Pro Dataset Format

{
    "original_context": "A large group of people are seen standing around a beach as well as several shots of cars and people riding bulls. various people"
    "original_choices": ["are seen seen diving into the water, hitting the bulls back and fourth as well as playing a game of volleyball and cheering along", "then run to the bull and the bull fights them off while one stands by and watches.", "are shown speaking to the camera and others riding bulls around one another.", "ride the bulls and sit in the cars as well as end with a game of volleyball and celebrating."],
    "original_label": 3,
    "perturbation_type": "reverse_conversion",
    "context": "Various people ride the bulls and sit in the cars as well as end with a game of volleyball and celebrating. Which could be the most possible context for this action?",
    "choices": ["A large group of people are seen standing around a beach as well as several shots of cars and people riding bulls.", "A crowd gathers at a local park for a community event featuring live music and food trucks.", "Tourists explore a busy marketplace, taking photos and buying souvenirs.", "Children play in a playground while parents watch from nearby benches."],
    "label": 0
}

Figure 8: An example of HellaSwag-Pro.

Chinese HellaSwag Dataset Format

{
    "context": "丽丽报名参加了日本京都的一趟文化之旅，深度体验了传统艺伎表演。她",
    "choices": ["学习了传统的日式剑道和弓道技巧","欣赏了京都著名的樱花季和红叶景观","深深地被茶道的精致仪式所吸引。","品尝了正宗的关西风味章鱼烧和大阪烧"],
    "label": 2,
    "broad_type": "休闲娱乐",
    "detailed_type": "旅游体验"
}
{
    "context": "Lili signed up for a cultural tour in Kyoto, Japan, and experienced a traditional geisha performance. She",
    "choices": ["learned traditional Japanese kendo and archery skills", "enjoyed Kyoto's famous cherry blossom season and red leaves", "deeply attracted by the exquisite rituals of the tea ceremony", "tasted authentic Kansai-style takoyaki and okonomiyaki"],
    "label": 2,
    "broad_type": "Leisure",
    "detailed_type": "Travel Experience"
}

Figure 9: An example of Chinese HellaSwag.

Prompt for Chinese HellaSwag Construction

Type requirements:
You are a Chinese teacher with rigorous logic and rich common sense. Please help me write a question about commonsense reasoning. Each question contains an incomplete context and ten options. The context describes a common [broad_type] [detailed_type] scenario in the Chinese context. The sentence ends with an entity, such as "she", "this man", "they", "Zhang San", etc. This entity has rich [detailed_type] knowledge scenario [detailed_type_definition].
The content in the options is the scenarios that may occur in this context, but only the first option is the correct option, which is possible in reality, while the other nine contain logical errors or are not applicable to the context scenario or contradict common sense, but do not contain supernatural phenomena.
The questions are returned in json format, similar to the following sample. Note that the attribute name must be contained in double quotes.
{5-shot examples}
Length requirements:
The context field should be {less than 20 words}.
The choice field should have similar words. You should be as creative as possible and generate as many questions as possible.
Pay attention to the fluency of the text, the clarity of the meaning, and the correctness of the grammar.

Figure 11: Prompt for Chinese HellaSwag construction.

Prompt For HellaSwag-Pro Construction

Total requirement:
Suppose you are a case generator. Given original_context, original_choices, original_label, your goal is to generate context, choices, label and explanation according to perturbation type. Your output should be in the following format: {original_context, original_choices, original_label, perturbation_type, context, choices, label and explanation}. I will provide some examples, and you should imitate my case generation process. You can be consistent with the perturbation_type provided to you.

{5-shot examples.}

Specific Variant Definition:
I hope you will concatenate the original_context and original_choices corresponding to the original_label into a complete paragraph and turn it into context; ending with Which could be the possible reason for the choice? and then explain the reason for such choice containing common sense as the correct option in choices. The correct option should be as concise as possible, and generate 9 other obviously wrong options according to the format and length of this option. The wrong option should contain wrong common sense. Put the correct option in the position of the first option and mark the label as 0. Note that context and choices should be fluent. The conversion process hopes to infer the possible reasons through the context and choices. So, how to convert the following prompt:
original_context: {} |original_choices: {}|original_label: {}

Figure 10: Prompt for HellaSwag-Pro construction.

Reviewer paper summaries: This paper presents a new large-scale bilingual benchmark based on HellaSwag for comprehensive and robust evaluation of LLMs in commonsense reasoning. With the help of LLMs, authors construct a Chinese version of HellaSwag and then extend samples in these two benchmarks with several variants designed to test the robust commonsense reasoning abilities. Authors conduct an evaluation of more than 40 models on their new benchmark and show that LLMs still struggle to robustly reason about commonsense knowledge.

This paper documents the creation of a new benchmark dataset for commonsense reasoning called HellaSwag-Pro, which consists of 7 types of adversarial variants for examples in both English and Chinese versions of HellaSwag (the latter also being created by the authors in this work). Altogether, this effort results in 11,200 4-way multiple-choice questions with adversarial variants, as well as 12,000 source questions for Chinese HellaSwag which were generated by LLMs across 56 categories. In addition to compiling HellaSwag-Pro and Chinese HellaSwag, the paper presents extensive experiments on 41 LLMs which measure their robustness across compiled variants, languages (of both data and models), and chain-of-thought style prompting strategies. The authors identify various trends in their results, but the strongest takeaway is that LLMs achieve high performance on source HellaSwag data (English and Chinese), but are largely not robust to adversarial variants of examples, which change the form of example content but not the meaning. This inflated performance may suggest that some LLMs have been trained directly on data in (English) HellaSwag, or more generally exhibit inconsistent understanding depending on the syntactic form of commonsense problems. The paper includes detailed fine-grained analyses of adversarial variant types and prompting configurations to bring further insights about these results.

This paper presents a substantial effort and the datasets collected would be valuable to the research community. However, I have some concerns, primarily about a lack of thoughtfulness in motivation and framing of the work and some experiments, as well as the perpetuation of data leakage in LLMs by just producing another dataset. Despite my concerns, this paper will be a strong contribution to work in commonsense reasoning evaluations of LLMs (after some revisions).
Summary of strengths: 1. Paper is well-written and appropriately structured.
2. Benchmark construction methodology based on minimally modifying the question and the answer choices is quite reasonable and valuable to test robustness in commonsense reasoning.
3. New benchmark is large-scale, high-quality (verified by human annotators) and supports two languages.
4. Authors have conducted a comprehensive evaluation of humans and more than 40 models on their benchmark and revealed systematic gaps in robust commonsense reasoning.
5. Authors have performed extensive analysis with different prompting strategies and question variants.

**Strength A: New useful dataset.** This paper proposes a dataset that shows how LLMs are inconsistent commonsense reasoners, opening new avenues for model improvement in future work. It is rare to find benchmark datasets that LLMs can’t solve to high accuracy, and thus 

**Strength B: Sound and thorough data collection process.** The authors took several steps to ensure diversity and quality of the (largely LLM-generated) dataset. While there are some details missing related to human annotation and data quality (see Question E under “Summary of Weaknesses”), at first glance this dataset seems to be a high-quality, valuable resource.

**Strength C: Thorough experiments.** The paper establishes baselines from an impressive breadth of common LLMs at various sizes, including both English and Chinese LLMs, and open- and closed-source LLMs. Furthermore, it includes additional analyses on the types of adversarial variants that LLMs found most challenging, as well as experiments with various types of in-context learning and chain-of-thought prompting strategies. While I have some issues with how the results are presented, this is a substantial experimental effort that strengthens the contribution of the paper.
Summary of weaknesses: There are no major weaknesses with this work. However, no details have been provided about the human evaluation. How many samples were evaluated by humans? How many humans evaluated each sample? How were they compensated?

**Weakness A: Various issues with framing and motivation of the work.** First, I wanted to highlight various related issues around how the paper motivates and frames its contributions:

1. The motivation of the work presented in the abstract and Section 1 feels ill-defined and circular. For example, at L009 and L044, the authors highlight that it’s unclear whether LLMs’ high performance on commonsense benchmarks is due to a “genuine understanding of commonsense knowledge, or … memorizing superficial patterns” (L043-044). I would like to better understand what the authors mean by this… Why would they memorize superficial patterns? Are they implying that LLMs have been pre-trained on benchmark data (which would only be possible for the pre-existing English HellaSwag data)? Or perhaps that LLMs, as models of the distribution of words, exhibit inconsistent results depending on the specific form commonsense knowledge is expressed in? Being specific here would help. Related to this, the sentence beginning at L005 in the abstract says “In fact, LLMs are non-robust to the question variants with the same commonsense knowledge but different reasoning forms, a phenomenon not fully investigated by existing studies.” This implies that the authors already know the answer to their research question, and should be rephrased (as it sounds circular). While I know that this statement is based on the example in Figure 1, this was not an arbitrarily selected example - it used adversarial filtering, and thus was generated specifically to fool LLMs (so this is not surprising). As I already said, I would instead motivate the work with why we think LLMs might be “memorizing superficial patterns” or exhibiting only surface-level understanding.

2. The third paragraph of Section 1 completely minimizes the broad space of work on robustness and consistency in language models (both within and beyond commonsense). Some references that are missing here and in the Related Work are [1,2,3,4,5], and additional references may be found in those papers’ Related Work sections. To ensure the paper is adequately contextualized, I would strongly recommend that the authors take a closer look at work in this space, and think about how their work contributes to it.


3. At L071 and other places, the authors claim that their dataset is “the first comprehensive evaluation on the robustness of commonsense reasoning for LLMs.” The dataset only covers 8 types of adversarial variations, and its content is limited to the domains of the source data (only on the order of tens of thousands of instances). Commonsense knowledge and reasoning is believed to be impossibly vast, and thus while I do believe this is the first work that created a robust commonsense dataset with this much breadth, I do not believe that it can be considered “comprehensive.”

4. The paper claims to draw inspiration from the Bloom Cognitive Model in creating the dataset. While this sounds interesting and principled, the connection is not obvious to me after looking more into this model. Could the authors provide more details on this? I would also recommend including this information in the appendix.

5. At L501, the authors claim that “research indicates that when LLMs are utilized for database generation, the resulting datasets are more accurate and fluent … than those created by crowd-sourced annotators.” This claim sounded fishy to me, so I dug into the references here (Lu et al, 2022 and Min et al., 2022). As I expected, neither paper demonstrates this. Both papers study aspects of prompts used in in-context learning settings, not LLM-generated datasets. Lu et al. (2022) discusses in its related work section that automating prompt design yields better performance than human-crafted prompts (which is not the same as saying that LLM-generated data is more accurate and fluent than humans). Min et al. (2022) does not even mention humans or crowd-sourced annotations in the body of the paper. I’m not sure where the authors got the impression that these references said that, but hopefully this was an honest mistake or mixup of references. If I’m not missing something, this egregious mischaracterization needs to be removed.

**Weakness B: May perpetuate issues in language modeling.** This paper applies adversarial filtering to create examples that LLMs struggle to solve, and uses it to show that LLMs can exhibit inconsistent understanding across various forms of the same commonsense knowledge. While the paper seems to be motivated by LLMs memorizing spurious patterns in data, what may ultimately happen is that this dataset is directly used to train future LLMs so that they exhibit a consistent understanding. This would necessitate the creation of another dataset like this, perpetuating the underlying issues in language models, which this paper offers no solution to. If the test data is going to be publicly released, I would like to see some discussion of this issue in the paper, including a note about how the authors hope/expect the dataset to be used in the future (e.g., for evaluation only, not training).

Related to this, the authors claim to “introduce a human-in-the-loop adversarial filtering method” (L183-184), implying some novelty. In fact, the paper that proposed adversarial filtering (Zellers et al., 2018) also applied human verification, so this is not new. I urge the authors to rephrase this sentence.

**Weakness C: Conclusions on model size in RQ1 results are not well justified.** In the paragraph beginning at L378, the authors observe that while larger models have higher scores, they also have higher RLA values (indicating lack of robustness). However, I do not see such a trend in Table 3. For Qwen2.5, InternLM, and Gemma, the OA and ARA scores do increase with model size, but the RLA score seems to go up and down. For Yi-1.5, this trend does seem to hold, but the variation in all metrics is quite small (RLA only varies by less than 1%), so the significance is unclear. The other models only have no more than 2 data points, so it is difficult to make such conclusions. I would recommend that the authors remove this conclusion, as it is not justified.

**Weakness D: Unclear motivation and conclusions around RQ3.** I appreciated the authors’ effort to organize their results around research questions (RQs). However, RQ3 (*How robust are LLMs to diverse prompts during evaluation?*) is super vague and does not really connect with the experiments that are performed, which include various types of few-shot and chain-of-thought prompts in both English and Chinese. I do not think the question here is whether LLMs are robust to these variations, but instead what type of in-context learning configuration yields the best robustness in LLMs. I would like to see some better motivation for the specific types of approaches tried. For example, at L451, the authors conclude that “conducting CoT reasoning in the LLM’s native language generally yields better results.” Why wouldn’t it? Further, as I mention later in “Comments Suggestions And Typos”, the way these results are presented (i.e., in Table 4, which has many acronyms and various types of results mixed up together) makes it very hard to confirm the observations the authors make when discussing the results.

**Question E: Details on data quality and human annotation.** I have some slight concerns on the quality of the data collected. While I’m aware that “professional NLP task annotators” (L941) filtered generated questions, data filtering is not a cognitively demanding task, and I worry that there could still be a large number of low-quality examples in the dataset. While the human performance measurement in Table 3 shows that humans can get high accuracy on the task, and thus the data quality must be decent, I can’t find details on how human performance is collected. Is human performance collected from the same annotators that worked on the dataset, and are these annotators also authors of the paper? Someone who is very familiar with the dataset may overlook poor quality examples, or it may be possible that poor quality examples (which fool LLMs) are actually still easy for humans to solve. I would like to see more details about the human performance measurement in the paper, and I also wonder whether it is necessary to perform one last data quality check with a pool of annotators that did not work on the dataset (possibly including inter-annotator agreement measurements). This is listed as a Question instead of a Weakness since I don’t have sufficient details - looking forward to getting more details in the author-reviewer discussion.

[1] Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. https://aclanthology.org/D17-1215/

[2] Ruben Branco, António Branco, João António Rodrigues, and João Ricardo Silva. 2021. Shortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1504–1521, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. https://aclanthology.org/2021.emnlp-main.113/ 

[3] Shane Storks and Joyce Chai. 2021. Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3169–3177, Punta Cana, Dominican Republic. Association for Computational Linguistics. https://aclanthology.org/2021.findings-emnlp.272/

[4] Alexis Ross, Matthew Peters, and Ana Marasovic. 2022. Does Self-Rationalization Improve Robustness to Spurious Correlations?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7403–7416, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. https://aclanthology.org/2022.emnlp-main.501/


[5] Jacob K. Johnson and Ana Marasović. 2023. How Much Consistency Is Your Accuracy Worth?. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 250–260, Singapore. Association for Computational Linguistics. https://aclanthology.org/2023.blackboxnlp-1.19/
Comments, suggestions and typos: 1. I think you should consider citing CRoW (https://aclanthology.org/2023.emnlp-main.607/) which also presented a benchmark constructed with a similar methodology (applying perturbations to examples in existing datasets) and designed to test the robust commonsense reasoning abilities of LLMs.
2. Why didn't you simply translate HellaSwag to Chinese, but instead opted to construct a similar benchmark from scratch?

**General/higher-level comments:**

Tables 3 and 4 are *very* hard to read and get insights from with so many results. For Table 3, at minimum, there should be more sub-division of rows, e.g., first divide into sections by language/openness, then divide by model size/version. For Table 4, all the different variants and abbreviations are so hard to compare. I would recommend to remove some of these abbreviations and instead have columns for each aspect, e.g., CoT language, # shots, LLM language, etc. To save space, you can remove the first column and instead use the dataset names as “section headers” like you did in Table 3 (e.g., “Closed-source LLMs vs.”Chinese open-source LLMs” vs. “English open-source LLMs”). If possible, it would be great to break tables down into one or more graphs, e.g., to show trends with respect to model size in Table 3.

Additionally, is it possible to add a random or majority-class baseline row to tables? It would be helpful to know the lower bounds of the proposed ARA, RLA, and CRA metrics and compare to results.

Figure 4 and the discussion about it at L395-403 are a bit confusing. First, the graphs say “Hella” and “HellaSwag”, not “HellaSwag” and “HellaSwag-Pro” as mentioned in the discussion paragraph - please make sure you’re consistent here. It would actually be most appropriate to move the explanation of the graph legend to the Figure 4 caption, and make the discussion in this paragraph more natural rather than quoting the graph repeatedly (which makes it hard to parse). For example, rather than saying “For all LLMs, ‘HellaSwag✓ HellaSwag-ProX’ occupy a significant proportion, indicating a lack of robustness for current LLMs,” you can say something like “For all LLMs, a significant proportion of HellaSwag-Pro variants were answered incorrectly despite LLMs being able to solve the source example.” (No need to copy that verbatim, I leave it up to you how to explain.)

In Section 4.3, you suddenly switch to using bullet points to introduce results. I found this a bit choppy as a reader, and would suggest using separate paragraphs as you had done before.

A lot of your references appear to be arXiv papers. Where possible, please replace these with references to the publication venues/proceedings these papers appeared in.

**Specific comments (typos, etc.):**

It’s possible I missed some typos, so please use my feedback to make another pass and correct common issues where needed.

L001: “the” -> “their”

L006: “the” not needed before “question variants.”

L020: “resentative” -> “representative”

Figure 1 caption: The terms “negative transformation,” “scenario refinement,” etc. are not yet defined, so it’s a bit confusing to the reader to see this on the first page. I may suggest expanding the caption slightly to refer to “its variants” as “several meaning-preserving variants” or something like this.

L035: Not necessary, but I myself prefer to avoid back-to-back and nested parentheses using \citealp in LaTeX.

L052: No need for “partly” here.

L151: “LLM” to “the LLM”

L168: “selects” -> “select”

L200: “the Appendix  D” -> “Appendix D” (be sure to remove extra space)

L230: “LLM” -> “LLMs”

L314, L316, L318, maybe other places: Please stylize the package names somehow when mentioning them, e.g., bold, italics, or \texttt (I tend to use the latter). It feels harder to parse when they’re in plain text.

L318: “default value of the hugging-face library” … so, zero? Be specific and provide the value.

L324-331: Please add spaces after numbers.

L367: Maybe say “near-zero” instead of “negative”, as this doesn’t seem to be *significantly* negative.

L369: “human” -> “humans”

L398: “occupy” -> “occupies”?

L400: “Detailed speaking” does not make sense - maybe “More specifically”?

L431-433: I found the phrase about “tiny data proportion” to be confusing. It also may need support - is there any prior work suggesting that these two types are not represented in the data? If not, make sure you’re clear that this is just speculation. Additionally, the claim about the “intrinsic challenge of counterfactual reasoning” is unsupported. Either cite something, or again make it clear this is speculation.

L449: This sentence has a mix of plural and singular words. A possible rewrite would be “Besides, translating datasets into LLMs’ native languages before reasoning does not enhance performance.”

Figure 6 caption: Please spell out these acronyms to avoid the reader having to go back in the paper and find them.

L466: “targets” -> “target”

L474: “Notable” -> “Notably”

L1206: “casual” -> “causal”
Overall assessment scores: 4

3.5
Metareview score: 4
Metareview: The paper introduces HellaSwag-Pro, a large-scale bilingual benchmark for evaluating the robustness of commonsense reasoning in LLMs. The benchmark includes a newly created Chinese version of HellaSwag along with adversarial variants of both the original English examples and the new Chinese examples. The authors conducted a comprehensive evaluation of 41 LLMs of different sizes and families with multiple prompting methods. The results reveal valuable insights regarding the commonsense reasoning skills of LLMs, in part that (a) LLMs are largely not robust to adversarial variants, suggesting that their understanding of commonsense problems is inconsistent across different syntactic forms of the input, (b) the results may suggest that some LLMs have been trained directly on instances from the English HellaSwag. 

Overall, the paper makes solid contributions. It introduces a valuable large-scale benchmark, the data construction process is sound and thorough, and the experiments are comprehensive and reveal systematic gaps.

At the same time, the paper could benefit substantially from the revisions suggested by the reviewers. These include revisions of the positioning of the paper and motivation, extending discussions on potential impact and usage, tuning down some of the arguments, extending related work, and adding missing details about the human evaluation. During the rebuttal, the authors have addressed most of the reviewers’ concerns, through clarification and extensive discussions.

I have been contemplating whether to assign 3 or 4 here, as it is not clear to me whether making all these important revisions could be considered “minor”. Given the positive scores by the reviewers, the value of the benchmark to the community, and the fact that the suggestions do not seem to require additional experiments or analysis but mostly focus on concrete writing adjustments, I believe that in terms of contributions the paper is ready for publication. However, I strongly encourage the authors to keep their promise and incorporate the suggested revisions in the paper to enhance its overall quality.
Summary of reasons to publish: - The paper introduces a challenging  large-scale bilingual benchmark for the robustness of commonsense reasoning in LLMs
- The data construction process is sound and thorough
- The experiments are comprehensive and reveal systematic gaps