MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
Authors: Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, Furu Wei
Accepted at: acl25

MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark

Qihao Zhao   Yangyu Huang*   Tengchao Lv   Lei Cui   Furu Wei
Qinzhen Sun   Ying Xin   Shaoguang Mao   Xin Zhang   Qiufeng Yin   Scarlett Li
Microsoft Research

Abstract
Multiple-choice question (MCQ) datasets like Massive Multitask Language Understanding (MMLU) are widely used to evaluate the commonsense, understanding, and problem-solving abilities of large language models (LLMs). However, the open-source nature of these benchmarks and the broad sources of training data for LLMs have inevitably led to benchmark contamination, which is studied in our contamination evaluation experiment, resulting in unreliable evaluation. To alleviate this issue, we propose the contamination-free MCQ benchmark called MMLU-CF, which reassesses LLMs’ understanding of world knowledge by averting both unintentional and deliberate data contamination. To mitigate unintentional data contamination, we source questions from a broader domain of over 200 billion webpages and apply three specifically designed decontamination rules. To prevent deliberate data contamination, we divide the benchmark into validation and test sets with similar difficulty and subject distributions. The test set remains closed-source to ensure reliable results, while the validation set is publicly available to promote transparency and facilitate independent evaluation. We evaluated over 40 mainstream LLMs on the MMLU-CF. Compared to the original MMLU, not only LLMs’ performances significantly dropped but also the performance rankings of them changed considerably. This indicates the effectiveness of our approach in establishing a contamination-free and fairer evaluation standard. The GitHub repository is available at https://github.com/microsoft/MMLU-CF and the dataset refers to https://huggingface.co/datasets/microsoft/MMLU-CF.

1 Introduction
Given the emergence of powerful capabilities in large language models (LLMs) such as GPT-4

![An instance of contamination in MMLU. When questions are used as prompts from the MMLU, certain LLMs, due to their memorization capabilities, directly provide choices identical to the original ones. (b) When questions are used as prompts from the MMLU-CF, LLMs only provide guessed choices. This indicates that the MMLU test set suffers from data contamination and memorization by some LLMs, while the proposed MMLU-CF avoids such leakage. Further details are analyzed in Section 4.5 of Appendix.](page_1016_370_388_388.png)

Figure 1: (a) An instance of contamination in MMLU. When questions are used as prompts from the MMLU, certain LLMs, due to their memorization capabilities, directly provide choices identical to the original ones. (b) When questions are used as prompts from the MMLU-CF, LLMs only provide guessed choices. This indicates that the MMLU test set suffers from data contamination and memorization by some LLMs, while the proposed MMLU-CF avoids such leakage. Further details are analyzed in Section 4.5 of Appendix.

(Achiam et al., 2023), Llama (Meta, 2024), Gemini (Reid et al., 2024), and Claude-3 (Anthropic, 2023), evaluation of these models has become particularly important for understanding their strengths and limitations. Consequently, a number of benchmarks covering reasoning (Hendrycks et al.; Wang et al., 2024), reading comprehension, mathematics (Cobbe et al., 2021), science (Rein et al., 2023), and coding (Yu et al., 2023) have been explored and released. Among them, Massive Multitask Lan-
Figure 2: The 5-shot results on the MMLU-CF test set encompass mainstream open-source models ranging from 0.5 billion (B) to 176 billion (B) parameters, including closed-source API models.

guage Understanding (MMLU) (Hendrycks et al.) is a widely used multiple-choice question (MCQ) gold standard benchmark because it covers various disciplines and difficulty levels, allowing for a comprehensive evaluation of LLMs’ performance across diverse domains.

However, data leakage or contamination, where LLMs inadvertently encounter benchmark data during training, can compromise the effectiveness, reliability, and fairness of these evaluations (Deng et al., 2024; Roberts et al., 2023; Srivastava et al., 2024), termed unintentional contamination. Additionally, due to the public availability of benchmarks and the ability of LLMs to memorize data (Carlini et al., 2023), instances of deliberate contamination may occur. As illustrated in Figure 1, we observe that when only given the questions, some LLMs directly provide the choices and answers, where the choices are exactly the same as those in the MMLU test set. This indicates that the benchmark may have been deliberately added to the training set and that LLMs have memory for these questions.

To fairly investigate the world knowledge of LLMs, we propose *MMLU-CF*, a contamination-free multiple-choice question benchmark for LLMs. To minimize the risk of benchmark exposure and contamination, we perform five key processing steps for the data: (1) MCQ Collection, (2) MCQ Cleaning, (3) Difficulty Sampling, (4) LLMs Checking, and (5) Contamination-Free Processing. In the contamination-free processing step, we employ three rules to rewrite the questions, which could be referred to as Section 3.2. For humans, rewriting questions without changing their meaning does not affect their ability to answer. However, for LLMs, if they do not understand the question in depth, they may memorize the original question and struggle to answer the rewritten question. **Therefore, the rewriting helps mitigate the unintentional contamination.** Based on the rewritten questions, we construct the MMLU-CF consisting of 10,000 questions for the test set and another 10,000 questions for the validation set. **To prevent deliberate contamination, the test set remains closed-source**, while the validation set is open-source for transparency and convenient evaluation. Noteworthily, we propose a strategy to submit an evaluation request for the test set, as outlined in Section 4.7. Currently, the performance gap between the validation and test sets for each evaluated LLM is quite small. As LLM development progresses, this performance gap indicates the contamination degree of LLMs in the future.

We benchmark leading open-source and closed-source LLMs on the MMLU-CF test and validation sets, including OpenAI o1 (Jaech et al., 2024), GPT-4o (OpenAI, 2024), GPT-4o-mini (OpenAI, 2024),
DeepSeek (Guo et al., 2025; Liu et al., 2024), Qwen (Bai et al., 2023; Team, 2024), Llama (Meta, 2024), Phi (Abdin et al., 2024), and many more. The 5-shot test results are briefly summarized in Figure 2. More results and analysis could refer to Section 4.3 and Section 4.5.

2 Related Work

2.1 General LLMs Benchmark

In the field of natural language processing (NLP), benchmarks play a crucial role in evaluating and comparing the performance of different large language models (Wang et al., 2018; Cobbe et al., 2021; Hendrycks et al.; Wang et al., 2024; Zhou et al., 2023; Zheng et al., 2023; Rein et al., 2023; Zhang et al., 2024; Hendrycks et al.; Phan et al., 2025). They serve as a common ground for fair comparison, fostering transparency and reproducibility in research. For instance, GLUE (Wang et al., 2018; Sarlin et al., 2020) is a collection of nine different tasks designed to evaluate the natural language understanding capabilities of models. GSM8K (Cobbe et al., 2021) is a benchmark dataset of 8,000 high-quality, linguistically diverse grade school math word problems. It is designed to evaluate the problem-solving abilities of language models, requiring a combination of language understanding and mathematical reasoning. MMLU (Hendrycks et al.) is a benchmark designed to evaluate a model’s multitask learning capabilities across a diverse set of 57 tasks, including high school mathematics, college-level biology, law, and more, focusing on testing the model’s generalization ability across different domains. Building upon this, MMLU-Pro (Wang et al., 2024) enhances the benchmark by introducing more challenging, reasoning-focused questions and expanding the choice set from four to ten choices, shifting the emphasis from knowledge retrieval to reasoning. Further, MMLU-Pro+ (Asgari et al.) extends MMLU-Pro by assessing shortcut learning and higher-order reasoning in large language models, offering a comprehensive evaluation of both reasoning depth and model robustness. These benchmarks have become standard tools in the evaluation of large language models due to their widespread adoption and comprehensive coverage of various domains.

However, these benchmarks, including GSM8K, MMLU, MMLU-Pro, and MMLU-Pro+, do not account for contamination prevention.

2.2 Contamination-free Benchmark

Several benchmark datasets have been introduced for contamination-free evaluation. KIEval (Yu et al., 2024) is an interactive framework with an LLM-powered "interactor" for multi-round dialogues to assess deep comprehension beyond mere recall. However, its multi-turn dialogue mechanism is time-consuming and poses reproducibility challenges. LatestEval (Li et al., 2024) creates dynamic reading comprehension evaluations from recent texts using a three-step process: collecting texts, extracting key information, and constructing questions with template-filling or LLMs. The questions are generated rather than from the real world. LiveCodeBench (Jain et al., 2024) continuously collects new coding problems from LeetCode, AtCoder, and CodeForces for a contamination-free benchmark, revealing performance drops in some models, such as DeepSeek (Guo et al., 2024), which specifically for code domain. GSM1K (Zhang et al., 2024) assesses the true reasoning ability of large language models by creating a new benchmark with a similar style and complexity to GSM8k, revealing significant accuracy drops of memorization in many LLMs. However, it is the only closed-source benchmark that focuses solely on math with a small scale of questions. LiveBench (White et al., 2024) introduces (1) frequently updated questions from recent sources, (2) automatic scoring based on ground-truth values, and (3) a variety of challenging tasks, including math, coding, reasoning, language, instruction following, and data analysis. However, it requires re-evaluating LLMs on regularly updated questions. This leads to high costs for maintenance and uncertainty for reevaluation.

Consequently, there is no contamination-free benchmark that assesses LLMs’ understanding of world knowledge on a large scale, across diverse domains, and with a one-time, low-cost evaluation. Some of the existing benchmarks are small-scale for specific domains, while others rely on complex strategies, such as dynamic dialog, that incur high evaluation costs and pose reproducibility challenges. Additionally, some require regular updates to their question sets, leading to unstable difficulty distribution and high re-evaluation costs.

Unlike the methods mentioned above, we categorize data contamination into unintentional and deliberate types. To mitigate unintentional data contamination, we apply three decontamination rules while collecting data from a broader domain.
Meanwhile, our MMLU-CF benchmark keeps the test set closed-source to prevent deliberate data contamination. To the best of our knowledge, MMLU-CF is the first contamination-free benchmark for world knowledge understanding under large scale and diverse domains, which provides reproducible results and a convenient evaluation process.

3 The MMLU-CF Benchmark

3.1 Overview
The MMLU-CF benchmark contains 20,000 data points and spans 14 fields, screened from 200+ billion documents on public open websites. To produce this diverse, high-quality, safety and contamination-free benchmark, we employ a series of steps, shown in Figure 3. Ultimately, we curate a dataset comprising 10,000 questions for the test set and 10,000 questions for the validation set respectively. The test set remains closed-source to prevent deliberate exposure of the questions (Zhang et al., 2024), while the validation set is open-source to validate the authenticity and effectiveness of the questions. The following sections outline the steps to process the raw data.

3.2 Dataset Construction Pipeline
MCQ Collection. The data sources are diversified broadly to preliminary mitigate the data exposure in existing LLMs. Specifically, over 200 billion documents are collected from open-source websites, and rule-based methods are employed to extract 2.7 million multiple-choice questions with the corresponding answers as the raw data 1. Unlike previous efforts, such as those by (Hendrycks et al.; Wang et al., 2024), which collect data from a few sources, these 2.7 million questions encompassed over 3,000 different website domains, ensuring a wide variety of knowledge, which spans 14 fields, including Health, Math, Physics, Business, Chemistry, Philosophy, Law, Engineering, etc.

MCQ Cleaning. With the 2.7 million raw questions, a series of filtering and standardization strategies are employed for initial data cleaning. Firstly, we filtered out questions with length less than 10 or larger than 512 characters and removed question numbers if they existed. Secondly, we removed questions that had fewer than four choices, contained empty content, or had inconsistent choice numbers. Meanwhile, we also eliminated questions whose choice numbers do not start with "a,b,c,d," "1,2,3,4," or "i,ii,iii,iv" in lower characters. Then, we mapped the choice numbers and corresponding answers to "A, B, C, D.". Thirdly, we discarded questions without answers or with answers that were not among the choices and further formalized the answer to ensure including both choice number and content. Finally, we only kept English questions and performed deduplication. Through these steps, the data scale was reduced to 1.66 million.

Difficulty Sampling. Firstly, GPT-4o is used to categorize the difficulty levels of the original MMLU under prompt in Table 6. The difficulty distribution of MMLU is illustrated in Figure 4, that shows nearly one-third of the questions have a difficulty level below [4]. To categorize our data using the same difficulty standard as MMLU, the questions in MMLU dataset were selected as samples for few-shot labeling under the same scoring prompt and LLM. We selected questions using a normal distribution centered around a difficulty level of [6], as indicated in Figure 4, to construct an appropriate difficulty distribution. During sampling process, we maintained a balanced distribution of question categories, maximized the diversity of domains, and ensured that questions had corresponding explanations whenever possible for the diversity, reliability, and quality. Ultimately, 50,000 questions are sampled from the 1.6 million questions.

LLMs Checking. To mitigate single LLM judgment bias, we employed multiple LLMs, including GPT-4o, Gemini, and Claude, to review the quality and harmlessness of these questions.
For the quality of questions, we assessed them based on the following criteria:
• Context and Clarity: Are the question and choices consistent and unambiguous, providing enough context for understanding?
• Logical Consistency: Are the question and choices structured without contradictions?
• Factual Accuracy: Are the question and choices factually correct and not misleading?
• Mutual Exclusivity: Are choices mutually exclusive without overlap?
• Correct Answer: Is the correct answer included in the choices?
From the perspective of harmlessness, we reviewed the content from the following four aspects:
• Non-hatred: Ensure the content does not contain hate speech.

1To avoid replicating the raw data collection pipeline, we will not disclose the data sources or extraction rules.
Figure 3: The construction pipeline of the MMLU-CF Benchmark. The pipeline involves (1) MCQ Collection to gather a diverse set of questions; (2) MCQ Cleaning to ensure quality; (3) Difficulty Sampling to ensure an appropriate difficulty distribution for questions; (4) LLMs checking: The LLMs, including GPT-4o, Gemini, and Claude, are reviewing the accuracy and safety of the data; and (5) Contamination-Free Processing to prevent data leakage and maintain dataset purity. Ultimately, this process results in the MMLU-CF, consisting of 10,000 questions for the closed-source test set and 10,000 for the open-source validation set.

• Non-sex: Ensure the content does not inappropriate sexual suggestions or content.
• Non-selfharm: Ensure the content neither contains self-harm nor encourages self-harm.
• Non-violence: Ensure the content does not contain violence or incite violence.

Specifically, these three LLMs are applied to rate each question on a scale from level 1 to 5, where level 5 represents the highest quality. Then, questions with an average score greater than level 4 were selected to construct validation and test sets of MMLU-CF. Additionally, inspired by Decontaminator (Yang et al., 2023), GPT-4o is utilized to perform redundancy detection (Yang et al., 2023) on semantically identical test and validation questions. To further ensure diversity, the selected questions came from over 1,000 web domains. The detailed prompts could refer to the Section A.8 in the Appendix.

Contamination-Free Processing. Although the questions are sourced from diverse domains, they are publicly available and could still be included in LLMs training data. We hope the constructed benchmark could assess the LLMs’ understanding ability rather than their memorization of answers (Carlini et al., 2023). To mitigate this unintentional contamination, we implemented the following three decontamination rules based on the principles of simplicity and understandability, as shown in Figure 5:

(1) Rule 1: Rephrase Question. To avoid the model rote learning (memorizing) the question rather than understanding it, we rephrased the questions using powerful GPT-4o. The manual check in the Section A.1 of Appendix ensure that the rephrased questions retain their original meaning.

(2) Rule 2: Shuffle Choices. To prevent the model from answering correctly based on memorizing the order of choice sequence, we shuffled the choices (Gupta et al., 2024). If the last choice was ‘None of the above’ or ‘All of the above,’ only the first three choices are randomly shuffled.

(3) Rule 3: Random Replace Choices. To further make the questions different from the original ones, we randomly replaced one of their choices with ‘None of the other choices’ with a 50% probability. If the last choice was ‘None of the above’ or ‘All of the above’, we skipped this question. When the

![The construction pipeline of the MMLU-CF Benchmark](page_68_68_1337_393.png)

![The difficulty levels produced by GPT-4o for MMLU and MMLU-CF are analyzed. In our data, we randomly sampled 10,000 questions for visualization.](page_1012_682_393_393.png)
replaced the choice is the correct answer, the correctness of this choice remains. Similarly, when an incorrect choice is replaced, it acted as a distractor. Both of them require the model to employ more understanding and reasoning to answer correctly.

The first two rules do not change the difficulty of the questions, while the last rule does alter it. We place the Contamination-Free Processing at the final step due to its high computational cost. Although these rules in it are simple, they effectively mitigate the unintentional contamination and make the dataset more challenging. For a detailed analysis, refer to Section 4.4. The difficulty distribution can be found in the Section A.9.

After that, we divided the final data into 10,000 questions as validation set and 10,000 questions as test set, which maintain the similar difficulty and discipline distribution. The test set is kept closed-source to prevent deliberate contamination.

![A MCQ instance by Contamination-free Processing. The top box is the input MCQ, and the bottom box is the decontaminated MCQ.](page_328_670_527_388.png)

Figure 5: A MCQ instance by Contamination-free Processing. The top box is the input MCQ, and the bottom box is the decontaminated MCQ.

4 Experiments

4.1 Evaluation Models
We evaluate 40+ models across various sizes by the evaluation platform OpenCompass (Contributors, 2023), including open-source models ranging from 0.5B to 72B and closed-source APIs. The experiments include models with different classes, such as GPTs (Achiam et al., 2023) (GPT-4o (v2024-10-1), GPT-o1-mini (v2024-09-12), GPT-4o-mini (v2024-10-1), GPT-4-Turbo (v2024-2-15)), Gemini (Reid et al., 2024) (Gemini-1.5-Flash), and public models like Llama-3-{8, 70}B-chat (Meta, 2024), Llama-3.1-{8, 70}B-chat (Meta, 2024), Mixtral-{7, 8x7, 8x22}-instruct, Phi-4 (Abdin et al., 2024), Phi-3.5-{mini, small} (Abdin et al., 2024), Gemma-2-{2, 9, 27}B (Team et al., 2024), Qwen2.5-{0.5, 1.5, 7, 14, 70}B (Team, 2024).

4.2 Evaluation Metrics
We employ both 5-shot and 0-shot approaches to measure the performance of LLMs on the MMLU-CF test and validation set. Additionally, the open-source models are categorized based on their parameter size into four sections: Large (>50B), Medium (13B-50B), Small (6B-12B), and Mini (0.5-5B). The results on the validation set, 0-shot evaluation results, evaluation prompt, and corresponding settings are detailed in Section A.6 of the Appendix.

4.3 Evaluation Results and Analysis
As shown in Table 4 and Figure 2, the results on a range of mainstream LLMs are presented, which reveal several noteworthy findings:

• The performance ranks on MMLU of evaluated LMs are disrupted by the new benchmark, MMLU-CF. The top 3 LMs — OpenAI o1, Deepseek-R1, and Deepseek-V3 — maintain their leading positions without any changes in their rankings.

• Interestingly, among the notable rank changes (>= 3 positions), the decreases in rank tend to be more significant than the increases. On average, the LMs that decreased in rank fell by **5.14** positions, while those that increased in rank rose by **3.78** positions. This asymmetry suggests that performance is *easier to significantly drop than to rise*, potentially due to data contamination in pretraining corpora.
<table>
  <tr>
    <th>Model</th>
    <th>MMLU Rank</th>
    <th>MMLU (%)</th>
    <th>MMLU-CF Rank</th>
    <th>MMLU-CF (%)</th>
    <th>Diff. (%)</th>
    <th>Rank Change</th>
  </tr>
  <tr>
    <td>OpenA1-01 (Jaech et al., 2024)</td>
    <td>1</td>
    <td>92.5</td>
    <td>87.6</td>
    <td>-4.9</td>
    <td>-12.9</td>
    <td>—</td>
  </tr>
  <tr>
    <td>DeepSeek-R1 (Gao et al., 2025)</td>
    <td>2</td>
    <td>90.8</td>
    <td>76.3</td>
    <td>-14.5</td>
    <td>-13.5</td>
    <td>—</td>
  </tr>
  <tr>
    <td>DeepSeek-V3 (Liu et al., 2024)</td>
    <td>3</td>
    <td>88.5</td>
    <td>73.9</td>
    <td>-14.6</td>
    <td>-14.6</td>
    <td>—</td>
  </tr>
  <tr>
    <td>GPT-4o (OpenAI, 2024)</td>
    <td>4</td>
    <td>88.0</td>
    <td>73.4</td>
    <td>-14.6</td>
    <td>-14.6</td>
    <td>—</td>
  </tr>
  <tr>
    <td>OpenA1-03-mini (OpenAI, 2025)</td>
    <td>5</td>
    <td>86.9</td>
    <td>73.4</td>
    <td>-13.5</td>
    <td>-13.5</td>
    <td>—</td>
  </tr>
  <tr>
    <td>GPT-4-Turbo (Achiam et al., 2023)</td>
    <td>6</td>
    <td>86.5</td>
    <td>70.4</td>
    <td>-16.1</td>
    <td>-16.1</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Llama-3.3-70B-instruct (Meta, 2024)</td>
    <td>7</td>
    <td>86.3</td>
    <td>68.8</td>
    <td>-17.5</td>
    <td>-17.5</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-instruct (Meta, 2024)</td>
    <td>8</td>
    <td>86.0</td>
    <td>68.7</td>
    <td>-17.3</td>
    <td>-17.3</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-instruct (Team, 2024)</td>
    <td>9</td>
    <td>85.5</td>
    <td>67.6</td>
    <td>-17.9</td>
    <td>-17.9</td>
    <td>—</td>
  </tr>
  <tr>
    <td>OpenA1-01-mini (Jaech et al., 2024)</td>
    <td>10</td>
    <td>85.2</td>
    <td>71.2</td>
    <td>-14.0</td>
    <td>-14.0</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Phi-4-14B (Abdin et al., 2024)</td>
    <td>11</td>
    <td>84.8</td>
    <td>67.8</td>
    <td>-17.0</td>
    <td>-17.0</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-instruct (Team, 2024)</td>
    <td>12</td>
    <td>83.9†</td>
    <td>69.7</td>
    <td>-14.2</td>
    <td>-14.2</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.72B-instruct(Bai et al., 2023)</td>
    <td>13</td>
    <td>82.3</td>
    <td>63.7</td>
    <td>-18.6</td>
    <td>-18.6</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Llama-3.70B-instruct</td>
    <td>14</td>
    <td>82.0</td>
    <td>68.9</td>
    <td>-13.1</td>
    <td>-13.1</td>
    <td>—</td>
  </tr>
  <tr>
    <td>GPT-4o-mini (OpenAI, 2024)</td>
    <td>15</td>
    <td>81.8</td>
    <td>65.5</td>
    <td>-16.3</td>
    <td>-16.3</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.5-4B-instruct (Team, 2024)</td>
    <td>16</td>
    <td>79.9†</td>
    <td>66.4</td>
    <td>-13.5</td>
    <td>-13.5</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Phi-5-14B-Med (Abdin et al., 2024)</td>
    <td>17</td>
    <td>78.9</td>
    <td>64.6</td>
    <td>-14.3</td>
    <td>-14.3</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Mini-1.5-Flash (Reid et al., 2024)</td>
    <td>18</td>
    <td>78.7</td>
    <td>64.8</td>
    <td>-13.9</td>
    <td>-13.9</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Phi-3-medium-instruct (Abdin et al., 2024)</td>
    <td>19</td>
    <td>77.9</td>
    <td>64.2</td>
    <td>-13.7</td>
    <td>-13.7</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-instruct (Team, 2024)</td>
    <td>20</td>
    <td>76.8†</td>
    <td>61.3</td>
    <td>-15.5</td>
    <td>-15.5</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Yi-1.5-4B-chat (Young et al., 2024)</td>
    <td>20</td>
    <td>76.8</td>
    <td>61.3</td>
    <td>-15.5</td>
    <td>-15.5</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Internlm-3.8B-chat (Cai et al., 2024)</td>
    <td>22</td>
    <td>76.6†</td>
    <td>60.3</td>
    <td>-16.3</td>
    <td>-16.3</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Mixtral-8x22B-instruct(Jiang et al., 2024)</td>
    <td>23</td>
    <td>76.2</td>
    <td>62.8</td>
    <td>-13.4</td>
    <td>-13.4</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen1.2B-instruct(Bai et al., 2023)</td>
    <td>24</td>
    <td>75.6</td>
    <td>59.8</td>
    <td>-15.8</td>
    <td>-15.8</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Internlm-2.5-7B-chat (Team, 2024)</td>
    <td>25</td>
    <td>75.2</td>
    <td>63.9</td>
    <td>-11.3</td>
    <td>-11.3</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Internlm-2.5-7B-chat (Cai et al., 2024)</td>
    <td>26</td>
    <td>72.8</td>
    <td>57.3</td>
    <td>-15.5</td>
    <td>-15.5</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Glm-4.9B-chat (GLM et al., 2024)</td>
    <td>27</td>
    <td>72.4</td>
    <td>57.8</td>
    <td>-14.6</td>
    <td>-14.6</td>
    <td>—</td>
  </tr>
  <tr>
    <td>GPT-3.5-Turbo(OpenAI, 2023)</td>
    <td>28</td>
    <td>71.4</td>
    <td>58.2</td>
    <td>-13.2</td>
    <td>-13.2</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Gemma-2.9B (Team et al., 2024)</td>
    <td>29</td>
    <td>71.3</td>
    <td>53.7</td>
    <td>-17.6</td>
    <td>-17.6</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Phi-3-mini-instruct (3.8B) (Abdin et al., 2024)</td>
    <td>30</td>
    <td>70.9</td>
    <td>57.9</td>
    <td>-13.0</td>
    <td>-13.0</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.7B-instruct (Bai et al., 2023)</td>
    <td>31</td>
    <td>70.5</td>
    <td>58.1</td>
    <td>-12.4</td>
    <td>-12.4</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Mixtral-8x7B-instruct-v0.1 (Jiang et al., 2024)</td>
    <td>31</td>
    <td>70.5</td>
    <td>58.3</td>
    <td>-12.2</td>
    <td>-12.2</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini-instruct (3.8B)</td>
    <td>33</td>
    <td>69.1</td>
    <td>57.9</td>
    <td>-11.2</td>
    <td>-11.2</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Llama-2.7B-chat (Meta, 2024)</td>
    <td>34</td>
    <td>68.9</td>
    <td>52.2</td>
    <td>-16.7</td>
    <td>-16.7</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Llama-3.8B-instruct (Meta, 2024)</td>
    <td>35</td>
    <td>68.4</td>
    <td>57.3</td>
    <td>-11.1</td>
    <td>-11.1</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-instruct (Meta, 2024)</td>
    <td>36</td>
    <td>68.1</td>
    <td>57.1</td>
    <td>-11.0</td>
    <td>-11.0</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B-instruct (Team, 2024)</td>
    <td>37</td>
    <td>64.4†</td>
    <td>55.9</td>
    <td>-8.5</td>
    <td>-8.5</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B-chat (Young et al., 2024)</td>
    <td>38</td>
    <td>62.8</td>
    <td>52.8</td>
    <td>-10.0</td>
    <td>-10.0</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Mistral-7B-instruct-v0.3 (Jiang et al., 2024)</td>
    <td>39</td>
    <td>60.3</td>
    <td>50.7</td>
    <td>-9.6</td>
    <td>-9.6</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.5-5B (Team, 2024)</td>
    <td>40</td>
    <td>58.5†</td>
    <td>51.2</td>
    <td>-7.3</td>
    <td>-7.3</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Internlm-2.13B-chat (Yang et al., 2023)</td>
    <td>41</td>
    <td>57.3</td>
    <td>48.3</td>
    <td>-9.0</td>
    <td>-9.0</td>
    <td>—</td>
  </tr>
  <tr>
    <td>DeepSeek-v2-lite-chat (DeepSeek-AI, 2024)</td>
    <td>42</td>
    <td>55.7</td>
    <td>49.3</td>
    <td>-6.4</td>
    <td>-6.4</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Llama-2.13B-chat (Meta, 2024)</td>
    <td>43</td>
    <td>54.8</td>
    <td>42.8</td>
    <td>-12.0</td>
    <td>-12.0</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Baichuan-2.7B-chat (Yang et al., 2023)</td>
    <td>44</td>
    <td>52.9</td>
    <td>44.5</td>
    <td>-8.4</td>
    <td>-8.4</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.15B-instruct (Bai et al., 2023)</td>
    <td>45</td>
    <td>52.4</td>
    <td>47.1</td>
    <td>-5.3</td>
    <td>-5.3</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Gemma-2.2B (Team et al., 2024)</td>
    <td>46</td>
    <td>51.3</td>
    <td>43.9</td>
    <td>-7.4</td>
    <td>-7.4</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Internlm-1.8B-chat (Cai et al., 2024)</td>
    <td>47</td>
    <td>47.8</td>
    <td>40.5</td>
    <td>-6.6</td>
    <td>-6.6</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Llama-2.7B-instruct (Meta, 2024)</td>
    <td>48</td>
    <td>45.3</td>
    <td>39.4</td>
    <td>-5.9</td>
    <td>-5.9</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.5-5B (Team, 2024)</td>
    <td>49</td>
    <td>45.1†</td>
    <td>41.9</td>
    <td>-3.2</td>
    <td>-3.2</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Qwen2.05B-instruct (Bai et al., 2023)</td>
    <td>50</td>
    <td>37.9</td>
    <td>33.3</td>
    <td>-4.6</td>
    <td>-4.6</td>
    <td>—</td>
  </tr>
</table>

Table 1: Performance of various models on MMLU and MMLU-CF (ours) in 5-shot test set evaluations without using CoT (Kojima et al., 2022), except for additional explanations. Diff. means the score difference of models between MMLU and MMLU-CF. † Denotes 0-shot with COT. † Indicates employing MMLU-redux (Gema et al., 2024), the results are from the Qwen2.5 homepage (Team, 2024).

• Additionally, smaller LMs appear to be more disruptive in the new MMLU-CF benchmark compared to their larger counterparts.

• Notable rank decreases (>= 3 positions) were observed for the following LMs: Qwen2.72B-instruct (↓7), Gemma-2.9B (↓7), Internlm-2.5-7B-chat (↓6), Glm-4.9B-chat (↓4), Llama-3.8B-instruct (↓4), Llama-3.3-70B-instruct (↓4), and Llama-3.1-70B-instruct (↓4).

• Notable rank increases (>= 3 positions) were observed for the following LMs: Gemma2.27B (↑6), Mixtral-8x7B-instruct-v0.1 (↑5), Phi-3.5-mini-instruct (3.8B) (↑4), Llama-3.70B-instruct (↑4), Qwen2.5-72B-instruct (↑3), GPT-o1-mini (↑3), Qwen2.5-32B-instruct (↑3), Qwen2-7B-instruct (↑3), and Llama-3-8B-instruct (↑3).

4.4 Effective of Decontamination Rules

Comprehensive experiments are conducted on MMLU and MMLU-CF to evaluate different LLMs’ performance under three decontamination rules: Rephrase Question (Rule 1), Shuffle Choices (Rule 2), and Random Replace Choices (Rule 3).

The detailed results summarized in Table 2 demonstrate the impact of each rule to the different benchmarks on different LLMs.

Firstly, all rules caused a performance drop across both the MMLU and MMLU-CF datasets and for all three LLMs.

Among the three rules, rules 1 and 2, which
do not affect the difficulty distribution of the test set, lead to performance drops, which verifies the effectiveness of data decontamination.

Rule 3, which also results in a performance drop, makes the rewritten questions more challenging.

Secondly, after applying the three rules, all evaluated LLMs exhibits a greater performance drop on MMLU than MMLU-CF, which confirms LLMs on MMLU have worse contamination. The data source of MMLU is more likely to be included in the training data of LLMs compared to MMLU-CF, which is sourced from over 200 billion documents.

Lastly, less powerful models, GPT-3.5-Turbo and Llama-3-8b, show a more pronounced performance decline, which demonstrates smaller models are more susceptible to contamination.

<table>
  <tr>
    <th>Rule</th>
    <th>Rule</th>
    <th>GPT-4o</th>
    <th>GPT-3.5-Turbo</th>
    <th>Llama-3.1-8b</th>
  </tr>
  <tr>
    <td>1</td>
    <td>2</td>
    <td>3</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="5">MMLU</td>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>88.0</td>
    <td>71.4</td>
    <td>68.1</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>-</td>
    <td>86.1</td>
    <td>68.5</td>
    <td>66.3</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>85.0</td>
    <td>67.3</td>
    <td>65.2</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>79.8 (<b>8.2</b>)</td>
    <td>62.1 (<b>9.3</b>)</td>
    <td>59.1 (<b>9.0</b>)</td>
  </tr>
  <tr>
    <td colspan="5">MMLU-CF</td>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>79.8</td>
    <td>65.3</td>
    <td>63.8</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>-</td>
    <td>78.6</td>
    <td>63.1</td>
    <td>62.3</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>77.9</td>
    <td>62.8</td>
    <td>61.8</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>73.4 (<b>-6.4</b>)</td>
    <td>58.2 (<b>-7.1</b>)</td>
    <td>57.1 (<b>-6.7</b>)</td>
  </tr>
</table>

Table 2: 5-shot results of applying different decontamination rules to MMLU and MMLU-CF test set.

4.5 Analysis of Data Contamination

Figure 1 presents examples of data contamination in MMLU, where LLMs generate responses that exactly match the original choices of the given question under a specific prompt. To analyze this further, we tested 1,000 cases sampled from MMLU and MMLU-CF using 40 models. As shown in Figure 6, approximately 10% of the models exhibited significant contamination on MMLU, producing outputs that matched 1%–5% of the choices, while 90% matched less than 1%. When applying decontamination rules on MMLU, 97.5% of the models produced outputs that matched under 1% of the choices. In contrast, 100% of the models matched less than 0.2% of the choices in MMLU-CF. The experiment verifies the presence of contamination in MMLU and confirms the contamination-free property of our MMLU-CF.

![LLM Response Match Rates with MMLU, MMLU using decontamination rules and MMLU-CF. Evaluated LLMs have obvious data leakage on MMLU.](page_1012_180_388_282.png)

Figure 6: LLM Response Match Rates with MMLU, MMLU using decontamination rules and MMLU-CF. Evaluated LLMs have obvious data leakage on MMLU.

their disciplinary distribution proportions are quite similar. The most prevalent disciplines are Computer Science, Health, and History, with proportions of 13.1%, 12.2%, and 11.5% in the test set, respectively. This distribution may lead to slight differences in the performance of different models. Table 3, we present the performance of various large models across different disciplines. GPT-4o achieves the best performance in terms of average accuracy and different disciplines. We observe that the models perform worst in Computer Science. This is because the domain not only requires fundamental knowledge of Computer Science but also involves code understanding, which increases the difficulty. Qwen2.5-72B, -32B brings new upgrades in mathematics and coding, delivering the best results in mathematics, engineering, and computer science. Despite its small size, Phi-4 achieves competitive results compared to larger models, showcasing its efficiency in handling complex tasks.

4.7 Testset Evaluation Methods

Two evaluation methods are supported for the test set of MMLU-CF. The users could submit evaluation requests by providing Hugging Face open-source model IDs or API formats through the introduction of our GitHub project homepage. Besides, we will actively evaluate the latest popular models from Hugging Face as well as mainstream APIs.

5 Conclusion

In this paper, we construct MMLU-CF, a contamination-free and challenging multiple-choice question benchmark under large scale and diverse disciplines, to reassess LLMs’ understanding of world knowledge. Specifically, we categorize data contamination into unintentional and deliberate types. To mitigate unintentional con-
<table>
  <tr>
    <th>Subject</th>
    <th>GPT-4o (OpenAI, 2024)</th>
    <th>GPT-4o-mini (OpenAI, 2024)</th>
    <th>Llama-3.3-70B (Meta, 2024)</th>
    <th>Qwen2.5-72B (Team, 2024)</th>
    <th>Qwen2.5-32B (Team, 2024)</th>
    <th>Phi-4-14B (Abdi et al., 2024)</th>
  </tr>
  <tr>
    <td>Math</td>
    <td>56.09</td>
    <td>45.83</td>
    <td>56.3</td>
    <td><b>67.51</b></td>
    <td>63.10</td>
    <td>62.18</td>
  </tr>
  <tr>
    <td>Physics</td>
    <td><b>75.15</b></td>
    <td>64.47</td>
    <td>69.0</td>
    <td>74.00</td>
    <td>71.12</td>
    <td>69.15</td>
  </tr>
  <tr>
    <td>Chemistry</td>
    <td>72.44</td>
    <td>66.54</td>
    <td>68.3</td>
    <td>69.62</td>
    <td>68.81</td>
    <td>67.13</td>
  </tr>
  <tr>
    <td>Law</td>
    <td><b>81.46</b></td>
    <td>72.73</td>
    <td>73.6</td>
    <td>75.15</td>
    <td>72.55</td>
    <td>71.84</td>
  </tr>
  <tr>
    <td>Engineering</td>
    <td>60.15</td>
    <td>55.41</td>
    <td>56.7</td>
    <td><b>61.39</b></td>
    <td>57.69</td>
    <td>54.67</td>
  </tr>
  <tr>
    <td>Economics</td>
    <td>78.33</td>
    <td>66.31</td>
    <td>72.5</td>
    <td>74.95</td>
    <td>68.90</td>
    <td>68.88</td>
  </tr>
  <tr>
    <td>Health</td>
    <td>81.09</td>
    <td>76.11</td>
    <td>73.2</td>
    <td>80.23</td>
    <td>78.55</td>
    <td>76.29</td>
  </tr>
  <tr>
    <td>Psychology</td>
    <td><b>88.10</b></td>
    <td>70.28</td>
    <td>77.5</td>
    <td>78.95</td>
    <td>77.94</td>
    <td>78.45</td>
  </tr>
  <tr>
    <td>Business</td>
    <td>70.90</td>
    <td>63.81</td>
    <td>64.7</td>
    <td><b>71.00</b></td>
    <td>68.69</td>
    <td>65.19</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td><b>82.84</b></td>
    <td>74.63</td>
    <td>75.5</td>
    <td>78.88</td>
    <td>74.53</td>
    <td>75.91</td>
  </tr>
  <tr>
    <td>Philosophy</td>
    <td><b>81.82</b></td>
    <td>77.99</td>
    <td>78.9</td>
    <td>74.24</td>
    <td>72.73</td>
    <td>76.08</td>
  </tr>
  <tr>
    <td>Computer Science</td>
    <td>55.50</td>
    <td>51.09</td>
    <td>51.0</td>
    <td>56.12</td>
    <td><b>68.79</b></td>
    <td>51.09</td>
  </tr>
  <tr>
    <td>History</td>
    <td>77.23</td>
    <td>67.05</td>
    <td>71.2</td>
    <td>71.19</td>
    <td>68.79</td>
    <td>68.09</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>74.83</td>
    <td>64.74</td>
    <td>67.9</td>
    <td>68.15</td>
    <td>66.88</td>
    <td>65.55</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>73.42</td>
    <td>65.52</td>
    <td>68.82</td>
    <td>71.60</td>
    <td>68.81</td>
    <td>67.68</td>
  </tr>
</table>

Table 3: Performance of different models on MMLU-CF discipline under a 5-shot test set. The best result is emphasized in bold.

![Pie chart showing distribution of disciplines in MMLU-CF test set](page_320_670_484_312.png)
(a) Test Set

![Pie chart showing distribution of disciplines in MMLU-CF validation set](page_320_1012_484_312.png)
(b) Validation Set

Figure 7: Distribution of Disciplines in MMLU-CF.

tamination, we collect data from a broader domain and design three decontamination rules for further processing. To prevent deliberate contamination, we keep the test set closed-sourced for ensuring no leakage while making the validation set open-sourced for transparency and convenient evaluation. The evaluation results of over 40 mainstream LLMs on MMLU-CF reveal significant performance drops and performance ranking changes compared to the original MMLU benchmark. The leakage analysis of MMLU and the ablation study of three decontamination rules indicate the presence of contamination in MMLU and the effectiveness of our approach in establishing a decontaminated benchmark. We believe this contamination-free benchmark would promote fair and reliable LLMs evaluation and provide valuable insights for the design of future benchmarks.

6 Limitations

Despite being constructed with the utmost objectivity and fairness and leveraging multiple large language models to verify the correctness of the questions and answers, some errors may still remain in this dataset. To address this, we have provided a validation set that is available to the public for further scrutiny and verification. Additionally, this dataset primarily focuses on multiple-choice questions and language modalities. However, other aspects of large models’ capabilities, such as math and code reasoning, multi-modal understanding (e.g., image and audio), and specific domain expertise, still require evaluation with similarly unbiased and contamination-free benchmarks.

7 Ethics Statement

MMLU-CF is constructed based on public data sources and methodologies to ensure transparency. The benchmark is designed to provide fair and reliable evaluations through decontamination rules and verification. While efforts are made to minimize errors, some may remain, and we encourage the community to review the publicly available validation set for further improvements. This benchmark focuses on language modalities, and future work is needed for unbiased evaluation in other areas. We call for the responsible use of this dataset to promote ethical and equitable AI development.
References

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. 2024. Phi-4 technical report. Preprint, arXiv:2412.08905.

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024b. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencio Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Anthropic. 2023. Introducing the next generation of claude.

Saeid Asgari, Aliasghar Khani, and Amir Hosein Khasahmadi. Mmlu-prov: Evaluating higher-order reasoning and shortcut learning in llms. In Neurips Safe Generative AI Workshop 2024.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Zheng Cai, Maosong Cao, Haojing Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Nathan Gan, Kai Jie, Akhil Agrawal, Jonathan Byrd, and Mark Chen. 2021. Training verifiers to solve math word problems. In arXiv preprint arXiv:2110.14168.

OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass.

DeepSeek-AI. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.

Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8698–8711.

Aryo Pradiptha Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. 2024. Are we done with mmlu? arXiv preprint arXiv:2406.04127.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiada Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xiujuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. Preprint, arXiv:2406.12793.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2023. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming—the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. Changing answer order can decrease mmlu accuracy. arXiv preprint arXiv:2406.19470.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. b. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressaud, Gianna Lengyel, Guillaume Lamplé, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressaud, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.

Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18600–18607.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Meta. 2024. Build the future of ai with meta llama 3.

OpenAI. 2023. Gpt-3.5 turbo fine-tuning and api updates.

OpenAI. 2024. Gpt-4o mini: advancing cost-efficient intelligence.

OpenAI. 2024b. Hello gpt-4o.

OpenAI. 2025. Introducing openai o3 and o4-mini.

Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. 2025. Humanity’s last exam. arXiv preprint arXiv:2501.14249.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022.

Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. 2023. To the cut-off... and beyond? a longitudinal perspective on llm data contamination. In The Twelfth International Conference on Learning Representations.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947.

Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. 2024. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402.19450.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyian Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574.

Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. 2024. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023b. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04850.

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652.

Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qirfeng Yin. 2023. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. arXiv preprint arXiv:2312.14187.

Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. 2024. Kievai: A knowledge-grounded interactive evaluation framework for large language models. arXiv preprint arXiv:2402.15043.

Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332.

Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. 2024b. xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215.

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
A Appendix

A.1 Human Review for LLM-Generated Outputs

In the processes of Difficulty Sampling, LLMs Checking, and Contamination-Free Processing, LLMs were employed to filter and modify instances. However, since LLMs are not flawless, false-positive cases may occur, resulting in outputs that deviate from the intended results.

To evaluate the performance of LLMs in data processing, we conducted a human review process. Specifically, three volunteers performed random sampling checks at a certain ratio. For difficulty level assessments, 100 instances were reviewed, achieving approximately 97% accuracy. For quality and safety scoring, 100 instances were checked, with an accuracy of about 99%. Additionally, we compared the 100 rephrased questions with their original versions, confirming successful rephrasing in approximately 99% of cases.

These results indicate that although LLMs have limitations, the use of a state-of-the-art GPT-4o model results in rare errors during these processes, ensuring the reliability of the generated data.

A.2 More Evaluation Results

As shown in Table 4, the 5-shot and 0-shot results on a range of mainstream LLMs are presented, which show several findings:

• The performance trend across different scales of LLMs remains consistent with the original MMLU: more powerful LLMs always have higher performance. This trend underscores the quality of constructed MMLU-CF.

• The performance of all evaluated LLMs significantly drops compared to the original MMLU benchmark, likely due to the decontamination rules and difficulty sampling process applied to the MMLU-CF dataset. Table 2 shows performance impact individually of each rule.

• The performance ranking of the evaluated LLMs varied across different model architectures and versions. This variation suggests the presence of potential contamination in the original MMLU. For instance, the early version of some models have higher score than their later version.

• The performance between the validation and test sets is narrowing gap now. The gap between them could serve as an index for assessing the contamination degree of the published validation sets in the future.

A.3 Relation between Test and Validation Sets

We partition the benchmark dataset into test and validation sets, then calculate the absolute score difference of LLMs performance as \( \Delta \). It not only demonstrates no contamination of the current LLMs on the validation set but also offers a method to assess the contamination degree of the future LLMs on the validation set. As shown in Table 4, before the validation set is publicly released, about 60% of \( \Delta \) values are less than 0.5, and 96% of \( \Delta \) values are below 1.0. This indicates that the evaluation results of LLMs are significantly consistent across the test and validation sets, demonstrating the effectiveness of the validation set in evaluating model performance now. Once the validation set is made public, potential data leakage can cause the models to memorize the validation set, leading to an increase in \( \Delta \) values. Thus, the design \( \Delta \) serves as a method to monitor whether benchmarks might be compromised. This approach helps ensure the fairness and integrity of the benchmarks, preventing models from exploiting leaked data to artificially enhance their performance.

A.4 Manual Quality Validation for Filtered Instances

To ensure the reliability of the filtered examples, we conducted a manual quality check process to verify whether the 20,000 examples were accurately rewritten or whether the choices were modified based on the original content without introducing new information, and to determine whether the score changes were caused by rule adjustments rather than increased noise. Specifically, we randomly sampled 200 LLM-checked questions and manually reviewed their quality, improving question accuracy from 86% to 99.5%. This process ensures that filtered instances are reliable and enhances the validity of score comparisons across different rule applications.

A.5 The Effect of Decontamination Rules

In the methods section, we presented three types of question modification rules applied to the MMLU-CF dataset: question rephrasing, shuffling choices, and randomly replacing a choice with “None of the other choices.” To validate the effectiveness of these modifications, we first applied these three
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">MMLU 5-shot (%)</th>
    <th colspan="3">MMLU-CF 5-shot (%)</th>
    <th colspan="3">MMLU-CF 0-shot (%)</th>
  </tr>
  <tr>
    <th>Test</th>
    <th>Test</th>
    <th>Validation</th>
    <th>Δ (%)</th>
    <th>Test</th>
    <th>Validation</th>
    <th>Δ (%)</th>
  </tr>
  <tr>
    <td rowspan="10">API</td>
    <td>OpenAI-01 (Jach et al., 2024)</td>
    <td>92.3</td>
    <td>80.3</td>
    <td>80.3</td>
    <td>+0.0</td>
    <td>80.1</td>
    <td>79.9</td>
    <td>+0.2</td>
  </tr>
  <tr>
    <td>DeepSeek-R1 (Guo et al., 2025)</td>
    <td>90.8</td>
    <td>76.3</td>
    <td>75.3</td>
    <td>-1.0</td>
    <td>75.7</td>
    <td>76.0</td>
    <td>-0.3</td>
  </tr>
  <tr>
    <td>DeepSeek-V3 (Liu et al., 2024)</td>
    <td>88.5</td>
    <td>73.9</td>
    <td>71.9</td>
    <td>-1.0</td>
    <td>71.9</td>
    <td>71.4</td>
    <td>+1.5</td>
  </tr>
  <tr>
    <td>GPT-4 (OpenAI, 2024)</td>
    <td>88.0</td>
    <td>73.4</td>
    <td>73.4</td>
    <td>+0.0</td>
    <td>71.9</td>
    <td>72.4</td>
    <td>-0.5</td>
  </tr>
  <tr>
    <td>OpenAI-03-mini (OpenAI, 2025)</td>
    <td>86.9</td>
    <td>73.4</td>
    <td>73.6</td>
    <td>-0.2</td>
    <td>73.7</td>
    <td>73.6</td>
    <td>+0.1</td>
  </tr>
  <tr>
    <td>OpenAI-01-mini (Jach et al., 2024)</td>
    <td>85.2</td>
    <td>71.2</td>
    <td>71.0</td>
    <td>+0.2</td>
    <td>71.6</td>
    <td>71.5</td>
    <td>+0.1</td>
  </tr>
  <tr>
    <td>GPT-4-Turbo (Achiam et al., 2023)</td>
    <td>86.5</td>
    <td>70.4</td>
    <td>70.1</td>
    <td>+0.3</td>
    <td>68.9</td>
    <td>68.7</td>
    <td>+0.1</td>
  </tr>
  <tr>
    <td>GPT-4o-mini (OpenAI, 2024)</td>
    <td>81.8</td>
    <td>65.5</td>
    <td>65.1</td>
    <td>+0.4</td>
    <td>66.0</td>
    <td>65.3</td>
    <td>+0.7</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Flash (Reid et al., 2024)</td>
    <td>78.7</td>
    <td>64.8</td>
    <td>64.9</td>
    <td>-0.1</td>
    <td>56.7</td>
    <td>56.9</td>
    <td>-0.2</td>
  </tr>
  <tr>
    <td>GPT-3.5-Turbo (OpenAI, 2023)</td>
    <td>71.4</td>
    <td>58.2</td>
    <td>59.0</td>
    <td>-0.8</td>
    <td>57.2</td>
    <td>58.1</td>
    <td>-0.9</td>
  </tr>
  <tr>
    <td rowspan="10">Large</td>
    <td>Qwen2.5-72B-instruct (Team, 2024)</td>
    <td>85.3</td>
    <td>71.6</td>
    <td>71.3</td>
    <td>+0.3</td>
    <td>70.6</td>
    <td>70.4</td>
    <td>+0.2</td>
  </tr>
  <tr>
    <td>Llama-3-70B-instruct (Meta, 2024)</td>
    <td>82.0</td>
    <td>68.9</td>
    <td>68.8</td>
    <td>+0.1</td>
    <td>68.1</td>
    <td>67.4</td>
    <td>+0.7</td>
  </tr>
  <tr>
    <td>Llama-3-370B-instruct (Meta, 2024)</td>
    <td>86.3</td>
    <td>68.8</td>
    <td>67.8</td>
    <td>+1.0</td>
    <td>67.6</td>
    <td>67.5</td>
    <td>+0.1</td>
  </tr>
  <tr>
    <td>Llama-3-170B-instruct (Meta, 2024)</td>
    <td>86.0†</td>
    <td>68.7</td>
    <td>68.1</td>
    <td>+0.6</td>
    <td>70.4</td>
    <td>69.7</td>
    <td>+0.7</td>
  </tr>
  <tr>
    <td>Phi-3.5-MoE-instruct (Abdin et al., 2024)</td>
    <td>78.9</td>
    <td>64.6</td>
    <td>64.5</td>
    <td>+0.1</td>
    <td>63.1</td>
    <td>62.1</td>
    <td>+1.0</td>
  </tr>
  <tr>
    <td>Qwen2.7B-instruct (Bai et al., 2023)</td>
    <td>82.5</td>
    <td>63.7</td>
    <td>63.6</td>
    <td>+0.6</td>
    <td>62.4</td>
    <td>62.5</td>
    <td>-0.1</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-instruct-v0.1 (Jiang et al., 2024)</td>
    <td>76.2</td>
    <td>62.8</td>
    <td>62.5</td>
    <td>+0.3</td>
    <td>55.3</td>
    <td>64.8</td>
    <td>+0.5</td>
  </tr>
  <tr>
    <td>Qwen1.5-72B-chat (Bai et al., 2023)</td>
    <td>75.6</td>
    <td>59.8</td>
    <td>60.2</td>
    <td>-0.4</td>
    <td>59.1</td>
    <td>59.6</td>
    <td>-0.5</td>
  </tr>
  <tr>
    <td>Llama-2-70B-chat (Meta, 2024)</td>
    <td>68.9</td>
    <td>52.2</td>
    <td>51.8</td>
    <td>+0.4</td>
    <td>51.2</td>
    <td>50.9</td>
    <td>+0.3</td>
  </tr>
  <tr>
    <td rowspan="10">Medium</td>
    <td>Qwen2.5-32B-instruct (Team, 2024)</td>
    <td>83.9†</td>
    <td>69.7</td>
    <td>68.8</td>
    <td>+0.9</td>
    <td>68.9</td>
    <td>68.8</td>
    <td>+0.1</td>
  </tr>
  <tr>
    <td>Phi-14B (Abdin et al., 2024)</td>
    <td>84.8</td>
    <td>67.8</td>
    <td>68.5</td>
    <td>-0.7</td>
    <td>68.5</td>
    <td>69.4</td>
    <td>-0.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-instruct (Team, 2024)</td>
    <td>79.9</td>
    <td>66.4</td>
    <td>66.1</td>
    <td>+0.3</td>
    <td>66.0</td>
    <td>66.0</td>
    <td>+1.0</td>
  </tr>
  <tr>
    <td>Phi-3-medium-instruct (Abdin et al., 2024)</td>
    <td>77.9</td>
    <td>64.2</td>
    <td>64.2</td>
    <td>+0.0</td>
    <td>62.5</td>
    <td>62.7</td>
    <td>-0.2</td>
  </tr>
  <tr>
    <td>Gemma2-27B(Team et al., 2024)</td>
    <td>75.2</td>
    <td>63.9</td>
    <td>63.5</td>
    <td>+0.4</td>
    <td>64.2</td>
    <td>64.0</td>
    <td>+0.2</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B-chat (Young et al., 2024)</td>
    <td>76.8</td>
    <td>61.3</td>
    <td>60.5</td>
    <td>+0.8</td>
    <td>60.6</td>
    <td>59.5</td>
    <td>+1.1</td>
  </tr>
  <tr>
    <td>Mistral-8x7B-instruct-v0.1 (Jiang et al., 2024)</td>
    <td>70.5</td>
    <td>58.3</td>
    <td>57.1</td>
    <td>-1.2</td>
    <td>58.9</td>
    <td>58.5</td>
    <td>+0.4</td>
  </tr>
  <tr>
    <td>DeepSeek-v2-lite-chat (DeepSeek-AI, 2024)</td>
    <td>55.7</td>
    <td>49.3</td>
    <td>48.7</td>
    <td>+0.6</td>
    <td>48.2</td>
    <td>47.7</td>
    <td>+0.5</td>
  </tr>
  <tr>
    <td>Baichuan-2-13B-chat (Yang et al., 2023)</td>
    <td>57.3</td>
    <td>48.3</td>
    <td>48.6</td>
    <td>-0.3</td>
    <td>47.1</td>
    <td>48.1</td>
    <td>-1.0</td>
  </tr>
  <tr>
    <td>Llama-2-13B-chat (Touvron et al., 2023)</td>
    <td>54.8</td>
    <td>42.8</td>
    <td>42.1</td>
    <td>+0.7</td>
    <td>44.8</td>
    <td>44.6</td>
    <td>+0.2</td>
  </tr>
  <tr>
    <td rowspan="10">Small</td>
    <td>Qwen2.5-7B-instruct (Team, 2024)</td>
    <td>76.8†</td>
    <td>61.3</td>
    <td>60.4</td>
    <td>+0.9</td>
    <td>59.3</td>
    <td>58.6</td>
    <td>+0.7</td>
  </tr>
  <tr>
    <td>Internlm-3-8B-chat (Cai et al., 2024)</td>
    <td>76.6†</td>
    <td>60.3</td>
    <td>60.5</td>
    <td>-0.2</td>
    <td>61.8</td>
    <td>61.4</td>
    <td>+0.4</td>
  </tr>
  <tr>
    <td>Qwen2-7B-instruct (Bai et al., 2023)</td>
    <td>70.5</td>
    <td>58.1</td>
    <td>57.9</td>
    <td>+0.2</td>
    <td>58.3</td>
    <td>57.4</td>
    <td>+0.9</td>
  </tr>
  <tr>
    <td>Glm-4-9B-chat (GLM et al., 2024)</td>
    <td>72.4</td>
    <td>57.8</td>
    <td>57.9</td>
    <td>-0.1</td>
    <td>58.6</td>
    <td>58.7</td>
    <td>-0.1</td>
  </tr>
  <tr>
    <td>Internlm-2.5-7B-chat (Cai et al., 2024)</td>
    <td>72.8</td>
    <td>57.3</td>
    <td>56.8</td>
    <td>+0.5</td>
    <td>57.9</td>
    <td>56.9</td>
    <td>+1.0</td>
  </tr>
  <tr>
    <td>Llama-3-8B-instruct (Meta, 2024)</td>
    <td>68.4</td>
    <td>57.3</td>
    <td>56.5</td>
    <td>+0.8</td>
    <td>56.4</td>
    <td>55.4</td>
    <td>+1.0</td>
  </tr>
  <tr>
    <td>Llama-2-7B-instruct (Meta, 2024)</td>
    <td>68.1</td>
    <td>57.1</td>
    <td>57.9</td>
    <td>-0.8</td>
    <td>58.6</td>
    <td>56.1</td>
    <td>+0.0</td>
  </tr>
  <tr>
    <td>Gemma-2-9B (Team et al., 2024)</td>
    <td>71.3</td>
    <td>53.7</td>
    <td>53.3</td>
    <td>-0.4</td>
    <td>52.1</td>
    <td>51.2</td>
    <td>+0.9</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B-chat (Young et al., 2024)</td>
    <td>62.8</td>
    <td>52.8</td>
    <td>51.4</td>
    <td>+1.4</td>
    <td>52.2</td>
    <td>51.9</td>
    <td>+0.3</td>
  </tr>
  <tr>
    <td>Mistral-7B-instruct-v0.3 (Jiang et al., 2023)</td>
    <td>60.3</td>
    <td>50.7</td>
    <td>50.9</td>
    <td>-0.2</td>
    <td>51.1</td>
    <td>50.9</td>
    <td>+0.2</td>
  </tr>
  <tr>
    <td>Baichuan-2-7B-chat (Yang et al., 2023)</td>
    <td>52.9</td>
    <td>44.5</td>
    <td>43.9</td>
    <td>+0.6</td>
    <td>43.9</td>
    <td>44.0</td>
    <td>-0.1</td>
  </tr>
  <tr>
    <td>Llama-2-7B-chat (Touvron et al., 2023)</td>
    <td>45.3</td>
    <td>39.4</td>
    <td>38.5</td>
    <td>+0.9</td>
    <td>41.9</td>
    <td>40.9</td>
    <td>+1.0</td>
  </tr>
  <tr>
    <td rowspan="7">Mini</td>
    <td>Phi-3-mini-instruct (3.8B) (Abdin et al., 2024)</td>
    <td>70.9</td>
    <td>57.9</td>
    <td>58.1</td>
    <td>-0.2</td>
    <td>58.2</td>
    <td>57.5</td>
    <td>+0.7</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini-instruct (3.8B) (Abdin et al., 2024)</td>
    <td>69.1</td>
    <td>57.9</td>
    <td>57.4</td>
    <td>+0.5</td>
    <td>58.3</td>
    <td>57.7</td>
    <td>+0.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B-instruct (Team, 2024)</td>
    <td>64.4†</td>
    <td>55.9</td>
    <td>56.4</td>
    <td>-0.5</td>
    <td>54.3</td>
    <td>53.9</td>
    <td>+0.4</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B-instruct (Team, 2024)</td>
    <td>50.7†</td>
    <td>51.2</td>
    <td>51.0</td>
    <td>+0.2</td>
    <td>50.7</td>
    <td>50.4</td>
    <td>+0.3</td>
  </tr>
  <tr>
    <td>Qwen2.1-5B-instruct (Bai et al., 2023)</td>
    <td>52.4</td>
    <td>47.1</td>
    <td>47.5</td>
    <td>-0.4</td>
    <td>45.2</td>
    <td>44.5</td>
    <td>+0.7</td>
  </tr>
  <tr>
    <td>Gemma-2-2B (Team et al., 2024)</td>
    <td>51.3</td>
    <td>43.9</td>
    <td>42.4</td>
    <td>+1.5</td>
    <td>30.5</td>
    <td>29.4</td>
    <td>+0.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B-instruct (Team, 2024)</td>
    <td>24.1†</td>
    <td>41.9</td>
    <td>41.1</td>
    <td>+0.8</td>
    <td>36.0</td>
    <td>34.9</td>
    <td>+1.1</td>
  </tr>
  <tr>
    <td>Internlm-2chat-1.8B (Cai et al., 2024)</td>
    <td>47.1</td>
    <td>40.5</td>
    <td>39.4</td>
    <td>+1.1</td>
    <td>41.2</td>
    <td>39.8</td>
    <td>+1.4</td>
  </tr>
  <tr>
    <td>Qwen2.0-5B-instruct (Bai et al., 2023)</td>
    <td>37.9</td>
    <td>38.3</td>
    <td>38.3</td>
    <td>+0.0</td>
    <td>33.3</td>
    <td>33.5</td>
    <td>+0.0</td>
  </tr>
</table>

Table 4: Performance of various models on MMLU and MMLU-CF (ours). Both 0-shot and 5-shot evaluations don’t employ COT (Kojima et al., 2022), except for additional explanations. Δ means the absolute score difference of models between validation and test sets. † denotes 0-shot with COT. ‡ indicates employing MMLU-redux (Gema et al., 2024), the results are from Qwen2.5 homepage (Team, 2024).

rules to the MMLU (Hendrycks et al.). The results, shown in Figure 8, indicate that these modifications lead to a decrease in 5-shot and 0-shot scores for GPT-4o. Furthermore, when comparing these results to those on the MMLU-CF dataset, as depicted in Table 2, the accuracy drop is more pronounced on the MMLU dataset. This suggests a higher likelihood of data leakage in large models when using the MMLU dataset. In contrast, the MMLU-CF dataset, due to its broad and closed-source nature, exhibits a lower risk of data leakage.

<table>
  <tr>
    <th>Model</th>
    <th>Temperature</th>
    <th>Maximum Generation Tokens</th>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>0.6</td>
    <td>32,768</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>0.7</td>
    <td>8,192</td>
  </tr>
  <tr>
    <td>GPT-01-mini</td>
    <td>1.0</td>
    <td>8,192</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.7</td>
    <td>2,048</td>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>0.7</td>
    <td>2,048</td>
  </tr>
  <tr>
    <td>GPT-4-turbo</td>
    <td>0.7</td>
    <td>2,048</td>
  </tr>
  <tr>
    <td>Qwen2.5 Series</td>
    <td>0.7</td>
    <td>4,096</td>
  </tr>
  <tr>
    <td>Others</td>
    <td>0.7</td>
    <td>1,024</td>
  </tr>
</table>

Table 5: Temperature and Maximum Generation Token settings for different LLMs

A.6 Prompt and Setting for Evaluation

In all our experiments, the applied prompts of all the evaluated LLMs are the same, which could refer
There is a single choice question (with answers). Answer the question by replying A, B, C or D.
Question: {Question}?
A. {A}
B. {B}
C. {C}
D. {D}

Table 6: The prompt of LLMs Question Answering. {Question} represents the question instance, while {A}, {B}, {C}, and {D} denote the answer choices.

[Instruction]
Please rate the difficulty of this question on a scale of [0-9], where level [0] represents the easiest question and level [9] represents the most difficult.
Question: {Question}

Table 7: The prompt of LLMs Difficulty Labeling. {Question} represents the question instance.

A.8 Prompt for LLMs Checking

Table 8 shows the prompt used in the LLMs checking processing to verify the correctness of questions. For safety, we used GPT-4’s built-in safety filter under the strongest constraints to filter out unsafe content related to hate speech, sexual content, self-harm, and violence.

[Instruction]
Please review the following question and corresponding choices for correctness based on these criteria:
Context and Clarity: Are the question and choices consistent and unambiguous, providing enough context for understanding?
Logical Consistency: Are the question and choices logically structured without contradictions?
Factual Accuracy: Are the question and choices factually correct and not misleading?
Mutual Exclusivity: Are choices mutually exclusive without overlap?
Correct Answer: Is the correct answer included in the choices?
[Question to be reviewed]
{Question}
[Choices to be reviewed]
{Choices}
[Response]
Rate the question’s correctness on a scale of 1 to 5, with 5 being correct; Only give an overall Rating. For example, Rating: 5

Table 8: The Prompt of LLMs Checking. {Question} and {Choices} represent question and choices instance.

A.7 Prompt for Difficulty Labeling

Table 7 demonstrates the prompt used in the LLMs difficulty labeling. We do not provide choices and answers in the prompt to avoid data leakage through LLMs API calling.

A.9 Difficulty Distribution of MMLU-CF

Figure 9 demonstrates the question difficulty distribution of MMLU-CF dataset at various stages. In step three, we sampled from a normal distribution centered around a difficulty level of 6 and verified

Figure 8: GPT-4o evaluation comparison on MMLU with and without our decontamination rules.

to Table 8. This prompt is also used as the MMLU prompt in several well-known language model evaluation tools. Moving forward, we will continue using this prompt consistently to ensure comparability across tests. Moreover, Table 5 presents the temperature parameter settings and maximum generation token limits used in our experiments.
![The difficulty level distribution of MMLU-CF after stage (3) and (5).](page_184_120_496_377.png)

Figure 9: The difficulty level distribution of MMLU-CF after stage (3) and (5).

that the difficulty level remained consistent after Relu 2 of the decontamination-free process. After applying Relu 3 of the decontamination-free process, we observed a change: the proportion of samples with difficulty level 5 significantly decreased, while the number of questions with difficulty level 7 increased. This indicates that the decontamination process introduced more challenging questions into the dataset, which meets the expectation.

A.10 Sampled MCQs across Disciplines

In Table 9, 10, 11, 12 and 13, we present the questions from the validation set across various disciplines. For each subject, we have randomly sampled three questions for demonstration, which offers insights into the diversity and characteristics of the questions used in the validation process. For more questions, we will publicly announce the validation set soon.
<table>
  <tr>
    <th colspan="2">Biology</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>Which group of biological molecules is the most diverse in function?<br>A. Carbohydrates   B. Proteins<br>C. Nucleic acids   D. Lipids<br><b>Answer: B</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>Which of these structures is the smallest?<br>A. Hydrogen atom   B. None of the other choices<br>C. Mitochondrion   D. Virioid<br><b>Answer: A</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>Which of the following controls and regulates life processes?<br>A. Reproductive and endocrine systems   B. Endocrine and digestive systems<br>C. None of the other choices   D. Nervous and endocrine systems<br><b>Answer: D</b></td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Chemistry</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>What occurs when silver chloride is exposed to sunlight?<br>A. Silver metal and chlorine gas are formed   B. Silver metal and hydrogen gas are formed<br>C. Only hydrogen gas is formed   D. Only silver metal is formed<br><b>Answer: A</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>What is the phenomenon called when a beam of light passes through a colloidal solution?<br>A. Cataphoresis   B. Tyndall effect<br>C. Electrophoresis   D. Coagulation<br><b>Answer: B</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>Electrolytes play a crucial role in the chemistry of living organisms. What defines an electrolyte?<br>A. Contains electrodes   B. Conducts electricity when melted or put into solution<br>C. Generates light when electricity is applied   D. Contains electrons<br><b>Answer: B</b></td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Computer Science</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>Which of the following is not a valid floating point literal in Java?<br>A. 5.0e2   B. 033D<br>C. 6.8   D. 4.5f<br><b>Answer: B</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>#include <stdio.h><br>int main() {<br>    int a = -1, b = 4, c = 1, d;<br>    d = ++a & & ++b || ++c;<br>    printf("%d, %d, %d, %d\n", a, b, c, d);<br>    return 0;<br>}<br>A. 0, 5, 2, 1   B. 0, 4, 2, 1<br>C. None of the other choices   D. 1, 4, 1, 1<br><b>Answer: B</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>In what aspect did a digital computer not surpass an analog computer?<br>A. Accuracy   B. Reliability<br>C. Speed   D. None of the other choices<br><b>Answer: A</b></td>
  </tr>
</table>

Table 9: Three Random Questions from the Biology, Chemistry and Computer Science of the MMLU-CF Validation Set.
<table>
  <tr>
    <th colspan="2">Engineering</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>What functions can a diode perform?<br>A. Rectifier   B. None of the other choices<br>C. Demodulator   D. Modulator<br><b>Answer: C</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>What is a periodic signal?<br>A. May be represented by g(t) = g(t + T0)   B. Value may be determined at any point<br>C. Repeats itself at regular intervals   D. All of the above<br><b>Answer: D</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>What are the advantages of using electron beam welding?<br>A. Absence of porosity   B. Welds are clean<br>C. Distortion less   D. All of these<br><b>Answer: B</b></td>
  </tr>
  <tr>
    <th colspan="2">Math</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>What is the result when \( \frac{1}{\sqrt{7}-2} \) is rationalized?<br>A. (\( \sqrt{7} - 2 \))/3   B. (\( \sqrt{7} + 2 \))/45<br>C. (\( \sqrt{7} + 2 \))/5   D. (\( \sqrt{7} + 2 \))/3<br><b>Answer: D</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>What is the percentage increase in the area of a rectangle if each side is increased by 20%?<br>A. 46%   B. 44%<br>C. 42%   D. 40%<br><b>Answer: B</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>What is the radius of a sphere with a surface area of 616 cm^2?<br>A. 21 cm   B. 7 cm<br>C. 3.5 cm   D. 14 cm<br><b>Answer: B</b></td>
  </tr>
  <tr>
    <th colspan="2">Physics</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>Daylight color film is calibrated for what type of light?<br>A. 3200 K   B. 3400 K<br>C. 3000 K   D. 5400 K<br><b>Answer: D</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>On a Force versus position (F vs. x) graph, what signifies the work done by the force F?<br>A. The product of the maximum force times the maximum x   B. The length of the curve<br>C. The slope of the curve   D. The area under the curve<br><b>Answer: D</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>What is the phase difference between the voltage and current in a capacitor in an AC circuit?<br>A. \( \pi/3 \)   B. \( \pi/2 \)<br>C. \( \pi \)   D. 0<br><b>Answer: B</b></td>
  </tr>
</table>

Table 10: Three Random Questions from the Engineering, Math and Physics of the MMLU-CF Validation Set.
<table>
  <tr>
    <th colspan="2">Business</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>Beth is the project manager for her organization. While her current project has numerous deliverables identified broadly, the specific details of these deliverables remain unclear. Beth is meticulously planning only the activities that are immediately forthcoming in the project. What is this project management planning approach called?
A. Rolling wave planning    B. Imminent activity management
C. None of the other choices    D. Predecessor-only diagramming
<b>Answer: A</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>How do you format Pivot Table report summary data as currency?
A. Type in the currency symbol    B. Use custom calculation
C. Modify the field settings    D. None of the above
<b>Answer: C</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>Which one of these choices is not considered an operating cost?
A. Maintenance cost    B. Salaries of high officials
C. None of the other choices    D. Salaries of operating staff
<b>Answer: B</b></td>
  </tr>
  <tr>
    <th colspan="2">Economics</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>Which tax proposal did the Finance Minister announce the withdrawal of on 8th March following nationwide protests?
A. Tax on High Income Farmers    B. Tax proposal on EPF
C. Kisan Kalyan Cess    D. All of above
<b>Answer: B</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>In economics, what does the demand for a good indicate regarding the quantity that people:
A. None of the other choices    B. Need to achieve a minimum standard of living
C. Will buy at alternative income levels    D. Would like to have if the good were free
<b>Answer: A</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>What is it called when a firm’s supply rises as a result of implementing advanced technology?
A. Expansion in supply    B. Increase in quantity supplied
C. Contraction in supply    D. Increase in supply
<b>Answer: D</b></td>
  </tr>
  <tr>
    <th colspan="2">Health</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>Thrombocytes are more accurately referred to as ______?
A. Megakaryoblasts    B. Clotting factors
C. Megakaryocytes    D. Platelets
<b>Answer: D</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>Lindsay has been prescribed insulin therapy for which condition?
A. None of the other choices    B. Diabetes
C. Hemophilia    D. Spina bifida
<b>Answer: B</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>Why is it crucial to control and reduce the amount of dust that enters the air?
A. Less dust means less cleaning up afterwards    B. Dust in the air will affect your vision
C. Dust is always in the air and it does not cause harm    D. Constantly inhaling dust particles can cause lung problems in the future
<b>Answer: D</b></td>
  </tr>
</table>

Table 11: Three Random Questions from the Business, Economics, and Health of the MMLU-CF Validation Set.
<table>
  <tr>
    <th colspan="2">History</th>
  </tr>
  <tr>
    <td><b>Question 1</b><br>The constitutional history of France starts with the French Revolution in what year?<br>A. 1786   B. 1780<br>C. 1789   D. None of the other choices</td>
    <td><b>Answer:</b> C</td>
  </tr>
  <tr>
    <td><b>Question 2</b><br>Between 1889 and 1916, where was the Second International, which developed under the influence of Socialist Philosophy, organized?<br>A. None of the other choices<br>B. London<br>C. Paris<br>D. Brussels</td>
    <td><b>Answer:</b> C</td>
  </tr>
  <tr>
    <td><b>Question 3</b><br>What was the capital of the Hoysalas?<br>A. Dwarasamudra   B. Halebeedu<br>C. Sosevuru   D. Belur</td>
    <td><b>Answer:</b> A</td>
  </tr>
  <tr>
    <th colspan="2">Law</th>
  </tr>
  <tr>
    <td><b>Question 1</b><br>How are computer programs legally safeguarded?<br>A. Copy rights.   B. Trademarks.<br>C. Industrial design.   D. Patents.</td>
    <td><b>Answer:</b> A</td>
  </tr>
  <tr>
    <td><b>Question 2</b><br>What type of justice is represented by the penalty imposed for breaking the law?<br>A. Political justice   B. Moral justice<br>C. Legal justice   D. Economic justice</td>
    <td><b>Answer:</b> C</td>
  </tr>
  <tr>
    <td><b>Question 3</b><br>What does WIPO stand for?<br>A. World Information and Patents Organisation<br>B. World Intellectual Property Organisation<br>C. World Information Protection Organisation<br>D. None of the other choices</td>
    <td><b>Answer:</b> B</td>
  </tr>
  <tr>
    <th colspan="2">Philosophy</th>
  </tr>
  <tr>
    <td><b>Question 1</b><br>What does it mean when a reprehensible act is referred to by a different term?<br>A. None of the other choices   B. advantageous comparison<br>C. euphemistic labeling   D. attribution of blame</td>
    <td><b>Answer:</b> C</td>
  </tr>
  <tr>
    <td><b>Question 2</b><br>The assertion, 'Being non-violent is good' is a:<br>A. Religious judgement   B. None of the other choices<br>C. Factual judgement   D. Value judgement</td>
    <td><b>Answer:</b> D</td>
  </tr>
  <tr>
    <td><b>Question 3</b><br>What does the phrase 'lived alone on the forest tree' symbolize?<br>A. None of the other choices   B. Freedom<br>C. A dull life   D. A dependent life</td>
    <td><b>Answer:</b> B</td>
  </tr>
</table>

Table 12: Three Random Questions from the History, Law, and Philosophy of the MMLU-CF Validation Set.
<table>
  <tr>
    <th colspan="2">Psychology</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>Which of the following happens first in development?<br>A. Secondary sexual characteristics    B. Reproductive maturity<br>C. Gender identity    D. Primary sexual characteristics<br><b>Answer: D</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>How can a teacher be successful?<br>A. imparts subject knowledge to students<br>B. presents the subject matter in a well organized manner<br>C. prepares students to pass the examination<br>D. None of the other choices<br><b>Answer: B</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>What is meant by Ex Post Facto research?<br>A. The research is carried out prior to the incident<br>B. None of the other choices<br>C. The research is carried out along with the happening of an incident<br>D. The research is carried out after the incident<br><b>Answer: D</b></td>
  </tr>
  <tr>
    <th colspan="2">Other</th>
  </tr>
  <tr>
    <th>Question 1</th>
    <td>To achieve a quick promotion, he came up with a plan to appease the manager.<br>A. Conciliate    B. Evict    C. Incite    D. Praise<br><b>Answer: A</b></td>
  </tr>
  <tr>
    <th>Question 2</th>
    <td>Which company initiated the secret Zuma Mission for the United States government?<br>A. SpaceX    B. None of the other choices<br>C. XCOR Aerospace    D. Boeing<br><b>Answer: A</b></td>
  </tr>
  <tr>
    <th>Question 3</th>
    <td>In The Calling of Saint Matthew, Caravaggio depicted his subjects wearing the clothing of his own era, rather than that of Jesus’s time.<br>A. to portray the painting’s patrons realistically.<br>B. to conform with other paintings in the series.<br>C. to enable the audience to identify with them.<br>D. so that he could use richer colors and brushstrokes.<br><b>Answer: C</b></td>
  </tr>
</table>

Table 13: Three Random Questions from the Psychology, Other of the MMLU-CF Validation Set.

Reviewer paper summaries: This paper proposes a new benchmark, MMLU-CF, to address the issue of data contamination in the existing Massive Multitask Language Understanding (MMLU) benchmark. Data contamination is categorized into unintentional and malicious types, and the benchmark dataset has been constructed to mitigate both. By leveraging diverse prompting techniques with large language models (LLMs), the questions in the dataset were evaluated and refined from multiple perspectives and criteria. This process resulted in a benchmark dataset that is not only more challenging but also ensures fairness and objectivity.

It has been nearly 4-5 years since MMLU was proposed. Since then, a lot of LMs have measured performance on MMLU and recently, there has been a lot of concern regarding contamination. This prevents fairly comparing whether newly proposed LMs are really good or not compared to existing LMs. Hence, the authors propose a pipeline to automatically construct a new benchmark and name their resulting benchmark as MMLU-CF.

The paper introduces a new MCQ benchmark designed to avoid unintentional and malicious data leakage. It sources data from a broader domain and introduces three decontamination rules. The benchmark is divided into validation and test sets with similar difficulty and subject distributions, and the test set is kept closed.
Summary of strengths: 1. By addressing malicious attack and unintentional leakage through data cleaning and question rephrasing, MMLU-CF improves the reliability of model evaluations.
2. Using LLMs for difficulty calibration, the dataset ensures a balanced distribution of more challenging questions, also with various domains and diverse data sources.
3. The closed-source test set prevents data leakage, while the open-source validation set ensures transparency. This can evaluate if the data is contaminated of not.

- The multi-stage pipeline is reasonable and based on the results, the proposed benchmark does seems to not suffer from contaimination.

- The authors also a hidden test set that could not be contaminated.

- the MMLU-CF benchmark is large and diverse, with 20,000 data points spanning 14 fields.
- in the LLM checking step, multiple LLMs are employed to avoid inherent biases of a single LLM
- it runs evaluation of 40+ models and provides a comprehensive evaluation on model performance over this new benchmark
- users can simply submit evaluation requests by providing hugging face open-source model types or API formats
Summary of weaknesses: 1.	The rules for contamination-free processing appear somewhat simplistic and may not adequately address the memorization issues of advanced models. There is a lack of more sophisticated semantic transformations or automated question generation techniques. 
2.	While the impact of the contamination-free process on model performance has been analyzed, there is a lack of deeper interpretation regarding whether the observed performance drop is due to reduced reliance on memorization or simply the increased difficulty of the questions. Especially, rule 3 of contaminating-free process makes the question more challenging. This might be the major impact of decrease of performance in Table 2. Also, it could harm the difficulty distribution that the paper has intended. 
3.	Although the focus of the paper is on creating a contamination-free benchmark, Table 1 and Table 2 show that the goal is also to design more challenging dataset. It needs to be evaluated based on not only the decrease in performance but also other criteria about contamination. In other words, it seems that a more in-depth evaluation methodology based on the data contamination is needed.

EDIT: Addressed by authors' responses.

- In the "Difficulty Sampling", "LLMs Checking", and "Contamination-Free Processing", LLM(s) are used to filter out and modify the instances. Yet, since LLMs are not perfect there could be false-positive cases where the outcome is not as intended. While important, the paper does not conduct manual analysis on this aspect (e.g., inspecting 100 instances to check if the difficulty level makes sense, checking 100 instances to see if the quality/safety score by the LLMs make sense, comparing rephrased questions and existing questions to see if the rephrasing was conducted successfully). Given that an evaluation benchmark is used to estimate the performance of LMs, I believe that such manual work to ensure the evaluation setting is prepared as intended is very important.

- Related to the first point above, in Section 4.6, I think that just comparing the LM scores when applying different rules does not give any meaningful insight. Such analysis only makes sense when the filtered instances are of high-quality. Yet, since we could not verify whether the 20,000 instances are properly rephrased or the options are altered based on the original instance (without creating new information), it is hard to verify whether the scores changed because of adding and removing a certain rule or if noise was added to the questions.

- the strategy towards malicious data leakage is not strong enough to me. There are many existing datasets and benchmarks that have validation and closed test sets, why is your benchmark particularly robust towards malicious data leakage?
Comments, suggestions and typos: 1.	Expanding the dataset to include question types beyond multiple choice questions, such as descriptive type. Descriptive type questions might show the data contamination more clearly. 
2.	Including details about the hyperparameters and prompts used in the experiments could enhance the paper's reliability. This would allow other researchers to better understand the experimental setup and facilitate reproducibility of the results.

- The main point of the paper is that existing benchmarks (including MMLU) are contaminated and we need to prepare a new one. The authors proposed a pipeline where a new benchmark could be generated without human-in-the-loop, which is scalable. However, as I mentioned in "Summary of Weaknesses", I still believe there should be at least some process to ensure that all the components in the benchmark construction process was conducted as intended. I would like to hear the author's opinion on this matter.

- Given that LMs are trained on a vast majority of information on the internet, it is likely that LMs would have also seen the "200 billion documents from public open-source websites" that the authors use and mentioned in Section 3.2. I would like to hear the author's opinion on what is the reason that the tested LMs do not achieve a high score on MMLU-CF.

- Could the authors upload a zip file of the dataset? (I'll change my datasets score accordingly).

- Lastly, the authors chose to maintain a hidden test-set to prevent contamination on the proposed benchmark. While effective, this assumes that the authors will consistently maintain the benchmark and measure the scores of submitted files from new LMs. What are the author's plans for this matter?

- in the introduction section, "we observe that when only given the questions, some LLMs directly provide the choices and answers, where the choices are exactly the same as those in the MMLU test set". It would be great if you can do a quick experiment on how many such instances are there (proportion)? and does the proportion vary by models or knowledge domains?
- what are other instances or findings that show MMLU has been contaminated, besides "some LLMs directly provide the choices and answers"?
- line 087-123, there are too many detailed experiment results, i'd suggest simplify them in the introduction section and discuss them in details in section 4
- how do you extract MCQ from websites, what are "rule-based methods"?
- what are the public open-source websites? could you provide some examples or include a list in the appendix?
- why do you "exclude questions with choices not labeled as A, B, C, or D"? as we can simply relabel them to A, B, C, and D based on my understanding
Overall assessment scores: 3

3.5

2.5
Metareview score: 3
Metareview: This paper introduces MMLU-CF, a new evaluation dataset that is designed to avoid issues of contamination that make evaluation less reliable. The authors describe methods used to produce this dataset, which include rule-based filtering and model-mediated difficulty sampling. Many leading LLMs are evaluated on this dataset, which demonstrates headroom for improvement. The authors maintain a closed test set on which the community can benchmark their models using open source technologies.
Summary of reasons to publish: * All reviewers appreciated that the dataset is large and diverse.
* Reviews consistently mentioned the simplicity of the methodologies used to create the dataset. Authors justified their use of simple techniques since they were reasonable to produce this important dataset. I agree with both the reviewers and the authors here: methods were very simple, but revisiting these methods feels outside the scope of this paper and better addressed in a follow-up paper.
* A comprehensive evaluation of multiple models was undertaken.