KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks
Authors: Kaijing Ma
Xeron Du
Yunran Wang
Haoran Zhang
ZhoufutuWen
Xingwei Qu
Jian Yang
Jiaheng Liu
minghao liu
Xiang Yue
Wenhao Huang
Ge Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Reasoning; Knowledge-Orthogonal; Rule-Based
Primary area: datasets and benchmarks
TL;DR: We introduce the concept of Knowledge Orthogonal Reasoning (KOR) and propose five types of rule-based reasoning tasks to construct a KOR-Bench to fully evaluate the intrinsic reasoning ability of the model.

Abstract: In this paper, we introduce Knowledge-Orthogonal Reasoning (KOR), a concept aimed at minimizing reliance on domain-specific knowledge, enabling more accurate evaluation of models' reasoning abilities in out-of-distribution settings. 
Based on this concept, we propose the Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench), encompassing five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. 
KOR-Bench emphasizes models' effectiveness in applying new rule descriptions to solve novel rule-driven questions. O1-Preview and O1-Mini achieve accuracies of 72.88\% and 70.16\%, surpassing Claude-3.5-Sonnet and GPT-4o (58.96\% and 58.00\%), highlighting the effectiveness of KOR-Bench. 
We perform detailed analyses, identifying bottlenecks in the Cipher task with Stepwise Prompting, where two rounds of Self-Correction yield optimal results. 
We evaluate performance across three integrated tasks, explore the impact of Tricks on the Puzzle task, and visualize rule-focused attention. Additionally, we conduct an ablation study on dataset size, benchmark correlations, and zero-shot and three-shot "only questions" experiments. 
KOR-Bench aims to enhance reasoning evaluation and support further research in this area.

Published as a conference paper at ICLR 2025
KOR-BENCH: BENCHMARKING LANGUAGE MODELS KOr
ON KNOWLEDGE-ORTHOGONAA REASONING TASKS
Kaijing Mat.5,, Xinrun Du Yunran Wang Haoran Zhang*.7, Zhoufutu Wen', Xingwei Qu Jian Yang* Jiaheng Liu Minghao Liu Xiang Yuel.10, Wenhao Huang! 23, Ge Zhang! 23
MMultimoall Art Projection Research Community, "ByteDance.Inc, '01AI, 20077.A,, TTnngii University, 'Ecole Polytechnique, 'University of Illinois at Urbana-Champaign, Univvessity Manchester, "Nanjing University. "Cammegie Mellon University
mk mk:3385003egmai2 com, muxinrrun2000@maii1.
gezhang@umich.c ch..michhco edu
https://kor-bende hit.hnch..ithu........
ABSTRACT
In this paper, we introduce KNOWLEDGE-- -OREHDDNOOOOTTOONLALLLLLLOOO REASONING (KOR), a concept aimed at minimizing reliance on domain-specific knowledge, enabling more accurate evaluation of models' reasoning abilities in out-of-distribution set- tings. Based on this concept, we propose the KNOw LROWLEDGEE-O ORTHOGONAL REA- SONING BENCHMARK
(OOR-BENCH), encompassing five task categories: Op- eration, Logic, Cipher, Puzzle, and Counterfactual. -DR-Bench emphasizes mod- els' effectiveness in applying new rule descriptions solve novel rule-driven questions. O1-Preview and O1-Mini achieve accuracies of 72.88% and 10%,, surpassing M.ndee-.5--Soneee and GPT-4o (58.96% and 58.00%). highlighting the effectiveness of -on-Bench. We perform detailed analyses, identifying bot- tlenecks in the Cipher task with Stepwise Prompting, where two rounds of Self- Correction yield optimal results. We evaluate performance across three integrated tasks, explore the impact of Tricks on the Puzzle task, and visualize fule-focused attention. Additionally, we conduct ablation study dataset size, benchmark correlations, and zero-shot and three-shot "only questions" experiments. KOR- Bench aims enhance reasoning evaluation and support further research in this area.
INTRODUCTION
HARRY POTTER
MATOL
Reasoning is a fundamental aspect of human intelligence, and research indicates that when models reach a sufficient scale, they exhibit emergent
POKEMON
POWER
advanced rea- soning capabilities such understanding com- plex scenarios, strategic planning, and multi- step execution, making this capability crucial indicator of an intelligent system's 's ability to handle complex tasks (Huang & Chang, 2022; Gui et al., 2024).
WTTCK
WRENEESS
Adveeceed
bevil
Chollange
DRONN GOOR
Counter
Operation
lavet
Propuritionll
Spatial
Kop
Puctte Sertee
Logie
Math
Cipher
Indar tive
1001 INOUSOOON
When learning new tasks and solving new prob- lems, humans are never "starting from scratch"; rather, they "nearly starting from scratch". This phenomenon is evident in various scenar- ios: by understanding game rules, humans can quickly master the gameplay (Nam & McClel- land, 2024); by learning the basic rules of ad- dition, humans can easily solve the problem of adding two numbers of any length (Hu et al., 2024); by giving restrictions and constraints, humans can apply thoughtful methods such as
Verbal
Infermal
Nedure
Clacciooll
WORD
HOLLALLLORSS LINS RSSRMN
AWNSSNC NMOM VINCA
Figure I: Overview of KOR-Bench
Equal Technical Contributions.
Corresponding Authors.

---

Published as a conference paper at ICLR 2025
Reductio ad absurdum and Elimination to solve puzzles (Bill Yuchen Lin, 2024). Human society is abundant with OOD defined-requiring requiring continuous adaptation and the ability to navigate new paradigms. Humans have abilities like abstract, rule-based, and explanatory reasoning, enabling them to learn rules efficiently and adapt quickly specific areas.
tasks (Liu et al., 2021b) those that are novel and un-
Similarly, we expect models to develop similar capabilities so that they can still effectively handle OOD tasks when encountering unfamiliar rules and frameworks, and generate results that conform to specific rules or settings in real-world applications (Sun et al., 2024). Despite the models' re- markable achievements on certain reasoning tasks, the study Mondorf & Plank (2024) points out that they are still challenged by conceptual errors and limitations when dealing with scenarios be- yond the training data. While the incorporation of large amounts of code and data during model training improves the performance of given task, this improvement is based more on the model's memory of the patterns of the training data than on its increased ability follow rules or reason. This reliance in in-domain knowledge limits the effectiveness of existing evaluation benchmarks in accurately measuring a model's s reasoning ability (Wu et al., 2023; Zhang et al., 2023; Dziri et al., 2023). Therefore, there is an urgent need to establish more comprehensive and effective evaluation benchmark to measure the ability of models to understand, follow new rules and solve problems efficiently, while reducing the reliance on pre-trained knowledge.
Inspired by a deeper understanding of the human learning process, we propose the concept of Gnhowledge-Orthogonn
Reasoning" (KOR) explore model's capabilities in reading compre- hension, immediate learning, knowledge transfer, logical reasoning, and problem-solving. while reducing the reliance on the existing knowledge base. Knowledge Orthogonality, formally defined in Appendix A, refers to the independence between background/bomiin-speccff knowledge (e.g.. general knowledge or skills acquired during pre-training) and the rules explicitly defined to solve particular task. It ensures that tosk-solving relies on understanding and reasoning about the task rules, while background knowledge only aids the reasoning process. "Knowledge-Orthogee soning Benchmark" solve new rule-driven questions, rather than relying on data retrieval or information memorization.
Rea- focuses on evaluating how models apply newly-defined rules to
Specifically, we design series of tasks challenge and demonstrate the model's reasoning ability by introducing new elements and rules. These tasks are divided into five categories, each based on one of the following new elements: new symbols, new concepts, new execution rules, new problem- solving frameworks, and new story-context settings. The specific categories are as follows:
Operation Reasoning Task: Understand new definitions of mathematical symbols and
apply this knowledge to perform calculations in mathematical reasoning tasks.
Logic Reasoning Task: Reason and solve problems based on new logical rules and newly categorized logical concepts in logical reasoning tasks.
Cipher Reasoning Task: Perform encryption and decryption operations according to new execution rules in cryptography reasoning tasks.
Puzzle Reasoning Task: Solve puzzles and intellectual games based on newly defined problem-solving frameworks in conditional constraint and combinatorial reasoning tasks.
Counterfactual Reasoning Task: Engage in hypothetical thinking and reasoning within new story contexts in conjectural scenario reasoning tasks.
These tasks push models beyond traditional reasoning frameworks by customizing rules and prob- lems, demonstrating their innovation and adaptability in the face of non-standard problems. We plan to increase the size of the dataset in the future, explore parameterized rules, deepen the inference hierarchy, refine the evaluation of the reasoning process, and expand the multimodal version.
RELATED WORK
To comprehensively assess the reasoning capabilities of large language models, researchers have evaluated them through various benchmark tests, including aspects such as commonsense reasoning (Bang al., 2023; Bian al., 2023; Clark et al., 2018). logical reasoning (Tian et al., 2021; Liu et al., 2021a; 2023), multi-hop reasoning (Yang et al., 2018; Chen et al., 2020; Khashabi et al., 2018), and mathematical reasoning (Hendrycks al., 2021; Arora et al., 2023; Wei et al., 2023).
2

---

Published as a conference paper at ICLR 2025
According to Chen et al. (2024), the realization of reasoning ability hinges on core components: (1) possessing extensive general knowledge of the world, and (2) effectively integrating new infor- mation into existing knowledge base. This framework provides crucial lens through which we can evaluate the reasoning capabilities of L.MSs.
Knowledge-Depender Based Evaluation. Most knowledge-dependeen benchmarks, such as MMLU (Hendrycks et al., 2020), MMLU-Pro (Wang et al., 2024), GPQA (Rein et al., 2023), Com- monsenseQA (Talmor al., 2018), and SciQ (Pedersen et al., 2020), assess model's ability to accumulate and recall data, often struggling to distinguish between true reasoning and simple recall. Designing reasoning benchmarks is challenging because Nomainn-specife reasoning performance. This raises the question: Is the model reasoning or recalling learned pat- terns? Benchmarks like GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) target mathematical reasoning, while FOLIO (Han et al., 2022) and Multi-LogiiEva (Patel et al., 2024) focus on logical reasoning- However, these still rely heavily on domain knowledge, potentially masking genuine reasoning capabilities.
knowledge can obscure
Information Integration Based Evaluation. Moreover, there relatively little research on the ability of (2) models to integrate new information. This imbalance in evaluation hinders a compre- hensive understanding of the model's adaptability and creativity in unfamiliar environments. Some studies have begun addressing this by testing models classic puzzles within specific tasks, such as Zebral Logic (Bill Yuchen Lin, 2024; Berman et al., 2024), Math word problems (Xu et al., 2024), Mathador-LM Benchmark (Kurtic et al., 2024), BeyondX Benchmark (Kao et al., 2024), Connec- tions Game (Todd et al., 2024), Cryptic Crosswords (Sadallah et al., 2024), GridPuzzle (Tyagi et al., 2024), and Crossword Puzzles (Saha et al., 2024). These challenges assess the model's logical rea- soning, spatial cognition, and creative thinking by testing its ability to recognize patterns, apply logic, and derive insights from given information, highlighting divergent and lateral thinking. Addi- tionally, Natural Plan (Zheng et al., 2024) and TravelPlanner (Xie et al., 2024), evaluate the models' information integration and decision-making skills in complex planning scenarios.
Rule-Following Based Evaluation. Recent evaluations are expanding from cntrructionfollowiini to focusing on rule-following capabilities. This trend exemplified by benchmarks such as RuleBench (Sun et al., 2024) for general rule following, LOGICGAME (Gui et al., 2024) for execution and planning reasoning, SearchBench (Borazjanizadeh et al., 2024) for search and problem-solving. and PuzzleBench (Mittal et al., 2024) for combinatorial reasoning. This shift reflects a growing interest in assessing models' reasoning and problem-solving abilities in complex, dynamic environments.
Knowledge Orthogonality Based Evaluation. Building on these research trends, we introduce the concept of "knowledge orthogonality" to address the limitations of current assessment meth- ods. Our approach aims to reduce the impact of domain-specific knowledge on reasoning ability assessment, thoroughly examine rule-following capabilities in OOD scenarios, and provide a more comprehensive and fair evaluation framework.
KNOWLEDGE-ORTHOCON REASONING BENCHMARK
3.1 OVERVIEW
KOR-Bench contains five categories, each containing 25 manually defined rules that are suitably modified to ensure that they do not appear in common pre-training data, maintaining setting that is orthogonal to domain-specific knowledge, Each rule is accompanied by 10 problem instances de- signed to evaluate reasoning based on the rule. For detailed classification of the five task categories in KOR-Bench including the number of corresponding rules and the distribution of answer formats, please refer to Tables 4 and 6 in Appendix C.
3.2 DATA CONSTRUCTION PROCESS
Data construction for -OR-Bench follows three main phases: (1) Rule Design, (2) Rule-Driven Q&A Design, and (3) Quality Validation, shown in Figure 2. The entire data creation process is carried out primarily through manual annotation, with large language models (LLMs) used only for quality validation and difficulty filtering. Details of each phase are in Appendix B.
3

---

Published as a conference paper at ICLR 2025
Data
Rule Design
Kop
Rule Extraction
Rule Redefinition
Quality Validation
Puzzle Websites
Human
Rule-Driven Q&A Design
Human Verification
Books Texts
Q&A
Adaptation
Format Specification
Human
T
Human
Generation
Prior Knowledge
LLM Verification
Human&Code
Figure 2: Overview of the KOR-Bent Data Construction Process.
3.3
DATASET CATEGORIES
3.3.1 OPERATION REASONING
In operation reasoning task, new symbolic operators and corresponding rules are defined, typically involving an operator and its associated equations. These rules are derived from classical mathe- matical operations but have been combined adjusted to align with the concepts and framework of KOR. These rules cover various levels of difficulty and knowledge domains, ranging from elemen- tary arithmetic advanced mathematics. This section only assesses the model's comprehension of the novel rules but also evaluates its reasoning capabilities mathematical operations. The model must be acquainted with classical mathematical operations and apply its understanding of mathemat- ical knowledge in accordance with the newly defined rules to solve these rule-driven questions. For specific descriptions of each rule, please refer to Table
3.3.2 LOGIC REASONING
Rules in the logic section are based on traditional logic textbooks and refined with symbolic ad- justments and innovative definitions to address the specific challenges of KOR-Bench. These rules assess the model's understanding of classical logic and its ability to apply new rules to unconven- tional problems, demonstrating flexibility and innovation. Ten problems of varying difficulty have been designed for each rule. A detailed description of each rule is provided in Table 9.
3.3.3 CIPHER REASONING
Cipher section consists of traditional and modern cryptographic methods, which have been modified address the specific challenges of KOR- Bench. These methods are based on uncommon encryp- tion and decryption techniques found on the Braingle and dCode websites. They have been adapted by altering substitution tables and adjusting certain steps in the encryption process. We verify their accuracy with encryption and decryption programs and generate examples based on these rules. This section tests the model's ability to understand new rules and reason step-by-step according to them. Encryption and decryption involve techniques like transposition and rotation, further testing the model's spatial understanding. Table 10 lists the details of each cipher rule.
3.3.4 PUZZLE REASONING
Rules for the puzzle section are divided into three categories: classic paper puzzles (c.g.. star bat- tle), number games (e.g., sudoku and 24-point), and word games (e.g., anagram). The puzzles
4

---

Published as a conference paper at ICLR 2025
Operation
Logic
Cipher
Rule
Rule
Rule
Define operation such that when multiple
Propositional Symbolization Rules: Equivalence represented by 10=
Encryttion
Convert the message to Morse code, with Morse characters separated by slash "Y" and words separated by double slashes '/f".
ab8b a/b
Negation represented Implication represented by
whon muitiple of
attb b/a
single character remaining at the end, added directly to the end of the ciphertext.
If
if is not multiple of and b is not
Basic Equivalence: (10) A ::::: 8
multiple at8b Both and are integers.
Rule Driven Question
-Diien Question
Rule-Driven Question
Compute 25355
Using Basic Equivalence (10)
Plaintext: "IWWNCXRTWUU"
what equivalent expression is obtained by removing occurrences of in q) i?
Please provide the encrypted answer in the formet [I.J].
XIE M=5 Find
Rule
Rule
game played under each which mine may be hidden or empty.
grid.
Professor Oak renowned Pokérno world for his extensive research Pokémon and their work, relationships with humans, particularly in the field of Pokémon behavior and genetics, is considered groundbreaking and has paved the way for future studies.
squares indicating the number of mines around them (8 squares including the diagonal),
number
You need find all the where mines are located.
Rule-Drivee Question
Rule-Driven Question
X2X3X
considered pioneer study of gonetics?
12332
Gregor Mendel 3. Charles Darwin Professor Oak Bill the Pokémaniac
XXXX2
Puzzle
Counterfactual
Figure Illustration and Examples of the Five Task Categories in KOR-Bench.
sourced from Braingle and Puzzle Prime, two sites that offer classic and original puzzles, as well as challenging and entertaining brain games. A detailed description each puzzle rule provided in Table 11. These rules examine not only mathematical, verbal, and spatial reasoning skills but also the model's understanding of the rules and their use complex, integrated problems. most cases, the model has to use combination of abilities to find the answer. Under each rule, ten problems of varying difficulty were designed based on that rule.
3.3.5 COUNTERFACTUAL REASONING
Counterfactual reasoning aims test the model's ability to navigate hypothetical scenarios and adapt to new rules and environments. This section leverages 25 selected works from anime, television, film, and game foundational world settings. Within these settings, the model must derive answers based on the established worldviews and story rules from the given text information under these new conditions. the model to engage in counterfactual thinking. This tests the model's 's ability to adapt to new rules, interpret fictional contexts, and engage complex reasoning beyond conventional real-world logic. The rule setting for counterfactual reasoning is shown in Table 12.
each case, the questions are crafted to deviate from real-life answers, requiring
3.4 STATISTICS
Table details the KOR-Bench statistics, covering the number and length rules and questions. Ap- pendix C provides further details KOR-Bench. In particular, Table 4 gives a statistical overview of the number of rules different categories, illustrating the distribution rules each category. In addition, Table 8, 9, 10, 11, 12 provide detailed summary summaries of the rules for each category of tasks. Table shows the mean and standard deviation the input and output tokens for each task type in
using GPT-4o example. These statistics not only reveal the characteris-
5

---

Published as a conference paper at ICLR 2025
ties of different task types but also help us assess the differences in their specific demands the computational resources of the model.
Category
Total Rs
Avg. R Len
Max. R Len
Total Qs
Avg. Q Len
Ans. Fmt
Operation
25
51.32
208
250
170.81
NR. ME, SD
Logic
25
1549.12 12
3338
250
411.54
NR. TR, MC TR
Cipher
25
2436.64
6454
250
157.2
Puzzle
25
473.16
767
250
394.9
NR, ME, TR, SD MC
Counterfactual
25
4572.56
9472
250
388.66
Overview of KOR-Bench Statistics. *Note: This table presents the total number of rules, average rule length, maximum rule length, total number of questions, and average question length for five types of reasoning tasks, along with the involved answer formats. The lengths all refer to the number of characters. We define five answer formats: NR (Numerical Response). ME (Mathematical Expression), TR (Textual Response), MC (Multiplc Choice), and SD (Structured Data). Appendix C.2 provides detailed explanation of the answer formats and the proportions of each format across the different tasks.*
Table I:
EXPERIMENT SETUP
Prompting Strategy. Zero-shot prompting strategy for chat model generates responses based on newly defined rules and questions, as outlined in prompt template in Appendix D. Base model uses three-shot strategy, providing three generic Q&A pairs for each rule to support in-context learning.
Evaluation Methodology. We parse the output by regular expression' to try to match the contents of the double square brackets, and if not found, try to match the single square brackets and clean the extraction results. To further improve the accuracy of the analysis, we customise the design of the evaluation script by observing the model's output and processing the problems under some specific rules. After completing the output extraction and special rule processing, it is compared with the answer. Specifically, for mathematical expressions, SymPy (Meurer et al., 2017) is used for parsing in LaTeX format and simplifying the expressions for comparison. The accuracy of the model on each type of task and the overall accuracy on the entire test set are calculated. Comprehensive details regarding the extraction and evaluation can be found in Appendix C.4.
RESULT ANALYSIS
Table presents the performance of the frontier models on COR-Bench, revealing several key in- sights. Overall, accuracy varies significantly across models and task types.
Chat Model Performance. Within the landscape of chat models, Ol-Preview (72.88% and 01- Mini (70.16%) currently demonstrate the best overall performance, especially excelling in the Ci- pher and Puzzle reasoning tasks. In the Cipher category, their accuracies reach 82.80 % and 79.60% significantly outperforming GPT-4o's 42.80 %. On the Puzzle category, they also achieve superior accuracies of 36.80% % and 35.60 %, far surpassing GPT-4o's 16.80 %. further highlighting the advan- tages of the 01 series models in creative reasoning tasks. Meanwhile, Claude-3.5-Sonme (58.96% (6) and GPT-4o (58.00%) follow as the next best-performing models. Claude-3.5-Somae shows better results on Operation and Logic reasoning tasks, especially Logic reasoning. On the other hand, GPT-4o performs better on Cipher and Puzzle reasoning tasks, particularly in Cipher reasoning, dominance that may related its native multimodal nature. This suggests that Claude-3.5-Somme more accurate in understanding and applying rules, while GPT-4o is better at handling tasks that require in-depth analysis and creative thinking. Additionally, Qwen2.5-322 IIIttuct outperforms Swen2.5-72B-Innstuue suggesting that model size alone doesn't 't ensure better performance (McKen- zie et al., 2023).
Base Model Performance. For base models, Meta-Llama 1attmaa3.1-40555. 1-405B achieves the highest overall accuracy at 39.68%. Additionally, the performance of the base model and its associated chat model
'The specific regular expression used r \ (Ns (.*?) \\]

---

Published as a conference paper at ICLR 2025
Open
Operation
Logle
Cipher
Counterfactual
Char Model
O1-preview-2024 09-129229900 (OpenAI, 2024b)
X
72.88
98. 50
63.20
82.80
36,80
92.80 (5.20)
01-mini-2022-099-1 (OpenAI, 2024b)
70.16
82.80
61.20
79.60
35.60
91. (6115.60)
setiie...-somme--200000 2024060002200200 (Anthropic, 2024)
x
58.96
$8.40
67.20
33.20
14.80
91. 91.20(6.00)
GPT-40-2022-05-1 (OpenAI, 2024a)
86.00
92.00k
Mcte-Llana-3,, La----l---3.1-055 1-405 B- Instruct (Dubey ot al,, 2024)
405B
55.36
87.82
$6.80
31.20
13.95
87.66099.00
Qwen2.5. inttt-tstttt (Team, 2024)
36.80
GPT-4-Tuubo-200 1-44-Too--222-0--0000 (OpenAl, 2023)
53.52
90.40
$4.00
23.20
12.80
87. 20/2 60)
Mistral- Large- Dastruct-2409 tteam, 2024)
123B
53.12
$6.80
51,20
22. 80
15.60
89.20/6.80)
Qwen2.5.72BB Instttt (Team, 2024)
87,2018,44)
Meta-LLoma-3, A.mm--3.1-708- Instrutttructttccc (Dibey et al, 2024)
706
50.00
84, 800
49. 20
20:40
7,60
88, 001 40)
Yi-Large
50,00
84,00
47.60
20.80
11,20
86, 40.
Qwen2.5. 14B-Instruct (Team, 2024)
50.00
Meta: Meta-Llama Llama-3-70BB Inntu-lnsttuu (At@ 2024)
70B
49.20
82. 40
46.40
20.40
7.20
89. 39.60/5.201
Domihoo Pro- Pro-1284 128k
48 48.08
$5.20
46, .40
11.20
7.60
50.00)
DeepScek- (DeyScckAA. 2024)
86.80110.00
Qwen2-72B5 72B--22--B8 3---nnttrctttt (Yang et et. 2024)
2024)
72,71B
47.04
78.00
45.60
12.80
9.20
39.60(7.20 20)
Gemma-2-27h- (Team, 2024)
27B
44.48
73.60
49.20
7.20
5.20
87. 22/20/9.22 20)
Phi-3 1-3.5-MM 11--5--Moll-nstone
2024)
16x3.88
88
Gemini- 1.niii----- L.5-Pro (Team et 2024)
13.36
81 60
46.10
6.80
10.80
71 20.20088.
Gemma-2-99-II (Team, 2024)
98
41.60
70.00
39, 60
6.40
6.40
85. 6009 201
Yi- T1-1.5-3BBChh (AI al., 2024)
34B
39.76
79.60
24.40
8.00
3.20
63.6016.800
Phi-3.5-mini-Instmn (Abdin
2024)
Qwen2.5-78-Insse 5-8--nsttr [Team, 2024)
7,618
388 56
55 66)
39, 20
6.40
6,00
85 660 80)
Meta: Li--m--mmm--88B 1033-nstructttt (Dubey 2024)
8B
37.20
60.10
28.80
8.40
2.00
86.40/8.00 00)
Yi.1.5.9B.Chr
al., 2024)
7.60
80.80 10.00)
Meta L.ama-3-344 I-S---nntt Instroct MIMMMes,200
32.80
46.00
20.00
7.60
4.00
86.4015.44 40)
CAni-Commant Brrmannd---Plu--8----
1048
32.72
30.00
34.40
6.80
2,00
90. 5.660)
Yi-L.566--Chut sl.,
77.20:12880
MHii-Command-RRoe R---mmmm----02224
32B
31.12
29.60
28.80
$.20
3.60
88. 88.40(8.00)
Qwen2. Bben2-7B.uee IBB-Instactt (Yang et al. 2024)
7.07B
30.72
28.80
28.00
3.20
4,80
88.80 200
(emmma-2-2--11 2024)
83
Mistral-7BB TB-Imstract-0.33 (Jiang et 2023)
7B
24.16
13.20
19.20
4,80
2.40
81,20011 10
Qwen2 5-1.5B--nstrug 3-8-Iastruct( [Team, 2024)
1,548
20.40
14.80
10.00
D.80
0.80
75 60 60)
OLMo-7B--00 1io-7BB-0724-nstmccs hf (Groeneveld
2024)
7B
18.48
13.20
6.40
1.20
1,20
70.40(8.80)
MAP.Nee-7D 1--e-----nttttt--..... (Zhang et 2024)
10.40
2.00
38, 00..
Qwen2-1 58-Instract (Yang et al.,, 2024)
1.548
14.32
6.80
6.80
0.40
0.80
56. 56.80,14444 40)
Qwen2.5e 0--0.BBBB Instruct (Team, 2024)
0.49B
9.04
4,40
3.20
0.00
0.80
36.80(14.00)
Qwen2-0 Snstrrttctttt (Yang al.. 2024)
0.50
2.00
1.60
12.80(14.40)
Bise Model
Mcta-LJama- 1-405B (Dubey et al., 2024)
4058
68
39.20
51.20
11.20
88:40
858.40 6.000
Qwen2 5-32B (Team, 2024)
32.5B
37.28
38.40
50.00
9.20
6.80
82.00 (1))
2.722 (Team, 2024)
72.7B
37.28
38.80
49.20
10.80
5.20
82.40( 10.80)
Meta-I Jam0-3-708 (At@Meta, 2024)
70B
35 20
30.00
44, 40
7.60
8.00
$6.00 6,00
Qwen2-72B (Yang 2024)
72.71B
34.32
34.00
45.60
7.60
4,80
79.60( 19.60112.440 12.40)
Mcti-L.aama-3117000 (Dubey al., 2024)
708
33
46.40
7.20
13,20,10.000
Gemma-2-276 (Team, 2024)
27B
33.36
26.10
42.40
7.60
5.60
81. 81.50((766)
Qwen2.5.14B (Team, 2024)
14.7B
33.28
30.80
44,80
6.40
5.20
79,20( 14,00)
Yi-1.5-348 al. 2024)
39.
7.20
76.00) 76.00)(14.40
Yi-1,5-9B (A] cl al., 2024)
9B
29 29.20
22. 00
39. 20
8.00
2,80
74.00(1 1020
Qwen2.5-7B (Team, 2024)
766.B
28.80
24.40
34,00
8.00
2,00
75 73.60013.60
Qwen2-7B (Yang
2024)
20.40
76.40014480)
Meta- (Dubey 2024)
8B
26.00
14.00
32.00
5.20
3,20
(..60012440
Gemma-2-9b (Team, 2024)
98
25 52
16.80
35,
6.00
2,80
800 80)
Mcta-Llama-3-8B
2024)
74.40(12.80
Mistral-BB-vO.1 [Jiang et 2023)
7B
21.60
11.20
28.80
2.80
2.40
S0/ 22.80018.880 18.80)
Yi-1.5-6B (Al et al. 2024)
68
20.88
11.60
27.
3.20
2,80
59.60 59.60)22.44)
MAP-Nco-7B (Zlang
2024)
44,00,31,
Owwen255-1.58 (Team, 2024)
1,54B
15.12
12.00
16.00
1.60
40,30
OLMo-78-0724--a (Groenevdd 2021)
7B
14,80
4,80
22.00
1.20
0.80
45 21.20(199 60)
Gemma-2-26 (Team, 2024)
13.20
7.20
15.60
1.60
0.40
21.20/22.800 22.00)
Qwen2-1 55 (Yang et al., 2024)
1.548
12.32
8.80
15.20
0.80
(2,060,,66 80)
Qwen2-0.55 (Yang et al., 2024)
0.19B
9.92
5.20
12.40
0.50
0.10
30,80/22 80)
Dween25500.55 (Team, 2024)
0.49B
6.00
10.80
0.40
1.20
27.20/26.40)
Table 2: Models Performance on KOR-Bench. Noote: Values in parentheses represent the pro-
portion of real-life answers in the counterfactual setting, where lower proportions are better; for all other values, higher proportions are better. For Chat models, the best result is in blue, for Base models, it's in green The second-best is bold, and the third-best is underlined.
shows less decline in the Logic category, compared to a significant drop in other inference tasks. This difference is likely due the shallower depth of inference required in the Logic category.
Reasoning Process Performance. When evaluating reasoning abilities, larger models often trigger Chain-of-Thour (CoT) reasoning automatically, applying rules step-by-step and demonstrating a clear reasoning process in their responses. While they occasionally make execution errors on complex tasks, their overall rule application remains strong. In contrast, smaller models often fail to
7

---

Published as a conference paper at ICLR 2025
activate CoT reasoning. Especially in the Cipher task, smaller models often output "Hello World" as the answer without any reasoning.
Across the five types of reasoning tasks, models generally perform best on the Counterfactual reasoning task, indicating an apparent strength in literal reasoning com- pared to tasks involving mathematical, logical, or theoretical reasoning. Following that, they also perform well on Operation and Logic reasoning tasks, which typically involve one or two levels of reasoning. However, aside from the O1 series models, the models struggle with Cipher and Puzzle reasoning tasks, with maximum accuracy of 42.80 % on the Cipher task and just 16.80% on the Puzzle task, revealing significant weaknesses handling deeper reasoning challenges.
Reasoning Tasks Performance.
Single Task Analysis. Models struggle with algebraic problems involving unknowns but perform better in forward symbolic computation in Operation reasoning. Logic reasoning, construct- ing correct logical expressions remains difficult due to symbolic complexity. In Cipher reasoning, errors are most frequent Position Mapping, Transpose Writing, and Mathematical Calculation, along with Split Connection and Multi-Step Execution. Puzzle reasoning reveals strengths in single- solution tasks challenges multi-step and spatial reasoning. Counterfactual reasoning, as overall model accuracy increases, the ratio of real-life answers decreases, suggesting error from the models' fixed knowledge. Chat models' real-life answer ratios stay below 15%, while base mod- els improve to 36.8% accuracy drops (see Figures 6 and 7 in Appendix E). Appendix G provides emoor case studies for each task.
FURTHER ANALYSIS
We select 16 models for a detailed analysis of their reasoning behaviors, including Claude-3.5. Sonnet, GPT-4o, DeepSeek-V2. V2.5, and six model series: Meta-Lllmma-3.1, Qwen2.5, Qwen2, Yi, Command-R, and Mistral. For each series, we include one large model and one small model. The experiments aim to examine their characteristics, with further details in Appendix F.
6.1 STEPWISE PROMPTING ANALYSIS OF CIPHER TASK BOTTLE BOTTLENECKS
100
Sub Step Accuracy
30
80
Sub-Step Error
20
40
Clude- Someet
Qwen25 326 instrcc
Qwi255 5-78--888800 instrec
Meta- 31-77 -08 insrrst
Go
Mistal Lagg insrinn 2407
DrepSank V2.5
Goant Intteer
Clgi Commend Plus
Meta- 3ama-33 dlt instrud
Large
Gomn25 78 instrecr
Mistral Incturt Ye- g8 Char
Clai-Con Command if
Qwan2 78 manuc
Conditional Filling
Conditonal Reading
Rotation
Compulation
Mapping
Subatitution
Staft
Partition
Encoding
(a) Model Accuracy in Cipher Sub-Steps.
(b) Average Error Rates in Sub-Steps.
Figure 4: Model Performance in Cipher Stepwise Prompting: (a) Accuracy and (b) Error Rates.
In the Cipher Reasoning task, we select five highly erroneous rules and, with human expertise, break down the solution process into sequential sub-steps to guide the LLM in solving the problem step by step. This allows us to perform stepwise prompting analysis, pinpointing challenges and bottlenecks in the reasoning process. There are types of these sub-steps, detailed in Table 13. Figure 4 shows the accuracy of models on cipher sub-steps and the error rates across nine types of sub-steps. An example of dividing problem into sub-steps provided in the Appendix F.1.2.
8

---

Published as a conference paper at ICLR 2025
Results show that error rates for Encoding and Partition relatively low, indicating these not major factors in Cipher reasoning. Eπror rates for Shift, Mapping, and Substitution are higher, suggesting these sub-steps plex calculations affect reasoning. Error rates for Rotation, Conditional Filling, and Conditional Reading are nearly 100%, suggesting spatial operations are bottleneck. Model error rates across all sub-steps are detailed in Appendix F.1.3.
more challenging. High error rates for Calculation indicate com-
6.2
ANALYSIS ON SELF-CORRECTION
Origin
Self Correction
Overall Accuracy(
60
44.96
58.96
66.4
47.76
60.48
68.24
50
57.52
49.36
53.12
64.48
47.04
56.64
54.72
62.32
52.16
62.48
53.68
47.04
50
57.76
42.32
42.32
40
37.2
38.56
35.2
31.12
32.72
31.12
30.72
24.15
20
Cáai- Cdai-Command-R
CAal- Mail-ommand-- R Plus
Claude 3.5 5-Sonnet
DeepSeek-V22 V2.5
Meta- Llama-3 3.1 70-00--o Instruct
Meta- Lan-L-aaaa3311 -.1m--..11 1-88- Instruct
Gpt-48
Mistral-78 Instruct-v0.3 3
Mistral- Hairal-Large-Bie Instinsst-tt22 2407
Qwen2-722 72B Instruct
Qwen2 78- Insttructt
Qwen2.5-32 5--e2.5-332B--2 Instrucct
Qwen255-22 7ann255772-Innttuu Instrutt
Qwen2.5 Dwen2.5-7B-Inas 50l22557B--nntee RR-Instruuct
Yi-1.5-98-CK Chat
Vi-Large
Figure Correcoorsssson's Impact on Overall Accuracy,
We conduct the Self-Correction experiment guide the model in identifying errors, reflecting on their causes, and improving reasoning accuracy. Figure 5 illustrates the results of model self- correction KOR-Bench. With a maximum of rounds, the history may exceed the model's context window, requiring the extraction of the previous round's 's response for re-input. This process involves identifying the relevant response for inclusion in the next input sequence. Appendix F.4.1 provides the self-con offfccorection prompt template used for this purpose.
All models show a significant performance improvement after self-correction, with an average in- crease of 10.36%. Detailed results are in Appendix F.4.2. Figure 11 shows the correction rate from the model's perspective, with the most significant improvement in the first two rounds, and limited gains in later rounds. Figure 12 presents the correction rate by task category, with the Counterfactual category achieving the highest rate of 44.05 % in the first round, and strong corrections in the first two rounds for the other categories, diminishing in the last two rounds.
6.3 ANALYSIS ON COMPLEX TASK PROCESSING
The Complex Task Processing experiment evaluates the model's ability to apply rules to solve mul- tiple problems, manage longer reasoning chains, and test reasoning robustness. It includes three settings: (1) Multi-Q: 1 rule, 1-10 questions; (2) Multi-R: 2-3 rules, 1 question; (3) Multi-RQ: 2-3 rules, 1-3 questions. See Appendix F.5.1 and Appendix F.5.2 for evaluation details. Each set- ting contains random combinations of five reasoning task types, with 1000 examples per type. The model's 'ss task to extract relevant information, reason deeply, and solve problems efficiently.
Table 3 displays the model performance. Claude-3.5-Sonne consistently performs the best across all settings, demonstrating a robust overall capability and resilience against interference. Yi-Large and GPT-40 show similar performance. Mistral-7B-Innttratt-oll performs significantly worse in

---

Published as a conference paper at ICLR 2025
Model
Size
Overall
Multi-Q
Multi-R
Multi-RQ
Close Model
souude-..-soonet-222 GPT-50-2024-05-13
31.37 (43.24)
23.40 ( (42.25)
45.20
25.50
42.28
21.80 (29.40)
15.00 (25.39)
31,20
19.20
(31.62)
Yi-Large
22.73 (31.11)
14.90 (29.09)
33.40
19.90
(30.85)
Open Model
Deepseek-V2. V2.5
236B
21.23 (31.12)
16,50 (31.88)
28.70
18.50 (32.77)
Mistral- Lanruu-Lurge-Istuutt--2e-
123B
18.27 (26.31)
14.80 (27.91)
25.10
14.90 (25.92)
Saii-Command-Re --eieaaddd------------- P-l2--02-222
104B
11.00 (22.94)
9.60
(19.58)
Qwen2-72B- Insttruuctt
72.71B
17.73 (27.03)
14.70 (28.46)
24.60
13,90 (28.03)
Twen2.5-72B-Insttee
72.7B
13.53 (21.26)
13.30 (25.58)
16.00
11.30 (22.20)
Meta-Llama-3.1-10
1attumm---.-70--nntttte Instructt
70B
17.60 (24.71)
14.70 (24.59)
23.90
14.20 (25.63)
Qwen2.5-32B-Instnge
32B
23.97 (33.96)
20.00 (35.13)
33.40
19,90 (33.33)
C4ui-Command-R-08-
32B
16.13 (23.64)
10.40 (21.79)
26.10
11.90 (23.03)
Yi-1.5-9B--hat
9B
(9.47)
5.30 (16.16)
4,90
2.10
8B
7,00 (9.06)
7,60 (11.32)
8.10
5.30 (7.77)
1wen2.5577BInntrruct 1estral7B--Isttuutt-0.333
7,61B
6.77 (12.34)
5,40 (13.79)
9.80
5.10
7.07B
7.47 (14.03)
7,50 (17.87)
8.90
6.00 (15.33)
7B
9.57 (15.52)
4.20 (13.36)
17.70
6.80 (15.50)
Table 3: Evaluation Model Performance Across Complex Task Processing Settings. *Note: The overall accuracy is shown outside the parentheses, while the pass rate for individual sub- problems is inside. The Multi-R Setting has multiple rules but only one question, so has a single value. The best accuracy is in blue, the best pass rate is in green the second-best results are bolded, and the third-best are underlined.*
the Multi-Q setting compared to Multi-R and Multi-RQ, suggesting limitations in handling mul- tiple problems simultaneously. (44ai-Command R-Plus performs poorly Multi-R and Multi-RQ settings, indicating weaknesses in multi-task switching.
6.4 MORE EXPERIMENTS AND ANALYSES
Appendix F.2 provides an analysis of model performance after the introduction of the Trick field in the puzzle task. Appendix F.3 gives the experimental setup and analysis of the Cule-Focusee Attention Visualization based on Retrieval Head (Wu al., 2024), which can be an effective tool for improving interpretability. The generated file is PDF highlighting the attention distribution, which can also be utilized for future expansions the vision version. Appendix H includes some generated examples for reference. Appendix I demonstrates the robustness of through an ablation study. Correlations with other benchmarks show a stronger alignment with reasoning- focused benchmarks, particularly we evaluate the model's s ability to recognize patterns and extract reasoning rules through zero-shot and three-shot "only questions" experiments. In the zero-shot setting, models rely solely prior knowledge, often struggling with accuracy due to insufficient information. In the three-shot setting, models infer rules from three examples, improving performance, as detailed in Appendix K.
to size variations
Appendix J for details). Finally,
7 CONCLUSION
By maintaining orthogonality with domain -specffc knowledge, we introduce KOR-Bench evalu- ate models' reasoning abilities in reading comprehension, immediate learning, knowledge transfer, logical reasoning, and problem-solving, while minimizing the influence of pre-existing knowledge. KOR-Bench provides substantial differentiation and poses significant challenge, as evidenced by Ol-Preview and O1-Mini achieving 72.88% and 70.16%, respectively, while advanced models like Claude-3.5-Somtt and GPT-4o score only 58.96% and 58.00%. We aim for KOR-Bench be comprehensive and challenging benchmark that evaluates models' reasoning abilities while decou- pling them from intrinsic knowledge, ultimately advancing research in reasoning and planning.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our work on KOR-Bench and the associated experiments:
The complete KOR-Bench dataset, including all rules, questions and answers, will be made publicly available upon publication. Detailed information about the data collection process, annotation guidelines, and quality control measures are provided in subsection 3.3.
Dataset:
Code: We have developed and will release comprehensive codebase that includes: Scripts for data loader; Implementation of all evaluation metrics; Code for running experiments.
Model Evaluation: For all baseline models evaluated, provide detailed specifications. For proprietary models, we specify the exact API versions used.
Reproducibility Challenges: We acknowledge that exact reproduction of results for some proprietary models may be challenging due to potential API changes.
We plan to continuously expand the dataset and introduce dynamic initial- ization parameters, such as varying keys and text lengths in the Cipher reasoning task, to enhance rule flexibility and reasoning depth. Additionally, we aim to add more observation dimensions and extend the evaluation multimodal version, including the visual domain.
Future Plans:
By providing these resources and detailed documentation, we aim to facilitate the reproduction of results and encourage further research in this area. We welcome feedback from the community on any aspects that require additional clarification to ensure full reproducibility.
ETHICS
Our research prioritizes ethical considerations in the development of the KOR -encBench dataset. We ensure that all data used is collected responsibly and that participant privacy is maintained. Ad- ditionally, we are committed to transparency in our methodology to prevent biases and promote fairness in the evaluation of models. We recognize the importance of ongoing ethical oversight as we refine and expand the dataset.
In the future, we plan to continuously update and expand the dataset. We also plan to introduce dynamically configurable initialization parameters, such as implementing dynamic keys and text length variations in the Cipher reasoning task. This will enhance the flexibility the generated rules, thereby influencing the required depth of reasoning. We plan to add more observation dimensions enhance the evaluation of the reasoning process and extend it to the visual domain, developing into a multimodal version.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter. Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu. Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song. Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward. Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu. Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Ji- long Xue, Sonali Yaday, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan,
11

---

Published as a conference paper at ICLR 2025
Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https::////...... aaxii.iv org/ abs/ / 22402444440 14219.
01. AI, :. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen. Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by O1.ai, 2024. URL Bolhhttss:/arxiiv abs/2403. 04652.
AetMeta. Llama 3 model card. 2024. URL https://gtttttiieeee http:////ggtttbb.ccommmeaaaallla
1lama3/blok
CARD md.
Anthropic. Claude 3.5 sonnet model card addendum, 2024. URL https://ww,
paperswi www.de.tthodd...oo..o.ooo...e... com hatmm//pee/claud--555555 we----onnt--oeeeeea------e----e- Accessed: 2024-09-21.
Daman Arora, Himanshu Gaurav Singh, al. Have ilms advanced enough? challenging problem solving benchmark for large language models. arXiv preprint vrxiv:2305. 2023.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity. arXiv preprint DXXix2332.040233 2023.
Shmuel Berman, Baishakhi Ray, and Kathleen McKeown. Solving zebra puzzles using constraint- guided multi-agent systems. arXiv preprint arXiv:2407.03300 2024.
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, Ben He, Shanshan Jiang, and Bin Dong. Chatgpt is knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. arXiv preprint arXiv:2303. 16421, 2023.
Yejin Choi Bill Yuchen Lin, Ronan Le Bras. Zebralogic: Benchmarking the logical reasoning ability of language models, 2024. URL ZebraLogic.
ctt.e/accceccceeeeeeeeeeooooeeee http://www......././///eaaaaaaaa pataces/allende
Nasim Borazjanizadeh, Roei Herzig, Trevor Darrell, Rogerio Feris, and Leonid Karlinsky. Navigat- ing the labyrinth: Evaluating and enhancing llms' ability to reason about search problems. arXiv preprint arXiv:2406. 20l.:2200..1172... 2024.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. Hy- bridqa: A dataset of multi-hop question answering over tabular and textual data, arXiv preprint arXiv:2004. 07347, 2020.
Zeming Chen, Gail Weiss, Eric Mitchell, Asli Celikyilmaz, and Antoine Bosselut. Reckoning: reasoning through dynamic knowledge encoding. Advances in Neural Information Processing Systems, 36, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: 0188800355577 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL http::/urxiv. org/abs/21 10. 14168, 2021.
DeepSeek-AL. Deepseek-v2 A strong, economical, and efficient mixture-of-expprts language
model, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint 47 2024.
12

---

Published as a conference paper at ICLR 2025
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang. et al. Faith and fate: Limits of transformers compositionality (2023). arXiv preprint arXiv:2305.18654 2023.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff. Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lam- bert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannanch Hajishirzi. Olmo: Accelerating the science of language models. Preprint, 2024.
Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Logicgame: Benchmarking rule-based reasoning abilities of large language models. arXiv preprint arXiv:2400S 15778, 2024.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, et al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.000 00040, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arv: 103.03874, 2021.
Yi Hu, Xiaojuan Tang, Haotong Yang, and Muhan Zhang Case-based or rule-based: How do transformers do the math? arXiv preprint arXiv:2402. 2024.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint 12 10403, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv.org org/ /aabbs/2333 22310. 06825.
Kuei-Chun Kao, Ruochen Wang, and Cho-Jui Hsich. Solving for x and beyond: Can large lan- guage models solve complex math problems with more-than-two unknowns? arXiv preprint arXiv:2407.055 05.007005/344 2024.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 252-262,
2018.
Eldar Kurtic, Amir Moeini, and Dan Alistarh. Mathador-Im: dynamic benchmark for mathemati- cal reasoning on large language models. arXiv preprint arXiv:2406. 1255222 2024.
Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. Natural language inference in context- investigating contextual reasoning over long texts. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11388-13394 2021a.
Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0-an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-o---istiibuttin generalization: survey. arXiv preprint arXiv:2 108. (08./300 /3644, 2021b.
13

---

Published as a conference paper at ICLR 2025
Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. Inverse scaling: When bigger isn't t better. arXiv preprint arXiv:2306.09000000 006.09996 2023.
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rath- nayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:3 January 2017. ISSN 2376-5992. doi: 10.7717/peerj-cs. 103. URL https: doii org/oo/g///ooooo.ooo 771 717771/ppeee ppeerjeeeeeee cs 103.
Chinmay Mittal, Krishna Kartik, Parag Singla, et al. Puzzlebench: Can Ilms solve challenging first-order combinatorial reasoning problems? arXiv preprint arXiv:2402.0261/ 2024.
Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models-a survey, arXiv preprint arXiv:2404.01869, 2024.
Andrew Nam and James I. McClelland. Systematic human learning and generalization from a brief tutorial with explanatory feedback. Open Mind, 8:148-176, 2024.
OpenAI. Gpt-4 technical report. Technical report, OpenAI, 2023. Utthtps:://cee opp.a/ com/papers/gpte .dff.
OpenAL. Gpt-4o system card. Technical report, OpenAI, 2024a. research/ httpchh/pppp////
OpenAl. Openai ol: Learning to reason with Ilms, 2024b. URL https::/openai.. com/
index/learning to reason with- 1lms/.
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varsh- ney, and Chitta Baral. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models. arXiv preprint arXiv:2406. 17169, 2024.
Pedersen, M Otokiak, I Koonoo, J Milton, E Maktar, A Anaviapik, M Milton, G Porter, A Scott, C Newman, al. Sciq: invitation and recommendations to combine science and inuit qauji- majatugangit for meaningful engagement of inuit communities in research. Arctic Science, 6(3): 326-339, 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. arXiv preprint arXiv:2311. 12022, 2023.
Abdelrahman Sadallah, Daria Kotova, Ekaterina Kochmar, et al. Are llms good cryptic crossword solvers? arXiv preprint arXiv:2403. 1.04003./099. /2094, 2024.
Soumadeep Saha, Sutanoya Chakraborty, Saptarshi Saha, and Utpal Garain. Language models are crossword solvers. arXiv preprint arXiv:2406.0900 09043, 2024.
Wangtao Sun, Chenxiang Zhang, Xueyou Zhang, Ziyang Huang, Haotian Xu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. Beyond instruction following: Evaluating rule following of large language models. arXiv preprint aTXiv:2407.084400 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. arXiv preprint arXiv: 2018.
Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabriela Surita. Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornaahho Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Manaal Faruqui, Natalie Clay, Justin Gilmer, JDD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki
14

---

Published as a conference paper at ICLR 2025
Morioka, Kevin Hui, Krishna Haridasan, Victor Campos, Mahdis Mahdieh, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Pagannni, Xavier Garcia, Fan Yang. Dasha Valter, Maja Trebacz, Kiran Vo- drahalli, Chulayuth Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Sid- dhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junbyuk Oh, Tom Hennigan. George van den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, San- tiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Khah Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puigdomenech Badia, Piyush Paril, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da- Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Ma- teo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Quoc Le. Arjum Kar. Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lam- prou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang. Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope. Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May, Zhen Yang. Johan Schalkwyk, Christina Butter- field, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Mar- vin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackimnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap. Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu. Ross Mcllroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojeiech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mineu, Yujia Li, Ravi Addanki, Kalpesh Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, Hyun- Jeong Choe, Alex Tomala, Chalence Sifrannek-Shrader Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli. Connie Tao, Rajkumar Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakićević, Do- minik Grewe Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, lain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, An- geliki Lazaridou, Xi Xiong, Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang. Basil Mustafa, Albert Webson, Hyo Lee. Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay. Anudhyan Boral, Lisa Anne Hendricks, Corey Fry, Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RI Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkellsso, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen. Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan, Zeyncep Cankara, Soo Kwak, Yun- han Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Fan, Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananey, Anastasija Ilie. Taylor Tobin, John Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty,
Krishna,
15

---

Published as a conference paper at ICLR 2025
Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, El- naz Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak, Su- san Zhang, Michael Azzam, Khe Chai Sim, Sergi Caelles, James Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil Has- sas Yeganch, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang, Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Due Dung Nguyen, James Svensson, Le Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong. Filip Pavetic. Sebastien Cevey, Ramona COmanescu, Richard Ives, Sherry Yang. Cosmo Du. Bo Li, Zizhao Zhang, Mariko linuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Saputro, Anita Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei. Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Blo- niarz, Jaehoon Lee, Pedram Pejman. Paul Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong. Nikolay Savinov, Charlotte Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker. Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shrivastava, James Cobon-Kerr, Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misiunas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, An- ton Briukhov, Neil Houlsby, Mia Glaese, Balaji Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaek Lim, Fangxiaoyu Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghami, Jenny Brennan, Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Amold,, Solomon Chang. Julian Schrittwieser, Elena Buchatskaya, Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hellendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksei Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yiin Chang, Craig Swanson, Biao Zhang, Benjamin Lee, Paul Kishan Ruben- stein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aish- warya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi Hashemi, Jessica Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger. Dennis Daun, Khuslen Baatarsukh, Matthew Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Fe- lix de Chaumont Quitry, Charline Le Lan, Tom Hudson, Qingze Wang. Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Iwan- icki, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srini- vasan, Hannah Muckenhirn, Honglong Cai, Amol Mandhane, Mukarram Tariq. Jack W. Rae, Gary Wang, Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Gar- rette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hut- ter, Vahab Mirrokni. Nanxin Chen, Kaisheng Yao, Zoltan Egyed, Francois Galilee, Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thomton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank V. Gabriel Carvajal, Josef Broder, Iulia Comsa, Alena Repina, William Wong, Warren Weilun Chen, Peter Hawkins, Egor Filonov, Lucia Loher, Christoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Bums, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, lonel Gog, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis C. Cobo, Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaixiu Steven Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Contamaria-Femanddee Wojeiech Fica, Xinyun Chen, Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, S. M. Ali Eslami, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen Srini-
16

---

Published as a conference paper at ICLR 2025
vasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fermando, Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang Luong, Boyu Wang. Sujeevan Rajayogam, Julian Fisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Di- ane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann- Rice, Krystal Kallarackal, Rosamne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou, Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzoca, Shourya Sarcar. Hi- lal Dib, Jeff Stanway, Frank Perbet, Neje Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka, David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Uy. Andrew Bolt, Olivia Wiles, Mi- lad Nasr, Ili Shumailov, Marco Selvi, Francesco Piccinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galic, John Carpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mah- moud Alnahlawi, Brian McWilliams, Sankalp Singh, Annie Louis, Wen Ding. Dan Popovici, Lenin Simicich. Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatchi, Jo- vana Mitrovic, Alex Grills, Joseph Pagadora, Dessie Petrova, Danielle Eisenbud, Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tripuraneni, Yannis Assael, Thomas Brovelli, Prateek Jain, Mihajlo Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi, Yi Yao. Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Elisseeff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Latorre-Chimoto Solomon Kim, William Zeng, Ken Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher A. Choquuette-hhoo James Manyika, Brona Robenek, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko Velic. Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Wooyeol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Kaplanis, XiangHai Sheng, Yuri Chervonyi, Caglar Unlu, Diego de Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dan- gyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garmon, Adrian Goedeckemeyer, Adam R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh Jazayeri. Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan, David Engel, Francesco Pongetti, Dario de Cesare, Dongseong Hwang, Lily Yu, Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan Li, Asaf Aharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loie Matthey, Braman- dia Ramadhana, Austin Matthews, CJ Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang. Manish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree Boppana, Taylan Bilal, Yuma Koizumi, Ying Xu. Yasemin Altun, Nir Shabat, Ben Bariach, Alex Korchemniy, Kiam Choo, Olaf Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene Cai, Shariq lqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chaitanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica Pa- traucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang Nguyen, Ji Liu, Madhavi Se- wak, Bryce Petrini, DongHyun Choi, Ivan Philips, Ziyue Wang. Ioana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka Agrawal, Xiaowei Li. Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach, Sadh MNM Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar, Alek Wenjiao Wang, Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Fred- erick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei. Riham Mansour, Tomasz Kepa,
17

---

Published as a conference paper at ICLR 2025
François- François-Xavier Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejezuk, Mark Geller, Niccolo Dal Santo, Valentin Anklin, Majd Al Merey, Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
05530.
Gemma Team. Gemma. 2024. doi: 10.34740KKAAGGL////33300 URL
com/m/33 3301.
Mistral AI team. Mistral
large 2, 2024.
URL https://mistral.
mistral- - Laaalarrrg00
Accessed: 2024-07-24.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhtttss://gwenlm
github. ia/blog///////////oo hwwpbl/g///ee.......c. qwen2.5/
Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first-order logical reasoning ability through logicnli. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 3738-3747, 2021.
Graham Todd, Tim Merino, Sam Earle, and Julian Togelius. Missed connections: Lateral thinking puzzles for large language models. arXiv preprint arXiv:2404.11733 2024.
Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin RRV, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, and Chitta Baral. Step-by-step reasoning to solve grid puzzles: Where do Ilms falter? arxiv preprint arXiv:2407.14:: 14790, 2024.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et Mmlu-pro: more robust and challenging multi-task language understanding benchmark. arXiv preprint a7Xtv:2400.01154,,000 2024.
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. Cmath: Can your language model pass chinese elementary school math test? arXiv preprint arXiv:2306. /6636, 2023.
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanisti- cally explains long-context factuality. arXiv preprint arXiv:2404. /5574, 2024.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.0244 2023.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:240..01622, 2024.
Xin Xu, Tong Xiao, Zitong Chao, Zhenya Huang, Can Yang, and Yang Wang. Can Ilms solve longer math word problems better? arXiv preprint arXiv:2405. 14804, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei. Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin. Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang. Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https:/ / //tr///avvvv///////////ee///aavl wwt./aas.////ooooooooooo.....aa/ 2407 10671.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: / 1809.89600 2018.
18

---

Published as a conference paper at ICLR 2025
Chiyuan Zhang. Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. Advances in Neural Information Processing Systems, 36:39321 2023.
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que. Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv: 2400.19321 2024.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Tze Cheng, Quoc V Le, Ed Chi, et al. Natural plan: Benchmarking Ilms on natural language planning. arXiv preprint arXiv:2406. 04520, 2024.
19

---

Published as a conference paper at ICLR 2025
Appendix
Formal Definition of "Knowledge Orthogonality"
22
Data Construction Process
23
B.1 Rule Design
23
B.2 Rule-Driven Q&A Design
................................
23
B.3 Quality Validation
23
Details of KOR-Bench
24
C.I Rule Distribution Across Task Types
................................
24
C.2 Answer Format Distribution Across Task Types
24
C.3 Statistical Overview of Input and Output Tokens
24
C.4 Detailed Extraction and Evaluation
26
C.5 Summary of Rule Descriptions for Five Task Types
26
Prompt Templates
32
D.I Operation Prompt
32
................................
D.2 Logic Prompt
................................
33
D.3 Cipher Prompt
34
................................
D.4 Puzzle Prompt
D.5 Counterfactual Prompt
36
Analysis of Real-life Answer Ratios for Counterfactual Task
37
Details of Further Analysis
38
F.1
Stepwise Prompting Analysis of Cipher Task Bottlenecks
38
F1.1
Detailed Explanations of Nine Key Sub-Step Types
E.1.2
Example of Cipher Question Split into Sub-Steps
39
F.1.3
Model Results on Sub-Step Error Rates
40
F2
Impact Analysis of Tricks Puzzle Task Performance
42
F.2.1
A Case Study of Incorporating Tricks in Puzzle Reasoning Task
42
F3
Attention Focus Visualisation
43
F.4
Analysis on Self-Correction
44
F.4.1
Self Correction Prompt Template
44
F.4.2
Impact of Round Count on Self-Correction Accuracy
44
F.5
Analysis on Complex Task Processing
46
F.5.1
Complex Task Processing Prompt
46
F.5.2
Complex Task Processing Evaluation
46
Fun-Filled Analysis of Slip-Ups
47
G.I
Operation Error Cases Analysis
48
G.2 Logic Error Cases Analysis
53
G.3 Cipher Error Cases Analysis
63
G.4 Puzzle Error Cases Analysis
76
20

---

Published as a conference paper at ICLR 2025
G.5 Counterfactual Error Cases Analysis
85
H Attention Focus Visualisation Cases
89
H.1 Cases for Qwenn.5--B-Inssrruct
"...............................
89
H.2 Cases for Qwenn.5-77B-Inntrruet
89
1 Ablation Study on Dataset Size
97
Correlation analysis with other benchmarks
99
K Zero-Shot and Three-Shot "Only Questions" Experiments
100
21

---

Published as a conference paper at ICLR 2025
A FORMAL DEFINITION OF "KNOWLEDGE ORTHOGONALITY"
For a task 7, the required reasoning information consists of:
K: General background/domain--peeer knowledge acquired during pre-training, exclud-
ing common sense.
R: Core rule information designed to solve T",
Q: Rule-Driven question.
A: Answer the question Q.
Notational Definitions:
- Represents the cognitive process of deriving A from Q.
P: Represents the belief strength that A is a valid answer to Q based on R and/or K.
P(Q - A R): Belief A driven solely by R.
-(QQ - A K): Belief in A based solely on K.
PPQQ A R. K): Combined belief in A, integrating R and K.
T' satisfies knowledge orthogonality under the following conditions:
Knowledge-Rund Decoupling: Rule
logically self-contained and independent of K.
R K
2. Knowledge Assistiveness: Background knowledge K may support or interfere with the derivation of A from Q, but does not play a central role in reasoning. The extent of this influence is quantified by the Knowledge Impact Factor (8), defined as:
P(Q + A R. K) P(Q + A R)
P(Q - A R)
3 ranges from (-1,c), where e a very small positive number.
When 3 is positive and close to 0, has little impact, with R being dominant.
When 3 is negative, it can range from small negative values to approaching -1, where K increasingly undermines reasoning.
3. Rule Centrality: Correctness relies understanding and applying R, with R having sig-
nificantly greater influence than K.
P(Q ~ R, K) P(Q
R) P(Q A K)
4. Derivation Adjustment: This formula adjusts the reasoning process based on R, incorpo- rating the influence of K with reflecting its effect.
P(Q A K) P(Q + A R) (1+ B)
22

---

Published as a conference paper at ICLR 2025
B DATA CONSTRUCTION PROCESS
KOR-Bench data construction unfolds three phases: (1) Rule Design, (2) Rule-Driven Q&A
Design, and (3) Quality Validation, shown in Figure Manual annotation drives the process, with large language models (LLMs) used for quality validation and difficulty filtering. Detailed explanations of these phases follow.
B. RULE DESIGN
Core rules are extracted from logic puzzles, textbooks, domain knowledge, or virtual world settings and defined as natural language descriptions.
Rule Extraction:
Rule Redefinition:
Expand redefine existing rules by incorporating symbols, concepts, constraints, execution steps, or introducing novel story contexts.
B.2 RULE-DRIVEN Q&A DESIGN
Q&A Adaptation: Existing questions are adjusted align with the extracted rules, and both ques- tions and answers are annotated.
Q&A Generation: Questions and answers are either manually crafted (e.g., Counterfactual prob- lems where answers differ from real-world facts) programmatically generated (e.g., Cipher prob- lems).
Answer Format Specification: Answers to different questions are assigned specific formats, in- cluding NR (Numerical Response), ME (Mathematical Expression), TR (Textual Response), MC (Multiple Choice), and SD (Structured Data).
B.3 QUALITY VALIDATION
Human Verification: Human evaluators assess the quality of rules and Q&A pairs.
LLM Verification:
We evaluate the dataset using LLMs to assess its difficulty and discriminative power. Tasks where models often fail may indicate excessive difficulty or unclear descriptions, while universally correct answers may suggest overly simple setups data leakage. Throughout the dataset construction process, we repeatedly revise these issues after each evaluation.
23

---

Published as a conference paper at ICLR 2025
C DETAILS OF KOR-BENCH
C.1
RULE DISTRIBUTION ACROSS TASK TYPES
The following table 4 shows the distribution of rule counts across categories within the five task types.
Category
Subcategory
Description
Rule Count
Total Rules
Total Questions
Elementary arithmetic
Basic Level
Power and square root
Exponential and logarithmic
Operation
Advanced Level
Operation on complex numbers
25
Derivative
Operation
Calculus
Challenging Level
Operation on matrices
Propositional Logic
Predicate Logic
Formal Logic
Modal Logic
Logic
25
250
Inductive Logic
Informal Logic
Informal Logic
5
Monoulphahetic Cipher
Polyalphahetic Cipher
Clussical Cryptograhhy
Polygraphic Cipher
Transposition Cipher
Cipher
25
250
Symmetric Cipher
Modern Cryptography
Asymmetric Cipher
Hash Function Cipher
Verbal only
Verbal
Verhal Mathematical
Verbal & Spatial
2
Puzzle
25
Mathematical only
Mathematical
Mathematicul Sputial
Spatial
Spatial only
Counterfactual
25
25
250
Total
125
1250
Table Statistical Overview of Rule Distribution. This table presents the hierarchical categoriza- tion of rules within five task categories, including subcategories and tertiary classifications, along with their corresponding rule counts.
C.2 ANSWER FORMAT DISTRIBUTION ACROSS TASK TYPES
Table gives explanations and examples of the five answer formats.
Table 6 shows the distribution of different categories of answer formats across the five task types.
C.3 STATISTICAL. OVERVIEW OF INPUT AND OUTPUT TOKENS
Table presents the number of input and output tokens generated by GPT-4o across the five task types. The tokenizer used is the cl/00k base from OpenAl's tiktoken library, which is specifically designed for efficiently encoding and decoding text for GPT-4 and GPT-3.5 models.
24

---

Published as a conference paper at ICLR 2025
Category
Explanation
Cases
Numerical Response(NR)
An answer format contains one more numeric values and contains only purely numeric values.
[[1//]]| [[24]]. [[4]]
Mathematical Expression(ME)
An answer format that mathematical notations, symhols, and operations to rep- relationship equation.
[[2zsin(x) + + cos())]] - 5|| S
Textual Response(TKe
answer format composed entirely of text, including complete sentences other paragraphs of characters.
||I|
[SI163:::22]],
[334665209922222cc5555bb
Multiple Choice(MC)
An answer format which one set multiple choices is selected as the answer.
IAIL [1BII. I[C]
Structured Data(SD)
An answer format that organises the output into specific structure set by the question.
10 = N,N = = = 2]}
[[ (12,
[11694]594727211]]
Table 5: Explanation and Examples of Answer Formats. This table provides explanations and examples for the five answer formats.
Category
Numerical Response
Mathematical Expression
Textual Response
Multiple Choice
Structured Data
Operation
177 (70.80%5)
43(17.20%)
30 (12.00%)
Logic
(5.20%)
(34.80%)
150 (60.00%)
Cipher
255 (100,005)
Puzzle
10 (4.00%)
30 (12.00%)
40 (16.00%)
170 (68.00%)
Counterfactual
(100.005%)
Table 6: Statistical Overview of Answer Format Distribution. This table shows several answer formats and their numbers and percentages for the five types of tasks.
Category
Input Tokens
Output Tokens
Mean
Std. Dev.
Mean
Std. Dev.
Operation
179.51
27.28
316.52
157.94
Logic
628.18
169.04
230.23
237.57
Cipher
823.41
409.16
451.24
340.77
Puzzle
345.38
102.84
629.14
288.03
Counterfactual
1138.23
417.67
68.332
50.96
Table 7: Token Statistics for KOR-Bench. This table shows the mean and standard deviation of the number of input and output tokens for GPT-4o for each type of task problem.
25

---

Published as a conference paper at ICLR 2025
C.4 DETAILED EXTRACTION AND EVALUATION
To ensure evaluation accuracy, we establish set of detailed extraction rules. First, we use the regular expression r' AAANN x?) Mss>]Vy N' parse the output, attempting to match the content within double brackets. If this fails, we try to match single brackets and clean the extracted result, including removing quotation marks, line breaks, and spaces. To further enhance the precision of the analysis, we tailor the evaluation script based on the characteristies of the model output and specific rules. Below are some of the main settings:
Multiple Answer Handling: If the question allows multiple answers separated by "or", we remove the "[[]"" and split both the response and the answer by "or". Then, we trim the whitespace, sort the resulting parts, and compare the sorted lists determine if they match.
Mathematical Expression Handling:
For equation-basee based questions, we only need to ensure that the result equals a specific value. We extract the mathematical expression, process the symbols, and directly calculate to check correctness.
For questions requiring mathematical expression (such derivative), SymPy (Meurer et al., 2017) library's 's carse_latex function to parse both the response and the answer, then simplify the results using the simplify function before comparing them.
the
For inequality- questions (such as 2 6), we use the regular expression r' $) \s* ( [[-]? <\d+) ?\dx) to extract the inequality and compare the extracted results.
Unordered List Handling: If the order of the answers is unimportant, we extract the text content from both the response and the answer, normalize the data (such as cleaning and sorting), and then compare them.
C.5 SUMMARY OF RULE DESCRIPTIONS FOR FIVE TASK TYPES
The following five tables Table 8, 9, 10, 11, 12 present summaries of rule content for five task types. Each table provides detailed list of specific rules and their descriptions for the corresponding task type.
26

---

Published as a conference paper at ICLR 2025
Rule ID
Title
Description
Define operation such when a multiple a/b + when is a multiple of a, axb = b/a 2; if a is not a multiple of and is not multiple of a, ambb = 24
*
2
AOB=AA+3BXAAB)
<abb,c.b=2ab+o-d
#
a#b is the average all even numbers between and
5
8
acch=a² +
operation $ means select the larger of the two numbers operation $ means select the smaller of the two numbers
6
Multiple Operators
Multiple Operators
agb=(a-h))2:ad/bbxxxxx
Multiple Operators
abb= Vā 12 a22b=
9
abb=
10
acb=log, + log, b
11
aWb=a" - 8a
12
%
a8bb=n' + vab
a@b=a + bi
14
a0h=(a + bi)²
15
A
(Ag=(J((()))))
16
(0==)(())) +''(())
(Vg=f(>) + +g)(x)
18
AEB=(AU) B) (An
19
a=b= 1 2x do
20
b= (x)
21
D= f(a, da dy
A&B denotes the clement-by-cllmmet power operation
(A& A. of matrix A and matrix
A L@B denotes the alement-by-elemenn maximization operation
(A&B);; max(A.j. B,₁) of matrix A and matrix B
E
AEB=2A+3B,
are matrices.
Table 8: Summary of 25 Rules for Operation Reasoning Task. This table gives the Rule IDs, titles, and brief descriptions of the 25 rules under the Operation Reasoning task for review.
27

---

Published as a conference paper at ICLR 2025
Rule ID
Title
Deseription
Propositional Logic Formalization
Introduce propositional lngic symhols with precedence. Introduce customised notion of formula lovel for A, B, C, differing from standard definitions, specifying truth/fallse assignments.
Equivalence Calculus
Introduce unique symbols for logical operations, differing from standurd defimitions. Specify 16 hasic equivalence equations, restrictions on simplest expression, and Truth Value Judgment Steps.
3
Disjunctive Normal Form and Conjunctive Normal Form
Define and denote simple/naree conjunctiveddiiinnttii forms and principal disjunctive normal form, differing from standard definitions. Five types include tautology, contradiction, basic, all-even, all-odd formulus.
4
Resolution
Definitions and arithmetic rules for Literal, Complenment, and Resolution. Detailed steps for determining that conjunctive normal form Resolution Algorithm with true assignment
5
Circuit Diagram
A simplified circuit diagram illustrating logical operators, with symbotic representations of inpuls and oulputs, as well as indications for powered and unpowered states.
Predicate Logic Formalization
Use unique symbols for quantifiers, logical operators, differing from standard definitions. Formalise prodicate logic representation under individual domains with n meta predicates, properties, relations.
7
Interpretation of Propositions
Composition of logical language M C.Calullation steps for Formulas under interpretation J.
8
Propositional I Logic Concepts
Compose Direct Propositions with unique elements: S. P. C. Q. Intreduce Logical
Forms: A, E, I, O, Singular Aft/Neg. Outline preroquisites for relationships. Introduce four unique types of relationships.
Derivative Reasoning Propositional I Logic
Definitions conversion steps and applicable propositions three straightforward propositional conversion methods A,B.C.
10
Figure of the Syllogism
Symbolic representation of four propositional types O,E.I.O.Fom and Valid Moods of the Four Figures of the sy syllogism.
11
Truth Value Modal Propositions
Introduce unique symbuls for necessity, possibility. proposttions logical operators. 4 unique Modal Proposition Relationships. 16 Logic Inference Formulas.
Canonical Propositions
Introduce unique symbols for obligation, permission, prohibition modalities. Propositional pairs and properties of four types of normative propositional relations. 12 Normative reasoning formulas.
13
Temporal Propositions
Unique symbols for past/fiture paints/perinds and present. unique Time Proposition Relationships. 24 Time Proposition Inference Formulas.
p
Epistemic Logic
Unique logical symbols for Belief, Common Belief, and Douht Components Cognitive Logic Model and Definition of Common Belief, Cognitive Logic Axioms Axioms,Advaneee Axions,Axioms of Douht.
15
Dynamic Logic
Fornal notation for commands, propusitions. Dynamic operators of necessity. possibility. 12 Axioms and Rules.
Enumerative Inductive Ressoning
Definition, symbolic representation, rules key differences between Enumerative Induction and Complete Induction.
17
Logical Methods for Exploring Cause and Effect Relationships
5 Methods for Explocing Causal Relationships that differ from the standard definition.
18
Analogical Reasoning
2 types of analogical reasoning, and the symbolisation of properties under both reasoning methods.
Statistical Reasoning
Statistical Reasoning Categories and Symbolization. Rule Descriptions Sample. Based Inference of Statistical Properties.
20
Induction Paradox
Definitions, rules and symbolic representations of three inductive paradoxes GB Paradox, BC Paradox,1.S 1.S Puradox.
21
Speech Acts
Purpose, Adaptive Directions. Formulas, and Common Verbs for 5 Speech Act Classification Rules: Assertives Directives, Commissives, Expressives, and Declarations.
22
Cooperative Principle
Speaker's Criterion and Hearer's Inference for the three Cooperation Principles:C** Principle,C" Principle, CC Principle.
23
Definitions
6 Intensional 2efinitions,2 2 Extensional Definitions.3 3 Lexical Definitions.
24
Argumentation
4 Direct Argumentation Methods.
25
Formal Fallacies
10 Formal Fallacy Naming Rules.
Table 9: Summary of 25 Rules for Logic Reasoning Task. This table gives the Rule IDs, titles, and brief descriptions of the 25 rules under the Logic Reasoning task for review.
28

---

Published as a conference paper at ICLR 2025
Rule ID
Tille
Description
Custom Inverse Shift Substitution Cipher
Customised Caesar cipher variants based on alphabetical substitution and inverse order mapping, combined with keys and fixed shifting digits.
Custom Pigpen Masonic Cipher
Each letter replaced with a symbol its corresponding position according to the encryption_tab table..
3
Custom Multi tap Phone Code
Using the Correspondence Table, letters replaced by keycode power representations, with numbers indicating keystrokes.
4
Custom Polybius Square Cipher
Letters are eacrypted using Polybius _qqure rowcolumn numbers.
5
Custom Affine Cipher
Letters are converted to numerical values using the aftine function for encoding, then converted back to letters according to the affine alphabet to complete encryption.
6
Custom Solitaire Cipher
A key stream is generated using dock of 52 suit cards and 2 trump cards via shufting and cutting, combined with message characters for eacry pion//dcrypttion
Custom Phillips Figure Cipher
Eneryphion/derrppion uses different 5x5 grids. Each block of five characters encrypted using its corresponding grid, finding its position, then enerypting as if shifted one grid to the lower right.
8
Custom Parta Cipher
13 alphahets are used, each associated with two letters. Each letter in the plaintext is
replaced with letter in the correponndng position according to the alphabet corresponding to the key letter.
9
Custom Alberti Cipher
Encryption uses fixed and moving alphabets. Each letter is replaced by its inner disc counterpart. The inner disc rotates after each period.
10
Custom Jefferson Cipher
For encryption and decryption, 25 reels are used in a eyclic manner, where each character is replaced by the next character in its position the current reel.
11
Custom Four-Square Cipher
generated by keys. Encryption result found by matching positions in squares based on double letter set.
Eneryption uses 4 squares: I & 4 fixed,
12
Castom Morbit Cipher
A key of 9 unique letters establishes number associations. The message is converted to Moese code and encrypted by indexing
string of mumbers.
13
Custom Bifid Cipher
Letters' row and columm coordinates are vertically aligned to form a new sequence, which is used to find corresponding letters the 5x5 grid form the ciphertext.
14
Custom Digrafid Cipher
Using shuflled character and grids, churacters are grouped into binary groups. Each group calculates ternary (call, num3, row2). Ciphertext is formed by reading all temaries by columns.
15
Custom Collon Cipher
Find the position of each letter in a 5x5 grid, concatenate the corresponding row header and column footer characters to form a binary. and concatenate all the binaries to form an encrypted message.
16
Custom Redefence Figure Cipher
The plaintext is filled to a predetermined number of lines in Zig-Zag mode and then read line by line form the cipberiext.
Custom Path Cipher
The serpentine path filled to a predeternined number rows, which are read columa by column to form the ciphertext.
18
Custom Rotating Grid Cipher
Hide messages arranging the letters of the message grid and using a rotatable overlay with holes to select the letters to be read/written
19
Custom ADEGVX Cipher
Using a 6x6 matrix, plaintext churacters' row/celumn numbers are replaced with ADRIVX churacters. Ciphertext is formed by reading all rows columns.
20
Custom Transposition Cipher
Using a transposed sequence list, plaintext written line by line, columns are reordered, then read line by line to form ciphertext.
21
Custom XOR Cipher
Each plaintext character converted binary, XOR with fixed key, replaced by a Permutation Table, and merged to form the encrypted binary string.
22
Custom S-BOX Cipher
After padding and chunking, plaintext is encrypted thruugh ASCII encoding,
XORing with key, S_ BOX substitution. replacement, XORing again, and converted
to hexadecimal string for ciphertext.
23
Customn RSA Cipher
Each plaintest letter's ASCII cade is converted decimal, RSA encrypted
mondd))), concatenated with commas to form the ciphertext.
24
Custom ECC Cipher
Convert each plaintext letter's 's ASCII to decimal, multiply by k_y,x, concatenate with commas to form ciphertext.
25
Custom SHA Cipher
Convert plaintext byte sequenee, perform XOR with looped SHA256 hash key, convert bexadecimal string for ciphertext.
Table 10: Summary of 25 Rules for Cipher Reasoning Task. This table gives the Rule IDs, titles, and brief descriptions of the 25 rules under the Cipher Reasoning task for review.
29

---

Published as a conference paper at ICLR 2025
Rule ID
Title
Description
Word Brain Teasers
Find similarities in a group of words.
Word Roots and Affixes
Find the same prefix or suffix before or after the letter combinations to form meaningful words.
Connect words
Form words by following the letter requirements.
Anagram
Rearrange the letters to form new words
Crypto-Math
Solve a formula of letters, find out the numbers represented by letters
6
Word ladder
Stepbystep changing of letter converts one word to another, and each step must form a valid word.
7
Logic puzzle
Map elements to attributes by given clues.
Word Search
Find hidden words in matrix of letters that can he arranged
horizontally, vertically or diagonally.
Math Path
Find the correct numbers to make the equation equal to the given number.
10
24 points
Use the four given numbers and the four operations of addition, subtraction, multiplication and division, combine them into an expression equal to 24.
11
Survo
Fill the grid with numbers to satisfy given sum the boundaries of the rows and columns.
12
kukurasu
Fill a grid with black squares, each filled with a different weight, and satisfy the puzzle requirements by summing these weights.
13
Numbrix
Fill in the grid with numbers to 81 in sequence, the path can be moved horizontally or vertically.
14
Number Wall
Build walls to separate the cue figures so that each figure's 's island is isolated from each other and the walls can be connected into continuous path.
15
Sudoko
Fill the 9x9 grid so that each row, column and each 3x3 subgrid contains all the numbers from I to 9 without duplication.
16
Caleudoko
In addition to following the standard Sudoku rules, the combinations of numbers a given area must satisfy specified mathematical requirements.
17
Futoshiki
Fill in the grid with numbers that do not repeat in each row and column and satisfy the inequality constraints hetween neighhouring cells.
Vector puzzles
Place vectors or arrows in the mesh, following specific direction and length constraints.
19
Star battle
Place stars in the grid to meet the required number of stars in each row,
column and region.
20
Campsite
Based on the given hints. place the tents in the grid such that each tent is adjucent to a tree and the tents do not touch each other.
21
Minesweeper
Mark all mine locations without stepping them by following the numerical cues that indicate the number of mines surrounding them.
22
Arrow Maze
Follow the arrows in the maze to find the path from the start to the end.
Norinori
Fill the grid with 2x1 dominoes such that each row and column contains the required number of dominoes.
24
Wordscapes
Fill in the grid with letters from the Across and Down word lists, ensuring that words intersect correctly and the first letter of each word
corresponds to its clue number.
25
Skyscrapers
Fill the grid with buildings of different heights so that each row and column contains a unique height while satisfying the given visible building number cue on the outside.
Table 11: Summary of 25 Rules for Puzzle Reasoning Task. This table gives the Rule IDs, titles, and brief descriptions of the 25 rules under the Puzzle Reasoning task for review.
30

---

Published as a conference paper at ICLR 2025
Rule ID
Title
Description
Pokemon
Based on the worldview of Pokémon, this worldview depicts world where people and magical creatures Pokémon live together.
Harry Potter
Based on the worldview of Harry Potter, this worldview depicts a wizarding world that includes wizards, magical creatures, etc.
Lord of the Rings
Based on the worldview of The Lord of the Rings, this worldview depicts a fantasy world filled with multiple races and ancient powers, with epic bottles waged around the mighty Supreme Ring.
Attuck on Titan
Based the worldview of Attack Titun, this worldview shows bleak, anti utopian world that includes gigantic man eating giants.
5
Avengers
Bused on the worldview of The Avengers, this worldview brings together a group superhemes, as well as some powerful enemies that threaten the sufety of the planet.
Star Wars
Based on the Star Wars worldview, this worldview depicts a galaxy-wide universe that includes fierce battle between the forces of light and darkness, Jedi Knights, and more.
7
Dragon Ball
Based on the worldview of Dragon Ball, this worldview depicts universe filled with powerful warriors and magical forces.
Street Fighter
Based on the Street Fighter worldview, this worldview focuses on fighting tournaments around the globe, including many top fighters.
Plants Zombies
Based on the Zombies worldview, this worldview includes many diffcrent types of plants with invading zombies.
Final Fantasy
Based on the Final Fantasy worldview, this worldview presents fantasy world filled with miagic, technology and complex human relationships.
Three Body
Based on the worldview of 'Three Bodies', this worldview includes humans and different civilisations in the universe.
12
Ling Cage
Based on the worldview of Ling Cage, this worldview depicts post conttappcalyptic world.
Sturcraft
Based on the worldview of Starcruft, this worldview shows interstellar race wars and political games.
14
Avatur
Based on the worldview of Avatur, this worldview shows vibrant alien planet, including humans and various local creatures.
Zootopia
Based on the worldview of Zootopia. this worldview depicts an authropomorphic animal society.
16
Don't 't Starve
Bused on the worldview of Famine, this worldview shows mysterious and dangerous world of survival.
How Train Your Dragon
Based on the worldview of How to Train Your Drugon, this worldview depicts a world where people and dragons coexist.
18
Incurnation
Bused on the worldview of Incarnation, this worldview explores the struggle for survival and human choices of survivors in a post-upoculyptic world, revealing the conflict between technology and ethics.
19
The Legend of Zelda Tears of the Kingdom
Bused on the worldview of The Legend of Zelda: Tears of the Kingdom, this worldview depicts a vast fantasy world.
20
Qin's Moon
Based the worldview of Qin's 's Moon, this worldview shows the end of the Warring States period and includes many historical figures and fictional charucters.
21
The Wandering Earth
Based on the worldview of Wandering Earth. this worldviow depicts the interstellar migration of mankind in order to survive.
22
SpongeBob SquarePants
Based oa the worldview of SpongeBob SquarePants, this worldview depicts the fun life the underwater world and includes vuriety of cartoon characters.
23
Howl's Moving Castle
Based on the worldview of Howl's Moving Castle, this worldview shows a world of magic and fantasy.
Transformers
Based on the worldview of Transformers, this worldview depicts a fierce battle between two robot camps on Earth.
25
World of Warcraft
Based on the World of Wareraft worldview, this worldview presents a fantasy world filled with magic and epic battles, including various races on the continent of Azeroth.
Table 12: Summary of 25 Rules for Counterfactual Reasoning Task. This table gives the Rule IDs, titles, and brief descriptions of the 25 rules under the Counterfactual Reasoning task for review.
31

---

Published as a conference paper at ICLR 2025
D PROMPT TEMPLATES
Content below shows the prompt templates used in our KOR-Bennch.
D.1 OPERATION PROMPT
Operation
Zero-shot
You are an intelligent assistant specializing in evaluating custom operations. Below is specific rule defined for custom operation. Your task is to apply this rule accurately to the provided question.
### Instructions:
Carefully read and understand the definitions the new operations in the rule.
2. If the question does not specifically ask for it, your answer should be number group of numbers.
3. Double-check your final answer to ensure it follows the rule accurately.
### Operation Rule:
(A Operation Rule.)
## Question:
(A Operation Rule-Driven Question.)
### Answer:
Three-shot
You are an intelligent assistant specializing in evaluating custom operations. Below is specific rule defined for custom operation. Your task is to apply this rule accurately to the provided question.
### Instructions:
1. Carefully read and understand the definitions of the new operations in the rule.
the question does not specifically ask for it, your answer should be number or group of numbers.
Double-check your final answer ensure it follows the rule accurately.
### Operation Rule:
(A Operation Rule.)
#*# Question:
(A Sample Question.)
### Answer:
(A Sample Answer.)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer.)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer.)
### Question:
(A Operation Rule-Driven Question.)
### Answer:
32

---

Published as a conference paper at ICLR 2025
D.2 LOGIC PROMPT
Logic
Zero-shot
You are an intelligent assistant that helps with various logical reasoning tasks. Below a custom- defined rule for specific type of logic. When responding, please ensure that your output adheres to the specified logical rules and format.
### Instructions:
Identify the relevant properties and objects as specified in rule.
2. Apply the given logical Logics or reasoning patterns.
Ensure your output formatted according to the specified notation and symbols.
### Logic Rule:
(A Logic Rule.)
### Question:
(A Logic Rule-Driver Question.)
### Answer:
Three-shot
You are an intelligent assistant that helps with various logical reasoning tasks. Below custom- defined rule for specifie type logic. When responding, please ensure that your output adheres to the specified logical rules and format.
###f Instructions:
Identify the relevant properties and objects as specified in the rule.
Apply the given logical Logics or reasoning patterns.
3. Ensure your output is formatted according the specified notation and symbols.
### Logic Rule:
(A Logic Rule.)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer:)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer:)
## Question:
(A Sample Question.)
### Answer:
(A Sample Answer.)
### Question:
(A Logic Rule-Driven Question.)
### Answer:
33

---

Published as a conference paper at ICLR 2025
D.3 CIPHER PROMPT
Cipher
Zero-shot
You are intelligent assistant that specializes in encryption and decryption tasks, Below are the rules for a specific cipher. When responding, please ensure that your output adheres the specified encryption and decryption rules and format.
### Instructions:
1. Identify the relevant propertics and objects specified in the rule, including the plaintext, keyword, and ciphertext.
2. Follow the specified encryption decryption operations precisely as described in the rules.
Ensure your output is formatted according to the specified notation and symbols.
### Cipher Rule:
(A Cipher Rule.)
### Question:
(A Cipher Rule-Driven Question.)
### Answer:
Three-shot
You are intelligent assistant that specializes encryption decryption tasks. Below are rules for a specific cipher. When responding, please ensure that your output adheres the specified encryption and decryption rules and format.
###f Instructions:
Identify the relevant propertics and objects specified in the rule, including the plaintext, keyword, and ciphertext.
2. Follow the specified encryption decryption operations precisely as described in the rules.
Ensure your output is formatted according to the specified notation and symbols.
### Cipher Rule:
(A Cipher Rule.)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer:)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer:)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer.)
### Question:
(A Cipher Question.)
### Answer:
34

---

Published as a conference paper at ICLR 2025
D.4 PUZZLE PROMPT
Puzzle
Zero-shot
You are an intelligent assistant specializing solving custom puzzle problems. Below is specific rule defined for a custom puzzle. Your task is to apply this rule accurately to the provided question.
### Instructions:
1. Thoroughly understand the rule provided. needed, break down the rule into simpler com- ponents or steps.
2. Apply the rule carefully to address the question presented.
Verify your answer ensure aligns with the rule and context the puzzle.
###fi Puzzle Rule:
(A Puzzle Rule.)
### Question:
(A Puzzle Rule-Driver Question.)
### Answer:
Three-shot
You intelligent assistant specializing in solving custom puzzle problems. Below is specific rule defined for a custom puzzle. Your task to apply this rule accurately to the provided question.
### Instructions:
1. Thoroughly understand the rule provided. needed, break down the rule into simpler com- ponents or steps.
Apply the rule carefully to address the question presented.
3. Verify your answer to ensure it aligns with the rule and the context of the puzzle.
### Puzzle Rule:
(A Puzzle Rule.)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer:)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer.)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer.)
### Question:
(A Puzzle Rule-Driven Question.)
### Answer:
35

---

Published as a conference paper at ICLR 2025
D.5 COUNTERFACTUAL PROMPT
Counterfactual
Zero-shot
You are an advanced assistant with expertise storytelling and rule-based reasoning. Your task is carefully analyze the provided story, which includes specific rules and details, and use this information to accurately answer related questions.
### Instructions:
1. Thoroughly review the story to identify and understand the relevant details and rules.
Use context provided by story offer precise insightful answers.
3. Ensure your responses align with the rules and information given in the story.
### Counterfactual Rule:
(A Counterfactual Rule.)
#*# Question:
(A Counterfactual Rule-Driven Question.)
### Answer:
Three-shot
You are an advanced assistant with expertise storytelling rule-based reasoning. Your task is to carefully analyze the provided story, which includes specific rules and details, and use this information to accurately answer related questions.
### Instructions:
1. Thoroughly review the story to identify and understand the relevant details and rules.
Use the context provided by the story to offer precise and insightful answers.
3. Ensure your responses align with the rules and information given in the story.
### Counterfactual Rule:
(A Counterfactual Rule.)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer.)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer:)
### Question:
(A Sample Question.)
### Answer:
(A Sample Answer:)
### Question:
(A Counterfuctual Rule-Driven Question.)
### Answer:
36

---

Published as a conference paper at ICLR 2025
E ANALYSIS OF REAL-LIFE ANSWER RATIOS FOR COUNTERFACTUAL TASK
To analyze the ratio of real-life answers in counterfactual reasoning task, we conduct trend analysis on the models listed in Table 2. Figure 6, below show the trend of change for Chat Model and Base Model. This analysis reveals that counterfactual accuracy decreases, the real-life answer ratio increases. For chat models, the ratio remains low, not exceeding 15%, while for base models, it rises significantly 36.8% accuracy declines.
Counterfactual (ccuracy(%) Real Life Answer Ratio(%)
100
80
60
Models (sorted by decreasing Counterfactual Accuracy)
Figure 6: Trend of Real-life Answer Ratio Increasing as Counterfactual Accuracy Decreases for Chat Model.
Counterfactual Accuracy(%) Real Life Answer Ratio(%)
60
40
0
Models (sorted by decreasing Counterfactual Accuracy)
Figure 7: Trend of Real-life Answer Ratio Increasing as Counterfactual Accuracy Decreases for Base Model.
37

---

Published as a conference paper at ICLR 2025
F DETAILS OF FURTHER ANALYSIS
Appendix F provides details of supplementary experiments that provide additional insight into the
KOR-Bench results.
F.1 STEPWISE PROMPTING ANALYSIS OF CIPHER TASK BOTTLENECKS
In exploring the bottlenecks of the Cipher Reasoning task, we choose five highly erroneous rules with Rule IDs 1,9,,, and can check Table 10 to see the cipher rules corresponding to these Rule IDs, and disassemble Cipher Question step by step by constructing consecutive sub-steps datasets.
F.1.1 DETAILED EXPLANATIONS OF NINE KEY SUB-STEP TYPES
Table 13 gives specific explanation of the major types of sub-steps after splitting Cipher Ques- tion into multiple steps.
Sub-Step Type
Explanation
Substitution
Replace one or more symbols with others.
Mapping
Map input to a new position or value using predefined tables or rules.
Shift
Move data left right by characters.
Rotation
Rotate data clockwise counterclockwise.
Partition
Divide data into multiple blocks.
Conditional Filling
Fill data in specific order based on certain conditions.
Conditional Reading
Read data in specific order based on conditions.
Encoding
Convert data another format or representation, such as ASCII.
Computation
Perform mathematical operations.
Table 13: Explanations Sub-Step Types in Stepwise Prompting Analysis of Cipher Task. This table gives an explanation of the 9 major types of steps when Cipher Question is split into multiple steps.
38

---

Published as a conference paper at ICLR 2025
F.1.2 EXAMPLE OF CIPHER QUESTION SPLIT INTO SUB-STEPS
The following content F.1.2 illustrates how encryption or decryption question in the Cipher rea- soning task is broken down into several different types of sub-steps.
Cipher Sub-Step Example
Rule:
Other Fields:
Eneryption Rules:
Question: Plaintext: "O"
Please provide the encrypted answer, encapsulated in double square bruckets. For example, the format should be: Hencrypped answer]].
Input:
Uppercase letters string without punctuation and spaces.
Plaintext:
Answer: [[N]]
Output:
Steps:
Ciphertext: Uppercase letters string.
Step 1: For letter O in the given Plaintext: (1) Use reversed catccabb_phbeettteee for reverse mapping. Find its position in the standard standard_alphaann aapphbett and replace it with the letter in the corresponding position in reversed alphabet. For example, mapped to Z and B is mapped give your answer after performing (1) in for- mat [I...]I.
Preparation:
standard_ale stanaard_alphabtt:
"ABCDEFGHUKKM
NOPORSTUVWXYZ"
Please
reversed allphabt:
"IXXWVUTSRQPON
MLKJIHGFEDCBA"
Atom: O
Answer: [[L]] Type: Substitution
substitution -alphahet:
"RFDJUHABCEGIK
LMNOPQSTVWXYZZ"
Step 2: For letter O in the given Plaintext: Execute (1) to obtain the letter (2) Move letter obtained in (1) forward 4 places the standard alphabet order. For example, if p=A, after (1) is mapped Z, then 7. is shifted for- ward 4 positions in the standard standard_alphaber get D. Please give your answer after perform- ing (2) in the format I....].
Encryption Steps:
For each letter p in given Plaintext: (1) Use reversed reversed_alphabet for reverse mapping. Find its position standard stanhard_olphabet and replace it with the letter in the correspond- ing position in reversed alphabet. For example, is mapped and B is mapped to Y.
Atom: L
Answer: [[P]]
(2) Move the letter obtained in forward 4 places dard_alphabet order. For example, if p=A, after (1) is mapped to Z, then Z is shifted forward 4 posi- tions in the standard alphabet to get
Type: Shift
stan-
Step 3: For letter O in the given Plaintext: Execute (1)(2) to obtain the letter P. (3) Re- place letter obtained from (2) by finding its position standard _appaabe and using the corresponding letter in substitution. alphabet, resulting the final ciphertext letter. For example, the letter obtained by going through (2) is it is mapped as J. Please give your answer after performing (3) in the format [[...].
subsiitutio_oaphabee,
(3) Replace the lettr obtained from (2) by finding its position in standard alphabet and using corresponding letter in substitu- tion_alphin aan__pphhbet, resulting in the fi- nal ciphertext letter. For exam- ple, if letter obtained by going through (2) is D, it mapped as J.
Atom: P
Answer: [[N]]
Type: Substitution
39

---

Published as a conference paper at ICLR 2025
F.1.3 MODEL RESULTS ON SUB-STER ERROR RATES
In the Stepwise Prompting setting, Figure below gives the error rates the model on nine key Sub-Steps to reveal its weaknesses in the Cipher Reasoning task.
C4ai-Command-
C4ai-Command R-Plus
100
Step Error Rate (%)
Claude-3.5-Sone
DeepSeek-V2.55
100
Sub-Step Rate(
Gpt-4o
Meta-Llama-3.1-70-- t.rrol--rtt
100
Sub-Step Rate(%)
1oka-Llama-311-888 3.1-8B-Instruct
Mistral- 1
100
Step Error Rate (%)
Sub-Step
Substitutiio
Shift
Partition
Conditional Filling
Mapping
Conditional Reading
Rotation
Encoding
Computation
Substitution
Shift
Partition
Conditional Filling
Mapping
Conditional Reading Encoding
Rotation
Computation
Figure 8: Model Sub-Step Error Rates in Cipher Task Stepwise Prompting. This Figure shows the error rates for some models over the nine categories of sub-steps, and the results for the other models shown Figure 9.
40

---

Published as a conference paper at ICLR 2025
Mistrall-Lagge-lsstrur-----0
1eonn-722-Instruule
100
Step Rate(
Swve2--BBInstten
(wen2.5-328-Insste
100
Sub-Step Rate
Swen2.5-72B-Insteen
Qwen2.5-7B-Instruct
Sub Sub-Step Rate(
Vi-1.5-9B-Chat
Yi-Large
100
Step Rate(%)
Substitution
Shift
Partition
Conditional Filling
Mapping
Conditional Reading
Rotation
Encoding
Computation
Substitution
Shift
Partition
Conditional Filling
Mapping
Conditional Reading Encoding
Rotation
Computation
Figure 9: Model Sub-Step Error Rates in Cipher Task Stepwise Prompting. This Figure shows the error rates for some models over the nine categories of sub-steps, and the results for the other models shown in Figure 8.
41

---

Published as a conference paper at ICLR 2025
F.2 IMPACT ANALYSIS OF TRICKS ON PUZZLE TASK PERFORMANCE
In this experiment, we introduce a "trick" field as additional input to explore its impact on puzzle task performance. For complex puzzle tasks such mazes and sudoku, identifying and executing key initial steps can often dramatically simplify the entire process. Figure 10 gives some results on the accuracy of the model on the Puzzle task without Trick and with Trick given.
Without Trick
With Trick
20
18
Puzzle Accuracy
15
11.2
10.4
9.2 9.2
8.4
7.6
6.4
4.4
4.8
4.4
3.6
2
2.4
2.4
C4ai- CAal-Commane Command-R
Caai- Command -- Plus
Claude-35 Sonnet
Meta Llama- 3.1 1-700- 11-70B-Instruce Instruct
Meta Llm-l-lmmaaa 1-88- Instruct
Mistral: 78 Instruct-v0.3
Qwen2-728 1omm2-72B-hn 7tnsstrt- Instruct
Qwen2 5. 728 (00-722--Innntt Instruct
Qwen2 1-n225--78 78-Instruct
Vi-1.5-BB -h--at Chat
Yi-Large
Figure 10: Impact of Trick Field Puzzle Task Accuracy. The figure gives the accuracy without and with Trick for some models that improve in Puzzle Task Accuracy after Trick Field is given.
F.2.1 A CASE STUDY OF INCORPORATING TRICKS IN PUZZLE REASONING TASK
The following content F.2.1 demonstrates the addition of the trick field in the Puzzle reasoning task, along with an example illustrating the difference in responses from Claude 3.5 Sonnet when the trick provided versus when it is not. The trick field is appended the question in the prompt.
Puzzle Trick Example
Rule: A rectangular grid is given, which is pre-filled with numbers, while each row and column has
number at the end indicating the sum of the numbers in that row or column. The player needs fill the remaining spaces with specified of numbers. Each number can only be used once.
The numbers must be filled in such that the sum of the numbers each row and column matches the intended sum.
The puzzle given in the form of matrix with blank squares filled with X and the player replaces the squares with numbers. Question:
X
2
X
4
23
9
X
X
11
24
X
8
31
23
The final answer should be given in order from left right, top bottom with each element separated by a space and different lines separated by ",". Wrap your final answer in double square brackets, like this: [[your answer]].
Answer: [[7 2 10 4,9 11,5 12 811
Observe the second column, 12 then the X value of the second row and second column is 1, fill the matrix and continue the solution.
Trick:
42

---

Published as a conference paper at ICLR 2025
Response without Trick:
Response with Trick:
To solve this puzzle, I'll apply the given rule and fill in the blanks (X) with the appropriate num- bers ensure each and column sums up to given total. Let's break down step by step: First, let's ss identify the missing numbers: need to fill in 5 X's with numbers that aren't al- reudy used in the grid. The unused numbers are:
Let's approach this step-by-slep:
1. We start with the observation that in the sec-
ond column, 3+X+12=13 X
Now let's fill that and solve the rest of the puzzle:
X
2
X
4
23
X
1, 5, 6,
X
12
X
8
31
Now, let's solve each row and colunn: Row 1: X 2 + X + 23; We need two numbers that
21
15
19
23
4); The only combination that works 6 and 11.
add to 17 (2.3
3. For the first row: X + 2
+
+
4
23; We
need two numbers that add up 17 (23 2 4
Row 2: 9
X
24; We need num-
17); The only unused numbers that satisfy this
bers that add up (24 9- 11): The only com-
are 7 and
bination that works is and 3.
4. For the second row: 9 + 1 X + 11 24; So
Row 3: X 12 X 8 31; We need Iwo num-
X must be 3.
bers that add up to 11 (31 12 8); The only remaining number that fits is and
5. For the third row: X + 12 X 31; We
need two numbers that add up to (31 12
Column I: X + 9 (conftrmed).
21; + 9 21
11); The only unused numbers that satisfy this
are 5 and
Column 2: 2 X 12 15; 2
12 15
6. Now can fill in the entire grid:
(conftrmed).
2
10
Column 3:
+ X 19: 11
1
3
11
24
(confirmed).
5
12
6
8
31
Column 4: 4 11 8 23 (confirmed)
21
15
19
23
3. The completed grid looks like this:
7. We can verify that all row and column sums are correct.
2
11
4
23
1
3
11
24
Therefore, the final answer is: [[7 2 10 4, 13 11, 8]]
12
5
31
21
15
19
23
Now, we format the answer as requested: [[6 11 11, 62
F.3 ATTENTION FOCUS VISUALISATION
In ReR-Bench, the rules are text-based descriptions, and each rule-driven question focuses on a specific part of the rule particular branch of the rule. Consequently, the model must concentrate on these core elements to have higher likelihood of answering the question accurately. While focusing on these core elements does not guarantee correct answers, it is necessary condition.
Ancuumalatdd
Σ
Σ
(teep Aeaall i] if rulestar S S ruleend
(I)
decode, steps (eayr,.heod)Etp
In the experiment, we add "needle" field to each question-answer sample, indicating the core parts the model needs to focus on during the answering process. We first use the Retrieval Head (Wu al., 2024) code calculate the Attention Head Retrieval Score, which ranks each of the model's retrieval heads. Next, we accumulate the attention matrices of the top 50 ranked retricval heads within the rule's specified range (max decode len 2000), as shown in Equation I. Finally, map the attention scores assigned to tokens back to the rule text for visualization, where the "needle" field is underlined, and the intensity of the color indicates the level of attention.
Combining the results of attention visualization helps to understand the model's 's output better, partic- ularly the reasons for its errors. Appendix H provides several case studies of attention visualizations.
43

---

Published as a conference paper at ICLR 2025
F.4 ANALYSIS ON CELF-CORECTTON
F.4.1 SELF CORRECTION PROMPT TEMPLATE
The following content F.4.1 gives prompt template for the Self-Correction experiment. Due to the limit on the number of tokens, each round of dialogue, the model's responses are extracted and reintegrated into the dialogue history. The maximum number of interactions is 5, implying correction rounds.
Self-Correction Prompt Template
Round 0: User: (The default prompt includes the following components:) (Rule)
(Question)
Assistant: (Extracted [Extracted_ResponseeeHeee Beapoonse_Roud____.
Round 1:
User: Your answer is incorrect, please check your answer and provide correct one. Assistant: (Extracted
Round 2:
User: Your answer is incorrect, please check your answer and provide correct one. Assistant: (Extracted Responsee_eeeeeeeeeeee "esponse_RRond_2]
(Subsequent rounds omitted...)
F.4.2 IMPACT OF ROUND COUNT ON SELF-CORRECTION ACCURACY
Figure 11 below gives the effect of the number of rounds on the rate of correction from the model's point of view, and it can be seen that for most models, self-correcting for two rounds gives the highest gains.
Figure displays the correction rates categorized by task type. The counterfactual reasoning task maintains higher level of correction rates across all four rounds. In contrast, the other four task types show a significant correction effect the first and second rounds, with less pronounced effeets subsequently.
44

---

Published as a conference paper at ICLR 2025
Round_y
Round 2
Round 3
Round 4
Correct Rate(%)
10
5
Cdai-Command- R
C4ai- Commond--n-RR Rrrmannd-R-Plus R-Plus
S.5nnntttt Claude-3.5 Sonnet
DeepSeek-V2
Meta Llama-3, 3,1 11-708--nn Ins--nntt
Meta- Meta-Llamar Llama-3311111 3.1 1-18188--- Instructtttt
Gpt-4o
Mistral 78 ---nsttuuut-0.33
Mistral- Mistral-Large-Istteee large--nsstruct2200
Qwen2 Qwen2-72B-Insse 1ann2--2B-Innstrae Instrrct
Qwen2-78 Instruct
Qwen2 1oom22--38--nman 100-322--nnstt I-ssustrcct
Qwen2 --- 10--722--nssttt Instructtt
Qwen2.5 (wen2.5-78-Inae 10555-77--ssete R-Innttrcct
Yi- 1. 98 Chat
Yi-Large
Figure 11: Self- Correctont Rates of Multiple Models Over Four Rounds.The self-correction rate defined as the number of problems successfully corrected in a given round divided by the number of problems still unresolved from the previous round.
operation
logic
cipher
puzzle
counterfactual
43
40
Correct Rate(%)
30
27.65
24.34
20
13.21
10
8.7
7.16
2.67
2.82
4.65
3.14
3.84
1.73
1.25 0.65
0
Round
Round 2
Round 3
Round 4
Figure 12: Self-Correctin Rates of Five Task Types Over Four Rounds.Tha self-correction rate defined as the number of problems successfully corrected in a given round divided by the number of problems still unresolved from the previous round.
45

---

Published as a conference paper at ICLR 2025
F.5 ANALYSIS ON COMPLEX TASK PROCESSING
F.5.1 COMPLEX TASK PROCESSING PROMPT
Content F.5.1 gives prompt template for complex task processing. Three Settings for complex task processing:(1) Multi-G Q: rule, 1-10 questions; (2) Multi-R: 2-3 rules, question; (3) Multi-RQ: 2-3 rules, 1-3 questions all use this template.
Complex Task Processing Prompt Template
You are an intelligent assistant capable handling all types of reasoning and problem- solving tasks. Below is the text of set of rules. Your task is to apply the appropriate rules to solve a series of problems.
### Instructions:
1. Read each question carefully and rules to find something relevant to that question.
2. Use the relevant rules to answer each question accurately.
3. Provide the final answers to all questions JSON format.
({
"question1": "your answer", : "your answer", "question3": "your answer",
### Rules:
(A series of rules.)
### Questions:
(A series of questions.)
### Answers:
F.5.2 COMPLEX TASK PROCESSING EVALUATION
For the evaluation of complex task processing, since the output format of the answers in all three settings is JSON, the system first uses the regular expression r' {.*}* to extract the answer portion. Next, it traverses the question list and, for each sub-question, extracts its category and index to retrieve the specific question content, including the correct answer, from the corresponding data source. The system then conducts assessment of each sub-question, individually judging and recording the passing status of each one. Finally, the overall pass rate of the question is calculated based on the number sub-questions that pass. If the length of the response results is less than the number of sub-questions, the evaluation process terminates promptly. All assessment results for the sub-questions, including the question content, response text, correct answer, and correctness, are systematically recorded.
46

---

G FUN-FILLED ANALYSIS OF SLIP-UPS
Appendix presents a series of interesting error cases occurring in five types of tasks from the Claude-3.5-Sonee or GPT-4o. Each case is accompanied by detailed analyses and in-depth expla- nations designed to reveal potential weaknesses in the model and provide insights into improving model performance.
List of Case Study Boxs
Operation Error Case
48
2 Operation Error Case 2
49
3 Operation Error Case 3
50
4 Operation Error Case 4
51
5 Operation Error Case 5
52
6 Logic Error Case
1
53
7 Logic Error Case
55
Logic Error Case
57
9 Logic Error Case
59
10 Logic Error Case 5
61
11 Cipher Error Case
63
12 Cipher Error Case 2
65
13 Cipher Error Case 3
67
14 Cipher Error Case 4
71
15 Cipher Error Case
73
16 Puzzle Error Case 1
76
17 Puzzle Error Case 2
78
18 Puzzle Error Case 3
79
19 Puzzle Error Case 4
81
20 Puzzle Error Case 5
83
21 Counterfactual Error Case 1
85
22 Counterfactual Error Case 2 Back to Table Contents
87
47

---

Published as a conference paper at ICLR 2025
G.1
OPERATION ERROR CASES ANALYSIS
Error Case 1
Rule:
Define an operation such that when is a multiple of b, asb a/b
When b is multiple of a, a&b bla + 2.
If is multiple of b and b is not multiple of a, a&b 24. Both and b are integers.
Question:
Compute 11.
Please ensure the answer single number and wrap double square brackets, like this: [[your answer]].
Answer:
[[4]]
Response:
Let's break this down step by step:
1. First, we need to calculate 25
25 is multiple of 5, so we use the first part of the rule.
25 = 25/5 2 = 2
2. Now we have 7:14:
7 is not multiple 14, and 14 is not multiple of 7.
So we use the third part of the rule.
1: 24.
3. Therefore, 15 24
4. The final answer, wrapped in double square brackets, is: [[24]]
Analysis:
The sentence "7 is not a multiple of 14, and 14 is not a multiple of 7" is wrong. In fact, 14 is multiple 7f 7, The misjudgement of the multiplicity has led to the wrong choice of using the third part of the rule.
Back to List of Cases
Back to Table of Contents
48

---

Published as a conference paper at ICLR 2025
Error Case 2
Rule:
Define an operation such that when is a multiple b, asb a/b
When b is multiple of asb bla + 2.
If a not a multiple of b and b is not multiple of a, ash = 24.
Both a and b are integers.
Question:
If # find X.
When providing your answer, please enclose in double square brackets, like this: [[answer]].
If there is more than one correct answer, separate the answers with 'or' like this: [[lor2]].
Answer:
[[650er5or125]]
Response:
Let's approach this step-by-step:
1) We know that 25 25
Let's consider the possible values for that could make this equation true.
3) For the final result to be we need either:
a) 255 xx = 2 and 2 2 = 4, or
b) 25 50 and 50 4
4) Let's examine case a:
For 2 to equal 4, 14 must a multiple of which is true. not this case doesn't work. 5) Let's examine case b:
14/2 + 2
For 50*14 to equal 4, 14 must be multiple of 50, which is not true. So 50 = 24, which is not 4. This case doesn' I work either.
6) Therefore, we need to find X such that 25 50
For this be true, must be multiple of 25, and 25/X 2 50
8) Solving this equation:
25/X + = 50
25/X = 48
X 25/48 0.52083...
9) But X must be an integer. The only integer that satisfies all conditions is 25.
10) Let's verify:
25 25/25 + + =
3 24 (since neither is multiple of the other) Therefore, the value of X is 25. [[25]]
Analysis:
The analysis is fundamentally flawed due unreasonable initial assumptions about the two scenarios. Back to List of Cases
Back to Table of Contents
49

---

Published as a conference paper at ICLR 2025
Error Case 3
Rule:
AQB=(A+3))((++B)
A and are integers.
Question:
Compute 20304.
Please ensure the answer is single number and wrap it in double square brackets, like this: [[your answer)).
Answer:
[[3953]]
Response:
Let's approach this step-by-step:
1) First, we need to calculate 44
3O4 (3+3x4) (3+4)
= =3++12) x 7
15 x 7
105
Now we have 1010
20 (2+3x105) x (2+105)
=2+3155 107
=17 107
33,919
Therefore, the final answer is:
[[33919]]
Analysis:
The response demonstrates a critical ervor in the order of operations. The calculation should proceed from left to right, but the answer incorrectly prioritizes the rightmost op- eration.
Back to List of Cases
Back Table of Contents
50

---

Published as a conference paper at ICLR 2025
Error Case 4
Rule:
acb=log, a + log, b a and b are positive integers.
Question:
Compute 9e243.
If the answer is a fraction, write it in "a/b' text Format.Decimer are not allowed. Please wrap the answer in double square brackets, like this: Hyouu answer)). Answer:
[[29/10]]
Response:
Let's apply the given operation rule to compute 9c243:
acb=log, a + loga b
Here, 9 and b 243
9c243 = 1og243 9 + logg 243
1) log243 9 = 1/2 (since 242) = 1)1)
2) logg 243 5/ ((since 9M(5/2) 243)
Adding these:
9c243 1/2 5/2 3/2
Therefore, the answer is:
[[3/2]]
Analysis:
The response contains a partial calculation error that propagates through the solution.
Back to List of Cases
Back to Table Contents
51

---

Published as a conference paper at ICLR 2025
Error Case 5
Rule:
acb=log, a + log, b a and b are positive integers.
Question:
If 3cX=17/4 find X.
The answer should only be given as number.
Please wrap the answer in double square brackets, like this: yyour answer)).
Answer:
[[81]]
Response:
Let's approach this step-by-step:
We' given that 3 17/4
2) Using the definition of the operation, can write:
3eX = log 3 + loga X = 17/4
We need solve this equation for X. This is complex equation that doesn' 't have a simple algebraic solution.
4) However, we can use trial and error with some logical deduction. We know X must be positive integer.
5) Let's try
9.
loga + log, 9 R 0.5 +
2.5
This is very close to 17/4 4.25
7) We can verify that no other integer value of X gives a result closer to 17/4.
Therefore, the value of X that satisfies the equation 3 = 17/4 is 9.
[[9]]
Analysis:
The response attempts solve the problem through trials. After testing, correctly identi- fies that the proposed solutions do not meet the conditions. However, it presents this answer as the final solution.
Back to List of Cases
Back Table Contents
52

---

Published as a conference paper at ICLR 2025
G.2
LOGIC ERROR CASES ANALYSIS
Error Case 1
Rule:
Propositions represented using p1. P2.-...Pu:
Let pı be proposition, the compound proposition "not p1 is represented P1- Let and p2 be two propositions. the compound proposition "pı and p2" is repre- sented as &1&-
Let pa and p2 be two propositions, the compound proposition "pr p2 " is represented P2-
as Pi
Let pı and p2 be two propositions. the compound proposition Pi, then p2" repre- sented P1 P2-
Let pt and p2 be two propositions, the compound proposition "pı if and only if p2" is represented as P1 = P2-
A single proposition and proposition constants can be called a formula. Formulas represented using Fi, F2..2...., Fu.
If Fi is formula, then Fi is also formula. If F1 and F2 are formulas, then F2& F1 formulas.
P2,
F1
+
F2.
F1
F2 are also
Level A Formula: The most basic proposition unit, without logical connectives nested structures.
Level B Formula: A formula containing one logical connective, and the connected two propositions are both Level A formulas. For example, P1.
Level C Formula: A formula containing nested logical connectives and at least one Level B formula. For example, ~ P1-
Other levels of logic follow by analogy; when higher than Level they are classified as Z+n (n 2 1). For example, ~ (~~ p1).
True assignment of proposition: proposition Pr is assigned as v, indicating that pı is true.
False assignment of proposition: A proposition is assigned as x, indicating that Pi is false.
True assignment of a formula: the formula is (~ pı& p2& ps) || (p1&tp2). then x| x |xx, V ///ix are true assignments of the formula.
False assignment of a formula: If the formula is (~ p&& ~ pr& ~ ps) || &pp))
then x x V VV are false assignments of the formula.
For p1 = P2, only V x is false assignment of the formula.
A formula that is true under all assignments is called a Truth Formula.
A formula that is false under all assignments is called a Falsehood Formula.
Recursive definition of formulas: Any formula containing nested logical connectives can be decomposed recursively to obtain its subformulas and their logical connective structures.
Priority of logical connectives: The priority of logical connectives from high to low is follows: (not), & (and), || (or), = (if...heen) = (if and only if).
Without parentheses, operations are performed according to priority.
Equivalence of formulas: Two formulas are equivalent if they have the same truth value under all assignments.
Equivalent formulas can be interchanged.
53

---

Published as a conference paper at ICLR 2025
Simplification of formulas: Formulas can be simplified through logical rules to obtain a more concise form without changing the truth value of the formula.
Question:
Find the falsity values of the following formulas:
(1) (~ PI ~ Ps
(2) (~ A (p1 * P2)
(3) (pi - 12) (~ (p) Pa) Vp)
The answer format is [IF....::.E....IEEE......
If there are multiple values in F, they should be separated by commas (,). For example
[[F:x] VLLLE/////// [x./VVVII
Answer:
[[Fx XX
M X XIII X X MJ
Response:
To fon the falsity values of the given formulas, need to evaluate each formula for all possible truth value assignments and identify the assignments that make the formula false. Let's analyze each formula:
(I) (1)~~~p) A P2)V ~
This formula is false when (~> A p2) is true and p3 is true.
So, it' false when pl is false, p2 is true, and p3 true.
(2) ((~ P2 V p3) (p1 + p2)
This formula is false when either (~ P2 V pa) is false or (p1 = P2) is false.
(~ P2 V Ps) is false when p2 is true and Ps is false.
(p1 + P2) is false when p1 is true and p2 is false.
So, it's false when pı is true, p2 is true, and ps is false, or when pt is true, P2 is false,
and pa is any value.
(3) (pr P2) (~ Aa)) pi)
This formula is false when (p1 = P2) is false or (~ (p1 A Pa) P1) is false.
(pL + P2) is false when is true and p2 is false.
(~ (pr A A3) V p1) is always true (it's 's a tautology), so doesn't t contribute to making the formula false.
So, the formula is false when is true, p2 is false, and ps can be any value.
Now, let's 's formal the answer according to the specified format:
[[ [E x,
V x x] F.V
V, x X]]
Analysis:
The model accurately identifies two different scenarios during the analysis, which should theoretically result in four possible answers. However, the model fails to fully cover all scenarios in its final consideration, a phenomenon that reveals its inadequacy in exploring all possible paths in terms of logical reasoning.
Back to List of Cases
Back to Table of Contents
54

---

Published as a conference paper at ICLR 2025
Error Case 2
Rule:
a simple conjunctive form (simple disjunctive form) containing n propositional vari- ables. if each propositional variable and its negation appear exactly once. and the propositional variables or their negations are arranged in ascending order of sub- scripts or in lexicographical order, such simple conjunctive form (simple disjunctive form) is called a paired conjunctive term (paired disjunctive term).
If the true assignment of paired conjunctive term corresponds to a binary number equal to hexadecimal number i, this paired conjunctive term is denoted my For example, the true assignment of p/q is 11, and the binary number is 11, corresponding hexadecimal number 3, denoted as mys.
If the false assigument of a paired disjunctive term corresponds binary number
equal to hexadecimal number i, this paired disjunctive term is denoted as M; For example, the false assignment of - -q -r 111, and the binary mumber 111. corresponding to hexadecimal number 7, denoted as M7.
The disjunctive normal form (conjunctive normal form) consisting of all paired con- junctive terms (paired disjunctive terms) is called the principal disjunctive normal (principal conjunctive normal form).
Given formula A containing propositional variables:
If the principal disjunctive normal form of A includes all 2n paired conjunctive terms, A is a tautology.
If the principal disjunctive normal form form of A includes no paired conjunctive terms, A is contradiction.
If the principal disjunctive normal form of includes mo, A is basic formula. If the indices of the paired conjunctive terms included in the principal disjunctive normal form of A are all even, A is an all-even formula.
If the indices i of the paired conjunctive terms included in the principal disjunctive normal form of A are all odd, A is an all-odd formula.
Question:
Given that formula contains 4 propositional variables, what should it be denoted as if it is both tautology and a basic form? The answer format [[]].
Answer:
[mmo Vm, Vmna Vmn Vm4 Vmng VmmrVmmaammn hwwp://mwwwwaannnnaaaaaavvvvovvm
Response:
To answer this question, let's apply the given logic rule:
1. Formula A contains 4 propositional variables.
2. For it be tautology, its principal disjunctive normal form must include all 2n4
= 16 paired conjunctive terms.
3. For it to be basic formula, its principal disjunctive normal form must include m0.
Given these conditions, the formula A would be denoted as:
[[mo V my V Vmna V ma ma V ma V ma V my V mmg V my m V mii V Vmia V
144 mis]]
This notation includes all 16 paired conjunctive terms (m0 m15), satisfying the con- dition for a tautology, and includes m0, satisfying the condition for basic formula.
55

---

Published as a conference paper at ICLR 2025
Analysis:
The model may not be sufficiently limited in its understanding of the rules to accurately capture the specific requirement of 'hexadecimal' leading to errors.
Back to List of Cases
Back to Table of Contents
56

---

Published as a conference paper at ICLR 2025
Error Case 3
Rule:
Between propositions pl and p2. the representation is defined as follows:
A: Vpı P2
E: Vp1 + 7/
I: 3p1 - P2
0: Ap1 - P2
The figures and moods of the syllogism are as follows:
1. Figure
(1) Form:
MOP
SOM
SOP
(2) Valid Moods:
AAA
EAE
All
EIO
2. Figure II
(1) Form:
POM
SOM
J. SOP
Valid Moods:
AEE
EAE
EIO
AOO
3. Figure III
(1) Form:
M()P
MOS
SO)
Valid Moods:
All
EIO
IAI
OAO
4. Figure IV
(1) Form:
POM
MOS
SOP
(2) Valid Moods:
AEE
EIO
IAI
Question:
Given the logical statements:
57

---

Published as a conference paper at ICLR 2025
AP M, AM -S VS PP
Determine which figure and mood of syllogism the argument corresponds to, using the format [[/n///V)
Answer:
[[[]]
Response:
To determine the figure and mood of the syllogism for the given logical statements, let's s analyze them step by step:
We have:
First premise: PP - M
Second premise: AM + -
Conclusion: AS + -P
Let's match these to our syllogism figures:
The first premise has as the subject and as the predicate. The second premise has as the subject and as the predicate (negated).
The conclusion has S as the subject and P as the predicate (negated).
This structure matches Figure
M()
SOM
:SOPP
Now, let's determine the mood:
First premise: VP + M -This is A statement
Second premise: vM = -S- This is an E statement
Conclusion: VS = - This is an E statement
So, the mood is AEE.
6. Checking the valid moods for Figure we see that AEE is not listed as a valid mood.
However, the question asks to determine which figure and mood the argument cor- responds to, even if it's 's not valid.
Therefore, the answer the requested format is:
[[]][AEE]]
This indicates that the argument corresponds to Figure I with the mood AEE, although it's worth noting that this is not a valid syllogism in classical logic.
Analysis:
After a matching error occurs in the early stage, although is eventually found to be unsat- isfactory, an incorrect conclusion is still given without correction.
Back to List of Cases
Back to Table of Contents
58

---

Published as a conference paper at ICLR 2025
Error Case 4
Rule:
Between propositions pl and p2. the representation is defined as follows:
A: Vpı P2
E: Vp1 + 7/
I: 3p1 - P2
0: Ap1 - P2
The figures and moods of the syllogism are as follows:
1. Figure
(1) Form:
MOP
SOM
SOP
(2) Valid Moods:
AAA
EAE
All
EIO
2. Figure II
(1) Form:
POM
SOM
J. SOP
Valid Moods:
AEE
EAE
EIO
AOO
3. Figure III
(1) Form:
M()P
MOS
SO)
Valid Moods:
All
EIO
IAI
OAO
4. Figure IV
(1) Form:
POM
MOS
SOP
(2) Valid Moods:
AEE
EIO
IAI
Question:
Please fill in the parentheses below
59

---

Published as a conference paper at ICLR 2025
()AP
MI(
:.SIP
Provide the answer in the format of [[ J:[ 11.
Answer:
[[M]:[]]
Response:
To answer this question, we need to identify the figure and mood of the syllogism based on the given information and the valid moods for each figure.
Let's analyze the given syllogism:
()AP
MI()
SIP
This structure matches Figure IV:
POM
MOS
..SP)P
Now, we need to find a valid mood for Figure IV that results conclusion of SIP. Looking at the valid moods for Figure IV (AEE, EIO, IAI), we can see that IAI is the only one that gives us an conclusion.
So, we'll 'lll the IAI mood:
I: 3p1 P2 (for the first premise)
A: Vp = p2 (for the second premise)
1: 3p + (for the conclusion)
Applying this to our syllogism:
IAP
MAI
SIP
Therefore, the answer in the requested format is:
[[]][[]]
This fills in the parentheses in the original syllogism as follows:
(I)AP
MA(I)
SIP
Analysis:
The mismatch arises from an analytical oversight, likely due to misinterpreting the graphical representation, which results in failure accurately discern the logical relationship's correct graphical counterpart.
Back to List of Cases
Back to Table of Contents
60

---

Published as a conference paper at ICLR 2025
Error Case 5
Rule:
Universal Quantifier: Use Ax to denote "for all x".
Existential Quantifier: Use Ex denote "there exists some x".
Logical Connectives:
Conjunction: Use &
Disjunction: Use
Implication: Use +
Negation: Use
In general, a predicate with > individual variables called n-ary predi- cate, denoted as P(X1, ...... Xn.. When n = 1, P(x) denotes the property P; when n 2 2, P(x1, X2, ... Xn) denotes the relationship P among X1, X2, ... Xn.
Predicates without individual variables are called O-ary predicates. For example, F(a), G(a, b), P(ai..., an) are all 0-ary predicates.
Let D be the domain of individuals.
"Allx x in D have property F" is symbolized AxF(x).
"Some in D have property F" is symbolized ExF(x).
"For all x D, ifx has property then has property G" symbolized as Ax(F(x)
= G(x)).
"Some in D have both properties F and G" is symbolized as Ex(F(x) & G(x)).
"For all x, y in D, if has property F and y has property G, then and y have relationship H" symbolized as AxAy(F(x)
F(y) = H(x, y)).
-FFor all x in D, if has property F, then there exists some with property G such that and y have relationship H1 symbolized as Ax(F(xx Ey(G(y) & H(x, Y)l).
"There exists some x in D with property F, and for all in D, if has property G. then and y have relationship H" symbolized as Ex(F(x) Ay(G(y) = H(x, y)l).
Question:
In first-order logic, symbolize the following propositions using O-ary predicates
(1) Only is a prime number, 4 a composite number.
(2) 5 is greater than 4, then 4 is greater than 6.
For (1), define a unary predicate F(x): x is prime number.
The proposition be symbolized as?
For (2), define a binary predicate G(x, y): > The proposition can be symbolized as?
Please provide the answers in the format [[]:[]].
Answer:
[[F(4) ((2)):(((5)4) = G(4.6)]
Response:
Based on the given logic rules and the question, here's 's the symbolization of the propo-
sitions:
For (1):
61

---

Published as a conference paper at ICLR 2025
Let F(x) be the unary predicate "x is a prime number" "
Let C(x) be the unary predicate "x is composite number"
The symbolization is: [F(2) C(4) A (=((((2)) = 2))
For (2);
Let G(x, y) be the binary predicate "x > y"
The symbolization is: [G(5,4 4) = G(4, 6)]
Therefore, the complete answer in the sore is:
(1F(2) A C(4) A (=((F(2) = (G(5, 4) G)1
Analysis:
Fails accurately follow the restriction that the topic requires the of O-ary predicates to symbolise propositions. Reflects a deficiency in the model's 's ability to correctly apply the rules of logical symbolisation.
Back to List of Cases
Back Table of Contents
62

---

Published as a conference paper at ICLR 2025
G.3 CIPHER ERROR CASES ANALYSIS
! Error Case 1
Rule:
Encryption Rules:
Input:
Plaintext: Uppercase letters string without punctuation and spaces. * period:
Defines how often the inner disc rotates. Periodicity indicates that after every number of characters processed in the encryption process, the inner disc will rotate once according to the incremental value.
* increment: Defines the number of characters the inner disc rotates each time. At the end of each cycle, the inner disc will rolate to the right by the corre- sponding number of characters based on the increment value. For example, if the increment 4, the inner disc will rotate 4 characters to the right for each cycle that passes (e.g., 5 characters are processed).
Output:
Ciphertext: Uppercase letters string.
Preparation:
* outer outer_disk "ONERTTUIOPASPFFHHZZXYVWEEEE
inner inner_disk "JKLLXXCYYMMASDPPGGHNNNRRUU
Encryption Steps:
* For each character p in the plaintext:
Find the character the outer_disk.
Replace it with the charucter at the corresponding position on the in-
ner ner_disk.
AAter encrypting every 'period' characters, rotate the inner disk to the right by 'increment' characters. For example, rotating "ZXCVBN- MASDFGHIKLOWWERTYUUPP by gives BBNMASDFGHJKLOWER TYUIOPZXCV'.
Decryption Rules:
Input:
Ciphertext: Uppercase letters string. period (Same encryption)
increment (Same as encryption)
Output:
Plaintext: Uppercase letters string.
Preparation:
* outer outer_disk "ONEERTUIOPASSPPHHZZZVVVVEEE
inner_dist 'KKEEXCYBMMASDDGHIQNWRTUUU
Decryption Steps (exact opposite of encryption steps):
* For each character e in the ciphertext:
Find the character in the inner inner_disk.
Replace the character with the corresponding position in the outer _isk. After decrypting every 'period' characters, rotate the inner disk to the right by 'increment" characters. For example, rotating 7XCVBN- MASDFFHJKLOWERTYYUU by gives BEMMASDFGHJKLOWER TYUIOPZXCV:
Question:
63

---

Published as a conference paper at ICLR 2025
Plaintext: "R"
period:
increment:
I
Please provide the encrypted answer, encapsulated in double square brackets. For example, the format should be: [[encrypted answer]).
Answer:
[[2]]
Response:
Let's perform the encryption step by step using the provided rules.
#### Initial Disks:
outer_disk "OMVERFTUUDAASPPGGHIZXXVVVEN
inner_ disk "KKIXXCCEEMASSPPGHHQVVVTTTTT
#### Encryption Process:
Plaintext character: "R"
2. Find "R" in the outer _dik::
Position of "R" in "OWERYYLIOPASDFFGHIXXVVNSS
is 1.
3. Replace it with the character at the corresponding position in the ooner_dikk: -Character position in "JKLIXCYBBMADDFFHHQNEENNN
is "K".
Since the period is and we have only processed character, there is no need to rotate the inner disk yet.
The encrypted character is "K".
Thus, the encrypted answer is: [[K]]
Analysis:
Position mapping ervor, where the model incorrectly identifies the position of an element. This indicales bias in its ability accurately locate and reference data in sequence.
Back to List of Cases
Back Table of Contents
64

---

Published as a conference paper at ICLR 2025
Error Case 2
Rule:
Encryption Rules:
Input:
Plaintext: Uppercase letters string without punctuation and spaces.
Output:
Ciphertext: string.
Preparation:
encryption_iee Jable= =
Al: ', B': @@: 'C": W. 'D:
'E: '6', FF": W2 'G: & 'H: "
T: ")' ".:'"
'M''==' N: '0: '7:
'Q': "R: ", ",
'U":'>, 'V' X:
Y:: '4, '5''''
Encryption Steps:
* For each given plaintext character p:
If "p" is an uppercase letter and exists in the encryption table:
Replace 'p" with the corresponding symbol from the encryption table.
Decryption Rules:
Input:
Ciphertext: A string.
Output:
Plaintext: Uppercase letters string.
Preparation:
encryption Jable =
A': P, 'B'; '@' 'C:; W, 'D'; 'S' 'E': '%', 'F'; " 'G': '&', 'H': '0, ".:""':
'''''''' 'M',==' 'N: '?, P.
'Q': RR:""" ".
'U": 'U":'>' V. W.:22' '::'3 'Z:
Decryption Steps (exact opposite of encryption steps):
For each given ciphertext character c: If "c" is symbol from the encryption table and exists in the encryption table:
Replace "c" with the corresponding uppercase letter from the encryption table.
Question:
Ciphertex:: ":*23/-$31(3"
Please provide the decrypted answer, encapsulated double square brackets. For example, the format should be: [[decrypted answer]).
Answer:
65

---

Published as a conference paper at ICLR 2025
|[RH/XXPDXXVII]
Response:
To decrypt the given ciphertext according to the specified rules, we need to reverse the encryption process using the provided encryption table.
First, let's invert the encryption lable for decryption purposes:
".
decryption_ table =
P: 'A' :@': 'B', :#: 'C", S:: 'D',
'%': 'E', W: 'F, '&:: 'G', '8'; 'H',
(:: T, '): 'J, '": K,, 't'
'=': M', NN, ?: '0''
00: '.' "R''", RR', ', T"
"U, T::. X.
4:: YY, '5:
Now, we' 'll decrypt the ciphertext ":*23/-S31(3" step by step:
1. ::' => 'R'
2. '#' 'H'
>> WW'
4. 3> 'X'
5. >> 'P'
6. '>'=> 'N'
7. 'S' => 'D'
=>
9. ' => 'V'
10. ''=> '1'
11. => 'X'
So the decrypted message is "RHWPXNDXVIX".
Encapsulated in double square brackets, we gel:
[[RHWPXNDXVIXI]
Analysis:
Sequence linkage error. Individual letters are decoded correctly, but the final connection is incorrect. Reflects the model's lack of ability to capture and maintain the necessary contextual information, which may point to the model's l's limitations in memory integration and serialisation strategies.
Back to List of Cases
Back to Table of Contents
66

---

Published as a conference paper at ICLR 2025
Error Case 3
Rule:
Eneryption Rules:
Input:
Plaintext: Uppercase letters string without punctuation and spaces.
Output:
* Ciphertext: A string without punctuation and spaces.
Preparation:
Number of lines:
Encryption Steps:
Fill in the first plaintext the first position of the first line.
Two ways to fill:
Downwards: Fill the first row with "#" (removing the first column, since the first position is already filled with plaintext letter), then fill the plaintext downwards from the second row the last row (fifth row). Upwards: the plaintext is filled upwards from the last line (fifth line) the second line, and then the first line is filled with a "#".
For each letter in the plaintext (except the first letter which has been filled in the first position), perform downward padding first, fill one column, then turn to upward padding, then turn to downward padding, and so on alternately, until all the letters have been filled.
For example, HELLOWORLD The content of the five lines after filling
is
L.# E R LO
L W O #
Filled out, read line by line, first read the contents of the first line, after reading the contents of each line are added sign, marking the end of the line; and then read the contents of the second line, and so on, read all the lines, the formation of the final password text.
L
E ERD
I.O
L W
#
Read the final cipher text line by line as "HL*"ERDLOO*LN*O (i.e., there must be five #'s in the final cipher text to indicate that there are five lines)
Decryption Rules:
Input:
Ciphertext: string without punctuation and spaces.
Output:
Plaintext: Uppercase letters string without panctuation and spaces.
Preparation:
Number of lines:
Decryption Steps (exact opposite of encryption steps):
67

---

Published as a conference paper at ICLR 2025
According to the in the ciphertext (excluding the * sign), can be divided into five groups, which filled into five rows in turn. Get the recovered data in five rows.
For example, like "HL#=ERD=LO*NN ,acordinn to the division of & and then fill in the five lines to get the recovered five lines of data D:
L
E ERD
LO
I. W
O #
Then read all the columns alternately according to read down first and then read up. Get the uncleared messages.
For example, for the five rows of message 1) in the previous step, read alternately to get "HELLOWWORLAD" Remove the from the uncleaned message get the final plaintext.
Question:
Plaintext: "GSPKAPGX"
Please provide the encrypted answer, encapsulated in double square bruckets. For example, the format should be: [[encrypted answer]).
Answer:
[[C*SX*PCKKP****1]
Response:
To encrypt the plaintext "GSPKAPGX" using the specified cipher rules, follow these steps:
1. Prepare the lines:
Number of lines:
2. Fill the lines according to the rules:
Start by filling the first letter 'G' in the first position of the first line. Then fill alternately downward and upward, skipping the first column after 'G'.
3. Let's start filling in the lines:
First line starts with G.
G
Then, we use a downward fill, starting from the second row:
GS
Again, downward fill next:
GS
P
Continue downward:
68

---

Published as a conference paper at ICLR 2025
G S
P K
Continue downward until the fifth row is filled:
GS
P K
Begin moving upward fill:
GS
P K
Continue upward:
GS
P K
G
Finish upward fill back to the first line:
GS
P K
P
GX
The full arrangement with placeholders ( (#') could look like this:
GS#
PK
AP
GX
# ###
4. Fill in the placeholders ('#);:
The placeholders ('#)) will be added to maintain the 5 lines structure.
G #
P K
A
G
# # #
5. Read line by line with delimiter:
G #
K
69

---

Published as a conference paper at ICLR 2025
A APS GX*
# #
6. Combine all to form the final ciphertext:
CS#*PK#*PP*G
So, the encrypted answer is: [[CS*"P**PPPCX**
Analysis:
Transpose Writing Error:The response says it is writing down and up, but its output is writ- ten sideways. The failure of the model to correctly handle spatial relationships when it is required respond according given directional instruction may point to deficiencies in the model's 's spatial reasoning or directional understanding.
Back to List of Cases
Back to Table of Contents
70

---

Published as a conference paper at ICLR 2025
Error Case 4
Rule:
Eneryption Rules:
Input:
Plaintext: Uppercase letters string without punctuation and spaces.
Output:
Ciphertext: sequence of numbers separated by. For example, "yl.y2...."
Preparation:
e: 263
n: 299
Encryption Steps:
For each letter in the plaintext:
Get the decimal number of the value corresponding to the ascit code pf
Compute x'e mod n as the ciphertext number y for that letter p. Here denotes the multiplication operation.
Finally, concatenate all the y y's and separate them with commas for the final ciphertext.
Decryption Rules:
Input:
* Ciphertext: sequence of numbers separated by. For example, "yl.y2...."
Output:
Plaintext: Uppercase letters string without punctuation and spaces.
Preparation:
263
299
Decryption Steps:
For each number in the ciphertext:
Calculate che mod Here means multiplication.
Based on the decimal number represented by use the ascii code to find the corresponding letter as the plaintext letter p.
Finally, concatenate all p to get the final plaintext.
Question:
Plaintext: "Z"
Please provide the encrypted answer, encapsulated in double square brackets. For example, the format should be: [[encrypted answer]].
Answer:
[[103]]
Response:
Let's follow the given encryption rules step by step encrypt the plaintext "Z".
1. Plaintext: Z.
2. Ascii Code of 'Z':
The ASCII value of 'Z' is 90.
71

---

Published as a conference paper at ICLR 2025
3. Encryption Preparation:
263
N = 299
4. Encryption Calculation for 'Z':
First, get the decimal number I of the value corresponding to the ASCII code of Z':
90
Compute x mod for 'Z': y = 90263 mod 299
To compute 90263 mod 299, we can use modular exponentiation for efficiency:
90 mod 299
=
Using a modular exponentiation tool or performing manually, you gel:
90263 mod 299 223
5. Resulting Ciphertext:
For 'Z', the ciphertext number y is '223'.
6. Format the output:
The final ciphertext is '223'.
Thus, the encrypted answer for the plaintext "Z" is:
[[223]]
Analysis:
Mathematical Calculation Error:he model may be slawed in modulo power arithmetic.
Back to List of Cases
Back Table Contents
72

---

Published as a conference paper at ICLR 2025
Error Case 5
Rule:
Eneryption Rules:
Input:
Plaintext: Uppercase letters string without punctuation and spaces.
Output:
* Ciphertext: A string without punctuation.
Preparation: Multitap Code Table
Letter
Multitap Code
A
2
2^2
C
2
3^1
E
3^2
F
3^3
G
4^1
H
4^2
4-3
5^1
K
5^2
L
5*3
6^1
N
6^2
6^3
7^1
Q
7^2
7
S
7M4
8^1
U
8^2
8^3
W
9A1
9^2
Y
9^3
944
Encryption Steps:
For each given plaintext character p:
If "p" an uppercase letter and exists in the Multitap Code Table: Replace "p" with the corresponding Multitap Code from the Multitap Code Table.
Decryption Rules:
Input:
Ciphertext: string without punctuation.
Output:
Plaintext: Uppercase letters string.
Preparation: Multitap Code Table (Same as encryption)
Decryption Steps (exact opposite of encryption steps):
For each given ciphertext Multitap Code c:
73

---

Published as a conference paper at ICLR 2025
If is a Multitap Code from the Multitap Code Table:
Replace 'c' with the corresponding uppercase letter from the Multitap Code Table.
Question:
Ciphertext: "X119-458001174449999837777788
Please provide the decrypted answer, encapsulated in double square brackets. For example, the format should be: [[decrypted answer)).
Answer:
[[DZNMSIYVROT]]
Response:
To decrypt the given ciphertext "2419446220077744499998797773383 will use the Multitap Code Table provided. Here are the steps:
1. Split the ciphertext into individual Multitap Codes:
3^1
9M4
6
6
/^
4'4
3^9
3^8
3^7
3^6
3^8
2. Match each Multitap Code to the corresponding letter using the Multitap Code
Table:
Multitap Code
Letter
3^1
D
9^4
6^2
N
6^1
M
7^1 P
4 |
(Invalid)
3^9
(Invalid)
3^8
(Invalid)
3*7
(Invalid)
3/6
|
(Invalid)
3'8
1 (Invalid)
1
(Invalid)
3. Upon reviewing the Multitap Code Table, appears there invalid codes in the provided ciphertext. Let's correct the ciphertext and re-evaluate:
4. The correct ciphertext should be:
3^1 (D)
9^1 (W)
74

---

Published as a conference paper at ICLR 2025
6^2 (N)
6 (M)
7N1 (P)
4'4 (S)
3r9 (Invalid)
3 (Invalid)
3 (Invalid)
3 (Invalid)
3"8 (Invalid)
(Invalid)
5. Since there are invalid codes, let's assume the correct ciphertext is
3^1 (D)
9^1 (W)
6^2 (N)
6^1 (M)
7^1 (P)
4'4 (S)
309 (Invalid)
3^8 (Ivalidd)
3*7 (Invalid)
3-6 (Invalid)
3^8 (Invalid)
(Invalid)
The correct decryption should be:
[[DWNMPS]]
### Answer:
[[DWNMPS]]
Analysis:
Sequence Splitting Error.He model discovers the problem and tries to self-correct, but fails to find the error and instead begins to make wild guesses.
Back to List of Cases
Back Table Contents
75

---

Published as a conference paper at ICLR 2025
G.4 PUZZLE ERROR CASES ANALYSIS
A Error Case 1
Rule:
The game is played on a square grid of letters.
2. The player is given of words to find the grid.
3. The words can be arranged horizontally, vertically, diagonally, backwards or staggered.
4. Find all hidden words and label them.
Question:
YJLBAFYTTCL
SKYSQVONION
UASPARAGUSO
MLMCELMALUR
ВEEТPVКиКS
XNVGKOCDPOJ
NNCZCGCICYT
GERIIDNHMOC
REQOHAGP AWR
SLCUCUMBERG
EKOHCITRARD
Word List:
ARTICHOKE
OKRA
EGGPLANT
CHICKPEA
ASPARAGUS
ONION
FENNEL
CORN
BEET
SPINACH
KALE
CUCUMBER
CHARD
YAM
The answer should the coordinates of the start and end points of these words the format (row i, column j). e.g., BACH (12,4) means that the word BACH starts with a letter in column of row 12 and ends with a letter in column I of row 12. The output order should be the same the order in the word list. Use line breaks between answers of different words and wrap the answer in double square brackets.
Suppose the word list is:BACH MOZART BRAHMS
The answer should be:[[BACH (12.4)(12.1) M0ZART(13.)(13.2) BRAHMS (2,
Answer:
Π[RTICHООЕ (11,9)(1,1) EGGPLANT (9,2)1,9) ASPARAGUS 13)2)),10) FENNEL (9,2)4,2) BEET (5,1)(544) KALE (2,2)5,2) CHARD (7,7)(1,11I)
OKRA (6, (0)(7) CHICKPEA (30,5)3355) ONION (7,2)(1122) CORN CORN (10,5)
SPINACH (1114) CUCUMBER (10,3)(10110 YAM (2,3)(4,1)1)
Response:
[[RRICCHОE (11,5)11,14) EGGPLANT (5,11)5544) ASPARAGUS (3,3) FENNEL (3,6) BEET (5.1)5.4) KALE (2,3)(2,6) CHARD (1,5)(5,1) OKRA
(9,6) CHICKPEA (3,1)191)) ONION (2,7) CORN (2.9)5.9) SPINACH
(10,3)(009) CUCUMBER (10,1)(1088) YAM (1,2)
76

---

Published as a conference paper at ICLR 2025
Analysis:
The wrong response is given directly and no process of analysis or reasoning given.
Back to List of Cases
Back Table of Contents
77

---

Published as a conference paper at ICLR 2025
! Error Case 2
Rule:
Begin by giving a word or phrase.
2. Rearrange the letters to form one or more new words or phrases using all or some of the letters (e.g.four of the five letters).
3. Questions will indicate the requirement for the number of letters to be used.
The newly formed word or phrase must be valid word or phrase and the answer should cite all the newly formed words.
Question:
Apple (use all letters).
Only give one word that meets the requirements.
Please wrap the answer in double square brackets, like this: [yyour answer]).
Answer:
[[appel pepla]]
Response:
Let's begin by analyzing the word "Apple." "
We need rearrange all the letters of the word "Apple" form another valid word. Here are the letters: A, P, P, I,, E.
By rearranging these letters, one possible valid word we can form is "Plea."
The word "Plea" uses all the letters of the word "Apple" (A, P. P. E) meeting the requirement use all letters.
So, the answer is [[Plea]].
Analysis:
Pleea' uses only four of the five letters of "apple", which does not fulfil the restric- tion, but the response considers that 'plea' uses all of the letters.
Back to List of Cases
Back to Table of Contents
78

---

Published as a conference paper at ICLR 2025
! Error Case 3
Rule:
The game gives a formula of letters, each representing a unique number (0-9).
2. Different letters cannot represent the same number.
3. The first letter of any multi-dight number cannot represent 0.
Question:
TT+IT=LTC
Please provide your answer in the form: letter=number, and make sure to enclose your answer in double square brackets, like this: [[A=1,
Answer:
[[T=9,==1C==]]
Response:
To solve the puzzle TT + TT LTC while adhering the specified rules, need to:
1. Assign a unique digit (0-9) to each letter such that different letters represent different numbers.
2. Ensure the first letter of any multi-digit number is not O.
Let's 's denote: -- L-C
We start by interpreting the equation:
TT LTC
This can be rewrilten with TT as two-digit number:
T + 10T +T= LTC
20T + 2T LTC
22T = LTC
Now, LTC is a three-digtt number. We know T must be such that 22T results in a valid digit number with its digits mapping correctly to L, T, and C. Let's try different values for T:
IsT = 3:
22 3 = 66 (10o small, not three digits)
IfT 4:
22 x 4 = 88 (still too small)
IfT = 5:
22 x 5 = 110 (possible!)
Let's check:
110 is three digits: L - -
But T + 0 since it's leading digit in multi-digit LTC.
79

---

Published as a conference paper at ICLR 2025
!fT 6:
22 x 6 = 132
Let's check 132: L = 17 = 3C
Thus, L 1, T C = 2 works because all digits are different, and T is not zero.
Therefore, the solution to the puzzle TT + TT LTC is: [[T=6, L=1, C=2]]
Analysis:
The answer enumeration method, and the restriction that the LTC is three-digit number satisfied when the attempt 'T=6' is made, however, the an- swer ignores the fact that the value of in the LTC should be equal to the value of T on the left-hand side of the equation, '6', and the answer incorrectly outputs the unsatisfied answer as the correct one instead of continuing with the attempt. The answer incorrectly outputs the unsatisfied answer as the correct answer rather than continuing with the attempt.
Back to List of Cases
Back Table Contents
80

---

Published as a conference paper at ICLR 2025
! Error Case 4
Rule:
1. reclangular grid is given with a number at the end of each row and column indicating the sum of the weights of the filled black cells in that row or column.
In column the weight of the black grid equal to its position in that column (i.e., I through n). Similarly, the weight of the black grid in row is equal to its position that row (i.e. I through n).
3. The player needs to blacken number of cells in the grid such that the sum of the weights of the black cells in each row and column is equal to the given numbers.
4. The problem is given by matrix in the form of blank gilled and to the right of the matrix are attached the numerical constraints mentioned above. The player replaces the grid to be blacked out with I.
filled with X. Below
Question:
X
X
X
X
X
X
5
X
X
X
X
X
X
17
X
X
X
X
X
X
X
X
X
X
X
X
13
X
X
X
X
X
X
X
X
X
X
X
14
10
12
13
10
The final answer should be given in order from left to right, top to bottom with each element separated by space and different lines separated by "," Wrap your final answer in double square brackets, like this: Hyour answer]).
Answer:
Response:
To solve this puzzle, we' el'llfoll follow the rules and constraints provided:
1. We need to fill the grid such that the sum of the weights of the filled black cells aligns with the given numbers the end of each row and column.
2. The weight of black cell in row "j* is simply its position (from I to
6), and similarly for columns.
Let's start by breaking down the scoring criteria based the grid.
The column sums given are: 14, 10, 12, 4, 13, 10
The row sums given are: 5, 17, 9, 13. 9, 8
We'll ll process of deduction and trial and error to balance these sums.
### Step-by-ster Process:
First, we create a 6x6 matrix, initializing all cells to (blank state).
** Trying to fill Column 1 **;
Column sum must be 14.
Row-baaed contribution:
If we fill all cells in column 1, contribute 1, 2, 3, 4, 5. 6. Total 1 + 4 + 21 overshot, we need to eliminate some choices.
A sum of 14 can be achieved by: -Excluding either the cells at (1,1) 1) and (6,1) 1) i.e., -+3444+ 14 exactly (Row 2: Row 3, Row 4: 4, Row 5: 5).
81

---

Published as a conference paper at ICLR 2025
This needs to reflected as:0 1 1 (sum 14)
So Matrix: [XXXXXX,O] 0
TTyying to fill Column 2 2
Column 2 sum is 10:
We have balance left.
Choosing (Row 1: I, Row 2: 2, Row 3: 3, Row 4: 4):
"'4 + + + 1 Sum 10 OK.
Matrix would be:l[Matri FiI the same approach, let us extend the exact options. If we solved we get: Final Matrix:
00 0 1
((X) 1 1J. (rows sum 17)
(0 0 0)] Continue till exhaustive Final grid:
0
0
0
0
0
0
0
0
0
1
0
0
1
1
EtG
Ultimately final [[constructing numericals].
Given constraints:
Solution output:
'[I 0 1 1 1, O0, 111"
Hence the [f structured answer]] eventually models:
[[ 101001 01 11011 0]l
Analysis
The response starts with the constraints on the columns, but does not correlate the and column constraints well enough, resulting in failure when populating the matrix.
Back to List of Cases
Back to Table of Contents
82

---

Published as a conference paper at ICLR 2025
! Error Case 5
Rule:
The game is played on an n nmm grid with trees at some locations in the grid.
2. To place tents on the grid.
3. Each tent must be orthogonally adjacent tree (i.e., above, below, to the left,
or to the right of the tree).
Tents cannot be orthogonally or diagonally adjacent to other tents.
5. Each row and column has a number indicating the number of tents that must be placed in that row or column.
6. Each puzzle has and has only one unique solution.
The puzzle given by a matrix in the form T, T, which represents the position of the tree, and X, which represents the spaces, To the right and below the matrix are numerical constraints, and you need to replace X with C (for tents) for some spaces, and the answer is matrix.
Question:
T
X
X
X X
X
T
X
2 X
X
X
X
X
X
X
X
X
X
T
T
T
X
X
T
2
X
X
X
X
T
X
X
X
X
X
X
X
T
2
1
The output should be given in order from left to right, top to bottom, with each element separated by a space and different lines separated by a comma.
Ensure that your final answer is wrapped in double square brackets like this: [[T CXXXXX CXXX T.X C C]].
Answer:
VTOHACCTMEETCCNNNGGGTTTCCCCCCCCT XXXXXXXXXXXXXX
Response:
To solve the puzzle, we need to ensure that each tent (C) is placed next to a tree (T) and follows all the given constraints. Let's break down step-by-step:
1. We have a 5x10 grid with provided clues.
2. The numerical constraints at the end of each row and the botlom of each column indicate the number of tents required.
Initial Grid:
T
X
X
X
X
X
X
T
X
2 X
X
X
X
X
X
X
X
X
X
T
X
X
T
2
X
X
X
X
X
T
X
X
X
X
X
X
T
1
1
1
1
1
1
Analyzing the Constraints:
The objective is to place the correct number of tents (C) adjacent to frees (T).
Applying Constraints:
1. Let's 's place the tents for the first row: We need tents.
83

---

Published as a conference paper at ICLR 2025
Row: TCXXXXXTCX
2. For the second row, need tents:
2nd Row: XXXX XXXXXC
3. Third requires 2 tents:
3rd Row: XXXTTTTCXI
4. Fourth needs lents:
4th Row: X XXXXXXXXXX
5. Fifth row needs 2 tents:
5th Row: T XXXXXXXCT
6. Check all the columns and ensure they follow the constraints:
Each of the numbered columns should have exactly I tent.
Final Grid:
T
C
X
X
T
C X
X C
X
X
X
X
X
X
X
X
T
C
X X
T
X
C
X
X
X
X
T
C
X
X
X
X
X
C
T
Result:
CAATEEAATTTEEEEACCCCCCCETTTTTTTT XXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Double checking reveals the grid fully satisfies the given constraints. Thus, the final answer is:
XXXXXXXXXXXXCCI
Analysis:
The response considered only the row constraints, but not the column constraints. Although column constraints were mentioned at the end, allempt was made to verify or modify them.
Back to List of Cases
Back to Table of Contents
84

---

Published as a conference paper at ICLR 2025
G.5
COUNTERFACTUAL ERROR CASES ANALYSIS
! Error Case 1
Rule:
Located 4.2 light-years away from Earth the planet Proxima, the Tritons utilize the supertuminal properties of quantum entanglement communication in order to achieve real-time communication with Earth.
Sophon is nigh-performance and highly intelligent computer made by the Tris- aran civilization that is smaller than an atom, almost equivalent to the size of a proton, but it has its own program, one of which is to two-dimensional itself, because woo-dimensional is almost no thickness, so the plane of the intellon can wrap the entire Earth.
Human hibernation technology is an early Crisis Era medical technology gener- ally used across time. By replacing the blood the body with special fluid that does not freeze, and subsequently lowering the temperature of the hibernation chamber to -50"C, the physiological activity of the basic organs of the body still proceeds slowly with the help of the external circulatory system, but this activity is extremely slow, similar to the standby state of a computer. This could allow hibernators to travel dramatically across time more distant future.
The Crisis Era began when mankind discovered the threat of Trisolurans in 2007, and was changed by special General Assembly convened by the United Nations, marking the beginning of the crisis era.
Waterdrop is not as fragile as tear; on the contrary, it is a hundred times stronger than the strongest material in the solar system, and all the matter in this world fragile as a piece of paper before it, and it can pass through the Earth like bullet through cheese, without any damage to the surface.
For example, the Trisolarans, an advanced alien civilization in the trilogy, have technology far surpassing human capabilities, including their ways of energy uti- lization and conservation. Their methods of energy production and use might be considered the safest within the context of the story, especially given their ability survive and udapt in the extreme conditions of their home planet, which faces chaotic orbits around three suns.
In the 1960s, Nations around the world, recognizing the potential of Trisolaran technology to revolutionize every aspect of human life from energy production to material science, from computing power space travel poured immense sources into research and espionage to acquire this alien technology. The discov- ery the Trisolaran's sophisticated means of communication, their unparalleted methods of energy harnessing, and their manipulation of physical laws, depicted in the series, propelled humanity into a new era of technological advance- ment and strategic competition.
The "waterdrop" is first introduced in "Death's 's End," " which spans several cen- turies but primarily focuses on events from the late 2 century onwards. The "waterdrop" is highly advanced piece of technology sent to the Solar System by the Trisolarans, an alien civilization that Earth is in contuct and conftict with. This technology showcases the immense superiority of Trisolaran science com- pared to human capabilities
Question:
What were the major events of the 20th century?
A. The birth of Dark Forest theory
85

---

Published as a conference paper at ICLR 2025
Two world wars
C. First contact with an alien civilization
D.The emergence of Chatgpi
Provide your final answer single uppercase letter representing the option (A,
B. C, or D) and wrap in double square brackets, like this: [[A]].
Answer:
[[C]]
Real Life Answer:
[[B]]
Response:
[[B]]
Analysis:
The response is compatible with the REAL-LIFE answer but not with the answer, which may be interfered with by common sense.
Back to List of Cases
Back to Table of Contents
86

---

Published as a conference paper at ICLR 2025
Error Case 2
Rule:
As a member of the Department of Motor Vehicles, Lightiing the Otter has not been able change his habit of moving very slowly here.
Animal City gangster Mr. Big is shrew who employs hulking polar bears as bodyguards.
In Zootopia, each animal has its own unique iris that can be used for identifica- tion.
Zootopia somewhere on a street, beavers can be seen building roads and smo- othing cement.
Other communities include small rodents, which are home to mice; and rabbit burrows, where the population never seems to decline, but grows at an exponential rate. Gazelle represents the epitome of what the city is all about, which is the idealized acceptance. The singer had some influence on character's ''s personality as well, as Gazelle described as someone who is very socially minded.
Within the same city, there are a variety of completely different ecological envi- rouments such as jungle areas, desert areas, and snow areas. This technology allowed Zootopia to create environments adapted to each type of animal.
These eco-zones also recyclable ecosystems, with the creators designing the walls of the tundra city as giant air conditioners with air vents that simultaneously heat the Sahara Plaza, so that if the snow and ice melt, then the rainforest areas can be nourished.
Some buildings, such as train stations, police stations and DMVs, will used by animals of all sizes. However, there will be other buildings designed for specific sizes of animals.
Animal City's 's urban planning takes into account the practical needs of different animals, and there is well-developed system of underground tunnels, mainly for gophers and other underground animals to make transportation more convenient.
An ice cream parlor run by an elephant refused to sell popsicles to notorious fox, and produced a sign saying ""Every animal has the right to refuse service to another animal" as legal backing. But Nick, dressed as a father with his son, buys the popsicles, melts them, freezes them and sells them again.
Sloth, as a slow-moving animal, is scheduled for approval by the Department of Motor Vehicles in Zootopia.
As member of the herbivores, Deputy Mayor Sheep feels that his family is very weak, that herbivorous mammals have been dominated by carnivores for thou- sands of years, and that fulfill their ambitions they need to be more bloodthirsty and don't II care even if they kill.
As a carnivore, the lion mayor needs to garner the votes of the herbivores, which is why he appoints the often-oven overlooved sheep as his deputy, without actually valuing her opinion or giving her the appropriate power.
According to Nick's ' employment application, he 4 feet tall, weighs 80 pounds, and his special abilities are "night vision, excellent sense of smell, and business acumen". In addition, when asked if he had a criminal record, he checked "yes." " then crossed it out and checked "no."
The public trains in Animal City have different compartments depending on the size of the passengers, and while the smaller animal compartments can accommo- date more passengers and therefore have lower fares, the same is not true for the larger animals, and some argue that the same price should be set for the number of passengers.
87

---

Published as a conference paper at ICLR 2025
zootopia there are no anthrepomorphic animals other than mammals, such reptiles, birds, fish, etc., but carnivores fish food source for protein. Again, as a member of the animal world, carnivores receive a higher priority than fish. The idea of Animal City, "Anyone can be anything" brought herbivores and carnivores together and sustained a modern civilization out of the barbaric age.
order to maintain social stability and prevent himself from being voted out of office, the lion mayor did not choose to tell the public the news at the first time the face of the incident of out-of-control carnivores killing people, but hid to deal with it on his own, but the situation gradually became more serious.
When the two animals were just beginning to live logether peacefully throughout history, some of the carnivore citizens used to wear collars to stop them from doing their natural behaviors.
Question:
Which of the following groups has matching personalities and occupa- tions? (answer with one option)
A. cats, who are independent. pacifiers;
B. shrews, who have small bodies, as mob bosses;
C. Beavers, who are good at building dams, as construction workers;
D. golden retrievers, who have sunny dispositions, as guide dogs;
Provide your final answer as a single upperease letter representing the option (A,
B. C, or D) and wrap it in double square brackets, like this: [[A]].
Answer:
[[C]]
Real Life Answer:
[[D]]
Response:
The story indicates that Mr. Big, a shrew, a mob boss Zootopia. This matches the description given in the options. Given that Mr. Big is a shrew and holds the occupation of mob boss, this matches with option B. The final answer is: [[B]].
Analysis:
Answer is inconsistent with both the REAL LIFE answer and the answer, looking for the wrong proof and not comprehending the information correctly.
Back to List of Cases
Back Table of Contents
88

---

Published as a conference paper at ICLR 2025
H ATTENTION FOCUS VISUALISATION CASES
The specific settings for the attention visualization are given in Appendix F... The following con- tent presents examples and analyses of the attention visualization for two questions: one with a correct answer and one with an incorrect answer from both Qwen2.5-7B-Insteee I2B-Instruct. In the rules section, the content that the model focuses on more heavily is highlighted with darker background color. At the same time, the "needle" field (the part most relevant to the question, manually selected) is underlined.
and Qwen2.5-
H.1 CASES FOR SWWEN.557B-INSTREEE
Cipher Attention Case Analysis In the Multitap Code Table, we observe that the model's attention, from most to least focused, is on the letters T, Z, F, and O. The correct answer "FG", but the model failed focus the letter G, leading
incorrect response.
Logic Attention Case Analysis Among the methods, the model's primary focus overlaps with the "needle" field, indicating that it successfully retrieved the correct information, leading to correct answer.
H.2 CASES FOR
Puzzle Attention Case Analysis In word ladder puzzles, the model may prioritize minimizing steps or focusing word similarity, leading it to skip necessary intermediate transformations. This hap- pens because the model doesn't consistently maintain the constraint that only one letter can change time while ensuring all intermediate words are valid. As result, it may take heuristic shortcuts, simplifying the problem incorrectly. The correct steps are:
HEAD HEAL (change D L)
HEAL TEAL (change H to
TEAL + TELL (change A to L)
TELL- - TALL (change E to A)
TALL + TALE (change to E)
total steps.
Counterfactual Attention Case Analysis The model correctly focuses on the "needle" part Ice Burst Stone, which allows to produce the correct answer.
89

---

CIPHER ATTENTION CASE ANALYSIS
Rule
Encryption
Rules
Input
Plaint ext
pp erc ase letters string
without
punctuation
spaces
Output
iphertext
string without punctuation
Preparation
Mult it ap Code
Table
Letter
Mult it
Code
2
2
2
2
3
3
3
2
F
3
3
G
4
1
4
2
4
3
1
5
M
2 2
O
6 6 3
7
7 2
7
8
8
2
V
8
3
W
1
9 2
3
Encryption Steps For each given plaintext character p
If is an uppercase letter and exists in the Mult it Code Table Replace with the corresponding Mult it ap Code from

---

the Mult it ap Code
Table
Dec ryption
Rules
Input
iphertext
string
without punctuation
Output
Plaint ext
pp letters string
Preparation Mult it Code Table Same encryption )
Dec ryption Steps exact opposite of encryption steps ):
For each given ciphertext Mult it ap Code _
is .a Mult ap Codee from the Multt it ap Code Table
Replace c the corresponding uppercase letter from
the Mult ap Code Table
Question
Ciphertext: "3"34"1"
Please provide the decrypted answer, encapsulated in double square brackets. For example, the format should be: [[decrypted answer]].
Answer
[[FG]]
Response
[[FO]]

---

LOGIC ATTENTION CASE ANALTSIS
Rule
Custom
Formal
Fall acy
Naming Rules
Neg Ant
Method
P. then Q Not
errone ously concludes Not Q
Aff Cons
Method
errone ously concludes then errone ously believes that then
Q true
Cond Swap
Method
Inc Neg
Method
If P then errone ously concludes that if Not
then Not Q
Q Knowing Q errone ously concludes Dis S yl Method Either P or
Not P
Quant Switch Method
x
(x Y therefore y x (x V L
one ously changes the order quant ifiers leading to an invalid conclusion
III Trans Method (S erroneous to infer all are S from all S are P.. Similarly. from x S A Px it is erroneous to infer x ( Px A S x.). ). Err one ously converts the terms the proposition. leading to an Invalid conclusion
Px ) therefore,
Px S x ). It is
Inc orr Inf Method From x S A Px infer x ( A Px erroneous to
from x (Sx Px infer (Sx Px). infer some are P from some are P and vice versa An made about propositions with existential quant ifiers
invalid inference Inv Sub Error Method K (x y) indicates that individual knows that y is true R (x, z)' indicates that has a relationship with Sub Error (x y. z) indicates a substitution error when incorrectly applying knowledge or attributes about to z
Let Clause Shift Method When the structure a statement incorrectly adjusted or interpreted, causing the original intent logical relationship be misrepresented For example, shift in the structure of let clause leads to invalid inference
Question
Considering the domain individuals natural numbers and representing the 'less than" relationship. x yR(x, y) states that for any natural number, you can find another natural number greater than it, meaning there no largest natural number. However, y xR(x y) suggests that there is a natural number greater than any other natural number, implying the existence of a largest natural number. Here, the premise is true, but the conclusion is false,

---

making the reasoning invalid.
What type of formal fallacy is this?
A. NegAnt Method
AffCons Method
CondSwap Method
IncorrNeg Method
E. DisjSyl Method
F. QuantSwitch Method
G. IliTrans Method
H. Incorrinf Method
I. InvSubError Method
J. LetClauseShift Method
Please give your answer in the format [IA/B/C/D/EFF//HHIIIJII
Answer
[[]]]
Response
[[F]]

---

PUZZLE ATTENTION CASE ANALYSIS
Rule
.Te
game starts with a word and specifies an ending word
2
Only one letter can be changed at a time and each Intermediate step must be 8 valid word
Change from the start word the word by few est steps
4 The question give the start end words Lanswr the minimum number of steps needed change from the start word to the end word
Question
From "HEAD" to "TALE". Output number in double brackets. example, takes steps from the start word the end word, present the answer as [[]]].
Answer
[[5]]
Response
To solve this puzzle, need to change the word "HEAD' to "TALE" by changing letter at time, ensuring each intermediate step valid word. Let's break down the steps:
**HEAD** **HEAL** (change to
2. **HEAL** *TEAL*** (change to
3. *TEAL** *TALE** (change E L)
So, the minimum number of steps required to change 'HEAD' to "TALE" is 3.
Therefore, the answer is: [[]]

---

COUNTERFACTUAL ATTENTION CASE ANALYSIS
Rule
The Om nid irectional Mobility Gear in Attack on Titan allows soldiers to move quickly and navigate through complex environments especially useful three -dimensional space
for travers ing large distances In Attack on Titan large walls protect humanity from the threat Titans serving as the primary defense mechanism for cities In Attack Titan the Fall of Wall Maria
significant event that led to massive loss of life and territory drastically altering the course of human history within the series
In Attack on Titan signal ares are used for long -distance communication, especially during exped itions and battles where other forms communication are not feasible Ice burst Stone unique resource Attack on Titan known its extreme .oldd making it valuable for preservation in the absence of modern refriger ation In Attack on Titan the presence of Titan Shift ers on the battlefield can significantly sway the outcome due to their immense power and strategic value
Attack on Titan the Royal Government holds substantial power and influence over the society 's political decisions and direction Survey Corps in Attack on Titan specializes exploring and mapping charted territories beyond the safety of the walls often facing the risk of Titan encounters
The Military Police in Attack on Titan are responsible for maintaining order within the walls overseeing internal security and enforcement of laws
Attack on Titan " the ossal Titan sheer size and power make a game -changing factor warfare capable of causing massive destruction and aying the outcome of battles
The design of cities in Attack Titan includes multiple layers of massive walls providing essential defense against Titan attacks and ensuring the safety of urban areas
In Attack on Titan " Histor ia Re iss connection to the Found ing Titan and her royal blood line allows her to access a wealth of historical and cultural memories.. ensuring their preservation
Attack on Titan reloc ating refugees inside various Walls Maria Rose and ina is critical strategy for managing large populations displaced by Titan attacks
In Attack on Titan " the Cad et Corps provides comprehensive military training to young recruits preparing them for the challenges they will face as soldiers..

---

In Attack on Titan " understanding Titan biology including their weaknesses
and behaviors crucial for developing effective strategies to combat them ensure long -term survival
AAttack on Titan choosing location far from areas prone Titan attacks is critical consideration for safety and survival effective means communicate imminent threats.. especially Titan sightings. to large number of people quickly
In Attack Titan " flare signals used
In Attack on Titan " significant historical events and knowledge are preserved through the memory inheritance of the Re iss family a unique method of historical record keeping their world In Attack Titan the uncover ing of Gr isha ager 's secrets the basement marks the beginning of new about the world and changes the course of the story
it reveals the truth
In Attack on Titan ." the unique power of mind -w iping possessed by the Found ing Titan has led to the forgotten history of entire segments of the population er asing crucial parts their past
Question
What is the most reliable method for preserving food for long periods?
Options: A. Refrigeration##s. Heating##C. lcaburst Stone##D. Buried in the soil
Provide your final answer single uppercase letter representing the option (A, B, C, D) and wrap in double square brackets, like this: [[A]].
Answer
[[C]]
Response
[[C]]

---

Published as a conference paper at ICLR 2025
I
ABLATION STUDY ON DATASET SIZE
Modklt
Out Modes
Eree
204
15%
220 190%
20%
19%
600
1005
Loge
866
39%
966
Proportion
Proportion
Mardh chet Modes
Wieer Mokes
ADuuDO Sroe Ei1 CorPicse
Sroe Gie CHMPPI SA4
205
396
47%
50%
96%
X0
80
596
186
205
30C
496
50%
58%
75%
80%
305
10t
Figure 13: Ablation study on dataset size. The x-axis represents the proportion of the subset size
relative to the entire dataset, while the y-axis in the three rows represents (from top to bottom): the mean error of model scores for the subset compared the full dataset, the standard deviation of the error, and the Gini coefficient of model scores for the subset. The left column employs a rule-based sampling strategy, selecting specified proportion of questions from 10 questions under each rule to maintain the dataset's original diversity. The right column uses category- based sampling strategy, randomly selecting a specified proportion from all 250 questions each category mitigate differences in difficulty across rules. The "ALL Models" curve reflects metric changes for all models in Table 2, while other curves correspond to metrics for subsets of models categorized from the full The results show that, regardless of the sampling strategy, both the mean and standard deviation of errors remain small, stabilizing at around 2 once the dataset propor- tion reaches 20% the full set. Similarly, the Gini coefficient exhibits minimal fluctuation, with maximum variation of approximately 0.02.
We conduct an ablation study on dataset size, which proves that our evaluation exhibits strong ro- bustness variations in dataset size, shown in Figure 13.
Three key metrics:
Accuracy Error Average: This metric calculates the mean the differences in accuracy between the model's 's performance the full dataset and its performance various subsets. It provides an overall measure of how much accuracy is lost when the model is evaluated smaller portions of the dataset.
97

---

Published as a conference paper at ICLR 2025
Accuracy Error Standard Deviation: This measures the variability of accuracy enrors, capturing how consistent the model's performance across different subset sizes. A smaller standard deviation indicates more stable performance, while a larger one suggests greater inconsistency.
The Gini coefficient used to assess the dispersion of model scores within each subset. It represents the inequality of the score distribution, reflecting how dif- ferentiated or homogeneous the model's performance is across different subsets. A higher Gini coefficient indicates more uneven performance, while lower value suggests more evenly distributed performance across subsets. The calculation formula is:
Gini Coefficient:
j=1 1 Msl
G
2n
where
y 1311 32 .... ynl
is the model score.
Two Sampling Strategies:
Rule-Based Sampling(Left Column): The left column of the figure corresponds to a rule- based sampling strategy, where specified proportion of questions is sampled from 10 questions under each rule to maintain the dataset's s diversity.
Category-Basee Sampling Column): The right column represents a category- based sampling strategy, where specified proportion of questions is randomly selected from all 250 questions in each category to reduce the impact of rule-specific difficulty dif- ferences on model scores, the difficulty of questions within the same rule also vary.
The results show that model score differences are minimally affected by dataset size. The mean and standard deviation of errors remain around and the Gini coefficient exhibits maximum variation of approximately 0.02. This indicates that KOR-Bende is highly robust to variations in dataset size. This robustness is consistent across models, regardless of their size or whether they are fine-tuned. However, we still emphasize the importance of continuously increasing the diversity the dataset.
Result Analysis:
98

---

Published as a conference paper at ICLR 2025
J CORRELATION ANALYSIS WITH OTHER BENCHMARKS
200
KoR(Overall)
100.0
88.2
83.8
85 .4
86.5
76.9
86.9
94.3
85.5
88.6
89.6
89.2
(Oporraat
100.0
55.0
65.7
64.1
63.3
79.7
80.1
75.6
82.9
76.7
KoR(Logic)
83.8
55.0
100.0
78.9
83.3
58.9
92.2
88.7
72.0
83.3
67.5
74.8
KOR(CIpher)
85.4
65.7
78.9
100.0
94.3
48.1
76.5
81.1
62.1
76.7
70.2
65.7
KOR(Puzale)
KOR[Counterfactual)
76.9
58.6
58.9
48.1
54.0
100.0
67.6
66.8
67.3
62.5
75.9
82.9
MMLU
86.9
63.3
92.2
76.5
79.7
67.6
100.0
94.6
86.4
89.0
79.4
88.5
79.7
88.7
92.9
GSMBK
85.5
80.1
72.0
62.1
66.1
67.3
86.4
92.6
100.0
90.7
89.5
93.0
MATH
88.6
75.6
83.3
76.7
80.6
62.5
89.0
92.9
90.7
100.0
86.2
86.7
HumanEvaf-
89.6
82.9
67.5
70.2
75.5
79.4
89.3
89.5
86.2
100.0
87.5
MBPP
89.2
76.7
74.8
65.7
68.2
82.9
88.5
90.6
93.0
86.7
87.5
100.0
KoR(Overall)
KOR(Operation)
KOR(Logic)
kOR(Cipher)
ROR(Puzzie)
coR(Counterfaee (aluuuaattr
MMLU
MMLU-Pro
GSMAK
MATH
HumanE val
MBpp
Figure 14: Heatmap of KOR-Bench correlations with various benchmarks. The heatmap shows the correlation of KOR- -OR-Bench with MMLU, MMLU-Pro, GSM8K, MATH, HumanEval, and MBPP, using 21 models of varying sizes. The results indicate that KOR-Bench is most closely correlated with reasoning- focused benchmarks, particularly MMLU-Pro, which emphasizes logical reasoning over prior knowledge. The difference in the correlation between KOR-Bench and these two bench- marks aligns with KOR-Bench's stated focus emphasizing logical reasoning while minimizing reliance on prior knowledge. This demonstrates KOR-Bench's alignment with reasoning-oriented benchmarks, further validating its design focus.
We further calculate the correlations between KOR-Bench and several benchmarks, including MMLU, MMLU-Pro, GSM8K, MATH, HumanEval, and MBPP. The correlations measure the simi- larity in score distributions of various models across different benchmarks. We calculate these using 21 models of varying sizes, including GPT4o, Slaude-3.5-Somee and the Qwen, Llama, Yi, Mistral, Phi, and Gemma series.
The results, shown in the Figure 14, indicate that KOR-Bench more closely related reasoning- focused benchmarks and shows the highest correlation with MMLU-Pro, the newest and most chal- lenging version of MMLU. The main difference between MMLU-Pro and MMLU that MMLU- Pro places a greater emphasis on reasoning ability. It includes large number of computational problems that require strong logical reasoning to solve.
The difference in the correlation between KOR-Bench and these benchmarks aligns with KOR- Bench's 's stated focus on emphasizing logical reasoning while minimizing reliance on prior knowl- edge. This demonstrates KOR-Bench's alignment with reasoning-orientee benchmarks, further val- idating its design focus.
99

---

Published as a conference paper at ICLR 2025
K ZERO-SHOT AND THREE-SHOP "ONLY QUESTIONS" EXPERIMENTS
Model
Size
Open
Overall
Operation
Logic
Cipher
Puzzle
Counterfactual
Gpt-4o
12.56
12.80
27.20
0.80
12.80
9.20(81.60)
Qwen2.5-77B-Instruu
72.7B
12.40
32.80
0.40
7.20
6.80(84.40)
Claude=33..Soonet
8
11.04
10.40
27.20
0.00
8.40
9.20(80.40)
Meta-LLama--3.
3.1-303----amm III-Instruct
70B
10.80
12,00
26.80
0.40
3.60
11,20(76.00)
Qwen2.5-32BB Instrucctt
32B
10.72
11.60
27.20
0.80
6.00
8.00(82.00) (0))
DeepSeck-V2.
236B
10.48
12.00
24.40
0.80
4.40
10.80
Yi-Large
10.32
10,80
28.40
0.40
5,60
6.40(81.60)
Large-Instrutt2247
123B
10.24
8.00
25.20
0.80
8.40
8, 8.80(80.40)
Owee2..5-7B--msstee
7.61B
25.60
9.20(78.00)
Owen2-72B-Instuun
72.71B
8,96
8.80
23.60
0.00
4.40
8.00181.60)
Owwn2-7H-oo Instruct
7.07H
8.16
7.20
20.80
0.40
2.40
10.00(73.60)
Meta-Llama-3. -.8-sttnttttt
8B
7.60
5.60
19.20
0.00
1.60
11.60(72.00
C4ai- Command-RR ---------22--
32B
7.28
15.60
13,20(70.80
Maii-Command-ee Plus-08-2024
104B
6,88
4.00
17.20
0.40
0.80
12.00(66.40)
Yi-1.5 Vi-1.5-9B- 1.5-9--hht
9B
6.08
6.80
10.40
0.00
2.40
10. (0(800)1.20
Mistral-7B 1--d7----sntttt--.....
7B
4.48
2.40
8.00
0.80
11.20 11.20(70.80)
Table 14: Model Performance on KOR-Bench Zero-Shor Setting with Only Questions.
Model
Size
Open
Overall
Operation
Logic
Cipher
Puzzle
Counterfactual
Gpt-4o
29.92
24.80
43.20
5.20
16.00
60.40( (9)60)
1aen2..-728-Instreee
72.78
25.44
32.80
47.20
4,00
8.80
34,40(54.80)
Qwen2.5-32B--nsstte
32B
24.48
29.20
43.60
4.40
7.60
37.60(43.60) 60)
Mistral-Large-Instuee
Instruc-24000
123B
22.48
36.80
11.60
43.20(30.80)
Qwen2-72B-Instme
72.718
21.92
24.40
44.00
6.00
7.60
27.60K61.2 (0).00
Yi-Large
X
21.12
14.40
32.80
3.20
8.40
46.80(21.66 (0)
Meta-Llama-3.--70B-nnmmmm
70B
20.08
12.40
33.60
1.20
8.00
45, (2220022200)
DecpSeck-V2.5
236B
19.12
16.40
41.60
2.40
8.80
26.40(53.22
Claude-3,5-Sonet
18.64
13.20
22.00
3,20
15.20
39,60 (2.60028.00)
C4ui-Command- Siw--oommnd--O--220 --00--2024
32B
15.36
12,00
27.60
2.40
3.20
31 61(6048844)
C4ai- Saii-Command-Be P-P2---08--22222
1041
26.40
27.60(54.80)
Qwen2.5-7B 1eoe2.55-B-iie Instrructt
7.61B
14.64
17.20
30.40
3,60
2.40
19.60(64.00)
Qwen2-7B Instructttt
7.078
14.48
14,80
30.80
2,80
3.20
20.80(66.88 60)
Yi-1.5-9B- Chat
9B
14.08
15.20
26.40
2.80
3.60
22.40(56.80)
Mistral-7B. Misstal-7B-Instuutt-v.... Instruct-vOO.
7B
11.44
25.60
18.80
Micta-Laama-,11411 3.1--BIIsssI
8B
10.88
2.80
13.60
0.80
0.00
37.20(21.20)
Table 15: Model Performance KOR-Bench in Three-Shot Setting with Only Questions.
We conduct zero-shot and three-shot "only questions" experiments, without explicit rules, to assess the model's ability recognize patterns and extract abstract reasoning rules.
Experimental Setup:
Zero-Shot Setting: Present the model with problem without explicit rules or guidance. The model relies solely its prior knowledge and reasoning to generate an answer.
Three-Shot Setting: Provide the model with three examples and their answers. The model infers a pattern or rule from these, then applies it to solve new problem.
Result Analysis:
In the zero-shot setting: In this setup, models struggle to answer questions accurately because the information provided in the questions, combined with their prior knowledge, is insufficient.
In the three-shot setting: When given three examples, models can infer patterns and solve some problems correctly. The success depends the task and the relatedness of the examples. For tasks like Counterfactual, Logic, and Operation, where examples are closely connected, models uncover rules and apply them effectively, resulting higher scores. However, for tasks like Cipher and Puzzle, which involve more abstract rules and weaker correlations between examples, models struggle deduce the rules, leading to lower scores.
100


Reviews:
[{"review_id": "O0W1oTbaOP", "reviewer": "Reviewer_D3fb", "summary": "This paper presents KOR-Bench, a new benchmark that tests LLMs' reasoning abilities across five categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. The key innovation in the design of the benchmark is the concept \"knowledge orthogonality\", which refers to the property of problems requiring minimal dependence on the domain-specific knowledge the model is exposed to during training. The authors maintain that their knowledge orthogonal tasks can examine how LLMs reason about new rules, concepts, and situations, thus constituting a good testbed for the out-of-distribution reasoning abilities of LLMs. A large number of LLMs, including both chat and base models are evaluated on this benchmark. While SoTA models such as Claude 3.5 and GPT-4o perform the best, all models have large room for improvement. In particular, the cipher and puzzle categories are especially challenging for LLMs based on the evaluation results. The authors also carried out fine-grained and targeted analysis to better understand model behaviors and error patterns. Overall, the experimental results contribute to diagnosing current model limitations, and the benchmark may help assess reasoning capabilities of future models.", "questions": "See the \"Weaknesses\" section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation of the paper is generally clear and thorough. It provides good examples of the categories and problems, reasonable details of the benchmark statistics and creation process, and informative tables and figures.\n\n- Benchmarking reasoning in reliable ways is an important topic and is timely, which this work targets.\n\n- The proposed benchmark is fairly diverse with respect to reasoning. Some of the five categories are related, but they together cover a wide range of reasoning tasks.\n\n- The authors conducted comprehensive evaluations across many models. I appreciate the effort of running models of different sizes and different kinds (open vs closed, chat vs base). The results suggest that most of the benchmark is challenging for most of the models.", "weaknesses": "- The biggest weakness of the current version of the paper is that the notion of \"knowledge orthogonality\" is not well articulated enough. There are two aspects. One is about what the notion means. The authors write it \"means that the rules within the\nbenchmark are independent of the domain-specific knowledge the model is exposed to during pre-training\" (Lines 76-77). What do you mean by \"independent\" here? Does it just mean \"likely absent\"? Could you clarify this central notion more in-depth? The other aspect is whether the five categories satisfy knowledge orthogonality. For example, one could argue that the Puzzle example in Figure 2 is basically the minesweeper game, which is a famous and popular game LLMs know a lot about. I am wondering how you would argue that is \"knowledge orthogonal\". Moreover, in the Counterfactual category, most of the cover stories are culturally well-known (e.g., Pokemon and Avengers). In what sense are they \"independent\" of pre-training data? \n\n- Relatedly, in Section 3.2 the authors mention that the raw rules/problems taken from other sources are \"adjusted/refined/modified/etc.\" for this benchmark. How was that process done? What did the authors do to ensure that the end results \"meet the specific challenges of KOR-Bench\"? It would be good to see more explanations, since this is important given the nature of the proposed benchmark.\n\n- For the evaluation, in terms of prompting strategies this work only studies zero-shot and few-shot settings. We know that prompting matters for LLM reasoning, and there are many methods that can improve reasoning performance (e.g., most notably chain-of-thought/CoT). It would have been a stronger paper if the authors had evaluated the problems with at least one more sophisticated method. Or, if CoT sometimes/often happens automatically and the authors intend that to the case, it should be discussed explicitly in the paper.\n\n- The size of the benchmark is not particularly large, as the authors note. It is still good, and the authors indicate the plan to expand it.\n\n- I think the writing of Section 6 can be improved, especially 6.2 and 6.3. \"Self-correction\" is introduced rather abruptly with minimal setup/context, and I am not sure that I fully understand what \"complex task processing\" is about. I recommend that the authors elaborate on those analyses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g0n2toie9w", "reviewer": "Reviewer_MyDX", "summary": "This paper introduces the concept of Knowledge-Orthogonal Reasoning (KOR), aiming to reduce the interference of existing pre-trained knowledge by introducing new rules that are orthogonal to it, thus allowing for a more precise assessment of a model's intrinsic reasoning capabilities. Based on this concept, the paper proposes the Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench), which covers five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. The benchmark is designed to evaluate how well models can apply new rule descriptions to solve novel, rule-driven problems. The study reveals that even state-of-the-art models, such as Claude-3.5-Sonnet and GPT-4o, perform far from optimally on KOR-Bench, achieving accuracies of 58.96% and 58.00%, respectively. The paper also conducts in-depth analyses using methods like Stepwise Prompting to identify performance bottlenecks in specific tasks, such as the Cipher task, and explores the impact of self-correction techniques and tricks on puzzle-solving, along with visualizing rule-focused attention. These contributions provide new perspectives and tools for assessing and enhancing models' reasoning abilities, fostering further research in the field.", "questions": "1. Can the authors provide more experimental evidence or theoretical proof to ensure the fairness and objectivity of the model evaluation?\n2. How does the author intend to implement the plan proposed in line 539, and briefly explain how the results of this paper can be extended to multimodality?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a new method for testing the reasoning capabilities of large models and conducts extensive experiments and tests using both open-source and proprietary large models across multiple aspects. During the assessment, this work reduces the impact of pre-training knowledge. Most existing benchmarks, such as MMLU, GPQA, and CommonsenseQA, primarily evaluate a model's ability to accumulate and recall data, often making it difficult to distinguish whether the model is performing genuine reasoning or simply recalling learned patterns. KOR-Bench reduces this dependency by introducing new rules independent of existing pre-trained knowledge, thereby more accurately assessing the model's intrinsic reasoning and planning capabilities.\n2. This assessment is more comprehensive and challenging. KOR-Bench covers a variety of reasoning scenarios and task types, including operations, logic, cryptography, puzzles, and counterfactual reasoning across five categories. These tasks are designed to be novel and challenging for models, testing their ability to handle unseen rules and frameworks. Even the most advanced models achieve relatively low accuracy rates (e.g., Claude-3.5-Sonnet scores 58.96%, and GPT-4o scores 58.00%), demonstrating its significant challenge and discriminative power.\n3. The authors provide a comprehensive and detailed introduction to the code and data through appendices and open-source releases.\n4. The tables and graphs in this article are clear and beautiful.", "weaknesses": "1. There are clerical errors in some parts of the article, such as the model name \"Clause-3.5-Sonnet\" on lines 308 and 309, and it is critical to be rigorous when doing scientific research.\n2. This paper lacks theoretical and experimental evidence on how to ensure the objectivity and impartiality of the assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KHhLsHVJbx", "reviewer": "Reviewer_4jdU", "summary": "The paper introduces KOR-Bench, a benchmark specifically designed to evaluate the reasoning and problem-solving abilities of LLMs independent of their pre-existing, pretrained knowledge. This benchmark employs a “Knowledge-Orthogonal Reasoning” approach to assess LLMs across five challenging task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual Reasoning. Each task type introduces new rules and problem-solving frameworks that are orthogonal to domain-specific knowledge, focusing on the model’s ability to reason based on novel rules rather than relying on memorized patterns.", "questions": "I have no further questions; the analysis in the main body and appendix is highly comprehensive.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The introduction of knowledge orthogonality is an innovative approach to eliminate biases from pre-existing knowledge, offering a fresh perspective on evaluating true reasoning capabilities in LLMs.\n2. By incorporating diverse tasks such as logical puzzles, cryptographic challenges, and hypothetical scenarios, KOR-Bench provides a robust testbed for examining LLMs’ adaptability to unfamiliar rules.\n3. This paper conducts comprehensive experiments and provides a thorough analysis of model errors, identifying specific challenges within task sub-steps. This analysis offers valuable insights into model limitations and potential areas for improvement.", "weaknesses": "1.\tThere are already existing benchmarks for evaluating the similar capabilities of current LLMs. For example, as mentioned in the related work, we have logic games [1] for logical reasoning and datasets like [2] for counterfactual reasoning. If the authors argue that these datasets might have been included in current model training data, it would be helpful to explain how KOR-Bench avoids this, such as by concealing the test set or employing other safeguards.\n2.\tThe data scale of KOR-Bench may be a concern, as the current version includes only a small dataset for each task category, potentially limiting the robustness and generalizability of the benchmark findings.\n\n****References****\n\n[1] Gui J, Liu Y, Cheng J, et al. Logicgame: Benchmarking rule-based reasoning abilities of large language models[J]. arXiv preprint arXiv:2408.15778, 2024.\n\n[2] Wu Z, Qiu L, Ross A, et al. Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024: 1819-1862.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kYJIJKS7QZ", "reviewer": "Reviewer_YDMq", "summary": "The paper introduces the concept of Knowledge-Orthogonal Reasoning (KOR), which focuses on evaluating a model's intrinsic reasoning and planning abilities by minimizing reliance on pre-existing knowledge. This is achieved by introducing new rules that are independent of the knowledge the model was exposed to during pretraining. The authors propose the Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench), which includes five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench challenges models by introducing new elements and rules, pushing them beyond traditional reasoning frameworks.", "questions": "See questions in Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Important Direction**: The paper addresses a crucial aspect of evaluation by focusing on how to avoid shortcuts when constructing reasoning benchmarks, which is vital for accurately assessing model capabilities.\n\n2. **Comprehensive Scope**: It involves a substantial workload, providing a thorough examination across various dimensions, ensuring a well-rounded evaluation of model reasoning abilities.\n\n3. **Detailed Presentation**: The benchmark is presented with detailed statistics and information, offering clear insights into its structure and the performance of different models.", "weaknesses": "1. **Ensuring Orthogonality**: There is a lack of clarity on how the constructed problems are guaranteed to be orthogonal to the knowledge in LLMs. The paper does not provide evaluation results to support this claim.\n\n2. **Evaluation of Base Models**: The purpose of evaluating base models alongside chat models is unclear. The distinct advantages of base models over chat models are not well articulated.\n\n3. **Lack of Benchmark Comparison**: The study does not compare results with other benchmarks, which makes it difficult to highlight the unique contributions of KOR-Bench.\n\n4. **Benchmark Construction Process**: The process of constructing the benchmark is unclear, particularly whether it was done manually or with LLM assistance.\n\n5. **Missing Related Work**: The paper does not reference a relevant study https://arxiv.org/abs/2306.09479# that proposes a similar concept, which could help in emphasizing the novelty of this work.\n\nMinor comments:\n1. **Section 5 Clarity**: The section is somewhat disorganized. The \"Reasoning Process Performance\" paragraph lacks insights and supportive data, while \"Single Task Analysis\" is presented without deeper discussion.\n\n2. **Conclusion Content**: The conclusion includes too much focus on future plans, which might be better placed elsewhere in the paper.\n\n3. **Appendix Content**: The appendix contains irrelevant materials, which should be organized more effectively.\n\n5. **Few-Shot Testing**: It would be interesting to test if LLMs can solve problems based on few-shot examples without explicit rules, to assess their ability to derive abstract rules.\n\n6. **Knowledge Orthogonality vs. Contradiction**: The concept of knowledge orthogonality is confused with knowledge contradiction, especially in counterfactual questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)