Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
Authors: Kai Xiong
Xiao Ding
Ting Liu
Bing Qin
Dongliang Xu
Qing Yang
Hongtao Liu
Yixin Cao
Venue: NeurIPS 2024 poster (2024)
Keywords: Abstract Reasoning
Large Language Models
Question Answering
Primary area: natural_language_processing
TL;DR: Large language models (LLMs) struggle with abstract reasoning. We created a specialized dataset and learning method to improve their abstract reasoning skills, achieving more than simple memorization.

Abstract: Large language models (LLMs) have developed impressive performance and strong explainability across various reasoning scenarios, marking a significant stride towards mimicking human-like intelligence. Despite this, when tasked with several simple questions supported by a generic fact, LLMs often struggle to abstract and apply the generic fact to provide consistent and precise answers, revealing a deficiency in abstract reasoning abilities. This has sparked a vigorous debate about whether LLMs are genuinely reasoning or merely memorizing. In light of this, we design a preliminary study to quantify and delve into the abstract reasoning abilities of existing LLMs. Our findings reveal a substantial discrepancy between their general reasoning and abstract reasoning performances. To relieve this problem, we tailor an abstract reasoning dataset (AbsR) together with a meaningful learning paradigm to teach LLMs how to leverage generic facts for reasoning purposes. The results show that our approach not only boosts the general reasoning performance of LLMs but also makes considerable strides towards their capacity for abstract reasoning, moving beyond simple memorization or imitation to a more nuanced understanding and application of generic facts. The code is available at https://github.com/Waste-Wood/MeanLearn.

Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
Kai Xiong
Xiao Ding Ting Liu Bing Qin
Dongliang Xu
Qing Yang* Hongtao Liu
Yixin Cao
Research Center for Social Computing and Information Retrieval
Harhin Institute of Technology, Harbin, China
Fudan University, Shanghai, China
Du Xiaoman (Beijing) Science Technology Co., Ltd., Beijing, China (kriong, xding, tliu, qimb}-ir www.tuu...u......... cn (xudongliang, yangqing, liuhongtac01 @duxiiamman com yıcaoθfudan. edu. cn
Abstract
Large language models (LLMs) have developed impressive performance and strong explainability across various reasoning scenarios, marking significant stride wards mimicking human-like intelligence. Despite this, when tasked with several simple questions supported by generic fact, LLMs often struggle abstract and apply the generic fact to provide consistent and precise answers, revealing a deficiency in abstract reasoning abilities. This has sparked vigorous debate about whether LLMs are genuinely reasoning or merely memorizing. In light of this, we design preliminary study to quantify and delve into the abstract reasoning abilities of existing LLMs. Our findings reveal substantial discrepancy between their general reasoning and abstract reasoning performances. To relieve this problem, we tailor an abstract reasoning dataset (AbsR) together with meaningful learning paradigm to teach LLMs how to leverage generic facts for reasoning purposes. The results show that our approach not only boosts the general reasoning perfor- mance of L.MSs but also makes considerable strides towards their capacity for abstract reasoning, moving beyond simple memorization imitation more nuanced understanding and application generic facts. The code is available at https://gitiiei http://tthub..c//atteteeeee comm/kaste-oood/Maalll..o Wood/Meannearn
Introduction
Humans proceed with meaningful learning to induce common patterns or high-level abstractions to acquire abstract reasoning abilities (2 37 Such capabilities allow us apply broad principles across diverse situations, demonstrating deep understanding and versatile application of knowledge. As shown Figure (a). humans can deduce "rock dissolved" and "the skin suffers pain" when given "adding rock into hydrochloric acid" and "acid touches human skin",, respectively. This stems from the established expertise in the human mind (the generic fact "acid corrosive"), and extends to applications in different scenarios. Of late, remarkable headways of L.MSs have pushed AI much further towards human- like intelligence 22 47 Interestingly, can L.LMs act like humans to instinctively consider the generic fact for flexible applications in different scenarios?
In our pilot investigation, observed that LLMs appear to lack satisfying capabilities in abstract betwenn vanilla
reasoning, As illustrated in Table there is notable discrepancy-exceede
"Conresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table I: The vanilla accuracy and abstract rea- soning accuracy (AbsAcc) on e-CARE (14) We will formally define AbsÅcc in Sec. 2.1
Generie Fact: Acid is corrosive
Questionl: :: Adding rock into hydrochloric acid
Rock dissolved
Rock corroded
Methods
Size
Vanilla Accuracy AbsÅcc
Question2: Acid touches human skin
7B
51.67
24.82
LLaMA-2
The skin suffers pain
Skin moisturized
13B
68.45
42.94
(a) Humans conduct reasoning
(b) LLMs conduct reasoning
7B
74.23
51.19
Orca-2
35
13B
79.83
59.47
The responses of (a) Humans and (b) LLMs when facing two questions which are sup- ported by the same generic fact.
Figure 1:
GPT-3355
>20B
83.25
66.08
Human 14
92.00
89.90
accuracy and abstract reasoning accuracy (AbsAcc) for all LLMs, while humans show only small disparity. Despite the extensive pre-training LLMs, which equips them with a vast repository of generic facts, they seem unable to leverage this information as flexibly as humans do (Figure I (b)).
In this paper, we aim to systematically and quantitatively investigate the abstract reasoning of LLMs. We first formally define the abstract reasoning metric. Then we carry out abstract reasoning and knowledge probing tasks for a preliminary study. Our investigation comprises two main components: the reasoning tasks can quantify the abstract reasoning capabilities of LLMs. The generic fact probing task can give further analysis from the perspective of generic fact mastery. Hereafter, we improve abstract reasoning in LLMs based on the analysis of the preliminary study. Unlike existing chain-of-thought (CoT) methods 52 277. which conduct reasoning with uncontrolled step-by-step explanation, we create a dataset AbsR tailored for abstract reasoning. AbsR not only includes generic facts but also their guided explanations, offering coached approach to understand the process of reasoning with common patterns. Finally, to make LLMs subconsciously exploit the generic fact like humans, we design a simple but effective learning paradigm called meaningful learning (MeanLearn) to simulate the implicit knowledge learning process [3]. enabling LLMs to implicitly learn and utilize generic facts for reasoning without requiring generic facts as input.
We evaluate MeanLearn on six -it-of--ommin (OOD) reasoning and language understanding bench- marks. Experimental results demonstrate that MeanLearn not only improves general reasoning of LLMs but also excels in abstract reasoning. This distinction in performance is particularly pronounced in the realm of abstract reasoning, underscoring the unique efficacy of our approach in nurturing the higher-order thinking skills that are essential for sophisticated cognitive processing in artificial intelligence systems. Further analysis and ablation studies yield additional evidence supporting the effectiveness of our approach. We summarize our contributions as follows:
We provide systematic and quantitative analysis of abstract reasoning in current LLMs, and we develop an abstract reasoning dataset with generic-fact-guieee explanations.
We propose meaningful learning to improve abstract reasoning in LLMs with generic fact.
We achieve significant improvement in general reasoning and abstract reasoning various OOD reasoning and language understanding benchmarks.
2 Abstract reasoning study
In this section, we initially define the evaluation metric of abstract reasoning to establish quantifiable standards. Then introduce the LLMSs used in this study, Finally, we conduct experiments: one to directly estimate the abstract reasoning abilities of LLMSs, and the other to further analyze from the perspective of general fact mastery.
2.1 Abstract reasoning metric
Abstract reasoning requires models to apply general patterns of high-level abstractions to different scenarios or questions. Different from general reasoning. which focuses each single example, abstract reasoning takes the examples supported by a generic fact as whole. We define abstract reasoning accuracy due to the lack of proper metric.

---

Table The categorized abstract reasoning accuracy based on whether the generic facts are known.
Table 2: Accuracy of generic fact probing.
LLaMA-2
Orca-2
GPT-3.5
LLaMA-2
Orca-2
GPT-3.5
LLMs
Category Metries
7B
13B
7B
13B
>20B
7B
13B
7B
13B
>20B
Accu. 62.44
70.32
26.97
12.78
77.30
Known
AbsÅcc 34.26 43.51
64.98
81.51
68.31
Unknown AbaAcc 17.19
31.84
49.00
58.38
58.49
Given dataset D with generic facts R = (21,72.- ,.. rny. for the i-th generic fact Tı, supports my examples S; D.
Generic Fact Fi
Generie Fact 12
Generie Fact r3.
Example s
Example sf
Example s
Label: A
Label: C
Label: D
Prediction:
Prediction:
Prediction: DO
As shown in Figure 2 akin to Qiu et al. 37 we suppose an LLM LM has grasped the generic fact if and only if LM can cor- rectly answer all examples in S.. Hence, we define abstract reasoning accuracy (denoted as AbsAcc) as the proportion of generic facts that LM has grasped:
Example s
Example s²
Example s
Label: B
Label: B
Label: C
Prediction:
Prediction:
Prediction:
Example s
Label: A
Vanilla : (1=1-0+0-001++))/ 0.57
Prediction:
AbsAcc: (0 - /) 0.33
AbsAcc = I((M,, (S.))
(1)
Figure 2: Computation of abstract reasoning metric.
n
where o quantifies the number of examples in S; that L.M answers correctly. II serves the indicator function, taking when the specified condition is met and 0 otherwise.
2.2 Large language models for evaluation
We employ several LLMs in our preliminary experiments: (1) LLaMA-2 47 is an open-access LLM. We use 7B and 13B LLaMA-2 for experiments; (2) Orea-2 (35] an open-access LLM finetuned on
LLaMA-2 with over 80K reasoning-specife examples. We use 7B and 13B Orca-2 for experiments. (3) GPT-3.5 7 is limited access LLM, we use gpt-3. ..- 10-555-tuoo000 0125 for experiments.
2.3 Experiment I: abstract reasoning
We formulate the first preliminary experiment to gain an initial insight into the abstract reasoning abilities of current LLMSs.
To achieve this goal, we choose a multiple-choice question- answering dataset e-CARE [14 for implementation. e-CARE is large-scale explainable causal reasoning dataset with generic fact in each example, and a generic fact can support more than one example. The full set (train, test, and dev) of e-CARE is selected for experiments. We filter out the examples that the corresponding generic facts support only one example. Finally, we obtain 13,045 examples supported by 5,608 generic facts.
We follow zero-shot setting prompt the LLMs for answer selection from multiple choices, and the generic facts are not provided in the inputs (refer to Appendix B.I.. Table Tableesshn shows overall results:
(1) The significant disparity (over 17%) between the vanilla accuracy and AbsÅcc for all LLMMs proves two aspects: on the one hand, vanilla accuracy is not ideal for estimating abstract reasoning. On the other hand, although LLMs could answer questions accurately, they still cannot capture the common patterns or generic facts behind the questions.
(2) Larger LLMs outperform smaller ones due to their richer knowledge and stronger reasoning ability. Orca-2, additional trained on LLaMA-2, notably outperforms LLaMA-2, indicating that supervised fine-tuning enhances both reasoning and abstract reasoning abilities. However, even with large-scale post-training, the significant gap between vanilla accuracy and AbsAcc remains unbridged.
(3) The minimal gap between vanilla accuracy and AbsAce for humans suggests the large gap for LLMs is not due to varying difficulties of the example supported by the same generic fact.

---

Generic Fact: Unusual conditions affect behavior
Question:
Which of following scenarios most likely to influence person's typical shopping habits?
Question: In psychological study, which condition
most likely to yield atypical results in participants?
Options:
Options:
A) A regular day with no significant changes weather
A) study conducted during stressful event
or social conditions
B) A study conducted in a controlled environment with
B) A holiday season with sales and promotions in stores
no distractions
C) When the person has not received their paycheck yet
study conducted with the same participants
D) day with consistent weather and social conditions
previous similar study
as the previous week
D) A study conducted with used questionnaires
Answer: B)
Answer: A)
Explanation: Holiday seasons with promotions and sales present unusual conditions that can significantly alter person's 'ss typical shopping behavior, often encouraging spending.
Explanation: A stressful event like natural disaster creates an unusual condition that can significantly affect participants behavior and responses, leading to atypical results in psychological study.
Figure 3: Two samples of AbsR guided by the generic fact about "unusual conditions".
2.4 Experiment II: generic fact probing
To delve into the abstract reasoning from the perspective of generic fact mastery, we design a generic fact probing task to ask LLMs whether they possess the given generic fact (refer to Appendix [B22for more details). Drawing inspiration from knowledge uncertainty estimation [3) we formulate this task as yes or no problem. The overall results are shown in Table 2
We could conclude as follows:
(1) LLaMA-2 and GPT-3.5 have substantial knowledge reserves. This indicates the poor Abs of LLaMA-2 might come from lack of ability to apply the generic fact to different reasoning scenarios. This problem also exists in GPT-3.5, but it is not as serious as LLaMA-2.
(2) In contrast, Orca-2 knows limited generic facts, and the larger the model, the less it knows. This contrasts sharply with its strong reasoning abilities (vanilla accuracy). We suppose training Orca-2 is more like rote learning rather than meaningful learning, which implies that Orca-2 tends to lose knowledge while acquiring reasoning.
To deeply investigate the relation between generic facts and abstract reasoning, we categorize the examples in e-CARE according whether LLMs know the generic facts. The results are represented in Table from which we can draw the following conclusions:
(1) The performance of ""nownn" category has superiority over "Unknown" category across all LLMs. This demonstrates knowing the generic facts is helpful for abstract reasoning, which provides a potential avenue for abstract reasoning enhancement through generic fact retrieval.
(2) Nonetheless, most of the results the "Unknown" category still have undeniable performance, suggesting that LLMs may engage in reasoning based on suspicious correlations or rely on memo- rization.
In conclusion, we can enhance the abstract reasoning abilities of LLMs in two ways: (1) Knowledge, injecting generic facts through additional training prompting; (2) Reasoning, teaching LLMs how to utilize the generic facts to provide more precise responses to questions.
3 AbsR: abstract reasoning dataset
To enhance the abstract reasoning abilities of L.LMs through Knowledge and Reasoning ways, we need a dataset that can not only provide generic facts but also teach LLMs how to exploit the generic fact to derive the correct answer in different scenarios (meaningful learning). Since there is no suitable dataset to assist in achieving the goals, we create an Abstract Reasoning dataset (AbsR).
3.1 Generic fact collection
To obtain the generic facts, We chose GeneriesKB 5 as the foundational generic fact base for AbsR. GeneriesKB is large-scale knowledge base with over 3.4 million sentences (c.g. Dogs bark)

---

expressing general truths, each of which also includes the corresponding concept (e.g. Dog) and the confidence score (between and 1). Hence, we filter and then sample collection of 4,613 high-quality generic facts of different concepts from GeneriesKB (details of generic fact filtering and sampling can refer Appendix
Table 4: The statistics of AbsR.
3.2 Dataset construction
Examples Questions Generic Facts
To construct the whole dataset, and inspired by previous works 49 [56), choose GPT- 4 12) as our data annotator. The API we used is 100-4-110-ppreiess
Train
18.020
9,010
4,613
Test
200
200
104
Specifically, for each sampled generic fact rr. we would ask GPT-4 to create samples S {......, ,,, & m, S 3) in various scenarios based on Ti. Each sample s) contains the fol- lowing terms: (I) question Xj with a few options, (2) response Yj with an answer and an explanation guided by r;. All terms form a triple =< Xj, rs. Yi > for each sample. The prompt for dataset creation can refer to Appendix D Figure shows illustration of the created samples.
Finally, without loss of generality, given the j-th sample of of generic fact rú, we can create two kinds of examples for meaningful learning. One is predicting Vj given < Xj, ri > (K-example), while the other is predicting Y given only X; (R-example). The examples can implicitly enhance abstract reasoning in LLMs through Knowledge and Reasoning ways. Table shows the statistics of AbsR.
3.3 The quality of AbsR
We conduct human evaluations to measure the quality of AbsR from the following dimensions (details and more statisties such as evaluation criteria, agreements, and pay can refer to Appendix B andd Fr:
Human Performance: humans can achieve vanilla accuracy of 95% and AbsAcc of 93.27%;
Support Rate: 89% (the rate of examples which can supported by the generic fact);
Diversity: 88.5% (the diversity of the examples supported by the same generic fact. Greater diversity indicates greater differences among the examples of the same generic fact).
For comparison, e-CARE 14] is also human-annotatee annotated, with examples generated from generie facts. On e-CARE, humans reached 92% vanilla accuracy, 89.9% AbsAcc, and an 87% support rate [14), showing that AbsR matches the quality of the aamannannotttedd dataset.
4 Method: meaningful learning
To imitate humans' instinctive use general facts in reasoning, we develop a simple but effective learning paradigm called meaningful learning (MeanLearn). It can enhance the abstract reasoning abilities of LLMs in Knowledge and Reasoning ways. This is inspired by the implicit knowledge learning process, which uses hidden variables to learn event background knowledge (13). MeanLearn can make LLMs implicitly learn generic facts and solve problems under the guidance of generic facts.
Specifically, LLM LM with parameters 0 and the input construct conditional probability for the output y:
L.M(E, 0) - Σ log Po (v Vee).
(2)
In Meanl earn, for K-example < X, r, Y and R-example < X,Y > pair guided by generic fact we send the example pair into LM to model two conditional probabilities:
Y,,,
LM(X, Y,0)
Σ log (Y,),,
(3)
LM(X, Y,0) =
Σ log (Y, Ye.).
Hereafter, on the one hand, we train LM to learn policies Po (V Yee) and go (V/)X, r, Yet) together to enable LM implicitly leam the generac fact On the other hand, LM can learn how
5

---

to apply r into different scenarios by learning the explanations in Y of different X supported by r. Finally, we can reason with the implicit guidance of when only given X, just like humans instinctively answer questions without explicitly giving the general facts.
Experiments
5.1 Experimental details
Training setup. We use LoRA [20] for parameter effcient finetuning. 7B and 13B MeanLearn are trained on 7B and 13B Orea-2, respectively, 8B MeanLearn is trained on 8B LLaMA-3, with batch sizes of 256 for 7B and 8B, and 240 for 13B. McanLearn of various sizes are trained for one epoch at 5e-5 learning rate with AdamW 26). 7B and 8B Meanl trained on NVIDIA A100 80GB PCIe GPUs for hours. 13B MearnLearn is trained on such GPUs for 4 hours.
Baselines. We adopt a wide range of open and limited access LLMs across different sizes as base- lines. Open Access LLMs: (1) LLaMA-2 [47] 7B and 13B; (2) LLaMA-3 (34 8B; (3) Vicuna [8]
7B and 13B (finetuned on GPT I data); (4) WizardLM (66) 7B and 13B (fintuned with evolved
instruction data); (5) Orca-2 (35] 7B and 13B (prograssively finetuned on massive reasoning data).
Limited Access LLMs: (1) GPT-3.5 (gpt-3. 5-t.--rrrroooooe.
(2) P-LM feexx--bisonn0111
Evaluation benchmarks. In addition AbsR, we also include a wide range of benchmarks of
reasoning and natural language understanding for evaluation: (1) AGIEval [60] consists of tests
ranging from college admission tests, to national civil service examinations; (2) RACE 28 consists
of tests with reading comprehension questions; (3) BBH 43 is subset of Big-Bench 42 which
contains 23 hardest tasks focusing on challenging scenarios; (4) Com. [55] is a collection of
commonsense reasoning datasets (oNLI [4), CSQA 44), COPA 339 e-CARE [4], SociallQa 40
PIQA (6), and StrategyQA [66); (5) MMLU [88] is a massive multitask language understanding
benchmark; (6) ARC 10
is a benchmark of easy (ARC-e) and challenge (ARC-c) science questions.
Evaluation setup. Since generation paased evaluation is time-consuming due to limited computing resources, we follow OpenCompass [[T] to have hybrid evaluation criteria with high reproducibility: (1) For PaLM-2 and GPT-3.5 on all tasks, and open access LLMs on 4 generation tasks (4 tasks in BBH), greedy decoding and exact match are utilized for evaluation; (2) For open access LLMss on classification and multiple-choice tasks, perplexity (PPL. adopted for prediction. The option or category with the lowest PPL is chosen as the answer. For all baselines, evaluation is conducted once since there is no randomness on such evaluation criteria. For MeanLearn, results are averaged of MeanLearn(s) trained with 3 different random seeds. The evaluation prompts can refer to Appendix H
5.2 Results: vanilla accuracy
Table 5 shows the Vanilla Accuracy of MeanLearn and baselines:
(1) Additional training usually brings improvements (Vicuna, WizardLM, Orca-2, and MeanLeam), but it still can result in performance losses on some benchmarks (c.g. AGIEval), This signifies the effectiveness of massive crasoning-speciffe data and the necessity of data coverage.
(2) MeanLeam has superiority over all open access baselines on nearly all benchmarks. This demonstrates that MeanLearn can use generic facts to effectively guide LLMs to reason more properly and logically, making them more flexible and task-adaptive.
(3) The more challenging the henchmark (e.g. BBH), the less pronounced the improvement in the performance of MeanLearn. This is mainly because the improvement is mostly constrained by the complex scenarios in harder benchmarks, which require mightier base LLMs or more complex training data to perform and learn multi-step and compositional reasoning.
(4) MeanLearn does not have many advantages over LLaMA-3. LLaMA-3 stores dense knowledge in its parameters, making it harder learn new knowledge without forgetting existing knowledge.
(5) Larger open- LLMs do not always yield better results. We suppose the main factors behind this are in two ways: on the one hand, LLMs with different parameter sizes might be good at different tasks, and excessive contemplation may lead to decreases on some tasks (53]. On the other hand, the crplexxty-bbsser evaluation setting might have moderate cross-scale generalization ability.

---

Table 5: Overall vanilla accuracy and AbsAcc (%) of baselines and MeanLearn. Due to the space limit, we only report the standard deviation of Average performance of each method.
Size
LLMs
AbsR
AGIEval
RACE
BBH
Com.
MMLU
ARC- ARC-e
Average
Vanilla Accuracy
PaLM-2
75.76
15.83
75.82
48.73
78.05
58.73
89.95
82.37
65.77 + 0.00
>20B
GPT-3.5
84.00
25.84
83.36
54.15
74.80
65.98
94.71
88.81
71.46 0.00
LLaMA-2
50.00
27.89
38.85
26.00
55.37
41.65
58.73
43.05
42.69 1: 0.00
Vicuna
75.00
32.56
65.75
31.93
64.77
49,40
74.78
57.63
56.48
WizardLM
65.00
22.27
22.11
25.99
43.63
23.26
25.57
22.37
31.28 0.00
Orca-2
73.50
38.57
73.32
34.51
71.58
50.11
79.19
73.90
61.84 = 0.00
MeanLearn
77.00
38.15
77.15
35.64
76.59
52.98
86.67
78.06
65.28 0.34
LLaMA-3 82.50
36.33
76.75
36.60
71.76
61.86
88.54
75.25
66.20 + 0.00
8B
MeanLearn
84.50
38.53
77.02
37.67
71.73
62.35
88.89
77.97
67.12 + 0.18
ILaMA-2
73.50
34.41
59.46
30.61
61.00
51.87
71.25
55.25
54.67 + 0.00
Vicuna
74.00
33.23
63.54
33.22
63.10
49.34
77.78
59.77
56.75 + 0.00
13B
WizardLM
74.00
33.23
63.54
33.22
63.10
49.34
77.78
59.77
56.75
Orca-2
66.50
43.75
66.86
39.39
65.92
47.27
82.19
70.85
60.34 0.00
McanLeam
67.17
43.92
70.24
40.36
69.75
51.68
88.69
82.00
64.23 = 0.25
AbsÅcc
PaLM-2
64.58
9.59
61.54
27.69
69.34
44.39
85.68
77.10
$4.99 + 0.00
>20B
GPT-3.5
77.08
14.48
66.79
25.45
65.74
52.09
92.16
85.05
59.86 0.00
LLaMA-2
35.42
16.09
14.08
9.56
47.42
25.48
50.27
35.51
29.23 0.00
Vicuna
58.33
19.90
38.68
14.28
58.07
32.50
65.95
49.53
39.84 0.00
WizardLM
53.24
12.94
5.57
9.77
32.96
11.55
20.00
17.29
20.42 0.00
Orca-2
60.42
26.27
50.90
15.67
64.02
33.68
72.16
67.76
48.85 88.86-22 + 0.00
Mcanl leam
64.58
25.27
57.10
16.67
71.17
37.24
81.35
73.83
53.39 0.32
LLaMA-3
72.92
23.78
55.61
18.13
66.35
47.31
83.51
70.09
54.71 + 0.00
8B
MeanLearn
73.96
26.57
55.09
19.24
66.48
46.76
83.78
73.36
55.66 + 0.17
LLaMA-2
61.46
21.01
32.17
13.74
52.87
35.72
62.43
44.86
40.53 + 0.00
Vicuna
61.46
19.70
36.50
16.33
55.26
32.21
70.27
52.34
40.37 0.00
13B
WizardLM
59.38
16.96
28.59
18.61
55.73
29.47
$7.84
42.06
38.58 0.00
Orca-2
50.00
29.50
42.12
20.61
56.41
31.11
75.95
62.15
45.98 0.00
McanLeam
45.82
30.01
47.65
21.85
61.04
35.21
85.41
77.08
50.51 = 0.23
(6) LLMs possess more than 20B parameters still struggle with abstract reasoning. MeanLearn can achieve comparable performance to PaLM-2 despite their huge gap in model sizes.
5.3 Results: abstract reasoning accuracy
Since the generic facts are not given in the OOD benchmarks, we train RoBERTa-Large 32 with
constrastive learning to cluster examples supported by the same generic fact. In particular, we utilize e-CARE the training set, examples guided by the same generic fact will be pushed closer, or they will be pushed away. Cosine similarity used measure the distance between examples (training details can refer to Appendix [ For clustering, examples sharing similarity above 0.6 are considered be supported by the same generic fact. Each cluster contains no more than 3 examples.
Hereafter, we calculate the abstract reasoning metrics based on the results of clustering. The overall results are shown in Table 5 AbsAcc). we can have the following observations:
(1) AbsAcc is much lower than vanilla accuracy, indicating the large gap between general reasoning and abstract reasoning. More efforts should be made to fill this gap in the future.
(2) MeanLearn can surpass all baselines, usually exhibiting greater advantages in AbsAcc than in vanilla accuracy. This suggests that the improvement in vanilla accuracy is more likely due to the enhancement of abstract reasoning abilities. Meanl.ean Learn can tell L.LMs the generic facts and teach them how to implicitly utilize them in different scenarios, resulting in better abstract reasoning.

---

Table 6: The overall reasults of ablation studies. "woo" denotes without.
Size
Method
AbsR AGIEval
RACE
BBH
Com.
MMLU
ARC-
ARC-c Average
Vanilla Accuracy
MeanLearn
77.00
38.15
77.15
35.64
76.59
52.98
86.67
78.06
65.28
w/o Knowledge
73.50
38.89
74.10
33.80
73.28
50.97
79.72
75.25
62.44
7B
w/o Reasoning
72.50
38.37
72.60
33.39
71.68
49.55
77.25
73.22
61.07
w/ AbsR'
73.50
38.53
73.14
33,65
73.57
50.96
80,60
75.25
62.40
MeanLearn
67.17
43.92
70.24
40.36
69.75
51.68
88,69
82.00
64.23
w/o Knowledge
63.50
44.07
67.19
36,80
70.45
47.88
84.13
74.58
61.08
13B
w/o Reasoning
56.00
43.95
58.05
35,91
68.09
45.31
82.54
69.83
57.46
w/ AbsR"
66.50
43.75
66.86
36.92
65.92
47.27
82.19
70.85
60.03
AbsAcc
Meanl .arn
64.58
25.27
57.10
16.67
71.17
37.24
81.35
73.83
53.39
w/o Knowledge
61.46
26.67
52.91
14.74
66.26
34.84
72.70
69.16
49.84
7B
w/o Reasoning
59.38
26.18
49.66
14.15
64.13
33.27
69.19
67.29
47.91
AbsR' 61.46
26.44
52.24
14.76
55.56
34.88
74.05
69.16
48.57
MeanLearn
45.82
30.01
47.65
21.85
61.04
35.21
85,41
77.08
50.51
w/o Knowledge
46.88
30.43
42.87
18.83
62.95
32.10
78.11
66.82
47.37
13B
w/o Reasoning
36.46
30.81
31.19
18.26
59.70
29.12
75.68
61.68
42.86
w/ AbsR"
50.00
29.58
42.12
18.82
56.41
31.11
75.95
62.15
45.77
Further analysis
To further investigate the pros, cons, and ef- fectiveness of MeanLearn, we conduct several analyses and ablation studies. LLaMA-2, Orca- 2, and MeanLearn are adopted for analysis.
Physies & Astronomy
Law
Medical
Finuace & Mamgemmnn
Society
Polities
Philosophy
Biology
Computer Science
Engineering
History
MeanLeam
Geogmphy
Prychology
Chemistry
Others
6.1 Performance on different domains
We categorize tasks of MMLU based on their respective domains to demonstrate the pros and cons of MeanLearn for future improvements, The results are shown in Figure with a total of 16 categories:
Vanilla-Ace
(1) All LLMs possess poor vanilla accuracy and AbsÅcc in math and chemistry. We con- jecture the primary factors are the complexity of math problems and the shortage of chem- istry knowledge.
AbsAco
(2) Compared to Orca-2, MeanLeam possesses considerable advantages in engineering, geog- raphy, and physics & astronomy domains. We suppose this is related the domain coverage of AbsR.
Figure 4: Visualization of reasoning capabilities on
MMLU, which is categorized by task domians.
(3) Since the overall AbsAcc is not relatively high, for potential improvement in the future, apart from focusing more on the domains with poor vanilla accuracy and AbsAcc, we should also increase the overall volume of finetuning data.
6.2 How much do knowledge and reasoning ways contribute to MeanLearn?
We utilize Knowledge and Reasoning ways to enhance abstract reasoning LLMs. To investigate the effect of them, we conduct ablation study of AbsR to respectively remove the Knowledge (generic facts) and Reasoning (explanations) each example. Results are shown in Table (w/o Knowledge and w/o Reasoning),
observe:

---

Table Performance on mathematical reasonin datasets from MMLU [18].
Table 7: MeanLearn trained based on Mistral.
Size Method
AA
CM
EM
HSS
HSM
Average
Method
AbsR Com, MMLU RACE Average
Vanilla Accuracy
Orca-2
23.00
32.00
33.07
37.95
28.15
Vanilla Accuracy
MeanLearn
36.00
35,00
33.07
40.28
28.89
34.56
Mistral
83,00
66.53
57.02
75.33
70.47
MeanLearn (Ours)
84,50 74.56
58.28
76.65
73,50
Mistral
25.00
30.00
38.89
46.76
32,96
34.72
McanLearn
31.00
34,00
37.83
47.69
34,07
36,92
AbaAce
Mistral
72.92
58.87
40.19
52.69
56.17
Meanl earn (Ours)
73.96
68.59
41.14
55.77
59.87
Orca-2
0.00
4.88
25.00
25.17
14.63
13.94
MeanTLearn
2,86
24.02
27.97
16.26
7B
Mistral
2,86
7.32
25.49
36.36
14.63
17.33
MeanI.eam
5.71
9.76
24.55
37.76
16.26
18.81
(I) Removing Knowledge (generic facts) or Reasoning (explanation) leads to a decrease in perfor- mance. This is because the generic facts and their guided reasoning process in AbsR can teach LLMs to learn and utilize the generic fact behind the questions to reason more logically and properly.
(2) In MeanLearn, Knowledge contributes less in boosting the vanilla and abstract reasoning than Reasoning. It indicates the importance teaching LLMs how to reason under the guidance of generic facts. Furthermore, it reveals the bottleneck of small-scale LLMs reasoning is caused more by reasoning itself rather than knowledge.
(3) MeanLeean without Knowledge can defeat Orca-2 on vanilla accuracy and AbsAcc, while MeanLeam without Reasoning cannot. This implies the synergy effects between Knowledge and Reasoning (MeanLearn) can push LLMs further.
(4) In AGIEval, Knowledge and Reasoning are less critical due to their focus complex reading com- prehension without needing vast knowledge. Moreover, Knowledge seems useless in Commonsense. When providing LLMs with knowledge, we should also teach them how use it.
6.3 Is the improvement performance solely attributed additional data?
Supervised fine-tuning can also improve AbsAcc (Table so we design another ablation study to investigate whether the improvement of MeanLearn only comes from additional data.
To be precise, for each sample in AbsR, we use GPT-4 to generate new explanation based on the question and label without the guidance of the generic fact. Hereafter, we obtain AbsR*. The only difference between AbsR and AbsR" is the explanations in AbsR are guided by the generic fact. Table (w/ AbsR*) shows the results Meanl.earn trained with AbsR", we can infer:
(1) Training 7B and 13B Orea-2 with AbsR" can still outperform Orca-2 on vanilla accuracy. However, cannot bring stable improvements on AbsAcc. This underscores that reasoning-speeiffe post-training can boost performance, while improving abstract reasoning performance might depend on more careful design such as MeanLearn.
(2) MeanLeamn without Knowledge outperforms MeanLearn trained with AbsR*, showing the su- periority of training with generic-face waa--afact----aaaee explanation. This stems from the seneric-fact-guidee explanation that can teach LLMS to use the regular pattern behind the questions for better reasoning.
6.4 Expanding Meanl Learn to LLMs besides the LLaMA series
To investigate the universality of MeanLeamn we train MeanLearn with Caatral-7--Instructt-t 21 (denoted as Mistral). All settings consistent with See 5 We choose AbsR, Com., MMLU, and RACE for evaluation. As shown in Table we can find MeanLearn has good applicability on Mistral.
6.5 Expanding reasoning domains to mathematical reasoning
Although we do not incorporate math examples in AbsR, we want to investigate whether Meanl .earn can improve the performance of mathematical reasoning by enhancing abstract reasoning in LLMs.

---

Following [33, we select five mathematical reasoning datasets from MMLU [8] to demonstrate this: abstract algebra (AA), college mathematics (CM), elementary mathematics (EM), high school statistics (HSS), and high school mathematics (HSM). We choose 7B version of Orca-2 and Mistral for comparison. Results are shown Table is interesting to find that Meanl earn can outperform baselines even if there is no math data in AbsR. This demonstrates the effectiveness of Meanleearn in enhancing abstract reasoning in LLMs.
Laan
Related work
7.1
Reasoning with LLMs
LLMs have overturned the research paradigm of NLP, which makes reasoning more accessible. Using LLMs to perform reasoning can be categorized into tunning-based and prompt-based methods.
Tunning-basse based methods, originating from TO [8 and FLAN [52], finetuned T5 [38 and LaMDA- PT 46 on massive NLP tasks, achieving notable zero-shot performance on unseen tasks. Flan- PaLM and TN-INSTRUCT [50] further scaled up the model and task size for improvement. Orea
36
and Orea-2
35
leveraged vast instruction data to teach small-scale L.MMs reasoning skills.
As for prompt-based methods, Wei et al. [53] introduced CoT prompt to elicit reasoning in LLMs by encouraging step-by-step thinking. Additionally, Kojima et al. (27) presented zero-shot CoT to bypass manual annotation issues. Subsequently, Zhang et al. (59) proposed AutoCoT to automatically generate few-shot examples with CoT for reasoning. Various CoT-based methods [22 5
have since
emerged, such as complex CoT [1], and the tree of thought [57], etc.
These methods designed for improving the general reasoning abilities of LLMs, while focus on the abstract reasoning capabilities of LLMs.
7.2 LLMs--s-annotators
LLMs are increasingly used annotators create data for training and evaluation due to their advanced instruction-folloing skills.
Some studies distilled various CoT data from LLMs to train student models (3
19 58 inter alia),
achieving impressive performance. For instance, Li et al. 30) distilled symbolic CoT from GPT-3 and observed enhanced performance in commonsense reasoning. Dai al. [2] used ChatGPT to generate augmentation data to enhance BERT (2). With the advent of open-source LLMs like LLaMA [47], other studies distilled instruction data from large-scale LLMs to train small-scale 31
[3 [7], inter alia). For example, Xu et al. (56) and Luo et al. (33 employed an evolving strategy to obtain complex data, trained LLaMA, and achieved strong performance. Tang et al. 45 constructed a concept graph and used GPT-3.5 to synthesize math data with high diversity, We employ LLMs to generate training data and develop AbsR tailored for abstract reasoning.
models [56] 36 25
Conclusion
In this paper, we investigate the abstract reasoning of existing LLMs. To achieve this, we first define the evaluation metric of abstract reasoming and design a series of preliminary experiments. Then we discover a significant performance gap between general reasoning and abstract reasoning. Hence, we tailor an abstract reasoning dataset AbsR with the help of GPT-4 to enhance LLMs. Finally, we devise a simple but effective paradigm (MeanLearn) to teach LLMs abstract reasoning in Knowledge and Reasoning ways. Extensive experiments demonstrate the superiority and effectiveness of MeanLearn in vanilla and abstract reasoning accuracies. The limitations of MeanLeamn are discussed in Appendix
Acknowledgements
We would like to acknowledge the editors and reviewers for their efforts and advice, and gratefully ac- knowledge the support of the National Natural Science Foundation of China under Grants U22B2059 and 62 17009, Natural Science Foundation of Heilongjiang Province under Grant YQ2022F005.
10

---

References
[1] Sharegpt: Share your wildest chatgpt conversations with one click, 2023.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[3] Alfonso Amayuelas, Liangming Pan, Wenhu Chen, and William Wang, Knowledge of knowl- edge: Exploring noown-unknowns uncertainty with large language models. arXiv preprint 2.ii:22005/3712, 2023.
[4] Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In International Conference on Learning Representations, 2019.
[5] Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. Genericskb: A knowledge base of generic statements. arXiv preprint arXiv:2005, 00660, 2020.
[6] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages
7439, 2020.
[7] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[9] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finetum language models. arXiv preprint arXiv:22 /416, 2022.
[10] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint wrXiv:1803.05457, 2018.
[11] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models, 2023.
[12] Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, et al. Auggpt: Leveraging chatgpt for text data augmentation. arXiv preprint 2rXiv:2302.13000 2023.
[13] Li Du, Xiao Ding, Ting Liu, and Zhongyang Li. Modeling event background for if-then com- monsense reasoning using context-aware variational autoencoder. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 2682-2691, 2019.
[14] Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. e-care: a new dataset for exploring explainable causal reasoming. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432-446 2022.
[15] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-bun based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, 2023.
[16] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
11

---

[17] Sonam Gupta, Yatin Nandwani, Asaf Yehudai, Mayank Mishra, Gaurav Pandey, Dinesh Raghu, and Sachindra Joshi. Selective self-rehearsal: A fine-tuning approach to improve generalization in large language models. arXiv preprint Kiv:22409004887, 2024.
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
[19] Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852-14882, Toronto, Canada, July 2023. Association for Computational Linguistics.
[20] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[21] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 1rX.v22310.06825, 2023.
[22] Ziqi Jin and Wei Lu. Self- heff-harmonized chain of thought. arXiv preprint arXiv:2409.04055 2024.
[23] Hoang Anh Just, Mahavir Dabas, Lifu Huang, Ming Jin, and Ruoxi Jia. Dipt: Enhancing Ilm reasoning through diversified saapeettivetaaking arXiv preprint (rXiv:2409.06241, 2024.
[24] Jacob Devlin Ming-Was Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota, 2019.
[25] Bumjun Kim, Kunha Lee, Juyeon Kim, and Sangam Lee. Small language models are equation reasoners. arXiv preprint arXiv:2409. 123393,,0. 2024.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[27] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
[28] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, 2017.
[29] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[30] Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wai Chang, and Yejin Choi Symbolic
distillation: Small models can also "think" step-by-step. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 2665-2679, Toronto, Canada, July 2023, Association for Computational Linguistics.
[31] Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, and Jun Zhao. Neural-symbolic collaborative distillation: Advancing small language models for complex reasoning tasks. arXiv preprint arXiv:2409. 13203, 2024.
[32] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint
2019.
12

---

[33] Haipeng Luo, Qingteng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023.
[34] Meta AL. Introducing meta Ilama 3: The most capable openly available Ilm to date, 2024.
[35] Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibicdina, Erik Jones, Kriti Aggarwal, et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.1104 2023.
[36] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.027777 027077 2023.
[37] Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang. Yoon Kim, Yejin Choi, Nouha Dziri, et al. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.0855, 2023.
[38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551 2020.
[39] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alter- natives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
[40] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463-4473, 2019.
[41] Jihao Shi, Xiao Ding, Li Du, Ting Liu, and Bing Qin. Neural natural logic inference for interpretable question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3673-3684, 2021.
[42] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities language models. Transactions Machine Learning Research, 2023.
[43] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguisties: ACL. 2023, pages 13003-13051) Toronto, Canada, July 2023. Association for Computational Linguistics.
[44] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages
2019.
[45] Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. In Forty-first International Conference on Machine Learning. 2024.
[46] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et Lamda: Language models for dialog applications. arXiv preprint
2022.
13

---

[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09988: 2023.
[48] Sanh Victor, Webson Albert, Raffel Colin, Bach Stephen, Sutawika Lintang, Alyafeai Zaid, Chaffin Antoine, Stiegler Arnaud, Raja Arun, Dey Manan, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[49] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprini arXiv:2212. 22.22000..00000 10560, 2022.
[50] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super-naturalinsruuioons Generalization via declarative instructions on 1600+ nip tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, 2022.
[51] Yu Wang, Shiwan Zhao, Zhihu Wang, Heyuan Huang, Ming Fan, Yubo Zhang, Zhixing Wang, Haijum Wang, and Ting Liu. Strategic shain-of-thoughhe Guiding accurate reasoning in llms through strategy elicitation. arXiv preprint arXhv:2409.03271, 2024.
[52] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Reprexentations, 2022.
[53] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. -fainn-ff-though prompting elicits reasoning in large language models. Advances Neural Information Processing Systems, 35:24824 24837724444877, 2022.
[54] Kai Xiong, Xiao Ding, Zhongyang Li, Li Du, Ting Liu, Bing Qin, Yi Zheng, and Baoxing Huai. Reco: Reliable causal chain reasoning via structural causal recurrent neural networks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6426-6438, 2022.
[55] Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7572-7590, 2023.
[56] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.1224 12244, 2023.
[57] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
[58] Jiahao Ying, Mingbao Lin, Yixin Cao, Wei Tang, Bo Wang. Qianru Sun, Xuanjing Huang, and Shuicheng Yan. Lims-as-instructors Learning from errors toward automating model improvement. Findings of EMNLP, 2024.
[59] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Represenia- tions, 2023.
[60] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human- benchmark for evaluating foundation models. arXiv preprint arXiv:2304.0mm 303400664444. 2023.
14

---

A Limitations
There are several limitations of our work: First, the size of the training data should be larger to obtain better and more stable results. Second, there should be some examples (whether MeanLearn can generate meaningful explanations) to demonstrate the superiority of MeanLearn. Third, more efforts should be made focus more on abstract reasoning itself. Finally, complex scenarios (with complex generic facts) such as causal chain reasoning [54]]]]]]] and logical reasoning 41 is potential direction of abstract reasoning.
B Asbstract reasoning study
B.1 Prompt for experiment
B.1.1 Orca-2
[lin_star/>ssyter
You are Orca, an AI language model created by Microsoft. You are cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive b <nd|>
(premise) Hypothesisi or Hypothesis2?
Hypothesisi: (hypothesist)
Hypothesis2: (hypothesis2)
Your answer should follow the format like "Answer: Hypothesis(1 2)
is more plausible.
Explanation: -__ im_dim_
ilimsttart/ asssistant
B.1.2 LLaMA-2 and GPT-3.5
System: You are a helpful assistant.
User: {premise] Hypothesisi Hypothesis2?
Hypothesisi: (hypothesist)
Hypothesis2: [hypothesis2) Your answer should follow the format like "Answer: Hypothesis 2) is more plausible.
Explanation:
The placeholders {premise], (hypothesisi), and [hypothesis2) will be filled with the corre- sponding terms in each example of c-CARE dataset.
15

---

B.2
Prompt for experiment II: Orca-2
B.2.1 Orca-2

syytee
You are Orca, an AI language model created by Microsoft. You are cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior.< < <lim_start/>üser
You given fact, do you know this fact? Just Yes No, do not give any additional information.
Fact: [fact] ilm__nd//>>> <lim_start assistant
B.2.2 LLaMA-2 and GPT-3.5
System:
You helpful assistant.
User: are given fact, you know this fact? Just answer or No, do not give any additional information.
Fact: [fact]
The placeholders [premise], (hypothesisl), (hypothesist), and [fact] will be filled with the corresponding terms in each example of c-CARE dataset.
C Details for generic facts filtering
We apply the following steps to filter generics facts from GeneriesKB:
(1) Exclude sentences with confidence < 0.7.
(2) Categorize the sentences by the concepts.
(3) Randomly sample 4,613 concepts, and then randomly sample one sentence as the generic fact from the category each sampled concept.
Hereafter, we can obtain collection of 4,613 generic facts.
D Prompt for AbsR construction
We use the following prompt to generate our AbsR examples.
You expert in creating questions, you should first offer question together with some options based on the fact the user gives. Second, you should give an answer and an explanation guided by the given fact. You can propose questions in any area, including but not limited to history, law, medicine, math, science, computer science, psychology, AI, politics, economics, etc. Your response should follow the following format: "Question: Explanation:
Options:
Answer:
". NOTE that the fact should be an implicit explanation for obtaining the true answer, which means the fact SHOULD NOT appear explicitly in the questions or the options. The explanations should be short. Please create [number) examples.
The placeholder [number] will be randomly filled with "one", "two" or "three".
16

---

E Details of human evaluation
We choose three annotators with good backgrounds in textual inference and event reasoning. The whole test set of AbsR is involved in the human evaluation (200 instances). For each annotator, we pay $10 per hour, while in our country. the minimum wage is less than $5 per hour. All annotators agreed to let us use their annotations. The evaluation agreements are 95%, 91 %, and 90% for the question answering, generic fact supportance, and diversity tasks, respectively.
F Instructions of human evaluation
E1
Performance
You are given some questions each with some options, all the questions are about commonsense and are generated by GPT-4. Please choose the most plausible option for each question. Just type your choice (such as (A)) in the Answer column.
F.2 Support rate
You are given some questions each with some options and generic fact, the questions are about commonsense and are generated by GPT-4. Please judge whether the generic support to answer the question. Just type your choice (Yes or No) in the Answer column.
F,3
Diversity
You given generic facts, each generic fact possesses several samples (each sample consists of a question and few options). The samples can be supported by the corresponding generic fact. Please judge whether the samples within generic are similar and are just different in nouns or expressions. If sample and sample are similar and are just different in nouns or expressions, just type tuple (1, in the Answer column. If there are multiple similar sample pairs, just separate them with a "l,"". For example, (1, ((3, 4).
G K K-example and R-example
G.1 K-example
You are a cautious assistant. You carefully follow instructions. You given question, few options, and rule, you should follow the given rule to answer the question. Your response should follow the format like "Explanation: Answer: __."/ii "<lim_end>> <lim_start
Fact: Cookie files are simple text files that can viewed Windows Notepad.
Question: What type application can used to open view cookie files on Windows computer? Options:
A) Microsoft Excel B) Adobe Photoshop C) Windows Notepad D) 3D Modeling boftware<lim_end]] /im_sstar/>assiiitet
Explanation: Windows Notepad is a text editor that is capable of opening and displaying the contents of simple text files, which is the format of cookie files.
Answer: C) Windows Notepad< |
17

---

G.2 R-example
</im_start systt
system
You are a cautious assistant. You carefully follow instructions. You are given a question together with a few options, you should give explanation first and then answer the question. Your response should follow the format like "Explanation: Answer:: ---- "__"<m/mm_en>>>>
[lin_start/>üBrr
Question: What type of application can be used to open and view cookie files on Windows computer? Options:
A) Microsoft Excel B) Adobe Photoshop C) Windows Notepad D) 3D Modeling boftware<lim_ende
<I <lim_start/> assisstant
Explanation: Windows Notepad is text editor that is capable of opening and displaying the contents of simple text files, which is the format of cookie files.
Answer: C) Windows Notepad< im_end|>
Templates for evaluation
H.1 LLaMA-2, LLaMA-3, and MeanLearn (8B)
The following are multiple-choice questions (with answers) about abstract algebra.
Find the degree for the given field extension Q(sqrt (2), sqrt(3), sqrt (18)) over Q. A. B. 4 C. 2 D.
Answer:
H.2
Orca-2 and Meanl (7B and 13B)
<l >> /start/>system system
The following are multiple-choice questions (with answers) about abstract <ggabra.<lim_emn
<limstaat\>uuser
Question: Find the degree for the given field extension (sqrrt (3), sqrt(18)) over Options:
A. B. C. D. @<lim_end]]
<lim_start|>>as >>asisstant
Answer: A<lim_end|>
I Training details of RoBeRTa-Large--assd clusterer
We use the RoBERTa-Large
released by Meta. The batch size is 256. We use AdamW 266 to optimize the model with learning rate of le-5 for 5 epochs. The computing device is one NVIDIA A100-SXM-64GB. The running time is about 12 minutes.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims can be supported by the experiments and ablation studies.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have provided limitations of the work in Sec. 6.1 and Appendix, A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
We have provided complete proof in all the tables and figures in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We have provided the training and evaluation details for reproducibility in We also submit the code and data in the supplementary material. We also run experiments for multiple times to report the mean and standard deviation of MeanLearn.
Justification:
Sec. []
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided the link for the data and code at the end of the Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the training and test details of all baselines and MeanLean in Sec. 5 Some other details such as the prompts and human evaluation are provided the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported the mean and standard deviation results across 3 runs of MeanLearn in 5 There is no randomness with all the baselines since the evaluation criterias we used is greedy decoding and PPL-based prediction.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the compute resources and time of execution in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: We have provided the detials human evaluation (such as the pay) in Ap- pendix E We also obtain the agreements of the human annotators to use their annotation. We also follow the other terms in the code ethics which would be covered in our paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We focus on the reasoning abilities of LLMs, which is foundational research and does not concentrate specific application scenarios.
Guidelines:
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We use publicly accessible data and LLMs for experiments.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite every resource (such LLMs and benchmarks) used in this work. Please refer to Sec. 5 We also follow the the license and terms of use of corresponding
LLMs and data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssie thcode cwwc datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have provided the structure of the AbsR dataset in See. 3.2 Figure also shows two samples of AbsR. We also provide the create AbsR dataset with a readme file in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We have provided the instructions of human evaluations in Appendix We also offer the information such the pay in Appendix E
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our human evaluations are answering questions about commonsense.
Guidelines:

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Rto60nAr2W", "reviewer": "Reviewer_XVL7", "summary": "This paper addresses the challenge that LLMs face in abstract reasoning, where they often struggle to apply general facts to new situations despite their impressive performance in other reasoning tasks. To tackle this issue, the authors introduce an abstract reasoning dataset called AbsR, which incorporates generic facts and guided explanations to teach LLMs how to leverage such facts for reasoning. They also propose a learning paradigm named Meaningful Learning (MeanLearn) that simulates the human process of implicit knowledge acquisition, enabling LLMs to implicitly learn and apply generic facts without explicit input. Through experiments on various out-of-distribution reasoning and language understanding benchmarks, the paper demonstrates that MeanLearn improves the general and abstract reasoning capabilities of LLMs, moving beyond simple memorization towards a more nuanced understanding and application of knowledge.", "questions": "See above,", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Useful Resource: This paper introduces an abstract reasoning dataset (AbsR) that provides generic facts and guided explanations for reasoning tasks. The dataset and code will be publicly available, facilitating reproducibility and further community research.\n2. Empirical Evidence: Comprehensive experimental results and ablation studies that validate the effectiveness of the proposed method.  Improvements are observed in both general and abstract reasoning performance of LLMs across various benchmarks.\n3. Broad Applicability: The approach's effectiveness is shown on multiple LLMs of varying sizes, indicating broad applicability.", "weaknesses": "1. Evaluation on Advanced LLMs: The observation that the benefits of MeanLearn seem to diminish with better-trained LLMs, like LLaMA3, raises a crucial question about its broader applicability. While the method shows promise with smaller models, it's essential to assess its performance on more powerful LLMs like Mistral, Phi3, and Qwen 2. This would provide a clearer picture of its potential contribution in the context of rapidly advancing language models.\n\n2. Data Augmentation and Training Dynamics: The use of GPT4 for constructing the AbsR dataset, with annotation quality comparable to human annotators, opens up interesting possibilities for data augmentation. Exploring the impact of increasing the training data size, potentially using more cost-effective models like GPT4o or open-source LLMs, could reveal the potential for further performance gains. Additionally, analyzing the training dynamics of MeanLearn by varying the proportion of training data (e.g., 20%, 50%, 80%) would provide valuable insights into its saturation point and the diminishing returns of additional data.\n\n3. Expanding Reasoning Benchmarks: The paper's focus on commonsense reasoning benchmarks (Com. and ARC) should be complemented by evaluation on other reasoning domains, such as arithmetic reasoning. Including benchmarks like GSM8K or MATH would provide a more comprehensive understanding of MeanLearn's capabilities across different reasoning tasks.\n\n4. Presentation Clarity and Improvements: The paper's presentation could benefit from several improvements. The introduction should explicitly connect abstract reasoning with other types of reasoning, providing a broader context for the research. The pilot investigation in the introduction is quite confusing as it does not introduce the datasets, metrics, and setup used. Confusing figures, like Figure 2, should be explained in detail to ensure clarity and understanding.\n\n5. Updated Related Work: The related work section should be updated to include recent efforts in tuning-based methods, ensuring a comprehensive overview of the current landscape in abstract reasoning research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments, the following is the detailed response:\n# 1. Evaluation on Advanced LLMs (Weakness 1)\nThe benefits of our method MeanLearn do not diminish, but it is LLaMA-3 may need more data compared with other base models. The reason is because LLaMA-3 breaks conventional data scaling laws [1] by achieving a token-to-parameter (T/P) ratio of 1875:1, far surpassing the 286:1 ratio of LLaMA-2. This results in dense knowledge in the parametric memory of LLaMA-3. However, to ensure fair comparisons across different LLMs, we trained MeanLearn using LLaMA-3 with the same dataset size as the other LLMs.\n\nDue to limited time, we train MeanLearn based on Mistral-7B-Instruct-v0.2, and evaluate them on Com., MMLU, RACE and AbsR, the results are shown in the following table:\n\n|Method|AbsR|Com.|MMLU|RACE|*&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Average*|\n|---------------------|:-----:|:-----:|:-----:|:-----:|:---------:|\n|***Vanilla Accuracy***||||||\n|Mistral-7B-Instruct|83.00|66.53|57.02|75.33|70.47|\n|MeanLearn(Ours)|84.50|74.56|58.28|76.65|**73.50 (+3.03)**|\n|***AbsAcc***||||||\n|Mistral-7B-Instruct|72.92|58.87|40.19|52.69|56.17|\n|MeanLearn(Ours)|73.96|68.59|41.14|55.77|**59.87 (+3.60)**|\n\nOn average, our proposed MeanLearn can outperform Mistral by 3.03% in vanilla accuracy and 3.70% in AbsAcc, which demonstrates the superiority of MeanLearn. The main reason is that Mistral does not have a dense knowledge in parameters as LLaMA-3.\n\n[1] Pearce and Song, 2024. Reconciling Kaplan and Chinchilla Scaling Laws\n# 2. Data Augmentation and Training Dynamics (Weakness 2)\nGPT-4o was released about a week before the NeurIPS 2024 deadline, making it too late to use for synthesizing AbsR. Indeed, leveraging cost-effective models like GPT-4o or open-source LLMs for data generation is crucial for future performance improvements. We are enthusiastic about exploring this and the training dynamics in the future to achieve continual performance gains.\n# 3. Expanding Reasoning Benchmarks (Weakness 3)\nThere are mathematical tasks in MMLU, following [2], we select five mathematical reasoning datasets (abstract algebra, college mathematics, elementary mathematics, high school statistics, and high school mathematics) from MMLU to demonstrate the superiority of MeanLearn. The results are shown in the following table:\n|Size|Method|Abstract Algebra|College Mathematics|Elementary Mathematics|High School Statistics|High School Mathematics|*Average*|\n|------------------|-----------|:----------------:|:-------------------:|:----------------------:|:----------------------:|:-----------------------:|:-------------:|\n|***Vanilla Accuracy***||||||||\n|7B|Orca-2|23.00|32.00|33.07|37.95|28.15|30.83|\n||MeanLearn (Ours)|36.00|35.00|33.07|40.28|28.89|**34.65 (+3.82)**|\n||Mistral|25.00|30.00|38.89|46.76|32.96|34.72|\n||MeanLearn (Ours)|31.00|34.00|37.83|47.69|34.07|**36.92 (+2.20)**|\n|8B|LLaMA-3|30.00|36.00|41.27|50.00|40.74|39.60|\n||MeanLearn (Ours)|31.00|33.00|44.44|49.54|40.74|**39.74 (+0.14)**|\n|13B|Orca-2|27.00|35.00|35.45|34.72|28.15|32.06|\n||MeanLearn (Ours)|27.00|37.00|37.30|43.06|27.78|**34.43 (+2.37)**|\n|***AbsAcc***||||||||\n|7B|Orca-2|0.00|4.88|25.00|25.17|14.63|13.94|\n||MeanLearn (Ours)|2.86|7.32|24.02|27.97|16.26|**15.69 (+1.75)**|\n||Mistral|2.86|7.32|25.49|36.36|14.63|17.33|\n||MeanLearn (Ours)|5.71|9.76|24.55|37.76|16.26|**18.81 (+1.48)**|\n|8B|LLaMA-3|2.86|9.76|28.43|38.46|18.70|19.64|\n||MeanLearn (Ours)|5.71|12.20|30.88|34.97|19.51|**20.65 (+1.01)**|\n|13B|Orca-2|2.86|4.88|24.02|24.48|13.82|14.01|\n||MeanLearn (Ours)|2.86|17.07|22.55|29.37|10.57|**16.48 (+2.47)**|\n\nOn average, MeanLearn outperforms baselines on both vanilla accuracy and AbsAcc. It is interesting to note that we do not synthesize math questions to train MeanLearn, improvements on math reasoning are largely due to the enhancement of abstract reasoning.\n\nMeaningful Learning is one of the main contributions of our work, and we are excited about the improvements brought by MeanLearn. Building on this foundation, we are now conducting a new work to expand our investigations with additional tasks such as mathematical and logical reasoning.\n\n[2] MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark\n\n# 4. Presentation Clarity and Improvements (Weakness 4)\nAbstract reasoning is not a specific type of reasoning; rather, it measures the ability of employing generic facts to solve problems. For each reasoning type, an abstract reasoning performance metric can be derived. For example, in math problems, abstract reasoning can help estimate the ability of LLMs to use the generic fact “x + y = y + x” to solve a series of problems in various scenarios that rely on this fact. While in commonsense problems, it can help estimate the ability of LLMs to use the generic fact “acid is corrosive” to solve a series of problems in various scenarios that rely on this fact.\n\nThe pilot investigation results presented in introduction serve to intuitively highlight the disparity (Table 1) between general and abstract reasoning, which is used to demonstrate our motivation that LLMs have a substantial discrepancy between their general reasoning and abstract reasoning performances. Detailed discussions of this investigation are available in sections 2.1 and 2.3. \n\nFurthermore, Figure 2 illustrates the calculations for vanilla accuracy and AbsAcc, employing the same symbols and formulas outlined in section 2.1 and equation (1).\n\n# 5. Updated Related Work (Weakness 5)\nDue to page limitations and to place more experimental results, we have condensed some sections of the related work. We will expand the related work in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fTUszIdHwH", "reviewer": "Reviewer_vYe8", "summary": "This paper introduces a novel framework aimed at enhancing the abstract reasoning capabilities of large language models (LLMs) through a method called \"Meaningful Learning.\" It specifically targets the challenge LLMs face in abstract reasoning despite their robust general reasoning abilities. The authors identify a notable gap in performance between general and abstract reasoning tasks and propose a structured approach to narrow this gap by using AbsR, a tailored dataset that includes generic facts coupled with guided explanations to foster deeper learning and understanding.\nKey contributions include:\n1. Introduction of the Meaningful Learning framework for improving abstract reasoning in LLMs.\n2. Development of the AbsR dataset for training.\n3. New metrics and Empirical evaluation across different settings", "questions": "1. How did you determine that perplexity-based evaluation was the best approach for classification tasks, given its limitations mentioned in Appendix A? Were other methods considered?\n2. Have you considered comparing MeanLearn to more recent reasoning techniques, such as tree-of-thoughts or other advanced prompting methods\n3. The paper mentions that \"MeanLearn can make LLMs implicitly learn generic facts and solve problems under the guidance of generic facts\" (Section 4). Can you provide more concrete examples or analysis of how this implicit learning occurs? \n4. How do you envision MeanLearn scaling to larger models (100B+ parameters), and what challenges or benefits do you anticipate in applying this method to more advanced LLMs? \n5. how do you ensure that the AbsAcc metric is truly capturing abstract reasoning abilities rather than other factors like memorization or pattern matching?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel approach to improving abstract reasoning in LLMs through generic fact guidance.\n2. Comprehensive evaluation across multiple settings demonstrating reasonable performance improvements.", "weaknesses": "1. Limited scale: The experiments are conducted on relatively small models (7B-13B parameters) compared to state-of-the-art LLMs. Without such it's hard to judge such method's applicability in real world and more complex tasks as mentioned in 5.1.\n2. Human evaluation scale: The human evaluation of the AbsR dataset is conducted on a relatively small sample (200 instances) (Appendix E).\n3. Lack of comparison to more recent reasoning techniques: The paper doesn't compare MeanLearn to recent advances in LLM reasoning capabilities.\n4. Weaknesses in Evaluation Metrics: It's not convincing that Perplexity-based evaluation for classification tasks is good and the use of AbsAcc needs more clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments, the following is the detailed response:\n# 1. Limited scale (Weakness 1 and Question 4)\n## 1.1 Limited scale\nDue to limited computational resources, we conduct our experiments on small LLMs (7B-13B). It is worth noting that our main focus is also to improve small LLMs, which is also crucial, as evidenced by considerable research focus in this area [1][2]. Small LLMs have a broader range of applications and are preferred for deployment on devices like cell phones due to their lower resource demands. Despite their wider applicability, small LLMs are significantly weaker than their larger counterparts, underscoring the importance of prioritizing their improvement.\n\n[1] Orca 2: Teaching Small Language Models How to Reason\n\n[2] WizardLM: Empowering Large Language Models to Follow Complex Instructions\n## 1.2 Applying MeanLearn to 100B+ LLMs\nWe are attempting to discuss to apply our method to 100B+ LLMs, but it is out of the scope of this paper. Applying MeanLearn to 100B+ LLMs would presents both challenges and benefits:\nChallenges: (1) Resource Requirements: the post-training of such large LLMs demands additional and more stable computational resources to manage the high costs associated with their scale; (2) Data Demands: according to the data scaling law [3], there is a significant need for extensive training data to achieve adequate coverage and ensure the generalization capabilities of these large models. Acquiring such large datasets is often costly.\nBenefits: (1) Enhanced Learning Abilities: larger LLMs possess stronger learning capacities, enabling them to more effectively comprehend and assimilate the knowledge presented to them; (2) Superior Performance: these models typically outperform smaller LLMs, offering better overall performance in tasks due to their advanced capabilities.\n\n[3] Pearce and Song, 2024. Reconciling Kaplan and Chinchilla Scaling Laws\n# 2. Human evaluation scale (Weakness 2)\nWe mainly follow previous works [4] [5], which conduct human evaluation on 100-200 instances. We choose 200 instances for evaluation, which is a balance between maintaining evaluation quality and managing labor costs.\n\n[4] Du et al., e-CARE: a New Dataset for Exploring Explainable Causal Reasoning\n\n[5] Ying et al., Intuitive or Dependent? Investigating LLMs' Robustness to Conflicting Prompts\n# 3. Lack of comparison to more recent reasoning techniques (Weakness 3 and Question 2)\nWe adhere to conventional standards by selecting baselines of similar types and scales to ensure a fair evaluation. Our focus is on enhancing the fundamental capabilities of LLMs through post-training techniques. In contrast, methods like Tree-of-Thought (ToT) primarily target improvements during the decoding stage, which complements our proposed method.\n# 4. Perplexity-based evaluation (Weakness 4 and Question1)\nWe align with the evaluation criteria established by prior research [6][7] and leaderboards [8][9], adopting perplexity as our evaluation metric. The advantages of employing perplexity are twofold: \n* Clarity in Evaluation: evaluation with perplexity can extract an answer for each example, while generation-based evaluation is hard to do this, since LLMs might either refuse to provide an answer or treat any option as valid;\n* Efficiency: Calculating perplexity requires only a single forward pass through the LLMs, making it more cost-efficient compared to generation-based methods, which require text to be generated in an autoregressive manner. This streamlined process enhances both the speed and the resource efficiency of the evaluation.\n\n[6] Sun et al., 2024: A Simple and Effective Pruning Approach for Large Language Models\n\n[7] Zhao et al., 2024: Deciphering the lmpact of Pretraining Data on Large Language Models through Machine Unlearning\n\n[8] OpenCompass, 2023: OpenCompass: A Universal Evaluation Platform for Foundation Models\n\n[9] Huggingface 2024: Open LLM Leaderboard\n# 5. About AbsAcc (Weakness 4 and Question 5)\nTheoretically, AbsAcc offers a more reliable measurement of abstract reasoning. For each generic fact, we have multiple examples to test LLMs, effectively reducing the influences of memorization and pattern matching. Ideally, increasing the number of examples for each generic fact would enhance the reliability of AbsAcc. We do not incorporate more examples per generic fact for the balance between cost and effectiveness. Please note that, our method is scalable with respect to the number of examples for each generic fact.\n# 6. Examples or analysis of of the implicit learning process (Question 3)\nHumans can apply the grasped generic fact to solve problems in different scenarios. For example, in Figure 1 of our paper, if humans grasp “acid is corrosive”, they can deduce “rock dissolved” and “the skin suffers pain” when respectively given “adding rock into hydrochloric acid” and “acid touches human skin”. MeanLearn is designed to imitate the above meaningful learning process of humans.\n\nAs for analysis, this question is indeed interesting and merits exploration. However, it falls outside the scope of the main contributions of our work. Our focus is not on demonstrating the implicit learning processes within a black-box LLM, as that pertains more to the field of interpretability. Addressing this would require a significantly different research approach and substantial additional effort.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gzosYc6aCt", "reviewer": "Reviewer_zthW", "summary": "This paper explores the abstract reasoning abilities of LLMs by creating a specific evaluation metric and a dataset called AbsR, developed using GPT-4. It presents a method, Meaningful Learning (MeanLearn), to improves both general and abstract reasoning accuracies of LLMs by teaching them to apply generic facts in varied scenarios.", "questions": "- How the methods introduced in the paper can be compare to [1]?\n- The paper states that \"Our findings reveal a substantial discrepancy between their general reasoning and abstract reasoning performances\". However, based on [2], LLMs \"know\" more than they can \"say\". Please justify how your statement remains valid.\n- Considering that MeanLearn does not have many advantages over LLaMA-3, how do you justified the use of MeanLearn?\n\n- [1] Xu, Yudong, et al. \"LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations.\" Transactions on Machine Learning Research.\n\n- [2] Li, Kenneth, et al. \"Inference-time intervention: Eliciting truthful answers from a language model.\" Advances in Neural Information Processing Systems 36 (2024).", "limitations": "- Lack of sufficient experiments.\n- Limited related work, and background sections.\n- A thorough writing revision would be beneficial.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Proposing a dataset to study the potential of LLMs in abstract reasoning.\n- Proposing MeanLearn to improve the abstract reasoning capabilities of LLMs by teaching them to use generic facts for reasoning.", "weaknesses": "- The paper is rushed, and not well organized. For instance, Some experimental results are presented in section 2, and others appear later in section 5 and after.\n- The methodology section is half a page. More details on the model is needed. For instance, in Eq. 3, how the first and second equations will be utilized in the model.\n- Incomplete related work section, specifically, insufficient related work on Abstract Reasoning.\n- Some experimental results are in section 2: Abstract reasoning study, some later in section 5: Experiments. In section 2, the paper jumps into some results and experiments too early and without proper background.\n- Experiments are limited. For example, it is unclear how the methods compare to [1] \n\n- [1] Xu, Yudong, et al. \"LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations.\" Transactions on Machine Learning Research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to clarify some critical misunderstandings in the preliminary review, which may have negative impacts on your assessment of our contributions.\n# 1. The paper is rushed (Weakness 1)\nBoth sections 2 and 5 contain results, and this arrangement is intentional and not rushed. As in many prior works [1][2], the results in section 2 are preliminary and are used to demonstrate our motivations and introduce our following method. Meanwhile, the results in section 5 are related to our proposed methods and baselines. Thus, the results in sections 2 and 5 are independent and serve different purposes within our paper.\n\n[1] Zhang et al., 2023: Automatic Chain of Thought Prompting in Large Language Models\n\n[2] Jin et al., 2024: The Impact of Reasoning Step Length on Large Language Models\n# 2. The methodology section is half a page (Weakness 2)\nDue to the page limit, we are trying to put more important content in the main paper. To accommodate as much as experimental results, we describe our approach as concisely as possible (More details can refer to Appendix C and D). It is worth noting that our method has two sections, the construction method of AbsR dataset (section 3) and MeanLearn pipeline (section 4), a totally of 1.5 pages.\n\nIn Eq. (3), the first equation describes using input (X) to autoregressively model output (Y), the second is using X and generic fact r to autoregressively model Y, which is usually adopted by current LLMs posting-training works [3][4].\n\n[3] Orca 2: Teaching Small Language Models How to Reason\n\n[4] WizardLM: Empowering Large Language Models to Follow Complex Instructions\n\n# 3. Insufficient related work on Abstract Reasoning (Weakness 3)\nTo the best of our knowledge, this is the first work to highlight the importance of abstract reasoning in the field of natural language processing. Thus, we include the most similar research directions: “Reasoning with LLMs” and “LLMs-as-annotators” in the related work section. Due to page limitations, we have condensed the related work. These sections will be expanded in the revised version.\n\n# 4. Comparison to paper [5] (Weakness 4 and Question 1)\nWe believe there may be some critical misunderstandings concerning our paper and your referred paper. A comparison with this reference is unnecessary because: (1) it focuses on the reasoning on images or symbols, whereas our work is centered on abstract reasoning in natural language; (2) the paper does not propose a new method but merely presents an empirical study using ChatGPT and LLaMA on image tasks.\n\n[5] LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations\n\n# 5. The statement in our paper \"Our findings reveal a substantial discrepancy between their general reasoning and abstract reasoning performances\" and \"LLMs know more than they can say\" in [6] (Question 2)\nThese two statements indeed do not conflict. Most researches follow the conventional evaluation process to directly evaluate LLMs based on their outputs, quantifying their reasoning abilities. Our proposed abstract reasoning metric AbsAcc based on cognitive theory, and our results show a substantial disparity between general and abstract reasoning. On the other hand, the statement “LLMs know more than they can say” may be derived from the domain of interpretability. It concerns the LLMs’ inference processes on tasks, which may not be in a form understandable to humans, leading to performance loss when translating these processes into natural language. The focal points of our work and the work you referred to are distinct, each addressing different aspects of LLM capabilities and behaviors.\n\n[6] Inference-time intervention: Eliciting truthful answers from a language model.\n# 6. the effectiveness of MeanLearn (Question 3)\nThe benefits of our method MeanLearn do not diminish, but it is LLaMA-3 may need more data compared with other base models. The reason is because LLaMA-3 breaks conventional data scaling laws [7] by achieving a token-to-parameter (T/P) ratio of 1875:1, far surpassing the 286:1 ratio of LLaMA-2. This results in dense knowledge in the parametric memory of LLaMA-3. However, to ensure fair comparisons across different LLMs, we train MeanLearn based on LLaMA-3 with the same dataset size as the other LLMs. Furthermore, MeanLearn achieved satisfied improvements on LLaMA-2 and Orca-2.\n\n[7] Pearce and Song, 2024. Reconciling Kaplan and Chinchilla Scaling Laws", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8iEK7o82xJ", "reviewer": "Reviewer_Fwhs", "summary": "The study introduced a specialized dataset and learning approach that aimed to enhance LLMs' use of generic facts in reasoning. The results indicate that this methodology not only improved their general reasoning skills but also significantly developed their abstract reasoning abilities, suggesting a shift from mere memorization to a deeper, more nuanced understanding and application of information. The paper is well composed and demonstrates good, consistent performance across several benchmarks.", "questions": "- see the questions in weakness section", "limitations": "- some of the terms mentioned in the paper maybe a little bit fancy and overestimated, please refer to the weakness section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- the concept of AbsAcc is interesting and natural, as demonstrated by the experiments in Table 1, where the small difference between vanilla accuracy and AbsAcc for human subjects highlights that vanilla accuracy alone may not be an adequate metric for assessing abstract reasoning. \n- the paper is well-composed and straightforward to comprehend.", "weaknesses": "- the experiments currently focus on earlier versions of LLMs, including LLaMA-2 and GPT-3.5-turbo-0125. It is advisable to also include more recent models like LLaMA-3 and GPT-4/4o to ensure the consistency of the performance evaluations.\n- some of the terminology used appears complex; for instance, in the memory learning section, the distinction between Knowledge and Reasoning examples (K-example and R-example) seems minimal. In the K-example, when a user poses a question, a supporting fact is cited while the R-example not. Additionally, I'm confused by the definition of abstract reasoning. It looks akin to tasks in commonsenseQA (https://aclanthology.org/N19-1421.pdf). The authors might need to elucidate the differences between their approach and that body of research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments, the following is the detailed response:\n# 1. More experiments (Weakness 1)\nWe have conducted experiments on LLaMA-3 which is shown in Table 5, our MeanLearn can improve the performance of LLaMA-3 on both vanilla accuracy and AbsAcc. We do not test GPT-4/4o is because: (1) our main focus is to improve small LLMs, which is also crucial and have a broader range of application. They are preferred for deployment on devices like cell phones due to their lower resource demands. (2) GPT-4o is released about a week before the deadline of NeurIPS 2024, leaving insufficient time to conduct experiments with GPT-4o. Furthermore, we are glad to offer more exploration in the future.\n\n# 2. Some of the terminology used appears complex (Weakness 2)\nYour understandings about K-example and R-example are correct. While for abstract reasoning, it is not a specific reasoning type. We can compute the performance of abstract reasoning for any task or benchmark, not just CommonsenseQA. For example, in math problems, abstract reasoning can help estimate the ability of LLMs to use the generic fact “x + y = y + x” to solve a series of problems in various scenarios that rely on this fact. While in commonsense problems, it can help estimate the ability of LLMs to use the generic fact “acid is corrosive” to solve a series of problems in various scenarios that rely on this fact.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper represents a relatively substantial amount of work about reasoning in LLMs. While one reviewer believes there are issues with the paper to be resolved, it seems like the overall consensus is that the paper represents sufficient amount of work. I ask that the authors work specifically on the representational issues that are mentioned by the reviewers.
Decision: Accept (poster)