Information Re-Organization Improves Reasoning in Large Language Models
Authors: Xiaoxia Cheng
Zeqi Tan
Wei Xue
Weiming Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
Information Re-organization
Multi-hop Reasoning
Primary area: natural_language_processing

Abstract: Improving the reasoning capabilities of large language models (LLMs) has attracted considerable interest. Recent approaches primarily focus on improving the reasoning process to yield a more precise final answer. However, in scenarios involving contextually aware reasoning, these methods neglect the importance of first identifying logical relationships from the context before proceeding with the reasoning. This oversight could lead to a superficial understanding and interaction with the context, potentially undermining the quality and reliability of the reasoning outcomes. In this paper, we propose an information re-organization (\textbf{InfoRE}) method before proceeding with the reasoning to enhance the reasoning ability of LLMs. Our re-organization method involves initially extracting logical relationships from the contextual content, such as documents or paragraphs, and subsequently pruning redundant content to minimize noise. Then, we utilize the re-organized information in the reasoning process. This enables LLMs to deeply understand the contextual content by clearly perceiving these logical relationships, while also ensuring high-quality responses by eliminating potential noise. To demonstrate the effectiveness of our approach in improving the reasoning ability, we conduct experiments using Llama2-70B, GPT-3.5, and GPT-4 on various contextually aware multi-hop reasoning tasks. Using only a zero-shot setting, our method achieves an average absolute improvement of 4\% across all tasks, highlighting its potential to improve the reasoning performance of LLMs.

Information Re-Organization Improves Reasoning in Large Language Models
Xiaoxia Cheng, Zeqi Tan, Wei Xue, Weiming Lu" College of Computer Science and Technology Zhejiang University {zjucxx, zqtan, lokilanka, (omm))-zj.edd.cm
Abstract
Improving the reasoning capabilities of large language models (LLMs) has at- tracted considerable interest. Recent approaches primarily focus improving the reasoning process to yield a more precise final answer. However, in scenarios involving contextually aware reasoning, these methods neglect the importance of first identifying logical relationships from the context before proceeding with the reasoning. This oversight could lead to superficial understanding and interac- tion with the context, potentially undermining the quality and reliability of the reasoning outcomes.
this paper, we propose an information re-organization (InfoRE) method before proceeding with the reasoning to enhance the reasoning ability of LLMs. Our re-organization method involves initially extracting logical relationships from the contextual content, such as documents or paragraphs, and subsequently pruning redundant content minimize noise. Then, we utilize the re-organized information in the reasoning process. This enables LLMss to deeply understand the contextual content by clearly perceiving these logical relationships, while also ensuring high-quality responses by eliminating potential noise. To demonstrate the effectiveness of our approach improving the reasoning ability, we conduct experiments using Llama2-70B, GPT-3.5, and GPT-4 on various contex- tually aware multi-hop reasoning tasks. Using only a zero-shot setting, our method achieves average absolute improvement of 4% across all tasks, highlighting its potential to improve the reasoning performance of LLMs. Our source code is available httth::///e h/tp:::////ttbb com/ http///sstxx//xxt InfoRE
Introduction
Large language models (LLMs) demonstrate powerful generative capabilities and achieve remarkable performance across a range of linguistic tasks [13]. However, their capabilities in performing complex reasoning tasks
essential aspect of advanced language understanding and intelligent decision-making - still present substantial challenges 4 [5). This has spurred researchers to explore innovative strategies [6] improve the reasoning capabilities of these models.
Recently, diverse methods have been developed enhance the reasoning ability of LLMs. For example, notable method Chain-of-Thoughh (CoT) [6]. incorporates a series of intermediate reasoning steps into the reasoning. CoT (6) allows for more transparent and understandable path to the final answer, making it easier to follow the logic behind the conclusion. Building upon this foundation, subsequent approaches such as Tree of Thoughts (ToT) 7 and Graph of Thoughts (Go'T) 8 are proposed to further refine the reasoning steps and enhance the accuracy and reliability of LLMss. Different from the sequential intermediate steps of CoT (6. ToT [ and GoT [8 model the problem solving process into structured tools of tree and graph, respectively. A critical observation
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

is that these existing approaches primarily focus on improving the reasoning process of LLMs, as Figure[[(left),,
shown in Figure (left). However, in scenarios involving contextually aware reasoning, is equally important to first identify logical relationships of context before proceeding with the reasoning, not just improve their reasoning process. This is because logical relationships, such as parallelism, causal connections, contrasts, etc., are essential elements of reasoning [9]. Nevertheless, these existing methods often neglect this crucial step. Such an oversight can lead to a superficial understanding and interaction with the context, potentially undermining the quality of the reasoning results.
Existing
InfoRE
Context:
Context:
[TEXT]
[Re- [Re-Ongad TEXT]
Question:
[TEXT]
[QUESTION]
Question:
Answer:
[QUESTION]
First,
Answer:
Next,
Final answer is [ANSWER]
Final answer is [ANSWER]
Figure I: InfoRE (Ours) vs existing methods. In contrast to the existing methods that primarily focus on the reasoning process, our InfoRE emphasizes the re-organization of context information. The [TEXT] in italics indicates that it is optional in the reasoning process.
Inspired by the fact that when faced with context-aware reasoning tasks humans often first re-organize existing contextual information to uncover the logical relationships, eliminate noises, and enhance their understanding of the context, we propose an information re-organization (InfoRE) method, to ground reasoning by the re-organized information. As shown in Figure I (right), different from previous methodologies that primarily focus on refining the reasoning steps to enhance the reasoning capabilitiis of L.LMs, our approach takes novel direction of context re-organization. We emphasize the utilization of re-organized contextual content to explicitly present the logical relationships that often implicit within the plain text, promoting more effective reasoning. Specifically, our re- organization method comprises two operations: extraction and pruning. The extraction first uncovers the implicit logical relationships within the contextual content by transforming the content into MindMap structure [TO) We employ this structure because it is rich in logical relationships and encompasses multi-hop connections. Pruning is then used to further minimize noise that irrelevant to the reasoning objective. The pruning operation uses pre-trained BERT [1] based model trained with reinforcement learning (RL). Finally, we utilize the re-organized context reason. This enables LLMs to deeply understand the context by clearly perceiving these logical relationships, facilitating the quality and reliability of reasoning. Besides, our information re-organization method can be integrated with existing prompt methods, like CoT (6) to further improve the reasoning ability of L.Mss. To verify the efficacy of our proposed InfoRE method, we conduct experiments using Llama2-70B [2). GPT-3.5 M. and GPT-4 [3] on various contextually aware multi-hop reasoning tasks, including claim verification [2]. question answering [13), and reading comprehension [14]. Using only zero-shot setting, our method achieves average improvement of 4% across all tasks, highlighting its potential to improve the reasoning performance of LLMs.
Our main contributions are as follows:
In contrast to existing methods that primarily focus on refining the reasoning steps to enhance the reasoning capabilities of LLMs, we take a novel direction of context re-organization.
The re-organization method initially uncovers the logical relationships that encompass multi-hop connections in the contextual content by extraction, and subsequently minimizes noise by pruning.
Experiment improvements on contextually aware multi-hop reasoning tasks across claim verification, question answering, and reading comprehension show the efficacy of our proposed method.
2 Related Work
have revolutionized the field of natural language processing (NLP) and demonstrate remarkable proficiency across range of linguistic tasks. To further improve
Reasoning with LLMSs L.MMS 1

---

the reasoning ability of LLMs has attracted considerable interest. In-context learning [ as promising approach enhance the reasoning abilities of LLMs, has been verified in mathematical reasoning task [6]. In addition, CoT [ incorporates coherent series of intermediate reasoning steps to improve the reasoning ability of LLMss. Following this paradigm, ToT 7 and GoT [ have also been proposed to focus on improving the structure of the intermediate chain. Then self-consistency [15] and plan-to-solve [15] focus on the reliability of the chain. Recently, the step-back prompting [16] is proposed, which obtains the high-level concept and first principles from instances by abstraction the first step, then guides the reasoning of LLMs with the obtained concept and principles. For tasks that require multi-hop reasoning, current approaches [17] 18] generally decompose multi-hop problems into simpler sub-tasks problems relying on the in-context learning ability of LLMs. In contrast to existing methods, we take novel direction of context re- or-gaaiization to enhance the reasoning capabilities of L.Mss for contextually aware reasoning tasks.
Information Re-organization Information re-organization is technique that leverages other structures to improve the clarity and comprehensibility of information. Moreover, it can also reveal conceptual relationships implicit in the original textual text, making valuable strategy for various tasks. For example, SG [19] re-organizes the document to scattered knowledge graph triples for explainable multi-hop question answering. GraphRAG [00 build community hierarchy and generate summaries for these communities after extracting knowledge graph, then leverage these structures on RAG-based tasks. MindMap [T0 is a powerful structure method for representing knowledge and concepts, it can be used to construct a hierarchical abstraction of natural language text (21). Previous method [22] uses it to organize large amount of scientific material enhance the clarity of the text material. Different from the scattered knowledge graph triples used in SG [19]. MindMap is more centralized, which aggregates content related to the same topic together. In this paper, we use the structure of re- re-organized information to uncover the logical relationships and multi-hop connections implicit within the plain context with one step, as opposed to GraphRAG 20 separately aggregates information with multiple steps.
3 Methodology
In this section, we first give formulation of a context-awan reasoning task in $3.1 and then describe our method in detail. As shown in Figure 2 the framework of our method consists of two components, an information re-organization $3.2 and reasoning step using the re-organized context $3.3
Context: Julius Caesar is a 1953 epic Metro-Goldwyn. Mayeddyy----yrrrraee film adapiation of the play by Shakespeare, directed by Joseph L. Mankiewicz, who also wrote the uncrodited screenplay, and produced by John Houseman...
was born on September 22, 1902, Bucharest, Romania... He was educatod in England Clifton College, bocame a British subject, and worked the grain trade in London before emigrating to the United States 1925, where took the stage name of John Houseman. became United States citizen in 1943...
Question: Where did the producer of Jullus Caesar study or work?
Re-organization
Extraction
Train with RL
Reward (F1)
J Julius Caesar:
Production Company: Metro- Metno-Goldyyn- Mayer
Pruning (BERT)
LLM (eg.GPT)
Reasoning
A ccording to the Re- organized information, under the tag of Education, J John Houseman, the producer of Julius Caesar, stadied Clifton College, England. y
Director: Joseph Mankiewicz
Producer:
Name: I ohn H Houseman
Julius Caesar:
Education: Clifton College, England Occupation: Grain Trade, London
Producer:
Name:
John Houseman
ddaptation: Play by Shakespeare
Education:
Clifton College, England
Figure 2: Illustration of our information re-organization method with two modules: Innorrmtioon Re- Re-Orranization, which includes logic relationship extraction and noise pruning. 2) Reasoning using re-organized context. The re-organized context in black italicized text is relevant to the question.
3.1 Task Formulation
Given a sample (c, (c,g,a) from sample space (C, 2, where denotes gold context, denotes question, the task aims to obtain answer a to the question g, based on gold context e with a large language model. In particular, it requires multi-hop reasoning to get the answer to question g.
3

---

3.2
Information Re-Organization
The purpose of the information re-organization is obtain the logical relationships from the context and minimize the noise irrelevant to the question. This goal is achieved through two operations: extraction and pruning.
3.2.1 Extraction
For a question if the derivation of its answer a relies on context c, then a deeper understanding of c crucial. Logical relationships, such as parallelism, causal connections, contrasts, etc., are essential elements of understanding and reasoning [9]. However, the logical relationships in the plain context are often implicit. Therefore, we perform extraction operation on the plain context to uncover the logical relations in it using language model. The process can be defined as:
g fo P...)
(1)
where fo(+) is language model parameterized by 6, P.n. is input task prompt. The specific task prompt P.n. used in our paper is displayed in the Appendix [A] We perform Equation for each c in the sample space to obtain the extracted context space G. We use the MindMap [T] structure to display the reorganized content, because it serves as powerful structure for representing knowledge, concepts, and perspectives, contains not only logical relationships but also multi-hop connections.
As shown in Figure 2 (bottom left), the extracted context g E g contains not only parallel logical relationships, e.g., Produce,&&roducerr but also causal relationships, e.g., Julius Direetr Furthermore, it also describes the three-hop connections, such as Julius Deiesaa--iieetor---- Julius Caesar-+ dareetor-+Occupattin and Julius Caesar --Drrertor-Eduueeee with logical relationships enables LLMs to deeply understand the contextual content by clearly perceiving these logical relationships, facilitating the quality and reliability of reasoning. Additionally, this multi-hop connection corresponds to the complex multi-hop problem and therefore helps to solve this multi-hop question y.
This information
3.2.2 Pruning
As described in Section 3.2 the extracted context E G contains various logical relationships and rich attributes. However, not all logical relationships and attributes help answer the question q. On the contrary, some may even interfere with the response to the question. For example, consider the question in Figure 2 the content "Julius Caesar -> Production Company" is distracting element, and "Julius Caesar > Adaptation" is irrelevant to the question.
To further reduce the interference of distracting irrel-
evant logical relations and attributes on the retrieval of answers for question 9, we use a pruning model trained through reinforcement learning (RL). The pruning model based on the pre-trained BERT [ due its high gen- cralizability, as shown in Figure 3 Its input consists of concatenated logical relationships, their corresponding at- tribute values from g. and the question q. For example, to prune the relation Julius Caesar -> Adaptation, the input is: [CLS] Julius Caesar Adaptation Play by Shakespeare (SEP) Question [SEP]. We give a detailed demonstration of input format in Appendix B
Action Probabilities
Softmax
Linear
ecLS
Cr
eers
Crz
Cra
eq
esep
BERT
[CLS] relation attributes [SEP] [CLS] mlaaion attributes [SEP] Question [SEP]
RL Formulation We formulate the pruning policy model optimization as RL problem and employ proxi- mal policy optimization (PPO) [23]. The action is keeping deleting the logical relations. The policy decides the action probability given the question and g. We fine-tune the policy model ı by optimizing the reward
Figure 3: Illustration of Pruning model. The representation of [CLS] is used to obtain action probabilities.
[r] =
Reward Function Our goal to maximize LLM's generation toward the desired target by an alignment measure R, and we use this as reward. In this paper, the alignment metric we have

---

chosen is the Fl score. To keep the policy network π from moving too far from the old result, we add a clipped term in the reward. Therefore, the final reward becomes:
r(z ②) min (R(z, q), clip([T(= = a; q), -, +))
where c is a hyperparameter indicating the range of CLIP to be performed. After pruning, the extracted contextual content becomes the context g which closely related to the reasoning objective q. We perform a pruning operation for each E 9 to obtain the pruned context space G'.
3.3 Reasoning
After the re-organization process in Section 3.2 we get the re-organized context g E G Then the re-organized context g can be used as a context alone or combined with the original context e get the final answer of The reasoning process can be defined as:
solg [d] Pz)
where P. denotes prompt, and the content within [ denotes that it is optional. The specific prompt used reasoning is displayed in the Appendix
4 Experimental
4.1 Tasks and Datasets
To verify the effectiivnesss of our information re-organization method, we conduct experiments across a range contextually aware multi-hop hopp reasoning tasks and datasets, including claim verification [12],
question answering [13], and reading comprehension [14. The detailed dataset information, including
data splits and statistics, available in Appendix D
Claim Verification The task involves assessing given claim against a set of evidence documents to
determine whether they support or refute the claim [12]. We consider HOVER 24 and FEVEROUS
[25]. which comprise complex claims that necessitate multi-hop reasoning for verification. Besides, we also take into account the SCIFACT 26 dataset, notable for its inclusion of scientific claims.
Question A Aswering For this task, we consider the following datasets: 2WikiMultiHopQA (27).
Strategy QA (28). MuSiQue [29]. and HotpotQA [30]. To answer the questions in these datasets requires not only multi-hop reasoning but also cross-document analysis.
Reading Comprehension Machine reading comprehension task requires model to process docu- ments and select an answer from the provided candidates to a question about the content [14]. We primarily consider WIKIHOP B1 in the task, which necessitates multi-hop reasoning to derive the final answer. Additionally, HotpotQA [3) is also frequently considered as part of the QA domain.
4.2 Baselines
In our paper, we compare our method InfoRE to two reasoning baselines: 1) Standard. 2) CoT. The Standard approach is method that directly reasons with the original textual context. The CoT 6 method involves augmenting standard reasoning methods by adding step-by-ster thought process. In our paper, we conduct the CoT strategy by appending the sentence "Let's think step by step." at the end of the question.
The baseline methods and InfoRE both adopt zero-shot setting counteract the potential randomness associated with demonstrations in a few-shot setting. We also design an answer-format instruction within the prompts for various tasks to standardize the structure of the final answer, thereby enhancing the precision of answer extraction. Moreover, all results reported in the paper use only the reorganized contextual information reason. Comprehensive details about prompts and answer-format instruction are available in Appendix C Following previous methods [17 [8 25 we run the official evaluation scripts of each dataset get the F1 measure the results.
4.3 Implementation Details
In our paper, the L.MSs employed in the extraction and reasoning process include Llamm2-70B (2). GPT-3.5 (text-davinci-003) [32] and GPT-4 [3]. We use the official version of Llama2-70B The
5

---

specific version of GPT-4 is GPT-4-0613. We configure all models with top_p parameter as 1.0 and temperature as 0.0. In the policy model, we use the BERT-base version all tasks and datasets. In RL training, we calculate the F1 score between the generated answer and the reference answer as the reward, with rescaling coefficient of 10. We train the model for 1000 episodes. We conduct training for epoch 5, batch size of 4, and learning rate of 2e-6. The parameter of is set to 0.2. All experiments are conducted on an NVIDIA RTX A6000.
Results and Analysis
5.1 Main Results
Claim Verification Table poolepreeentss a comprehensive performance comparison between our InfoRE and existing zero-shot techniques. For the HOVER dataset, we segment it into the 2-hop, 3-hop, and 4-hop levels following previous methods [17. As depicted in Table T our InfoRE demonstrates
significant improvements in the zero-shot claim verification task. The CoT
[6] approach offers
lightly increase 0.62% in the HOVER 4-hop using GPT-4, which indicates its marginal utility in more complex reasoning scenarios. Yet, our InfoRE achieves 3.02% improvement the HOVER 4- hop using GPT-4 showing remarkable performance on contextual aware understanding and reasoning. This improvement is further increased to 73.62% in combination with CoT, suggesting that the methods complement each other effectively. In the case of Llama2- 70B, the combined application of InfoRE and CoT yields score of 53.20% on the 2-hop HOVER task, surpassing its CoT-only score of 50.02%. This pattern of improvement is consistent with GPT-3.5. GPT-4 shows superior performance across all methods and datasets, suggesting inherent advanced reasoning ability. This performance mirrored in the specialized benchmarks, where GPT-4 with InfoRE attains near-perfie serr-pe accuracy on FEVEROUS (95.62% and very high accuracy SCIFACT (93.67%). These figures solidify the notion that the InfoRE indeed enhances the reasoning capabilities of LLMs.
Table Zero-shot performance on claim verification task across three LLMs.
HOVER
LLMs
Methods
FEVEROUS
SCIFACT
2-hop
3-hop
4-hop
Standard InfoRE
49.41
48.35
47.82
63.39
60.70
LLAMA2 (70B)
52.83
51.42
50.04
67.84
63.81
3.42
t 3.07
2.22
4.44
3.11
CoT InfoRE + CoT
50.02
48.76
48.01
64.53
61.24
53.20
51.70
50.15
68.12
64.02
3.188
t 2.94
2.14
3.59
2.78
Standard InfoRE
64.74
63.04
61.54
87.67
77.42
68.21
66.45
64.91
91.31
81,54
GPT-3.5
1 3.47
3.41
3.37
3.64
1.122
CoT InfoRE + CoT
66.70
64.52
62.69
88.67
78.49
69.02
67.53
65.66
91.53
82.26
2.32
3.01
2.97
2.86
3.77
Standard InfoRE
72.40
71.02
70.06
92.33
91.40
75.87
74.06
73.08
95.62
93.67
GPT44
1 3.47
3.04
3.02
3.29
2.27
CoT
73.82
72.07
70.68
92.67
92.47
InfoRE CoT
76.69
75.16
73.62
95.67
94.32
1 2.87
3.09
2.94
3.00
1.85
Question Answer and Reading Comprehension Different from claim verification task, this task involves using multiple documents context, presenting a challenge in cross-documen reasoning. Intuitively, when the reasoning process involves multiple documents, information re-organization can effectively merge the information from different documents and uncover logical relationships that are not apparent in plain text. Performance Table 2 verifies the intuitive. We can see that after applying information re-organization, all the results have significant improvement. GPT-4 outperforms

---

the other models. with the highest F1 score of 76.52%, 71.20%, and 83.22% when employing InfoRE on 2WikiMuliHHoQAA, StrategyQA, and HotpotQA, respectively. This consistent with the performance improvements observed with GPT3.5 (text-davinci-003) and Llama2-700B when applying InfoRE. Different from conventional QA tasks, reading comprehension tasks require L.LMs to not only deeply understand the context but also identify distractors among the candidates, increasing the reasoning challenge. Our InfoRE consistently shows improvements on this task. Specifically, in the WIKIHOP dataset, G with InfoRE outperforms other methods with an Fl of 51.87%. This improvement further verifies the effectiveness our method.
Table 2: Zero-shot results on Question Answering and Reading Comprehension tasks. 2WMHQA, SQA, and HQA are abbreviations for 2WikiMuliHHopQA, StrategyQA, and HotpoiQA, respectively,
LLMs
Methods
2WMHQA
MuSiQue
SQA
HQA
WIKIHOP
Standard InfoRE
52.56
49.55
51.23
66.07
40.32
LLAMA2
57.62
52.78
55.32
69.98
42.90
(70B)
5.06
3.23
1 4.00
3.91
1.288
CoT InfoRE + CoT
52.99
52.90
56.80
66.80
41.07
57.72
56.10
59.93
70.60
43.37
4.73
3.20
3.13
1 3.80
2.30
Standard InfoRE
58.25
55.01
59.39
73.30
48.92
64.58
58.03
63.16
77.12
51.87
GPT-3.5
1 6.33
3.02
1 3.77
3.82
2.95
CoT InfoRE + CoT
59.37
57.05
67.51
73.90
49.65
65.13
60.52
70.45
77.74
52.70
1 5.76
3.47
1 2.94
3.84
3.05
Standard InfoRE
72.69
62.65
68.32
79.33
55.46
76.52
66.36
71.20
83.22
58.01
GPT-4
1 3.83
3.71
1 2.88
1 3.89
1 2.55
CoT InfoRE + CoT
74.08
64.36
68.50
80.66
56.02
78.60
69.11
71.54
84.26
58.91
4.52
4.75
3.04
1 3.60
2.89
5.2 Analysis
Ablation studies In our paper, the re-organization comprises two components: extraction and pruning. To investigate the impact of each component in detail, we conduct a series ablation experiments using GPT-3.5 on the 2WikiMuliHHpQAA dataset. First, we directly remove extraction and pruning from our method, and the results are shown in the second and third rows of Table 3 respectively. It is worth mentioning that in the experiment where extraction is removed, we directly prune the sentences in the original context. Furthermore, we replace the reinforcement learning- based pruning method with similarity-bar based pruning method to demonstrate its effectiveness. Specifically, the similarity- based pruning method uses Siamese- BERT, which takes the original question and each logical rel relationship as inputs separately, and then generates the corresponding representations. Then, we calculate the cosine similarity between these two representations. Finally, we removed the 30% of logical relationships with the lowest similarity, the result is shown in the last row of Table 3
The results in Table show that removing the ex- traction and pruning operations leads to perfor- mance drops of 2.94% and 1.53%, respectively. This demonstrates the effectiveness of both com- ponents in our methods. The larger performance drop after removing extraction highlights the im- portance of extracting logical relationships for effective reasoning. After replacing the pruning model, the performance dropped by 1.26%, but the results were still better than without pruning.
Table 3: Fl performance of ablation studies.
Methods
2WikiMultiHHopQA
Full model
64.58
w/o extraction
61.64
w/o pruning
63.05
similarity-based pruning
63.32
7

---

This not only demonstrates the necessity of pruning but also highlights the effectiveness of the reinforcement learning-biis based pruning method.
Quality of Re-Organized Information To assess the quality of the re-organized information, perform a quantitative evaluation of the re-organized information with GPT-4 (gpt-4-32k) on the 2WikiMuliHopQA dataset. Specifically, we select 100 samples from the dataset, GPT-4 (gpt-4- 32k) is asked to rank re re-organized information produced by GPT-3.5 (text-davinci-003 [[] and GPT-4, as well as original context following criteria: (1) Depth: The information present multiple relationships of topic, offering insightful perspectives or in-depth understanding of the subject. (2) Clarity: Information clear and precise, making easy to understand without ambiguity. All of the information is ranked 1, 2, and 3 with 3, 2, and scores, respectively. Finally, get weighted average score for each information to measure the overall quality.
The results in Table 4 demonstrate that the re- organized information outperforms the original textual context in terms of depth and clarity, which justifies the motivation of our paper. In ad- dition, the re-organized information from GPT-4 outperforms GPT-3.5, which proves in another way that GPT-4 is indeed more capable than GPT-3.5.. The evaluation results further validate the effectiveness of our method. Furthermore, the 22.22% improvement in depth more ob- vious than the 15.14% improvement in clarity, which indicates that the re-organization of the information has been particularly effective in enhancing the logical relationships the infor- mation. The improvement in clarity suggests that the information is now presented in a more direct and streamlined manner, making it easier for LLMs to grasp the essential points without wading through unnecessary details.
Table 4: Qualitative evaluation results 2Wiki- MultiHopQA dataset. Avg R denotes the weighted average ranking score. The larger ranking score denotes better information quality.
Depth
Methods
2nd
3rd
Avg
Original GPT-3.5 GPT-4
0.22
0.36
0.42
1.80
0.32
0.36
0.32
2.00
0.46
0.28
0.26
2.20
Clarity
Methods
1st
2nd
3rd
Avg R.
Original GPT-3.5
0.25
0.35
0.40
1.85
0.35
0.32
0.33
2.02
GPT-4
0.40
0.33
0.27
2.13
Effect of Re-organized Information Quality
To explore the effects the quality of re-organized context on model reasoning capabilities, we utilize both GPT-3.5 (text-davinci-003. and GPT-4 to re-organize context information. Reasoning processes are then independently executed on each model. Moreover, employing this cross-validation technique also allows effectively evaluate the robustness of method.
The cross-validation results, shown in Table 55 indicate that the use of ei- ther GPT-3.5 or GPT-4 for informa- tion re-organization leads to marked improvement in the performance of LLMs reasoning tasks. Addition- ally, it is observed that using GPT-4 for information re-organization while reasoning with the GPT-3.5 results in further increase in reason results by 1.19% and 2.03% on FEVEROUS and 2WiiiMultHHoQAA datasets, respec- tively. This implies that enhancing the quality of re-organized informa- tion can improve the performance of LLMs, pointing to a promising direc- tion for future research in refining in- formation synthesis and re-organization. Conversely, when GPT-3.5 employed for information re-organization in conjunction with reasoning using the GPT-4 model, there is a decrease in reason ability by 0.95% and 1.45% respectively. addition, the magnitude of the decrease lower than the increase, which implies that re-organization strategy may have greater impact on models with
Table 5:
Fl performance of cross- validatio,,,,, where InfoRE' denotes reason with GPT-3.5 but information re-gaaniizttion with GPT-4, InfoRE' denotes reason with GPT-4 but infor- mation re-organization with GPT-3.5 (text-davinci-003)
Methods
FEVEROUS 2WikiMultiHopQA
GPT-3.5 (t)
Standard
87.67
58.25
InfoRE
91.31
64.58
InfoRE
92.50
66.61
GPT-4 (*)
Standard
92.33
72.69
InfoRE
95.62
76.52
InfoRE
94.67
75.07
8

---

weaker reasoning abilities. This finding aligns with our understanding that models with inherently weaker reasoning abilities tend to rely more heavily on external strategy. For models with stronger inherent capabilities, our method still further improve its reasoning ability.
Error Analysis To better comprehend where the errors in our InfoRE methodology come from and where they are fixed, we annotate 100 wrong predictions made by both InfoRE and Standard methods with GPT-3.5 on 2WikiMuliiHoPQA dataset. We categorize the errors into 4 classes:
Contextual Misunderstanding (CM): This happens when the model fails to interpret or connect multiple pieces of information from different parts of the documents. Multi-hop reasoning requires synthesizing information from various segments, and recognizing logical relations, and any misunderstanding can lead to incorrect conclusion.
Factual Error (FE): The model may provide an answer that is factually incorrect or not supported by the given documents. This is often due to the model's reliance on its training data, which may not always align with the specific facts in the context.
Mathematical Error (ME): The error occurs when math calculations are involved in deriving the final answer.
Unanswerable Question (UQ): It's a specific type of error or limitation in dataset design, where the context does not contain enough information to provide valid answer to the posed question.
When engaging in reasoning with LLMs, all four error categories are present. As shown in Figure there are 6% unanswerable question
0.80
Standard
InfoRE
errors in the dataset, and more than 90% errors
0.70
0.80
occur in the reasoning in baseline method.
0.60
0.67
Among the four types of error, contextual mis-
0.50
understanding is the primary source of errors
in the baseline, highlighting the importance of
0.30
an in-depth understanding of context in solv-
0.20
ing reasoning tasks with LLMs. This finding
0.10
0.01
0:13
0.05
0.14
0.00
is consistent with our motivation presented in the introduction section. Moreover, comparing the results of errors between our method and the baseline method, our method mainly cor- rects 14% of errors coming from the baseline method, most of the corrected errors are textual misunderstanding errors. This further indicates that our InfoRE method assists LLMs in understanding context, signifying the neces- sity and effectiveness of conducting information re-organization before directly addressing the original question.
CM
FE
ME
UQ
Corrected
Figure 4: Error Analysis of InfoRE 2Wiki- MultiHopQA against Standard baseline method. The first four rectangles are error categories, while "Corrected" on the far right denotes the percentage of errors originally made by the baseline method that our method InfoRE has successfully corrected.
Conclusion
In this paper, we propose an information re-organization method improve the reasoning ability of LLMs.. Compared with previous approaches primarily focus on improving the quality of intermediate steps. our method emphasizes uncovering the logical relationships, multi-hop connections, and pruning the irrelevant information through information re-organization. This approach enables LLMs explicitly perceive the logical relationships and multi-hop connections of concepts within the context, promoting deeper integration and understanding of the context, which results in more robust reasoning outcomes. To verify the effectiveness of our method, we conduct experiments using various LLMs across range of contextually aware multi-hop reasoning tasks. The experiment results demonstrate the potential of our method improve the reasoning ability of LLMS.. Additionally, our method has a positive impact on various tasks involving context understanding, such as academic research, legal analysis, and medical diagnostics. However, it also important to be aware of potential negative impacts, such as the propagation of misinformation.

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (No. 62376245), the "Pioneer" and "Leading Goose" R&D Programs of Zhejiang (No. 2024C03255), the Fundamental Research Funds for the Central Universities (226-2024-00170) the project of the Donghai Laboratory (Grant no. DH-2022ZY0013), National Key Research and Development Project of China (No. 2018AAA0101900) and MOE Engineering Research Center of Digital Library.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. I Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URI httphtttt///rrrrcedinggs newr.p............e............. cc/ wcc/paper__. f files/ Daape__filessppper//20000 pappr//020/faleee
[2] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini. Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
09288
[3] OpenAL. Gpt-4 technical report, 2024. URL
[4] Konstantine Arkoudas. Gpt-4 can't reason, 2023. URL http://arxiv/ http://xxivv.rr/aa//20200 abs/ 03762
[5] Andrew Blair-Stanck, Nils Holzenberger, and Benjamin Van Durme. Can gpt-3 perform statutory reasoning? In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL '23, page 22 31, New York, NY. USA, 2023. Association for Computing Machinery. ISBN 9798400701979.
doi: 10. 10.1145/35945363355566 URL https
//doi. 1145/359453366 3594536 35951633
[6] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of tfouuhtthtttttt prompting elicits reasoning large language models. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824 24837. Curran Associates, Inc., 2022. URL httpt:tt//oooceeetttetioo mew.eeeddddngneeeuippepeeee...ee cc/ http://aarrfflle///e/ppp. / Coraaperr--oofeeenee.ee pdf
[7] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I.. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliherate problem solving with large language models, 2023.
[8] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski. Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 38616) Mar 2024. doi: 10. 609/aaai. vaaaaaaii.viii11 ww8.1..29722202 URL. httpt:://///// aaai org/ index, phppp/AA articlevviess v
10

---

[9] M. D. S. Braine. On the relation between the natural logic of reasoning and standard logic. Psychological Review, 85:1-21, 1978. doi: 10.1037/0033-295xx
[10] Tony Buzan, Barry Buzan, and James Harrison. The mind map book: Unlock your creativity, boost your memory, change your life. 2010. URL httLhhtts//appe http::://app.eeeatttccccooooooo www.seeimmcmoooooooooo.o
org/
It:pusID:14198600
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 00.18653/v1/N14 1423.
URL latts://///c http://cccaaahooooooo.oooooo hwwpollloyoooooooooooooooo org/N N19- 1423
[12] Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. survey on automated fact- checking. Transactions of the Association for Computational Linguistics, 10:178- 2022. doi: 10.1162/taclaa0000 a
org 2022, tac1-1.11,
[13] Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho. and Douwe Kiela. Unsupervised question decomposition for question answering. In Bonnie Webber, Trevor Cohn. Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864-8880 Online, November 2020. Association for Computational Linguistics. doi: 1//0////1//220mmmmpmmmooo
URL https://
aclanthology wttg/022.emmmmpmmaaiii.. emnlp-main.n...... 713
[14] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2368- 2378. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. N0.18653/v1/111112200 1246.
org/N19- 1246.
[15] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisenncy improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: net/ DdF7id-1PLLNINWWE
[16] Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng- -ze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. Take step back: Evoking reasoning via abstraction in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
[17] Liangmingg Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min- Yen Kan, and Preslay Nakov. Fact-checking complex claims with program program-guided reasoning. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6981-7004, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.1 00.18653/v1/2033 www.2220222.l-looo...2 URL http:/tttps://ccaattin hww.etthhoooooooogo/oooo 2023 2 ac1-1ong. 386
org/
[18] Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Her- nandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilé Lukošiūté, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer Showk, Tamera Lanham, Tim Maxwell, Venkatesa Chandrasckaran, Zac Hatfield- Hatficld-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. Question decomposition improves the faithful- ness of model-generated reasoning, 2023, URL httt:://t:::iii org 2207. 11768
[19] Ruosen Li and Xinya Du. Leveraging structured information for explainable multi-hop question answering and reasoning. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5779-6789, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10. 10.186611/2222.. findings-emmlp.452 452. URL htttp:://aclataaaaate org. Dtl.a//ree//ccaaaaoooooor//g///p findings emnlp. 452

---

[20] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach query-focused summarization, 2024. uttpstts::// http///axvvvv.. org/ abs 2404. 16130
[21] Mohamed Elhoseiny and Ahmed M. Elgammal. Text to multi-level mindmaps: A new way for interactive visualization and summarization of natural language text. CoRR, abs/14088.n 1031,
2014. BRLLhtt://aar// org/ abs/aa/s/ 1031
[22] Theodore Dalamagas, Tryfon Farmakakis, Manolis Maragkakis, and Artemis G. Hatzigeorgiou. Freepub: Collecting and organizing scientific material using mindmaps. ArXiv, abs/1012.1623, 2010. /ttts://://app wwwanticschol......... org/ Corpus ID: 12397418
[23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL http:////aaxi org/ abs/ 0////17770004444 06347
[24] Yichen Jiang. Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. HoVer: A dataset for many-hop fact extraction and claim verification. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguisties: EMNLP 2020, pages 3441-3460, Online. November 2020. Association for Computational Linguisties. doi: 10.18653/v1/20200 hun.://5ww1.//0200mnnnnnemmmmmpp 2020. windinnnnnnnnnnlppp 309
URL http::://aalaattollllooooo/ org/
[25] Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thoree, Andreas Vla- chos, Christos Christodoulopoulos, Oana Cocarasou, and Arpit Mittal. Feverous: Fact extraction and verification over unstructured and structured information. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Pro- cessing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL cew.pps.cc/papprrr.. pdff
[26] David Wadden, Kyle Lo, Bailey Kuchl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannanch Hajishirzi. SciFact-open: Towards open-domain scientific claim verification. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7719-4734 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 http://.......nnsooeooooo Anddings-emnl..477 URL https::/acclanthollggg org/ 2022. www.ffffiiinnnene emnlp 347
[27] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.186633//02222oool maingn-mmai.55800
URL htRLhttt:///acaatoolooo www.g..orgggg. wttp:///00/o.ooooogooooooooooee. main.580
[28] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? question answering benchmark with implicit reasoning strategies, 2021.
02235
[29] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022. doi: 10. 10.1162/tacl_aa0000 Lacpltaclla_004770 URL https
/ //aclanthology. org. htrg/2222.5eom tacl...........1
[30] Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun innichhi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium, Linguistics. doi: 10.18653/v S006553v1/D11/ 1/D18- 1259. URL httpps:
2018, Association for Computational
wwwoorrg////// 1259
[31] Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi- hop reading comprehension across documents. CoRR, abs/1710.06481, 2017. URL http! //arxiv. org/aog.///..ooooooooo /0b0/ 06481
12

---

[32] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL http:///a//a org/ abs/ab/s.3 02155
[33] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Natural Language Processing
pages 3730-3740 3740, Hong Kong, China, November 2019. Association for Computational Linguistics.
International Joint Conference
doi: 10.18653/v1/D19-1133 13377 1387. URL
13

---

A Prompt for Extraction
The specific prompt of extraction is shown in Table 6 The purpose of the prompt is to obtain logical relationships and multi-hop connections from the context. In the prompt, the content within the brackets should be replaced with specific example content.
Table 6: Specific prompt for obtaining logical relationships. During execution, "[EVIDENCE]" needs to be replaced with the specific document, and "[CLAIMM]" replaced with the specific question.
Prompt Content for Logical Relationship Extraction
Given claim and corresponding evidence, please summarize the evidence as mind map according to the claim.
The output must be in a strict JSON format: ("mind map": "mind map"). CLAIM: [CLAIM]
EVIDENCE: [EVIDENCE]
To clearly demonstrate the content of the LLM after performing the extraction, we give an example in the MuSiQue dataset.
Document: Julius Caesar is a 1953 epic Metro-Golldyn-Mayer film adaptation of the play by Shakespeare, directed by Joseph L. Mankiewicz, who also wrote the uncredited screenplay, and produced by John Houseman. The original music score by Miklos Rozsa. The film stars Marlon Brando Mark Antony, James Mason Brutus, John Gielgud
Cassius, Louis Calhern as Julius Caesar, Edmond Brien as Casca, Greer Garson as Calpurnia, and Deborah Kerr as Portia. Houseman was born September 22, 1902, in Bucharest, Romania, the son of May (Davies) and Georges Haussmann, who ram a grain business. His mother was British, from a Christian family of Welsh and Irish descent. His father was Alsatian-born Jew. He educated in England Clifton College, became a British subject, and worked in the grain trade London before emigrating to the United States in 1925, where he took the stage name of John Houseman. He became United States citizen in 1943.
JSON format after extraction:
"Julius Caesar {
Production Company "Metro Mettro-Goldwyn-aayee "Adaptation ": "Play by Shakespeare".
"Director": "Joseph L. Mankiewicz", " "Screenplay":
"Writer": "Joseph L. Mankiewicz", " "Credit": "Uncredited"
Producer";
"John Houseman": ":
Birth "Date "September 22, 1902",
"Place": " "Bucharest, Romania"
Parents
"Mother"
"Name", "May Davies". "Nationality ", "British",
"Religion" ": "Christian"
"Descent". " "Welsh and Irish"
Father
"Name": "Georges Haussmann", "Occupation" ": "Grain Business", "Religion"; "Jew".
14

---

"Birthplace": "Alsace"
} "ddccttion" ": "Clifton College, England", Citizenship"
"Original ": "British",
"Naturalized": "United States, 1943"
Occupation":
"Original ": "Grain Trade, London",
"Stage Name": "John Houseman"
Emigration": "United States, 1925"
Music Score "; {
" "Composer"; "Miklos Rozsa"
"ast":
"Marlon Brando ": "Mark Antony",
" "James Mason ": " "Brutus",
"John Gielgud". " "Cassius"
" "Louis Calhern". "Julius Caesar",
"Edmond O' Brien "Casca",
Greee Garson": "Calpurnia",
"Deborah Kerr": "Portia"
B Input Format For Pruning
The extracted context g includes logical relationships and corresponding attribute values. First, we iterate through all logical relationships and attribute values, and following the previous method [33] we concatenate them using (SEP) [CL.S], combining them with the question to input the pruning model. Then, we use the representation of the [CLS] token to represent the logical relationships for subsequent operations.
We use an example in Figure 2 to demonstrate the input format of the extracted context g in the pruning model. The question is: Where did the producer of Julius Caesar study or work? The the extracted context g is:
Julius Caesar:
Production Company: Metro-Goldwyn-
Director: Joseph L. Mankiewicz
Producer:
Name: John Houseman
Education: Clifton College, England
Occupation: Grain Trade, London
Adaptation: Play by Shakespeare
We only traverse logical relationships of the first-level progressive type. Therefore, the logical
relationships contained in extracted context g of example include 1) Production Company: Metro- Goldwyn-Mayer, 2) Director: Joseph L. Mankiewicz, 3) Producer: Name: John Houseman, 4)
Producer: Education: Clifton Englang,England, 5) Producer: Occupation: Grain Trade, London, 6)
Adaptation: Play by Shakespeare. Then, we concatenate them using [SEP][CLS].
The input format after concatenation [CL.S] Production Company: Metro-Goldwyn-Me [SEP]
[CLS] Director: Joseph L. Mankiewicz [SEP] [CLS] [CLS] Producer: Name: John Houseman [SEP]
15

---

[SEP] [CLS] Producer: Education: Clifton College [SEP] [CLS] Producer: Occupation: Grain Trade, London [SEP] [CLS] Adaptation: Play by Shakespeare [SEP] [CLS] Where did the producer of Julius Caesar study work? [SEP].
C Prompt for Multi-hop Reason
During the reasoning stage using large language models, to accommodate more context-aware reasoning tasks while ensuring comparability of results, we designed universal prompt template. The prompt template consists of three components: original context, e.g., documents or paragraphs, reorganized information, and question. The prompt template is as follows:
Documents:
Drganiznd TEXT]
[TEXT] [0ptional]
Question:
[QUESTION]
please the question based on the documents.
Answer:
The specific prompts for Standard, chain-of-thougle (CoT), our InfoRE, and InfoRE CoT to reason are shown in Table 7 In the prompt, the content within the brackets [] should be replaced with specific example content.
Table Specific prompts and answer format instructions of Standard, CoT, InforRE, and InfoRE CoT in our paper. During execution, "[EVIDENCE]" needs to be replaced with the specific document, "[CLAIM]" replaced with the specific question, and "[MindMap]" is replaced with the specific MindMap.
Methods
Prompt Content
Documents: [EVIDENCE]
Question: [CLAIM]?
Please answer the question based on Documents.
Standard
Your final answer should be enclosed in XML tag <answer></answer, like this:
<answer> (fnnal_aswwer)</asssseesss at the end of your response.
Answer:
Documents: [EVIDENCE]
Question: [CLAIM]?
Please answer the question based on Documents.
CoT
Your final answer should be enclosed in XML tag <answer></answer,, like this:
<answer> canswer>[fmal (afffallanswertt [isrral__aseerr//aswwww.. at the end of your response.
Let's think step by step.
Answer:
Documents: [MindMap]
Question: [CLAIM]?
Please answer the question based on Documents.
InfoRE
Your final answer should be enclosed in XML tag <answer></answe>> like this:
<answer> (finall__waaaaaaswwaaa /twnss/wwww///w//////3333 </answer>, at the end of your response.
Answer:
Documents: [MindMap]
Question: [CLAIM]?
Please answer the question based on Documents.
InfoRE CoT
Your final answer should be enclosed in XML tag <answer></aswwr>> like this:
<answer> [final _assarr </answer/// at the end of your response.
Let's think step by step.
Answer:
16

---

D Detailed Dataset Information
In our experiments, due the resource limitations of large language models, we sample portion from each dataset, following previous methods 6 16
Table 8: Details statics information of evaluation datasets we used in the paper. Pairs denote the number of examples. 2WMHopQA is the short name of 2WikiMuliHHoQA..
Pairs
Tasks
Dataset
train
test
FEVEROUS
2000
2959
Claim Verification
HOVER
2000
4000
SCIFACT
200
212
2iikiMultHHoPQA
2000
500
Question Answering
StrategyQA
1000
229
MusiQue
2000
2417
HotpotQA
2000
500
Reading Comprehension
WIKIHOP
2000
500
Claim Verification The task involves assessing given claim against set of evidence documents determine whether they support or refute the claim [12]. We consider HOVER (24) and FEVEROUS [2]], which comprise complex claims that necessitate multi-hop reasoning for verification. Besides, we also take into account the SCIFACT (26) dataset, notable for its inclusion of scientific claims.
Question Answering For this task, consider the following datasets: 2WikiMulliHHpQA 27
StrategyQA QA (28). MuSiQue [29]. and HotpotQA [30]. To answer the questions in these datasets requires not only multi-hop reasoning but also cross-document analysis.
Reading Comprehension Machine reading comprehension task requires a model to process documents and select answer from the provided candidates a question about the content [14] We primarily consider WIKIHOP 3i in the task, which necessitates multi-hop reasoning to derive the final answer. Additionally, HotpotQA dataset [30 also frequently considered part of the question answering domain.
E
Limitations
propose an information re-organization approach to improve the reasoning of large language models, which performs well some context-aware reasoning tasks but still has some limitations. Firstly, the structures information re-organization are limited. Generally, there are multiple structures for information reorganization, such tables, timelines, etc. Next, will extend the re-organization structure to include more types. Secondly, the re-organization process relies on large language models. If method will become more generalizable. This is another direction we need to focus on the future.
implement this re-organization using smaller language models,
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract, Introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
Supplementary material
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuiu wwwticccguie//eeeeeeeeeii CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experimental
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:r/ps....ppuooot//cccccoaaaa
Answer: [Yes]
Justification: Supplementary material
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Conclusion
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Introduction, Related Work, Methodology, Experimental, Results and Analysis.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer: [NA]
Justification:
The paper does not involve the release of new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "oMxdknJImc", "reviewer": "Reviewer_wQkW", "summary": "The paper introduces an \"Information Re-organization\" (InfoRE) method aimed at improving the reasoning abilities of large language models (LLMs). It highlights the deficiencies of existing approaches that focus on intermediate reasoning steps without adequately addressing the preliminary organization of contextual information. The proposed InfoRE method involves extracting logical relationships and pruning redundant content from contextual data, guiding the reasoning process. This restructuring allows LLMs to recognize and use these logical relationships effectively, potentially enhancing reasoning quality. The method's efficacy is demonstrated using various LLMs like GPT-3.5 and GPT-4 across multiple reasoning tasks, showing an average improvement of 4% in performance.", "questions": "1. What are the differences between your extraction method and extracting to knowledge graph triples? Leveraging Structured Information ... (Li et al., EMNLP 2023) proposed a method to extract knowledge graph triples from documents. Moreover, you should add this paper to your related work since it also extracts knowledge from the document.\n2. Since generations from LLMs always change, can you provide a standard variance or significant score of your results to show your improvement is substantial?\n3. Do you limit the type of entities and relations during extraction? Do you apply open relations or closed relations during extraction? I only find the prompt in Appendix A. Is this all about your extraction part?\n4. How do you check the completeness and quality of your extraction results? Can you provide more details about this?\n5. LLM answers are always long, even with the format instruction. How do you compare your generated answers with golden answers? Do you apply exact match or other methods?", "limitations": "The paper adequately describes its limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel method of restructuring information before reasoning, which could be foundational for future developments in LLM reasoning processes.\n2. Results indicate that InfoRE significantly improves reasoning tasks across multiple datasets and model architectures.\n3. The paper provides a comprehensive breakdown of the method's components and their contributions to the overall performance.", "weaknesses": "1. Information extraction and pruning might require intricate tuning and be computationally expensive, making it less practical for real-time applications.\n2. The paper is well structured. However, authors should provide more details about their experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for reviewing our work and providing these valuable feedback and comments.\n\n**W1**: In our method, information extraction is implemented using closed-source or publicly available LLMs with frozen parameters, significantly reducing computational expense.\nThe primary resource-intensive component of our method is the BERT-based pruning model. However, it's important to highlight that this model contains only 110M parameters. This makes our approach less demanding in terms of computational resources. The frozen parameters LLMs-based information extraction and BERT-based pruning allow our method for real-time applications.\n\n**W2**: Due to the paper space limitation, we describe the primary details of the experiments in Section 4 (Experimental), including the statistical information of the datasets, the methodologies of the baseline approaches, the evaluation metrics, the detailed parameters, and the environment setting of the model training process. To enhance the clarity and understanding of our experiments, we also provide the code in the supplementary materials. In addition to these main experimental details, the other details of our experiments include: we use the Adam optimizer, and set the hidden size of the linear layer to 2048. In the revised version, we will include these experimental details in the appendix.\n\n**Q1**: The primary difference between our extraction and extraction into knowledge graph triples is that our extraction results are more centralized, whereas the knowledge graph triples are more scattered. The centralized extraction results of our method have two advantages. Firstly, our results aggregate content related to the same topic together. For example, as shown in Figure 2, all information related to the movie \"Julius Caesar\" is concentrated under the \"Julius Caesar\" node. Secondly, our extraction results include multi-hop connections, such as the path Julius Caesar → Producer→ Education in Figure 2, which is very helpful for answering complex the multi-hop question: Where did the producer of Julius Caesar study or work?. In the revised version, we will include a description of the differences between our method and the knowledge graph triples extraction method  (Leveraging Structured Information ... (Li et al., EMNLP 2023)) in the related work.\n\n**Q2**: The reported results in the paper are the means of the three times of each experiment. Based on the three times results of each experiment, the specific standard deviations for all datasets and LLMs involved in the paper are as follows:\n\n| LLMs | Methods | HOVER-2 | HOVER-3 | HOVER-4 | FEVEROUS | SCIFACT |\n|----------|----------|----------|----------|----------|----------|----------|\n| LLAMA2-70B    | InfoRE     | 52.83 &plusmn; 0.6     | 51.42 &plusmn; 1.64     | 50.04 &plusmn; 0.37     | 67.84 &plusmn; 0.16     | 63.81 &plusmn; 0.27     |\n| GPT-3.5    | InfoRE     | 68.21 &plusmn; 1.89     | 66.45 &plusmn; 1.53     | 64.91 &plusmn; 1.82     | 91.31 &plusmn; 0.57     | 81.54 &plusmn; 0.4     |\n| GPT-4    | InfoRE     | 75.87 &plusmn; 0.66     | 74.06 &plusmn; 1.62     | 73.08 &plusmn; 1.71     | 95.62 &plusmn; 0.25     | 93.67 &plusmn; 1.99     |\n\n\n| LLMs | Methods | 2WMHQA | MuSiQue | SQA | HQA | WIKIHOP |\n|----------|----------|----------|----------|----------|----------|----------|\n| LLAMA2-70B    | InfoRE     | 57.62 &plusmn; 0.52     | 52.78 &plusmn; 1.11     | 55.32 &plusmn; 1.77     | 69.98 &plusmn; 0.55     | 42.90 &plusmn; 0.12     |\n| GPT-3.5    | InfoRE     | 64.58 &plusmn; 1.62     | 58.03 &plusmn; 1.91     | 63.16 &plusmn; 1.31     | 77.12 &plusmn; 0.89     | 51.87 &plusmn; 0.32     |\n| GPT-4    | InfoRE     | 76.52 &plusmn; 1.1     | 66.36 &plusmn; 1.34     | 71.20 &plusmn; 1.44     | 83.22 &plusmn; 0.64     | 58.01 &plusmn; 0.93     |\n\n**Q3**: We don't limit the types of entities and relationships during extraction and apply open relationships. This is because closed relations restrict the relationships between entities to specific categories, making it challenging to express complex textual content. Limiting entity types further exacerbates this issue. We use a large language model based on the prompts in Appendix A for extraction. This extraction method was adopted after extensive analysis and experimentation. The extraction results aggregate content related to the same topic together and include multi-hop connections.\n\n**Q4**: In our experiments, we don’t separately assess completeness but instead focus on consistency. Specifically, we use GPT-4-32K to determine whether the extraction results are consistent with the original documents. If the results are found to be inconsistent, we re-execute the extraction process until consistent results are obtained. According to our statistics, we found that over 98% of the extraction results are consistent with the original text after the first extraction. This indicates that inconsistencies are rare. Due to the high consistency rate, we initially did not include this content in the paper. To improve the clarity of our method, we will include these details in the revised version.\n\n**Q5**: In our experiments, we use special tags <answer>{final_answer}</answer> to distinguish the final reasoning answer from intermediate results, as shown in Table 7. After using regular expressions to extract the final answer between the <answer></answer> tags, we combine it with the gold labels provided in the dataset and input them into the evaluation script to obtain the evaluation results. For the datasets 2WikiMultiHopQA, StrategyQA, HotpotQA, MuSiQue, and WIKIHOP, we run the official evaluation scripts provided by these datasets to get the evaluation results, including EM (exact match), F1, Precision, and Recall. For the HOVER, FEVEROUS, and SCIFACT datasets, we extract True or False predictions from final answer and use run classification evaluation scripts to get the evaluation results, including F1, Precision, and Recall.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oca3JKLxB0", "reviewer": "Reviewer_cK3D", "summary": "This paper proposes a method called information re-organization (InfoRE) to enhance the performance of large language models on some reasoning tasks. Unlike existing approaches that primarily focus on refining the reasoning process, InfoRE emphasizes re-organizing contextual information to identify logical relationships before reasoning. This method involves extracting logical relationships from the context and pruning redundant content to minimize noise. The re-organized information is then used in the reasoning process. The authors demonstrate the effectiveness of InfoRE through experiments on several contextually aware multi-hop reasoning tasks, achieving an average improvement of 4% in a zero-shot setting.", "questions": "See weaknesses.", "limitations": "I don't see potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of information re-organization is intuitive.\n- The demonstrated improvement in reasoning performance by an average of 4% across tasks, as well as the ablation study, verifies the functionality of the proposed method.\n- This paper is easy to follow.", "weaknesses": "- The idea of this paper can be connected to existing prompt engineering works, e.g., performing retrieval from context and query/context rewriting. I feel this paper lacks technical depth (though this may be the fashion of a prompt engineering work, i.e., insightful but lacking technical depth). And the insight is not significant enough to compensate for the limited technical depth.\n- Though a 4% performance improvement may be considered significant, the baseline in this paper is quite weak (e.g., standard prompting or vanilla CoT). More baselines may need to be included and compared.\n- From the limited ablation study on 2WikiMultiHopQA, we can observe that the improvement from pruning is not significant. This makes me doubt whether such a complex/expensive design is really necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our work and providing these valuable comments.\n\n**W1**: Our paper introduces a strategy specifically designed to enhance multi-hop reasoning capabilities by effectively organizing contextual information before reasoning processes begin. This approach addresses a noticeable gap in the literature, which traditionally emphasizes intermediate reasoning steps without sufficient focus on the initial organization of relevant contexts. Our method is simple yet effective and reflects a deliberate design choice aimed at maximizing efficiency and applicability across various tasks and datasets. We have rigorously tested our approach on multiple datasets, demonstrating its effectiveness in improving reasoning outcomes, with substantial improvements over baseline methods.\n\nContrary to existing works that focus primarily on query rewriting (Query Rewriting for Retrieval-Augmented Large Language Models, EMNLP 2023) to better retrieve context that is more relevant to the query, and context rewriting (Learning to Compress Prompts with Gist Tokens, NeurIPS 2023) aims to compress the length of the context, our approach innovates by post-processing context to extract and prune information precisely. This fine-grained manipulation of context is both technically challenging and critical for the nuanced understanding required in multi-hop reasoning tasks.\n\nFurthermore, we focus on developing a foundational precedent for future developments in large language model reasoning processes. The practical implications of our research offer a scalable solution that can be integrated with existing and future LLM frameworks to facilitate more robust reasoning capabilities.\n\nWe hope that this clarification addresses your concerns and further illustrates the technical rigor and innovative insights our work contributes to the field of prompt engineering.\n\n\n**W2**: Firstly, while the Chain of Thought (CoT) method is indeed simpler than some other methods, it has consistently demonstrated its effectiveness across a variety of reasoning tasks. Its widespread use and acknowledged efficacy in the research community make it an appropriate baseline for evaluating novel methodologies like ours. We aim to build upon well-established foundations to ensure that our contributions are both measurable and meaningful.\nSecondly, following the reviewer's suggestion, we add a stronger baseline method Tree-of-Thoughts (ToT). According to the original Tree-of-Thoughts paper, we adopt BFS search to solve the complex question. Specifically, we use GPT-4 to decompose a multi-hop question into a tree in BFS order, where the root node is the original question, and the other nodes are sub-questions. The specific results are as follows:\n\n| Model | Methods | HOVER-2 | HOVER-3 | HOVER-4 | FEVEROUS | SCIFACT |\n| --- | --- | --- | --- | --- | --- | --- |\n| GPT-4 | ToT | 74.36 | 72.43 | 70.64 | 92.78 | 92.33 |\n| GPT-4 | ToT+InfoRE | 76.96 | 74.18 | 72.30 | 95.82 | 94.42 |\n\n| Model | Methods | 2WMHQA | MuSiQue | SQA | HQA | WIKIHOP |\n| --- | --- | --- | --- | --- | --- | --- |\n| GPT-4 | ToT | 75.32 | 65.12 | 68.96 | 81.73 | 57.46 |\n| GPT-4 | ToT+InfoRE | 79.45 | 69.57 | 72.28 | 85.26 | 60.24 |\n\nThe inclusion of the Tree-of-Thought baseline has provided a more challenging comparison, and our method still demonstrates nearly a 4% improvement in performance on the same datasets, reinforcing the effectiveness and robustness of our approach.\n\n**W3**:  Our current experiments are done on normal data without much noise. This is why the impact of the pruning operation is not very pronounced, although it still results in a 1.53% improvement on the 2WikiMultiHopQA dataset. To further demonstrate the significance of the pruning effect, we use a retrieval tool to construct contexts with more noise for the 2WikiMultiHopQA dataset, replacing the currently used golden contexts. Specifically, we use BM25, implemented with the Pyserini toolkit, to retrieve the top-3 paragraphs from the October 2017 Wikipedia corpus as context documents. We then apply our method on this dataset and conduct the ablation study as described in section 5.2 using GPT-4. From the ablation study results, we observe a notable increase in performance. Specifically, in these noisier conditions, the pruning operation resulted in a significant improvement of 2.55%, which is comparable with the ability of extraction.\nThe ablation study results are as follows:\n\n| Methods | 2WikiMultiHopQA |\n| --- | --- |\n| Standard |  43.70 |\n| Full model (InfoRE) |  48.82 |\n| w/o extraction |  46.14($&darr; 2.68$ ) |\n| w/o pruning |  46.27($&darr; 2.55$) |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nS0XDvhitm", "reviewer": "Reviewer_x5hg", "summary": "The paper proposes a simple yet effective method to work with most of the current reasoning strategies. It automatically organizes the content into structural form, excluding noises and unused information during this process. The experiments are done using three models across ten datasets. The method consistently improves a vanilla setting and also a CoT setting. Ablation study shows the effectiveness of both the extraction part and the pruning part.", "questions": "1.\tWhy do you name the method RE-organization? I don’t see any other organization process before you’re the process of your method in the pipeline. Will it be more proper to name it “Information Organization?”\n2.\tWhat version of LLaMA-2 are you using? The temperature of the hugging face version needs to be a positive value so it cannot be zero.\n3.\tLine 317-319: Is it also because GPT-4 has a better ability to do the re-organization job than GPT-3?\n4.\tAre the 100 wrong predictions you annotated randomly sampled?\n5.\tCould you please give some mind map examples that LLM output in JSON format? Probably you can append them in the Appendix for a clear illustration.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe method is compatible with other reasoning strategies such as CoT and can further improve them. The method is easy to follow implement.\n2.\tThe paper includes extensive experiments. The proposed method is verified on all the models (3 LLMs) across all selected tasks (10 datasets) and shows consistent improvement in all scenarios.\n3.\tThe ablation study shows that each of the two components in the proposed method can improve the overall performance independently.", "weaknesses": "1.\tDo you consider trying gpt-3.5-0613/1106/0125 instead of text-davinci-003? The text-davinci-003 model is no longer usable in OpenAI APIs.\n2.\tSome rationales need further explanation in the design of the proposed method. Do you consider using LLM for the information pruning part instead of a pre-trained BERT? Will there be any improvement if using LLM? Do you consider using an end-to-end model to do extraction and pruning at the same time?\n3.\tIt is great to see those analysis in section 5.2. It will greatly improve the insight of your paper if you can include:\na.\tWill the extraction process introduce some hallucination?\nb.\tThe paper focuses on zero-shot setting currently. How can few-shot setting help? It may help from two perspectives: (1) Better information extraction; (2) Better downstream task performance.\nc.\tThe paper uses a very simple reasoning technique, the CoT prompting. Will the proposed method help in more advanced on such as ToT?\n4.\tI am particularly interested in the aspect of minimizing noise. However, currently the experiments are done on dataset with normal data without many noises. Could you please try on something like web-crawled data? Another choice can be manually injecting some noises in the datasets you use, i.e., adding some meaningless tags (<a></a>), or some irrelevant sentences. I am interested to see how the proposed method can automatically exclude these noises.\n5.\tAlso, since the method can formulate the content in a structural data. Could you please try something like changing the order of sentences in the content? Will it affect how the method extract data?\n\nMinor suggestions to presentation:\n\n1.\tA space is missing between “2” and “(” at line 125.\n2.\tThe “Large language models (LLMs)” appears at line 22, line 75, and line 353. Please do the abbreviation once at the beginning and use the abbreviation directly afterwards. Same suggestion applies to others like “Chain of Thought (CoT)” at line 190.\n3.\tThere seems to be an extra period in the “Prompt Content for Logical Relationship Extraction” in appendix A.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to the reviewer for giving these valuable feedback and comments.\n\n**W1**: We supplement the results across all datasets using GPT-35-Turbo-0613, including the main experimental results and ablation results. The results are shown in Tables 1, 2, and 3 in the PDF file under the Author Rebuttal section.\n\n**W2**: We chose pre-trained BERT for information pruning due to its high generalizability. In our experiments using LLAMA-2-70B, GPT-3.5, and GPT-4 for the pruning on 2WikiMultiHopQA, only GPT-3.5 and GPT-4 surpass the current method. However, we aim to develop a foundational and general method, not one limited to specific conditions. Additionally, we are concerned about self-preference bias in LLMs. As Arjun Panickssery et al. noted in \"LLM Evaluators Recognize and Favor Their Own Generations,\" LLMs tend to favor their own outputs. Using LLMs for both extraction and pruning could intensify this bias, making it hard to determine if improvements are due to methodological enhancements or bias. \n\nAn end-to-end approach combining extraction and pruning was tested in our method. As shown in Table 6 of Appendix A, our prompt for context extraction included the specific question to filter unrelated content. However, the ablation study in Table 3 shows that additional pruning improves performance, indicating that simultaneous extraction and pruning is less effective than our current separate method.\n\nWe will supplement the rationale analysis in the revised version of the paper to enhance the rationale of the approach.\n\n\n**W3**: a. LLM-generated content often contains hallucinations. In our experiments, we used GPT-4 to verify the factual consistency of the extracted information. If inconsistencies were found, we re-extracted the information until it was consistent. According to our statistics, 98% of the extraction results are consistent with the original text after the first extraction.\n\nb. The context of the dataset used in our experiments is relatively long. To prevent the text length from exceeding the maximum limit of LLMs, we adopt a 1-shot setting during the information extraction on the 2WikiMultiHopQA dataset, followed by direct reasoning with GPT-4. This approach resulted in an F1 score of 74.67%, which is lower than the existing zero-shot information extraction results of 76.52%. Considering the diversity of the samples in the dataset, using the 1-shot setting limited the effectiveness of the information extraction.\n\nc. To explore whether our method contributes to the ToT method, we conduct experiments with GPT-4 on all datasets. Specifically, we use GPT-4 to decompose a multi-hop question into a tree in BFS order, where the root node represents the original question and the other nodes represent sub-questions. The results are shown in Tables 1 and 2 in the PDF file under the Author Rebuttal section. The results show that our method, combined with ToT, can further enhance reasoning capabilities.\n\n\n**W4**: Following the reviewer's suggestion, we test our method on a noisier dataset constructed using retrieval tools. Due to time constraints, this is validated only on the 2WikiMultiHopQA dataset. Specifically, we use BM25, implemented with the Pyserini toolkit, to retrieve the top-3 paragraphs from the October 2017 Wikipedia corpus as context documents. We then apply our method and conduct the ablation study as described in section 5.2 using GPT-4. The results indicate that when the context contains more noise, the pruning operation brings a significant 2.55% improvement. The specific ablation results are as follows:\n| Methods | 2WikiMultiHopQA |\n| --- | --- |\n| Standard |  43.70 |\n| Full model (InfoRE) |  48.82 |\n| w/o extraction |  46.14($&darr; 2.68$ ) |\n| w/o pruning |  46.27($&darr; 2.55$ ) |\n\n**W5**: If there are no dependencies between sentences, changing their order will not affect the extraction results. Conversely, if there are dependencies, such as reference relationships, changing the sentence order will affect the extraction results.\n\n**presentation suggestions**: \n\nThanks to the reviewer for the presentation suggestions, we will address them in the revised version.\n\n**Q1**: We chose the term \"Re-Organization\" to emphasize the characteristics of reintegration and optimization of the information process of our method. Although no prior organization process is explicitly mentioned before the pipeline, we consider that the raw data processing and analysis are usually organized and processed in some initial form. Therefore, our method further \"re-organizes\" this to enhance the relational richness and usability of information.\nHowever, we greatly appreciate your suggestion to use the name \"Information Organization\" to more intuitively reflect the function of the method. We will consider this change to more accurately describe our method. \n\n**Q2**: We use the official version of the model. Specifically, we register on the Meta website (https://llama.meta.com/llama-downloads/) to download the model. Following the usage example in example_chat_completion.py on the official Llama GitHub (https://github.com/meta-llama/llama), we set the temperature to 0.\n\n**Q3**: More specifically, it is because GPT-4 inherently has stronger capabilities than GPT-3, whether in terms of organizing information or reasoning.\n\n**Q4**: Yes, these 100 samples are randomly selected from our results of prediction errors. This random sampling ensures an unbiased analysis of the error types our model is making, providing us with a comprehensive view of the potential areas for improvement.\n\n**Q5**: Due to the rebuttal length limit of 6000 tokens, we do not have enough space to display the JSON format mind map requested by the reviewer. However, the JSON format mind map we used is very similar to the example shown in Figure 2 of the paper, except that the example in Figure 2 omits the curly braces \"{}\".\nMore samples will be included in the appendix of the revised version for a clear illustration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you very much to the reviewers for reviewing our work and providing valuable feedback and suggestions.

In the PDF file, we supplement the results of our method on the GPT-35-Turbo-0613 version.

Meta-review: After carefully reading the paper and the author responses, all reviewers agree that this paper presents a novel method for re-organizing information before reasoning. The proposed algorithm demonstrated an average improvement of 4% in reasoning performance across 10 datasets and 3 large language models. It is independently effective and compatible with other reasoning strategies like CoT, offering a comprehensive breakdown of its components and their contributions.

There are still some rooms to further improve the paper. For example, it would be great if an in-depth comparison with knowledge-graph-based algorithms (e.g., GraphRAG (https://microsoft.github.io/graphrag/) can be included in the camera ready.
Decision: Accept (poster)