Information Re-Organization Improves Reasoning in Large Language Models

Authors: xiaoxia  cheng
zeqi  tan
wei  xue
weiming  lu
julius  caesar
joseph l mankiewicz
john  houseman
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 11:31:40

Abstract: Improving the reasoning capabilities of large language models (LLMs) has attracted considerable interest. Recent approaches primarily focus on improving the reasoning process to yield a more precise final answer. However, in scenarios involving contextually aware reasoning, these methods neglect the importance of first identifying logical relationships from the context before proceeding with the reasoning. This oversight could lead to a superficial understanding and interaction with the context, potentially undermining the quality and reliability of the reasoning outcomes. In this paper, we propose an information re-organization (\textbf{InfoRE}) method before proceeding with the reasoning to enhance the reasoning ability of LLMs. Our re-organization method involves initially extracting logical relationships from the contextual content, such as documents or paragraphs, and subsequently pruning redundant content to minimize noise. Then, we utilize the re-organized information in the reasoning process. This enables LLMs to deeply understand the contextual content by clearly perceiving these logical relationships, while also ensuring high-quality responses by eliminating potential noise. To demonstrate the effectiveness of our approach in improving the reasoning ability, we conduct experiments using Llama2-70B, GPT-3.5, and GPT-4 on various contextually aware multi-hop reasoning tasks. Using only a zero-shot setting, our method achieves an average absolute improvement of 4\% across all tasks, highlighting its potential to improve the reasoning performance of LLMs.

Main paper: 
 introduction: 
 Large language models (LLMs) demonstrate powerful generative capabilities and achieve remarkable performance across a range of linguistic tasks [1] [2] [3] . However, their capabilities in performing complex reasoning tasks -an essential aspect of advanced language understanding and intelligent decision-making -still present substantial challenges [4, 5] . This has spurred researchers to explore innovative strategies [6] [7] [8] to improve the reasoning capabilities of these models.Recently, diverse methods have been developed to enhance the reasoning ability of LLMs. For example, a notable method Chain-of-Thought (CoT) [6] , incorporates a series of intermediate reasoning steps into the reasoning. CoT [6] allows for a more transparent and understandable path to the final answer, making it easier to follow the logic behind the conclusion. Building upon this foundation, subsequent approaches such as Tree of Thoughts (ToT) [7] and Graph of Thoughts (GoT) [8] are proposed to further refine the reasoning steps and enhance the accuracy and reliability of LLMs. Different from the sequential intermediate steps of CoT [6] , ToT [7] and GoT [8] model the problem solving process into structured tools of tree and graph, respectively. A critical observation is that these existing approaches primarily focus on improving the reasoning process of LLMs, as shown in Figure 1 (left). However, in scenarios involving contextually aware reasoning, it is equally important to first identify logical relationships of context before proceeding with the reasoning, not just improve their reasoning process. This is because logical relationships, such as parallelism, causal connections, contrasts, etc., are essential elements of reasoning [9] . Nevertheless, these existing methods often neglect this crucial step. Such an oversight can lead to a superficial understanding and interaction with the context, potentially undermining the quality of the reasoning results.Context:Question:[QUESTION] Answer: …… Final answer is [ANSWER] .Context:[TEXT] Question:[QUESTION] Answer:Final answer is [ANSWER] .Existing Figure 1 : InfoRE (Ours) vs existing methods. In contrast to the existing methods that primarily focus on the reasoning process, our InfoRE emphasizes the re-organization of context information. The [TEXT] in italics indicates that it is optional in the reasoning process.Inspired by the fact that when faced with context-aware reasoning tasks humans often first re-organize existing contextual information to uncover the logical relationships, eliminate noises, and enhance their understanding of the context, we propose an information re-organization (InfoRE) method, to ground reasoning by the re-organized information. As shown in Figure 1 (right), different from previous methodologies that primarily focus on refining the reasoning steps to enhance the reasoning capabilities of LLMs, our approach takes a novel direction of context re-organization. We emphasize the utilization of re-organized contextual content to explicitly present the logical relationships that are often implicit within the plain text, promoting more effective reasoning. Specifically, our reorganization method comprises two operations: extraction and pruning. The extraction first uncovers the implicit logical relationships within the contextual content by transforming the content into a MindMap structure [10] . We employ this structure because it is rich in logical relationships and encompasses multi-hop connections. Pruning is then used to further minimize noise that is irrelevant to the reasoning objective. The pruning operation uses a pre-trained BERT [11] based model trained with reinforcement learning (RL). Finally, we utilize the re-organized context to reason. This enables LLMs to deeply understand the context by clearly perceiving these logical relationships, facilitating the quality and reliability of reasoning. Besides, our information re-organization method can be integrated with existing prompt methods, like CoT [6] , to further improve the reasoning ability of LLMs. To verify the efficacy of our proposed InfoRE method, we conduct experiments using Llama2-70B [2] , GPT-3.5 [1] , and GPT-4 [3] on various contextually aware multi-hop reasoning tasks, including claim verification [12] , question answering [13] , and reading comprehension [14] . Using only a zero-shot setting, our method achieves an average improvement of 4% across all tasks, highlighting its potential to improve the reasoning performance of LLMs.Our main contributions are as follows:• In contrast to existing methods that primarily focus on refining the reasoning steps to enhance the reasoning capabilities of LLMs, we take a novel direction of context re-organization. • The re-organization method initially uncovers the logical relationships that encompass multi-hop connections in the contextual content by extraction, and subsequently minimizes noise by pruning. • Experiment improvements on contextually aware multi-hop reasoning tasks across claim verification, question answering, and reading comprehension show the efficacy of our proposed method.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: Abstract, Introduction Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Reasoning with LLMs LLMs [1] [2] [3] have revolutionized the field of natural language processing (NLP) and demonstrate remarkable proficiency across a range of linguistic tasks. To further improve the reasoning ability of LLMs has attracted considerable interest. In-context learning [1] , as a promising approach to enhance the reasoning abilities of LLMs, has been verified in mathematical reasoning task [6] . In addition, CoT [6] incorporates a coherent series of intermediate reasoning steps to improve the reasoning ability of LLMs. Following this paradigm, ToT [7] and GoT [8] have also been proposed to focus on improving the structure of the intermediate chain. Then self-consistency [15] and plan-to-solve [15] focus on the reliability of the chain. Recently, the step-back prompting [16] is proposed, which obtains the high-level concept and first principles from instances by abstraction in the first step, then guides the reasoning of LLMs with the obtained concept and principles. For tasks that require multi-hop reasoning, current approaches [17, 18] generally decompose multi-hop problems into simpler sub-tasks problems relying on the in-context learning ability of LLMs. In contrast to existing methods, we take a novel direction of context re-organization to enhance the reasoning capabilities of LLMs for contextually aware reasoning tasks.Information Re-organization Information re-organization is a technique that leverages other structures to improve the clarity and comprehensibility of information. Moreover, it can also reveal conceptual relationships implicit in the original textual text, making it a valuable strategy for various tasks. For example, SG [19] re-organizes the document to scattered knowledge graph triples for explainable multi-hop question answering. GraphRAG [20] build a community hierarchy and generate summaries for these communities after extracting a knowledge graph, then leverage these structures on RAG-based tasks. MindMap [10] is a powerful structure method for representing knowledge and concepts, it can be used to construct a hierarchical abstraction of natural language text [21] . Previous method [22] uses it to organize a large amount of scientific material to enhance the clarity of the text material. Different from the scattered knowledge graph triples used in SG [19] , MindMap is more centralized, which aggregates content related to the same topic together. In this paper, we use it as the structure of re-organized information to uncover the logical relationships and multi-hop connections implicit within the plain context with one step, as opposed to GraphRAG [20] separately aggregates information with multiple steps.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Limitations in Appendix Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In this section, we first give a formulation of a context-aware reasoning task in §3.1 and then describe our method in detail. As shown in Figure 2 , the framework of our method consists of two components, an information re-organization §3.2 and a reasoning step using the re-organized context §3.3. Given a sample (c, q, a) from sample space (C, Q, A), where c denotes gold context, q denotes question, the task aims to obtain answer a to the question q, based on gold context c with a large language model. In particular, it requires multi-hop reasoning to get the answer a to question q.The purpose of the information re-organization is to obtain the logical relationships from the context and minimize the noise irrelevant to the question. This goal is achieved through two operations: extraction and pruning.For a question q, if the derivation of its answer a relies on context c, then a deeper understanding of c is crucial. Logical relationships, such as parallelism, causal connections, contrasts, etc., are essential elements of understanding and reasoning [9] . However, the logical relationships in the plain context are often implicit. Therefore, we perform a extraction operation on the plain context to uncover the logical relations in it using a language model. The process can be defined as:EQUATIONwhere f θ (•) is a language model parameterized by θ, P in is input task prompt. The specific task prompt P in used in our paper is displayed in the Appendix A. We perform Equation 1 for each c in the sample space to obtain the extracted context space G. We use the MindMap [10] structure to display the reorganized content, because it serves as a powerful structure for representing knowledge, concepts, and perspectives, contains not only logical relationships but also multi-hop connections.As shown in Figure 2 (bottom left), the extracted context g ∈ G contains not only parallel logical relationships, e.g., Director&Producer, but also causal relationships, e.g., Julius Caesar→Director. Furthermore, it also describes the three-hop connections, such as Julius Caesar→Director→Name, Julius Caesar→ Director→Occupation, and Julius Caesar→Director→Education. This information with logical relationships enables LLMs to deeply understand the contextual content by clearly perceiving these logical relationships, facilitating the quality and reliability of reasoning. Additionally, this multi-hop connection corresponds to the complex multi-hop problem and therefore helps to solve this multi-hop question q.As described in Section 3.2, the extracted context g ∈ G contains various logical relationships and rich attributes. However, not all logical relationships and attributes help answer the question q. On the contrary, some may even interfere with the response to the question. For example, consider the question in Figure 2 , the content "Julius Caesar → Production Company" is a distracting element, and "Julius Caesar → Adaptation" is irrelevant to the question.… BERT 𝒆 𝑪𝑳𝑺 𝒆 𝒓 𝟏 Linear 𝒆 𝒓 𝟐 𝒆 𝒓 𝒏 𝒆 𝒒 𝒆 𝑪𝑳𝑺 𝒆 𝑺𝑬𝑷Action ProbabilitiesSoftmax To further reduce the interference of distracting or irrelevant logical relations and attributes on the retrieval of answers for question q, we use a pruning model trained through reinforcement learning (RL). The pruning model is based on the pre-trained BERT [11] due to its high generalizability, as shown in Figure 3 . Its input consists of concatenated logical relationships, their corresponding attribute values from g, and the question q. For example, to prune the relation Julius Caesar → Adaptation, the input is: [CLS] Julius Caesar Adaptation Play by Shakespeare [SEP] Question [SEP]. We give a detailed demonstration of input format in Appendix B.We formulate the pruning policy model optimization as an RL problem and employ proximal policy optimization (PPO) [23] . The action is keeping or deleting the logical relations. The policy decides the action probability given the question and g. We fine-tune the policy model π by optimizing the reward r:EQUATIONReward Function Our goal is to maximize LLM's generation toward the desired target by an alignment measure R, and we use this as a reward. In this paper, the alignment metric we have chosen is the F1 score. To keep the policy network π from moving too far from the old result, we add a clipped term in the reward. Therefore, the final reward becomes:EQUATIONwhere ϵ is a hyperparameter indicating the range of CLIP to be performed. After pruning, the extracted contextual content g becomes the context g ′ , which is closely related to the reasoning objective q. We perform a pruning operation for each g ∈ G to obtain the pruned context space G ′ .After the re-organization process in Section 3.2, we get the re-organized context g ′ ∈ G′ . Then the re-organized context g ′ can be used as a context alone or combined with the original context c to get the final answer of q. The reasoning process can be defined as:EQUATION) where P r denotes prompt, and the content within [] denotes that it is optional. The specific prompt used in reasoning is displayed in the Appendix C.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
tasks and datasets: 
 To verify the effectiveness of our information re-organization method, we conduct experiments across a range of contextually aware multi-hop reasoning tasks and datasets, including claim verification [12] , question answering [13] , and reading comprehension [14] . The detailed dataset information, including data splits and statistics, is available in Appendix D.The task involves assessing a given claim against a set of evidence documents to determine whether they support or refute the claim [12] . We consider HOVER [24] and FEVEROUS [25] , which comprise complex claims that necessitate multi-hop reasoning for verification. Besides, we also take into account the SCIFACT [26] dataset, notable for its inclusion of scientific claims.Question Answering For this task, we consider the following datasets: 2WikiMultiHopQA [27] , StrategyQA [28] , MuSiQue [29] , and HotpotQA [30] . To answer the questions in these datasets requires not only multi-hop reasoning but also cross-document analysis.Reading Comprehension Machine reading comprehension task requires a model to process documents and select an answer from the provided candidates to a question about the content [14] . We primarily consider WIKIHOP [31] in the task, which necessitates multi-hop reasoning to derive the final answer. Additionally, HotpotQA [30] is also frequently considered as part of the QA domain.In our paper, we compare our method InfoRE to two reasoning baselines: 1) Standard. 2) CoT. The Standard approach is a method that directly reasons with the original textual context. The CoT [6] method involves augmenting standard reasoning methods by adding a step-by-step thought process. In our paper, we conduct the CoT strategy by appending the sentence "Let's think step by step." at the end of the question.The baseline methods and our InfoRE both adopt a zero-shot setting to counteract the potential randomness associated with demonstrations in a few-shot setting. We also design an answer-format instruction within the prompts for various tasks to standardize the structure of the final answer, thereby enhancing the precision of answer extraction. Moreover, all results reported in the paper use only the reorganized contextual information to reason. Comprehensive details about prompts and answer-format instruction are available in Appendix C. Following previous methods [17, 18, 25] , we run the official evaluation scripts of each dataset to get the F1 to measure the results.In our paper, the LLMs employed in the extraction and reasoning process include Llama2-70B [2] , GPT-3.5 (text-davinci-003) [32] and GPT-4 [3] . We use the official version of Llama2-70B. The specific version of GPT-4 is GPT-4-0613. We configure all models with top_p parameter as 1.0 and temperature as 0.0. In the policy model, we use the BERT-base version on all tasks and datasets. In RL training, we calculate the F1 score between the generated answer and the reference answer as the reward, with a rescaling coefficient of 10. We train the model for 1000 episodes. We conduct training for epoch 5, a batch size of 4, and a learning rate of 2e-6. The parameter of ϵ is set to 0.2. All experiments are conducted on an NVIDIA RTX A6000. 
main results: 
 Claim Verification Table 1 presents a comprehensive performance comparison between our InfoRE and existing zero-shot techniques. For the HOVER dataset, we segment it into the 2-hop, 3-hop, and 4-hop levels following previous methods [17] . As depicted in Table 1 , our InfoRE demonstrates significant improvements in the zero-shot claim verification task. The CoT [6] approach offers a lightly increase of 0.62% in the HOVER 4-hop using GPT-4, which indicates its marginal utility in more complex reasoning scenarios. Yet, our InfoRE achieves 3.02% improvement on the HOVER 4hop using GPT-4 showing remarkable performance on contextual aware understanding and reasoning. This improvement is further increased to 73.62% in combination with CoT, suggesting that the methods complement each other effectively. In the case of Llama2-70B, the combined application of InfoRE and CoT yields a score of 53.20% on the 2-hop HOVER task, surpassing its CoT-only score of 50.02%. This pattern of improvement is consistent with GPT-3.5. GPT-4 shows superior performance across all methods and datasets, suggesting an inherent advanced reasoning ability. This performance is mirrored in the specialized benchmarks, where GPT-4 with InfoRE attains near-perfect accuracy on FEVEROUS (95.62%) and very high accuracy on SCIFACT (93.67%). These figures solidify the notion that the InfoRE indeed enhances the reasoning capabilities of LLMs. Ablation studies In our paper, the re-organization comprises two components: extraction and pruning. To investigate the impact of each component in detail, we conduct a series of ablation experiments using GPT-3.5 on the 2WikiMultiHopQA dataset. First, we directly remove extraction and pruning from our method, and the results are shown in the second and third rows of Table 3 , respectively. It is worth mentioning that in the experiment where extraction is removed, we directly prune the sentences in the original context. Furthermore, we replace the reinforcement learning-based pruning method with a similarity-based pruning method to demonstrate its effectiveness. Specifically, the similarity-based pruning method uses Siamese-BERT, which takes the original question and each logical relationship as inputs separately, and then generates the corresponding representations. Then, we calculate the cosine similarity between these two representations. Finally, we removed the 30% of logical relationships with the lowest similarity, the result is shown in the last row of Table 3 . The results in Table 3 show that removing the extraction and pruning operations leads to performance drops of 2.94% and 1.53%, respectively. This demonstrates the effectiveness of both components in our methods. The larger performance drop after removing extraction highlights the importance of extracting logical relationships for effective reasoning. After replacing the pruning model, the performance dropped by 1.26%, but the results were still better than without pruning.This not only demonstrates the necessity of pruning but also highlights the effectiveness of the reinforcement learning-based pruning method.To assess the quality of the re-organized information, we perform a quantitative evaluation of the re-organized information with GPT-4 (gpt-4-32k) on the 2WikiMultiHopQA dataset. Specifically, we select 100 samples from the dataset, GPT-4 (gpt-4-32k) is asked to rank re-organized information produced by GPT-3.5 (text-davinci-003) [1] and GPT-4, as well as original context following criteria: (1) Depth: The information present multiple relationships of a topic, offering insightful perspectives or in-depth understanding of the subject. (2) Clarity: Information is clear and precise, making it easy to understand without ambiguity. All of the information is ranked 1, 2, and 3 with 3, 2, and 1 scores, respectively. Finally, we get a weighted average score for each information to measure the overall quality. The results in Table 4 demonstrate that the reorganized information outperforms the original textual context in terms of depth and clarity, which justifies the motivation of our paper. In addition, the re-organized information from GPT-4 outperforms GPT-3.5, which proves in another way that GPT-4 is indeed more capable than GPT-3.5. The evaluation results further validate the effectiveness of our method. Furthermore, the 22.22% improvement in depth is more obvious than the 15.14% improvement in clarity, which indicates that the re-organization of the information has been particularly effective in enhancing the logical relationships of the information. The improvement in clarity suggests that the information is now presented in a more direct and streamlined manner, making it easier for LLMs to grasp the essential points without wading through unnecessary details.To explore the effects of the quality of re-organized context on model reasoning capabilities, we utilize both GPT-3.5 (text-davinci-003) and GPT-4 to re-organize context information. Reasoning processes are then independently executed on each model. Moreover, employing this cross-validation technique also allows us to effectively evaluate the robustness of our method. The cross-validation results, shown in Table 5 , indicate that the use of either GPT-3.5 or GPT-4 for information re-organization leads to a marked improvement in the performance of LLMs on reasoning tasks. Additionally, it is observed that using GPT-4 for information re-organization while reasoning with the GPT-3.5 results in a further increase in reason results by 1.19% and 2.03% on FEVEROUS and 2WikiMultiHopQA datasets, respectively. This implies that enhancing the quality of re-organized information can improve the performance of LLMs, pointing to a promising direction for future research in refining information synthesis and re-organization. Conversely, when GPT-3.5 is employed for information re-organization in conjunction with reasoning using the GPT-4 model, there is a decrease in reason ability by 0.95% and 1.45% respectively. In addition, the magnitude of the decrease is lower than the increase, which implies that our re-organization strategy may have a greater impact on models with weaker reasoning abilities. This finding aligns with our understanding that models with inherently weaker reasoning abilities tend to rely more heavily on external strategy. For models with stronger inherent capabilities, our method still further improve its reasoning ability.Error Analysis To better comprehend where the errors in our InfoRE methodology come from and where they are fixed, we annotate 100 wrong predictions made by both InfoRE and Standard methods with GPT-3.5 on 2WikiMultiHopQA dataset. We categorize the errors into 4 classes:• Contextual Misunderstanding (CM): This happens when the model fails to interpret or connect multiple pieces of information from different parts of the documents. Multi-hop reasoning requires synthesizing information from various segments, and recognizing logical relations, and any misunderstanding can lead to incorrect conclusion.• Factual Error (FE): The model may provide an answer that is factually incorrect or not supported by the given documents. This is often due to the model's reliance on its training data, which may not always align with the specific facts in the context.• Mathematical Error (ME): The error occurs when math calculations are involved in deriving the final answer.• Unanswerable Question (UQ): It's a specific type of error or limitation in dataset design, where the context does not contain enough information to provide a valid answer to the posed question. The first four rectangles are error categories, while "Corrected" on the far right denotes the percentage of errors originally made by the baseline method that our method InfoRE has successfully corrected.When engaging in reasoning with LLMs, all four error categories are present. As shown in Figure 4 , there are 6% unanswerable question errors in the dataset, and more than 90% errors occur in the reasoning in the baseline method.Among the four types of error, contextual misunderstanding is the primary source of errors in the baseline, highlighting the importance of an in-depth understanding of context in solving reasoning tasks with LLMs. This finding is consistent with our motivation presented in the introduction section. Moreover, comparing the results of errors between our method and the baseline method, our method mainly corrects 14% of errors coming from the baseline method, most of the corrected errors are contextual misunderstanding errors. This further indicates that our InfoRE method assists LLMs in understanding context, signifying the necessity and effectiveness of conducting information re-organization before directly addressing the original question. 
conclusion: 
 In this paper, we propose an information re-organization method to improve the reasoning ability of LLMs. Compared with previous approaches primarily focus on improving the quality of intermediate steps, our method emphasizes uncovering the logical relationships, multi-hop connections, and pruning the irrelevant information through information re-organization. This approach enables LLMs to explicitly perceive the logical relationships and multi-hop connections of concepts within the context, promoting a deeper integration and understanding of the context, which results in more robust reasoning outcomes. To verify the effectiveness of our method, we conduct experiments using various LLMs across a range of contextually aware multi-hop reasoning tasks. The experiment results demonstrate the potential of our method to improve the reasoning ability of LLMs. Additionally, our method has a positive impact on various tasks involving context understanding, such as academic research, legal analysis, and medical diagnostics. However, it is also important to be aware of potential negative impacts, such as the propagation of misinformation.NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Introduction, Related Work, Methodology, Experimental, Results and Analysis. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not involve the release of new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: After carefully reading the paper and the author responses, all reviewers agree that this paper presents a novel method for re-organizing information before reasoning. The proposed algorithm demonstrated an average improvement of 4% in reasoning performance across 10 datasets and 3 large language models. It is independently effective and compatible with other reasoning strategies like CoT, offering a comprehensive breakdown of its components and their contributions.

There are still some rooms to further improve the paper. For example, it would be great if an in-depth comparison with knowledge-graph-based algorithms (e.g., GraphRAG (https://microsoft.github.io/graphrag/) can be included in the camera ready.

Reviews: review: paper_summary: paper_summary: The paper introduces an "Information Re-organization" (InfoRE) method aimed at improving the reasoning abilities of large language models (LLMs). It highlights the deficiencies of existing approaches that focus on intermediate reasoning steps without adequately addressing the preliminary organization of contextual information. The proposed InfoRE method involves extracting logical relationships and pruning redundant content from contextual data, guiding the reasoning process. This restructuring allows LLMs to recognize and use these logical relationships effectively, potentially enhancing reasoning quality. The method's efficacy is demonstrated using various LLMs like GPT-3.5 and GPT-4 across multiple reasoning tasks, showing an average improvement of 4% in performance.
strength_weakness: strength_weakness: 1. Information extraction and pruning might require intricate tuning and be computationally expensive, making it less practical for real-time applications.
2. The paper is well structured. However, authors should provide more details about their experiments.
questions: questions: 1. What are the differences between your extraction method and extracting to knowledge graph triples? Leveraging Structured Information ... (Li et al., EMNLP 2023) proposed a method to extract knowledge graph triples from documents. Moreover, you should add this paper to your related work since it also extracts knowledge from the document.
2. Since generations from LLMs always change, can you provide a standard variance or significant score of your results to show your improvement is substantial?
3. Do you limit the type of entities and relations during extraction? Do you apply open relations or closed relations during extraction? I only find the prompt in Appendix A. Is this all about your extraction part?
4. How do you check the completeness and quality of your extraction results? Can you provide more details about this?
5. LLM answers are always long, even with the format instruction. How do you compare your generated answers with golden answers? Do you apply exact match or other methods?
limitations: limitations: The paper adequately describes its limitations.
paper_summary: paper_summary: This paper proposes a method called information re-organization (InfoRE) to enhance the performance of large language models on some reasoning tasks. Unlike existing approaches that primarily focus on refining the reasoning process, InfoRE emphasizes re-organizing contextual information to identify logical relationships before reasoning. This method involves extracting logical relationships from the context and pruning redundant content to minimize noise. The re-organized information is then used in the reasoning process. The authors demonstrate the effectiveness of InfoRE through experiments on several contextually aware multi-hop reasoning tasks, achieving an average improvement of 4% in a zero-shot setting.
strength_weakness: strength_weakness: - The idea of this paper can be connected to existing prompt engineering works, e.g., performing retrieval from context and query/context rewriting. I feel this paper lacks technical depth (though this may be the fashion of a prompt engineering work, i.e., insightful but lacking technical depth). And the insight is not significant enough to compensate for the limited technical depth.
- Though a 4% performance improvement may be considered significant, the baseline in this paper is quite weak (e.g., standard prompting or vanilla CoT). More baselines may need to be included and compared.
- From the limited ablation study on 2WikiMultiHopQA, we can observe that the improvement from pruning is not significant. This makes me doubt whether such a complex/expensive design is really necessary.
questions: questions: See weaknesses.
limitations: limitations: I don't see potential negative societal impact of this work.
paper_summary: paper_summary: The paper proposes a simple yet effective method to work with most of the current reasoning strategies. It automatically organizes the content into structural form, excluding noises and unused information during this process. The experiments are done using three models across ten datasets. The method consistently improves a vanilla setting and also a CoT setting. Ablation study shows the effectiveness of both the extraction part and the pruning part.
strength_weakness: strength_weakness: 1.	Do you consider trying gpt-3.5-0613/1106/0125 instead of text-davinci-003? The text-davinci-003 model is no longer usable in OpenAI APIs.
2.	Some rationales need further explanation in the design of the proposed method. Do you consider using LLM for the information pruning part instead of a pre-trained BERT? Will there be any improvement if using LLM? Do you consider using an end-to-end model to do extraction and pruning at the same time?
3.	It is great to see those analysis in section 5.2. It will greatly improve the insight of your paper if you can include:
a.	Will the extraction process introduce some hallucination?
b.	The paper focuses on zero-shot setting currently. How can few-shot setting help? It may help from two perspectives: (1) Better information extraction; (2) Better downstream task performance.
c.	The paper uses a very simple reasoning technique, the CoT prompting. Will the proposed method help in more advanced on such as ToT?
4.	I am particularly interested in the aspect of minimizing noise. However, currently the experiments are done on dataset with normal data without many noises. Could you please try on something like web-crawled data? Another choice can be manually injecting some noises in the datasets you use, i.e., adding some meaningless tags (<a></a>), or some irrelevant sentences. I am interested to see how the proposed method can automatically exclude these noises.
5.	Also, since the method can formulate the content in a structural data. Could you please try something like changing the order of sentences in the content? Will it affect how the method extract data?

Minor suggestions to presentation:

1.	A space is missing between “2” and “(” at line 125.
2.	The “Large language models (LLMs)” appears at line 22, line 75, and line 353. Please do the abbreviation once at the beginning and use the abbreviation directly afterwards. Same suggestion applies to others like “Chain of Thought (CoT)” at line 190.
3.	There seems to be an extra period in the “Prompt Content for Logical Relationship Extraction” in appendix A.
questions: questions: 1.	Why do you name the method RE-organization? I don’t see any other organization process before you’re the process of your method in the pipeline. Will it be more proper to name it “Information Organization?”
2.	What version of LLaMA-2 are you using? The temperature of the hugging face version needs to be a positive value so it cannot be zero.
3.	Line 317-319: Is it also because GPT-4 has a better ability to do the re-organization job than GPT-3?
4.	Are the 100 wrong predictions you annotated randomly sampled?
5.	Could you please give some mind map examples that LLM output in JSON format? Probably you can append them in the Appendix for a clear illustration.
limitations: limitations: Yes.

Comments: title: Appreciation for Reassessment
A Gentle Reminder
Appreciation for Positive Feedback
Appreciation for Positive Feedback
Reply to Rebuttal
comment: Dear Reviewer cK3D,

Thank you for taking the time to reassess our work. We sincerely appreciate your patience and professionalism throughout the entire review process, which have been instrumental in helping us further improve our research.

Once again, thank you for your time and effort in reviewing our paper. If you have any further questions, please feel free to reach out at any time.

Best regards,

The authors of InfoRE
Thank you for your response. It solves some of my concerns. I have updated my score.
Dear Reviewer cK3D,

Thank you for your efforts in reviewing our paper. We greatly value the feedback from each reviewer and have provided detailed responses to the concerns raised. Up to this point, we have received feedback from reviewers wQkW and x5hg, and are pleased to have their positive recognition. However, we have not yet received any feedback from you.

As the discussion period is nearing its end, we would greatly appreciate it if you could let us know whether we have adequately addressed your concerns. Your insights are important to us, and we believe they will help us further improve the quality of our paper.

Thank you once again for your time and consideration.

Best regards,

The authors of InfoRE
Dear Reviewer x5hg,

Thank you for your positive reassessment. We greatly appreciate your recognition and insightful suggestions.  We will incorporate suggestions and additional analysis results into the revised paper to enhance its completeness and clarity.

Once again, thank you for your time and effort in reviewing our paper. If you have any further questions, please feel free to reach out at any time.

Best regards,

The authors of InfoRE
Thanks very much. Most of my concerns have been adressed. I raise my score by one point.
Dear Reviewer wQkW,

Thank you for your positive reassessment, and we appreciate your acknowledgment of our work. We are truly grateful for your insightful feedback throughout the entire review process, which has helped us further improve our work.

Best regards,

The authors of InfoRE
Thank you for the response. It has solved my primary concern. I have updated my rating score.