Enhancing LLM’s Cognition via Structurization

Authors: kai  liu
zhihang  fu
†  chao
wei  zhang
rongxin  jiang
fan  zhou
yaowu  chen
yue  wu
jieping  ye
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 05:28:43

Abstract: When reading long-form text, human cognition is complex and structurized. While large language models (LLMs) process input contexts through a causal and sequential perspective, this approach can potentially limit their ability to handle intricate and complex inputs effectively. To enhance LLM’s cognition capability, this paper presents a novel concept of context structurization. Specifically, we transform the plain, unordered contextual sentences into well-ordered and hierarchically structurized elements. By doing so, LLMs can better grasp intricate and extended contexts through precise attention and information-seeking along the organized structures. Extensive evaluations are conducted across various model architectures and sizes (including a series of auto-regressive LLMs as well as BERT-like masking models) on a diverse set of NLP tasks (e.g., context-based question-answering, exhaustive hallucination evaluation, and passage-level dense retrieval). Empirical results show consistent and significant performance gains afforded by a single-round structurization. In particular, we boost the open-sourced LLaMA2-70B model to achieve comparable performance against GPT-3.5-Turbo as the halluci- nation evaluator. Besides, we show the feasibility of distilling advanced LLMs’ language processing abilities to a smaller yet effective StruXGPT-7B to execute structurization, addressing the practicality of our approach. Code is available at https://github.com/alibaba/struxgpt.

Main paper: 
 introduction: 
 Large language models (LLMs) have emerged with remarkable language capabilities [6, 53, 1] , yet remain at a discernible distance from human-level intelligence, especially when handling long-form, sophisticated contexts as inputs [3, 36] . Scaling up the model size has significant benefits for boosting context-comprehension and instruction-following abilities for LLMs [47, 58] . However, it is generally resource-intensive on both model training and inference. This paper presents another perspective on enhancing LLMs' cognition capability without altering the models: context structurization.The idea of structurization is motivated by neurocognitive science [51, 5, 17] . In human cognition, as indicated in Fig. 1 , sophisticated text sequences will be processed and consolidated into a structured knowledge tree, with factual elements well-organized hierarchically [28, 15] . This process is defined as structurization. People can precisely search information from general concepts to specific details and make connections and comparisons along structures. We thus aim to transform plain texts into structurized inputs, helping LLMs recognize and understand contexts in a human manner [71] .As Fig. 1 illustrates, input sequences are reorganized in a simple but generic three-layer structure: scope, aspects, and descriptions. The scope summarizes the topic and contents, unfolding into several main aspects with corresponding detailed descriptions. The structurized results can be freely assembled into various natural language forms, depending on the specific downstream tasks. Fig. 2 provides an exemplar of our overall framework: after structurizing and reassembling the vanilla context to highlight its scope and aspects of data imbalance handling techniques, LLMs become able to grasp the target information about the dynamic weighting strategy and generate reliable responses.The idea of language identification is to classify a given audio signal into a particular class using a classification algorithm. Commonly language identification task was done using i-vector systems. A well known approach for language identification proposed by N. Dahek et al. uses the GMM-UBM model to obtain utterance level features called ivectors. Recent advances in deep learning have helped to improve the language identification task using many different neural network architectures which can be trained efficiently using GPUs for large scale datasets … Figure 1 : Structured cognition on sequential contexts. Humans may easily identify a given passage's topic/scope, break down the text sentences into several aspect points with detailed descriptions, and form a tree-like knowledge structure.We first execute the structurization by prompting advanced commercial LLMs (e.g., GPT-3.5-Turbo1 or Qwen-Max2 ) with a few examples, and then collect the results to train a smaller 7B-parameter LLaMA2 [53] or Qwen [2] model as our StruXGPT. This is motivated by [48] , where the fundamental syntactic processing ability from giant LLMs is distilled into a responsive and affordable StruXGPT-7B. Comprehensive evaluations indicate that StruXGPT-7B inherits 97% of the structurization ability from the teacher model, showing our method's feasibility.Empirical experiments are conducted on a diverse set of NLP tasks (e.g., context-based question-answering, exhaustive hallucination evaluation, and passage-level dense retrieval). The results show that with a single-turn structurization by our StruXGPT, the cognition performance of vanilla large language models witnesses consistent improvements regardless of the model architecture and size variation. In particular, we boost the open-sourced LLaMA2-70B [53] to achieve comparable performance against GPT-3.5-Turbo as a hallucination evaluator, and demonstrate the compatibility with other advanced prompting techniques, such as CoT [59] . We hope this paper can bring new insights to the community on building a more powerful and critical language model with human cognition.Our contribution can be summarized as follows:• We propose the concept of structurization, in order to enhance LLM's cognition capability without altering the models themselves. • We present the feasibility of distilling the structurization ability from giant commercial LLMs into a responsive and affordable StruXGPT-7B model, making our approach practical. • With structurization, we empirically demonstrate the consistent cognition enhancement for various LLMs across model architecture and size variation on diverse NLP tasks.The data to train our StruXGPT is carefully curated from two main aspects. First, to ensure diversity, we randomly sample 15,000 passages from Wikipedia across various domains, whose token length ranges from 1K to 3K. Then, we also supply round 15,000 statement passages from the domainspecific model-and human-generated content, including the filtered CAMEL-AI [31] , FiQA [40] ,and MedQuad [4] datasets. With Qwen-max API, we collect the structurized results and eliminate the failed cases (usually due to internet inaccessibility and unexpected output format), resulting in 22,547 pieces of training data in total to derive our StruXGPT, and 200 test samples for evaluation.StruXGPT is built upon LLaMA2-or Qwen-7B-chat models, which had been aligned to humans with promising instruction-following capabilities with their prior knowledge. The 22,547 pieces of input-output pairs are utilized to distill the structurization ability to StruXGPT via supervised fine-tuning (SFT). Specifically, StruXGPT is trained with a constant learning rate of 5 × 10 -6 for LLaMA and 1 × 10 -5 for Qwen for 1 epoch. The batch size is 128, and other hyper-parameters follow the default settings from Touvron et al. [53] and Bai et al. [2] . The training is resource-friendly, which can be done on 8 NVIDIA V100 (16G) GPUs for 3.5 hours.For all the inference experiments, we leverage 1-2 NVIDIA A100-80G GPUs for model deployment. 
related work: 
 Large language models (LLMs). LLMs' emergent abilities [58] has recently received extensive attention in the literature [6, 61, 9, 53, 23, 1] , which are found closely related to the scaling law [26] .When the scale reaches a certain level, the performance on complex NLP tasks significantly rises due to the superior in-context learning, instruction following, and reasoning abilities [71, 47] . Numerous efforts have been made to boost the model capacity with training and prompting strategies [37, 38, 59, 14] , and this paper presents a new perspective, context structurization, to encourage LLMs to perceive, recognize, and communicate like humans [55, 71] , without altering the model themselves.Context augmentation. Recent studies have proposed several context augmentation methods to enhance LLM's cognition ability [59, 41] on when taking the long-form context (with thousands of tokens) as inputs [3, 36, 30] . Specifically, aspect-based summarization (ABS) and query-based summarization (QBS) [64, 69] are designed to extract important information from lengthy text data as well, but they require pre-defined or user-input aspect/query lists to conduct targeted summarization, and the detailed information will be inevitably lost during the summarization process. In contrast, the Figure 2 : Framework overview. When instructed to generate responses based on vanilla long-form and sophisticated contexts, LLMs often lose their focus and give unreliable answers due to their limited cognition capability. In contrast, we structurize the vanilla context by using our StruXGPT to identify its main scope and aspect points, facilitating the original LLMs to comprehend the context and generate accurate responses.paper develops context-wise structurization to highlight the knowledge structure, running a single-turn structurization on the context to enhance LLMs' cognition abilities on a diverse set of NLP tasks.Structurization. In the conventional NLP literature, the term structured data usually refers to entity-relation-entity triplets or properties extracted from plain texts, which are utilized to construct knowledge graphs or databases with special data formats or schemas [24, 32] . On the contrary, the structurization in this paper does not focus on entity-level information extraction and aggregation. Instead, it suggests reorganizing the input sentences into a three-layer structure based on their inner linguistic relations. Similar to discourse analysis and constituency parsing [10, 27] , the main purpose of our structurization is capturing the dependencies and relations of the elements within specific long-form text inputs, so as to enhance LLMs' cognition of the knowledge structure and relations.Knowledge Distillation. In the era of large language model learning, distilling specific knowledge from giant LLMs' outputs has commonly been used to derive a more affordable but still powerful language model [63, 19] . Previous attempts, such as Standford Alpaca [50] and Vicuna [8] , show the feasibility of collecting instruction-response pairs from GPT-3.5 or GPT-4 to train a smaller fundamental [46] or domain-specific (e.g., with reasoning [43] or coding [20] skills) language models. Our work also distills the structurization capability from the giant Qwen-Max model to a smaller yet effective StruXGPT-7B model, with extensive evaluations to show the efficiency and efficacy. 
structurization: 
 The substantial purpose of structurization is to mimic the human cognition process and transform plain, sequential text sentences into a well-organized, hierarchical knowledge structure. Inspired by the linguistic discourse analysis [11, 18] , this paper develops a three-layer hierarchy exemplar to present the cognition/knowledge structure, as introduced in Fig. 1 :(1) Scope summarizes the topic and boundary of the textual context. It outlines the central issues of knowledge throughout the text and the scope of the discussion that will be covered. (2) Aspect further subdivides the input context into several parts. It presents the aspects or dimensions that must be considered to fully understand the topic and scope. (3) Description is the most specific and detailed layer. It provides in-depth descriptions and analyses to support each aspect of the context scope.This generic three-layer structure is derived for efficacy and efficiency in dealing with diverse textual inputs. There might be some elaborated structures (such as a knowledge mindmap [60] ) to better deconstruct specific text sources, but the difficulty and complexity of defining, extracting, and utilizing those structures to aid in practical problems are dramatically increased. We leave this exploration in our future work.Next, we will present how to implement effective structurization with minimal cost in Sec. 3.1, and demonstrate the utilization of the structurization results to enhance LLMs' cognition abilities in different downstream tasks in Sec. 3.2.We have explored two approaches to execute the structurization process by leveraging the extraordinary capability of large language models: few-shot prompting on commercial LLMs and direct instructing our developed StruXGPT models.Few-shot commercial LLM. In the initial stage, we use two in-context examples to query the commercial Qwen-Max model to structurize the input corpus, since it shows promising instructionfollowing and textual analyzing capability with over 200B parameters 3 . Here is a simplified example of prompting structurization in Fig. 3 , and the full template is displayed in Appendix E.1. However, commercial LLMs are usually slow and expensive, and sending user data to LLM APIs may cause privacy and security problems due to information leakage. Thus, we train a smaller 7b-parameter model (e.g., LLaMA2 [53] or Qwen [2] ) to inherit the structurization ability from giant commercial LLMs, which can be deployed locally for efficiency and privacy.Fine-tuned StruXGPT. Our tailored model is named by StruXGPT, where Stru is the abbreviation of structurization, and X implies we do not specify the model architecture. We carefully curate 22,547 raw data pieces from Wikipedia 4and CAMEL-AI dataset [31] to ensure diversity, and collect the structurized results from Qwen-Max to train our StruXGPT via supervised finetuning (SFT). From the collected samples, 200 are utilized for evaluation (including human verification), and the remaining training samples are adopted to distill the structurization capability from Qwen-Max to our StruXGPT-7B. It is practical since the structurization only relies on fundamental syntactic understanding and processing ability, which has already been learned from the large-scale corpus. We merely teach the 7B-parameter models how to reorganize the input text via SFT, without introducing new memorizing or creative overloads. In addition, the ultimate StruXGPT does not require few-shot examples, and it reduces the input lengths for further efficiency. The training details are described in Appendix A.1.The identified knowledge structure (i.e., the scope, aspect, and description hierarchy) from the raw context is initially parsed in JSON format and unsuitable for direct inputs to handle massive lengthy elements. Therefore, we use a unified template to transform the structured data back into natural language sentences as models' inputs to fit their intrinsic processing patterns, as LLMs are pre-trained and aligned with mostly natural language data. Concurrently, to preserve and highlight the knowledge structure, we harness specific linguistic markers to signal hierarchy and relationships among concept elements, such as numbered lists for order, bullet points for categorization, and indentation to depict nesting levels of information. Fig. 4 showcases some typical examples.The first row of Fig. 4 provides a unified template to transform structurization results into natural languages. The top-level hierarchy scope is presented as a standalone sentence, serving as the introduction to the structured context and highlighted with bold markers. Subsequently, the secondary aspect are organized with numerical markers and bolded, attaching with its corresponding tertiary descriptions through subclauses or separate sentences. This method not only signifies the rank and relation of each piece of information relative to others but also provides clear, navigable paths for the LLMs to follow and process the information efficiently (e.g., when examining the long-form comprehension capability). Moreover, the second row of Fig. 4 introduces another variation for transformation, where each description elements are further broken down and enumerated for the delineation of fine-grained details, making it easier for language models to discern and retain specific nuances associated with each aspect (e.g., when examining the hallucination detection capability). After transformation, the linguistic input retains its structured knowledge through systematic cues but is presented in a comprehensible manner for LLMs. This not only facilitates an enhanced understanding and interaction with complex data but also enables the models to leverage their existing natural language capabilities to generate more accurate and contextually relevant responses. 
experiments: 
 In this section, we conduct extensive experiments on a series of downstream NLP tasks to comprehensively demonstrate the efficacy of structurization. We hope the results can bring new insights to enhancing LLM's cognition via structurization, regardless of model architecture and size variation.Three representative natural language understanding and processing tasks are investigated, including context-based question-answering Sec. Question-answering based on a long-form context is an emerging research area within QA, which requires large language models to precisely seek the target information and generate reliable responses to the question [3, 30] . It is an immediate measure of LLM's cognition ability to handle intricate and sophisticated contexts. In this section, we comprehensively evaluate how structurization boosts QA ability on seven datasets from the LongBench benchmark [3] with a variety of LLMs to examine.Dataset setup. LongBench [3] Evaluated models. Following [3] , we evaluate three representative large language models on long context comprehension ability: LLaMA2-7B-4k [53] , Qwen-7B-8k [2] , and ChatGLM3-6B-32k [67] , and extend the larger LLaMA2-13B-4k [53] model. The four LLMs have a relatively similar parameter capacity with different model architectures and window sizes. All the models to examine are pre-trained chat models, and we just employ our StruXGPT to structurize the input contexts to enhance those LLMs' cognition capabilities. To ensure reproducibility and reduce uncertainty, greedy search is adopted during LLM's decoding process when generating responses. The accuracy between models' responses and ground-truth answers is measured by ROUGE-L and F1-score.Experimental results. Tab. 1 suggests structurized contexts bring consistent improvements on almost all 3 tasks and 7 subsets across the model architectures and window sizes. Specifically, structurization leads to relatively greater improvement for the MultiDoc-QA subtask (with a 3% performance gain on average), revealing the potential promotion of LLMs' multi-hop reasoning abilities. Despite the negligible decline on the Musique [54] subset (see Appendix B.2 for analysis), the advanced ChatGLM3-6B-32k is also boosted, showing structurization's efficacy for powerful models.Comparison with other baselines. To further evaluate our structurization augmentation, we compare our method against the typical summarization-based methods that also employ LLMs for context augmentation. The results are presented in Appendix B.3, which further demonstrates our superiority in highlighting the knowledge structure without loss of key information.A: Domain experts with legal training.Q: Who were the experts used for annotation?A: Unanswerable.This passage talks about the PrivacyQA dataset and its characteristics: Investigating structurization from the attention perspective. Fig. 5 reveals how structurization can aid LLM's cognition from the attention perspective. In particular, we compare the attention maps for the same tested LLaMA2-7B model with different contexts as input. At the position of the model's first token prediction, we average the attention maps across the 32 attention heads for each layer of LLaMA's last 16 layers [72] , and visualize the attention scores in Fig. 5 . Specifically, when handling vanilla contexts, LLaMA2-7B loses its focus on the target information of the experts. On the contrary, the structurized context clearly presents the content structure of the introduced PrivacyQA dataset, and LLaMA2-7B immediately grasps the target aspect and its detailed descriptions of experts with legal training. In this way, LLM's cognition capability is successfully enhanced via context structurization.Hallucination has raised wide attention in the community [70, 56] . In general, evaluating hallucinations involves verifying atomic claims against supportive materials (e.g., Wikipedia passages [66, 42] ), yet even advanced GPT-3.5-Turbo and GPT-4 cannot always accurately make the judge, as LLMsevaluators often struggle to extract relevant information due to the complexity of passage contexts. We introduce how to improve LLM evaluators' assessing ability by context structurization below. Dataset setup. AttrScore [66] and FactScore [42] datasets are adopted for evaluation. We take the AttrEval-GenSearch test set with 245 examples from AttrScore, where each example comprises a statement and its reference passage, and is annotated by Attributable (abbreviated as Attr.), Contradictory (abbreviated as Contra.), and Extrapolatory (abbreviated as Extra.). FactScore collected 4,726 atomic claims/statements of people biographies generated by InstructGPT (abbreviated as InstGPT), 5,426 by ChatGPT, and 5,888 by PerplexityAI (abbreviated as PPLAI). For each input sample, we leverage StruXGPT to structurize the reference to identify its main aspects and detailed descriptions.The numerically ordered structure is preserved, as displayed in Fig. A3 , since we explicitly ask the evaluator to check the information along the structure for judgment.Evaluated models. We mainly investigate the open-sourced LLaMA2-7B and LLaMA2-70B models as the LLM-evaluator, and also explore the integration with the close-source GPT-3.5-Turbo-1106 [6] via API access. The main results are presented in Tab. 2. We also report the results with Qwen models on AttrScore and FactScore and the incorporation to more powerful GPT-4 in Appendix B.4.Experimental results. According to Tab. 2, our structurization brings significant enhancements to both LLaMA2-7B and 70B models (for 6.4% and 4.3% on average, respectively). And the powerful GPT-3.5 model also gains 4.1% (from 58.0% to 62.1%). Furthermore, we incorporate the Chain-of-Thought (CoT) technique into the prompt template to clarify the evaluation steps (such as Carefully read the claim and double-check the answer.) (denoted as "GPT-3.5-1106 + CoT"). After that, the GPT-3.5 model immediately obtains an improvement of 4.0% (from 58.0% to 62.0%). Consequently, on top of the advanced CoT prompt, our method further enhances the model to a higher accuracy of 65.4% on average, demonstrating our method's compatibility with advanced prompting techniques.Retrieval-augmented generation (RAG) has been empirically validated to significantly bolster LLM's domain knowledge [29, 62] , where precise document retrieval plays a vital role. We now investigate how structurization can facilitate dense retrieval for BERT-like masking language models.Dataset setup. BEIR dataset [52] is a popular benchmark for evaluating dense retrievers' zeroshot effectiveness [39, 33] , where retrievers are trained on MS MARCO's passage-retrieval training split [44] while directly tested on the BEIR benchmark without finetuning. We focus on our evaluation of the 5 subsets from BEIR i.e., NFCorpus, FiQA, ArguAna, SciDocs, and SciFact.Evaluated models. BERT [13] , SimLM [57] , and coCondenser [16] are chosen for evaluation, since they achieve state-of-the-art performance on MS MARCO's development split. To convert the structurized passages from our StruXGPT into natural languages, we eliminate the numerical indicators (such as "1.", "1.1", etc.) and attach the description statements in the third layer with their aspects. Fig. A3 presents an example. We only structurize passages to enhance retrievers' cognition, and the queries remain unchanged. Following the literature, nDCG@10 results are reported in Tab. 3. Experimental results. Structurization boosts all three retrievers on most subsets, yielding a maximum performance improvement of 4.5% on SciFact for SimLM. The results suggest that structurization not only augments decoder-only generative LLMs with explosive parameters (at least 7B), but also benefits encoder-decoder masked language models with constrained parameters (around 110M). It implies that the patterning of linguistic and semantic structurization may be a fundamental mechanism for enhancing language models, transcending distinctions in their architectural design and scale.In this section, we assess various structurization methods through exhaustive experiments on five approaches, including prompting Qwen-max with few-shot exemplars (serves as our teacher model), few-shot Qwen-7B and LLaMA2-7B pre-trained chat models, and our fine-tuned Qwen-7B and LLaMA2-7B (student) models. As introduced in Sec. 3.1, we use 200 validation cases and have the five models generate 1,000 structurized outputs for analysis.A good structurization should effectively deconstruct the vanilla input text to clearly identify its knowledge structure, so as to facilitate LLM's cognition. The resulting content should be faithful to the original texts, neither dismiss the factual information nor fabricate statements or opinions that do not exist. To this end, we revise four evaluation metrics to investigate the efficacy of different structurization approaches, and the results are displayed in Tab. 4. Lexical evaluation (LexicalEval). We first leverage the widely-used ROUGE-L [34, 35, 25] to assess recall and precision between structured content and original text. However, lexical metrics from the methods, ranging 0.6 to 0.7, inadequately reflect structurization quality where LLMs will paraphrase the words but lexical scores miss the semantic consistency. For instance, a statement pair "They adopt ROUGE" and "ROUGE is adopted" only receives a 0.33 f1-score for ROUGE-L. Therefore, a crucial human evaluation is developed to obtain a trustworthy conclusion.Human evaluation (HumanEval). We recruited 17 well-trained natural language annotators from the PAI-iTAG platform 5 to evaluate the structurization quality on a 0-5 scale across three dimensions: completeness (ensuring no loss of information from the original text), factuality (accurate three-layer deconstruction), and anti-hallucination (avoiding fabricated content). As annotating structurization only involves linguistic and syntactic level judgments, annotators do not need professional expertise to check the information of a given text itself. The detailed evaluation criteria are displayed in Appendix A.3, and we report the labeling results in Tab. 4.The commercial Qwen-max shows a promising instruction-following and in-context learning ability, generally scoring 4.5 at the three dimensions. However, the pre-trained 7B models from Qwen and LLaMA2 immediately decline the scores to below 4.0, as they struggle to understand the instruction to build the three-layer structure and tend to hallucinate responses due to the limited model capacity.More structurization examples can be found in Appendix E.2. Notably, the fine-tuned StruXGPT-7B-Q(wen) and StruXGPT-7B-L(LaMA2) both obtain a 4.35 -4.45 score on average. They inherit 97% of structurization capability from the Qwen-max teacher model, evidencing the effectiveness of training a specialized 7B-parameter model for structurization to aid in efficiency and privacy.Evaluation with downstream application (AppEval). Since our main motivation is to utilize structurization to enhance LLM's cognition, a further evaluation is conducted to investigate those different structurization methods. Specifically, we compare how much improvement (∆) those methods can bring to downstream natural language processing applications. On the Qasper subset [12] from LongBench [3] , we instruct an independent LLaMA2-7B chat model for reading comprehension with long-context structurized by different approaches. LLaMA2-7B receives a 19.6 F1-score for QA accuracy when taking the vanilla context as input, which serves as the baseline performance.The evaluation results are displayed in Tab. 4, which are consistent with the human evaluation presented above. In particular, the few-shot Qwen-max achieves an over 3% improvement in answer quality, while the pre-trained 7B-parameter chat models fail to generate validated structurizations. Meanwhile, our fine-tuned StruXGPT-7B-LLaMA and StruXGPT-7B-Qwen models bring comparable enhancements against the few-shot Qwen-max, emphasizing the efficacy of distilling the structurization ability from giant teacher models to a more responsive and affordable student model.Evaluation with semantic embedding (SemEval). At last, we explore the structurization quality evaluation in the semantic embedding perspective, as a supplementary to lexical evaluation. Following Zhang et al. [68] , we calculate the semantic similarity between original and structurized contents with the embedding similarities, and the results in Tab. 4 show consistent measure against HumanEval and AppEval with a much lower cost. Hence, BERTScore [68] can be further leveraged as an effective and efficient quality-assessment tool for training-data filtering and structurization quality evaluation to derive a better StruXGPT model. Sec. 4.6 presents some preliminary investigations.Through the comprehensive evaluation of three protocols, we demonstrate the feasibility and efficacy of training a specialized StruXGPT. It is more resource-friendly to deploy for efficiency and privacy, meanwhile inheriting 97% of the ability from giant teacher models to perform structurization. To further verify it, we investigate the number of in-context examples with two evaluation protocols (as in Tab. 4): AppEval (an improvement on Qasper subset with context structurization) and BERTScore (semantic similarity with raw and structurized texts in the validation set). We also report the error rate when parsing structurization results from the teacher model's outputs (denoted as "FormatError").According to Tab. 5, 1-shot is apparently insufficient to illustrate structurization, while 2-and 3-shot achieve comparable structurization quality under AppEval and BERTScore. Notably, 3-shot receives a 2% lower FormatError than 2-shot, in trade for the increased inference cost (because of increased few-shot samples). We argue that the 2% gap (around 400 samples) does not make a difference for the final StruXGPT training, which can be verified in Appendix A.2. Therefore, we recommend users to apply 3-or even more shots when prompting teacher LLMs if available, otherwise 2-shot is also a good choice to balance the inference cost and structurization quality. On the other hand, since poor structurization results can lead to suboptimal application performance, we statistic the enhancement variance in the Qasper subset from LongBench [3] in Tab. 8 for structurized context inputs. Our method merely causes degradation on 3.5% of samples (with relatively lower structurization quality), but ultimately receives a +3.6 improvement overall test samples. Furthermore, we filter out the structurization results with a low BERTScore (e.g., <0.05) and take back the original context as input. In this way, the degradation can be alleviated (from 3.5% to 3.0%), further improving the final enhancement to +3.7. The lower bound of our StruXGPT can thus be ensured. 
conclusion and discussion: 
 This paper presents a novel concept of context structurization to enhance LLM's cognition capability.Extensive evaluations of various representative NLP tasks reveal the consistent enhancement across language model's architectural designs and capacity scales. We demonstrate the feasibility of distilling the structurization ability from giant commercial LLMs into a responsive and private StruXGPT-7B model, addressing the practicality problem. The limitation and future work are discussed in Appendix C. We hope this paper can bring new insights into how to build a more powerful and reliable language model with human cognition and intelligence. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We show consistent improvements across three datasets with various models in Sec. 4. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We reported the compute resource requirements in Appendix A.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discussed the broader impacts of our method in Appendix D.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Please refer to https://github.com/alibaba/struxgpt.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All our experimental data and models are open source and obtainable.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.To further investigate the training sample size and its impact, we have conducted additional ablation studies on StruXGPT's training process, and report the question-answering capability enhancement for the same LLaMA2-7B-Chat model on LongBench's Qasper subset, as introduced in Sec. 4.4. Specifically, for the model trained with 2K samples, the generated outputs frequently contain incorrect formatting and struggle to parse the three-tier structure effectively. In these cases, we resort to using the original text as a fallback input for downstream LLMs, which accounts for the somewhat lower performance drop (-0.6%) compared to the model trained on 5K samples (-1.3%).Criteria. A good structurization should effectively deconstruct the vanilla input text to clearly identify its knowledge structure, and the structurized results should be faithful to the original texts, neither dismissing the factual information nor fabricating statements or opinions that do not exist. Therefore, we devise three dimensions to evaluate the structurization quality, and each of the dimensions can be scored from 0 to 5 (higher is better). The detailed criteria are displayed in Fig. A1 , and the annotating screenshot is provided in Fig. A2 .Assess the quality of structured outputs produced by different sources, and evaluate whether the structured output is sufficiently "good" by using a score ranging from 0 to 5. To clarify what constitutes a "good" structured result, please consider and score the following aspects, with each aspect also scoring from 0 to 5. Scoring Criteria: Please see Fig. A1 .Payment. The payment for annotators is 2,000¥ in total, which is higher than the minimum wage in our country.The content is consistent to the original text, with no fabrications/hallucinations.The structure is clear and standard with appropriate granularity as expected.The core content information is largely retained with the omission of 2-3 descriptive clauses or adverbials permitted.The overall content is faithful, but it includes a few expanded sentences not present in the original text.The hierarchical structure is retained but with minor formatting issues, such as extraneous spaces or line breaks; granularity is generally suitable, with occasional instances of a subsection containing only one description or exceeding seven descriptions.There is an omission of 20% of sentences or the absence of introductory and concluding overview/summary paragraphs.Considerable content is fabricated, such as offering additional large sections of ``suggestion'' text.Poor structure: lack of aspect/description numbering, excessively long aspect name, or overly short description sentences (lists of words); Inappropriate granularity: all sentences attached to one aspect, or dividing each sentence into a separate aspect.There exist considerable omissions of sentences or paragraphs are present, or each paragraph lacks 40% of its sentences.3Severe repetition or meaningless phrases are present, significantly reducing the quality of the result.The basic structure is preserved, but there is a frequent lack of numbering and non-compliance with hierarchical organization.Extensive information loss is evident, with more than half of the original sentences removed.Essentially nonsensical, with only a few words or phrases retained from the original text.Chaotic structure: content is organized without any adherence to hierarchical structure.The majority of information is omitted, with only a few key words or sentences preserved. This passage talks about the PrivacyQA dataset and its characteristics: Detailed dataset description. To focus on the investigation of context structurization, we choose 7 subsets from LongBench across single-document QA, multi-document QA, and synthetic QA tasks in English, and the remaining Chinese subsets or code-orientated tasks are eliminated:• Single-Doc QA. For single-document QA, we take two subsets from LongBench: (1) Qasper [12] , featured by question-answering over NLP technical papers and annotated by NLP practitioners; (2) MultiFieldQA, manually curated from multiple data sources and annotated by Ph.D. students. MultiFieldQA contains 150 Context-Question-Answer triplets to test, and Qasper and other adopted subsets include 200 pieces of test samples respectively. • Multi-Doc QA. Multi-document QA requires LLMs to extract and combine information from multiple documents to derive the answer, which is generally more challenging than single-doc QA. We take three multi-hop QA datasets: (1) HotpotQA [65] , containing 2-hop questions written by native speakers given two related paragraphs; (2) 2WikiMultihopQA [22] , involving up to 5-hop questions synthesized through manually designed templates on Wikipedia passages; and (3) MuSiQue [54] , carefully composed with up to 4-hop reasoning on an increased number of supporting and distracting context evidence. • Synthetic QA. We employ two extra synthetic tasks to test LLM's long-context handling ability on specific scenarios and patterns: (1) PassageCount, requiring models to count the number of unique passages from a shuffled passages pool; and (2) PassageRetrieval, randomly choosing one of 30 passages to obtain its summarization with GPT-3.5-Turbo, and asking tested LLMs to determine the original passage with which the summarization is crafted.Performance analysis on MuSiQue. On the LongBench dataset [3] , ChatGLM-3-32K [67] receives a slight decline (around 0.1%) on the MuSiQue subset [54] when taking structurized contexts as input. The main reason is that MuSiQue is too complicated. It requires up to 4-hop reasoning on manually paraphrased questions on a large amount of supporting and distracting context evidence. For ChatGLM-3-32K that has already achieved a considerable cognition ability, context structurization in the inference stage is temporally unable to bring significant advances in such a complex scenario. Next, we will delve into developing methodologies for training LLMs to capture the intrinsic structure of context, unlocking structurization's potential in enhancing LLMs on more complicated NLP tasks.Aspect-Based Summarization (ABS) and Query-Based Summarization (QBS) are classical context augmentation methods in the NLP literature, which aims to gradually extract important information from lengthy text data based on pre-defined aspect list or user-input queries. The summary that contains key information will be integrated into context inputs to augment tested LLMs to generate reliable responses, rather than lost in irrelevant sentences in the original lengthy inputs. We take ABS and QBS as the comparative baseline to further evaluate the efficacy of our proposed structurization augmentation strategy.We investigate ABS and QBS on five document-QA subsets in LongBench [3] . Specifically, as the pre-defined aspect list is unavailable for each passage in LongBench, we turn the traditional ABS to input-agnostic paragraph-based summarization to lengthy paragraphs, where the prompt template is: Summarize the following text with no more than three sentences. Passage: {text}; Summary: . For QBS, the prompt template focuses on specific queries: Summarize the following text to answer the query with no more than three sentences. Query: {query}; Passage: {text}; Summary:. The {text} and {query} are placeholders, and the output summaries are respectively concatenated to form the augmented passages as inputs. We report the performance variance of the same test model (i.e., LLaMA2-7B-Chat [53] ) when taking vanilla, summarized, and our structurized passages as inputs to answer the same given questions, as displayed below (due to the space limitation, we merely report the averaged enhancement on the five subsets). Based on Tab. A2, we observed that ABS, without guidance from pre-defined aspects lists or userinput queries, failed to preserve critical information, and led to performance declines in four out of the five subsets. Conversely, QBS, with user query guidance, achieved improvements in the Qasper, HotpotQA, and Musique subsets. However, substantial information loss during the summarization process resulted in decreased performance on the MultiFieldQA subset. In contrast, our structured approach delivered consistent improvements across all subsets, demonstrating its efficacy.In addition, ABS and QBS are applicable for passage-based question-answering tasks but do not extend well to other tasks such as hallucination assessment, where every piece of information counts and should not be summarized. Our approach is task-agnostic, highlighting the generalizability.Qwen models as the LLM evaluators. In Sec. 4.2 in the manuscript, we mainly study structurization with LLaMA2 and GPT-3.5-Turbo on the AttrScore benchmark. For a thorough comparison, this section provides the investigation of employing another series of models, i.e., Qwen-7B and Qwen-72B [2] , on AttrScore and FactScore datasets to further demonstrate our method's efficacy. Results are presented in Tab. A3 and Tab. A4, respectively.For the 7B-and 72B-parameter Qwen evaluators, structurization can immediately improve judgment accuracy on both AttrScore and FactScore datasets. The smaller Qwen-7B model receives a more significant enhancement, along with a nearly 10% improvement in all metrics (except the Attr. criteria in Tab. A3) on the two datasets. It indicates smaller LLMs may be more reliant on structurization to enhance their cognitive and information-seeking capabilities than large-scale LLMs. Simultaneously, the larger Qwen-72B evaluator also witnesses a considerable improvement with a 3%-4% increase.In particular, based on the structurized references, Qwen-72B achieves a competitive evaluation performance against GPT-3.5-Turbo on the FactScore benchmark, and even slightly outperforms GPT-4 on the Contra. criteria in AttrScore. Those results substantiate the effectiveness of leveraging structurization to further enhance powerful language models with considerable size capacities. Accordingly, our method can also boost the Llama-7b baseline for 3%-7% without training, while AdaptLLM requires costly continual training of the baseline model on each domain corpus. In particular, for the general reading comprehension task, AdaptLLM-Fin does not introduce significant boosts, while AdaptLLM-Bio/Law even cause performance drops, which is mainly because AdaptLLM's domain-adaptation tuning will harm the general capability more or less. In contrast, our method does not alter the baseline model, but only structurizes the input context to enhance LLM's cognition ability on downstream tasks, showing stable and consistent improvements (e.g., a 2.5% increase on BoolQ). Although our final performance is slightly inferior to the domain-specialized AdaptLLM, our generalizability emphasizes the contribution of our work, as we bring consistent enhancement across downstream domains and cause no degradation on general tasks.Training-time aggregation. This paper mainly investigates structurization during LLM's inference stage. Despite the universal enhancement across language models and NLP tasks, structurization has not yet fully tapped into its potential. Specifically, developing methodologies for training LLMs to capture the intrinsic structure of context and extrinsic correlations among training corpus presents an unresolved challenge in the field. We will delve into this problem in our future work.Inference efficiency. We acknowledge that the introduced inference cost is a common limitation for the methods adopting LLMs for data augmentation, which depends on computing resources and the optimization techniques employed. Here we conduct extra experiments with the competitive augmentation methods (i.e., ABS and QBS in Appendix B.3) for an in-depth comparison on the Long-Bench [3] dataset for question-answering evaluation. Besides, we also investigate the AttrScore [66] dataset for hallucination evaluation to assess the impact of input length and output length, as well as the model size. The inference time, measured in seconds per sample, is calculated on an NVIDIA A100 GPU with vllm6 acceleration (except for the LLaMA2-70B model, which demands at least two A100 GPUs for deployment). LongBench contains input passages with an average length of 14K tokens (computed with the LLaMA2 tokenizer), which burdens the cost for all the data augmentation methods. According to Tab. A8, only the task-specific QBS can bring a positive enhancement of 0.2% on average, which is negligible compared with the 2.5% boost by our task-agnostic structurization approach. However, the LLM-based data augmentation methods generally introduce an extra 2.5-2.9 seconds cost. To alleviate this problem, we have recently noticed a new decoding acceleration technique named Selection-of-Thought (SoT) [45] , which is also motivated by the aspect-based thinking and writing process of humans and is naturally compatible with our structurization process. With the 2.39× speed up and negligible quality decrease (denoted as " † Ours + SoT"), the extra cost of our method can be further reduced to an acceptable 1.2 seconds, which we believe makes it more practical. AttrScore requires a relatively longer output length (with a detailed explanation for hallucination evaluation), and the vanilla time cost increases compared to LongBench. According to Tab. A9, our structurization earns a 6.4% gain on the average hallucination evaluation accuracy (detailed values are shown in Tab. 2), in trade of the 1.5× increased time cost for the vanilla LLaMA2-7B evaluator.As the base evaluator is scaled up to a larger LLaMA2-70B model, our method merely introduces 12% extra cost (2.1s) and still gains a considerable enhancement of 4.3%. Our method showcases more advantages by incorporating larger base models.Performance on General Evaluation Benchmarks As our method focuses on context structurization to enhance LLM's cognition ability, for the tasks where no context is provided, our method may not bring significant enhancement and even get worse. The commonly used MMLU benchmark [21] is a typical scenario, where LLMs are asked to answer questions without context references but requiring their parametric knowledge (learned during large-scale pre-training), and context structurization does not help. If we insist on structurizing the question alone and feeding it into LLM's inputs, the model may be disturbed by the introduced information from StruXGPT and generate wrong answers. As shown in Tab. A10, our method causes a slight 0.1% decrease (measured by OpenCompass protocol 7 ) when taking LLaMA2-7B-Chat [53] as the baseline model. On the other hand, we have also tested another common benchmark, BBH [49] , which is designed to evaluate LLMs' reasoning capability when dealing with several logical sentences/statements. In this case, our method can adapt well to highlight the logical relations and boost LLM's reasoning abilities by 0.8%.In conclusion, we suggest users apply structurization to long-form or logically complex contexts, while taking the original question as inputs when there is no context provided.We discuss the positive and negative societal impacts as follows:Positive Societal Impacts. Through the innovative approach of context structurization, this work significantly enhances the comprehension abilities of Large Language Models (LLMs), leading to more effective and efficient applications in various sectors such as education, healthcare, and customer service. Avoiding the need for larger models, it not only curtails the environmental impact associated with training sophisticated AI systems but also democratizes access to cutting-edge AI technologies. This fosters a broader base for innovation and empowers smaller entities with the tools to contribute meaningfully to technological advancement. Moreover, our method, rooted in human cognitive processes, promises to enrich human-AI collaboration, paving the way for solving complex societal issues by leveraging AI's improved problem-solving capabilities.Negative Societal Impacts. On the flip side, the advances made through structurization bear potential risks, including the augmentation of disinformation campaigns and the creation of more sophisticated deepfakes, which could undermine trust in digital content. The enhanced capabilities of LLMs might also be co-opted for intrusive surveillance, raising substantial privacy and ethical dilemmas. Furthermore, the inherent biases in training data could be deepened, potentially automating and perpetuating social inequalities. The concentration of advanced AI technologies in the hands of a few could limit competition and place immense power over information dissemination with those entities, while increasing reliance on AI for critical decision-making could inadvertently erode human analytical skills over time. To mitigate these concerns, it is essential to establish rigorous ethical standards and robust oversight mechanisms that govern the deployment of LLMs, ensuring transparency and accountability in their use. Furthermore, active research and the implementation of bias detection, correction frameworks, and the development of media literacy programs are crucial steps to preemptively address the potential misuse, privacy violations, and societal impact of these advanced AI systems.E.1 Prompt Templates Fig. A4 displays the few-shot prompt template to query giant commercial LLMs to execute structurization. We first introduce the instruction for structurization (including the task definition and output formats), then provide two representative examples to further illustrate the process. The first example in Fig. A5 tells the model to focus on the existing numerical or enumeration indicators to assist in constructing the aspect-description structure. The second example in Fig. A6 teaches the model to automatically summarize the aspects and attach their corresponding descriptions from the raw text sequences. Finally, in Fig. A4 we ask commercial LLMs to structurize the input_statement from user input as expected.To ensure completed and faithful structurization on long-form input statements (usually with thousands of tokens), we provide two detailed exemplars in Fig. A5 and Fig. A6 for commercial LLMs' few-shot learning. However, these two exemplars take a considerable context length, which may even confuse the distilled StruXGPT-7B with a smaller size capacity. We thus remove the examples in StruXGPT-7B's prompt template, as StruXGPT-7B has learned how to perform structurization as desired and the functionality of those examples is negligible. Specifically, the example_1 and example_2 (with their introductions) are removed for both efficiency and efficacy.In this section, we present several structurization examples from different models as discussed in Sec. 3 in the manuscript. Given an input statement about Facebook's stock, Fig. A7 shows that Qwen-max produces a factual and faithful structurization result with the help of few-shot exemplars. It clearly identifies the main aspects of the original statement, and attaches the corresponding descriptions into each aspect. On the contrary, Fig. A8 indicates the smaller pre-trained 7B-size Qwen and LLaMA2 models failed to follow the instructions to produce qualified structurizations. There exists a fabricated "Note" content in Qwen-7B's response. Meanwhile, LLaMA2-7B is even unable to follow the desired format, and information like "Facebook doesn't sell anything tangible" is missing. Fig. A9 demonstrate the effectiveness of our fine-tuned 7B-parameter StruXGPT model, which is able to structurize the input statement as expected, with comparable performance to the giant Qwen-max model with over 200B-size.You are a helpful NLP assistant.Help me rephrase a given statement to identify:(1) a summary of the statement's scope, which should generally be a noun phrase with a few words.(2) a list of main aspects on which the statement is discussing. Each main aspect should be a noun or noun phrase.The aspect list should be precise and concise to conclude the statement, and the number of aspects should be limited.(3) an enumeration of descriptive sentences regarding each aspect above, which display the details of those aspects.Each description sentence must be completed and faithful to the original statement. You should NOT remove any descriptive segment in this layer.Given an original statement, the rephrased structure should strictly follow this format: ere is an example to illustrate how to rephrase an input statement as the desired structure. Pay attention to numerical or enumeration indicators, like '1. ', '2. ', '(1) ', '(2) ', '-', etc. {example_1}## Statement'Here is another example to illustrate how to rephrase an input statement as the desired structure. When there are no explicit words indicating the statement's scope and main aspects, please use a few words to precisely summarize the scope as well as the main aspects. Then you may carefully attach the descriptive sentences to each main aspect. {example_2} Now summarize the "scope" of the following statement with a few words, and then rephrase the input statement to its "main aspects" and "corresponding descriptions" in the numerically ordered format strictly. Note that the aspect list should be precise and concise to conclude the statement, and the number of aspects should be limited. Each description sentence must be completed and faithful to the original statement, and you should NOT remove any descriptive segment in this layer. # Input: ``T he water absorption curve of rice after soaking is an unimodal curve. The inflection point of the curve is the peak period of water absorption. The relationship between rice water absorption and time is non-linear and can be expressed by the following formula: a*t+b=c, where a, b, and c are constants. Under different humidity conditions, the change in the water absorption rate of rice with time is basically similar, that is, the water absorption rate is between 0 and the point d inflection, and the rate of change accelerates over time. After the inflection point, the increase in water absorption gradually stabilizes. The changing rules of water absorption and moisture content of rice are similar, but under different humidity conditions, the relationship between water absorption of rice and moisture content is different. When the moisture content is low, the water absorption of rice increases as the moisture content increases. When the moisture content is high, the increase in water absorption of rice gradually stabilizes. There are three obvious steps for rice seeds to absorb water: First, at the beginning of water absorption, the water content of the seeds gradually increases, and the water absorption rate slowly increases. Second, during the peak water absorption period, the water absorption rate increases rapidly. Third, in the later stage of water absorption, the water content of seeds slowly increases, and the water absorption rate increases. The water absorption rate of rice seeds is closely related to temperature. In general, water absorption increases as temperature increases. The relationship between water absorption and temperature can be expressed as: unsaturated water absorption (%) = 1.2 The price increase can be unstable, as the company's value is dependent on user growth and trends. 2. Evaluation of Facebook's business model 2.1 Facebook's revenue is primarily generated through advertising and is highly dependent on user behavior and trends.2.2 Their income is single-product oriented, so they are vulnerable to changes in market conditions. ``Ǹ ote: The original statement was quite long and complex. It discusses the influence of hype and uneducated investors on stock prices, before moving on to an evaluation of Facebook's business model. The scope of the statement includes the analysis of Facebook stock, but it could be more precise by narrowing it down to just "stock prices." This is why the first part of the rephrased structure includes the phrase "market analysis of Facebook stock." In the second part, I identified two main aspects: the influence of hype and uneducated investors on stock prices, and the evaluation of Facebook's business model. Each aspect is further divided into several descriptive sentences that are faithful to the original statement. } • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://openai.com/https://dashscope.aliyun.com/https://rank.opencompass.org.cn/leaderboard-llm-v2We use the 20231020-en dump from https://dumps.wikimedia.org/.An open platform at https://www.aliyun.com/product/bigdata/learn/itaghttps://github.com/vllm-project/vllmhttps://github.com/open-compass/opencompass 


Decision: This paper introduces enhancing the comprehension capabilities of LLMs by structuring contextual information hierarchically. The authors propose a three-layer structure to summarize input text into a Scope-Aspect-Description format. They optimize a 7B model (StruXGPT) by distilling the structuring capabilities of LLMs to reduce computational costs. The method improves performance in some NLP tasks, including QA and passage-level dense retrieval, and the fine-tuned 7B model inherits most of the structuring capabilities of large LLMs. The aurhors findings also suggest that the structured prompt transformation reduces model hallucinations and improves performance in context-based QA benchmarks. 
There were several questions about the experiments, and strengthening of the findings through better ablations was asked. The authors have provided additional evidence by including different model sizes, and included additional evidence on common benchmarks (such as MMLU and BBH). Reviewers have asked comparison to different models, which authors have agreed to include in the final submission.

Reviews: review: paper_summary: paper_summary: This paper introduces the concept of context structurization to enhance the comprehension capabilities of large language models (LLMs) for long texts. The authors propose summarizing the input text into a three-layer structure of Scope-Aspect-Description using LLMs, and then inputting this three-layer structure as an enhanced version into the LLM. Additionally, the authors propose to optimize a 7B model by distilling the structuring capabilities of giant commercial LLMs to reduce computational costs. The effectiveness of the method is validated through experiments on multiple NLP tasks. The experiments also demonstrate that the fine-tuned 7B model can inherit most of the structuring capabilities of the giant commercial LLMs.
strength_weakness: strength_weakness: The experiments are not sufficiently comprehensive. Firstly, many tables only compare the scenarios with and without StruXGPT (except for Table A2 and A5), without comparing against other advanced prompt engineering methods. Therefore, it is hard to determine whether StruXGPT enhances the cognitive abilities of LLMs more effectively than other methods. Secondly, the authors did not validate the effectiveness of StruXGPT on state-of-the-art LLMs such as GPT-4, so it is unclear whether GPT-4 would still perform well without context structurization.

There are also some related summarization-based augmentation methods that could be discussed or compared [1].

[1] Cheng, Daixuan, Shaohan Huang, and Furu Wei. "Adapting Large Language Models via Reading Comprehension." In The Twelfth International Conference on Learning Representations.
questions: questions: How does the number of in-context examples affect the results of structurization, and why was the number set to 2?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper presents a novel approach to improve the cognitive capabilities of large language models (LLMs) without inferring the model by structuring contextual information hierarchically. The authors propose transforming plain, sequential text into a structured format, enabling LLMs to process and understand complex information more effectively. This method is empirically tested across various NLP tasks, demonstrating consistent performance gains. Additionally, the paper introduces StruXGPT, a distilled 7B model that efficiently executes the structurization process, enhancing the results on context-based QA benchmark, Hallucination Evaluation and Passage-level dense retrieval.
strength_weakness: strength_weakness: 1. Lack of parameter-efficiency analysis: authors provide 7B model for structurization process, but it is not clear whether smaller or bigger models would significantly hurt or benefit the approach.
2. While the proposed approach shows stable improvements, it is still a lot of additional parameters, which is a concern for scalability to large amount of texts.
questions: questions: 1. Did you experiment with smaller/bigger models for structurization? If yes, what were the results? 
2. Are there specific tasks in which such structurization could be worse? 
3. Did I understand correctly that the data for training StruXGPT was gather by quering bigger model? Did you analyze the training set in details and do you think the proposed approach would benefit from training data gather or annotated by humans? I am personally worried here about the hallucinations that could be inherited from the bigger model.
4. Do you plan to publish your StruXGPT model? 

Also in Table 1 you write "The prefix Struct- indicates..." but there is not such prefix in the table. Did you mean "+StruXGPT (ours)" here?
limitations: limitations: 1. Paper investigates only inference stage 
2. Introducing StructXPGT is still additional computation cost, however authors provide the analysis of extra cost in appendix
paper_summary: paper_summary: This paper proposes a new technique for prompting Large Language Models (LLMs) called StruXGPT. The basic idea is to transform the original prompt into a more structured description of the request which contains three levels of information: Scope, Aspects, and Descriptions. While the **Scope** provides an outline summary of the request, the **Aspects** present an itemized and well-ordered list of topics that are associated with their respective **Descriptions** which provide the details. In the paper, the prompt transformation is achieved by either using another large model or by a smaller model obtained by knowledge distillation. The experiments show that this type of structure can help LLMs find the right information needed to answer the requests, improving performance in question-answering tasks and reducing model hallucinations.
strength_weakness: strength_weakness: In the abstract and introduction of the paper, there are claims about how human cognition works that are poorly backed up. This is an area of debate and I believe this discussion could be avoided without any harm to the main point of the paper.

The proposal assumes that the prompt transformation process is an easier task for an LLM than directly addressing the request. Although this seems to be the case for question-answering problems considered, there is no discussion about cases in which this may not be adequate. I believe more experiments with other types of datasets would help us to better understand that.

Finally, as there are other papers investigating similar approaches, the novelty of the approach is not a strong point.
questions: questions: The legend of Table 1 mentions that the prefix Struct- indicates the data fed into LLMs is structured by StruXGPT-7B. However, I don't see this prefix in the table.
limitations: limitations: Yes, the limitations were well addressed in the paper.
paper_summary: paper_summary: The paper presents a method to improve the cognitive capabilities of large language models (LLMs) by organizing input context into a hierarchical structure. The method is called context structurization, and it involves transforming unordered contextual sentences into hierarchically structured elements to mimick human cognitive processes. The paper shows the effectiveness of the approach through evaluations on various NLP tasks, such as context-based question-answering, hallucination evaluation, and passage-level dense retrieval, using different model architectures and sizes. The study shows performance improvements and introduces StruXGPT-7B, a distilled model, to perform structurization efficiently.
strength_weakness: strength_weakness: - The performance improvements are heavily dependent on the quality of the structurization process, so a poor structurization could lead to suboptimal model performance or even confusion.
- Even though there is a performances improvement the process of structurizing context might introduce additional computational overhead. 
- The approach lack a bit of novelty, it doesn't introduce fundamentally new concepts and resembles methodologies from previous studies.
- The study relies on metrics such as ROUGE-L and human evaluation, that are useful but they might not fully capture the more complex improvements, such as a cognitive-like processing that structurization aims to achieve.
questions: questions: - How does the proposed three-layer structure handle texts with more complex relationships or multiple topics?
- What are the computational costs associated with context structurization?
limitations: limitations: - The study heavily relies on specific models like LLaMA2 and Qwen, so i wonder about the generalizability of the results to other LLMs.
- The structurization may be less effective working with extremely long contexts or when the context cannot be easily divided into clear hierarchical segments.

Comments: title: Response for the rebuttal
Response for the rebuttal
Thanks for your timely feedback
Thank you for your reply.
comment: Dear Reviewer 7Sjn,

Thank you for your positive feedback and for raising your score! We will include more comparisons in the final version.

---

Best regards,

Authors
Thank you for providing more results. I have raised my score. I look forward to seeing more comparisons with other approaches.
Dear Reviewer 7Sjn,

We sincerely thank you for your feedback.  As it takes us some time to re-produce AdaptLLM's results and implement our method, we have currently compared with AdaptLLM on three subsets respectively in the medicine/finance/law domain.

|  Domain  |   Subset  |    Metric   |  Baseline |  AdaptLLM |    Ours   |
|:--------|:---------|:-----------|:---------|:---------|:---------|
| Medicine |  PubMedQA |     Acc     |    59.6   |    **63.3**   |    63.0   |
|  Finace  | ConvFinQA |      EM     |    29.2   |    **41.5**   |    36.5   |
|    Law   |   SCOTUS  | mic-F1/mac-F1 | 28.3/10.8 | 30.0/**17.8** | **30.6**/15.6 |

According to the results above, our method can also boost the Llama-7b baseline for 3%-7% **without training**, while AdaptLLM requires _costly continual training_ of the baseline model on each domain corpus. Although our final performance is slightly inferior to the domain-specialized AdaptLLM, our **generalizability** emphasizes the contribution of our work (we bring _consistent enhancement across downstream domains_ and cause _no degradation on general tasks_, as stated in our previous response).

Due to the time limitation, we can temporally present the performance on three subsets (plus the aforementioned extra general subset).  In our revised paper, we will provide further comparisons on other datasets and with other approaches to emphasize our method's efficacy.

We hope those results can address your remaining concerns.  And if there is any further question, please do not hesitate to tell us.

---

Best regards,

Authors
I appreciate the authors' responses to my questions, which addressed part of my concerns. However, merely comparing chain-of-thought (CoT) and only comparing AdaptLLM on a single dataset is not sufficiently convincing for me. In AdaptLLM paper, there are other datasets where AdaptLLM performs significantly better than the baseline. How does the authors' method perform on these datasets?
Dear Reviewer, 

We sincerely appreciate your acknowledging our responses, and we will continue to polish our paper based on your and other reviewer's constructive suggestions.

---

Best regards,

Authors
Your explanations are very clear, and I appreciate that you plan to add the experiments and discussions about W1 in the final version. Because of this, I’ve decided to raise my rating to 6.
Dear Reviewer 8sM9,

We are glad that our detailed responses and additional experiments have helped to address your concerns and clarify any lingering questions.

Once again, thank you for your time and valuable feedback.

Best regards,
Authors
Thank you for your quick and very detailed reply, full of convincing experiments. I believe the score is already high enough, but you have fully answered all the questions. I find it very interesting that increasing the model size affects results in such a way, implying that probably bigger models learns structurization by itself.

Best wishes to your paper, and let me know if I can be of any help.