Title: Enhancing LLM’s Cognition via Structurization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models

Structurization

Augmentation
Abstract: When reading long-form text, human cognition is complex and structurized. While large language models (LLMs) process input contexts through a causal and sequential perspective, this approach can potentially limit their ability to handle intricate and complex inputs effectively. To enhance LLM’s cognition capability, this paper presents a novel concept of context structurization. Specifically, we transform the plain, unordered contextual sentences into well-ordered and hierarchically structurized elements. By doing so, LLMs can better grasp intricate and extended contexts through precise attention and information-seeking along the organized structures. Extensive evaluations are conducted across various model architectures and sizes (including a series of auto-regressive LLMs as well as BERT-like masking models) on a diverse set of NLP tasks (e.g., context-based question-answering, exhaustive hallucination evaluation, and passage-level dense retrieval). Empirical results show consistent and significant performance gains afforded by a single-round structurization. In particular, we boost the open-sourced LLaMA2-70B model to achieve comparable performance against GPT-3.5-Turbo as the halluci- nation evaluator. Besides, we show the feasibility of distilling advanced LLMs’ language processing abilities to a smaller yet effective StruXGPT-7B to execute structurization, addressing the practicality of our approach. Code is available at https://github.com/alibaba/struxgpt.

Enhancing LLM's Cognition via Structurization

Abstract

When reading long-form text, human cognition is complex and structurized. While large language models (LLMs) process input contexts through a causal and sequential perspective, this approach can potentially limit their ability to handle intricate and complex inputs effectively. To enhance LLM's cognition capability, this paper presents a novel concept of context structurization . Specifically, we transform the plain, unordered contextual sentences into well-ordered and hierarchically structurized elements. By doing so, LLMs can better grasp intricate and extended contexts through precise attention and information-seeking along the organized structures. Extensive evaluations are conducted across various model architectures and sizes (including a series of auto-regressive LLMs as well as BERT-like masking models) on a diverse set of NLP tasks ( e.g. , context-based question-answering, exhaustive hallucination evaluation, and passage-level dense retrieval). Empirical results show consistent and significant performance gains afforded by a singleround structurization. In particular, we boost the open-sourced LLaMA2-70B model to achieve comparable performance against GPT-3.5-Turbo as the hallucination evaluator. Besides, we show the feasibility of distilling advanced LLMs' language processing abilities to a smaller yet effective StruXGPT-7B to execute structurization, addressing the practicality of our approach. Code is available at https://github.com/alibaba/struxgpt .

1 Introduction

Large language models (LLMs) have emerged with remarkable language capabilities [6, 53, 1], yet remain at a discernible distance from human-level intelligence, especially when handling long-form, sophisticated contexts as inputs [3, 36]. Scaling up the model size has significant benefits for boosting context-comprehension and instruction-following abilities for LLMs [47, 58]. However, it is generally resource-intensive on both model training and inference. This paper presents another perspective on enhancing LLMs' cognition capability without altering the models: context structurization .

The idea of structurization is motivated by neurocognitive science [51, 5, 17]. In human cognition, as indicated in Fig. 1, sophisticated text sequences will be processed and consolidated into a structured knowledge tree, with factual elements well-organized hierarchically [28, 15]. This process is defined as structurization . People can precisely search information from general concepts to specific details and make connections and comparisons along structures. We thus aim to transform plain texts into structurized inputs, helping LLMs recognize and understand contexts in a human manner [71].

As Fig. 1 illustrates, input sequences are reorganized in a simple but generic three-layer structure: scope , aspects , and descriptions . The scope summarizes the topic and contents, unfolding into several main aspects with corresponding detailed descriptions. The structurized results can be freely

assembled into various natural language forms, depending on the specific downstream tasks. Fig. 2 provides an exemplar of our overall framework: after structurizing and reassembling the vanilla context to highlight its scope and aspects of data imbalance handling techniques, LLMs become able to grasp the target information about the dynamic weighting strategy and generate reliable responses.

We first execute the structurization by prompting advanced commercial LLMs ( e.g. , GPT-3.5Turbo 1 or Qwen-Max 2 ) with a few examples, and then collect the results to train a smaller 7B-parameter LLaMA2 [53] or Qwen [2] model as our StruXGPT. This is motivated by [48], where the fundamental syntactic processing ability from giant LLMs is distilled into a responsive and affordable StruXGPT-7B. Comprehensive evaluations indicate that StruXGPT-7B inherits 97% of the structurization ability from the teacher model, showing our method's feasibility.

Empirical experiments are conducted on a diverse set of NLP tasks ( e.g. , context-based question-answering, exhaustive hallucination evaluation, and passage-level dense retrieval). The results show that with a single-turn structurization by our StruXGPT, the cognition performance of vanilla large language models witnesses consistent improvements regardless of the model architecture and size variation. In particular, we boost the open-sourced LLaMA270B [53] to achieve comparable performance

The

idea

of

language

identification

is

to

classify  a  given  audio  signal  into  a  particular

class

using

a

classification

algorithm.

Commonly  language  identification  task  was

done  using  i-vector  systems.  A  well  known

approach for language identification proposed

by N. Dahek et al. uses the GMM-UBM model

to  obtain  utterance  level  features  called  i-

vectors. Recent advances in deep learning have

helped to improve the language identification

task

using

many  different

neural

network

architectures  which  can  be  trained  efficiently

using GPUs for large scale datasets …

Context

Perceiver

Scope

:

Language Identification

Aspect: Method Overview

Desc: i-vector is a well-known

approach using GMM-UBM …

Aspect: Pooling Strategies

Aspect: Model Architecture

Desc: Deep learning advances

have improved language…

Desc: NetVLAD pooling:

replaces hard assignment…

Desc: GhostVLAD pooling:

An extension of NetVLAD…

Desc: convolution layers with

ReLU activation functions…

Desc: feature cube is converted

into a 2D feature maps…

Cognition

against GPT-3.5-Turbo as a hallucination evaluator, and demonstrate the compatibility with other advanced prompting techniques, such as CoT [59]. We hope this paper can bring new insights to the community on building a more powerful and critical language model with human cognition.

Our contribution can be summarized as follows:

We propose the concept of structurization, in order to enhance LLM's cognition capability without altering the models themselves.

We present the feasibility of distilling the structurization ability from giant commercial LLMs into a responsive and affordable StruXGPT-7B model, making our approach practical.

With structurization, we empirically demonstrate the consistent cognition enhancement for various LLMs across model architecture and size variation on diverse NLP tasks.

2 Related Work

Large language models (LLMs) . LLMs' emergent abilities [58] has recently received extensive attention in the literature [6, 61, 9, 53, 23, 1], which are found closely related to the scaling law [26]. When the scale reaches a certain level, the performance on complex NLP tasks significantly rises due to the superior in-context learning , instruction following , and reasoning abilities [71, 47]. Numerous efforts have been made to boost the model capacity with training and prompting strategies [37, 38, 59, 14], and this paper presents a new perspective, context structurization , to encourage LLMs to perceive, recognize, and communicate like humans [55, 71], without altering the model themselves.

Context augmentation . Recent studies have proposed several context augmentation methods to enhance LLM's cognition ability [59, 41] on when taking the long-form context (with thousands of tokens) as inputs [3, 36, 30]. Specifically, aspect-based summarization (ABS) and query-based summarization (QBS) [64, 69] are designed to extract important information from lengthy text data as well, but they require pre-defined or user-input aspect/query lists to conduct targeted summarization, and the detailed information will be inevitably lost during the summarization process. In contrast, the

Vanilla

Structurized

How are weights dynamically adjusted?

The article does not provide information.

To address the dominating influence

of easy-negative examples, each

training sample is assigned a weight

proportional to $(1-p)$ …

This passage talks about the data imbalance handling in NLP

tasks, which involves several aspects:

1. Introduction to data imbalance.

A common issue in NLP tasks …

2. Training objectives.

CE or MLE objectives are widely adopted …

3. Handling easy-negative examples:

A dynamic weight adjusting

strategy is proposed to associates each training example with …

4. Combining strategies.

It significantly boosts performance …

StruXGPT

Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine

reading comprehension. … Data imbalance results in the following two issues: (1) the training-

test discrepancy: … The cross-entropy objective (CE for short) or maximum likelihood (MLE)

objective, which is widely adopted as the training objective for data-imbalanced NLP tasks …

Inspired by the idea of focal loss in computer vision, we propose a dynamic weight adjusting

strategy, which associates each training example with a weight in proportion to $(1-p)$   …

Combing both strategies, we observe significant performance boosts on a wide range of data

imbalanced NLP tasks….

Context

LLM

Response

paper develops context-wise structurization to highlight the knowledge structure, running a single-turn structurization on the context to enhance LLMs' cognition abilities on a diverse set of NLP tasks.

Structurization . In the conventional NLP literature, the term structured data usually refers to entity-relation-entity triplets or properties extracted from plain texts, which are utilized to construct knowledge graphs or databases with special data formats or schemas [24, 32]. On the contrary, the structurization in this paper does not focus on entity-level information extraction and aggregation. Instead, it suggests reorganizing the input sentences into a three-layer structure based on their inner linguistic relations. Similar to discourse analysis and constituency parsing [10, 27], the main purpose of our structurization is capturing the dependencies and relations of the elements within specific long-form text inputs, so as to enhance LLMs' cognition of the knowledge structure and relations.

Knowledge Distillation . In the era of large language model learning, distilling specific knowledge from giant LLMs' outputs has commonly been used to derive a more affordable but still powerful language model [63, 19]. Previous attempts, such as Standford Alpaca [50] and Vicuna [8], show the feasibility of collecting instruction-response pairs from GPT-3.5 or GPT-4 to train a smaller fundamental [46] or domain-specific ( e.g. , with reasoning [43] or coding [20] skills) language models. Our work also distills the structurization capability from the giant Qwen-Max model to a smaller yet effective StruXGPT-7B model, with extensive evaluations to show the efficiency and efficacy.

3 Structurization

The substantial purpose of structurization is to mimic the human cognition process and transform plain, sequential text sentences into a well-organized, hierarchical knowledge structure. Inspired by the linguistic discourse analysis [11, 18], this paper develops a three-layer hierarchy exemplar to present the cognition/knowledge structure, as introduced in Fig. 1:

Scope summarizes the topic and boundary of the textual context. It outlines the central issues of knowledge throughout the text and the scope of the discussion that will be covered.

Aspect further subdivides the input context into several parts. It presents the aspects or dimensions that must be considered to fully understand the topic and scope.

Description is the most specific and detailed layer. It provides in-depth descriptions and analyses to support each aspect of the context scope.

This generic three-layer structure is derived for efficacy and efficiency in dealing with diverse textual inputs. There might be some elaborated structures (such as a knowledge mindmap [60]) to better deconstruct specific text sources, but the difficulty and complexity of defining, extracting, and utilizing those structures to aid in practical problems are dramatically increased. We leave this exploration in our future work.

Next, we will present how to implement effective structurization with minimal cost in Sec. 3.1, and demonstrate the utilization of the structurization results to enhance LLMs' cognition abilities in different downstream tasks in Sec. 3.2.

3.1 Efficient Implementation of Structurization

We have explored two approaches to execute the structurization process by leveraging the extraordinary capability of large language models: few-shot prompting on commercial LLMs and direct instructing our developed StruXGPT models.

Few-shot commercial LLM. In the initial stage, we use two in-context examples to query the commercial Qwen-Max model to structurize the input corpus, since it shows promising instructionfollowing and textual analyzing capability with over 200B parameters 3 . Here is a simplified example of prompting structurization in Fig. 3, and the full template is displayed in Appendix E.1.

However, commercial LLMs are usually slow and expensive, and sending user data to LLM APIs may cause privacy and security problems due to information leakage. Thus, we train a smaller 7b-parameter model ( e.g. , LLaMA2 [53] or Qwen [2]) to inherit the structurization ability from giant commercial LLMs, which can be deployed locally for efficiency and privacy.

Fine-tuned StruXGPT. Our tailored model is named by StruXGPT, where Stru is the abbreviation of structurization, and X implies we do not specify the model architecture. We carefully curate 22,547 raw data pieces from Wikipedia 4 and CAMEL-AI dataset [31] to ensure diversity, and collect the structurized results from QwenMax to train our StruXGPT via supervised finetuning (SFT). From the collected samples, 200

are utilized for evaluation (including human verification), and the remaining training samples are adopted to distill the structurization capability from Qwen-Max to our StruXGPT-7B. It is practical since the structurization only relies on fundamental syntactic understanding and processing ability, which has already been learned from the large-scale corpus. We merely teach the 7B-parameter models how to reorganize the input text via SFT, without introducing new memorizing or creative overloads. In addition, the ultimate StruXGPT does not require few-shot examples, and it reduces the input lengths for further efficiency. The training details are described in Appendix A.1.

3.2 Effective Utilization of Structurization

The identified knowledge structure ( i.e. , the scope, aspect, and description hierarchy) from the raw context is initially parsed in JSON format and unsuitable for direct inputs to handle massive lengthy elements. Therefore, we use a unified template to transform the structured data back into natural language sentences as models' inputs to fit their intrinsic processing patterns, as LLMs are pre-trained and aligned with mostly natural language data. Concurrently, to preserve and highlight the knowledge structure, we harness specific linguistic markers to signal hierarchy and relationships among concept elements, such as numbered lists for order, bullet points for categorization, and indentation to depict nesting levels of information. Fig. 4 showcases some typical examples.

The first row of Fig. 4 provides a unified template to transform structurization results into natural languages. The top-level hierarchy scope is presented as a standalone sentence, serving as the introduction to the structured context and highlighted with bold markers. Subsequently, the secondary aspect are organized with numerical markers and bolded, attaching with its corresponding tertiary descriptions through subclauses or separate sentences. This method not only signifies the rank and relation of each piece of information relative to others but also provides clear, navigable paths for the LLMs to follow and process the information efficiently ( e.g. , when examining the long-form comprehension capability). Moreover, the second row of Fig. 4 introduces another variation for transformation, where each description elements are further broken down and enumerated for the delineation of fine-grained details, making it easier for language models to discern and retain specific nuances associated with each aspect ( e.g. , when examining the hallucination detection capability).

The life and career of director George Cukor can be deconstructed as:

1. Early life and background

-

Cukor was born on the Lower East Side of Manhattan, the younger child

and only son of…

-

His parents selected his middle name in honor of Spanish-American War

2. Career milestones

-

Cukor won the Academy Award for Best Director for "My Fair Lady"

(1964), which was …

-

He continued to work into the 1980s….

This passage talks about {Scope}:

1.

**{Aspect 1}**:

{Desc 1.1}. {Desc 1.2}.

2.

**{Aspect 2}**:

{Desc 2.1}. {Desc 2.2}.

3.

**{Aspect 3}**:

…

4. …

{Scope} can be deconstructed as:

1.

**{Aspect 1}**

-

{Desc 1.1}.

-

{Desc 1.2}.

2.

**{Aspect 2}**

-

{Desc 2.1}.

3. …

This passage talks about the PrivacyQA dataset and its characteristics:

1. Introduction to Privacy Policies

. Privacy policies are legal documents

that disclose how companies gather, use, share, and manage user data…

2. Privacy Policy Misuse and Lack of Awareness

. Lack of awareness …

3. PrivacyQA Dataset

. Motivated by the need for quick identification …

4. Data Collection Methodology

. PrivacyQA comprises 35 mobile

application privacy policies collected from the Google Play Store…

Examples

After transformation, the linguistic input retains its structured knowledge through systematic cues but is presented in a comprehensible manner for LLMs. This not only facilitates an enhanced understanding and interaction with complex data but also enables the models to leverage their existing natural language capabilities to generate more accurate and contextually relevant responses.

4 Experiments

In this section, we conduct extensive experiments on a series of downstream NLP tasks to comprehensively demonstrate the efficacy of structurization. We hope the results can bring new insights to enhancing LLM's cognition via structurization, regardless of model architecture and size variation.

Three representative natural language understanding and processing tasks are investigated, including context-based question-answering Sec. 4.1, exhaustive hallucination evaluation Sec. 4.2, and passage-level dense retrieval Sec. 4.3. When instructing tested LLMs to perform target tasks, we merely structurize the vanilla textual inputs, transform the results back into natural languages, and immediately feed the results to LLMs to make responses. The tested LLMs themselves are not fine-tuned on structurized data corpus. We adopt our StruXGPT-7B model to execute structurization in all experiments in this section. The evaluation and ablation of StruXGPT model are demonstrated in Sec. 4.4, Sec. 4.5, and Sec. 4.6, and more comparison with related augmentation-based methods can be found in Appendix B.3 and Appendix B.5.

4.1 Application on Context-based Question-Answering

Question-answering based on a long-form context is an emerging research area within QA, which requires large language models to precisely seek the target information and generate reliable responses to the question [3, 30]. It is an immediate measure of LLM's cognition ability to handle intricate and sophisticated contexts. In this section, we comprehensively evaluate how structurization boosts QA ability on seven datasets from the LongBench benchmark [3] with a variety of LLMs to examine.

Dataset setup. LongBench [3] is a multi-task benchmark tailored for long context understanding evaluation, composed of 6 major task categories and 21 different tasks. To focus on the investigation of context structurization, we choose 7 subsets from LongBench across single-document QA, multidocument QA, and synthetic QA tasks in English, and the remaining Chinese subsets or codeorientated tasks are eliminated. Except for the MultiFieldQA subset with 150 testing samples, each subset contains 200 pieces of context-question-answer triplets to evaluate, resulting in 1,350 samples to test in total. Each subset has a 4K-18K context length on average. If the context length exceeds an LLM's window size, we truncate from the middle of the text and preserve information at the beginning and end, as suggested by LongBench. Detailed dataset description is displayed in Appendix B.2.

Evaluated models. Following [3], we evaluate three representative large language models on long context comprehension ability: LLaMA2-7B-4k [53], Qwen-7B-8k [2], and ChatGLM3-6B32k [67], and extend the larger LLaMA2-13B-4k [53] model. The four LLMs have a relatively similar parameter capacity with different model architectures and window sizes. All the models to examine are pre-trained chat models, and we just employ our StruXGPT to structurize the input contexts to enhance those LLMs' cognition capabilities. To ensure reproducibility and reduce uncertainty, greedy search is adopted during LLM's decoding process when generating responses. The accuracy between models' responses and ground-truth answers is measured by ROUGE-L and F1-score.

Experimental results. Tab. 1 suggests structurized contexts bring consistent improvements on almost all 3 tasks and 7 subsets across the model architectures and window sizes. Specifically, structurization leads to relatively greater improvement for the MultiDoc-QA subtask (with a 3% performance gain on average), revealing the potential promotion of LLMs' multi-hop reasoning abilities. Despite the negligible decline on the Musique [54] subset (see Appendix B.2 for analysis), the advanced ChatGLM3-6B-32k is also boosted, showing structurization's efficacy for powerful models.

Comparison with other baselines. To further evaluate our structurization augmentation, we compare our method against the typical summarization-based methods that also employ LLMs for context augmentation. The results are presented in Appendix B.3, which further demonstrates our superiority in highlighting the knowledge structure without loss of key information.

Investigating structurization from the attention perspective. Fig. 5 reveals how structurization can aid LLM's cognition from the attention perspective. In particular, we compare the attention maps for the same tested LLaMA2-7B model with different contexts as input. At the position of the model's first token prediction, we average the attention maps across the 32 attention heads for each layer of LLaMA's last 16 layers [72], and visualize the attention scores in Fig. 5. Specifically, when handling vanilla contexts, LLaMA2-7B loses its focus on the target information of the experts . On the contrary, the structurized context clearly presents the content structure of the introduced PrivacyQA dataset , and LLaMA2-7B immediately grasps the target aspect and its detailed descriptions of experts

A

: Domain experts with legal training.

Q

: Who were the experts used for annotation?

A

: Unanswerable.

Structurized Context

This passage talks about the PrivacyQA

dataset and its characteristics:

1. Introduction to Privacy Policies. …

2. …

3. PrivacyQA Dataset

.…

4. Data Collection Methodology.

PrivacyQA comprises 35 mobile application

privacy policies collected from the Google

Play Store. Seven experts with legal training

provide answers to Turker questions, which

are then meta-annotated for relevance,

subjectivity, and legal soundness…

5. …

Privacy policies are the documents which

disclose the ways in which a company…

…

…

…

… To identify legally sound answers, we

recruit seven experts with legal training to

construct answers to Turker questions.

Experts identify relevant evidence within

the privacy policy, as well as provide meta-

annotation on the question's relevance,

subjectivity, OPP-115 category BIBREF49,

and how likely any privacy policy is…

with legal training . In this way, LLM's cognition capability is successfully enhanced via context structurization.

4.2 Application on Exhaustive Hallucination Evaluation

Hallucination has raised wide attention in the community [70, 56]. In general, evaluating hallucinations involves verifying atomic claims against supportive materials ( e.g. , Wikipedia passages [66, 42]), yet even advanced GPT-3.5-Turbo and GPT-4 cannot always accurately make the judge, as LLMsevaluators often struggle to extract relevant information due to the complexity of passage contexts. We introduce how to improve LLM evaluators' assessing ability by context structurization below.

Dataset setup. AttrScore [66] and FactScore [42] datasets are adopted for evaluation. We take the AttrEvalGenSearch test set with 245 examples from AttrScore, where each example comprises a statement and its reference passage, and is annotated by Attributable (abbreviated as Attr. ), Contradictory (abbreviated as Contra. ), and Extrapolatory (abbreviated as Extra. ). FactScore collected 4,726 atomic claims/statements of people biographies generated by InstructGPT (abbreviated as InstGPT), 5,426 by ChatGPT, and 5,888 by PerplexityAI (abbreviated as PPLAI). For each input sample, we leverage StruXGPT to structurize the reference to identify its main aspects and detailed descriptions. The numerically ordered structure is preserved, as displayed in Fig. A3,

since we explicitly ask the evaluator to check the information along the structure for judgment.

Evaluated models. We mainly investigate the open-sourced LLaMA2-7B and LLaMA2-70B models as the LLM-evaluator, and also explore the integration with the close-source GPT-3.5-Turbo-1106 [6] via API access. The main results are presented in Tab. 2. We also report the results with Qwen models on AttrScore and FactScore and the incorporation to more powerful GPT-4 in Appendix B.4.

Experimental results. According to Tab. 2, our structurization brings significant enhancements to both LLaMA2-7B and 70B models (for 6.4% and 4.3% on average, respectively). And the powerful GPT-3.5 model also gains 4.1% (from 58.0% to 62.1%). Furthermore, we incorporate the Chain-ofThought (CoT) technique into the prompt template to clarify the evaluation steps (such as Carefully read the claim and double-check the answer. ) (denoted as 'GPT-3.5-1106 + CoT'). After that, the GPT-3.5 model immediately obtains an improvement of 4.0% (from 58.0% to 62.0%). Consequently, on top of the advanced CoT prompt, our method further enhances the model to a higher accuracy of 65.4% on average, demonstrating our method's compatibility with advanced prompting techniques.

4.3 Application on Passage-level Dense Retrieval

Retrieval-augmented generation (RAG) has been empirically validated to significantly bolster LLM's domain knowledge [29, 62], where precise document retrieval plays a vital role. We now investigate how structurization can facilitate dense retrieval for BERT-like masking language models.

Dataset setup. BEIR dataset [52] is a popular benchmark for evaluating dense retrievers' zeroshot effectiveness [39, 33], where retrievers are trained on MS MARCO's passage-retrieval training split [44] while directly tested on the BEIR benchmark without finetuning. We focus on our evaluation of the 5 subsets from BEIR i.e. , NFCorpus, FiQA, ArguAna, SciDocs, and SciFact.

Evaluated models. BERT [13], SimLM [57], and coCondenser [16] are chosen for evaluation, since they achieve state-of-the-art performance on MS MARCO's development split. To convert the structurized passages from our StruXGPT into natural languages, we eliminate the numerical indicators (such as '1.', '1.1', etc. ) and attach the description statements in the third layer with their aspects. Fig. A3 presents an example. We only structurize passages to enhance retrievers' cognition, and the queries remain unchanged. Following the literature, nDCG@10 results are reported in Tab. 3.

Experimental results. Structurization boosts all three retrievers on most subsets, yielding a maximum performance improvement of 4.5% on SciFact for SimLM. The results suggest that structurization not only augments decoder-only generative LLMs with explosive parameters (at least 7B), but also benefits encoder-decoder masked language models with constrained parameters (around 110M). It implies that the patterning of linguistic and semantic structurization may be a fundamental mechanism for enhancing language models, transcending distinctions in their architectural design and scale.

4.4 Evaluation of the Structurization Approach Itself

In this section, we assess various structurization methods through exhaustive experiments on five approaches, including prompting Qwen-max with few-shot exemplars (serves as our teacher model), few-shot Qwen-7B and LLaMA2-7B pre-trained chat models, and our fine-tuned Qwen-7B and LLaMA2-7B (student) models. As introduced in Sec. 3.1, we use 200 validation cases and have the five models generate 1,000 structurized outputs for analysis.

A good structurization should effectively deconstruct the vanilla input text to clearly identify its knowledge structure, so as to facilitate LLM's cognition. The resulting content should be faithful to the original texts, neither dismiss the factual information nor fabricate statements or opinions that do not exist. To this end, we revise four evaluation metrics to investigate the efficacy of different structurization approaches, and the results are displayed in Tab. 4.

Lexical evaluation (LexicalEval). We first leverage the widely-used ROUGE-L [34, 35, 25] to assess recall and precision between structured content and original text. However, lexical metrics from the methods, ranging from 0.6 to 0.7, inadequately reflect structurization quality where LLMs will paraphrase the words but lexical scores miss the semantic consistency. For instance, a statement pair 'They adopt ROUGE' and 'ROUGE is adopted' only receives a 0.33 f1-score for ROUGE-L. Therefore, a crucial human evaluation is developed to obtain a trustworthy conclusion.

Human evaluation (HumanEval). We recruited 17 well-trained natural language annotators from the PAI-iTAG platform 5 to evaluate the structurization quality on a 0-5 scale across three dimensions: completeness (ensuring no loss of information from the original text), factuality (accurate three-layer deconstruction), and anti-hallucination (avoiding fabricated content). As annotating structurization

only involves linguistic and syntactic level judgments, annotators do not need professional expertise to check the information of a given text itself. The detailed evaluation criteria are displayed in Appendix A.3, and we report the labeling results in Tab. 4.

The commercial Qwen-max shows a promising instruction-following and in-context learning ability, generally scoring 4.5 at the three dimensions. However, the pre-trained 7B models from Qwen and LLaMA2 immediately decline the scores to below 4.0, as they struggle to understand the instruction to build the three-layer structure and tend to hallucinate responses due to the limited model capacity. More structurization examples can be found in Appendix E.2. Notably, the fine-tuned StruXGPT7B-Q(wen) and StruXGPT-7B-L(LaMA2) both obtain a 4.35 - 4.45 score on average. They inherit 97% of structurization capability from the Qwen-max teacher model, evidencing the effectiveness of training a specialized 7B-parameter model for structurization to aid in efficiency and privacy.

Evaluation with downstream application (AppEval). Since our main motivation is to utilize structurization to enhance LLM's cognition, a further evaluation is conducted to investigate those different structurization methods. Specifically, we compare how much improvement ( ∆ ) those methods can bring to downstream natural language processing applications. On the Qasper subset [12] from LongBench [3], we instruct an independent LLaMA2-7B chat model for reading comprehension with long-context structurized by different approaches. LLaMA2-7B receives a 19.6 F1-score for QA accuracy when taking the vanilla context as input, which serves as the baseline performance.

The evaluation results are displayed in Tab. 4, which are consistent with the human evaluation presented above. In particular, the few-shot Qwen-max achieves an over 3% improvement in answer quality, while the pre-trained 7B-parameter chat models fail to generate validated structurizations. Meanwhile, our fine-tuned StruXGPT-7B-LLaMA and StruXGPT-7B-Qwen models bring comparable enhancements against the few-shot Qwen-max, emphasizing the efficacy of distilling the structurization ability from giant teacher models to a more responsive and affordable student model.

Evaluation with semantic embedding (SemEval). At last, we explore the structurization quality evaluation in the semantic embedding perspective, as a supplementary to lexical evaluation. Following Zhang et al. [68], we calculate the semantic similarity between original and structurized contents with the embedding similarities, and the results in Tab. 4 show consistent measure against HumanEval and AppEval with a much lower cost. Hence, BERTScore [68] can be further leveraged as an effective and efficient quality-assessment tool for training-data filtering and structurization quality evaluation to derive a better StruXGPT model. Sec. 4.6 presents some preliminary investigations.

Through the comprehensive evaluation of three protocols, we demonstrate the feasibility and efficacy of training a specialized StruXGPT. It is more resource-friendly to deploy for efficiency and privacy, meanwhile inheriting 97% of the ability from giant teacher models to perform structurization.

4.5 Ablation Studies on StruXGPT's Establishment

This section studies two major factors of training a StruXGPT model: data quality and model capacity.

Using two few-shot examples is sufficient to collect high-quality training data. In this work, we choose 2 in-context examples to prompt commercial LLMs (as a teacher) to generate data pairs of raw/structurized texts to train our StruXGPT-7B model (as a student). We think it is enough for teacher models to understand the structurization process and generate valid training samples, as the 2 examples respectively describe the 2 most common types of real-world text ( i.e. , with/without existing indicators like '1.', '2.', etc), which is displayed in Appendix E.2.

To further verify it, we investigate the number of in-context examples with two evaluation protocols (as in Tab. 4): AppEval (an improvement on Qasper subset with context structurization) and BERTScore (semantic similarity with raw and structurized texts in the validation set). We also report the error rate when parsing structurization results from the teacher model's outputs (denoted as 'FormatError').

According to Tab. 5, 1-shot is apparently insufficient to illustrate structurization, while 2- and 3-shot achieve comparable structurization quality under AppEval and BERTScore. Notably, 3-shot receives a 2% lower FormatError than 2-shot, in trade for the increased inference cost (because of increased few-shot samples). We argue that the 2% gap (around 400 samples) does not make a difference for the final StruXGPT training, which can be verified in Appendix A.2. Therefore, we recommend users to apply 3- or even more shots when prompting teacher LLMs if available, otherwise 2-shot is also a good choice to balance the inference cost and structurization quality.

StruXGPT-7B balances parameter capacity and structurization quality. As Qwen [2] provides a series of models varying sizes, in Tab. 6, we implemented StruXGPT on Qwen-1.8B/7B/14B respectively to investigate the relationship between model capacity and structurization quality.

Compared with the 7B model capacity, the smaller 1.8B model, despite its positive enhancement on downstream applications, shows slight inferiority in both AppEval (+2.7 v.s. +3.6) and BERTScore (0.299 v.s. 0.313), and presents 5% error rate when parsing structurization results. On the other hand, the 14B model brings further improvement to BERTScore, (the structurization content is relatively more faithful to original texts), but the boost on AppEval is insignificant. Hence, the 7B model is a good trade-off between model capacity (training/inference efficiency) and structurization quality.

4.6 Utilization of Structurization Quality Assessment

As the structurization quality simultaneously influences StruXGPT's training performance and application improvements, this section investigates the quality assessment tool BERTScore (as discussed in Sec. 4.2) on StruXGPT's training and inference stages, respectively.

On one hand, as the training data quality determines StruXGPT's upper bound, we statistic the BERTScore of the 22K training entries, and around 94.45% raw/structurized text pairs present positive scores (normalized by the baseline score of 0.83, and a positive BERTScore presents a benign similarity), demonstrating the high quality of our training data. Consequently, we eliminated around 5% of data with negative scores and trained another StruXGPT model, and the results in Tab. 7 indicate this part of data does not affect the final performance. According to Appendix A.2, data quantity may play a vital role in further improving our StruXGPT.

On the other hand, since poor structurization results can lead to suboptimal application performance, we statistic the enhancement variance in the Qasper subset from LongBench [3] in Tab. 8 for structurized context inputs. Our method merely causes degradation on 3.5% of samples (with relatively lower structurization quality), but ultimately receives a +3.6 improvement overall test samples. Furthermore, we filter out the structurization results with a low BERTScore ( e.g. , < 0.05) and take back the original context as input. In this way, the degradation can be alleviated (from 3.5% to 3.0%), further improving the final enhancement to +3.7. The lower bound of our StruXGPT can thus be ensured.

5 Conclusion and Discussion

This paper presents a novel concept of context structurization to enhance LLM's cognition capability. Extensive evaluations of various representative NLP tasks reveal the consistent enhancement across language model's architectural designs and capacity scales. We demonstrate the feasibility of distilling the structurization ability from giant commercial LLMs into a responsive and private StruXGPT7B model, addressing the practicality problem. The limitation and future work are discussed in Appendix C. We hope this paper can bring new insights into how to build a more powerful and reliable language model with human cognition and intelligence.

Acknowledgments and Disclosure of Funding

This work was supported in part by the Fundamental Research Funds for the Central Universities, in part by Alibaba Cloud through the Research Intern Program, and in part by Zhejiang Provincial Natural Science Foundation of China under Grant No. LDT23F01013F01.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 , 2023.

Asma Ben Abacha and Dina Demner-Fushman. A question-entailment approach to question answering. BMC bioinformatics , 20(1):1-23, 2019.

S.M. Breedlove, N.V. Watson, and M.R. Rosenzweig. Biological Psychology: An Introduction to Behavioral, Cognitive, and Clinical Neuroscience . Sinauer Associates, Incorporated Publishers, 2010. ISBN 9780878933242. URL https://books.google.com/books?id=VYRFAQAAIAAJ .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading comprehension. In International Conference on Learning Representations , 2023.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6, 2023.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1-113, 2023.

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. arXiv preprint arXiv:1804.05685 , 2018.

Malcolm Coulthard and CN Condlin. An introduction to discourse analysis . Routledge, 2014.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011 , 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixtureof-experts. In International Conference on Machine Learning , pages 5547-5569. PMLR, 2022.

John Duncan. The structure of cognition: Attentional episodes in mind and brain. Neuron , 80(1):3550, 2013. ISSN 0896-6273. doi: https://doi.org/10.1016/j.neuron.2013.09.015. URL https://www. sciencedirect.com/science/article/pii/S0896627313008465 .

Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540 , 2021.

M.S. Gazzaniga, R.B. Ivry, and G.R. Mangun. Cognitive Neuroscience: The Biology of the Mind (5th edition) . W.W. Norton, 2018. ISBN 9780393603170. URL https://books.google.com/books?id= ZbtotwEACAAJ .

James Paul Gee. An introduction to discourse analysis: Theory and method . routledge, 2014.

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations , 2023.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644 , 2023.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060 , 2020.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645 , 2023.

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 , 2017.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. arXiv preprint arXiv:1805.01052 , 2018.

David R Krathwohl. A revision of bloom's taxonomy: An overview. Theory into practice , 41(4):212-218, 2002.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledgeintensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459-9474, 2020.

Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, E. Gonzalez Joseph, Stoica Ion, Xuezhe Ma, and Hao Zhang. How long can open-source llms truly promise on context length?, June 2023. URL https://lmsys.org/blog/2023-06-29-longchat .

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 13067-13075, 2023.

Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. Llara: Aligning large language models with sequential recommenders. arXiv preprint arXiv:2312.02445 , 2023.

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74-81, 2004.

Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 , 2023.

Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.13169 , 2023.

Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv e-prints , pages arXiv-2308, 2023.

Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. arXiv preprint arXiv:2310.08319 , 2023.

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www'18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018 , pages 1941-1942, 2018.

Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896 , 2023.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251 , 2023.

Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707 , 2023.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. choice , 2640:660, 2016.

Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, and Yu Wang. Skeleton-of-thought: Large language models can do parallel decoding. arXiv preprint arXiv:2307.15337 , 2023.

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 , 2023.

Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 , 2021.

Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam. WikiChat: Stopping the hallucination of large language model chatbots by few-shot grounding on Wikipedia. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 2387-2413, Singapore, December 2023. Association for Computational Linguistics. URL https: //aclanthology.org/2023.findings-emnlp.157 .

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 13003-13051, 2023.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html , 3(6):7, 2023.

P. Thagard. Mind: Introduction to Cognitive Science . Bradford Bks. MIT Press, 1996. ISBN 9780262201063. URL https://books.google.com/books?id=-SQJngEACAAJ .

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 , 2021.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics , 10:539-554, 2022.

Alan M Turing. Computing machinery and intelligence . Springer, 2009.

Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521 , 2023.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval. arXiv preprint arXiv:2207.02578 , 2022.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824-24837, 2022.

Yilin Wen, Zifeng Wang, and Jimeng Sun. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. arXiv preprint arXiv:2308.09729 , 2023.

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025 , 2023.

Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116 , 2024.

Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081 , 2023.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 , 2018.

Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311 , 2023.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 , 2022.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, 2020.

Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics , 12:39-57, 2024.

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 , 2023.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 , 2023.

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405 , 2023.

A Implementation Details of Structurization

A.1 Implementation Details

The data to train our StruXGPT is carefully curated from two main aspects. First, to ensure diversity, we randomly sample 15,000 passages from Wikipedia across various domains, whose token length ranges from 1K to 3K. Then, we also supply round 15,000 statement passages from the domainspecific model- and human-generated content, including the filtered CAMEL-AI [31], FiQA [40], and MedQuad [4] datasets. With Qwen-max API, we collect the structurized results and eliminate the failed cases (usually due to internet inaccessibility and unexpected output format), resulting in 22,547 pieces of training data in total to derive our StruXGPT, and 200 test samples for evaluation.

StruXGPT is built upon LLaMA2- or Qwen-7B-chat models, which had been aligned to humans with promising instruction-following capabilities with their prior knowledge. The 22,547 pieces of input-output pairs are utilized to distill the structurization ability to StruXGPT via supervised fine-tuning (SFT). Specifically, StruXGPT is trained with a constant learning rate of 5 × 10 -6 for LLaMA and 1 × 10 -5 for Qwen for 1 epoch. The batch size is 128, and other hyper-parameters follow the default settings from Touvron et al. [53] and Bai et al. [2]. The training is resource-friendly, which can be done on 8 NVIDIA V100 (16G) GPUs for 3.5 hours.

For all the inference experiments, we leverage 1-2 NVIDIA A100-80G GPUs for model deployment.

A.2 Additional Ablation Studies on Training Data

To further investigate the training sample size and its impact, we have conducted additional ablation studies on StruXGPT's training process, and report the question-answering capability enhancement for the same LLaMA2-7B-Chat model on LongBench's Qasper subset, as introduced in Sec. 4.4.

Our results in Tab. A1 reveal an interesting pattern: as the training sample increases, StruXGPT first causes a decline in the LLaMA2-7B model's QA performance, and then brings stable enhancement with larger training data (over 10K samples). We attribute this to the fact that we train StruXGPT for only one epoch in each experiment to prevent overfitting. Consequently, models trained with too few samples often exhibit subpar structurization quality, commonly resulting in information loss or hallucination, which adversely affects downstream performance.

Specifically, for the model trained with 2K samples, the generated outputs frequently contain incorrect formatting and struggle to parse the three-tier structure effectively. In these cases, we resort to using the original text as a fallback input for downstream LLMs, which accounts for the somewhat lower performance drop (-0.6%) compared to the model trained on 5K samples (-1.3%).

A.3 Human Evaluation

Criteria. Agood structurization should effectively deconstruct the vanilla input text to clearly identify its knowledge structure, and the structurized results should be faithful to the original texts, neither dismissing the factual information nor fabricating statements or opinions that do not exist. Therefore, we devise three dimensions to evaluate the structurization quality, and each of the dimensions can be scored from 0 to 5 (higher is better). The detailed criteria are displayed in Fig. A1, and the annotating screenshot is provided in Fig. A2.

Instruction. The full annotation instruction is displayed below:

Assess the quality of structured outputs produced by different sources, and evaluate whether the structured output is sufficiently 'good' by using a score ranging from 0 to 5. To clarify what constitutes a 'good' structured result, please consider and score the following aspects, with each aspect also scoring from 0 to 5. Scoring Criteria: Please see Fig. A1.

Payment. The payment for annotators is 2,000¥ in total, which is higher than the minimum wage in our country.

Table A1: Ablation on training samples.

←语料结构化标注-v1...

目预览中(1/5)

分组圆

横排竖排

卡片大小○

内容占比-0

内容大小一0

字体大小○

20条/页

source：复制原文本禁用html渲染

pred：复制原文本禁用html渲染

①预览中不支持修改，点击修改答案后可修改

甲烷（CH)在氧（O2)存在下燃烧的化学方程为：

主题概念：

甲烷燃烧的化学方程式及产物

·（单选）内容完整

Ch+20→c0+2hz0

声明的主要方面和相应的说明：

在这种燃烧反应中，生成的产物是二氧化碳(CO)和水(HO)。

1.甲烷燃烧的化学方程

·(单选）结构合理

1.1.反应涉及到甲烷（CH)和氧（O2)。

3

1.2.平衡式为：CH+2O→CO+2HO。

·（单选）没有编撰

2.燃烧过程的产物

2.1.形成的主要产物是二氧化碳(CO2)。

012

2.2.另一个产物是水(H2O)。

历史标注

修改答案

source：复制原文本禁用html渲染

pred：①复制原文本禁用html渲染

①预览中不支持修改，点击修改答案后可修改

甲烷（CH)在氧（O2)存在下燃烧的化学方程为：

主题概念：

甲烷(CH4)燃烧的化学方程

·（单选）内容完整

Cha+202→cOz+2h0

声明的主要方面和相应的说明：

0123

在这种燃烧反应中，生成的产物是二氧化碳（CO2)和水(HO)。

1.化学方程式

·（单选）结构合理

1.1方程式为：CH+2O2→CO2+2HO。

3

·（单选）没有编撰

3

历史标注

修改答案

<1>

source：复制原文本禁用html渲染

pred：复制原文本禁用html渲染

①预览中不支持修改，点击修改答案后可修改

B Details for Structurization Applications

B.1 Structurization Examples

Fig. A3 provides several examples of how to structurize the long-form context for downstream applications, including reading comprehension in Sec. 4.1, hallucination detection in Sec. 4.2, and passage retrieval in Sec. 4.3.

Structurized Context

This passage talks about the PrivacyQA

dataset and its characteristics:

1. Introduction to Privacy Policies. …

2. Privacy Policy Misuse and …

3. PrivacyQA Dataset

. …

4. Data Collection Methodology.

PrivacyQA

comprises 35 mobile application privacy

policies collected from the Google Play Store.

Seven experts with legal training provide

answers to Turker questions…

Q

: Who were the experts used for annotation?

Instruction

A

: Domain experts with legal training.

Response

The life and career of director George Cukor:

1. Early life and background

1.1 Cukor was born on the Lower East Side of

Manhattan, the younger child and only son of…

1.2 His parents selected his middle name in honor of

Spanish-American War hero George…

2. Career milestones

2.1 Cukor won the Academy Award for Best Director

for "My Fair Lady" (1964), which was …

2.2 He continued to work into the 1980s….

Structurized Context

Judge

: Cukor won the award for Best Director. True or

False?

Instruction

A

:

True

. George Cukor won the Academy Award for

Best Director for "My Fair Lady" (1964).

Response

DNA methylation in human PBMCs.

**Role of DNA methylation**:

DNA

methylation plays a significant role in human

health and disease processes.

**Technological advancements**:

Recent

technological advancements have significantly

improved the accuracy and.….

**Whole-genome bisulfite sequencing**:

Using the whole-genome bisulfite sequencing at

24.7-fold coverage (12.3-fold per strand)…

Structurized Context

Instruction

Query

: What is the role of DNA methylation

in human health and diseases?

A

:

Matched.

Response

Reading Comprehension

Hallucination Detection

Document Retrieval

B.2 Details for Context-based Question-Answering on LongBench

Detailed dataset description. To focus on the investigation of context structurization, we choose 7 subsets from LongBench across single-document QA, multi-document QA, and synthetic QA tasks in English, and the remaining Chinese subsets or code-orientated tasks are eliminated:

Single-Doc QA. For single-document QA, we take two subsets from LongBench: (1) Qasper [12], featured by question-answering over NLP technical papers and annotated by NLP practitioners; (2) MultiFieldQA , manually curated from multiple data sources and annotated by Ph.D. students. MultiFieldQA contains 150 Context-Question-Answer triplets to test, and Qasper and other adopted subsets include 200 pieces of test samples respectively.

Multi-Doc QA. Multi-document QA requires LLMs to extract and combine information from multiple documents to derive the answer, which is generally more challenging than single-doc QA. We take three multi-hop QA datasets: (1) HotpotQA [65], containing 2-hop questions written by native speakers given two related paragraphs; (2) 2WikiMultihopQA [22], involving up to 5-hop questions synthesized through manually designed templates on Wikipedia passages; and (3) MuSiQue [54], carefully composed with up to 4-hop reasoning on an increased number of supporting and distracting context evidence.

Synthetic QA. We employ two extra synthetic tasks to test LLM's long-context handling ability on specific scenarios and patterns: (1) PassageCount , requiring models to count the number of unique passages from a shuffled passages pool; and (2) PassageRetrieval , randomly choosing one of 30 passages to obtain its summarization with GPT-3.5-Turbo, and asking tested LLMs to determine the original passage with which the summarization is crafted.

Performance analysis on MuSiQue. On the LongBench dataset [3], ChatGLM-3-32K [67] receives a slight decline (around 0.1%) on the MuSiQue subset [54] when taking structurized contexts as input. The main reason is that MuSiQue is too complicated. It requires up to 4-hop reasoning on manually paraphrased questions on a large amount of supporting and distracting context evidence. For ChatGLM-3-32K that has already achieved a considerable cognition ability, context structurization in the inference stage is temporally unable to bring significant advances in such a complex scenario. Next, we will delve into developing methodologies for training LLMs to capture the intrinsic structure of context, unlocking structurization's potential in enhancing LLMs on more complicated NLP tasks.

B.3 Comparison with Aspect- and Query-Based Summarization

Aspect-Based Summarization (ABS) and Query-Based Summarization (QBS) are classical context augmentation methods in the NLP literature, which aims to gradually extract important information from lengthy text data based on pre-defined aspect list or user-input queries. The summary that contains key information will be integrated into context inputs to augment tested LLMs to generate reliable responses, rather than lost in irrelevant sentences in the original lengthy inputs. We take ABS and QBS as the comparative baseline to further evaluate the efficacy of our proposed structurization augmentation strategy.

We investigate ABS and QBS on five document-QA subsets in LongBench [3]. Specifically, as the pre-defined aspect list is unavailable for each passage in LongBench, we turn the traditional ABS to input-agnostic paragraph-based summarization to lengthy paragraphs, where the prompt template is: Summarize the following text with no more than three sentences. Passage: {text}; Summary: . For QBS, the prompt template focuses on specific queries: Summarize the following text to answer the query with no more than three sentences. Query: {query}; Passage: {text}; Summary: . The {text} and {query} are placeholders, and the output summaries are respectively concatenated to form the augmented passages as inputs. We report the performance variance of the same test model ( i.e. , LLaMA2-7B-Chat [53]) when taking vanilla, summarized, and our structurized passages as inputs to answer the same given questions, as displayed below (due to the space limitation, we merely report the averaged enhancement on the five subsets).

Based on Tab. A2, we observed that ABS, without guidance from pre-defined aspects lists or userinput queries, failed to preserve critical information, and led to performance declines in four out of the five subsets. Conversely, QBS, with user query guidance, achieved improvements in the Qasper, HotpotQA, and Musique subsets. However, substantial information loss during the summarization process resulted in decreased performance on the MultiFieldQA subset. In contrast, our structured approach delivered consistent improvements across all subsets, demonstrating its efficacy.

In addition, ABS and QBS are applicable for passage-based question-answering tasks but do not extend well to other tasks such as hallucination assessment, where every piece of information counts and should not be summarized. Our approach is task-agnostic, highlighting the generalizability.

B.4 Additional Experiments for Hallucination Evaluation

Qwen models as the LLM evaluators. In Sec. 4.2 in the manuscript, we mainly study structurization with LLaMA2 and GPT-3.5-Turbo on the AttrScore benchmark. For a thorough comparison, this section provides the investigation of employing another series of models, i.e. , Qwen-7B and Qwen72B [2], on AttrScore and FactScore datasets to further demonstrate our method's efficacy. Results are presented in Tab. A3 and Tab. A4, respectively.

For the 7B- and 72B-parameter Qwen evaluators, structurization can immediately improve judgment accuracy on both AttrScore and FactScore datasets. The smaller Qwen-7B model receives a more significant enhancement, along with a nearly 10% improvement in all metrics (except the Attr. criteria in Tab. A3) on the two datasets. It indicates smaller LLMs may be more reliant on structurization to enhance their cognitive and information-seeking capabilities than large-scale LLMs. Simultaneously, the larger Qwen-72B evaluator also witnesses a considerable improvement with a 3%-4% increase. In particular, based on the structurized references, Qwen-72B achieves a competitive evaluation performance against GPT-3.5-Turbo on the FactScore benchmark, and even slightly outperforms GPT-4 on the Contra . criteria in AttrScore. Those results substantiate the effectiveness of leveraging structurization to further enhance powerful language models with considerable size capacities.

Incorporation with GPT-3.5/4 models and CoT techniques. In Sec. 4.2, we the popular chain-ofthought (CoT) technique for prompt augmentation on the advanced GPT-3.5-Turbo model on the AttrScore [66] benchmark, where Tab. A5 is a simply copy of the experimental results. Our method presents comparable performance against CoT, and achieves better enhancement after integrating with CoT, illustrating the compatibility and extensibility to more advanced strategies.

Consequently, Tab. A6 further presents the incorporation of our StruXGPT with the more powerful GPT-4 model and the CoT technique, showing consistent benefits for powerful GPT-3.5 and GPT-4 models, either with or without the CoT strategy. The experiments and discussions further validate the effectiveness of our StruXGPT approach.

B.5 Comparison with Other Augmentation Methods

Besides the summarization-based and prompting-based augmentation methods (in Appendix B.5 and Appendix B.4 respectively), here we also compare our structurization approach with AdaptLLM [7], which developed several domain-specific LLMs (in BioMedicine, Finance, and Low) via the proposed reading-comprehension training technique. The experimental results are presented in Tab. A7.

Accordingly, our method can also boost the Llama-7b baseline for 3%-7% without training , while AdaptLLM requires costly continual training of the baseline model on each domain corpus. In particular, for the general reading comprehension task, AdaptLLM-Fin does not introduce significant boosts, while AdaptLLM-Bio/Law even cause performance drops, which is mainly because AdaptLLM's domain-adaptation tuning will harm the general capability more or less. In contrast, our method does not alter the baseline model, but only structurizes the input context to enhance LLM's cognition

ability on downstream tasks, showing stable and consistent improvements ( e.g. , a 2.5% increase on BoolQ). Although our final performance is slightly inferior to the domain-specialized AdaptLLM, our generalizability emphasizes the contribution of our work, as we bring consistent enhancement across downstream domains and cause no degradation on general tasks.

C Limitation and Future Work

Training-time aggregation. This paper mainly investigates structurization during LLM's inference stage. Despite the universal enhancement across language models and NLP tasks, structurization has not yet fully tapped into its potential. Specifically, developing methodologies for training LLMs to capture the intrinsic structure of context and extrinsic correlations among training corpus presents an unresolved challenge in the field. We will delve into this problem in our future work.

Inference efficiency. We acknowledge that the introduced inference cost is a common limitation for the methods adopting LLMs for data augmentation, which depends on computing resources and the optimization techniques employed. Here we conduct extra experiments with the competitive augmentation methods ( i.e. , ABS and QBS in Appendix B.3) for an in-depth comparison on the LongBench [3] dataset for question-answering evaluation. Besides, we also investigate the AttrScore [66] dataset for hallucination evaluation to assess the impact of input length and output length, as well as the model size. The inference time, measured in seconds per sample, is calculated on an NVIDIA A100 GPU with vllm 6 acceleration (except for the LLaMA2-70B model, which demands at least two A100 GPUs for deployment).

LongBench contains input passages with an average length of 14K tokens (computed with the LLaMA2 tokenizer), which burdens the cost for all the data augmentation methods. According to Tab. A8, only the task-specific QBS can bring a positive enhancement of 0.2% on average, which is negligible compared with the 2.5% boost by our task-agnostic structurization approach. However, the LLM-based data augmentation methods generally introduce an extra 2.5-2.9 seconds cost. To alleviate this problem, we have recently noticed a new decoding acceleration technique named Selection-of-Thought (SoT) [45], which is also motivated by the aspect-based thinking and writing process of humans and is naturally compatible with our structurization process. With the 2.39 × speed up and negligible quality decrease (denoted as ' † Ours + SoT'), the extra cost of our method can be further reduced to an acceptable 1.2 seconds, which we believe makes it more practical.

AttrScore requires a relatively longer output length (with a detailed explanation for hallucination evaluation), and the vanilla time cost increases compared to LongBench. According to Tab. A9, our

structurization earns a 6.4% gain on the average hallucination evaluation accuracy (detailed values are shown in Tab. 2), in trade of the 1.5 × increased time cost for the vanilla LLaMA2-7B evaluator. As the base evaluator is scaled up to a larger LLaMA2-70B model, our method merely introduces 12% extra cost (2.1s) and still gains a considerable enhancement of 4.3%. Our method showcases more advantages by incorporating larger base models.

Performance on General Evaluation Benchmarks As our method focuses on context structurization to enhance LLM's cognition ability, for the tasks where no context is provided, our method may not bring significant enhancement and even get worse. The commonly used MMLU benchmark [21] is a typical scenario, where LLMs are asked to answer questions without context references but requiring their parametric knowledge (learned during large-scale pre-training), and context structurization does not help. If we insist on structurizing the question alone and feeding it into LLM's inputs, the model may be disturbed by the introduced information from StruXGPT and generate wrong answers. As shown in Tab. A10, our method causes a slight 0.1% decrease (measured by OpenCompass protocol 7 ) when taking LLaMA2-7B-Chat [53] as the baseline model.

On the other hand, we have also tested another common benchmark, BBH [49], which is designed to evaluate LLMs' reasoning capability when dealing with several logical sentences/statements. In this case, our method can adapt well to highlight the logical relations and boost LLM's reasoning abilities by 0.8%.

In conclusion, we suggest users apply structurization to long-form or logically complex contexts, while taking the original question as inputs when there is no context provided.

D Broader Impacts

We discuss the positive and negative societal impacts as follows:

Positive Societal Impacts . Through the innovative approach of context structurization, this work significantly enhances the comprehension abilities of Large Language Models (LLMs), leading to more effective and efficient applications in various sectors such as education, healthcare, and customer service. Avoiding the need for larger models, it not only curtails the environmental impact associated with training sophisticated AI systems but also democratizes access to cutting-edge AI technologies. This fosters a broader base for innovation and empowers smaller entities with the tools to contribute meaningfully to technological advancement. Moreover, our method, rooted in human cognitive processes, promises to enrich human-AI collaboration, paving the way for solving complex societal issues by leveraging AI's improved problem-solving capabilities.

Negative Societal Impacts . On the flip side, the advances made through structurization bear potential risks, including the augmentation of disinformation campaigns and the creation of more sophisticated deepfakes, which could undermine trust in digital content. The enhanced capabilities of LLMs might also be co-opted for intrusive surveillance, raising substantial privacy and ethical dilemmas. Furthermore, the inherent biases in training data could be deepened, potentially automating and perpetuating social inequalities. The concentration of advanced AI technologies in the hands of a few could limit competition and place immense power over information dissemination with those entities, while increasing reliance on AI for critical decision-making could inadvertently erode human analytical skills over time. To mitigate these concerns, it is essential to establish rigorous ethical standards and robust oversight mechanisms that govern the deployment of LLMs, ensuring transparency and accountability in their use. Furthermore, active research and the implementation of bias detection, correction frameworks, and the development of media literacy programs are crucial

steps to preemptively address the potential misuse, privacy violations, and societal impact of these advanced AI systems.

E Prompt Examples for Structurization

E.1 Prompt Templates

Fig. A4 displays the few-shot prompt template to query giant commercial LLMs to execute structurization. We first introduce the instruction for structurization (including the task definition and output formats), then provide two representative examples to further illustrate the process. The first example in Fig. A5 tells the model to focus on the existing numerical or enumeration indicators to assist in constructing the aspect-description structure. The second example in Fig. A6 teaches the model to automatically summarize the aspects and attach their corresponding descriptions from the raw text sequences. Finally, in Fig. A4 we ask commercial LLMs to structurize the input _ statement from user input as expected.

To ensure completed and faithful structurization on long-form input statements (usually with thousands of tokens), we provide two detailed exemplars in Fig. A5 and Fig. A6 for commercial LLMs' few-shot learning. However, these two exemplars take a considerable context length, which may even confuse the distilled StruXGPT-7B with a smaller size capacity. We thus remove the examples in StruXGPT-7B's prompt template, as StruXGPT-7B has learned how to perform structurization as desired and the functionality of those examples is negligible. Specifically, the example _ 1 and example _ 2 (with their introductions) are removed for both efficiency and efficacy.

E.2 Structurization Examples

In this section, we present several structurization examples from different models as discussed in Sec. 3 in the manuscript. Given an input statement about Facebook's stock , Fig. A7 shows that Qwen-max produces a factual and faithful structurization result with the help of few-shot exemplars. It clearly identifies the main aspects of the original statement, and attaches the corresponding descriptions into each aspect. On the contrary, Fig. A8 indicates the smaller pre-trained 7B-size Qwen and LLaMA2 models failed to follow the instructions to produce qualified structurizations. There exists a fabricated 'Note' content in Qwen-7B's response. Meanwhile, LLaMA2-7B is even unable to follow the desired format, and information like 'Facebook doesn't sell anything tangible' is missing. Fig. A9 demonstrate the effectiveness of our fine-tuned 7B-parameter StruXGPT model, which is able to structurize the input statement as expected, with comparable performance to the giant Qwen-max model with over 200B-size.

```

The water absorption curve of rice after soaking is an unimodal curve. The inflection point of the curve is the peak

period of water absorption. The relationship between rice water absorption and time is non-linear and can be

expressed by the following formula: a*t+b=c, where a, b, and c are constants. Under different humidity conditions,

the change in the water absorption rate of rice with time is basically similar, that is, the water absorption rate is

between 0 and the point d inflection, and the rate of change accelerates over time. After the inflection point, the

increase in water absorption gradually stabilizes. The changing rules of water absorption and moisture content of

rice are similar, but under different humidity conditions, the relationship between water absorption of rice and

moisture content is different. When the moisture content is low, the water absorption of rice increases as the

moisture content increases. When the moisture content is high, the increase in water absorption of rice gradually

stabilizes. There are three obvious steps for rice seeds to absorb water: First, at the beginning of water absorption,

the water content of the seeds gradually increases, and the water absorption rate slowly increases. Second, during

the peak water absorption period, the water absorption rate increases rapidly. Third, in the later stage of water

absorption, the water content of seeds slowly increases, and the water absorption rate increases. The water

absorption rate of rice seeds is closely related to temperature. In general, water absorption increases as

temperature increases. The relationship between water absorption and temperature can be expressed as: unsaturated

water absorption (%) = 14.289T-10.719 (where T is temperature)

```

# Output:

## Statement's scope:

```characteristics of soaking rice seeds to absorb sufficient water```

## Statement's main aspects and corresponding descriptions:

```

1. The water absorption curve

1.1 The water absorption curve of rice after soaking is an unimodal curve.

1.2 The inflection point is the peak period of water absorption.

1.3 The relationship between rice water absorption and time is non-linear, which can be expressed by a*t+b=c,

where a, b, and c are constants.

2. Rice water absorption as time changes

2.1 Under different humidity conditions, the change in the water absorption rate of rice with time is basically

similar.

2.2 The water absorption rate is between 0 and the point d inflection.

3.3 The rate of change accelerates over time

2.3 After the inflection point, the increase in water absorption gradually stabilizes.

3. Relationship between rice water absorption and moisture content

3.1 The changing rules of water absorption and moisture content of rice are similar

3.2 Under different humidity conditions, the relationship between water absorption of rice and moisture content

is different.

3.3 When the moisture content is low, the water absorption of rice increases as the moisture content increases.

3.4 When the moisture content is high, the increase in water absorption of rice gradually stabilizes.

4. Three stages for rice seeds to absorb water

4.1 At the beginning of water absorption, the water content of the seeds gradually increases, and the water

absorption rate slowly increases.

4.2 During the peak water absorption period, the water absorption rate increases rapidly.

4.3 In the later stage of water absorption, the water content of seeds slowly increases, and the water absorption

rate increases.

5. Relationship between water absorption rate and temperature

5.1 The water absorption rate of rice seeds is closely related to temperature.

5.2 Water absorption generally increases as temperature increases.

5.3 The relationship can be expressed as: unsaturated water absorption (%) = 14.289T-10.719 (where T is

temperature)

```

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We accurately made the claims to reflect the paper's contributions and scope in Sec. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitation of our work in Appendix C.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not involve theorems.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the experimental details in Sec. 4, Appendix A, and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to https://github.com/alibaba/struxgpt .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided the experimental details in Sec. 4, Appendix A, and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We show consistent improvements across three datasets with various models in Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We reported the compute resource requirements in Appendix A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed the broader impacts of our method in Appendix D.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: Please refer to https://github.com/alibaba/struxgpt .

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All our experimental data and models are open source and obtainable.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Please refer to https://github.com/alibaba/struxgpt .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: Please refer to Appendix A.3.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We only hire annotators to label the structurization quality, as introduced in Appendix A.3, which does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method to improve the cognitive capabilities of large language models (LLMs) by organizing input context into a hierarchical structure. The method is called context structurization, and it involves transforming unordered contextual sentences into hierarchically structured elements to mimick human cognitive processes. The paper shows the effectiveness of the approach through evaluations on various NLP tasks, such as context-based question-answering, hallucination evaluation, and passage-level dense retrieval, using different model architectures and sizes. The study shows performance improvements and introduces StruXGPT-7B, a distilled model, to perform structurization efficiently.\n\nSTRENGTHS:\n- The study covers a wide range of tasks, providing a comprehensive assessment of the method's effectiveness.\n- I like the idea of distilling the structurization capability into a smaller model (StruXGPT-7B).\n- Even though the approch lack a bit of novelty it seems to work.\n\nWEAKNESSES:\n- The performance improvements are heavily dependent on the quality of the structurization process, so a poor structurization could lead to suboptimal model performance or even confusion.\n- Even though there is a performances improvement the process of structurizing context might introduce additional computational overhead. \n- The approach lack a bit of novelty, it doesn't introduce fundamentally new concepts and resembles methodologies from previous studies.\n- The study relies on metrics such as ROUGE-L and human evaluation, that are useful but they might not fully capture the more complex improvements, such as a cognitive-like processing that structurization aims to achieve.\n\nQUESTIONS:\n- How does the proposed three-layer structure handle texts with more complex relationships or multiple topics?\n- What are the computational costs associated with context structurization?\n\nLIMITATIONS:\n- The study heavily relies on specific models like LLaMA2 and Qwen, so i wonder about the generalizability of the results to other LLMs.\n- The structurization may be less effective working with extremely long contexts or when the context cannot be easily divided into clear hierarchical segments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new technique for prompting Large Language Models (LLMs) called StruXGPT. The basic idea is to transform the original prompt into a more structured description of the request which contains three levels of information: Scope, Aspects, and Descriptions. While the **Scope** provides an outline summary of the request, the **Aspects** present an itemized and well-ordered list of topics that are associated with their respective **Descriptions** which provide the details. In the paper, the prompt transformation is achieved by either using another large model or by a smaller model obtained by knowledge distillation. The experiments show that this type of structure can help LLMs find the right information needed to answer the requests, improving performance in question-answering tasks and reducing model hallucinations.\n\nSTRENGTHS:\nThe proposed technique is useful and easy to implement. As hallucination is currently one of the main challenges faced by real-world systems that employ LLMs, any technique that reduces it should be considered.\n\nWEAKNESSES:\nIn the abstract and introduction of the paper, there are claims about how human cognition works that are poorly backed up. This is an area of debate and I believe this discussion could be avoided without any harm to the main point of the paper.\n\nThe proposal assumes that the prompt transformation process is an easier task for an LLM than directly addressing the request. Although this seems to be the case for question-answering problems considered, there is no discussion about cases in which this may not be adequate. I believe more experiments with other types of datasets would help us to better understand that.\n\nFinally, as there are other papers investigating similar approaches, the novelty of the approach is not a strong point.\n\nQUESTIONS:\nThe legend of Table 1 mentions that the prefix Struct- indicates the data fed into LLMs is structured by StruXGPT-7B. However, I don't see this prefix in the table.\n\nLIMITATIONS:\nYes, the limitations were well addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to improve the cognitive capabilities of large language models (LLMs) without inferring the model by structuring contextual information hierarchically. The authors propose transforming plain, sequential text into a structured format, enabling LLMs to process and understand complex information more effectively. This method is empirically tested across various NLP tasks, demonstrating consistent performance gains. Additionally, the paper introduces StruXGPT, a distilled 7B model that efficiently executes the structurization process, enhancing the results on context-based QA benchmark, Hallucination Evaluation and Passage-level dense retrieval.\n\nSTRENGTHS:\n1. While structurization of text is not a novel concept itself, the development of StruXGPT, a smaller and more efficient model for structurization, is a significant contribution and it demonstrates the practical applicability of the proposed method.\n2. Evaluation and experiments are strong parts of paper, authors provide many detailed experiments with proper comparisons. The demonstrate stable performance gains using proposed method.\n3. The paper thoroughly describes the structurization process, including the use of linguistic markers and transformation templates, ensuring clarity and reproducibility.\n4. Authors made the research deeper by analyzing attention mechanism on structured texts.\n\nWEAKNESSES:\n1. Lack of parameter-efficiency analysis: authors provide 7B model for structurization process, but it is not clear whether smaller or bigger models would significantly hurt or benefit the approach.\n2. While the proposed approach shows stable improvements, it is still a lot of additional parameters, which is a concern for scalability to large amount of texts.\n\nQUESTIONS:\n1. Did you experiment with smaller/bigger models for structurization? If yes, what were the results? \n2. Are there specific tasks in which such structurization could be worse? \n3. Did I understand correctly that the data for training StruXGPT was gather by quering bigger model? Did you analyze the training set in details and do you think the proposed approach would benefit from training data gather or annotated by humans? I am personally worried here about the hallucinations that could be inherited from the bigger model.\n4. Do you plan to publish your StruXGPT model? \n\nAlso in Table 1 you write \"The prefix Struct- indicates...\" but there is not such prefix in the table. Did you mean \"+StruXGPT (ours)\" here?\n\nLIMITATIONS:\n1. Paper investigates only inference stage \n2. Introducing StructXPGT is still additional computation cost, however authors provide the analysis of extra cost in appendix", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of context structurization to enhance the comprehension capabilities of large language models (LLMs) for long texts. The authors propose summarizing the input text into a three-layer structure of Scope-Aspect-Description using LLMs, and then inputting this three-layer structure as an enhanced version into the LLM. Additionally, the authors propose to optimize a 7B model by distilling the structuring capabilities of giant commercial LLMs to reduce computational costs. The effectiveness of the method is validated through experiments on multiple NLP tasks. The experiments also demonstrate that the fine-tuned 7B model can inherit most of the structuring capabilities of the giant commercial LLMs.\n\nSTRENGTHS:\nThe paper is well-written and well-structured, making it easy to understand.\n\nThe proposed method is technically sound.\n\nThe effectiveness of context structurization has been validated across multiple datasets in Context-based Question-Answering, Context-based Summarization, and Passage-level Dense Retrieval, showing improvements on a number datasets.\n\nWEAKNESSES:\nThe experiments are not sufficiently comprehensive. Firstly, many tables only compare the scenarios with and without StruXGPT (except for Table A2 and A5), without comparing against other advanced prompt engineering methods. Therefore, it is hard to determine whether StruXGPT enhances the cognitive abilities of LLMs more effectively than other methods. Secondly, the authors did not validate the effectiveness of StruXGPT on state-of-the-art LLMs such as GPT-4, so it is unclear whether GPT-4 would still perform well without context structurization.\n\nThere are also some related summarization-based augmentation methods that could be discussed or compared [1].\n\n[1] Cheng, Daixuan, Shaohan Huang, and Furu Wei. \"Adapting Large Language Models via Reading Comprehension.\" In The Twelfth International Conference on Learning Representations.\n\nQUESTIONS:\nHow does the number of in-context examples affect the results of structurization, and why was the number set to 2?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}