Enhancing LLM’s Cognition via Structurization
Authors: Kai Liu
Zhihang Fu
Chao Chen
Wei Zhang
Rongxin Jiang
Fan Zhou
Yaowu Chen
Yue Wu
Jieping Ye
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Structurization
Augmentation
Primary area: natural_language_processing

Abstract: When reading long-form text, human cognition is complex and structurized. While large language models (LLMs) process input contexts through a causal and sequential perspective, this approach can potentially limit their ability to handle intricate and complex inputs effectively. To enhance LLM’s cognition capability, this paper presents a novel concept of context structurization. Specifically, we transform the plain, unordered contextual sentences into well-ordered and hierarchically structurized elements. By doing so, LLMs can better grasp intricate and extended contexts through precise attention and information-seeking along the organized structures. Extensive evaluations are conducted across various model architectures and sizes (including a series of auto-regressive LLMs as well as BERT-like masking models) on a diverse set of NLP tasks (e.g., context-based question-answering, exhaustive hallucination evaluation, and passage-level dense retrieval). Empirical results show consistent and significant performance gains afforded by a single-round structurization. In particular, we boost the open-sourced LLaMA2-70B model to achieve comparable performance against GPT-3.5-Turbo as the halluci- nation evaluator. Besides, we show the feasibility of distilling advanced LLMs’ language processing abilities to a smaller yet effective StruXGPT-7B to execute structurization, addressing the practicality of our approach. Code is available at https://github.com/alibaba/struxgpt.

Enhancing LLM's Cognition via Structurization
Kai Liul Zhihang Fu Chao Chen', Wei Zhang', Rongxin Jiang' Fan Zhou', Yaowu Chen', Yue Wu?, Jieping Ye*
1 ZZhejiang University,
2 AAlibbaa Cloud
Abstract
When reading long-form text, human cognition complex and structurized. While large language models (LLMs) process input contexts through a causal and sequen- tial perspective, this approach can potentially limit their ability to handle intricate and complex inputs effectively. To enhance LLM's cognition capability, this paper presents a novel concept of context structurization. Specifically, we transform the plain, unordered contextual sentences into well-ordered and hierarchically structurized elements. By doing so, LLMs can better grasp intricate and extended contexts through precise attention and information-seekie structures. Extensive evaluations are conducted across various model architectures and sizes (including series of auto-regressive LLMs well as BERT-like masking models) on diverse set of NLP tasks (e.g., context-based question-ansween exhaustive hallucination evaluation, and passage-level dense retrieval), Empirical results show consistent and significant performance gains afforded by single- round structurization. In particular, we boost the open-sourced L.LMMA2-0BB model to achieve comparable performance against GPT-3.5- -errbo the halluci- nation evaluator. Besides, we show the feasibility of distilling advanced LLMs' language processing abilities to a smaller yet effective StruXGPT-7B to execute structurization, addressing the practicality our approach. Code is available https://gitte com/ aliaabaabb htt.aaabbbttbugggptttt
along the organized
1 Introduction
Large language models (LLMs) have emerged with remarkable language capabilities (6 53] Il. yet remain at a discernible distance from human-level intelligence, especially when handling long-form. sophisticated contexts as inputs S 36 Scaling up the model size has significant benefits for boosting context-comprehensioon and instruction-followii abilities for LLMs 47 58). However, is generally resource-intensive on both model training and inference. This paper presents another perspective on enhancing LLMs cognition capability without altering the models: context structurization.
The idea structurization motivated by neurocognitive science [5] [7
In human cognition, as indicated in Fig. [ sophisticated text sequences will be processed and consolidated into a structured knowledge tree, with factual elements well-organized hicrarchically 28 [5]. This process is defined structurization. People precisely search information from general concepts specific details and make connections and comparisons along structures. We thus aim to transform plain texts into structurized inputs, helping LLMs recognize and understand contexts in human manner (7]].
As Fig. T Tillustrates, input sequences are reorganized in simple but generic three-layer structure: scope, aspects, and descriptions. The scope summarizes the topic and contents, unfolding into several main aspects with corresponding detailed descriptions. The structurized results can be freely
Work done during Liu's research internship at Alibaba Cloud. Email: kai Kail@zju.edn.cn. Conresponding authors. Email: rongxinj zju.edu.cn, zhihang.fzh@ @ww.hhaabbbbbbbbccccommm
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

assembled into various natural language forms, depending on the specific downstream tasks. Fig. provides an exemplar of our overall framework: after structurizing and reassembling the vanilla context to highlight its scope and aspects of data imbalance handling techniques, LLMs become able to grasp the target information about the dynamic weighting strategy and generate reliable responses.
We first execute the structurization by prompt- ing advanced commercial LLMs (e.g., GPT-3.5- Turbd or Qwen-Max) with a few examples, and then collect the results to train smaller 7B 8-parmmettr LLaMA2 [53] or Qwen 2 model as our StruXGPT. This is motivated by [48], where the fundamental syntactic processing abil- ity from giant L.LMs is distilled into responsive and affordable StruX ...7B...mmeeeeni evaluations indicate that StruXGPT-7B inher- its 97% of the structurization ability from the teacher model, showing our method's feasibility.
Context
dlssif a olven sado sigad too pertictar using deantticcion alganthem Uamersely ther waay
Perceiver
Mestirratin woeen sofl by N. or aue the LX156 modd suetee, Roare afrence in doep loaning hare leleel seprove ber kenmour
usay cilivee actaal sesook Cit
cas vaind offiiend
Cognition
Scope: Language Identification
Empirical experiments are conducted on a di- verse set of NLP tasks (e.g., bostext-based question-answert answering, exhaustive ballucination evaluation, and passage-level dense retrieval). The results show that with single-turn struc- turization by our StruXGPT, the cognition per- formance of vanilla large language models wit- nesses consistent improvements regardless of the model architecture and size variation. In particular, we boost the open-sourced LLaMA2- 70B [53] to achieve comparable performance against GPT-3.5-Tutee as a hallucination evaluator. and demonstrate the compatibility with other advanced prompting techniques, such CoT [59]. We hope this paper can bring new insights to the community on building a more powerful and critical language model with human cognition.
Aspect: Malloed Overview
Aspect: Paslling Stratagies
Aspett Mudel Architecture
Dosc evettor a vidi-zanm apprach uaingg CIMM 1180M
Docc: NeVVLAD socting: replacce ausprrian....
Dac convoheacs layers with Ralii acas fincoons...
anpresed langaagg....
NEYLLA......
IDDoa Konauz
Figure 1: Structured cognition on sequential contexts. Humans may easily identify given pas- sage's s topic/scope, break down the text sentences into several aspect points with detailed descrip- tions, and form a tree-like knowledge structure.
Our contribution can be summarized as follows:
We propose the concept of structurization, in order to enhance LLM's cognition capability without altering the models themselves.
We present the feasibility of distilling the structurization ability from giant commercial L.MSs into responsive and affordable StruXGPT-7B model, making our approach practical.
With structurization, we empirically demonstrate the consistent cognition enhancement for various LLMs across model architecture and size variation diverse NLP tasks.
2 Related Work
Large language models (LLMs). LLMs' emergent abilities 588 has recently received extensive attention in the literature 61 61 9 53 23 D. which are found closely related to the scaling law [26).
When the scale reaches certain level, the performance
complex NLP tasks significantly rises due to the superior in-context learning, instruction following. and reasoning abilitics [7]] Numerous efforts have been made to boost the model capacity with training and prompting strategies [37 38
59
14), and this paper presents a new perspective, context structurization, to encourage LLMs to recognize, and communicate like humans [55 71, without altering the model themselves.
Context augmentation. Recent studies have proposed several context augmentation methods to enhance LLM's cognition ability [59 41 on when taking the long-form context (with thousands
of tokens) as inputs 3 36
(0). Specifically, aspect-based summarization (ABS) and query-based
summarization (QBS)
664 69 are designed to extract important information from lengthy text data as
well, but they require pre-defined or user-input aspect/query lists to conduct targeted summarization. and the detailed information will be inevitably lost during the summarization process. In contrast, the
httpp://openai. com/
https://dashscopa wel.saaliiil aliyun.caolooo com/

---

Context
LLM
Response
Dasa inbukinee a commen less is valker of NLP sala wth as tagikn and caciiis
Vanilla
dhergenog The mes extepe diperse (CB for shord mesiters lidc lichheel DILE
Ingend alss Seca low compreer poie
which wakky
provid
dynamti waiget cokprting
cach moning carrpic weghh propartinn 50-e/5 enoree which immppoo, absome isdadinned NLP ualgen
porfermane bosin wale
How are weigtt dysamically adjoand!
This passage udla chout the des infialance handling in NEP
Lado, whish invelesseeven severd tapoes:
intedance. cmeree loue NLPtala 2. Truinins ebjetivv.. CE MILE chidtoo acvildy akptell
Structurized
To addires domisating leflaenne of easy tege--ttgtteeeee enarglens eari training sampll audgned weight praporionall S13-pi5
StruXGPT
3. lieeriing nog--eggatie enappic A-yyamicvan wangeegggee adjating
amiagy is perpoctts moccan och meeing ccarple with 6. Comntiing straingto hrggriiastt boars poffrmanee...
Figure 2: Framework overview. When instructed to generate responses based vanilla long- long-form and sophisticated contexts, LLMs often lose their focus and give unreliable answers due to their limited cognition capability. In contrast. we structurize the vanilla context by using our StruXGPT to identify its main scope and aspect points, facilitating the original LLMs to comprehend the context and generate accurate responses.
paper develops context-wise structurization to highlight the knowledge structure, running single-turn structurization on the context to enhance LLMs' cognition abilities on a diverse set of NLP tasks.
Structurization. In the conventional NLP literature, the term structured data usually refers to triplets or properties extracted from plain texts, which are utilized to construct knowledge graphs databases with special data formats schemas [24 32) On the contrary, the structurization in this paper does not focus on entity-level information extraction and aggregation. Instead, it suggests reorganizing the input sentences into three-layer structure based their inner linguistic relations. Similar to discourse analysis and constituency parsing [0] (2),, the main purpose of structurization is capturing the dependencies and relations of the elements within specific long-form text inputs, so as to enhance LLMs' cogmition of the knowledge structure and relations.
Knowledge Distillation. In the era of large language model learning, distilling specific knowledge from giant LLMs' outputs has commonly been used to derive a more affordable but still powerful language model [63] [9]. Previous attempts, such as Standford Alpaca [50] and Vicuna [8], show the feasibility of collecting instruction-respons pairs from GPT-3.5 or GPT-4 to train a smaller fundamental [46] domain-specifhe (e.g., with reasoning
4 or coding 20 skills) language models. Our work also distills the structurization capability from the giant Qwen-Max model to a smaller yet effective StruXGPT-7B model, with extensive evaluations to show the efficiency and efficacy.
3 Structurization
The substantial purpose of structurization is to mimic the human cognition process and transform plain, sequential text sentences into well-organized, hierarchical knowledge structure. Inspired by the linguistic discourse analysis [11 [8]. this paper develops three-layer hierarchy exemplar to present the cognition/knowledge structure, as introduced in Fig.
(1) Scope summarizes the topic and boundary of the textual context. It outlines the central issues of knowledge throughout the text and the scope of the discussion that will be covered.
(2) Aspect further subdivides the input context into several parts. It presents the aspects or dimen- sions that must be considered to fully understand the topic and scope.
(3) Description the most specific and detailed layer. It provides in-depth descriptions and analyses to support each aspect of the context scope.
This generic three-layer structure is derived for efficacy and efficiency in dealing with diverse textual inputs. There might be some elaborated structures (such as a knowledge mindmap [60]) to better deconstruct specific text sources, but the difficulty and complexity of defining, extracting, and utilizing those structures aid in practical problems are dramatically increased. We leave this exploration in our future work.
Next, we will present how to implement effective structurization with minimal cost in Sec. 3.1 and demonstrate the utilization of the structurization results to enhance L.MSs cognition abilities in different downstream tasks in Sec. 3.2
3

---

3.1 Efficient Implementation of Structurization
We have explored two approaches to execute the structurization process by leveraging the extraor- dinary capability of large language models: few-shot prompting on commercial LLMs and direct instructing our developed StruXGPT models.
Few-shot commercial LLM. In the initial stage, we use two in-context examples to query the commercial Qwen-Max model to structurize the input corpus, since it shows promising instruction- following and textual analyzing capability with over 200B parameters [3] Here is a simplified example of prompting structurization in Fig. B and the full template is displayed in Appendix E.I
However, commercial LLMs are usually slow and expensive, and sending user data to LLM APIs may cause privacy and security problems due to information leakage. Thus, train smaller 7b-parameter model (e.g., LLaMA2 [53] or Qwen (2) to inherit the structurization ability from giant commercial L.Mss, which can be deployed locally for efficiency and privacy.
Given sequential statement, you supposed to identify:
#v Statement's 's scope: [geterally noun phrose]
... Statement" main aspects and corresponding descriptions:
[the first espect the statement]
1.1 [a descriptive sentence]
1.3 Lanother deseriptive sentence]
[the second aspect of the statemen:]
Fine-tuned StruXGPT. Our tailored model is named by StruXGPT, where Stru is the abbre- viation of structurization, and X implies we do not specify the model architecture. We carefully curate 22,547 raw data pieces from Wikipedia and CAMEL-AI dataset [3] to ensure diversity, and collect the structurized results from Qwen- Max to train our StruXGPT via supervised fine- tuning (SFT). From the collected samples, 200
2.1 la descriptive santence]
3. lonother aspect of the statesent]]
SEB Input: Impput siatement)
Outpus:
Figure 3: Prompt template for structurization.
are utilized for evaluation (including human verification), and the remaining training samples are adopted to distill the structurization capability from Qwen-Max to our StruXGPT-7B. It is practical since the structurization only relies on fundamental syntactic understanding and processing ability, which has already been learned from the large-scale corpus. We merely teach the 7B-parameter models how to reorganize the imput text via SFT, without introducing new memorizing or creative overloads. In addition, the ultimate StruXGPT does not require few-shot examples, and it reduces the input lengths for further efficiency. The training details are described in Appendix A. A.
3.2 Effective Utilization of Structurization
The identified knowledge structure (i.e., the scope, aspect, and description hierarchy) from the raw context is initially parsed in JSON format and unsuitable for direct imputs handle massive lengthy elements. Therefore, we use a unified template to transform the structured data back into natural language sentences models' inputs to fit their intrinsic processing patterns, LLMs are pre-trained and aligned with mostly natural language data. Concurrently, to preserve and highlight the know ledge structure, we harness specific linguistic markers signal hierarchy and relationships among concept elements, such as numbered lists for order, bullet points for categorization, and indentation to depict nesting levels of information. Fig. 4 showcases some typical examples.
The first row of Fig. 4 Arroviees a unified template to transform structurization results into natural languages. The top-level hierarchy scope is presented as standalone sentence, serving as the introduction to the structured context and highlighted with bold markers. Subsequently, the secondary aspect are organized with numerical markers and bolded, attaching with its corresponding tertiary descriptions through subclauses or separate sentences. This method not only signifies the rank and relation of each piece of information relative to others but also provides clear, navigable paths for the L.MSs to follow and process the information efficiently (e.g., when examining the long-form comprehension capability). Moreover, the second row of Fig. [4] introducss another variation for transformation, where each description elements are further broken down and enumerated for the delineation of fine-grained details, making it easier for language models discern and retain specific nuances associated with each aspect (e.g., when examining the hallucination detection capability).
hhttps://rank opencompass. org. www.eeeeerbaarddaeeeoooeeaa 11m-v2
We use the 20231020-en dump from http::::p:///tt //dumps. wikimedia.oog org/

---

passage (Srope)s
This passage talks about PrivacyQA datavet characteristie:
1. Introduction Privacy Policies Privacy policies are logal documents
1. "'Asspect 11 (Dew LJ), (Desc 1.2).
disclose companics grller, slane, menage datn...
2. *'[Asppct 2) (Dese (Dese 2.2).
2. Privacy Policy Misuse and Lack of Awareness. Lack of avereene
PrivacyOA Dataset, Metivated by need for ideniification
""(Aspect 3j*he...
4. Data Collection Mechodology PrivacyQA compeises mobile application privacy policies collected Bor the Google Play Store...
[Sosp]] can be deconstructed av:
career directer George
deconstructed
1. Early life and background
"''([spect 1)**
- Cuker was born the Lower East Side of Manitraan, the younger child and of...
(Desc 1.1).
purests sclected middle rome honor of Spuriid-Ammeriin heco Gearge...
[Desc 1.2).
2. "AAspect
2. Carrer milestones
[Dese 2.1).
wen Academy (1964), which was
Dest Director "My Lady"
He continsed to week into the 1980s....
Templates
Examples
Figure 4: Left: templates to transform structurization results into natural languages, with special linguistic markers to preserve and highlight the context examples with clear information structure for long-form reading comprehension (upper) and hallucination detection (lower) tasks.
extracted knowledge structure. Right: transformed
After transformation, the linguistic input retains its structured knowledge through systematic cues but is presented in comprehensible manner for LLMs. This not only facilitates an enhanced understanding and interaction with complex data but also enables the models to leverage their existing natural language capabilities to generate more accurate and contextually relevant responses.
Experiments
In this section, we conduct extensive experiments on a series of downstream NLP tasks to compre- hensively demonstrate the efficacy of structurization. We hope the results can bring new insights to enhancing LLM's cognition via structurization, regardless of model architecture and size variation.
Three representative natural language understanding and processing tasks are investigated, includ- ing context-basee question- question-answeringg answering Sec. 4.1 exhaustive hallucination evaluation Sec. 4.2, and
passage-level dense retrieval Sec. 4.3 When instructing tested LLMs to perform target tasks, merely structurize the vanilla textual inputs, transform the results back into natural languages, and immediately feed the results LLMs make responses. The tested LLMs themselves are not fine-tuned on structurized data corpus. We adopt our StruXGPPT-7 model to execute structurization in all experiments in this section. The evaluation and ablation of StruXGPT model are demonstrated in Sec. 4.4] Sec. 4.5 and Sec. 4.6, and more comparison with related augmentation-bal methods can be found in Appendix B.3 and Appendix B.5
4.1 Application on Context-based Question-Answering
Question-answeriig based on a long-form context is an emerging research area within QA, which requires large language models to precisely seek the target information and generate reliable responses to the question 30). is an immediate measure of LLM's cognition ability to handle intricate and sophisticated contexts. In this section, we comprehensively evaluate how structurization boosts QA ability on seven datasets from the LongBench benchmark [3] with a variety of LLMs to examine.
Dataset setup. LongBench (3] is a multi-task benchmark tailored for long context understanding evaluation, composed of 6 major task categories and 21 different tasks. To focus on the investigation of context structurization, we choose 7 subsets from LongBench across single-document QA, multi- document QA, and synthetic QA tasks in English, and the remaining Chinese subsets or code- orientated tasks are eliminated. Except for the MultiFieldQA subset with 150 testing samples, each subset contains 200 pieces of context-question-nssse wnwwwrr triplets to evaluate, resulting in 1,350 samples to test in total. Each subset has a 4K-18K context length on average. If the context length exceeds an I.L''s window size, we truncate from the middle the text and preserve information the beginning and end, suggested by LongBench. Detailed dataset description is displayed in Appendix B.2
5

---

Table Performance on LongBench datasets. The indicator StruXGPT means the data fed into LLMs is structurized by our StruXGPT-77, while the evaluated LLMs themselves are unchanged. The results are acquired by LongBench's official protocol. Higher is better.
SingleDoc QA
MultiDoc QA
Synthetic Tasks
Method
Average
Qasper
MFQA
HpQA
2Wiki
Musique
PsgCnt
PsgRet
LLaMA2-7B-4k
19.5
34.6
30.4
27.3
10.7
2.0
9.0
19.1
+StruXGPT (ours)
23.1
35.9
32.7
29.9
13.4
3.0
12.0
21.4
LLaMA2-13B-41
26.9
34.5
38.9
34.4
13.9
10.0
22.9
+StruXGPT (ours)
28.5
35.3
40.0
39.4
18.9
3.5
16.0
26.0
Qwen-7B-8k
19.6
34.1
20.4
12.5
7.5
2.0
15.5
15.9
+StruXGPT (ours)
22.3
37.0
25,4
14.7
8.2
2.5
17.5
18.2
ChaGGLM3-66B332
43.3
51.7
54.4
44.9
40.4
2.0
99.0
47.9
+StruXGPT (ours)
44.6
52.1
57.2
47.6
40.1
4.0
99.5
49.3
Evaluated models. Following (3. we evaluate three representative large language models on long context comprehension ability: LLaMA2-7B-4K (53]. Qwen-7B-8X 22. and ChaiGLM3-6B- 32k [67] and extend the larger LLaMA2-133 1---A-11B-4- [33 model. The four LLMs have relatively similar parameter capacity with different model architectures and window sizes. All the models examine are pre-trained chat models, and we just employ our StruXGPT to structurize the input contexts to enhance those L.Ms'' cognition capabilities. To ensure reproducibility and reduce uncertainty, greedy search is adopted during LLM's decoding process when generating responses. The accuracy between models' responses and ground-truth answers is measured by ROUGE-L and Fl-score.
Experimental results. Tab. I Suggests structurized contexts bring consistent improvements on almost all 3 tasks and 7 subsets across the model architectures and window sizes. Specifically, structurization leads to relatively greater improvement for the MultiDoc- - subtask (with a 3% performance gain on average), revealing the potential promotion of LLMs' multi-hop reasoning abilities. Despite the negligible decline on the Musique [54] subset (see Appendix B.2 for analysis), the advanced is also boosted, showing structurization' 's efficacy for powerful models.
ChatGL
Comparison with other baselines. To further evaluate our structurization augmentation, we compare our method against the typical summarization- based methods that also employ LLMs for context augmentation. The results are presented in Appendix B.3] which further demonstrates our superiority highlighting the knowledge structure without loss of key information.
Investigating structurization from the atten- tion perspective. F 5 how structuriza- tion can aid LLM's cognition from the attention perspective. In particular, we compare the at- tention maps for the same tested LLaMA2-7B model with different contexts input. At the position of the model's first token prediction, we average the attention maps across the 32 atten- tion heads for each layer of LLaMA's last 16 layers [72). and visualize the attention scores in Fig. 5 Specifically, when handling vanilla con- texts, LLaMA2-7B loses its focus on the target information of the experts. On the contrary, the structurized context clearly presents the content structure of the introduced PrivacyQA dataset, and LLaMA2-7B immediately grasps the target aspect and its detailed descriptions of experts with legal training. this way, L.M''s cognition capability successfully enhanced via context structurization.
Vimilla Context
Structurized Context
RNSRER policies ar the decereets which
paiage about PrivacyQA
dlucioo says in which company.
charadittitiss
Introduction Privaey Policko...
ideetify soud
C Collection Muthedslony.
with legil
PrivacyAA ourpries mobile application
policio collated from Geogle
Esperti deanity mileont werienee withn the penoey palicy w-tolll se proside (eats ausitaan on qucaied adbectiviy, by, 1889 115 eatepory BIBREFA, and how lioely any provcy palicy
Play Store Senmn expert with tanin poovide upoaas to Turker queations whsch an thw.eeteomeeeee for relevmos.. eabjocnutty and logal ouninee...
A. Usauoverable
Figure 5: Attention maps on vanilla and structur- ized contexts for the same LLaMA2-7B. The sam- ple comes from the QAsper subset.

---

4.2 Application on Exhaustive Hallucination Evaluation
Hallucination has raised wide attention in the community 770 56
general, evaluating hallucina-
tions involves verifying atomic claims against supportive materials (e.g.. Wikipedia passages [66] 42 yet even advanced GPT-3.5-Turbo and GPT-4 cannot always accurately make the judge, as LLMSs- evaluators often struggle to extract relevant information due to the complexity of passage contexts. We introduce how to improve LLM evaluators' assessing ability by context structurization below.
Dataset setup. AttrScore [66] and FactScore [42] datasets are adopted for evaluation. We take the AurEEal- GenSearch test set with 245 exam- ples from AttrScore, where each ex- ample comprises statement and its reference passage, and is annotated by Attributable (abbreviated as Attr), Contradictory (abbreviated Con- tra.), ). and Extrapolatory (abbreviated as Extra.). FactScore collected 4,726 atomic claims/statements of people bi- ographies generated by InstructGPT (abbreviated as InstGPT), 5,426 by ChatGPT, and 5,888 by PerplexityAl (abbreviated as PPLAÍ). For each in- put sample, leverage StruXGPT to structurize the reference to identify its main aspects and detailed descriptions. The numerically ordered structure is preserved, as displayed in Fig. [A since we explicitly ask the evaluator to check the information along the structure for judgment.
Table 2: Hallucination Evaluation on AttrScore.
Evaluator
Attr.
Contra.
Extra.
Average
GPT-4
87.3
45.0
89.6
74.0
GPT-3.5-Turoo
61.2
20.6
53.3
45.0
Alpaca-13B
50.6
6.1
19.3
25.3
Alpaca-7B
50.7
8.6
3.6
21.0
LLaMA2-7B
51.5
9.1
20.1
26.9
+StruXGPT (ours)
54.5
15.0
30.4
33.3
LLaMA2-70B
70.9
31.1
74.1
58.7
+StruXGPT (ours)
75.4
35.6
78.1
63.0
GPT-3.5-1106
72.0
30.4
71.7
58.0
+StruXGPT (ours)
77.1
31.8
77.4
62.1
GPT-3.5-1106 CoT
76.4
35.3
74.4
62.0
+StruXGPT (ours)
78.9
42.9
74.5
65.4
Evaluated models. We mainly investigate the open- sourced LLaMA2-7B and LLMMA2-70B models as the LLM-evaluator, and also explore the integration with the source GPT-3.5-Tume 1106 [6] via API access. The main results presented in Tab. We also report the results with Qwen models on AutSScore and FactScore and the incorporation to more powerful GPT-4 in Appendix B.4
Experimental results. According to Tab. 2 our structurization brings significant enhancements to both LLaMA2-7B and 70B models (for 6.4% and 4.3% on average, respectively). And the powerful incorporate the Chain-of-
5 model also gains 4.1% (from 58.0% to 62.1%... Furthermore, Thought (CoT) technique into the prompt template to clarify the evaluation steps (such as Carefully read the claim and double-check the answer) (denoted as "GPT-3.5- 100 CoT"). After that, the GPT-3.5 model immediately obtains an improvement of 4.0% (from 58.0% to 62.0%). Consequently, on top of the advanced CoT prompt, method further enhances the model to higher accuracy of 65.4% on average, demonstrating our method's compatibility with advanced prompting techniques.
4.3 Application on Passage-level Dense Retrieval
Retrieval- -errreval-ugmmettee augmented generation (RAG) has been empirically validated to significantly bolster LLM's domain knowledge 29 62 where precise document retrieval plays a vital role. We now investigate
how structurization
facilitate dense retrieval for BERT-like masking language models.
Dataset setup. BEIR dataset [52] is a popular benchmark for evaluating dense retrievers' zero- shot effectiveness 33 33 where retrievers trained MS MARCO's passage-retrieval training split 44] while directly tested on the BEIR benchmark without finetuning. We focus on our evaluation of the subsets from BEIR i..,, NFCorpus, FIQA, ArguAna, SciDocs, and SciFact.
Evaluated models. BERT 13], SimLM [57], and coCondenser [16] are chosen for evaluation, since they achieve state-of-the the- performance on MS MARCO's development split. To convert the structurized passages from our StruXGPT into natural languages, we eliminate the numerical indicators (such as "1.", "1.1", etc.) and attach the description statements in the third layer with their aspects. Fig. A3 presents an example. We only structurize passages to enhance retrievers' cognition, and the queries remain unchanged. Following the literature, nDCG@ 10 results are reported in Tab.
7

---

Table 3: Performance on BEIR subsets. Retrievers are trained with MS MARCO corpus and directly evaluated on BEIR without fine-tuning.
Retriever
NFCorpus
FIQA
ArguAna
SciDocs
SciFact
Average
BERT
24.4
23.7
36.2
11.4
50.8
29.3
+StruXGPT (ours)
24.4
24.9
40.0
11.4
50.7
30.3
SimLM
17.3
34.2
48.2
26.7
+StruXGPT (ours)
22.9
19.8
34.6
52.7
28.3
coCondenser
28.2
22.8
40.5
12.8
55.6
32.0
+StruXGPT (ours)
28.8
23.5
43.4
13.1
56.8
33.1
Experimental results. Structurization boosts all three retrievers on most subsets, yielding maximum performance improvement of 4.5% on SciFact for SimLM. The results suggest that structurization not only augments decoder-only generative LLMs with explosive parameters (at least 7B), but also benefits encoder-decoder masked language models with constrained parameters (around 110M). implies that the patterning of linguistic and semantic structurization may be fundamental mechanism for enhancing language models, transcending distinctions their architectural design and scale.
4.4 Evaluation of the Structurization Approach Itself
In this section, assess various structurization methods through exhaustive experiments five approaches, including prompting Qwen-max with few-shot exemplars (serves as our teacher model), few-shot Qwen-7B and LLaMA2-7B pre-trained chat models, and fine-tuned Qwen-7B and LLaMA2-7B (student) models. As introduced in Sec. 3.1 we use 200 validation cases and have the five models generate 1,000 structurized outputs for analysis.
A good structurization should effectively deconstruct the vanilla input text to clearly identify its facilitate LLM's cognition. The resulting content should be faithful to
knowledge structure, the original texts, neither dismiss the factual information nor fabricate statements or opinions that do not exist. To this end, revise four evaluation metrics investigate the efficacy of different structurization approaches, and the results are displayed Tab.
Table 4: Comprehensive comparison structurization approaches.
LexicalFval
HumanFval
AppFval
Approach
Model
recall?
precision?
completeness"
factuality
anti-halluf
Af
bertscoref
Qwen mux
0.63
0.68
4.58
4.49
4.57
+3.3
0.31
Few-shot
Qwen-7B
0.56
0.67
3.77
3.67
3.96
-0.1
0.22
LLaMA2-7B
0.61
0.72
4.09
3.98
4.12
+0.3
0.24
StuXGGFT-B--Q
0.63
0.67
4.41
4.36
4.48
+3.6
0.31
Fine-tuned
SmxXGPT-7B-L.
0.61
0.66
4.37
4.38
4.36
+2.8
0.30
Lexical evaluation vaa.). We first leverage the widely-used ROUGE-L 34
35 25
to
assess recall and precision between structured content and original text. However, lexical metrics from the methods, ranging from 0.6 to 0.7, inadequately reflect structurization quality where LLMs will paraphrase the words but lexical scores miss the semantic consistency. For instance, a statement pair "They adopt ROUGE" and "ROUGE is adopted" only receives 0.33 fl-score for ROUGE-L. Therefore, a crucial human evaluation is developed to obtain a trustworthy conclusion.
Human evaluation (HumanEval). We recruited 17 well- -rained natural language annotators from the PAI-ITAG platform to evaluate the structurization quality on a 0-5 scale across three dimensions: completeness (ensuring no loss of information from the original text), factuality (accurate three-layer deconstruction), and anti-hallucination (avoiding fabricated content). As annotating structurization
An open platform at hethhttps.. ://www http:////wwaaaaaaooooo/////////a

---

only involves linguistic and syntactic level judgments, annotators do not need professional expertise to check the information of a given text itself. The detailed evaluation criteria are displayed in Appendix
A.3
and we report the labeling results in Tab. 4
The commercial Qwen-max shows promising instruction-followiin and in-context learning ability, generally scoring 4.5 at the three dimensions. However, the pre-trained 7B models from Qwen and LLaMA2 immediately decline the scores to below 4.0, as they struggle to understand the instruction to build the three-layer structure and tend to hallucinate responses due to the limited model capacity, More structurization examples can be found in Appendix E.2 Notably, the fine-tuned StruXGPT. 7B- 7B-Q(wen) and
both obtain 4.35 4.45 score on average. They inherit 97% of structurization capability from the Qwen-man max teacher model, evidencing the effectiveness of training a specialized 7B- 1--parameter model for structurization to aid in efficiency and privacy.
Evaluation with downstream application (AppEval). Since our main motivation is to utilize structurization to enhance L.M's cognition, a further evaluation is conducted to investigate those different structurization methods. Specifically, we compare how much improvement (A) those methods can bring to downstream natural language processing applications. On the Qasper subset [22 from LongBench [R] we instruct an independent LLaMA2-7B chat model for reading comprehension with long- congecontext structurized by different approaches. LLaMA2-7B receives a 19.6 Fl-score for QA accuracy when taking the vanilla context as input, which serves as the baseline performance.
The evaluation results are displayed in Tab. 4 which are consistent with the human evaluation presented above. In particular, the few-shot Qwen-max achieves over 3% improvement in answer quality, while the pre-trained 7B-parameter chat models fail to generate validated structurizations. Meanwhile, our fine-tuned BruXGGPT-7B-LLaMME parable enhancements against the few-shot Qwen-max, emphasizing the efficacy of distilling the structurization ability from giant teacher models more responsive and affordable student model.
and StruXGPT-7B-Qwen models bring com-
Evaluation with semantic embedding (SemEval). At last, we explore the structurization quality evaluation in the semantic embedding perspective, a supplementary lexical evaluation. Follow- ing Zhang et al. [68], we calculate the semantic similarity between original and structurized contents with the embedding similarities, and the results in Tab, 4 show consistent measure against HumanEval and AppEval with a much lower cost. Hence, BERTScore [68] can be further leveraged as an effective and efficient muality-aseessmen tool for training-data filtering and structurization quality evaluation to derive a better StruXGPT model. Sec. 4.6 presents some preliminary investigations.
Through the comprehensive evaluation of three protocols, we demonstrate the feasibility and efficacy of training a specialized StruXGPT. It is more resource-friendly to deploy for efficiency and privacy, meanwhile inheriting 97% of the ability from giant teacher models to perform structurization.
4.5 Ablation Studies on StruXGPT's Establishment
This section studies two major factors of training a StruXGPT model: data quality and model capacity.
Using two few-shot examples is sufficient to collect high-quality training data. In this work, we choose in-context examples to prompt commercial LLMs (as a teacher) to generate data pairs of raw/structurized texts to train our StruXGPT-7B model (as student). We think it is enough for teacher models to understand the structurization process and generate valid training samples, as the 2 examples respectively describe the 2 most common types of real-world text (i.e., with/without existing indicators like "1.", "2.", etc), which is displayed in Appendix E.2
To further verify it, we investigate the number of in-context examples with two evaluation protocols (as in Tab. 4 AppEval (an improvement on Qasper subset with context structurization) and BERTScore Score (semantic similarity with raw and structurized texts in the validation set). We also report the error rate when parsing structurization results from the teacher model's outputs (denoted as "FormatError").
According to Tab. 1-shot is apparently insufficient to illustrate structurization, while 2- and 3-shot achieve comparable structurization quality under AvpEval and ScoT Notably, 3-shot receives a 2% lower FormatError than 2-shot, in trade for the increased inference cost (because of increased few-shot samples). We argue that the 2% gap (around 400 samples) does not make a difference for the final StruXGPT training, which can be verified in Appendix A.2 Therefore, we recommend users to apply 3- or even more shots when prompting teacher L.MMs available, otherwise 2-shot is also a good choice to balance the inference cost and structurization quality.

---

Table 5: Number of few-shot examples.
Table 6: Parameter capacity of StruXGPT.
nShot
AppEval
BERTScore
FormatError
StruXGPT
AppÉval
BERTScore
FormatError
1-shot
+1.8
0.282
25.4%
Qwen-
+2.7
0.299
5.0%
2-shot
+3,2
0.308
7.4%
Qwen-7B
+3.6
0,313
0.0%
3-shot
+3.3
0.302
5.5%
Qwen-14B
+3.8
0.323
0.0%
StruXGPT-7B balances parameter capacity and structurization quality. As Qwen provides a series of models varying sizes, in Tab. 66] we implemented StruXGPT on Qwen-1.8B/7BB 1.8B/7B/14B respectively to investigate the relationship between model capacity and structurization quality.
Compared with the 7B model capacity, the smaller 1.8B model, despite its positive enhancement on downstream applications, shows slight inferiority in both AppEval (+2.7 v.s. +3.6) and BERTScore (0.299 v.s. 0.313), and presents 5% error rate when parsing structurization results. On the other hand, the 14B model brings further improvement BERTScore, (the structurization content relatively more faithful to original texts), but the boost on AppEval is insignificant. Hence, the 7B model is good trade-off between model capacity (training/iffrrncce efficiency) and structurization quality.
4.6 Utilization of Structurization Quality Assessment
As the structurization quality simultancously influences StruXGPT's training performance and application improvements, this section investigates the quality assessment tool BERTScore (as discussed in Sec. 4.2) on StruXGPI'ss training and inference stages, respectively.
On one hand, as the training data quality determines StruXGPT's upper bound, we statistic the BERTScore of the 22K training entries, and around 94.45% raw/structurized text pairs present positive scores (normalized by the baseline score of 0.83, and positive BERTScore presents benign similarity), demonstrating the high quality of our training data. Consequently, we eliminated around 5% of data with negative scores and trained another StruXGPT model, and the results in Tab. indicate this part of data does not affect the final performance. According to Appendix A.2 data quantity may play vital role in further improving StruXGPT.
Table 7: Training data filtering.
Table 8: Inference results filtering.
Training Data
AppEval
BERTScore
Context
Declined Ratio
Overall Enhance
vanilla
+3.6
0.313
vanilla
3.5%
+3.6
filtered
+3.4
0.316
filtered 3.0%
+3.77
On the other hand, since poor structurization results lead to suboptimal application performance, we statistic the enhancement variance in the Qasper subset from LongBench [3] in Tab. B for structurized context inputs. Our method merely causes degradation relatively lower structurization quality), but ultimately receives +3.6 improvement overall test samples. Furthermore, we filter the structurization results with low BERTScore (e.g., <0.05) and take back the original context as input. In this way, the degradation can be alleviated (from 3.5% 3.0%), further improving the final enhancement to +3.7. The lower bound of our StruXGPT can thus be ensured.
3.5% of samples (with
5 Conclusion and Discussion
This paper presents novel concept of context structurization enhance LLM's cognition capability. Extensive evaluations of various representative NLP tasks reveal the consistent enhancement across language model's 's architectural designs and capacity scales. We demonstrate the feasibility of distilling the structurization ability from giant commercial LLMs into a responsive and private StruXGPT. 7B model. addressing the practicality problem. The limitation and future work are discussed in Appendix C
We hope this paper can bring new insights into how to build more powerful and reliable language model with human cognition and intelligence.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the Fundamental Research Funds for the Central Universities, in part by Alibaba Cloud through the Research Intern Program, and in part by Zhejiang Provincial Natural Science Foundation of China under Grant No. LDT23F01013F01.
References
[1] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv 2133:23033 2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint N..iv:2300.166000 2023.
[3] Yushi Bai, Xin Lv, Jiajic Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang. Zhengxiao Du, Xiao Liu, Aohan Zeng. Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
[4] Asma Ben Abacha and Dina Fushman. A question-entailnee approach to question answering. BMC bioinformatics, 20( 10)3331233 2019.
[5] S.M. Breedlove, N.V. Watson, and M.R. Rosenzweig. Biological Psychology: An Introdaction to Behav- ioral, Cognitive, and Clinical Neuroscience. Sinauer Associates, Incorporated Publishers, 2010. ISBN 9780878933242. URL hettps://books.goee google wowp//bookksid-VVVAAAAAAA.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subhiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading comprehen- sion. In International Conference on Learning Representations, 2023.
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Schastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113,, 2023.
[10] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. arXiv preprint arXiv:1804.00 104.180400000 05685, 2018.
[11] Malcolm Coulthard and CN Condlin. An introduction to discourse analysis. Routledge, 2014.
[12] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint 00X.0.2105.03011, 2021.
[13] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-traininn training of deep bidirec- tional transformers for language understanding, arXiv preprint 1010.081000 2018
[14] Nan Du, Yamping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture- of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
[15] John Duncan. The structure of cognition: Attentionaa episodes in mind and brain, Neuron, 80(1):35- 50, 2013. ISSN 0896-6273. doi: sciencedirect.com http://www.......caac.aaaaaaaaaa
w0t....rroo..1.0000..00. URL httpp:///ww/
[16] Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540 2021.
[17] M.S. Gazzaniga, R.B. Ivry, and G.R. Mangun. Cognitive Neuroscience: The Biology of the Mind (5th edition). W.W. Norton, 2018. ISBN 9780393603170.. http::////////ok.gggo wwt.ggl....mmeeeeooooeeeeeeeeeee com www/books ? 7id- ZototWEACAAJ

---

[18] James Paul Gee. An introduction to discourse unalysis: Theory and method. routledge, 2014.
[19] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2023.
[20] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 11644, 2023.
[21] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representationss 2021.
[22] Xanh Ho, Anh- Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:201.001600 2020.
[23] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplor, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0683 2023.
[24] Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. Structgnt: general framework for large language model to reason over structured data, arXiv preprint arXiv:2305.096453 2023.
[25] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint (eXXiv:705.035.. 2017.
[26] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.085 008360,000000,....... 2020.
[27] Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. arXiv preprint urXiv:1805.00. 2018.
[28] David R Krathwohl. A revision of bloom's taxonomy: An overview. Theory into practice, 41(4):212-218, 2002.
[29] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459 9474, 2020.
[30] Dacheng Li. Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, E. Gonzalez Joseph. Stoica Ion, Xuezhe Ma, and Hao Zhang. How long open-source lls truly promise context length?, June 2023. URL https://mmsys.. org/blog 202--06-2222222 20--299-llonee longchat
[31] Guohao Li. Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[32] Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13067-13075, 2023.
[33] Jiayi Liao, Sihang Li. Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. Llara: Aligning large language models with sequential recommenders. arXiv preprint arXiv:2312.02444 2023.
[34] Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oud, pages 74 81, 2004.
[35] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07955 2021.
[36] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 2023.
[37] Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph. Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint 231., 2023.
12

---

[38] Keming Lu, Hongyi Yuan. Zheng Yuan, Runji Lin. Junyang Lin, Chuanqi Tan. Chang Zhou, and Jingren Zhou. instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv e-prints, pages arXiv-2308, 2023.
[39] Xueguang Ma. Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. arXiv preprint urXiv:2310.08319, 2023.
[40] Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www'1 18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018 pages 1941-1942. 2018.
[41] Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint XrXix 08896, 2023.
[42] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14255 2023,
[43] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, arXiv preprint arXiv:2306.02700 2023.
[44] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary. Rangan Majumder. and Li Deng. Ms marco: A human generated machine reading comprehension dataset. choice, 2640:660,, 2016.
[45] Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang. and Yu Wang. Skeleton-of-thought: Large language models can parallel decoding. arXiv preprint (xXi:2337.15337, 2023.
[46] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03211 032777, 2023.
[47] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 1rX1v:2112.1144..200 2021.
[48] Sina Semmani, Violet Yao, Heidi Zhang, and Monica Lam. WikiChat: Stopping the hallucination of large language model chatbots by few-shot grounding on Wikipedia. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023. pages 2387-2413, Singapore, December 2023, Association for Computational Linguistics. URL https //aclanthology. wwaptthhoooy.orgg////oo www.org//00 2023. findings emnlp. 157]
[49] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoe Le, Ed Chi, Denny Zhou et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023,
pages 13003-13051, 2023.
[50] Rohan Taori. Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: strong, replicable instruction cnstruction-oollowingg model. Stanford Center for Research on Foundation Models. http://crfim stanford. edu/2023/033/1 3/alppaca// hmml, 366)7, 2023.
[51] P. Thagard. Mind: Introduction to Cognitive Science. Bradford Bks. MIT Press, 1996. ISBN 9780262201063. UtR./https://bol hww.b/oww...oogg..ooo.oo.oo.oo.. www.oooll..ooooooooooooooooo www..////wooddooooooooeoooeeeeee
[52] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heteroge- nous benchmark for sero evaluation of information retrieval models. arXiv preprint arXiv:2104.0666 2021.
[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[54] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-555 2022.
[55] Alan M Turing. Computing machinery and intelligence. Springer, 2009.
[56] Cunxiang Wang. Xiaoze Liu, Yuanhao Yue, Xiangru Tang. Tianhang Zhang, Cheng Jiayang. Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. Survey on factuality in large language models: Knowledge. retrieval and comman-sppccffcci arXiv preprint ar11vv2310007521, 2023.
13

---

[57] Liang Wang. Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval. arXiv preprint arXiv:2207.0257 2022.
[58] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud. Dani Yogatama. Maarten Bosma, Denny Zhou, Donald Metzler, Emergent abilities of large language models. arXiv preprint 2ii::220000062222.. 2022.
[59] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
[60] Yilin Wen, Zifeng Wang, and Jimeng Sun. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. arXiv preprint arXiv:2308.09729 2023.
[61] BigScience Workshop. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b- 766b-parameter open-access multilingual language model. arXiv preprint arXiv:22 1,05 2022.
[62] Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025, 2023.
[63] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou survey on knowledge distillation of large language models. arXiv preprint arXiv:2402. 13116, 2024.
[64] Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. Exploring the limits of chatgpt for query or aspect-hased text summarization. arXiv preprint 2rx00:2302.000811, 2023.
[65] Zhilin Yang. Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop hop question answering. arXiv preprint Xiv: 189. 09600, 2018.
[66] Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.063// 2023.
[67] Aohan Zeng. Xiao Liu, Zhengxiao Du. Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130h: An open bilingual pre-trained model. arXiv preprint 2rXiv:2210.02414, 2022.
[68] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, 2020.
[69] Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39-57, 2024.
[70] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.0122 2023.
[71] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[72] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazcika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprini arXiv: 2023.
14

---

A Implementation Details of Structurization
A.1 Implementation Details
The data to train our StruXGPT is carefully curated from two main aspects. First, to ensure diversity, we randomly sample 15,000 passages from Wikipedia actoss various domains, whose token length ranges from 1K to 3K. Then, we also supply round 15,000 statement passages from the domain- specific model- and human-generated content, including the filtered CAMEL-AI [3,, FiQA [40], and MedQuad [ datasets. With Qwen-max API, we collect the structurized results and eliminate the failed cases (usually due internet inaccessibility and unexpected output format), resulting in 22,547 pieces of training data in total to derive our StruXGPT, and 200 test samples for evaluation.
StruXGPT is built upon LLaMA2- or Qwen-7B-chat models, which had been aligned to humans with promising esstructionfoolloving capabilities with their prior knowledge. The 22,547 pieces of input-output pairs are utilized to distill the structurization ability to StruXGPT via supervised fine-tuning (SFT). Specifically. StruXGPT is trained with a constant learning rate of 5 x 10 for LLaMA and 1 10 for Qwen for 1 epoch. The batch size is 128, and other hyper-parameters follow the default settings from Touvron et al. [53] and Bai et al. [2]. The training is resource- frieurce-friendly, which can be done on 8 NVIDIA V100 (16G) GPUs for 3.5 hours.
For all the inference experiments, we leverage 1-2 NVIDIA A100-80G GPUs for model deployment.
A.2 Additional Ablation Studies on Training Data
To further investigate the training sample size and its impact, we have conducted additional ablation studies on StruXGPT's training process, and report the question-ann ansserrin capability enhancement for the same LLaMA2-7B-Chan model on LongBench's 's Qasper subset, as introduced in Sec. 4.4
Our results in Tab. A reveal an interesting pattern: as the training sample increases, StruXGPT first causes a decline in the LLaMA2-7B model's 'ssse QA performance, and
Table Al: Ablation on training samples.
Training Samples
Enhancement
then brings stable enhancement with larger training data (over 10K samples). We attribute this to the fact that we train StruXGPT for only one epoch each experiment to prevent overfitting. Consequently, models trained with too few samples often exhibit subpar structurization quality, commonly resulting in information loss hallucination, which adversely affects downstream performance.
2K
-0.6
5K
-1.3
10K
+2.6
15K
+3.3
22K
+3.6
Specifically, for the model trained with 2K samples, the generated outputs frequently contain incorrect formatting and struggle to parse the three-tier structure effectively. In these cases, we resort to using the original text as a fallback input for downstream LLMs, which accounts for the somewhat lower performance drop (-0.6%) compared to the model trained on 5K samples (-1.3%).
A.3 Human Evaluation
Criteria. A good structurization should effectively deconstruct the vanilla input text clearly identify its knowledge structure, and the structurized results should be faithful to the original texts, neither dismissing the factual information nor fabricating statements opinions that do not exist. Therefore, we devise three dimensions to evaluate the structurization quality, and each of the dimensions can be scored from 0 to 5 (higher is better). The detailed criteria are displayed in Fig. Al and the annotating screenshot is provided in Fig. A2
Instruction. The full annotation instruction is displayed below:
Assess the quality of structured outputs produced by different sources, and evaluate whether the structured output is sufficiently "good" by using score ranging from 0 to 5. To clarify what constitutes "good" structured result, please consider and score the following aspects, with each aspect also scoring from to Scoring Criteria: Please see Fig. AT
Payment. The payment for annotators 2,000% in total, which higher than the minimum wage in our country.
15

---

Score
Completeness
Factuality
Anti-Hallucination
in information largely retained with the omission 2-3 descriptive clauses adverbials permitted.
The structure is clear and standard with appropriate granularity as expected.
Thee content is consistent to the original text, with no aabricationsshalluciattii
The hierarchical structure retained but with minor formatting issues, such as extrancous spaces or line breaks; granularity generally suitable, with occasional instances of subsection containing only description exceeding seven descriptions.
There is an omission 20% of sentenees or the absence of introductory concluding overview//ummmary paragraphs.
The overall content is faithful, but it includes few expunded sentences not present in the original text.
4
Poor structure: lack of nspectidescription numbering, excessively long aspect nume, or overly short description sentences (lists of words); Inappropriate granularity: sentences uttached one aspect, dividing each sentence into separate aspect.
There considcrable omissions of sentences or paragraphs are present, each paragraph lacks sentences.
Considerable content is fabricated. such as offering additional large sections of "suggestion" text.
3
The basic structure preserved, but there frequent lack of numbering non-compliance with hierarchical organization.
Extensive information loss is evident, with more than half of the original sentences removed.
Severe repetition meaningless phrases are present, significantly reducing the quality of the result.
2
The majority of information is omitted, with only few key words or sentences preserved.
Chaotic structure: content is organized without adherence
Essentially nonsensical, with only few words phrases retained from the original text.
hicrarchical structure.
If the content empty completely hallueination, give it of
Figure A1: Detailed descriptions for human evaluation criteria on structurization quality.
PR
KIAAY
28
SOurCe:
prodi INDXAS
S:R::
LIID ARRE
Olte
CUEiP MA dtr2tt 18: (1K)52))
PE 20-+00 2400.
1 BHOOL.
2.2 (ULA
B892
BODE
JOUTEE:
ped:
1::::
+-R22) 4 1786
1820 MBRE
City
19
11,6-HH' 20, 400 IHO.
Souree:
prod:
JERABKEUSE
Figure A2: Screenshot for human evaluation.
16

---

B Details for Structurization Applications
B.1
Structurization Examples
Fig. A3 provides several examples of how to structurize the long-form context for downstream
applications, including reading comprehension in Sec. 4.1 hallucination detection in Sec. 4.2 and
passage retrieval in Sec. 4.3
Structurized Context
Structurized Context
Structurized Context
This paoage uila about the Privasy
The life and caces deocue Geonge Caker
DNA laiso is humas PBMC. "Bale of DNA melliylatio" DNA undio leios plays signdcastt sole bunaa beath and disaase processss.
daasc: and in clanstordiic
1. Early life and bockgrosnd
1. Intt eduction to Privacy Policies...
1.1 Cikor was boo ou the Lower East of
2. Privacy Polics Misuse and
Mathotin, the younge chid and onl sen oc...
3. Privacs Dateset...
1.2 His pareets selcttdd rildle same botor Sparssb-Ameeica War Gearge...
"Techoological adrancencots" i Recest tecmnoloiical advancenenta significuntly crenoved accuracy
Data Callection Methodolagy. Pr
comreriees mobile upilication prinacy
Career mikotonns
policces collected
Google
Cikor was the Acaiensy Award Directoe
hole-genomee bisulfite scqucnciag"*:
Soven expers legal traising provide
MM Far Lady (1964), was
Usina whole gatooge
sequescina
stowes Turker qucitions...
contneed into the 1950s....
24.7-3 corerage (12.3-fld pe
Instruction
Instruction
Instruction
Q Who some the rapers aoo for anncttion?
Judge Cricrr was the sound for Bost Dineaee Trur or Fille?
Query: What is the of INA mathybrron Tunnenn tealfs and dieeses?
Respouse
Respouse
Response
Doraain with logal traising.
A: True. George Cukor wos the Acadensy Award Sur Iee Dineesor for "My Iser Lady" (1954)
A. Matched.
Reading Comprehension
Hallucination Detection
Document Retrieval
Figure A3: Examples leverage the structurized results for downstream aookucatuibs
B.2 Details for Context-based Question-Answering on LongBench
Detailed dataset description. To focus on the investigation of context structurization, we choose 7 subsets from LongBench across single-document QA, multi-document QA, and synthetic QA tasks in English, and the remaining Chinese subsets or code-orientated tasks are eliminated:
Single-Doc QA. For single-document QA, we take two subsets from LongBench: (I) Qasper [1], featured by question-
over NLP technical papers and annotated by NLP practitioners; (2) MultiFieldQA, manually curated from multiple data sources and anno- triplets to test,
tated by Ph. students. MultiFieldQA contains 150 Context- -eeattt-uuetton-nnneeeee and Qusper and other adopted subsets include 200 pieces of test samples respectively.
Multi-Doe QA. Multi dolti-d QA requires LLMs to extract and combine information from multiple documents derive the answer, which is generally more challenging than single-doc QA. We take three multi-hop QA datasets: (1) HotporQA (65) containing 2-hop questions written by native speakers given two related paragraphs; (2) 2 WiikiMulihhoQQA
(22), involving
up to 5-hop questions synthesized through manually designed templates on Wikipedia passages; and (3) MuSiQue (54), carefully composed with up 4-hop reasoning on an increased number of supporting and distracting context evidence.
Synthetic QA. We employ two extra synthetic tasks to test LLM's long-context handling ability on specific scenarios and patterns: (1) PossageCoumt, requiring models to count the number of unique passages from shuffled passages pool; and (2) PassageRetrieval, randomly choosing one of 30 passages to obtain its summarization with GPT-3.5-Turbe and asking tested LLMs to determine the original passage with which the summarization is crafted.
Performance analysis on MuSiQue. On the LongBench dataset a slight decline (around 0.1%) on the MuSiQue subset [54] when taking structurized contexts as input. The main reason is that MuSiQue is too complicated. It requires up to 4-hop reasoning on manually paraphrased questions on a large amount of supporting and distracting context evidence. For Ch33GL that has already achieved considerable cognition ability, context structurization in the inference stage is temporally unable to bring significant advances in such complex scenario. Next, we will delve into developing methodologies for training L.LMs to capture the intrinsic structure of context, unlocking structurization's 's potential in enhancing LLMs on more complicated NLP tasks.
3 CharGLM-3-32K 67 receives
17

---

B.3 Comparison with Aspect- and Query-Baser Summarization
Aspect- Baseed Summarization (ABS) and Query- Based Summarization (QBS) are classical context augmentation methods in the NLP literature. which aims to gradually extract important information from lengthy text data based on pre-defined aspect list or user-input queries. The summary that contains key information will be integrated into context inputs to augment tested LLMs to generate reliable responses, rather than lost irrelevant sentences in the original lengthy inputs. We take ABS and QBS as the comparative baseline to further evaluate the efficacy of our proposed structurization augmentation strategy.
We investigate ABS and QBS on five document- QQA subsets in LongBench [3. Specifically, as the pre-defined aspect list unavailable for each passage in LongBench, we turn the traditional ABS to input-agnostic paragraph-based summarization to lengthy paragraphs, where the prompt template is: Summarize the following text with no more than three sentences. Passage: [text]; Summary: For QBS, the prompt template focuses on specific queries: Summarize the following text to answer the query with no more than three sentences. Query: [query); Pussage: [text); Summary: The [text) and (query) are placeholders, and the output summaries are respectively concatenated to form the augmented passages as inputs. We report the performance variance of the same test model (i.e., LLaMA2-7HH M-aaa--B-hha (53) when taking vanilla, summarized, and our structurized passages as inputs to answer the same given questions, as displayed below (due to the space limitation, we merely report the averaged enhancement on the five subsets).
Table A2: Comparison with ABS and QBS on subsets of LongBench.
DataAug
Qasper
MultifieldQA
HotpotQA
2WikiMQA
Misque
Average
19.5
34.6
30.4
27.3
10.7
24.5
ABS
15.6
24.9
29.1
27.7
8.5
21.2
QBS
21.4
30.1
31.2
27.5
13.1
24.7
Ours
23.1
35.9
32.7
29.9
13.4
27.0
Based on Tab. A2 we observed that ABS, without guidance from pre-defined aspects lists or user- input queries, failed preserve critical information, and led performance declines in four out of the five subsets. Conversely, QBS, with user query guidance, achieved improvements in the Qasper, HotpotQA, and Musique subsets. However, substantial information loss during the summarization process resulted in decreased performance on the MultiFieldQA subset. In contrast, our structured approach delivered consistent improvements across all subsets, demonstrating its efficacy.
In addition, ABS and QBS are applicable for passage-based question- guestion-answering tasks but do not extend well to other tasks such as hallucination assessment, where every piece of information counts and should not be summarized. Our approach is task-agnostic, highlighting the generalizability.
B.4 Additional Experiments for Hallucination Evaluation
Qwen models as the LLM evaluators. In Sec. 4.2 in the manuscript, we mainly study structurization with LLaMA2 and GPT-3.5-Turke on the AttrScore benchmark. For thorough comparison, this section provides the investigation of employing another series of models, i.e., Qwen-BB and Qwen- 72B (2) on AttrScore and FactScore datasets to further demonstrate our method's efficacy. Results are presented in Tab. and
and Tab. A4 respectively.
For the 7B- and 72B 2-B-arraetter Qwen evaluators, structurization can immediately improve judgment accuracy on both AttrScore and FactScore datasets. The smaller Qwen-7B model receives a more significant enhancement, along with nearly 10% improvement all metrics (except the Attr: criteria in Tab. A3 on the two datasets. It indicates smaller LLMSs may be more reliant on structurization to enhance their cognitive and ceermaationseeeiinn capabilities than large-scale LLMs. Simultaneously, the larger Qwen-72B evaluator also witnesses considerable improvement with 3%-4% increase. In particular, based on the structurized references, Qwen-72B achieves a competitive evaluation performance against GPT-3.5-Turbb on the FactScore benchmark, and even slightly outperforms GPT-4 on the Contra, criteria in AttSScore. Those results substantiate the effectiveness of leveraging structurization to further enhance powerful language models with considerable size capacities.
18

---

Table A3: Results AttrScore evaluation.
Table A4: Results on FactScore evaluation.
Method
Attr.
Contra.
Extra.
Method
InstGPT
CharGPT
PPLAI
GPT-4
87.3
45.0
89.6
GPT-4
Gurro,5-Turbo
61.2
20.6
53.3
GPT-3.5-Turbo
87.5
80.2
65.8
Alpaca-13B
50.6
6.1
19.3
LLaMA-65B
54.6
42.1
36.1
Alpaca-7B
50.7
8.6
3.6
Inst..LaMA-78
67.1
55.1
Qwen-7B
60.9
11.6
34.2
Qwen-7B
68.5
50.8
40.8
+Ours
61.9
25.9
44.4
+Ours
79.2
64.8
46.0
Qwen-72B
75.8
37.7
77.6
Qwen-72B
86.9
75.2
60.5
+Ours
77.8
47.5
80.2
+Ours
89.0
79.4
61.8
Incorporation with GPT-3.5/4 models and CoT techniques. In Sec. 4.2 we the popular chain-of- thought (CoT) technique for prompt augmentation on the advanced GPT-3.5-Turbo model on the AttrScore (66 benchmark, where Tab. AS is a simply copy of the experimental results. Our method presents comparable performance against COT,, and achieves better enhancement after integrating with CoT, illustrating the compatibility and extensibility more advanced strategies.
Consequently, Tab. A6 further presents the incorporation of our StruXGPT with more powerful GPT-4 model and the CoT technique, showing consistent benefits for powerful GPT-3.5 and GPT-4 models, either with or without the CoT strategy. The experiments and discussions further validate the effectiveness of our StruXGPT approach.
Table A5: Integration with GPT-3.5 and CoT.
Table A6: Integration with GPT-4 and CoT.
Method
Attr.
Contra
Extra.
Avg.
Method
Amr
Contra.
Extra.
Avg.
72.0
30.4
71.7
58.0
86.2
43.3
88.3
72.6
+Ours
77.1
318
77.4
62.1
+Ours
87.6
89.7
75.2
+CoT
76.4
35.3
74.4
62.0
+CoT
88.8
48.9
89.7
75.8
+CoT+Ours
78.9
42.9
74.5
65.4
+CoT+Ours
88.5
52.8
90.3
77.2
B,5
Comparison with Other Augmentation Methods
Besides the summarization baased and prompting-basee based augmentation methods (in Appendix B.5 and Appendix (eetrddxxBBBresspeepeee respectively), here we also compare our structurization approach with AdaptLLM 7 which developed several domain-specific L.MSs (in BioMedicine, Finance, and Low) via the proposed ceading-oomprehension training technique. The experimental results are presented in Tab. A7
Table A7: Comparison with AdaptLLM various domain benchmarks.
Domain
Subset
Metric
Baseline
AdaptLLM
Ours
Medicine
PubMedQA
Acc
59.6
63.3
63.0
Finance
ConvFinQA
EM
29.2
41.5
36.5
Law
SCOTUS
mic-F1/mac-Fl
28.3/10.8
30.0/17.8
30.6/15.6
General
BoolQ
Ace
55.7
53.9
58.2
Accordingly, our method can also boost the Llama-7b baseline for 3%-7% without training, while AdaptLLM requires costly continual training of the baseline model on each domain corpus. In particu- lar, for the general reading comprehension task, AdaptL.LM-Fi does not introduce significant boosts, while MdaptLLL-BBoo// -io-Bio/Lav even cause performance drops, which is mainly because AdaptLLMM domain- addattttnon tuning will harm the general capability more or less. contrast, our method does not alter the bascline model, but only structurizes the input context to enhance LLM's cognition
19

---

ability on downstream tasks, showing stable and consistent improvements (e.g., 2.5% increase on BoolQ). Although our final performance is slightly inferior to the domain-specialized AdaptLLM. our generalizability emphasizes the contribution of our work, as we bring consistent enhancement across downstream domains and cause no degradation on general tasks.
C Limitation and Future Work
Training-time aggregation. This paper mainly investigates structurization during LLM's inference stage. Despite the universal enhancement across language models and NLP tasks, structurization has not yet fully tapped into its potential. Specifically, developing methodologies for training LLMs to capture the intrinsic structure of context and extrinsic correlations among training corpus presents an unresolved challenge in the field. We will delve into this problem in our future work.
Inference efficiency. We acknowledge that the introduced inference cost is common limitation for the methods adopting LLMs for data augmentation, which depends on computing resources and the optimization techniques employed. Here conduct extra experiments with the competitive augmentation methods (i.e., ABS and QBS in Appendix B.3) for an in-depth comparison on the Long- Bench (3] dataset for guestion-answering evaluation. Besides, also investigate the AttrScore [66] dataset for hallucination evaluation to assess the impact of input length and output length, as well as the model size. The inference time, measured in seconds per sample, is calculated on NVIDIA A100 GPU with vllm acceleration (except for the LLaMA2-70B model, which demands at least two A 100 GPUs for deployment).
Table A8: Time cost analysis for reading comprehension in LongBench.
DataAug
Enhancement
Task-Agnostic
Extra Cost
Total Cost
+0.0
0.7s
ABS
-3.3
V
2.7s
3.4s
QBS
+0.2
X
2.5s
3.2s
Ours
+2.5
V
2.9s
3.6s
t Ours SoT
+2.5
V
1.2s
1.9s
LongBench contains input passages with an average length of 14K tokens (computed with the LLaMA2 tokenizer), which burdens the cost for all the data augmentation methods. According to Tab. A8 only the task-specific QBS can bring a positive enhancement of 0.2% on average, which is negligible compared with the 2.5% boost by our task-agnostic structurization approach. However, the LLM-based data augmentation methods generally introduce an extra 2.5-2.9 seconds cost. To alleviate this problem, we have recently noticed a new decoding acceleration technique named Selection of- Thought (SoT) [5), which is also motivated by the aspect-based thinking and writing process of humans and is naturally compatible with our structurization process. With the 2.39 x speed up and negligible quality decrease (denoted as "f Ours + SoT"",, the extra cost of our method can be further reduced to an acceptable 1.2 seconds, which we believe makes more practical.
Table A9: Time cost analysis for hallucination evaluation in AttrScore.
LLM-Evaluator
Enhancement
Total Cost
Extra Cost (%)
LLaMA2-7B
+0.0
1.4s
+Ours
+6.4
3.5s
150%
LLAMA2-70B
+0.0
17.6s
+Ours
+4.3
19.7s
12%
AutrScore requires a relatively longer output length (with a detailed explanation for hallucination evaluation), and the vanilla time cost increases compared to LongBench. According Tab. A9 our
"haps://eithub..om//lln projecc///i
20

---

structurization earns 6.4% gain on the average hallucination evaluation accuracy (detailed values are shown Tab. 2). in trade of the 1.5x increased time cost for the vanilla LLaMA2-78 evaluator. As the base evaluator is scaled up to larger L.LMA2-70B model, our method merely introduces 12% extra cost (2. Is) and still gains a considerable enhancement of 4.3%. Our method showcases more advantages by incorporating larger base models.
Performance on General Evaluation Benchmarks As our method focuses on context structurization to enhance LLM's cognition ability, for the tasks where no context is provided, our method may not bring significant enhancement and even get worse. The commonly used MMLU benchmark [2]] is typical scenario, where LLMs are asked to answer questions without context references but requiring their parametric knowledge (learned during large-scale pre-traue are-training) and context structurization does not help. If we insist on structurizing the question alone and feeding it into LLM's inputs, the model may disturbed by the introduced information from StruXGPT and generate wrong answers. As shown in Tab. A10 our method causes slight 0. % decrease (measured by OpenCompass protoco when taking L.a.MA-7B--CI [53] as the baseline model.
Table A10: Evaluation on general benchmarks.
Model
MMLU
BBH
LLaMA2-7B-Char
45.93
30.47
+Ours
45.84
31.30
On the other hand, we have also tested another common benchmark, BBH [49]. which is designed to evaluate LLMs' reasoning capability when dealing with several logical sentences/statemments In this case, our method can adapt well to highlight the logical relations and boost LLM's 's reasoning abilities by 0.8%.
In conclusion, we suggest users apply structurization to long-form or logically complex contexts, while taking the original question as inputs when there no context provided.
D Broader Impacts
We discuss the positive and negative societal impacts as follows:
Positive Societal Impacts. Through the innovative approach of context structurization, this work significantly enhances the comprehension abilities of Large Language Models (LLMss,, leading to more effective and efficient applications in various sectors such as education, healthcare, and customer service. Avoiding the need for larger models, not only curtails the environmental impact associated with training sophisticated AI systems but also democratizes access cutting-edge AI technologies. This fosters a broader base for innovation and empowers smaller entities with the tools to contribute meaningfully to technological advancement. Moreover, our method, rooted in human cognitive processes, promises to enrich human- AI collaboration, paving the way for solving complex societal issues by leveraging Al's improved problem problem-solvingg capabilities.
Negative Societal Impacts. On the flip side, the advances made through structurization bear potential risks, including the augmentation of disinformation campaigns and the creation of more sophisticated deepfakes, which could undermine trust digital content. The enhanced capabilities of L.LMs might also be co-opted for intrusive surveillance, raising substantial privacy and ethical dilemmas. Furthermore, the inherent biases in training data could be deepened, potentially automating and perpetuating social inequalities. The concentration of advanced AI technologies in the hands of few could limit competition and place immense power over information dissemination with those entities, while increasing reliance on AI for critical decision-makinn making could inadvertently erode human analytical skills over time. To mitigate these concerns, it is essential to establish rigorous ethical standards and robust oversight mechanisms that govern the deployment of LLMs, ensuring transparency and accountability in their use. Furthermore, active research and the implementation of bias detection, correction frameworks, and the development of media literacy programs crucial
21

---

steps to preemptively address the potential misuse, privacy violations, and societal impact of these advanced AI systems.
E Prompt Examples for Structurization
E.1 Prompt Templates
Fig. A4 displays the few-shot prompt template to query giant commercial LLMs to execute structur- ization. We first introduce the instruction for structurization (including the task definition and output formats), then provide two representative examples to further illustrate the process. The first example in teltell the model to focus the existing numerical enumeration indicators to assist in constructing the espect-description structure. The second example in Fig. A6 teaches the model to automatically summarize the aspects and attach their corresponding deseriptions from the text sequences. Finally, in Fig. A4 we ask commercial LLMs to structurize the input_statenent from user input expected.
To ensure completed and faithful structurization on long-form input statements (usually with thou- sands of tokens), we provide two detailed exemplars in Fig. A5 and Fig. A6 commercial LLMs' few-shot learning. However, these two exemplars take a considerable context length, which may confuse the distilled StruXGPT-7B with smaller size capacity. We thus remove the examples in StruXGPT-775 s prompt template, as StruXGPT-7B has Iearned how to perform structurization desired and the functionality of those examples is negligible. Specifically, the erample_ 1 and erample 2 (with their introductions) are removed for both efficiency and efficacy.
F,2 Structurization Examples
In this section, we present several structurization examples from different models as discussed in See. B in the manuscript. Given an input statement about Facebook's 's stock, Fig. A7 shows that Qwen-max produces factual and faithful structurization result with the help of few-shot exemplars. clearly identifies the main aspects
the original statement, and attaches the corresponding descriptions into each aspect. On the contrary, Fig. A8 indicates the smaller pre-trained 7B-size Qwen and LLaMA2 models failed to follow the instructions to produce qualified structurizations. There exists fabricated "Note" content in Qwen-7B's response. Meanwhile, LLaMA2-78 is even unable to follow the desired format, and information like "Facebook doesn't sell anything tangible" missing. Fig. A9 demonstrate the effectiveness of our fine- sene-tune 7B 18-parameter StruXGPT model, which is able to structurize the input statement as expected, with comparable performance to the giant Qwen-max model with over 200B-size.
22

---

You are helpful NLP aesistant.
Help rephrase given statenent identily:
(1) summary of the staterent's scope, which should generally be phrase with few words.
iain aspects shich the statenent discussing. main aspect should be The aspect list should be precise and concise to conclude the statement, and the rumber of aspects should be linited.
noun parase.
(3) enumeration of descriptive sentences regarding each espect above, which display the details of those aspects. Each description ion sentence must be completed and faithful to the original statement. You should NOT rouove any descriptive seguent this layer.
Given an original statement, the rephrased stracture should strictly follow this format:
## Statement' scope:
jpenerally noun, phrase]
## Statement's rain espects and corresponding deseriptions:
1. [the First aspect of the statemea:]
descriptive sentence iorrusponding to asport]
1.2 [a deseriptive sentence corrosponding to this aspent]
1.3 (snother descriptive sentence, necessary]
[the secand aspect the statement]
2. deseriptive sentence corresponding ta this aspect]
lenother descriptive sentence, necessary]
Lonother aspect of the statenent, if necessary]
Here example to illustrate how rephrase an Input starement the destred structure. Pay attention to nunerical eomeratiion indicators, like 'I. '2. (1) (2) etc.
eexample_t
Here is another exapll ta Illustrate how to rephrase an input statement as the desired structure. When there are icit words indinating statement seope aspects, please fex worda pracisely scope 8s well as the main aspects. Then you carefully attach the descriptive sentences to each noin aspect.
eexapple 2]
Now sumarize the "scape" of the fol Fowing statement with few wards, and then rephrase the input. statement to its "main aspects" "corresponding descriptions' the munercallly ly ordered fornet strictly. Note that the aspect list should be precise and concise to conclude the statenent, and the nunher of asports should be linfted.
Each description sentence zust be completed and faithful to the original statenent, and should NOT renove any deseriptive segnent in this layer.
Iaput:
iinutt statement]
Outpot:
Figure A4: Prompt template for few-shot structurization.
23

---

Input:
Comprehensive prevent ion for malignat diseuses in the rice seedling stage lollows:
Choose dieease-free seeds. Do not leeve seeds in diseased rields and nearby rice fields. Choose healthy rice and elininate diseosed, dead, and jnrer rice.
Seed disinfection. Before soving, sook the seeds with 23% 100g (Xibek) EC 3000 times liquid for I to days, grans Dexinging ve.lable powder every kilograns rice seeds. Soek seeds kg water for 60 hours.
secsed strox. not cover germinated dry seedlings with diseased straw.
Outpet:
## Statenent's 's scope:
The comprebensive prevention neasures for malignent discases In rice seedling"
## Statement's is main aspecta and carresponding descriptions:
Choose disease-free seeds
1.1 Do not leave seeds in diseased fields and nearby rice fields.
Choose healthy
elininate ciseased, dead, and injured rice.
Seec disinfection
100g (Kibok) EC tixes liquid For 2.2 An alternative is taking 20 grans of 176 Dexinging wettabla pander for every 6 kilograns of rice seeds. the seeds kg of water for 60 hours.
sowing,
Deal with sesssnd rice strax
3. not cover gernimted or dry smellings with diseased stran.
Figure A5: The first example for few-shot structurization.
24

---

Input:
The water ahsorption turve of rice after soaking is an uninndal suree. The inflection point of the curve is the poak period vater absorption.
absorption and tine nor-lineer can expressed by the folluwing forull:: wstib-c, stere a, h, and are constants. Lnder different hunidity conditions. change the water ahsorption rate of rime betxeen D and the point d inflection, and the rate of change accelerates over tine. After the inflection point, the increase In water absorption gradually stabflizes. The changing rules sater absorption and noisture content of rice sixilar, but under different hunidity conditi tion, the relationship between water absorption of rice and uoleture content is different. Mee the relsture content is los, the water absorption of rice increases as the moisture content increases. Vhen the noisture content is high, the increase in
relationship betveen
basically siriler, that is, the sater ahsorption rate
absurption of rice gradually
stabilizes. There are three obvious steps for rice seeds to absorb vater: First, at the begiming of vater absorption, the cuntent of the seeds gradually increuses, and the the peak vater absorptica period, the vater abeorption rate increases repidly. lhird, in the later stage of sater sbsorption
absurption rute slevly increuses, Second, during
absoration, the vater abscrption rate of rice seeds is closely related to texperature. In generol, water absorption increases as terperature
seeda slovly incresses, and the
increases,
relationship betseet. vater absorption and temperature be expressed unsaturated voter absorption (3) 14. 2891-10. 719 (where T is temprrtture)
Outzut:
Statement's scope:
charactoristies of soaking rice seeds to absorh sufficient water"
## Statement's 's moin aspects and correaponding deseriptions:
1. The water absorption curve
1.1 water absorption curve rice after sooking is an uninodal curve.
1.2 The inflection point is the peak period of water ebsorption.
1.3 The relationship betveen rice vater absorption and tine is non-linear, which can be expressed by antibre, constants,
where b, and
2. Ricn water absorption as time changes
2.1 Under different hunidity condi tions, the change in the water absorption rate of rice with time is basically
sini lar.
The water absorption
betxeen and the point inflection.
3.3 The rate of change ceaalerate over time
2. After the inflection. point, the incresse vater absorption gradually stabilizes.
Re Belationship hetwoen rine sater ahsorpt ion and misture content
water absorption misture content 3.2 Inder di different "unidity candi tions, the relationship between sater ahsorption of rice and misture content is different.
changing
are sinilar
3.3 Khen the moisture content is Tox, the water shsorption of rice increases as the moisture content increases.
noisture concent high, iacresse in water absurption rice graduilly stabil 4. Three stages for rice seeds to ahsorh sater
4. ät the beginning of vater absorption, the water content of the seeds gradwlly increases, and the water
ahsoration rate slouly fneraases.
4.2 During the pesk vater absorption perioó, the vater absorption rate increases rapidly.
4.3 In the later stage of sater absorption, the vater content of seeds elonly increuses, and the water absorption rate increases.
Relationship betwern
absorption rate and tempratare
5. The water absurption rate of rice seeds is closely related to tempersture.
5.2 Kater ahsorption gorerally increases terperature incrusses.
5,3 The relationship cur. be expressed as: unsaturated water absorption (3) 14. 289T 10. 719 (shere T. tesperatura)
Figure A6: The second example for few-shot structurization.
25

---

Source Statenent
The amount of hope and uneducated investors/ speculators driving its prices up. Just by that, I would say its priens Facebook doesn't sell anytaing tengible. They down as
inflated. Bear Nost of their income is ad-besed and single-prodict orlented, and such hishly dependent on usage and trends (remenber MySpuce?)) Having said that, all the other "classic" veluation techniques are still valid end you should iiilixe them.
they up.
Qwen max's is Output [ Statenent scope:
Valcation concerns for Facebook due kype and lock of tangible products"
## Statement' uain aspects end corresponding descriptions:
lype uneducated investors/ investorssspoculstrre
1.1. The amount of hype and uneducated Investors/speculators is significant, which drives up the prices.
1.2. suggests that the current prices are inflated besed this fector alone.
Lack tangibbee products
2. Facebook doesn't t sell any tangible goods, noking volue potentielly unstable.
2.2. Prices drop rapidly, similar to their of incressn.
Ad based income and dependency
3.
Facebook' incone ad-based focused on single product.
3.2. Their revenue is highly dependent on usage Lrends and popularity (citing the deel iie of MySpore). valuation tachniques
4. I. Despite these concerns. :raditional valuation nethods remain valid for assessing Facebook 's value.
4.2. recomended to utilize classic valuation: techniques when evalcating the company.
Figure A7: An example for source inputs and structurized results from few-shot Qwen-max max.
Qwen 7B's Output
## Statement 's soope:
warket analysis of Pacehook stock""
#2 Statement's is main aspects and corresponding deseriptions:
1. Inf'luence of hype and uncdurated investors on stock prices
stack. ancunt hype uneducated investors driving prices up to an infleted
price increase can unstable, as company'
dependent on user grouth
Eraluation Facehook's business model
2.1 Fecebook's 's revense is primarlly generated through advertisiag and is highly degendent on user behavior and trends.
2.2 Their incone is single-product product oriented, so they are vulnerable to changes in narket conditions.
The original statement quite long and corplex, It discusses the influence of and unedurated investors on stock prices, before noving on to an evaluation Pacebook's business rodel, The scope the statment includes the analysis Facebuck stock, but could be precise by narrowing it dumn just "stock prices." This is why the First part of the repheased structure Includes the phrase 'aurket enalysis of Facebook stock." In the second part, prices,
Note:
investors Facebook's ss bus iness model. Bach espect is further divided into several descriptive sentences that are faithful to the original statement.
identified
aspects:
aype
LLaMA2_ "B'__'''s Output
Scope: **
reasons behind high vallution of Faceboook"
Main Aspects and Corresponding Destriptions:
Tae influence of hype und uneducated Investors/ /tttor////peetoorr
1.1 Just by that vould say its prices are inflated.
1.2 Facebook's 's reliance
ad-based and single- product--riented Income exposes it narket trends and
Deperdence on usage and trends
2. Remerber VySppuce?
Validity of alternative valuation techniques
3. Baving said
the other "classic" valuation techniques still valid you should utflize then.
Figure A8: Corresponding structurization results from few-shot Qwen-7B and L.MMA2-7B.
26

---

StruXGPT_TT Oren's Output
## Statement" scope:
valuation Facebook's stock considering umpractioal factors"
's aspects and corresponding descriptions:
Price inflation to hype and Inexperienced investers/ sappccllators.
1.1. The vast anounts
surrounding Facebook's 's stock drive Sts prices
1.2. This suagests that the nateriality its assets not Justified.
Facebook's is Finaccial stability and potential For volatility.
2. L. Facebook doesn't t sell any tangible gooda.
2.2. They decline rapidly
have grown
2.3. The jorlity of its income comes froe advertising and single product model, noking it highly dependent on (parallels
usage
NySpace's decline).
3. Validity and appl Fration of valuation techniques.
traditional valuation
are appl Pacebook's
3.2. Investors advised to utilize these techniques despite the platfom'ss 's unique situation.
StruXGPT_ 78, ILLMM's is Outgut
## Statementt 's scope:
scconceptions about the narket veluee of Facebook based on uneducated sentinent and issues related to Its business model
Statement's aspecta corresponding descriptions:
Hype unedueated investars
1. The amount of hype around Pacebook is driving its price.
1.2. Inedunated investors/spennt iomuultta stors are also contributing to the inflated prices.
Innene dependence.
2. 1. Over 50% of Facehook's as income comes from adverti sements.
signifieant portion Facebook's 3. Business model volatility
is rellent upon usege and trenda.
not sell any tangible products, hence 3.2. As such, gnins can be just as quick as they wore achiaved.
The company
inherently volatils.
Classic valuation techniques
4. Despite hypo and inssn urpredictahility, other "classic" valuation techniguns renein agpl icabla assess
norket value.
4.2. They should iantinue he utilixed for coaprehensive
Facehook's worth.
Figure A9: Corresponding structurization results from our fine-tuned StrUXGPT-7B-Qwar and SruXGGPT-B--LLabe
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We accurately made the claims to reflect the paper's contributions and scope
in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation of our work in Appendix C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
This paper does not involve theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the experimental details in Sec. Appendix A and Ap- pendix B
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification:
Please refer to
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
www.SSubbissooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the experimental details in Sec. Appendix A and Ap- pendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We show consistent improvements across three datasets with various models in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported the compute resource requirements in Appendix A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discussed the broader impacts of our method in Appendix D Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please refer to
com/ commalibaba/strure
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All our experimental data and models open source and obtainable.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer hutp:s///gittubb.ccmmaaaaabaaaba
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Please refer to Appendix A.3
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We only hire annotators to label the structurization quality, as introduced in Appendix A.3 which does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "dISyZoSzwB", "reviewer": "Reviewer_7Sjn", "summary": "This paper introduces the concept of context structurization to enhance the comprehension capabilities of large language models (LLMs) for long texts. The authors propose summarizing the input text into a three-layer structure of Scope-Aspect-Description using LLMs, and then inputting this three-layer structure as an enhanced version into the LLM. Additionally, the authors propose to optimize a 7B model by distilling the structuring capabilities of giant commercial LLMs to reduce computational costs. The effectiveness of the method is validated through experiments on multiple NLP tasks. The experiments also demonstrate that the fine-tuned 7B model can inherit most of the structuring capabilities of the giant commercial LLMs.", "questions": "How does the number of in-context examples affect the results of structurization, and why was the number set to 2?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and well-structured, making it easy to understand.\n\nThe proposed method is technically sound.\n\nThe effectiveness of context structurization has been validated across multiple datasets in Context-based Question-Answering, Context-based Summarization, and Passage-level Dense Retrieval, showing improvements on a number datasets.", "weaknesses": "The experiments are not sufficiently comprehensive. Firstly, many tables only compare the scenarios with and without StruXGPT (except for Table A2 and A5), without comparing against other advanced prompt engineering methods. Therefore, it is hard to determine whether StruXGPT enhances the cognitive abilities of LLMs more effectively than other methods. Secondly, the authors did not validate the effectiveness of StruXGPT on state-of-the-art LLMs such as GPT-4, so it is unclear whether GPT-4 would still perform well without context structurization.\n\nThere are also some related summarization-based augmentation methods that could be discussed or compared [1].\n\n[1] Cheng, Daixuan, Shaohan Huang, and Furu Wei. \"Adapting Large Language Models via Reading Comprehension.\" In The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 7Sjn:\n\nWe thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:\n\n> **W1**: Many tables only compare the scenarios with and without StruXGPT (except for Table A2 and A5), without comparing against other advanced prompt engineering methods. There are also some related augmentation methods that could be discussed or compared [1].\n\n**A**: In our manuscript, we mainly focus on consistent improvements across downstream tasks with our StruXGPT, and also compared with summary-based strategies for question-answering (as mentioned in Table A2 and A5), as well as the popular chain-of-thought (CoT) technique for hallucination evaluation (in Table 2). Below is a simple copy of the experimental results built on top of GPT-3.5-Turbo on the AttrScore dataset:\n\n|Evaluator|Attr.|Contra.|Extra.|Average|\n|:--|:--:|:--:|:--:|:--:|\n|GPT-3.5-Turbo|72.0|30.4|71.7|58.0|\n|GPT-3.5-Turbo + Ours|**77.1**|**31.8**|**77.4**|**62.1**|\n|GPT-3.5-Turbo + CoT|76.4|35.3|74.4|62.0|\n|GPT-3.5-Turbo + CoT + Ours|**78.9**|**42.9**|**74.5**|**65.4**|\n\nAccordingly, our method presents comparable performance against CoT, and more importantly, our method does not conflict with existing advanced prompt engineering methods.\nAs shown in the above table, we achieves better enhancement after integrating with CoT, illustrating the compatibility and extensibility to more advanced strategies.\n\nHere, we further compare with AdaptLLM[1] on the BoolQ dataset for reading comprehension. In particular, AdaptLLM developed several domain-specific LLM (in BioMedicine, Finance, and Low) via the proposed training technique, which however causes inferiority in general reading comprehension capability. Compared to the baseline model (LLaMA-7B), AdaptLLM-Fin does not introduce significant boosts, while AdaptLLM-Bio/Law even cause performance drops, which is mainly because AdaptLLM's domain-adaptation tuning will harm the general capability more or less.\nIn contrast, our method does not alter the baseline model, but only structurizes the input context to enhance LLM's cognition ability on downstream tasks, showing stable and consistent improvements (e.g., a 2.5% increase on BoolQ). \n\n|Dataset|Metric|Baseline|AdaptLLM (Bio/Fin/Law)|Ours|\n|:--|:--:|:--:|:--:|:--:|\n|BoolQ|Acc|55.7|50.7 / 55.8 / 53.9|**58.2**|\n\nWe will add those comparisons and discussions in our revised manuscript. \n\n> **W2**: The authors did not validate the effectiveness of StruXGPT on state-of-the-art LLMs such as GPT-4.\n\n**A**: Thanks for this suggestion. As discussed above, we have evaluated our efficacy on the powerful GPT-3.5-Turbo model in our manuscript, and here we further extend StruXGPT to GPT-4-Turbo to investigate the effectiveness:\n\n|Evaluator|Attr.|Contra.|Extra.|Average|\n|:--|:--:|:--:|:--:|:--:|\n|GPT-4-Turbo|86.2|43.3|88.3|72.6|\n|GPT-4-Turbo + Ours|**87.6**|**48.3**|**89.7**|**75.2**|\n|GPT-4-Turbo + CoT|**88.8**|48.9|89.7|75.8|\n|GPT-4-Turbo + CoT + Ours|88.5|**52.8**|**90.3**|**77.2**|\n\nAs shown above, our method presents consistent benefits for powerful GPT-3.5 and GPT-4 models, with or without the CoT strategy.\nWe believe the experiments and discussions can further validate the effectiveness of our StruXGPT approach.\n\n> **Q1**: How does the number of in-context examples affect the results of structurization, and why was the number set to 2?\n\n**A**: In our paper, we choose 2 in-context examples to prompt commercial LLMs (as a teacher) to generate data pairs of raw/structurized text for training our StruXGPT-7B model (as a student). We think it is enough for teacher models to understand the structurization process and generate valid training samples, as the 2 examples respectively describe the 2 most common types of real-world text (i.e., with/without existing indicators like `1`, `2`, etc). \n\nTo further verify it, we investigate the number of in-context examples for teacher models with two evaluation protocols (as in Table 4 in our manuscript): AppEval (an improvement on Qasper subset with context structurization) and SemEval (semantic similarity with raw and structurized texts in the validation set, captured by BERTScore). Specifically, AppEval evaluates how much the structurization can enhance baseline models' cognition capability, and BERTScore verifies hallucinations during the structurization process. Besides, we also report the error rate when parsing structurization results from the teacher model's outputs (denoted as _FormatError_). We respectively adopt 1/2/3 few-shot examples to evaluate the structurization quality, and the results are displayed as follows:\n\n|nShot|AppEval|BERTScore|FormatError|\n|:--|:--:|:--:|:--:|\n|1-shot|+1.8|0.282|25.4%|\n|2-shot|+3.2|**0.308**|7.4%|\n|3-shot|**+3.3**|0.302|**5.5%**|\n\nAccording to the results, 1-shot is apparently insufficient to illustrate structurization, while 2- and 3-shot achieve comparable structurization quality evaluated by AppEval and BERTScore.\nNotably, 3-shot receives a 2% lower FormatError than 2-shot, in trade for the increased inference cost (because of increased few-shot samples).\nWe argue that for the final StruXGPT training, the 2% gap (around 400 samples from 22K in total) does not make a difference, as we will eliminate the samples with the wrong structurization format from the training set.\n\nIn conclusion, we recommend users to apply 3- or even more shots when prompting teacher LLMs if available, otherwise 2-shot is also a good choice to balance the inference cost and structurization quality. We will add the experiments and discussions in our revised paper.\n\nWe hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.\n\nBest regards,\n\nAuthors\n\n---\n\n[1] Cheng, Daixuan, Shaohan Huang, and Furu Wei. \"Adapting Large Language Models via Reading Comprehension.\" In The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqNaIYyPYR", "reviewer": "Reviewer_8sM9", "summary": "The paper presents a novel approach to improve the cognitive capabilities of large language models (LLMs) without inferring the model by structuring contextual information hierarchically. The authors propose transforming plain, sequential text into a structured format, enabling LLMs to process and understand complex information more effectively. This method is empirically tested across various NLP tasks, demonstrating consistent performance gains. Additionally, the paper introduces StruXGPT, a distilled 7B model that efficiently executes the structurization process, enhancing the results on context-based QA benchmark, Hallucination Evaluation and Passage-level dense retrieval.", "questions": "1. Did you experiment with smaller/bigger models for structurization? If yes, what were the results? \n2. Are there specific tasks in which such structurization could be worse? \n3. Did I understand correctly that the data for training StruXGPT was gather by quering bigger model? Did you analyze the training set in details and do you think the proposed approach would benefit from training data gather or annotated by humans? I am personally worried here about the hallucinations that could be inherited from the bigger model.\n4. Do you plan to publish your StruXGPT model? \n\nAlso in Table 1 you write \"The prefix Struct- indicates...\" but there is not such prefix in the table. Did you mean \"+StruXGPT (ours)\" here?", "limitations": "1. Paper investigates only inference stage \n2. Introducing StructXPGT is still additional computation cost, however authors provide the analysis of extra cost in appendix", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. While structurization of text is not a novel concept itself, the development of StruXGPT, a smaller and more efficient model for structurization, is a significant contribution and it demonstrates the practical applicability of the proposed method.\n2. Evaluation and experiments are strong parts of paper, authors provide many detailed experiments with proper comparisons. The demonstrate stable performance gains using proposed method.\n3. The paper thoroughly describes the structurization process, including the use of linguistic markers and transformation templates, ensuring clarity and reproducibility.\n4. Authors made the research deeper by analyzing attention mechanism on structured texts.", "weaknesses": "1. Lack of parameter-efficiency analysis: authors provide 7B model for structurization process, but it is not clear whether smaller or bigger models would significantly hurt or benefit the approach.\n2. While the proposed approach shows stable improvements, it is still a lot of additional parameters, which is a concern for scalability to large amount of texts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 8sM9:\n\nWe thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:\n\n> **W1/Q1**: Lack of parameter-efficiency analysis: authors provide a 7B model for structurization process, but it is not clear whether smaller or bigger models would significantly hurt or benefit the approach.\n\n**A**: Thanks for this suggestion. We have implemented our StruXGPT on Qwen-1.8B/7B/14B respectively. We follow the setting in our ablation section to investigate the mode size with two evaluation protocols: AppEval (an improvement on the Qasper subset with context structurization) and SemEval (semantic similarity with raw and structurized texts in the validation set, captured by BERTScore). Specifically, AppEval evaluates how much the structurization can enhance baseline models' cognition capability, and BERTScore verifies hallucinations during the structurization process. In addition, we also report the error rate when parsing structurization results from the trained StruXGPT model's outputs (denoted as _FormatError_).\n\n|StruXGPT|AppEval|BERTScore|FormatError|\n|:-|:--:|:--:|:--:|\n|Qwen-1.8B|+2.7|0.299|_5.0%_|\n|Qwen-7B|+3.6|0.313|0.0%|\n|Qwen-14B|**+3.8**|**0.323**|0.0%|\n\nCompared with the 7B model chosen in our paper, the smaller 1.8B model, despite its positive enhancement on downstream applications (+2.7), shows slight inferiority in both AppEval (+2.7 v.s. +3.6) and BERTScore (0.299 v.s. 0.313), and presents 5% error rate when parsing structurization results. \nOn the other hand, the 14B model brings further improvement to BERTScore, meaning that the structurization content is relatively more faithful to the original text. But the boost on AppEval is insignificant, reaching the upper bound enhancement of structurization.\n\nTherefore, the chosen 7B model is a good trade-off between model capacity (training/inference efficiency) and structurization quality.\nWe will add the experiments and discussions in our revised paper.\n\n> **Q2**: Are there specific tasks in which such structurization could be worse?\n\n**A**: As our method focuses on context structurization to enhance LLM's cognition ability, for the tasks where no context is provided, our method may not bring significant enhancement and even get worse.\nThe commonly used MMLU benchmark is a typical scenario, where LLMs are asked to answer questions without context references but requiring their parametric knowledge (learned during large-scale pre-training), and context structurization does not help. If we insist on structurizing the question alone and feed it into LLM's inputs, the model may be disturbed by the introduced information from StruXGPT and generate wrong answers. As shown below, our method causes a slight 0.1% decrease (measured by OpenCompass protocol) when taking LLaMA2-7B-Chat as the baseline model.\n\n|Model|MMLU|BBH|\n|:--:|:--:|:--:|\n|LLaMA2-7B-Chat|**45.93**|30.47|\n|+Ours|45.84|**31.30**|\n\nBesides, we have also tested another common benchmark, BBH, which is designed to evaluate LLMs' reasoning capability when dealing with several logical sentences/statements.\nIn this case, our method can adapt well to highlight the logical relations and boost LLM's reasoning abilities by 0.8%.\n\nIn conclusion, we suggest users apply structurization to long-form or logically complex contexts, while taking the original question as inputs when there is no context provided.\n\n> **Q3**: Did I understand correctly that the data for training StruXGPT was gather by quering bigger model? Did you analyze the training set in details and do you think the proposed approach would benefit from training data gather or annotated by humans? I am personally worried here about the hallucinations that could be inherited from the bigger model.\n\n**A**: Yes, the training data for StruXGPT is distilled from a bigger commercial model (Qwen-max in our paper), and we agree that our method can benefit from human annotations. However, only a large amount of annotations on structurization data can bring significant benefits, which are usually inefficient and unaffordable. Our current data collection and filtering strategy is sufficient to construct a qualified training dataset.\n\nHere we delve into the quality of our training data by taking BERTScore as a proxy metric, as it shows a high degree of consistency with human annotation (see Table 4 in our manuscript). According to the statistics below, over 94% of the data pairs have a positive BERTScore (normalized by the baseline score of 0.83, and a positive BERTScore presents a benign similarity), demonstrating the high quality of our training data.\n\n|Score|>0.0|>0.1|>0.2|>0.3|\n|:--|:--:|:--:|:--:|:--:|\n|Ratio|94.45%|75.89%|51.43%|32.68%|\n\nFurthermore, we eliminated around 5% of data with negative scores and trained another StruXGPT model, and the evaluation below indicates this part of data does not hurt the final performance. \n\n|Training Data|AppEval|BERTScore|FormatError|\n|:--|:--:|:--:|:--:|\n|vanilla|+3.6|0.313|0.0%|\n|filtered|+3.4|0.316|0.0%|\n\nOn the other hand, according to the ablation in Table A1, data quantity may play a vital role in further improving our StruXGPT.\nWe can curate more raw texts and prompt teacher LLMs to generate structurization candidates, and apply our filtering strategies to construct a large-scale and high-quality dataset for training.\nWe will add the experiments and discussions.\n\n> **Q4**: Do you plan to publish your StruXGPT model?\n\n**A**: Sure! The model, code, and data will be made all public soon, so as to facilitate future research in the LLM community.\n\n> **Q5**: In Table 1 there is not such prefix of _Struct-_. Did you mean \"+StruXGPT (ours)\" here?\n\n**A**: Yes, it is a typo, and we will fix it in the revised version. Thanks for pointing out that.\n\nWe hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8bMcuQyyce", "reviewer": "Reviewer_yxZb", "summary": "This paper proposes a new technique for prompting Large Language Models (LLMs) called StruXGPT. The basic idea is to transform the original prompt into a more structured description of the request which contains three levels of information: Scope, Aspects, and Descriptions. While the **Scope** provides an outline summary of the request, the **Aspects** present an itemized and well-ordered list of topics that are associated with their respective **Descriptions** which provide the details. In the paper, the prompt transformation is achieved by either using another large model or by a smaller model obtained by knowledge distillation. The experiments show that this type of structure can help LLMs find the right information needed to answer the requests, improving performance in question-answering tasks and reducing model hallucinations.", "questions": "The legend of Table 1 mentions that the prefix Struct- indicates the data fed into LLMs is structured by StruXGPT-7B. However, I don't see this prefix in the table.", "limitations": "Yes, the limitations were well addressed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed technique is useful and easy to implement. As hallucination is currently one of the main challenges faced by real-world systems that employ LLMs, any technique that reduces it should be considered.", "weaknesses": "In the abstract and introduction of the paper, there are claims about how human cognition works that are poorly backed up. This is an area of debate and I believe this discussion could be avoided without any harm to the main point of the paper.\n\nThe proposal assumes that the prompt transformation process is an easier task for an LLM than directly addressing the request. Although this seems to be the case for question-answering problems considered, there is no discussion about cases in which this may not be adequate. I believe more experiments with other types of datasets would help us to better understand that.\n\nFinally, as there are other papers investigating similar approaches, the novelty of the approach is not a strong point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer yxZb:\n\nWe thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:\n\n> **W1**: In the abstract and introduction of the paper, there are claims about how human cognition works that are poorly backed up. This is an area of debate and I believe this discussion could be avoided without any harm to the main point of the paper.\n\n**A**: Thanks for this suggestion. We will eliminate some debatable descriptions (e.g., \"In human cognition, sophisticated text sequences will be processed and consolidated into a structured knowledge tree, with factual elements well-organized hierarchically\").\n\n> **W2**: The proposal assumes that the prompt transformation process is an easier task for an LLM than directly addressing the request. Although this seems to be the case for question-answering problems considered, there is no discussion about cases in which this may not be adequate. I believe more experiments with other types of datasets would help us to better understand that.\n\n**A**: Thanks for this suggestion. Here we further evaluate our method on two commonly used multi-choice datasets (i.e., MMLU and BBH).\n\nThe MMLU benchmark is a typical scenario, where LLMs are asked to answer questions without context references but requiring world knowledge, such as `In 2016, about how many people in the United States were homeless?`. LLMs have to use their parametric knowledge (learned during large-scale pre-training) to find the answer, and context structurization does not help. If we insist on structurizing the question alone (such as `1. Number of homeless people: in 2016, about how many people in the United States were homeless?`), and feed it into LLM's inputs, the model may be disturbed by the repeated information (such as the `1. Number of homeless people:`) and generate wrong answers.\n\nTo quantify the results, we take LLaMA2-7B-Chat as the baseline model and feed the structurized question. The performance variation (measured by OpenCompass protocol) is reported below, which shows our method causes a slight 0.1% decrease on the MMLU benchmark.\n\n|Model|MMLU|BBH|\n|:-------|:---------:|:---------:|\n|LLaMA2-7B-Chat|**45.93**|30.47|\n|+Ours|45.84|**31.30**|\n\nOn the other hand, we have also tested another common benchmark, BBH, which is designed to evaluate LLMs' reasoning capability when dealing with several logical sentences/statements. Here is an example of structurized question from BBH:\n```\nThe finishing positions of seven golfers in a tournament:\n1. Golfers' names: The seven golfers were Ana, Eve, Ada, Dan, Rob, Amy, and Joe.\n2. Dan's finishing position: Dan finished third in the tournament.\n3. Ana's finishing position relative to Ada: Ana finished above Ada.\n...\n7. Rob's finishing position relative to Joe: Rob finished below Joe.\nWho finished first?\n```\nIn this case, our method can adapt well to highlight the logical relations (by abstractive hints like `Dan's finishing position:`) and boost LLM's reasoning abilities.\nThe enhancement on multi-choice datasets is consistent with that on question-answering datasets evaluated in our manuscript, demonstrating our method's generalizability.\n\nIn conclusion, we suggest users apply structurization to long-form or logically complex contexts, while taking the original question as inputs when there is no context provided.\n\n> **W3**: Finally, as there are other papers investigating similar approaches, the novelty of the approach is not a strong point.\n\n**A**: There have been recently concurrent papers utilizing structurization-like techniques to reduce LLM hallucinations through two major approaches: retrieval-augmented generation[1][2] and prompt-augmented generation[3][4], and our method presents at least two core improvements over the rivals:\n\n1. **Our method is proven effective across various NLP tasks**: All of those works are specifically designed for a single task ([1][2] for RAG and [3][4] for reasoning), while our method is verified to consistently enhance various LLMs across the question-answering (including multi-hop reasoning), hallucination evaluation, as well as knowledge retrieval tasks.\n2. **We presented an affordable and scalable StruXGPT-7B model for structurization**: The concurrent rivals usually adopt LLaMA3-70B/ChatGPT/GPT4 for prompt augmentation, which is neither inference-efficient nor cost-friendly. In contrast, our paper has proposed and validated an efficient and effective solution to train a StruXGPT-7B model for structurization, which is more affordable for deployment and may even exceed the giant teacher LLMs (e.g., with 0% format error in structurization.) We will open-source the model, data, and code soon to further facilitate the research in the LLM community.\n\nWe hope the discussion with related works can better establish the contribution and novelty of our paper.\n\n> **Q1**: The legend of Table 1 mentions that the prefix Struct- indicates the data fed into LLMs is structured by StruXGPT-7B. However, I don't see this prefix in the table.\n\n**A**: Thanks for pointing out that. It is a typo and we will fix it in our revised manuscript.\n\nWe hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.\n\nBest regards,\n\nAuthors\n\n---\n\n[1] Dong et al. Multi-view Content-aware Indexing for Long Document Retrieval. ArXiv'24.\n\n[2] Sarthi et al. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. ICLR'24.\n\n[3] Cheng et al. Information Re-Organization Improves Reasoning in Large Language Models. ArXiv'24.\n\n[4] Zhong et al. Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Perfect Reasoners. ArXiv'24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ej7vPvZ3H", "reviewer": "Reviewer_eQho", "summary": "The paper presents a method to improve the cognitive capabilities of large language models (LLMs) by organizing input context into a hierarchical structure. The method is called context structurization, and it involves transforming unordered contextual sentences into hierarchically structured elements to mimick human cognitive processes. The paper shows the effectiveness of the approach through evaluations on various NLP tasks, such as context-based question-answering, hallucination evaluation, and passage-level dense retrieval, using different model architectures and sizes. The study shows performance improvements and introduces StruXGPT-7B, a distilled model, to perform structurization efficiently.", "questions": "- How does the proposed three-layer structure handle texts with more complex relationships or multiple topics?\n- What are the computational costs associated with context structurization?", "limitations": "- The study heavily relies on specific models like LLaMA2 and Qwen, so i wonder about the generalizability of the results to other LLMs.\n- The structurization may be less effective working with extremely long contexts or when the context cannot be easily divided into clear hierarchical segments.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The study covers a wide range of tasks, providing a comprehensive assessment of the method's effectiveness.\n- I like the idea of distilling the structurization capability into a smaller model (StruXGPT-7B).\n- Even though the approch lack a bit of novelty it seems to work.", "weaknesses": "- The performance improvements are heavily dependent on the quality of the structurization process, so a poor structurization could lead to suboptimal model performance or even confusion.\n- Even though there is a performances improvement the process of structurizing context might introduce additional computational overhead. \n- The approach lack a bit of novelty, it doesn't introduce fundamentally new concepts and resembles methodologies from previous studies.\n- The study relies on metrics such as ROUGE-L and human evaluation, that are useful but they might not fully capture the more complex improvements, such as a cognitive-like processing that structurization aims to achieve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer eQho:\n\nWe thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:\n\n> **W1**: The performance improvements heavily depend on the quality of the structurization process. Poor structurization can lead to suboptimal model performance or confusion.\n\n**A**: Thanks for pointing out this.\nAs performance variation on specific tasks and input samples is a trivial problem, our aim is to build statistically consistent improvements across tasks, datasets, and models. \nDespite that, we also statistic the performance variance in the Qasper dataset when taking our structurized context as input. As reported below, our method merely causes degradation on 3.5% of samples (with relatively lower structurization quality), but ultimately receives a +3.6 improvement over all test samples.\n\n|Struct|Declined|Overall|\n|:--|:--:|:--:|\n|default|3.5%|+3.6|\n|filtered|**3.0%**|**+3.7**|\n\nIn addition, as BERTScore has showcased an efficient but reliable proxy measure of the structurization quality, we can filter out the structurization results with a low BERTScore (e.g., lower than 0.05) and take back the original context as input. In this way, the degradation can be alleviated (from 3.5% to 3.0%), further improving the final enhancement to +3.7.\n\nWe will add the experiments and discussions.\n\n> **W2/Q2**: While there is a performance improvement, the process of organizing context may lead to additional computational overhead.\n\n**A**: As discussed in Appendix C, the increased inference cost is a common limitation for methods using LLMs for augmentation., which depends on computing resources, acceleration techniques, and data patterns. In comparison with summary-based competitors, our experiments on the question-answering task in Table A5 show that our method significantly outperforms competitors at a comparable extra cost. To investigate the impact of model size, we use LLaMA2-7B/70B as baseline models for hallucination evaluation, with the results displayed here for convenient reference:\n\n|LLM-Evaluator|Enhancement|Total Cost|Extra Cost (%)|\n|:--|:--:|:--:|:--:|\n|LLaMA2-7B|-|1.4s|-|\n|+Ours|+6.4|3.5s|150%|\n|LLaMA2-70B|-|17.6s|-|\n|+Ours|+4.3|19.7s|12%|\n\nOur method showcases more efficiency (e.g., 12% extra cost in trade of +4.3 improvement) when facilitating larger baseline models.\n\n> **W3**: The approach lack a bit of novelty.\n\n**A**: There have been several concurrent papers utilizing structurization-like techniques to enhance LLMs through either retrieval- or prompt-augmented generation[1-4], and our method has two core improvements over the rivals:\n\n1. **Our method is proven effective across various NLP tasks**.\n2. **We presented an affordable and scalable StruXGPT-7B model for structurization**.\n\nDue to the space limit, please kindly refer to our response to Reviewer yxZb (R3)'s Weakness 3 (W3) for more details.\n\n> **W4**: The study relies on metrics such as ROUGE-L and human evaluation, that are useful but they might not fully capture the more complex improvements, such as a cognitive-like processing that structurization aims to achieve.\n\n**A**: If we misunderstand your concern, please feel free to correct us immediately. We assume that you are worrying about how to effectively evaluate the structurization quality of our StruXGPT model besides the ROUGE-L and human evaluation metrics. \nIn Table 4, we also provide AppEval (improvements on downstream applications, which is our goal to achieve) and BERTScore (semantical similarity for raw and structurized texts) as proxy measures, because they show a high degree of consistency with human evaluation. \nWe believe our systematical evaluation metrics can better capture the improvements of our structurization process.\n\n> **Q1/L2**: How does the proposed three-layer structure handle texts with more complex relationships or multiple topics? The structurization may be less effective working with extremely long contexts.\n\n**A**: Thanks for pointing out this. To avoid information loss for very long contexts (e.g., with 32K length), we automatically split the raw text into several chunks (e.g., using paragraph identifiers like `\\n\\n`), perform structurization on those chunks in parallel, and integrate the structurized chunks into one structure (by concatenating the aspects in those chunks) to capture the whole context. This is exactly our strategy when handling the context in the MuSiQue subset from the LongBench dataset. As for the text with more relations or multiple topics, we can adopt a similar strategy to construct the semantic structure in a bottom-up manner. And as discussed in our paper, we will continue to explore more flexible approaches (such as a MindMap) to capture complicated structures in our future work.\n\n> **L1**: The study heavily relies on specific models like LLaMA2 and Qwen, so I wonder about the generalizability of the results to other LLMs.\n\n**A**: If we misunderstand your concern, please feel free to correct us immediately. If you worry about the downstream generalizability, we have investigated LLaMA, Qwen, ChatGLM, as well as ChatGPT/GPT4 models across various tasks; as for our structurization model (StruXGPT), we have evaluated different architectures (LLaMA/Qwen) and sizes (1.8B/7B/14B in our new experiments).\nWe hope those experiments can help confirm our method's generalizability. \n\nWe hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.\n\nBest regards,\n\nAuthors\n\n---\n\n[1] Dong et al. Multi-view Content-aware Indexing for Long Document Retrieval. ArXiv'24.\n\n[2] Sarthi et al. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. ICLR'24.\n\n[3] Cheng et al. Information Re-Organization Improves Reasoning in Large Language Models. ArXiv'24.\n\n[4] Zhong et al. Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Perfect Reasoners. ArXiv'24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable time and constructive suggestions when evaluating our manuscript. We are really encouraged to see **ALL** reviewers find our method **technically solid**, **extensively validated**, and **well-presented**. 

We have provided point-to-point responses to reviewers' comments below, and here is a brief summary of the included experiments and explanations:

* **Comparisons with other augmentation approaches**. We have presented further comparisons with the recent AdaptLLM approach, and extended to incorporating the GPT-4 model as well as the CoT technique to demonstrate our method's efficacy and compatibility.

* **Ablations on training corpus and model size for StruXGPT**. We have additionally investigated the impact of StruXGPT's model size on structurization quality, and also quantify and guarantee the data quality for StruXGPT's training corpus with in-depth analysis.

* **Discussion with concurrent structurization works**. We have discussed several concurrent works to emphasize our novelty and contribution: effectiveness across various downstream models and tasks with a unified structurization, and the affordable and scalable StruXGPT model for this structurization.

* **Exploration of structurization's capability boundary**. We have supplemented extra evaluations on common benchmarks (such as MMLU and BBH) to further study the capability boundary of our method, so as to provide practical suggestions in real-world applications.

We believe reviewers' comments have made our paper much stronger, and we hope our work can further inspire the LLM community to a deeper study in model cognition and generalized artificial intelligence via structurization.

Meta-review: This paper introduces enhancing the comprehension capabilities of LLMs by structuring contextual information hierarchically. The authors propose a three-layer structure to summarize input text into a Scope-Aspect-Description format. They optimize a 7B model (StruXGPT) by distilling the structuring capabilities of LLMs to reduce computational costs. The method improves performance in some NLP tasks, including QA and passage-level dense retrieval, and the fine-tuned 7B model inherits most of the structuring capabilities of large LLMs. The aurhors findings also suggest that the structured prompt transformation reduces model hallucinations and improves performance in context-based QA benchmarks. 
There were several questions about the experiments, and strengthening of the findings through better ablations was asked. The authors have provided additional evidence by including different model sizes, and included additional evidence on common benchmarks (such as MMLU and BBH). Reviewers have asked comparison to different models, which authors have agreed to include in the final submission.
Decision: Accept (poster)