Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs
Authors: Zhuowen Liang
Xiaotian Lin
Zhengxuan Zhang
Yuyu Luo
Haixun Wang
Nan Tang
Venue: ICLR 2026 Poster (2026)
Keywords: Information Extraction
Document Analysis
Small Language Models
Reinforcement Learning
Primary area: interpretability and explainable AI

Abstract: Large language models (LLMs) are widely applied to data analytics over documents, yet direct reasoning over long, noisy documents remains brittle and error-prone. Hence, we study document question answering (QA) that consolidates dispersed evidence into a structured output (e.g., a table, graph, or chunks) to support reliable, verifiable QA. We propose a two-pillar framework, LiteCoST, to achieve both
high accuracy and low latency with small language models (SLMs). Pillar 1: Chain-of-Structured-Thought (CoST). We introduce a CoST template, a schema-aware instruction that guides a strong LLM to produce both a step-wise CoST trace and the corresponding structured output. The process induces a minimal structure, normalizes entities/units, aligns records, serializes the output, and verifies/refines it, yielding auditable supervision. Pillar 2: SLM fine-tuning. The compact models are trained on LLM-generated CoST data in two stages: Supervised Fine-Tuning for structural alignment, followed by Group Relative Policy Optimization (GRPO) incorporating triple rewards for answer/format quality and process consistency. By distilling structure-first behavior into SLMs, this approach achieves LLM-comparable quality on multi-domain long-document QA using 3B/7B SLMs, while delivering 2–4× lower latency than GPT-4o and DeepSeek-R1 (671B). The code is available at https://github.com/HKUSTDial/LiteCoST.

Published as a conference paper at ICLR 2026
LONG-DOCUMENT QA WITH CHANN-OF-STRUCTURED THOUGHT AND FINE-TUNED SLMS
Zhuowen Liang', Xiaotian Lin', Zhengxuan Zhang', Yuyu Luo', Haixun Wang', Nan Tang'*
The Hong Kong University of Science and Technology (Guangzhou), 2EvenUp, USA
ABSTRACT
Large language models (LLMs) are widely applied to data analytics over documents, yet direct reasoning over long, noisy documents remains brittle and error prone. Hence, we study document question answering (QA) that consolidates dispersed evidence into structured output (e.g., table, graph, or chunks) to support reliable, verifiable QA. We propose two-pillar framework, LITECOST. to achieve both high accuracy and low latency with small language models (SLMs). Pillar 1: (CoST), We introduce CoST template. schema- aware instruction that guides a strong LLM to produce both step-wise CoST trace and the corresponding structured output. The process induces minimal structure, normalizes entitics/units, aligns records, serializes the output, and verifies/refines it, yielding auditable supervision. Pillar 2: SLM fine-tuning. The compact models are trained on LLM -.ge-rgeeeteeed CoST data in two stages: Supervised Fine- Tuning for structural alignment, followed by Group Relative Policy Optimization (GRPO) incorporating triple rewards for answer/format quality and process con- sistency. By distilling structure-first behavior into SLMs, this approach achieves LLM- CoLparabeer quality on multi-domall long- document QA using 3B/7B SLMs, while delivering 2-4x lower latency than GPT and DeepSeek-FR R1 (671B). The code is available at htttps://gt Httpbb/com/HHISDiia/LLtteStS
INTRODUCTION
Zhu et al. 2025), yet direct reasoning over long documents is brittle and opaque, prone errors in high-stakes domains such as finance and legal (Chew et al., 2023 Qin et al., 2024 Edge et al.
Large language models (LLMs) are increasingly used for analytics (Chen et al.
2023
2024 Tang et al.) 2024ah. We therefore study cong-document QA where explicit structured data helps. In this regime, the system constructs query-specific structured data-e.g. e.g., table, graph, or chunks-fin frrsss-rommmo which final answer directly derivable with explicit explanations. As shown Fig. [ extracting structured data for long-document QA (Edge et al. 2024 Zhang et al. 2025 Li et al. 2025c improves reliability, interpretability, and reuse by exposing evidence and enabling routine verification.
We instantiate a query-conditioned pipeline: given natural question Q and documents D, the system induces a minimal schema tailored to Q. populates it with normalized evidence (e.g., units, entities, time), and serializes it into a structured output S. The answer A is then computed from S. Unlike fixed and pre-defined schemas, structures are assembled dynamically for each query, thereby excluding open-ended narrative questions that are not amenable structured representations.
natural idea to directly leverage powerful LLMS (e.g., GPT-4 or (eepSeck-RI) to emit the structured artifact. However, direct prompting is not ideal: (1) evidence is dispersed across long, multi- document contexts, leading to omissions or hallucinations; (2) values appear in heterogeneous units and formats, requiring normalization; and (3) long-context reasoning must remain consistent across the entire structure. As shown in Fig. a)a). direct prompting often yields brittle results-omissions hallucinations, and format drift (Wei et al. 2023) Wang et al., 2025). In contrast, Fig, b) illustrates our CoST template, which guides the LLM to produce both (i) schema- aaligned CoST trace and (ii) a query-specific serialized structured output (SSO) (e.g., table/graph/chunks), ensuring field completeness and format consistency, for robust and interpretable analytics over long-document QA.
"Corresponding author: Nan Tang (F-mail: nantang @ hhusttgz.eedu.c))

---

Published as a conference paper at ICLR 2026
(4) Information Extraction
(b) Structured knowledge- -based reasoning
Q
Endezwery,
Operiting saperos for 2013 and those lafe, de gn
Inc's profit margin
from
Drow Cppeeet
QL,
O
Endeowers Froe andfi exrgle fivm then secinad 2t packac 12 p45 in 2013 then to 180%
Dacamants:
Q1,T1
10-4 flasarid reports fown 2021-2004
ADM Undeavere the varkout ohiege sit fleedeede dikur wi spenching rg ceperaca o 250 150 in 2021.
Graph-91
We to know how thes eetel ssec s4 their rasrroendiae are und can
Q2,
QZ
In Abii Exceseens of departments, icod
ScoreIe
Rine
Q2.61
LLM
complctt preceds SED/we consiit cremimum of three steges.
(bes
firal outhyat?
Structured data makes QA more accurate and reliable. (a) From raw document D, we extract a table Tı for query Q1 and a graph G₁ for query Q2. (b) LLMs often fail when reasoning directly over unstructured text (Q1, D; Q2, D), but succeed with structured inputs (Q1, TT: Q2, G1).
Figure 1:
Seriatzed Structurei Output
User Query
Grea Elettic
Cempary
Tinal Navcamer Asals
4,77 33015
Year
2004
Which compony hac the lnngee "ahd Nam curvent Asdeet 2025-20240
Your tesk is Long obcurment QuA Please extrect afr structened kodge from the decummnn fully orower question.
(a) Direct Prompting
rmad Yech
2004
EC43
Vil
Shan by Ag
2014
Documents
neew tlot The rerel san corvert
2004
Astwar The lorget curpary
Shonley Agt with 7:133
16 317.7555 375 2034.
wdett i 4. 510.55 and 2283..
Strutture Amblyrs
CoST
Seridisee
ditiniy Agr Stateremt cosp,, 2003) usate 7,37 200, Otfer
seart.
Aligs entrica with so coherra
Senily dl schemo reloort ont
LLM
Shiture
EaCippy Tueee nour
Company
8129 445
oumer
2024
2004
reperted bslunce exscte (i trasandd) of December, 2008 luocad sed) careent adsets
Vertyy consictency and retain for mons.
Ki 2 211,00 Nih.
Ayn
7,733 389
pww.cunolloot (590)
2009
280 /02
2024
Input
(b) Chain-of- SoffSStuctured-Teeeee Thought
(Template)
Deery Aha dev Ra00000 2006
Dract Alggear?
Hase Deangy
Correct Pormal
4,321 poo
2024
lingot campany
Figure 2: (a) Direct prompting LLMs often causes hallucinations and format errors. (b) (uestionn, Document, CoST Template) = LLM + (CoST Trace SSO), yielding verifiable and auditable QA.
While CoST prompts with strong LLMs can yield accurate, verifiable SSOs, this effectiveness comes with substantial cost: repeated large- mnge-model calls increase token/compute budgets, add latency, and limit throughput-undesirabb undesirable for practical deployments that require low-latency, high-throughput service (Li et al. 2024 Xu al. 2024). Reliance hosted LLM APIS can also introduce privacy concerns for sensitive data. A natural response is to adopt small language models (SLMs []]]]]] for on-premises (on-prem) deployment, enabling cost-efficient inference in local or private environments; however, off-the-shelf SLMs struggle with the very skills CoST demands, including schema- extraction across long contexts, unit/entity normalization, record alignment, and step-consistent serialization, making naive LLM-+SLM substitution ineffective (Tang et al. 2024b Li al. 2025b a).
To balance effectiveness with efficiency, we introduce LITECOST, a two-pillar framework that equips SLMs with strong QA-by-structuring capabilities. Pillar 1 invokes powerful LLM once as a structure-firs trace generator: proposes a concise, query-conditioned schema and produces an auditable CoST trace together with structured data that makes evidence and formats explicit. Pillar 2 transfers this ability to an SLM via a lightweight adaptation pipeline: supervised fine-tuning (SFT) to instill structure, format, and step discipline, followed by group-relative policy optimization (GRPO) that jointly rewards answer quality and process consistency.
Contributions. We propose LITECoST with three notable contributions:
CoST for Da-by--Structuring: a structure-firs prompting paradigm that leverages LLMs to elicit step-wise, schema-guide CoST traces and SSOs from long, noisy documents-yyildinng auditable supervision and machine-checkable outputs.
2. SLM adaptation via structured dual signals: two-phase SFT SFT- recipe that introduces a novel dual-level reward, encompassing structured output quality as well as process consistency, to instill CoST-style, schema-aware structured reasoning into compact models.
3. Empirical validation: We have evaluated our approach across multiple domains, including finance, legal, and scientific literature. On the financial subset of the Loong benchmark (Wang) let al. 2024), our CoST recipe substantially improves small models: LLaMA-3B gains +27.6 accuracy points and +0.29 perfect rate (PR), while Qwen-7B gains +17.8 accuracy and +0.22 PR. with the 7B model slightly surpassing GPT-4o. Inference is 2-4x x faster than
We use small language models (SL to denote compact models (e.g., 3B-7B).
2

---

Published as a conference paper at ICLR 2026
GPT-4o/DeepSeek-R.. Extensive experiments further show that LITECOST framework delivers significant improvements while showcasing excellent generalization capabilities.
PRELIMINARY AND PROBLEM FORMULATION
Bong-Document QA as D--by-Structtrring We study practically important regime of long- document QA in which the system turns a question Q and a collection of long, noisy, multi-source documents D into a compact, query- serialized structured output (SSO) with provenance. Structure becomes the interface: the system first induces a minimal schema tailored to Q, populates with normalized, aligned evidence, and then derives the final answer from the structure with explicit support. This framing mitigates noise and dispersion, improves interpretability and reuse, and foregrounds four desiderata: accuracy (correct answers). faithfulness (evidence-groundee grounded), auditability (verifiable traces), and efficiency (bounded compute/token cost).
(CoST), our design, the CoST template the input language model: a schema-aware instruction that specifies step-wise, structure-fisy requirements. When exe- cuted, the language model produces two complementary outputs: CoST trace, i.e., the auditable, step-wise reasoning record that documents schema selection, evidence alignment, normalization, and verification; and (ii) serialized structured output (SSO), i.., the machine-checkable artifact (table, graph, list, or record set) linked with provenance to the source documents. This input-output separation ensures that the LLM's role is well-defined: given a CoST template, it must emit both reasoning trace and structured output, enabling supervision, verification, and reuse. The whole CoST procedure consists of four key steps: (A1) structure analysis, (A2) trace generation, (A3) quality verification, and (A4) iterative refinement (see Sec. 3.1 3.1
more details).
Two Research Goals. Our research goals are as follows.
(G1) Accurate and verifiable QA. Obtain high-quality CoST traces and SSOs iee. schema-complete, format-consistent, and serreeanncegrounndee outputs-from which we can compute correct answers.
(G2) Low latency via SLMs. Achieve CoST-style reasoning SLM speeds. While strong LLMs are effective CoST generators, their latency/cost hinder deployment. Our objective is to transfer this structure-first behavior compact models (SLMs) through fine-tuning, so that SLM-generated structures SSLM are as useful for answering as their LLM counterparts SIIM, at much lower latency:
LL,
LLM(Q, STLM) and Latency(Sst.)
Latency(SSIM)
(1)
Operationally. we will (i) use LLMs once to generate high-quality CoST traces/SSOs (Pillar 1) and (ii) fine-tune SLMs to internalize schema/format/step discipline and process consistency (Pillar 2). enabling accurate, auditable QA at low cost.
3 LITECOST: FROM LLM COST GENERATION TO SLM ADAPTATION
Next, we present LITECOST (see Fig. B two-stage framework designed to achieve the dual goals in Sec. 2 (G1) accurate and verifiable QA through high-quality CoST traces and SSOs, and (G2) latenat execution via compact SLMs. Stage strong LLM executes the CoST template as input and produces auditable CoST traces and
SSOs outputs. These outputs serve as supervision signals that capture schema, normalization, alignment, and verification. In Stage B, transfer this structure-first reasoning behavior into an SLM through lightweight two-phase recipe: supervised fine-tuning (SFT) for schema/format/step compliance, followed by group-relative policy optimization (GRPO) jointly reward answer quality and process consistency.
3.1 STAGE A (G1): CoST (STRUCTURE-FIRS) REASONING AND TRACE GENERATION)
As illustrated in Fig. operationalize CoST as a structure- input procedure: given a question, document, the ground truth answer, and CoST template, a strong LLM generates two outputs auditable CoST trace and serialized structured output SSO.
(A1) Structure Analysis. The first step is to dynamically select the most suitable data structure and instantiate accurate schema to support answering given question. Specifically, LITECoST incorporates a cuesttion-oieettee structure selection mechanism that, for example, chooses tables for statistical comparison graphs for relational reasoning, without exhaustively processing the entire
3

---

Published as a conference paper at ICLR 2026
Stage A: CoST: Structure-Firss Reasoning and Trace Generation
Input
Quality Verification
Output
Question
Guided Instraction: Generate table from adhere To (schums) give the extraction steps
Documents
CoST
Ground Truth
COST Template
REPATT
Trath
Yes
Trace
No
Let's think step by step,
Seridlized
Table Chunks
Graph
First, entitt extraction:
Attribute A Entity Al, A2
Attribate Entity 31, B2
Structured
(SSPu
Second antity algnment:
Iterative Structurlizer
Bl...) Third, sumnerization:
Schhmo]
(Attmibuts A 0.C
Grean Correct Redi Whong
A1. Structure Analysis
A2. Trace Generation
A4, Iterative Refinement
DE
Complex Reasoning
Policy Oprinization
Preference Algement
Process Reward
PPO
Outcome Reward
General-Purpose SLM
SFT Training
GRPO Training
LiteCoST-Mode
Stage B: SLM F SFT
GRPO
Figure 3: Overview of LITECOST, containing two stages: (1) CoST: Structure-First Reasoning and Trace Generation through structure analysis, trace generation, quality verification, and iterative refine- ment; and (2) SLM Fine- Tunin: SFT - GRPO process, including SFT for structure/formattsten followed by GRPO with dual signals for answer/format quality and process consistency.
corpus. Once the structure type is chosen, we invoke a dynamic schema construction procedure in which the LLM parses the question and enumerates task-specific attributeseenittes (e.g.. Company, Asset, Year), ensuring precise alignment with the question semantics.
(A2) CoST Trace Generation. Following structure analysis, we adopt an instruction-based chain-of- thought paradigm that performs step-by-step reasoning to progressively generate the trace guide schema-aligned extraction. The task is specified through three key components: 1) task description, template specifying step-wise requirements; 2) input text, the source documents; and 3) the dynamically generated schema. Guided by schema-ing informed instructions, strong LLM extracts, aligns, and serializes into a deterministic structured format, emitting both the reasoning trace and the final structured output. The template of trace generation is provided in Appendix A.2
(A3) Quality Verification. The module aims to assess the quality of the generated structured data by evaluating its ability to answer the original question. Since ground-truth structured data is unavailable, we adopt an LLM-assJudge approach (Zheng et al., (023), where a strong L.M evaluator (e.g., GPT- 40) assesses the extracted responses. Inference outputs that exactly match the reference answers are deemed correct and retained for subsequent training. Further details are provided in Appendix A.3
(A4) Iterative Refinement. At its core, the module employs Iterative Structuralizer that refines low-quality samples by regenerating structured knowledge for GRPO training. Rather than discarding flawed but challenging cases, it reuses them recursively with the question and context, reframing the task as supplemental extraction and providing richer supervision than vanilla fine-tuning. The iterative update rule, sufficiency evaluator, and stopping criteria are detailed in Appendix A.4
Final Output. After the CoST pipeline, the final output is (c", ("), where c* is the CoST trace (generated in A2), and S* the structured output refined through quality verification (A3) and iterative refinement (A4). This pair provides high-quality supervision for training and downstream reasoning.
3.2 STAGE B (G2): SLM FINE- TUNING (SFT GRPO)
Given the supervised training data, LITECOST first warms up the model with Supervised Fine- Line-Tuning (SFT). We then apply reinforcement learning with Group Relative Policy Optimization (GRPO), introducing dual-level reward that jointly optimizes (1) outcome reward, which evaluates the format compliance and answer correctness, and (2) process reward, which scores step-wise reasoning against ground-truth evidence to enforce a reliable extraction path.
Training Data Template. Each training sample is defined as (i, d, c², ,²,,","), where is the question, the document input, c* the CoST reasoning trace (enclosed by <reasoning>.. </reasoning?), and y the structured output (enclosed by <answer>... ... </answer>). In SFT, the model learns to generate (c*, T from (i, d), while GRPO also conditions on (i, d) and optimizes with dual-level rewards (process, outcome) against the verified targets.
4

---

Published as a conference paper at ICLR 2026
GRPO
KL Reference Model
O
A
Policy Model
O
Reward Model
Group Computation
A
Question
3
(+) Format Reward
(b) Answer Reward
(c) Process Reward
Soft Format + 0.5
Prefie
NoW
699
572
2018
creasoning </recsonings conswers
Wie
763,075
2018
</answer>
2019
NFA
562
362
2500
Trojentory prajuctery 2 pectory
Hord Format + 1.0
WGW
Ground Truth
582,082
099 677
2019
Eyh RAcerlon Attribatel
Entinyp Ralat Astribte
Bat Reieren
Step lt..Step2...Stepp
Answer
2025
Attriste
Figure 4: The GRPO training pipeline based on dual-level reward.
Supervised Fine-tuning (SFT). We initially performed Supervised Fine- Fine-Tuning (SFT) on a general- purpose base model, specifically enhancing its capability for CoT-driven information extraction. This process enables the model to acquire fundamental extraction capability (e.g., handling structure, format, and step-wise reasoning) in a specific domain, thus substantially mitigating the errors observed when deploying the base model complex extraction tasks.
Group Relative Policy Optimization (GRPO). We then employ GRPO via a three-level reward mechanism, as illustrated in Fig. 4
Formulation. For each question q, GRPO samples a group of outputs {0r, 02...., oG) from the old policy my. Each output receives reward rı, yielding of G rewards {ri, r., r. ). From these rewards, we compute the group-relative advantage A; = Ti (x-mean(r....r)) and then optimizes the policy model by maximizing the following objective:
std(ri..-re)
(G)PO(0) = E [v P(V), forl Thaa (Olv)
(min (r.atio A;, clip (reatio, 1 e, 1 + e) A.) DRL (mol(
(2)
where 3 and hyper-parameters, ratioo is the importance sampling ratio comparing the likelihood of output under the new and old policies, and A; is the group-relative advantage. The clipping operator would stabilize updates within trust region, and the minimum operation ensures conservative yet
effective policy updates (Shao el al. 2024). Further details are provided in Appendix B.I
Formal Compliance. Fig. 4 (a) illustrates the hierarchical design of the format reward: soft reward (0.5) is given if the output contains a single reasoning sequence in <reasoning ing? and a final answer in <answer> without extrancous content; hard reward (1.0) assigned the reasoning further structured with explicit step labels (e.g., Step 1, Step 2); otherwise, the score is 0.
Answer Correctness. As illustrated in Fig. bb), address the limitations of rule-based evaluation by adopting hybrid metric that combines structural alignment and semantic similarity:
fscore S (1 ol) Ssom
For Sauuet. we use rule-based checks (e.g., row-column alignment in tables) to verify structural correctness, For Syem. we adopt GPT-4o-mini an automatic evaluator, comparing the content within <answer>... </answer> tags against the reference; outputs with higher semantic similarity receive higher rewards. The raw score fscore is scaled from jo, 100] to [o. 1). with NULL rewards assigned to empty outputs. The detailed LLM-based evaluation prompts are provided in Appendix B.2
Process Reward. Outcome rewards alone are sparse and insufficient for fine-grained guidance. We therefore introduce consistency-basee process reward to supervise reasoning at the step level. Consistency is evaluated from both the entity-level and the tuple-level, enabling the model to capture fine fine-grained errors such as partially incorrect entities or mismatched relations. For each step i, LLM is prompted with an instruction Jconsistency to judge whether the predicted step result 8; is consistent with the corresponding ground truth s² If the consistency holds, the step is assigned score of otherwise, it receives 0. The overall process reward is formally defined as:
N
Aprocess (8:)
1 [Cons(st s7 Iconsistency)]]
(4)
N
5

---

Published as a conference paper at ICLR 2026
where NN denotes the total number of reasoning steps, Cons(-) is the LLM-based consisteney function, and [[: is the indicator function that returns if the consistency check is satisfied and 0 otherwise. This formulation provides dense and fine-gres ane-grained training signal, guiding the model towards faithful step-by-step extraction while complementing the sparse outcome reward, as shown in Fig. (c).
Overall Reward. The overall reward defined as the sum of the format compliance, answer correct- ness, and process rewards. To prevent training dynamics from being dominated by other reward signals, we introduce a scaling factor that modulates the process reward along each trajectory, Rprocess (s,) = (x)) (T.)) Here, y(T)) trajectory-level coefficient: positive for correct answers to reinforce reasoning, negative for incorrect or overthought trajectories to discourage such behaviors, and 1 for format errors to isolate penalties to specific steps.
EXPERIMENTS ON LONG-DOCUMENT QA WITH CoST AND SLMS
In this section, we evaluate the performance of our proposed LITECOST framework on the Loong benchmark (Wang al. 2024), which effectively captures the challenges of generating serialized structured output (SSO) across varying context lengths. Rather than directly answering questions, we focus on the ability of LITECOST to produce reliable SSO that supports long-document QA. We further assess its efficiency and conduct ablation studies to analyze contributing factors. Specifically, we aim to address the following research questions:
(1) Benefits of Structured Data: How do structured outputs enhance long-document QA?
(2) Effectiveness: How effective is LITECOST in generating high-quality SSO for long-document
QA, compared with current LLMs and state-of-the-arl methods?
(3) Efficiency: How efficient LITECOST relative to LLMs in terms of SSO generation speed?
(4) Ablation Study: What factors contribute to performance gains in structured output generation?
(5) Generalization: How well does the framework generalize to other datasets and domains?
4.1 EXPERIMENTAL SETUP
Training Dataset. To support two-phase training, we construct two domain-specific datasets via
LITECOST from four large-scale multi-task resources: FINQA Chen al. 202T), TAT- QA (Zhu et al. 2021), SQUAD (Rajpurkar et al. 2016), and LEGALBENCH (Pipitone & Alami 2024). The datasets target the finance, legal and general knowledge, capturing diverse reasoning patterns (e.g.. aggregation, comparison, multi-hop inference) in realistic settings drawn from diverse documents.
Evaluation Dataset. We adopt the Loong benchmark (Wang et al. 2024). a real-world multi- document QA dataset with 1,600 test samples spanning three domains (Finance, Legal, Paper), four task categories (Spotlight Locating, Comparison, Clustering, Chain of Reasoning), and four document length settings where longer contexts disperse relevant information. Our analysis focuses primarily on an in-depth analysis of the finance domain, with legal results provided in Appendix E.
Evaluation Details. Defining ground truth for the structured output from long- congecontext documents, poses significant challenges. To address this, we adopt a 2-hop evaluation paradigm by leveraging downstream QA tasks (Jain al., 2024). Specifically, we employ the GPT-4o an automatic judge, scoring model responses from 0 to 100 based on accuracy, hallucination, and completeness. It also introduces the Perfect Rate, which measures the proportion of responses that achieve a perfect score.
Baselines. To comprehensively evaluate the generation capability of LITECOST, we compare the performance gains achieved through both LLMs and SLMs. Specifically, we consider two categories of baselines: the first targets reasoning, where LLMs are prompted to generate structured output (e.g., Zero-shot and Chain-of-Thhughhe (CoT) (Wei al. ). ). The second focuses on improvements with SLMs, where we compare against several state-of of the-art models, including Llama3.2 B- Instruct, Meven2-7B-Instruune and DeepSeek-R1 1 We further include two categories of baselines: (1) Fine-tuned IE models, such as ODIE (Jiao et al., 2023), IEiile (Gui et al. 2024b), and Struc-bench (Tang et al. 2024b); and (2) Modular extraction frameworks that leverage component modules to extract structured knowledge, including StruetRAG (Li et al. 2025c). For a fair comparison, we evaluate the baseline methods using the same backbones (i.e., LLaMA-3.2-3BB 3.MA--3.23B-Instrucc. Instrruct, Owen2-7B-Instee our LITECOST. These baselines act as structured output generator, with GPT-4o employed as the reasoning model to produce responses. More details provided in Appendix C.2
Llama-3.1 I- Qwen2.5-144BInnsee
GPT40-mini, GPT-4o,
as those used for

---

Published as a conference paper at ICLR 2026
Table I: Comparison of different models generating structured outputs for long-document QA on the
Finance Subset of Loong. Green highlights the remarkable improvements over the base model.
Model
Spotlight Locating
Comparison
Clustering
Chain of Reasoning
Overall
Model
Size
AS
PR
AS
PR
AS
PR
AS
PR
AS
PR
Close Sonucee Models & Large Language Models
LLaMA-3.1-8B 11..1-8---nssruse
8B
55.03
0.20
51.60
0.15
$1.50
0.04
44.75
0.02
$1.32
0.10
GPT-4o-mini
8B
84,42
0.70
80.40
0.67
77.38
0.40
65.35
78.08
0.51
Swen2.5-14B-Instruuc GPT-4o (Abacha al. 2025
14B
83,74
0.57
82.12
0.56
69.96
0.24
66.41
0.10
75.60
0.38
200B
84.10
0.73
80.53
0.60
81.50
0.50
64.30
0.25
79.32
0.54
DecpsceK-RI (Guo et al 2025
671B
84.27
0.62
78.97
0.55
75.42
0.34
74.40
0.35
78.18
0.46
LLaMA-332-3B-nnttuu Instrcctt (Base)
3B
49.90
0.16
52.10
0.14
47.89
0.07
46.85
0.06
49.37
0.11
LLaMA-3.2-3B-Imm Instruct (Ours)
3B
81.27
0,53
78.08
0.49
78,34
0.36
64.75
0.16
76.95
0.40
131.37
10.37
125.98
10:35
130.45
10.29
117.90
10.10
127.58
10.29
Qwen2-7B- Ween2-7B-Instrue (Base)
7B
63.10
0.36
67.85
0.37
60.83
0.18
52.25
0.09
62.10
0.26
Qwen2-7B-Instie (Ours)
7B
83,97
0.62
81.55
0.59
81,00
0,43
67.98
0.18
79.93
0.48
120.87
10.26
133.70
10:23
120.17
10.25
115.73
10.09
117.83
10.22
65
LLM Score
LLM Score
w/
60A576755R POPRE
su
CP
10
+400))
Butte
532
CUR
CBR
40.4
010000
0.15
024
023
(a) Qwen2.5 Qwen2255148--- 114--Inruut
(b) GPT-4o
Querr
Figure 5: Radar plot of detailed scores for Figure 6: Quality assessment of CoST-generated struc- different prompting methods on 4 subtasks tured data via reasoning performance on Loong across popular LLMs (SD denotes structured data).
the Finance subset of Loong.
Implementation Details. During the training phase, LITECOST employs a two-phase training pipeline comprising LoRA fine-tuning followed by GRPO optimization using trl and verl. The model is trained in two stages: first, fine-tuning for 3 epochs with a learning rate of 2e-4 and batch size of 16 using a LoRA adapter (rank 16, lora alpha 32); followed by reinforcement learning with GRPO using a learning rate of le-5, batch size of 16, and 5 sampled generations per query. In Equation 3 the weighting parameter set to 0.3; the training cost about $20, and the maximum generation length is extended to 2,048 tokens to support CoTT-style reasoning.
4.2 THE BENEFITS OF STRUCTURED DATA
This experiment evaluates how structured data improves model performance on thowledge-intensse reasoning, emphasizing the need for accurate serialized structured output (SSO). The data is curated by LITECOST using GPT-4o as the base model; structure distributions are detailed in Appendix C.5
High-quality SSO from CoST improves LLM Reasoning. As shown in Fig. 6 all models achieve stronger reasoning when leveraging the structured data rather than raw long documents. With LITECOST, overall scores rise by 12.41, 8.77, 9.04, and 8.47 points, and perfect rates improve by +0.10, +0.05, +0.04, and +0.08 for Qwen2-72B Insstruutt GPT-4o-mini, GPT-4o, and Claude-3.5- Sonnet, respectively. These consistent gains highlight the value of high-quality structured knowledge in improving both accuracy and reliability. Full results are provided in Appendix C.5
4.3 EFFECTIVENESS: How GOOD LITECOST FOR SSO GENERATION?
In this section, evaluate the effectiveness of LITECOST across both LLMs and compact SLMs on the Finance subset of Loong, compared with state-of-the-art models and bascline methods. The in-depth analysis demonstrate that LITECOST consistently outperforms other comparable strategies, achieving substantial gains in correctness, and Sec. further confirms its broader generalization.
Efficacy of Stain-of-Structured. Thought. Fig. ppresents radar chart comparing the performance of different prompting methods across four task categories, evaluated on two backbone LLMs. The results show that step-wise reasoning substantially improves SSO generation and yields high-quality
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance of the Finance subset of Loong compared with other state-of-ther art methods
Spotlight Localing
Comparison
Clastering
Chain of Reasoning
Overall
Backbone
Method
AS
PR
AS
PR
A$
PR
AS
PR
AS
PR
ODIE
Jiao ct ml
2023
68.89
0.39
61.30
0.30
61.11
0.15
49.75
0.07
61.21
0.23
lEpdee
0.37
65.10
63.12
0.14
0.02
61.90
0.22
LLaMA-3.23B-Inn
Struc-beach Lubg et 2024bb
55.13
0.15
51.05
0.15
48.16
0.06
44.10
0.08
49.90
0.11
StrucRAGG [[ ( al. 20250
39.50
0.01
39.70
0.02
31.08
0.00
35.95
0.00
36.04
0.01
LITECOST (Ours)
81.27
0.53
78.08
0.49
78.34
0.36
64.75
0.16
0.40 76.95
ODIE O et 2023
82.13
0.59
73.85
0.48
73.54
0.32
55,30
0.12
72.86
0.40
|Epile Quu
qra 202410
71.83
0.45
72.60
0.46
70.86
0.29
54.20
0.11
69.19 0.35
Qwen2-7B-Ins
Struc-hench
202.4b
81.60
0.63
74.90
0.53
73.40
0.36
60.35
0.19
73.72
0.44
StructRAG @ al. 20256
48.83
0.03
46. 80
0.02
55.34
0.06
42.55
0.00
49.68
0.03
LITECUST (Owis)
0.62
81.00
0.18
79.93
supervision signals, with CoT consistently outperforming Zero-Shot, especially on Qwen2.5-144
Instruct. Beyond this, our structured prompting paradigm (CoST) yields the strongest improvements, consistently reaching the outer boundary (green line) and achieving top performance across nearly all tasks, notably in Chain of Reasoning (GPT-40), Clustering (wven..5-14BInng and Spotlight Locating, underscoring consistent effectiveness. See full numerical results in the Appendix C.3
that our model consistently outperforms all evaluated small language models (defined as open-source models with fewer than 8 billion parameters). Both fine-tuned variants, LLaMA-LiteCoST and improvements over their base models, with gains of (+27.58, +0.29) and (+17.83, +0.22), and consistent enhancements across all sub-tasks. Compared with other small scales, the LLaMA- LiteCoST significantly outperforms the 7B, and 8B models by (+14.85, +0.14) and (+25.63, +0.30) in overall score and perfect rate, respectively, while Qwen-LiteCoST delivers even larger improvements, underscoring the effectiveness our approach.
Our fine-tuned SLMs >> other SLMs.
achieve substantial
Our fine-tuned SLMs LLMs. On one hand, both variants surpass @wen-14BBInntrrue despite having far fewer parameters, with improvements of (+1.35, +0.02) on the LLaMA backbone and (+4.33, +0.10) on the Qwen backbone. On the other hand, Qwen-LiteCoST achieves the best overall performance among all evaluated models, surpassing all small baselines and even exceeding GPT-4o- mini by 1.85, Deepseek-RR R1 by 1.75, and GPT-4o by 0.61. It achieves top-2 performance on 5 out of 8 evaluation points across the four subtasks, highlighting the effectiveness of our training strategy in narrowing the gap between lightweight and large-scale models for structured output generation. Full results across different document sizes are provided in Appendix
RL-enhanced gains for SLMs. Compared with other state-of-the-a methods, including fine-tuned models and modular extraction frameworks, LITECOST achieves superior performance across all tasks, as shown in Table particular, LITECOST substantially outperforms the strong baseline StructRAG, with gains of (+30.91, +0.39) on LLaMA and (+30.47, +0.46) on Qwen, Moreover, relative to the previous best fine-tuned methods, sets new state of the art, achieving improvements of (+15.05, +0.18) 18) over IEPile on I.LaMA and (+6.41, +0.05) over Struchench on Qwen. These results indicate that our RL-enhanced framework provides a principled advancement over conventional fine-tuning for information extraction, further underscoring its effectiveness and robustness.
4.4 EFFICIENCY: HOW FAST ARE SLMS COMPARED WITH LLMS FOR SSO GENERATION?
SLMs are much faster than LLMs. The la- tency comparison in Fig. 7 demonstrates that our model offers an optimal trade-off between ac- curacy and efficiency in structured output gen- eration tasks, measured as the average time per sample on the Loong dataset. OWTT attains lower latency (12.09s) than LLaMA3.. 1- 8B-Instruct (13.19s) and Qwwn2.5-14BB-Innn (14.71s), while delivering substantial accuracy gains. Notably, maintains latency comparable to its base model (Qwen2-7B-Instruee at 11.89s),
Latency (0thers) Latency (Ours) Acccuracy
Accurssy
21.1
7.8
11.9
13.2
14.7
12.1
LLaMAA38 Queer-78 LB Qeeci110
GF1-4
b5-2t Quen
Comparison of extraction time and accuracy across different scale models.
Figure 7:
while running nearly 2x faster than GPT-4o (21.15s) and 4x faster than DeepSeek-RI (44.44s),
without relying on proprietary APIs. For scenarios requiring faster extraction, LLaMA-LiteCSSTi is preferable, running in just 8.04s while achieving performance comparable to 14B-scale models.
8

---

Published as a conference paper at ICLR 2026
Table 3: Effect of different reward designs in ablation study on the Finance subset of Loong.
Spotlight Locating
Comparison
Clustering
Chain of Reasoning
Overall
Model
AS
PR
AS
PR
AS
PR
AS
PR
AS
PR
LLaMA-Ours
81.27
0.53
78.08
0.49
78.34
0.36
64.75
0.16
76,95
0.40
w/o Process Reward
79.10
0.48
77.95
0.47
76.03
0.30
63.98
0.13
75.52
0.37
w/o Outcome Reward
76.07
0.45
73.10
0.41
71.72
0.23
68.22
0.15
72.55
0.32
Owen-Ours
83.97
0.62
81.55
0.59
81.00
0.43
67.98
0.18
79.93
0.48
w/o Process Reward
87.93
0.63
77.95
0.55
77.32
0.42
60.60
0.15
77.39
0,46
w/o Outcome Reward
86.29
0.57
75.65
0.45
73.56
0.31
63.35
0.18
75.43
0.39
Table 4: Performance Comparison on the Legal subset of Loong. Green highlights the remarkable improvements over the base model, while Red indicates relative drops.
Model Spotlight Locating
Comparison
Clustering
Chain of Reasoning
Overall
Model
Size
AS
PR
AS
PR
PR
AS
PR
PR
SoseeSo Models & Large Language Models
GPT-44-mini Qwen2.5-14B-Insiree
8B
46.55
0.10
28.05
0.00
48.68
0.13
42.56
0.11
0.09
14B
48.45
0.08
21.90
0.01
57.31
0.16
26.74
0.02
37.51
0.06
GPT-4o
200B
50.05
0.06
27.00
0.01
61.16
0.14
33.11
0.03
42,06
0.06
LLaMA-3.2-30B-nnstnn (Base)
3B
41.00
0.08
25.10
0.01
31,74
0.02
27.29
0.01
30.67
0.03
-LaMA-332-B--nsttruu (Ours)
3B
62.20
0.30
45,20
0.02
45,00
0.09
36.55
0.02
45.45
0.09
121.20
10.22
120.10
10.01
113:26
10:07
19.26
10.01
114.78
10.06
Qwen2-7B- ---nstuctt (Base)
7B
37.90
0.05
18.90
57.85
0.18
35,44
0.08
38.05
0.08
Qwen2-7B-Instnun (Owrs)
7B
52.85
0.12
31,00
0.00
60.37
0.22
37,88
0.06
44.94
0.10
114.95
10.07
112.10
10.00
12.52
10.04
12:44
10.02
16.89
10.02
4.5 ABLATION STUDY
Effect of Different Reward. To identify the key drivers of LITECOST 's performance, we ablated reinforcement learning configurations (Table 3). When we remove the process reward component from the complete model, performance drops by 1.43 points on the L.LaMA backbone 77.955+75.5 and 2.54 points on Qwen (79.93-77.39).. This demonstrates that fine-grained process rewards effectively guide step-wise extraction, complementing outcome-based signals to yield stronger overall performance. Similarly, excluding the outcome reward causes much larger drop (-4.40 on LLaMA, .4500 on Qwen), underscoring its importance for answer correctness. These results highlight the synergistic effects of process- and outcome-level supervision, yielding more robust training strategy. Case studies in Appendix D further illustrate how RI RL-enhancement improves extraction quality.
4.6 GENERALIZATION
To further demonstrate the broad applicability of LITECOST beyond financial analysis, we extend our evaluation to two additional distinct domains: Legal and Scientific Question Answering.
Legal Domain. We evaluate performance on Loong legal subset (Wang et al. 2024 to assess the capability of LITECOST in generating serialized structured outputs within complex legal contexts. As shown in Table our LITECOST-tunel 3B and 7B models substantially outperform their base versions, achieving Average Score and Perfect Rate gains of (+14.78, +0.06) and (+6.89, +0.02), respectively. Remarkably, both compact variants surpass significantly larger models despite having far fewer parameters. Specifically, the LLaMA- model outperforms Qwen2.5- 1iren2.5-14B-B11 IB-ttsttuct GPT- 4o-mini, and GPT-4o by (+7.94, +0.03), (+3.51, +0.00), and (+3.39, +0.03), respectively, with the Owen-based variant demonstrating similar superiority. More details are discussed in Appendix E.
Results on Open-domain QA. addition to the Loong benchmark, we verify the performance of CoST and LITECOST on LongBench (Bai et al., 2024). a comprehensive benchmark tailored for multi-task long-document QA that covers key long-text application scenarios. Our analysis focuses on both single-document and multi-document QA tasks across four datasets. We compare LITECOST against state-of-the -ftt---rr----ttttt- models using the standard Fl score, which assesses the quality of reasoning results derived from the extracted information. Consistent with the experimental setup in Loong, GPT-4o is employed as the reasoning agent to generate final answers based on the structured outputs.
As presented in Table 5 the results align with findings in other domains, highlighting two consistent insights: (1) CoST enhances LLM reasoning (Table5a): applying CoST consistently boosts performance for both Qwen2.5-14p) Instruct and GPT-4o across all datasets, yielding F1

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison on LongBench benchmark
(a) Quality assessment of CoST-generated struc- tured data reasoning performance on Long- Bench across LLMs (SD denotes structured data).
(b) Performance Comparison Between LiteCoST-Tuned
Models and LL--Based Baselines on LongBench.
Single-doc
Multi-docs
IE Model
Single-doc
Multi-docs
NarQA
Qasper
HotpotQA
2Wiki
Model
NarQA
Qasper
HotpotQA
2Wiki
LLaMA-3,2-38
16.94
34,46
54.92
51.82
Qwen2-7B
45.06
41.40
Qwen- 14B
29.78
45.07
62.59
60.00
GPi-4o-mini
24.38
40.28
65.03
65.15
WSSD
31.77
47.17
67.41
67.11
GPT.40
28.68
43.39
67.68
68.29
GPT-4o
32.59
46.80
70.93
67.75
LLaMA-LieCCSTT
27.24
41.37
66.86
67.52
WISD
35.09
49.28
73.47
72.98
Owen-LieCoST
30.40
44.64
68.39
65.73
gains of up to +7.11 and +5.23 points, respectively. (2) SLMs rival proprietary models (Table 5b): Qwen-LiteCoST achieves the best performance on Narrative Qasper, and HotpoiQA, surpassing GPT-4o by 0.71-1.72 points. Notably, it improves over its base model by substantial margins (up to +23.33), with both LITECOST variants consistently ranking in the top three across all datasets. Collectively, these results demonstrate the strong effectiveness and broad generality of CoST and LITECOST across diverse domains and varying task complexities.
RELATED WORK
LLong-Documen Question Answering is critical test of LLM reasoning (Wang et al.
2024 Zhang
ct al. 2024), where dispersed evidence, noise, and complex reasoning make it more challenging than
short-passage QA. Existing approaches. including long-context models (Yang et al.) 2024; Guo et al.
2025). retrieval augmentation (Lewis et al. 2020), and chain-of-thoughtt prompting (Wei et al. 2022
mitigate these challenges but remain brittle, often yielding hallucinations in high-stakes domains.
Structured knowledge has also been explored (Li et al., 2025c Panda et al.) [024] [Edge et al. 2024]
Chen et al. 2023), yet such methods require repeated large-LLM calls, leading to high cost and limited scal ability. To address this, we propose a structure-firss design with efficient SLM execution. LLMs for Long-Context Information Extraction. Information Extraction (IE) underpins many downstream NLP tasks (Xu et al., 2024 Zhang et al.). While large language models (LLMs) perform well on diverse IE tasks, even in zero- and few-shot settings (Lu et al. [023] WWii et al.. 2023 Ashok
& Lipton 2023 Wang et al. 2023b 2025 Jain et al.) 2024). existing methods remain confined to short texts. In long contexts, dispersed evidence and noise impede reliable integration. Prior "QA-
by-structuring" systems (Li et al., 2025c Tang et al., 2024b) focus on structured generation rather than verifiable step-wise extraction, and thus do not achieve reliable evidence tracing. To address this, we propose the Chain-of-Se -itti-offf-uuuueed---- Though (CoST) paradigm, which uses step-wise reasoning for structured extraction, yielding schema-aligned outputs and rich supervision for fine-tuning.
Fine-tuned Lightweight Models. While LLMs provide high-quality extraction, their computational cost and latency limit real-time use. Fine-tuned smaller models improve efficiency (Gui et al.) 2024a
Xiao et al.) 2023 Wang et al. 2023a Wu et al.) 2025), but instruction-tuning
short texts (Wu Tang et al. 2024b Lin et al. 2025) yields shallow supervision and struggles with long-document reasoning. Reinforcement learning (RL) offers stronger alternative by refining models with rewards (Shao et 2024 Xie et al. 2025). enhancing outcome correctness and
et al. 2022; Gui et al. 2024b
step-wise reasoning. Building on this (Trung al. 2024 Liu al. 2025), we propose RL-enhanced lightweight models with a two-phase, dual-reward scheme to bridge the accuracy- gap.
CONCLUSION
In this work, we present LITECOST, a reinforcement learning-enhanced framework that fine-tunes lightweight small language models (SLMs) to generate high-quality structured output for long- document QA. Through Chain-of Structured-Thouse Optimization (GRPO), LITECOST enables a 3B-scale model to approach GPT-4o-mini and 7B models to achieve GPT-4o-level performance, while substantially reducing inference latency and resource consumption. We further discuss the potential applications of LITECOST across diverse domains in Appendix E. This works demonstrate the potential of scalable, cost-efficient long-document QA and paves the way for effective LLM reasoning grounded in structured representations.
(CoST) procedure and Group Relative Policy
Limitations. While LITECOST demonstrates strong performance across financial, legal, and open- domain QA, its generalization to other distinct domains remains to be fully verified. Despite the current scarcity of comain-specific document QA datasets suitable for constructing training data, we reserve the exploration of broader domain adaptation and more diverse QA scenarios for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is supported by National Key R&D Program of China under Grant No. 46.2202YYFA1112700 Guangdong provincial project 2023CX10X008, NSF of China (62402409), Youth S&T Talent Support Programme of Guangdong Provincial Association for Science and Technology (SKXRC2025461 and the Young Talent Support Project of Guangzhou Association for Science and Technology (QT- 2025-001).
REFERENCES
Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen- Seidiin--Yildiz, Fei Xia, and Thomas Lin. Medec: A benchmark for medical error detection and correction in clinical notes, In Findings of the Association for Computational Linguistics, pp. 22539-22550, 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Dhananjay Ashok and Zachary Lipton. Promptner: Prompting for named entity recognition. arXiv preprint arXivv:335. 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 31 19-3133,, 2024.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: dataset of numerical reasoning over financial data. arXiv preprint w..:::2019000/11 2021.
Zui Chen, Zihui Gu, Lei Cao, Ju Fan, Samuel Madden, and Nan Tang. Symphony: Towards natural language query answering over multi-modal data lakes. In 13th Conference on Innovative Data Systems Research, 2023.
Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. Lim-assisted content analysis: Using large language models support deductive coding. arXiv preprint arXiv:2306. 14924, 2023.
Pradeep Dasigi. Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of informattio-sseeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.030// 2021.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local global: graphrag rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
Honghao Gui, Shuofei Qiao, Jintian Zhang, Hongbin Ye, Mengshu Sun, Lei Liang, Jeff Z Pan, Huajun Chen, and Ningyu Zhang. Instructie: A bilingual instruction-based information extraction dataset. In International Semantic Web Conference, pp. 59-79, 2024a.
Honghao Gui, Lin Yuan, Hongbin Ye, Ningyu Zhang, Mengshu Sun, Lei Liang, and Huajun Chen. IEPile: Unearthing large scale schema- dahema-conditionee information extraction corpus. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024b.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 122225511..29.... 2025.
11

---

Published as a conference paper at ICLR 2026
Xanh Ho. Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference Computational Linguistics, 2020.
Parag Jain, Andreea Marzoca, and Francesco Piccinno. STRUCTSUM generation for faster text comprehension. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Yizhu Jiao, Ming Zhong, Sha Li, Ruining Zhao, Siru Ouyang, Heng Ji, and Jiawei Han. Instruct and extract: Instruction tuning for on-demand information extraction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. CoRR, abs/2310.03714, 2023.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-322 2018.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. eetrieval-augmeent generation for coowledge-intessiie NLP tasks. In Advances in Neural Information Processing Systems, 2020.
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to SQL: are we fully ready? [experiment, analysis & benchmark J. Proc. VLDB Endow., 17(11): 3318-33 1031-333,,220 2024.
Boyan Li, Chong Chen, Zhujun Xue, Yinan Mei, and Yuyu Luo. Deepeye-sql: A software- eegineering-inspiiee text-to-sql framework. CoRR, abs/2510. 17586, 2025a.
Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. Alpha-sql: Zero-shot text-to-sql using monte carlo tree search. In ICML. OpenReview.net, 2025b.
Zhuoqun Li, Xuanang Chen, Haiyang Yu, Hongyu Lin, Yaojie Lu, Qiaoyu Tang, Fei Huang, Xianpei Han, Le Sun, and Yongbin Li. Structrag: Boosting knowledge intensive reasoning of Ilms via inference-tine hybrid information structurization. In The Thirteenth International Conference on Learning Representations, 2025c.
Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, and Yuyu Luo. LEAD: iterative data selection for efficient LLM instruction tuning. CoRR, hts//505500437, 2025.
Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng. Jinyi Niu, Zixuan Wang, Jiajic Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, et al. Fin-rl: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXiv:2503. 16252, 2025.
Di Lu, Shihao Ran, Joel R. Tetreault, and Alejandro Jaimes. Event extraction as question generation and answering. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1666-1688, 2023.
Pranoy Panda, Ankush Agarwal, Chaitanya Devaguptapu, Manohar Kaul, and A. P. Prathosh. HOLMES: hyper-relational knowledge graphs for multi-hop question answering using llss. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 13263-13282. 2024.
Nicholas Pipitone and Ghita Houir Alami. Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain. arXiv preprint arXiv:2408. 103433,, 2024.
Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S. Yu. Large language models meet NI.P: A survey. CoRR, abs/2405. 12819, 2024.
12

---

Published as a conference paper at ICLR 2026
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.052500 2016.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Nan Tang, Chenyu Yang, Ju Fan, Lei Cao, Yuyu Luo, and Alon Y. Halevy. Verifai: Verified generative AI. In 14th Conference on Innovative Data Systems Research, 2024a.
Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, and Mark Gerstein. Struc-hench: Are large language models good at generating complex structured tabular data? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024b.
Luong Quoe Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7601-7614, 2024.
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. Leave no document behind: Benchmarking long-context ulms with extended multi-doc QA. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, and Chen Guo. GPT-NER: named entity recognition via large language models. In NAACI ( (Findings), pp. 4257-4275 2025.
Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjic Ye, Qi Zhang, Tao Gui, al. Instructuie: Multi-task instruction tuning for unified information extraction. arXiv preprint arXiv:2304.0800 08085,, 2023a.
Xingyao Wang. Sha Li, and Heng Ji. Code4struct: Code generation for few-shot event structure prediction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3640-3663, 2023b.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou, -fthouht prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, 2022.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. Zero-shot information extraction via chatting with chatgpt. arXiv e-prints, pp. arXiv-2302, 2023.
Xueqing Wu, Jiacheng Zhang, and Hang Li. Text-to-table: A new way of information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2518-2533, 2022.
Yifan Wu, Jingze Shi, Bingheng Wu, Jiayi Zhang, Xiaotian Lin, Nan Tang, and Yuyu Luo. Concise reasoning, big gains: Pruning long reasoning trace with difficulty-aware prompting. CoRR, abs/2505. 19716 1916,,771116 2025.
Xinglin Xiao, Yijie Wang, Nan Xu, Yuqi Wang, Hanxuan Yang, Minzheng Wang. Yin Luo, Lei Wang, Wenji Mao, and Daniel Zeng. Yayi-uie: A chat-enhanced instruction tuning framework for universal information extraction. arXiv preprint urXiv:2312.15544 2023.
Yupeng Xie, Zhiyang Zhang, Yifan Wu, Sirong Lu, Jiayi Zhang, Zhaoyang Yu, Jinlin Wang, Sirui Hong. Bang Liu, Chenglin Wu, et al. Visjudge-bence bench: Aesthetics and quality assessment of visualizations. arXiv preprint arXiv:2510.22373 2025.
13

---

Published as a conference paper at ICLR 2026
Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang. Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang. and Enhong Chen. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 18(6):186357, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpoiqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369-2380, 2018.
Zhengxuan Zhang, WU Yin, Zhuowen Liang, Chengliang Chai, Yuyu Luo, and Nan Tang. Cycleie: Robust document information extraction through iterative verification and refinement.
Zhengxuan Zhang, Yin Wu, Yuyu Luo, and Nan Tang. Mar: Matching-augmentt reasoning for enhancing visual-based entity question answering. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1520-15220 2024.
Zhengxuan Zhang, Zhuowen Liang, Yin Wu, Teng Lin, Yuyu Luo, and Nan Tang. Datamosaic: Explainable and verifiable multi-modal data analytics through reaso--rrason--vriily. arXiv preprint ar0iv:2504. /0036, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging -ln-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595546623, 2023.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference natural language processing (volume 1: long papers), pp. 3277-3287, 2021.
Yizhang Zhu, Liangwei Wang, Chenyu Yang, Xiaotian Lin, Boyan Li, Wei Zhou, Xinyu Liu, Zhangyang Peng, Tianqi Luo. Yu Li, Chengliang Chai, Chong Chen, Shimin Di, Ju Fan, Ji Sun, Nan Tang, Fugee Tsung, Jiannan Wang, Chenglin Wu, Yanwei Xu, Shaolei Zhang, Yong Zhang, Xuanhe Zhou, Guoliang Li, and Yuyu Luo. A survey of data agents: Emerging paradigm or overstated hype? CoRR, abs/2510.2225 13.22500,,3587, 2025.
14

---

Published as a conference paper at ICLR 2026
Appendix Contents
Appendix A. Implementation of Chain-of-Structureed Thought (CoST)
16
A.1
The Prompt of Structure Analysis
T6
A.2
The Prompt of CoST Trace Generation
16
A.3
The Prompt of Quality Verification
16
A.3
The Prompt of Iterative Refinement
18
Appendix B. Additional Details of Reinforcement Learning
18
B.1
Group-Relative Advantage and KL Divergence
18
B.2 The Prompt of Answer Completeness
19
Appendix C. Experimental Details
C.I Evaluation Metrics and Scoring Criteria
19
C.2 Details of Experimental Setting
20
C.3 Numerical Results of The Radar Chart
C.4 Robust Long-Document Handling
C.5 Full Results of Reasoning w/Structured Data
C.6 Computational Resource
23
Appendix D. Case Studies on Practical Long-Document QA Tasks
D.1
Case Study RL-Enhancement
Appendix E. Generalization on Other Domains
E.1
Legal Domain
24
E.2
Open-domain QA
E.3
Others
Appendix Full Performance of Effectness on Loong (Finance)
26
15

---

Published as a conference paper at ICLR 2026
The Prompt for Structure Selection
This is data structure selection task. Based given question', choose most suitable data structure to answer the question. You can choose from the following options:
Table (statistical comparisons, mulli-source data)
Graph (entity connections, network analysis)
Text chunks (simple facts, single-step queries)
Return your answer format: ("answer": "data structure', "reason": "concise explanation")
The question Is: (question)
Guidelines:
1. the question requires aggregating and comparing numbers/attribute from multiple
sources Use Table
question focuses on connections between entities Use Graph
4. the question can be answered with direct text extraction -> Use Text chunks
Figure 8: The prompt for selecting the most optimal structure.
A IMPLEMENTATION OF CHANN-OF-STTRCCTURDD-HHHONG (CoST)
During the entire CoST: Structure- First Reasoning and Trace Generation process, construct prompts in four key processes respectively. Firstly, in the structure analysis stage, we construct the prompt about structure selection and schema construction, as shown in Fig. and Fig. / Secondly, we adopt an instruction-basse progressively extract structured knowledge and generate the CoST trace, as shown in Fig. [T0] Finally, in order to obtain high-quality serialized structured outpul (SSO), we conduct the quality verification, followed by iterative refinement. The details are shown Fig. 11 12
ofiin-of-though paradigm that performs step-by step-by-step by- reasoning to
A.1 THE PROMPT OF STRUCTURE ANALYSIS
To dynamically select appropriate data structures and perform question preprocessing, we conduct structure analysis, which includes both structure selection and schema construction. On one hand, we design prompts that enable question-oriented structure selection. On the other hand, we construct an accurate, task-specific schema through careful preprocessing of the question, before instruction- information extraction. The prompts of structure analysis are shown in Fig. g
A.2 THE PROMPT OF CoST TRACE GENERATION
To generate high-quality trace and structured output, we adopt an schema guided chain-of- thought paradigm composed of: (1) step-by-step task instructions, (2) input text, and (3) schema dynamically generated from the question. GPT-4o is prompted with these schema-informe intermediate reasoning traces, which then used supervise instruction-tune -uned models zero-shot or few-shot settings. Fig. 10 illustrates this reasoning process for the structured output (e.g., table).
instructions to produce
A.3 THE PROMPT OF QUALITY VERIFICATION
To evaluate the quality of the generated structured output, we employ an L.LM-as-Judge framework. Given the original question and the model-generated answer derived from the extracted structure, we prompt GPT-4o to assess whether the answer correctly addresses the question. An inference considered correct and retained only if it exactly matches the expected answer based the prompt instructions. The prompt includes the original question, the extracted result, and evaluation instructions guiding the model to make binary decision, detailed in Fig. I1
16

---

Published as a conference paper at ICLR 2026
The Prompt for Schema Construction
You are schema generation assistant. Given query, analyze its task and generate a structured schema to guide entity extraction from documents. The schema should include all relevant enlities and their types, even for simple queries.
### Key Rules:
"'Eniiy Extraction"::
the query involves single entity (e.g., "Find the company with the highest revenue"), extract the entity type (e.g.. "Company', "Revenue").
If the query contains composite entities (e.g., mathematical formulas or nested metrics),
them into separate entities (e.g., "Accounts Payable", "COGS", "Inventory"). '"Language Support"":
The schema should support both Chinese and English queries. Use language of query for entily names.
3. "OOuipu Format"":
Ignore output requirements in the query. The output must strictly follow this format:
(EntityTypa) EntilyType2,
strictly
(EntityType1, Type2, ...).
The schema should include only entity types, listed order separated by commas.
Entity types should be clear and specific (e.g., 'Company", "Revenue", "Year").
#*# Examples:
(oxamples)
Data:
Query: (query)
### Output:
Figure 9; The prompt for dynamically constructing schema.
The Prompt for Table Extraction
Ubjective
Consruct struclured iabre ron comeett that strictly adhemee provided achema..
hppi Schere
Schema Columes: (schoms)
Slop Stop Indtuutions
Shep "LEaity Extrecion"
Icentily relevant enti
alinn
using ssact keyword matching
Blrwctt lin
they spawar parteriing any aidilonal
no direct mutch. Use aprtortually to ated valsos marked with (INFERPED)
Generate uneemed sle resull that octpute the pe between entiies Inn the torm of lap was, ecotamiy mg how euch entiyy cornected,
Shep 2: "Sunnutrratin""
Bosed the endtees and livinng relationc from the provius steps, cummarie the intomaaiio and organizo it into tants
vircctrred fornel that contoms the Schuma.
morreddite ensuring that Schema correctty populated. Seap "*iinal Formatting"
Generste
Utiple
deintteeedeuuteeelllpp. wwt.miaapsseeeeetptooesee...ee _delmber)
(how"Cupte attmitrj-VALLE Txtliple soinitrr-WALUE 2x Jrreordd delimiter)
[camplation delintrr
Criicall Rtules
STRUCTTLY POLLD OW SCHEMA EOL UMN ORCER
NEVER inon data prosent row scnnont
"noo NO1
addtional computation simply edraction."
"TThe chaug-ot-hhough reasoning
shouts dieatly decumented intemmddate results befors generating frai table.
(content)
Duisit Strctures
Figure 10: The prompt for extracting table step-by-step.
17

---

Published as a conference paper at ICLR 2026
The Prompt for Data Verification
answer, provided answer, your verily answer
according the standard answer. You should base your evaluation strictly on the content and correctness of the provided answer in relation to the standard answer
<standard answer>
(std_ans)
<standard answer>
<provided
(ane) <provided answer>
Important
<provided
can't get
any similar phrase, consider
answer incerrect.
2. the sprovided answer> is vague, irrelevant or does not answer the question properly, return "False".
3. the <standard answer> is a numerical value, and the format the <provided answer> different answor> the <standard 98%. it considered that the meanings consistent, return 'True".
from
values the same,
considered that the and <provided
meanings consistant. example,
4. If the <slandard answerz is a numerical value, and the final result of the sprovided answerz is consistent with the <standard answers atter rounding. then is considered that the meanings are consistent. example, <standard answer> "True".
<provided answers It considered
meanings
## Output Format:
Please respond with the result in the following formal: ((check: True False))
provide any additional explanation
result spacified format.
Figure 11: The prompt for verifying the data quality.
A.4 THE PROMPT OF ITERATIVE REFINEMENT
To enhance Group Relative Policy Optimization (GRPO) training with more challenging learning signals, we introduce an Iterative Structuralizer module that refines low-quality samples through recursive structured knowledge regeneration. The prompt for refining structured data extraction (e.g., table) is detailed in Fig. [2 Formally, this process is implemented as a recursive function over the evolving extraction state, gradually improving coverage and accuracy across iterations:
((t))
if (C(S .q) True otherwise,
g(++1)
(5)
(e. S(O).
where (C((()) q) is sufficiency evaluator that returns True if the current structured knowledge can answer the question; fextreet is the structured knowledge extraction function, and is the context. The process terminates when S(K((t,, .q) True when a predefined maximum number of iterations is reached. The final structured knowledge S* is then used for downstream reasoning.
B ADDITIONAL DETAILS OF REINFORCE LEARNING
B.1 GROUP-RELATIVE ADVANTAGE AND KI. DIVERGENCE
For completeness, we provide additional details of the GRPO optimization. The importance sampling ratio is defined as r.mito ne(o: [v) Tee (os VV which quantifies the relative likelihood of generating output o; under the new policy compared with the old policy л The group-relative advantage A, is calculated based the relative rewards of outputs within the same group only.
To ensure stable optimization, the clipping operator clip(r:atio, 1- c, + c) restricts the update magnitude within the trust region [1 - c, + cl, thereby avoiding destabilizing large parameter changes. Finally, taking the minimum between the unclipped term X and its clipped counterpart enforces conservative update, which balances aggressive improvements with training stability.
18

---

Published as a conference paper at ICLR 2026
The Prompt for Data Refinement
Reconstruct by combining Original Data and Table STRICTLY MAINTAINING the existing format siructure.
Format Requirements
1. Follow this EXACT structure:
("Iable ("taaleluuiieedeiiiieeeeeea
(record_ delimiter)
rrecorddellimitrr)
("header"(tull (tuuleee(tpeepreeeee wew.m//errr-OLLUUNNNaAaa (ittpll
http:////....ooooooopppppoooooou (etpppee__clmmmee//CLLLUUUM .t..eerrrrddddddd .e...Krecord_delmmtter delimieerr
""roletuple.co "twp""tuue_delmiiiiiiee
(completion_dee dellimt_eeliiter
2. Strict Rules:
PRRESERVE original column order and headers
USE
jillessouree/descripiinn unless new metadata found
DO NOT add/remove columns
ONLY modity rows with missing/confficting data
# Input Context
Original Data:
(content)
Current Table Structure: (current_table)
# OutputA
Figure 12: The prompt for refining low-quality data.
In addition, the KI. divergence term DEL ( Fret) plays critical role in regularizing policy updates. penalizes large deviations from a stable reference policy Tref, preventing overfitting to noisy reward signals and maintaining alignment with the base model's distribution. The coefficient controls the strength of this regularization, striking a balance between exploration (deviation from the reference) and stability (consistency with the pretrained policy). This regularization is particularly important when rewards are sparse or noisy, such process rewards, as it prevents the model from overfitting unstable signals while still enabling gradual improvement (Shao al. 2024).
B.2 THE PROMPT ANSWER COMPLETENESS
To measure the semantic similarity between the generated structured outputs and the ground truth during the GRPO process, we employ GPT-4o-mini as an automatic evaluator. The evaluator examines outputs along four dimensions: null check, core field coverage, semantic alignment, and semantic equivalence-ane and then produces score within the range [0, 100). This score, denoted as Seem in Equation 5, provides an accurate quantification the semantic similarity the structured data, with the prompting details shown in Fig. 13
C EXPERIMENTAL DETAILS
C. EVALUATION METRICS
Following the approach of Loong (Wang et al. 2024). we prompt GPT-4o as judge to against the golden answer and question requirements three dimensions: Accuracy, Hallucinations, and Completeness, on 0-100 scale, as detailed in Figure 15 With this evaluation method, the Judger model would output a percentage score along with corresponding explanation. Given the limited ground-truth annotations for information extraction on long-contes documents, adopt 2-hop evaluation following the principle of STRUCTSUM (Jain et al.) 2024), which uses the structured outputs answer QA pairs derived from the input text. Our core intuition is that if LLM can accurately answer questions based solely on the extracted data, the extraction process has likely preserved the essential information-thus reflecting its quality.
19

---

Published as a conference paper at ICLR 2026
The Prompt for Semantic Verification
[Task Description]
[Generated aligns [Ground Truth Data] content and moaning. Evaluate step by step, then output score (0-100), key, differences, and warnings.
semantle consistency evalluator.
[Steps]
Null Check
[Generated Datal is empty or trivial B..B. "N/A"), return score 0 with warning "Generated data emply"
Field Coverage
Check key fields (e.g.. company, revenue, time) in ground fruth are novered.
Missing core fields greatly lowers the score: non-critical fields have less impact. meaning (egg.
Different
"Operating Income") equivalent.
Semantic Alighment
Venty if fields express the same meaning in:
Numbers (10M 10,000.000)
Unita/ratioa/lime
Entities (Microsaft vs Microsaft Corp.)
expressions
Factusl relations (Founder Jack Ma vs Jack Ma founded the company)
Semantic Equivalince (No Panally)
Acceptable varitiion: numbor formatting. abbreviatione, time formats, synonyms.
[Ground Truth Date]
(Ground Truth)
[Generated Datal
(Generated Dale)
[Output Format]
Return JSON only:
"score":
dere_diffeeeneee' ["...],
"warnings": ["..7]
Figure 13: The prompt for verifying the similarity of generated output and ground truth.
For experiments LongBench (Bai al. 2024), we additionally use the standard Fl score assess the quality of reasoning based on the extracted information. Consistent with the Loong setup, GPT-4o employed as the reasoning agent generate final answers from the structured outputs.
C.2 DETAILS EXPERIMENTAL. SETTING.
Base Models. To comprehensively evaluate the extraction ability of LITECOST, we compare with several state-of-the-art models, including
(Grattafiori et al. 2024). GPT mini,,, GPT-4o (Achiam et al. 2023), and Deepseek-R1 (Guo et al.) (025). For fairness, all models are evaluated under the same prompting setup, where each is guided by task-specific schema derived from the given question or instruction. To avoid exceeding the context window, all models perform document-level extraction and merged the resulting structured sub-knowledge.
Awen2-7BBInnstrul (Yang al. 2024). -lamaa-.11-8B-Intttue
Fine-tuned IE Models. For comparison with fine-tuned IE models, both LITECOST and the baselines trained on our CoST-curated dataset under identical conditions. We then evaluate their extraction performance to highlight differences in fine-tuning strategies and demonstrate the relative advantages of our approach.
Modular Frameworks. We further consider modular extraction frameworks such as StructRAG (Li al. 2025c). In this setup, we configure the Router and Structurizer components with the same while employing GPT-4o as
backbones (i.e., M..MA--.--BB-Insrruu and Qwen2-7B-Instree the Utilizer for reasoning. This setup ensures that the comparison between AtructRAG and our LITECOST remains both fair and rigorous.
Baseline Prompting Template. To ensure fairness, all baselines and our model use the same input prompting format for structured extraction, as shown in Fig. 14] The LLM baselines are evaluated in zero-shot setting, and entries labeled "LLM baseline" reter to this zero-shot performance. Our LiTECoST-tue -tuned models compared directly against these baselines because both rely the identical prompt template, ensuring that performance differences arise from CoST distillation and the two-stage training procedure rather than from prompting variations model-capacity differences.
20

---

Published as a conference paper at ICLR 2026
Prompling Template
Graph Baseline Prompling Template
Prompling Template
You are an expart in tablee constructon. Plaase aniifies the definiion the inpul and generate a structured
You are expert in graph condtruction Pleace edtract
excert chuniss construction. Please eatract the text chunks that can answer finally
relafionhipp triples in form (head, relattonnni. teal that match achema deliition from the input, and fraily the structured graph.
given question from
generate the structured text, chunks...
Schema:
[date]
(yyp
Input Text:
Input (dans)
(ontitt__yyp
(question)
Question:
Input Text:
Please output the final tabre structured format.
Please oupput the final graph in stuctured formal.
(dato)
Figure 14: The baseline prompt template for different structured extraction.
Table 6: The performance of various LLMs under different prompting settings, showing Average Scores (AS, 0-100) and Perfecr Rate (PR, 0-1) on the Finance subset of Loong benchmark.
Backhone
Method
Spotlight Locating Comparison
Clustering
Chain of Reasoning
Overall
PR
Zero-Shot
83,74
0.57
82.12
0,56
69.96
0.24
66.41
0.10
75.60
0.38
Qwen2,5-14B-Ing
CoT Wei et 2022
85,93
0.63
81.38
0.57
73.28
0.30
67.70
0.26
77.51
0.44
Dspy KKliaaao at 2023
80.90
0.55
80,90
0.60
76,86
0.34
63.60
0.21
76.99
0.44
LITECOST (COST)
88.03
0.72
80.45
0.56
76.72
0.37
68.35
0.25
79.01
0.48
Zero-Shot
84.10
0.73
80.53
0.60
81.50
0.50
64.30
0.25
79.32
0.54
GPT-4o
CoT Wei ct 2022
85.57
0.69
84,40
0.66
79.58
0.44
64.45
0,30
80.08
0.54
Depy Khalaab ot al. 2023
86.17
0.74
82.65
(1.2
80.00
0.47
67.25
0.27
80.26
0.54
LITECOST (CoST)
88.80
0.75
84.50
0.65
82.24
0.50
68.90
0.28
82,39
0.56
C.3 NUMERICAL RESULTS OF THE RADAR CHART
Table 6 provides the detailed numerical results on the Finance subset of Loong across backbone LLMs. Relative to Zero-Shot prompting, Shain-of-Thoughhe (CoTT) consistently improves overall performance, raising the average score (AS) from 75.60 to 77.51 on Owen2.5-14B-Ins and from 79.32 to 80.08 on GPT-4o. Building further on CoT. our CoST-based LITECOST achieves the strongest results across all backbones, with overall AS/PR reaching 79.01/0.48 on Qwen2.5-14B-Im and 82.39/0.56 on GPT-4o. The improvements are consistent across subtasks for example, CoST yields a gain of +6.62 AS on Clustering with GPT-4o and +8.62 AS on Spotlight Locating with Qwen2.5-14B-Ing InB-Ins, demonstrating its robustness in enhancing raasoning-intensive extraction.
We have also integrate prompt-optimization methods, including DSPy (Khattab et al. 2023), into our evaluation. Specifically, we use the same zero-shot stractured-extraction instructions as the initial input prompts for GPT-4o and -.een..s-14B--nstruet,,, and then apply the MiPRO optimizer to refine these prompts-automatiinn automatically generating structured prompts. MiPRO conditioned on examples drawn from a randomly sampled 10% subset of the LITECOST training data (finance). We observe that although DSPy provides slight improvements over the zero-shot and CoT baselines, our method still performs substantially better. This highlights both the effectiveness and robustness of CoST, and further illustrates the challenges of applying automated prompt optimization to structure-aware extraction tasks.
reasoning steps and optimized
C.4 ROBUST LONG-DOCUMENT HANDLING.
The results in Fig. show that both siiin-of--Stuutureddee Thought (CoST) and LITECOST exhibit strong robustness as document length increases.
LLM+CoST. The overall score of LLM equipped with CoST decreases by only 25.21 points when moving from Setl (91.46) to the most challenging Set4 (66.25), whereas zero-shot prompting exhibits much sharper decline (92.33 58.42) and CoT similarly drops from 87.90 to 60.49, On Sct4, CoST surpasses zero-shot and CoT by 5.76 and 7.83 points, respectively, indicating that CoST provides substantially greater robustness under increasing difficulty.
LiTECoST-tuned SLM. Our LiTECoST-tunel SLM also demonstrates strong robustness, exhibit- ing only modest 24.46-poin decline document length increases. Even on the most challenging Set4, achieves a score of 64.89, outperforming all LLM-based baselines: +6.47 over GPT-4o, +4.77 over Qwen2.5- 1-BBBB.55.BBB-.... Instrrct, and + +1001 over A-..-...11---- 1-81-----stttees
21

---

Published as a conference paper at ICLR 2026
The Prompt for LLM Score Evaluation
[Gold Answer]
(std ans)
[The Start of Assistant's Predicted Answer] (ans)
End Assistant's Predicted Answer]
[System] We would like request your feedback the performance the user question displayed above according to the gold answer. Please use the following listed aspects and their descriptions as evaluation criteria: Accuracy and Hallucinations: The assistant's answer is semantically consistent with the gold answer; The numerical value and order need to be accurate, and there should be no hallucinations. Completeness: Referring to the reference anewers, the key points needed answer the user's question; further omitted.
assistant in
assistant's answer should contain elaboration these key points question. Please note that the gold The assistant receives overall score on scale of 1 to 100, where higher score indicates better everall performance. Please note that if the assislant's answer and the gold answer fully meet the above critoria, Its overall ratiing should be the full marks (100). Please first provide comprehensive explanation of your evaluation, avoiding any potential bias. Then, outout line indicating the score of the Assistant. SCALE
suitable the the question.
cansidered correct
PLEASE OUTPUT THE FOLLOWING FORMAT, WHERE "[[scor]]", EXAMPLE "Rating: [[100]]":
100
STRICTLY FOLLOWING
<Start Output>
Evaluation evidence: evluation explanation here, no more than 100 words Rating: [[score]]
<End Outputs
New, start your evaluation:
Figure 15: The prompt for evaluation using LLM Score.
Aama3-2-b-ire
lamad lamm331-- 1-4b-ins
gpt-4o
LLM
LLM-COT
LLM+CoST
qwen2-7b-ii 7b-ns
Qwen-LtecuST
90
90
Performance
Performance
80
80
70
70
60
50
Set1 (10k-50k)
Set2 (50k-100k)
Set3 (100k-200k)
Set4 (200k-250k)
Set1
Set2 (50k-100k)
Set3 (100k-200K)
Set4 (200k-250k)
(10k-50k)
Figure 16: Performance (LLM score) versus document-set size for (a) CoST with bascline methods (zero-shot and CoT) and (b) LiteCoST-tuned models with different base models.
These results underscore the particular strength of CoST and LITECoST in handling long- docummee inputs, consistently maintaining high reasoning accuracy even as context length increases.
C.5 FULI RESULTS OF REASONING w/ STRUCTURED DATA.
By selecting the optimal structure for each QA sample the Loong benchmark, the dataset comprises 823 tables, 400 graphs, and 377 text chunks. The purpose of this experiment is to explore how serial- ized structured output (SSO) contributes to model performance on knowledge-intensiee reasoning tasks. Therefore, we curate such structures through our LITECOST framework with GPT-4o as the base model, and then evaluate the results of LLMS reasoning over these structured representation.
Table 8 shows that all models achieve better reasoning performance when leveraging the extracted structured information, compared to the baseline that directly processes the raw long-form documents, underscoring the necessity of accurate and effective structured knowledge extraction.
The gains are particularly notable in comparison and clustering tasks, where all models show improvements in LLM scores, along with least 0.11 and 0.17 increases in Perfect Rate, respectively. These results highlight the value of structured knowledge in capturing entity relationships and aggregating discrete information for complex reasoning.
Pros.
22

---

Published as a conference paper at ICLR 2026
Cons. We observe slight performance drops in certain subtasks (Spotlight Locating and Chain-of- Reasoning) after integrating structured data. These results reflect a trade-off between structured clarity and the flexibility of unstructured reasoning:
Improves aggregation and clustering tasks by enabling more effective comparison and summarization.
Advantages of structured data:
Challenge in Spotlight Locating: Fine-grained localization may degrade when structured data omits subtle contextual cues, reducing recall.
Challenge in Chain-of Sacnn-of-Rasssee Reasoning:: Complex multi-step inference may lose nuance when context compressed into structured formats.
Here, provide
illustrate this issue with the following Spotlight Locating task:
Question: What the name of the company with $30, 179 in Accounts Receivable?
Ground Truth: CIRTRAN CORP
LLM Answer: The information provided in the context does not list company with $30, 179 in
Accounts Receivable. Therefore, the answer is Not Provided
Table 7: Extracted table for the given query.
Company
Accounts Receivable
AXIM Biotechnologies, Inc.
$23,642
CIRTRAN CORP.
None
Arax Holdings Corp.
$453,837
High Wire Networks, Inc.
$4,483,1
In this case, due value corruption omission during extraction, CIRTRAN CORP's relevant numerical value is missing in the structured table, leading the model to respond incorrectly. A large language model performing direct reasoning over the full text, in contrast, can flexibly match partial evidence (e.g., approximate figures or nearby mentions) without being constrained by structure.
C.6 COMPUTATIONAL RESOURCE.
Model Deployment. We deployed and run models ranging from 3B 14B model size cluster equipped with eight NVIDIA RTX 4090 GPUs, each with 24GB of VRAM. For closed sourced large language models such as GPT and Claude, we accessed them via API calls.
Computational cost. Our cost mainly comes from two components:
Data Generation. This includes structure analysis, CoST trace generation, quality verification, and iterative refinement using GPT-4o. The average cost is approximately $30 per domain.
GRPO Fine-tuning. Reward computation relies on GPT-4o-mini to evaluate structural alignment, format compliance, and answer correctness. This adds additional $10 per training run.
The total cost of $40 necessary and acceptable because amortizes extremely well: once LiteCoST is trained, downstream inference relies solely on compact SLMs, eliminating repeated LLM calls and yielding substantial savings during deployment.
D CASE STUDY ON PRACTICAL LONG-DOCUMENT QA TASKS.
D.1 CASE STUDY ON RL-ENHANCEMEN
As shown in Fig. 17 we present representative long-context QA example to evaluate the model's information extraction capabilities. This question is particularly challenging, as it requires the model to retrieve multiple pieces of evidence from the input text and accurately integrate them into coherent structured information. Specifically, the case study includes the question and documents input (grey box), predictions from the base LLaMA-3.2-23 3--BB-Instuct model (blue box), its finetuned variant (orange box) and our LITECOST model (green box).
In addition, incorrect predictions are highlighted red, while correct ones are marked green. The results show that the Llama base model performs poorly at information extraction and fails
23

---

Published as a conference paper at ICLR 2026
Table 8: Structured data quality evaluation of LITECOST via Structured-Touught (CoST) on the Loong benchmark, compared against popular LLMs. SD denotes structured data.
Context
Spollight Locating
Comparison
Clustering
Chain of Reasoning
Overull
Model
Length
AS
PR
AS
PR
AS
PR
AS
PR
AS
PR
Qwen2-72B-Ins
128k
54.17
0.36
42.38
0.20
36.71
0.04
47.76
0.18
43.29
0.15
SD + Qwen2-72B-Ins
128k
57.30
0.31
54.80
0.38
61.46
0.23
46.36
0.16
55.70
0.25
GPT-4o-mini
128k
59.46
0.49
51.90
0.27
34.55
0.04
64.28
0.39
49.25
0.24
SD + GPT-4o-mini
128k
63.23
0.44
53.04
0.38
59.63
0.21
55.98
0.26
58.02
0.29
GPT-4o
128k
73.95
0.62
50.50
0.28
44.29
0.09
57.05
0.28
63.47
0.26
SD GPT-4o
128k
62.11
0.33
63.27
0.41
68.06
0.29
53.52
0.22
62.51
0.30
Claude3.5 Sonnnet
200k
58.45
0.49
54.21
0.25
45.77
0.07
43.92
0.25
48.85
0.23
SD +
200k
47.60
0.34
54.64
0.41
66.95
0.31
50.45
0.23
57.32
0.31
Question: Which company has the highest total amount of non-current assets?
Documents: The financial reports of eight companies: CDFCO Engineering, Pioneer Electronics, Aerospace Intelligent Manufacturing, Gree Electric, China Express Airlines, DunAn Environment, Huafeng Microfiber, and Wuhan Fangu.
Llama-3.2 2.2- 38-Instruct (Base)
Llama 3.2 3B Instruct (SFT)
Llama-33 1..ma-3.2-20 -B-IInstruct (Ours)
&
Extracted Structured Data
H
Extracted Structured Data
Extracted Structured Data
COFCO Engineering Co., Ltd., country-owned company, holds 39.48% of the shares, amounting 202,243,856 00, with limited sale condition and no locked or cartified status. China Express Airlines Company, domestic non-state-owe owned entity, holds 7.33% of the shares, totaling 37,572,900.00..
Company
Non Non-currettio Assets
Company
Non Non-current Assets
COFCO...
3. 3,582, 768.20
3,582.668.20
COFCO ...
629,581663397
DunAn
296,305,753.57
2,145.171.199.66
China Express...
15,185,444,226 64
China Express...
15,185,444,29.66
Pioneer...
3,750,842.05
Pioneer
279,422.490.92
Wuhan Fangu
2,301,98839
Wuhan Fangu
858,871,89228
Electric...
149.673,530.15
Electric...
145.555,317,8786
Aerospace
232,197,577.22
Aerospace...
2,907,34,,11.35
Huafeng...
5,001,871,249.05
Huafeng... ...
5,091,871,249.05
Structured Reasoning Output
Structured Reasoning Output
Structured Reasoning Output
The table lacks specific entries for total company, so It's not possible to identify which has the highest. X
The company with the highest "Total con-ccurent Assets" is China Express Airlines Co. Ltd., with an amount of 15,185,444,222 244,449.4444
Among these companies, Gree Ellccticc Appllances, Inc. has the highest total non-current assets, amounting 145,555,317,678.8
assots per
Figure 17:
Case study on representative information extraction tasks for large language models, comparing the information extraction capabilities of three Llama-3.2-3B-Inste variants.
to organize content into structured formats such tables. The fine-tuned model exhibits partial extraction ability but remains inaccurate in long-context, multi-document settings. In contrast, our LITECoST accurately integrates dispersed information into high-quality structured tables, with gains largely attributed to supervised CoT reasoming and reinforcement learning. As shown in Fig. I7 LITECoST provides significantly enhanced interpretability compared its base model.
E
GENERALIZATION ON DIFFERENT DOMAINS
E. LEGAL DOMAIN.
To examine the generalization capabilities of LITECOST, we apply it to the legal domain by curated by from LegalBenchRAGG (Pipitone & Alami] 2024), dataset of 6,858 query-answer pairs over corpus of over 79M characters, entirely human-annotated by legal experts. Leveraging the training data described in Sec. 4.1 we perform RL fine-tuning on small language models. The fine-tuned
models are subsequently evaluated on the legal subset of Loong (Wang et al. 2024) to assess their ability to generate structured outputs in complex legal contexts.
Table 4 shows that our LITECoST-tunce 3B/7B models, LLaMA-
and Qwen-LiteCoST, achieve substantial improvements over their respective base models and deliver performance compa- rable to proprietary LLMs. From these results, we further derive two key insights:
(1) Notably, our method elevates LLaMA variant to surpass Qwen in overall performance. The two backbones, however, exhibit complementary strengths: LLaMA excels on the Spotlight Locating and Comparison subtasks, while Qwen achieves superior results on Clustering and Chain-of Reasoning.
24

---

Published as a conference paper at ICLR 2026
Table 9: An overview of the dataset statisties for subset of LongBench. 'Source' denotes the origin of the context. 'Avg len' (average length) is computed using the number of words for the English (code) datasets and the number of characters for the Chinese datasets. 'Accuracy (CLS)' refers to classification accuracy, while 'Accuracy (EM) refers to exact match accuracy.
Datasct
ID
Source
Avg len
Metric
Language
#data
Single-Docunng QA
NarrativeQA
1-1
Literature, Film Science
18,409
F1
English English
200
Qasper
1-2
3,619
F1
200
QA 2WikiMultihopOA 2-2
HotpotQA
2-1
Wikipedia
9,151
F1
English
200
Wikipedia
4,887
F1
English
200
(2) Qwen2-7B-Inge Inntstuutttt performs exceptionally well on the legal domain, even surpassing its 14B counterpart, potentially due to: stronger domain adaptation from larger proportion of legal corpora in training; @ the tendency of larger models to over-generate in long, highly structured legal texts, leading to hallucinations or format drift, whereas the smaller 7B model more faithfully adheres to schemas and maintains consistency.
E.2 OPEN-DOMAIN QA.
Beyond the Loong benchmark, we further evaluate our method on LongBench (Bai et al., 2024), a widely recognized multi-task benchmark for long-document QA that covers key real-world application scenarios across literature, science, encyclopedias, etc. These datasets contain far more nuanced information. As shown in Table[5] our LITECOST effectively extends to settings with richer and more subtle semantics, demonstrating strong generalization beyond strictly structured environments. Our analysis focuses both single- and multi-doc QA tasks across four datasets, as shown in Table 9
Single-Doc QA. For single-document QA, we focus on datasets containing longer and more challeng- ing documents. We evaluate on Narrative (Kočiský al. 2018), consisting full--ength stories paired with questions designed to test deep reading comprehension. We also include Qasper (Dasigi al. 202T). a dataset featuring QA over NL.P research papers, annotated by domain experts.
Multi-Doc QA. molti-document QA requires models to extract and combine information from several documents to obtain the answer, which is usually more challenging than single-doc QA. We evaluate on two Wikipedia-based multi-hop QA datasets: HotpotQA (Yang al. 2018). 2WikiMulti- hopQA (Ho et al., 2020). HotpotQA involves a number of 2-hop questions directly written by na- tive speakers given two related paragraphs. 2Wiki-MulthhoPQA consists of up to 5-hop questions that are synthesized through manually designed tem- plates to ensure that they cannot be solved through shortcuts. Each question in the original datasets is supplemented by 2-4 supporting paragraphs that provide one-step reasoning evidence and several distracting paragraphs.
E.3 OTHERS.
Beyond the finance, legal, and scientific QA tasks, LITECOST can be applied wide range of fields, enabling effective and efficient structured extraction from large-scale unstructured corpora:
Healthcare: extracting patient attributes, treatment outcomes, and adverse event reports to
enhance clinical decision-making and pharmacovigilance.
Scientific literature: supporting literature understanding, analysis, and question answering through the extraction of experimental settings, results, and methodological details.
structuring entities and relations from legislative and regulatory docu- ments to facilitate compliance monitoring and policy evaluation.
Policy and government:
Enterprise analytics: organizing information from reports, manuals, and support logs into structured forms to improve knowledge management and retrieval- -uggeetee applications.
Together, these applications highlight the broad adaptability of LITECOST in transforming unstruc- tured knowledge into structured representations that directly support diverse downstream tasks.
25

---

Published as a conference paper at ICLR 2026
Table 10: Full results of Table ! AS represents Arg Scores 100) and PR denotes Perfect Rate (0"1). Bold indicates the best result within each setting, and underlined indicates the second best.
Model
Model
Spotlight Locating
Comparison
Clustering
Chain of Reasoning
Overall
Size
AS
AS
AS
PR
AS
PR
AS
All Set
(10K-250K)
LLaMA- 3.2-3B Instruct
3B
49.90
0.16
52.10
0.14
47.89
0.07
46.85
0.06
49.37
0.11
Qwen2-7B-Instre
7B
63.10
0.36
67.85
0.37
60.83
0.18
52.25
0.09
62.10
0.26
LLaMA-3.1 3.1.--11-55 3.1.8B Instructtt
8B
55.03
0.20
51.60
0.15
51.50
0.04
44.75
0.02
51.32
0.10
GPT-4o-mini
8B
84,42
0.70
80.40
0.67
77.38
0.40
65.35
0.18
78.08
0.51
Qwen2.5. 14B Insttnct
14B
83.74
0.57
82.12
0.56
69.96
0.24
66.41
0.10
75.60
0.38
GPT-4o
200B
84,10
0.73
80.53
0.60
81.50
0.50
64.30
0.25
79.32
0.54
Deepseek-R2 R1
671B
84,27
0.62
78.97
0.55
75.42
0.34
74,40
0.35
78.18
0.46
LLaMA-3.2.3B Instruct (SFT)
75.53
72.27
LLaMA-3.2 1-----3.-----nttrrce (Ours)
38
81.27
0.53
78.08
0.49
78.34
0.36
64.75
0.16
76.95
0.40
Qwen2- 7B (SPT)
82.23
81.15
76.83
Qwen2-7B-Instee Instructt (Ours)
7B
83,97
0.62
81.55
0.59
81.00
0.43
67.98
0.18
79.93
0.48
Set: (10K-50K)
LLaMA-3.22 1.MM--3.2-3B- I-3--nsrrctt
3B
54.13
0,17
43,33
0.13
44.25
0.07
55,50
0.10
47.28
0.12
Owea2-7B-Instruut
7B
73.26
0.48
80.17
0.53
68.25
0.28
65.00
0.30
72.52
0,40
LLaMA-3. I-8B-Instrict
8B
46.09
0.04
35,50
0.00
47.38
0.00
40,50
0.00
42,96
0.01
GPT-44-miii
8B
96.09
0.91
93.00
0,90
86.62
0.70
75.50
0.40
89.51
0.78
Qwen2,5-14B-Bi 1-4-Intrret
14B
95.00
0.74
87,00
0.63
84,45
0.53
84,30
0.20
87.53
0.57
GPTT-0
200B
96.09
0.91
90.00
0,80
91.25
0.78
95.00
0.90
92.33
0.83
DoepseckRR
67
91,96
0,78
90.50
0.80
88.75
0.68
99,50
0,90
91.02
0.76
LLaMA-3.2-38 3.2t---r-nt (SFT)
3B
80.43
0.61
83,10
0.60
82.70
0.47
67.00
0.30
80.79
0.52
LLaMA-3.2 22 Instruct (Ours)
3B
91.52
0.78
81.33
0.57
88.95
0.60
75.00
0.50
85.95
0.62
70-rn-2---Insttuut (SFT)
7B
91.30
0.83
89.17
0,73
81.75
0.53
81.50
0.20
86.02
0.62
Qwen2-7B Instruct (Ours)
7B
92.17
0.83
89.67
0.77
88.75
0.72
84.80
0.60
89.40
0.75
Sett2 (50K-100K)
LLaMA-3.2-3B--E
57.16
52.61
Qwen2-7B-Instre
7B
80.00
0.62
71.13
0.45
63.11
0.24
58.75
0.12
67.61
0.35
LLaMA-3.1
1LaMA-3.1-88-Instee 1-8t-Instruc
8B
55.15
0.03
52.86
GPT-4o-mini
8B
96.12
0.90
88.93
0.80
86.07
0.58
64.62
0.28
85.09
0.65
Dwen2.5-14B-Innstn
14B
0.28
GPT-4o
200B
90.38
0.80
96.27
0.72
89.67
0.71
66.88
0.33
85.02
0.67
Deepseek-R1
83.04
82.07
L.MMA-1.2.3P Instttrreet (SFT)
3B
82.83
0.57
77.93
0.49
79.09
0.40
65.12
0.15
77.07
0.42
(Ours)
3B
88. 12
0.60
81.33
83.98
0.48
65.25
0.20
80.79
Qwen2-78.Instrmet Instruct (SFT)
7B
87.58
0.70
84.73
0.63
80.53
0.41
66.12
0.20
80.67
0.40
Qwen2-7B-Instee (Ours)
7B
90.00
0.72
85,53
85.28
0.56
68.50
0.23
83.39
0.50
Set
(1 -200)
LLaMA-3.2-3B 2-0--...-3B-
3B
54.42
0.22
51.24
0.09
44.06
0.02
45.14
0.09
48.67
0.10
Qwen2-7B Instruct
7B
56.00
0.30
63,73
0.28
59.63
0.13
45.57
0.03
58.08
0.20
LLaMA-3. LLaMA-3.1-8B-Instun 1-8B-Instruet
8B
57.92
0.27
55.13
0.17
52.06
0.04
39.71
0.00
$2.63
0.13
GP'T-4o-mini
8B
84.00
0.77
77.40
0.57
71.78
0.22
64.57
0.09
75.25
0.43
Qwen2.5-14BB 1-14-Instttt
14B
88.93
0.68
76.93
0.49
63.39
0.16
64.14
0.00
73.29
0.35
GP'T-o
200B
88.33
0.83
76.13
0.49
77.00
0.32
53.43
0.06
76.19
0.45
Deepseek-R1 R1
671B
91.17
0.73
76.80
0.47
71.22
0.16
67.57
0.20
76.94
0.38
LLLaMA-12-BB-nstrruc (SFT)
3B
75,25
0.48
74.24
67.67
0.16
55,14
0.09
69.63
0.29
1L.MA-312-3B-nnstmn (Ows)
3B
83.75
0.60
77.73
0.47
72.37
0.22
62.43
0.06
75.20
0.36
Owen2-7B-Instruee (SFT)
7B
85,33
0.63
78.60
74.06
0.24
56.29
0.03
75.58
0.37
Owen2-7B-Instren (Ours)
7B
86.92
0.68
8:73
0.53
79.
0.33
63.86
0.11
78.75
0.40
Set4
(200K-2500K)
LLaMA-3.2 3.MA--.2-30 IB-IIntruct
3B
48.52
0.11
49,50
0.15
36.50
0.00
50.33
0.00
45.
0.07
Qwen2-7B-Inste In--ntrut
7B
52.50
42,00
47.07
LLaMA-3.1 1.1..8 Instruct
8B
55.00
0.15
49.25
0.15
55.50
0.07
48.67
0.07
$2.88
0.11
GPT-4o-mini
41.50
55.83
54.65
Qwen2.5. 14B Instruct InB-srstuutt
14B
55.55
0.00
59.75
0.25
51.93
0.00
65.53
0.00
56.75
0.05
GPT.4p
55.19
61.25
$7.50
0.03
62.33
0.07
58.42
Deepseek-RI R1
671B
60.19
0.15
$4,50
0.25
53,00
0.00
62,33
0.07
56.96
0.11
LLaMA-332-3B-Insttee Instruct (SFT)
3B
54,81
0.07
60.00
0.20
63.17
0.13
47.33
0.00
$7.45
0.11
LLaMA-3.2-3B 2-3--.----- (Ours)
3B
56.89
0.04
2.25
0.20
65.17
0.10
62.00
0.07
61.39
0.10
77-n--77B-nsttucct (SFT)
7B
59.26
0.07
65.25
0.30
59.83
0.10
54.00
0.00
59.89
0.12
Qwen2-7B-Innaee Insstructtt (Ours)
7B
61.48
0.15
67.75
0.30
66.67
0.07
65,33
0.00
65.16
0.13
F FULL PERFORMANCE OF EFFECTIVENESS ON LOONGFIN
Table 10 presents the full results of the Table T in the main manuscript.
26


Reviews:
[{"review_id": "XwsXuF3L7a", "reviewer": "Reviewer_1HZK", "summary": "This paper introduces Chain-of-Structured-Thought (CoST) a structured prompting method for generated serialized structured outputs (SSOs) along with a verifiable reasoning chain. These outputs are further used to train LiteCost, a model (with two-stage SFT -> GRPO) that integrates this behavior within an SLM, allowing for more efficient inference without significant performance hit. The evaluation is performed on financial data (along with legal). Authors find that CoST -> SSO is an effective mechanism for enhancing reasoning in popular models for long/multi document reasoning.", "questions": "- Have you performed any failure mode analysis when applying this method to domains requiring 'fuzzier' reasoning? I'm curious if the text chunk representation acts as the fallback mechanism for these cases, or if there is still too much loss of information even in this format? Do you have any suggestions on how to make your method generalize to a broader set of applications?  \n\n- Apologies if I missed this, but do you perform any analysis that plots task complexity (table size, document-set size, etc...) versus performance of your techniques (CoST, and LiteCosT)? Are the improvements more apparent over the baselines for harder splits, and have you observed any practical ceiling on what your method can support?\n\nA few presentation notes:\n- Some figures have typos (e.g. 'Strcture', 'Structurlizer')\n- expand 'on-prem'\n- Figure 5: define the axes somewhere\n- Figure 6: perhaps report both values, and add explicit +Δ notations in parentheses.\n- In general, the figures are pretty dense and tricky to follow. There are also four of them. Given your multi-faceted contribution this is fine, but I think some more work needs to be put into readability.\n- It was also sometimes unclear when/whether your analysis was focused on CoST analysis (e.g. using this with off-the-shelf models) versus when your analysis was comparing your trained LiteCost model. The paper is fairly dense read, so it would be great if you can be extremely precise with your wording and the specific components that you contribute (e.g. \"High-quality SSO improves LLM Reasoning\" -- is the SSO high-quality because of CoST, or because of the model used to generate the CoST?)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this work offers a meaningful contribution for their domain, enhancing complex reasoning performance through several methods, both 1) in enhancing off-the-shelf methods via CoST prompting and improved SSO generation and 2) by effectively distilling this behavior into small language models.", "weaknesses": "- While the contributions are clear, I worry that the scope of the work is quite limited. The authors specifically explore the financial domain which has clear structure and a heavy emphasis on numeric reasoning. As noted in the intro, this structured reasoning approach doesn't extend to situations with information that is more nuanced. Similarly, as noted in Appendix C.4, the structured reasoning can be a lossy mapping, impacting performance on non-aggregative / numeric tasks. As such, it seems one must be quite careful in ensuring the applicability of this method to their task. \n\n- Similarly, I am curious if there are cases where a one output structure type isn't appropriate for an example (e.g. perhaps you ideally need a composition of tabular, graph, and textual data for one task), and if there any ways to address these by making the output structure more flexible. In this vein, I'd also like to see how CoST and LiteCoST performance scales with task complexity (e.g., table size, graph breadth/depth, input length).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dEpB066vWR", "reviewer": "Reviewer_ufYT", "summary": "This paper studies document question answering (QA) that consolidates dispersed evidence into structured outputs to support reliable and verifiable QA. Based on this, the authors propose a two-pillar framework, LITECOST, aiming to achieve both high accuracy and low latency with small language models (SLMs). First, they introduce a Chain-of-Structured-Thought (CoST) template—a schema-aware instruction guiding a strong LLM to generate both step-wise CoST traces and corresponding structured outputs. Subsequently, compact models are trained on the LLM-generated CoST traces and structured data via supervised fine-tuning (SFT) and a reinforcement learning approach, GRPO.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a structure-first prompting template that leverages LLMs to elicit step-wise, schema-guided CoST traces and structured outputs from long, noisy documents, producing auditable supervision and machine-checkable results.\n\nThe two-stage training paradigm, involving supervised fine-tuning followed by GRPO, effectively transfers CoST behavior to compact models using reward signals on both answer/format correctness and process consistency.", "weaknesses": "The paper does not clarify how the process reward is computed.\n\nTable 2 only compares a subset of the Loong benchmark with other state-of-the-art models, significantly limiting the validity of LITECOST’s effectiveness claims.\n\nThe novelty of LITECOST is limited; the paradigm of first applying SFT and then enhancing performance via GRPO is already well-established, and the paper does not sufficiently highlight any unique contributions.\n\nIt is unclear whether the reported parameter count of 200 billion for the GPT-4o model is accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LU249mQPwI", "reviewer": "Reviewer_NvTs", "summary": "The paper proposes a method for distilling LLM traces and final structured responses (tables, graphs, etc.) into SLMs. This is in two key steps - (1) schema aware prompting that generates a structured output along with a step by step trace, (2) finetuning with SFT and RL using both outcome and process rewards. Authors demonstrate the performance of their method over other finetuned SLMs, LLMs on the Loong Benchmark legal and finance subsets, while maintaining a lower latency compared to LLMs.", "questions": "Kindly see weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and is easy to follow\n2. At a high level the problem is important -- long document QA with SLMs is a longstanding challenge, and typically SLMs do not perform well on this task.\n3. The method seems sound, there are no obvious technical flaws with the approach. \n4. Based on my understanding of the paper there are two key insights in this work. First, that training an SLM on LLM traces and a structured output that is easy parse significantly improves the SLMs ability to generate the correct answer on long document QA. Second is that a cold start SFT and GRPO with process rewards provides a better learning signal for the SLM. These seem to be sound, nothing unexpected.\n5. The empirical validation is convincing - the reported results on Loong benchmark indicate a significant performance improvement over baselines.", "weaknesses": "1. The contributions of the paper lack novelty. Though well engineered, the contribution of the work lies in applying existing techniques to Loong benchmark and not fundamentally proposing a new strategy. For example: It is well known that cold-start SFT followed by RL training using fine-grained supervision is an effective mechanism for training SLMs (Luong et al., 2024). It might be misleading to present this as a core novel contribution of the paper. \n\n2. Similar to the first point, structured chain of thought prompting has been explored before. Prior works for example (Li et al, 2023) have tried it and confirmed its effectiveness.\n\n3. Details of prompting LLM-based baselines (GPT-4o, GPT-4o-mini, DR1, etc) are not provided. For a fair comparison, both the proposed method and untrained baselines should leverage the same prompting strategy to confirm the effectiveness of training. The authors should also consider prompting LLM-based baselines with structured COT to establish the effectiveness of distillation.\n\n4. Figure 7 compares latency primarily between LITECOST and large closed models. However, a latency comparison against other SLM-based baselines (e.g., Struc-Bench, IEPile) would clarify the efficiency gains.\n\n5. It remains unclear whether the CoST schemas generalize beyond Loong. The paper’s own limitations section (Appendix E) acknowledges the narrow domain focus. Additional evaluation on non-financial datasets (e.g., academic papers or scientific QA) would strengthen the claim of generalizability.\n\n6. The prompting technique in the work is hand-engineered. It should be compared against popular prompt optimization frameworks like dspy, promptim, etc. for a fair assessment. \n\n7. The paper lacks a comprehensive comparison against SLM based baselines on the legal subset of Loong. The main paper emphasizes financial QA results, deferring legal subset results to the appendix. Including a concise summary of those findings in the main text would improve completeness and support claims of cross-domain applicability.\n\n8. The paper could benefit from a clearer comparison to previous “QA-by-structuring” systems such as StructRAG (Li et al., 2024) and StrucBench (Tang et al., 2024), beyond reporting numeric improvements.\n\nOverall:\n\nThis paper addresses an important practical problem: enabling efficient, auditable long-document QA using small models. The experiments show that LITECOST narrows the performance gap between SLMs and LLMs. However, the novelty is primarily incremental, with several well-known components engineered into a working pipeline. Strengthening the baselines, clarifying prompt design choices, comparing the structured prompt with prompt optimization, and demonstrating cross-domain generalization would make the paper more compelling.\n\n[1] Structured Chain-of-Thought Prompting for Code Generation. Li et al\n\n[2] ReFT: Reasoning with Reinforced Fine-Tuning. Luong et al", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WacQO0WskK", "reviewer": "Reviewer_MkZ4", "summary": "The paper proposes LiteCoST, a two-pillar framework for long-document question answering that combines structured reasoning with small-language-model (SLM) fine-tuning. Pillar 1, Chain-of-Structured-Thought (CoST), uses schema-aware prompting to make a strong LLM generate an auditable reasoning trace and a serialized structured output (SSO), e.g., a table or graph, from which the answer is explicitly derivable. Pillar 2 transfers this structure-first behavior to compact models through two phases: SFT to instill schema and format discipline, followed by GRPO using a dual-reward setting targeting both answer correctness and process consistency. Empirical results on benchmarks on finance and legal domains (the Loong benchmark) show that 3B / 7B SLMs trained with LiteCoST substantially improve over base models and even approach GPT-4o performance, while achieving 2 to 4 times lower latency.", "questions": "[Q1] How sensitive is the GRPO process to the weighting between process and outcome rewards? The ablation indicates both are important, but does the balance depend on domain complexity?\n\n[Q2] During CoST trace generation, does the model ever perform schema selection incorrectly (e.g., choosing a graph when a table is needed)? If so, how is this corrected or reflected in the reward?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] Valuable problem and novel idea. The problem of reliable QA over long, noisy documents is valuable because direct LLM prompting often fails. The idea of QA-by-structuring (introducing minimal schemas dynamically per query to improve interpretability and verifiability) is conceptually novel. \n\n[S2] Strong empirical results. On the Loong finance subset, both 3B and 7B LiteCoST variants outperform their base SLMs by +27.6 and +17.8 accuracy points respectively, and even slightly surpass GPT-4o-mini and DeepSeek-R1 while running 2 to 4 times faster. The consistent improvements across multiple subtasks (Spotlight Locating, Comparison, Clustering, Chain of Reasoning) demonstrate the effectiveness of transferring CoST behavior to small models.\n\n[S3] The paper is generally well written and easy to follow.", "weaknesses": "[W1] Limited domain and benchmark scope. While the results on Loong (finance/legal) are strong, the framework has not been tested beyond these domains. Both training and evaluation rely on tabular data; it is unclear whether CoST and LiteCoST generalize to less structured settings (e.g., scientific literature QA, narrative multi-hop QA, or multimodal reports). When relevant evidence cannot be serialized, the \"QA-by-structuring\" assumption may not always hold.\n\n[W2] Complexity and reliance on LLM supervision. Although the paper claims cost-efficiency, generating CoST traces still depends on large-LLM supervision (GPT-4o) for data construction and reward evaluation. The overhead of repeated calls during Stage A and GRPO reward computation could limit reproducibility and offset some of the efficiency gains. A more explicit accounting of total compute (LLM cost + SLM training) would strengthen the argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)