STRUCTSUM Generation for Faster Text Comprehension (2024)

Abstract: We consider the task of generating structured representations of text using large language models (LLMs). We focus on tables and mind maps as representative modalities. Tables are more organized way of representing data, while mind maps provide a visually dynamic and flexible approach, particularly suitable for sparse content. Despite the effectiveness of LLMs on different tasks, we show that current models struggle with generating structured outputs. In response, we present effective prompting strategies for both of these tasks. We introduce a taxonomy of problems around factuality, global and local structure, common to both modalities and propose a set of critiques to tackle these issues resulting in an absolute improvement in accuracy of +37pp (79%) for mind maps and +15pp (78%) for tables. To evaluate semantic coverage of generated structured representations we propose AUTO-QA, and we verify the adequacy of AUTO-QA using SQuAD dataset. We further evaluate the usefulness of structured representations via a text comprehension user study. The results show a significant reduction in comprehension time compared to text when using table (42.9%) and mind map (31.9%), without loss in accuracy.

heading: 1 Introduction
text: The overwhelming amount of information available online poses a significant challenge for users seeking to quickly grasp and process relevant information. Current large language models (LLMs), such as PALM-2 (PaLM2, 2023), Gemini (Gemini Team, 2023) and ChatGPT (OpenAI, 2022), while capable of providing text-based responses to user queries, often fail to adequately structure and organize this information in a way that facilitates comprehension (Tang et al., 2023). This can lead to information processing bottlenecks that hinder
* Work done while interning at Google DeepMind.
eration prompts. The prompting steps are colored . Figure (a) illustrates the divide-and-generate prompt. The input passage is initially segmented into subpassages, followed by the generation of multiple tables. Figure (b) demonstrates the generation process for mind maps. After the main concept has been generated, an iterative expansion phase ensues, during which the mind map is expanded until termination.
users’ ability to efficiently extract meaningful insights from text. To address this issue, we introduce the notion of structured summaries, or STRUCTSUM in short. STRUCTSUMs are derived by hierarchically organizing information and inducing semantic connections from an input text passage. Without loss of generality, we focus on tables (Wu et al., 2022; Li et al., 2023) and mind maps (Buzan, 1996; Huang et al., 2021) as possible STRUCTSUM instantiations:
• Tables are well-studied in the NLP literature. However the vast majority of the work focused on simpler tasks where tables are inputs – such as QA (Herzig et al., 2020), semantic parsing (Bogin et al., 2019), NLG (Andrejczuk et al., 2022; Puduppully and Lapata, 2021; Laha et al., 2020), etc. – rather than outputs. Indeed, faithfully transforming an arbitrary text passage into a table is a difficult task as the model must deal with different challenges, such as reasoning at multi-
7876
ple levels, dealing with missing information, and visually consistent formatting. Motivated by the limitations above, we propose to generate multiple tables instead. We argue that this is a simpler task for an LLM, as shown in Figure 2, which compares single-table and multi-table generation side by side. We therefore propose a divide-andgenerate prompting approach (see Figure 1) that first divides the input text into multiple text passages, each representing a sub-topic, followed by an LLM prompt to generate a table-caption pair for each smaller passage. This decomposition allows the model to generate smaller, focused and more informative tables, especially for complex text passages with multiple sub-topics.
• Mind maps (Hu et al., 2021; Wei et al., 2019) are less studied in the literature, but are helpful for comprehension and learning (Buzan, 1996; Dhindsa et al., 2011). Mind maps are complementary to tables in their structure, allowing for more flexibility and dynamism than tables, as they are inherently schema-less. However, generating mind maps with LLMs presents several challenges: (i) the model first need to select a central concept, that is the fulcrum of all the successive extractions, as mind maps revolve around a central root node; (ii) being a schemaless abstraction, each connecting branch has its own independent sub-topic, making it difficult to automatically add branches all at once; (iii) to ensure readability and well-structuredness each leaf node should terminate the path in a way that concludes the idea or sub-topic; (iv) depending on the information density, some paths may be shorter than others. Therefore, the model should decide whether or not a branch is worth expanding. Following the structure of these observations, we propose an iterative prompting technique for mind map generation. As show in Figure 1, we initialize the mind map by generating the root concept. At each iteration, we decide either to expand the current mind map further or stop the process. During the expansion step, we prompt the model to add branches to the current leaf nodes. We represent the mind map as a JSON object, as it is easy to parse and verify.
Through extensive experimentation with PALM2 (PaLM2, 2023), we show that LLMs are not always effective at generating STRUCTSUMs that are factual and structurally correct. To overcome these
issues we propose a pipeline for structured data generation. Our pipeline consists of structure-specific prompts followed by critics to assess output quality along three different dimensions, that are common both to tables and mind maps: (i) Factuality, (ii) Local Structure and (iii) Global Structure. We found that our proposed critics improved the overall quality of the generated output by +37pp for mind maps and +15pp for tables.
To ensure the usefulness of STRUCTSUM for text-comprehension tasks, we propose Auto-QA as a measure of output coverage. We automatically generate QA pairs from input text and use structured outputs to answer these questions. Furthermore, we verify the appropriateness of using Auto-QA by comparing Auto-QA with human generated QA pairs on SQuAD (Rajpurkar et al., 2016) development set.
Finally, starting from the initial hypothesis that STRUCTSUMs can enhance the effectiveness of information-seeking scenarios, we conducted a user study to evaluate their impact on users’ ability to process information, using a text comprehension user study. Results demonstrate how STRUCTSUMs improve information seeking, specifically on timed text comprehension metrics. We found that by using the structured representation, users can answer questions 42.9% faster for tables and 31.9% for mind maps.

heading: 2 Related Work
text: Structured Output. Generating structured output from text has been explored in the context of information extraction (Li et al., 2023; Pietruszka et al., 2022). Most of the work focus on textto-table (Wu et al., 2022) generation using the model trained on domain specific dataset. Ni and Li (2023) use LLM for information extraction by generating key-value pairs. Tang et al. (2023) evaluate different models on table generation from text by prompting where table structure is provided as format instructions. Mind map generation has been explored in the form of relation graph structure (Hu et al., 2021; Wei et al., 2019) to summarize new articles (Cheng and Lapata, 2016; Hermann et al., 2015). In contrast, we focus on a generation pipeline applicable for multiple structured outputs types by prompting LLM given a text input. We keep the output structure flexible and domain independent by not instructing the model with specific format.
Prompting. Our prompting strategy is rooted in task decomposition techniques. Least-tomost (Zhou et al., 2023), in contrast with chain-ofthought (Wei et al., 2022), progresses from easiest to hardest questions eventually answering the complete question, while successive prompting (Dua et al., 2022) iteratively generate new questions based on previous answers. Unlike least-to-most, decomposed prompting (Khot et al., 2023) doesn’t restrict task decomposition from easiest to the hardest and iteratively generate next steps that can be executed by different systems. Most of the prior work is focused on reasoning for solving QA type problems, in contrast, we are interested in transforming text to structured formats. Our divide-andgenerate prompting for multiple table generation (similar to least-to-most) uses an initial prompt to divide the input passage into different topical sub-passages that simplifies the table generation in next step. Different from these tasks our iterative prompting for mind maps requires reasoning over current structured output at each step.
Factuality. Attribution is used as a tool for assessing the reliability of LLMs and identifying potential sources of inaccuracy or fabrication in their generated outputs. Current work apply attribution on unstructured text generation settings, such as, question answering (Bohnet et al., 2022) and text generation tasks (Gao et al., 2023a). Diverging from that, our work require verifying the factuality of generated structured outputs.
Evaluation. Due to the cost of human evaluation, LLMs are used to critique the generated outputs (Wang et al., 2023). Recent instructions tuned models, such as, GPT-4 (OpenAI, 2023) and Chat-
GPT (OpenAI, 2022) are shown to be strong evaluators. To avoid using external APIs, Kim et al. (2023); Wang et al. (2023) fine-tune a smaller pretrained model to critic model responses. We are interested in evaluating the quality structured outputs using critics and self-correct based on the feedback. As a part of data generation pipeline, our focus is on filtering instances that are incomplete and are not factually grounded.

heading: 3 Generating STRUCTSUMs
text: We focus on tables and mind maps as a possible STRUCTSUM instantiations.

heading: 3.1 Tables: Divide & Generate prompting
text: Given an input text we would like to transform it into multiple tables. Although generating multiple tables from text may seem unnecessary, singletable generation lead to several issues, as shown in Figure 2 (bottom left). The model often produces complex table structures, resulting in missing cell values or the exclusion of relevant information. Additionally, complex tables are difficult to verify for factual accuracy and can require additional mental effort from the user to understand.
To address these limitations, we propose a divide-and-generate approach that dynamically partitions the passage into smaller subtopic segments. While deterministic rule-based chunking methods (e.g., based on word or sentence count) can be employed, they often produce suboptimal results due to potential under-chunking, over-chunking, and the absence of division for certain instances. Therefore, the chunking must be adaptive and depend on the input text and its sub-topic distribution. We use a one-shot prompt for this step, as shown in
Appendix B (Figure 14). After the chunking, we prompt the model to generate a table along with its caption for each sub-passage obtained in the previous step.
Algorithm 1 Mind maps Iterative Prompting Require:
input text passage: input maximum number of steps: max_steps
1: step← 0 2: mindmap← GENERATE-ROOT(input) 3: while step < max_steps do 4: step← step + 1 5: if CONTINUE-PROMPT(input, mindmap) then 6: expansions← EXPAND(input,mindmap) 7: mindmap← JSON-CRITIC(expansions) 8: else 9: return mindmap
10: end if 11: end while 12: return mindmap

heading: 3.2 Mind maps: Iterative Prompting
text: Contrary to tables, mind maps (see example in Figure 3) are more flexible and present a different set of challenges. The first challenge is representation. We desire a representation that is (i) close to a familiar format, and (ii) is easily parsable and verifiable using current tools. JSON meets both of these requirements. The second challenge is that mind maps, unlike tables where each row can be produced linearly, necessitate attaching information in different locations depending on which branch is being expanded. This requires the model to think radially.
We propose an iterative prompting for mind maps generation. Algorithm 1 shows the overall procedure. Details of each prompt is in Appendix B. We start by generating the root concept that becomes the central node for the mind map. This
separate step allows the model to independently reason about the theme of the passage. After generating the root, at each step we prompt the model to decide if current mind map can be expanded further. If the model decides to expand (line 5), we prompt the model using the current mind map to add more branches. Otherwise, the procedure terminates and we return the current mind map. At each expansion step we sample multiple mind maps. Utilizing the fact that JSON verification is cheaper we select the topmost JSON that is parsed correctly. In the rare case, when none of the samples are parsable we call a critic prompt to correct the top JSON (line 7).

heading: 4 Data Generation Pipeline
text: We now present our STRUCTSUM data generation pipeline. Although each STRUCTSUM is seemingly different, we identify three dimensions that are common to both table and mind map modalities: (i) Factuality, (ii) Local Structure, and (iii) Global Structure. We use a set of critics, implemented via prompts, to ensure sufficient quality across each dimension. Through our initial experiments we find that tweaking each critic according to the structure is more helpful.

heading: 4.1 Factuality Critic
text: We use post-attribution (Gao et al., 2023a) to verify factuality, as we found that jointly generating and attributing (Gao et al., 2023b) results in (i) unnatural text output and (ii) in the model copying verbatim from the input text passage.
Critic cost is one aspect that requires consideration. For example, for tables, verifying each cell could be more robust, however, it increases the number of LLMs calls (listed in Table 1), from O(1) to O(#number of cells).
For simplicity, we choose a single prompt per STRUCTSUM: for tables we ask the LLM to attribute each row, while for mind maps we ask to attribute each path from root to leaf. We convert the input text passage to a list of sentences and ask the model to cite, following the [x,y] format for attribution, the source sentence(s) where the information can be found or [NA] in case this is not possible. The prompts are reported in Figure 15.

heading: 4.2 Local Structure Critic
text: For tables, a common issue arises from the model misplacing values in incorrect columns. For example, placing “66 years” in the Birth date column or
an address in the Company Name column. To detect such errors, we leverage each column header as a category and verify whether all cell values within that column belong to the same category. For mind maps, we observed that a well-defined terminal node can often represent the entire path leading to it. We use this fact and prompt the model to verify whether the terminal node is a specific value, rather than a general concept. The prompts are reported in Figure 16.

heading: 4.3 Global Structure Critic
text: Global critic allows us to verify the overall structure of the output. This means understanding whether all the information contained in a STRUCTSUM makes sense globally. For tables, we simply verify whether the table is well formatted: e.g. we verify equal number columns in the header and subsequent rows, therefore ignoring semantic content of the table and only focusing on form rather than the content. This is realized via simple heuristics implemented in Python (we do not prompt the model for these). For mind maps, we used a stricter approach, to ensure that information were semantically valid on a global level. Specifically, we convert the mind map into a familiar format like table of contents (ToC), which we hypothesize is more likely to be seen during the pre-training phase of existing LLMs, and ask the model to check if the ToC is at right level of abstraction. The prompts are reported in Figure 17.

heading: 5 Semantic Coverage using AUTO-QA
text: In this section we propose an automatic way to assess the quality and the general usefulness of STRUCTSUMs introducing AUTO-QA coverage as proxy metric. 1 This metric measures the semantic
1We do not present AUTO-QA as a substitute for human evaluations of quality. Instead, we propose AutoQA as a coverage metric that allows us to use synthetic question generation.
coverage or percentage of questions that are answerable when using a STRUCTSUM s, instead of the full text passage t. Formally it is defined as:
COV (s) = 1
|GenQA(t)|
|GenQA(t)|∑
i=1
1Eai [Q(s, qi)]
where GenQA(x) is a function that generates (q, a) pairs given the input text passage t, Q(s, qi) is a function that generates an answer given in input a STRUCTSUM s and the question qi, whereas the indicator function 1Eai (x) asses the answer equivalence between ai and x. Figure 18 in Appendix B, show all the prompts associated with AUTOQA module (Deutsch et al., 2021; Fabbri et al., 2022).
Independently of perceived quality, it is worth noting that this simple metric can be thought as an abstractiveness measure or compression quality for a given STRUCTSUM s. A value of 1 indicates no information loss at the expense of no compression/abstraction, whereas a value of 0 indicates theoretically maximum compression at the expense of not providing any useful information. A target value is therefore application specific and must be adjusted accordingly 2.
QA pairs generation GenQA(t) is implemented by prompting the LLM to generate a list of question-answer (QA) pairs conditioned on the input text t. To ensure that the quality of QA pairs is sufficient, after generation, we we apply a threestep procedure. First, we removed duplicate questions via string match. Second, we removed answers if none of the words appeared in the input text, thereby ensuring with reasonable certainty that the answer is grounded in the text without being overly stringent. Third, we performed a cyclic consistency check, where we prompted the model to answer the generated question based on input text.
Question answering We use a simple prompt for function Q(s, qi). For tables, we convert the table representation to a markdown table format, whereas for mind maps we simply serialize the information as a JSON object.
Answer Equivalence As the model might generate verbose answers, verifying whether two an-
2It is possible to include coverage as a critic. But we opted not to do so, as the threshold for coverage depends on the specific use case. This also allowed us to analyze coverage independently, without being influenced by other factors.
swers are the same is a problem of semantic similarity. Instead of using lexical matching, that is 1Eai
(x) := ai = x, we prompt the model to check if two answers are equivalent.

heading: 6 Model
text: For all the experiments, we use the Unicorn (PaLM2-unicorn, 2023) variant of PALM-2, a fine-tuned transformer-based model with UL2 (Tay et al., 2022) like objectives. PALM-2 improves on PaLM (Chowdhery et al., 2023) through optimized scaling, richer training data and instructing tuning (Wei et al., 2021; Chung et al., 2022).

heading: 7 Dataset
text: To test our pipeline on a diverse set of input passages, we selected Wikipedia text as the source. Specifically, we started with the English split of the WIKI40B (Guo et al., 2020) dataset 3. The dataset is cleaned up by page filtering to remove disambiguation pages, redirect pages, deleted pages, and non-entity pages. Input to our prompts are passages that are obtained by splitting the Wikipedia text using the _START_PARAGRAPH_ symbol that is already provided as part of the dataset.

heading: 7.1 Filtering for Tables Generation
text: Not all input paragraphs are well suited for table generation. As a proxy for selecting adequate passages, we used regex-based filters to only include passages with more that 20 numeric values and removed passages with less than three sentences. In a real world setting, we would like a systematic way of deciding which modality is adequate for a given text. We leave this exploration as future work.

heading: 8 Results
text: In this section, we present the results of our experiments using PALM-2.

heading: 8.1 Quality impact of prompting style and automated critics
text: We assessed the quality of generated structured data through manual human ratings. The study was conducted on 100 instances for mind maps. For multi-table generation, we choose 100 individual
3We used the version that is available via the Tensorflow datasets https://www.tensorflow.org/datasets/ catalog/wiki40b.
table-text pair for annotation 4. Input passages were obtained via data filtering strategy described in Section 7.
Guidelines Annotators were asked to rate each instance as “Good“ or “Bad” by checking the overall quality of the output. For both modalities, annotators were asked to check for factuality as well as the structural quality of the output. To help the annotators measure the structural quality we asked the annotators to check “table structure”, “table header”, “column header-value match” for tables. For mind maps, they were asked to check “incomplete branches”, “not a good main concept”, “too dense / too sparse” and “wrong edge connections”. We also encouraged the annotators to mark the instance as bad if they find any other issues.
Prompt style Table 2 show the results for both the modalities. For table generation task we find that annotators prefer multiple tables generation outputs compared to single table generation. This can be attributed to the fact that multi-table generation enables the model to generate more concise, focused and informative tables. For mind map, we compare chain-of-thought (Wei et al., 2022) with our proposed iterative generation strategy described
4We made sure that the input passages are the same for the different ablations within modalities. For multi-table generation, we choose 100 text-table pairs generated using 52 input passages.
in Algorithm 1. We find that iterative generation were preferred over simpler prompt outputs.

heading: 8.2 Do Critics Align with Human Ratings?
text: Through our human annotations results in Section 8.1, we find that many generated outputs are not of acceptable quality. To improve the quality of the generated data and to avoid costly human annotations, we propose to use a combination of critics as a measure of data quality. To verify the efficacy of our critics, we first filtered the generated dataset with our critics. Specifically, we performed a logical AND of individual critics and filtered the instances that do not pass the criterion. We then sampled 100 instances from filtered examples and conducted the same evaluation as in Section 8.1.
Results in Table 3 show that using the proposed critics the overall quality is improved by a significant margin. We observe that data filtered using Structure (Global and Local) and Factuality critics improve the percentage of acceptable instances generated using the pipeline. We find that the quality of mind maps improve by absolute +37pp. Similarly, for tables quality improves by absolute +15pp. These results indicate that the critics were able to retain good examples and that the selection criterion is in agreement with human judgement.

heading: 8.3 Measuring Coverage via Auto-QA
text: Results in Figure 4 show AUTO-QA coverage for mind maps and tables. The curve shows for a particular coverage threshold what percentage of data meets that threshold. Overall, we observe that tables have better coverage compared to mind maps, meaning that they have an higher abstractiveness or information retention capacity. Interestingly, even
though both modalities are perceptually different, we notice that both of them follow similar trends.

heading: 8.4 Is Auto-QA a reasonable metric?
text: We investigate the feasibility of using AUTO-QA as a surrogate for manually written QA pairs. We aim to determine whether AUTO-QA can generate QA pairs of comparable quality to those written by humans, and leading to a similar evaluation of semantic coverage. To verify the same, we use randomly selected 1000 <passage, question, answer> triples from the SQuAD (Rajpurkar et al., 2016) validation set (common for all the experiments). Using the text passage as input we generate different STRUCTSUMs. Next, we generate a QA pair corresponding to each text passage. This QA pairs acts as a substitute for human written QA pair for AUTO-QA study. The goal is to check whether, keeping the passage and output STRUCTSUM the same, there is a correlation in performance between human generated QA pairs and automatically generated QA pairs.
Table 4 shows the overall results. Second (Mind maps) and third (Multi-Table) row show the comparison between Human QA and Auto QA for our proposed divide-and-generate prompt for tables and iterative prompt for Mindmap generation. We can see that AUTO-QA has comparable results and is a reasonable substitute for human generated questions as a measure of semantic coverage. We further study the limitations of AUTO-QA, and the difference in Human vs Auto QA scores in Section 10.

heading: 8.5 Multiple Tables vs Single Table
text: To check whether generating multiple tables is better at covering more information, we perform a comparison between the ability to answer questions by generating single or multiple tables. On
comparing Multi-Table and Single Table row in Table 4, we observe that for both AUTO-QA and Human QA generating multiple table provides more coverage. So in addition to the benefits such as comparatively better verifiability and robust generation, multiple table generation are also better at covering more semantic information.

heading: 8.6 Query Focused Generation
text: In many cases user intent is known in advance, for example, a user query to search or LLM-based Assistant interface (e.g., ChatGPT, Gemini, etc.). We explore the possibility of generating structured data in the presence of a query. We perform a preliminary analysis by adding the query in single table generation prompt. As we can see in last row in Table 4, query focused generation improve the performance by more than 20 for AUTO-QA and 25 points for Human-QA. Since this requires further investigation in terms of prompting and output quality analysis, we leave a comprehensive exploration of query-focused structured data generation as future work.

heading: 8.7 Are STRUCTSUMs useful?
text: We evaluate whether STRUCTSUM are useful abstractions for the users. For this we design a timed text comprehension based user study. We assume that user is looking to answer a specific query, i.e. has a specific intent. We measure time taken to satisfy the user intent as a proxy of usefulness. Our evaluation team consists of 12 volunteers who are
affiliated with our institution. Five of these volunteers are female, and seven are male. All volunteers are proficient in English, although not native speakers.
We create an intent in the form of a question along with different context combinations. For example, for a question q, we create 〈q, s〉, 〈q, t〉, 〈q, s + t〉 as possible combinations, where s is a STRUCTSUM and t is the input text passage. Each of these combinations are presented to different annotators while ensuring that no annotator see the same question twice. We then measure how long it takes to answer the question in each scenario.
STRUCTSUMs for the study were generated using our data generation pipeline and critic-based filtering, as discussed in Section 4. In total 600 instances were annotated, equally divided into different context combinations for mind maps and table generation. Annotators consistently answered correctly across all context combinations (Appendix A), suggesting that the level of context did not significantly impact their accuracy. Figure 5 shows the overall results. The plots show 95% confidence interval of time taken by the annotators when using different modality: • Tables. Figure 5a shows that on average anno-
tators with access to tables were able to answer almost 42.9% time faster on average compared to annotators with only text. Furthermore, we observe that presenting both table and text is also useful to the annotators. • Mind maps. Figure 5b shows the results for
the study with mind map. A similar trend can be observed, with a reduction of approximately 31.9% in average time between annotators with mind maps compared to annotators that only used text to answer the question. We note that 〈q, s+ t〉 performs worse than 〈q, s〉. We believe this is due to the fact that the annotators cross-checked the answer from both the modalities, leading to increase in time to answer the question.

heading: 9 Data Generation statistics
text: Table 6 shows different statistics of data generated using our prompts. For tables generation we observe that our methods generate almost two (∼ 1.9) tables per instance and the tables have 7.1 rows and 3.3 columns on average. Mind maps have an average of 11.8 nodes with a depth of 2.2. We show example mind map and table generation in Figure 7 and Figure 6 respectively.

heading: 10 Limitations of AUTO-QA
text: We propose AUTO-QA as a coverage metric, in order to measure how much information from the original passage is retained in the STRUCTSUM.
Note that the metric is not intended as a substitute for evaluations of overall quality. The primary benefit of AUTO-QA is the synthetic question generation component, which can be run at scale without costly human annotations. In Table 4, we measured how closely the synthetic AUTO-QA aligns with human generated question-answer pairs, revealing certain limitations of the metric. Notably the quality of generated questions is not always reasonable, which is mitigated using specialized filters, as discussed in Section 5. Although we generate several question-answer pairs, it is possible that none pass the filters, which we see for fewer than 10% of input passages. Such instances adversely affect the AUTO-QA coverage score, contributing to the score discrepancies in Table 4. When the analysis is restricted to those subsets where the AUTO-QA filtering is successful, the observed differences are diminished and fall within the bounds of experimental noise, as reported in Table 5.

heading: 11 Conclusion
text: In this work we study the potential of structured representations like tables and mind maps to enhance information comprehension. Utilizing our divideand-generate prompting and iterative expansion, we achieved significant improvements in output quality (+37pp for mind maps, +15pp for tables) using structure-specific prompts and critics. We proposed AUTO-QA based coverage metric that automatically generates QA pairs from the input text and uses STRUCTSUM outputs to answer them.

heading: 12 Acknowledgment
text: We would like to thank Massimo Nicosia and Yasemin Altun for their feedback and comments on the paper. We are grateful to all the volunteers who participated in our annotation experiments.

heading: 13 Limitations
text: We outline the limitations of our work to ensure transparency and inspire future research. First, the structured output representations we experimented with are limited to tables and mind maps. However, to comprehensively evaluate the effectiveness of our critics and pipeline, it is desirable to also evaluate other input and output modalities, e.g. image and video, considering the recent advances in VLMs. Secondly, our work and experimental findings are limited to only English sources. We plan to also explore multilingual structured summaries in future work. Third, we would to warn against the risk of blindly trusting models to generate structured summaries from an input accurately. Although we take extra care to increase the factuality of the outputs via the use of critics, and experimentally validate QA coverage, we believe that special care should be taken to verify outputs in accuracysensitive applications. Finally, our STRUCTSUM generation is performed using a LLM with fixed prompts, however, prior work have shown a reasonable portability of prompts across similar models (Zhou et al., 2023; Khot et al., 2023). Despite these limitations, our work serves as an initial step in constructing reliable structured summarization evaluations, models and applications. We hope future research can greatly benefit from this starting point.

heading: A User Study
text: Usually, mind maps are represented as a graph as shown in Figure 7. However, for the text comprehension user study described in Section 8.7, to avoid bias due to color or orientation, we simplify the representation as a tree (Figure 10). To establish the known query intent, annotators’ are first shown with input question, e.g., Figure 8. Next, on clicking Show content button, annotators are shown context in the form of either text (Figure 9), structure (Figure 10), or structure + text (Figure 11). The question-answer pairs were generated automatically conditioned on input text (Section 5). Annotators were also allowed to mark an instance un-answerable. The user study for tables is performed in a similar manner. We annotated 100 question-answer pairs for both mind maps and tables. Each input instance is annotated with three different context combinations, leading to 600 total annotations. We filtered instances that were marked un-answerable by the annotators (32% and 22% for tables and mind map study resp.). To avoid penalizing for spelling errors or other typing mistakes, the answers were evaluated via human evaluation.
We adopt timed-comprehension for answering free-form questions as a proxy measure for usefulness of generated structured representation. This makes it different from categorical data annotations. We calculate rater agreement for the questionnaire responses. We find that 89.9% of questions had full rater agreement regarding the correct response, with the only differences in the time taken to respond. Each question was shown to three raters. Table 7 shows the overall accuracy as percentage of questions answered correctly in different context. Irrespective of context combinations, annotators were able to answer the questions correctly with a high accuracy.
Mind map 97.7 Text 94.3 Mind map+Text 97.7
Table 7: Answer accuracy (as percentage) for different context combinations. Structure context performs on par/better compared to text.

heading: B Prompts
text: We include the different prompts used in this study. In our implementation we use Jinja (https:
//jinja.palletsprojects.com/) to specify the prompt template. The prompts can be found in Figures 12 to 18.
The Mersey-class cruisers were improved versions of the Leander class with more armour and no sailing rig on a smaller displacement. Like their predecessors, they were intended to protect British shipping. The cruisers had a length between perpendiculars of 300 feet (91.4 m), a beam of 46 feet (14.0 m) and a draught of 20 feet 2 inches (6.1 m). They displaced 4,050 long tons (4,110 t). The ships were powered by a pair of two-cylinder horizontal, direct-acting, compound-expansion steam engines, each driving one shaft, which were designed to produce a total of 6,000 indicated horsepower (4,500 kW) and a maximum speed of 18 knots (33 km/h; 21 mph) using steam provided by a dozen cylindrical boilers with forced draught. The Mersey class carried enough coal to give them a range of 8,750 nautical miles (16,200 km; 10,070 mi) at a speed of 10 knots (19 km/h; 12 mph). The ships’ complement was 300 to 350 officers and ratings. Their main armament consisted of two breech-loading (BL) 8-inch (203 mm) guns, one each fore and aft on pivot mounts. Their secondary armament was ten BL 6-inch (152 mm) guns, five on each broadside in sponsons. Protection against torpedo boats was provided by three quick-firing (QF) 6-pounder Hotchkiss guns and three QF 3-pounder Hotchkiss guns. The ship was also armed with a pair of submerged 14-inch (356 mm) torpedo tubes and carried a pair of 14-inch torpedo carriages. The Mersey-class ships were protected by a lower armoured deck that was 2 inches (51 mm) on the flat and 3 inches (76 mm) on the slope. It sloped down at the bow to reinforce the ram. The armoured sides of the conning tower were 9 inches (229 mm) thick.
Kathleen "Kay" Daly (January 8, 1919 – October 16, 1975) was an Irish-born American advertising executive and one of the four "celebrated Daly sisters". At Norman, Craig & Kümmel she was the creative force behind the famous Maidenform "I Dreamed ..." campaign and Revlon’s legendary 1952 Fire And Ice campaign, working with photographer Richard Avedon. She also was responsible for the line ”Every woman alive loves Chanel Number Five". She went on to join Revlon in 1961 as vice president and creative director. Kathleen Daly was born in Castlecaufield, County Tyrone, Ulster, Ireland, in 1919. Northern Ireland was created two years later with Tyrone one of its six counties. The family emigrated early in the 1920s. She grew up as one of four sisters, Maggie, Kay, Maureen, and American-born Sheila. They became known for their writing and work in journalism, fashion, and advertising, and were called "the celebrated Daly sisters" by Time magazine in 1966. Life magazine ran a feature story on them in 1949 and a follow-up in 1959. All four were at least once employed by the Chicago Tribune. When she moved to San Francisco after World War II, Kay Daly famously rented space on a billboard to advertise for an apartment. It not only netted her an apartment, but netted her nationwide fame and countless marriage proposals. She had a brief marriage to BMW executive and film producer Richard Bradford (part of the famous Bradford family of Plymouth Colony), who fathered her sons John (Kelly), Richard, and Peter. She then was married to journalist and executive Warren Leslie, who adopted and raised her sons, until her death on October 16, 1975, of pancreatic cancer. She was survived by husband Warren, sons Kelly, Peter, and Richard Bradford, and stepsons Warren and Michael Leslie.
(a) Input text for mind map generation.

heading: Future Work
text: We outline the limitations of our work to ensure transparency and inspire future research. First, the structured output representations we experimented with are limited to tables and mind maps. However, to comprehensively evaluate the effectiveness of our critics and pipeline, it is desirable to also evaluate other input and output modalities, e. g. image and video, considering the recent advances in VLMs. Secondly, our work and experimental findings are limited to only English sources. We plan to also explore multilingual structured summaries in future work. Third, we would to warn against the risk of blindly trusting models to generate structured summaries from an input accurately. Although we take extra care to increase the factuality of the outputs via the use of critics, and experimentally validate QA coverage, we believe that special care should be taken to verify outputs in accuracysensitive applications. Finally, our STRUCTSUM generation is performed using a LLM with fixed prompts, however, prior work have shown a reasonable portability of prompts across similar models (Zhou et al. , 2023; Khot et al. , 2023). Despite these limitations, our work serves as an initial step in constructing reliable structured summarization evaluations, models and applications. We hope future research can greatly benefit from this starting point