Hierarchical Catalogue Generation for Literature Review: A Benchmark
Authors: kun Zhu
Xiaocheng Feng
Xiachong Feng
Yingsheng Wu
Bing Qin
Venue: EMNLP 2023 Findings (2023)
Keywords: scientific document processing
multi-document summarization
datasets
metrics

Abstract: Scientific literature review generation aims to extract and organize important information from an abundant collection of reference papers and produces corresponding reviews while lacking a clear and logical hierarchy.
We observe that a high-quality catalogue-guided generation process can effectively alleviate this problem.
Therefore, we present an atomic and challenging task named Hierarchical Catalogue Generation for Literature Review as the first step for review generation, which aims to produce a hierarchical catalogue of a review paper given various references. 
We construct a novel English Hierarchical Catalogues of Literature Reviews Dataset with 7.6k literature review catalogues and 389k reference papers. To accurately assess the model performance, we design two evaluation metrics for informativeness and similarity to ground truth from semantics and structure.  Our extensive analyses verify the high quality of our dataset and the effectiveness of our evaluation metrics. We further benchmark diverse experiments on state-of-the-art summarization models like BART and large language models like ChatGPT to evaluate their capabilities. We further discuss potential directions for this task to motivate future research.

Hierarchical Catalogue Generation for Literature Review: A Benchmark
Kun Zhu', Xiaocheng Feng'1, Xiachong Feng', Yingsheng Wu', Bing Qin Peng Cheng Laboratory
"Harbin Institute of Technology xcfeng, xiachongfeng, yswu, qinb]@ir hit. edu. cn
Abstract
Pioneer studies on scientific literature review generation explore citation sentence generation (Xing et al., 2020; Luu et al., 2020; Ge et al., 2021; Wu et al., 2021) and related work generation (Hoang and Kan, 2010; Hu and Wan, 2014; Li et al., 2022; Wang et al., 2022). However, these methods can only generate short summaries, while a litera- ture review is required to provide comprehensive and sufficient overview of particular topic (Web- ster and Watson, 2002). Benefiting from the devel- opment of language modeling (Lewis et al., 2019), recent works directly attempt survey generation (Mohammad et al., 2009; Jha et al., 2015; Shuaiqi et al., 2022) but usually suffer from a disorganized generation without hierarchical guidance. As il- lustrated in Figure I(A), we take gpt-3.5-turbo', large language model trained massive amounts of diverse data including scientific papers, the summarizer to generate scientific literature reviews. Direct generation can lead to disorganized reviews with content repetition and logical confusion.
Scientific literature review generation aims to extract and organize important information from abundant collection of reference pa- pers and produces corresponding reviews while lacking a clear and logical hierarchy. We ob- serve that high-quality catalogue-guided gen- eration process can effectively alleviate this problem. Therefore, we present an atomic and challenging task named Hierarchical Catalogue Generation for Literature Review as the first step for review generation, which aims pro- duce hierarchical catalogue of review paper given various references. We construct a novel English Hierarchical Catalogues of Literature Reviews Dataset with 7.6k literature review cat- alogues and 389k reference papers. To accu- rately assess the model performance, we de- sign two evaluation metrics for informativeness and similarity to ground truth from semantics and structure. Our extensive analyses verify the high quality our dataset and the effec- tiveness of our evaluation metrics. We further benchmark diverse experiments on state-of-the- art summarization models like BART large language models like ChatGPT to evaluate their capabilities. We further discuss potential direc- tions for this task to motivate future research.
Since hierarchical guidance is effective for text generation (Yao et al., 2019), as shown in Figure I(B), we observe that the catalogue, representing the author's understanding and organization of ex- isting research, is also beneficial for scientific liter- ature review generation. Therefore, the generation process can be divided into two steps. First gen- crating a hicrarchical catalogue, and next cach part of the review. Figure ICC) reveals that even state-of -tate-of-the-ard language models can not obtain re- liable catalogue, leaving valuable and challenging problem for scientific literature review generation.
1 Introduction
Today's researchers can publish their work not only in traditional venues like conferences and journals but also in e-preprint libraries and mega-journals, which is fast, convenient, and easy to access (Fire and Guestrin, 2019). This enables rapid develop- ment and sharing of academic achievements. For example, statistics show that more than 50,000 pub- lications occurred in 2020 in response to COVID- 19 (Wang and Lo, 2021). Therefore, researchers are overwhelmed by considerable reading with the explosive growth in the number of scientific papers, urging more focus on scientific literature review generation (Altmami and Menai, 2022).
To enable the capability of generating reason- able hierarchical catalogues, we propose novel and challenging task of Hierarchical Catalogue Generation for scientific Literature Review, named as HiCatGLR, which is a first step towards au- tomatic review generation. We construct the first benchmark for HiCatGLR by gathering html for-
'A version of ChatGPT, openai. cow.blog//chagppt

---

Till: A Survey of Large Longuage Models
Oracle Catalogue Introduction
[B) Given orodle Catalogue
[C] Given generated Catalogue 3
A) Generate the roview directly
2 Overview
Intnodacion
Lang argeane motes (ML as pro-rinnnd on
Irkostacion
Larae Lanzuaze Modsle (L.1M) are peverful
Introdactonn
Resources of LLMs
specific
Lenging Procesing (NI.2) teols
NLP.
Me ruch GPT-3 hase neockrtenizad of
langaay undontssding
Pre-training
Ressurees of ULSAs
interal procnsing (NIP).
Dewelgment of LUbs
insceiiggy
sppeilitee....
4.2 Archilecture
LLMis ace publicll available ctrorat matel chesiporate se Alta Commonty saze corpera
Eatly lansusee medele sect as Hidden Marion Midels and Noau modal lacked the expacity to lightre
bevelammen and Traening of LLMss
4.11 Marsiroam Aechitectrees
Like are developod bong doip leaning tncrizzes, each
Large laeplege koading the devdopeact of tozalomnon baicd LLM
Aogl
in nenmi networts, tua cam from larg arreent can..
4.33 Prediaiiing Tesks
4.3. Bammary and Bioussion
accos Likb and fet
Biggling Froe,
LLMs
dayloyel wile
Fro traitiinngg
uec as car7-
including toxt sonemation, madine imsalation, sentimere
Adaptation Tuning LLMs Utilization
Bera collection.. M1Ms employ maleetmoon
Types LLMb
anulyiss queeion ennnerng and stattont.. Ertigue MMMe
7 Capacity Evaluation
ralling tachiium tebs oparniio key modek...
model
ingeesseee perto axonon,
6 Conclusion
Adaciation Tuning of LLMf
dela-dkiect appeoackes geazrate lest warrees
Hivieel nember of
.. Alganneet taiing cino clige mode extpals
soed laggaas modek use deep loaning arcenncunce acc Rexrment Nerra Netwerts and Tranetemen
thil.
kith hamas expectation through releforcement Utliization
lnakin is basel suppes Eaere io alro 1eak des LtM
Generated Catalogue
sodele as greenoe
2x1 ba usdd naurratt tennfrenstion fikn Deis.
> Introduction
paiing
Gaps in Knowledge
ntploseent, petrucy, and faimace.
Thane machis fom ahout LLMs inchadity how-
stlasel
sooeasiegly yewertl they
prictiaa
v3 Types of LLMs
>2 Development of LLMss
4.1 Statistioal Longuage Models
Negram Modals
mecturon the adaple LLSt to uner oucroneleeenee.
Eizzaat-t proeptina allons LLMe to unlinc
In In-suetest lecrning invelves onder lyigg
kndkyth browldgg holg
preceation
Asplicattora of LiMMs
For www.decchisiii Artifcaa Eocoed beefigncc
1MMs lave lece wed dookly laagaage rodel
knowledge
langnnges,
(bey cos efficciiell finecuuned differest teaks and
immnow thar mitepertailitty mrmerranccy
4.9 Bute-Baeed Largunge Modele
4.1. Tranefomee bated bloreis
Capuaity Evaluation
skgnnen, sncraciis with csteral cav ivement
Adhance abiliyy evaluation inooleex buran
teraads mikiing dats mostis findiation
Chu lenges of LLMs
Oes key dhaleege LLM is thitr compatational
Additorulty there is need for more reccach oqtines bh
4.3 Iruindatty flavet Sastorn
eusipelatoo Concistion Futurm Disccions
socal and citical mepliaatios of Lib,, inciddig thair
>-4 Applications of LLMs
Futuro Directino
repect clncttion yourell deneanses
5 Challenges of LLMs
Futrre rewandd to snqured to adloss dse Lanitaiinn and challonges LMMs incleding eraiical coepaaastona ufficeacy...
LLMi hove becune cratal corpenest of Nu?
imsperring the officiency and scalabusity and opability ofroswning bwyord beggage Concassoo
conduaiion LLMe havd emeraed poncrtal in
Future Directions
MLPand have born wakks adopted in vanso Sckb...
Conclusion
L.MS lane beonse cwartd sods
Figure I: Examples of scientific literature review generated by gpt-3.5-turbo, given the title "A Survey of Large Language Models." (A) represents direct generation with the title only. Duplicated contents (underlined) exist two different chapters without proper hierarchical guidance. Besides, the title "Gaps in Knowledge" should be part of "Critique of LLMs" rather than alongside it. (B) denotes the review based on both title and oracle (human-written) catalogue. Under the guidance of the hierarchical catalogue, the model can generate transparent and rational reviews. (C) consists of two steps: first generate a pseudo catalogue given the title and then obtain the entire review. It's apparent that the quality of generated catalogue is not satisfying since traditional language modeling methods, "statistical" and "rule-based," should have belonged the chapter "Development of LLMs" rather than "Type of LLMs." This consequently causes the degeneration of reviews due to the unreliable catalogue.
mat² of survey pupers' catalogues along with ab- stracts of their reference papers from Semantics Scholar². After meticulous filtering and man- ual screening, we obtain the final Hierarchical Catalogue Dataset (HiCaD) with 7.6k references- catalogue pairs, which is the first to decompose the review generation process and seek to explicitly model a hierarchical catalogue. The resulting Hi- CaD has an average of 81.1 reference papers for each survey paper, resulting in an average input length of 21,548 (Table 1), along with carefully curated hierarchical catalogues as outputs.
end-to-end (one-step) generation and step-by-step generation for the hierarchical catalogues, where the former generally works better and the latter al- lows for more focus on target-level headings. We benchmark different methods under fine- fine-tuned or zero-shot settings to observe their capabilities, in- cluding the recent large language models.
In summary, our contributions are threefold:
We observe the significant effect of hierarchi- cal guidance on literature review generation and propose new task, HiCatGLR (Hierar- chical Catalogue Generation for Literature Re- view), with the corresponding dataset HiCaD. We design several evaluation metrics for in- formativeness and structure accuracy of gen- crated catalogues, whose effectiveness is en- sured with detailed analyses.
Due to the structural nature of catalogues, tradi- tional metries like BLEU and ROUGE can not accu- rately reflect their generation quality. We specially design two novel evaluations for the catalogue gen- eration, where Catalogue Edit Distance Similarity (CEDS) measures the similarity to ground truth and Catalogue Quality Estimate (CQE) measures the degree of catalogue standardization from the frequency of catalogue template words in results.
We study both fine-tuned and zero-shot set- tings to evaluate models' abilities on the Hi- CatGLR, including large language models.
To evaluate the performance of various meth- ods on our proposed HiCaIGLR, we study both
Datasets: HICAD
We now introduce HiCaD, including the task def- initions, data sources, and processing procedures. We also provide an overall statistical analysis.
Phttps://arSiv. labs. labs. arxiv. orrg/ Phttps: //www/ www.we.scaniiccchlaarr.co org/

---

2.1 Definitions
all words lowercase for subsequent generation and evaluation. Finally, after removing data with less than 5 catalogue items and less than 10 valid references, we obtain 7,637 iiferences-cataloge pairs. We count the fields to which each paper be- longs (Table 6). We choose the computer science field with the largest number of papers for the ex- periment and split the 4,507 papers into training (80%), validation (10%), and test (10%) sets.
The input of this task is the combination of the title i of the target survey S and representative in- formation of reference articles {R), R² R?...., R") ). where n the number of reference articles cited by S. Considering the cost of data collection and ex- periments, we take abstracts as the representation of corresponding references. Besides, we restrict each abstract to 256 words, where the exceeding part will be truncated. The output is the catalogue ) of S where k is the number of catalogue items. ct is an item in the catalogue con- sisting of level mark 1' (Li, L2, L3}. which represents the level of the catalogue item, and the content [wf, wch .... ) with p number of words. As shown in Figure 1, "Pre-training" is the first- level heading, "Architecture" is the second-level heading, and Mainstream Architectures" is the third-level heading. In our experiment, we only keep up the third-level headings and do not con- sider further lower-level headings.
2.3 Dataset Statistics and Analysis
C
Taking the popular scientific dataset example, present the characteristics of the different multi- document scientific summarization tasks in Table 1. Dataset Multi-Xscience proposed by Lu et al. (2020), which focuses on writing the related work section of a paper based on its abstract with 4.4 articles cited in average. Dataset BigSurvey-MDS is the first large-scale multi-document scientific summarization dataset using review papers' intro- duction section as target (LIU et al., 2022), where previous work usually takes the section of related work as the target. Both BigSurvey and our HiCat- GLR task have more than 70 references, resulting in over 10,000 words of input, while their output is still the scale of a standard text paragraph, similar to Multi- Xscience. A natural difference between our task and others is that our output contains hier- archical structures, which place high demands on logic and conciseness for generation.
2.2 Dataset Construction
Our dataset collected from two sources: arXiv" and Semantics Scholar.. We keep papers contain- ing the words "survey" and "review" the title and remove ones with "book review" and "com- ments". We finally select 1 11,435 papers that are considered to be review papers. It is straightfor- ward to use crawler to get all 11,435 papers in PDF format according to the arxiv-id. However, ex- tracting catalogue from PDF files difficult where structural information is usually dropped during converting. Therefore, we attempt arSiv? to get the papers in HTML format. This website processes articles from arXiv as responsive HTML web pages by converting from LaTeX via LaTeXMLS. Some authors do not upload their LaTeX code, we have to skip these and collect 8.397 papers.
To measure how abstractive our target catalogues are, we present the proportion of novel n n-grams in the target summaries that do not appear in the source (Table 2). The abstractiveness of HiCaD is lower than that of BigSurvey-MDS and Multi- XSeience, which suggests that writing catalogues focus on extracting keywords from references. This conclusion is in line with the common sense that literature review is closer to reorganization than in- novation. Therefore, our task especially challenges summarizing ability rather than generative ability.
For the output part, we obtain the original cata- logues by cleaning up the HTML files. Then we replace the serial number from the heading with the level mark <L,> using regex. For input, we collate the list of reference papers and only keep the valid papers where titles and abstracts exist. We convert
We also analyze the share of each level of the cat- alogue in the whole catalogue. Figure 2 shows the value and proportional relationship of the average number of catalogue items as well as average word length at different levels. It can be seen that the second-level headings have the most weight, being 44.32% of the average number and 48.50% of the average word length. Table shows the weight of the headings at each level in the catalogue from the perspective of word coverage. We calculate
htttps://arxiv.rre
'https: sewanticscholall .rgg/
'These survey papers' metadata obtained from Kaggle up to the end of April 2023. https://wwe www..kkggle..com com/ Trressttstoonell---uuvvrvssttttt "https://arSiv.nn http://ggttoo..oombrmummleee///e

---

#Sents
#Words
#Sents
#Words
Datasets
Task
Pairs
Refs
Form
(input)
(input)
(output)
(output)
Mulii-XScience
Related Work
40,828
4,4
778.1
Paragraph
116.4
BigSurvey-MDS
Survey Introduction
4,478
76.3
450.1
11,893.1
Paragraph
38.8
1,051.7
HiCal
Survey Catalogue
7,637
81.1
471.4
21,548.2
Hicrarchical Catalogue
27.4
103.2
Table I: Comparison of our HiCaD dataset other multi-document scientific summarization tasks and their datasets. Pairs means the number of examples. Refs stands for the average number of input papers for each sample. Sents and Words indicate the average number of sentences and words in input or output calculated by concatenating all input output sources. Form is the form of the output text.
% of novel n-grams in target summary unigrams bigrams trigrams
La
La
Datasets
4-grams
R- R-1/R-2/RRL
R-1/R-2/RRLL
R-1/R-2/R-L
Multi- XSeience
42.33
81.75
94.57
97.62
L1
1815/
6.2/1.3/6.1
BigSurvey-Mil
37.39
76.46
93.87
98.04
L2
18.5/6.1/18.1
11.0/3.2/10.9
HiCaD
23.13
64.66
87.11
95.22
La Total
6.2/1.3/6.1 11.0/3.2/10. 57.9/53.0/57.9
45.3/39.1/45.3
25.1/23.0/25.1
Table The proportion of novel n-grams target sum-
maries across different summarization datasets.
Table 3:
ROUGE scores between different levels of
headings. Total represents the whole catalogue.
bewdt 28.
oracle ones, respectively.
6m
LON0-12 12.
#8.2hh
3.1 Catalogue Quality Estimate
There are some fixed templates in the catalogue, such as introduction, methods, and conclusion, which usually do not contain information about domain knowledge and references. The larger the percentage of template words in the catalog, the less valid information is available. Therefore, the proportion of template words indicate the in- formation content of the catalogue some extent. We collate a list of templates and calculated the percentage of templates in the catalogue items as Catalogue Quality Estimate (CQE). CQE evalua- tion metric measures the informativeness of the generated catalogue through template words statis- tics. Table 7 lists all template words. The CQE of oracle catalogues in the test set is 11.1%.
(a) item number
(b) word length
Figure 2: The value and proportional relationship of the average number of catalogue items as well as average word length different levels.
ROUGE scores between different levels of head- ings (L1, L2, L3) and the general catalogue "Total". Similar to the above, the secondary headings have the highest Rouge-1 score of 57.9 for the entire catalogue. Moreover, the Rouge-1 score of 18.5 for the first and second-level headings L indicates some overlaps between the first and second levels. The low Rouge scores of Lı-L3 and L2-La reveal that there are indeed different usage and wording distributions between different levels.
3.2 Catalogue Edit Distance Similarity
Catalogue Edit Distance Similarity (CEDS) mea- sures the semantic similarity and structural simi- larity between the generated and oracle catalogues. Traditional automatic evaluation methods sum- marization tasks (such as ROUGE (Lin, 2004) and BERTScore (Zhang* et al., 2020)) can only mea- sure semantic similarity. However, catalogues are texts with hierarchical structure, so the level of headings in catalogues also matters.
3 Metrics
To evaluate the quality of generated catalogues, we propose two evaluation methods, Catalogue Qual- ity Estimate (CQE) and Catalogue Edit Distance Similarity (CEDS). The former takes textual per- spective, while the latter integrates text and struc- ture. We use these evaluation methods to assess the informativeness of the generated catalogues and the gap between the generated catalogues and the
We are inspired by the thought of edit distance which is commonly used in the ordered labeled tree similarity metric. An ordered labeled tree is one in which the order from left to right among siblings is significant (Zhang and Shasha, 1989). The tree
"The ROUGE scores in this paper are all computed by ROUGE-I.5. script with the option "-c 95 -r 1000 -n -m"

---

edit distance (TED) between two ordered labelled trees Ta, Tb is defined the minimum number of node edit operations that transform Ta into To (Paaßen, 2018). There are three edit operations on the ordered labeled tree: deletion, insertion, and modification, each with a distance of one. Similar to TED, the definition of catalogue edit distance (CED) is the minimum distance of item edit oper- ations that transform catalogue Cn into another catalogue Co, where each entry in the catalogue is a node. The difference is that we calculate the dis- tance between two items according to the similarity nodes when modification:
Survey
encoder
Reference
Survey Title
decoder
Catalogue
encoder
Refenence
Survey Title
encoder
Reference N
Figure 3: Architecture of the step-by-step approach.
and the information from its reference papers input before generating the survey catalogue. The title and references are concatenated together and fed to the encoder. However, existing transformer- based models with O(N2) computational complex- ity show an explosion in computational overhead as the input length increases significantly.
Distance(z, y) mmmn11, ax (1- Siiilarity((,
We leverage BERTScore obtain the similarity between two items of the catalogue based on the embeddings from SciBERT (Beltagy et al., 2019):
We choose Fusion-in-Decodd (FiD) (Izacard and Grave, 2021), a framework specially designed for long context in open domain question answer- ing, to handle the end-to-end catalogue generation. As shown Figure 3, the framework processes the combination of the survey title and information from each reference paper independently by the encoder. The decoder pays attention to the con- catenation of all representations from the encoder. Unlike previous
Similarity (2, y) = (Sci-)BERT y)-
SciBERT is more suitable for scientific literature than BERT because it was pretrained a large multi-domain corpus of scientific publications. We take hyperparameter
1.2 during experiments. Therefore, we can define Catalogue Edit Distance Similarity (CEDS) as:
CED Ch) CEDS(C,, Cb) 100x (1 max([Cal, (c&l)
models, the FiD model processes papers separately in the encoder. Therefore, the input can be extended a large num- ber of contexts since it only performs self-attention over one reference paper each time. This allows the computation time of the model to grow linearly with the length of the input. Besides, the joint processing of reference papers in the decoder can better facilitate the interaction of multiple papers.
We use the library Python Edit Distances 10 pro- posed by Paaßen et al. (2015) for the implemen- tation of algorithms. A detailed example of node alignment and the conversion process between two catalogues is given in Appendix
4 Models
4.2 Step-by-Step Approach
We now introduce our explorations in hicrarchical catalogue generation for literature review, includ- ing end-to-end and step-by-step approaches.
Another main challenge in generating the catalogue is modeling relationships between catalogue items at different levels. Motivated by Tan al. (2021), we explore an effective approach for incremen- tal generation. Progressive generation divides the complicated problem of generating a complete cat- alogue into more manageable steps, namely the generation of hicrarchical levels of the catalogue. Different from generating everything in one step, the progressive generation allows the model to per- form high-level abstract planning and then shift attention to increasingly concrete details. Figure 5 illustrates the generation process.
4.1 End-to-End Approach
One of the main challenges in generating the cata- logue is handling long input contexts. The intuitive and straightforward generation method is the end- to-end model, and we experiment with two models that specialize in processing long text. We con- sider an encoder-decoder model for the end-to-end approach. The model takes the title of survey
htttt::://gitlab.uu wwt.uuu..uunbee uni-bielefell wmi-bbelefallddd/bppasssee beaasaasssen/ sattoo--ddi--isttneeee

---

ROUGE:
Method
BERTScore f
CEDST
CQE +
L3
Total
Extractive Methods
LexRank
10.5/1.7/10.0
47.5
2.4
TextRank
2/12.1
47.2
2.1
End-to End
base
37.8/12.2/37 41
1
18/0.3/1.5
26. 9/2.5/
64.2
31.41
22.2
LED
large
39.3/12.8/38.7
18.9/4.0/18.8
5.2/1.0/5.2g
29.5/8.0/29.3
64.7
34.1
16.9
base
35.1/10.3/34.8
15,1/3 85.1//.8/15.0
3.00,442.9
25. 1/250/
64.31
29.5
21.4
FD(BART)
large
34.6/10.9/34.1
16.9/4.3/16.7
5.6/1.1/5.6
27.0/7.6/26.95
64.43
31.0
18.5
Step-by-Siep
base
36.3/12.0/36.0
13.0/2.9/12.9
0.7/0.100 1/0.7
14.2/6.//24.0
63.0
30.1
23.0
LED
large
37.0/11.8/36
13.3/3.2/13.2
1880.5/1.8
26.4/7.3/26.2
63.6
32.3;
20.8
base
36.8/12.7/36.5
9,9/2.5/9.8
0.0/0.00..0
22.0/6.4/21..
63.2
27.1
29.1
FiD(BART)
large
37.7/11.8/37.21
8.8/2.188.7
0.600.000.6
24.3/6.4/24.0
63.3
28.9
21.6
Large Language Models
Galactica-6.7b ChatGPT
16.0/4.7/15.5
2.0/0.3/1.9
0.870.000.7 4.8/0.8/4.8f
17.7/4.3/17.3
55.6
28.0 28.0
13.81
18.2/3.9/18.02 1/00334
26.6/6.8/26.3
62.6
13.2
Table 4: Automatic evaluation results HiCaD. Bold indicates the best value in each setting. $ represents the second best result and j represents the third best.
5 Experiments
5.2 Implementation Details
We dropout with the probability 0.1 and learn- ing of 4e 5. The optimizer is Adam with 31 0.9 and B2 0.999. We also adopt the learn- ing rate warmup and decay. During the decoding process, we use beam search with a bcam size 4 and tri-gram blocking to reduce repetitions. We adopt the implementations of LED from Hugging- Face's Transformers (Wolf et al., 2020) and FiD from SEGENC (Vig al., 2022). We maximize the input length to 16,384 tokens. All the models are trained on one NVIDIA A100-PCIE-80GB.
We study the performance of multiple models on the HiCaD dataset. Detailed analysis of the gen- eration quality is provided, including correlation validation of the proposed evaluation metrics with human evaluation and ROUGE for abstractiveness.
5.1 Baselines
Due the large input length, we choose encoder- decoder transformer model that can handle the long text and its backbone model to implement FiD besides various extractive models in our ex- periments.
5.3 Results
(I) LexRank (Erkan and Radev, 2004) is an unsupervised extractive summarization ap- proach based on graph-based centrality scoring of sentences.
As shown in Table 4, we calculated the ROUGE scores, BERTScore, Catalogue Edit Distance Sim- ilarity (CEDS) and Catalogue Quality Estimate (CQE) between the generated catalogues and ora- cle ones separately. We especially remove all level mark symbols (e.g. <L,>) for evaluation. In or- der to compare the ability of different methods to generate different levels of headings in detail, we calculated ROUGE scores for each level of head- ings with the corresponding level of oracle ones.
(II) TextRank (Mihalcea and Ta- rau, 2004) is graph-based ranking algorithm im- proved from Google's PageRank (Page al., 1999) for keyword extraction and document summariza- tion, which uses co-occurrence information (se-
mantics) between words within a document to ex- tract keywords. (III) BART (Lewis et al., 2019) is pre-trained sequence-tor to--nee-----euuccee Transformer model to reconstruct the original input text from the corrupted text with denoising auto-encoder. (IV) Longformer-Enee Beeermeer-Eneee-----e Encoder-Decoder (LED) (Belt- agy et al., 2020) is built on BART but adopts the sparse global attention mechanisms in the encoder part, which alleviates the input size limitations of the BART model (1024 tokens) to 16384 tokens.
We first analyze the performance of traditional extractive approaches. Since these extractive mod- cls can not generate catalogues with explicit hier- archy as abstractive methods, only calculate the ROUGE scores of extracted results the entire acle catalogues (Column Total in Table 4). We take the ROUGE score (1 11.99 9.99//5// 1.4) between titles of literature reviews and entire oracle catalogues

---

threshold because titles are the most concise and precise content related to reviews. LexRank (10.5/1.7/10.0) and TextRank (12.9/1.//12.) only achieves similarly results as the threshold. This means extractive methods are not suitable for hier- archical catalogue generation.
First, we test the human evaluations and CEDS of corresponding data for normal distribution. Af- ter the Shapiro- Shakiro-Wilk test, the p-values for the two sets are 0.520 and 0. 2,250, all greater than 0.05 (Ta- ble 8). That means these data groups can be consid- ered to meet the normal distribution, which enables further Pearson correlation analysis. Person corre- lation analysis shows that p-values between CEDS and Human are 0.027, more diminutive than 0.05 (Table 9). The r-value 0.634, which represents a strong positive correlation. Therefore, we consider CEDS as a valid evaluation indicator.
By comparing all evaluation scores, the end-to- end approach achieves higher similarity with the ground truth than step-by-step on the whole cata- logue. Generally, the model with a larger number of parameters performs better. However, there are still duplication and hierarchical errors in the cur- rent results (See case studies in Appendix F).
5,4.2 Automatic Evaluation
Large language models have shown excellent performance on many downstream tasks due to massive and diverse pre- pre-training. We also test two representative large language models: Galactica (Taylor et al., 2022) and ChatGPT (gpt-3.5-turbo). Galactica (GAL) is a large language model for sci- ence that achieve scientific tasks. The corpus it used includes over 48 million papers, textbooks, scientific websites and so on. ChatGPT the best model recognized by the community and also trained on ultra large scale corpus. The corpus used by these two models in the pre-training phase contains scientific literature and thus we consider that the knowledge of the models contain the reference papers. From evalua- tion results, instructions understanding and answer readability, ChatGPT generates far better results than GAL. It's worth noting that large language models can not outperform models (LED-llrre) specially trained for this task. This reveals that sim- ply stacking knowledge and parameters may not be a good solution for catalogue generation which requires more capabilities on logic and induction. See Appendix D for specific details and analysis.
We also conduct a pair-wise consistency test be- tween all the automatic indicators measured (Table 5) by Pearson's 's correlation coefficient, where only ROUGE-L in ROUGE was computed.
First, a noticeable trend is that the ROUGE-L of the first-level catalogue (LIRL) is not corre- lated with any other metrics. We infer that this is due to the ease of generating first-level headings, which perform similarly across methods and reach bottlenecks. The second & third-level catalogue (L2RL,, L3RL) exhibit an extremely strong positive correlation with the total one (TotalRL), which sug- gests that the effectiveness of generating second & third-level headings affects the overall perfor- mance of catalogue generation. Second, Catalogue Quality Esstimate (CQB) negatively correlates with the ROUGE-L in three levels (L.2RI,, L,RII, To- taIRL). This indicates that domain knowledge and reference information are mainly found in the sec- ondary and tertiary headings, which is in line with human perception. Finally, we study the differ- ence between TotalRL and BERTScore. We find that TotalRL and BERTScore not relevant, but they are correlated with CEDS respectively. This means that CEDS can combine both ROUGE and BERTScore indicators and better describe the simi- larity between generated results and oracle ones.
tete-----hhhrrrn results over many
5.4 Consistency Check
5.4.1 Human Evaluation
To demonstrate the validity of our proposed eval- uation metrics, we conduct consistency tests on CEDS with human evaluation. We generate 50 catalogues for each implementation with total number of 450 samples, each sample is evaluated by three professional evaluators. We skip Galactica- 6.7b for subsequent experiments since it can hardly generate reasonable catalogues. Evaluators are re- quired assess the quality of catalogues based on the similarity (ranging from one to five, where five represents the most similar) to the oracle one.
Related Work
Survey generation Our work belongs to the multi- document summarization task of generation, which aims to reduce the reading burden of rescarchers. Survey generation is the most challenging task in multi-document scientific summarization. The early work was mainly an extractive approach based on various content selection ways (Moham- mad et al. (2009), Jha et al. (2015)). The results

---

LIRI.
T.2RI.
L3RI.
TotalRI.
BERTScore
CHDS
CQE
-0.240
-0.357
0.118
0.348
0.511
0.335
LIRI
P
0.534
0.346
0.763
0.358
0.160
0.378
-0.240
889
8474*
0,424
0,511
-777*
L2RL
P
0.534
0.001
0.004
0.256
0.160
0.014
-0,357
.89**
826**
0,434
0,409
82748
L3RL
P
0.346
0.001
0.006
0.243
0.275
0.006
0.118
8847*
.826**
0.586
808**
790*
TotaIRI.
0.763
0.004
0.006
0.098
0.008
0.011
0.348
0,424
0.434
0,586
.707
-0.082
BERT
0.358
0,256
0.243
0.098
0.033
0,834
0.511
0.511
0.409
808**
.70**
-0.338
CEDS
P
0.160
0.160
0.275
0.008
0.033
0.374
0.335
-7774
-82744
7790*
-0.082 0.834
-0.338 0.374
CQF
P 0.378
0.014
0.006
0.011
Table 5: Results of Pearson's 's consistency test. represents a significant correlation at the 0.01 level (two-tailed).
denotes a significant correlation the 0.05 level (two-tailed).
generated by unsupervised selection models have significant coherence and duplication problems.
crate slides from a multi-modal document. They generate slide titles, i.e., structures of one target document, via summarizing and placing object. These efforts don't really model the relationships between multiple documents, which far from the hierarchical structure of review paper. This paper focuses on the generation of a hicrarchical cata- logue for literature review. Traditional automatic evaluation methods for summarization calculates similarity among unstructured texts from the per- spective of overlapping units and contextual word embeddings (Papineni al. (2002), Banerjee and Lavie (2005), Lin (2004), Zhang* al. (2020), Zhao al. (2019)). Tasks involving structured text generation are mostly measured using the accuracy, such as SQL statement generation (He et al. (2019), Wang et al. (2019)), which ignores semantic infor- mation. Assessing the similarity of catalogues to standard answers should consider both structural and semantic information.
To make the generated results have clear hi- erarchy, Hoang and Kan (2010) additionally in- puts associated topic hierarchy tree that de- scribes a target paper's topics to drive the creation of an extractive related work section. Sun and Zhuge (2019) proposes a template-based frame- work for survey paper automatic generation. allows users to compose a template tree that con- sists of two types of nodes, dimension node and topic node. Shuaiqi et al. (2022) trains classifiers based on BERT(Devlin et al., 2018) to conduct category-based alignment, where each sentence from academic papers is annotated into five cat- egories: background, objective, method, result, and other. Next, each research topic's 's sentences are summarized and concatenated together. However, these template trees are either inflexible or require the users to have some background knowledge to give, which could be more friendly to beginners. Also, this defeats the original purpose of automatic summarization to aid reading.
Conclusion
In this work, we observe the significant effect of hicrarchical guidance on literature review gener- ation and introduce new task called hierarchi- cal catalogue generation for literature review (Hi- CatGLR) and develop a dataset HiCaD out arXiv and Semantic Scholar Dataset. We em- pirically evaluate two methods with eight imple- mentations for catalogue generation and found that the end-to-end model based LED-large achieves the best result. CQE and CEDS are de- signed measure the quality of generated results from informativeness, semantics, and hierarchi- cal structures. Dataset and code are available at http://gtthb..com/huuum 102//iiCaD. Our analy-
Text structure Generation There are some ef- forts involving the automatic generation of text structures. Xu et al. (2021) employs a hierarchi- cal model to learn structure among paragraphs, se- lecting informative sentences to generate surveys. But this structure is not explicit. Tan et al. (2021) generate long text by producing domain-specifir keywords and then refining them into complete passages. The keywords structure's guide for the subsequent generation. Trairatvorakul et al. reorganize the hierarchical structures of three to six articles to generate a hier- archical structure for a survey. Fu et al. (2022) gen-
be considered

---

sis illustrates how CEDS resolves some of the limi- tations of current metrics. In the future, we plan to explore a better understanding of input content and the solution to headings level error, repetition, and semantic confict between headings.
References
Nouf Ibrahim Altmami and Mohamed El Bachir Menai. 2022. Automatic summarization of scientific articles: A survey, Journal of King Saud Srivpusity-Compute and Information Sciences, 34(4): 1-1008..1.....
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- rion, pages 65-72.
Limitations
The hierarchical catalogue generation task can help with literature review generation. However, there is still much room for improvement in the current stage, especially in generating the second & third- level headings, which requires a comprehensive understanding of all corresponding references. Be- sides, models will need to understand each refer- ence completely rather than just their abstracts in the future. Our dataset, HiCaD, does cover a comprehensive range of domains. In the future, we need to continue to expand the review data in other fields, such as medicine. Currently, we only experiment on a single domain due to limitations of collected data resources, where knowledge transfer across domains matters for catalogues generation.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNI.P) pages 3615-3620.
Beltagy, Matthew Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arxhv:2004.05150
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint urXiv:1810.0480
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience text sum- marization. Journal of artificial intelligence research, 22:457-479.
Ethics Statement
In this paper, we present a new dataset, and we dis- cuss here some relevant ethical considerations. (1) Intellectual property. The survey papers and their reference papers used for dataset construction are shared under the CC BY-SA 4.0 license. It is free for research use. (2) Treatment of data annotators. We hire the annotators from proper channels and pay them fairly for the agreed-upon salaries and workloads. All hiring is done under contract and in accordance with local regulations.
Michael Fire and Carlos Guestrin. 2019. Over- optimization of academic publishing metrics: ob- serving goodhart's law in action. GigaScience, 8(6)gi22533
Tsu-Jui Fu, William Yang Wang, Daniel McDuff, and Yale Song. 2022. Doc2ppt: Automatic presentation slides generation from scientific documents. In Pro- ceedings of the AAAI Conference on Artificial Intelli gence, volume 36, pages 634 642.
Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021. Baco: A back- ground knowledge-and content-based framework for citing sentence generation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume 1: Long Papers), pages 1466-1478.
Acknowledgements
Xiaocheng Feng is the corresponding author of this work. We thank the anonymous reviewers for their insightful comments. This work was sup- ported by the National Key R&D Program of China via grant No. 20.0040000000502,,, National Natural Science Foundation of China (NSFC) via grant 62276078, the Key R&D Program of Heilongjiang via grant 2022ZX01A32 the International Cooper- ation Project of PCL, PCL2022D01 and the Funda- mental Research Funds for the Central Universities (Grant No.HIT.OCEE:20230111
Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-sql: reinforce schema representation with context. arXiv preprint arXiv:1908.08715
Cong Duy Vu Hoang and Min- Sen Kan. 2010. Towards automated related work summarization. In Coling 2010: Posters, pages 427-435.
Yue Hu and Xiaojun Wan. 2014. Automatic genera- tion of related work sections in scientific papers: an

---

optimization approach. In Proceedings of the 2014 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP)), pages 1624-1633.
Proceedings of human language technologies: The 2009 annual conference of the North American chap- ter of the association for computtional linguistics. pages 584-592.
Gautier Izacard and Edouard Grave. 2021. Leveraging
passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online, Association for Computa- tional Linguistics.
Benjamin Paaßen. 2018 tance and its backtracing: A tutorial. arXiv preprint 1805.080000666
Revisiting the tree edit dis-
Benjamin Paaßen, Bassam Mokbel, and Barbara Ham- mer. 2015. A toolbox for adaptive sequence dis- similarity measures for intelligent tutoring systems. In Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015), page 632-632. International Educational Datamining So- ciety.
Rahul Jha, Catherine Finegan-Dollaks Ben King, Reed Coke, and Dragomir Radev. 2015. Content models for survey generation: a factoid-based evaluation. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 441 4114500
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation rank- ing: Bringing order to the web. Technical report, Stanford InfoLab.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to-seee to-sequence pre-trate pre-training for natural language generation, translation, and comprehension. arXiv preprint urXiv:1910. 1346l.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
Pengeheng Li, Wei Lu, and Qikai Cheng. 2022. Gen- erating a related work section for scientific papers: optimized approach with adopting problem and method information. Scientometrics, 127(8))4397- 4417.
LIU Shuaiqi. Jiannong Cao, Ruosong Yang, and Zhiyuan Wen. 2022. Generating a structured sum- mary of numerous academic papers: Dataset and method.
Chin-Yew Yew Lin. 2004. ROUGE: A package for auto-
matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Xiaoping Sun and Hai Zhuge. 2019. Automatic gen- eration of survey paper based on template tree. In 2019 15th International Conference on Semantics, Knowledge and Grids (SKG), pages 89-96. IEFF.
Shuaiqi LIU, Jiannong Cao, Ruosong Yang, and Zhiyuan Wen. 2022. Generating a structured sum- mary of numerous academic papers: Dataset and method. In Proceedings of the Thirty-First In- ternational Joint Conference on Artificial Intelli- gence, IJCAI-22, pages 4259-4265. International Joint Conferences on Artificial Intelligence Organi- zation. Main Track.
Bowen Tan. Zichao Yang, Maruan Al-Shedivat, Eric Xing, and Zhiting Hu. 2021. Progressive generation of long text with pretrained language models. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Haman Language Technologies, pages 4313 1313-4324, Online. Association for Computational Linguistics.
Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi- xscience: large-scale dataset for extreme multi- document summarization of scientific articles. arXiv preprint arXiv:2010.1423:
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert. Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09088
Kelvin Luu, Xinyi Wu. Rik Koncel-Kedaan Kedziorskkii.ii Kyle Lo, Isabel Cachola, and Noah A Smith. 2020. Explaining relationships between scientific documents. arXiv preprint arXiv:2002.000 00317.
Prawat Trairatvorakul, Alexander Fabbri, and Dragomir R Radev. Surveytree: Automatic generation of survey structures for nlp and ai topics.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bring- ing order into text. In Proceedings of the 2004 con- ference on empirical methods in natural language processing, pages 404-411.
Jesse Vig, Alexander Fabbri, Wojciech Kryscinski, Chien-Sheng Wu, and Wenhao Liu. 2022. Exploring neural models for mocusfo summarization. Findings of the Association for Computational Lin- guisties: NAACL 2022, pages 1455-1468, Seattle, United States. Association for Computational Lin- guistics.
Saif Mohammad, Bonnie Dorr, Melissa Egan, Ahmed Hassan, Pradeep Muthukrishnan, Vahed Qazvinian, Dragomir Radev, and David Zajic. 2009. Using cita- tions to generate surveys of scientific paradigms. In

---

Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson, 2019. Rat-sql: Relation-aware schema encoding and linking for text- to-sql parsers. arXiv preprint arXiv: 1eee19911..99 1.04922.
Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNIP-IJCNIPP) pages 563-578, Hong Kong, China. Association for Computational Lin- guistics.
Lucy Lu Wang and Kyle Lo. 2021. Text mining ap- proaches for dealing with the rapidly expanding lit-
A Dataset Domain
erature covid-19. Briefings in Bioinformatics, 22(2):781-799.
We count the fields to which the dataset belongs, and the computer science field have the most pa- pers, followed by mathematics. In this paper, we use only papers from the computer science domain for experiments.
Pancheng Wang. Shasha Li, Kunyuan Pang, Lian- gliang He, Dong Li, Jintao Tang, and Ting Wang. 2022. Multi-document scientific summarization from a knowledge graph-centric view. arXiv preprint arXiv:2209.03319
Jane Webster and Richard Watson. 2002. Analyzing the past prepare for the future: Writing literature review, MIS quarterly, pages xiii-xxiii.
Domain Computer
Subject
Amount
Computing Research
4,507
Science
Repository
Mathematics
Mathematics
922
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2020. Transformers:
Physics
284
High Energy Physics
235
-Phenomenology
the-art natural language processing. Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38-45.
Condensed Matter
High Energy Physics
173
-Theory
Physics
Quantum Physics
General Relativity
103
and Quantum Cosmology
Jia Yan Wu, Alexander Te- Wei Shich. Shih-Ju Hsu, and Yun-Nung Chen. 2021. Towards generating citation sentences for multiple references with intent control. arXiv preprint 12.00321
High Energy Physics
90
-Experiment
Mathematical Physics
Nuclear Experiment
Nuclear Theory
Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020. Automatic generation of citation texts in scholarly papers: A pilot study. In Proceedings of the 58th Annual Meeting of Association for Computational Linguistics, pages 6181-6190.
Nonlinear Sciences
High Energy Physics
23
Lattice
Electrical Engineering & Systems
Electrical Engineering and Systems Science
Huiyan Xu, Zhongqing Wang, Yifei Zhang, Xiaolan Weng, Zhijian Wang, and Guodong Zhou. 2021. Doc- ument structure model for survey generation using neural network. Frontiers of Computer Science, 15:1- 10.
Science
Statistics
Statistics
267
Quantitative Biology
Quantitative Biology
83
Quantitative
Quantitative Finance
33
Finance
Economics
Economics
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, and Rui Yan. 2019. Plan-and-write: To- wards better automatic storytelling. Proceedings of the AAAI Conference Artificial Intelligence, 33:7378 7385.
Table 6: Domain distribution of literature review papers in the dataset HiCaD. Category division comes from arXiv.org.
Kaizhong Zhang Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM journal computing, 18(6):225-.) 1262.
B Template Words
The list of template words used when calculating CQE in this paper is listed in Table 7.
Tianyi Zhang", Varsha Kishore Felix Wu*, Kilian Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
C Consistency Check
D Large Language Models
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris-
tian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in
In this section, we experiment with the effect of two foundation models on the hierarchical catalogue generation task.

---

introduction preliminaries background overview definition(s) methodology (sthod(s) sources purpose related work dataset(s) outlook eassy benefit(s) advantage sss challenge(s) application(s) future summary notation(s) result(s) discussion analysis observation(s) conclusion(s) references
preliminary
CEDS sample
Pearson 0.634
Human sample
Pearson P
0.027
Table Consisteney evaluation results between pro- posed metries and corresponding human judgements.
Table 7: All template words used to calculate Catalogue Quality Estimate (CQE).
catalogue. Finally, it generates the whole catalogue with method similar to the previous step. The generation process corresponds to a decomposition of the conditional probability as:
Shapiro-Win Wilk test
Human sample
0.250
CEDS sample
0.027
Table 8: P-values of Shapiro-Wii Wilk test for CEDS and human evaluation results.
P(t)s) = Pa L12))
D.1 Galactica
Case Study
Galactica has five sizes ranging from 125M to 120B parameters and the standard size 6.7B is the max size can use. We experimented one-shot test two generation samples in Figure 4. It can be seen that there is lot of repetition in the results.
Table 10 and Figure 6 present an example of the best catalogues generated by end-to-end models and step-by-step models about the title" "a survey of domain adaptation for neural machine translation". Table gives an example of aligmment between catalogue items when calculating the Catalogue Edit Distance (CED). If an item does not match any other entity, then the required action for this item is insertion or deletion, so the cost is 1. If node matches a node, then the required action is modification. The operation cost of modification is calculated according to the similarity. It is worth noting that Node[2] in the generated result does not match Node[1] the ground truth, even though they have exactly the same content. It is because the former is a secondary heading, and the latter is primary heading. If the alignment is forced, the cost required will be greater than the current result. Thus this example shows that not only does the metric CEDS measure catalogues from mantic perspective, but also takes into account the hierarchical structure of the catalogue.
test set. There
D.2 ChatGPT
The prompt of what we use generate a directory using ChatGPT is:
Your task to write a table contents for the re- view paper by recalling relevant papers, organizing and classifying them according to the given review paper topic. Only the first, second and third level headings need to be written, no detailed explana- tion is required. Please ensure that your catalogue is well-structured, clear, and concise, and accu- rately represents the topic's main research findings and methodologics.
Title: A Survey of Large Language Models Table of Contents:
E Step by Step Method
The parts marked in red in Figure are the gener- ation problems we observe. The first problem is du- plicating the content in a single catalogue item, e.g. "monolingual monolingual data". The second one is the hierarchy error between sibling nodes. For ex- ample, "evaluation metrics" conceptually contain "automatic evaluation". Finally, there is recur- rence of the heading "applications of nmt domain adaptation". In summary, there are still duplication and hierarchical errors the current results.
Figure illustrates one the generation processes the step-by-step generation method. Instead of generating the whole catalogue C directly, we propose to generate step-by-steny source - L1 + 11,2 1,,2,3 (target),, where L, is the i-th level headings of the catalogue. For example, in the first step. we input survey title and information about its reference documents and generate the first- level headings Jn of the catalogue for that survey. Then, L1 added to the input, and the next step generates the first two levels headings L,2 of the

---

Prompt:
Prompt:
Large
Title: Survey
Kodels
Nodels
Table of contents:
Table of conterts:
Tstle: A SuSueey of Lange Longunge Models
Tiele: the are of soctal bots: rectes
Table of contents:
Table
1. [JabshOO]
2. Intreduction
Response:
tro-iig
4. Fine-tuning
Entreduction
Model Compression
Taxonomy Social Bets
6. Model Scoling
3 Types of Social Bots
7. Model Architecture
4.1 Automation
4.11 Automation by Hunans
9. Model Evaluction
Model Depleyment
Automation Mathines
Conclusion
Automation Hybrid
4.2 Autonation Humons
4,23 Automation Humond
The few have witnessed the rapid development deep learsing, which has ochieved
fielss,
4.2 Automation by Humons
much os In cose, necessory to develop deep learning models with good generdizaation obility and high efficiency.
4233 Automation by Humons
Longunge
class doep
widely
longunge processing.
Automation by Humani
LMd are usuclly trained on acgle uricheled led text corporo and then fine fuuned on dowestrean tusks.
Automation by Humans
The pre-troining and fine- tuning panadigm achiaved great
natural longunge processing,
Humons
The pre- training stage can general language rupresentations from large- unlabeled next corpons, the fine-turing stage can adapt the pre trained language representations downstream tasks.
4.27 Automation Humons
4.28 Automation by Humons
4.29 Automation by Humons
In recent years, The pre traiiiiiingg and fine-tuiing paradigm has been widely used in natural larguage processing The training fine tuning paradign has achiaved scale uniobeled el text corporo. fine-tuning odept pre-troined longunge representiatins to downstroam tosks
4.230 Automation by Humons
42231 Automation by Humans
Autatration --.
this survey, We first introduce the pre training and fine-tuning paradion for lange scole longuage nodels.
pre-trainin furin paradigm lorge- longuage models.
Then, intreduce the pre-troining and fine-tuning perodign for lorg-stosls longunge models.
Then, we introduce the pre-troining and fire turin parodigm for large scole longuage models.
Figure 4: Two catalogue generation results of GAL model.
Servay Title
Introckction
Ixtrataction
otaptaticn adopaatias
tranclation
odppottton
adopaation
odaphationn
Reference
acerence
cdophratio
colptation
Nevil exciie cusclatior
funare siructioes corckaton
cato centric
madal
4.3 monaiingul carpors
donaii odynaatinn in rediwandd
systhetic parcila carpora gesaration
aconattd
Fature dirachone
4.1.3 using sut doms parcllel carpora
feaits preamal
6,1 coepis sdaytenien for shate-ot- thavort
4.2 meddl cestric
Survey Title
6.2 net ardhtecttines
4.32 tedinng objectiive cantric
desion acasrationn tramiantie
domcin speofic exchionrry
4.2. orchitecture conniis
incorpanation
diceddng centric
malttingual and muhiidamain
demain edephation neal acanamie Fature direchoou
Referesce
odeptation
Bwoll wikhoe troestation
adiarsanaal densioodoosttion
domein sdapttaiio shote-of-tha-
andirtectunal
Referesce
reahtlingan
poslt grenalaatiin sits
odig
adversand daman adaptistion doman generation
Survey Title
canayy deads sturatian for
condiugos
treslattes in few
Reforence App/ bedidg paine frie
Progressive Generation
Final Result
Figure Architecture of step-by- step approach.

---

introduction
1 Introduction
domain adaptation for statistical machine
2 neural machine translation
translation
2,1 preliminaries
2.1 overview of domain adaptation for smt
2.2 neural network
2.2 problem definition
architecture
2.3 domain adaptation challenges in smt
2.3 adversarial learning
datasets for domain adaptation
2.4 parallel data
3.1 monolingual monolingual data
2.5 monolingual data
3.2 parallel parallel data
2.6 discrete translation
3.3 out-of-domain parallel data
lexicons
4 data selection for nmt
2.7 other techniques
4.1 data selection methods for statistical nmt
3 domain adaptation
4,1,1 domain weighting methods
3.1 overview of domain adaptation
4.1.2 instance-based methods
4.1.3 multi-task learning methods
3.2 types of domains
4,1.4 domain-adversarial training methods
3.3 adaptation tasks
4.1.5 other data selection methods
3.4 evaluation metrics
4.2 multi-domain nmt methods
3.5 applications of nmt domain
model adaptation for domain-specific tasks
adaptation
5.1 word-level adaptation
4 applications of nmt domain adaptation
5.2 phrase-based adaptation
5.3 sentence-level adaptation
4.1 challenges and future directions
5.4 other adaptation methods
6 evaluation
6.1 dataset-based evaluation
6.2 evaluation metrics
6.3 human evaluation
6.4 automatic evaluation
conclusion and future direct
Figure Examples of end-to-end and step-by-step

---

Generated Result
Ground Truth
distance
introduction
introduction
neural machine translation
background
domain adaptation for
0.50
</2 neural machine translation
> data centric [3]
0.48
domain adaptation
model centric [4]
0.35
survey of domain adaptation
domain adaptation for nmt
0.12
parallel data adaptation
data centric [6]
0.34
fine-tuning
>
using monolingual corpora [7]
0.44
weighting [7]
synthetic parallel corpora generation [8]
0.35
multi-task learning [8]
using out-of-domain parallel corpora
0.38
model centric [10]
> training objective
[11]
monolingual data adaptation
architecture centric [12]
0.44
2 > synthetic parallel adaptation [10]
[g > decuding centric [13]
> domain adaptation real- world scenarios [14]
> conclusion directions [1]]
> directions [15]
12 >
adaptation state-of-the-ad
architectures [16]
12 > domain specific dictionary incorporation [17]
1 > multilingual and mulli-domain adaptation [18]
12 > adversarial domain adaptation and domain generation [19]
> conclusion [20]
Table 10: An example of aligned nodes and distances between them when calculating CED.


Reviews:
[{"review_id": "NZgqENDe8a", "reviewer": "Reviewer_SJpq", "summary": "", "questions": "1. Page 3, column 2, is ambiguous on the final size of the dataset after limiting it to computer-science only papers. is the final size 7637 or 4507?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestion.\n\n1. **Dataset Size and Domain**\n\nOur dataset comprises a total of 7,637 reference-catalogue pairs, spanning a broad spectrum of domains (Computer Science, Mathematics, Physics, Electrical Engineering & Systems Science, Statistics, Quantitative Biology, Quantitative Finance, Economics).  In Table 6, we list the areas of papers included in the dataset and their corresponding quantities.\n\nWith a majority share of 4,507 articles, Computer Science emerges as the most represented domain. Given the complexity of dataset annotation, which necessitates a high level of expertise to assess the quality of generation, we opt to conduct the catalogue generation task experiments within the most representative domain, i.e., Computer Science. Besides, surveys in other fields of the dataset provide a solid foundation and can further stimulate future research.\n\nA lot of effort has been put into collecting and processing the data (Section 2.2). Both Review 2 and 3 agree with our contribution. Besides, the size of our dataset exceeds that of other datasets related to surveys, as shown in the table below. Our data collection methodology is not only scalable but also adaptable to future expansion in terms of field and scale. In our final version, we will enhance the clarity of our expression.\n\n| Dataset | Size |\n| --- | --- |\n| HiCaD(Ours) | 7,637 |\n| BigSurvey-MDS[1] | 4,478 |\n| TutorialBank(survey)[2] | 390 |\n| Surveyor[3] | 297 |\n| Surfer100[4] | 100 |\n| DocStructure[5] | 27 |\n\n[1] Generating a Structured Summary of Numerous Academic Papers: Dataset and Method. (Liu et al., IJCAI 2022)\n\n[2] TutorialBank: A Manually-Collected Corpus for Prerequisite Chains, Survey Extraction and Resource Recommendation. (Fabbri et al., ACL 2018)\n\n[3] Surveyor: A System for Generating Coherent Survey Articles for Scientific Topics. (Jha et al., AAAI 2015)\n\n[4] Surfer100: Generating Surveys From Web Resources, Wikipedia-style. (Li et al., LREC 2022)\n\n[5] Document structure model for survey generation using neural network. (Xu et al., Frontiers of Computer Science)\n\n2. **Method**\n\nThe most important two contributions are as follows: \n\n(1) We observe that direct survey generation can lead to disorganized reviews with content repetition and logical confusion, an issue evident even in top-rated models like ChatGPT (Figure 1). \n\n(2) To enable the capability of generating reasonable hierarchical catalogues, we **are the first to** propose a novel and challenging task of Hierarchical Catalogue Generation for scientific Literature Review, which is far different from the previous tasks. We also propose a dataset as well as two evaluation metrics and adapt existing methods to the catalogue generation task.\n\nNote that, we devote our efforts in proposing a new task as well as a stable solution. We thoroughly compare extractive and abstractive approaches, end-to-end generation and step-by-step generation. In addition, we analyze the performance of large language models on this task. We believe our work can contribute to the development of the text summarization community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. nice study on the problem, and the claims,\n2. proposing a new dataset for the task.", "reasons_to_reject": "1. limited dataset size, and biased toward survey papers only, in computer science\n2. very simple method, and idea. it doesn't make a remarkable improvement in the field.", "questions_to_authors": "1. Page 3, column 2, is ambiguous on the final size of the dataset after limiting it to computer-science only papers. is the final size 7637 or 4507?", "paper_topic_and_main_contributions": "They present a task named Hierarchical Catalogue Generation for Literature Review as the first\nstep for review generation, which aims to produce a hierarchical catalogue of a review paper given various references. They construct a novel English Hierarchical Catalogues of Literature Reviews Dataset with 7.6k literature review catalogues and 389k reference papers. To accurately assess the model performance, They design two evaluation metrics for informativeness and similarity to ground truth from semantics and structure.\nThey get their papers from Arxiv and semantic scholars and extract the catalogues from HTML sources of the papers. They focus on surveys and review papers (appearance in the title) in the field of computer science.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SPyuBe6n25", "reviewer": "Reviewer_bVQ3", "summary": "", "questions": "see the reasons to reject\n\nBesides,  what organization of surveys are good and what are not good do you think after analyzing the surveys you collected?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and afﬁrmation of this work.\n\n1. **Organizations behind a survey**\n\nIntuitively, there are potential ways of organizations when writing a literature review. However, the underlying organization of writing requires in-depth knowledge of the specific field in order to mark it out, which is difficult to collect and analyze in large quantities.\n\nWe observe a usually used organization style is \"deduction and summary.” The definition of the topic to be synthesized is presented first. Then present the methods, data, and experiment results of the field. Advanced-level survey papers usually conduct their own experiments rather than simply citing data from other sources. The survey ends with the author's prospective outlook and reflections on the topic.\n\nThe catalogue is the one that reveals how authors organize and can be collected quickly and in large quantities. Beyond standard section headings (Introduction, Background, Conclusion, References), the catalogue usually employs succinct, informative phrases to encapsulate chapter content.\n\nAlthough different people (or different research fields) have different styles, we collect review-catalogue pairs as much as possible and let the generative model learn by itself to implicitly derive the most appropriate organization. In the future, we plan to delve deeper into the organization and prior knowledge required for crafting literature reviews.\n\nWe also plan to conduct a more comprehensive analysis of the quality of the survey papers in our dataset. This will include factors such as the authors' influence, the number of citations to the paper, and its publication status. Additionally, we're considering the incorporation of supplementary writing aids outside of the catalogue, such as human annotations and review paper labels, to assist in survey generation.\n\nThese discussions and results will be integrated into our final version.\n\n2.  **Main findings after experiments**\n\n(1) The hierarchical structure provided by a catalogue is crucial and beneficial in the process of generating literature reviews.\n\n(2) The task of generating a catalogue is indeed challenging, necessitating the generative models to possess an advanced comprehension of the references. Even for sophisticated large language models such as ChatGPT, this task remains a hard one.\n\n(3) End-to-end generation methods are superior to step-by-step generation methods based on BART-like models. There is still a lot of room for improvement in the future.\n\n3. **Missing reference**\n\nWe will cite this work in our final version: Surveytree: Automatic Generation of Survey Structures for NLP and AI topics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research problem is well motivated.\n2. The authors take in-depth analysis of the problem and propose new evaluation methods, which is quite innovative.\n3. The authors construct a dataset to facilitate the future research.", "reasons_to_reject": "1. Different people (or different research fields) have different styles of writing the literature review. So, before jumping into generating the survey catalogs, maybe analyzing what organizations are more common, or investigating what is the logic behind organizing a survey is important. (I think more investigation of the survey papers in the dataset is needed)\n\n2. What are the main findings after the experiments (summarizing the reasons of outperformance of each model )?  I think this part is not clearly presented in the result analysis.", "questions_to_authors": "see the reasons to reject\n\nBesides,  what organization of surveys are good and what are not good do you think after analyzing the surveys you collected?", "paper_topic_and_main_contributions": "This paper addresses the issue of lacking a clear and logical hierarchy in scientific literature reviews by proposing a task called Hierarchical Catalogue Generation for Literature Review. They construct a dataset and evaluation metrics, and benchmark different models to evaluate their capabilities in generating hierarchical catalogs for literature reviews.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Surveytree: Automatic Generation of Survey Structures for NLP and AI topics"}, "extra_text": {}}, {"review_id": "n4XHLzJqva", "reviewer": "Reviewer_9uwx", "summary": "", "questions": "1. How did you control the quaility of the constructed dataset?\n2. How do you guarantee the evaluators are professional?\n3. Will you consider build a human-annotated dataset and release it?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments.\n\n1. **Track of paper**\n\nThank you for your suggestion. Our initial idea is that hierarchical catalogue generation for literature review is a challenging task highly relevant to text summarization. We believe that our work can contribute a lot to the scientific summarization community.\n\n2. **The reliability of proposed metrics and human evaluation**\n\nThe reliability of the proposed metrics is assessed by rigorous  human evaluation. Survey catalogue generation is a challenging task, as it requires specialized background knowledge of relevant domains. Therefore, evaluating the survey catalogue costs a lot, which limits the sample size (450) of our evaluation. We invited three professional Ph.D. students who have published survey papers to make this evaluation reliable. The results of the three evaluators' assessments showed general statistical consistency with a Kappa coefficient of 0.28.\n\n3. **Baselines before 2021?**\n\nWe tested the performance of chatGPT on the catalog generation task.  Despite BART's earlier introduction, it remains a benchmark model for summarization tasks.\n\n4. **Control the quality of the constructed dataset**\n\nIn Section 2.2 (Dataset Construction), we provide an exhaustive explanation of our data collection process, including the decisions and compromises made to ensure optimal dataset quality. \n\nInitially, we identified 11,435 papers from arXiv, deemed suitable as literature review papers. PDF files of these papers are easy to collect but extracting catalogue from PDF files is difficult where structural information is usually dropped during converting. We finally choose to get the papers in HTML format available on the ar5iv website which processes articles from arXiv as responsive HTML web pages by converting from LaTeX via LaTeXML. However, we encountered a limitation where some authors had not uploaded their LaTeX code, resulting in a reduced dataset of 8,397 papers. \n\nOur final dataset, after eliminating data with fewer than five catalogue items and less than ten valid references, comprised 7,637 reference-catalogue pairs. To verify the accuracy of our extraction process, we manually inspected 100 randomly selected samples, confirming the correctness of all extracted catalogues.\n\n5. **Human-annotated dataset**\n\nLarge-scale data labeling is difficult and expensive to perform. We accept Review 2's comment and consider proposing a higher-level human-annotated test set  in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed task is quite interesting.\n2. The dataset could be helpful for future research in this filed.", "reasons_to_reject": "1. This work is more suitable for the resource and evaluation track, as it does not focus on proposing new methods.\n2. The reliability of the proposed metrics, i.e., Catalogue Edit Distance Similarity and Catalogue Quality Estimate, is not well verified. 450 samples are not enough.\n3. The baselines in Sec. 5.1 are all before 2021.", "questions_to_authors": "1. How did you control the quaility of the constructed dataset?\n2. How do you guarantee the evaluators are professional?\n3. Will you consider build a human-annotated dataset and release it?", "paper_topic_and_main_contributions": "This work proposes Hierarchical Catalogue Generation for Literature Review and builds a dataset for it. It also proposes two evaluation metrics for this task, i.e., Catalogue Edit Distance Similarity and Catalogue Quality Estimate.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new task of generating a "catalogue" for literature review given reference papers. For this task, authors provide a dataset, evaluation measures, and evaluated performances of baseline models. 
Reviewers agree on the novelty of the proposed task, and the value of the proposed dataset for further research in this field.
Reviewers also question the choice of baselines, data size, and the validity of the proposed measure, which are addressed by the authors. That being said, constructing a high-quality human-annotated test set is suggested.
Decision: Accept-Findings