Towards a Unified Framework for Reference Retrieval and Related Work Generation
Authors: Zhengliang Shi
Shen Gao
Zhen Zhang
Xiuying Chen
Zhumin Chen
Pengjie Ren
Zhaochun Ren
Venue: EMNLP 2023 Findings (2023)
Keywords: related work generation
lexicon-enhance retrieval
instruction tuning
TL;DR: A Unified Reference Retrieval and Related Work Generation Framework

Abstract: The task of related work generation aims to generate a comprehensive survey of related research topics automatically, saving time and effort for authors. Existing methods simplify this task by using human-annotated references in a large-scale scientific corpus as information sources, which is time- and cost-intensive. To this end, we propose a Unified Reference Retrieval and Related Work Generation Model (UR3WG), which combines reference retrieval and related work generation processes in a unified framework based on the large language model (LLM). Specifically, UR3WG first leverages the world knowledge of LLM to extend the abstract and generate the query for the subsequent retrieval stage. Then a lexicon-enhanced dense retrieval is proposed to search relevant references, where an importance-aware representation of the lexicon is introduced. We also propose multi-granularity contrastive learning to optimize our retriever. Since this task is not simply summarizing the main points in references, it should analyze the complex relationships and present them logically. We propose an instruction-tuning method to leverage LLM to generate related work. Extensive experiments on two wide-applied datasets demonstrate that our model outperforms the state-of-the-art baselines in both generation and retrieval metrics.

Towards a Unified Framework for Reference Retrieval and Related Work Generation
Zhengliang Shi-
Shen Gao Zhen Zhangi Xiuying Chen
Zhumin Chen' Pengjie Ren' Zhaochun Ren**
Shhnndnn University, Qingdao, China "Leiden University, Leiden, The Netherlands
3 *Computational Bioscience Reseach Center, KAUST shizhl@mail sdu edu. cn shengao@sdu.edu..cr xiuying. chen@kaust. edu. zhen. zhang www.smauueal..... sd@ee. com
chenzhumi n@sdu. edu. cn jay. www@eeeeooo..cooooo.ooooooo.o.. com z. rerrliicce leidenuniv.m nl
Abstract
Witting
Abstract
The task of related work generation aims to generate a comprehensive survey related re- search topics automatically, saving time and effort for authors. Existing methods simplify this task by using human-annotatee references in large-scale scientific corpus informa- tion sources, which is time- and cost-intensive. To this end, propose Unified Reference Retrieval and Related Work Generation Model (UR$WG), which combines reference retrieval and related work generation processes in uni- fied framework based on the large language model (LLM). Specifically, UR first lever- ages the world knowledge of LLM to extend the abstract and generate the query for the subse- quent retrieval stage. Then lexicon-enhanced dense retrieval is proposed to search relevant references, where an importance-aware repre- sentation of the lexicon is introduced. We also propose multi-gu multi-granularity contrastive learning optimize our retriever. Since this task not simply summarizing the main points in refer- ences, should analyze the complex relation- ships and present them logically. We propose an instruction-tune -tnnig method to guide LIM to generate related work. Extensive experiments wide-applied datasets demonstrate that our 'WG outperforms the state-of-the-art baselines both generation and retrieval met- rics.
Reforence
Releted
Mathod
Abstroct
Willing
Related Work
Cuary
Curry
Reefernnce
Figure I: Comparison between the existing related work generation methods and our UR'WG. We propose to retrieve references and generate related work without references in unified model.
However, most existing RWG methods (Chen et al., 2022a, 2021; Wang et al., 2019) only focus on the second stage, where the human-annotated refer- papers are taken as input via summariza- tion model. Consequently, retrieving related papers from the sarge-scale corpus remains underempha- sized. To address this problem, we focus on amalga- mating the above two stages: retrieving the related papers and generating the related work. Our unified model, unlike previous RWG research, alleviates the user's burden by requiring only high-level abstract of the target paper input. This approach minimizes the workload for practical purposes.
1 Introduction
An intuitive retrieval practice in RWG is tak- ing the abstract as a query and employing dense retrieval methods to obtain the related papers ac- cording to semantic similarity. However, the ab- stract given by the user is too vague to be clear intention for reference. Some related topics are inherently correlated with the paper but not explicitly mentioned in the abstract, indicating mantic mismatches. Moreover, the presence of rare proper nouns, e.g., specialized terminologies in sci- entific papers, makes it difficult for existing dense
The automatic related work generation (RWG) sys- tem, which aims at generating a related work sec- tion for target paper, may help the readers go through the cutting-edge research progress (Hoang and Kan, 2010). Two main adjacent stages considered fundamental disciplines in RWG task: (1) selecting related reference papers and (2) fig- uring out the logical relation to write a summary to present the evolving process of a specific field.
Corresponding author.

---

retrieval methods to capture the semantic nuances. To generate the related work section, common- used alternative is leveraging the summarization models (Chen et al., b0. Nevertheless, an ideal related work section should reason intricate relations among various references, instcad of sim- ply enumerating their main contributions. There- fore, training the model to grasp the concept of RWG while producing high-quality related work section remains challenging.
ority of our proposed model.
Related work
2.1 Related work generation
Similar to multi-doeument summarization, the task of related work generation (RWG) which usually takes multiple reference papers input and gener- a related work section for target paper (Chen et al., 2022a; Hoang and Kan, 2010) via summa- rizes the related information in a logical order. Ex- isting RWG methods can be divided into extractive and abstractive methods. Specifically, extractive methods select subset of words or sentences most relevant to the input abstract to form the final re- lated work (Hoang and Kan, 2010; Hu and Wan, 2014; Deng et al., 2021) With the emergence of neural-based models (Lewis al., 2019; Raffel et al., 2019; Touvron al., 2023), more abstractive methods are utilized to solve the RWG task (Zhao et al., 2020a; Chen et al., 2022b). For example, Chen et al. (2022a) leverage the information of the target paper and propose target- aware graph en- coder to model the relationship between reference papers. Although these methods have shown exem- plary performance, they rely on the human-labeled references in the target paper, which still requires a lot of manual retrieval work.
In this work, we leverage the large language model (LLM) to enhance retrieval and generation stages the RWG task to tackle the above chal- lenges. As shown in Figure 2, we propose an Unified Reference Retrieval and Related Work Generation framework (UR$WG), Specifically, we leverage the LLM to extend the vague abstract and generate query containing more relevant background information since the LLM has shown strong knowledge association ability (Wang al.. 2023). The generated query is used subsequent retrieval stage. To mitigate the diminished perfor- mance of dense semantic retrieval methods, we pro- pose a Lexicon-Enbanced dense Retrieval (LER) method to integrate the advantages of lexical re- trieval, where a learnable lexical text-matching al- gorithm is introduced. We also propose multi- granularity contrastive learning method to opti- mize our retriever, including group-wise and pair- wise contrast. Since the related work section should comprehensively introduce the relationship between references, instead of simply summarizing the main points, we propose an instruction-tuning method to guide LLM to generate high-quality related work section. Extensive experiments con- ducted on two benchmark datasets show that our model significantly outperforms all the strong base- lines, e.g., pushing the ROUGE-1 to 31.59 (8.4% relative improvement) and BERTScore to 0.70 (6.9% relative improvement..
2.2 Retrieval argument generation
Information retrieval has been widely used in many knowledge-intensive natural language generation tasks, e.g., question answering (Guu et al., 2020; Gao et al., 2023) and mhowledge-groundee logues (Zhao et al., 2020b; Meng et al., 2020; Li et al., 2021). For example, Lewis et al. (2020) combine parametric and non-parametric knowl- edge, where the former employs the pre-train lan- guage model as a knowledge base while the lat- ter is dense vector index of Wikipedia, accessed with pre-trained neural retriever. Izacard and Grave (2021) propose to first retrieve passages via sparse, e.g., BM25 or dense retrieval, e.g., DPR (Karpukhin et al., 2020) to obtain the relevant passages. Then sequence-to-sequee model is employed to generate the answer based on the concatenation of the representations of all the retrieved passages.
dia-
Our contributions are as follows: (1) We propose the UR3WG, the first unified reference retrieval and related work generation model in the RWG task. (2) We propose a cexicon-enhance dense re- trieval method with an LL---ased query extension method to retrieve highly related reference papers, supervised by gualiligranulrrity contrastive learn- ing. (3) We propose an Iestrcction-tningg method to incorporate the references and guide the L.M to generate related work logically. (4) Experimental results on two benchmark datasets show the superi-
(seq2seq)
These retrieval- -uggeeeedd methods are usually designed for scenarios with an explicit and clean re- trieval goal. However, there have been few attempts at the RWG task. On the one hand, the model only

---

Abstract
Lexicon Matching
Query
Query
Index Mapping
Group-Wise Max
Lexicon Meltching
Semantic Matching
Max Pooling
Max Pooling
LLMs
Lineer
Semantic
Document Index
Matching
Top
[CLS]
[CLS]
Related Work
Query Encoder
Document Encoder
Instruction Following
Query
Document
Figure 2: Left: The architecture of proposed framework for reference retrieval and related work generation. Right: the Lexicon-Enhance Dense Retrieval (LER).
uses the abstract as input, and the abstract does not indicate which topic should be surveyed in the re- lated work. On the other hand, it is noteworthy that
inition of the RWG task and generate the high- quality related work section.
4.2 Query extension
the citation relation among multiple documents is implicit in nature, thereby necessitating the reason- ing understand.
It has been proven that achieving better retrieval performance feasible if the query can be rewrit- ten to a more similar form to the candidate docu- ments (Yu et al., 2022). Since the LLMs trained on massive textual corpora have shown strong knowl- edge association ability (Wang al., 2023), we leverage the knowledge of L.MM to extend the ab- stract X and formulate pseudo references R as an extended abstract, which contains more rich infor- mation in the RWG task. The R. is taken as a query to search relevant references accurately. Specif- ically, we first design instruction IQE which contains two parts: (1) DQE is demonstration ample which describes the definition of the query extension task, and (2) X is the abstract of the target paper:
3 Task formulation
In this paper, we generate the related work section of the target paper and only use the abstract X as the input without the ground-truth references R Specifically, we propose a unified reference retrieval and text generation framework, which con- sists of three steps: (I) leverage the knowledge of LLMs to extend the input abstract and generate the query used for the subsequent retrieval stage. (2) retrieve reference papers R {ri, F2, TpR from the document corpus D = {d1, d2, d/p) according to the similarity S to query at seman- tic level S₃ and lexicon level Sü (3) generate the related work section Y of the target paper by sum- marizing the retrieved references R.
IQE: Demonstration (DQE)
Input {X).
4 Methodology
Then, we leverage the LLM to generate pseudo references R. on the condition of instruction Iqe- Since R. is highly relevant to the abstract X and can be potentially cited, we take R. as the query during the subsequent retrieval stage, assisting in query disambiguation and guiding the retriever (Wang et al., 2023). And we optimize the proposed UR3WG via standard language modeling objec- tive:
4.1 Model overview
As shown in Figure 2, UR3WG first generates the reference retrieval query by extending the abstract X. Then a lexicon dense retriever (LER) retrieves relevant references by using the extended abstract the query. Since RWG is different from simple text summarization tasks, the logical rela- tion between input abstract and references should be carefully considered. We employ an instruction- tuning method to train LLM to understand the def-
|R8 Iral
LoE
log (o (t) (1(()))))
,IQE), (1)

---

where the yp is the learnable parameters and R* is the set of ground-truth references.
4.3.2 Lexicon-oriented retrieval
To capture the information of local phrases and en- tities, we propose a token-level interaction network with guided interaction mechanism that calcu- lates the lexicon-level similarity. The architecture illustrated in Figure 2. Specifically, we construct a continuous bag-of- -f-ddds representation for each token in q based on Hg:
4.3 Lexicon-enhanced dense retrieval
Current trainable dense retrieval models built on language models have shown remarkable capa- bility in capturing sentence-level similarity. De- spite these advances, they exhibit diminished per- formance when retrieving references due to their omission significant local phrases and enti- ties (Zhang et al., 2023), which leads to outcomes worse than those obtained via traditional lexicon- based retrieval methods (Shen et al., 2022; Thakur et al., 2021; Lin and Ma, 2021a). To alleviate this problem, introduce Lexicon-Enhanced dense Retrieval (LER) method that explicitly incorpo- rates importance-uwwrre lexicon representations into dense semantic representations. We also design an approximation to accelerate this method for appli- cation in the large-scale corpus.
Ey softmax H. Wg) E RIIXM (5)
where Wg E RMXM the trainable parameter, and the [V] is the vocabulary size. The E, indi- cates the importance of the input token to all the vocabulary. Then lexicon representation of query can be constructed via max-pooling layer along each token of Eg:
0g MaxPooling(E.)) E R
(6)
Similarly, we obtain the lexicon representation 0₂ for the document d via:
4.3.1 Semantic-oriented retrieval
Ed = softmax W E 4 0. MaxPooling (Ea) RR
(7)
Inspired by the dense retrieval methods (Khattab and Zaharia, 2020; Santhanam al., 2023), we em- ploy dual independent dense encoders: Encoderg and Encodera, which map the query and candi- date reference d (a..a., document) to token-level representation, respectively. Specifically, we add the [CLS] the beginning of and d. The query representation H,, is obtained via:
(8)
where the W C RRmx is the trainable parameter. Although the bag-of-words representation con- tains tne-ggained importance of lexicon, the repre- sentation 0g over the entire vocabulary is too sparse and contains much noisy information. Therefore, propose group-wise local threshold mech- anism that partitions the sparse representation fg into several groups containing & tokens, and only remains the maximum value each group:
H = Encoder,
(2)
where H, = (h n2 .,......//b))) E Rkix The hy indicates the representation of each token, and m is the feature size. Similarly, we derive the token- level representation of each document d through:
| k
0 = (max(0 = nk, n S
)
(9)
V
Tq = (argmax(0) ( nk, -
k ). (10)
H₂ (ncodera(d).
where the Tg is the index of the maximum of each group. indicating the most important lexicon for query q in this group. And the interaction between the sparse representation of and d can be guided by Ta to calculate the lexicon-level similarity:
where H. = 13....1144 E kid and |d] is the length of the document. We take the embedding of (CLS) token as the semantic representation. The semantic similarity between query and document d can be defined based on the inner product:
ITal
Si d)
0'
(11)
1=0
Sz d) Lin (h))
(4)
Finally, we combine the similarity score of se- mantic and lexicon levels as the final matching score for the query g and document d:
where Lincar denotes linear layer with an activa- tion function to compress and extract features from the raw representation.
S(q, d) = (.(g.d) + Si(g.d).
(12)

---

For each query q, we retrieve the top-k candidate reference R as input. Considering the scientific paper corpus is large-scale in real-world scenarios while the online interaction mentioned in Equa- tion 11 is time-intensive, we design an approximate algorithm for offline matching. The details and qualitative analysis are provided in Appendix A.I.
where the À is the margin that we defined explic- itly constrain the gap between positive and negative references. Our contrastive loss is the combination of the LG and Lp:
Lcr. = pCa + (1- p4)
(15)
where is the weight of two losses.
4.3.3 Multi-granularity contrastive learning Unlike the existing passage retrieval task which assumes there is only one relevant document for query (Karpukhin et al., 2020; Khattab and Za- haria, 2020), in the RWG task, a scientific pa- per has multiple diverse references. We propose a multi-gramuuarrty contrastive learning method to alleviate this problem, including group-wise and pair-wise contrast. Specifically, we denote the reference cited in the target paper as R+ (ri r ................................ | and the irrelevant paper as R (ri :r::12 ........ )R))
4.4 Instruction-tunnng for generation
Instruction tuning has been shown to improve the performance and generalization of LLMs in com- plex tasks for supervising the LLMs following con- crete instructions (Chung et al., 2022; Wei et al., 2021). Since the definition of the RWG task is complex and different from summarizing text di- rectly, we design an instruction-tuning method to help LLM understand the approach of generating related work. Specifically, we describe the defini- tion of the RWG task via the demonstration DGEN- The DGEN paired with abstract X and reference R are concentrated construct the instruction IGEN, which can be formulated as:
Group-wise contrastive learning. Given the ab- stract X of the target paper, the goal of our group- wise contrastive learning to minimize the dis- tance between the abstract X and the positive ref- erence group R and maximize the distance to the irrelevant document group R- Specifically, we employ negative log-likelihood as the loss function for positive references:
IGEN = Demonstration {DGEN)
Input : {abstractX) Reference [R].
We then leverage the LLM to generate related work on the condition of the IGEN, prompting LLM to understand the definition of the RWG task and gen- erate the related work section logically, which is supervised via the instruction-tuniin loss:
Lo(g, R* RR
(13)
R+1 == e slar!)
Σ
loo E
((gr()
IIR-1
RR- ((rr7)
Pair-wise contrastive learning. Most of the ex- isting retrieval methods (Qin et al., 2022; Hossain et al., 2020) obtain the relevant documents by rank- ing the candidate documents based on the similar- ity to the query. They only minimize the negative log-likelihood of the positive documents, implic- itly optimizing their model to binary classify the documents. However, the ranks of documents also provide a training signal to optimize the retrieval module. Specifically, we propose pair-wise con- trastive loss Lp which contrasts each positive ref- erence rit in R+ and negative reference rī in Rt. explicitly optimizing the model rank r/ before rT- The Lp can be formulated as:
M log Po(U, (<0) (9 10,000 IGEN). (16)
LGEN =
Σ 1=1
The is the ground truth related work of the target paper and yp is the parameters shared with Equa- tion 1.
4.5 Multi-task learning
We jointly optimize URèWG with two tasks, i.e., query extension in 4.2 and instruction-followii generation in $ 4.4. The final training objective is defined as:
Lp(g. R
(14)
J + MLGNN
(17)
R R
max(0, ((g.r,,)--((4 + +),,
where the a and are the syper-parameters, denot- ing the weights of the two losses, respectively.
0 0=0

---

Table 1: Results on two datasets. We abbreviate ROUGE as R. We underline the best results each category baseline methods. All ROUGE scores have 95% confidence interval reported by the official ROUGE script.
TAS2 Dataset
TAD Dataset
Methods
R-1
R-2
R-SU
R-1
R-2
R-L
R-SU
Multi-document sunmarizztion methods
NES (Wang et al., 2019)
26.04
3.39
22.46
6.14
26.13
3.24
23.20
MGSum (Jin et al., 2020)
25.54
3.75
23.16
6.49
27.49
4.79
25.21
7.29
LexRank (Erkan and Radev, 2004)
25.74
2.81
22.43
6,03
25.70
2.86
22.68
5.90
BertSumEXT (Liu and Lapata, 2019)
25.85
2,90
22.66
6.21
25.95
2.91
23.05
6.25
BertSumABS (Liu and Lapata, 2019)
25.45
3.82
23.04
6.39
27.42
4.88
25.15
7.22
EMS (Zhou et al., 2021)
26.17
4.16
23.63
6.67
28.21
5.15
25.74
7.56
Large language models
Llama-7B (Touvron et al., 2023)
18.28
2.05
15.29
4.01
15.45
1.62
13.48
Claude
27.59
3,88
24.99
7.28
30.69
5.11
28.00
Vicuna-7B
28.40
4.01
25.39
7.01
31.09
4,91
27.62
Davinci-text-003
22,80
2.49
20,26
5.08
24.59
3.15
21.97
CharGPT
29.13
4.08
25.59
7.89
31.41
5.34
28.15
9.33
Related work generation methods
RRG (Chen et al., 2021)
26.79
4.43
24.46
6.85
28.94
5.59
26.46
TAG (Chen al., 2022a)
28.04
4.75
25.33
7.69
6.16
27.79
8.89
UR WG
31.59
5,86
26.13
9.62
32.68
7.74
28.87
9.54
Ablation study
w/o Los
28.30
4.25
7.35
31.05
5.77
27.64
w/o LOEN
30.42
5.02
25.11
8.10
30.77
4.97
26.78
5 Experiment Setup
well-educated Master students are invited to judge 40 randomly sampled examples with a three-scale in the four aspects: Relevance (Rel.), Fluency (Flu.). Coherence (Cohe.), Informativenes: (Info.). The details of these aspects can be found in Ap- pendix A.5.
5.1 Datasets
We conduct experiments on two widely-applied datasets: TAS2 and TAD (Chen et al., 2022a). Specifically, the TAS2 consists of 117,700 scien- tific publications from several fields, whereas TAD consists of 218,255 scientific publications from the computer science field. For each example in datasets, includes (I) the abstract of the article, (2) a paragraph of the related work, and (3) the reference paper cited in this paragraph (four refer- ences per paragraph on average).
5.3 Baselines
We mainly compare our UR3WG with three types of baselines: multi-document summarization meth- ods, related work generation methods, and large language models.
The multi-document summarization baselines in- clude extractive methods, such as LexRank (Erkan and Radev, 2004), BertSunExt (Liu and Lapata, 2019) and NES (Wang et al., 2019) as well as ab- stractive methods like BertSumAbs (Liu and La- pata, 2019) and EMS (Zhou et al., 2021). These baselines take the references as input. The related work generation methods include RRG (Chen al., 2021) and TAG (Chen et al., 2022a), which take the abstract and references as input. The large lan- guage models include Llama-7B (Touvron et al., 2023), Vicuna Claude 2 Davinci-rext-003, and ChatGPT which take the instruction IGEN input. We also compare our retrieval method LER with strong baselines, including traditional term-based
5.2 Evaluation metrics
Following Chen et al. (2018): Zhou et al. (2021); Chen et al. (2022a), mainly employ ROUGE (1,2,L,SU) for evaluation. Since only using the ROUGE metric to evaluate generation quality can be misleading (Shi al., 2023), also con- sider the other metrics, including the lexicon-based metric, e.g., BLEU (Papineni et al., 2002) and semantic-based metrics, e.g., BERTScore (Zhang et al., 2019) and BARTScore (Yuan et al., 2021). We evaluate the retriever with Recall@t @ K (k=5, 10. 20) metrics. The statistical significance of differ- ences observed between the performance of two runs is tested using two-tailed paired t-test at a 0.01 and p < 0.05.
htttps://githhb. wwwm//mmss//astttttitico http::://ww..attroph.. http://pppaai..cmmm//////ettttg
we also conduct the human evaluation. Three

---

Table 2: The recall 10,5,10,20) score for retricval methods on two datasets.
TAS2 Dataset
TAD Dataset
Methods
Recall @
Recall 10
Recall (8 20
Recall
Rocall @ 10
Recall 20
BM25 (Robertson ul., 2009)
6.32
9.79
13.82
2.62
7.10
DPR (Karpukhin et al., 2020)
13.23
18.30
27.89
6.28
16.9
DLR (Lin and Lin, 2022)
23.68
25.77
30.57
15.61
18.87
27.95
SPLADE (Formal et al., 2021)
24.21
30.51
37.31
15.70
20.80
26,29
ColBERT (Khattab and Zaharia, 2020)
15.56
21.14
27.89
23.34
24.12
25.11
Condenser (Gao and Callan, 2021)
20.84
28.94
38.48
11.22
16.61
23.94
UniCOIL (Lin and Ma, 2021b)
15.35
19.33
24.25
8,74
11.60
15.16
Lexicon-enhanced retrieval (LER)
37.35
41.56
44.17
31.20
40.20
49.47
Ablation study
w/o Lou
29.30
33.70
37.28
23,47
23.22
27.77
w/o Cp
33,24
39.15
40.21
25.41
38.01
45.14
w/o St
34.10
40.23
41.03
29.16
35.26
44.87
methods like BM25 (Robertson et al., 2009), and dense retrievers, such DPR (Karpukhin et al., 2020), ColBERT (Khattab and Zaharia, 2020), Con- denser (Gao and Callan, 2021), as well as lexicon- aware retrievers, e.g., DLR (Lin and Lin, 2022), SPLADE (Formal ct al., 2021) and UnICOIL (Lin and Ma, 2021b). Since the abstract too long with many irrelevant words, we take concatenation of the keywords extracted from the abstract as the query. To ensure the fairness of comparison, all the baselines are finetumed with the same datasets our proposed method. More details about the baselines are provided in Appendix A.3.
datasets, with 8.4% and 25.6% relative improve- ment compared to existing state-of-the-a baseline. As shown in Table 4, also select several strong baselines for more comprehensive evaluation. The proposed UR WG outperforms the strong base- lines terms of the lexicon-level and semantic- level metrics, e.g., pushing BLEU-I to 23.22 and BERTScore to 0.70 in TAS2 dataset. These results demonstrate that our UR fits well in generat- ing high-quality related work with the assistance of curated instruction.
Retrieval performance evaluation. As shown in Table 2, our retrieval module outperforms the strong retrieval baselines by large margin. For ex- ample, LER reaches Recall @5537.35 in the TAS2 dataset with 13.1 14 absolute improvement compared to the state-of-the -frtt---rttt baseline SPLADE, which il- dense
5.4 Implementation details
We initialize the parameters with Llama-7B (Tou- vron et al., 2023). We vary the weight of contrastive learning loss in {0.5, 0.6, 0.7, 0.8, 0.9), and find that 0.7 achieves the best performance. Follow- ing the Sun al. (2022), we tune the weight to 0.5, 7) to 0.5 initialization, respectively, and linearly decrease to 0.3 and 0.2. We optimize the model using AdamW optimizer with parameters 31 = 0.98, 32 = 0.99, the learning rate of 25-5, and the weight decay coefficient of 0.01. The train- ing of our model can be done within 28 hours on the TAD dataset and 17 hours the TAS2 dataset using four NVIDIA A100 PCIE GPUs.
lustrates that our proposed retrieval is more suitable for reference retrieval sce- narios. We also consider several variant models for more comprehensive comparison in Section 7.
6.2 Human evaluation
Table 6 shows the results of the human evalua- tion. We find that the UR²WG outperforms the best RWG baselines TAG in four aspects, e.g. push- ing Relevance to 2.77 (0.47 absolute improvement) in TAD dataset. We also observe that UR3WG achieves comparable and better performance with ChatGPT (175B) with only 7B parameters, indi- cating the effectiveness of our method. The erage Kappa statistics for four evaluation metries are 0.72, 0.70, 0.71, and 0.73, illustrating agree- ment among the annotators. More results details are provided in Appendix A.5.
6 Result and analysis
6.1 Experiment result
Generation performance evaluation. Table shows the details of the results. Overall, WR achieves the best performance compared to the other baselines. For example, UR gets HOUGE-1=32.66 ROUGE-2-7.74 in TAD

---

Table 3: The recall score for retrieval on two datasets. We take the input abstract, extended abstract (denoted as abstract') and keywords extracted from input abstract as the query search relevant documents, respectively.
TAS2
TAD
Method
Query
@
Recall @
@
Recall@
Recall
Recall
abstract
7.13
11.49
17.48
3.99
6.81
11.04
ConDenser
abstract"
29.27
32.56
36,04
19.6
21.89
24.49
keywords
11.22
16.61
23.94
20.84
28.94
38.48
abstract
2.93
5.16
8.07
1.91
3.31
5.34
ColBERT
abstract*
32.98
33,92
35.33
23.34
24.12
25.11
keywords
12.47
17.14
22.63
10.55
14.70
19.26
abstract
6.08
9.08
13.10
4.06
6.48
9.59
SPLADE
abstract*
36.38
41.92
47.20
30.42
36.14
41.36
keywords
24.21
30.51
37.31
15.70
20.80
26.29
abstract
2.13
4.62
6.57
1.76
2.67
4.13
UniCOIL
ubstract*
19.18
21.33
24.15
12.72
14.26
16.16
keywords
15.35
19.33
24.25
8.74
11,60
15.16
Approximation
LER (w. approx.)
abstract'
35.59
40.50
43.23
29.14
40.11
48.32
LER
abstract"
37.35
41.56
44.17
31.20
40.20
49.47
Table 4: More evaluation results. B1: BLEU-1; B2:
references and final related work section.
BLEU-2: BES: BERTScoor, BAS: BARTScore.
-w/o LGEN. We replace the instruction-tuning
loss LGEN with standard language modeling ob- jective (Zhao al., 2020c; Johansen and Juselius,
Methods
B1
BB2
BES
BAS
B1
B2
BES
BAS
TAG
17.23
7.50
0.660
-4.61
18.82
7.61
D653
-4.76
2010), where take the concatenation of abstract
Vicuna
14.17
4.34
0.504
-185
17.61
6.18
0.651
-5.00
and references as input, supervising the model pre- diet the ground- -ound related work. As shown
Davincisext-Ons
0.589
ChaGGT
16.77
5.18
0.645
-5.37
15.93
5.25
0.654
55200
UR"WG
23.22
8.71
0.700
-3.74
24.31
7.64
0,674
-4.00
Table 1, the performance significantly declines, demonstrating that the instruction we design fits the LLMs well in the RWG task and improves the capability in complex generation scenarios.
Ablation smudy
wio Cqp
22:47
8.09
0.564
-4.34
23.72
7.01
0,656
-497
wio CorN
22.47
8.64
0.650
4.00
22.98
0.632
6.3 Ablation study
6.4
Case study
To better understand the impact of different com- ponents of our UR$WG, we employ the following modifications to the architecture.
We conduct several case studics and find that UR2WG is more effective at retrieving relevant ref- erences and generating more informative text than baselines. Details are provided in Appendix A.6.
-w/o Cp. We remove the loss Lp mentioned in Equation 15. We observed an average decrease in recall of 10 on both datasets, e.g., the Re- c 10 drops from 41.56 to 39.15 with 6. 6.16% relative decrease, 40.20 38.01 with 5.76% rel- ative decrease, which indicates the effect our multi-granularity contrastive.
7 Discussion
Variant model for reference retrieval. For each retrieval, we take the abstract, keywords of abstract, and extended abstract as the query search rele- vant documents. Specifically, the keywords are extracted via KeyBert (Grootendorst, 2020), and extended abstract is generated via the proposed UR 3 WW. As shown in Table 3, for each retrieval method, taking the extended abstract as the query reaches the best performance, which indicates that our query extension method can be combined with different retrieval methods. And this result further illustrates that the extended abstract contains rich information about the target paper. We also observe a significant decline when we replace the query
-w/o S₁. We remove the lexicon matching score Sı(g,) d) (in 4.3.2). From the results shown Ta- ble 2, the recall metrics have a significant decline, which indicates that the lexicon information is criti- cal to retrieve relevant references the RWG task.
-w/o LQE- We remove the loss Loe for query extension. As shown in Table and Table both the generation and retrieval performance suffers from significant decrease. This result indicates that the external knowledge injected via LQe assists the U WW in generating more informative pseudo

---

(a) Recalle
(b) TAS2/Recal
(c) TAD/Recall@s
(d) TAD/Recal 10
0.40
0.45
0.33
0.43
0.34
0.40
0.27
0.37
0.27
0.35
0.21
0.31
0.20
0.30
0.15
0.25
128 512 11127681022 2048
128 512 7681024 2048
128 1024 2048
128
512 768 1024 2048
Figure 3: The retrieval performance of TAS2 dataset (a and and TAD dataset (c and d) based on different slice. For each figure, the horizontal axis indicates the group size (128, 512, 768, 1024, 2048) and the vertical axis indicates the corresponding Recall@k score (k=5, 10).
from abstract to keywords. The potential reason is that the abstract is too long with many irrelevant words, hindering the performance of retrieval.
Since RWG is different from simple text summa- rization tasks, employ instruction-tuning method, enabling the LLM understand the def- inition of RWG task and generate high-qualing related work section. Experiments on two datasets demonstrate that our model establishes a new state- of the-rtt with automatically retrieved references.
The impact of lexicon representation group size. we conduct experiments to alternate the sparse lex- icon representation into different dimensions. As shown in Figure 3, find that the recall keeps increasing and peaks 1024 (TAD datasets) and 768 (TAS2 datasets), indicating that more repre- sentative lexicon information can be used match relevant documents in the retrieval stage. How- ever, when the dimension increases to 2048, the recall decreases, suggesting that some low-featured lexicon information is retained.
Limitations
The main limitation of this paper is the requirement of massive computation resource. We will explore how to employ our method in low-resource scenar- ios. Another limitation is that we only consider the single-lingual retrieval scenario. In the future, we plan to extend our model to multi-lingus applica- tions, which is common in real-world.
Qualitative analysis about the approximation We explore the impact by comparing the perfor- mance before and after approximation, and results are are shown in Table 3. Comparing with the vanilla LER, we observe that LER (w. approx.) an average decrease of about 3. 131% and 3.68% in TAS2 and TAD datasets, respectively. This result illustrates that the approximation can improve re- trieval efficiency a slight performance cost. More details can be found in Appendix A.2.
Ethics Statement
The paper proposes UR UGG, a unified reference retrieval and related work generation framework. Given the abstract as input, the UR²WG first searches relevant papers as references and generate related work. Moreover, reference papers should be cited appropriately.
Acknowledgements
8 Conclusion
This work was supported by the National Key R&D Program of China with grant No. 2020YFB1 1406744 the Natural Science Founda- tion of China (22293773, 62272274, 61972234, 62072279, 62102234, 62202271). the Natu- ral Science Foundation of Shandong Province (ZR2022QF004), the Key Scientific and Techno- logical Innovation Program of Shandong Province (201)22YY111129) the Fundamental Research Funds of Shandong University.
this paper, we propose a UR3WG, which fuses the reference retrieval and related work generation into a unified framework based on large language models (LLMs). Concretely, UR UWG first extend the input abstract and generate the query used for next retrieval stage. Then a lexicon-enhanced dense retrieval (LER) is proposed retrieve the relevant document the semantic and lexicon level. The LER is optimized via multi multi-granularity contrastive learning, including group- and pair-wise contrasts.

---

References
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expan- sion model for first stage ranking. In SIGIR.
Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Rui Yan, Xin Gao, and Xiangliang Zhang, 2022a. Target-aware abstractive related work generation with contrastive learning. In SIGIR.
Luyu Gao and Jamie Callan. 2021. Condenser: a pre- training architecture for dense retrieval. In ACI..
Xiuying Chen, Hind Alamro, Li Mingzhe, Shen Gao, Xi- angliang Zhang, Dongyan Zhao, and Rui Yan. 2021. Capturing relations between scientific papers: An abstractive model for related work section generation. ACL.
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit exact lexical match information retrieval with contextualized inverted list. In NAACI.
Shen Gao, Zhengliang Shi, Minghang Zhu, Bowen Fang. Xin Xin, Pengjie Ren, Zhumin Chen, and Jun Ma. 2023. Confucius: Iterative tool learning from intro- spection feedback by easy-to-difficllt curriculum.
Xiuying Chen, Shen Gao, Chongyang Tao, Yan Song, Dongyan Zhao, and Rui Yan. 2018. Iterative docu- ment representation learning towards summarization with polishing. In EMNLP.
Maarten Grootendorst. 2020. Self-supervised con- textual keyword and keyphrase retrieval with self- labelling.
Xiuying Chen, Mingzhe Li. Shen Gao. Zhangming Chan, Dongyan Zhao, Xin Gao, Xiangliang Zhang, and Rui Yan. 2023a. Follow the timeline! generating abstractive and extractive timeline summary in chronological order. ACM Transactions on Informa- tion Systems.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In ICML.
Xiuying Chen, Mingzhe Li, Shen Gao, Xin Cheng, Qiang Yang, Qishen Zhang. Xin Gao, and Xiangliang Zhang. 2023b. A topic-aware summarization frame- work with different modal side information. Proc. of SIGIR.
Cong Duy Vu Hoang and Min- Yen Kan. 2010. Towards automated related work summarization. In COLING.
Nabil Hossain, Marjan Ghazvininejad, and Luke Zettle- moyer. 2020. Simple and effective retrieve-edit- rerank text generation. In ACL.
Xiuying Chen, Mingzhe Li, Shen Gao, Rui Yan, Xin Gao, and Xiangliang Zhang. 2022b. Scientific pa- per extractive summarization enhanced by citation graphs. arXiv.
Yue Hu and Xiaojun Wan. 2014. Automatic genera- tion of related work sections in scientific papers: An optimization approach. In EMNLP.
Hyung Won Chung, Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai. Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra. Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu. Slav Petrov, Fdd H. Chi, Jeff Dean, Jacoh Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction finetuned language mod- els.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In EACL.
Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020. Multi-granulane granularity interaction network for extractive and abstractive multi-document summarization. In ACI.
Søren Johansen and Katarina Juselius. 2010. Maximum likelihood estimation and inference cointegration with applications to the demand for money. Oxford Bulletin of Economics and Statisties.
7ekun Deng, Zixin Zeng, Weiye Gu, Jiawen Ji, and Bolin Hua. 2021. Automatic related work section generation by sentence extraction and reordering. In Alf@:Conference.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Lewis, Patrick andw Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage re- trieval for open-domain question answering. EMNLP.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over BERT. In SIGIR.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregres- sive blank infilling. In ACI.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising requence-to-sqqeeme pre-training for nat- ural language generation, translation, and compre- hension. In ACI.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text sum- marization. Journal of Artificial Intelligence Re- search.

---

Patrick S. Lewis. Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Michael Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Searieval-augmentee generation for snowledge-intensir nip tasks. In NewrIPS.
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiao- long Huang, Binxing Jiao, Linjun Yang. and Daxin Jiang. 2022. Lexmae: eexicon-bottleneckee pretrain- ing for large-scale retrieval. ArXiv.
Zhengliang Shi, Weiwei Sun, Shuo Zhang. Zhen Zhang. Pengjie Ren, and Zhaochun Ren. 2023. Rade: weference-assister dialogue evaluation for open- domain dialogue. In ACL.
Jinpeng Li. Yingce Xia, Rui Yan. Hongda Sun, Dongyan Zhao, and Yan Liu. 2021. Stylized dialogue gen- cration with multi-pass dual learning. Advances in Neural Information Processing Systems, 34:28470- 28481.
Weiwei Sun, Zhengliang Shi, Shen Gao, Pengjie Ren, Maarten de Rijke, and Zhaochun Ren. 2022. Con- trastive learning reduces hallucination in conversa- tions, arXiv.
Jimmy Lin and Xueguang Ma. 2021a. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv.
Nandan Thakur, Nils Reimers, Andreas Rückllé Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, arXiv.
Jimmy Lin and Xueguang Ma. 2021b. A few brief notes on deepimpact, coil, and conceptual framework for information retrieval techniques, arXiv: Information Retrieval.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 2rX1v.230.113971.
Sheng- Chich Lin and Jimmy Lin. 2022. A dense repre- sentation framework for lexical and semantic match- ing. ACM Transactions on Information Systems.
Sheng-Chieh
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. EMNLP-IJCNEL
Liang Wang, Nan Yang. and Furu Wei. 2023. Query2doc: Query expansion with large language models. arXiv preprini uxXv:2303.0000
Chuan Meng, Pengjie Ren, Zhumin Chen, Weiwei Sun, Zhaochun Ren, Zhaopeng Tu, and Maarten de Ri- jke. 2020. Dukenet: A dual knowledge interaction network for mnowledge-rroudeer conversation. In SIGIR.
Yongzhen Wang, Xiaozhong Liu, and Zheng Gao. 2019. Neural related work summarization with a joint context-driven attention mechanism. In EMNLP.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M Dai, and Quoe Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.0163
Stephen Mussmann and Stefano Ermon. 2016. Learning and inference via maximum inner product search. In ICMI..
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method automatic evalu- ation of machine tramslation. In ACL.
W. Yu, Dan Iter, Shuohang Wang. Yichong Xu, Mingx- uan Ju, Soumya Sanyal. Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models strong context generators. ArXiv.
Jiarui Qin, Jiachen Zhu, Bo Chen, Zhirong Liu, Weiwen Liu, Ruiming Tang, Rui Zhang, Yong Yu, and Weinan Zhang. 2022. Rankflow: Joint optimization of multi- stage cascade ranking systems as flows. In SIGIR.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In NIPS.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text text trans- former. Journal of Machine Learning Research.
Kai Zhang, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, and Daxin Jiang. 2023. Led: Lexicon-enlightened dense retriever for large-scale retrieval. In Proceedings of the ACM Web Conference 2023, pages 3203-3213.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends Information Re-
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinherger, and Yoav Artzi. 2019. Bertscore: Evalu- ating text generation with bert. ICLR.
trieval.
Jinming Zhao, Ming Liu, Longxiang Gao, Yuan Jin, Lan Du, He Zhao, He Zhang, and Gholamreza Haffari. 2020a. Summpip: Unsupervised multi-document summarization with sentence graph compression. SIGIR.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2023. Col- bertv2: Effective and efficient retrieval via lightweight late interaction. In NAACL.

---

Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao. Dongyan Zhao, and Rui Yan. 2020b. Knowledge- grounded dialogue generation with pre-trained lan- guage models. In EMNLP.
Yufan Zhao. Wei Wu, and Can Xu. 2020c. Are pre- trained language models knowledgeable to ground open domain dialogues? arXiv.
Hao Zhou, Weidong Ren, Gongshen Liu, Bo Su, and Wei Lu. 2021. Entity-aware abstractive multi- document summarization. In ACL.

---

A Appendix
A.2 Qualitative analysis about the approximation
A.1 Retrieving on large-scale corpus
To support the retrieval the large-scale corpus, approximate the interaction to adapt the vector retrieval method in Faiss (in Eq A.1), which un- avoidably introduces the loss of information. We explore the impact by comparing the performance before and after approximation. and the results are shown in Table 3 as LER (w. approx.). Compar- ing with the vanilla LER, we can observe that the performance of the algorithm has decreased, with average decrease of about 3.13% and 3.68% in 'AS2 and TAD datasets, respectively. This result illustrates that the approximation can improve re- trieval efficiency at a slight performance cost. With this approximation, we can use Faiss to perform semantic matching on the large-scale corpus. Oth- erwise, it will be difficult for us to retrieve on a scrge-scale corpus with acceptable latency.
Since the scientific paper corpus is usually large- scale in the real-world scenario, our UR$WG should support a billion-scalc similarity search more efficiently. Following the Lewis al. (2020), we first encode all candidate documents into vector representations using the encoder Encoder and utilize Faiss to store and conduct the real-time vec- tor search. Faiss a library for efficient similarity search and clustering of dense vectors. Faiss em ploys the MIPS (Maximum Inner Product Search) to find the document vectors with the highest in- ner product with the query vector (Mussmann and Ermon, 2016).
Since the online interaction mentioned in Equa- tion 11 is time-intensive, we can approximate by creating an index of group-wise maximum and minimum values of 0:
A.3 Details for baselines
= [arg max
max (iiith) nk. < }}
k
We mainly compare our UR$WG with three types of baselines: multi multi-document summarization meth- ods, related work generation methods, and large language models.
|
0z.max
nk, S Odmin (min(0) nk. n S }},
}}.
k
[V]
The multi- document summarization baselines in- clude: LEAD, which concatenates the first sentence of each reference; LexRank (Erkan and Radev, 2004), which extract summarization based on the graph representation of sentences; NES (Wang et al., 2019), a extractive method which measures the relevance via bibliography graph; MGSum (Jin et al., 2020), an abstractive method based on multi- granularity interaction network; BertSum (Liu and Lapata, 2019). a summarization system built on a pre- pre-trained BERT (Devlin et al., 2019); EMS (Zhou et al., 2021), which augments the encoder-decoded framework with heterogencous graph.
where the I.] indicates the lower rounding.
The essence of our approximation method is that the bag-of-words representation is highly sparse, so we can assume that only one individual element in each group is significantly larger than the other elements, while the other elements are only slightly different. Therefore, the Equation 11 can be further simplified as:
S =
Σ oi Bg 1=
ommaa
(1 oz) Br Jamin
where og is label defined as:
The related work generation methods include: RRG (Chen et al., 2021) which enhanced by a iteratively refined relation aware graph between references; TAG (Chen et al., 2022a), a transformer- based model with target-aware graph encoder.
0,
T x T
(18)
1,
For i-th group of Oa during the matching, if the Tå equals to ryj there no bias. If the rá not equals to Th. the value ri-th element in Og can be alternate by the minimum value of this group based on our assumption. With this approximation, we can use Faiss to perform semantic matching on the large- scale corpus.
The large language models (LLMs) include flan-t5 (Chung al., 2022), Llama-7B (Touvron et al., 2023), ChatGLM (Du et al., 2022). Vicuna $ claude 6, Davinci-text-003, and ChatGPT 7, where the instruction IGEN is taken as input.
htttps://githhb. com/ wttp///nnss////attthaaaaae com/ http://ppaai..comm//o///chhtttt

---

Table Results on the TAS2 and TAD dataset. We abbreviate ROUGE as R We underline the best results in each category of baseline methods. w/ denotes the method which uses both references and abstract as input. All ROUGE scores have 95% confidence interval as reported by the official R script.
TAS2 Dataset
TAD Dataset
Methods
R-1
R-2
R-L
R-SU
R-1
R-2
R-L
R-SU
Summarization methods
LEAD
19.63
1.75
16.88
3.73
22.74
2.32
20.15
4.87
NES w/ T (Wang al., 2019)
26.04
3.39
22.46
6.14
26.13
3.24
23.20
6.18
MGSum (Jin al., 2020)
25.54
3.75
23.16
6.49
27.49
4.79
25.21
7.29
LexRank (Erkan and Radev, 2004)
25.74
2.81
22.43
6.03
25.70
2.86
22.68
5.90
LexRank w/ T (Erkan and Radev, 2004)
27.04
3.18
23.48
6.55
27.29
3.50
24.06
6.61
BertSumEXT (Liu and Lapata, 2019)
25.85
2,90
22.66
6.21
25.95
2.91
23.05
6.25
BertSumEXT w/ T (Liu and Lapata, 2019)
27.43
3.56
24.01
6.97
27.60
24.51
6.98
BertSumABS (Liu and Lapata, 2019)
25.45
3.82
23.04
6.39
27.42
25.15
7.22
BertSumABS w/ T (Liu and Lapata, 2019)
26.35
3.23
23.73
6.65
28.48
25,91
7.63
EMS (Zhou al., 2021)
26.17
4.16
23.63
6.67
28.21
25.74
7.56
EMS w/ T (Zhou al., 2021)
26.50
4.22
23.90
6.84
28.74
5.36
26.37
7.89
Large language models
Llama-7B (Touvron al., 2023)
18.28
2.05
15.29
4.01
15.45
1.62
13.48
2.91
Claude
27.59
3.88
24.99
7.28
30.69
5.11
28.00
9.04
Vicuna-7B
28,
4.01
25.39
7.01
31.09
4.91
27.62
8.9
Davinci-text-003
22.80
2.49
20.26
5.08
24.59
3.15
21.97
5.89
ChatGPT
29.13
4.08
25.59
7.89
31.41
5.34
28.
9.33
Related work generation methods
RRG (Chen al., 2021)
26.79
4.43
24,46
6.85
28.94
5.59
26,46
7.92
TAG (Chen al., 2022a)
28.04
4.75
25.33
7.69
30.48
6.16
27.79
8.89
UR'WG
31.59
5.86
26.13
9.62.
32.68
7.74
28.87
9.54
We compare our lexicon-enhance re-
erence. As shown Table 5, the ROUGE metrics increase when the abstract concatenated to the reference as input, e.g., an average 4% relative im- provement in ROUGE-1 metrics. However, it still lags behind the proposed UR3WC.
trieval method with the following baselines: BM25 (Robertson et al., 2009), a classical sparse retrieval; DPR (Karpukhin et al., 2020), a dense retrieval method with dual-encoder framework; SPLADE (Formal et al., 2021), passage ranker based sparsity regularization and log-saturation effect; ColBERT (Khattab and Zaharia, 2020), which searches relevant passages via contextualized late interaction. UniCOIL (Lin and Ma, 2021b), an extension of classical sparse retrieval COIL (Gao et al., 2021). Since the abstract is too long with many irrelevant words, we take concatenation of the keywords extracted from the abstract the query.
A.5
Human evaluation
A.5.1 Details for human evaluation
We conduct human evaluation, where three well- educate Master students are invited to judge 40 ran- domly sampled examples. Specifically, we show each annotator the abstract, related work, and cor- responding references. Each annotator is asked to rate the related work with three-scale in the following metrics: (1) Relevance (Rel.): the rel- evance between the raw abstract and the related work. (2) Fluency (Flu.): whether the related work is fluent with no grammatical errors. (3) Coherence (Cohe.): coherence refers to how well the sentences connected and how they flow together to form coherent and understandable text. (4) Informative- ness (Info.): Whether related work covers the key information mentioned the article can provide more extensions. Table shows the details the human evaluation results.
A.4 Variant model
Since the existing summarization baselines only use the references as input, the abstract the target paper also contains much useful informa- tion. To make fair comparison, we develop a variant model for each multi-document summariza- tion method which uses the concatenation of the abstract of the target paper and references in- put. We add the notion (w/ T) to each original summarization method to denote the variant model. We also implement a baseline, denoted as LEAD, which concatenates the first sentence of each ref-

---

Table Human evaluation results on two datasets.
TAD dataset
TAS2 dataset
Methods
Rel.
Flu.
Cohe.
Info.
Rel.
Flu.
Cohc.
Info.
TAG (Chen et al., 2022a)
2.30
2.50
2.54
2.40
2.55
2.35
Claude
2.15
2.70
2.45
2.33
2.35
2.77
2.35
2.39
Vicuna
2.25
2.75
2.32
2.35
2.34
2.77
2.44
2.47
Davinci-text-003
1.90
2.51
1.97
1.97
2.05
2.60
2.01
2.05
ChatGPT
2.65
2.86
2.40
2.65
2.54
2.83
2.44
2.65
UR'WG
2.77
2.79
2.45
2.60
2.57
2.70
2.54
2.70
Human
2.84
2.85
2.78
2.79
2.74
2.95
2.75
Table An example of generated related work by our model and baselines. The same color denotes the same key information.
Abstract target paper (truncated):
we explore simple method for predicting genre of artists based on co-occurrence analysis, i.e. analyzing
co-occurrences of artist genre names on music related pages. We investigate four different query schemes
two probabilistic approaches for predicting genre of artists.
Ground Truth Reference Papers (the same as our retrieved reference):
[1] Detecting artists that can considered as prototypes particular genres or styles music an interesting task. [2] We use relation hetween the prototypicaiiyy ranking and ranking obtained by page counts of search google contain artist and genre.
Ground truth related work Co-occurrence analysis has also been applied to thedetecting prototypical artists task for a given genre[1]. I.. In [2]. they used a technique based on a page rank citation ranking on page count estimates to derive the prototypicality of each of artist for given genre.
Baseline TAG output: Co-occurrences of artist names on web pages have been used for artist co occurrencce analysis in [1]. In [2]. the authors use the co-occurrence analysis of artist and genre names to determine artist similarity. They use cosine similarity to find co-occurrences.
Baseline ChatGPT output: This survey focuses web-based Co-nccurrences technique for genre prediction of music artists. evaluating the effectiveness of this approach and considering the integration of objective rating methods like Pagerank, aims contribute the development of accurate and reliable genre prediction models.
UR-WG output: [1] use co-occurrences artist and genre names on music related web pages to predict the genre a given artist. [2) propose a method for predicting the genres of an artist which relies on the page counts provided by google to estimate the relatedness of an arbitrary artist to each of set of genres.
A.6 Case study
Table 7 presents an example of the generated re- lated work by UR and the start-of-the-art base- line TAG. And we highlight the same key informa- tion with the same color. We first observe that UR$WG retrieves the ground-truth references cor- rectly, even though the keyword "page rank" is not mentioned in the abstract, which verifies the effi- ciency of our retriever. Moreover, the related work generated by UR3 captures more highly related key information than the TAG baseline.


Reviews:
[{"review_id": "4UvnjPUbkq", "reviewer": "Reviewer_Vyrb", "summary": "", "questions": "See Reasons to reject above -- Happy to hear what the authors think, and will adjust my ratings accordingly.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments that help us improve the work. Below we address the concerns mentioned in the review.\n\n**Question 1: Fairness of comparison to baselines.**\n\nTo ensure the fairness of comparison, all the retrieval baselines in Table 3 are finetuned with the same datasets as our proposed LER.\nWe will add these details in the camera-ready version.\n\n**Question 2.1: More analyses and justify why we should use your system instead of ChatGPT for related work generation in general.**\n\nDespite the ChatGPT achieving remarkable performance in the related work generation task, it also encounters the following limitations:\n1. For data security reasons[1,2], not all applications can transmit user data, such as reference papers and unpublished content, to LLM service vendors. It restricts the use of proprietary LLMs in such applications.\n2. The proprietary large language model, e.g., ChatGPT, is cost-intensive during inference for its huge parameters, leading to a tremendous carbon footprint. According to the [3] and [4], the ChatGPT exhibits a carbon footprint that is ten times greater than Llama-7B. In contrast, our system achieves higher performance while utilizing only 4% parameters of ChatGPT. Given that our method is model-agnostic, it is intuitive that the performance of our system will improve along with the increase of the parameter scale. Considering the environmental concern posed by carbon emissions[3-6], we only use Llama-7B for validation within our research.\n\n**Question 2.2: The cross-domain evaluation.**\n\nWe further conduct the cross-domain evaluation, where we train our model with one dataset and evaluate it in another. As shown in Table 8, our proposed method exhibits potential generalizability. For example, the *UR3WG w/cross* outperforms Claude with a 9.2%~11.5% average relative improvement on the two datasets. Moreover, our method achieves comparable and even better performance compared with ChatGPT. These results illustrate the capacity of our system to generalize across different related work generation datasets effectively.\n\nTable 8: Results on two datasets, where the R is the  abbreviation of ROUGE. We conduct the cross evaluation, where we train the model on one dataset and evaluate the other dataset (UR3WG w/cross).\n|                  |        | TAS2 | Dataset |      |       |  TAD | Dataset |      |\n|----------------|:------:|:----:|:-------:|:----:|:-----:|:----:|:-------:|:----:|\n|                  |   R-1  |  R-2 |   R-L   | R-SU |  R-1  |  R-2 |   R-L   | R-SU |\n|      Claude      |  27.59 | 3.88 |  24.99  | 7.28 | 30.69 | 5.11 |  28.00  | 9.04 |\n|     Vicuna-7B    | 28.40  | 4.01 | 25.39   | 7.01 | 31.09 | 4.91 | 27.62   | 8.9  |\n| Davinci-text-003 | 22.80  | 2.49 | 20.26   | 5.08 | 24.59 | 3.15 | 21.97   | 5.89 |\n| ChatGPT          | 29.13  | 4.08 | 25.59   | 7.89 | 31.41 | 5.34 | 28.15   | 9.33 |\n| UR3WG            | 31.59  | 5.86 | 26.13   | 9.62 | 32.68 | 7.74 | 28.87   | 9.54 |\n| UR3WG w/cross    | 30.01  | 4.77 | 25.50   | 8.34 | 31.75 | 6.74 | 27.79   | 9.27 |\n\n\n**Question 3: More discussion for the human evaluation.**\n\nTo further examine the effectiveness of our method, we employ a harder baseline which feeds the ground truth references into ChatGPT to get a more competitive baseline, while our method only takes the model retrieved references as input. Notably, employing ground truth references yields better performance than utilizing retrieval references. Due to space constraints, we only present the superior result (where ground truth references are used as input) in the paper. For a more comprehensive comparison, the results with model retrieval reference as input are shown in Table 9 and Table 10. We can observe that our method outperforms baselines with the same setting in terms of automatic and human metrics.\nWe will add these experimental details in the final version in the main body of the paper. \n\n&nbsp;\n\nTable 9: We feed ChatGPT with two types of reference, i.e., retrieval references (denoted as ChatGPT w/retrieval) and ground truth references (denoted as ChatGPT w/oracle).\n|                     |       | TAS2 | Dataset |      |       | TAD  | Dataset |      |\n|----------------|:------:|:----:|:-------:|:----:|:-----:|:----:|:-------:|:----:|\n|                     | R-1   | R-2  | R-L     | R-SU | R-1   | R-2  | R-L     | R-SU |\n| ChatGPT w/retrieval | 26.48 | 3.42 | 23.90   | 6.92 | 28.45 | 4.51 | 25.60   | 7.99 |\n| ChatGPT w/oracle    | 29.13 | 4.08 | 25.59   | 7.89 | 31.41 | 5.34 | 28.15   | 9.33 |\n| UR3WG               | 31.59 | 5.86 | 26.13   | 9.62 | 32.65 | 7.74 | 28.87   | 9.54 |\n\n&nbsp;\n\nTable 10: Human evaluation results on two datasets.\n|                     |      | TAS2 | Dataset |       |      | TAD  | Dataset |       |\n|----------------|:------:|:----:|:-------:|:----:|:-----:|:----:|:-------:|:----:|\n|                     | Rel. | Flu. | Cohe.   | Info. | Rel. | Flu. | Cohe.   | Info. |\n| ChatGPT w/retrieval | 2.59 | 2.70 | 2.24    | 2.50  | 2.47 | 2.55 | 2.14    | 2.52  |\n| ChatGPT w/oracle    | 2.65 | 2.86 | 2.47    | 2.65  | 2.54 | 2.83 | 2.44    | 2.70  |\n| UR3WG               | 2.64 | 2.75 | 2.35    | 2.60  | 2.57 | 2.70 | 2.30    | 2.60  |\n| Human               | 2.84 | 2.85 | 2.78    | 2.89  | 2.79 | 2.74 | 2.95    | 2.75  |\n\n&nbsp;\n\nWe will also add these experimental results and discussion in the final version of the paper. It would be very kind of you to raise the current score if you find our responses useful and helpful.\n\n&nbsp;\n\n[1] The False Promise of Imitating Proprietary LLMs\n\n[2] WebCPM: Interactive Web Search for Chinese Long-form Question Answering\n\n[3] Reducing the Carbon Impact of Generative AI Inference (today and in 2035)\n\n[4] ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review\n\n[5] Measuring the environmental impacts of artificial intelligence compute and applications\n\n[6] Artificial intelligence and the climate emergency: Opportunities, challenges, and recommendations", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed framework presents state of the art results on two of the related work generation datasets. \nUsing the same backbone model for query extension and related work generation in a joint learning setting is an interesting idea, and has some intellectual merit. \nThe paper presents solid evaluation results and ablation studies for us to understand the performance of the model.", "reasons_to_reject": "Though the proposed framework presents state-of-the-art results, it's a bit hard to tell from the evaluation in its current form if the improvements can be attributed to the conceptual idea, or LLM itself. Specifically, IMHO here are some comments and suggestions on where the authors can potentially improve their evaluations. \n\n- For reference retrieval, the LER system seems to be the only one that's finetuned on scientific papers specifically, all the other retrieval baselines are evaluated in zero-shot settings. So it's hard to tell if the LER design specifically is contributing to the improvements or not.\n- Given the relatively small improvements in terms of ROUGE over ChatGPT (especially considering the ++ amount of supervision that the proposed system used compared to LLM baselines), it might be worth doing more analyses and justify why we should use your system instead of ChatGPT for related work generation in general. One way to strengthen the evaluation could be doing a cross-domain generalization study on your system, e.g. train on one dataset and evaluate on the other, so that we can see if your system is able generalize or not. \n- Along the previous point, the human evaluation presented in Table 6  in Appendix seems to favor vanilla ChatGPT over your proposed system even for in-domain train+eval setting.  This is fine, but it's worth dedicate more discussion + analyses on why it could be in the main body of the paper?", "questions_to_authors": "See Reasons to reject above -- Happy to hear what the authors think, and will adjust my ratings accordingly.", "paper_topic_and_main_contributions": "The paper proposes a unified, joint learning framework UR3WG for Reference Retrieval and Related Work Generation.  The framework takes three steps. Given a user query (e.g. abstract of a paper), UR3WG first learns to generate an extended query, retrieve related work, and generate a summary for all the related work.  \n\nThe paper presents SOTA results on TAS2 and TAD datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall I feel like the paper could use more space for results and error case analysis in the main body of the paper, and truncate/trim the description on the methods."}, "extra_text": {}}, {"review_id": "oerXVj8hKJ", "reviewer": "Reviewer_5RZj", "summary": "", "questions": "See Reasons To Reject", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments that help us improve the work. Below we address the concerns mentioned in the review:\n\n\n**Question 1: Comparison with previous retrieval method.**\n\nThe DLRs[1] is a classical lexicon-enhance retrieval method that shares a similar motivation with approaches such as LexMAE[2], SPLADE[3], and ColBERT[4]. Inspired by these methods, we propose a novel method LER to measure the relevance between query and document on both semantic and lexical levels.\n\nTo tackle the specific challenges in the RWG task, there are three main contributions in our proposed retrieval method:\n1. Considering the user-provided abstract is vague without a clear intention for reference, we cannot directly take the abstract as the retrieval query. We leverage the knowledge of the LLM to re-generate an extended abstract that contains more related topics inherently correlated with the target paper.\n2. Scientific papers entail multiple diverse references, in contrast to the existing passage retrieval task that assumes a single relevant document per query. Therefore, we propose a multi-granularity contrastive learning method to train our retriever to tackle this problem.\n3. Due to the size of the scientific paper corpus is usually large and it is time-intensive to conduct the interaction (Equation 12) in real-world applications. We design an approximate algorithm for semantic matching.\n\nAs you suggested, we further compare with the DLRs in Table 8 for a more comprehensive analysis. We also include two more recent retrieval methods: SPLADE v2 and SPLADE++. From Table 8, we can find that our proposed LER outperforms these baselines, such as achieving 13.81 absolute improvement  compared with SPLADE v2 on average. Since the DLRs is also a typical lexicon-enhance retrieval method, we will definitely add this citation in our final version.\n\n&nbsp;\n\nTable 8: The recall score for retrieval on two datasets. We compare with more recent baselines to further examine the effectiveness of our method.\n|           |          | TAS2     | Dataset   |          | TAD | Dataset   |\n|------|:---------:|:--------:|:---------:|:---------:|:--------:|:---------:|\n|           | Recall@5 | Recall@10 | Recall@20 | Recall@5 | Recall@10 | Recall@20 |\n| DLR[1]      |   23.68 |   25.77     |     30.57     |  15.61  |  18.87  |      27.95   |\n| SPLADE v2[5]|    27.67 |  33.95     |     40.68     |  14.88   |   19.41   |     24.50    |\n| SPLADE ++[6]|     24.48  |   29.64  |    34.87   |  16.10  |   20.77    |    41.10  |   \n| LER         |    37.35  |   41.56     |     44.17     |  31.20   |   40.20    |      49.47     |\n\n&nbsp;\n\n**Question 2: Limited technical contribution.**\n\nOur work aims to build a unified framework for relation work generation task, which consists of two main parts: reference retrieval and related work generation.\n\nSince the reference retrieval in RWG task has some specific challenges, the existing dense retrieval methods cannot suit the RWG task well. As introduced in the response to Question #1, we propose the LER model to retrieve more relevant references in this scenario.\n\nIn the related work generation module, according to the definition of related work generation task[7,8], the related work section should comprehensively introduce the correlation between the references, instead of simply summarizing the primary points for multiple documents. We propose an instruction-tuning based generation method to generate a high-quality related work section, which leverages the instruction-following capability of LLM.\n\nWe will add these details for a clearer explanation and highlight the contribution of our work in the camera-ready version. It would be very kind of you to raise the current score if you find our responses useful and helpful.\n\n&nbsp;\n\n[1] A Dense Representation Framework for Lexical and Semantic Matching\n\n[2] LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval\n\n[3] SPLADE: Sparse lexical and expansion model for first stage ranking\n\n[4] ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT\n\n[5] SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval\n\n[6] From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective\n\n[7] Towards Automated Related Work Summarization\n\n[8] Target-aware Abstractive Related Work Generation with Contrastive Learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Propose the first unified reference retrieval and related work generation model.\n(2) Experiments demonstrate URWG outperforms the baselines in both generation and retrieval metrics.", "reasons_to_reject": "(1) Similar technologies without proper comparison. The authors claim that they propose Lexicon-Enhance Dense Retrieval (LER), however, the technologies in LER is very similar to DLRs [1]. I don't think these two can be considered as contemporaneous work, and hope that the authors make proper clarifications and citations of the duplicated technical points, instead of declaring these as their contributions as well.\n\n(2) Limited technical contribution.\n\n\n\n[1] A Dense Representation Framework for Lexical and Semantic Matching", "questions_to_authors": "See Reasons To Reject", "paper_topic_and_main_contributions": "This propose a Unified Reference Retrieval and Related Work Generation Model (URWG), which combines reference retrieval and related work generation processes in a unified framework based on the large language model for the related work generation task. Experiments on two wide-applied datasets demonstrate that URWG outperforms the baselines in both generation and retrieval metrics.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "A Dense Representation Framework for Lexical and Semantic matching", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "azoUPMlSXr", "reviewer": "Reviewer_FLmC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments that help us improve the work. Below we address the concerns mentioned in the review:\n\nWe provide the comparison with three recent baselines, such as SPLADE v2[1] and SPLADE ++[2]. And the Table 8 summarizes the results. We find that our method outperforms the baslines in two datasets. For example, the LER achieves Recall@5=37.35 , with 12.87 absolute improvement compared to SPLADE++ in the TAD dataset. These results further illustrate the effectiveness of our retrieval method.\n\nWe will add these baselines to our main experiment section in the final version. And it would be very kind of you to raise the current score if you find our responses useful and helpful.\n\n&nbsp;\n\nTable 8: The recall score for retrieval on two datasets. We compare with more recent baselines to further examine the effectiveness of our method.\n|           |          | TAS2     | Dataset   |          | TAD | Dataset   |\n|------|:---------:|:--------:|:---------:|:---------:|:--------:|:---------:|\n|           | Recall@5 | Recall@10 | Recall@20 | Recall@5 | Recall@10 | Recall@20 |\n| DLR[3]      |   23.68 |   25.77     |     30.57     |  15.61  |  18.87  |      27.95   |\n| SPLADE v2[1]|    27.67 |  33.95     |     40.68     |  14.88   |   19.41   |     24.50    |\n| SPLADE ++[2]|     24.48  |   29.64  |    34.87   |  16.10  |   20.77    |    41.10  |   \n| LER         |    37.35  |   41.56     |     44.17     |  31.20   |   40.20    |      49.47     |\n\n[1] SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval\n\n[2] From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective\n\n[3] A Dense Representation Framework for Lexical and Semantic Matching", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Smart unified framework to combine related work generation and retrieval.\n2. This paper is well-written and easy to follow.\n3. The experimental results are strong.", "reasons_to_reject": "1. Missing several important lexicon-aware baselines in Table 3 like SPLADE v2, SPLADE ++, and LED. It's a bit weird to me that there’s no methods after 2021 in Table 3.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework unifying reference retrieval and related work generation based on LLMs. The proposed method reformulate the query and retrieval the documents as references for better related work generation. Specifically, the lexicon-enhanced retriever significantly helps the retrieved documents, thereby delivering better summarization.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed method reformulate the query and retrieval the documents as references for better related work generation. The method is smart and useful. I personally respect the three reviewers' opinion, and suggest the paper accepted as Findings in the conference.
Decision: Accept-Findings