HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction
Authors: Qianyue Hao
Jingyang Fan
Fengli Xu
Jian Yuan
Yong Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Citation prediction
large language model
text embedding model
Primary area: natural_language_processing
TL;DR: This paper designs a hybrid language model workflow integrating representative and generative LMs to precisely predict which previous papers (candidates) will an ongoing new paper (query) cite from vast candidate set.

Abstract: Citation networks are critical infrastructures of modern science, serving as intricate webs of past literature and enabling researchers to navigate the knowledge production system. To mine information hiding in the link space of such networks, predicting which previous papers (candidates) will a new paper (query) cite is a critical problem that has long been studied. However, an important gap remains unaddressed: the roles of a paper's citations vary significantly, ranging from foundational knowledge basis to superficial contexts. Distinguishing these roles requires a deeper understanding of the logical relationships among papers, beyond simple edges in citation networks. The emergence of large language models (LLMs) with textual reasoning capabilities offers new possibilities for discerning these relationships, but there are two major challenges. First, in practice, a new paper may select its citations from gigantic existing papers, where the combined texts far exceed the context length of LLMs. Second, logical relationships between papers are often implicit, and directly prompting an LLM to predict citations may lead to results based primarily on surface-level textual similarities, rather than the deeper logical reasoning required. In this paper, we introduce the novel concept of core citation, which identifies the critical references that go beyond superficial mentions. Thereby, we elevate the citation prediction task from a simple binary classification to a more nuanced problem: distinguishing core citations from both superficial citations and non-citations. To address this, we propose $\textbf{HLM-Cite}$, a $\textbf{H}$ybrid $\textbf{L}$anguage $\textbf{M}$odel workflow for citation prediction, which combines embedding and generative LMs. We design a curriculum finetune procedure to adapt a pretrained text embedding model to coarsely retrieve high-likelihood core citations from vast candidate sets and then design an LLM agentic workflow to rank the retrieved papers through one-shot reasoning, revealing the implicit relationships among papers. With the two-stage pipeline, we can scale the candidate sets to 100K papers, vastly exceeding the size handled by existing methods. We evaluate HLM-Cite on a dataset across 19 scientific fields, demonstrating a 17.6\% performance improvement comparing SOTA methods. Our code is open-source at https://github.com/tsinghua-fib-lab/H-LM for reproducibility.

HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction
Qianyue Hao, Jingyang Fan, Fengli Xu, Jian Yuan, Yong Li" Department of Electronic Engineering, BNRist, Tsinghua University Beijing, China
Abstract
Citation networks are critical infrastructures modern science, serving as intri- cate webs of past literature and enabling researchers to navigate the knowledge production system. To mine information hiding in the link space of such net- works, predicting which previous papers (candidates) will cite critical problem that has long been studied. However, an important gap remains unaddressed: the roles of paper's citations vary significantly, ranging from foundational knowledge basis to superficial contexts. Distinguishing these roles requires deeper understanding of the logical relationships among papers, beyond simple edges in citation networks. The emergence of large language models (LLMs) with textual reasoning capabilities offers new possibilities for discerning these relationships, but there are two major challenges. First, in practice, a new paper may select its citations from gigantic existing papers, where the combined texts far exceed the context length of LLMs. Second, logical relationships between papers are often implicit, and directly prompting an LLM to predict citations may lead to results based primarily on surface-level textual similarities, rather than the deeper logical reasoning required. In this paper, we introduce the novel concept of core citation, which identifies the critical references that go beyond superficial mentions. Thereby, we elevate the citation prediction task from simple binary classification to more nuanced problem: distinguishing core citations from both superficial citations and non-citations. To address this, we propose HLM-Cite, a Hybrid Language Model workflow for citation prediction, which combines embed- ding and generative LMs. We design a curriculum finetune procedure to adapt a pretrained text embedding model to coarsely retrieve high-likelihood core citations from vast candidate sets and then design an LLM agentic workflow to rank the retrieved papers through one-shot reasoning, revealing the implicit relationships among papers. With the two-stage pipeline, we can scale the candidate sets to 100K papers. vastly exceeding the size handled by existing methods. We eval- uate HL.M-Cite on dataset across 19 scientific fields, demonstrating a 17.6% performance improvement comparing SOTA methods, Our code is open-source www.m/siingha--i--laabb. fib-lab/H LM for reproducibility.
paper (query)
cww/tsinghua
1 Introduction
With the rapid development of modern science, the volume of research papers is increasing an- nually (1) As links between papers, citations network connects vast literature and bridge newly emerging knowledge with existing ones. Due to the critical role of citations, citation prediction is an important problem that has long been studied 2 6 71, where the goal predict which
Fengli Xu and Yong Li are corresponding authors.
Email:
fenglix denglixu@ts inghua, edu, en.
1iyong07etsinghus. edu,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

papers from a set of previous papers (candidate set) will an emerging new paper (query) cite. Accurate citation prediction can help reveal information hiding in link space of citation networks 2 owning value in aiding citation- based computational social science studies regarding the patterns of paper publication and scientific innovation TO
13 14, On the other hand, citation prediction of practical significance for assisting researchers in writing manuscripts, providing high-likelihood citation suggestions, and thereby saving massive literature searching time.
Despite abundant studies on citation prediction, there is critical problem that remains unconsidered. While one paper typically cites multiple previous papers, the roles of citations vary significantly. The most important citations serve as research foundations of the query paper, assisting researchers in tracing the lineage of knowledge production. In contrast, some less relevant citations are only mentioned superficially in context. Existing works treat citation prediction as a simple binary classification problem and neglect such varying roles 22 distract attention from the important ones. However, such nuanced roles cannot be adequately reflected by simple edges in citation networks, but require understandings on the logical relationships among papers. In this paper, we aim to predict citations with various roles based on in-depth content understanding, where the emerging textual reasoning ability of LLMs provides a possible approach.
7, letting superficial citations
Predicting citations with LLMs faces two major challenges. (1) Vast candidate sets. The real-world scientific database consists of gigantic papers, and researchers need retrieve possible citations from millions of previous papers. With limited context length, it is impractical to feed the vast candidates' contents into LLMs and expect reasoning on logical relationships among them. (2) Implicit logical relationships. The logical relationships among papers lie implicitly within the content of papers. Directly prompting an LLM to predict key-role citations for query papers is likely to get sunk into simple content similarity rather than reasoning actual logical relationships among papers.
In this work, we define the novel concept of core citation with inspiration from rich science of science research [9] 10 [2), depicting the varying roles of citations. We analyze 12M papers across 19 scientific fields and illustrate core citations' significantly closer relationships with the query papers. Based on this definition, we develop the task of citation prediction from simple binary classification between citations and non- citatiitioons into more challenging but meaningful version. i... distinguishing core citations from superficial citations and non-citation cinnciiaaioos. To solve this task on vast candidate sets, we propose integrating embedding and generative LMS as HLM-Cite, a two-stage hybrid language model workflow. We design curriculum fine-tuning procedure to adapt a pretrained text embedding model to analyzing research papers, initially retrieving high- sebb-iikelihood core citations from vast candidate sets in the first stage. Subsequently, we design LLM agentic workflow, consisting of a Guider, an Analyzer, and a Decider, for the second stage. Guided by a one-shot example, the LLM agents analyze the papers' implicit logical relationships through textual reasoning and rank the retrieved papers by citation likelihood. In HLM-Cite, we incorporate the capability of both embedding and generative LMs, enabling precise extraction of core citations from tremendous candidate sets. We conduct extensive experiments on cross-field papers, and the results show a 17.6% performance improvement of our method compared to SOTA baselines. Also, experimental results prove that our workflow can scale up to 100K candidates, thousands of times more than existing works, owning the potential to cover entire research domain for practical implementation. In summary, the main contributions of this work include:
We define the novel concept of core citation to depict the varying roles of citations. Thereby, we develop the citation prediction task from simple binary classification into distinguishing core citations, superficial ones, and non-citations, giving it more practical significance.
We design hybrid language models workflow to integrate the capabilities of embedding and generative LMs, where two categories of models form a two-stage pipeline that cascades retrieval and ranking to predict core citations. This design enables our method to handle very large candidate sets with high precision.
We conduct extensive experiments on a cross-field dataset with up to 100K paper candidate sets. The results prove the scalability of our design and illustrate 17.6% performance improvement comparing SOTA methods.
2

---

Superficial citations
Future Now Previous
citations
35-
Overlap of beywordd
12-
1.0
Non-citations
0.8
Query
-0-
Nomber
Superficia ciaatons
Superficial chtaions
O
A cites B
Natural science Social science
Overal
0.0
Natural science Socia science
Overal
(a) Definition of core citation
(b) Overlap keywords
(c) Number of mentions the main texl
Figure 1: (a) Definition of core citation. (b) (c) Statistical difference between core citations and superficial citations. In all panels, 95% CI are shown as error bars.
2 Problem Formulation
2.1 Definition Core Citation
We first provided some notations about paper citation relationships. Considering set of papers G, query paper g E G cites small subset of G' including My previous papers, denoted as (s] ...8 (8.99) a
S c CGY, {q], while the rest papers are not cited by 9. which we denote them as {p).... $ Pa
Co Sq. Also, my subsequent papers cite denoted as {"""....""" & Fg G (S₃ (g))-
As mentioned above, the roles of each element in S, may vary significantly, where there exist ky ,
elements in Sy have major importance. We name them as core citations, denoted as "..""..."" 2
Sg C. Sg. Naturally, we name the rest of the citations, i.c., S, S.y. as superficial citations. Enlighten by previous computational social science studies regarding citation networks [IO] [2]. following-up papers of q, i.e., gg. are likely also cite the critical foundations of q, namely q ss core citations. On the other hand, less relevant citations of such as some background knowledge, are typically not followed by Fg. Therefore, we mathematically identify the core citations according to such local citation relationships (Figure Th):
3, - (sg E Sy 3p E Fg, let q E Sps Sq E Sp}.
(1)
To verify the rationality this definition, we draw statistics on 12M papers across 19 scientific fields in the Microsoft Academic Graph (MAG) [15] (See dataset details in Section From the results in Figure Dp and c, we find that, with statistical significance, in both natural and social science domains, the query paper has more overlapped keywords with its core citations than its superficial citations, and the core citations are also more frequently mentioned in the main texts of query papers. This illustrates that the core citations identified from citation networks, are consistent with the important citations in the papers' content, proving feasibility of predicting core citations purely from the texts.
2.2 Core Citation Prediction Task
Considering the difference between citations and superficial citations, focus predicting the core citations, which are most meaningful links among literature for scientific research. We formally define the task of core citation prediction follows.
Definition (Core Citation Prediction) Given a query paper q, and a candidate set C.p where IC, tgp Cg includes tl core citations and f superficial citations of 9, ensuing tl < k < n4 - kg and fg + fg < ty. and its rest elements, if any, are non-citations. The goal of core citation prediction is to pick out 12 elements from Cg. maximizing the number of picked core citations.
In such setting, superficial citations actually become hard negative samples against the core citations, adding to the challenges of the task.
In this paper, we focus on text-based citation prediction, where we only use citation networks to obtain the ground truth of core citations and do not include any network features other than the papers' textual content in the prediction. In this way, our model learns to extract the logical relationships
3

---

purely from the texts, predicting which citations of q are likely to be valued by future papers citing q without requiring any information about the exact future citations, which have not happened yet. Therefore, although we construct the ground truth of core citations in training and testing sets with previously published papers where we already know the subsequent papers that cite them, i.e., Fg. our models is feasible for ongoing manuscripts without Fg-
Query V Superficial citations
Title + Abstract
Stage 1: Classification task
Stage 2: Ranking task
Core citations Non-citations
Encoder
Pretrained arraained model
1- c
G
Frooder
Candidate
(uVGoenoven)
Encoder 8
Cp
Retrieval module
Encoder 32
5101
Finetune
768. dimensional ambedding
Embeddlig similarity [Representative]
(b) Curriculum finetuning of the retrieval module
Guider
supervision
Why q cibes these?
What ranking?
Retrieval Rg
LVV [
1. heuraa Frobabiicss Language Modal
2 Attonion You Need o: Transformer Example
To adnnswiddge Ehe fistonccl contest di.. jalnost (ralove))
Founcist ona model which Transformer XL.. Ivery relovand
2- x1 (corecti
B. BERT
... within recont advancements in pransformer... Oess relesont
Ranking module LLMs
VZ Guider
Retrieval PREC85-0.6
Textual reasoning (Generative)
Analyser (LLM)
One-shot learning
Decider (LLM)
Decider
Retrieved condidane
3 Foundarions the query paper... Every relevand)
Retrleved candidane
provide interciscipien liarr insggts.. (almost imelovan)) admowledge the rulated contont Qis. relesant) paper... Drory relevint)
Improved PRECeS-0.8
Reseaced condidiree
Final prediction
(DDDv)}
Beineced condiitiee Quarry
6... inform the dislgn
(a) Overview
(c) LLM agentic ranking module
Figure 2: Illustration of the proposed hybrid language model (HLM-Cite) workflow.
3 Methods
3.1 Overview
To effectively predict core citations from large-scale candidate sets, we integrate the capability of both embedding and generative LMs, forming hybrid language models workflow (HLM-Citee. We illustrate designs of the workflow in Figure
As shown in Figure 2h. the HLM-Cite workflow consists of two major modules, i.c., the retrieval module (Section 3.2) and the LLM agentic ranking module (Section 3.3). When given query q and a candidate set Cg with the size of t.. we first call the retrieval module, a pretrained text embedding model finetuned with training data. We calculate the embedding vectors of q and each paper in Cy.
denoted as V, and V, = [v] {v.,..., v }, where we concatenate the title and abstract as inputs. Based on the inner products between V, and each vector Vg. we retrieve Tq papers with the highest probability of being core citations of from Cg, forming the retrieval set Rg Subsequently, we employ LLM agents in the ranking module collaboratively analyze the retrieved papers in Rg and rank them according to their likelihood of being core citations, improving accuracy. Finally, we take the top tl papers as the prediction result.
3.2 Retrieval Module
3.2.1
Model Structure
Here, we introduce the structure of the text embedding model used in the retrieval module. We employ the GTE-base pretrain model [16), one of the top models on the Massive Text Embedding Benchmark (MTEB) leaderboard [17]. Its 10M parameters are initialized from BERT (18) and trained with multi-stage contrastive learning tasks, embedding input text into 768 f68-dimensional dense vector. We freeze the lower 7 layers of the GTE-base model and only finetune parameters in the higher layers, as shown in Figure P.. As empirically proven in previous research (19) such design can reduce computational consumption while maintaining the transfcrability in finetuning.

---

3.2.2 Curriculum Finetuning
As mentioned above, superficial citations act as hard negatives, adding to the difficulty of distin- guishing core citations. Therefore, instead of directly transferring the GTE-base model to pick core citations from superficial citations and non-citations, we designed a two-stage curriculum finetuning as Figure P to gradually adapt the general-corpus model to our specific task, from easy to hard.
In the first stage, we finetune the model via classification task that only distinguishes the core citation from non-citations, excluding the interference of superficial citations, i.e., the hard negatives. We construct each training data with one query, one of its core citations, and numerous non-citations, and we use cross-entropy loss for classification error in this stage.
In the second stage, we fully consider the ranking task of distinguishing core citations, superficial citations, and non-citations. We include query together with its multiple core citations, superficial citations, and non-citations in each training data, and we apply NeuraINDCG loss function, a differentiable approximation of NDCG (20), to measure the difference between the model output and the ground-truth ranking. In both stages, we use in-batch negative sampling (2T) to obtain non-citations for each query reduce the embedding cost.
3.3
LLM Agentic Ranking Module
3.3.1 Overall Procedure
To improve the accuracy of core citation prediction, we incorporate LLMs' textual reasoning capability to rectify the ranking of papers retrieved in the previous stage by core-citation likelihood. As we illustrate in Figure 2 the LLM agentic ranking module consists of three agents, the analyzer, the decider, and the guider, which are all driven by LLMs and collaborate via natural language communications. Given query paper and its possible core citations retrieved from the candidate set, we first employ the analyzer to analyze the logical relationship between each individual paper in the retrieval set and the query paper. Then, we feed the analysis to the decider to obtain revised ranking of their likelihood of becoming core citations, drawing final prediction results. In addition, design guider to enhance complex reasoning, where it produces one-shot example under human supervision, assisting the analyzer and the decider via the chain of thought (CoT) method [22].
Also, we find that one useful technique in the LLM agentic ranking module is not to rank all retrieved candidates. Specifically, with the retrieval size of rg and t, core citations in the candidate set, we exempt the (2t) - r4) retrieved candidates with largest inner products from reranking, and then we rerank the remaining 2 - (gg) retrieved candidates with the LLM agents and selected the top (rg - (2)) ones, resulting in lg selected candidates in total. For example, when retrieval size is 7, we keep top-3 candidate unchanged and only rank the latter candidates; when retrieval size 8, we keep top-2 candidate unchanged and only rank the latter 6 candidates; and so on. The intuition for this is that the top candidates retrieved by the text embedding model tend to be core citations more safely. Therefore, only adjusting the latter ones is rational solution that reduces the text length inputted into the LLMs and thereby improves the accuracy. We provide the detailed prompts used for the agents in Appendix A.I
3.3.2 Design of LLM Agents
Analyzer: from textual similarity to logical relationship. Intuitively, predicting citations requires in-depth understandings of the logical relationships among the papers, rather than only focusing on the textual similarity between their titles and abstracts. Therefore, we design the analyzer to extract why the query paper cites each of the candidates. Since plentiful knowledge has been encoded in the LLM as an implicit knowledge base, the agent can perform such analysis without domain-specific
finetuning (23 24 25
Decider: final ranking for core citation prediction. Based on the obtained analysis of paper relationships, employ the decider to generate the final ranking of core-citation likelihoods. Besides simple ranking results, we prompt the agent to output corresponding explanations alongside, improving the rationality of its results 26 227.
Guider: one-shot learning. To provide one-shot example for the analyzer and decider, we first select one representative query paper and several candidates outside the test set. As shown in Figure 2 the
5

---

28 include papers about (I) Neural Probabilistic where the ground truth ranking is 2-3-1. The guider goes through the analyze-decide procedure and produces group of exemplary analysis and rectified ranking. We manually review and revise the obtained analysis and ranking texts, making sure they correctly reveal that (2) serves as the research foundation of the query, (3) discusses related recent advancements, while (I) only provides some historical contexts. Then we respectively feed the texts to the analyzer and decider via the chain of thought (CoT) [22] method, concatenate them at the beginning of the prompts, Here we only summarize the essence of guider's exemplary output due to limited space, and the full texts are available in Appendix A.6
candidates of query paper about Transformer-X XL
Model (29). (2) Transformers [30]. and (3) BERT 18
4 Experiments
4.1 Dataset
We conduct experiments based on Microsoft Academic Graph (MAG) [15] which archives hundreds of millions of research papers across 19 major scientific domains, forming huge citation network. We traverse the dataset and filter 12M papers with abundant core citations and superficial citations, from which randomly sample 450,000 queries and subsequently sample 5 core citations and 5 superficial citations for each query. We randomly divide the sampled queries into 8:2 as training and testing sets. Categorizing the scientific domains into natural science (biology, chemistry, computer science, engineering, environmental science, geography, geology, materials science, mathematics, medicine, physics) and social science (art, business, economics, history, philosophy, political science, psychology, sociology), we show statistics of the dataset in Table [ Please note that a natural science query paper may cite some papers from the social science domain and vice versa.
Table I: Dataset statisties
Training set
Testing set
Scientific domain
Total
Query
Candidate
Query
Candidate
Natural science
386655
3830273
48388
479596
4744912
Social science
13345
169727
1612
20404
205088
Total
400000
4000000
50000
500000
4950000
4.2 Baselines
We mainly evaluate our methods against three categories of baselines: simple rule-based method, LMs specifically designed for scientific texts, and pretrained LMs for peneroe-purpose tasks. In the first category, we mainly predict core citation based on the degree of keyword overlap, i.e., the more overlap the candidate paper's keywords have with the query paper, the more likely it is to be a core citation. The second category includes SciBERT [1., METAG (6), PATTON, SciPATTON (5). SPECTER 3 32), SciNCL 4.. and SciMult (33 SciBERT is pretrained on millions of research papers from Semantic Scholar with the same approaches as BERT; METAG learns to generate multiple embeddings for various kinds of patterns of citation network relationships; PATTON and SciPATTON are finetuned with network masked language modeling and masked node prediction tasks citation networks from BERT and SciBERT respectively; SPECTER is continuously pretrained from SciBERT with a contrastive objective; SciNCL is an improvement of SPECTER by considering hard-to-learn negatives and positives in contrastive learning; and SciMult is multi-task contrastive learning framework, which focuses on finetuning models with common knowledge sharing across different scientific literature understanding tasks. The third category and ApenAll-mbedding pretrained with masked language modeling and next sentence prediction objectives on Wikipedia and BookCorpus; GTE is a series of top embedding models finetuned from BERT with multi-stage contrastive learning task; and the latter two are advanced universal embedding models proposed by OpenAl. We access these models from off-the-shelf pretrained parameters or API calls and include different scale versions of each model when available.
includes BERT [18, GTE [16] 34),
BERT
httpp://ppeaas..oo//////eeeoere/ www.:eewwweedinnnoodeee--------e mwdels and poodels-and-api-uppaaeess api-updates/

---

4.3 Overall Performance
We conduct the curriculum finetuning of our retrieval module with the batch size of 512 and 96 respectively in two stages, and each train for 10 epochs. The training process takes approximately 12 hours on 8XNVIDIA A100 80G GPUs in total. Then, we call OpenAI API to access GPT models for L.M agentic ranking, where we keep using GPT-4 as the guider but alternate two versions of GPTs for the analyzer and the decider. For more implementation details, please refer to Appendix A.2
Table 2: Overall performance. Bold and underline indicate the best and second best performance.
Model
Natural
science
Social
science
Overall
PREC
3/5
NDCG@35 3/5
PRECØ
3/5
NDCG@3 3/5
PRECC@3/5
NDCG@ 3/5
Keywords overlap
0.334
0.267
0.302
0.262
0.402
0.322
0.359
0.311
0.336
0.269
0.304
0.264
SCIBERT 31
0.033
0.046
0.036
0.050
0.083
0.089
0.087
0.076
0.054
0.046
0.057
0.031
METAG G
0.112
0.089
0.124
0.104
0.180
0.142
0.196
0.166
0.114
0.090
0.126
0.106
PATTON
0.248
0,201
0.266
0.229
0,407
0.341
0,429
0.378
0.253
0.205
0.271
0.234
SciPATTON [
0.444
0,368
0.470
0,410
0.529
0.448
0,548
0.487
0.447
0,371
0.472
0.413
SPECTER [3]
0.542
0.457
0.567
0.502
0.620
0.537
0.641
0.579
0.545
0.460
0,570
0,504
SciNCL
0.575
0.495
0.598
0,537
0.634
0,558
0.655
0.597
0.577
0,497
0.600
0.539
SciMult-vanilla
33
0.568
0,483
0.591
0.527
0.623
0.547
0.644
0.586
0.569
0,485
0.593
0.529
SolMult-Mol
33
0.578
0.493
0.601
0.537
0.637
0.558
0,658
0.598
0.579
0,496
0,603
0.539
SPECTER-2.0
12
0.600
0,512
0.625
0.558
0.654
0,579
0.674
0.617
0.602
0.515
0.627
0,560
BERT-basc
18
0.036
0.034
0,036
0.035
0.129
0,115
0.133
0.122
0.039
0.036
0.039
0.038
BERT-large
18
0.025
0.027
0.024
0.026
0,055
0.062
0,051
0.057
0.026
0.029
0.025
0.027
OpenAl-ada-002
0.623
0,534
0.646
0.579
0.671
0.590
0,692
0.631
0.624
0,536
0.648
0,581
OpenAl-3
0.632
0.543
0.655
0.588
0.671
0.592
0.691
0.632
0.633
0.545
0.656
0,589
GTE-buse [6
0.638
0.555
0.659
0.596
0.669
0.596
0.688
0.633
0.639
0,556
0.659
0.597
GTE-basc-v1.5 134
0.637
0.549
0.660
0.593
0.670
0.591
0.692
0.631
0.638
0.551
0.661
0.594
GTE-large [T6
0.640
0,556
0.661
0.597
0.669
0.593
0.690
0.632
0.641
0,557
0.662
0.599
GTE-large-vL.. 1.5 [4
0.647
0.562
0.669
0.605
0.690
0.606
0.707
0.645
0.649
0.563
0.671
0.606
H-LM (GPT3.5)
0.725
0.644
0.734
0.677
0.743
0,661
0.751
0.693
0,725
0.644
0.735
0.677
H-LM (GPT4o)*
0.736
0.655
0.743
0.686
0.756
0.670
0.763
0.702
0.736
0.655
0.743
0.686
In evaluation, we set vast candidate sets with by = 10K (t) 22 5) for all models and set
the retrieval size to be rg- 8 in our workflow. We evaluate the performance via PREC@3/5 and NDCG@ 3/5, and show the results in Table 2 The results illustrate that our method significantly surpasses all the baselines across all scientific domains with all metrics, with an overall PREC @5 improvement up to 17.6%. We verify the statistical significance of the performance improvement in Appendix I Mentioning that without loss of statistical significance, we only randomly test 10% of the testing set with GPT-40 due to API rate limits.
Retrieval set R. PRECOS-O.6
Analysis
Ranking
A DNA Nanorabnt Functions as Cancer Therapeutic
unchanged
Moteculltly Self-aseeblln blddd Nucleic Acid Nanopartic
Structural DNA Nanotechnology.
Callis
Top
design mechanisma ?... inspine DNA nscorobats and targeted drug delivery.. (less relevan))
inform the development of DNA nanorobots... (very relecant)
draw insights from DNA... almost instaveet)]
nanorobot... (very relevant)
Improved PRECa
7. Self-Asemmlldd DNA Hydr Hydrogels with...
unchanged
Figure 3: Case study of the LLM agentic ranking module.
In order to verify the rationality of LLM agentic ranking process, we provide the summary of a representative testing sample. We show the query paper, which designs DNA Nanorobot 35),
and the retrieved candidates in Figure 3 It turns out that our analyzer correctly reveals that the
two candidates with core-citation ground truth inform the key design or the query paper [36] 37];
the candidate with auperficial-citation ground truth inspires some design details [38]; while the
non-citation candidate only mentions some very broad context that is almost irrelevant 39 Based on
the rational analysis, the decider correctly ranks the retrieval set and improves the precision. Please refer Appendix to access the full texts of this case study.
4.4 Ablation Studies
In order to verify the validity of our designs, we conduct ablation studies regarding both curriculum finetuning of the retrieval module and LLM agents design in the ranking module. We show the results in Table In the former part, we respectively delete the first and second stages of the curriculum and calculate the metrics the retrieval set. The performance drop in both ablations indicates that our
7

---

curriculum design does enable the udaption of the pretrained model from easy to hard, improving its transfer performance from general corpus to scientific documents. the latter part, we respectively remove the analyzer and the guider. Specifically, without the analyzer, the decider directly ranks the retrieved candidates based on their raw titles and abstracts; without the guider, the analyzer and decider perform their tasks without the guidance of the one-shot example. It turns out that the absence of any agent leads to performance degradation, proving the essential role of each of them. We verify the statistical significance of the performance degradation in Appendix A.5.2
Table 3: Ablation studies. Bold indicates the best performance.
Natural science
Social science
Overall
Model
PREC@33 3/5
NDCG@3/5
PREC@3/5
NDCG@33 3/5
PREC
NDCG@3/5
Full curriculum
0.683
0.598
0.705
0.641
0.704
0.623
0.724
0.663
0.684
0.598
0.706
0.641
w/o Stagel
0.682
0.595
0.703
0.638
0.706
0.623
0.724
0.662
0.682
0.596
0.704
0.639
w/o Stage2
0.666
0.587
0.686
0.626
0.614
0.705
0.650
0.667
0.588
0.687
0.627
Full workflow
0.725
0.644
0.734
0.677
0.743
0.661
0.751
0.693
0.725
0.644
0.735
0.677
Analyzer
0.723
0.629
0.733
0.666
0.736
0.648
0.747
0.684
0.723
0.630
0.734
0.667
w/o Guider
0.686
0.594
0.707
0.638
0.702
0.618
0.723
0.660
0.686
0.595
0.708
0.639
w/o Guider
0.659
0.580
0.688
0.626
0.686
0.608
0.712
0.651
0.660
0,581
0.689
0.627
4.5 Analysis
In this section, we provide in-depth analysis of various key elements in the HLM-Cite workflow, enabling a better understanding of our design. Here, if there no special explanation, we all employ GPT-335 as our analyzer and decider. Mentioning that due to API rate limits, we only test 10% of the testing set this section.
Natural science
Social science
Natural science
Social science
0.70
0.68-
0.05
0.60
8000
B.6E:
6000
PRECOS
PRECBS
6
0.55
0.35
0.62
soco
Token
0.54
t0co
0.50
0.45
0.40
2000
2000
10*
0.40
Candidate
Candidale lg
Retrieval size G
Retrieval size f
(a) Effect of candidte size
(b) Effect of retricval size
Figure 4: Effect of candidate size and retrieval size. In all panels, 95% CI are shown as error bars.
4.5.1 Effeet of Candidate Size
To illustrate the advantage of our method large-scale candidate sets, which are normal in real- world applications, we keep 12 = 12 = consistent and change the number of non-citations to construct candidate sets with tg 1k, 10K, and 100K. As shown in Figure 4,. regardless of the candidate size, our method significantly surpasses all top baselines and even achieves higher relative performance improvement on larger candidate sets (up 18.5% in t, 100K). We provide results with other metrics in Appendix A.3 where the conclusion is consistent.
4.5.2 Effect of Retrieval Size
In our hybrid workflow, retrieval size ry is a key hyper-parameter that balances the work between the retrieval module and the LLM agentic ranking module. To explore the effect of rg. we alter it from 10 and show the performance together with LLM token consumption per query in Figure 4p. The results indicate that when rq increases, the performance increases at the cost of more token consumption. Larger r, leads to a higher recall rate of core citations in the retrieval set, and thereby, LIM agents have the potential to pick out more core citations from the texts with increased length. However, when ry large enough, continuing to increase leads to performance drop while consuming more tokens. We believe this is because too many retrieved candidates surpass the

---

reasoning ability of LLMs, leading to confused analysis and low-quality ranking. Generally observed from the results, the optimal value of rg supposed to be 8 and for natural and social science, respectively. Results with other metrics in Appendix A.4 show consistent conclusion.
4.5.3 Effect of One-shot Example
As studied in previous research [40), CoT enhances the performance of LLMs by demonstrating the logical structure of reasoning rather than providing specific knowledge content. Here, we investigate whether this is true in our hybrid workflow. We extend one-shot learning into a few-shot version. In this version, we produce individual example for each scientific domain, where full texts available in our GitHub repository. This provides more domain knowledge while maintaining an identical logical structure. The results in Table 4 show significant performance difference between one-shot and few-shot learning, proving that what matters in CoT prompting is the logical structure of reasoning rather than specific domain knowledge.
Table 4: Comparison between mee-shot and few-shot learning. Bold indicates the best performance.
Natural science
Social science
Overall
Model
PREC@3/5
NDCG@ 3/5
PREC@3M
NDCG@ 3/5
PREC@3/5
NDCG@3/5
One-shot
0.713
0.635
0.726
0.669
0.727
0.645
0.741
0.682
0.713
0.635
0.727
0.670
Few-shot
0.720
0.633
0.731
0.668
0.731
0.649
0.744
0.684
0.720
0.633
0.732
0.669
4.5.4 Effect of LLM Types
We explore the effect of substituting GPT-3.5 in our workflow with other open-source and lightweight LLMs. Here, we keep using GPT-4 as the guider to provide a high-quality one-shot example and change the analyzer and decider to various open-source LLMs We explore using two versions of Llama3, one of the most famous open-source LLMs; two versions of Mixtral, a mixture of experts (MoE) model; and ChatGLM2-66 -2..B. and find that although larger LLMS perform slightly better, i.e., Llama3- 70B wins Llama3-8B, and Mixtral-8 x 22B wins Mixtral-8 8 7B, these lightweight LLMs all perform significantly worse than GPT models. This highlights the importance of implicit knowledge in L.L''s large-scale parameters, which is crucial for solving tasks like citation prediction that require strong professional knowledge.
Engllsh bilingual model. We show the results in Table
Table 5: Comparison between different types of LLMs as agents. Bold indicates the best performance.
Natural science
Social science
Overall
Model
PREC@3/5
NDCG@3/5
PREC@3/5
NDCG@36
PREC@3/5
NDCG@3/5
GPT-40
0.736
0.655
0.743
0.686
0.756
0.670
0.763
0.702
0.736
0.655
0.743
0.686
GPT-4
0.721
0.649
0.732
0.680
0.723
0.660
0.738
0.690
0.721
0.649
0.732
0.680
GPT-3.5
0.713
0.635
0.726
0.669
0.727
0.645
0.741
0.682
0.713
0.635
0.727
0.670
Llama3-70B
0.681
0.593
0.704
0.637
0.688
0.604
0.713
0.649
0.681
0.593
0.704
0.637
Llama3-8B
0.668
0.590
0.695
0.634
0.679
0.604
0.707
0.648
0.669
0.590
0.695
0.634
Mixtral-8+22
0.678
0.592
0.702
0.636
0.690
0.604
0.715
0.649
0.678
0.592
0.702
0.636
Mixtral-8*7B
0.678
0.591
0.701
0.635
0.692
0.601
0.716
0.647
0.678
0.591
0.702
0.636
CharGLM2-61
0.671
0.585
0.697
0.631
0.673
0.589
0.703
0.637
0.671
0.585
0.697
0.631
Related Works
5.1 Pretrained Language Models (PLMs)
Pretrained language models have long been studied and reached great success. Various small-scale embedding models have been trained via different objectives, such as masked token prediction 4I).
contrastive learning 3 42 4 [6], and permutation language modeling 43.. These models require
fewer computational resources and especially suitable for wide range of tasks the large-scale
corpus, including classification, clustering, retrieval 17,, etc. On the other hand, generative large
/tttps:/1llam. meta. com https://nistral. ai and https: wttp.tc.ocm///////aaaaooooo

---

language models (LLMs) have developed unprecedentedly in recent years. Pretrained on vast corpus, LLMs exhibit strong shot [3 and zero-shot [24 learning ability, reaching superior performance
on text analyzing 44
45
46
code generation 47 48 and even solving math problems 49 50
However, most of the existing works lack the combination of these two categories of models. this paper, we design the hybrid language workflow, incorporating small embedding models" advantage of efficient large-scale retrieval and generative LLMs' capability of textual reasoning.
5.2 LLM Agents
Utilizing the strong reasoning capability and human-like behavior of LLMSs, researchers have explored various applications based on agents driven by LLMs. First, LLM agents for decision-making reach success in sandbox games [51 (2), robot controlling [53]. and navigation [54]. Besides, group of LLM agents can simulate daily social life [55], generate physical mobility behavior [56), and reveal macroeconomic mechanisms [57], providing insights for social science research. Closer our task, role-fused LLM agents can collaboratively solve natural language processing tasks via analysis and discussions 45 27 [88. However, due the limited context length in LLM reasoning, existing studies face difficulty handling tasks with extremely long texts, such as citation precision on vast candidate sets. In this paper, we incorporate generative LLMs with embedding models, enabling our hybrid workflow to work on very large candidate sets.
6 Conclusions
In this paper, we investigate the task of scientific citation prediction. We first define the novel concept of core citation and thereby evolve the conventional citation prediction task into a more meaningful version of distinguishing the core citations. Then, we propose hybrid language model workflow that incorporates the capability of both embedding and generative LMs. Through extensive experiments and in-depth analysis, we verify the validity of our design and illustrate its superior performance in tasks with gigantic candidate sets. One major limitation of our method lies in LLMs' illusion problem. Despite average performance improvement, LLMs may output unfaithful analysis under certain circumstances and poison specific samples. Therefore, how to verify the output of LLM agents and improve the reliability of our hybrid workflow worth future studies.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under 231AA02114, U20B2060, 62272260, and Beijing National Research Center for Information Science and Technology.
References
[1] Johan SG Chu and James Evans. Slowed canonical progress in large fields of science. Proceedings of the National Academy of Sciences, 1 2222111 18, 2021.
[2] Xiao Yu, Quanquan Gu, Mianwei Zhou, and Jiawei Han. Citation prediction in heterogeneous bibliographic networks. In Proceedings of the 2012 SIAM international conference on data mining, pages 1119-1199 1130. SIAM, 2012.
[3] Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. SPECTER: document-level representation learning using Extation-informn transformers. In ACL, pages 2270-2282. Association for Computational Linguistics, 2020.
[4] Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp. and Georg Rehm. Neighbor- hood contrastive learning for scientific document representations with citation embeddings. In EMNLP, pages 11670-11660 1168. Association for Computational Linguistics, 2022.
[5] Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, and Jiawei Han. Patton: Language model pretraining on text-rich networks. ACL (1), pages 7005-7020. Association for Computational Linguistics, 2023.
10

---

[6] Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Han Zhao, and Jiawei Han. Learning multiplex embeddings on text-rich networks with one text encoder. arXiv preprint arXiv:2310.06684 2023.
[7] Ronald Seoh, Haw-Shiuan Chang, and Andrew McCallum. Encoding multi-domain scientific papers by ensembling multiple cls tokens. arXiv preprini arXiv:2309.0433; 2023.
[8] Jingtao Ding, Chang Liu, Yu Zheng, Yunke Zhang, Zihan Yu, Ruikun Li, Hongyi Chen, Jinghua Piao, Huandong Wang, Jiazhen Liu, et al. Artificial intelligence for complex network: Potential, methodology and application. arXiv preprint arXiv: 240.::2400..888/7 2024.
[9] Santo Fortunato, Carl T Bergstrom, Katy Börner, James A Evans, Dirk Helbing, Staša Milojević, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, et al. Science of science. Science, 359(6779) 2018.
[10] Lingfei Wu, Dashun Wang, and James A Evans. Large teams develop and small teams disrupt science and technology. Nature, 566(7744):378-388 382, 2019.
[11] Unai Rodrrezz Federico Battiston, Guilherme Ferraz de Arruda, Yamir Moreno, Matjaž Perc, and Vito Latora. Evolutionary dynamies of higher-order interactions in social networks. Nature Human Behaviour, 5(5):586-595, 2021.
[12] Michael Park, Erin Leahey, and Russell Funk. Papers and patents are becoming less disruptive over time. Nature, 613(7942):1388 13444444)3111 2023.
[13] Fengli Xu, Lingfei Wu, and James Evans. Flat teams drive scientific innovation. Proceedings of the National Academy of Sciences, 1 119029))2200922119 2022.
[14] Yiling Lin, Carl Benedikt Frey, and Lingfei Wu. Remote collaboration fuses fewer breakthrough ideas. Nature, 623(79899:987-991, 2023.
[15] Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pages 243 443-246, 2015.
[16] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang, Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.0328) 2023.
[17] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07311 2022.
[18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
[19] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi Revisiting few-sample BERT fine-tuning. In ICLR. OpenReview.net, 2021.
[20] Przemyslaw Pobrotyn and Radostaw Biatobrzeski. Neuralndeg: Direct optimisation of ranking metric via differentiable relaxation of sorting. arXiv preprint
[21] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1). pages 6769-6781, Association for Computational Linguistics, 2020.
H. Lewis, Ledell Wu, Sergey Edunov,
[22] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Ahain-of-though prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
[23] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 2020.
11

---

[24] Takeshi Kojima. Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, Advances in neural information processing systems, 15:22199-22213, 2022.
[25] Songwei Li, Jie Feng, Jiawei Chi, Xinyuan Hu, Xiaomeng Zhao, and Fengli Xu. Limp: Large language model enhanced intent-aware mobility prediction. arXiv preprint arXiv:2408, /2832, 2024.
[26] Md Rizwan Parvez. Evidence to generate (e2g): A single-agent two-step prompting for context grounded and retrieval augmented reasoning. arXiv preprint arXiv:2401.05775 2024.
[27] Lin Chen, Fengli Xu, Nian Li, Zhenyu Han, Meng Wang. Yong Li, and Pan Hui. Large language model-driven meta-structure discovery in heterogeneous information network, Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 307-318 2024.
[28] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint 1901.09000028000 2019.
[29] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[31] Iz Beltagy, Kyle Lo, and Arman Cohan, Scibert: A pretrained language model for scientific text. In EMNLP/JJCNLP (I), pages 3613-3618. Association for Computational Linguistics, 2019.
[32] Amanpreet Singh, Mike D' Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. Scirepeval: multi-format benchmark for scientific document representations. In EMNLP, pages 5548- 5566. Association for Computational Linguistics, 2023.
[33] Yu Zhang, Hao Cheng, Zhihong Shen, Xiaodong Liu, Ye-Yi Yi Wang, and Jianfeng Gao. Pre- training multi-task contrastive learning models for scientific literature understanding. In EMNLP (Findings). pages 12259- Association for Computational Linguisties, 2023.
[34] Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, text represen-
Baosong Yang. Pengjun Xie, Fei Huang, al. mgte: Generalized tation and reranking models for multilingual text retrieval. arXiv preprint arXiv:2407.1966) 2024.
[35] Suping Li, Qiao Jiang, Shaoli Liu, Yinlong Zhang, Yanhua Tian, Chen Song, Jing Wang, Yiguo Zou, Gregory Anderson, Jing- Yan Han, et al. dna nanorobot functions as a cancer therapeutic in response to molecular trigger in vivo. Nature biotechnology, 36(3):258-264, 2018.
[36] Andre V Pinheiro, Dongran Han, William M Shih, and Hao Yan. Challenges and opportunities for structural dna nanotechnology. Nature nanotechnology. 6(12):763-772, 2011.
[37] Hyukjin Lee, Abigail KR Lytton-Jean, Yi Chen, Kevin T Love, Angela I Park, Emmanouil D Karagiannis, Alfica Sehgal, William Querbes, Christopher S Zurenko, Muthusamy Jayaraman, et al. Molecularly self-assembled nucleic acid nanoparticles for targeted in vivo sirna delivery. Nature nanotechnology. 76)):399-933, 2012.
[38] Yongzheng Xing, Enjun Cheng, Yang Yang, Ping Chen, Tao Zhang, Yawei Sun, Zhongqiang Yang, and Dongsheng Liu. Self-assembled dna hydrogels with designable thermal and enzymatic responsiveness. Advanced Materials, 23(9): 1117-1121, 2011.
[39] Songming Peng, Thomas Derrien, Jinhui Cui, Chuanying Xu, and Dan Luo. From cells to dna materials. Materials Today, 1555)1190-144, 2012.
12

---

[40] Aman Madaan, Katherine Hermann, and Amir Yazdanbakhsh, What makes chain -ffiin--f--nne oia--ff-tttho thought prompting effective? A counterfactual study. In EMNLP (Findings), pages 1448-1535. Associa- tion for Computational Linguistics, 2023.
[41] Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint 1903. 10676, 2019.
[42] Yu Meng, Chenyan Xiong, Payal Bajaj. Paul Bennett, Jiawei Han, Xia Song, et al. Coco-Im: Correcting and contrasting text sequences for language model pretraining. Advances in Neural Information Processing Systems, 34:23102-2311 14, 2021.
[43] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[44] Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. Llm-assisted content analysis: Using large language models to support deductive coding. arXiv preprint arXiv:23066. /4924, 2023.
[45] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.0720. 2023.
[46] Xiaochong Lan, Chen Gao, Depeng Jin, and Yong Li. Stance detection with collaborative role-infused lim-based agents. arXiv preprint arXiv:2310. 10467 2023.
[47] Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. Llm is like a box of chocolates: the non-determinism of chatgpt in code generation. arXiv preprint arXiv:2308.02822 2023.
[48] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang. and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances Neural Information Processing Systems, 36, 2024.
[49] Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:23303.00 053399 2023.
[50] Yiran Wu, Feiran Jia, Shaokun Zhang, Qingyun Wu, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, and Chi Wang. An empirical study on challenging math problem solving with gpt-4. arXiv preprint
2023.
[51] Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
[52] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305 17.5..17444.. /744,4144444 2023.
[53] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embod- ied reasoning through planning with language models. In CoRL, volume 205 of Proceedings of Machine Learning Research, pages 1769-1782. PMLR, 2022.
[54] Qingbin Zeng, Qinglong Yang, Shunan Dong, Heming Du, Liang Zheng, Fengli Xu, and Yong Li. Perceive, reflect, and plan: Designing I1m agent for goal-directed city navigation without instructions. arXiv preprint arXiv:2408. 000008004168.. 168, 2024.
[55] Joon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In UIST; pages 2:1-2: ACM, 2023.
13

---

[56] Chenyang Shao, Fengli Xu, Bingbing Fan, Jingtao Ding, Yuan Yuan, Meng Wang, and Yong Li. Beyond imitation: Generating human mobility from context-aware reasoning with large language models. arXiv preprint arXiv:2402.098: 2024.
[57] Nian Li, Chen Gao, Yong Li, and Qingmin Liao. Large language model-empowered agents for simulating macrocconomic activities. arXiv preprint arXiv:2310.10430 2023.
[58] Xiaochong Lan, Yiming Cheng, Li Sheng, Chen Gao, and Yong Li. Depression detection on social media with large language models. arXiv preprint arX1v:2403.10750, 2024.
14

---

A Appendix
A.1 Prompts for the LLM agents
The specific prompt for the analyzer is as follows:
System prompt: Now you sophisticated researcher and information analyst, and going to investigate the problem of a specific paper citation. Your analysis should be based on the following steps: Explore citation conventions and standards in academic fields. For example, citation serve to acknowledge prior work, provide evidence or support, facilitate further exploration and allow readers trace the development and history of ideas methodologies.
Prompt: Here the title and abstract of the query paper. Title: (QueryPaperTite) Abstract: (QueryPaperAbstracll Now you are doing research following up this paper above. Here are some other rescarch papers which have been already cited by the query paper. Paper 1 Title: Paper 2 Title: /CandidatePPperIIIii Abstract: /Candidate Abbstractt, Try to think abductively and convince yourself as researcher. Figure out why the query paper cite these one by one. Try to think step by step before giving the answer.
|CandidatePaper | Abstract:
PandiddaeeaperIAAsaee
The specific prompt for the decider is as follows:
System prompt: Your role is to assist in predicting which research papers most likely be cited together based on a given set of papers or topics. Strive for faimess and objectivity.
Prompt: Here is the title and abstract of the query paper. Title: /QueryPaperTite] Abstract:
(QueryPaperAbstrract. There are some other candidate papers and the analysis of why this query paper cites these. Paper Title: [CandidatePaperIIIi Title) Analysis: (CandidatePaper/ IAnalyssss Paper Title: PaandidaaePaper2iiiiii
Now you are doing research following up this query paper. Use the analysis to identify patterss themes that suggest potential citation relationships. Rank these candidate papers the order you are most likely to cite from the perspective of research follower and provide explanations or justifications for your reasoning.
Analysis:
15

---

A.2 Implementation Details
In this section, we provide all implementation details for reproducibility in Table
Table 6: Implementation details
Module
Element
Detail
OS
Ubuntu 22.04.2
CUDA
11.7
System
Python
3.11.4
Pytorch
2.0.1
Device
8*NVIDIA A100 80G
Batch size
512
Number of epochs
10
Max token length
512
Curriculum stage
Selected model epoch
10
Optimzer
Adam
Learning rate
0.00001
Random seed
2024
Batch size
96
Number of epochs
10
Max token length
512
Curriculum stage
Selected model epoch
Optimzer
Adam
Learning rate
0.00001
Random seed
2024
Model name Temperature
Analyzer
0.0
Model name Temperature
Decider
0.0
Model name Temperature
gpt-4-0125-previeet
Guider
0.0
16

---

A.3 Supplementary Figures in Effect of Candidate Size Analysis
We provide performance comparisons among our method and several top baselines with different candidate sizes t in Figure | In both natural and social science domains, measured by both PREC@S and NDCG@5, 5, our method significantly surpasses all baselines regardless of the candidate size. The conclusion is consistent with the main text.
Natural science
Social science
Overall
0.70
0.65
0.65
0.60
0.60
PREC0S
PREC@S
0.60
PRECOS
0.55
0.50
0.50
0.45
0.45
0.40
10a
10
10a
10a
10'
10a
101
10*
10a
Candidate size fq
Candidate size to
Candidate size tq
Natural science
Social science
Overall
0.75
0.70
0.70
NDCG@5
NDCG@5
NDCG@5
0.60
0.60-
0.55
0.55
0.50
0.50
0.50
0.45
0.40
10
10
10a
10'
101
10a
105
Candidate size 4
Candidate size
Candidate
HLM-Cite (GPT-3.5)
GTE-large
GTE-base
OpenAl-3
OpenAl-ada-002
Figure 5: Analysis the effect of candidate size ty. In all panels, 95% CI are shown errorbars.
17

---

A.4 Supplementary Figures in Effect of Retrieval Size Analysis
We provide the performance of our method together with LLM token consumption per query with different retrieval sizes rg in Figure 6 In both the natural and social science domains, when 7q increases, the performance increases the cost of more token consumption. However, when rg is large enough, continuing increase leads to performance drop, while consuming even more tokens. Measured by both PREC@S and NDCG@S @5, the optimal value of rg supposed be 8 and 7 for natural and social science, respectively. The conclusion is consistent with the main text.
Natura science
Social science
Overall
0.64
10000
10000
10000
0.66
Token count
Token count
PREC@5
PREC
6000
6000
6000
Token
0.62
4000
0.62
4000
4000
0.61
2000
2000
2000
0.60
10
10
10
Retrieva size fa
Retrieval size rg
Retrieval size rg
Natura science
Social science
Overall
0.675
10000
0.675
10000
0.670
0.670
0.685
8000
8000
Token count
0.665
8000
Token count
NDCG@S
NDCG@5
NDCG@5
0.660
6000
6000
0.660
6000
0.655
4000
4000
0.650
4000
0.64
0.645
0.640
0.640
10
Retrieva size
Retrieval size
Retrieval size
Prompt token
Output token
Figure 6: Analysis on the effect of retrieval size ry. In all panels, 95% CI are shown as errorbars.
18

---

A.5 Statistical Significance
A.5.1 Overall Performance
We conduct statistical significance tests to better compare our model with the strongest baseline. In Table we used the two-tailed t-test between the performance of our model and the strongest baseline. Our method surpasses the top baselines in all fields (p < 0.01 in most individual fields, and p < 0.001 averaging all fields through t-test), demonstrating its general applicability to a wide range of fields.
Table 7: Overall performance. Bold and underline indicate the best performance and the best baseline.
Our method performs significantly (p- < 0.01, < 0.1) ) better than the best baseline in majority of fields. Baseline methods include M1: Keywords Overlap. M2: SciBERT, M3: METAG, M4:
PATTON, M5: SciPATTON, M6: SPECTER, M7: SciNCL, M8: SeiMult-Vanillee M9: SciMult-
MOE, M10: SPECTER-2.0, M11: BERT-base, M12: BERT-large, M13: OpenAl-ada-002, M14;
OpenAI-3, M15: GTE-base, M16: GTE-base-vl.5, M17: GTE-large, M18: GTE-large-vI.s. Our methods include OursI: HLM-Cite (GPT-3.5), Ours2: HLM-Cite (GPT-40).
Fieh
Matex
M1
M2
MB
M44
M5
M6
M7
385
MP
306
MI1
M12
M15
M144
MI5
316
301
MIS
Dami
Oued
201
E1901
6.3%
0.505
8:55
1:95
3164
E/913
F164
F164
3061
0.388
Billegs
B.504
B914
3403
5968
(152)
169
0.64
Chenterr
6/99
8:11
B-166
0.75
0.70
PRUCC 9-
B400
0:51
3.51
0.51
5324
tum
0.56
0.566
5.51
5361
5301
0.65
0.61
M
3.544
01156
01174
6511
5:40
6534
5331
1114
DA1
5803
5327
5308
E510
0,330
0.51
(K:7
NDCUMS
8:55
0.300
8.665
B480
0351
0346
5311
5559
5559
526)
5,00
5900
5300
(40)
0,55
0.644
3:414
B451
2415
3510
2233
2649
1.00
1454
0.730
0.756
D091
8.500
8101
640
5310
5446
5.66
b6th
546
0759
0.090
0.717
B313
130h
0913
0113
1301
1901
0303
Gouemaper
3:48
B446
Bm0
536
5363
6111
1011
1900
500
0.533
Natural
Godlggr
PRRC
3.45
0.38
0400
[401
0483
5.16
1.50
1-50
2400
0106
0.667
Seltice
HUR 9
1401
5110
3411
0,20
1:1h
8511
0.06
0266
0-41
5-71
2415
2433
0,60
5240
0333
0258
0254
B1867
0.1%%
3461
2364
512
B671
260
2003
360
2061
0.53
MUBC
1.64
6351
E331
611%
B464
1360
6556
8760
(411
6111
5/4m
1/46
1/48
3001
6677
11L 9
0.42
5040
5,00
D421
2004
2503
U004
2566
2001
0.713
NIKGIS
3270
D0066
0.001
030
839
0004
556
5200
5700
BAIT
5951
0.71
0.455
0:201
5391
010
0100
5401
5943
5505
5100
0.118
(omb)))
8.41
01164
8:16
B634
Sall
544
6154
8671
1011
0.311
Sub-ertnge
3.000
1131
676
6/30
030
[401
5/00
030
560
5760
3100
3161
STER
0.113
2200
Badeyes
3.341
SURT
0.13
8400
1500
0551
5511
0551
5203
2300
5161
311
200
0.684
0.344
6450
159
0/66
2615
2306
1660
366
6.97
0.087
3.35
8:211
146
260
5511
0601
2614
516
2651
200
(e=1
NDCUNS
3.562
0.125
0/366
0550
1566
1556
6554
0464
1155
1746
1700
0,91
0.48
Poliicl
RUBC 3
3-354
BU61
0.165
8534
0566
0556
0.56
3361
61:1
0601
240
0055
0.50
8 3.50
1155
0/15
556
5.63
6111
661
5.6
5.60
002
040
5,04
0.75
Scance
Prccelorr
1310
5710
6113
000
1374
0.355
N
3.56
EURC1
B468
5540
566
5406
5348
5360
2375
5.11
5006
3,60
3360
33m
3x60
51155
01761
0.762
Others
8:23 214
0:213
0441
8350
0381
5405
5261
0173
0601
5931
0126
0.56
0.581
0266
3.56
0.14
B341
5704
5371
5111
5.60
5788
1261
500
03511
0.65
NDCUN3
0.33
D07T6
5.16
0451
5200
1236
1296
280
612
5001
285
027
0.782
6114
B:47
0440
5455
3162
1621
363
5634
5608
2614
0.25
NDCON3
3:244
(Uc]
5.166
0:34
8413
0.44
5350
5555
555
3340
1006
TUCT
1:61
5.60
5.64
1.66
6677
0.86
19

---

A.5.2 Ablation Studies
We conduct statistical significance tests to better compare our model with the ablation versions. In T we used the two-tailed t-test between the performance of our full design and the ablation versions. Generally, all parts of our designs are valid with significance (p < 0.01 p < 0.1 in overall performance through I-test). Moreover, we notice that when focusing on social science papers, which only comprise small proportion of all papers, Stage I of curriculum finetuning is only slightly beneficial. Therefore, when only applying to social science papers, it is an alternative for users to skip Stage I if they want save computational cost with the cost of a slight performance drop. In contrast, when applying to natural science papers, is necessary to keep Stage for better performance.
Table 8: Ablation studies. Bold indicates the full design. The performance of ablations drops significantly < 0.01 p < 0.1 compared to the full design.
Natural science
Social science
Overall
Model
35
N/CG@B5
PREC@
3DCG@B3/5
PREC@3/5
NDCG@ 3/5
curriculum
0.683
0.705
0.704
0.623
0.724
0.684
0.598
0.706
w/o Stagel
0.682
0.595
0.703
0.638
0.706
0.623
0.724
0.662
0.682
0.596
0.704
0.639
w/o Stage2
0.666
0.587
0.686
0.626
0.685
0.614
0.705
0.650
0.667
0,588
0.687
0.627
Full agents
0,725
0.644
0.734
0.677
0.743
0.661
0.751
0.693
0.725
0.644
0,735
0.677
Analyzer
0.723
0.679
0.733
0.736
0.747
0.723
0.734
0.667
w/o Guider
0.686
0.594
0.707
0.638
0.702
0.618
0.723
0.660
0.686
0.595
0.708
0.639
w/o Analyzer& Guider
0.659
0,580
0.688
0,626
0.686
0.608
0.712
0.651
0.660
0,581
0.689
0.627
20

---

A.6 One-shot Example
Full texts of the one-shot example generate by the guider are as follows:
Query paper:
Title: Transformer-XI. Attentive I Language Models Beyond Fixed-Lengte Context
Abstract: Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-X0 that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of segment-level recurrence mechanism and novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and up 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-an results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on Wiki'Text- 103, 21.8 One Billion Word, and 54.5 Penn Treebank (without finetuning). When trained only on WikiText- 103, Transformer-X0 manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and Py PoTchh.
Candidate papers:
1. Title:
Attention is All you Need
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine trans- lation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28,4 4 BLEU on the WMT 2014 Sanglish-to-German translation task, improving over the existing best results, including ensembles by over BLEU
Abstract:
2. Title: Self-attention with relative position representations
Abstract: Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, distances between sequence elements.
3. Title:
Character-Lewe Language Modeling with Deeper Self-Attention
Abstract: LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that deep (64-layer) transformer model (Vaswani et al. 2017) with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses. both at intermediate network layers and intermediate sequence positions.
4. Title:
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre- are-trained BERT model can be fine- sunet with just one additional output layer to create of-he-of-the-art models for wide range tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. obtains new state-of -htt---rrtttt results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7%
21

---

point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement). SQuAD v1.1 question answering Test Fl to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test Fl to 83.1 (5.1 point absolute i improvement..
5. Title:
Adaptive input representations for neural language modeling
Abstract: We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText- 103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity.
6. Title: A Neural Probabilistic Language Model
Abstract: A goal of statistical language modeling is to learn the joint probability function of sequences of words. This is intrinsically difficult because of the curse of dimensionality: we propose to fight it with its own weapons. In the proposed approach one learns simulta- neously (I) distributed rep(cid: 173) resentation for each word (i.e. similarity between words) along with (2) the probability function for word sequences, expressed with these repre(cid:175) sentations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar words forming an already seen sentence.
Exemplary analysis:
1. Relevance: This paper introduces the Transformer model, which is the foundation upon which Transformer-XL builds. The Transformer model revolutionized natural language processing (NLP) by moving away from recurrent and convolutional networks, focusing instead on attention mechanisms to process sequences of data. The query paper extends the Transformer model to handle longer contexts, which is direct expansion of the work introduced in this paper.
Reason for Citation: To acknowledge the foundational model (Transformer) which Transformer-X XL is based and to discuss the limitations of the original Transformer model that the query paper aims overcome.
2. Relevance: The introduction of relative position representations in self-attention mecha- nisms is a key innovation that allows Transformers to better understand the relationships between different parts of sequence. This concept is important for the -ransformer-XL, which seeks to improve the model's ability to handle long-term dependencies. Reason for Citation: To discuss advancements in self-attention mechanisms that are rele- vant to the development of Transformer- XLL, particularly the handling of position information in sequences, which is crucial for modeling longer contexts.
3. Relevance: This paper demonstrates the effectiveness of deep Transformer models for character-level language modeling, highlighting the potential of Transformers capture complex dependencies in text. It sets a precedent for using Transformers in language modeling, particularly at the character level, and discusses techniques to improve their performance.
To support the argument that deep Transformer models, like Transformer-XL, are effective at language modeling tasks and to build on the idea of enhancing Transformer architectures for better performance in NLP tasks.
Reason for Citation:
4. Relevance: BERT represents a significant leap forward in pre-training language representa- tions, using bidirectional Transformer. While BERT focuses on understanding language context in both directions, Transformer-Xi aims extend the context length that models can effectively process.
Reason for Citation: To highlight the importance of deep bidirectional Transformers in NLP and to position Transformer-XI. within the broader context of recent advancements in Trasseermer-based models. It may also cite BERT to discuss differences in approach, particularly regarding context length and model architecture.
22

---

5. Relevance: This paper explores adaptive input representations, which can make models more efficient and effective by adjusting the capacity input representations based on the complexity of the input. Such techniques are relevant for Transformer-Xi XI,, which seeks to improve efficiency and performance language modeling.
Reason for Citation: To discuss methods for improving the efficiency of neural language models, particularly the context of Transformer-base architectures. The query paper might leverage or build upon these adaptive techniques enhance Trasformer-XL's perfor- mance.
6. Relevance: This work foundational in the field of neural language modeling, introducing the concept of learning distributed representations for words alongside the probability function for word sequences. lays the groundwork for subsequent developments in language modeling, including the use of Transformers.
Reason for Citation: To acknowledge the historical context and evolution of language modeling techniques leading up to the development of Transformer and Transformer-XI. models. It may also cite this work to discuss the importance of distributed representations in understanding language.
Exemplary ranking: Ranked order: paper 1, paper 2, paper 3, paper 4, paper 5, paper 6
1. Explanation:
This paper is the cornerstone of Transformer models. Any research following Transformer-XL would likely reference this to acknowledge the foundational model and its limitations that the follow-up work seeks to address or build upon.
2. Explanation: The methodological relevance of improving self-attention mechanisms, es- pecially for handling longer contexts in Transformer models, makes this paper critical citation for discussing technical advancements or modifications follow-up study.
3. Explanation: This paper's focus on deep Transformer models for character-level language modeling aligns closely with the objectives of Transformer-Xe XL, making it a likely citation for discussions on model depth and granularity language modeling.
4. Explanation: Given the significant impact of BERT on the NLP field and its methodological similarities and differences with Transformer-2 XL, a follow-up study would likely cite it to discuss further advancements or comparisons Transformer-basee model architectures.
5. Explanation: Techniques for improving model efficiency and input representation crucial for advancing Transformer models. A follow-up study might cite this work to explore introduce new adaptive techniques for enhancing Transformer-XI. efficiency or performance.
6. Explanation: While foundational to the field neural language modeling, this paper might be cited less frequently in direct follow-up to Transformer-Xe except to provide historical context or discuss the evolution of language modeling techniques leading up to Transformer models.
23

---

A.7 Full case Study
The full texts of the representative case in the main text are as follows. In this case, the analyzer correctly reveals the relationships between the query paper and each candidate, and the ranker outputs a revised ranking, which increases the PREC@5 from 0.6 in the retrieval set to 0.8 in the final prediction. Notice that due to fluctuations in dataset quality, there may be few garbled characters.
Query paper:
Title: A DNA nanorobot functions as a cancer therapeutic in response to molecular trigger in vivo. Abstract: Nanoscale robots have potential as intelligent drug delivery systems that respond to molecular triggers. Using DNA origami we constructed an autonomous DNA robot programmed to transport payloads and present them specifically in tumors. Our nanorobot is functionalized on the outside with DNA aptamer that binds nucleolin, a protein specifically expressed on tumor- associated endothelial cells, and the blood coagulation protease thrombin within its inner cavity. The mucleolin-targeting aptamer serves both as a targeting domain and as a molecular trigger for the mechanical opening of the DNA nanorobot. The thrombin inside is thus exposed and activates coagulation at the tumor site. Using tumor-bearing mouse models, we demonstrate that intravenously injected DNA nanorobots deliver thrombin specifically to tumor-associated blood vessels and induce intravascular thrombosis, resulting in tumor necrosis and inhibition of tumor growth. The nanorobot proved safe and immunologically inert in mice and Bama miniature pigs. Our data show that DNA nanorobots represent promising strategy for precise drug delivery in cancer therapy.
Candidate papers:
1. (Core citation)
Title:
Universal computing by DNA origami robots in a living animal Abstract: Biological systems are collections of discrete molecular objects that move around and collide with each other. Cells carry out elaborate processes by precisely controlling these collisions, but developing artificial machines that can interface with and control such interactions remains a significant challenge. DNA is natural substrate for computing and has been used to implement a diverse set of mathematical problems, logic circuits and robotics. The molecule also interfaces naturally with living systems, and different forms of DNA-based biocomputing have already been demonstrated. Here, we show that DNA origami can be used to fabricate nanoscale robots that are capable of dynamically interacting with each other in a living animal. The interactions generate logical outputs, which are relayed to switch molecular payloads on or off. As proof of principle, we use the system to create architectures that emulate various logic gates (AND, OR, XOR, NAND, NOT, CNOT and half adder). Following ex vivo prototyping phase, we successfully used the DNA origami robots in living cockroaches (Blaberus discoidalis) to control a molecule that targets their cells.
2. (Non-citation)
Title:
In Situ SiRNA Assembly in Living Cells for Gene Therapy with MicroRNA Triggered Cascade Reactions Templated by Nucleic Acids.
Abstract: In Situ SiRNA Assembly in Living Cells for Gene Therapy with MicroRNA Triggered Cascade Reactions Templated by Nucleic Acids. The in situ generation of siRNAs in living cells can greatly enhance the specificity and efficiency of gene therapy. Inspired by the natural molecular machines that organize different compartments sequentially in a limited space to facilitate cellular process, this work constructs DNA nanomachine (DNM) by alternately hybridizing two pairs of DNA/RNA hybrids to DNA scaffold generated by rolling circle amplification for highly efficient in situ siRNA assembly in living cells. After target cell-specific delivery of DNM, intracellular specific microRNA can work as a trigger to operate the DNM by initiating DNA cascade displacement reaction between DNA/RNA hybrids along the scaffold for continuous generation of siRNAs. Using miR 21 as model, efficient siRNAs generation is achieved via DNA templated cascade reaction, which demonstrated impressive suppressions to VEGF mRNA and protein expressions in cells and in vivo tumor growth and indicated promising application of the designed strategy in gene therapy.
3. (Core citation) Title: Cellular Immunostimulation by SOGG-Sequence-Couue Coated DNA Origami Structures
24

---

To investigate the potential of DNA origami constructs as programmable and non- cytotoxic immunostimulants, we tested the immune responses induced by hollow 30-helix DNA origami tubes covered with up to 62 phoosine-hoosphae-eee -uannine (CpG) sequences in freshly isolated spleen cells. Unmethylated CpG sequences that are highly specific for hac- terial DNA are recognized by specialized receptor of the innate immune system localized in the endosome, the Toll-like receptor 9 (TLR9). When incubated with oligonucleotides containing CpGs, immune cells are stimulated through TLR9 to produce and secrete cy- tokine mediators such as interleukin-6 (IL-6) and interleukin- 12p70 (IL- 10),,,70), a process associated with the initiation of immune response. In our studies, the DNA origami tube built from an 8634 nt long variant of the commonly used stngle-stradded DNA origami scaffold M13mp18 and 227 staple oligonucleotides decorated with 62 CpG-containing oligomucleotides triggered a strong immune response, characterized by eyt...
Abstract:
4. (Core citation)
Title:
Challenges and opportunities for structural DNA nanotechnology
DNA molecules have been used build a variety of nanoscale structures and devices over the past 30 years, and potential applications have begun to emerge. But the development of more advanced structures and applications will require number of issues to be addressed, the most significant of which are the high cost of DNA and the high error rate of self-assembly. Here we examine the technical challenges the field of structural DNA nanotechnology and outline some of the promising applications that could be developed if these hurdles can be overcome. In particular, we highlight the potential use of DNA nanostructures in molecular and cellular biophysics, as biomimetic systems, in energy transfer and photonics, and in diagnostics and therapeutics for human health.
Abstract:
5. (Non-citation)
Title:
From cells to DNA materials
Materials need to be specially engineered to interface with cells; on the other hand, cells provide great inspiration for new material designs. Here, using examples mainly from our own research, we demonstrate that DNA can be used as both a genetic and generic material for various cell-related applications, including diagnostics, drug delivery, cell culture, protein production, and immuno-modulation We envision that other cell-based materials such as RNA, proteins, polysaccharides, and lipids can be more pervasively employed in materials science and engineering.
Abstract:
6. (Core citation)
Title: Molecularly self-assembled nucleic acid nanoparticles for targeted in vivo siRNA delivery
Abstract: DNA strands can self-assemble into tetrahedral nanoparticles that can deliver small interfering RNA molecules to cells and suppress genes in tumours.
7.
(Superficial-citation))
Title: Self-Assembled DNA Hydrogels with Designable Thermal and Enzymatic Respon- siveness
or used as a programmable template to direct the assembly of nanoparticles. [ 14-17 ] Recently, the concept of DNA assembly has been expanded to construct "DNA hydrogels", which are crosslinked networks swollen in an aqueous phase. [ [18-31] ] Though hydrogels have great potential in biological and medical applications. 32-36) such drug and gene delivery, biosensing, and tissue engineering, studying the preparation of DNA hydrogels with designable properties is still in its early stages. In the past, several methods have been reported to prepare DNA hydrogels, for example, DNA directly extracted from the nucleus nature, behaves like long linear polymer and forms hydrogel via physical entanglement or by chemical crosslinking of small molecules. [ 18-20] Similarly, DNA can be used negatively charged polymer and form a complex with cationic electrolesssssses through electrostatic interactions. 21 22 ] However, both methods treated DNA as polymer and did not take advantage of the self-assembly of DNA into ordered structures. therefore, the resulting hydrogels lacked precise structural control and specifi e responses. Instead of using physical interactions, DNA can covalently grafted onto synthetic poly- mers and serve as cross-linker, the recognition of complementary DNA strands leads to crosslinking of polymer chains and causes hydrogel formation. [ 23-28] general, the preparation of a DNA DNA-polymer hybrid requires laborious modifi cation steps, and easy
Abstract:
25

---

and fast strategy to build tailored DNA hydrogels is desired. Luo and his coworkers have developed a new approach to construct
8. (Non-citation)
Title: Aptamer-enabled efficient isolation of cancer cells from whole blood using a microflu- idic device.
Abstract: Circulating tumor cells (CTC) in the peripheral blood could provide impor- tant information for diagnosis of cancer metastasis and monitoring treatment progress. However, CTC are extremely rare in the bloodstream, making their detection and character- ization technically challenging. We report here the development of an sptamer-mediated micropillar-basee microfluidic device that is able to efficiently isolate tumor cells from unprocessed whole blood. High-affiniyy aptamers were used as an alternative to antibodies for cancer cell isolation. The microscope-slide-sii which enhanced the probability of the interactions between aptamers and target cancer cells. The device geometry and the flow rate were investigated and optimized by studying their effects on the isolation of target leukemia cells from a cell mixture. The device yielded a capture efficiency of 95% with purity of 81 % the optimum flow rate of 600 nL/s. Further, we exploited the device for isolating col...
device consists of >59000 micropillars,
Analysis:
1. We do not involve the top-1 paper among retrieved candidates into LLM agentic ranking,
2. We do not involve the top-2 paper among retrieved candidates into LLM agentic ranking,
3. Relevance: This paper explores the immunostimulatory potential of DNA origami structures coated with CpG sequences. The immune response triggered by these structures is relevant to understanding the immune interactions and responses induced by DNA-based nanorobots, such as the one described in the query paper.
Reason for Citation: To discuss the immunostimulatory properties of DNA origami struc- tures and potentially draw parallels insights into the immune responses elicited by the DNA nanorobot in the query paper.
4. Relevance:
This paper addresses the technical challenges in the field of structural DNA nanotechnology, highlighting the need overcome hurdles such high costs and self- assembly errors. Understanding these challenges is crucial for advancing the design and application of DNA-based nanorobots.
Reason for Citation: To acknowledge the broader challenges and opportunities in DNA nanotechnology. which can inform the development and optimization of DNA nanorobots for targeted drug delivery in cancer therapy.
5. Relevance:
This work explores the use of DNA as a versatile material for various cell-related applications, including diagnostics, drug delivery, and mumuno-modulation Understanding the diverse applications of DNA materials can inspire innovative approaches to designing DNA nanorobots for targeted drug delivery.
Reason for Citation: To draw insights from the use of DNA in cell-related applications and potentially leverage the principles or techniques discussed in this paper for designing DNA nanorobots with specific functionalities for cancer therapy.
6. Relevance:
This paper focuses on the self-assembly of nucleic acid nanoparticles for targeted delivery of small interfering RNA (siRNA) molecules cells. The concept of targeted delivery using nucleic acid nanoparticles aligns with the targeted drug delivery approach of the DNA nanorobot described in the query paper
Reason for Citation: To explore strategies for targeted delivery of therapeutic molecules using nucleic acid nanoparticles, which can inform the design and delivery mechanisms of the DNA nanorobot in the query paper.
7. Relevance:
This paper discusses the design and properties of self-assembled DNA hydrogels with specific responsiveness to thermal and enzymatic stimuli. Understanding the principles of self- assemblyy and responsiveness in DNA structures can provide insights into designing functional DNA nanorobots with controlled release mechanisms.
Reason for Citation: To explore the concept of designable DNA structures with specific responsiveness, which can inspire the development of DNA nanorobots capable of controlled and targeted drug delivery in cancer therapy.
26

---

8. Relevance:
This work focuses the efficient isolation of cancer cells from whole blood using aptamer- mediated microfluidic devices. The use of aptamers for specific cell isolation aligns with the targeting strategy of the DNA nanorobot in the query paper, which utilizes DNA aptamer for tumor-specific binding- Reason for Citation: To explore innovative approaches for targeted cell isolation using aptamers, which can inform the targeting and delivery mechanisms of the DNA nanorobot designed for cancer therapy.
Ranking: Ranked order: paper 1, paper 2, paper 6, paper 3, paper 4, paper 7, paper 5, paper
1. We do not involve the top-1 paper among retrieved candidates into LLM agentic ranking.
2. We do not involve the top-2 paper among retrieved candidates into LLM agentic ranking.
3. Explanation: Understanding the immunostimulatory properties of DNA origami structures is essential for comprehending the immune responses induced by DNA-based nanorobots, providing insights into potential immune interactions and responses triggered by the nanorobot described in the query paper.
4. Explanation: While addressing the technical challenges in structural DNA nanotechnology is important for advancing the design and application of DNA-based nanorobots, this paper may be cited less frequently in a direct follow-up to the query paper, except to acknowledge broader challenges and opportunities DNA nanotechnology.
5. Explanation: Exploring the diverse applications of DNA materials in cell-related contexts can inspire innovative approaches to designing DNA nanorobots for targeted drug delivery, potentially offering new perspectives on how DNA-based nanorobots can be optimized for specific functionalities in cancer therapy.
6. Explanation: The focus on targeted delivery using nucleic acid nanoparticles resonates with the targeted drug delivery approach of the DNA nanorobot described in the query paper, making this paper highly relevant for understanding and improving targeted drug delivery mechanisms.
7. Explanation: Understanding the design and responsiveness of self-assembled DNA struc- tures can provide valuable insights into developing functional DNA nanorobots with con- trolled release mechanisms, enhancing the precision and efficacy of targeted drug delivery in cancer therapy.
8. Explanation: This paper directly aligns with the targeting strategy of the DNA nanorobot in the query paper, emphasizing the importance of aptamers for specific cell targeting and isolation, which crucial for the success of targeted drug delivery systems.
27

---

A.8
Discussions
A.8.1
Limitations
One major limitation of our method lies LLMs' illusion problem. Despite average performance improvement, LLMs may output unfaithful analysis under certain circumstances and poison specific samples. When researchers want high-likelihood citation suggestions in preparing manuseripts, these samples may cause confusion. Therefore, how to verify the output of LLM agents and improve reliability of our hybrid workflow worth future studies, Also, the curriculum finetuning process requires certain amount of computational resources. Therefore, how to lighten the computational load worth further investigation.
A.8.2 Code of Ethics
We fully use open-source models and datasets in the paper, which involve problem regarding privacy and copyright. We cite resources Section Section Section and Section Se.tion[4.3.4 Moreover, training and testing data randomly sampled from publications all around the world, which does not involve problems of bias and discrimination.
A.8.3 Broader Impacts
Our method has positive broader impacts. On the one hand, accurate citation prediction can help reveal information hiding in link space of citation networks, owning value in aiding citation-bar based computational social science studies. These studies may investigate the patterns of paper publication and scientific innovation, enlightening researchers with efficient research approaches and putting forward the advancement of modern science. On the other hand, the application of our hybrid workflow is not limited to the task of citation prediction. A wide range of natural language processing tasks may borrow experience from our work and improve their performance.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We summarized the paper's main contributions in the abstract and introduction. See Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of the work in Section 6 and Appendix A.8.I
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
We do not include theoretical proof in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided all implementation details for reproducibility Section 4.3 and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
We also made our source code available in an anonymous repository at httzps://anouymous. 4open scwence/r/H-L--77000
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
public/guides ottp://s/eoooeeeebeessssooooooo.
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
wetgguddddeeouuubbuneeee ssinoolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided all the training and test details in Section 4.3 and Appendix A.2 We also included all details the source code, which is anonymously available at https // //anonymouu 4open.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We reported 95% CI error bars in Figure I and Figure 4 We reported p-values in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported computer resources information in Section 4.3 and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We conformed the NeurIPS Code of Ethics. We discuss the following of code of ethics in Appendix A.8.2
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work has various potential positive broader impacts, We discussed the potential societal impacts in Section I and Appenix A.8.3
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited provided URLs for the original paper that produced the code,
dataset, and models in Section 3.2 Section 4.1 Section 4.2 and Section 4.5.4
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We made our source code available in an anonymous repository at https //anonymous wcwence/r/H-LM...
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not include crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not involve crowdsourcing nor research with human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "hdgcUJmFqA", "reviewer": "Reviewer_tEdQ", "summary": "This paper studies text-based citation prediction by exploring the varying roles of paper citations from foundational to superficial. The authors introduce the concept of core citations, emphasizing the most important citations over superficial ones. Then, they propose HLM-Cite, a hybrid language model workflow combining embedding and generative language models. The approach involves a two-stage pipeline: first, a pre-trained embedding model coarsely retrieves high-likelihood core citations, and then a large language model ranks these candidates through reasoning. Experiments on the Microsoft Academic Graph across 19 scientific fields demonstrate the effectiveness of HLM-Cite in comparison with scientific pre-trained language models, text-rich network representation learning methods, and general-domain language models for text embedding.", "questions": "- Could you compare HLM-Cite with SPECTER 2.0 and SciMult?\n\n- Could you report the results on the SciDocs benchmark dataset (or explain the reasons why you did not use it)?\n\n- Could you conduct a statistical significance test (e.g., two-tailed t-test) to compare your model with the strongest baseline/ablation version in Tables 2 and 3?", "limitations": "The authors discuss the limitations and potential negative societal impact of their work in the Checklist. I suggest they further consider the limitations mentioned in my review above.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed concepts of core citations and superficial citations are intuitive.\n\n+ The adopted retrieval-reranking paradigm is well-motivated, combining the strengths of pre-trained embedding models and large language models. The usage of curriculum finetuning is also reasonable in the retrieval module.\n\n+ Besides performance comparisons, the authors conduct various additional experiments to validate their motivation and design choices, including ablation studies, hyperparameter analyses, and the effect of examples/LLMs.", "weaknesses": "- Some scientific language model baselines are missing. To be specific, SPECTER 2.0 [1] and SciMult [2] have shown superior performance over SciNCL on various tasks including citation prediction, but neither of them is compared in the experiments.\n\n- The dataset used in this paper is constructed by the authors. It is unclear whether the proposed model works for other datasets. For example, the SciDocs dataset from the SPECTER paper [3] is a widely used benchmark dataset for citation prediction, but it is not considered in the experiments. Also, it is unclear why the authors cluster the 19 scientific fields into 2 big areas (i.e., natural science and social science). Would it be possible to directly show the results in each field (e.g., like [4])?\n\n- Statistical significance tests are not conducted. It is not clear whether the gaps between HLM-Cite and the baselines/ablation versions are statistically significant or not. In fact, in Tables 2 and 3, some gaps are subtle, therefore multiple runs are needed and the p-value should be reported.\n\n[1] SciRepEval: A Multi-Format Benchmark for Scientific Document Representations. EMNLP 2023.\n\n[2] Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding. Findings of EMNLP 2023.\n\n[3] SPECTER: Document-level Representation Learning using Citation-informed Transformers. ACL 2020.\n\n[4] The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study. WWW 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer tEdQ\n**Q1.** *Missing baselines, e.g., SPECTER 2.0 and SciMult.*\n\n**Response:** Thank you for providing these up-to-date baselines for scientific texts. We test SPECTER 2.0 [1] and SciMult (both vanilla and MoE) [2] on our tasks as suggested. Also, we investigate new baselines for general text embedding to compensate for the existing ones. We include GTE-v1.5, the upgraded version of GTE [3], where the latter one is our previous best baseline. We find the new baselines surpass our existing ones as expected, and still, our method outperforms all baselines. These new baselines provide stronger evidence of the advantage of our method, and they also indicate the possibility of updating the GTE model, which we used for initialization, to stronger pretrained LMs to further improve the performance of our method.\n\nHere, we show the overall performance for your convenience. Please refer to \"result.pdf\" in the global response for details.\n|Model|PREC@3|PREC@5|NDCG@3|NDCG@5|\n|:-:|:-:|:-:|:-:|:-:|\n|SPECTER|0.545|0.460|0.570|0.504|\n|SciMult-vanilla|0.569|0.485|0.593|0.529|\n|SciMult-MoE|0.579|0.496|0.603|0.539|\n|SPECTER 2.0|0.602|0.515|0.627|0.560|\n|GTE-base|0.639|0.556|0.659|0.597|\n|GTE-base-v1.5|0.641|0.557|0.662|0.599|\n|GTE-large-v1.5|0.649|0.563|0.671|0.606|\n|HLM-Cite (GPT-3.5)|0.725|0.644|0.735|0.677|\n|**HLM-Cite (GPT-4o)**|**0.736**|**0.655**|**0.743**|**0.686**|\n\n**Q2.** *SciDocs dataset.*\n\n**Response:** Thanks for mentioning SciDocs, which is a widely applied benchmark dataset in scientific text embedding. The major obstacle to testing our method on SciDocs is the lack of ground truth label of core/superficial citation. As reported in the SPECTER paper [4], SciDocs provides approximately 30K papers in total for the citation-prediction task, including 1K queries with a candidate set of 5 cited and 25 uncited papers for each query. However, the cited papers are regarded equally without more detailed information.\n\nIn our design, the core citations are identified on the citation network based on subsequent papers citing them. Regretfully, the 30K papers in SciDocs alone are not enough to build a citation network for identifying core citations, beause the citations among them are too sparse. To solve this dilemma, we built a citation network with hundreds of millions of research papers in Microsoft Academic Graph (MAG), identifying core/superficial citations in it. Since there is no direct mapping that bridges SciDocs and MAG, we cannot locate the raw 30K papers in MAG, and thus, we directly sample new queries and candidates from MAG for testing (Section 4.1). We compare our testing dataset and SciDocs from the following aspects:\n- **Data scale.** Our testing dataset is much larger in scale than SciDocs. We include 50K queries with a set of 10 cited (5 core and 5 superficial) papers for each query.\n- **Field coverage.** Like SciDocs, our testing dataset covers multiple scientific fields. We randomly sampled papers in MAG, where the sampled papers covered all 19 fields in MAG.\n- **Testing format.** Our testing uses the format of query-candidates, which is the same as the citation-prediction task in SciDocs.\n\nConsidering these, we believe that the testing results on the new dataset we constructed are convincing enough to compensate for SciDocs' absence.\n\nStill, we agree that it is meaningful to use SciDocs to test how various methods perform on the core-citation prediction task we proposed. To enable this in the future, we will work on adding labels of core/superficial to the cited papers provided in SciDocs. We will locate SciDocs's 30K papers in MAG by comprehensively matching the titles, authors, and keywords. Therefore, we can label the citations in SciDocs as core/superficial, providing our task with a more widely acknowledged dataset, as well as contributing an auxiliary applicable scenario to SciDocs.\n\n**Q3.** *Results in each field.* **& Q4.** *Statistical significance.*\n\n**Response:** Thanks for these suggestions. Like [5], we show the detailed performance in each field. Also, we conduct statistical significance tests to better compare our model with the strongest baseline/ablation version. We used the two-tailed t-test between the performance of (1) our model VS strongest baseline and (2) full design VS ablation versions. Please refer to \"result.pdf\" in the global response for details.\n\nThe results provide us with a deeper understanding of the performance and validity of our designs, illustrating that:\n- **Overall performance (our model VS strongest baseline, Table 2 in paper).** Our method surpasses the top baselines in all fields ($p<0.01$ in most individual fields, and $p<0.001$ averaging all fields through t-test), demonstrating its general applicability to a wide range of fields.\n- **Ablation studies (full design VS ablation versions, Table 3 in paper).** Generally, all parts of our designs are valid with significance ($p<0.01$ or $p<0.1$ in overall performance through t-test). Moreover, we notice that when focusing on social science papers, which only comprise a small proportion of all papers, Stage 1 of curriculum finetuning is only slightly beneficial. Therefore, when only applying to social science papers, it is an alternative for users to skip Stage 1 if they want to save computational cost with the cost of a slight performance drop. In contrast, when applying to natural science papers, it is necessary to keep Stage 1 for better performance.\n \n[1] SciRepEval: A Multi-Format Benchmark for Scientific Document Representations. *EMNLP 2023*.\\\n[2] Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding. *Findings of EMNLP 2023*.\\\n[3] Alibaba-NLP/gte-large-en-v1.5. *Hugging Face 2024*.\\\n[4] SPECTER: Document-level Representation Learning using Citation-informed Transformers. *ACL 2020*.\\\n[5] The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study. *WWW 2023*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qdwPFhOBXO", "reviewer": "Reviewer_3KQH", "summary": "The authors introduce the concept of core citations to distinguish important citations from superficial ones and non-citations. This shifts the citation prediction task from simple binary classification to the more subtle approach of identifying core citations. They then propose HLM-Cite, a hybrid language model workflow that combines embedding and generative language models. This two-stage pipeline first retrieves high-likelihood core citations from a vast set of candidates using a pretrained text embedding model and then ranks them using a logical reasoning process conducted by LLMs. With the two-stage pipeline, the authors show that they can scale the candidate sets to 100K papers, thousands of times larger than existing works. They evaluate HLM-Cite on a dataset across 19 scientific fields, demonstrating a 17.6% performance improvement comparing SOTA methods.", "questions": "Given that keyword overlap seems to be highly predictive of whether a paper is a core citation or not, I would have liked to see some comparisons to simpler baselines than what the authors ultimately showed. Also, is text analysis even necessary, or can network science be used here, as in so much of science of science research?", "limitations": "The authors consider a technical limitation, namely \"LLMs' illusion problem.\" LLMs may output unfaithful analysis under certain circumstances and poison specific samples. When researchers want high-likelihood citation suggestions in preparing manuscripts, these\n samples may cause confusion. The authors note that verifying the output of LLM agents and improving the reliability of their hybrid workflow could be pursued in the future. Also, the curriculum finetuning process requires a certain amount of computational resources. Lightening this load also merits further investigation. There are no significant social/ethical limitations of this work, including negative societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The problem is an interesting one, and right in the beginning of the paper, the authors show that the idea of core citations is an important one by empirically studying the Microsoft Academic Graph. \n\nThe paper makes a substantive contribution on the problem. The authors employ the GTE-base pretrain model [15], one of the top models on the Massive Text Embedding Benchmark (MTEB) leaderboard. Additionally, they have a curriculum finetuning module. To complete the architecture, the LLM Agentic Ranking Module aims to improve the accuracy of core citation prediction by incorporating LLMs’ textual reasoning capability to rectify the ranking of papers retrieved in the previous stage by core-citation likelihood.\n\nExperimentally, the authors are able to scale the candidate sets to 100K papers, thousands of times larger than existing works. The two-stage approach (called HLM-Cite) is evaluated on a dataset across 19 scientific fields, demonstrating a 17.6% performance improvement comparing SOTA methods. \n\nThe method could have positive broader impacts. Accurate citation prediction can help reveal information hiding in link space of citation networks, owning value in aiding citation-based computational social science studies. These studies may investigate the patterns of paper publication and scientific innovation, enlightening researchers with efficient research approaches and putting forward the advancement of modern science.", "weaknesses": "My main comment is that the terminology in Section 2.1 is overly dense. There is way too much use of superscripts and subscripts for relatively simple concepts. If possible, the authors should try to make the symbols simpler. \n\nOn the technical front, I still feel that the approach the authors are taking might be overly complex, and I'm unsure what impact it will have on actual computational science practice. I'm also wondering whether a simpler approach based on a core-periphery algorithm from network science, rather than text analysis, might not be useful for identifying core citations in existing papers. Obviously, this is less useful for prediction, but I am not sure there is any impact here in prediction; people writing papers already know what the core citations in their field are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer 3KQH\n**Q1.** *Overly dense terminology in Section 2.1.*\n\n**Response:** Thanks for pointing out this. In this paper, we follow previous computational social science [1,2] studies and define the core citations of a query paper from the citation network. We apologize for making the formulas too complicated when trying to express the intuitive definition mathematically. Here, we rephrase the terminology:\n\n*For a query paper $q$ and its citations $S_q$, if there exists a subsequent paper that cites $q$ and one of $q$'s citations $s_q\\in S_q$ at the same time, then $s_q$ is among the core citations of $q$. Therefore, $q$'s core citations can be expressed as:*\n$$\n\\tilde{S}_q=\\\\{s_q\\in S_q\\mid\\exists p, q\\in S_p, s_q\\in S_p\\\\}.\n$$\nHopefully, this simplified version will be easier for readers to understand.\n\n**Q2.** *Necessity and impact of text-based (rather than network science-based) core citation prediction.*\n\n**Response:** Thank you for this question. You are right that network science is helpful for identifying core citations of existing papers. As mentioned in Q1, we did utilize knowledge in network science and science of science to define the core citations given query paper $q$ based on subsequent papers that cite $q$ in the citation network. Then, we classify citations in the Microsoft Academic Graph (MAG) as core/superficial. However, this paper did not focus on analyzing these classifications of existing citations. Instead, we treat the network-based classifications as labels and train a text-based model, which can predict the core citations of $q$ without knowing subsequent paper that cites $q$.\n\nHere are two representative scenarios where the subsequent paper that cites $q$ is unknown for the given query $q$, illustrating the meaningful applications of our text-based model:\n- **Paper recommendation (for ongoing papers).** Our model can recommend suitable citations to researchers who are writing papers. You are right that researchers should know what the core citations in their field are. However, the rapid development of modern science brings about exponential growth in publications, where researchers may find an overwhelming number of publications that match their interests but are largely irrelevant to their needs in writing the specific paper [4]. The so-called \"burden of knowledge\" [2,3] makes manual identification of papers to read or cite challenging, relying on a researcher’s ability. Therefore, a lot of works have been done on paper recommendation systems [4,5]. Our model can serve as such a system, where users input their draft of the abstract as the query and set a group of possibly related papers as the candidates. Then, our model can help identify papers that are most suitable to cite in the ongoing paper. In this scenario, it is impossible to identify the core citations of the ongoing paper based on subsequent papers citing it.\n- **Science of science study (for existing papers).** As you mentioned, there exists abundant science of science researches that utilize citation networks to analyze published papers [1,2]. However, recent studies have recognized that the subsequent citations of a paper require three to five years after publication to accumulate, and a proportion of papers, named \"sleeping beauties\", may experience a burst of attention after up to ten years [6-8]. Several famous studies, e.g., measuring disruption, have reported the impact of delayed citation on their analyses [1]. Therefore, although identifying core citations based merely on the citation network is feasible for old papers that have accumulated sufficient subsequent citations, it is not applicable for the latest papers that have not yet been cited. In contrast, our text-based method can identify the core citations of a newly published paper without relying on subsequent papers citing it. This allows us to analyze the latest science using the idea of core citation, which is meaningful for understanding cutting-edge trends in scientific research.\n\nHopefully, these interpretations can better illustrate the potential impact and applications of our study.\n\n**Q3.** *Simpler baseline that directly utilizes keyword overlap.*\n\n**Response:** Thanks for proposing this simplified approach. Here, we test using the overlap ratio between candidate keywords and query keywords to predict the likelihood of the candidate becoming core citations.\n\n||PREC@3|PREC@5|NDCG@3|NDCG@5|\n|:-:|:-:|:-:|:-:|:-:|\n|PATTON|0.253|0.205|0.271|0.234|\n|Keywords Overlap|0.336|0.269|0.304|0.264|\n|SPECTER|0.454|0.460|0.570|0.504|\n|H-LM (GPT3.5)|0.725|0.644|0.735|0.677|\n|**H-LM (GPT4o)**|**0.736**|**0.655**|**0.743**|**0.686**|\n\nThe results prove that the keywords overlap ratio is truly predictive for core citations, achieving performance similar to medium baselines. However, its performance is still far behind that of advanced text-based methods, including our design. This illustrates the advantage and necessity of predicting core citations based on the texts rather than only considering the keywords.\n\n\n[1] Large teams develop and small teams disrupt science and technology. *Nature 2019*.\\\n[2] Papers and patents are becoming less disruptive over time. *Nature 2023*.\\\n[3] The burden of knowledge and the “death of the renaissance man”: Is innovation getting harder? *The Review of Economic Studies 2009*.\\\n[4] Scientific Paper Recommendation: A Survey. *IEEE Access 2019*.\\\n[5] Scientific paper recommendation systems: a literature review of recent publications. *International Journal on Digital Libraries 2022*.\\\n[6] Defining and identifying Sleeping Beauties in science. *PNAS 2015*.\\\n[7] Modeling citation dynamics of “atypical” articles. *Journal of the Association for Information Science and Technology 2018*.\\\n[8] New directions in science emerge from disconnection and discord. *Journal of Informetrics 2022*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JmLet7f0e0", "reviewer": "Reviewer_b1Uz", "summary": "The paper proposes a framework, HLM-Cite (Hybrid Language Model) for scientific citation prediction based on incorporating generative language models embeddings and LLMs as agents. The pretrained text embeddings are used to retrieve high likelihood core citations (a term the paper introduces to define as more meaningful citations, compared to superficial or non-citations). Three different LLM agents are then used to rank the papers by reasoning over them.", "questions": "- Since you are using MAG to finetune the various models, how are you annotating the entries in the graph with the three citation classes you proposed: \"core\", \"superficial\" and \"non-citations\", especially given the size of this knowledge graph?\n- In Section 3.3.2, it is mentioned that the analyses are manually reviewed and revised, to make sure that they correctly reveal the expected behavior (lines 193-195). How many papers is this done for? The entire set of papers? \n- In Section 4.1, it is mentioned that 5 core citations and 5 superficial citations are sampled for each of the 450k queries. Similarly to the first question - how are these citations initially annotated with one of the \"core\" vs \"superficial\" classes?\n- In several places in the paper (e.g. Section 4.5)  you mention \"Without loss of statistical significance\". How was this computed?", "limitations": "Authors mention LLM limitations, such as hallucinations, in their conclusion.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- This is an interesting use of language models, both smaller in scale and LLMs, for a task relevant to researchers. In particular, this type of work is very relevant and could benefit the open science community. \n- The system created is well designed, the choices are justified (e.g. the three agents)\n- The experiments are extensive, and multiple models are compared, which grounds the analyses scientifically\n- The results are strong and performance surpasses other models, showing that the proposed framework is indeed well designed\n- The paper is clearly written and the work is well presented", "weaknesses": "- The idea of a \"core\" citation is not necessarily novel, as citation classification systems have been around for a while. While there isn't one classification system the scientific community agrees upon, there have been multiple proposed. The paper should acknowledge this body of work around citation classification systems, especially if proposing a new one: \"core\", \"superficial\" and \"non-citations\" - which is essentially a binary classification on the citation type (\"core\" vs \"superficial\").  Some references: \n\n  - [1] Cohan, Arman, et al. \"Structural scaffolds for citation intent classification in scientific publications.\" arXiv preprint arXiv:1904.01608 (2019).\n\n  - [2] Garfield, Eugene. \"\" Science Citation Index\"—A New Dimension in Indexing: This unique approach underlies versatile bibliographic systems for communicating and evaluating information.\" Science 144.3619 (1964): 649-654.\n\n  - [3] Jurgens, David, et al. \"Measuring the evolution of a scientific field through citation frames.\" Transactions of the Association for Computational Linguistics 6 (2018): 391-406.\n\n  - [4] Moravcsik, Michael J. \"Citation context classification of a citation classic concerning citation context classification.\" Social Studies of Science 18.3 (1988): 515-521.\n\n  - [5] Nicholson, Josh M., et al. \"scite: A smart citation index that displays the context of citations and classifies their intent using deep learning.\" Quantitative Science Studies 2.3 (2021): 882-898.\n\n  - [6] Teufel, Simone, Advaith Siddharthan, and Dan Tidhar. \"Automatic classification of citation function.\" Proceedings of the 2006 conference on empirical methods in natural language processing. 2006.\n\n- While the paper provides an interesting and well designed system, it is definitely geared towards applications of LLMs and models, rather than novel scientific contributions. Might be more appropriate for a specialized workshop. This is why my recommendation is borderline.\n\n- Minor notes: \n\n  - Lines 13-15 are a bit hard to understand, could be rephrased e.g. [...an LLM to predict citations, which leads to results ..]\n\n  - Line 88 -> notations instead of notifications?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer b1Uz\n**Q1.** *Novelty of the core citation idea.*\n\n**Response:** Thanks for the literature. We agree that citation classification is not new. Existing works classified citations with traditional ML [2,3,5] and DL [1,4] according to the roles in the context (background, method, etc.) [1,2] and the author's attitude towards the citations (supporting, contrasting, etc.) [3-5]. However, they all focused on the semantics of each individual paper, which requires manual annotation, limiting the data size to thousands and consuming up to years of manual work [4].\n\nIn contrast, we defined core citations on the citation network among vast papers. For a query paper $q$ and its citations $S_q$, if there exists a subsequent paper that cites $q$ and one of $q$'s citations $s_q\\in S_q$ at the same time, then we regard $s_q$ as a core citation of $q$ (Section 2.1). Instead of subjective semantic labels by manual work, our design objectively reflects the collective behavior of the scientific community, i.e., the subsequent citations of $q$ indicate the degree of recognition of $q$ and its citations $S_q$ by the community.\n\nThus, we classified citations from a novel perspective. Also, because our classification criteria are objective, we require no subjective manual annotations and can label large datasets by automatic structural analysis on the citation network. After obtaining the labels from the citation network, we trained our model to predict them purely based on the texts. Hopefully, we have clarified the novelty of our core citation idea compared to existing works.\n\n**Q2.** *Annotation of core and superficial citations.*\n\n**Response:** Yes, we labeled citations in the entire MAG automatically and required no manual annotations (See Q1).\n\n**Q3.** *Scientific contributions and compliance with NeurIPS's scope.*\n\n**Response:** Thanks for recognizing our contribution to the application.\nHere, we want to clarify that our work is not only applications of LLMs and models but includes novel scientific contributions:\n- **Definition and identification of core citation.** AS we mentioned in Q1, our definition of core citation classifies citations from a novel aspect, which exhibits statistical rationality. Instead of massive human work, we can label large-scale datasets by automatic network structure analysis based on our idea.\n- **Hybrid workflow combining small and large language models.** More importantly than the finetuning and prompting details, we proposed the hybrid framework incorporating small and large LMs. Small models are computationally efficient but lack reasoning capability, only capturing textual similarities, while large models have text reasoning ability but are computationally expensive and limited in context length. Our framework combines the advantages of both models, ensuring computational efficiency in large-scale tasks and maintaining the reasoning ability beyond simple textual similarities. \n\nBesides, as mentioned in the NeurIPS call for papers, the conference's topics include:\n- Applications (e.g., language, etc.)\n- Machine learning for sciences (e.g., social sciences, etc.)\n\nIn this paper, we first proposed a novel and meaningful NLP task in social sciences, and then designed an applicable ML workflow that reached good performance on the proposed task, which we believe is within the scope of NeurIPS.\n\nAlso, we notice that a number of papers with similar scopes and contributions have been published in NeurIPS. For example:\n- [6] designed a communicative LLM agent framework via prompting, which can generate conversational data for studying.\n- [7] designed two failure modes to guide the jailbreak of LLMs, which enlightened the safe training of LLMs.\n\nConsidering these, we believe that our paper is suitable for NeurIPS, and hopefully, our interpretations can better illustrate the novelty and value of our work.\n\n**Q4.** *Details about manual review and revision.*\n\n**Response:** Thanks for the question. We only manually reviewed and revised one query paper (the selected Transformer-XL paper) and its several candidates. We used the manually reviewed and revised texts as the one-shot example for the analyzer and decider in processing other papers in the dataset, where no more manual work is needed. Please refer to Appendix A.6 for the full texts we manually reviewed and revised.\n\n**Q5.** *Statistical significance.*\n\n**Response:** Thanks for the question. We used the two-tailed t-test for statistical significance of the performance differences between different models. In detail, we tested in:\n- **Overall performance.** Ours VS the strongest baseline: $p<0.001$ averaging all fields.\n- **Ablation studies.** The full design VS different ablation versions: $p<0.01$ or $p<0.1$ averaging all fields.\n- **Other analyses.** One-shot VS few-shot, GPT models VS other models. Here, we only tested 10\\% of the testing set to reduce computational consumption, but small enough $p$ indicates that 10\\% samples can reflect the performance differences without loss of statistical significance.\n\n**Q6.** *Minor typos.*\n\n**Response:** Thanks. We will check and correct these.\n\n[1] Structural scaffolds for citation intent classification in scientific publications. *NAALC 2019*.\\\n[2] Measuring the evolution of a scientific field through citation frames. *Transactions of the Association for Computational Linguistics 2018*.\\\n[3] Citation context classification of a citation classic concerning citation context classification.*Social Studies of Science 1988*.\\\n[4] Scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. *Quantitative Science Studies 2021*.\\\n[5] Automatic classification of citation function. *EMNLP 2006*.\\\n[6] CAMEL: Communicative Agents for \"Mind\" Exploration of Large Language Model Society. *NeurIPS 2023*.\\\n[7] Jailbroken: How Does LLM Safety Training Fail? *NeurIPS 2023*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cwHr0PMQj3", "reviewer": "Reviewer_7Ub3", "summary": "This paper proposes a Hybrid Language Model workflow for citation prediction, where core citations are predicted from superficial citations and non-citations rather than using a simple binary classification approach. This method can handle candidate sets of up to 100K papers and demonstrates better performance compared to previous methods.", "questions": "1. How is the number of unchanged candidates in the decider stage selected?\n2. The paper uses a text-based method in the first stage to retrieve candidates. How does this compare to previous binary citation prediction methods? For example, what if binary citation prediction is used to retrieve citations first, then the LLM agentic ranking module is used to further distinguish between core citations and superficial citations?\n3. How are the numbers of tq1 and tq2 selected? Would this affect performance? Additionally, since superficial citations should be more than core citations in real cases, why are tq1 and tq2 kept equal?", "limitations": "The paper discusses limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper expands the citation prediction task from binary citation prediction to distinguishing core citations, superficial citations, and non-citations. This distinction is important as core citations form the main foundation of a paper. Core citations are defined through a citation network.\n2. The method uses two modules: a retrieval module and an LLM agentic ranking module. This design enables the method to handle large candidate sets efficiently.\n3. The paper employs a one-shot example as a guide and compares the effectiveness of one-shot and few-shot methods. The one-shot example of the Transformer is notable.\n4. The method is tested on real-world data through various experiments, demonstrating good performance.", "weaknesses": "1. The paper mentions in the appendix that they keep the top candidate unchanged and only rank the remaining candidates in the decider. This should also be mentioned in the main text to avoid confusion, especially in Figure 3, regarding the omission of the top 3 candidates.\n2. In the ablation study of the curriculum stage, the performance without Stage 1 is very close to the full curriculum, especially in Social Science. Moreover, the Prec@3 of Social Science is actually better without the full curriculum. This suggests that Stage 1 might not be that effective, especially considering its computational cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer 7Ub3\n**Q1.** *Design of unchanged candidates in the LLM decider.*\n\n**Response:** Thank you for carefully reading through the appendix. We apologize for not clearly explaining the the LLM decider's working process. Here, we explain the detailed designs.\n\nWith the retrieval size of $r_q$ and $t^1_q$ core citations in the candidate set, the decider needs to select $t^1_q$ candidates most likely to be core citations from the $r_q$ retrieved candidates. One approach is to directly reranking the $r_q$ retrieved candidates and select the top $t^1_q$. However, there are two shortcomings: (1) reranking all the $r_q$ retrieved candidates leads to long input context for the LLM decider, harming the performance; (2) directly reranking the $r_q$ retrieved candidates does not utilize any information from the embedding model. In other words, the $r_q$ retrieved candidates have different inner products by the embedding model, which are informative as the prediction of their core citation likelihood in the retrieval module. \n\nWe utilized the likelihood prediction in the retrieval module, regarding the $(2t^1_q-r_q)$ retrieved candidates with largest inner products as safe core citations and exempting them from reranking. Then we reranked the remaining $2(r_q-t^1_q)$ retrieved candidates with the LLM and selected the top $(r_q-t^1_q)$ ones, resulting in $t^1_q$ selected candidates in total.\n\nFor example, we used $t^1_q=5$ and $r_q=8$ in our main results. Here, the number of fixed top candidates is 2, and we reranked the remaining 6 retrieved candidates and selected the top 3. Also, we kept $t^1_q=5$ and tested $r_q=10$ in Section 4.5.2. Here, the number of fixed top candidates is 0, and we reranked all 10 retrieved candidates with the LLM agent and selected the top 5.\n\nWe will add these details in the main text of our paper. To avoid confusion, we will also modify Figure 3 accordingly, highlighting the unchanged top candidates and the reranking of the remaining ones.\n\n**Q2.** *Ablation study of the curriculum stage 1.*\n\n**Response:** Thanks for pointing out this. It is true that the performance without Stage 1 is very close to the full curriculum, especially in social science. To examine this, we use t-test to compare the full design with the ablation versions. Please refer to \"result.pdf\" in the global response for detailed results.\n\nThe results give us a deeper understanding of the role of each part in our designs. Generally, all parts of our designs are valid with significance ($p<0.01$ or $p<0.1$ in overall performance through t-test). Moreover, when focusing on social science papers, which only comprise a small proportion of all papers, Stage 1 of curriculum finetuning is only slightly beneficial. Therefore, when only applying to social science papers, it is an alternative for users to skip Stage 1 if they want to save computational cost with the cost of a slight performance drop.\nIn contrast, when applying to natural science papers, it is necessary to keep Stage 1 for better performance.\n\n**Q3.** *Comparison to previous binary citation prediction methods.*\n\n**Response:** Thanks for this question. Actually, the baseline embedding models we compared are binary citation prediction methods. In the retrieval stage, either the baselines or our models encode the query and candidates into vectors and then calculate each candidate's score via inner products. The only difference is that the baseline methods have not been trained to distinguish core/superficial citations. Therefore, they only try to ensure the score as cited papers>non-citations (binary citation prediction). In contrast, our model learned to distinguish citations and non-citations in the curriculum Stage 1 and further learned to distinguish core/superficial citations in the curriculum Stage 2. Therefore, our model can try to ensure the score as core citations>superficial citations>non-citations.\n\nAlso, the ablation study of curriculum Stage 2 is the case that our model only learned binary citation prediction. The significant performance drop indicates the limitation of merely using binary citation prediction model in our core citation prediction task.\n\nWe will add the explanations to the revised version of our paper. Hopefully, our interpretations can help readers understand the relationships and differences between our model and the binary citation prediction methods.\n\n**Q4.** *Details of the numbers of $t_q^1$ and $t_q^2$.*\n\n**Response:** Thanks for this question. In this paper, the values of $t_q^1$ and $t_q^2$ are manually set to be 5. We used this setting to align with SciDocs, which is a widely applied benchmark dataset in scientific text embedding and citation prediction [1]. In SciDocs, each query is provided with a candidate set of 5 cited papers. Therefore, we extended it into a candidate set of 5 core citations ($t_q^1$) and 5 superficial citations ($t_q^2$).\n\nOur model can directly adapt to testing sets with different values of $t_q^1$ and $t_q^2$. In all parts of our designs, our model did not fit to specific values of $t_q^1$ and $t_q^2$ but only learned to rank the candidates in the order of core citation>superficial citations>non-citations. Therefore, its performance would not be affected by the values of $t_q^1$ and $t_q^2$.\n\nBesides, according to our statistics over 12M papers across 19 scientific fields in the Microsoft Academic Graph (MAG), on average, each paper has 10.207 ($\\sigma=$0.007) core citations and 10.727 ($\\sigma=$0.007) superficial citations. Therefore, it is reasonable to set $t_q^1=t_q^2$.\n\nWe will add the explanations and statistics to the revised version of our paper. Hopefully, our interpretations will make it easier for readers to understand our experimental settings.\n\n[1] SPECTER: Document-level Representation Learning using Citation-informed Transformers. *ACL 2020*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thanks for taking the time to review our paper. We greatly appreciate your valuable feedback and insightful suggestions. Please find our detailed one-on-one responses to the raised questions in 'Rebuttal' following the reviews. In addition, we have attached here 'result.pdf', which includes the supplementary results suggested by the reviews. When we mention it in the responses to some specific questions, please refer to that file accordingly.

Once again, we want to express our gratitude for your efforts and professional advice.
We are committed to incorporating all your suggestions to enhance the quality of our paper.

Sincerely,\
The Authors

Meta-review: The paper proposed a complete system that leverages LLM and the concept of core citation to predict the suitable citations for a new paper. The paper is well written. Extensive experiments from the original manuscript and new results during rebuttal showed that the system outperform other solutions with statistical significance.

The paper has its practical value to demonstrate a state-of-the-art citation prediction solution, which is a well-known task in NLP with real world use cases. Some reviewers pointed out that this is work is more on the application side thus may lack the novelty for ML research, however application is also an explicitly listed item on the Call for Papers ("Applications (e.g., vision, language, speech and audio, Creative AI)", actually it's the first item on the list), so AC consider this as a non-factor.

AC suggests the author to carefully incorporate the new data into the paper, while simplifying the description of the algorithm for easier reading (as a few reviewers suggested).
Decision: Accept (poster)