HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction

Authors: qianyue  hao
jingyang  fan
fengli  xu
jian  yuan
yong  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:59:31

Abstract: Citation networks are critical infrastructures of modern science, serving as intricate webs of past literature and enabling researchers to navigate the knowledge production system. To mine information hiding in the link space of such networks, predicting which previous papers (candidates) will a new paper (query) cite is a critical problem that has long been studied. However, an important gap remains unaddressed: the roles of a paper's citations vary significantly, ranging from foundational knowledge basis to superficial contexts. Distinguishing these roles requires a deeper understanding of the logical relationships among papers, beyond simple edges in citation networks. The emergence of large language models (LLMs) with textual reasoning capabilities offers new possibilities for discerning these relationships, but there are two major challenges. First, in practice, a new paper may select its citations from gigantic existing papers, where the combined texts far exceed the context length of LLMs. Second, logical relationships between papers are often implicit, and directly prompting an LLM to predict citations may lead to results based primarily on surface-level textual similarities, rather than the deeper logical reasoning required. In this paper, we introduce the novel concept of core citation, which identifies the critical references that go beyond superficial mentions. Thereby, we elevate the citation prediction task from a simple binary classification to a more nuanced problem: distinguishing core citations from both superficial citations and non-citations. To address this, we propose $\textbf{HLM-Cite}$, a $\textbf{H}$ybrid $\textbf{L}$anguage $\textbf{M}$odel workflow for citation prediction, which combines embedding and generative LMs. We design a curriculum finetune procedure to adapt a pretrained text embedding model to coarsely retrieve high-likelihood core citations from vast candidate sets and then design an LLM agentic workflow to rank the retrieved papers through one-shot reasoning, revealing the implicit relationships among papers. With the two-stage pipeline, we can scale the candidate sets to 100K papers, vastly exceeding the size handled by existing methods. We evaluate HLM-Cite on a dataset across 19 scientific fields, demonstrating a 17.6\% performance improvement comparing SOTA methods. Our code is open-source at https://github.com/tsinghua-fib-lab/H-LM for reproducibility.

Main paper: 
 introduction: 
 With the rapid development of modern science, the volume of research papers is increasing annually [1] . As links between papers, citations network connects vast literature and bridge newly emerging knowledge with existing ones. Due to the critical role of citations, citation prediction is an important problem that has long been studied [2, 3, 4, 5, 6, 7] , where the goal is to predict which papers from a set of previous papers (candidate set) will an emerging new paper (query) cite. Accurate citation prediction can help reveal information hiding in link space of citation networks [2, 8] , owning value in aiding citation-based computational social science studies regarding the patterns of paper publication and scientific innovation [9, 10, 11, 12, 13, 14] . On the other hand, citation prediction is of practical significance for assisting researchers in writing manuscripts, providing high-likelihood citation suggestions, and thereby saving massive literature searching time.Despite abundant studies on citation prediction, there is a critical problem that remains unconsidered. While one paper typically cites multiple previous papers, the roles of citations vary significantly. The most important citations serve as research foundations of the query paper, assisting researchers in tracing the lineage of knowledge production. In contrast, some less relevant citations are only mentioned superficially in context. Existing works treat citation prediction as a simple binary classification problem and neglect such varying roles [2, 3, 4, 5, 6, 7] , letting superficial citations distract attention from the important ones. However, such nuanced roles cannot be adequately reflected by simple edges in citation networks, but require understandings on the logical relationships among papers. In this paper, we aim to predict citations with various roles based on in-depth content understanding, where the emerging textual reasoning ability of LLMs provides a possible approach.Predicting citations with LLMs faces two major challenges. (1) Vast candidate sets. The real-world scientific database consists of gigantic papers, and researchers need to retrieve possible citations from millions of previous papers. With limited context length, it is impractical to feed the vast candidates' contents into LLMs and expect reasoning on logical relationships among them. (2) Implicit logical relationships. The logical relationships among papers lie implicitly within the content of papers. Directly prompting an LLM to predict key-role citations for query papers is likely to get sunk into simple content similarity rather than reasoning actual logical relationships among papers.In this work, we define the novel concept of core citation with inspiration from rich science of science research [9, 10, 12] , depicting the varying roles of citations. We analyze 12M papers across 19 scientific fields and illustrate core citations' significantly closer relationships with the query papers. Based on this definition, we develop the task of citation prediction from simple binary classification between citations and non-citations into a more challenging but meaningful version, i.e., distinguishing core citations from superficial citations and non-citations. To solve this task on vast candidate sets, we propose integrating embedding and generative LMs as HLM-Cite, a two-stage hybrid language model workflow. We design a curriculum fine-tuning procedure to adapt a pretrained text embedding model to analyzing research papers, initially retrieving high-likelihood core citations from vast candidate sets in the first stage. Subsequently, we design an LLM agentic workflow, consisting of a Guider, an Analyzer, and a Decider, for the second stage. Guided by a one-shot example, the LLM agents analyze the papers' implicit logical relationships through textual reasoning and rank the retrieved papers by citation likelihood. In HLM-Cite, we incorporate the capability of both embedding and generative LMs, enabling precise extraction of core citations from tremendous candidate sets. We conduct extensive experiments on cross-field papers, and the results show a 17.6% performance improvement of our method compared to SOTA baselines. Also, experimental results prove that our workflow can scale up to 100K candidates, thousands of times more than existing works, owning the potential to cover an entire research domain for practical implementation.In summary, the main contributions of this work include:• We define the novel concept of core citation to depict the varying roles of citations. Thereby, we develop the citation prediction task from simple binary classification into distinguishing core citations, superficial ones, and non-citations, giving it more practical significance.• We design a hybrid language models workflow to integrate the capabilities of embedding and generative LMs, where two categories of models form a two-stage pipeline that cascades retrieval and ranking to predict core citations. This design enables our method to handle very large candidate sets with high precision.• We conduct extensive experiments on a cross-field dataset with up to 100K paper candidate sets.The results prove the scalability of our design and illustrate a 17.6% performance improvement comparing SOTA methods. 2 Problem Formulation 
definition of core citation: 
 We first provided some notations about paper citation relationships. Considering a set of papers G, query paper q ∈ G cites a small subset of G including n q previous papers, denoted as {s 1 q , ...s nq q } ≜ S q ⊂ G \ {q}, while the rest papers are not cited by q, which we denote them as {p 1 q , ...} ≜ P q = ∁ G\{q} S q . Also, m q subsequent papers cite q, denoted as {f 1 q , ...f mq q } ≜ F q ⊂ G \ (S q ∪ {q}). As we mentioned above, the roles of each element in S q may vary significantly, where there exist k q elements in S q have major importance. We name them as core citations, denoted as {s 1 q , ...s kq q } ≜ Sq ⊂ S q . Naturally, we name the rest of the citations, i.e., S q \ Sq , as superficial citations. Enlighten by previous computational social science studies regarding citation networks [10, 12] , following-up papers of q, i.e., F q , are likely to also cite the critical foundations of q, namely q's core citations. On the other hand, less relevant citations of q, such as some background knowledge, are typically not followed by F q . Therefore, we mathematically identify the core citations according to such local citation relationships (Figure 1a ):Sq ≜ {s q ∈ S q | ∃p ∈ F q , let q ∈ S p , s q ∈ S p }.To verify the rationality of this definition, we draw statistics on 12M papers across 19 scientific fields in the Microsoft Academic Graph (MAG) [15] (See dataset details in Section 4.1). From the results in Figure 1b and c, we find that, with statistical significance, in both natural and social science domains, the query paper has more overlapped keywords with its core citations than its superficial citations, and the core citations are also more frequently mentioned in the main texts of query papers. This illustrates that the core citations identified from citation networks, are consistent with the important citations in the papers' content, proving feasibility of predicting core citations purely from the texts.Considering the difference between core citations and superficial citations, we focus on predicting the core citations, which are most meaningful links among literature for scientific research. We formally define the task of core citation prediction as follows.Definition 1 (Core Citation Prediction) Given a query paper q, and a candidate set C q , where |C q | = t q . C q includes t 1 q core citations and t 2 q superficial citations of q, ensuing t 1 q ≤ k q , t 2 q ≤ n q -k q and t 1 q + t 2 q ≤ t q , and its rest elements, if any, are non-citations. The goal of core citation prediction is to pick out t 1 q elements from C q , maximizing the number of picked core citations.In such a setting, superficial citations actually become hard negative samples against the core citations, adding to the challenges of the task.In this paper, we focus on text-based citation prediction, where we only use citation networks to obtain the ground truth of core citations and do not include any network features other than the papers' textual content in the prediction. In this way, our model learns to extract the logical relationships purely from the texts, predicting which citations of q are likely to be valued by future papers citing q without requiring any information about the exact future citations, which have not happened yet. Therefore, although we construct the ground truth of core citations in training and testing sets with previously published papers where we already know the subsequent papers that cite them, i.e., F q , our models is feasible for ongoing manuscripts without F q .Finetune { }t q =10 t q 1 =3 t q 2 =2 Embedding similarity (Representative) Textual reasoning (Generative)Final prediction { } { } { } Candidate set C q q q q qQuery Query Core citations Core citationsNon-citations Non-citations Why q cites these? What is the best ranking?  
overview: 
 To effectively predict core citations from large-scale candidate sets, we integrate the capability of both embedding and generative LMs, forming a hybrid language models workflow (HLM-Cite). We illustrate designs of the workflow in Figure 2 .As shown in Figure 2a , the HLM-Cite workflow consists of two major modules, i.e., the retrieval module (Section 3.2) and the LLM agentic ranking module (Section 3.3). When given a query q and a candidate set C q with the size of t q , we first call the retrieval module, a pretrained text embedding model finetuned with training data. We calculate the embedding vectors of q and each paper in C q , denoted as v q and V q = {v 1 q , ..., v tq q }, where we concatenate the title and abstract as inputs. Based on the inner products between v q and each vector in V q , we retrieve r q papers with the highest probability of being core citations of q from C q , forming the retrieval set R q . Subsequently, we employ LLM agents in the ranking module to collaboratively analyze the retrieved papers in R q and rank them according to their likelihood of being core citations, improving accuracy. Finally, we take the top t 1 q papers as the prediction result.Here, we introduce the structure of the text embedding model used in the retrieval module. We employ the GTE-base pretrain model [16] , one of the top models on the Massive Text Embedding Benchmark (MTEB) leaderboard [17] . Its 110M parameters are initialized from BERT [18] and trained with multi-stage contrastive learning tasks, embedding input text into a 768-dimensional dense vector. We freeze the lower 7 layers of the GTE-base model and only finetune parameters in the higher 5 layers, as shown in Figure 2b . As empirically proven in previous research [19] , such design can reduce computational consumption while maintaining the transferability in finetuning.As mentioned above, superficial citations act as hard negatives, adding to the difficulty of distinguishing core citations. Therefore, instead of directly transferring the GTE-base model to pick core citations from superficial citations and non-citations, we designed a two-stage curriculum finetuning as Figure 2b to gradually adapt the general-corpus model to our specific task, from easy to hard.In the first stage, we finetune the model via a classification task that only distinguishes the core citation from non-citations, excluding the interference of superficial citations, i.e., the hard negatives.We construct each training data with one query, one of its core citations, and numerous non-citations, and we use cross-entropy loss for classification error in this stage.In the second stage, we fully consider the ranking task of distinguishing core citations, superficial citations, and non-citations. We include one query together with its multiple core citations, superficial citations, and non-citations in each training data, and we apply NeuralNDCG loss function, a differentiable approximation of NDCG [20] , to measure the difference between the model output and the ground-truth ranking. In both stages, we use in-batch negative sampling [21] to obtain non-citations for each query to reduce the embedding cost.To improve the accuracy of core citation prediction, we incorporate LLMs' textual reasoning capability to rectify the ranking of papers retrieved in the previous stage by core-citation likelihood. As we illustrate in Figure 2c , the LLM agentic ranking module consists of three agents, the analyzer, the decider, and the guider, which are all driven by LLMs and collaborate via natural language communications. Given a query paper and its possible core citations retrieved from the candidate set, we first employ the analyzer to analyze the logical relationship between each individual paper in the retrieval set and the query paper. Then, we feed the analysis to the decider to obtain a revised ranking of their likelihood of becoming core citations, drawing final prediction results. In addition, we design a guider to enhance complex reasoning, where it produces a one-shot example under human supervision, assisting the analyzer and the decider via the chain of thought (CoT) method [22] .Also, we find that one useful technique in the LLM agentic ranking module is not to rank all retrieved candidates. Specifically, with the retrieval size of r q and t 1 q core citations in the candidate set, we exempt the (2t 1 q -r q ) retrieved candidates with largest inner products from reranking, and then we rerank the remaining 2(r q -t 1 q ) retrieved candidates with the LLM agents and selected the top (r q -t 1 q ) ones, resulting in t 1 q selected candidates in total. For example, when retrieval size is 7, we keep top-3 candidate unchanged and only rank the latter 4 candidates; when retrieval size is 8, we keep top-2 candidate unchanged and only rank the latter 6 candidates; and so on. The intuition for this is that the top candidates retrieved by the text embedding model tend to be core citations more safely. Therefore, only adjusting the latter ones is a rational solution that reduces the text length inputted into the LLMs and thereby improves the accuracy. We provide the detailed prompts used for the agents in Appendix A.1.Analyzer: from textual similarity to logical relationship. Intuitively, predicting citations requires in-depth understandings of the logical relationships among the papers, rather than only focusing on the textual similarity between their titles and abstracts. Therefore, we design the analyzer to extract why the query paper cites each of the candidates. Since plentiful knowledge has been encoded in the LLM as an implicit knowledge base, the agent can perform such analysis without domain-specific finetuning [23, 24, 25] .Decider: final ranking for core citation prediction. Based on the obtained analysis of paper relationships, we employ the decider to generate the final ranking of core-citation likelihoods. Besides simple ranking results, we prompt the agent to output corresponding explanations alongside, improving the rationality of its results [26, 27] .Guider: one-shot learning. To provide one-shot example for the analyzer and decider, we first select one representative query paper and several candidates outside the test set. As shown in Figure 2c , the candidates of query paper about Transformer-XL [28] include papers about (1) Neural Probabilistic Model [29] , (2) Transformers [30] , and (3) BERT [18] , where the ground truth ranking is 2-3-1. The guider goes through the analyze-decide procedure and produces a group of exemplary analysis and rectified ranking. We manually review and revise the obtained analysis and ranking texts, making sure they correctly reveal that (2) serves as the research foundation of the query, (3) discusses related recent advancements, while (1) only provides some historical contexts. Then we respectively feed the texts to the analyzer and decider via the chain of thought (CoT) [22] method, concatenate them at the beginning of the prompts. Here we only summarize the essence of guider's exemplary output due to limited space, and the full texts are available in Appendix A.6.This paper explores the immunostimulatory potential of DNA origami structures coated with CpG sequences. The immune response triggered by these structures is relevant to understanding the immune interactions and responses induced by DNA-based nanorobots, such as the one described in the query paper.Reason for Citation: To discuss the immunostimulatory properties of DNA origami structures and potentially draw parallels or insights into the immune responses elicited by the DNA nanorobot in the query paper. 
dataset: 
 We conduct experiments based on Microsoft Academic Graph (MAG) [15] , which archives hundreds of millions of research papers across 19 major scientific domains, forming a huge citation network. We traverse the dataset and filter 12M papers with abundant core citations and superficial citations, from which we randomly sample 450,000 queries and subsequently sample 5 core citations and 5 superficial citations for each query. We randomly divide the sampled queries into 8:2 as training and testing sets. Categorizing the scientific domains into natural science (biology, chemistry, computer science, engineering, environmental science, geography, geology, materials science, mathematics, medicine, physics) and social science (art, business, economics, history, philosophy, political science, psychology, sociology), we show statistics of the dataset in Table 1 . Please note that a natural science query paper may cite some papers from the social science domain and vice versa. We mainly evaluate our methods against three categories of baselines: simple rule-based method, LMs specifically designed for scientific texts, and pretrained LMs for general-purpose tasks. In the first category, we mainly predict core citation based on the degree of keyword overlap, i.e., the more overlap the candidate paper's keywords have with the query paper, the more likely it is to be a core citation. The second category includes SciBERT [31] , METAG [6] , PATTON, SciPATTON [5] , SPECTER [3, 32] , SciNCL [4] , and SciMult [33] . SciBERT is pretrained on millions of research papers from Semantic Scholar with the same approaches as BERT; METAG learns to generate multiple embeddings for various kinds of patterns of citation network relationships; PATTON and SciPATTON are finetuned with network masked language modeling and masked node prediction tasks on citation networks from BERT and SciBERT respectively; SPECTER is continuously pretrained from SciBERT with a contrastive objective; SciNCL is an improvement of SPECTER by considering hard-to-learn negatives and positives in contrastive learning; and SciMult is multi-task contrastive learning framework, which focuses on finetuning models with common knowledge sharing across different scientific literature understanding tasks. The third category includes BERT [18] , GTE [16, 34] , OpenAI-embedding-ada-002, and OpenAI-embedding-3 2 . BERT is pretrained with masked language modeling and next sentence prediction objectives on Wikipedia and BookCorpus; GTE is a series of top embedding models finetuned from BERT with multi-stage contrastive learning task; and the latter two are advanced universal embedding models proposed by OpenAI. We access these models from off-the-shelf pretrained parameters or API calls and include different scale versions of each model when available.We conduct the curriculum finetuning of our retrieval module with the batch size of 512 and 96 respectively in two stages, and each train for 10 epochs. The training process takes approximately 12 hours on 8×NVIDIA A100 80G GPUs in total. Then, we call OpenAI API to access GPT models for LLM agentic ranking, where we keep using GPT-4 as the guider but alternate two versions of GPTs for the analyzer and the decider. For more implementation details, please refer to Appendix A.2. In evaluation, we set vast candidate sets with t q = 10K (t 1 q = t 2 q = 5) for all models and set the retrieval size to be r q = 8 in our workflow. We evaluate the performance via PREC@3/5 and NDCG@3/5, and show the results in Table 2 . The results illustrate that our method significantly surpasses all the baselines across all scientific domains with all metrics, with an overall PREC@5 improvement up to 17.6%. We verify the statistical significance of the performance improvement in Appendix A.5.1. Mentioning that without loss of statistical significance, we only randomly test 10% of the testing set with GPT-4o due to API rate limits. In order to verify the rationality of LLM agentic ranking process, we provide the summary of a representative testing sample. We show the query paper, which designs a DNA Nanorobot [35] , and the retrieved candidates in Figure 3 . It turns out that our analyzer correctly reveals that the two candidates with core-citation ground truth inform the key design or the query paper [36, 37] ; the candidate with superficial-citation ground truth inspires some design details [38] ; while the non-citation candidate only mentions some very broad context that is almost irrelevant [39] . Based on the rational analysis, the decider correctly ranks the retrieval set and improves the precision. Please refer to Appendix A.7 to access the full texts of this case study.In order to verify the validity of our designs, we conduct ablation studies regarding both curriculum finetuning of the retrieval module and LLM agents design in the ranking module. We show the results in Table 3 . In the former part, we respectively delete the first and second stages of the curriculum and calculate the metrics on the retrieval set. The performance drop in both ablations indicates that our curriculum design does enable the adaption of the pretrained model from easy to hard, improving its transfer performance from general corpus to scientific documents. In the latter part, we respectively remove the analyzer and the guider. Specifically, without the analyzer, the decider directly ranks the retrieved candidates based on their raw titles and abstracts; without the guider, the analyzer and decider perform their tasks without the guidance of the one-shot example. It turns out that the absence of any agent leads to performance degradation, proving the essential role of each of them. We verify the statistical significance of the performance degradation in Appendix A.5.2. In this section, we provide in-depth analysis of various key elements in the HLM-Cite workflow, enabling a better understanding of our design. Here, if there is no special explanation, we all employ GPT-3.5 as our analyzer and decider. Mentioning that due to API rate limits, we only test 10% of the testing set in this section. To illustrate the advantage of our method on large-scale candidate sets, which are normal in realworld applications, we keep t 1 q = t 2 q = 5 consistent and change the number of non-citations to construct candidate sets with t q = 1K, 10K, and 100K. As shown in Figure 4a , regardless of the candidate size, our method significantly surpasses all top baselines and even achieves higher relative performance improvement on larger candidate sets (up to 18.5% in t q = 100K). We provide results with other metrics in Appendix A.3, where the conclusion is consistent.In our hybrid workflow, retrieval size r q is a key hyper-parameter that balances the work between the retrieval module and the LLM agentic ranking module. To explore the effect of r q , we alter it from 6 to 10 and show the performance together with LLM token consumption per query in Figure 4b . The results indicate that when r q increases, the performance increases at the cost of more token consumption. Larger r q leads to a higher recall rate of core citations in the retrieval set, and thereby, LLM agents have the potential to pick out more core citations from the texts with increased length. However, when r q is large enough, continuing to increase it leads to a performance drop while consuming even more tokens. We believe this is because too many retrieved candidates surpass the reasoning ability of LLMs, leading to confused analysis and low-quality ranking. Generally observed from the results, the optimal value of r q is supposed to be 8 and 7 for natural and social science, respectively. Results with other metrics in Appendix A.4 show consistent conclusion.As studied in previous research [40] , CoT enhances the performance of LLMs by demonstrating the logical structure of reasoning rather than providing specific knowledge content. Here, we investigate whether this is true in our hybrid workflow. We extend one-shot learning into a few-shot version. In this version, we produce an individual example for each scientific domain, where full texts are available in our GitHub repository. This provides more domain knowledge while maintaining an identical logical structure. The results in Table 4 show no significant performance difference between one-shot and few-shot learning, proving that what matters in CoT prompting is the logical structure of reasoning rather than specific domain knowledge. We explore the effect of substituting GPT-3.5 in our workflow with other open-source and lightweight LLMs. Here, we keep using GPT-4 as the guider to provide a high-quality one-shot example and change the analyzer and decider to various open-source LLMs3 . We explore using two versions of Llama3, one of the most famous open-source LLMs; two versions of Mixtral, a mixture of experts (MoE) model; and ChatGLM2-6B, a Chinese-English bilingual model. We show the results in Table 5 and find that although larger LLMs perform slightly better, i.e., Llama3-70B wins Llama3-8B, and Mixtral-8×22B wins Mixtral-8×7B, these lightweight LLMs all perform significantly worse than GPT models. This highlights the importance of implicit knowledge in LLM's large-scale parameters, which is crucial for solving tasks like citation prediction that require strong professional knowledge. Pretrained language models have long been studied and reached great success. Various small-scale embedding models have been trained via different objectives, such as masked token prediction [18, 41] , contrastive learning [3, 42, 4, 16] , and permutation language modeling [43] . These models require fewer computational resources and are especially suitable for a wide range of tasks on the large-scale corpus, including classification, clustering, retrieval [17] , etc. On the other hand, generative large language models (LLMs) have developed unprecedentedly in recent years. Pretrained on vast corpus, LLMs exhibit strong few-shot [23] and zero-shot [24] learning ability, reaching superior performance on text analyzing [44, 45, 46] , code generation [47, 48] and even solving math problems [49, 50] . However, most of the existing works lack the combination of these two categories of models. In this paper, we design the hybrid language workflow, incorporating small embedding models' advantage of efficient large-scale retrieval and generative LLMs' capability of textual reasoning.Title: Challenges and opportunities for structural DNA nanotechnology Abstract: DNA molecules have been used to build a variety of nanoscale structures and devices over the past 30 years, and potential applications have begun to emerge. But the development of more advanced structures and applications will require a number of issues to be addressed, the most significant of which are the high cost of DNA and the high error rate of self-assembly. Here we examine the technical challenges in the field of structural DNA nanotechnology and outline some of the promising applications that could be developed if these hurdles can be overcome. In particular, we highlight the potential use of DNA nanostructures in molecular and cellular biophysics, as biomimetic systems, in energy transfer and photonics, and in diagnostics and therapeutics for human health. 
llm agents: 
 Utilizing the strong reasoning capability and human-like behavior of LLMs, researchers have explored various applications based on agents driven by LLMs. First, LLM agents for decision-making reach success in sandbox games [51, 52] , robot controlling [53] , and navigation [54] . Besides, a group of LLM agents can simulate daily social life [55] , generate physical mobility behavior [56] , and reveal macroeconomic mechanisms [57] , providing insights for social science research. Closer to our task, role-fused LLM agents can collaboratively solve natural language processing tasks via analysis and discussions [45, 27, 58] . However, due to the limited context length in LLM reasoning, existing studies face difficulty handling tasks with extremely long texts, such as citation precision on vast candidate sets. In this paper, we incorporate generative LLMs with embedding models, enabling our hybrid workflow to work on very large candidate sets.Abstract: We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. ( 2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity.Title: From cells to DNA materials Abstract: Materials need to be specially engineered to interface with cells; on the other hand, cells provide great inspiration for new material designs. Here, using examples mainly from our own research, we demonstrate that DNA can be used as both a genetic and generic material for various cell-related applications, including diagnostics, drug delivery, cell culture, protein production, and immuno-modulation. We envision that other cell-based materials such as RNA, proteins, polysaccharides, and lipids can be more pervasively employed in materials science and engineering.6. (Core citation) Title: Molecularly self-assembled nucleic acid nanoparticles for targeted in vivo siRNA delivery Abstract: DNA strands can self-assemble into tetrahedral nanoparticles that can deliver small interfering RNA molecules to cells and suppress genes in tumours.7. (Superficial-citation) Title: Self-Assembled DNA Hydrogels with Designable Thermal and Enzymatic Responsiveness Abstract: or used as a programmable template to direct the assembly of nanoparticles. [14] [15] [16] [17] Recently, the concept of DNA assembly has been expanded to construct "DNA hydrogels", which are crosslinked networks swollen in an aqueous phase. [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] Though hydrogels have great potential in biological and medical applications, [32] [33] [34] [35] [36] such as drug and gene delivery, biosensing, and tissue engineering, studying the preparation of DNA hydrogels with designable properties is still in its early stages. In the past, several methods have been reported to prepare DNA hydrogels, for example, DNA directly extracted from the nucleus in nature, behaves like a long linear polymer and forms a hydrogel via physical entanglement or by chemical crosslinking of small molecules. [18] [19] [20] Similarly, DNA can be used as a negatively charged polymer and form a complex with cationic (poly)electrolytes through electrostatic interactions. [ 21 , 22 ] However, both methods treated DNA as a polymer and did not take advantage of the self-assembly of DNA into ordered structures, therefore, the resulting hydrogels lacked precise structural control and specifi c responses. Instead of using physical interactions, DNA can be covalently grafted onto synthetic polymers and serve as a cross-linker, the recognition of complementary DNA strands leads to crosslinking of polymer chains and causes hydrogel formation. [23] [24] [25] [26] [27] [28] In general, the preparation of a DNA-polymer hybrid requires laborious modifi cation steps, and an easy and fast strategy to build tailored DNA hydrogels is desired. Luo and his coworkers have developed a new approach to construct 8. (Non-citation) Title: Aptamer-enabled efficient isolation of cancer cells from whole blood using a microfluidic device. Abstract: Circulating tumor cells (CTC) in the peripheral blood could provide important information for diagnosis of cancer metastasis and monitoring treatment progress. However, CTC are extremely rare in the bloodstream, making their detection and characterization technically challenging. We report here the development of an aptamer-mediated, micropillar-based microfluidic device that is able to efficiently isolate tumor cells from unprocessed whole blood. High-affinity aptamers were used as an alternative to antibodies for cancer cell isolation. The microscope-slide-sized device consists of >59000 micropillars, which enhanced the probability of the interactions between aptamers and target cancer cells.The device geometry and the flow rate were investigated and optimized by studying their effects on the isolation of target leukemia cells from a cell mixture. The device yielded a capture efficiency of 95% with purity of 81% at the optimum flow rate of 600 nL/s. Further, we exploited the device for isolating col... 
conclusions: 
 In this paper, we investigate the task of scientific citation prediction. We first define the novel concept of core citation and thereby evolve the conventional citation prediction task into a more meaningful version of distinguishing the core citations. Then, we propose a hybrid language model workflow that incorporates the capability of both embedding and generative LMs. Through extensive experiments and in-depth analysis, we verify the validity of our design and illustrate its superior performance in tasks with gigantic candidate sets. One major limitation of our method lies in LLMs' illusion problem. Despite average performance improvement, LLMs may output unfaithful analysis under certain circumstances and poison specific samples. Therefore, how to verify the output of LLM agents and improve the reliability of our hybrid workflow worth future studies.Abstract: A goal of statistical language modeling is to learn the joint probability function of sequences of words. This is intrinsically difficult because of the curse of dimensionality:we propose to fight it with its own weapons. In the proposed approach one learns simultaneously (1) a distributed rep(cid:173) resentation for each word (i.e. a similarity between words) along with (2) the probability function for word sequences, expressed with these repre(cid:173) sentations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar to words forming an already seen sentence.Exemplary analysis:1. Relevance: This paper introduces the Transformer model, which is the foundation upon which Transformer-XL builds. The Transformer model revolutionized natural language processing (NLP) by moving away from recurrent and convolutional networks, focusing instead on attention mechanisms to process sequences of data. The query paper extends the Transformer model to handle longer contexts, which is a direct expansion of the work introduced in this paper.Reason for Citation: To acknowledge the foundational model (Transformer) on which Transformer-XL is based and to discuss the limitations of the original Transformer model that the query paper aims to overcome. 2. Relevance: The introduction of relative position representations in self-attention mechanisms is a key innovation that allows Transformers to better understand the relationships between different parts of a sequence. This concept is important for the Transformer-XL, which seeks to improve the model's ability to handle long-term dependencies.Reason for Citation: To discuss advancements in self-attention mechanisms that are relevant to the development of Transformer-XL, particularly the handling of position information in sequences, which is crucial for modeling longer contexts. 3. Relevance: This paper demonstrates the effectiveness of deep Transformer models for character-level language modeling, highlighting the potential of Transformers to capture complex dependencies in text. It sets a precedent for using Transformers in language modeling, particularly at the character level, and discusses techniques to improve their performance.Reason for Citation: To support the argument that deep Transformer models, like Transformer-XL, are effective at language modeling tasks and to build on the idea of enhancing Transformer architectures for better performance in NLP tasks. 4. Relevance: BERT represents a significant leap forward in pre-training language representations, using a bidirectional Transformer. While BERT focuses on understanding language context in both directions, Transformer-XL aims to extend the context length that models can effectively process.Reason for Citation: To highlight the importance of deep bidirectional Transformers in NLP and to position Transformer-XL within the broader context of recent advancements in Transformer-based models. It may also cite BERT to discuss differences in approach, particularly regarding context length and model architecture. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We reported 95% CI error bars in Figure 1 , and Figure 4 . We reported p-values in Section 4.3.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cited or provided URLs for the original paper that produced the code, dataset, and models in Section 3.2 Section 4.1 Section 4.2, and Section 4.5.4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We made our source code available in an anonymous repository at https: //anonymous.4open.science/r/H-LM-7D36.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We did not include crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: We did not involve crowdsourcing nor research with human subjects in this paper.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The specific prompt for the analyzer is as follows:System prompt: Now you are a sophisticated researcher and information analyst, and going to investigate the problem of a specific paper citation. Your analysis should be based on the following steps: Explore citation conventions and standards in academic fields. For example, citation serve to acknowledge prior work, provide evidence or support, facilitate further exploration and allow readers to trace the development and history of ideas or methodologies.Prompt: Here is the title and abstract of the query paper. In this section, we provide all implementation details for reproducibility in Table 6 . We provide performance comparisons among our method and several top baselines with different candidate sizes t q in Figure 5 . In both natural and social science domains, measured by both PREC@5 and NDCG@5, our method significantly surpasses all baselines regardless of the candidate size. The conclusion is consistent with the main text.Figure 5 : Analysis on the effect of candidate size t q . In all panels, 95% CI are shown as errorbars.We provide the performance of our method together with LLM token consumption per query with different retrieval sizes r q in Figure 6 . In both the natural and social science domains, when r q increases, the performance increases at the cost of more token consumption. However, when r q is large enough, continuing to increase it leads to a performance drop, while consuming even more tokens. Measured by both PREC@5 and NDCG@5, the optimal value of r q is supposed to be 8 and 7 for natural and social science, respectively. The conclusion is consistent with the main text.Figure 6 : Analysis on the effect of retrieval size r q . In all panels, 95% CI are shown as errorbars.We conduct statistical significance tests to better compare our model with the strongest baseline.In Table 7 , we used the two-tailed t-test between the performance of our model and the strongest baseline. Our method surpasses the top baselines in all fields (p < 0.01 in most individual fields, and p < 0.001 averaging all fields through t-test), demonstrating its general applicability to a wide range of fields. We conduct statistical significance tests to better compare our model with the ablation versions. In Table 8 , we used the two-tailed t-test between the performance of our full design and the ablation versions. Generally, all parts of our designs are valid with significance (p < 0.01 or p < 0.1 in overall performance through t-test). Moreover, we notice that when focusing on social science papers, which only comprise a small proportion of all papers, Stage 1 of curriculum finetuning is only slightly beneficial. Therefore, when only applying to social science papers, it is an alternative for users to skip Stage 1 if they want to save computational cost with the cost of a slight performance drop. In contrast, when applying to natural science papers, it is necessary to keep Stage 1 for better performance. Full texts of the one-shot example generate by the guider are as follows:Query paper: Title: Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Abstract: Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18. 2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements.3. Title: Character-Level Language Modeling with Deeper Self-Attention Abstract: LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model (Vaswani et Relevance: This paper explores adaptive input representations, which can make models more efficient and effective by adjusting the capacity of input representations based on the complexity of the input. Such techniques are relevant for Transformer-XL, which seeks to improve efficiency and performance in language modeling. Reason for Citation: To discuss methods for improving the efficiency of neural language models, particularly in the context of Transformer-based architectures. The query paper might leverage or build upon these adaptive techniques to enhance Transformer-XL's performance. 6. Relevance: This work is foundational in the field of neural language modeling, introducing the concept of learning distributed representations for words alongside the probability function for word sequences. It lays the groundwork for subsequent developments in language modeling, including the use of Transformers.Reason for Citation: To acknowledge the historical context and evolution of language modeling techniques leading up to the development of Transformer and Transformer-XL models. It may also cite this work to discuss the importance of distributed representations in understanding language.Exemplary ranking: Ranked order: paper 1, paper 2, paper 3, paper 4, paper 5, paper 61. Explanation: This paper is the cornerstone of Transformer models. Any research following Transformer-XL would likely reference this to acknowledge the foundational model and its limitations that the follow-up work seeks to address or build upon. 2. Explanation: The methodological relevance of improving self-attention mechanisms, especially for handling longer contexts in Transformer models, makes this paper a critical citation for discussing technical advancements or modifications in a follow-up study. 3. Explanation: This paper's focus on deep Transformer models for character-level language modeling aligns closely with the objectives of Transformer-XL, making it a likely citation for discussions on model depth and granularity in language modeling. 4. Explanation: Given the significant impact of BERT on the NLP field and its methodological similarities and differences with Transformer-XL, a follow-up study would likely cite it to discuss further advancements or comparisons in Transformer-based model architectures. 5. Explanation: Techniques for improving model efficiency and input representation are crucial for advancing Transformer models. A follow-up study might cite this work to explore or introduce new adaptive techniques for enhancing Transformer-XL's efficiency or performance. 6. Explanation: While foundational to the field of neural language modeling, this paper might be cited less frequently in a direct follow-up to Transformer-XL, except to provide historical context or discuss the evolution of language modeling techniques leading up to Transformer models.The full texts of the representative case in the main text are as follows. In this case, the analyzer correctly reveals the relationships between the query paper and each candidate, and the ranker outputs a revised ranking, which increases the PREC@5 from 0.6 in the retrieval set to 0.8 in the final prediction. Notice that due to fluctuations in dataset quality, there may be a few garbled characters.Title: A DNA nanorobot functions as a cancer therapeutic in response to a molecular trigger in vivo.Abstract: Nanoscale robots have potential as intelligent drug delivery systems that respond to molecular triggers. Using DNA origami we constructed an autonomous DNA robot programmed to transport payloads and present them specifically in tumors. Our nanorobot is functionalized on the outside with a DNA aptamer that binds nucleolin, a protein specifically expressed on tumorassociated endothelial cells, and the blood coagulation protease thrombin within its inner cavity.The nucleolin-targeting aptamer serves both as a targeting domain and as a molecular trigger for the mechanical opening of the DNA nanorobot. The thrombin inside is thus exposed and activates coagulation at the tumor site. Using tumor-bearing mouse models, we demonstrate that intravenously injected DNA nanorobots deliver thrombin specifically to tumor-associated blood vessels and induce intravascular thrombosis, resulting in tumor necrosis and inhibition of tumor growth. The nanorobot proved safe and immunologically inert in mice and Bama miniature pigs. Our data show that DNA nanorobots represent a promising strategy for precise drug delivery in cancer therapy.Candidate papers:1. (Core citation) Title: Universal computing by DNA origami robots in a living animal Abstract: Biological systems are collections of discrete molecular objects that move around and collide with each other. Cells carry out elaborate processes by precisely controlling these collisions, but developing artificial machines that can interface with and control such interactions remains a significant challenge. DNA is a natural substrate for computing and has been used to implement a diverse set of mathematical problems, logic circuits and robotics. The molecule also interfaces naturally with living systems, and different forms of DNA-based biocomputing have already been demonstrated. Here, we show that DNA origami can be used to fabricate nanoscale robots that are capable of dynamically interacting with each other in a living animal. The interactions generate logical outputs, which are relayed to switch molecular payloads on or off. As a proof of principle, we use the system to create architectures that emulate various logic gates (AND, OR, XOR, NAND, NOT, CNOT and a half adder). Following an ex vivo prototyping phase, we successfully used the DNA origami robots in living cockroaches (Blaberus discoidalis) to control a molecule that targets their cells. Abstract: To investigate the potential of DNA origami constructs as programmable and noncytotoxic immunostimulants, we tested the immune responses induced by hollow 30-helix DNA origami tubes covered with up to 62 cytosine-phosphate-guanine (CpG) sequences in freshly isolated spleen cells. Unmethylated CpG sequences that are highly specific for bacterial DNA are recognized by a specialized receptor of the innate immune system localized in the endosome, the Toll-like receptor 9 (TLR9). When incubated with oligonucleotides containing CpGs, immune cells are stimulated through TLR9 to produce and secrete cytokine mediators such as interleukin-6 (IL-6) and interleukin-12p70 (IL-12p70), a process associated with the initiation of an immune response. In our studies, the DNA origami tube built from an 8634 nt long variant of the commonly used single-stranded DNA origami scaffold M13mp18 and 227 staple oligonucleotides decorated with 62 CpG-containing oligonucleotides triggered a strong immune response, characterized by cyt...1. We do not involve the top-1 paper among retrieved candidates into LLM agentic ranking.2. We do not involve the top-2 paper among retrieved candidates into LLM agentic ranking.Relevance: This paper addresses the technical challenges in the field of structural DNA nanotechnology, highlighting the need to overcome hurdles such as high costs and selfassembly errors. Understanding these challenges is crucial for advancing the design and application of DNA-based nanorobots.Reason for Citation: To acknowledge the broader challenges and opportunities in DNA nanotechnology, which can inform the development and optimization of DNA nanorobots for targeted drug delivery in cancer therapy.Relevance: This work explores the use of DNA as a versatile material for various cell-related applications, including diagnostics, drug delivery, and immuno-modulation. Understanding the diverse applications of DNA materials can inspire innovative approaches to designing DNA nanorobots for targeted drug delivery.Reason for Citation: To draw insights from the use of DNA in cell-related applications and potentially leverage the principles or techniques discussed in this paper for designing DNA nanorobots with specific functionalities for cancer therapy.Relevance: This paper focuses on the self-assembly of nucleic acid nanoparticles for targeted delivery of small interfering RNA (siRNA) molecules to cells. The concept of targeted delivery using nucleic acid nanoparticles aligns with the targeted drug delivery approach of the DNA nanorobot described in the query paper Reason for Citation: To explore strategies for targeted delivery of therapeutic molecules using nucleic acid nanoparticles, which can inform the design and delivery mechanisms of the DNA nanorobot in the query paper. is important for advancing the design and application of DNA-based nanorobots, this paper may be cited less frequently in a direct follow-up to the query paper, except to acknowledge broader challenges and opportunities in DNA nanotechnology. 5. Explanation: Exploring the diverse applications of DNA materials in cell-related contexts can inspire innovative approaches to designing DNA nanorobots for targeted drug delivery, potentially offering new perspectives on how DNA-based nanorobots can be optimized for specific functionalities in cancer therapy. 6. Explanation: The focus on targeted delivery using nucleic acid nanoparticles resonates with the targeted drug delivery approach of the DNA nanorobot described in the query paper, making this paper highly relevant for understanding and improving targeted drug delivery mechanisms. 7. Explanation: Understanding the design and responsiveness of self-assembled DNA structures can provide valuable insights into developing functional DNA nanorobots with controlled release mechanisms, enhancing the precision and efficacy of targeted drug delivery in cancer therapy. 8. Explanation: This paper directly aligns with the targeting strategy of the DNA nanorobot in the query paper, emphasizing the importance of aptamers for specific cell targeting and isolation, which is crucial for the success of targeted drug delivery systems.One major limitation of our method lies in LLMs' illusion problem. Despite average performance improvement, LLMs may output unfaithful analysis under certain circumstances and poison specific samples. When researchers want high-likelihood citation suggestions in preparing manuscripts, these samples may cause confusion. Therefore, how to verify the output of LLM agents and improve the reliability of our hybrid workflow worth future studies. Also, the curriculum finetuning process requires a certain amount of computational resources. Therefore, how to lighten the computational load worth further investigation.We fully use open-source models and datasets in the paper, which involve no problem regarding privacy and copyright. We cite the resources in Section 3.2, Section 4.1, Section 4.2, and Section 4.5.4. Moreover, our training and testing data are randomly sampled from publications all around the world, which does not involve problems of bias and discrimination.Our method has positive broader impacts. On the one hand, accurate citation prediction can help reveal information hiding in link space of citation networks, owning value in aiding citation-based computational social science studies. These studies may investigate the patterns of paper publication and scientific innovation, enlightening researchers with efficient research approaches and putting forward the advancement of modern science. On the other hand, the application of our hybrid workflow is not limited to the task of citation prediction. A wide range of natural language processing tasks may borrow experience from our work and improve their performance.Justification: We also made our source code available in an anonymous repository at https://anonymous.4open.science/r/H-LM-7D36.Guidelines:• The answer NA means that paper does not include experiments requiring code. We also included all details in the source code, which is anonymously available at https: //anonymous.4open.science/r/H-LM-7D36.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.https://openai.com/index/new-embedding-models-and-api-updates/https://llama.meta.com, https://mistral.ai, and https://github.com/THUDM/ChatGLM-6B 


Decision: The paper proposed a complete system that leverages LLM and the concept of core citation to predict the suitable citations for a new paper. The paper is well written. Extensive experiments from the original manuscript and new results during rebuttal showed that the system outperform other solutions with statistical significance.

The paper has its practical value to demonstrate a state-of-the-art citation prediction solution, which is a well-known task in NLP with real world use cases. Some reviewers pointed out that this is work is more on the application side thus may lack the novelty for ML research, however application is also an explicitly listed item on the Call for Papers ("Applications (e.g., vision, language, speech and audio, Creative AI)", actually it's the first item on the list), so AC consider this as a non-factor.

AC suggests the author to carefully incorporate the new data into the paper, while simplifying the description of the algorithm for easier reading (as a few reviewers suggested).

Reviews: review: paper_summary: paper_summary: This paper studies text-based citation prediction by exploring the varying roles of paper citations from foundational to superficial. The authors introduce the concept of core citations, emphasizing the most important citations over superficial ones. Then, they propose HLM-Cite, a hybrid language model workflow combining embedding and generative language models. The approach involves a two-stage pipeline: first, a pre-trained embedding model coarsely retrieves high-likelihood core citations, and then a large language model ranks these candidates through reasoning. Experiments on the Microsoft Academic Graph across 19 scientific fields demonstrate the effectiveness of HLM-Cite in comparison with scientific pre-trained language models, text-rich network representation learning methods, and general-domain language models for text embedding.
strength_weakness: strength_weakness: - Some scientific language model baselines are missing. To be specific, SPECTER 2.0 [1] and SciMult [2] have shown superior performance over SciNCL on various tasks including citation prediction, but neither of them is compared in the experiments.

- The dataset used in this paper is constructed by the authors. It is unclear whether the proposed model works for other datasets. For example, the SciDocs dataset from the SPECTER paper [3] is a widely used benchmark dataset for citation prediction, but it is not considered in the experiments. Also, it is unclear why the authors cluster the 19 scientific fields into 2 big areas (i.e., natural science and social science). Would it be possible to directly show the results in each field (e.g., like [4])?

- Statistical significance tests are not conducted. It is not clear whether the gaps between HLM-Cite and the baselines/ablation versions are statistically significant or not. In fact, in Tables 2 and 3, some gaps are subtle, therefore multiple runs are needed and the p-value should be reported.

[1] SciRepEval: A Multi-Format Benchmark for Scientific Document Representations. EMNLP 2023.

[2] Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding. Findings of EMNLP 2023.

[3] SPECTER: Document-level Representation Learning using Citation-informed Transformers. ACL 2020.

[4] The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study. WWW 2023.
questions: questions: - Could you compare HLM-Cite with SPECTER 2.0 and SciMult?

- Could you report the results on the SciDocs benchmark dataset (or explain the reasons why you did not use it)?

- Could you conduct a statistical significance test (e.g., two-tailed t-test) to compare your model with the strongest baseline/ablation version in Tables 2 and 3?
limitations: limitations: The authors discuss the limitations and potential negative societal impact of their work in the Checklist. I suggest they further consider the limitations mentioned in my review above.
paper_summary: paper_summary: The authors introduce the concept of core citations to distinguish important citations from superficial ones and non-citations. This shifts the citation prediction task from simple binary classification to the more subtle approach of identifying core citations. They then propose HLM-Cite, a hybrid language model workflow that combines embedding and generative language models. This two-stage pipeline first retrieves high-likelihood core citations from a vast set of candidates using a pretrained text embedding model and then ranks them using a logical reasoning process conducted by LLMs. With the two-stage pipeline, the authors show that they can scale the candidate sets to 100K papers, thousands of times larger than existing works. They evaluate HLM-Cite on a dataset across 19 scientific fields, demonstrating a 17.6% performance improvement comparing SOTA methods.
strength_weakness: strength_weakness: My main comment is that the terminology in Section 2.1 is overly dense. There is way too much use of superscripts and subscripts for relatively simple concepts. If possible, the authors should try to make the symbols simpler. 

On the technical front, I still feel that the approach the authors are taking might be overly complex, and I'm unsure what impact it will have on actual computational science practice. I'm also wondering whether a simpler approach based on a core-periphery algorithm from network science, rather than text analysis, might not be useful for identifying core citations in existing papers. Obviously, this is less useful for prediction, but I am not sure there is any impact here in prediction; people writing papers already know what the core citations in their field are.
questions: questions: Given that keyword overlap seems to be highly predictive of whether a paper is a core citation or not, I would have liked to see some comparisons to simpler baselines than what the authors ultimately showed. Also, is text analysis even necessary, or can network science be used here, as in so much of science of science research?
limitations: limitations: The authors consider a technical limitation, namely "LLMs' illusion problem." LLMs may output unfaithful analysis under certain circumstances and poison specific samples. When researchers want high-likelihood citation suggestions in preparing manuscripts, these
 samples may cause confusion. The authors note that verifying the output of LLM agents and improving the reliability of their hybrid workflow could be pursued in the future. Also, the curriculum finetuning process requires a certain amount of computational resources. Lightening this load also merits further investigation. There are no significant social/ethical limitations of this work, including negative societal impacts.
paper_summary: paper_summary: The paper proposes a framework, HLM-Cite (Hybrid Language Model) for scientific citation prediction based on incorporating generative language models embeddings and LLMs as agents. The pretrained text embeddings are used to retrieve high likelihood core citations (a term the paper introduces to define as more meaningful citations, compared to superficial or non-citations). Three different LLM agents are then used to rank the papers by reasoning over them.
strength_weakness: strength_weakness: - The idea of a "core" citation is not necessarily novel, as citation classification systems have been around for a while. While there isn't one classification system the scientific community agrees upon, there have been multiple proposed. The paper should acknowledge this body of work around citation classification systems, especially if proposing a new one: "core", "superficial" and "non-citations" - which is essentially a binary classification on the citation type ("core" vs "superficial").  Some references: 

  - [1] Cohan, Arman, et al. "Structural scaffolds for citation intent classification in scientific publications." arXiv preprint arXiv:1904.01608 (2019).

  - [2] Garfield, Eugene. "" Science Citation Index"—A New Dimension in Indexing: This unique approach underlies versatile bibliographic systems for communicating and evaluating information." Science 144.3619 (1964): 649-654.

  - [3] Jurgens, David, et al. "Measuring the evolution of a scientific field through citation frames." Transactions of the Association for Computational Linguistics 6 (2018): 391-406.

  - [4] Moravcsik, Michael J. "Citation context classification of a citation classic concerning citation context classification." Social Studies of Science 18.3 (1988): 515-521.

  - [5] Nicholson, Josh M., et al. "scite: A smart citation index that displays the context of citations and classifies their intent using deep learning." Quantitative Science Studies 2.3 (2021): 882-898.

  - [6] Teufel, Simone, Advaith Siddharthan, and Dan Tidhar. "Automatic classification of citation function." Proceedings of the 2006 conference on empirical methods in natural language processing. 2006.

- While the paper provides an interesting and well designed system, it is definitely geared towards applications of LLMs and models, rather than novel scientific contributions. Might be more appropriate for a specialized workshop. This is why my recommendation is borderline.

- Minor notes: 

  - Lines 13-15 are a bit hard to understand, could be rephrased e.g. [...an LLM to predict citations, which leads to results ..]

  - Line 88 -> notations instead of notifications?
questions: questions: - Since you are using MAG to finetune the various models, how are you annotating the entries in the graph with the three citation classes you proposed: "core", "superficial" and "non-citations", especially given the size of this knowledge graph?
- In Section 3.3.2, it is mentioned that the analyses are manually reviewed and revised, to make sure that they correctly reveal the expected behavior (lines 193-195). How many papers is this done for? The entire set of papers? 
- In Section 4.1, it is mentioned that 5 core citations and 5 superficial citations are sampled for each of the 450k queries. Similarly to the first question - how are these citations initially annotated with one of the "core" vs "superficial" classes?
- In several places in the paper (e.g. Section 4.5)  you mention "Without loss of statistical significance". How was this computed?
limitations: limitations: Authors mention LLM limitations, such as hallucinations, in their conclusion.
paper_summary: paper_summary: This paper proposes a Hybrid Language Model workflow for citation prediction, where core citations are predicted from superficial citations and non-citations rather than using a simple binary classification approach. This method can handle candidate sets of up to 100K papers and demonstrates better performance compared to previous methods.
strength_weakness: strength_weakness: 1. The paper mentions in the appendix that they keep the top candidate unchanged and only rank the remaining candidates in the decider. This should also be mentioned in the main text to avoid confusion, especially in Figure 3, regarding the omission of the top 3 candidates.
2. In the ablation study of the curriculum stage, the performance without Stage 1 is very close to the full curriculum, especially in Social Science. Moreover, the Prec@3 of Social Science is actually better without the full curriculum. This suggests that Stage 1 might not be that effective, especially considering its computational cost.
questions: questions: 1. How is the number of unchanged candidates in the decider stage selected?
2. The paper uses a text-based method in the first stage to retrieve candidates. How does this compare to previous binary citation prediction methods? For example, what if binary citation prediction is used to retrieve citations first, then the LLM agentic ranking module is used to further distinguish between core citations and superficial citations?
3. How are the numbers of tq1 and tq2 selected? Would this affect performance? Additionally, since superficial citations should be more than core citations in real cases, why are tq1 and tq2 kept equal?
limitations: limitations: The paper discusses limitations.

Comments: title: Reminder to reply
Reminder to reply.
Last chance to reply to the authors!
comment: Dear reviewer,

Thanks for taking the time to read our rebuttal. We are glad to hear that it has answered most of your questions.

To further clarify some issues, we want to briefly summarize our main novelty here:
- **Network-based method to identify core citations at scale.** Existing citation classification method requires manual annotation [1-5]. In contrast, our definition of core citation is based on citation network structure, which enables researchers to efficiently create large-scale datasets to fine-tune LLMs, and it performs well empirically. 
- **New citation prediction task.** To the best of our knowledge, we are the first to introduce the core citation prediction task [6-9]. We have empirically shown that it is a harder yet meaningful task because core citations have deeper overlap and more frequent interaction with the focal works (Figure 1).
- **Novel model design.** We propose a novel framework incorporating small and large LMs for citation prediction. Our framework combines the advantages of both models, ensuring computational efficiency in large-scale tasks and maintaining the reasoning ability beyond simple textual similarities. We are glad that you find our framework design interesting.

Also, please note that our work's novel contribution is not limited to the concept of core citation. Beyond the definition, we also proposed a novel training and prompting framework of language models to operationalize all the valuable features for precise citation prediction.

Hopefully, our summary can better illustrate the novelty of our work.

Best,\
The authors

[1] Structural scaffolds for citation intent classification in scientific publications. *NAALC 2019*.\
[2] Measuring the evolution of a scientific field through citation frames. *Transactions of the Association for Computational Linguistics 2018*.\
[3] Citation context classification of a citation classic concerning citation context classification.*Social Studies of Science 1988*.\
[4] Scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. *Quantitative Science Studies 2021*.\
[5] Automatic classification of citation function. *EMNLP 2006*.\
[6] SPECTER: Document-level Representation Learning using Citation-informed Transformers. *ACL 2020*.\
[7] Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings. *EMNLP 2023*.\
[8] Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding. *Findings of EMNLP 2023*.\
[9] Patton: Language Model Pretraining on Text-Rich Networks. *ACL 2023*.
Thanks for your valuable suggestions again. We will include the additional results and improve our paper accordingly.
I thank the authors for their detailed responses and additional experiments, which address some of my concerns. After reading the responses and the other reviews, I decided to increase my rating from 6 to 7. Meanwhile, I strongly suggest the authors include the additional results in their revised version.
Dear reviewer,

Thanks again for your valuable suggestions. Following your suggestions, we have rephrased the dense terminology and clarified the potential impact of our citation prediction task. Hopefully, our revision can help the readers better understand the contributions and potential applications of our study.

Please let us know if you have any questions. We are looking forward to further discussion.

The authors
Dear reviewer,

Thanks again for your insightful suggestions. Following your advice, we have incorporated the up-to-date baselines and clarified all other issues. Hopefully, our revision can help the readers better understand the contributions and potential impact of our study.

Please let us know if you have any questions. We are looking forward to further discussion.

The authors
Thank you to the authors for the response, which I have read carefully. While the response clarifies most of my questions and I believe the proposed framework is an interesting use of LLMs for citation prediction, I maintain my initial reservations around the novelty of the proposed concept of core citations. I will keep my score.
Thanks for your suggestions again. We will improve our paper accordingly.
Thanks to the authors' detailed response. Most of my questions have been answered. I will maintain a positive score and raise my confidence to 4.
Dear reviewer, if you have not clicked the reply button, please don't forget to do so as the deadline is approaching. Your input is important to the authors!  - AC
Dear reviewer, if you have not clicked the reply button, please don't forget to do so as the deadline is approaching. Your input is important to the authors!  - AC
Dear reviewer, if you have not clicked the reply button, please don't forget to do so as the deadline is approaching. Your input is important to the authors!  - AC