Beyond Retrieval: Topic-based Alignment of Scientific Papers to Research Proposal (2024)

Abstract: The inception of a research agenda typically commences with the creation of a comprehensive research proposal. The efficacy of the proposal often hinges on its ability to connect with the existing scientific literature that supports its ideas. To effectively assess the relevance of existing articles to a research proposal, it is imperative to categorize these articles into high-level thematic groups, referred to as topics, that align with the proposal. This paper introduces a novel task of aligning scientific articles, relevant to a proposal, with researcherprovided proposal topics. Additionally, we construct a dataset to serve as a benchmark for this task. We establish human and Large Language Model (LLM) baselines and propose a novel three-stage approach to address this challenge. We synthesize and use pseudolabels that map proposal topics to text spans from cited articles to train Language Models (LMs) for two purposes: (i) as a retriever, to extract relevant text spans from cited articles for each topic, and (ii) as a classifier, to categorize the articles into the proposal topics. Our retriever-classifier pipeline, which employs very small open-source LMs fine-tuned with our constructed dataset, achieves results comparable to a vanilla paid LLM-based classifier, demonstrating its efficacy. However, a notable gap of 23.57 F1 score between our approach and the human baseline highlights the complexity of this task and emphasizes the need for further research.

heading: 1 Introduction
text: Researchers frequently draft research proposals to present new ideas, define research agendas and seek funding grants. An integral part of the proposal writing process is reviewing relevant literature and relating it to different aspects of the proposal. Several existing approaches designed for automatic
* Equal contribution
retrieval of scientific articles can be applied to identify articles relevant to a proposal, with a detailed description (abstract) of the proposal serving as the query (Cohan et al., 2020b). However, there is often a further need to formulate high-level thematic categories (henceforth referred to as topics) relevant to the proposal and map retrieved relevant scientific articles to these categories for fine-grained contextualization. Such fine-grained mapping can further facilitate motivating the research problem, identifying its novelty, establishing baselines, synthesizing methods, and automatic literature review generation.
Given a set of reference article abstracts relevant to a proposal, Zhu et al. (2023); Martin-Boyle et al. (2024) auto-generates the thematic categories in a hierarchical form (termed as a catalogue) and organize references. However, the results demonstrate that the auto-generated catalogue does not match with the original-author-defined catalogue, leading to discrepancies in downstream literature review generation. This is due to inability of stateof-the-art LMs as well as subjectivity of the task. As opposed to this, we consider a more realistic setting, where we assume the availability of not only the reference papers retrieved to be relevant to a proposal; but also high-level topics provided by the researcher, for further literature categorization. With this assumption, we focus on the novel task of alignment of these reference papers, to one or more of these topics, offering a comprehensive understanding of its distinct contributions to the target proposal. This ensures a more personalized approach, aligning the cataloguing process closely with the researcher’s unique perspective representing their understanding of the field.
Figure 1 illustrates an example, showcasing proposal topics, relevant reference papers and their mappings to the topics, with distinct text spans from the reference paper representing the relevant context (henceforth termed as the reference text
70
span). Thus, a reference can be cited in more than one proposal topic with distinct context (reference text spans) (Li et al., 2023). Considering the token lengths of scientific articles, vanilla paid LLM-based approach for this task would be expensive. We consider the requirement of using smaller open-source LMs without compromising on the performance of LLMs. Hence, there is a need for a dataset to fine-tune domain-specific LMs for the task. As existing datasets for scientific document understanding (Clement et al., 2019; Lo et al., 2020; Saier et al., 2023; Kasanishi et al., 2023; Li et al., 2022) are not well-suited for this task, we extend the UnArXiv dataset (Saier et al., 2023), originally designed for literature review generation. We establish human and vanilla LLM baselines and define a novel three-stage approach for topic-based alignment of papers. Since ground truth labels for the reference text spans relevant to topics are not available, we augment the data by retrieving the reference text spans using the text around the article citation within the proposal topic (citation text), which is assumed to be available only for training. We use the synthesized pseudo pairs to train language models (LM) for retrieval of text spans from articles for a topic, and for classification of the retrieved articles to the topic. As opposed to all the prior approaches (Hoang and Kan, 2010; Hu and Wan, 2014; Chen and Zhuge, 2019; Wang et al., 2020; Pandey et al., 2022; Deng et al., 2021; Vajdecka et al., 2023), considering the realistic need of initial proposal writing stage, we neither assume the availability of citation text for the cited papers nor detailed description of the topics during inference. Our primary contributions are:
• We define the novel task of aligning reference
articles relevant to a target proposal with userdefined proposal topics.
• We construct a dataset using a set of research papers as proposals, the subsection headings of the related work sections as the ‘topics’ and the papers cited under those sections as the reference papers relevant to those ‘topics’.
• We establish human and LLM baselines and find that higher human F1 scores demonstrate task feasibility.
• We define a novel approach wherein we assume no citation text or detailed topic descriptions to retrieve reference text spans for a topic during inference. We devise a novel strategy using the citation text, available only for training, as a link between the topic and the reference text spans to create pseudo-labels for training a retriever and classifier pipeline.
• Our pipeline using much smaller LMs trained with pseudo-labels yields comparable performance to that of the LLM baseline, demonstrating the efficacy of the constructed dataset and the approach. In contrast, a significant gap (23.57 F1 score) exists between our approach and the human baseline indicating the need for more sophisticated solutions.

heading: 2 Related Work

heading: 2.1 Scientific Document Understanding
text: Literature Review Generation: Current approaches for automatic literature review generation either independently summarize articles (Hayashi et al., 2023; Urlana et al., 2022; Akkasi, 2022) or generate citation text (yan Wu et al., 2021; Jung
et al., 2022; Wang et al., 2022) for each article without considering their inter-relations (Li et al., 2022; Li and Ouyang, 2024). These approaches often produce monolithic extractive (Hu and Wan, 2014; Wang et al., 2018) or abstractive (Chen et al., 2021, 2022; Kasanishi et al., 2023; Chen et al., 2021; Liu et al., 2023) reviews lacking any structure. Kasanishi et al. (2023); Martin-Boyle et al. (2024) introduce a method for generating reviews with well-structured subsections, grouping relevant articles by specific topics. However, they either assume the availability of the mapping of research articles to proposal topics or take human inputs for the same. Our task serves as an upstream task for comprehensive literature review generation and focuses on the automatic alignment of relevant articles to a set of proposal topics.
Citation Text Generation: Citation text generation crafts sentences that cite reference articles based on their abstracts, aiming to integrate them into a literature review. However, existing methods (yan Wu et al., 2021; Jung et al., 2022; Wang et al., 2022) assume precise knowledge of intents (background, methodology, etc), behind citing a paper which is largely unavailable in the early stages of proposal writing. Moreover, these approaches rely solely on the reference paper’s abstracts, potentially lacking adequate information for appropriate mapping to an intent or topic. In contrast, our approach of extracting reference text spans from the reference paper ensures the availability of comprehensive information for alignment. Prior approaches to text span extraction from research papers use citation texts (Pandey et al., 2022; Zerva et al., 2020; Vajdecka et al., 2023) or queries (Li et al., 2023). In contrast, our approach considers the more practical setting at the proposal writing stage and retrieves text spans without relying on the availability of citation text or detailed queries assuming only the availability of high-level topics.
Citation Intent Detection: Citation intent detection (Lahiri et al., 2023; Roman et al., 2021; Berrebbi et al., 2022) presumes the presence of citation text to classify papers into predefined categories such as motivation, background, etc. However, during the proposal writing stage, citation text is unavailable. Moreover, the classification categories for potential reference papers differ for each target proposal. In contrast, our work introduces a new task of mapping potentially relevant papers to proposal-specific topics defined by users without assuming the availability of the citation text.
Scientific Paper Retrieval: Several approaches for retrieving scientific papers from a corpus rely either on abstracts and titles of a target paper (Singh et al., 2023; Cohan et al., 2020a), detailed textual queries (Sesagiri Raamkumar et al., 2017; Anand et al., 2017; Parisot and Zavrel, 2022; Medic and Šnajder, 2023), or relevant aspects such as the problem or methodology (Mysore et al., 2022; Ostendorff et al., 2022; Singh et al., 2023). These approaches employ strategies to generate suitable embeddings for queries and research papers, often focusing on pre-defined ’aspects’ common across the target papers. In contrast, our work assumes the availability of research articles relevant to a proposal and perfoms a more fine-granular mapping of these articles to a set of user-defined topics.

heading: 2.2 Existing Datasets
text: We evaluate existing datasets for our task. The SciReviewGen (Kasanishi et al., 2023) dataset is designed for literature review generation, is derived from S2ORC (Lo et al., 2020) and consists of only survey papers with their contents extracted from their PDFs, resulting in highly erroneous extractions. CORWA (Li et al., 2022), derived from ArXiv (Clement et al., 2019), extracts information from the LaTeX version of ACL conference papers, ensuring relatively error-free extractions. However, while it employs a tagger to classify transitions in text-forming paragraphs, it falls short of extracting the associated chapter heading (topic in our context) for each text span. (Medic and Šnajder, 2023) utilize the CORWA dataset for scientific paper retrieval, under the assumption that the first transition text spans of extracted paragraphs indirectly refers to the topic. Our analysis, however, reveals that this assumption holds only for a smaller subset of the samples. UnArXiv dataset (Saier et al., 2023), derived from ArXiv papers, struggles to correctly identify in-text citations for these papers required to derive the mapping of topics to the corresponding reference papers. We extend the UnArXiv dataset and overcome this limitation with our own parser (details in Section 4).

heading: 3 Task Definition
text: Consider a research proposal with its title and abstract R and with a set of topics forming the catalogue denoted as TR = t1, ..., tK . We assume a corpus of reference research papers PR = p1, ..., pn retrieved using R via existing retrieval methods
such as Specter (Singh et al., 2023). The task aims to classify the mapping of each research paper pi ∈ PR to each topic tj ∈ TR using binary labels yij ∈ 0, 1. The dataset consists of positive tuples < R, pi, tj , cij >, where cij is the citation text citing the paper pi under topic tj and is only available during training. We define a process to augment the dataset with the reference text span sij by using the citation text cij . The reference text span refers to the contents of the referenced paper pi, needed to determine whether pi is relevant to the topic tj . Again, we do not assume the availability of sij for training or inference in the dataset.

heading: 4 Dataset Construction
text: Figure 2 demonstrates the steps we have followed for the construction of our dataset. We utilize the papers in the UnArxiv dataset (Saier et al., 2023), with their titles and abstracts as target proposals R. We leverage the ArXiv tags of these papers to obtain their domains and focus on papers in AI and
Computer Vision. This was to ensure high-quality annotations since the human annotator who facilitated the establishment of the baseline (Section 6.2) had expertise in these domains.
We use UnArxiv IDs to retrieve papers from ArXiv, obtain LaTeX sources, and employ a LaTeX parsing technique with Regex statements to extract in-text citations within our simulated proposal papers. Our parser identifies section headings from these target papers and extracts topics (tj), from the subsection headings of the ‘Related Work’ or ‘Literature Review’ Sections. We selectively retain only those target proposals where multiple topics exist in the literature review. For each topic tj thus identified, we further identify in-text citations for referenced papers {pi} and extract the corresponding citation texts {cij}. We then match the citation text with the corresponding entry in the references section of the paper to obtain the referenced paper title. We obtained the PDFs of the referenced papers from various sources like ArXiv, ACL, and
Semantic Scholar. The contents of the referenced papers are extracted using our PDF extractor.
Thus, our final dataset consists of set-of research papers from selected domains from the UnArxiv dataset with each paper simulating a target proposal R. The dataset also includes the topics for each target proposal TR and reference papers relevant to the target proposal PR. Moreover, the ground truth labels of our task, in terms of alignments of reference papers to the topics are solicited from the in-text citations, explicitly specified by the authors of the target papers. For the training set, we assume the availability of the citation text cij for every pair tj and pi. Thus, a sample of the training data is depicted by the tuple < R, pi, tj , cij >. Whereas for the test set a sample is < R, pi, tj >. We split target proposals into train, validation, and test sets to prevent information leakage across splits. Dataset statistics are summarized in Table 1. The resulting corpus has target proposals pertaining to Artificial Intelligence (AI) (2.69 %), Machine Learning (ML) (15.56%), Computational Linguistics (CL) (7.28%), Computer Vision (CV) (73.23%) and a combination of CL and CV (1.24%). We make the dataset available at1.

heading: 5 Approach
text: We break down our approach into three stages ( Figure 2): (i) Augmenting the dataset with positive and negative reference text spans for a proposal topic, retrieved from papers relevant to the proposal, with the citation text as the query, using a state-of-the-art retriever model (SR) (ii) Training the Topic based reference text span Retriever (TR) and reference paper classifier (TC) using the augmented data (iii) Using TC to classify a reference paper for its relevance to a proposal topic in context
1https://github.com/NeuralNimbus/Beyond-Retrieval under license: GNU GPL v3
of the proposal and the reference text span retrieved from the paper relevant to the topic using TR .

heading: 5.1 Stage 1: Data Augmentation
text: To train the retriever TR, we need positive and negative pairs of topics tj and reference text spans sij . We do not have such pairs available in our training data. However, we do have the citation text cij , which we utilize as a ‘link’ to retrieve the reference text span sij from pi, relevant to cij and consequently relevant to tj . We assess the performance of existing retrieval models on equivalent tasks in the science domain, viz., (i) Retrieval of paragraphs from scientific documents given a question and (ii) Retrieval of a text span given the citation text. We identify a retriever model SR having the best zeroshot performance for both these tasks, making it generalizable for our task and dataset. Section 6.4, details the experiments performed to choose SR.
We chunk the reference paper pi using a slidingwindow approach choosing 7 sentences as a chunk with a stride of 3. The top-k chunks from the reference paper most relevant to the citation text cij retrieved using the best-performing retriever SR, serve as the retrieved reference text span ŝij of article pi for the topic tj . The similarity score between the citation text and a chunk of the reference paper is calculated on the lines of (Nogueira et al., 2020), by taking log softmax over ‘True’ and ‘False’ tokens to get the probability of ‘True’ token as the score and rank the chunks. The retrieved top-k chunks may or may not be contiguous, and we use a higher k value for good recall, avoiding information loss. The retrieved reference text span ŝij functions as a pseudo-positive pair for the topic tj . We consider three distinct types of pseudo-negative reference text spans sij for topic tj (Figure 3): (i) Type 1: The bottom-k chunks retrieved from pi, with citation text cij of topic tj as query, using SR serve as easy negatives for the topic tj in proposal R. (ii) Type 2: Text spans (top-k chunks) retrieved from the reference paper pl, NOT cited in topic tj , but cited in tk where k ̸= j, with the citation text clk as query using SR, serve as easy negatives for the topic tj in proposal R. (iii) Type 3: With each cij for topic tj citing pi as the query, we retrieve top-k chunks from reference articles pl, NOT cited in topic tj , but cited in tk where l ̸= i and k ̸= j using SR. The top-k chunks demonstrating maximum similarity with one of the cij serve as hard negatives for the topic tj in proposal R.
We augment our training dataset with these
pseudo positive and negative reference text spans for each topic leading to the resultant dataset, where a sample is depicted by the tuple < R, pi, tj , cij , ŝij , {sij} >. The statistics of the augmented dataset are illustrated in Table 2.

heading: 5.2 Stage 2: Training
text: Topic based Reference Text Span Retriever (TR): We train TR, which is a Language Model (LM) using the positive and negative topic and reference text span pairs. Following the methodology discussed in (Nogueira et al., 2020), we try to maximize the probability Pr(True|tj , sij) and Pr(False|tj , sij) using the cross-entropy loss. Due to the significantly higher number of negative samples compared to positives in the dataset, we include all positives but randomly sub-sample the negatives of each domain uniformly to maintain a balanced training dataset. Sampling is performed with replacement for every epoch to ensure that the model sees all negatives.
Proposal Topic based Reference Paper Classifier (TC): We train the TC using the augmented training dataset. We form samples < R, tj , ŝij >
with label yij = 1 and < R, tj , sij > with label yij = 0. The model learns to classify if the paper pi can be assigned to the topic tj in the context of R and sij , with the supervision of the label yij . We try to maximize the probability Pr(y = True|R, tj , sij) and Pr(y = False|R, tj , sij) using cross-entropy loss.

heading: 5.3 Stage 3: Inference
text: During inference, we feed the proposal title and abstract R and a topic tj from the test set to TR along with chunks chq of a reference article pi in the test set tagged as relevant to R. The model TR provides us with the a similarity score simjq = TR(tj , chq) for each chq. The score is calculated by applying a softmax on the logits of the ‘True’ and ‘False’ tokens and taking into consideration the probability P (True|tj , chq) (Nogueira et al., 2020). We rank chq for the given R and tj based on simjq and use the top-k ranked chq of paper pi as the retrieved reference text spans ŝij from paper pij, for topic tj in R. We further feed < R, tj , ŝij > to TC If probability P (y = True|R, tj , ŝij) ≥ 0.5, then only we consider pi to be aligned with tj .

heading: 6 Experimentation and Results

heading: 6.1 Evaluation Metric
text: To assess the performance of SR on evidence and reference text span retrieval tasks (Section 6.4), we compute the evidence F1 score (Dasigi et al., 2021). For the evaluation of our pipeline (TR followed by TC), we utilize the binary ground truth labels depicting the alignment of each proposal topic and each reference paper relevant to that proposal and the labels predicted by TC in the context of retrieved reference text spans to compute the confusion matrix for the binary classification task. We choose the F1 score as the metric, given the label imbalance.

heading: 6.2 Baselines
text: To establish baselines we evaluate the ‘Reference Paper Topic Classification’ task with human and LLM-generated annotations. Our human annotator is a researcher with expertise in AI and allied fields. The details of the annotation interface are discussed in Appendix B. Considering the complexity, cognitive load and LLM cost for the task, the baselines are developed only on a smaller evaluation subset
of the test set. We utilize GPT-3.5 Turbo2 LLM. To ensure deterministic and reproducible results, we set the temperature to 0. The structure of the prompt is detailed in Appendix A. Acknowledging the limitation of over-representation of CV papers in the dataset (Section 4), we construct the evaluation subset by uniformly sampling random four proposals from each of the five domains from the test set and thus ensuring balanced domain representation. This results in selecting 20 proposals with 52 topics citing 362 reference papers, forming 378 positive topic-reference paper pairs.
We have the following baselines: (i) Random: We randomly assign Yes/No labels to the test samples, (ii) TR + Human: The human annotator is provided with information about the target proposal’s title, abstract (R), topic tj , reference paper title pi, and the text span of the reference paper sij , retrieved using the topic tj with our trained TR model. The task is to assess the relevance of the reference paper to the given topic, labelling it 1 if the reference paper is aligned to the topic; and 0 otherwise, (iii) LLM: For vanilla LLM baseline we provide the complete reference paper pi to the LLM along with information about the target proposal’s title, abstract (R), topic tj as the part of the prompt (Appendix A) and ask the LLM to classify the reference paper’s relevance to the topic in the context of the proposal. We truncate the tail of the paper if the token length of the paper exceeds the maximum token length of the LLM. This baseline does not take into consideration text span from the reference paper, which is relevant to the topic and hence can be treated as is topic-agnostic reference text span based method, (iv) TR + LLM: This is the same has TR + Human except the relevant information mentioned above is provided to LLM as opposed to the human annotator. In this case, the LLM performs the job of classification of reference paper alignment to topics in the context of the proposal and retrieved reference text spans from the paper, as opposed to the complete paper, which is the case in vanilla LLM. Note that, on similar lines to the LLM baseline we could have had only the Human baseline, where the human gets to read the complete reference paper to perform the classification task. However, due to the very high cognitive load of this task, we skip this baseline.
2https://platform.openai.com/docs/models/gpt-3-5-turbo

heading: 6.3 Models and Hyperparameter Setting
text: Topic-Based Reference Text Span Retriever (TR): We fine-tune a T5-base model(Raffel et al., 2019), consisting of 223M parameters with a batch size of 120, learning rate of 3 × 10−4 with the AdamW optimizer, on Nvidia A100 for 20 epochs taking 84 hours, evaluated every 500 steps on the validation split. The model with the best validation F1 is obtained after fine-tuning for 1000 steps.
Research Paper Topic Classifier (TC): We chose RoBERTa (Liu et al., 2019) and Flan-T5 (Chung et al., 2022) base consisting of 125M and 248M parameters respectively, for their robust reasoning capabilities, as TC. Both models are finetuned on a class-balanced dataset with the batch sizes 240 and 120, using the AdamW optimizer and learning rates 5× 10−5 and 3× 10−4, respectively. Fine-tuning is performed for 50 epochs on an Nvidia A100 taking approximately 120 hours each. The models with the best validation F1 are obtained at 33 and 35 epochs for RoBERTa and Flan-T5, respectively.

heading: 6.4 Results
text: Citation Text based Reference Text Span Retrieval (SR): As discussed in Section 5.1, our dataset lacks ground truth reference text span labels sij from a reference paper pi, which are relevant to a topic tj . Given the availability of citation text cij belonging to a topic tj for the train set, we perform experiments to identify a state-of-the-art model to retrieve the reference text spans sij for that topic. We evaluate existing retrieval models on the test sets of QASPER (Dasigi et al., 2021) for evidence retrieval and CLScisumm (Chandrasekaran et al., 2019) for reference text span retrieval. We assess
the performance of models depicted in Table 3 in a zero-shot setting, to select the model most generalizable for our task and dataset. For the evidence retrieval task with QASPER, Specter 2 is used with the AdHoc Query Adapter for questions and the Proximity Adapter for paragraphs (Singh et al., 2022). For CLScisumm, Specter 2’s AdHoc Query Adapter is employed to embed both the citation text and candidate citation text spans. Top-3 sentences are retrieved for each model, following the approach in (Pandey et al., 2022). We observe that Mono T5 achieves the highest F1 Score among pretrained models for both datasets (Table 3). While not as good as the supervised state-of-the-art model CGSN (Nie et al., 2022) for QASPER, it surpasses the SOTA model CitRet (Pandey et al., 2022) in CL-SciSumm. Hence, we select Mono T5 as SR.
Topic based Reference Text Span Retriever (TR) and Reference Paper Classifier (TC): Table 4 illustrates the evaluation of our inference pipeline discussed in section 5.3 on the subset of test data (Section 6.2). A high F1 score achieved by the human baseline establishes the task’s feasibility. Higher F1 of our supervised models as compared to Random confirms task learnability. Higher F1 scores of TR + LLM as compared to LLM demonstrate the benefit of using our retrieval model. More importantly, comparable scores of our fine-tuned pipeline with very small (248M) LMs to that of pre-trained Large LM (175B), suggest the efficacy of utilizing our synthesized training data for task modelling. However, the noticeable difference in the human baseline concerning both the zero-shot LLM-based approach and our fine-tuned LM-based pipeline highlights the task’s inherent difficulty, indicating a need for more sophisticated techniques. For a comprehensive evaluation, we evaluate the pipeline on our complete test set. RoBERTa and
Flan-T5 yield F1 scores of 63.40% and 62.82%, respectively.
We conducted a detailed error analysis (Appendix C) of the errors made by the expert and identified author subjectivity, insufficient information in the retrieved reference text spans and lack of in-depth subject knowledge, as the challenges encountered by the expert. We further analyze the errors made by LLM and fine-tuned TC models for samples correctly classified by experts, detailed in Appendix C. The majority of errors are due to lexical overlap between topics and reference text spans with no semantic alignment causing False Positives or the models failing to perform complex reasoning required for the task.

heading: 7 Conclusion and Future Work
text: We have defined a novel task of mapping relevant scientific articles to research proposal topics as a precursor to the Literature Review Generation task for a new research proposal. We introduce a large-scale dataset for the task and establishment of competitive baselines by an expert and an LLM, underscoring task feasibility. We define a novel approach for the task and are the first ones to simulate the real-life scenario, at the early stage of proposal writing, of unavailability of citation text or detailed topic descriptions to retrieve topic-wise reference text spans from relevant articles. We came up with a novel strategy of using citation text (available for the training data) as a link between the topic and the text spans to create pseudo-labels for training a retriever. Our pipeline using much smaller open-source LMs trained with pseudo-labels yields comparable performance to that of the paid LLM baseline, demonstrating the efficacy of constructed data and designed pipeline. In contrast, a noticeable gap of 23.57 F1 score between our approach and the human baseline underscores the task’s complexity, demanding more sophisticated solutions.
While organizing reference articles by topic, the hierarchy among these topics (catalogue), is crucial for a comprehensive understanding. However, we found it difficult to capture this hierarchy due to the PDF parsing challenges, treating every topic as a standalone entity. In future, with more sophisticated PDF parsing and semantic paragraph segmentation techniques, we plan to capture the topic hierarchy for task completeness.

heading: A Prompt Structure for LLM Annotation
text: As outlined in Section 6.2, we employ GPT-3.5 Turbo to set a benchmark on the Annotated split of the dataset. After experimenting with various prompts, we find that the prompt depicted in Listing 1 yields the best results.
Listing 1: Prompt Structure for TR + LLM baseline. For vanilla LLM we replace the Text Span of paper ’B’ with the actual reference paper. system_prompt = ''' You are a Research Paper Relevance
Classifier. Given the 'Title ' and ' Abstract ' of Paper `A' and a 'Topic ' of Paper 'A', and a text span from Paper 'B' predict whether Paper 'B' is directly related to the topic of Paper 'A', in context of the 'Title ' and 'Abstract ' of Paper 'A '. Output the model's prediction as:"
Output: [1] (for 'yes ') or [0] (for 'no ')'''
user_prompt = ''' Title of Paper 'A': <Target Proposal Title > Abstract of Paper 'A': <Target Proposal Abstract >
Topic of Paper 'A': <Topic >
Text Span of Paper 'B': <Reference Paper Title > <Citation Text Span >
Is the content of Paper 'B' directly related to the topic of Paper 'A' (< Topic >)? '''

heading: B Annotation Interface
text: We have developed an Annotation tool aimed at offering a user-friendly interface to engage the human expert for annotating samples. The Annotator tool provides information on the target proposal’s title, topic, reference paper title, abstract and the retrieved citation text span of the reference paper, as illustrated in Figure 6 (b). Moreover, we mandate the expert to review the disclaimer prior to the annotation process, ensuring that they are well informed about the context and ethical guidelines associated with the samples they are annotating, as reflected in Figure 6 (a).

heading: C Error Analysis
text: We perform error analysis for the task of reference paper to topic mapping (classification). We use the sub-sampled test set discussed in Section 6.2 for the analysis.
C.1 Human Baseline
We analyze the False Positive and Negative samples with the annotations provided by the human expert.
The primary objective is to gain a profound understanding of the challenges inherent in the task. The identified 121 erroneous annotations provided by human are categorized into the following groups:
• Subjectivity: These errors, constituting 77.69% of total erroneous cases, stem from a misalignment between the thought processes of the author and the annotator. As shown in Figure 4 (a), the reference paper is not cited by the author under the given topic, however human annotator tends to map the reference paper to the topic due to the match between the semantics of the citation text span and the topic. In these instances, the annotator correctly deems the reference article relevant to the topic, while the author holds a contrasting view. This emphasizes the inherent subjectivity embedded in the task, where interpretations of relevance can diverge between individuals.
• Insufficiency: Errors in this category are 4.96% of total erroneous cases and are attributed to the insufficient information present in the retrieved citation text span. The experts face challenges in making well-informed decisions due to the lack of comprehensive details, impeding their ability to accurately assess the relevance of the reference article to the given topic. For example, as highlighted in Figure 4 (b), the citation text span does not carry enough information to let the author infer its mapping to the topic. We observe in some of these cases the citation text span may carry in-
formation semantically similar to the citation text and thus the topic, but the information is not sufficient to logically reason about the possible alignment of the reference paper to the topic. This problem can be addressed by having a better mechanism for citation text span retrieval given citation text, which would indirectly help us improve the results of topic based citation text span retrieval model to retrieve more relevant citation text spans. This can be improved by augmenting the pipeline with an entailment task following the retrieval, to evaluate which of the top-k retrieved citation text spans entails the citation text to accommodate the reasoning component. We leave this enhancement for future work.
• Inadequacy: Errors categorized under inadequacy, accounting for 7.44% of total erroneous cases, emerge due to the lack of subject matter knowledge of the expert annotator. This knowledge gap prevents her from accurately understanding the relevance of the reference article to the proposed topic.
• True Errors: This category, constitutes 9.92% of total erroneous cases, encompassing genuine mistakes made by human annotators. Despite concerted efforts to maintain accuracy, errors of this nature occur, underscoring the complexity and human cognitive load of the task.
The systematic categorization of these errors showcases the prevalence of subjectivity, arising from differences in the author’s writing style and thought process which constitutes the majority of mis-classifications by human annotators.
C.2 LLM baseline and fine-tuned models against Human baseline
We analyze instances where human annotators correctly classified the samples that are misclassified by the LLM and fine-tuned models, as outlined in Table 5. We majorly categorize the error into two categories:
• Requirement of complex reasoning: These errors are due to the absence of direct alignment between the topics and the citation text span of the reference paper. It requires multihop complex reasoning for the mapping task. As demonstrated in Figure 5 (a), the reference paper is not directly relevant to the topic
The primary factor contributing to misclassifications of models is lexical similarity or dissimilarity between the topic and the retrieved citation text span. This highlights the model’s challenge in comprehending contextual nuances, leading to misinterpretations.

heading: D Limitations
text: Our pipeline is fine-tuned with our constructed dataset containing papers in the AI (majorly CV) domain. The results are demonstrated with a test set containing papers in the AI domain only, leading to comparable results with the zero-shot vanilla LLM baseline. Our pipeline may not perform well in cross-domain settings. However, our dataset construction technique can be easily applied to the scientific articles belonging to any domain, based on the availability of the articles. Extrapolating the current results demonstrated for the AI - CV domain, the retrieval-classifier pipeline fine-tuned with such domain-specific synthesized dataset should lead us to comparable results with the zero-shot vanilla LLM baseline, for any domain.
We create pseudo-topic-reference text span labels for the training data by considering the text under that topic (section) which is written to cite a reference article (citation text) as an explicit link to connect the topic and the reference article. It clearly carries the information of the relevance of the scientific article with regards to the proposal, in the context of the given topic. The citations to the same reference article but at other places in the proposal may not carry information on the relevance of the reference article in the context of the given topic to which the reference article is to be aligned. However, we are not completely denying the possibility of some other sections carrying information about the relevance of the reference article to the proposal in the context of the given topic. However as there is no explicit signal that we can exploit to derive this connection, we cannot consider that information while establishing the connection.

heading: Future Work
text: We leave this enhancement for future work. • Inadequacy: Errors categorized under inadequacy, accounting for 7. 44% of total erroneous cases, emerge due to the lack of subject matter knowledge of the expert annotator. This knowledge gap prevents her from accurately understanding the relevance of the reference article to the proposed topic. • True Errors: This category, constitutes 9. 92% of total erroneous cases, encompassing genuine mistakes made by human annotators. Despite concerted efforts to maintain accuracy, errors of this nature occur, underscoring the complexity and human cognitive load of the task. The systematic categorization of these errors showcases the prevalence of subjectivity, arising from differences in the author’s writing style and thought process which constitutes the majority of mis-classifications by human annotators. C. 2 LLM baseline and fine-tuned models against Human baseline
We analyze instances where human annotators correctly classified the samples that are misclassified by the LLM and fine-tuned models, as outlined in Table 5. We majorly categorize the error into two categories:
• Requirement of complex reasoning: These errors are due to the absence of direct alignment between the topics and the citation text span of the reference paper. It requires multihop complex reasoning for the mapping task. As demonstrated in Figure 5 (a), the reference paper is not directly relevant to the topic
The primary factor contributing to misclassifications of models is lexical similarity or dissimilarity between the topic and the retrieved citation text span. This highlights the model’s challenge in comprehending contextual nuances, leading to misinterpretations