FINECITE: A Novel Approach on Fine-Grained Citation Context Analysis
Authors: Lasse M. Jantsch, Dong Jae Koh, Seonghwan Yoon, Jisu Lee, Anne Lauscher, Young-Kyoon Suh
Accepted at: acl25

FINECITE: A Novel Approach For Fine-Grained Citation Context Analysis

Lasse Jantsch1 Dong-Jae Koh1 Seonghwan Yoon1 Jisu Lee1 Anne Lauscher2* Young-Kyoon Suh1*
1School of Computer Science and Engineering, Kyungpook National University, South Korea
2Data Science Group, University of Hamburg, Germany
{lassejantsch, djkoh, shyoon0214, jisulee74, yksuh}@knu.ac.kr,
anne.lauscher@uni-hamburg.de

Abstract
Citation context analysis (CCA) is a field of research studying the role and purpose of citation in scientific discourse. While most of the efforts in CCA have been focused on elaborate characterization schemata to assign function or intent labels to individual citations, the citation context as the basis for such a classification has received rather limited attention. This relative neglect, however, has led to the prevalence of vague definitions and restrictive assumptions, limiting the citation context in its expressiveness. It is a common practice, for example, to restrict the context to the citing sentence. While this simple context conceptualization might be sufficient to assign intent or function classes, it fails to cover the rich information of scientific discourse. To address this concern, we analyze the context conceptualizations of previous works and, to our knowledge, construct the first comprehensive context definition based on the semantic properties of the citing text. To evaluate this definition, we construct and publish the FINECITE corpus containing 1,056 manually annotated citation contexts. Our experiments on established CCA benchmarks demonstrate the effectiveness of our fine-grained context definition, showing improvements of up to 25% compared to state-of-the-art approaches. We make our code and data publicly available.1

1 Introduction
Scientific research is inherently collaborative, with each discovery building upon a foundation of prior studies. To acknowledge previous work and provide credit, it is standard practice to include citations that connect past findings to new contributions. By embedding scientific progress and argumentation, citations serve a critical function that has been extensively examined—a research field known as citation context analysis (CCA) (Kunath et al., 2022; Swales, 1986).

Figure 1: A visual comparison of different citation context conceptualizations in the CCA literature

In computational linguistics, CCA is mainly concerned with the automatic classification of citations along various dimensions, such as citation function (Lauscher et al., 2022; Cohan et al., 2019; Jurgens et al., 2018; Teufel et al., 2006), sentiment (Lauscher et al., 2017; Abu-Jbara et al., 2013; Athar and Teufel, 2012), or influence (Pride and Knoth, 2020; Cohan et al., 2019). Given a passage of text surrounding a citation marker—referred to as the citation context—one or more classes from a predefined citation classification scheme are assigned.

Although a considerable amount of research has explored different classification schemes and methods, the citation context has received relatively little attention. This lack of focus has led to an absence of a comprehensive definition and datasets with overly simplistic and coarse-grained citation contexts (Pride and Knoth, 2020; Cohan et al., 2019).

To address this concern, we analyze different context conceptualizations in previous work and provide a new comprehensive definition based on the semantic information of the citing text. A visual comparison is provided in Figure 1.
With our research and artifacts, we hope to spark new interest in the exploration of citation context information. Given the drastic capabilities of LLMs in zero-shot text understanding (Brown et al., 2020; Lewis et al., 2020; Vaswani et al., 2017), and the emergence of advanced language processing systems (Lewis et al., 2020; Edge et al., 2024), we argue that an improved understanding of contextual citation information is essential for improving interactive exploration of scientific argumentation.

Our contributions are the following:

• We analyze and formalize citation context conceptualizations in previous work.

• We propose, to our knowledge, the first fine-grained citation context definition based on the semantic structure of the citing text.

• We construct and publish the FineCITE corpus comprising 1,056 manually annotated fine-grained citation contexts.

• We evaluate our context definition in two experiments and demonstrate its effectiveness on established benchmarks.

The rest of the paper is organized as follows. The subsequent section reviews relevant literature in the field of CCA and provides a formalization of task and context conceptualization. Section 3 introduces our citation context definition. In Section 4, we describe the curation process of the FineCITE corpus and provide core statistics. In Section 5, we assess the effectiveness of our context definition in both context extraction and citation classification. Section 6 summarizes our contributions and outlines directions for future research.

2 Related Work

CCA is the subject of a substantial body of research with (Garfield, 1972) often mentioned as one of the pioneering works. Reaching back to (Teufel et al., 2006), CCA research in computational linguistics is commonly conceptualized as learning a function \( F_C \) representing the relationship of a citation context spanning \( s \in S \) to a set of classes \( c \in C \). The task can thus be formalized as

\[
F_C(s) = \arg\max_{c \in C} P_{F_C}(c \mid s),
\]

where \( P_{F_C} \) are the class probabilities emitted by \( F_C \). The classes \( C \) can represent various citation attributes, such as function (Lauscher et al., 2022; Jurgens et al., 2018; Teufel et al., 2006), purpose (Pride and Knoth, 2020; Abu-Jbara et al., 2013), sentiment (Athar and Teufel, 2012), or intent (Cohan et al., 2019). For a comprehensive survey on CCA, refer to (Kunnath et al., 2022).

Despite the continued research in CCA, the introduction of new and larger datasets (Cohan et al., 2019; Jurgens et al., 2018), and updated methodology (Shui et al., 2024; Lauscher et al., 2022; Cohan et al., 2019), the simple modeling paradigm as described in Equation 1 prevailed. The popular SCiCITE benchmark (Cohan et al., 2019) even further simplifies the task by reducing the commonly used six-class framework of (Jurgens et al., 2018) to a three-class schema. This simplicity leads to a low task complexity; however, it often fails to adequately represent the rich information present in the scientific texts (Lauscher et al., 2022). To capture a wider range of information, it is necessary to move beyond prevalent context span constraints and conceptualization on mutually exclusive classes. Table 1 compares the relevant research.

Context Span Constraints. The optimal context spans \( S^* \) can be defined such that

\[
S^* = \left\{ \arg\max_{s_i \in S_i} P_F(c_i \mid s_i) \mid i \in I \right\}
\]

where \( S_i \) is the set of all possible context spans for one citation instance \( i \in I \), and \( P_F \) is the probabilities assigned by a function \( F \) representing some relationship between \( S \) and \( C \).

As it is infeasible to solve Equation 2, previous work uses various assumptions to extract an approximate optimal context \( \hat{S}^* \). The first common assumption is that \( S^* \) can be approximated by a fixed-sized window surrounding the citation marker. The size of the context window varies between one (Pride and Knoth, 2020; Cohan et al., 2019), or multiple sentences (Abu-Jbara et al., 2013; Athar and Teufel, 2012), a specific number of characters (Jurgens et al., 2018), or the whole paragraph (Teufel et al., 2006). Some approaches (Abu-Jbara et al., 2013; Athar and Teufel, 2012) allow for a non-context classification of context-window subsets, introducing a simple form of dynamic context spans. Only recent publications stress the importance of a fully dynamic approximation of \( S^* \) (Lauscher et al., 2022; Nambanoor Kunnath et al., 2022) to conform to the situated structure of scientific argumentation.
<table>
  <tr>
    <th>AUTHOR (YEAR)</th>
    <th>ASPECT</th>
    <th>NO. CLS.</th>
    <th>EXCL</th>
    <th>SEM</th>
    <th>DYN</th>
    <th>NON-C</th>
    <th>SUB-S</th>
  </tr>
  <tr>
    <td>Lauscher et al. (2022)</td>
    <td>function</td>
    <td>7</td>
    <td>✓</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Kunnath et al. (2022)</td>
    <td>function</td>
    <td>6</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Ferrod et al. (2021)</td>
    <td>intent</td>
    <td>5</td>
    <td>x</td>
    <td>✓</td>
    <td>(✓)</td>
    <td>(✓)</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Pride and Knoth (2020)</td>
    <td>purpose</td>
    <td>6</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Cohan et al. (2019)</td>
    <td>intent</td>
    <td>3</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Jurgens et al. (2018)</td>
    <td>function</td>
    <td>6</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Abu-Jbara et al. (2013)</td>
    <td>purpose</td>
    <td>6</td>
    <td>x</td>
    <td>x</td>
    <td>(✓)</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Athar and Teufel (2012)</td>
    <td>sentiment</td>
    <td>3</td>
    <td>x</td>
    <td>x</td>
    <td>(✓)</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Abu-Jbara and Radev (2012)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>✓</td>
    <td>(✓)</td>
    <td>(✓)</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Teufel et al. (2006)</td>
    <td>function</td>
    <td>11</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <th colspan="8">FINECITE (this work)</th>
  </tr>
  <tr>
    <td colspan="8">-</td>
  </tr>
</table>

Table 1: Structural comparison of previous work in computational linguistics on CCA (NO. CLS. = Number of classes, EXCL = Mutually exclusive labels, SEM = Semantic-based conceptualization, DYN = Dynamic context, NON-C = Non-contiguous context, SUB-S = Sub-sentence context)

The second common assumption is that \( S^* \) stretches continuously from the citation marker. Even though a notable number of publications technically allow for the extraction of non-contiguous contexts (Lauscher et al., 2022; Abu-Jbara et al., 2013; Athar and Teufel, 2012), only one study (Nambanoor Kunnath et al., 2022) particularly investigated the phenomenon. They directly compared a non-contiguous context window with a smaller contiguous version and found that the former slightly outperforms the latter.

Thirdly, \( S^* \) is often conceptualized with the sentence assumed to be the atomic unit of information (Cohan et al., 2019; Nambanoor Kunnath et al., 2022; Lauscher et al., 2022). In certain cases, however, this is not necessarily the case. Abu-Jbara and Radev (2012), for instance, shows evidently that sub-sentence segmentation is necessary to approximate \( S^* \) for sentences with multiple citations. While their focus lies on the multi-citation setting, we also observe sub-sentence context granularity in other settings.

Conceptual Restraints. Next to the restrictive assumptions imposed on the context span, there are conceptual restraints limiting the expressiveness of citation contexts. In nearly all previous work, the context is conceptualized as

\[
\hat{S}_C^* \approx \left\{ \arg \max_{s_i \in S'_i} P_{F_C}(c_i \mid s_i) \mid i \in I \right\},
\]

where

\[
S'_i = \{ s_i \in S_i \mid \exists c \in C : F_C(s_i) = c \}.
\]

This formulation captures that the context approximation \( \hat{S}_C^* \) only contains spans \( S' \) that have a clear association with a class in \( C \). In other words, the citation context is conceptualized based on the classification schema represented through \( F_C \) and not based on the semantic information of the text.

Most previous works additionally restrain their conceptualization by defining the relationship between \( S \) and \( C \) as mutually exclusive (Pride and Knoth, 2020; Cohan et al., 2019; Jurgens et al., 2018). This restricts the citation context further, as scientific discourse is faceted and can have multiple explanations (Lauscher et al., 2022). Lauscher et al. addressed this by creating the MULTIcite dataset, designed for multi-sentence, multi-function classification. They find that nearly one in five citations have at least two classes, with some reaching up to four. While this represents a step forward, it does not resolve the underlying limitation of defining citation context solely through the lens of the classification schema.

The only previous publication that defines a context based on semantic information from the vicinity of the citation marker is from Ferrod et al. (2021). They distinguish between the citation object and the context, where the former is the cited concept and the latter background information, or constraints on the citation object. While this goes in a similar direction to this work, their definition lags in completeness and only works on a subset of instances. To our knowledge, we are the first to propose a comprehensive citation context definition that is disjoint from the classification task.
3 Fine-Grained Citation Context

In this section, we propose and formalize our fine-grained context definition.

Semantic Dimensions. We base our context definition on previous research on argumentative structures in scientific texts. Teufel (2014) categorizes scientific argumentation along four principal dimensions: (i) statements about the author’s own work (citing paper), (ii) properties of existing solutions (cited papers), (iii) the relationships between existing solutions and the author’s contribution, and (iv) general properties of the research space. We apply this framework to the field of CCA and define the following three context dimensions.

The first dimension of the citation context is the information the citing author references from the cited paper. In the example

“Our paper extends the citation labeling scheme of <CITATION> and then reports similarities…”

the phrase, “the citation labeling scheme of <CITATION>,” describes here what information from the cited paper the author is referring to. This dimension highly correlates with (ii)—the properties of existing solutions, and is somewhat related to the citation object of Ferrod et al. (2021). In the following, we refer to this dimension as the Information Dimension (INF).

The second dimension describes the relationship between the citing and the cited work and corresponds to (iii) in Teufel (2014). In the excerpt

“Our paper extends the citation labeling scheme of <CITATION> and then reports similarities…”

the passage “our paper extends” describes how the author uses the cited information in their work. While use constitutes a substantial fraction of occurring relations, this dimension also includes other forms of perception, such as comparison, evaluation, and judgment. In the following, we refer to it as the Perception Dimension (PERC).

While these two dimensions cover the most critical aspects of a citation context,—what is cited and how is it perceived or used—they do not necessarily include the information of why the author chose to include a citation.

“Unlike recent language representation models <CITATION>, BERT is designed to pretrain deep bidirectional representations from…”

Here, the reason the author included this citation is to emphasize a novel property of the citing paper’s contribution. In Teufel’s (2014) framework, this falls under the semantic class (i)—properties of the citing work—and is neither considered in the INF nor the PERC dimension. In other instances, such a motivating factor could be related to a property of the research space (iv) or other direct citations (ii, iii). We categorize these passages, which explain why a citation was included, as the Background Dimension (BACK) of a citation.

Formal Definition. To formalize our fine-grained citation context, we expand upon Equation 2 by removing the task dependency and incorporating semantic dimensions outlined above. Specifically, we define the task-independent, approximately optimal citation context \( \hat{S}^* \) as:

\[
\hat{S}^* := \{ s_i^* \mid i \in I \},
\]

where

\[
s_i^* = \{ s_i \in S_i \mid \exists d \in D : F_D(s_i) = d \},
\]

\[
D = \{ \text{INF}, \text{PERC}, \text{BACK} \}
\]

is the set of semantic dimensions defined in this Section, and \( F_D \) represents the semantic relationship of the surrounding text to the citation.

We further formalize three structural properties of citation context spans \( s^* \in \hat{S}^* \):

• **Dynamicity:** The length \( |s^*| \) is dynamic and adapts to the situated structure of the enclosing argumentation.

• **Non-Contiguity:** \( s^* \) may consist of multiple disjoint spans allowing for skip-structured selection of relevant information.

• **Sub-Sentence Granularity:** \( s^* \) is constructed on sub-sentence granularity, enabling a fine-grained representation of the argumentative structure.

These properties collectively define a flexible and semantically motivated citation context that diverges from the constrained approximations of previous works. We provide a detailed empirical discussion of their relevance in Section 4.2 and 5.

4 FineCite Corpus

Using the definition in Section 3, we create the FineCite corpus. With the dataset creation, we aim to (i) assess whether the theoretical framework
practically applies to scientific texts, (ii) investigate the assumption on the semantic dimensions and structure of citation contexts, and (iii) create a resource for the evaluation of the framework on established CCA Benchmarks.

4.1 Dataset Construction
We construct the corpus in the following steps.

Step 1: Procurement. The FineCite dataset was built from a subset of ACL Anthology Network Corpus (Radev et al., 2009). The ACL Anthology Network contains over 80K papers from several ACL conferences and other venues in computational linguistics. We extracted the full paper text, including citations, using GROBID (GROBID, 2024). Documents containing faulty meta-information, languages other than English, and miscellaneous documents with <3 sections and <5 references were excluded. From the remaining documents, we sampled 1,056 paragraphs, each containing one citation marker highlighted for annotation.

Step 2: Guideline creation. The annotation guidelines comprise best practices and rules for the context annotation. The instructions were created based on the definition presented in Section 3 and further iteratively refined to better handle ambiguous cases. For each iteration, several annotators completed five to ten tasks separately and subsequently discussed differences. Afterwards, the guidelines were updated to reduce ambiguity for the next iteration. In total, five iterations were performed. The complete Annotation Guidelines can be found in Appendix E.

Step 3: Annotation. The annotation was performed for each paragraph separately. The annotator was asked to read the paragraph carefully and annotate the context of the highlighted citation based on the guidelines. To provide further information in case of ambiguity, additional information, like the surrounding paragraphs and metadata about citing and cited papers, was provided in the annotation tool. A detailed description of the annotation platform is provided in Appendix A.

All annotators had previous experience with scientific literature and were carefully trained on the Annotation Guidelines. The compensation followed locally typical rates for research assistants.

Step 4: Validation. To ensure the annotation quality, we monitored several inter-annotator agreement (IAA) metrics on 10% of the annotations. We measured both F-measure commonly used for span annotations with open bounds (Hripcsak and Rothschild, 2005), and Cohens \( \kappa \) (Cohen, 1960) for the agreement on label assignment above that expected by chance. To capture different aspects of the annotation process separately, we provide IAA for the whole context (\( F1_{total} \)), and for each scope separately (\( F1_{inf}, F1_{perc}, F1_{back} \)). The \( F1_{macro} \) is the mean over \( F1_{inf}, F1_{perc}, \) and \( F1_{back} \). While the metrics follow the standard definition, we provide a formal definition in Appendix B.

The \( F1_{total} \) after the annotation process was 0.75, indicating an overall good agreement. The separate IAA on the context dimensions, however, is considerably lower. While the \( F1_{inf} \) is with a score of 0.65 the highest, the \( F1_{perc} \) is at 0.42 and the \( F1_{back} \) at 0.34. The \( F1_{macro} \) lies at 0.48 and the \( \kappa \) on the validation samples was 0.55.

While these values are in the typical range for annotation of scientific literature (Lauscher et al., 2022; Ferrod et al., 2021; Lauscher et al., 2018), they highlight the task complexity. The moderate \( F1_{macro} \), despite a rather high \( F1_{total} \), indicates that while annotators often struggle to clearly distinguish between the dimensions, they have a good sense of what constitutes relevant information. PERC and BACK seem especially ambiguous.

4.2 Corpus Statistics
The FineCite corpus contains 1,056 fine-grained citation contexts for paragraphs from 72 scientific papers. Overall, INF accounts for 27%, PERC for 35%, and BACK for 38% of the annotated words. The average context length is ~45 words and is approximately normally distributed with a long tail toward the upper end. The main contribution to the longer contexts is the BACK dimension. While BACK comprises about 8 words (30%) in contexts shorter than 40 words, it expands to an average of 54 words (43%) in contexts exceeding 100 words. Combined with the low agreement score on BACK, this might indicate that a clearer delimitation of the dimension might be helpful. Figure 2 provides an expanded visualization of the corpus statistics.

To evaluate the context span properties presented in Section 3, we apply context restrictions commonly used in prior work to the contexts in FineCite and compare them to the fine-grained gold labels. We report F1-score and %-Match met-
Figure 2: Results of statistical analysis of the FineCITE dataset, showing the variation of context length and its interrelation with label distribution.

<table>
  <tr>
    <th>RESTRICTION</th>
    <th>F1-Score</th>
    <th>%-Match</th>
  </tr>
  <tr>
    <td>One Sentence</td>
    <td>0.679</td>
    <td>30.4</td>
  </tr>
  <tr>
    <td>Two Sentences</td>
    <td>0.716</td>
    <td>23.6</td>
  </tr>
  <tr>
    <td>Four Sentences</td>
    <td>0.704</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>Contiguous</td>
    <td>0.863</td>
    <td>64.6</td>
  </tr>
  <tr>
    <td>Sentence Segments</td>
    <td>0.951</td>
    <td>70.7</td>
  </tr>
</table>

Table 2: Experiments on context restrictions compared to the gold context from FineCITE

rics for fixed-size windows of one, two, and four sentences, as well as for the longest contiguous sub-context and contexts extended to the next sentence boundary. We ignore dimension classes to highlight the structural properties and allow a two-word tolerance in the %-Match metric. Results are shown in Table 2.

Restricting the context to a fixed number of sentences results in a considerable error in both the F1-score and %-Match. The %-Match scores suggest that single-sentence contexts offer the best performance among fixed-size context windows; however, they fall short of capturing a majority of instances. Contiguity exhibits a minor error compared to fixed context windows, indicating that non-contiguity occurs less, and non-contiguous segments are rather small in size. Surprisingly, the total F1-score error induced through sentence segmentation is relatively small. For the assignment of fine-grained dimension labels, sub-sentence segmentation is, however, a necessary property.

Overall, the results affirm the significance of the three structural assumptions—sub-sentence segmentation, non-contiguity, and dynamic context—for a fine-grained citation context extraction.

5 Experiments

In this section, we evaluate the FineCITE dataset on two tasks: (i) extraction of fine-grained citation contexts, and (ii) citation classification on standard CCA benchmarks using fine-grained context information extracted in (i).

5.1 Citation Context Extraction

Ensuring that common extraction models can reliably learn to identify citation contexts is crucial for the effective application of the presented fine-grained context definition.

Data preparation. We utilized the same samples used in the evaluation of the annotation process as the test set, with the remaining samples reserved for training and validation. We evaluate extraction on (i) uniform token labels and (ii) commonly used IOB (Inside–Outside–Beginning) labels.

Extraction model. For all extraction approaches, we use a SCIBERT (Beltagy et al., 2019) encoder model. SCIBERT is a BERT-like encoder-only transformer, pre-trained on scientific literature. To cover several common sequence extraction approaches, we evaluate three different classification heads: a linear, a Bi-LSTM (Hochreiter and Schmidhuber, 1997), and a conditional random field (CRF) (Lafferty et al., 2001) classifier.

Experiment setup. We used the pre-trained weights of SCIBERT from huggingface transformers (Wolf et al., 2020) and finetuned the whole model (encoder + cls-head) using AdamW (Loshchilov and Hutter, 2019) with a
<table>
  <tr>
    <th>APPROACH</th>
    <th>F1<sub>total</sub></th>
    <th>F1<sub>macro</sub></th>
  </tr>
  <tr>
    <th colspan="3">Inter Annotator Agreement</th>
  </tr>
  <tr>
    <td>Human (Annotation)</td>
    <td>0.75</td>
    <td>0.48</td>
  </tr>
  <tr>
    <th colspan="3">Extraction Task</th>
  </tr>
  <tr>
    <td>SciBERT w. Linear</td>
    <td>0.77</td>
    <td>0.557</td>
  </tr>
  <tr>
    <td>SciBERT w. BiLSTM</td>
    <td>0.759</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>SciBERT w. CRF</td>
    <td>0.787</td>
    <td>0.521</td>
  </tr>
</table>

Table 3: Extraction results on the FINECITE dataset

linear warm-up ratio of 5% and a peak learning rate of 5e-5. All models were fine-tuned using early stopping with a patience of three epochs, a batch size of 4, and a dropout rate of 0.1. To address class imbalance, we additionally applied weighted cross-entropy loss. The training was conducted on an NVIDIA A100 GPU. We evaluated the F1 scores described in Section 4.1.

Result. Table 3 shows the results of \( F1_{total} \) and \( F1_{macro} \). See Appendix D for extended results. We observe that all three extraction approaches reach higher F1 scores than those measured during the annotation process. The variance between the different classifiers is rather small. The CRF classifier exhibits the highest \( F1_{total} \) score of 0.787, while the Bi-LSTM classifier dominates the \( F1_{macro} \) metric with 0.56. The linear classifier achieves an \( F1_{macro} \) of 0.557 and an \( F1_{total} \) of 0.77, only slightly lower than the other approaches. The best results were achieved using IOB labels for linear and Bi-LSTM classifiers, whereas the CRF classifier worked better with uniform labels.

5.2 Citation Context Classification

To showcase the benefits of fine-grained contexts in a competitive setting, we provide a broad comparison with previous work using the citation classification task.

Data. We evaluate fine-grained contexts on four commonly used CCA benchmarks.

• ACL-ARC (Jurgens et al., 2018) comprises 1,933 labeled citations following a six-label classification schema introduced in the paper. All samples originate exclusively from the computational linguistics domain.

• ACT2 (N. Kunnath et al., 2021) is a larger, mixed-domain collection with 4,000 annotated citations labeled with the schema used for the ACL-ARC dataset.

• SciCITE (Cohan et al., 2019), also multi-domains, contains 11,000 samples, annotated with a simple three-class annotation schema.

• MultiCite (Lauscher et al., 2022) is a multi-sentence, multi-label dataset annotated with seven citation function classes based on the scheme used in ACL-ARC. With 12,653 annotated citations, it is the biggest dataset.

Although ACL-ARC and ACT2 are primarily modeled using the citing sentence alone, we perform extraction on an extended window containing multiple sentences before and after the citation. SciCITE does not provide text exceeding the citing sentence, which drastically restricts the extraction of our fine-grained context.

To reduce the model’s tendency to memorize author names, we conceal the targeted and other citations behind <TARGET_CITATION/> and <CITATION/> tags, respectively. Each dataset is divided into approximately 85% training and 15% testing. For the FINECITE approaches, we extract the fine-grained context using the extraction approach presented in Section 5.1.

Classification model. We considered four baselines for the classification task: (i) the scaffolding approach presented in Cohan et al. (2019), (ii) the best-performing citation classification model from the 3C classification task 2021 (N. Kunnath et al., 2021)—a SciBERT model with a linear classification head (Maheshwari et al., 2021), (iii) GPT-4o (Achiam et al., 2023), and (iv) SciTULU 70B (Wadden et al., 2024)—an instruction-tuned LLM for scientific literature. (i) and (ii) were fine-tuned on the training split, and (iii) and (iv) were evaluated in a zero-shot setting. The FINECITE approaches use SciBERT(Beltagy et al., 2019) embeddings and a linear classification head similar to (ii). Instead of using CLS pooling, we use mean pooling over tokens belonging to the same dimension. The resulting dimension embeddings were concatenated and passed to the classification head.

Experiment setup. We utilized the pre-trained SciBERT weights as mentioned above. The best performance was achieved using AdamW (Loshchilov and Hutter, 2019), early stopping, and a linear warm-up of 5%. The training was
<table>
  <tr>
    <th rowspan="2">APPROACH</th>
    <th colspan="3">ACL-ARC</th>
    <th colspan="3">ACT2</th>
    <th colspan="3">SciCITE</th>
    <th colspan="3">MULTICITE</th>
    <th rowspan="2">MEAN</th>
  </tr>
  <tr>
    <th>macro</th><th>st. dev.</th><th></th>
    <th>macro</th><th>st. dev.</th><th></th>
    <th>macro</th><th>st. dev.</th><th></th>
    <th>macro</th><th>st. dev.</th><th></th>
  </tr>
  <tr>
    <td colspan="14"><i>Baseline Approaches</i></td>
  </tr>
  <tr>
    <td>SCAFFOLDS</td><td>0.377</td><td>0.067</td><td>0.205</td><td>0.026</td><td>0.821</td><td>0.010</td><td>0.409</td><td>0.036</td><td>0.453</td>
  </tr>
  <tr>
    <td>SciBERT</td><td>0.517</td><td>0.018</td><td>0.242</td><td>0.012</td><td>0.841</td><td>0.005</td><td>0.584</td><td>0.006</td><td>0.546</td>
  </tr>
  <tr>
    <td>GPT 4o</td><td>0.401</td><td>-</td><td>0.117</td><td>-</td><td>0.766</td><td>-</td><td>0.434</td><td>-</td><td>0.43</td>
  </tr>
  <tr>
    <td>SciTULU 70B</td><td>0.37</td><td>-</td><td>0.114</td><td>-</td><td>0.783</td><td>-</td><td>0.353</td><td>-</td><td>0.405</td>
  </tr>
  <tr>
    <td colspan="14"><i>FINECITE Approaches</i></td>
  </tr>
  <tr>
    <td>SciBERT (Linear)</td><td>0.572</td><td>0.018</td><td>0.302</td><td>0.02</td><td>0.84</td><td>0.002</td><td>0.603</td><td>0.021</td><td>0.579</td>
  </tr>
  <tr>
    <td>SciBERT (BiLSTM)</td><td>0.584</td><td>0.014</td><td>0.282</td><td>0.014</td><td>0.845</td><td>0.003</td><td>0.601</td><td>0.005</td><td>0.578</td>
  </tr>
  <tr>
    <td>SciBERT (CRF)</td><td>0.563</td><td>0.007</td><td>0.274</td><td>0.024</td><td>0.841</td><td>0.002</td><td>0.606</td><td>0.010</td><td>0.571</td>
  </tr>
</table>

Table 4: Results of the citation classification task on the four benchmarks ACL-ARC, ACT2, SciCITE, and MULTICITE. The standard deviation (st. dev.) is calculated over five consecutive seeds.

conducted on an NVIDIA A100 GPU. The optimal learning rate, batch size, and dropout for each dataset are provided in Appendix C. For all fine-tuned models, the performance was evaluated over five consecutive seeds.

Result. Table 4 exhibits the macro-F1 and standard deviation for each dataset. Detailed results including class scores are shown in Appendix D.

Among the baselines, SciBERT achieves the highest average macro-F1 (0.546), followed by the SCAFFOLDS approach (0.453). Both GPT-4o (0.43) and SciTULU 70B (0.405) perform lower. These results show that finetuned encoder models have a considerably better conceptualization of the citation task than LLMs in a zero-shot setting. We further observe that the SCAFFOLDS approach exhibits a high standard deviation on the ACL-ARC tasks, as it struggles to predict minority labels correctly on the smaller dataset.

The FINECITE models introduced in this work outperform the baselines across all datasets. Among them, the context extracted with the Linear classification head achieves the best overall performance, with an average macro-F1 of 0.579. The context from the BiLST and CRF classifier only perform slightly lower with an average macro-F1 of 0.574 and 0.571, respectively. Comparing the performance on a per-dataset basis reveals a more nuanced pattern. The largest increase can be observed on the ACT2 dataset with a 25% increase over the strongest baseline, followed by a 13% increase on the ACL-ARC dataset. We explain the relatively low performance increases on MULTICITE by considering that the dataset already provides a dynamic context, leaving limited advantage for fine-grained contexts. The performance on the SciCITE benchmark further stresses that for the extraction of comprehensive fine-grained context, the citing sentence is not sufficient.

Overall, the results demonstrate that the fine-grained citation context proposed in this work captures a more comprehensive citation representation than other conceptualizations in previous work.

<table>
  <tr>
    <th rowspan="2">APPROACH</th>
    <th colspan="2">ACL-ARC</th>
    <th colspan="2">ACT2</th>
  </tr>
  <tr>
    <th>macro</th><th>st. dev.</th>
    <th>macro</th><th>st. dev.</th>
  </tr>
  <tr>
    <td colspan="5"><i>Context Dimensions</i></td>
  </tr>
  <tr>
    <td>w/o INF</td><td>0.556</td><td>0.017</td><td>0.277</td><td>0.013</td>
  </tr>
  <tr>
    <td>w/o PERC</td><td>0.563</td><td>0.019</td><td>0.259</td><td>0.036</td>
  </tr>
  <tr>
    <td>w/o BACK</td><td>0.56</td><td>0.019</td><td>0.253</td><td>0.024</td>
  </tr>
  <tr>
    <td colspan="5"><i>Mean Pooling</i></td>
  </tr>
  <tr>
    <td>Dimensions</td><td>0.584</td><td>0.014</td><td>0.302</td><td>0.02</td>
  </tr>
  <tr>
    <td>Weighted<sub>tok</sub></td><td>0.542</td><td>0.013</td><td>0.281</td><td>0.019</td>
  </tr>
  <tr>
    <td>Weighted<sub>dim</sub></td><td>0.573</td><td>0.015</td><td>0.28</td><td>0.015</td>
  </tr>
</table>

Table 5: Ablation on context dimensions and pooling

Ablation. We provide ablation on the context dimensions, pooling method, and domain shift for a further analysis of the proposed fine-grained citation contexts. The dimension and pooling ablation were done on the ACL-ARC and ACT2 datasets. We create two new datasets (\( ACT2, ACT2_D \)) for the evaluation on domain shift.

With the ablation on the citation dimensions (Table 5) we investigate the significance of the INF, PERC, and BACK dimensions for classification performance. Our analysis shows that removing any of the three citation dimensions leads to a performance drop for both datasets. While the decrease in performance on the ACL dataset is similar for all three dimensions, for the ACT2 benchmark PERC
<table>
  <tr>
    <th>APPROACH</th>
    <th>ACT2'</th>
    <th>ACT2_D</th>
    <th>Δ</th>
    <th>%</th>
  </tr>
  <tr>
    <td>SciBERT</td>
    <td>0.345</td>
    <td>0.228</td>
    <td>-0.117</td>
    <td>-33.9%</td>
  </tr>
  <tr>
    <td>FINECITE</td>
    <td>0.404</td>
    <td>0.263</td>
    <td>-0.141</td>
    <td>-34.9%</td>
  </tr>
  <tr>
    <th>Difference in Difference</th>
    <th></th>
    <th></th>
    <th>-0.024</th>
    <th>-1.0%<sup>2</sup></th>
  </tr>
</table>

Table 6: Ablation on Domain shift.

and BACK exert greater influence. This highlights that despite the low extraction performance, PERC and BACK contain essential information for the citation classification task.

The ablation on pooling strategies (Table 5) evaluates whether pooling citation dimensions separately improves performance over simpler alternatives. We compare this approach to token-weighted pooling, which ignores citation dimensions, and a dimension-weighted method. On both datasets, separate dimension pooling yields better results. Although the performance gap is modest, it indicates that modeling citation dimensions individually enhances representation quality, reinforcing the value of our context definition.

As the FINECITE dataset only consists of samples from the computational linguistics domain, there might be a domain bias in the context extraction. To evaluate whether this compromises domain adaptation performance on the classification task, we provide an ablation on two new datasets (ACT2', ACT2_D) constructed from the multi-domain ACT2 benchmark (Table 6). The ACT2_D contains samples from computational linguistics and STEM domains in its training split, and samples from medicine and social sciences in its test split, thus evaluating domain adaptation. The ACT2', on the other hand, contains samples from all domains in both splits while following the same split sizes. We provide the macro-F1 results on the test set for the strongest baseline and our approach, and analyze the difference-in-difference estimator between the two approaches.

For both models, we observe a substantial drop in performance when evaluated out-of-domain. Our approach retains a slightly larger margin, leading to a negative difference-in-difference estimate of -0.024. Despite this indicating that our model approach performs slightly worse on domain adaptation, the performance gains of using fine-grained contexts outweigh this drawback in overall effectiveness.

<sup>2</sup>Percentage Points

6 Conclusion

In this paper, we introduced a novel approach to defining citation contexts, aiming to foster new research in citation context analysis. We proposed a conceptual framework that characterizes citation context based on semantic dimensions and structural properties. Subsequently, we described the curation of the FINECITE corpus—a first resource for fine-grained citation contexts—and analyzed core statistics. Our experiments demonstrated that our context definition is practically applicable and leads to improved performance on established CCA benchmarks compared to state-of-the-art methods.

In future work, we will focus on expanding the dataset to a wider range of scientific texts and domains and further refining our context definition. Additionally, we plan to explore applications, such as retrieval-augmented generation (RAG) (Lewis et al., 2020; Edge et al., 2024) and question-answering (Q&A) frameworks (Lauscher et al., 2022; Dasigi et al., 2021), to support interactive exploration of scientific argumentation.

Limitations

This work presents the first dataset of its kind, albeit with limitations in both size and domain coverage. The accompanying evaluation and analysis should be understood within this restricted scope and may not generalize to broader contexts. The objective is to establish a comprehensive definition of citation contexts and provide a resource and baseline for further analysis. Additionally, although our context definition is intended to be task-independent, our evaluation is limited to a subset of tasks due to constraints in space and resources.

Acknowledgments

We thank Prof. Seonghyeon Lee for his insightful comments, and Hwanseong Joo and Jooyoung Yoon for their valuable feedback. We also acknowledge Joel Thomas, Minjoon Jin, and Jialun Zheng for their initial contributions. This study was supported by the NRF grant (RS-2018-NR031059) and the BK21 FOUR program (41202420214871), both funded by the Ministry of Education of Korea. Access to the A100 GPU used in the experiments was provided by the Department of Data Convergence Computing at Kyungpook National University, where Young-Kyoon Suh serves as an adjunct professor.
References

Amjad Abu-Jbara, Jefferson Ezra, and Dragomir Radev. 2013. Purpose and polarity of citation: Towards NLP-based bibliometrics. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 596–606, Atlanta, Georgia. Association for Computational Linguistics.

Amjad Abu-Jbara and Dragomir Radev. 2012. Reference scope identification in citing sentences. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 80–90, Montréal, Canada. Association for Computational Linguistics.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altshenmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.

Awais Athar and Simone Teufel. 2012. Detection of implicit citations for sentiment detection. In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, pages 18–26, Jeju Island, Korea. Association for Computational Linguistics.

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33:1877–1901.

Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics.

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37 – 46.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599–4610, Online. Association for Computational Linguistics.

Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. ArXiv, abs/2404.16130.

Roger Ferrod, Luigi Di Caro, and Claudio Schifanella. 2021. Structured Semantic Modeling of Scientific Citation Intents. In Extended Semantic Web Conference.

Eugene Garfield. 1972. Citation analysis as a tool in journal evaluation. Science, 178(4060):471–479.

GROBID. 2024. GROBID: A Machine Learning Software for Extracting Information from Scholarly Documents. https://github.com/kermitt2/grobid.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780.

George Hripcsak and Adam S. Rothschild. 2005. Technical brief: Agreement, the f-measure, and reliability in information retrieval. Journal of the American Medical Informatics Association : JAMIA, 12 3:296–8.

David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391–406.

Suchetha N. Kunmath, Drahomira Herrmannova, David Pride, and Petr Knoth. 2022. A meta-analysis of semantic classification of citations. Quantitative Science Studies, 2(4):1170–1215.

John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, page 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.

Anne Lauscher, Goran Glavaš, and Simone Paolo Ponzetto. 2018. An argument-annotated corpus of scientific publications. In Proceedings of the 5th Workshop on Argument Mining, pages 40–46, Brussels, Belgium. Association for Computational Linguistics.

Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Kai Eckert. 2017. Investigating convolutional networks and domain-specific embeddings for semantic classification of citations. In Proceedings of the 6th International Workshop on Mining Scientific Publications, WOSP 2017, page 24–28, New York, NY, USA. Association for Computing Machinery.
Anne Lauscher, Brandon Ko, Bailey Kuehl, Sophie Johnson, Arman Cohan, David Jurgens, and Kyle Lo. 2022. MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1875–1889, Seattle, United States. Association for Computational Linguistics.

Patrick Lewis, Ethan Perez, Aleksandara Pikutis, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. ArXiv, abs/2005.11401.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101.

Himanshu Maheshwari, Bhavyajeet Singh, and Vasudeva Varma. 2021. SciBERT sentence representation for citation context classification. In Proceedings of the Second Workshop on Scholarly Document Processing, pages 130–133, Online. Association for Computational Linguistics.

Suchetha N. Kunath, David Pride, Drahomira Hermannova, and Petr Knoth. 2021. Overview of the 2021 SDP 3C citation context classification shared task. In Proceedings of the Second Workshop on Scholarly Document Processing, pages 150–158, Online. Association for Computational Linguistics.

Suchetha Nambanoor Kunath, David Pride, and Petr Knoth. 2022. Dynamic context extraction for citation classification. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 539–549, Online only. Association for Computational Linguistics.

David Pride and Petr Knoth. 2020. An authoritative approach to citation classification. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL ’20, page 337–340, New York, NY, USA. Association for Computing Machinery.

Dragomir R. Radev, Pradeep Muthukrishnan, and Vahed Qazvinian. 2009. The ACL Anthology network corpus. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), pages 54–61, Suntec City, Singapore. Association for Computational Linguistics.

Zeren Shui, Petros Karypis, Daniel S. Karls, Mingjian Wen, Saurav Manchanda, Ellad B. Tadmor, and George Karypis. 2024. Fine-tuning language models on multiple datasets for citation intention classification. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16718–16732, Miami, Florida, USA. Association for Computational Linguistics.

John Swales. 1986. Citation Analysis and Discourse Analysis. Applied Linguistics, 7(1):39–56.

Simone Teufel. 2014. Scientific argumentation detection as limited-domain intention recognition. In Workshop on Frontiers and Connections between Argumentation Theory and Natural Language Processing.

Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 103–110, Sydney, Australia. Association for Computational Linguistics.

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems.

David Wadden, Kejian Shi, Jacob Daniel Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hanna Hajishirzi, and Arman Cohan. 2024. Sciriff: A resource to enhance language model instruction-following over scientific literature. ArXiv, abs/2406.07835.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. Preprint, arXiv:1910.03771.
A Annotation Interface

Figure 3 shows the annotation tool with an annotated example and different features, facilitating an efficient context annotation.

B Inter Annotator Agreement

The F-measure for IAA is calculated by

\[
F1 = \frac{2 \times precision \times recall}{precision + recall},
\]

where \( precision \) refers to the proportion of agreement on the annotation of annotator 1 and \( recall \) refers to the proportion of agreement on the annotation of annotator 2.

The three specific F-scores measure agreement on one distinct scope. More specifically, \( F1_{inf} \) relates to the information, \( F1_{perc} \) to the perception, and \( F1_{back} \) to the background scopes, respectively.

The aggregate metric, \( F1_{macro} \), is a macro F-score of the three context scopes:

\[
F1_{macro} = \frac{F1_{inf} + F1_{perc} + F1_{back}}{3}.
\]

The \( F1_{macro} \) measures the average class-specific agreement at one annotation task.

The second aggregate IAA is \( F1_{total} \), for which we ignore the scope classifications and only compare the agreement on the whole annotated area of the two annotators, represented by \( precision_{total} \) and \( recall_{total} \).

\[
F1_{total} = \frac{2 \times precision_{total} \times recall_{total}}{precision_{total} + recall_{total}}.
\]

The \( F1_{total} \) metric evaluates the class-unspecific agreement at one particular annotation task.

With Cohen’s Kappa (\( \kappa \)), we measure agreement on the label assignment for mutually annotated areas. We follow the common definition of

\[
\kappa = \frac{p_o - p_e}{1 - p_e},
\]

where \( p_o \) is the proportion of agreement and \( p_e \) is the expected proportion of agreement expected by chance.

C Hyperparamethers for the classification task

We explored the following hyperparameters for both baseline tasks, respectively.

Table 7 shows the hyperparameters (batch size, learning rate, dropout) that resulted in the optimal classification results for the ACL-ARC, ACT2, SCIcite, and MULTIcite datasets, respectively.

<table>
  <tr>
    <th>Dataset</th>
    <th>batch size</th>
    <th>learning rate</th>
    <th>dropout</th>
  </tr>
  <tr>
    <td>ACL-ARC</td>
    <td>4</td>
    <td>5e-05</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>ACT2</td>
    <td>16</td>
    <td>3e-05</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>SCIcite</td>
    <td>16</td>
    <td>3e-05</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>MULTIcite</td>
    <td>8</td>
    <td>5e-05</td>
    <td>0.1</td>
  </tr>
</table>

Table 7: Hyperparameters of each dataset

D Extended Results

The following tables show extended evaluation results. Table 8 shows the extended extraction results on the FineCite dataset. Tables 9, 10, 11, and 12 show the extended classification results for ACL-ARC, ACT2, SCIcite, and MULTIcite respectively.

E Annotation Guidelines

E.1 Introduction

We want to annotate the citation context of references in scientific literature to build a database for the training of an automatic citation context extraction model.

The scope of the annotation is to mark the context of a citation in a given paragraph. As the citation context, we understand the citation surrounding sentence segments that semantically relate to the target reference.

We use an online platform that supports the annotation process in its structure and functionality. In the following paragraphs, we describe the annotation task and briefly introduce the annotation platform.

E.2 The Task

E.2.1 What does the annotation task look like?

The task is to classify words of several sentences in the same paragraph and determine whether they relate to the citation marked as the target reference. An example of the annotation task might look like this:

Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing the modeling of dependencies without regard to their distance in the input or output sequences [GREF]. In all but a few cases [TREF], however, such attention mechanisms are
During the fine-tuning phase, we first apply finetuning on a small spoken corpus. For better domain adaptation, we adopt mixed fine-tuning [REF] , which trains on a mixed dataset that includes a subsampled general corpus and an upsampled spoken corpus.

Thirdly, we propose a method called “in-domain mixed fine-tuning”, which further improve the BLEU score than mixed finetuning.

Specifically, inspired by in-domain data filtering [GTREF] , we mixed upsampled spoken data with selected in-domain data from general corpus rather than random subsampled.

Figure 3: The Annotation Interface: Located on the left is the annotation toolbar, with the color-coded marker for each context scope, an ERASE tool, and the RESET button. The center is the working area where the annotation task is displayed and annotated. On the right side, meta-information regarding the citing and cited papers is provided, and alternatively, a comment section can be accessed to leave questions or notes. The navigation bar on the bottom gives (from left to right) access to the annotation guidelines, the comment section, and three buttons for returning to the previous task, skipping, or submitting the current task.

<table>
  <tr>
    <th>APPROACH</th>
    <th>F1<sub>macro</sub></th>
    <th>F1<sub>total</sub></th>
    <th>F1<sub>inf</sub></th>
    <th>F1<sub>perc</sub></th>
    <th>F1<sub>back</sub></th>
  </tr>
  <tr>
    <th colspan="6">Inter Annotator Agreement</th>
  </tr>
  <tr>
    <td>HUMAN (annotation)</td>
    <td>0.483</td>
    <td>0.758</td>
    <td>0.654</td>
    <td>0.416</td>
    <td>0.338</td>
  </tr>
  <tr>
    <th colspan="6">Extraction Task</th>
  </tr>
  <tr>
    <td>SciBERT w. Linear</td>
    <td>0.557</td>
    <td>0.771</td>
    <td>0.755</td>
    <td>0.495</td>
    <td>0.422</td>
  </tr>
  <tr>
    <td>SciBERT w. BiLSTM</td>
    <td>0.56</td>
    <td>0.759</td>
    <td>0.768</td>
    <td>0.496</td>
    <td>0.415</td>
  </tr>
  <tr>
    <td>SciBERT w. CRF</td>
    <td>0.521</td>
    <td>0.787</td>
    <td>0.738</td>
    <td>0.434</td>
    <td>0.391</td>
  </tr>
</table>

Table 8: Extended extraction results on the FINECITE Dataset.

used in conjunction with a recurrent network.

E.2.2 What is the meaning of the tags?

Four different types of tags can occur in the annotation task ([REF],[GREF],[TREF],[GTREF]). The ‘REF’ part of the tag generally refers to ‘Reference’, meaning that each tag is some kind of placeholder for one or multiple references. More particularly, the ‘[REF]’ tag replaces one single reference (e.g. (Goodfellow 2012) → [REF]), and the [GREF] tag replaces a Group of References (e.g. (Cohan et al. 2018, Jha et al. 2016) → [GREF]). Further, there are two different versions of the [REF] and the [GREF] tag, which indicate that they are the Target of the annotation task. The ‘T’ in the [TREF] and the [GTREF] tag means Target. Each annotation task will have only one target reference, but multiple other single or group references might exist.

E.3 What is the citation context?

The citation context is the text span in the citing document that describes the information used from the cited document, the way it is used, and how the author of the citing document perceives it. For the annotation process, we distinguish between three scopes:

• Citation information scope: describes the information of the cited document. It answers the question of what is cited. [GREEN]

• Citation perception scope: describes in what way the author perceived, used, or further analyzed the document. It answers the question of how something is cited. [YELLOW]

• Citation background scope: describes additional information required for putting the two previous scopes into the context they are used.
<table>
  <tr>
    <th rowspan="2">APPROACH</th>
    <th colspan="3">BACKGR.</th>
    <th colspan="3">COMPARE</th>
    <th colspan="3">EXTENSION</th>
    <th colspan="3">FUTURE</th>
    <th colspan="3">MOTIVATION</th>
    <th colspan="3">USE</th>
    <th colspan="3">MACRO</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
  </tr>
  <tr>
    <td colspan="21"><i>Baseline Approaches</i></td>
  </tr>
  <tr>
    <td>SCAFFOLDS</td><td>.682</td><td>.764</td><td>.720</td><td>.551</td><td>.311</td><td>.393</td><td>.285</td><td>.138</td><td>.177</td><td>.095</td><td>.160</td><td>.180</td><td>.147</td><td>.240</td><td>.180</td><td>.615</td><td>.745</td><td>.673</td><td>.396</td><td>.393</td><td>.377</td>
  </tr>
  <tr>
    <td>SciBERT</td><td>.754</td><td>.849</td><td>.798</td><td>.613</td><td>.368</td><td>.460</td><td>.755</td><td>.807</td><td>.780</td><td>.475</td><td>.012</td><td>.317</td><td>.196</td><td>.440</td><td>.272</td><td>.395</td><td>.600</td><td>.476</td><td>.534</td><td>.550</td><td>.517</td>
  </tr>
  <tr>
    <td>GPT 4o</td><td>.750</td><td>.677</td><td>.712</td><td>.393</td><td>.667</td><td>.494</td><td>.000</td><td>.000</td><td>.000</td><td>.400</td><td>.667</td><td>.500</td><td>.000</td><td>.000</td><td>.000</td><td>.776</td><td>.634</td><td>.898</td><td>.387</td><td>.441</td><td>.401</td>
  </tr>
  <tr>
    <td>SciTULU</td><td>.464</td><td>.684</td><td>.533</td><td>.661</td><td>.529</td><td>.587</td><td>.000</td><td>.000</td><td>.000</td><td>.400</td><td>.667</td><td>.500</td><td>.000</td><td>.000</td><td>.000</td><td>.862</td><td>.476</td><td>.613</td><td>.398</td><td>.393</td><td>.376</td>
  </tr>
  <tr>
    <td colspan="21"><i>FINECITE Approaches</i></td>
  </tr>
  <tr>
    <td>SciBERT (Linear)</td><td>.775</td><td>.804</td><td>.789</td><td>.727</td><td>.489</td><td>.582</td><td>.415</td><td>.213</td><td>.265</td><td>.566</td><td>.760</td><td>.633</td><td>.190</td><td>.440</td><td>.263</td><td>.714</td><td>.852</td><td>.775</td><td>.565</td><td>.593</td><td>.551</td>
  </tr>
  <tr>
    <td>SciBERT (BiLSTM)</td><td>.799</td><td>.800</td><td>.798</td><td>.692</td><td>.579</td><td>.625</td><td>.432</td><td>.225</td><td>.281</td><td>.524</td><td>.880</td><td>.638</td><td>.360</td><td>.480</td><td>.341</td><td>.795</td><td>.848</td><td>.819</td><td>.600</td><td>.635</td><td>.584</td>
  </tr>
  <tr>
    <td>SciBERT (CRF)</td><td>.811</td><td>.787</td><td>.797</td><td>.740</td><td>.496</td><td>.591</td><td>.341</td><td>.250</td><td>.264</td><td>.516</td><td>.880</td><td>.649</td><td>.206</td><td>.520</td><td>.282</td><td>.726</td><td>.876</td><td>.792</td><td>.557</td><td>.635</td><td>.563</td>
  </tr>
</table>

Table 9: Extended results of the citation classification task on ACL-ARC.

<table>
  <tr>
    <th rowspan="2">APPROACH</th>
    <th colspan="3">BACKGR.</th>
    <th colspan="3">COMPARE</th>
    <th colspan="3">EXTENSION</th>
    <th colspan="3">FUTURE</th>
    <th colspan="3">MOTIVATION</th>
    <th colspan="3">USE</th>
    <th colspan="3">MACRO</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
  </tr>
  <tr>
    <td colspan="21"><i>Baseline Approaches</i></td>
  </tr>
  <tr>
    <td>SCAFFOLDS</td><td>.513</td><td>.722</td><td>.600</td><td>.122</td><td>.071</td><td>.089</td><td>.102</td><td>.062</td><td>.076</td><td>.288</td><td>.300</td><td>.293</td><td>.281</td><td>.090</td><td>.136</td><td>.069</td><td>.026</td><td>.035</td><td>.229</td><td>.212</td><td>.205</td>
  </tr>
  <tr>
    <td>SciBERT</td><td>.527</td><td>.684</td><td>.595</td><td>.135</td><td>.108</td><td>.120</td><td>.340</td><td>.389</td><td>.363</td><td>.273</td><td>.092</td><td>.138</td><td>.326</td><td>.142</td><td>.198</td><td>.052</td><td>.021</td><td>.029</td><td>.298</td><td>.239</td><td>.240</td>
  </tr>
  <tr>
    <td>GPT 4o</td><td>.773</td><td>.511</td><td>.615</td><td>.017</td><td>.020</td><td>.018</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.038</td><td>.308</td><td>.068</td><td>.000</td><td>.000</td><td>.000</td><td>.138</td><td>.139</td><td>.117</td>
  </tr>
  <tr>
    <td>SciTULU</td><td>.753</td><td>.507</td><td>.605</td><td>.068</td><td>.053</td><td>.060</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.026</td><td>.014</td><td>.018</td><td>.141</td><td>.096</td><td>.114</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td colspan="21"><i>FINECITE Approaches</i></td>
  </tr>
  <tr>
    <td>SciBERT (Linear)</td><td>.535</td><td>.474</td><td>.495</td><td>.103</td><td>.186</td><td>.131</td><td>.475</td><td>.385</td><td>.414</td><td>.382</td><td>.554</td><td>.450</td><td>.296</td><td>.173</td><td>.208</td><td>.170</td><td>.087</td><td>.112</td><td>.327</td><td>.310</td><td>.302</td>
  </tr>
  <tr>
    <td>SciBERT (BiLSTM)</td><td>.532</td><td>.428</td><td>.471</td><td>.100</td><td>.186</td><td>.125</td><td>.393</td><td>.385</td><td>.381</td><td>.374</td><td>.495</td><td>.422</td><td>.219</td><td>.154</td><td>.176</td><td>.120</td><td>.123</td><td>.119</td><td>.290</td><td>.295</td><td>.282</td>
  </tr>
  <tr>
    <td>SciBERT (CRF)</td><td>.512</td><td>.320</td><td>.387</td><td>.087</td><td>.139</td><td>.104</td><td>.355</td><td>.354</td><td>.342</td><td>.324</td><td>.589</td><td>.417</td><td>.299</td><td>.250</td><td>.265</td><td>.113</td><td>.164</td><td>.128</td><td>.282</td><td>.303</td><td>.274</td>
  </tr>
</table>

Table 10: Extended results of the citation classification task on ACT2.

It answers the question of why something is cited. [VIOLET]

E.3.1 General Notes
To make the annotation process possible, we have to assume some facts as given:

1. All reference Markers have been set at the correct position, and none are missing.

2. Group references have the same (or at least sufficiently similar) information.

3. All the information mentioned in connection with the reference is from the cited document.

E.4 General Rules
1. Articles (<i>a</i>, <i>this</i>, and <i>the</i>) must be included in the scope of the following noun.

<table>
  <tr>
    <td><b>X</b> The architecture of the system is very similar to a large system built for the NIST Arabic/English task [TREF]</td>
  </tr>
</table>

<table>
  <tr>
    <td><b>✓</b> The architecture of the system is very similar to a large system built for the NIST Arabic/English task [TREF]</td>
  </tr>
</table>

2. The reference marker ([REF], [TREF], etc.) must be marked as well (adjacent scope).

<table>
  <tr>
    <td><b>X</b> BERT is a large language model (LLM) [TREF]</td>
  </tr>
</table>

<table>
  <tr>
    <td><b>✓</b> BERT is a large language model (LLM) [TREF]</td>
  </tr>
</table>

<table>
  <tr>
    <td><b>X</b> Following [TREF], the loss is a sum of binary cross-entropy losses over all entity types T over all training examples D.</td>
  </tr>
</table>

<table>
  <tr>
    <td><b>✓</b> Following [TREF], the loss is a sum of binary cross-entropy losses over all entity types T over all training examples D.</td>
  </tr>
</table>

3. Only marks what is relevant to the targeted reference marker in case one reference is mentioned multiple times.

4. If the text is ambiguous, it should be marked in the following hierarchy: Information scope, Perception scope, and Background scope.

5. In cases where it is unclear whether the information is a contribution of the cited paper or the author, it should be marked as the author’s contribution.
<table>
  <tr>
    <th rowspan="2">APPROACH</th>
    <th colspan="3">BACKGR.</th>
    <th colspan="3">METHOD</th>
    <th colspan="3">RESULT</th>
    <th colspan="3">MACRO</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
  </tr>
  <tr>
    <td colspan="13"><i>Baseline Approaches</i></td>
  </tr>
  <tr>
    <td>SCAFFOLDS</td><td>.863</td><td>.873</td><td>.868</td><td>.792</td><td>.792</td><td>.792</td><td>.827</td><td>.784</td><td>.804</td><td>.827</td><td>.816</td><td>.821</td>
  </tr>
  <tr>
    <td>SciBERT</td><td>.894</td><td>.862</td><td>.805</td><td>.805</td><td>.834</td><td>.819</td><td>.805</td><td>.855</td><td>.829</td><td>.835</td><td>.850</td><td>.842</td>
  </tr>
  <tr>
    <td>GPT 4o</td><td>.860</td><td>.810</td><td>.834</td><td>.725</td><td>.821</td><td>.770</td><td>.671</td><td>.719</td><td>.694</td><td>.785</td><td>.784</td><td>.766</td>
  </tr>
  <tr>
    <td>SciTULU</td><td>.803</td><td>.857</td><td>.829</td><td>.832</td><td>.726</td><td>.775</td><td>.720</td><td>.768</td><td>.743</td><td>.782</td><td>.784</td><td>.782</td>
  </tr>
  <tr>
    <td colspan="13"><i>MULTICITE Approaches</i></td>
  </tr>
  <tr>
    <td>SciBERT (Linear)</td><td>.886</td><td>.867</td><td>.876</td><td>.819</td><td>.812</td><td>.815</td><td>.796</td><td>.870</td><td>.831</td><td>.834</td><td>.850</td><td>.841</td>
  </tr>
  <tr>
    <td>SciBERT (BiLSTM)</td><td>.898</td><td>.862</td><td>.880</td><td>.823</td><td>.836</td><td>.829</td><td>.782</td><td>.875</td><td>.826</td><td>.834</td><td>.858</td><td>.845</td>
  </tr>
  <tr>
    <td>SciBERT (CRF)</td><td>.890</td><td>.863</td><td>.876</td><td>.827</td><td>.820</td><td>.822</td><td>.780</td><td>.874</td><td>.823</td><td>.832</td><td>.852</td><td>.841</td>
  </tr>
</table>

Table 11: Extended results of the citation classification task on SciCITE.

<table>
  <tr>
    <th rowspan="2">APPROACH</th>
    <th colspan="3">BACKGR.</th>
    <th colspan="3">MOTIVATION</th>
    <th colspan="3">USES</th>
    <th colspan="3">EXTENDS</th>
    <th colspan="3">SIMILARITY</th>
    <th colspan="3">DIFFEREN.</th>
    <th colspan="3">FUTUR.</th>
    <th colspan="3">MACRO</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
    <th>P</th><th>R</th><th>F</th>
  </tr>
  <tr>
    <td colspan="23"><i>Baseline Approaches</i></td>
  </tr>
  <tr>
    <td>SCAFFOLDS</td><td>.732</td><td>.762</td><td>.746</td><td>.140</td><td>.093</td><td>.106</td><td>.598</td><td>.670</td><td>.631</td><td>.303</td><td>.279</td><td>.285</td><td>.415</td><td>.356</td><td>.382</td><td>.573</td><td>.485</td><td>.523</td><td>.186</td><td>.200</td><td>.189</td><td>.421</td><td>.406</td><td>.409</td>
  </tr>
  <tr>
    <td>SciBERT</td><td>.821</td><td>.799</td><td>.810</td><td>.241</td><td>.334</td><td>.280</td><td>.740</td><td>.758</td><td>.749</td><td>.482</td><td>.552</td><td>.515</td><td>.607</td><td>.529</td><td>.565</td><td>.695</td><td>.644</td><td>.669</td><td>.437</td><td>.564</td><td>.492</td><td>.584</td><td>.597</td><td>.584</td>
  </tr>
  <tr>
    <td>GPT 4o</td><td>.514</td><td>.715</td><td>.598</td><td>.053</td><td>.227</td><td>.086</td><td>.702</td><td>.554</td><td>.619</td><td>.436</td><td>.473</td><td>.454</td><td>.195</td><td>.556</td><td>.289</td><td>.667</td><td>.574</td><td>.617</td><td>.273</td><td>.600</td><td>.375</td><td>.406</td><td>.528</td><td>.434</td>
  </tr>
  <tr>
    <td>SciTULU</td><td>.489</td><td>.712</td><td>.580</td><td>.011</td><td>.100</td><td>.019</td><td>.728</td><td>.557</td><td>.632</td><td>.257</td><td>.743</td><td>.382</td><td>.054</td><td>.440</td><td>.096</td><td>.699</td><td>.438</td><td>.539</td><td>.182</td><td>.286</td><td>.222</td><td>.346</td><td>.468</td><td>.353</td>
  </tr>
  <tr>
    <td colspan="23"><i>MULTICITE Approaches</i></td>
  </tr>
  <tr>
    <td>SciBERT (Linear)</td><td>.840</td><td>.788</td><td>.812</td><td>.404</td><td>.294</td><td>.338</td><td>.789</td><td>.706</td><td>.744</td><td>.536</td><td>.507</td><td>.518</td><td>.652</td><td>.464</td><td>.539</td><td>.737</td><td>.576</td><td>.643</td><td>.652</td><td>.582</td><td>.641</td><td>.659</td><td>.560</td><td>.602</td>
  </tr>
  <tr>
    <td>SciBERT (BiLSTM)</td><td>.830</td><td>.777</td><td>.802</td><td>.428</td><td>.323</td><td>.366</td><td>.753</td><td>.727</td><td>.739</td><td>.524</td><td>.525</td><td>.522</td><td>.622</td><td>.460</td><td>.526</td><td>.720</td><td>.601</td><td>.655</td><td>.600</td><td>.545</td><td>.571</td><td>.640</td><td>.564</td><td>.597</td>
  </tr>
  <tr>
    <td>SciBERT (CRF)</td><td>.827</td><td>.776</td><td>.799</td><td>.388</td><td>.415</td><td>.395</td><td>.784</td><td>.685</td><td>.729</td><td>.545</td><td>.515</td><td>.529</td><td>.647</td><td>.443</td><td>.526</td><td>.722</td><td>.598</td><td>.654</td><td>.690</td><td>.545</td><td>.606</td><td>.658</td><td>.568</td><td>.606</td>
  </tr>
</table>

Table 12: Extended results of the citation classification task on MULTICITE.

6. Conjunctions like “however,” “in fact,” “furthermore,” “hence,” “therefore,” “in that,” “on the other hand,” etc., should not be included.

<table>
  <tr>
    <td>✗ However, BERT is a large language model (LLM) [TREF]</td>
  </tr>
</table>

<table>
  <tr>
    <td>✓ However, BERT is a large language model (LLM) [TREF]</td>
  </tr>
</table>

E.5 What is the citation information scope?
The citation Information scope of the target citation is the part of the paragraph that describes objective facts directly from the cited paper. This information is objectively true and does not involve any judgment from the author. They can be attributed as a finding of the cited paper or describe a process or judgment in the cited paper.

E.5.1 INCLUDE
Information about the contribution of the cited paper:

CONTRIBUTION
This can also be seen in BERT [TREF].

CONTRIBUTION + FACT
BERT is a large language model (LLM) [TREF].

CONTRIBUTION + PURPOSE
The architecture of the system is very similar to a large system built for the NIST Arabic/English task [TREF].

CONTRIBUTION + OUTCOME
[TREF] trains a new model called BERT, and they can show it outperforms the current state-of-the-art model.

NOTE If slightly judgmental verbs (emphasizes, stresses-out, underlines) are in an otherwise non-judgmental sentence, they should be marked as information scope.
Keywords that are referenced by they, this, etc., and belong to the information scope.

SLIGHT JUDGEMENT
[TREF] does not discuss LSP costs for internal MT development. He emphasizes on margin shrinking, which is directly linked to investment gain.

REFERENCED KEYWORDS
Recently, many reports have described studies using deep learning for dialogue systems that have achieved good performance. They can generate fluent sentences based on a user’s utterances [GTREF].
E.5.2 INCLUDE
Information about used processes in the cited paper:

<table>
  <tr>
    <th>PROCESS</th>
    <td>[TREF] trains their proposed mode.</td>
  </tr>
  <tr>
    <th>PROCESS + FACT</th>
    <td>[TREF] trains their proposed model on a classification task.</td>
  </tr>
  <tr>
    <th>PROCESS + PURPOSE/REASON</th>
    <td>[TREF] trains their proposed model to achieve superior performance.</td>
  </tr>
</table>

E.5.3 INCLUDE
Information about outcomes or judgments in the cited paper: It should only be marked as information scope when it is clear that the judgment is from the cited paper and not from the author.

<table>
  <tr>
    <th>JUDGEMENT</th>
    <td>[TREF] shows their model works well.</td>
  </tr>
  <tr>
    <th>JUDGMENT + COMPARISON</th>
    <td>They show their model works better than the BERT model [TREF].</td>
  </tr>
  <tr>
    <th>JUDGMENT + FACT</th>
    <td>[TREF] have shown how parallel suffix arrays can be used to significantly reduce the large memory footprints that phrase-based SMT systems suffer from when attempting to use longer phrases.</td>
  </tr>
</table>

E.5.4 INCLUDE
Information about when, where, and by whom the paper was published: All information that gives clues about temporal, locational, or personal facts about the paper but does not judge the content in any way.

<table>
  <tr>
    <th>PERSONNEL</th>
    <td>The same research team developed BERT [TREF].</td>
  </tr>
  <tr>
    <th>TEMPORAL</th>
    <td>Recently, BERT was introduced [TREF].</td>
  </tr>
  <tr>
    <th>LOCATIONAL</th>
    <td>In a paper from the ACL Conference, BERT is introduced [TREF].</td>
  </tr>
</table>

E.5.5 EXCLUDE
Further Information:

<table>
  <tr>
    <th colspan="2">on SIBLING SOURCES</th>
  </tr>
  <tr>
    <td colspan="2">On a larger scale, event extraction has extended to many languages beyond English, including French [REF], Spanish [REF], Italian [TREF] and very recently, Hindi [REF].</td>
  </tr>
</table>

E.5.6 EXCLUDE
Non-attributable facts: Information that can not be clearly attributed to the cited paper.

<table>
  <tr>
    <th>RESULTS/FINDING</th>
    <td>Furthermore, the word embedding techniques used by [REF] or [TREF] have been shown to work well. (The position of the judgment after the ref marker makes it unclear).</td>
  </tr>
</table>

E.6 What is the citation perception scope?
The citation perception scope relates to the author’s subjective perception and use of the information in the cited document or a concept, the cited document is provided as an example.

E.6.1 INCLUDE
Use of the referenced information:

<table>
  <tr>
    <th>PROCESS</th>
    <td>We use a BERT model pre-trained on classification [TREF].</td>
  </tr>
  <tr>
    <th>PROCESS + FACT</th>
    <td>We analyze a BERT model pre-trained on classification [TREF] on our dataset.</td>
  </tr>
  <tr>
    <th>PROCESS + PURPOSE</th>
    <td>We use a BERT model pre-trained on classification [TREF] for classifying our dataset.</td>
  </tr>
  <tr>
    <th>PROCESS + REASON (for/against)</th>
    <td>To increase model performance, we use the text segmentation approach suggested by [TREF].</td>
  </tr>
</table>

E.6.2 INCLUDE
Judgment of the referenced information
PERFORMANCE JUDGMENT
[TREF] develop a promising classification method.
The proposed BERT model [TREF] is not reliable.

RELATIONAL JUDGEMENT
Recently, Neural Networks have been getting more attention. An example of this trend is BERT [TREF].

SCOPING JUDGEMENT
On a larger scale, ...; In particular...; Other common methods .; Most of...

NOT-MENTIONED JUDGMENT
[TREF] does not discuss LSP costs for internal MT development.

JUDGMENT + COMPARISON
[TREF] shows that BERT is a reliable model. Compared to RoBERTa [REF], which employs other metrics, it is less reliable.

E.6.3 INCLUDE
A concept the citation is an example of that is strongly judged (reason for a decision): These rules only apply when the concept is subjectively judged by the authors. Only if there is a strong connection between the concept and the example, strong connection words: such as, like, etc.

<table>
  <tr>
    <th>CONCEPT + USE</th>
    <td>We analyze automated metrics such as BLEU [TREF].</td>
  </tr>
  <tr>
    <th>CONCEPT + JUDGEMENT</th>
    <td>We consider actual human judgments to be preferable to automated metrics such as BLEU [TREF].</td>
  </tr>
  <tr>
    <th>CONCEPT + REASON</th>
    <td>Because we care about the adequacy of post-edited translations, we consider actual human judgments to be preferable to automated metrics such as BLEU [TREF].</td>
  </tr>
</table>

E.7 What is the citation background scope?
The citation background scope includes information about neither the contribution of the cited document nor how it is perceived or used, but is essential for understanding its use.

E.7.1 INCLUDE
Background Information

<table>
  <tr>
    <th>SCOPING BACKGROUND</th>
    <td>Text segmentation has been getting more attention recently. For example, [TREF] uses BERT to do text segmentation.</td>
  </tr>
  <tr>
    <th>PROCESS BACKGROUND</th>
    <td>We adopt the Lexical Conceptual Structure (LCS) of Dorr’s work and use a parameter-setting approach to account for the divergences. [TREF] describes a parametric approach.</td>
  </tr>
  <tr>
    <th>THIRD PARTY PROCESS/FACTS</th>
    <td>Following the SAMT approach, CCG-augmented HPB SMT [REF] uses CCG [TREF] to label non-terminals.</td>
  </tr>
  <tr>
    <th>BACKGROUND + JUDGEMENT</th>
    <td>In fact, several GANs have recently been proposed for text generation [GREF] and have achieved encouraging results in particular, RelGAN [TREF] has outperformed state-of-the-art (SOTA) results.</td>
  </tr>
  <tr>
    <th>BACKGROUND + COMPARISON</th>
    <td>In fact, several GANs have recently been proposed for text generation [GREF] and have achieved encouraging results in comparison to comparable maximum likelihood approaches, in particular, RelGAN [TREF] has outperformed state-of-the-art (SOTA) results.</td>
  </tr>
  <tr>
    <th>BACKGROUND + REASON</th>
    <td>For comparison with the most dominant coreference dataset, OntoNotes [REF], we also measure the MUC score on our dataset. The MUC score on our dataset is 83.6, compared to 78.4-89.4 in OntoNotes, depending on the domain [TREF].</td>
  </tr>
</table>

E.7.2 INCLUDE
Further information

as EXAMPLE of CONCEPT
Text segmentation [TREF] describes the process of segmenting text. An example of this would be to segment a sentence into two parts.
on COMPARISON
[TREF] shows that BERT is a reliable model. Compared to RoBERTa [REF], which employs other learning metrics, it is less reliable.

on JUDGMENT + FACT
We train another model on 80,000 Amazon kitchen reviews [TREF], and apply it on the kitchen review dev set and the Amazon electronics dev set, both having 10,000 reviews.

as SIBLING
The use of BERT has been shown to be reliable [REF] and effective [TREF].

on PROCESS + FACT
For comparison with the most dominant coreference dataset, OntoNotes [REF], which only reported the MUC agreement score [TREF].

on LOCATION IN PAPER
Table 1 displays the result of our BERT Model. We use BLUE for evaluation. BLUE [TREF] is a metric to evaluate…. The use of BLUE is described in the following section.

on USE of JUDGMENT
..service has over 50 million users [TREF]. As native speakers of English, both authors judged the documentation to be of reasonable quality and well-formed. These initial assumptions would be tested in the project.

on USE/JUDGEMENT in THIRD PAPER
[TREF] released XY. This method was later expanded by [REF], who did xx.

E.7.3 EXCLUDE
Background of Background

BG + FACT (further information on the background)
For comparison with the most dominant coreference dataset, OntoNotes [REF], which only reported the MUC agreement score [REF], we also measure the MUC score on our dataset. The MUC score on our dataset is 83.6, compared to 78.4-89.4 in OntoNotes, depending on the domain [TREF].

EXAMPLES of BACKGROUND
Automatic extraction of events has gained sizable attention in subfields of NLP and information retrieval such as automatic summarization, question answering, and knowledge graph embeddings [GREF], as events are a representation of temporal information and sequences in text. [TREF] applies BERT for event extraction.

SIBLINGS of BACKGROUND
We adopt the Lexical Conceptual Structure (LCS) of Dorr’s work and use a parameter-setting approach to account for the divergences. [TREF] describes a parametric approach.

LOCATION, PERSONA, TIME of BACKGROUND
In 2016, [REF] published Roberta based on BERT [TREF].

on LOCATION of non-attributed facts IN PAPER (it is not sure whether the part is from the paper)
Following the SAMT approach, CCG-augmented HPB SMT [REF] uses CCG [TREF] to label non-terminals. This section gives a brief introduction to CCG followed by a description of the approach of extracting non-terminal labels using the same.

E.7.4 EXCLUDE
Further information

on Siblings
They [TREF] and JBNU-CCLab (Lee and Na, 2022) achieved much higher performances thanks to SciBERT tokenizer because it is trained on scientific literature.

Reviewer paper summaries: The paper proposes a finetuning methodology for more accurate downstream standard citation classification task on community datasets. The claim is that if finer-granularity is adopted as annotation scheme (specifically, what, how, why) then it helps in the standard classification task where classes are of the sort Background, Comparison (Similarity, Differences), Extension, Future, Motivation, Use, Method, Result.

This paper focuses on the limited literature on citation context extraction in the task of citation context analysis. The authors compare previous works’ citation context definitions with theirs. They provide a large comparison of definitions and different citation context analysis tasks. They define and explain their novel strategy in fine grained citation context extraction that is built on multiple dimensions that answers the questions of ‘what’, ‘why’ and ‘how’ a reference is cited. They also provide comparisons with state of the art methods that both extract citation context and classify them. They report an increase in the performance of some citation context classification pipelines where they used their own context extraction, showing that finer grained citation context can increase models’ performance in citation context classification tasks.

This paper presents FINECITE, a novel approach to fine-grained citation context analysis (CCA). The authors identify a critical gap in the existing literature, where citation contexts have traditionally been defined in a restrictive manner, limiting their expressiveness in citation classification tasks. To address this, the paper introduces a structured definition of citation contexts based on three semantic dimensions (Information, Perception and Background) along with key structural properties (e.g., non-contiguity and sub-sentence granularity). To validate their approach, the authors construct and release the FINECITE corpus, which comprises 1,056 manually annotated citation contexts and demonstrate its effectiveness through evaluation on established CCA benchmarks.

Overall, this paper presents a promising and well-founded approach to fine-grained citation context analysis, supported by preliminary yet encouraging results. However, the work appears to be at an early stage, lacking the depth of evaluation and analysis expected for a high-impact contribution. I strongly encourage the authors to continue this research, addressing the identified limitations to strengthen the impact of their work, as it has the potential to make a significant contribution to the field of citation analysis.
Summary of strengths: The general idea of finetuning models on annotated datasets on finer context segment-based classes looks promising together with the experimental results that support the idea.

Focusing only on citation context definition and extraction is an important task. This paper provides a vast comparison of previous works in CCA and their own work, there is a good synthesis of the state of the art. Second strength is the wide experiments done on several steps, both for comparison of the extraction and for comparison of the classification performance after applying their extraction method. The manual annotation task is well-detailed and explained with examples.

The paper presents a well-motivated and novel approach to citation context extraction, advancing beyond conventional definitions to provide a more comprehensive representation of citation contexts. 

The discussion of existing limitations is well-grounded in the literature, supported by an adequate number of references. 

The introduction of a fine-grained annotation scheme, along with the release of a new dataset, constitutes a meaningful contribution to the field.

The empirical evaluation on widely used CCA benchmarks (i.e., ACL-ARC, ACT2, SCICITE, and MULTICITE) demonstrates that the proposed framework improves citation classification performance. However, while the corpus and the proposed framework is valuable, its size (1,056 annotated contexts) remains relatively small by NLP standards
Summary of weaknesses: I would like to point out to some technical issues that I think are very important to be addressed:

Lines 71-74: "*... making it difficult to separate citation-related content from unrelated information*" -- Very little in a research article is unrelated. The arguments are interlinked to establish a claim (or a set of claims). And citations provide supporting evidence to the arguments made. This holds even when the narration flows into a different section (i.e., an citation in one section acting as a support for a particular argument can also act as a support for another argument in some other subsequent section). In most disciplines the narrative is also linear with very little back-tracking that might be required to resume another line of argumentation. The proposed methodology seems to ignore this fact.

**General Comment**:  Also, the way to model the citation-content relationship should be more from the point of view of "content-related citation" (or more specifically, "<argument-claim> related citation" rather than "citation-related content". The problem with the latter is that the relevancy of the citation to the argument can be overlooked leading to lower accuracy.

Lines 246-247: For "how", it boils down to identifying the ways in which a citation is used for evidence to (a) *reinforce* the claim, (b) *extend* existing line of argument thereby leading to the claim, or (c) *oppose* existing claim to strengthen proposed claim. This has to come in pair with the "what" within the same context passage. At this point it is not clear if the classification is that of citation context or a segment within the context. Can there be any contiguous context that does not contain "how" but contains "what"?

Lines 266-268: As argued, the sole reason is to provide evidence to justify an argument made to establish a claim. Any context around a citation has to contain a rationale to make the narrative coherent. I am not quite getting why the distinction is needed at this point.

Lines 298-299: Isn't the 3rd property an obvious implication of the second property? More clarification is needed here.

Line 330: Should it be <?

Lines 339-341: The update process and its relationship to annotation differences should be more explicit. As per my understanding this is the most crucial bit of the paper since the claim that the proposed scheme would make annotation task lot easier for even non-experts depends on this guideline heavily.

Lines 347-354 (Annotation): Step 3: Two vital questions remain open - (a) how many annotators, (b) how many annotators per passage, (c) what about IAA between expert and non-expert in order to figure out the correctness of the annotation.

Lines 365-370: Regarding metrics, F-measure is not a strong statistic for IAA when compared to Krippendorf's alpha or Fleiss' Kappa. Also, the Cohen value is at the best moderate and does not indicate gold annotation agreement, particularly when dimension-wise breakdown is not given. Even with F1 the real challenge with this schema is getting high IAA for PERC and BACK.

Lines 412-415: It is unclear how the restrictions are imposed in a controlled environment during the annotation. Was the context-window shortened (i.e., text masked)? How was this done for effect of contiguity restriction and sentence-segmentation? 

Table 2: It is unclear how the models are compared w.r.t human IAA. Also, what does it mean to surpass human IAA?

Sec 5.2 (Ablation): I personally think that this should not be seen as Ablation but the main objective (since the challenges in the standard datasets are more application-driven) while FINECite should be seen as a finetuning technique that is created using less complex annotated training data. 

Section 5.2 (Ablation): Ablations should be on FINECite w/o INF, w/o PERC, w/o BACK to better understand the effect of each dimension on the standard tasks.

Table 3: Why not also use the same headers for the baseline models to create probably a stronger baseline? Also, it is unclear whether the baseline models were finetuned on the respective datasets. I am assuming that they are. In that case the train data size of FINECite vs. other datasets should be mentioned.

As the authors also state, this is built on computational linguistics literature so it is unclear that this strategy will reproduce well in other research fields’ papers. I think the authors should not have underlined this much that other works have made coarse grained definitions, or it is rather reduced to one sentence. Different research fields might require this type of reduction especially if you’re focusing on fine grained classification rather than the extraction. 

The paper lacks a detailed comparison (real examples) of extracted citation contexts from previous works, and their own work. The authors claim that the citation classification task advances with their context extraction method but we do not see the impact of the added dimensions. It is unclear whether or not the positive increase in the results is caused by the added dimensions or just the increased context. How it was possible to add dimensions to the classification tasks of other works is also unclear.

Given that the primary contribution of this work is the dataset itself, one would expect a larger-scale corpus and a more in-depth discussion on citation context extraction, which is currently limited to Table 2. Furthermore, while the evaluation relies on F1 scores, a more extensive analysis would better prove its usability, for example by encompassing domain coverage. As acknowledged by the authors, the dataset is limited in domain coverage, while the selected benchmarks span a broader range of disciplines. This discrepancy results in a negligible performance improvements on SCICITE, raising questions about the proposal’s effectiveness.
Comments, suggestions and typos: Abstract: It would be good to introduce what is meant by "elemental dimensions".

The comparisons among previous works in the related works section is great, but we don’t see the same comparison on a practical level. I would appreciate real examples that compare the dynamic context retrieved with MultiCITE, or other works, with what the authors receive using their methodology so we can truly see the difference caused by the addition of dimensions. 

Figure 1 is not referenced in the text

It is unclear/not well presented in the paper how their input data is converted to use the existing schemes that do not follow the same guides (e.g dimensions)

Line 135, use of tragic is really awkward, defining scope for research is not ‘tragic’

Line 956 mentions ‘citation intention scope’. Isn’t this supposed to be ‘perception’? 

There is a claim that an annotator's domain expertise is not necessary for this task. This is a very bold claim and it is not the conclusion of the many other works that the authors cite. How can they be so sure of this? Was there an evaluation by an expert afterwards?

Section 3 is hard to follow due to the format of examples, a lot of the information is in the appendix but it’s necessary to understand the paper 

There is no real ablation in this work even if there is a subsection about it, it is not clearly presented . More focus could have been given to comparing the extraction outputs of previous works before applying the classification schemes.

A more detailed per-class analysis of the INF, PERC and BACK dimensions, along with qualitative insights, would further enhance the evaluation. Such an analysis would provide a clearer understanding of how the fine-grained context influences classification performance and could reveal specific strengths and weaknesses of the proposed approach.
Overall assessment scores: 2

4

3.5
Metareview score: 2.5
Metareview: This paper proposes FINECITE, a fine-grained citation context extraction and classification framework, introducing a new annotated corpus and showing improvements on citation classification benchmarks.

Review summary:

Overall found the idea novel and promising, and deals with citation context extraction (a relatively neglected area and usually taken trivially to be a few lines around the marker), the careful annotation process, and the empirical gains shown on standard datasets.

However, key concerns were raised:
* Dataset size and domain generalization were considered weak (only ~1k examples, only the computational linguistics domain).
* Ablations and the impact of semantic dimensions were initially unclear but were added later during rebuttal.
* Reviewer (RJPw) argued about the task framing ("citation-related content" vs "argument-centered") and remained unconvinced even after rebuttal.
* Some issues with annotation process validation and moderate IAA (especially for difficult labels like BACK, PERC).
* Writing needed clarifications (some figures/tables not referenced, guidelines/process underspecified).

Meta reviewer post-all (including confidential comments/non-acknowledged responses) summary: While all the reviewers agree on the contribution to an arguably important aspect of citation analysis, the gains are empirically low (which raises the question, is this aspect of citation analysis the next avenue for improvement for this task - further questioned by fact that the annotated dataset is small). Further, there are several raised issues, including clarity. 
There are nuances on how citation-context is defined (functionally vs logically) - I have taken into account both points.

Meta-reviewers' concerns (apart from those raised by other reviewers):
Though this proposed a context-enhancement technique which can be orthogonally applied to any model, this paper is effectively a task paper where the core is to demonstrate new techniques to improve performance. However, I feel some strong baselines have not been considered. like [1] Zeren Shui, Petros Karypis, Daniel S. Karls, Mingjian Wen, Saurav Manchanda, Ellad B. Tadmor, George Karypis:
Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification. EMNLP (Findings) 2024: 16718-16732 which has reported higher perofmaance than the one reporetd here (though it uses multiple datasets togather - but same setup can be adopted)
Summary of reasons to publish: The paper proposed a novel technique  (via an annotated dataset) of considering context for better citation analysis.