The Noisy Path from Source to Citation: Measuring How Scholars Engage with Past Research
Authors: Hong Chen, Misha Teplitskiy, David Jurgens
Accepted at: acl25

The Noisy Path from Source to Citation:
Measuring How Scholars Engage with Past Research

Hong Chen
University of Michigan
Ann Arbor, MI, USA
hongcc@umich.edu

Misha Teplitskiy
University of Michigan
Ann Arbor, MI, USA
tepl1@umich.edu

David Jurgens
University of Michigan
Ann Arbor, MI, USA
jurgens@umich.edu

Abstract

Academic citations are widely used for evaluating research and tracing knowledge flows. Such uses typically rely on raw citation counts and neglect variability in citation types. In particular, citations can vary in their fidelity as original knowledge from cited studies may be paraphrased, summarized, or reinterpreted, possibly wrongly, leading to variation in how much information changes from cited to citing paper. In this study, we introduce a computational pipeline to quantify citation fidelity at scale. Using full texts of papers, the pipeline identifies citations in citing papers and the corresponding claims in cited papers, and applies supervised models to measure fidelity at the sentence level. Analyzing a large-scale multi-disciplinary dataset of approximately 13 million citation sentence pairs, we find that citation fidelity is higher when authors cite papers that are 1) more recent and intellectually close, 2) more accessible, and 3) the first author has a lower H-index and the author team is medium-sized. Using a quasi-experiment, we establish the “telephone effect”—when citing papers have low fidelity to the original claim, future papers that cite the citing paper and the original have lower fidelity to the original. Our work reveals systematic differences in citation fidelity, underscoring the limitations of analyses that rely on citation quantity alone and the potential for distortion of evidence.

1 Introduction

Citations are an essential part of scholarly communication. Given the assumption that authors cite prior work to acknowledge their intellectual debts to it (Kaplan, 1965; Harter, 1992), citations are widely used to measure scientific contributions and trace the flows of knowledge. In practice, analysts and scholars tend to rely on raw citation counts (Waltman, 2016), implicitly assuming that citations are similar in kind, or that any variation is minor.

![Diagram showing the relationship between citing and cited papers, including reporting citations and corresponding claims](page_749_384_377_246.png)

Figure 1: Citations do not always align with the original claims of cited works. Evidence on opioid addiction, originally restricted to a specific hospital setting, is often cited without critical contextual information. 1 We develop a computational pipeline to assess citation fidelity between citing and cited papers at scale.

However, a large literature has shown that citations vary substantially (Teufel et al., 2006; Valenzuela-Escarcega et al., 2015; Zhu et al., 2015; Pride and Knoth, 2017; Jurgens et al., 2018).

Citations may paraphrase, summarize, or even misrepresent original knowledge, thereby leading to varying degrees of fidelity between the original and the citing claims. This variability in citation fidelity is not merely a theoretical concern; it has tangible consequences for the transmission of scientific knowledge. For instance, a short letter published in 1980 documenting opioid use in an inpatient sample (Porter and Jick, 1980) was subsequently cited uncritically as evidence supporting the safe use of opioid prescriptions (Deyo et al.,

1 Example reporting citation (A) is from Kowal (1999), and (B) is from Portenoy and Foley (1986). The corresponding claim (C) is sourced from Porter and Jick (1980). The low-fidelity citation case overstates conclusions by failing to accurately specify the study population.
2015). This letter had been cited over 600 times by 2017, with fewer than 20% of citations acknowledging the critical information of the original context (Leung et al., 2017). Such citations collectively contributed to a misleading narrative of low addiction risk, which played a role in the subsequent opioid epidemic (West and Bergstrom, 2021).

Despite the importance of citation fidelity, it has been challenging to study at scale. In this study, we introduce a computational pipeline that overcomes the challenges. Specifically, using a large set of full-text research papers, we identify reporting citations from citing papers and their corresponding claims from cited papers, resulting in 13 million cited-citing sentence pairs. We then apply a sentence-level measure of scientific information change to evaluate the citation fidelity of each pair.

Our analysis reveals several patterns in citation fidelity: fidelity is higher when authors cite papers that are more recent and intellectually close and more accessible; within teams, citation fidelity is highest for medium-sized teams, negatively correlated with the seniority of first authors, and shows no significant correlation with the seniority of last authors. Additionally, we identify a “telephone effect” of intermediary citations. Specifically, using a quasi-causal experimental setting, we find that the fidelity of citations is lower when the authors cite the original paper and an intermediary one that cites the original as well. Moreover, citation fidelity is positively correlated with the fidelity of the intermediary paper’s citation to the original paper. This telephone effect suggests that intermediary engagement with original sources can result in information loss.2

2 Motivation and Related Works

Citations are a widely used bibliometric tool for evaluating the performance of various actors in the scientific community. To better understand citation behavior, researchers have increasingly examined the content of citations, especially the linguistic and contextual features of citing text, not only to infer authors’ intent and citations’ function, but also to interrogate a core assumption: to what extent do citations reflect genuine scientific influence on the citing authors and their work?

2The codes and models for this paper are available at https://github.com/hongcchen/citation.

2.1 Citation classification

The normative theory of citation (Kaplan, 1965; Merton, 1979) posits that authors cite papers to acknowledge their intellectual debts to them; thus citations should indicate a direct semantic relationship or relevance between citing and cited works (Garfield, 1979). However, citation practices are not always accurate or rigorous, with issues such as misquotation, miscitation, and misrepresentation frequently observed (Harzing, 2002; Horbach et al., 2021; Cobb et al., 2024).

A growing body of research has examined the relationship between citation and cited papers, revealing significant variations in citation fidelity (West and Bergstrom, 2021; Broadus, 1983; Greenberg, 2009; Zhang and Abernethy, 2024). Studies have investigated cases of citations deviating from the original findings with low fidelity and have revealed that between 1 in 5 and 1 in 10 citations are used to support claims incongruous with the results of the cited paper (Todd et al., 2010). For example, Cobb et al. (2024) examined 3,347 citing claims in top psychology journals and found that approximately 19% either omitted crucial nuances or entirely misrepresented the claims of previous research. Greenberg (2009) showed how hypotheses can be transformed into accepted facts through affirmative citation chains lacking direct supporting evidence in biomedical research. These findings reveal the variability in citation fidelity and low fidelity in citation can contribute to the noisiness of the knowledge transmission pipeline. However, these studies are often limited in scale and reliant on labor-intensive manual verification which requires substantial domain expertise, leaving gaps in our understanding of citation fidelity across broader academic discourse.

Existing citation text studies have scaled citation classification by automating the identification of citation intent or function, primarily leveraging the immediate context surrounding the reference or the metadata of the citing paper (Cohan et al., 2019; Lauscher et al., 2022; Pham and Hoffmann, 2003; Li et al., 2013; Pride and Knoth, 2017; Valenzuela-Escarcega et al., 2015; Teufel et al., 2006; Baig et al., 2021). For example, Jurgens et al. (2018) classified the broad thematic functions of citations into six categories including BACKGROUND, MOTIVATION, USES, EXTENSION, COMPARISION OR CONTRAST and FUTURE. While these studies have expanded the scale of citation analysis, they still
primarily focus on the citing paper, with limited efforts to use text of the cited paper (Luu et al., 2021), particularly for assessing citation fidelity.

Moreover, measuring citation fidelity extends beyond standard semantic textual similarity and requires assessing whether the citation faithfully conveys the same scientific information as the original text (Wright et al., 2022). While there are potential alternative metrics from related areas such as fact-checking and semantic similarity (Thorne et al., 2018; Ganitkevitch et al., 2013), they are typically designed for different goals (e.g., truth verification, general paraphrase accuracy) and do not directly address fidelity in the context of scientific claims. This task of measuring fidelity requires capturing more nuanced aspects of similarity, such as abstraction level, uncertainty, and contextual appropriateness (August et al., 2020).

In our work, we address this empirical gap in measuring citation fidelity by introducing a computational pipeline to examine this problem at scale. We analyze the relationship between citation pairs consisting of reporting citations and corresponding claims and use a measure of scientific information change to assess their fidelity systematically.

2.2 Author’s engagement with literature

A common criticism of using citations to indicate impact or trace knowledge flow is that authors may cite without directly engaging with the original source or being influenced by its content. There is widespread allegation that scientists may not fully read or only superficially review the works they reference (Simkin and Roychowdhury, 2002; Jergas and Baethge, 2015). Kaplan (1965) famously raised concerns about this practice, asking: “How often are the works of others cited without having been read carefully? How often are citations simply lifted from the bibliography in someone else’s work without either reading or giving credit to the man who did the original search of the literature?”

Existing research has primarily examined bibliographic errors as indicators of superficial engagement with the literature, which can contribute to questionable research practices. For example, Broadus (1983) found that 23% of citing papers reproduced the same bibliographic error when using an incorrect title from a well-known sociology work, suggesting that many authors copy references without verifying the original sources. Simkin and Roychowdhury (2005) modeled the spread of misprints in bibliographies in physics and estimated that 70–90% of citations are copied from other papers’ reference lists.

However, less attention has been paid to how authors’ engagement is reflected in subtle textual signals—specifically, how they report original findings in their citations. An author’s level of engagement may be revealed in how they reinterpret or modify original scientific information, whereas limited understanding or familiarity can result in lower alignment or fidelity. Studies suggest that such low-fidelity citations may contribute to broader issues in knowledge transmission, such as the perpetuation of dominant discourses (Mizruchi and Fein, 1999) or even the propagation of misinformation in science (West and Bergstrom, 2021).

In this study, we conceptualize the fidelity between citing and cited sentences as one reflection of authors’ engagement with literature, defined as the depth of interaction an author has with the referenced work. Prior studies have demonstrated that various factors can significantly influence how authors engage with and interpret prior research. For instance, temporal closeness has been shown to impact the depth of interaction, with recent and within-discipline works often fostering stronger connections (Medo et al., 2011; Wahle et al., 2025). Access barriers can also influence the depth of engagement with prior research (Evans and Reimer, 2009). At the same time, team dynamics, including team size and authorship roles, can affect how thoroughly prior studies are engaged and conveyed in writing, reflecting different levels of responsibility and expertise within collaborative research efforts (Larivière et al., 2016; Brand et al., 2015).

Building on these insights, we contribute to this question by hypothesizing and testing that fidelity can be shaped by several key factors indicative of the level of engagement of authors, including temporal and disciplinary closeness, accessibility, and reliance on intermediary sources. These dimensions influence the depth of authors’ engagement with original literature, thereby affecting the fidelity of citations.

3 Citation Sentence Pairs

Examining citation fidelity requires sentence-level analysis, so we leverage bibliometric datasets that provide both citation links and full-text content and extract citation sentence pairs—where the citing paper’s reporting sentence references a specific claim in the cited paper.
3.1 Dataset
In this study, we employ one of the most comprehensive academic datasets available: The Semantic Scholar Open Research Corpus (S2ORC: Lo et al., 2020). S2ORC is a general-purpose corpus containing metadata for 136 million scholarly articles spanning different fields. This dataset also provides access to the full text of articles, thus allowing us to analyze how authors make citations and identify which parts of the original text are being referenced on a large scale. After filtering out non-English articles and publications without full text, we obtained a dataset of approximately 42 million papers.

3.2 Reporting sentences from citing papers
Not all citations are intended to report the claims of prior literature. In this study, we focus on a specific category of citing sentences, referred to as reporting citations—sentences explicitly conveying the results or conclusions of a cited paper. To isolate these sentences, we apply two key filtering criteria: single-source citations and background citations.

Identifying single-source citations  We first locate the in-text citation by identifying the sentence where the reference appears, setting the citation context window to be a single sentence. Li et al. (2022) find that many citations tend to correspond to a very localized piece of the cited document typically ranging from roughly a short clause to spans of one or two sentences. Lauscher et al. (2022) also report that while references to prior literature can sometimes span multiple sentences, 83% of citation contexts occur within a single sentence. Given that most citations fall within this single-sentence window, we restrict our analysis to single-sentence citations for consistency and precision.

As citations occur at varying levels of aggregation and granularity (Cronin, 1994), especially when multiple sources are cited together, we focus on citations that explicitly report claims from a single source. To further refine this selection, we apply a rule-based filter, considering only parenthetical citations enclosed in [ ] or () at the sentence’s end. This structured approach helps isolate the core reporting function of citations and minimizes confounding factors, enhancing the precision of our analysis. See Appendix Table 2 for valid examples of citing sentences.

Identifying background citations  Citations can serve multiple functions beyond reporting claims from the original paper. Previous research has developed classification schemes for citations based on their roles within an argument. Among these, BACKGROUND citations are widely recognized as a category that provides essential context and relevant information for the domain under discussion (Teufel et al., 2006; Cohan et al., 2019; Lauscher et al., 2022). These citations often set the stage for a study by informing readers about existing knowledge, highlighting gaps, and establishing the context for the study’s purpose (Jurgens et al., 2018).

Leveraging their framework, we focus on BACKGROUND citations as a means to further identify reporting sentences. We construct a more comprehensive dataset by integrating annotated data from two prominent sources for citation function classification: SciCite (Cohan et al., 2019) and MultiCite (Lauscher et al., 2022), both of which annotate BACKGROUND as a citation function. The combined dataset includes 27,052 instances, with 11,635 (43%) labeled as BACKGROUND. We fine-tune a SciBERT model (Beltagy et al., 2019) based on the integrated dataset for a binary classification task to identify whether a sentence serves as a BACKGROUND citation. The model achieves an F1 score of 0.81 using an 80-20 train-test split, which shows its effectiveness in accurately detecting BACKGROUND citations.

3.3 Sentences of claims from cited papers
To match reporting citations with the corresponding claims in cited papers, we follow the definition that background citations typically refer to relevant findings or essential context from the cited work. Therefore, we focus on sentences of claims, which are operationalized as those labeled as “results” or “conclusions” based on the discourse categories defined by Dernoncourt and Lee (2017). We apply a sentence classifier to identify result or conclusion sentences within the full text of the cited publication. This classifier is fine-tuned from a RoBERTa model (Liu et al., 2019) and is trained to categorize sentences into five categories: methods, background, objective, results, and conclusions. The training dataset consists of 200,000 paper abstracts from PubMed, which were self-labeled with these categories (Canese and Weis, 2013). The classifier achieved an F1 score of 0.92 on a held-out 10% sample (details in Wright et al., 2022). Using this approach, we filter approximately 30% of sentences of results or conclusions from the full text as claims to be matched with citing sentences.
<table>
  <tr>
    <th>Citing Sentence</th>
    <th>Matched Sentence of Claim (Cited Paper)</th>
    <th>Fidelity</th>
  </tr>
  <tr>
    <td>A large survey study showed that regular antenatal exposure to music and talking to the baby might prevent traits of ASD [94].</td>
    <td>Our finding that exposing a fetus to music and maternal speech during pregnancy is associated with a broad reduction in autistic-like behaviors is congruent with previous findings of music training for children with ASD.</td>
    <td>4.147</td>
  </tr>
  <tr>
    <td>One limitation in rsfMRI is intra-individual variability, which may be due to a variety of causes, such as time-of-day, diet, blood pressure, or cognitive load (<i>Specht, 2019</i>) .</td>
    <td>However, one major disadvantage of rs-fMRI is that rs-fMRI studies still vary in their acquisition methods and whether they are conducted on a 1.5T, 3T, or 7T MR.</td>
    <td>2.817</td>
  </tr>
</table>

Table 1: Examples of citing and matched sentences with fidelity scores. The first pair from Applewhite et al. (2022) and Ruan et al. (2018) shows high fidelity as both sentences discuss prenatal exposure to music and ASD, despite wording differences. The second pair from Canario et al. (2021) and Specht (2020) shows lower fidelity due to differing focuses on rs-fMRI limitations. The second pair also reflects an interesting bibliographic inconsistency: the source is cited as 2019 though the paper was published in 2020.

4 Measuring Citation Fidelity

Assessing citation fidelity requires measuring how faithfully a citation reflects the original findings of the cited work. This involves first identifying the corresponding claims in the cited text, and then analyzing how the citation relates to that claim. Both are challenging due to the nuanced nature of scientific information and the absence of explicit sentence-level referential links in citations. To address this, we employ a sentence-level measure specifically designed for scientific findings, along with necessary approximations.

4.1 Measuring sentence-level scientific information change

Assessing fidelity between citing and cited claims requires more than standard semantic similarity measures. It involves capturing nuanced aspects often inherent in scientific claims to determine whether the citation faithfully conveys the original information (Luu et al., 2021; Wright et al., 2022; August et al., 2020).

For this purpose, we employ a sentence-level scientific information change measure developed by Wright et al. (2022). This measure quantifies the extent to which two semantically similar sentences describe the same scientific claims and capture topic alignment as well as both nuance and accuracy in how claims are reported. It is specifically designed for evaluating scientific texts and is well-aligned with our task of citation fidelity, with minimal domain or task shift.

The metric assigns a similarity score from 1 to 5, where 1 indicates that the two sentences describe completely different claims, and 5 indicates that the sentences express identical claims (see Appendix Table 3). The model is fine-tuned from MPNet (Song et al., 2020) on a human-annotated dataset (crowdsourced and expert-reviewed) and trained to minimize the cross-entropy over sentence embedding dot products. It achieves a mean square error of 0.489 and a Pearson correlation of 76.48 in predicting the scientific information change (details in Wright et al., 2022).

Upper-bound approach Pairing the reporting citation and the corresponding claim is challenging because authors typically cite the entire publication rather than referring to specific sentences, making it difficult to determine which exact claim the citing sentence refers to. The absence of explicit sentence-level referential links complicates the task of assessing citation fidelity.

To address this, we use an upper-bound approach, where the highest fidelity score between a citing sentence and any sentence in the cited work serves as a proxy for fidelity. This method identifies the best-matched corresponding claim in the original paper. While the highest-scoring sentence may not always be the exact one the author intended to reference, this method approximates the best-case scenario of fidelity and ensures that the most relevant claim is captured, providing a structured way to evaluate how much original information is represented in citations.
Figure 2: (a) Citation fidelity is higher when citing within-field studies compared to cross-field studies. (b) Citation fidelity is higher for self-citations compared to non-self-citations. (c) Citation fidelity is higher when citing open-access works compared to non-open-access works.

4.2 Interpreting citation fidelity

Two examples of citation sentence pairs from our dataset are presented in Table 1, illustrating how citation fidelity quantifies the extent to which citing sentences accurately and comprehensively reflect the claims of cited papers. High-fidelity citations demonstrate strong conceptual alignment, while low-fidelity citations often diverge in the amount of information conveyed as well as in meaning, emphasis, or interpretation.

By extracting reporting citations and corresponding claims in cited sources, we generate 13 million citation pairs. A random sample of fidelity score shows that the majority cluster around 3.5, following a roughly normal distribution ranging from 1.0 to 5.0 (See Appendix Figure 6). This suggests that most citations exhibit a reasonable level of fidelity, though the presence of fewer extreme high and low values shows the variability in how accurately authors represent cited claims. Moderate fidelity scores could reflect a balance between accurate representation and paraphrasing to fit the citing paper’s context, whereas lower scores might signal information loss or potential misrepresentation.

5 Higher Fidelity is Associated with Closer Proximity

How authors report claims when citing can vary based on factors such as disciplinary conventions, contextual relevance, and individual interpretation of original work. We are interested in how various factors—such as accessibility, and temporal and intellectual proximity—affect fidelity. These factors, which can indicate the level of engagement an author has with the referenced work, may influence the alignment of information between citing and cited claims, as reflected in our fidelity measure.

Experimental setup  We employ a standard regression analysis framework, with the model specified below. Citation fidelity is the dependent variable (\( F \)). Key factors such as accessibility, proximity, and team dynamics are included along with other relevant attributes (See Appendix Table 4 for more details on all variables).

\[
F = \beta_0 + \beta_1 \text{Field.of.Study} + \beta_2 \text{Publication.Year} \\
+ \beta_3 \text{Publication.Type} + \beta_4 \text{Open.Access} \\
+ \beta_5 \text{Context.Length} + \beta_6 \text{Reference.Frequency} \\
+ \beta_7 \text{Publication.Interval} + \beta_8 \text{Paper.Citation} \\
+ \beta_9 \text{Author.Seniority} + \beta_{10} \text{Team.Size} \\
+ \beta_{11} \text{Self.Citation} + \beta_{12} \text{Within.Field}
\]

Results  Figures 2, 3, and 4 present coefficient estimates from regression analyses, controlling for potential confounders. The regression coefficients are mapped back to the original 0–5 fidelity scale by adding back the intercept, in order to visualize the results in a more interpretable way. Error bars represent standard errors (See Figure 7 for more regression results). 3

First, our findings reveal that proximity—whether in terms of accessibility, disciplinary context, temporal closeness, or author familiarity—plays a critical role in shaping citation fidelity. One strong piece of evidence is that self-citations, which is arguably a straightforward indicator of engagement as authors are naturally more familiar with

3Key variables are defined as follows: Team dynamics is proxied using Team.Size and Author.Seniority, where Team.Size represents the number of authors in the team, and Author.Seniority represents the experience of the authors using their H-index. Accessibility is proxied by Open.Access, a binary variable indicating whether the cited paper is publicly accessible. Temporal and disciplinary proximity are proxied using Publication.Interval, Self.Citation, and With.Field, where Publication.Interval represents the number of years between the cited and citing papers, Self.Citation indicates whether the two papers share at least one author, and With.Field indicates whether they belong to the same discipline.
their own work, exhibit higher fidelity scores than non-self-citations. This supports the validity of our fidelity measure, as it reflects the expected pattern: authors who engage more deeply with cited work tend to cite with higher fidelity.

Similarly, open-access publications exhibit higher fidelity compared to non-open-access publications, likely because unrestricted access to the original sources allows more direct engagement and accurate representation of cited work (Nagaraj et al., 2020). Without access to the full content of the cited work, authors may only partially examine the original material, relying on overviews or others’ interpretations. Citation fidelity is higher for within-field citations compared to cross-field citations. This is likely because authors are more familiar with the methodologies, terminologies, or conceptual frameworks of their own disciplines, and are able to represent without much information change. In contrast, cross-field citations, while essential for interdisciplinary knowledge transfer (Thilakaratne et al., 2018), may introduce additional information loss due to differences in their research norms, jargon, or methodologies.

More recent publications tend to have higher fidelity scores, likely because they are more contextually relevant and aligned with current scientific discourse. Authors may be more inclined to engage directly with recent works, while older publications are often cited less thoroughly (Wahle et al., 2025). It is often suggested that for canonical works, authors may rely on secondary interpretations rather than consulting the original source, which can lead to inaccuracies or information loss (West and Bergstrom, 2021). We explore this dynamic further in Section 6.

Second, our findings suggest that the roles and collaborative dynamics within teams can shape how prior work is represented. Interestingly, citation fidelity decreases as the seniority of first authors increases, while showing no significant relationship with the seniority of last authors—the traditional role for team leaders or principal investigators. This distinction may reflect the different roles and responsibilities within team collaboration. Last authorship often involves advisory or supervisory duties, where responsibility for writing detailed citations is typically delegated to other team members, usually the first author. (Brand et al., 2015; Sauermann and Haeussler, 2017).

The decline in citation fidelity with increasing first-author seniority may indicate a shift in how senior researchers approach citations. More experienced authors might rely more on their established heuristics and prior knowledge, or adopt a more top-down writing approach, resulting in less engagement with the cited materials.

Citation fidelity is highest for medium-sized teams, while both small and large teams exhibit relatively lower fidelity. This pattern likely reflects differences in team dynamics. Small teams may lack the diversity of expertise or necessary capacity needed for critical engagement with cited works, while in large teams, distributed responsibilities or coordination costs can reduce individual accountability for accurate representation (Larivière et al., 2016; Jones, 2021). Medium-sized teams, however, may achieve a balance in between.

Figure 3: Citation fidelity is higher when the publication gap between the citing and cited works is shorter.

Figure 4: Citation fidelity is negatively correlated with the seniority of first authors and less correlated with the seniority of last authors. Citation fidelity is also higher for teams of medium size.
Figure 5: The left panel illustrates the citation relationship, where paper A (original) is cited by paper C and paper D, with some authors also citing paper B (treatment group). The right panel compares citation fidelity to paper A between the treatment group (authors citing both paper A and paper B) and the control group (authors citing only paper A). Citation fidelity in the treatment group is 0.06 lower than in the control group, suggesting that intermediary engagement with paper B reduces fidelity to paper A. Error bars represent standard errors.

6 “Telephone Effect” in Citation

A key concern in citation practices is that authors may not always engage directly with the sources they cite (Kaplan, 1965). Instead, they often rely on intermediary sources—papers that have already interpreted and cited the original work. This intermediary engagement may introduce misinterpretations or distortions, which may propagate across the literature through citation chains (Sinkin and Roychowdhury, 2002; West and Bergstrom, 2021). Much like the “telephone game,” where a message is whispered from person to person and gradually distorted, scientific claims can evolve or degrade, as they are repeatedly paraphrased, summarized, or reinterpreted in subsequent research. We employ a quasi-causal experiment setting to examine how intermediary engagement with prior work impacts knowledge transmission, as reflected in their citation fidelity.

We hypothesize that exposure to others’ interpretations of an original claim influences how accurately a paper reports that claim. Specifically, citing an intermediary source increases the likelihood of information change from the original claim in the source paper, compared to when the paper directly engages with the source. This leads to two key modes of engagement: direct engagement (Paper C cites Paper A directly) and intermediary engagement (Paper C cites Paper A and an intermediary source Paper B) that also cites Paper A. See Figure 5 for illustration.

To formalize this mechanism, we define the following hypotheses:

1. H1: Citation fidelity decreases when authors rely on an intermediary source. Specifically, one citation’s fidelity in citing paper A will be lower if they also cite paper B, an intermediary source that also cites paper A.

2. H2: The fidelity of the intermediary source affects the fidelity of subsequent citations. If paper B exhibits low fidelity in citing paper A, then a paper citing both A and B will have lower citation fidelity to A.

Experimental setup To test these hypotheses, we conduct a quasi-experimental study, identifying approximately 50K paper pairs (C and D) with the following setup: both papers cite the same claim from same original paper A, but paper D cites A directly (control group), while paper C cites both A and an intermediary paper B, which itself cites A (treatment group). To control for potential confounding factors, papers C and D are exactly matched by publication year and field, for comparability across both groups.

We use citations to an intermediary paper as a proxy for engagement with that paper. While this does not guarantee direct influence or even that the intermediary paper was read, it is associated with a higher chance of engagement. Consequently, citing an intermediary paper increases the likelihood of being influenced by its interpretation of the original claims compared to papers that do not cite any intermediary source.

Results In the treatment group (citing both paper A and paper B), citation fidelity to paper A is
0.06 lower than in the control group (citing only paper A). This finding supports H1, indicating that engagement with intermediary sources may contribute to low citation fidelity.

We further divide the treatment group based on the fidelity of the intermediary citation. Specifically, the fidelity of paper B’s citation to paper A is classified into three levels: high (score above 4), medium (scores of 3–4), and low (score below 3). We find the fidelity of paper C’s citation to paper A positively correlates with the fidelity of paper B’s citation to paper A, corroborating H2. Low citation fidelity in the intermediary source can lead to further information loss in subsequent citations.

These results show the inherent risks associated with relying on intermediary sources in scholarly communication. While intermediaries often serve as practical and time-efficient tools for navigating the large body of scientific literature, they can also become vectors for information loss or even misrepresentation (Sharkey et al., 2023). This is particularly concerning when intermediaries fail to accurately reflect the original claims, as their interpretations may cascade through subsequent citations, compounding distortions over time (West and Bergstrom, 2021). Such reliance not only affects the fidelity of knowledge transmission but also raises broader questions about the robustness of scholarly practices.

Our quasi-experimental results reflect the average effect of having any intermediary source on citation fidelity. However, intermediary effects are likely more nuanced. The number of intermediaries involved, as well as their characteristics, such as recency, prestige, and citation count, may differentially influence how interpretations are transmitted. Future work could explore these dimensions to better understand the role intermediary sources play and investigate the specific mechanisms through which these low fidelity occur, such as affirmative citation or selective reporting to understand the source of low fidelity.

7 Conclusion

In this study, we introduced a computational pipeline to analyze citation fidelity on a large scale and developed an automated measure to evaluate how authors report prior findings. By applying this method to 13 million citation sentence pairs, we uncovered several key insights into the dynamics of scholarly communication and showed citation fidelity is higher with closer proximity. Through a quasi-causal experiment, we further establish that the “telephone effect” contributes to information loss: when citing papers exhibit low fidelity to the original claim, future papers that cite both the intermediary and the original show even lower fidelity to the original.

This study addresses two critical research gaps: the lack of large-scale measurements to assess citation fidelity and the lack of empirical evidence on how authors engage with prior literature. By operationalizing citation fidelity and exploring its contextual drivers, we reveal systematic differences in citation fidelity and provide a more nuanced understanding of how information is preserved or lost during the citation process. These findings not only provide evidence on the mechanisms of knowledge transmission but also reveal the limitations of traditional citation analyses which often overlook dimensions of fidelity and authors’ engagement.

Finally, this study represents an initial step towards enriching citation analysis with text-level nuance. It could help advance our understanding of citation quality—as a potential source of distorted or misrepresented information, as a behavioral trace of author engagement with prior work, and as a valuable complement to existing metrics for research evaluation.

8 Limitations

Citations are made for different purposes and with different functions, and authors employ a variety of ways in writing to paraphrase, summarize, or reinterpret prior work. This variety makes systematically measuring citation fidelity challenging. In our analysis, we apply a series of filters to both citing and cited sentences to form appropriate citation pair instances—for example, limiting citing sentences to those with a single source cited at the end, and limiting cited sentences to single-sentence claims labeled as results or conclusions. These filters help us isolate citation pairs where claims are expected to be reported faithfully. They prioritize precision but at the expense of recall and they may also introduce potential selection bias and overlook alternative citation forms that are also interesting to investigate.

In particular, we restrict the analysis to one-to-one, sentence-level matches between citing sentences and candidate claims in the cited paper, with the fidelity score proxied as the highest alignment
of each sentence pair. However, in practice, authors may distribute their reporting text across multiple sentences or use only a brief clause. Likewise, cited claims may also be diffuse, spanning several segments of text in the cited work. As a result, this simplification could likely underestimate fidelity in more complex rhetorical structures. Future work could improve upon this by expanding the scope of citation context and claim spans.

Also, while our citation fidelity scores capture alignment across several dimensions—such as topical alignment, generality, and factual representation of the original claim—they reflect an aggregate measure rather than disentangling these specific components. So these scores may oversimplify nuanced aspects of citation behavior and may be influenced by factors such as varying levels of abstraction, synthesis with other sources, or factual inaccuracies in reported claims. These factors may blur the boundary between acceptable paraphrasing and substantive misrepresentation. Therefore, while our fidelity measure represents a meaningful first step toward identifying potential sources of misrepresented scientific evidence, it does not correspond directly to the widely discussed concept of “miscitation” or “reference error.” Future research should explore these finer-grained aspects, not only to identify such cases but also to gain deeper insights into how and why misinformation or questionable research practices emerge and propagate through the citation process.

9 Acknowledgments

We thank members of Blablalab and DiscoveryLab, as well as participants in the Computational Social Science Working Group at the University of Michigan, for their insightful feedback and suggestions on this project.

References

Briana Applewhite, Zeynep Cankaya, Annie Heiderscheit, and Hubertus Himmerich. 2022. A systematic review of scientific studies on the effects of music in people with or at risk for autism spectrum disorder. International journal of environmental research and public health, 19(9):5150.

Tal August, Lauren Kim, Katharina Reinecke, and Noah A. Smith. 2020. Writing strategies for science communication: Data and computational analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5327–5344. Online. Association for Computational Linguistics.

Yasa M. Baig, Alex X. Osterling, Rui Xin, Haoyang Yu, Angikar Ghosal, Lesia Semenova, and Cynthia Rudin. 2021. Multitask learning for citation purpose classification. In Proceedings of the Second Workshop on Scholarly Document Processing, pages 134–139, Online. Association for Computational Linguistics.

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.

Amy Brand, Liz Allen, Micah Altman, Marjorie Hlava, and Jo Scott. 2015. Beyond authorship: Attribution, contribution, collaboration, and credit. Learned Publishing, 28(2).

Robert N Broadus. 1983. An investigation of the validity of bibliographic citations. Journal of the American Society for Information Science, 34(2):132–135.

Edgar Canario, Donna Chen, and Bharat Biswal. 2021. A review of resting-state fmri and its use to examine psychiatric disorders. Psychoradiology, 1(1):42–53.

Kathi Canese and Sarah Weis. 2013. Pubmed: the bibliographic database. The NCBI handbook, 2(1).

Cory L Cobb, Brianna Crumly, Pablo Montero-Zamora, Seth J Schwartz, and Charles R Martínez Jr. 2024. The problem of miscitation in psychological science: Righting the ship. American psychologist, 79(2):299.

Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics.

Blaise Cronin. 1994. Tiered citation and measures of document similarity. Journal of the American Society for Information Science, 45(7):537–538.

Franck Dernoncourt and Ji Young Lee. 2017. PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 308–313, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Richard A Deyo, Michael Von Korff, and David Duhkrkoop. 2015. Opioids for low back pain. Bmj, 350.
James A Evans and Jacob Reimer. 2009. Open access and global participation in science. Science, 323(5917):1025–1025.

Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758–764, Atlanta, Georgia. Association for Computational Linguistics.

Eugene Garfield. 1979. Is citation analysis a legitimate evaluation tool? Scientometrics, 1:359–375.

Steven A Greenberg. 2009. How citation distortions create unfounded authority: analysis of a citation network. Bmj, 339.

Stephen P Harter. 1992. Psychological relevance and information science. Journal of the American Society for information Science, 43(9):602–615.

Anne-Wil Harzing. 2002. Are our referencing errors undermining our scholarship and credibility? the case of expatriate failure rates. Journal of Organizational Behavior, 23:127–148.

Serge P.J.M. Horbach, Kaare Aagaard, and Jesper Wi-borg Schneider. 2021. Meta-research: How problematic citing practices distort science.

Hannah Jergas and Christopher Baethge. 2015. Quotation accuracy in medical journal articles—a systematic review and meta-analysis. PeerJ, 3:e1364.

Benjamin F Jones. 2021. The rise of research teams: Benefits and costs in economics. Journal of Economic Perspectives, 35(2):191–216.

David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391–406.

Norman Kaplan. 1965. The norms of citation behavior: Prolegomena to the footnote. American documentation, 16(3):179–184.

Nancy Kowal. 1999. What is the issue?: pseudoaddiction or undertreatment of pain. Nursing Economics, 17(6):348–348.

Vincent Larivière, Nadine Desrochers, Benoît Macaluso, Philippe Mongeon, Adèle Paul-Hus, and Cassidy R Sugimoto. 2016. Contributorship and division of labor in knowledge production. Social studies of science, 46(3):417–435.

Anne Lauscher, Brandon Ko, Bailey Kuehl, Sophie Johnson, Arman Cohan, David Jurgens, and Kyle Lo. 2022. MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1875–1889, Seattle, United States. Association for Computational Linguistics.

Pamela TM Leung, Erin M Macdonald, Matthew B Stanbrook, Irfan A Dhalla, and David N Juurlink. 2017. A 1980 letter on the risk of opioid addiction. New England Journal of Medicine, 376(22):2194–2195.

Xiang Li, Yifan He, Adam Meyers, and Ralph Grishman. 2013. Towards fine-grained citation function classification. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, pages 402–407, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.

Xiangci Li, Biswadip Mandal, and Jessica Ouyang. 2022. CORWA: A citation-oriented related work annotation dataset. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5426–5440, Seattle, United States. Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.

Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.

Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, and Noah A. Smith. 2021. Explaining relationships between scientific documents. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2130–2144, Online. Association for Computational Linguistics.

Mattis Medo, Giulio Cimini, and Stanislao Gualdi. 2011. Temporal effects in the growth of networks. Physical review letters, 107(23):238701.

Robert K Merton. 1979. The normative structure of science. The sociology of science: Theoretical and empirical investigations, pages 267–278.

Mark S Mizruchi and Lisa C Fein. 1999. The social construction of organizational knowledge: A study of the uses of coercive, mimetic, and normative isomorphism. Administrative science quarterly, 44(4):653–683.

Abhishek Nagaraj, Esther Shears, and Mathijs de Vaan. 2020. Improving data access democratizes and diversifies science. Proceedings of the National Academy of Sciences, 117(38):23490–23498.
Son Bao Pham and Achim Hoffmann. 2003. A new approach for scientific citation classification using cue phrases. In AI 2003: Advances in Artificial Intelligence: 16th Australian Conference on AI, Perth, Australia, December 3-5, 2003. Proceedings 16, pages 759–771. Springer.

Russell K Portenoy and Kathleen M Foley. 1986. Chronic use of opioid analgesics in non-malignant pain: report of 38 cases. Pain, 25(2):171–186.

Jane Porter and Hershel Jick. 1980. Addiction rare in patients treated with narcotics. The New England journal of medicine, 302(2):123.

David Pride and Petr Knoth. 2017. Incidental or influential?–a decade of using text-mining for citation function classification.

Zeng-Liang Ruan, Li Liu, Esben Strodl, Li-Jun Fan, Xiao-Na Yin, Guo-Min Wen, Deng-Li Sun, Dan-Xia Xian, Hui Jiang, Jin Jing, et al. 2018. Antenatal training with music and maternal talk concurrently may reduce autistic-like behaviors at around 3 years of age. Frontiers in Psychiatry, 8:305.

Henry Sauermann and Carolin Haessler. 2017. Authorship and contribution disclosures. Science advances, 3(11):e1700404.

Amanda Sharkey, Balázs Kovács, and Greta Hsu. 2023. Expert critics, rankings, and review aggregators: The changing nature of intermediation and the rise of markets with multiple intermediaries. Academy of Management Annals, 17(1):1–36.

Mikhail V Simkin and Vwani P Roychowdhury. 2002. Read before you cite! arXiv preprint cond-mat/0212043.

Mikhail V Simkin and Vwani P Roychowdhury. 2005. Stochastic modeling of citation slips. Scientometrics, 62:367–384.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: masked and permuted pre-training for language understanding. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Karsten Specht. 2020. Current challenges in translational and clinical fmri and future directions. Frontiers in psychiatry, 10:924.

Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 103–110, Sydney, Australia. Association for Computational Linguistics.

Menasha Thilakaratne, Katrina Falkner, and Thushari Atapattu. 2018. Automatic detection of cross-disciplinary knowledge associations. In Proceedings of ACL 2018, Student Research Workshop, pages 45–51, Melbourne, Australia. Association for Computational Linguistics.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.

Peter A. Todd, James R. Guest, Junxiu Lu, and Loke Ming Chou. 2010. One in four citations in marine biology papers is inappropriate. Marine Ecology Progress Series, 408:299–303.

Marco Antonio Valenzuela-Escarcega, Vu A. Ha, and Oren Etzioni. 2015. Identifying meaningful citations. In AAAI Workshop: Scholarly Big Data.

Jan Philip Wahle, Terry Lima Ruas, Mohamed Abdalla, Bela Gipp, and Saif M. Mohammad. 2025. Citation amnesia: On the recency bias of NLP and other academic fields. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1027–1044, Abu Dhabi, UAE. Association for Computational Linguistics.

Ludo Waltman. 2016. A review of the literature on citation impact indicators. Journal of informetrics, 10(2):365–391.

Jevin D West and Carl T Bergstrom. 2021. Misinformation in and about science. Proceedings of the National Academy of Sciences, 118(15):e1912444117.

Dustin Wright, Jiaxin Pei, David Jurgens, and Isabelle Augenstein. 2022. Modeling information change in science communication with semantically matched paraphrases. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1783–1807, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Tianmai M Zhang and Neil F Abernethy. 2024. Detecting reference errors in scientific literature with large language models. arXiv preprint arXiv:2411.06101.

Xiaodan Zhu, Peter Turney, Daniel Lemire, and André Vellino. 2015. Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology, 66(2):408–427.
A Appendix

Inference and Training Details

All experiments were conducted on a single NVIDIA RTX A6000 GPU using HuggingFace Transformers 4.36.2 and PyTorch 2.1.2 in a CUDA 12.1 environment.

We trained a model for the BACKGROUND citation classification task. For fine-tuning the SciBERT model on the combined dataset, we trained for 10 epochs with a learning rate of \(10^{-5}\), 500 warmup steps, and a batch size of 16. The best model was selected based on the validation loss.

The fine-tuning process was computationally efficient, taking approximately 20 minutes to complete on the dataset of 27,052 annotated instances. For inference, the trained model processes citations at a rate of approximately 15K instances per minute, allowing for rapid annotation of large-scale citation data. This setup ensures both scalability and accuracy for identifying BACKGROUND citations in citation analysis tasks.

Examples of Reporting Citation

Table 2 presents examples of citing sentences that meet our criteria for single-source reporting citations. These cases are characterized by citations appearing at the end of a sentence, enclosed in brackets or parentheses, and explicitly attributing a claim to one single source.

By filtering for sentences dedicated to reporting findings from a single source, we create a controlled and precise dataset that isolates the core reporting function of citations, improving the reliability of our analysis.

Measure of Scientific Information Change

To assess the fidelity of citations, we employ a supervised model to evaluate the degree of scientific information change between two claims. This model is trained on annotated data that evaluate whether two scientific claims convey the same information. Table 3 outlines the annotation framework used to quantify scientific information change. The metric assigns a score from 1 to 5, where higher scores indicate greater alignment between the claims. Annotators were instructed to assess similarity based on the conveyed scientific content rather than surface-level semantic similarity. For further details on the annotation process, see Wright et al. (2022).

<table>
  <tr>
    <th>Citing sentence</th>
    <th>Valid?</th>
  </tr>
  <tr>
    <td>…past work has shown that active contributors in r/science are largely already involved in scientific activity [17] …</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>…Existing studies in NLP to help automate the study have examined exaggeration [17], certainty [18], and fact checking [19], among others …</td>
    <td>No</td>
  </tr>
  <tr>
    <td>… We use GROBID [12], a more commonly used and actively developed tool …</td>
    <td>No</td>
  </tr>
  <tr>
    <td>… The finding was in accordance with former studies (Lee et al. 2020) …</td>
    <td>No</td>
  </tr>
</table>

Table 2: We focus on a specific category of citing sentences that feature single-source background citations, with the reference placed at the end of the sentence. Valid examples of reporting citations are shown.

<table>
  <tr>
    <th>Annotation Instruction</th>
    <th>Score</th>
  </tr>
  <tr>
    <td>The information in the findings is completely the same.</td>
    <td>5</td>
  </tr>
  <tr>
    <td>The information in the findings is mostly the same.</td>
    <td>4</td>
  </tr>
  <tr>
    <td>The information in the findings is somewhat similar.</td>
    <td>3</td>
  </tr>
  <tr>
    <td>The information in the findings is mostly different.</td>
    <td>2</td>
  </tr>
  <tr>
    <td>The information in the findings is completely different.</td>
    <td>1</td>
  </tr>
</table>

Table 3: The metric assigns a similarity score from 1 to 5, where 1 indicates completely different scientific claims, and 5 indicates identical scientific claims.

Citation Fidelity Distribution

Figure 6 presents the distribution of citation fidelity across a random 10k sample from our dataset of sentence pairs. The histogram reveals a roughly normal distribution, with most scores clustering around 3.5. While a substantial portion of citations maintain a reasonable level of alignment with the original claims, the presence of both high- and low-fidelity cases shows variability in citation practices.
Figure 6: Distribution of fidelity scores across the dataset. A random 10k sample of fidelity scores reveals that the majority cluster around 3.5, following a roughly normal distribution ranging from 1.0 to 5.0.

B Regression Analysis Details and Results

We use a regression framework to analyze factors influencing citation fidelity, with fidelity as the dependent variable. Descriptions of all variables, including key factors (such as Open.Access, Within.Field, etc.) and control variables (such as Field.of.Study, Publication.Type, Reference.Frequency, etc.), are provided in Table 4.

Figure 7 presents the regression coefficients for additional factors (beyond those discussed in Section 5) affecting citation fidelity, with 95% confidence intervals. Categories within predictors (e.g., Field.of.Study and Publication.Type) are color-coded for clarity. The reference categories for categorical predictors are as follows: “Field: Physics,” “OpenAccess: False,” “Publication Type: Other,” “Publication Year: 2000,” “Within-Field: False,” and “Self-Cited: False.” For continuous variables, they are grouped into bins for visualization purposes to better illustrate trends (in these cases, the bin corresponding to the smallest value serves as the reference category).

Besides the major findings related to fidelity, disciplinary differences were also observed. Citation fidelity is higher in fields such as Biology and Medicine compared to Physics and Computer Science. This suggests that researchers in the life sciences may engage more closely with cited works. In contrast, fields like Computer Science, where rapid paradigm shifts and fast dissemination are common, exhibit lower levels of fidelity.

Publication type can also influence citation fidelity. Review articles demonstrate the highest fidelity, followed by journal articles, while conference papers tend to show lower fidelity. This pattern may reflect differences in citation reporting standards and conventions across publication venues. Review articles, by design, are expected to more accurately report existing claims with higher fidelity in their citations. And conference papers, which often emphasize novelty and prioritize brevity, may engage with prior work in a more flexible way with lower fidelity.

Citation fidelity decreases as the citation count of the cited paper increases. This trend suggests that heavily cited papers are often engaged with less deeply, possibly due to their canonical status or the reliance on a large number of secondary interpretations. Papers with high citation counts may be cited symbolically or to signal familiarity rather than as direct sources of content, leading to a low fidelity in their representation.

Citation fidelity also increases with longer citation contexts. However, the increase diminishes beyond approximately 100 words, indicating that longer contexts are associated with more thorough explanations of the cited work, but excessively long contexts may dilute focus or introduce irrelevant details. Similarly, frequent citation of a source within a paper is associated with lower fidelity, suggesting more extensive interpretation and paraphrasing of the cited content. When a source is referenced multiple times, authors may engage with different aspects of the cited work rather than reiterating the same claim, leading to greater variation in fidelity.
<table>
  <tr>
    <th>Variable</th>
    <th>Description</th>
  </tr>
  <tr>
    <td><i>Field.of.Study</i></td>
    <td>The assigned research discipline of the citing paper, as provided by the S2ORC dataset. S2ORC determines this by leveraging Microsoft Academic Graph fields of study, which are automatically assigned based on metadata and citation relationships. These fields represent the primary discipline of the paper (e.g., computer science, biology).</td>
  </tr>
  <tr>
    <td><i>Publication.Year</i></td>
    <td>The year in which the citing paper was published. This is treated as a categorical variable to capture temporal trends in citation behavior.</td>
  </tr>
  <tr>
    <td><i>Publication.Type</i></td>
    <td>A categorical variable indicating the document type or genre of the citing paper, such as journal articles, conference papers, review, etc.</td>
  </tr>
  <tr>
    <td><i>Open.Access</i></td>
    <td>A binary variable indicating whether the cited paper is publicly accessible.</td>
  </tr>
  <tr>
    <td><i>Context.Length</i></td>
    <td>The number of characters in the citing sentence containing the reference. This captures the verbosity of the citation context.</td>
  </tr>
  <tr>
    <td><i>Reference.Frequency</i></td>
    <td>The number of times the same cited paper appears within the citing paper. This quantifies the prominence of a reference within a document.</td>
  </tr>
  <tr>
    <td><i>Publication.Interval</i></td>
    <td>The time difference (in years) between the publication dates of the cited and citing papers, which captures the recency of citations.</td>
  </tr>
  <tr>
    <td><i>Paper.Citation</i></td>
    <td>The total number of citations that the cited paper has received at the time of dataset compilation. This is obtained from S2ORC’s citation metadata.</td>
  </tr>
  <tr>
    <td><i>Author.Seniority</i></td>
    <td>The seniority of the authors of the citing paper, measured using the highest H-index among the listed authors. This is obtained from S2ORC’s author metadata.</td>
  </tr>
  <tr>
    <td><i>Team.Size</i></td>
    <td>The number of authors listed on the citing paper.</td>
  </tr>
  <tr>
    <td><i>Self.Citation</i></td>
    <td>A binary variable indicating whether the citing and cited papers share at least one author.</td>
  </tr>
  <tr>
    <td><i>Within.Field</i></td>
    <td>A binary variable indicating whether the citing and cited papers belong to the same <i>Field.of.Study</i>.</td>
  </tr>
</table>

Table 4: Descriptions of independent variables used in our analysis. These variables are either directly obtained from S2ORC’s metadata or derived based on attributes within the dataset. See Lo et al. (2020) for details on how S2ORC constructs metadata.
<table>
  <tr>
    <th>Variables</th>
    <th>Coefficient</th>
  </tr>
  <tr>
    <td>Intercept</td>
    <td>3.348***</td>
  </tr>
  <tr>
    <td>Field.of.Study</td>
    <td></td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>0.245***</td>
  </tr>
  <tr>
    <td>Medicine</td>
    <td>0.231***</td>
  </tr>
  <tr>
    <td>Psychology</td>
    <td>0.129***</td>
  </tr>
  <tr>
    <td>Engineering</td>
    <td>0.116***</td>
  </tr>
  <tr>
    <td>Computer Science</td>
    <td>0.039***</td>
  </tr>
  <tr>
    <td>Environmental Science</td>
    <td>-0.003***</td>
  </tr>
  <tr>
    <td>Publication.Type</td>
    <td></td>
  </tr>
  <tr>
    <td>Review</td>
    <td>0.080***</td>
  </tr>
  <tr>
    <td>Journal</td>
    <td>0.058***</td>
  </tr>
  <tr>
    <td>Conference</td>
    <td>0.032***</td>
  </tr>
  <tr>
    <td>Open.Access</td>
    <td>0.024***</td>
  </tr>
  <tr>
    <td>Context.Length</td>
    <td>0.003***</td>
  </tr>
  <tr>
    <td>Reference.Frequency</td>
    <td>-0.016***</td>
  </tr>
  <tr>
    <td>Publication.Interval</td>
    <td>-0.007***</td>
  </tr>
  <tr>
    <td>Paper.Citation</td>
    <td>-4.10e-6 ***</td>
  </tr>
  <tr>
    <td>Author.Seniority</td>
    <td>-0.001***</td>
  </tr>
  <tr>
    <td>Self.Citation</td>
    <td>0.017***</td>
  </tr>
  <tr>
    <td>Team.Size</td>
    <td>6.86e-6</td>
  </tr>
  <tr>
    <td>Within.Field</td>
    <td>0.029***</td>
  </tr>
  <tr>
    <td>Publication.Year</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>Observations</td>
    <td>13587728</td>
  </tr>
  <tr>
    <td>\( R^2 \)</td>
    <td>0.054</td>
  </tr>
  <tr>
    <td>Adjusted \( R^2 \)</td>
    <td>0.053</td>
  </tr>
</table>

Note: *p<0.05; **p<0.01; ***p<0.001

Table 5: Regression results examining factors associated with variation in citation fidelity. The model estimates the relationship between fidelity and a set of explanatory variables. Significance levels indicated by asterisks.
Figure 7: Regression coefficients with 95% confidence intervals for predictors of citation fidelity. Each point represents the estimated effect of a predictor (e.g., discipline, publication type, context length) on citation fidelity. Horizontal lines indicate the 95% confidence intervals for each coefficient. The statistical significance level is indicated by asterisks (***) for p < 0.001.

Reviewer paper summaries: The authors compile a corpus of sentence pairs in scientific papers that link citing sentences to the most similar sentence from the cited paper. They build on earlier work inferring the faithfulness to the cited paper sentence of the citing sentence. They fit regression models to estimate the impact of various factors on this faithfulness measure.
Summary of strengths: The authors provide a clear account of their work and the datasets and models they rely on. They perform careful experiments showing the effect of different kinds of citations and different relationships between citing and cited paper lead to changes in faithfulness. Notably, they show how citing papers that mediate between a cited and citing paper reduces faithfulness.
Summary of weaknesses: In the telephone effect section (6), it's not clear whether the mediation of intermediate citations might be a linear effect. Do the authors only include paper pairs where there is exactly one mediating paper? If not, does the telephone effect attenuate when there are multiple mediating papers, because no one intermediate paper is likely to distort the interpretation of the original paper? Or are more intermediate papers likely to confound the original meaning?
In the main text, the authors state that Figures 2-4 plot regression coefficients, which they do on the scale of the dependent variable (Faithfulness) but in Figure 7 the raw coefficients are given. It would be useful to clarify what is being plotted in the captions to Figs. 2-4.
Given the 0.81 F1 on classifying background sentences, and the only 30% of sentences in cited papers classified as results or conclusions, it would be useful to see how robust these results are, at least to considering more sentences from the cited paper. Since the authors select the most faithful cited sentence for each citing sentence, this would hopefully show very similar results on a small test set (without necessarily having to rerun on the whole dataset).
Comments, suggestions and typos: In Table 1, is the Specht 2019/2020 inconsistency in the original? If so, it'd be good to point this out in the caption to make clear it's not the authors mistake.
Relatedly, the authors point out several earlier studies that use typos in citations to estimate a lower bound on unfaithful citations. Would it be possible to check at least title similarity in the paper pairs selected for this study? Or to find citations pairs with mismatched titles that had estimated high faithfulness?
Overall assessment scores: 4