Structurally Human, Semantically Biased: Detecting LLM-Generated References with Embeddings and GNNs
Authors: Melika Mobini
Vincent Holst
Floriano Tori
Andres Algaba
Vincent Ginis
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models (LLMs)
Citation Networks
Graph Neural Networks (GNNs)
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models are increasingly used to curate bibliographies, raising the question: are their reference lists distinguishable from human ones? We build paired citation graphs, ground truth and GPT-4o-generated (from parametric knowledge), for 10,000 focal papers ($\approx$ 275k references) from SciSciNet, and added a field-matched random baseline that preserves out-degree and field distributions while breaking latent structure. We compare (i) structure-only node features (degree/closeness/eigenvector centrality, clustering, edge count) with (ii) 3072-D title/abstract embeddings, using an RF on graph-level aggregates and Graph Neural Networks with node features. Structure alone barely separates GPT from ground truth (RF accuracy $\approx$ 0.60) despite cleanly rejecting the random baseline ($\approx$ 0.89--0.92). By contrast, embeddings sharply increase separability: RF on aggregated embeddings reaches $\approx$ 0.83, and GNNs with embedding node features achieve 93\% test accuracy on GPT vs.\ ground truth. We show the robustness of our findings by replicating the pipeline with Claude Sonnet 4.5 and with multiple embedding models (OpenAI and SPECTER), with RF separability for ground truth vs.\ Claude $\approx 0.77$ and clean rejection of the random baseline. Thus, LLM bibliographies, generated purely from parametric knowledge, closely mimic human citation topology, but leave detectable semantic fingerprints; detection and debiasing should target content signals rather than global graph structure.

STRUCTURALLY HUMAN, SEMANTICALLY BIASED: DETECTING LLM-GENERATED REFERENCES WITH
EMBEDDINGS AND GNNS
Melika Mobini!
Vincent Holst Floriano Tori'
Andres Algaba'
Vincent Ginis!2
Vrije Universiteit Brussel Harvard University (SEAS)
(Melika Mobini, Vincent Thorge Holst, Floriano. Tori Jevub.br be (Andres Algaba, Vincent Ginis @vub.
ABSTRACT
Large language models are increasingly used to curate bibliographies, raising the question: their reference lists distinguishable from human ones? We build paired citation graphs, ground truth and GPT-4o-generated (from paramet- ric knowledge), for 10,000 focal papers ( 275k references) from SciSciNet, and added field-matched random baseline that preserves out-degree and field dis- tributions while breaking latent structure. We compare structure-only node features (degree/cosennss/eiieneee
centrality, clustering, edge count) with (ii) 3072-D title/abstract embeddings, using an RF on graph-level aggregates and Graph Neural Networks with node features. Structure alone barely separates GPT from ground truth (RF accuracy x 0.60) despite cleanly rejecting the random base- line ( 0.89-0.92). By contrast, embeddings sharply increase separability: RF on aggregated embeddings reaches = 0.83, and GNNs with embedding node features achieve 93% test accuracy on GPT ground truth. We show the robustness of our findings by replicating the pipeline with Claude Sonnet 4.5 and with multi- ple embedding models (OpenAT and SPECTER), with RF separability for ground truth vs. Claude x 0.77 and clean rejection of the random baseline. Thus, LLM bibliographies, generated purely from parametric knowledge, closely mimic hu- man citation topology, but leave detectable semantic fingerprints; detection and debiasing should target content signals rather than global graph structure.
INTRODUCTION
Large Language Models (LLMs) (Brown et al., 2020) increasingly synthesize scientific knowledge and autonomously draft literature reviews Delgado-Chavee et al., 2025: Dennstädt al., 2024; Gougherty & Clipp, 2024; Kang & Xiong, 2024; Skarlinski et al., 2024; Susnjak et al., 2024). This raises question about what they internalize from science: beyond surface form, do they repro- duce the structural and semantic regularities of citation behavior that scaffold scholarly knowledge? To situate our evaluation, we connect broader capability frameworks for LLMs (Chang et al., 2024; McDermott, 1976). Prior work reports that LLM-written bibliographies often look struc- turally plausible while being semantically unreliable (Tang al., 2025; Li al., 2023; Chu al., 2024: Li et al., 2024a).
We make this tension concrete: How do citation graphs induced by LLM-suggested references com- pare, structurally and semantically, to ground truth citation graphs? Are LLM-generated graphs realistic enough to pass structural scrutiny yet betray detectable semantic shifts? Here we build on the earlier analyses of LLM bibliographies purely generated from parametric knowledge, i.e., without access to external databases (Algaba et al., 2024; Mobini et al., 2025; Algaba et al., 2025). These studies find close alignment with human citation networks across observable characteristics (title length, team size, citation and reference counts), alongside systematic differences: LLMS rein- force the Matthew effect (higher median cited-by counts, preference for prestigious venues), favor recency, shorter titles, and fewer authors, reduce self-citations, and match focal-paper similarity at the level of off- the-sflf embeddings. Local graph motifs likewise resembled human networks.

---

Fig2 Propertias
Fig4-GNN Comparboon
Node Propertes
Model Deusity
Genenedd Graphs
Geserated graphs
Ground truth grighs
Random graphs
Ground Truth
Graghs
Amalysis
Centrality Messurts
Figh-Textine Embedding
progerties
GNN
Accuracy
Random Graphi
Paper
CIB OCDN (0000
Non bolated
PCA
Moddl
Isolted
10000 Focal Papers
Rost
ReshufDed
Pupers Till
3072-Dimentionll Vectors
PCA.
Accurccy
Figure 1: Overview of the experimental pipeline comparing graph structure and semantic embeddings of LLM-generated and human citation networks. The study involves comparing the citation networks of focal papers created using ground truth references and LLM- -LM---enertted references. Random baselines generated by reshuffling ground truth references. Graph properties and semantic embeddings are compared, and various GNN models are employed to differentiate between generated and ground truth graphs.
What remains unresolved is whether can reliably distinguish LLM vs. ground truth reference lists using only their induced citation graphs and textual signals.
We address this, see figure I with progressive modeling strategy from interpretable features to deep graph learning. First, we aggregate node-level structural metries into graph descriptors and evaluate their discriminability with Random Forest (RF) (Breiman, 2001) over three classes: (i) ground truth graphs built from human references, (ii) generated graphs built from LLM-suggested references, and (iii) nomain-matched randomized graphs. These structural features cleanly separate both (i) and (ii) from random baselines but do not separate (i) from (ii) at statistically significant levels. Motivated by this, we next use title-based semantic embeddings with RF, which markedly improves discrimination. Finally, we train Graph Neural Networks (Gori et al., 2005; Scarselli al., 2005; Zhou et al., 2020: Wu et al., 2021) that learn jointly from structure and node text, yielding further gains. Unlike single-citation audits such as LLM-Check (Sriramanan et al., 2024), we evaluate entire reference lists via their induced citation citation-graph structure and embedding cohesion.
If LLMs mimic human-like topology yet retain distinct semantic fingerprint, detection and debi- asing should target content signals rather than coarse graph structure. This matters for automated literature-review tools, citation recommendation systems, and "LLM--in-the-loop" scientific work- flows.
2 RELATED WORK
There is fast-growing literature using LLMs research assistants support paper search/reading, drafting surveys, and even automating parts of reviewing, which also surfaces re- curring concerns about reliability and evaluation in scholarly tasks Lin al. (2023a); Zhuang al. (2025); Chen et al. (2025); Wu et al. (2025); Li et al. (2024b); Skarlinski et al. (2024); Susnjak et al. (2024). While most discussion starts from synthetic prose, the same problem appears more "struc- tured" scholarly artifacts that LLMs increasingly produce, most notably reference lists, where out- puts can look superficially plausible while still being systematically wrong, incomplete, or skewed. Recent work has begun to measure how L..M-udouueed bibliographies compare to human ones, of- ten finding that reference lists can match many coarse bibliometric and graph-structural regularities when selection differs in more latent ways Algaba al. (2024); Mobini et al. (2025); Algaba al. (2025). We position our contribution in this gap by using the induced citation graph and show why topology-only approaches can be weak, motivating approach that fuses network structure with semantic representations.
Another strand comes from scientometrics and citation network analysis, which treats citation struc- ture as a signal how knowledge and attention organize at scale, where we ask what kinds of distortions might appear when L.MSs mediate scholarly discourse Lepori et al. (2025); Bai et al.
2

---

(2025). This connects naturally to work citation generation and citation recommendation: as LLMs are increasingly asked to "suggest references," they effectively become recommenders whose choices may feed back into discovery, reading, and ultimately credit assignment. Recent studies already point to systematic disparities in who gets recommended and who gets cited-acros demo- graphics, geography, and majority/minority status-raising the prospect that LLM--riven reference selection could amplify existing inequalities in visibility Tian et al. (2025); He (2025); Liu et al. (2025).
CONSTRUCTING CITATION NETWORKS
We use the open-source dataset from Algaba et al. (2025) which consists of 10, 000 focal papers (with 274, 951 accompanying references) sampled from the SciSeiNet (Lin et al., 2023b) database. The sample is drawn from papers which are published in Q1 journals between 1999 and 2021, have between 3 and 54 references, and have at least one citation. Additionally, the selected focal paper must have a defined "top field." a valid DOI, and an available abstract. For each focal paper, GPT-4o is prompted to suggest references based on the paper's title, authors, publication year, venue, and abstract. The number of requested references corresponds to the number of ground truth references cited by each focal paper. The existence of the LLM- -----eeeertedd references is determined by cross- verification with the SciSciNet (Lin et al., 2023b) database through fuzzy matching with titles and authors, using conservative similarity thresholds. For more details, we refer to Algaba et al. (2025). In a robustness analysis, we repeat the entire generation pipeline with Claude Sonnet 4.5, using the same prompts, requested reference counts, and fuzzy-matching filters for GPT-4o. This yields second set of L.LM-generated citation graphs paired to the same focal papers. We reuse the same graph construction and structural descriptors.
As shown in Figure 2 the focal paper forms the primary node (blue node), with references catego- rized into:
Green nodes: References cited by the focal paper and suggested by GPT-4o.
Yellow nodes: Non-isolated GPPT-generted references, not cited by the focal paper but connected to other references.
Orange nodes: Isolated GPT-generated references, neither cited nor connected to other references.
Grey nodes: ground truth references not suggested by GPT-4o.
Edges between nodes represent citations retrieved from the SciSciNet dataset. Each focal paper initially has full graph comprising ground truth and GPT-generated references. This full graph then split into two sub-graphs:
Generated graph: Contains the focal paper and GPT-generated references (blue, green, yellow, and orange nodes). Some citations produced by the model were not present in the focal paper's s ground truth reference list and we represent these exactly as the model predicted, single edge from the focal paper each predicted citation node.
Ground truth graph: Contains the focal paper and ground truth references (blue, green, and grey nodes).
We quantified the semantic role of isolated nodes by comparing cosine-similarity distributions for random references, ground-truth references, non-isolated GPT references, isolated GPT references. and references shared between GPT and the ground-truthh bibliography, as shown Appendix Fig- ure 18. To evaluate GPT-generated reference lists, we need a random baseline that matches the visible statisties the model could exploit (e.g., citation frequencies, publication years, topical labels) while deliberately breaking the latent citation structure of the ground truth graph. We therefore introduce random baseline on the field level. line with (Lin et al., 2023b), field labels are derived from the MAG (Microsoft Academic Graph) fields-of-study records: we use the Level-0 fields as "top fields" and the Level-1 fields as "subfields", based on the field assignments provided in SeiSciNet. When paper has multiple field entries, ized citation count, implicitly assuming that this field more relevant to that paper. For all focal papers, then construct synthetic citation graph by reassigning each focal paper's references
keep the field with the highest normal-
3

---

to random papers from the same research field. Concretely, within each (top) field of study, uniformly permute the ground truth references, i.e., a without-replacemment shuffle at the field level, and then attach the permuted references back to their original focal papers. This preserves each fo- cal paper's out-degree (reference count) and the field-level distributions of citation frequencies and publication years, while potentially destroying the latent citation structure that makes ground truth graphs informative. We also construct an analogous subfield-level random baseline using the same procedure, but now within SciSciNet's finer classification into 292 subfields (nested within 19 top fields in total). This analysis yields qualitatively similar results, see Appendix figures 12 and 13. Finally, we construct temporally constrained random baseline in which resampled references are drawn from the same field but restricted to publication years < the focal paper's year. In the original field-level reshuffling, temporal order (reference published before the focal paper) is preserved for roughly 80% of focal-reference pairs, whereas under the temporally constrained baseline fewer than 1% of edges violate temporal order. compared to about 6% of GPT-generated suggestions that point to papers published after the focal paper. Structural scatter plots and RF performance for this tem- poral baseline (see Appendix Figure 14 and table 11) closely mirror the original random baseline: random graphs remain clearly separable from both ground truth and LLM--genratte graphs, even under explicit time constraints.
To ensure fair comparison, we randomly remove subset of references from ground truth graphs and random graphs to match the size of the generated graph. We remove 779 graphs as we only kept graphs with at least one existing generated reference from GPT-4o graphs and 89 from Claude graphs, and end up with 9,218 and 9908 graphs per group for GPT-4o and Claude respectively. We replace each directed edge with an undirected, yielding a simple graph. Doing so, the compar- isons reflect the topological organization of the network (who is connected to whom and patterns of clustering/centraltty). rather than directionality artifacts or trivial in/out- differences.
4 CLASSIFYING CITATION GRAPHS USING GRAPH TOPOLOGY
We conduct a comprehensive quantitative analysis of each citation graph to characterize global topol- ogy and test whether structure alone separates ground truth from GPT-generated instances. We de- liberately restricted ourselves to interpretable descriptors that scale well and transfer across graphs of different sizes: degree centrality. closeness centrality, eigenvector centrality (Freeman, 1978), clustering coefficient, and edge count. These capture complementary local and global aspects of citation structure (connectivity concentration, path accessibility, prestige amplification, local triadic closure, and overall density). For node-level quantities we used normalized definitions to ensure comparability when graph order and density vary; graph-level descriptors were formed by aggregat- ing node metrics using summary statistics (mean, median, interquartile range, and maxima-to-man ratios), so that scale differences do not require ad hoc rescaling.
Figure 2 reports distributions over all 9,218 graphs. The box plots reveal that centrality profiles in GPT GPT-generated graphs closely match ground truth in both medians and IQRs, with similarly oc- casional high-degree nodes. In other words, the concentration of visibility and reach-how much "attention mass" accumulates on few nodes-looks nearly identical across the two sets. For clus- tering, medians are again comparable, but the ground truth set exhibits a wider IQR indicating that the model sometimes produces unusually dense local closure. By contrast, the field-matched ran- dom graphs remain tightly centered at low clustering, with narrow dispersion, underscoring a lack of triadic closure.
Further, Figure 2(b) shows joint view of average degree centrality and clustering coefficient. Ground truth and GPT-generated graphs overlap almost entirely within structural constraints: as average degree rises, the combinatorics of shared neighbors make triadic closure more likely, push- ing points upward in clustering; simultaneously, extremely high average degree with only moderate clustering is scarce, which explains the low number of points in that portion of the plane. Random graphs, in contrast, collapse toward the now-degree/low-clsster corner, reflecting their tree-like sparsity.
Additional pairwise relations reinforce this picture. The coupling between mean degree and the maximum-to-man degree ratio in Figure 2(c) is similar in ground truth and GPT-generated graphs, indicating a comparable extent of hub dominance relative to overall connectivity. Likewise, edge count vs. mean degree and edge count vs. node count trace families of trends consistent with realistic
4

---

Genarated prophis
(8)
10-
Grann trath
Generated
Ground truth
Nodes ressoved
Sanders
Randem grophs
Nodes
removed
0.2
0.0
Degros
Closencia Egenvestoc Contra.
(b)
(d)
(e)
Moga Chusteting Dogttesgen
1.0
AA0 2001 Degres Taagroo,Toon Tanto
50
180
Complene
6..8
Nombeer of Edgpa
Nunber
40
0.2
0.25 0.75 Moan Dogrus Contrnity
Degree
Moun Degroe
Figure 2: Pipeline to generate the ground truth and GPT-generated citation graphs and fea- ture histograms (a) The distributions of four node-level metrics computed for each graph. The ground truth and generated references for degree centrality indicate similar medians and inter- quartile ranges, suggesting the frequent presence of high-degree hub nodes. Random graphs dis- tribution of degree centrality values differs in shape, as their hub nodes are only the focal papers. For closeness centrality ground truth graphs and generated graphs curves cluster in the upper half of the scale, indicating consistently short geodesic distances. Random graphs show lower values with minimal variance, revealing their sparse and less interconnected structure. All three sets display positive skew for eigenvector centrality with slightly lower median for random graphs. The clus- tering coefficients of ground truth and GPT graphs have similar mid-range medians, while random graphs are centered around zero, confirming the absence of triangle-based local cohesion. (b) The scatter plot with marginal histograms, each point represents one graph, ground truth and generated graphs both concentrate in the mid-range of degree centrality and clustering. Random graphs, by contrast, cluster tightly at low values on both axes, showing their consistently sparse and unstruc- tured topology. Marginal density plots along the top and right panels show the univariate distribu- tions of degree centrality and clustering coefficient, respectively. (c) The mean node degree in each graph against its max-to-mean degree ratio in random graph types concentrate low with ground truth and generated graphs showing nearly identical peaks in both histogram and den- sity. In ground truth and generated graphs, mean degree and ratio trade off smoothly whereas mean degree increases, the relative prominence typically diminishes. Random graphs differ by showing a wider range of ratio values and a consistently lower mean degree.
degrees,
(d) The total number of edges is plotted against its mean node degree. The red dashed line indicates a complete graph, while the green dotted curve represents a tree graph. Both the generated and ground truth graphs show clear upward trend, graphs with more edges unsurprisingly have higher average degrees, and their point clouds largely overlap. This shows that the generative model accurately reproduces the empir- ical scaling between graph density and por-node connectivity, mimicking the scaling of ground truth graphs, which lies between sparse tree graphs and fully-connected complete graphs. Random graphs break this pattern by clustering tightly low average degrees, indicating sparse attachment model that lacks the varied connectivity patterns seen in other sets. (e) The scatter of Edges vs. Nodes both ground truth and generated graphs lie predominantly between these extremes by covering the entire range, whereas the random graphs resemble tree-like growth representing uniformly minimal connectivity.
densities: both human and GPT-generated graphs fall between tree and complete-gen gracte bounds, whereas random graphs tend to cluster near the tree limit. The near-complete overlap of ground truth and GPT-generated point clouds across these projections indicates that the generative process reproduces the characteristic coupling between global connectivity and local cohesiveness that observe human bibliographies, not merely matching marginal distributions but also multivariate relationships among descriptors.
5

---

Graph properties
Ground truth vs. GPT
Ground truth vs. Random 0.8956 + 0.0024 0.8946 + 0.0026
GPT vs. Random 0.9275 = 0.0059 0.9272 = 0.0060
Mean Accuracy
0.6079 + 0.0058 0.6061 + 0.0055
Mean F1-score
Table 1: Performance of the RF on graph properties. The table shows the mean accuracy and Fl-score obtained from applying a RF classifier across 10 independent runs with different random seeds, utilizing graph-basee based features. The dataset was partitioned into training/validationn and testing subsets, with a train set of 0.7 and a validation/test size of 0.15. The number of estimators sets to 100. Given the balanced nature of the dataset and the binary classification setting, accuracy is robust, but we also used F1 to minimize false positives and negatives.
Importantly, these similarities persist despite topical matching of the random baseline: even when drawn from the same broad field, random graphs systematically exhibit lower connectivity, min- imal clustering, and sparser, star- or tree-like topologies. This separation highlights a qualitative distinction: random graphs fail to instantiate the hub-rich, locally cohesive patterns that empirical citation networks display, whereas GPT-generated graphs do capture these signatures at both local and global scales. In that sense, the structural realism of GPT-generated bibliographies extends beyond single-feature alignment to joint, topology-Ievel constraints.
We then assess whether these structural descriptors suffice for automated discrimination by training a Random Forest (RF) (Breiman, 2001) on the aggregated graph-level features and evaluated three binary tasks: Ground truth vs. GPT-generated, Ground truth vs. Random, and GPT- vs. Random (Table 1). Against random baselines, structure is highly informative: we obtain accuracies of 0.8956 + 0.0024 (Ground truth vs. Random) and 0.9275 0.0059 (GPT vs. Random), with comparable Fl-scores, confirming that realistic citation topology is far from random even under field controls. However, when directly contrasting GPT-generated with ground truth, performance drops to near-chance: Mean Accuracy 0.6079 + 0.0058, Mean F1-score 0.6061 + 0.0055. Within this descriptor family, then, structural properties alone do not reliably differentiate LLM from human reference lists at scale. Running the same experiment with Claude- Claude-generated graphs shows the same pattern: near-chance Claude vs ground truth on structure, but again a clean separation from the random baseline see Appendix Figure 5 and table4. Taken together, these results support two conclusions that guide the rest of the paper. First, GPT-generated bibliographies are structurally realistic: their global topology, centrality concentration, and local closure closely track ground truth citation graphs, and they occupy essentially the same region of the structural feature space. Second, any systematic differences that remain are unlikely to be captured by coarse structural summaries; to detect residual signatures, one must look beyond Copology-towar textual/semantic signals and models that can fuse node content with structure.
CLASSIFYING CITATION GRAPHS USING LLM EMBEDDINGS
To complement our structural analysis, we measure content-level alignment by extracting embed- dings of the title and abstract for all the focal papers, and embeddings of the titles of the ground truth and GPTT-generatdd references using the OpenAl sett--bbedding---arrg model. This results in a 3072-dimensional embedding vector for each focal paper and reference which used as the node features. We then proceed by computing the sum over all ground truth references embedding vectors, over all GPT-generated references embedding vectors, and over all random references em- bedding vectors for each focal paper. To assess the robustness of our analysis, we do the same anal- ysis with the SPECTER2 (Singh al., 2023) embedding model which results in a disedinnnsi embedding vector, see Appendix Figure 8 and Figure 10. For titles only, we use the adhoc-query fine-tuned version, and for title and abstract combinations of the focal papers, we use the proximity fine-tuned version.
Figure shows 2D PCA of the graph-level embeddings with contours showing substantial overlap between ground truth and LLM graphs, while random graphs occupy a distinct region. Importantly, this 2-D projection is purely illustrative: the first two principal components explain only about 6% of the variance in the are needed to explain more than 90% of the variance. The classifiers we report operate in this higher-dimensional semantic subspace, not in the compressed 2-D view, see Appendix Figure 16
title embeddings, and several hundred components

---

[6]
100
0.2
col Senbaaht
BMS
to c00000
0.0
0.5
PCA
Figure 3: PCA of graph embeddings with Cosine/EU Distances (a) Summing 3072-d node embeddings to graph level, visualize the embedding space with 2D PCA (contours). (b) Cosine alignment (node level). Three graph-wise alignment diagnostics: mean focal with reference, mean of reference with reference, and focal align with the focal paper and with each other. (c) Euclidean dispersion (node level). Distance- space counterparts of mean focal with reference, mean reference to reference, and focal sum of references, quantifying semantic spread.
of references, capturing how well references
and Figure 17. For every graph we measure the similarities and the distances between nodes. The cosine similarity and distance scores exhibit strong semantic alignment of generated references with ground truth references compare to weak alignment random graphs.
We then apply the same RF classifier the aggregated embedding vectors. As shown in Table 2, classification performance significantly surpasses the one achieved with structural features alone: distinguishing ground truth from GPTT hl'T--eenrated graphs reaches mean accuracy of 0.8346-+0. 0.0063, compared to z> 0.6 with purely structural metrics. Both ground truth and GPT-generated graphs are well-separated from random baselines, achieving over 0.90 accuracy in both cases. These effects persist across encoders: GPT-4o references embedded with SPECTER (Singh et al., 2023) and Claude Sonnet 4.5 references embedded with OpenAI/SPECTER remain separable from ground truth references (embedded with SPECTER, and OpenA//SPECTER, respectively), mirroring the backbone results of GPT-4o references embedded with OpenAI, see Appendix Figure and table and 5 and 8.
In our RF across all trees, the average leaf depth clusters around Rs 10 levels levels-showing that early splits do most of the discriminative work (See Appendix Figure 20). These results suggest that semantic embeddings capture highly discriminative information.
In both the structural and the semantic settings, performance deviated from field-matched random baseline that simulates bibliographies assembled by uniformly sampling references from the focal paper's area. This pattern provides initial evidence that LLMs retrieve topically relevant references across domains using their parametric knowledge. These observations motivate our next step to evaluate range of GNN architectures that can jointly exploit topology and semantics to test whether learned message passing (Gilmer et al., 2017) can match or surpass the signal exposed by these simpler baselines.
CLASSIFYING CITATION GRAPHS USING GNNS
Models Graph Neural Networks (GNNs) have become the leading deep learning methods to op- erate on graph-structured data. Standard GNN architectures include Graph Convolutional Networks (GCNs) (Kipf& & Welling, 2016), Graph Attention Networks (GATs) (Veličković et al., 2017), Graph- SAGE (Hamilton et al., 2017), and Graph Isomorphism Networks (GINs) (Xu et al., 2018). Each model offers unique strengths in capturing structural information, scalability, and representational power among various applications. Until now, evaluation these models has primarily focused on traditional citation networks (Shchur et al., 2018). Due their conceptual diversity and robust per- formance across various tasks, these models serve as standard benchmarks for a fair and thorough evaluation of GNNs (Zhou et al., 2020; Errica al., 2019).

---

Title embeddings
Ground truth vs. GPT
Ground truth vs. Random 0.9077 + 0.0062 0.9070 = 0.0063
GPT vs. Random 0.9527 = 0.0017 0.9526 = 0.0017
Accuracy
0.8346 = 0.0063 0.8345 = 0.0063
Mean Fi-score
Table Performance of the RF on embeddings. The table shows the mean accuracy and Fl-score obtained from applying RF classifier across 10 independent runs with different random seeds, utilizing textual embeddings over graphs.
Experimental setup Reliable comparison of GNN models is often hindered by inconsistent ex- perimental conditions, incomplete reporting, and the use of different data splits. Our results are pro- duced using consistent evaluation strategy to compare different GNN architectures on our dataset. Earlier work has demonstrated that performance in node classification tasks is highly sensitive the choice of training, validation, and test splits (Shchur et al., 2018). To reduce the influence of such variability, we adopt multiple random seeds and average results across different data splits and ini- tializations. Finally, we report the final results on the validation set for each hyperparameter setup. We present the hyperparameter search grid in the Appendix table 12. For each model we then select the best performing setup and compute the performance on the test set. This offers, combined with the distribution of the validation performance, a transparent way to compare different models over the entire hyperparameter grid while still maintaining a direct comparison on the test set.
Structural attributes are usually incorporated into node features in GNNs to improve performance. This strategy benefits from the expressive power of aggregation functions like summation, but can also limit a model's 's ability to adapt to graphs exhibiting unfamiliar degree distributions. For instance, (Cui et al., 2022) found that sum- busebased aggregation remarkably outperforms mean ag- gregation in structural node classification tasks, because it maintains neighborhood counts and thus embodies critical structural information. Meanwhile, (Lee & Yoon, 2022) indicate that average node degree is a key feature influencing GNN generalization to unseen graphs, and support for modular update mechanisms to better adapt to graphs with heterogeneous degree distributions. To preserve consistency, all models in our study are trained and tested using the same node feature representa- tions. We performed three sets of pairwise comparisons between graph types: GPT vs. Ground truth, GPT vs. Random, and Ground truth vs. Random once with graph properties and once with semantic based vectors. Following the construction of citation graphs, we extracted structural features and embedding based vectors for each node and assigned these as input features for downstream learn- ing tasks. Each node was assigned five-dimensional feature vector consisting of degree centrality, closeness centrality, eigenvector centrality. clustering coefficient, and the graph's total number of edges, which is a graph level features but here assigned as node feature in GNN training using the graph for structural features. And a 3072-dimensional feature vector was made for each node the graphs for GNN embedding-basee comparisons. For each pairwise comparison we used binary label to differentiate between GPT, ground truth or random graphs.
The dataset was split into training. validation, and test sets in proportions of 70, 15, and 15, respec- tively using stratified splits in such a manner that if a ground truth focal paper appeared the train dataset, its respective random graph also appeared in the same split set. To ensure reproducibility and consistency across multiple models and experiments, the random seed was fixed during split- ting. Although the splits were deterministic, internal ordering was shuffled before training to avoid learning biases tied to sequence. Each model was trained using a consistent optimization protocol. For all learning we used the Adam optimizer (Kingma & Ba, 2014).
Results Figure 4 summarizes the performance of four GNN architectures across our classification tasks, GCN, GAT, GIN, and GraphSAGE. The reported accuracies are the final validation accuracies over the sweep of hyperparameters. We choose to report the full performance on hyperparameters to provide a transparent performance overview instead of simply highlighting the top performer. We do provide the top performing result for completness in Table for the test dataset and the best hyperparameters in Table 13 and 14. The upper row of the figure shows the performance when we only use the graph properties as feature vectors, while the bottom row highlights the performances when feature vectors are the embedding vectors used previously. In this setting, GNNs achieve high separability for GPT vs. Random with a high density across GNN architectures, reflecting substantial structural differences between generated and randomized citation networks. However,
8

---

Model
GAT
GCN
GIN
GraphSAGE
0
uod
Grapph Proportins
12
0,99
0.00
VS
Deasity
10-
Random
059
0.000
Ground Truth
Models
100-
Models
30-
GUX
OImO
000
G128
GropiSaoz
C
GropSSAD2
0 (-oab
GAT
GAT
CIN
0.
0.5
0.5
0.7
0.6
0.9
1.0
0.4
0.5
0.5
0.7
0.8
0.9
1.0
(b)
GPT
GPT
121
12
Embeeddngg
10-
Random
10-
Ground Truth
Models
Models
OCN
ODD
GCN
Graphtaca
GraphTACR
GAT
CAT
GiN
05
G.B
3.0
Accuracy
Acouracy
Figure 4: Distribution of final validation accuracy over the sweep of hyperparameters using different models Each panel summarizing the performance of four GNN architectures on validation set across our binary classification tasks using different seeds. Above figures show the GNN using the graph properties and the bottom figures are related to the GNN results using embedding vectors. Within every panel, the top axis shows kernel-density estimates (KDEs) of per-run final accuracies, and the bottom axis shows each GNN architecture boxplots medians, interquartile ranges, whiskers, and outliers. The bar plots illustrating the mean standard deviation of accuracy on validation dataset across different models showing consistency over models. Each sweep consists of 500 hyperparam- eter setups.
performance drops sharply in GNN experiments using graph properties for the GPT vs. ground truth classification, with accuracies clustering around chance level. Using different models could increase the accuracies, indicating that structural features along with message passing are insufficient for distinguishing LLM-ggnnrrted graphs from ground truth citation graphs. We also show the results for the GNN performance of the Claude references in the Appendix Figures 7 and 9 and the performance of GPT-4o references using OpenAI embeddings (See Appendix Figure 11, and the performance of GNN in the test set 6. Training on GPT-4o and testing on Claude Sonnet 4.5 yields substantial above-chance generalization for all GNNs (See results in Appendix 8); an RF trained on GPT-4o also reaches x 0.72 when the generator is swapped at test time (See results in Appendix 9).
When replacing node embeddings with i.i.d. vectors of matched dimensionality, RI/GNN accuracy collapses to chance, showing that the gains are the result of semantic structure rather than the number of features, see Appendix results 15.
To assess robustness, we quantify distributional saturation as runs accumulate using permutation- averaged Wasserstein distance (see Appendix figure 19), showing that distributions saturate carly and additional sweeps have marginal impact.
By using embedding based features, we observe an improvement in the GPT vs. ground truth task: all GNN architectures surpass chance performance by significant margin. This indicates that, al- though GPT-generated citation graphs mimic the structural properties of ground truth networks, their semantic embeddings encode subtle but learnable differences in language patterns. These results

---

Table 3: Results on test (< Accuracy, FI).
Ground truth vs GPT
Random vs GPT
Model
Graph properties
Embeddings
Graph properties
Embeddings
57. 27.73
93. 40.50
88.51
95.23
GCN
56.83
13.10
88.38
95. 18.18.4 1..10
GraphSage
55.59 1.98
93.14 00.51
88.51
95.8542.58
55. 27.27-50 #2.18
93. 40.51
88.37 +0.30
95. 45..8
57.40+2.44
93.78+0.43
88.33
95.50 +0.66
GAT
57.03
93. 93.7840.48
88.41
95. 40.49
GIN
11.70
93. 20.557
88.47
97. 97.39.m 20.23
47. 47.23
93.28
88.34
97.40+0.28
suggest that the primary signature distinguishing LLM-generated from human-generated citation graphs lies not in topology but in semantic content.
7 DISCUSSION
We present large-scale, paired evaluation of LLM-producce bibliographies against human ground truth, supplemented by field-matched random baselines. Our analysis proceeds from interpretable graph-level descriptors to content-aware graph neural networks, cleanly decomposing what is cap- tured by citation topology versus node semantics. Across transparent descriptors and structure-only models, LLM-generated citation graphs are essentially indistinguishable from human ones, while the random baseline is cleanly rejected. Incorporating title/abstract embeddings changes the picture: content-an aware models reliably separate LLM graphs from ground truth, indicating that residual dif- ferences reside in semantics rather than topology. We strengthen our analysis by including further robustness checks. We checked that our results do not depend on the embedding backbone dimen- sionality of the embedding vectors, are robust under two additional random baselines controlling for subfield and publication year of the focal paper, and even generalize cross-generator experiments.
Practically, this hints that auditing and debiasing LLM-generated bibliographies should prioritize content signals (e.g., embedding distributions, topical drift, recency tilt) over coarse structural fea- tures. Detection pipelines built on text embeddings or text+graph hybrids are therefore the right tool; structure-only diagnostics will under-detect.
Methodologically, our protocol (paired graphs, domain-matched randomization, and stepwise path from transparent features to content-aware GNNs) constitutes general recipe for stress-testing bibliographic authenticity without relying on manual curation.
8 CONCLUSION, LIMITATIONS AND FUTURE WORK
Our work analyses two LLM families over multiple embedding backbones while focusing on ti- tle/abstract text rather than full-text. In current analysis, we focus solely on the parametrically retrieved references, allowing for stricter lab setting and probing directly the biases of the mod- els. We not consider the references that would be obtained when models have access to external databases. Future work could probe which semantic dimensions drive separability and what they could mean (recency, prestige, method vs. theory, author overlap). Taken together, our results suggest simple operating principle: today's LLMs convincingly mimie the shape of citation, but not yet its semantics and it is therefore this semantic fingerprint that reliable detection (and eventual correction) must operate.
10

---

ACKNOWLEDGEMENTS
This research was supported by funding from the Flemish Government under the "Onderzoekspro- gramma Artificiële Intelligentie (AI) Vlaanderen" program. Andres Algaba acknowledges support from the Francqui Foundation (Belgium) through a Francqui Start-Up Grant and a fellowship from the Research Foundation Flanders (FWO) under Grant No. 1286924N. Vincent Ginis acknowledges support from Research Foundation Flanders under Grant No. No.G032822N and GOK9322N. The re- sources and services used this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation Flanders (FWO) and the Flemish Government.
REFERENCES
Andres Algaba, Carmen Mazijn, Vincent Holst, Floriano Tori, Sylvia Wenmackers, and Vincent Ginis. Large language models reflect human citation patterns with heightened citation bias. arXiv preprint arXiv:2405.15733 2024. doi: 10.48550/arXiv 200.
Andres Algaba, Vincent Holst, Floriano Tori, Melika Mobini, Brecht Verbeken, Sylvia Wenmackers, and Vincent Ginis. How deep do large language models internalize scientific literature and citation practices? arXiv preprint arXiv:2504.02767, 2025. doi: 0.448500/rXiv.2504.000
Xiaomei Bai, Fuli Zhang, Jiaying Liu, Xiaoxia Wang, and Feng Xia. Revolutionizing scholarly impact: advanced evaluations, predictive models, and future directions. Artificial Intelligence Review, 58(10)3111, 2025. doi: 10. 10.1007/s10462, 024--466222-2--15----6---------0
Leo Breiman. Random forests. Machine Learning, 45(1):5-322 2001. doi: 10.1023/A:1010933400 324.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. doi: 10.5555/349552244493883.
Yupeng Chang, Xu Wang, Jindong Wang. Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang. Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024. doi: 10. 14/3644/664,
Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, et al. Ai4research: A survey of artificial intelligence for scientific research. arXiv preprint arXiv:2507.01900 2025. doi: 10.48550/arXiv. 250000500i............ 01903.
Zhibo Chu, Zichong Wang, and Wenbin Zhang. Fairness large language models: taxonomic survey, ACM SIGKDD Explorations Newsletter, 26(1):34-48, 2024. doi: 12.32 117.
Hejie Cui, Zijie Lu, Pan Li, and Carl Yang. On positional and structural node features for graph neural networks non-attributed graphs. In Proceedings of the 31st ACM Interna- tional Conference on Information & Knowledge Management, pp. 3898-3902, 2022. doi:
Fernando M. Gelgado-Chaves, Matthew J. Jennings, Antonio Atalaia, Justus Wolff, Rita Horvath, Zeinab M. Mamdouh, Jan Baumbach, and Linda Baumbach. Transforming literature screening: The emerging role of large language models in systematic reviews. Proceedings of the National Academy of Sciences, 122(2)ee41 1962122, 2025. doi: 10. 10713/nas..241 1ete2241192222 1962122.
Fabio Dennstädt, Johannes Zink, Paul Martin Putora, Janna Hastings, and Nikola Cihoric. Title and abstract screening for literature reviews using large language models: an exploratory study the biomedical domain. Systematic Reviews, 13(1):158, 2024. doi:
Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. arXiv preprint arXiv:1912.09899, 2019. doi: 10.48550
11

---

Linton C Freeman. Centrality in social networks: Conceptual clarification. Social Networks, 1(3): 215- 239, 1978. doi: 10.1016/0378 877788887777790221-7
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pp. 1263-1272. PMLR, 06-11 Aug 2017. doi: 10. 10.555/33033813300551
URL https: /ttpp//eoeinnsssee//e/ mlr www.:.www.......mm.mmm.mm. www.ppes//vvii. v70 /qi 1mer 7 .
Marco Gori, Gabriele Monfardini, and Franco Scarselli, new model for learning in graph domains. In Proceedings. 2005 IEEE international joint conference on neural networks, 2005., volume 2,
pp. 729-734. IEEE, 2005. doi: 10. L119//ICNN 15554499
Andrew V Gougherty and Hannah L Clipp. Testing the reliability of ai-based large language model to extract ecological information from the scientific literature. npj Biodiversity, 3(1):13, 2024. doi: 10.0138//4115-022-0000000
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs Advances in Neural Information Processing Systems, 30, 2017. doi: 10.48550/ARXIV. 11W.ARXXIII77766000 1706.02 216. URL pictts://aree
www.///g//////o/ee//oo//eee/eeee absa 02216 6.
Jiangen He. Who gets cited? gender-and majority-bias in llm-driven reference selection. arXiv preprint arXiv:2508.0277 027440, 2025, doi: 00.485500arXx1v2250.22277 02740..
Hao Kang and Chenyan Xiong. Researcharena: Benchmarking LLMs' ability to collect and organize information as research agents. arXiv preprint arXiv:2406.1002 2024. doi: 10.48550/arXiv.2 406. 10291.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2::1412.69000 2014. doi: 2005500arXiv.1412..66
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: (0..0.00097,,.. 2016. doi: 100550//rXii.1160..22222
Hyungeun Lee and Kijung Yoon. Towards better generalization with flexible representation of multi- module graph neural networks. arXiv preprint arXiv:2209.066: 06589, 2022. doi: 10.48550/arXiv.2 209.06589.
Benedetto Lepori, Jens Peter Andersen, and Karsten Donnay. Generative ai and the future of sci- entometrics: current topies and future questions. arXiv preprint 200022507,0078, 2025, doi: 11.48500/rxiiv25507.000
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. LLMss-s-judges: comprehensive survey on LLM-based evaluation methods. arXiv preprint arXiv:2412.0557., 2024a. doi: 10.48550/arXiv
Sihang Li, Jin Huang. Jiaxi Zhuang, Yaorui Shi, Xiaochen Cai, Mingjun Xu, Xiang Wang. Lin- feng Zhang. Guolin Ke, and Hengxing Cai. Scilitlim: How to adapt Ilms for scientific literature understanding. arXiv preprint arXiv:2408. 15545, 2024b. doi: 01.48550/rXiiv24400 15545.
Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on faimesss large language models. arXiv preprini urXiv:2308. 10149, 2023. doi: 10.48550/arXiv.23000 23140.0
Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. Automated scholarly paper review: Concepts, technologies, and challenges. Information fusion, 98:101830, 2023a. doi: 10. (nffjjinfffs..223....
Zihang Lin, Yian Yin, Lu Liu, and Dashun Wang. Sciscinet: A large-scale open data lake for the science of science research. Scientific Data, 10(1):315, 2023b. doi: 10. 111103//4119---22-2020210 -9.
Yixuan Liu, Ábel Elekes, Jianglin Lu, Rodrigo Dorantes Dorantes-Gilardi, and Albert-László Barabási. Un- equal scientific recognition in the age of llms. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 9.
2025. doi: 1..11853//112222 findings findings-emmlp1127
12

---

Drew McDermott. Artificial intelligence meets natural stupidity, MIGARTNewsse 57(55, 1976. doi: 10.1145/10453391045522
Melika Mobini, Vincent Holst, Floriano Tori, Andres Algaba, and Vincent Ginis. How deeply do LLMs internalize human citation practices? a graph-structural and embedding-based evaluation. In ICLR 2025 Workshop on Human-Al Coevolution, 2025. URL https: pettpp:://penreeee http///weeneee.eeeeeeeeee/e///ep
Franco Scarselli, Sweah Liang Yong, Marco Gori, Markus Hagenbuchner, Ah Chung Tsoi, and Marco Maggini. Graph neural networks for ranking web pages. In The 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WTO5), pp. 666-672, 2005, doi: 10.1109/W1.200
5.67.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günneemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: 2018. doi: 10.48550/arX iv.1811.05866
Amanpreet Singh, Mike D' D'Ary, Arman Cohan, Doug Downey, and Sergey Feldman. SeiRepE- val: A multi-forma benchmark for scientific document representations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5548-5566, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 12..1865////222...oo http://///222...mmmmamaaaaaa mminp- URL https:
/ //aclanthhology crg/ emnlp- comnlp-main.... 338.
Michael D Skarlinski, Sam Cox, Jon M Laurent, James D Braza, Michaela Hinks, Michael Ham- merling, Manvitha Ponnapati, Samuel G Rodriques, and Andrew D White. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409. 13740, 2024. doi:
Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kat- takinda, and Soheil Feizi. LLM-check: Investigating detection of hallucinations in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, pp- 34188-34216, 2024. doi: 10.52202/079017-1077. URL https: http:p::///ppeeeeeee /ttp///eeeeeeeettt.t.// netiiieewweeeeoooooooo um? um?1dd=Yxx43CCAgy
Teo Susnjak, Peter Hwang, Napoleon Reyes, Andre LC Barezak, Timothy R McIntosh, and Surangika Ranathunga. Automating research synthesis with domain domain-specific large language model fine-tuning. arXiv preprint arXiv:2404.0868%/ 2024. doi: 104485500axxxv.2444.86666
Xuemei Tang, Xufeng Duan, and Zhenguang G. Cai. Large language models for automated literature review: An evaluation of reference generation, abstract writing, and review composition, 2025.
URL https: /arxivv..i.. org/ abs 2ass//212..11 13612.
Yifan Tian, Yixin Liu, Yi Bu, and Jiqun Liu. Who gets recommended? investigating gender, race, and country disparities in paper recommendations from large language models. arXiv preprint arXiv:2501.00367. 2025, doi: 0085500arXiv.250100003..
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1711 2017. doi: 10.48550/arXiv 10900.
Yutao Wu, Xiao Liu, Yunhao Feng, Jiale Ding, and Xingjun Ma. Paperask: A benchmark for reliability evaluation of iims in paper search and reading. arXiv preprint arXi::2510.22242, 2025. doi: 20.55500arxxi.2510.2222.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. IEEE Trunsactions on Neural Networks and Learning Systems, 32(1):4-24 2021. doi: 10. N0.1009/NNNLS..2002277777
Keyulu Xu, Weihua Hu. Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint 181iv 2018. doi: 100550/arXiv.1110...0
13

---

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. Al Open, 1:57-81, 2020. doi: 1.0116/..ioppmm2221....0000
Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. Large language models for automated scholarly paper review: A survey. Information Fusion, 124: 103332, 2025. doi: 10.1016/j..oo.. ii..://ffffu2222555555 103332.
14

---

APPENDIX
PROMPTING GPT-40
The GPT-references in the dataset from Algaba al. (2025) were generated using the following prompt in pptt-4-2024-08-11
The system prompt for each focal paper is:
Below, we share with you the title, authors, year. venue, and abstract of a scientific paper. Can you provide {n} references that would be relevant to this paper?
The user prompt contains basic bibliographic metadata and the abstract:
Title: (PaperTitle)
Authors:
(Author_Name)
Year: (Year)
Venue: (Journal_Name
Abstract:
(Abstract)
Here [n} is set the ground truth number of cited works for that focal paper. We subsequently parse the model's suggestions for bibliometric data (title, authors, number of authors, venue, publication year) in postprocessing step using second prompt (called via gpt-4o-minn-2004--77111
Below, we share with you a list of references. Could you for each reference extract the authors, the number of authors, title, publication year, and publication venue? Please only return the extracted information in a markdown table with the authors, number of authors, title, publication year, and publication venue as columns. Do not return any additional information or formatting.
15

---

PROMPTING CLAUDE SONNET 4.5
Following the methodology of Algaba et al. (2025), we generate reference suggestions using the following prompt in Cladee-ommee-45222550999:: The system prompt for each focal paper is:
Below, we share with you the title, authors, year, venue, and abstract of a scientific paper. Can you provide {n} references that would be relevant to this paper?
The user prompt contains basic bibliographic metadata and the abstract:
Title: (PaperTitle)
Authors:
Mauthor_Nanc }
Year: (Year)
Venue: (Journal_Name)
}
Abstract: (Abstract)
Here {n] is set to the ground truth number of cited works for that focal paper.
16

---

CITATION NETWORKS
Let G = (V, E) a simple undirected graph, where V denotes the set of nodes and E denotes the of edges. Let [V] denote the number of nodes. For node V with degree das the degree centrality (v) can be defined as
d V
CD(c)
reflecting the fraction of possible connections that e actually has. The distance between two nodes u, E is denoted by d(u, v) and equals the numbers of edges in the shortest path between the two nodes. The closeness centrality of node E V determined by the reciprocal of the average shortest path from to all other nodes:
IV1-1
Colu) = Enjo d(u,v)
Let Cn(U) be the eigenvector centrality of node V defined as
Cε(e)
1 À uEN(()
Σ Cε(U),
where À is the largest eigenvalue of the adjacency matrix of the graph.
Let e(v) denote the number of edges among its neighbors. The clustering coefficient of u is defined
as
20(v) C(v) (d, - 1)
17

---

ALTERNATIVE GENERATORS AND ENCODERS
Extended to Claude and decoupled generator/cncoder pairs confirm that the semantic fingerprint is not tied to single generator/encoder pair.
Edpa
0.5-
Indral
02
0.0
Digrowectrr
(4)
6d)
Meae Choeteeag Coefiicant
1.0
Coreu grapt
6..6
0.4
to
02
3.00 0.25 6.50
Contrnity
1.00
30 Mear Degree
Moon Degrzo
Nunber of Nodes
Figure 5: Structural comparison of citation graphs when references are generated by Claude. The figure mirrors the pipeline and descriptors used in the GPT-4o analysis. These patterns confirm that Claude bibliographies closely mimic human citation topology and that structural summaries alone cannot reliably expose the generator.
Graph properties Mean Accuracy Mean Fl-score
Ground truth vs. Claude
Ground truth vs. Random 0.9601 0.0039 0.9600 0.0039
Claude vs. Random
0.5389 + 0.0084 0.5386 + 0.0085
0.9701 = 0.0024 0.9700 0.0025
Performance of the RF Claude graphs. The table reports mean accuracy and mean Fi-score over 10 runs with different random seeds, using the same structural descriptors and 70/15/15 train/validation/test split in the GPT-4o analysis. Structure-only features remain only weakly informative for Ground truth vs. Claude, while ground truth vs. random and Claude vs. ran- dom classified with high accuracy, confirming that Claude reproduces realistic citation topology but is hard to distinguish from human graphs based on coarse structure alone.
Table 4:
Gemorated
ANG CER
0.1
Kag DalL SIC.
0.:
2M eostantte
BC
-0.
PCA
Figure 6: Embedding-space comparison for Claude graphs using OPENAI Semantic compar- ison of citation graphs with Claude Claude-generated references. As in the GPT-40 case, these results demonstrate that Claude retrieves topically appropriate references yet leaves a consistent seman- tic fingerprint that can be exploited for detection, in contrast to the near-indistingsn wear-indistinguuhhablllyy at the purely structural level.
18

---

Title embeddings
Ground truth vs. Claude
Ground truth vs. Random
Claude vs. Random
Accuracy Mean Fl-score
0.7688 + 0.0067
0.9428 + 0.0038 0.9426 + 0.0038
0.9546 0.0031 0.9545 + 0.0031
0.7687 + 0.0067
Table 5: Performance of the RF on Claude graphs using OpenAi embeddings. Using the same 2072-dimensional OPENAI embeddings and RF protocol in the GPT-4o experiment, we obtain substantially higher separability than with graph properties: Ground truth vs. Claude reaches ~ 0.77 aceuracy, while both classes remain cleanly separable from the random baseline. This replicates the pattern that semantic content carries much stronger detection signal than topology, with slightly lower ground-truth-vs-- separability for Claude than for GPT-4o.
Model
GNT
GCN
GIN
GraphSAGE
(4)
Claude
0.19
ane
Claude vs Ground Truth
Graphh Properties Density
12
VS
0.83
0.1
10
Random
0.90
0.00
8
Models
Models
00%
*
GrapESAGE
GraphSACI
+
GAT
G1N
GIN
0.4
0.5
0.6
0.7
0.5
0.9
1.0
0.6
0.5
0.8
0.7
0.8
0.9
1.0
(b)
Claude
0,050 8.025 8,000
Claude
10
vs
0,65
Embeddings Density
Random
Ground Truth
0.0
Models
Models
DOW
GON
GripiSAGE
GraaSSaGg
GIN
0.4
0.5
0.6
0.7
0.5
0.9
1.0
0.6
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
Accuracy
Figure 7: Performance of the GNN for Claude dataset. Each panel summarizes the performance of four GNN architectures the validation set across our binary classification tasks using different seeds. The figures are related to the GNN results using OpenAl embedding vectors. Within every panel, the top axis shows kerne--density estimates (KDEs) of per-run final accuracies, and the bottom axis shows each GNN architecture's 's boxplots medians, interquartile ranges, whiskers, and outliers.
19

---

Table 6: Results on for Claude dataset (*
Accuracy, F1).
Ground truth Claude
Random vs Claude
Model
Graph properties
Embeddings
Graph properties
Embeddings
50.1640.7
94.50+0.5
94. 94.7942.00
95.23
GCN
+1.71
24.50.0.14
94.78
96.66
GraphSage
50.0841.00 11.6
44.5
95.44
96. 156115110 1.65
43.01 26.40
44.4
96.56+0.45
46..5
GAT
60.25
94.58
95.67 +0.32
+0.66
47.47
94.12+0.34
96.3 96.34+0.34
96. 96.4440.60
GIN
50.3940.93
94.22+0.44
94.81 1.008
97.3920.0
33.50
94.18+0.34
94.7941.09 +1.09
97.67 #0.44
20

---

(4)
CWE CIR
Earl to centrod
Figure 8: Embedding-space comparison for Claude graphs using SPECTER. Summing 768-d node embeddings graph level, visualize the embedding space with 2D) PCA (contours). Semantic comparison of citation graphs with Claude-generated references, As in the GPT-4o case, these results demonstrate that Claude retrieves topically appropriate references yet leaves consistent semantic fingerprint that can be exploited for detection, in contrast to the inw.iiiinnnuihabbbltty the purely structural level.
Title embeddings
Ground truth vs. Claude 0.6878 0.0099 0.6873 + 0.0100
Ground truth vs. Random
Claude vs. Random 0.9421 0.0039 0.9419 + 0.0039
Accuracy
0.9289 = 0.0058
Mean Fl-score
0.9286 + 0.0059
Table 7: Performance of the RF on Claude graphs using SPECTER embeddings. The table
shows the mean accuracy and Fi-score obtained from applying a RF classifier across independent runs with different random seeds, utilizing textual embeddings over graphs.
21

---

Model
GAT
GCN
GIN
GraphSAGE
(2
Claude
100
0.0
Claude
Gruph Prupertes
Random
0.50
0.00
Ground Truth
Models
Models
0C1
|
GrepSAAC
GraphSan1
:80
GAT
GAT
-1o
GN
6.5
Claude
Claude
12
Embedeings
20.
Random
0.80
Ground Trush
Models
Models
GON
CCN
GrsplSAGE
GrophSAGE
GAT
GIN
61N
0.4
0.5
0.6
1.0
0.6
0.9
1.0
Accurany
Accurany
Figure 9: Performance of the GNN for Claude dataset using SPECTER embeddings. Each panel summarizing the performance of four GNN architectures on validation set across our binary classification tasks using different seeds. The figures related to the GNN results using SPECTER embedding vectors. Within every panel, the top axis shows kernel-density estimates (KDEs) of per- final accuracies, and the bottom axis shows each GNN architecture boxplots medians, interquar- tile ranges, whiskers, and outliers.
22

---

(0)
100 10
SOTE
POV2
IN
Bock Cat TO
Figure 10: PCA for GPT-4o graph embeddings with Cosine/EU Distances on SPECTER em- beddings Summing 768-d node embeddings to graph level, visualize the embedding space with 2D) PCA (contours). The cosine alignment and Euclidean dispersion (node level) in three graph-wise alignment diagnostics: mean of focal with reference, mean of reference with reference, and focal vs. sum of references, capturing how well references align with the focal paper and with each other. Similar to graph properties, there is slight change regarding the embedding features in the random subfield baseline.
Title embeddings
Ground truth vs. GPT
Ground truth vs. Random
GPT vs. Random
Accuracy
0. 0.75 = 0.0093
0.8865 + 0,0060 0.8854 + 0.0062
0.9344 = 0.0035
Mean Fl-score
0.7498 = 0.0093
0.9342 = 0.0035
Table Performance of the RF on GPT-4o graphs using SPECTER. The table shows the mean accuracy and F1-score obtained from applying RF classifier across 10 independent runs with dif- ferent random seeds, utilizing textual embeddings over graphs.
Model
GCN
GIN
GraphSAGE
(n)
6,10
0.05
Embeddiagg
0.00
Density
Models
Models
10-
15-
10-
19
5
GUN
GCN
CO-00-
GoahSAGE
GraphSAdE
GAT
0
020
GNT
OH [|I+
GN
G13
0.4
0.3
1.0
0.4
0.5
0U6
0.7
Accuracy
Accuracy
Figure 11: Distribution of final validation accuracy over the sweep of hyperparameters using different models on GPT-4o graphs using SPCETER embeddings Each panel summarizing the performance of four GNN architectures on validation set across our binary classification tasks using different seeds. The figures are related to the GNN results using embedding vectors. Within every panel, the top axis shows kernel-densty estimates (KDEs) of per-run final accuracies, and the bottom axis shows each GNN architecture boxplots medians, interquartile ranges, whiskers, and outliers.
23

---

CROSS MODEL EXPERIMENT
We further probe cross-model generalization. Training GCN, GIN, GAT, and GraphSAGE on GPT- 4o vs. ground-truth graphs (using OpenAl title embeddings) and then evaluating on test sets where GPT-4o graphs are replaced by Claude graphs yields accuracies around 0.68-0.80. These results suggest that the semantic fingerprint exposed by our models is to a large extent shared across GPT. 4o and Claude, even though fine- eve-ggrined citation patterns remain model model-specific.
Title embeddings
Ground truth vs. Generated
Mean Accuracy
0.7205 + 0.0052 0.7163 + 0.0055
Mean Fl-score
RF cross-model table. Random Forest trained to distinguish ground-truth vs. generated citation graphs using summed 3072-dimensional title embeddings. Despite this generator swap, the classifier attains ss 0.72 accuracy.
Table 9:
Model
Accuracy
F1
GCNNet
0.8001
0.7947
GINNet 0.7619
0.7507 20.0233
GATNet
0.6793=0.0155
0.6500 0.6853
GraphSAGENet
0.704210.0125
GCN, GIN, GAT, and GraphSAGE trained to classify ground-truth vs. GPT-4o graphs using OpenAl title embeddings as node features, and then evaluated on generalise substantially above chance, even though their detailed citation patterns are not seen during training.
Table 10: Results test set for cross-model GNNs.
where the GPT-4o is replaced by Claude graphs. All architectures
24

---

RANDOM SUBFIELD BASELINE
As a robustness check, we provide the same analysis for classifying citation graphs using graph topology and LLM embeddings, but with second, more fine-grained, random baseline. This Subfield-Randon baseline restricts the sampling to papers within the same sub-field, provided that the sub-field contains at least 30 papers.
Generated
graphe
1.0
Garand Iruth
Genanied
Nedes removed
0.8
Sandom
grapis
Random
graphss
remowed
Deggnee Centra.
Clonattes Centra. Eegemnetoor Dentra.
Chisee
Mean chattriing Coettcees
1.0
8.
Nic Moan Dagges Rutlo
Do
Namber Edges
50
Number of Exdgess
Campiete graah
Trie griph
b0
62)
0.0
0.0 0.25 0.50 D55 1,00 Begree Centrality
6
Xeon Degree
30
Number Nodes
23 30 00 30
Figure 12: Pipeline to generate the citation graphs and feature histograms The box plot of distributions of four node-level metrics computed for each graph. Four scatter plots of joint features with marginal histograms where each point represents one graph, ground truth and generated graphs and random graphs. In all scatter plots ground truth and GPT-generated graphs overlap almost entirely whereas the random graphs exhibits differently, much like the random baseline in the same field that represent uniformly minimal connectivity, even though they were generated within the same fie-fields. The results exhibit quite similar properties to the random field shown in Figure 2.
Geberned
2NG DAL
6.5
(/44
PCA2
400
022
dinibertee CAMES
-0.2
Figure 13: PCA of graph embeddings with Cosine/EU Distances Summing 3072-d node embed- dings graph level, visualize the embedding space with 2D PCA (contours). The cosine alignment and euclidean dispersion (node level) in three graph-wise alignment diagnostics: mean of focal with reference, mean of reference with reference, and focal vs. sum of references, capturing how well references align with the focal paper and with each other. Similar to graph properties, there is slight change regarding the embedding features the random subfield baseline.
25

---

RANDOM TEMPORAL ORDERED PRESERVE
Here, we examine the temporal aspect of the random baseline in detail. In the original field-level reshuflling, temporal order (reference published before the focal paper) in fact preserved for roughly 80% of focal-reference pairs, even though this was not enforced by construction. We built new, stricter random baseline where we explicitly condition on publication year, so that focal papers are only assigned references from earlier years; under this construction, fewer than 1% of edges violate temporal order. For GPT-generated references themselves we find that about 6% of existing suggestions point to papers published after the focal paper. Importantly, using the temporally con- strained baseline leaves our main conclusions unchanged: random graphs remain easy separate from both ground truth and LLM graphs.
Goumdd piple
Edpa sbot
0.6-
0.0
Degree Centra.
Clasnnnce Centra, Eignowactor Centra
Clestoc Coed.
(b)
6d)
(6)
Mage Cheetereg Coafticeet
130
1500
0.6
02
20
0.0
0.0 D25 8.50 0.75 1.00
Degren Contrnitt
5 10
Moan Degree
20 20
Notes
60
50
Figure 14: structural comparison with temporally constrained random baseline. Structural comparison between ground-truth citation graphs, GPT-generated graphs, and temporally con- strained random graphs. In all scatter plots ground truth and GPPTgeneratee graphs overlap almost entirely whereas the random graphs exhibits differently, much like the random baseline in the same field that represent uniformly minimal connectivity, even though they were generated within explic- itly condition on publication year. Our main results remain unchanged, with random graphs still clearly separable from both ground truth and LLM LLM- generated graphs.
Graph properties
Ground truth vs. GPT
Ground truth vs. Random 0.8546 + 0.0064 0.8533 + 0.0066
GPT vs. Random
Mean Accuracy
0.6119 0.0103
0.8837 : 0.0052 0.8831 + 0.0051
Mean Fl-score
0.6099 0.0101
Table 11: Performance of the RF using temporal preserved random baseline graph properties. Random Forest performance graph-level structural descriptors for distinguishing ground-trutt GPT-generated, and temporally constrained random citation graphs.
26

---

RANDOM-VECTON CONTROL AND PCA-K ABLATION
To test whether the accuracy gains are merely a side-effect of high dimensionality, we run two con- trols. First, we replace all semantic node embeddings with 3072-dimensional i.i.d. random vectors of matching dimensionality to GPT-generated references and ground-truth references in this set- ting both RF and GNNs collapse to chance performance only by exploiting trivial size differences. Second, we project title embeddings to principal components and retrain GNNs: train and vali- dation accuracy closely tracks cumulative explained variance, dropping toward chance for small and recovering only when we retain large fraction of the semantic variance d E 8, 32, 512, 1024. Together, these controls indicate that separability comes from semantic content rather than from the sheer number of embedding dimensions. For the RF, ground-truth-vss seti---s--enerttee accuracy drops to 0.5047, ic. at chance. This shows that what matters is not the raw dimensionality but the structured semantic information in the embeddings.
Model
GAT
GCN
GIN
GraphSAGE
(a)
0.010
GPT
70
0.005
Embeddings
60
0.000
Ground Truth
Density
50
Models
40
30
20
10
col
GCN
GraphSAGE
tf
GAT
||
GIN
o|
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
Figure 15: Accuracy GNN classifiers trained on node embeddings to distinguish GPT- generated from ground-truth citation graphs. The inset reports the control experiment where semantic node embeddings are replaced by random vectors of the same dimensionality: in this set- ting, all architectures drop back to chance-level accuracy.
27

---

Train / Validation Accuracy
0.9
Accuracy
0.7
0.5
proj_ 8 val
proj. dim 312 val
proj
proj
proj_
pruj
proj, dim troin
proj_ dim
0.4
25
50
75
100
125
150
175
200
Figure 16: Effect of embedding projection dimension on classification accuracy Train and vali- dation accuracy for GNN classifier distinguishing GPT-generated from ground-trute truth citation graphs when the 3072-dimensional title embeddings are linearly projected to smaller dimensions. Curves show the mean across seeds. This dimeenionality-contool/o semantic embeddings is not merely due to having more features, but to the information they encode about citation content.
ablation indicates that the advantage of
28

---

Cumulative Explained Variance
1.0
Cumulative Variance Explained
0.8
0.6
0.4
0.2
500
1000
1500
2000
2500
3000
Number of PCA Components
Figure 17: Cumulative explained variance of title-embedding PCA. Cumulative fraction of vari- ance explained by principal components of the 3072-dimensional title embeddings, aggregated all focal and reference papers. The first two components used in the 2-D visualization of Figure capture only a small fraction of the total variance (6%), and the curve continues to rise steadily. A few hundred components explain over 90%. This confirms that the 2-D PCA plot is a heavily compressed view, while classifiers operate in much higher-dimensional semantic subspace.
29

---

ISOLATED NODE SEMANTIC ROLE
We examine the semantic role of isolated GPT-generated references by measuring cosine similarities between their embeddings and the of the graph.
Random
Ground-truth
Isolated
Non-Isolated
In Paper
Cosine similarity
Semantic role of isolated nodes For each reference node, we compute the mean cosine similarity between its title embedding and the embeddings of all other nodes (including the focal paper) in the same graph. The box violin indicates the empirical distribution with the median marked black. Random references have substantially lower similarity and narrower spread, while all three GPT categories much closer to ground truth. Isolated nodes sit slightly below non-isolated and in-paper references but remain far from the random baseline, suggesting that they semantically plausible but more peripheral additions rather than off-topic hallucinations.
Figure 18:
30

---

HYPERPARAMETER SETUP
Table 12: Hyperparameter search space and training setup for GNN experiments.
Hyperparameter
Search Space Setting
Batch Size
{8000, 10000, 13000}
Hidden Dimensions
{32, 64, 128}
Number of GNN Layers
{1, 2, 3, 4}
Learning Rate
Continuous [10- 10
Dropout Rate
Continuous [0.0, 0.5]
Weight Decay
Continuous in [0.0, 0.01]
Training Epochs
Maximum of 500
Early Stopping
Patience 15 epochs (no validation improvement)
Tuning Strategy
Random search per architecture/task
Table 13: Final hyperparameters (Ground truth vs GPT)
Graph properties
Embeddings
Hyperparameter
GCN
GraphSage
GAT
GIN
GCN
GraphSage
GAT
GIN
Learning rate (10 (10-A)
47.3668
8,8814
10.2697
47.3791
1.8859
5.7781
51.0714
15,4030
Hidden dim
32
64
32
128
128
128
Dropout
0.1494
0,2391
0.0003
0.0026
0,3582
0.4984
0.3426
0.2535
Num layers
1
4
3
3
Weight decay
0.0006
0.0001
0.0003
0.0026
0.0054
0.0049
0.0001
0.0005
Batch size
13000
8000
8000
13000
8000
10000
8000
8000
Table 14: Final hyperparameters (Random GPT)
Graph properties
Embeddings
Hyperparameter
GCN
GraphSage
GAT
GIN
GCN
GraphSage
GAT
GIN
Learning rate (10
48.4246
19.2918
69.6247
26,3361
84,7239
94.0900
27.2811
15.4219
Hidden dim
32
64
64
32
32
32
32
128
Dropout
0.0312
0.2455
0.2268
0,0075
0,4938
0.2636
0.0621
0.4876
Num layers
1
3
3
2
Weight decay
0.0006
0.0002
0.0010
0.0013
0.0006
0.0002
0.0016
0.0001
Batch size
13000
8000
13000
10000
10000
13000
8000
8000
31

---

SATURATION ANALYSIS
GIS
GAT
GapliAGE
OCN
OFF
Crach Prreetz
Kandee
0005
Greed tart
0004
382
341
1000
33
43
50
100
0.5
81-
O8
GAT
Grpes
OCk
6N
GM
3.55
Ranken
v
600
Trch
601
%afans
07-
8.-
0.5
85-
ost
GapiSAti
Figure 19: Saturation analysis across diffferent models Panels are arranged by task (rows: Graph Properties. Embeddings) and supervision (columns: GPT vs Random, GPT vs Ground Truth). Top panels: permutation-average
Wasserstein-1 distance between successive cumulative subsets of runs (20-100%). Curves are averaged over 500 random permutations; Smaller values and plateaus show distributional saturation. Across settings, the Wasserstein distance drops sharply by s 30-40% of runs and flattens beyond z 60-70%, suggesting the additional runs provide only limited marginal benefit; model-wist wise convergence rates are broadly similar, with differences reflected in the accuracy distributions shown in the bottom panels.
32

---

RANDOM FOREST
vallr 100 49.01
cati 5280 1006
APOREESEE de 120.25%
2038
MYDIERI 5%
Avaraga Eapth
Figure 20: Random-forest overview Example decision tree illustrating the early splits on embed- ding features where Gini is node impurity. Boxes use separate colors per class where blue presenting GPT class and orange presenting the ground truth class. Histogram of the average leaf depth per tree in the forest showing that most trees are relatively shallow, nearly levels on average; Cumulative share of total Gini decrease as function of depth, indicating that the vast majority of impurity re- duction occurs within the first 10-11 levels.
33


Reviews:
[{"review_id": "43hYL1X9Gu", "reviewer": "Reviewer_owXj", "summary": "This paper investigates whether LLMs when tasked with generating bibliographic references for scientific papers using only parametric knowledge, reproduce human-like citation patterns. Using 10,000 focal papers and around 275,000 references from SciSciNet, the authors construct paired citation graphs: ground-truth reference networks, GPT-generated reference networks, and a field-matched random baseline. Then they evaluate structural graph properties (centrality, clustering, density, etc. ), semantic embedding signals (OpenAI text-embedding-3-large), and multiple Graph Neural Networks (GCN, GAT, GIN, GraphSAGE). Their key results demonstrate that structural topology alone cannot reliably distinguish GPT-generated from human citation graphs, semantic embeddings do reliably distinguish them.\nOverall it is a well written paper.", "questions": "Below are a few comments to the authors:\n\n(a) Please answer the above 3 weak points.\n\n(b) The conclusion states differences lie in semantics, but the dimensions are not unpacked. Attention weights, probing tasks, or PCA factors would help interpretability. What are your thoughts on this? How do you go about addressing this issue?\n\n(c) Generate references with multiple models (Claude, LLaMA, DeepSeek), then train detectors on one and test on others. How the results would look in this case?\n\n(d) Considering temporal bias metrics would be nice. In other words, LLMs are known to over-cite recent literature. Is this really happening in your experiments? If yes, please quantify this directly.\n\n(d) Authors mention isolated nodes but does not explore their semantic role.. It would be nice to understand what such isolated nodes mean?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The following are 3 key strong points of this paper:\n1. Authors carried out rigorous experimental design by using paired graphs per focal paper and apply field-matched randomization to break subtle citation structure, making conclusions robust. This is one of the strongest points of this paper. \n\n2. Authors have done a good job of clearly decomposing Structural vs. Semantic signals. This leads to better interpretability and diagnostic insights.\n\n3. Their experiments provide strong empirical evidence at scale by considering 10,000 focal papers and 9,218 valid paired graphs; this is large for bibliometric graph learning and this adds confidence.", "weaknesses": "1. The analysis carried out in this paper is limited to only GPT-4o. Concept drift/divergence between models is noted but not tested. This is a serious limitation of this paper. \n\n2. While semantic embeddings separate classes, the authors do not identify what semantic dimensions differ (recency, methodology, prestige, jargon)?\n\n3. For GNNs, only titles and simple metrics are used. Full-text content or citation contexts might yield deeper insights.. Is there any reason for this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJxEpxLK6O", "reviewer": "Reviewer_13Z6", "summary": "The paper investigates whether large language models like GPT-4o produce bibliographies that can be distinguished from real, human-authored bibliographies when viewed through the lens of citation graph analysis.  The authors construct a large paired dataset of 9,218 citation graphs from SciSciNet, each with a ground-truth set of references, an equal-sized GPT-generated list, and a random baseline. The experiments show that using structural information alone cannot separate GPT-generated citations from the ground truth using either Random Forests or GNNs. However, when using semantic embeddings (from the title and sometimes the abstract), both RF and GNNs can distinguish the GPT-generated citations from the ground truth.", "questions": "1. Was temporal order preserved in the random baseline graphs?\n2. What percentage of variance is captured by the PCA projection in Figure 3? Perhaps t-SNE would be a better choice?\n3. Is the embedding (semantic of structure) learnable under the GNN models?\n4. If we append a random vector of size 3,067 to the structural node embedding, would it improve the performance of the GNNs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The findings of the paper are timely, as LLM-generated scientific bibliographies are emerging. Knowing the bias in the generated citations would be helpful to all researchers (and reviewers).\n2. The paper provides firm empirical grounding. Not only does the paper present the accuracy results, but it also includes descriptive statistics and visualization to confirm/motivate the results (Figures 2 and 3).\n3. The paper presents a carefully controlled dataset of ~9k citation graphs across three conditions. The results from two distinct machine learning methods (tree-based models and many GNN architectures) align closely.", "weaknesses": "1. The paper focuses on \"parametric knowledge\" and GPT-4o. This setting might not reflect all LLMs' behaviors nor citation recommendation systems that increasingly perform \"deep research\". \n2. While the paper argues that semantic embeddings reveal the key differences between human and GPT-generated citation graphs, this claim is potentially confounded by the extreme disparity in feature dimensionality (3072 vs 5). The observed accuracy gain in the Random Forest might partly reflect model capacity rather than genuine informational superiority. A dimensionality-controlled ablation or matched-feature comparison would strengthen this conclusion.\n3. There was some ambiguity in the text.\n    - The baseline citations are constructed based on the field of the papers. However, it is unclear whether the constructions follow the citations' chronological order.\n    - The paper states that the citation graphs of the three datasets have the same degree. However, summing over vectors is still an unusual option.\n    - Figure 3 shows that the projected embedding of the GPT and ground truth citations overlap, but the classification model can distinguish between them. This means that 2D PCA lost most of the variance.  Thus, it might be helpful to see the plot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5cr7yGRZgH", "reviewer": "Reviewer_cRu8", "summary": "### Summary\n\nThis study presents an approach to assess whether reference lists generated by large language models (LLMs) can be distinguished from those created by humans. The authors construct paired citation graphs using 10,000 papers from SciSciNet.  \n\nTo generate synthetic reference lists, they use GPT-4o, providing paper metadata such as title, abstract, authors, year, and venue. Three citation graphs are created:\n\n- (A) Human ground-truth references (extracted from each paper)  \n- (B) GPT-generated references  \n- (C) Field-matched random baseline (references randomly sampled from papers in the same field and year)\n\n---\n\n### Methodology\n\nUsing these three citation graphs, the authors introduce three classification models to determine whether a reference list was generated by GPT-4o or by humans. These are applied as pairwise binary classification tasks (A vs B, A vs C, B vs C):\n\n1. **Graph Topology:**  \n   A Random Forest (RF) classifier trained on hand-crafted graph-level statistics to distinguish citation networks by their structural topology.\n\n2. **Semantic Embeddings:**  \n   An RF classifier using aggregated text-embedding vectors of paper titles to classify graphs based on the semantic coherence of their reference papers.\n\n3. **Graph Neural Networks (GNNs):**  \n   Four GNN classifiers that jointly learn from citation structure and node embeddings through message passing, capturing both topological and semantic features.\n\n---\n\n### Key Findings\n\n1. The models achieved an accuracy of 0.6 when distinguishing between (A) and (B), showing that GPT mimics the structural patterns of human references.  \n2. Adding semantic features improved discrimination between (A) and (B), suggesting that main differences lie in semantics.  \n3. Structure-only GNNs could not effectively distinguish GPT-generated from human references, while embedding-based GNNs performed well when distinguishing (B) from (C).", "questions": "**Major suggestions**\n1. Include a comprehensive related work section.  \n2. Add a discussion section summarizing and comparing the key findings.\n\n**Minor suggestions**\n1. **Figure 2:** Simplify the figure to reduce textual complexity and improve readability; this could also free up space for expanded related work and discussion sections.  \n2. **Table 3:** Adjust the table to fit within the page margins specified by the template.  \n3. **References:** Add DOIs or URLs to allow readers to verify sources more easily.  \n4. **Grammar:** Correct the minor typo in line 465 (“have have”).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "### Strengths\n\n- **Originality:**  \n  This paper addresses the novel problem of distinguishing LLM-generated reference lists from human ones by constructing citation graphs and developing three classification approaches. While citation graphs and Random Forests are established methods, their application to this specific problem represents a new use case.\n\n- **Quality:**  \n  The study demonstrates a solid experimental setup by evaluating three distinct classification approaches across multiple models. The authors leverage a large-scale dataset of 10,000 papers and 275,000 references, enabling comprehensive and meaningful analysis.\n\n- **Clarity:**  \n  The paper is well-written and logically structured. Each approach is supported with clear explanations and informative figures.\n\n- **Significance:**  \n  The paper tackles a timely research question with little prior work. While earlier studies have examined the detection of LLM-generated content in general, the detection of LLM-generated references remains largely unexplored. This study thus provides a valuable foundation for future research.", "weaknesses": "### Weaknesses\n\n- **Related Work:**  \n  The paper lacks a dedicated related work section and does not fully contextualize its contribution. The authors briefly mention prior studies by Algaba et al. (2024), Mobini et al. (2025), and Algaba et al. (2025), but do not clarify the distinct contributions of each. A more thorough discussion is needed, especially regarding connections to related areas such as LLM-generated content detection, citation network analysis, citation generation, citation recommendation, and citation-related biases.\n\n- **Discussion:**  \n  Although the results of the three classification approaches are presented clearly within individual sections, the paper lacks a consolidated discussion that synthesizes the findings and compares the approaches. A dedicated discussion section would enhance interpretability and depth.\n\n**Overall:**  \nThe paper could be significantly improved by adding comprehensive related work and discussion sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1izwNa7B24", "reviewer": "Reviewer_XBHR", "summary": "The paper investigates whether citation graphs can distinguish between reference generated by LLM and those by humans. The author uses SciSciNet constructs 3 sets of paired graphs for each paper: the original graph, a graph generated by GPT-4o and a random graph. The research finds that only using structural features is difficult to distinguish GPT from humans, but the semantic information can increase the accuracy of distinguishing. Both human and GPT graphs are easily distinguished from random..", "questions": "See my weaknesses above. In addition, lines 132–137: why replace each directed edge with an undirected one? This removes information about “who came first” and “who cited whom,” which I think is the essential structural signal.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The story is interesting and timely, the work empirically investigates whether LLM-generated bibliographies hold detectable signals distinct from human ones.\n2. The paper has clear experimental design. For each paper, the authors build three paired graphs (human / GPT / random) and control reference count, field, and year. This makes it easy to see what structure explains and how much semantics adds.", "weaknesses": "Although the limitations are discussed fairly comprehensively, several issues should be addressed to make the work more well-rounded:\n1. Since the paper is about LLM capability, evaluating only GPT-4o is insufficient. Cross-model experiments are needed (e.g., generate with other LLMs; keep the embedding backbone fixed). Conversely, keep the generator fixed and vary the embedding backbone.\n2. No detailed analysis about the LLM bias. A literature field analysis would reveal performance differences across different disciplines, and a temporal analysis could test for newer vs. older references.\n3. Only a small subset of basic structural features is used in this paper. Will it omit distinguishable structural signals, and what motivate the selection of these features?\n4. Missing metadata normalization policy. The paper does not specify how reference fields are standardized/canonicalized before matching and analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)