Revisiting XRec: How Collaborative Signals Influence LLM-Based Recommendation Explanations
Authors: Cătălin-Emanuel Brița
Hieu Nguyen
Lubov Chalakova
Nikola Petrov
Venue: Accepted by TMLR ()

Abstract: Recommender systems help users navigate large volumes of online content by offering personalized recommendations. However, the increasing reliance on deep learning-based techniques has made these systems opaque and difficult to interpret. To address this, XRec (Ma et al., 2024) was introduced as a novel framework that integrates collaborative signals and textual descriptions of past interactions into Large Language Models (LLMs) to generate natural language explanations for recommendations. In this work, we reproduce and expand upon the findings of Ma et al. (2024). While our results validate most of the original authors’ claims, we were unable to fully replicate the reported performance improvements from injecting collaborative information into every LLM attention layer, nor the claimed effects of data sparsity. Beyond replication, our contributions provide evidence that the Graph Neural Network (GNN) component does not enhance explainability. Instead, the observed performance improvement is attributed to the Collaborative Information Adapter, which can act as a form of soft prompting, efficiently encoding task-specific information. This finding aligns with prior research suggesting that lightweight adaptation mechanisms can condition frozen LLMs for specific downstream tasks. Our implementation is open-source.

Published in Transactions on Machine Learning Research (06/2025)
Revisiting XRec: How Collaborative Signals Influence LLM- Based Recommendation Explanations
Catalin E. Brita" University of Amsterdam
catalin. Attalin.biita@@ssddend....m
Hieu Nguyen University of Amsterdam
hieu. die...sgsueennsssudddd.....
Lubov Chalakova"
University of Amsterdam
Nikola Petrov"
Universily of Amslerdam
Reviewed on OpenReview: https: openreview. net/ forum? id= g
Abstract
Recommender systems help users navigate large volumes of online content by offering person- alized recommendations. However, the increasing reliance on deep bearning-based techniques has made these systems opaque and difficult to interpret. To address this, XRec (Ma et al., 2024) was introduced a novel framework that integrates collaborative signals and textual descriptions of past. interactions into Large Language Models (LLMs) generate natural language explanations for recommendations. In this work, reproduce and expand upon the findings of Ma et al. (2024). While our results validate most of the original authors' claims, we were unable to fully replicate the reported performance improvements from in- jecting collaborative information into every L.M attention layer, nor the claimed effects of data sparsity. Beyond replication, our contributions provide evidence that the Graph Neural Network (GNN) component does not enhance explainability. Instead, the observed performance improvement is attributed to the Collaborative Information Adapter, which act form of soft prompting, efficiently encoding task-specific information. This finding aligns with prior research suggesting that lightweight adaptation mechanisms can condition frozen LLMs for specific downstream tasks. Our implementation is open-soured
1 Introduction
Recommender systems [Resnick & Varian 1997] help users filter through vast amounts of online content by providing personalized suggestions tailored to their preferences and interests (Cheng et al. 2021 2022 Explainable recommendation models (Zhang et al. 2020) have emerged to enhance trust, decision- makiiinnmaking, and satisfaction by providing explanations for their suggestions (Zhang 2019
Recent advancements in Large Language Models (LLMs) (e.g., Touvron al. 2023 Grattafiori al. 2024 have revolutionized the field of explainable recommendations.. Earlier work, such as PETER (Li et al. 2021), used transformers to generate explanations, while more recent works, like PEPLER (Li et al. 2023 integrated GPT-2 (Radford et al.) 2019) with user-item IDs as prompts to produce contextually rich and co- herent justifications. Building on these foundations, LLM2ER (Yang et al. 2024b) introduced a personalized prompt module, overcoming the limitations relying solely on user-ilem IDs for explanation generation.
Equal contribution.
https://gthuuu.cm//225//////af com/tr2512//e/ttet...

---

Published in Transactions on Machine Learning Research (06/2025)
A more recent study, XRec: Large Language Models for Explainable Recommendation (Ma et al. 2024 introduces a novel model-agnostic approach to address the limited availability of explanation data and the poor generalization of ID-based methods in zero-shot scenarios. This framework integrates collaborative signals (patterns of user interactions, such as revicws) into LLMs by injecting them into both the imput prompt and the model layers, allowing XRec to extract meaningful insights from user preferences. We chose to focus on Xeec because, to our knowledge, it was one of the first methods to combine Graph Neural Networks with LLMs to generate explainable recommendations, achieving state-of-the-arl performance.
This paper attempts to reproduce and extend the main findings of Ma et al. 2024 through the following:
Reproducing XRec results: We use the codebase of the original paper to replicate its experi- ments, verily the authors' claims, and analyze the required computational resources.
Extending the ablation study: We investigate the role of collaborative signals by evaluating dif- ferent GNN-based recommender systems that capture such signals from graph structures, including LightGCN He et al.] 2020) and NGCF (Wang et al.) 2019). Additionally, we test three ablations We extend the evaluation by using Sentence Transformer Similarity (STS) Thakur et al. 2020) and introducing LLMScore.
that entirely remove graph-based collaborative information.
We identified two instances of data leakage caused by the original data generation process. The first involves using the same review to generate both the ground truth and the input to the framework, while the second caused by the inclusion of test set information within the training data. We propose solutions to address these issues and analyze their impact.
Addressing data leakage:
Code adaptation and improvement: We modily and improve the original codebase to support open-source LLMs for dataset generation and evaluation, facilitating future extensions and repro- ducibility. Additionally, we add scripts for tasks like dataset creation and data-sparsity testing.
After performing the above steps, we verified that XRec (Ma et al. 2024) outperforms the baseline models, Our findings also confirm that incorporating that both the profile and the injection component can contribute to improved personalization. However, encountered challenges in reproducing the claims that injecting collaborative information across all LLM attention layers improves performance and that XRec is more effective under increased data sparsity.
and item profile information improves performance and
Furthermore, our in-depth analysis of the impact of collaborative information reveals that the performance improvement in XRec (Ma el al.) 2024) can be attributed to the Collaborative Information Adapter. We hy- pothesize this occurs because (1) collaborative signals may be redundant when unique identification features are available, even if randomly assigned, since the adapter can learn these signals as effectively as GNN- based models and (2) the adapter functions as form of soft prompting, efficiently encoding task-specific
information [Lester et al. 2021 Li et al. 2021).
While reproducing the datasets proposed by Ma et al. (024)) we idontified two instances of data Ieakage, which we subsequently addressed our attempt to reproduce the datasets. Despite the presence of these data leakage cases, our observations indicate that they did not significantly affect the overall claims or results.
2 Scope of reproducibility
Ma et al. (2024) introduced XRec, post-hoc technique that improves recommender system explainability
by providing user-level personalixation. Ma et al. (2024) make five key claims, which we investigate:
Claim 1: XRec improves over baselines in explainability and stability. The model im- proves explanations in recommender systems by integrating collaborative information and textual descriptions into LLMs, outperforming state-of-the-art baselines in both explainability and stability.
httpss://gtthub. httpa/u...oooo//////eeeeee
2

---

Published in Transactions on Machine Learning Research (06/2025)
Collaborative information injection improves explainability and stability. XRec achieves higher accuracy by injecting collaborative information across all attention layers of the LLM, rather than simply passing it in the imput prompt.
Claim 2:
User and item profiles improve explainability and stability. Adding user and item profiles to the prompt improves explainability and stability by providing contextual information.
Claim 3:
Claim 4: XRec has superior performance with increasing sparsity. The model demon- strates robust performance across varying sparsity levels, showing improved results user frequency decreases. Moreover, in zero-shot scenarios, its performance is comparable to other sparsity levels.
Claim 5: The model generates personalized explanations. XRec generates customized explanations for each distinct user-item interaction, offering users meaningful and unique insights.
Moreover, we extend the work by answering the following research questions:
How do Graph Neural Network (GNN)-based collaborative signals and the Collaborative Information Adapter influence the explainability and stability of XRec?
RQ 1:
RQ 2:
What are the effects of datass dataset reproduction in XRec on explainability and stability, especially when addressing sources of data leakage?
3 Methodology
3.1
Model description
XRec (Ma et al. 2024 is framework that uses Large Language Model (LLM) to generate natural language explanations of why a recommender system suggested an ilem to user. XRec (Ma el al. 2024 as visualized in Figure !
is a framework that uses an LLM to generate natural-languat explanations for recommendations. It consists of four main components: (i) Graph Neural Network (GNN)-based recommender system that learns user-item interaction patterns, (ii) an adapter that transforms these learned embeddings into the LLM's input space, (iii) textual profiles of users and items constructed via summarization, and (iv) an attention injection mechanism that reinforces collaborative signals during explanation generation.
GNN-based recommender system The GNN-based recommender system models user-item interactions as a graph, with users and items as nodes and interactions as edges. GNNs embed users and items into fixed-size vectors using messagg-passsng mechanism (Gilmer et al. 2017). which aggregates information from neighboring nodes. XRec employs LightGCN (He et al. 2020), model with simple prssage-passingg mechanism which lacks linear transformations and nonlinear activations during aggregation. This is shown by Eq. 1 where e!! is the embedding of node at layer I, and Nr is the set of neighbors of node
e (1+1)
Σ
(1)
IEN. V
To investigate how the GNN affects XRec's performance, we experiment by replacing LightGCN with NGCF Wang et al. 2019). NGCF has a more complex message-passe passing mechanism that uses nonlinear activations and learnable weight matrices (W1, W2), as shown in Eq. 2
(1+1) =0 Wie!! + Σ
W. (Wie (1) + W2 (e o e )))
s.N,
Both GNNs are trained with the Bayesian Personalized Ranking loss RRndlle al. 2009 (Eq. 3), which favors higher scores for observed interactions. lg regularization is also applied. Here, L denotes the loss
3

---

Published in Transactions on Machine Learning Research (06/2025)
Interaction Graph
Graph Neural Networks
Tolon Emdh.
Collaborative Information Empowered LLMs
Trainable Parameter
Afyere A Knteling
De
Liwr2
Freezed Parameter
80
Generated Explanation
lilk:
descrgtion....
g
prefierred by
nen
would interact with the New becouse
User Profile Generation
#4 cz
a,
Entenated hems:
(itle:
The would ftems which
Prompt:
recand = +USER EMBEDS
item recand: KITEM AMBEDS
Ground Truth Explaantion Generation
user profile: :...
profile
BaSt would innerat with Hem bocasse
<EXPLAIN POS
betwren
Figure 1: XRec framework consisting of (i) GNN-based recommender system (top left): embeds user- item interactions, (ii) Collaborative Information Adapter (MoE Adapter): transforms user and item embeddings to LLM input space. (iii) User and Item Text-Based Profiles (bottom left): text profile of users and items (iv) Attention Injection Mechanism: injects collaborative information into each LLM attention layer. Embeddings and profiles are combined into a prompt that is processed by the LLM, where collaborative signals are used throughout to generate the final explanation. Reprinted from Ma et al. (0024).
function, M the number of users, Oui refers to the predicted rating of user u for item i, a represents the sigmoid function, E(0) denotes the embedding matrix, and À is the l regularization parameter.
L L= M
Σ In In Duz) + X1| 8=1 W JEN.
Collaborative Information Adapter To effectively incorporate the collaborative information learned by the recommender system into the LLM, XRec uses an adapter which transforms the user and item embeddings into the input space of the LLM. This adapter uses Mixture-of-Experts (MoE) architecture which effectively captures diverse aspects of the collaborative information.
User and Item Text-Based Profiles In addition to the adapted GNN embeddings, XRec takes as input uaxtual profiles of users and items. Item profiles are created by prompting an LLM to summarize each item't 's description, while user profiles are generated by summarizing descriptions of items the user has interacted with. XRec's imput is constructed shown in Figure where <USER EMBED> and <ITEM EMBED> are the embeddings generated by the GNN.
Attention Injection Mechanism A potential issue with prompt construction is that collaborative in- formation, which is crucial for capturing user-item relationships, is limited to single token. To ensure the collaborative signal is not lost in a long prompt, it is injected into the key, query, and value of all self-attention layers. This is shown in Eq. 4 where W is the projection matrix, a represents the adapted user or item embedding, and poriginal and fnww (9.6.0) denote the query, key, and value before and after injection, respectively. [g.k.))
(a (xx) foriginal (x,) + W (g.k,)
(4)
3.2 Evaluation metrics
We evaluate the model using ALLM-based metries, as they better reflect human judgment and capture context- aware semantic similarity, rather than relying solely token-level analysis. First, to ensure a fair comparison
4

---

Published in Transactions on Machine Learning Research (06/2025)
with the bascline model performance reported by Ma et al. (202)) GPTScore uses an L.M as judge by prompting GPT-3.5 Turbo to assess the semantic similarity bet ween ground-truth explanations and generated explanations. Wang et al. (2023) showed that GP'TScore has a high correlation with human judgment and shows a doep semantic understanding, making it suitable for assessing the alignment between generated explanations and groundt explanations. However; GPTScore's ' use of a closod-source model limits accessibility and reproducibility and introduces modcl-specific bias.
use GPTSScre (Wang al. 2023
To address these limitations, we propose LLMScore, which uses multiple open-source LLMs jury (Verga et al. 2024), each outputting alignment score with the same prompt used by GPTScore. LLMScore averages these scores to mitigate model-specific biases and improve accessibility. Scores are obtained over five runs to ensure stability. The models used are LLaMA 3.1 8B Instruct, LLaMA 3.2 3B Instruct (Grattafiori
et al. 2024), Gemma 2 9B-IT (Rivière et al. 2024 and Qwen 2.5 7B- IB-Instruct (Yang et al. 2024a). The
prompt used for scoring provided in Appendix B
In addition to LLMScore, we use several other metries to capture explanation quality. BERTScore (Zhang et al.) 2019) computes cosine similarity over BERT (Kenton & Toutanova 2019) embeddings for token- level comparison of synonyms and paraphrases. Ma et al. (2024) reports Precision, Recall, and F1, but we use only F1 since it combines the other two. Precision and Recall details are reported in Appendix E.2 Complementing this, BA BARTScore (Yuan et al. 2021 measures the likelihood of regenerating ground-truth text using pre-trained BART (Lewis) 2019), making it effective for assessing fluency and coherence. To further align with human judgment, we include BLEURT (Sellam et al.) 2020), a fine-tuned BERT model trained on numan-annotanl text pairs, which enhances sensitivity to subtle differences in meaning.
Beyond semantic quality, assess personalization by mcasuring the diversity of generated explanations. We use USR (Unique Sentence Ratio) (Li et al., 2020) the ratio of unique to total explanations, providing insight into the variability of generated content. To build on this, we use STS (Sentence Transformer Similarity) (Thakur et al. 2020). which uses SBERT RReimers 2019) embeddings and cosine similarity to evaluate the semantic similarity of generated explanations.
Finally, to compare the performance of LightGCN (He et al. 2020 and NGCF (Wang et al. 2019), we employ four metrics: Precision@20 measures the proportion of relevant items among the top 20 recom- mendations, indicating accuracy; Recall@20 measures the proportion of relevant items retrieved, indicating coverage; NDCG@20 (Normalized Discounted Cumulative Gain) measures ranking quality by weigh- ing relevance and position, prioritizing relevant items appearing earlier and MRR@20 (Mean Reciprocal Rank) focuses on the rank of the first relevant item, rewarding higher-ranker relevant items.
4 Results
4.1 Datasets
Dataset processing To build XRec, Ma et al. (024) use three datasets, each containing user reviews for various items: Amazon Review Data Ni et al. 2019), Google Local Data [Li et al. 2022 Yan et al. 2023), and Yelp Open Datase 3 These were processed into what we refer to as [Og] Amazon, [Og] Google, and Yelp datasets. As we were unable to find the exact procedure used to create the
datasets, we independently reproduce them and refer to them as [Rc] Amazon, [Re] Google, and [Re]Yelp Yelp.
Detailed processing steps for these datasets, along with information about data splitting, can be found in
Appendix C Table presents statistical comparison between the original and reproduced datasets.
Train-test overlap XRec is trained in two phases: (1) training the recommender system using the full interaction dataset, and (2) training the Collaborative Information Adapter using the explanation dataset (a subset of the interaction dataset). We observed overlap between the train, validation, and test splits of the original explanation and interaction datasets proposed in Ma et al. (2024). with overlap percentages provided in Appendix C.3 This issue is addressed in the reproduced datasets.
$ http::///iaaieep11199ggghh.....,
v2/
5

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Dataset statistics from
Ma et al. (2024)
([Og]) and our reproduced version (Re)).
Dataset
[Og] Amazon
[Re] Amazon
[Og] Google
[Re]Google
[Og] Yelp
[Re] Yelp
# Users
15,349
15,069
22,582
19,503
15,942
15,962
# Items
15,247
15,028
16,557
18,998
14,085
14,085
# Interactions
360,839
350,644
411,840
400,038
393,680
393,680
Ground-truth and profile generation For the original datasets, ground truth explanations are gener- ated based on user reviews in order to make user preferences explicit. For item profiles, the Google and Amazon datasets use item names and descriptions, while the Yelp dataset also includes all user reviews of that item. User profiles are generated based on information from each item including the item's name, its associated profile, and the review the user left for that item. To improve dataset reproducibility, we use LLaMA 3.1 8B Instruct. (Grattafiori et al. 2024) for generation, replacing the closed-source GPT model used in Ma et al. (2024).
has interacted with,
Input contamination The user generation process rntrouuess dataset leakage across all original datasets, while the Yelp item profile generation process introduces additional leakage specific to Yelp. This occurs because user reviews are used to generate both the ground-truth explanations and the input (profiles). This raises two issues: (1) It creates an overlap between the input and ground truth, potentially causing the model to rely this overlap instead of learning meaningful patterns from user-item interactions, which limits generalization during testing this shortcut is unavailable (2) It compromises the evaluation process by leaking ground-truth test information into the test input profile. To address this in our reproduced datasets, we exclude all review information from input prompts used to generate profiles.
4.2 Training Configurations
The training process consists of two phases: training the GNN model and training the Collaborative In- formation Adapter module. The training process followed the hyperparameter configurations specified in Ma et al. (2024). An early stopping criterion is applied to the GNN-based models based on the Recall@20 metric. The LLM model used is LLaMA 2 7B (Touvron et al. 2023) which remains frozen throughout the training process. Appendix shows the exact hyperparameter values for each phase.
Training and inference are conducted using single NVIDIA A100 40GB GPU, while evaluation is performed two NVIDIA T4 16GB GPUs. Training takes approximately 15 hours for the Amazon and Google datasets and 12 hours for the Yelp dataset, for both the original and reproduced versions. Inference requires about 4 hours per dataset, and evaluation takes around 1.5 hours per dataset.
4.3 Results reproducing original paper
This section details our reproduction of the findings of Ma et al. (2024), where we retrain the XRec model and its ablations. Our goal is to validate key claims regarding the model's explainability and stability, ensuring consistency with the original results.
Claim 1: XRec improves over baselines in explainability and stability. We reproduce the results of Ma et al.
(2024) by retraining (1) XRec on the original datasets, and (2) XRoc (w/o Profile), where user and item profiles were excluded from the input prompts to ensure a fair comparison with the basclines that did not have access to profile information. Table 2 presents the original XRec (Ma et. al. 2024) results alongside the scores from our reproduced models. Our reproduction shows improvement in GPTScore and decrease in GPT across all datasots. While we camnot determine the cause of this change, because the metric relies on closed-source mode 6 we observe that the performance trends remain consistent with the original results.
For BERT-based metrics, BARTScore, and BLEURT, the results remain consistent with the original trends, with minor negligible variations. These metrics confirm that XRec maintains high explainability and stability,
6Thhi prevents from verifying whether the same model version and internal system prompt were used both cases.

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Comparison baseline models and our reproduced results. Values for all baselines and XRec (denoted [Rec) are from (Ma et al.) 2024). while our reproduced XRec results appear in italics.
BERT aggregates Precision and Recall, which are reported in Appendix E.2 score; underlined indicates the second highest.
Bold indicates the highest
Explainability 1
Stability J
Metrics
GPTScore
BERT
BARTScore
BLEURT
USR
GPT atd
BERT a/d
BART
BLEURT and
[Og]Amazon
AutSeq
76.08
0.3687
-3.9440
-0.3302
0.7757
12.56
0.1275
0.5080
0.299
NET
75.63
0.3443
-3.9806
-0.4073
0.5413
12.82
0.1321
0.5101
0.3104
PETER!
77.65
0.4043
-3.8908
-0.2937
0.8480
11.21
0.1098
0.5144
0.2607
PETER+1
76.07
0.3876
-3,9647
-0.3293
0.4493
11.99
0.1245
0.5131
0,2805
PEPLER!
78.77
0.35-13
-3,1142
-0.2950
0.9563
11.38
0.0893
0.5064
0.2195
Reec
82.57
0.4122
338035
-0.1061
1.0000
9.60
0.0800
0.4832
0.1780
(Orig]RRe (w/o profile)
81.77
0.4106
-382188
-0.1294
1.0000
9.60
0.0786
0.4799
0,1803
XRoc
83. 75.75
0.4095
3.7778
0.1
1.0000
9.40
0.0817
0.4985
0.2256
XRex (wo/ profle)
83.28
0.4094
3.8328
-0. 1839
1.0000
9. 16
0.0772
0.4801
0.2178
Yelp
Au22Sqq
63.91
0.2379
4.5316
0.6707
0.7583
15.62
0.1147
0.5616
0.247
NRT
61.94
0.1495
4.6142
0.7913
0.2677
16.81
0.1581
0.5612
0.2728
PETER
67.00
0.2513
-44100
-0.816
0.8750
15.57
0.2230
0.5800
0,3555
PETER+1
67.98
0,2533
-439733
+0,3555
0.8637
13.80
0,1701
0,5665
0.3421
PEPLER!
67.54
0.3052
-4.4633
-03354
14.18
0.1050
0,2524
RRee
74.53
0.3730
-43911
-0.2287
1.0000
11.45
0.0852
0.5770
0,2322
OOrig[Ree (w/o profile)*
71.81
0.3657
-44003
-02446
1.0000
12.71
0.0919
0.5717
0.2272
XRec
79. 72
0.3527
4.2969
-0.3089
1.0000
10.65
0.0967
0.5758
0.2334
XRee fw/o profile)
75.68
0.3636
4.3717
-0.3303
1.0000
11.13
0.1014
0,5676
0.2514
Google
AutSeeq'
61.31
0.3636
-426627
-0.4671
0.5070
17.47
0.1403
0.6663
0.3198
NRT
58.27
0.3496
-4.9115
-0.4888
0.2533
19.16
0.1571
0.6620
0,3118
PETER!
65.16
0.3881
4.1327
-0.3375
0.4757
17.00
0.2005
0.6701
0.3272
PETER+1
66.74
0.4047
4.1273
-0.3677
0.4887
15.23
0.1411
0.6515
0.3095
PEPLER1
69.71
0.3987
-4.1542
-0.2047
0.
17.11
0.1353
0.6800
0.3114
[Orig]
69.12
014311
4.1647
-0,2377
0.9993
14.24
0.0937
0.5700
0.2114
[ore (w/o profile)*
69.71
0,4310
-4.1142
-0,2026
0.997
14.09
0,1034
0.6465
0,2439
XRe
71.18
0.4331
1686
-0.2366
0.9993
13.29
0.0933
0.6569
0.2509
XRoe fw/o profile)
71.66
0.4395
-4.1237
-0.1829
1.000
13.05
0.0931
0.6329
0.2392
i This score was taken from Ma et al.
2024
as the best and second-best. scores are consistently achieved by our reproduced models. Additionally, while XRec (w/o Profile) performs slightly worse than XRec, it still consistently outperforms the baselines. This suggests that even when restricted to the same level of information as the basclines, XRec has superior performance. Overall, these results support CClim 1
Table 3: Comparison of GPTScore and LLMScore for XRec and XRec (w/o profile) on the original datasets, with LLMScore averaged over five runs. Given their high correlation, only LLMScore is used in later analysos.
Dataset
Model
GPTSccre
LLMScore
XRec
83.75
67.53 1: 11.65
Og Amazon
XRec (w/o profile)
83. 33.28
66.95 + 11.65
XRec
79.72
61.67 + 12.61
Og]
XRec (w/o profile)
75.68
59.82 + 12.11
XRec
71.13
55.81 + 13.04
Goolee
XRec (w/o profile)
71.66
56.49 +: 12.98
we use GPTScore for fair comparison with the basclines from Ma al. (2024) However, reproducing results is challenging since GPTScore closed-source. Instead, we propose LLMScore, which employs multiple open-source language models for scoring. Table that T.LMScore mirrors GPTScore's
To verify Claim

---

Published in Transactions on Machine Learning Research (06/2025)
patterns: best performance Amazon, worst Google, and reduced performance without profiles except Google. Therefore, I.M.Score is the primary evaluation metric for the remaining experiments.
XRec
XRec (w/o profile)
XRec (w/o injection)
XRec (w/o profile & injection)
0.44
0.12
BERTStability F_
0:10
LLMStability
LLMScore
BERTScore
0.44
0.00
0:38
50
6.07
[0g]Amazon
[og]Google
[Og]Yelp
[og]Amazon
toglGoogle
[Og|Yelp
[0g]Amazon
[0g]Google
[Og]Yelp
[og]Amazon
[og]Googie
[Oy]eolp
Ablation study results of the original datasets the full model and three variants: without profile, without injection and without both. The vertical bars represent the standard deviation obtained over 5 runs. BERT is deterministic and is shown without error bars
Figure 2:
Collaborative information injection improves explainability and stability. To verify this claim, we perform ablation studies using: (1) XRec, (2) XRec (w/o profile), where user and item profiles are not given as input, (3) XRec (w/o injection), where embeddings are not injected into the attention layers, and (4) XRec (w/o profile & injection). Following Ma et al. 2024), we evaluate them using LLMScore and BERT",
Claim 2:
Figure Fiowss that on LLMScore, XRec follows a similar trend reported by Ma et al. (0024). outperforming Showss the other configurations on Yelp and ranking second on Amazon. However, on the Google datasel, XRec performs the worst. Although there is high variance in LLMScore, include an explanation in Appendix E.1 and show that all scoring LLMs exhibit the same trend, confirming the conclusiveness of these results.
Moreover, in terms of BERTE: XRec consistently ranks last or lies for the worst performance across all three datasets. On the other hand, for LLMScore and LLMStability, XRec (w/o injection) outperforms all other ablations on two datasets (Amazon and Google), and ranks second on the other dataset (Yelp). These results contradict Claim 2 and suggest that, on average, injection reduces performance.
One possible explanation for the lack of performance improvement from injecting collaborative information lies in the characteristics of the underlying graph neural network (GNN). According to Qin et al. (2024), injection is most effective when the model can disentangle the various latent factors that shape user prefer- ences and item attributes. However, LightGCN (He et al. 2020) lacks this property, which may explain its negative impact on performance.
Claim 3:
User and ilem profiles improve explainability and stabilily. We evaluate this claim with the ablation study in Figure 2 XRec (w/o profile) performs worse than XRec in terms of LLMScore and LLMStability Amazon and Yelp. Similarly, XRec (w/o injection & profile) underperforms across all three datasets compared to XRec (w/o injection). However, models without profile exhibit slight yet. negligible improvement in FERT" compared to those that incorporate profiles. These findings support Claim 3
XRec has superior performance with increasing sparsity. We investigate the model's performance under varying levels of data sparsity. Specifically, as done in Ma et al. (2024), we divided the test data into six groups: one for zero-shot users (users absent from the training data) and five quantiles representing increasing ranges of user interactions. Appendix C.4 provides interaction frequency ranges and statistics for each group. As shown in Figure this grouping reveals that while BERTScore shows slight improvement for users with sparser interactions, the LLM-based metrics behave differently: explainability decreases and stability increases with growing sparsity. These conflicting trends suggest that Claim 4 is not reproducible and needs further investigation, though performance generally improves with higher interaction frequency.
Claim 4:
Claim 5: The model generates personalized explanations. Ma et al. (2024) measure personalization using USR (Li al. (2200, which captures lexical uniqueness but overlooks semantic equivalence. For instance, 'I am going to the store' and "I am heading to the store' both score 1.0, despite being semantically identical.
8

---

Published in Transactions on Machine Learning Research (06/2025)
[Og]Amazon
[Og]Google
[Og]Yelp
70
0.475
13-
0:450
BERTStability F1
0.10
LLMScore
LLMStablltty
12-
BERTScora
60
0.400
11-
0.375
0.08
10-
0.350
0.07
zero-shot
tst2 tst3
zero-shot
tst
tst2
tst3
tst4
tsts
zero-shot
tsts
zare-shot
tst
tst2
tst
tsté
tsts
Figure 3: XRec's performance on six data splits with increasing sparsity level on the original datasets.
To test this, we sampled user-item interactions from cach dataset and generated 100 explanations per sample. In all cases, USR was 1.0 even though the explanal ions conveyed the same meaning. This suggests that USR is not appropriate measure of personalization. To address this, we compute an aggregate STS score by averaging the pairwise STS values (hakur et al. 2020), providing an estimate of the overall personalization of generated explanations. On the same explanations, STS yields an average score of 0.78, indicating that the explanations are highly similar in meaning despite minor lexical differences. However, STS tends to assign higher similarity scores to sentences with similar structures, which is often the case with our generated explanations. Despite this limitation, STS provides a more meaningful measure of similarity than USR.
Table 4: Evaluation results of different XRec variations across datasets on the STS metric.
Dataset
XRec
XRec (w/o profile)
XRec (w/o injection)
XRec (w/o profile & injection)
XRee (w/o GNN)
Ogg] Amazon 0.5667
0.6563
0.6614
0.6936
0.4139
Ogg Yelp
0.3802
0.3601
0.3671
0.41419
0.2759
Google 0.5891
0.5910
0.6006
0.5863
0,2807
Table 4 presents the average pairwise STS scores for each dataset. The [Og]Amazon and [Ooglo explana- tions exhibit overall high STS, suggesting a lower degree of personalization in comparison to the explanations for [Og]Yelp This difference can be attributed to the higher level of detail in Yelp reviews compared to the other datasets. This shows that data quality affects the personalization capabilities of XRec. Addi- tionally, we observe that for Amazon, incorporating user-pro asor-erofiles and injection improves personalization. However, for Google, these factors do not significantly impact the STS score. Finally, removing the GNN makes generated explanations more diverse. We hypothesize that this is due to the fact that when XRec is trained with the GNN component, the MoF adapter learns to prompt the model in such a way that the explanations follow a similar structure. As result, removing this component allows the model to produce more diverse explanations, but at the cost of reduced explainability scores as shown in Table
notice that
4.4 Results beyond original paper
This section explores additional aspects of XRec's performance. We analyze the impact of GNN collaborative signals and the MoE adapter, and assess the effect of data leakage mitigation. These extensions provide deoper insights into the modcl's capabilities and potential limitations.
4.4.1 Impact GNN and Collaborative Information Adapter
How do Graph Neural Network (GNN)-based collaborative signals and the Collaborative Information Adapter influence the explainability and stability of XRec? To assess the impact of GNN-based collaborative signals on XRec, we replace its originally used LightGCN (He el al. 2020) with NGCF (aag et al. 2019) referring to this variant as XRec w/ NGCF. For clarity, we refer to XRec with its original GNN architecture this soction. NGCF (Wang et al.] 2019 is selected due to its similar architecture LightGCN (He et al. (2020). with only additional linear transformation and non-linear activation. This allows investigate the relationship between GNN recommendation performance and XRec explainability
RQ 1:
as XRoc w/ LightGCN

---

Published in Transactions on Machine Learning Research (06/2025)
while minimizing confounding factors from architectures. Additionally, test three collaborative-signal ablations: XRec (w/o GNN, fixed), where GNN outputs are replaced with fixed random embeddings. XRec (w/o GNN, random), where these are replaced with variable random embeddings, and XRec (w/o GNN & MoE), removing both the GNN and the Collaborative Information Adapter (MoE).
Table 5: Performance comparison of LightGCN and NGCF models. The best values are in bold.
Dataset
Model
Recall
NDCG
Precision
MRR
LightGCN
0.0719
0.0426
0.0175
0.0724
Ogg Amazon
NGCF
0.0514
0.0319
0.0131
0.0587
LightGCN
0.0614
0.0382
0.0160
0.0677
Yog]
NGCF
0.0571
0.0365
0.0156
0.0668
GCN
0.0791
0.0433
0.0129
0.0575
Google
NGCF
0.0571
0.0327
0.0094
0.0455
We compare the performance of the two recommender systems on the item recommendation task. Table shows that LightGCN outperforms NGCF across all evaluation metrics and datasets. We use the embeddings from these two models as imput to the MoE Adapter. Table that XRec (w/ NGCF) outperforms XRec (w/ Ligh on average, achieving higher BERT-based scores two datasets and higher LLMScore onc. This suggests that better item recommendations do not guarantee improved XRec explainability.
Table 6: Performance comparison of XRec with LightGCN and NGCF as collaborative information algo- rithms, as well as three variants where explicit collaborative information is removed. BERT aggregates Precision and Recall, which are reported in Appendix E.2. Values in bold indicate the highest score, while underlined values denote the second-best. LLMScore and LLMStability are averaged over five runs.
Explainability
Stability 1
Metrics
LLMSoore
B
BARTScore
BLEURT
STS
LLMStability
BERT
BART. BLEURT and
Amazon
Xloe (w/ LightGCN)
11.61
0.4095
-3.7773
-0.1783
0.57
10.67
4.50
0.0817
0.4985
0.2256
XRee (w/ NGCF)
67.97 + 11.66
0.4164
-3.7789
-0.1445
0.62
10.58
4.57
0.0806
0.4913
0.2118
XRec (wr/ GNN, fixed)
68 05 + 11.69
0,4089
-3.8171
-0.1466
0.67
10.19
+ 4.05
0.0764
0.4766
0.2077
XRec (w/o GNN, random)
65.33 + 11.81
0.1179
-3.8575
-0.1264
0.69
11.31
4.38
0.0816
0,1861
0.1990
XRec (w/o GNN & MoE)
62.66 1 12.14
0.3013
-4.0502
-0.4729
0.42
11.05
4.29
0.0804
0.4336
0.2439
YOg]
XRee (w/ LightGCN)
61.67 12.61
0.3527
-4,2969
-0.3089
0.38
10.65-+-0
4.73
0.0967
0.5758
0.2334
XRec (w/ NGCF)
12.19
0.3736
-1.2719
-0.3053
0.33
10.67
1.76
0.0841
0.5637
0.2128
XRec (w/o GNN, fixed)
61.20
+ 12.49
0.3658
-4.3070
-0,2971
0.36
10.61
1 4. 7.78
0.0897
0.5694
0.2101
XRoc (w/o GNN, random)
59.34
1 12.01
0.3926
-4.4425
-0.2608
0.45
11.00
1: 4.46
0.0877
0.5790
0.2112
XRooe (w/o GNN & MoE)
58.68
13.56
0.2710
4.6124
-0.3615
0.27
11.51
4.80
0.0953
0.5392
0.2174
OggGoogle
XRec (w/ LightGCN)
55.81 + 13.05
0.4331
-4.1686
-0,2366
0.59
11.84
4.35
0.0935
0.6580
0.2509
XRec (w/ NGCF)
0.4233
-4,2445
-0.2652
0.54
12.12
4.30
0.0946
0.6558
0.2635
XTte (w/o GNN, fixed)
57.25
12.87
0.4306
-4.1158
-0.1776
0.53
12.04
4.44
0.0959
0.6488
(L2398
XRoe (w/o GNN, random)
56.71
+ 12.89
0.4310
-4.1595
-0.1780
0.63
11.96
4.45
0.0937
0.6542
0.2390
XRee (w/o GNN & MoE)
50.40
+ 14.29
0.2456
-4.5504
-0.5188
0.28
12.17
+
4.58
0.0918
0.5725
0.2451
We observe that removing both the collaborative information and the adapter in XRec (w/o GNN & MoE) results the lowest performance, highlighting the importance of these components. However, this boosts personalization as measured by STS.
When comparing with XRee (w/o GNN, fixed), which uses a random embedding vector instead of embeddings generated by GNNs, we do not observe any performance drop. Instead, find that it achieves the best performance on two datasets and ranks second-best. on the other. Mcanwhile, XRec (w/o GNN, random) performs worse than models with the MoE adapter but better than those without it. This suggests that removing collaborative signals from the GNN has minimal impact on explainability and that the MoE effectively learns meaningful information, though it struggles when the input is random rather than fixed.
We hypothesize that this effect arises from two possible causes, though further research is needed to confirm the exact mechanism. First, the MoF adapter may interpret the embedding vector set of unique
10

---

Published in Transactions on Machine Learning Research (06/2025)
identifiers, enabling it to implicitly learn collaborative signals even without GNN-based embeddings. The poorer performance of XRec (w/o GNN, random) supports this hypothesis, as the randomized inputs disrupt the model's ability to retain meaningful identification information. Second, both STS scores and manual inspection revealed structural differences in sentence composition between models with and without the MoE adapter. This suggests that. incorporating the lightweight adapter, positioned almost as a prefix the prompt, may unintentionally act as a form of soft prompting. This effect aligns with prior work prompt tuming Lester et al. (2021); Li et al. (2021), which demonstrated that small, lightweight adapters condition LLMs for downstream tasks without requiring full fine-tuning.
4.4.2 Effect of Data Leakage Mitigation
What are the effects of dataset reproduction in XRec on ezplainability and stability, especially when addressing train-lest overlap and the use of shared reviews for both profile generalion and ground truth creation? We identified two cases of data leakage in the original datasets (Ma et al., 2024): Input Con- tamination, where the same reviews are used to generate both ground-truth explanations and input profiles; and Train-Test Leakage, where training, validation, and test sets overlap. To assess the impact of the leakage, we train XRec on the original datasets and our reproduced versions, where the issues are mitigated.
RQ 2:
Table 7: XRec performance on Original ([Og]) datasets with data leakage and Reproduced (Rel) dalasets with mitigated leakage. BERT aggregates BERTScore Precision and Recall, which are reported in Ap- pendix E.2
The best values are in bold. LLMScore and LLMStability are averaged over five runs.
Explainability 1
Stability
Metrics
LLMSeore
BERT"
BARTScore
BLEURT
STS
L.MSSability
BERT
BART
BLEURT std
Ogg Amazon
67.53
11.61
0.4095
-3.7773
-0,1793
0.5667
10.67 1: 4.59
0.0817
0.4985
0.2256
Rej Amazon
64.80 11.25
0.4396
-3.8135
-0.1474
0.7489
11.84 + 3.99
0.0943
0.5583
0.2599
Og] Yelp
61.67 12.61
0.3527
-4.2969
-0.3089
0.3802
10.65 4.73
0.0967
0.5758
0.2334
Rei Yelp
62.75
-4.0250
-0.1765
0.5186
10.78 4.52
0.0937
0.5813
0.2176
Ogg Google
55.81
13.05
0.4331
-4.1686
-0.2366
0.5891
11.84 + 4.35
0.0935
0.6560
0.2509
Ree] Google
59.72 12.45
0.4349
-3.9298
-0.1283
0.6074
12.57 + 4.43
0.0960
0.6089
0.2451
Table 7 demonstrates that mitigating data leakage improves the explainability of XRec, reflected in higher BERT and BLEURT values. This result is counterintuitive since input contamination typically inflates performance by providing the model with direct access to ground-trus ttrnh information. However, we cannot definitely attribute these improvements solely on data leakage mitigation, as other factors-sule as variations in selected reviews, or differences in the models used to generate the ground truth and explanations may have steered the results this direction. Additionally, Figure Ihouss shows similar data sparsity trend Figure indicating that data leakage does not impact the model's behaviour across sparsity levels. The consistency between these results indicates that mitigating leakage does not compromisc the validity of carlier findings.
[Re]Amazon
[Re]Google
[Re]Yelp
70
0.475
13
0.450
0..10
ILMScore
LuMttabllo
12
BEATScom
VERTStability.
0.400
0.375
0.08
55
10-
0.350
0.07
zero-shot tstl
1st2
1ot3
tstd
t5
zero-cher tot1
tot2
BB
12t4
1at5
zero-shat
tutl
tst2
1513
totd
tots
zero-shot
1st1
tst2
tst3
totd
tat5
Figure 4: XRee's performance on six data splits with increasing sparsity level on the reproduced datasets.
11

---

Published in Transactions on Machine Learning Research (06/2025)
5 Conclusion
Our reproduction study and analysis of XRec (Ma et al. 2024) confirm that the model improves over baselines in both explainability and stability. Although our results verify the impact of user and item profiles, they do not support the claim that collaborative information injection through the layers of the LLM improves explainability and stability, as XRec without injection performs better on two out of three datasets. Additionally, we find that explainability and stability improve with more information about the specific user, contradicting the claim that XRec performs better with increasing data sparsity.
Additional experiments show that removing the GNN-based recommendation system has a minimal impact on model explainability, as performance remains stable or slightly improves without it. We hypothesize that this occurs because (1) when distinet identification features of users and items are present, even if randomly assigned, the MoE can effectively capture collaborative patterns without relying on a GNN-based model, making those signals potentially redundant, and (2) the MoE adapter acts similarly to soft prompting, embedding task-specific knowledge in way that improves explainability (Lester et al. 2021 Li et al. 2021
Limitations and Future Work Our experiments show performance gains after removing data leakage, but these cannot be conclusively attributed to leakage alone, as factors such as review selection and model differences can also play role. While the results suggest data leakage may affect performance, its exact impact remains unclear. Future work should explore this through additional experiments.
Another limitation is in the personalization metrics. USR focuses on uniqueness, which may miss personal- ization quality, while STS can favor similar structures over truly personalized meaning. These issues limit conclusions about personalization effectiveness. Future work should develop metrics that better capture meaningful personalization beyond surface-level similarity.
Finally, while we hypothesize that the MoE acts as soft prompting, this may be amplified by metrics favoring structural similarity over meaning. Future work should examine the MoE's impact on explanation structure and meaning, and develop more robust evaluation metrics.
We believe future work explainable recommendations can build on our study by improving the evaluation metrics. We recommend adopting LLMScore as the primary evaluation metric to mitigate language models' inherent bias loward favoring their own generated text or models consistent with them (Panickssery et al., 2024). To reduce variation in scores, evaluations should use a limited rating scale (e.g., 0-5) and include in-context examples based on human-annotated judgments (Stureborg et al. 2024). Alternatively, an LLM could act as a discriminator in pairwise comparisons (Wu al.) 2024)). choosing the better of two generated explanations against ground-truth reference. Lastly, we encourage future work to perform data generation and evaluation using distinct models due to the same self-consistency issue (Panickssery et al. 2024
What was easy: Ma et al. 2024 provided most of the model implementation with documentation on run- ning the training process. The model architecture description was detailed and casy to follow. Additionally, the authors provided evidence linked to specific claims, simplifying the reproduction process.
What was difficult: Some parts of this study were more challenging than expected. The dataset pre- processing code was incomplete, making it difficult to rebuild the full pipeline. Ablation and evaluation scripts were also not provided, and docunentation lacked details on metrics and dataset-specific settings.
They promptly responded to our questions and clarified details about the data generation process and the hyperparameters we could alter to replicate their datasets.
Communication with original authors:
Acknowledgments
We thank Emmanouil Georgios Lionis for their feedback and inspiration, and Clara Rus for their insightful feedback this manuscript, which significantly contributed to the success of this project
12

---

Published in Transactions on Machine Learning Research (06/2025)
References
Min Cheng, Fei Yuan, Qingpeng Liu, Xia Xin, and Enhong Chen. Learning transferable user representations with sequential behaviors via contrastive pre-training. In 2021 IEEE International Conference Dala Mining (ICDM), pp. 51-60. IEEE, 2021.
Min Cheng, Zhiyong Liu, Qingpeng Liu, Shuyuan Ge, and Enhong Chen. Towards automatic discovering of deep hybrid network architecture for sequential recommendation. In Proceedings of the ACM Web Conference 2022, pp. 1923-1932, 2022.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, 1CML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70, pp. 1263 1272, 2017.
Aaron Grattafiori et al. The Ilama herd of models, 2024. URL https:://axii...ttt http://aax.v..orraa/////////// 21783
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 639 648, New York, NY, USA, 2020. ACM. doi: 00.1145/339721.1 34400066....
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of naacL-HLT, volume 1. Minneapolis, Minnesota, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-effiiint prompl luning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
doi: 10.18653/v1, 00/653/v1/221...// 2w200..1..mmmmmm.ee matillpmmii..2445 243.. URL httpp://aaclaathologgy .rg/ 2021/2221..v emnlp-main. 243/
Mike Lewis. Bart: Denoising sequenco-to-scquenee pre-training for natural language generation, translation, and comprehension. Xix preprint 1rXim:1910.34400 2019.
Jiacheng Li, Jingbo Shang, and Julian McAuley. Uctopic: Unsupervised contrastive learning for phrase represontations and topic mining. arXio preprint arXiw:200.. 2022.
Lei Li, Yongfeng Zhang, and Li Chen. Generate neural template explanations for recommendation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 755 165,764,22 2020.
Lei Li, Yongfeng Zhang, and Li Chen. Personalized transformer for explainable recommendation. Tn Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing (Volume Long Papers), pp. 4947-4957, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/202111 //tg://////21....oo..3..... long.
URL https://aclanthoooog. .rrg
2021. acl-long 1
383/
Lei Li, Yongleng Zhang, and Li Chen. Personalized prompt learning for explainable recommendation. ACM Transactions on Information Systems (TOIS), 2023.
Qiyao Ma, Xubin Ren, and Chao Huang. Xrec: Large language models for explainable recommendation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 391-402, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: https::/accllahhoooo...///////// htnp:/2224...niinnn--mmmmmmmm
findings-emnlp.22 URL
13

---

Published in Transactions on Machine Learning Research (06/2025)
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference. on empirical methods in natural longuage processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 188-197, 2019.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Lim evaluators recognize and favor their own gener- ations. arXiv preprint arXiv:2404. 19076, 2024. URL httpp:::/aa.....rr///aa/////////
13076.
Yijian Qin, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Disentangled representttteeennnnnn learning with large language models for lext-attributed graphs, 2024. URL http:s//aar.ii..or////a///330003
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and μyya Sutskever. Language mod- ols are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. URL https: openai. com/
ltr-llaggggg-rdddll/aagggaaaaaee models pooodelssar____ are are_ www.uuuuussiiddd.cmm htspepprrriieommtltttsssssk.ss. Ieltitastteearrrsssss pdf
N Reimers. Sentence-bert: Sentence embeddings using siamese berl-networks.
orXiv preprinl
arXiv:1908. 2019.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. BPR: bayesian per- sonalized ranking from implicit feedback. In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009, pp. 452-461, 2009.
Paul Resnick and Hal R. Varian. Recommender systems. Communications of the ACM, 40(3):56-58, 1997.
Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Auton Tsit- sulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlic Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette- Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Paacczaak-luciskaa, Harleen Batra, Harsh Dhand, Tvan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Mil- lican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjösund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, and Lilly McNcalus. Gemma 2: Improving open lan- guage models at a practical size. CoRR, abs/ hbs/2448.00014 2024. doi: 10.48550/ARXTV 240..00118. URL https:://oi.... org/ 48550//r50aiii 24w.r.iii.........mm 2408.00118
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint ar F5f:22004004600 2020.
Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and http://wwwww.arra/////.///.//... 01724
biased evaluators. arXiv preprint arXiv:2405.0172,, 2024. URL
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring lasks. arXiv preprinl arXiv:2010.082.0 2020.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine- tuned chat models. Xixiv preprint or orXi::2307.000 09288,, 2023. URL https::///// ttaas:///rxvvv arg/abss/307.0922
14

---

Published in Transactions on Machine Learning Research (06/2025)
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorod- sky, Minjie Xu, Naomi White, and Patrick Lewis. paaplccingjudee with juries: Evaluating 1lm generations with a panel of diverse models, 2024. URL https://arxxw..rr//abb///// 18776
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? preliminary study. arXiv preprint arXiv:2303.0m 04048. 2023.
Xiang Wang, Xiangnan Hc, Meng Wang, Fuli Fong, and Tat-Song Chua. Neural graph collaborative filter- ing. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019., pp. 165-174, 2019.
Zhuo Wu, Qinglin Jia, Chuhan Wu, Zhaocheng Du, Shuai Wang, Zan Wang, and Zhenhua Dong. Rec- Sys Arena: Pair-wisc recommender system evaluation with large language models. arXiv preprint arXiv:2412.11068 2024. URL httpss://axxiv.rrggt orgg abwpg/2bs/242..111 11068
An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. Personalized showcases: Generating multi-modal explanations for recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2251-222 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 12. 15115, 2024a.
Mengyuan Yang, Mengying Zhu, Yan Wang, Linxun Chen, Yilei Zhao, Xiuyuan Wang, Bing Han, Xiaolin Zheng, and Jianwei Yin. Fine-tuning large language model based explainable recomnendation with ex- plainable quality reward. Proceedings of the AAAI Conference on Artificial Intelligence, 38(8):9250 9259, 2024b. doi: 10.1609/amm
10009//aaivvaa v3818.......... 28777.
Weizhe Ynan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances in Newral Information Processing Systems, 34:27263 27277, 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. arXiv preprint arXit:1904.099 2019.
Yongfeng Zhang, Tutorial on explainable recommendation and search. In Proceedings of the 2019 ACM SIGIR International Conference Theory of Information Retrieval, pp. 255-256. ACM, 2019.
Yongleng Zhang, Xu Chen, et Explainable recommendation: survey and new perspectives. Foundalions and Trends0 in Information Retrieval, 14(1):1-101, 2020.
15

---

Published in Transactions on Machine Learning Research (06/2025)
A Hyperparameters
The models were trained and evaluated using the parameters listed in Table g
Table 8: Hyperparameter configurations for the GNN-based recommender system and the Information Col-
laboration Adapter module (MoE), both optimized using Adam (Kingma & Ba 2015).
Hyperparameter
GNN
MoE
Batch size
1024
Number of epochs
300
Optimizer
Adam
Adam
Learning rate
0.001
0.001
Number of layers
4
Embedding size
64
Early Stopping Patience
10 epochs
Number experts
Dropout rate
0.2
Gating Router Noise Factor
0.01
B GPTScore Prompt
To acquire GPTScore and LLMScore for each XRec explanation use the following prompt proposed by Ma et al. [024]] :
System Prompt:
Score the given explanation against the ground truth on a scale from 0 to 100, focusing on the alignment of meanings rather than the formatting. Provide your score as number and not provide any other text.
Prompt:
"prediction" <XREC explanation>,
"reference" <ground truth explanation>
16

---

Published in Transactions on Machine Learning Research (06/2025)
C
Datasets
C.1 Dataset Processing
To evaluate XRec, Ma et al. (2024) use three datasets from distinct domains, which they subsequently process. The first dataset, Amazon Review Data proposed by Ni et al (2019), includes reviews and product metadata, where only the book subsct is used for XRec. The sccond dataset, Google Local Data introduced by Li et al. (2022); Yan et al. (2023). contains Google Maps reviews and business metadata, where only the California subset used. Finally, the Yelp Open Dataset includes reviews and business metadata, and the entire dataset serves as input.
Ma et al. 2024 provide example processing code for the Yelp Open Dataset. Their approach involves:
1. Filtering out items without name, description, or categories;
2. Filtering out all reviews with rating of 3 or less;
3. Randomly subsampling the remaining users;
4. Retaining only interactions within the 8-core interaction graph
We follow these steps to reproduce the datasets, which we refer to as [Re]Amazn, Google, and Yeep to distinguish them from the original Amazon, Google, and Yelp datasets in XRec's repository. However, whereas Ma al. (2024) use the full Google dataset and work with a manually extracted 8-core subgraph, we the smaller 10-core version to reduce computational overhead. For consistency, we process the Amazon and Yelp datasets into 10-core versions.
The ground-truth explanations are generated from subset of interactions, taken from the pre-processod dataset, following the original procedure in Ma et al. (2024). For the Amazon and Google datasets, we include reviews with more than 10 words in order to reduce noise.
C.2 Data Splitting
We provide the Train, Validation, and Test splits for the reproduced interaction datasets [mezona [Re] GRogl,,, and Ree|Yelp-in Tableg These interaction datasets are used to train the Recommender System. From each interaction dataset, extract subset containing interactions with ground truth explanations, which we refer to as the explanation dataset. This explanation dataset is used to train the XRec framcwork. To ensure consistency, the splits for the explanation dataset were designed to match the numbers of the corresponding original datasets (Ma et al. 2024), and the same split ratios were applied to the interaction datasets.
Table 9: Train, Validation, and Test splits for Amazon, [Re]Google, and Yelp datascts.
Amazon Explanations
[Re] Google
[Re] Yelp
Split
Interactions
Explanations
Interactions
Explanations
Interactions
Train
95,841
303,246
94,663
345,846
74,212
337,797
Validation
11,980
37,905
11,833
43,231
9,277
42,227
Test
3,000
9,493
3,000
10,961
3,000
13,656
Sum
110,821
350,644
109,496
400,038
86,489
393,680
v2/
httpp::/bbsineess. http://pppp.o.aaaaee/eeorooooeo wtwp://eeeeeppedaaaataeeeeeeeeee
10 "-corre" graph subgraph which every node degree least meaning each node connected
least. k other nodes. For example, in 8-core graph, every user item must have least interactions be included.
17

---

Published in Transactions on Machine Learning Research (06/2025)
C.3 Overlap Between Original Dataset Splits
Ma et al. (2024) train their system using two dalasets: (1) the full interaction dataset (for training the recommendation system) and (2) the explanation dataset (a subset of the interaction dataset, used to train the Collaborative Information Adapter). The explanation dataset includes textual explanations of reviews corresponding to specific interactions. However, we observed that during the splitting of these datasets, overlaps occasionally occur, leading to instances where data from one dataset exposed to the other. In this section, we present the percentage of overlap between the train, validation, and test splits of the explanation and interaction datasets. Table 10 summarize these overlaps for the [Og] Amazon, [Og]Google, and [Og] Yelp datasets introduced by Ma et al. 2024
Table 10: Percentage of overlap between the splits of Explanations and Interactions for [Og] Amazon, [Og] and [Og] Yelp datasets.
Dataset Expl. split
Interactions (%)
Train
Validation Test
Train
79
10
10
Amazon
Validation
79
10
9
Test
79
10
Train
80
10
Google
Validation
80
Test
78
10
10
Train
79
10
10
Yelp
Validation
80
9
Test
81
9
C.4 Data sparsity statistics
Figure provides the statistics of all sparsity sets used reproduce Claim 4
[0g]
[Og] Google
[Gg] Wip
(Sa] Google
[Re]
1FL
20
140
14.9
12-4
17
[9.0)
161
31
GA 2011
11.
2050
R8
02 L34
D4
Viradon
Maracca orga
Vteriion ag
Raaaaiic rage
Figure 5: Data sparsity set statistics including user interaction frequency intervals and the percentage of data allocated to each sparsity set from the test set consisting of 3000 interactions. Each panel corresponds to one dataset and displays the proportion of interactions each sparsity set (zero-shot, istl (st5) for both original ([Og)) and reproduced ([Re)) datasets. The bottom x-axis shows the exact frequency intervals, while the top x-axis indicates sparsity-set names.
D Carbon Emissions
We estimate the carbon emissions from the experiments using the CO2c = CI x PUE x P x formula, where the carbon intensity (CT) in the Netherlands is 0.370 CO2e emissions per kWh. The power usage effectiveness (PUE) values for the different setups are 1.2 and 1.1. The power consumption is 0.25 kW for A100 GPUs and 0.07 kW for T4 GPUs, with runtime durations of 280 hours and 160 hours, respectively. Based on these caleulations, the estimated carbon emissions for the experiments amount approximately 31 kg CO2e and 4.5 kg CO2e. In total, the experiments resulted in approximately 35.5 kg CO2e emissions.
18

---

Published in Transactions on Machine Learning Research (06/2025)
E Extended Results
E.1
Ablation Results per LLM Judge
The error bars are computed by first obtaining five stochastic predictions from the LLM for cach individual test instance, caleulating & within-inslance standard deviation (o:), and subsequently averaging these across the dataset. While informative for capturing prediction-level noise, this procedure inadvertently inflated uncortainty estimates the datasct level. The inflation occurs for two primary reasons: (1) each os computed from only five samples, inherently possesses significant sampling error; and (2) averaging of across all instances preserves prediction-level variability without benefiting from the typical vn reduction that occurs when averaging metrics at the corpus level.
Model
Aggregated
Gemma2 4b-it
Unma 3.1 Abiinstartt
- Liama 3.2 Tb-intruct
+ Quee2.5 7b-inotruct
[Og]Amazon
[og]Google
[Og]Yelp
80
LLNSCrrS
Alee beco prottel
one seected)
Morr weth profiy lppettor)
can pesti
aeter ted lpjetion)
tedo sraffer agletioni
abee Coll pertle
Aibe des Ryjettoon
West redo andite oriection
Figure 6: Scores from individual LLM judges (colored lines) evaluating XRec explanations across ablations the Amazon, [Og] Google and [Og]Yelp datasets. The average score across these judges ('Aggre- gated" dashed line) represents the final LLMScore reported in Figure - All LLMs show agreement on the Amazon and [Og]Google datasets, while for the Yelp datasel, all LLMs agreed except for the Gemma2 model.
E.2 BERTScore Precision and Recall
In this section, present. the extended results with BERT Precision (Zhang al., 2019 2019 (BERTP), BERT Recall (Zhang al. 2019) (BERT") metric. Given the predicted sentence BERT representation is a [R1, f.,, f..,,,., ...6. and the reference sentence representation is x = [x1, D2, .m.) BERT Precision, BERT Recall The detailed BERT Precision and Recall are presented
and BERT F1 is defined as in Equations
6 and 7
in Tables
11
12 and
13
BERT
Σ max x, &, 8GEZ D:Ex
(5)
BERT
max X &j ZE
(6)
m
8
BERT R BERT =2 "ERT + BERTR
(7)
19

---

Published in Transactions on Machine Learning Research (06/2025)
Table 11: Comparison of baseline models and reproduced results for BERTScore Precision (BERTP) and Recall (BERT*). Baseline and original XRRc results (denoted as [Orig]XRec) are taken from (Ma et al. 2024h, while our reproduced results (XRec) are shown in italics. Bold indicates the highest score, and underlined indicates the second highest. Table aggregates these results under BERTFI (F1 Score).
Explainabilityy 1
Stability +
Metrics
BERT" BERT
BERT std
BERTH. std
[Og] Amazon
Att2Seq
0.3746
0.3624
0.1691
0.1051
NRT
0.3444
0.3440
0.1804
0.1035
PETER
0.4279
0.3799
0.1334
0.1035
PETER+
0.4119
0.3626
0.1576
0.1077
PEPLER
0.3506
0.3569
0.1105
0.0935
OOigg]Ree
0.4193
0.4038
0.0836
0.0920
XRec (w/o profile)
0.4194
0.4004
0.0819
0.0955
XRec
0.4139
0.4037
0.0863
0.0933
XRec (w/o profile)
0.4207
0.3967
O. 0.0802
0.0888
Yelp
Au2Seq
0.2099
0.2658
0.1583
0.1074
NRT
0.0795
0.2225
0.2293
0.1134
PETER
0.2102
0.2983
0.3315
0.1298
PETER+"
0.2594
0.3097
0.2522
0.1174
PEPLER
0.2920
0.3183
0.1476
0.1044
XRec
0.3946
0.3506
0.0969
0.1048
Orig]XRec (w/o profile) f
0.3879
0.3427
0.1087
0.1072
Recc
0.3541
0.3504
0.1126
0.1108
XRec (w/o profile)
0.3814
0.3451
0.1218
0.1110
[Og] Google
Att2Seq
0.3619
0.3653
0.1855
0.1247
NRT!
0.3509
0.3495
0.2176
0.1267
PETER
0.3892
0.3905
0.2819
0.1356
PETER+
0.4125
0.3975
0.1893
0.1244
PEPLER
0.3806
0.4093
0.1602
0.1154
(Orig]XRes
0.4546
0.4069
0.0972
0.1031
(OrigiRRce (w/o profile)
0.4427
0.4187
0.1180
0.1171
XRec
0.4632
0.4025
0.0986
0.1163
XRec (w/o profile)
0.4594
0.4191
0.1001
0.1150
This score was taken from Ma et al.
2024
20

---

Published in Transactions on Machine Learning Research (06/2025)
Table 12: BERTScore Precision (BERTP) and Rocall (BERTR) comparison of XRec with LCghtGCN and NGCF collaborative information algorithms, well as three variants where explicit collaborative informa- tion is removed. Values in bold indicate the highest score, while underlined values denote the second-best. Table aggregates these results under BERT (F1 Score).
Explainability
Stability
Metrics
BERT BERT
BERT std
BERTH sld
[Og] Amazon
XRec (w/ LightGCN)
0.4139
0.4037
0.0863
0.0933
XRec (w/ NGCF)
0.4277
0.4088
0.0825
0.0934
XRec (w/o GNN, fixed)
0.4200
0.3963
0.0077
0.0899
XRec (w/o GNN, random)
0.4361
0.3984
0.0842
0.0940
XRec (w/o GNN & MoE))
0.2909
0.3102
0.1007
0.0796
[Og]
XRec (w/ LightGCN)
0.3541
0.3504
0.1126
0.1108
XRec (w/ NGCF)
0.3828
0.3635
0.0953
0.1047
XRec (w/o GNN, fixed)
0.3737
0.3572
0.1103
0.1039
XRec (w/o GNN, random)
0.4376
0.3476
0.0967
0.1098
XRec w/o GNN & MoF)
0.2796
0.2609
0.1045
0.1088
Google
XRec (w/ LightGCN)
0.4632
0.4025
0.0986
0.1163
XRec (w/ NGCF)
0.4517
0.3943
0027
0.1136
XRec (w/o GNN, fixed)
0.4414
0.4192
0. 0.1081
0.1141
XRec (w/o GNN, random)
0.4520
0.4094
0.0958
0.1187
XRec (w/o GNN & MoEE)
0.2094
0.2806
0.1049
0.1019
Table 13: BERTScore Precision (BERT?) and Recall (BERT comparison of XRec on (1) Original ((Og)
datasets with data leakage and (2) Reproduced ([Re) datasets with mitigated leakage. The best values are in bold. Table 7 aggregates these results under FI (F1 Score).
Explainability BERT" BERT
Stability
Metrics
BERT std
BERT std
Ogg Amazon
0.4139
0.4037
0.0863
0.0933
Re Amazon
0.4563
0.4218
0.0986
0.1049
[Og Yelp
0.3541
0.3504
0.1126
0.1108
[Re] Yelp
0.4382
0.4207
0.1002
0.1076
O0g Google
0.4632
0.4025
0.0986
0.1163
[Re] Google
0.4450
0.4238
0.1004
0.1107
21


Reviews:
[{"review_id": "gvVMekd0Qv", "reviewer": "Reviewer_EzUY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. Revisiting XRec provides evidence that the GNN component does not enhance explainability. Instead, the observed performance improvement is attributed to the Collaborative Information Adapter, which acts as a form of soft prompting, efficiently encoding task-specific information.\n2. The experiments are comprehensive, not only providing extensive validation and ablation studies on the original XRec but also offering expanded experiments.\n3. The paper is well-written, clear, and understandable.\n4. The code is open-sourced and well-documented, facilitating reproducibility.\n\nWeaknesses:\n1. The experimental conclusions require further analysis. For instance, injecting collaborative signals into all attention layers of the LLM did not significantly enhance performance, contradicting XRec's hypothesis. The role of collaborative signals in the explainability of recommender systems needs more investigation.\n3. The paper's primary contribution is the reproducibility of XRec; other contributions are limited. Proposing better metrics for explainability or a more comprehensive method would make revisiting XRec more solid.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reproduces and expands upon the findings of XRec, a framework that integrates collaborative signals—patterns of user interactions such as reviews—into Large Language Models (LLMs) by incorporating them into both the input prompt and the model layers. The authors further explore XRec by investigating the role of collaborative signals through the evaluation of different Graph Neural Network (GNN)-based recommender systems, such as LightGCN and NGCF, that capture these signals from graph structures. To assess the impact of entirely removing graph-based collaborative information, the authors conducted three comparative experiments. The conclusion drawn is that the GNN component does not enhance explainability. Instead, the observed performance improvement is attributed to the Collaborative Information Adapter, which acts as a form of soft prompting, efficiently encoding task-specific information.\n\nIn summary, this reproducibility paper provides a novel perspective on XRec, demonstrating that removing the GNN-based recommendation system has minimal impact on model explainability, as performance remains stable or slightly improves without it.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Correct some proprietary nouns, e.g., Llama -> LLaMA\n2. It would be beneficial to explain metrics such as BERT_f1, BERT_recall in the Appendix.\n3. More explicitly display the Limitations and Future Work.\n4. Additional contributions beyond reproducibility would be advantageous.", "broader_impact_concerns": "This paper does not raise any ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ae5ux8Vy1", "reviewer": "Reviewer_776v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper conducts a comprehensive reproduction of the XRec study. By replicating experiments, extending ablation studies, and addressing data leakage, it provides a detailed and systematic examination of the original work.\n* The discovery that the Collaborative Information Adapter is the main driver of performance improvement, rather than the GNN component as might have been initially assumed, is a significant contribution. This finding not only clarifies the inner workings of XRec but also aligns with prior research on lightweight adaptation mechanisms for LLMs.\n* Making their implementation open-source enables other researchers to build on this work, further validate the findings, and explore new directions in explainable recommendation systems.\n\nWeaknesses:\n* While the findings regarding reproduction are interesting and the conclusion deviates from the original paper, the paper focuses solely on validating and supplementing XRec without proposing new methods, which could make the paper less significant.\n* While the authors challenge the claims that injecting collaborative information across all LLM\nattention layers improves performance, and XRec is more effective under increased data sparsity, could the authors also provide some insights on how to improve XRec to improve the performance on sparse data and mitigate the complexity of injecting collaborative information in all attention layers?\n* For the new observations such as the extending of the ablation study, could they also be able to generalize to other similar models or different datasets outside of those used in the original study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Recommender systems, especially those using deep learning, have become complex and hard to interpret. XRec (Ma et al., 2024) was introduced to address this by integrating collaborative signals and textual descriptions into Large Language Models (LLMs) for generating natural language explanations for recommendations. This paper aims to reproduce and expand on the findings of Ma et al. (2024). The authors replicated XRec experiments, extended the ablation study, addressed data leakage issues in the original data generation, and adapted the codebase. Their results validated that XRec outperforms baseline models and that incorporating user and item profile information improves performance. However, they could not fully replicate claims about injecting collaborative information into every LLM attention layer, improving performance and the effects of data sparsity. Their analysis showed that the Collaborative Information Adapter, rather than the Graph Neural Network (GNN) component, was responsible for the performance improvement, acting as a form of soft prompting. The paper concludes by highlighting areas for future research, such as developing better evaluation methods for personalization and examining the influence of the adapter on explanation structure and meaning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please kindly refer to the Weaknesses for potential requested changes.", "broader_impact_concerns": "No potential negative societal impacts are explicitly discussed, which is suggested to be included.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WncDfETeJE", "reviewer": "Reviewer_PzX6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- [S1] The paper is generally well-explained, clearly stating its claims (Claim1-Claim5) and research questions (RQ1-RQ2).\n- [S2] The study significantly focuses on reproducibility, which is crucial in recommendation system research (e.g., the well-known RecSys’19 paper), and provides ample appendices and information.\n- [S3] The paper addresses the often overlooked aspect of data processing reproducibility, identifying and resolving issues.\n\nWeaknesses (See also Requested Changes)\n\n- [W1] There are several underexplained points and concepts, possibly due to trying to fit the content within 12 pages.\n- [W2] The readability of tables and figures has decreased, likely due to space constraints within 12 pages.\n- [W3] Explanations of terms like GPTScore and LLMScore are insufficient for researchers unfamiliar with TMLR or RS backgrounds.\n- [W4] The reasoning behind choosing NGCF for the ablation study is unclear.\n- [W5] It is uncertain whether the discussion on CO2e is necessary, especially if it leads to dropping other explanations.\n- [W6] Related to [W2], there are difficulties in quickly understanding the consistency of scores and the assertions of claims from the tables and figures; improving the explanation and presentation would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Explainable recommendation systems are widely studied today as essential technologies. The method at the core of this paper, XRec, uses a large-language model  (LLM)-based approach proposed by Ma et al. in 2024. The original XRec generates collaborative signals using user interactions such as reviews and injects these into the language model to effectively model user preferences for recommendations. \n\nBeyond the above original work, the authors revisited XRec, reproducing its implementation and experiments to compare performance. The replication study added ablation experiments not conducted in the original XRec paper and considered data leakage issues that emerged during reproduction. The results confirmed the value of LLM-based explainable recommendation systems and provide deeper insights into the effectiveness of the GNN component previously assumed in existing research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**[Critical to securing my recommendation for acceptance]**\n\nWhile I respect the authors' contribution to reproducibility, the paper contains numerous underexplained points, likely due to page limitations. In its current state, I cannot confidently recommend acceptance.\n\nBelow are my concerns regarding explanations, experimental result reliability, and presentation:\n- The paper cites Figure 1 from XRec, but inadequately explains its individual components. This creates a gap in demonstrating the approach's value and undermines trust in the work. The selection of XRec appears arbitrary, the GPTScore explanation is insufficient (leaving its measurement purpose unclear), and the choice of NGCF lacks motivation—all of which weaken the overall contribution. Most critically, the unexplained selection of NGCF and the minimal explanation of GPTScore present significant concerns.\n- As highlighted in [W2], the tables' current size and density impair readability. This makes it impossible to directly verify the consistency of results claimed in Claim 1, diminishing the paper's persuasiveness.\n- The paper needs to explain the vertical bars in Claim 2, as Fig. 3's current description lacks conviction. For example, while it discusses the inverse relationship between increasing LLM Score and decreasing BERT Score F1, insufficient explanations of Fig. 2 and GPTScore make it unclear whether this represents a consistent pattern or mere coincidence. The observed stability issues suggest the latter. Furthermore, the paper should justify its transition to using only F1—if Recall/Precision are not necessary, simplifying this throughout could strengthen the paper's arguments.\n\n**[Comments to strengthen the work in my view]**\n- As noted in [W5], I am unsure whether discussing CO2e is necessary, especially at the cost of omitting important technical explanations.\n- For Table 13, visualizations (e.g., using histograms) could be helpful to show the division is valid and reliable.", "broader_impact_concerns": "Not applicable to this paper in my opinion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper re-examines XRec, a popular recommendation model using GNNs and LLMs. It investigates several aspects, including reproducing experiments,  ablation study,  addressing data leakage and etc. The reviewers all agree that the paper has made insightful observations and will benefit the community of recommender systems. For final version, the authors are requested to add discussions on how to develop better metrics for explainability and/or directions for a more comprehensive method that would make revisiting XRec more robust.
Decision: Accept (poster)