Understanding Embedding Scaling in Collaborative Filtering
Authors: Yicheng He
Zhou Kaiyu
Haoyue Bai
Fengbin ZHU
Yonghui Yang
Venue: Accepted by TMLR ()

Abstract: Scaling recommendation models into large recommendation models has become one of the most widely discussed topics. Recent efforts focus on components beyond the scaling embedding dimension, as it is believed that scaling embedding may lead to performance degradation. Although there have been some initial observations on embedding, the root cause of their non-scalability remains unclear. Moreover, whether performance degradation occurs across different types of models and datasets is still an unexplored area. Regarding the effect of embedding dimensions on performance, we conduct large-scale experiments across 10 datasets with varying sparsity levels and scales, using 4 representative classical architectures. We surprisingly observe two novel phenomenon: double-peak and logarithmic. For the former, as the embedding dimension increases, performance first improves, then declines, rises again, and eventually drops. For the latter, it exhibits a perfect logarithmic curve. Our contributions are threefold. First, we discover two novel phenomena when scaling collaborative filtering models. Second, we gain an understanding of the underlying causes of the double-peak phenomenon. Lastly, we theoretically analyze the noise robustness of collaborative filtering models, with results matching empirical observations.

Published in Transactions on Machine Learning Research (12/2025)
Understanding Embedding Scaling in Collaborative Filtering
Yicheng Hc
yh84 Buduc. edu
University of Mlinois Urbena- ChannnCn
Kaiyu Zhou Nanyang Technological University
kaigu001 Gc. ntu. www.nuuuuuuu.... edu.sg
Haoyue Bai Arizona State University
bashaoyue621 @gmaiil com
Fengbin Zhu National University of Singapore
shfengbinngymaillccmm com
Yonghui Yang"
yh yang Drrus.
National University of Singapore
Reviewed on OpenReview: https: /ttps:://ppnneen opw.r/vie...eeeeee net/ forum? id= 3f5HtLQRaY
Abstract
Scaling recommendation models into large recommendation models has become one of the most widely discussed topics. Recent efforts focus components beyond the scaling bedding dimension, as it is believed that scaling embedding may lead to performance degra- dation. Although there have been some initial observations on embedding, the root cause of their non-scalability remains unclear. Moreover, whether performance degradation occurs across different types of models and datasets is still an unexplored area. Regarding the effect of embedding dimensions on performance, we conduct large-scale experiments across 10 datasets with varying sparsily levels and scales, using representative classical architec- tures. We surprisingly observe two novel phenomena: double-peak and logarithmic. For the former, the embedding dimension increases, performance first improves, then declines, rises again, and eventually drops. For the latter, it exhibits perfect logarithmic curve. Our contributions are threefold. First, we discover two novel phenomena when scaling col- laborative filtering models. Second, we gain an understanding the underlying causes of the double-peak phenomenon. Lastly, we theoretically analyze the noise robustness of collaborative filtering models, with results matching empirical observations.
1 Introduction
Scaling up Transformer (Vaswani 2017 paramcters boost performance in NLP tasks is key reason behind the success of large language models. Therefore, natural question arises: can replicate similar success in collaborative filtering? It is widely agreed that embedding dimension is one of the key controllable parameters in collaborative filtering. However, there is also a consensus that scaling the embedding dimension
may leads to performance degradation (Guo et al. 2024 Ardalani et al.] 2022 Zhang et al.) 2024 Ye et al.,
2025 Xu et al., 2025; Peng et al. 2025). For example, Guo al. (2024) argues: "Worsely, increasing the embedding dimension does not sufficiently improve the performance or even hurts the model", and [Zhang et al. (2024) believes: Merely exponding the sparse component of model does not enhance its ability to capture the complez interactions ...". As result, recent works (Zhang et al., 2024 Guo et al. 2024] scaling recommendation models bypass the seemingly unviable path of expanding embedding dimension.
"Corresponding author

---

Published in Transactions on Machine Learning Research (12/2025)
A further question arises: why do scaling embedding dimensions fail to improve recommendation perfor- mance? Guo et al. (2024) observe that when the embedding dimensions are scaled, embedding will collapse. Specifically, they conduct a spectral analysis based on singular value decomposition of the learned embed- ding matrices and find that most of the singular values are extremely small. Furthermore, several studies on elficient recommendation Liu et al. ) Qu et al. 2023); Wang et al. (2024); Luo el al. 2002 Liu et al. (202T) treat the embedding dimensionality as a learnable parameter and optimize through prede- fined objectives, thereby enhancing the model's 's overall effectiveness. Because, they think that there exists optimal dimensionality for embeddings. In other words, performance exhibits single-peak phenomenon as the embedding is scaled. Overall, previous works observed that increasing the dimensionality not only worsens performance but also causes changes in the singular values of the embedding.
In this paper, we are more interested in understanding why the performance deteriorates with scaling and what the underlying causes are. Would it follow the previously observed single-peak phenomenon?
To enable a comprehensive investigation, we conducted large- scale experiments across 10 datasets with varying sizes and sparsity levels (Table 3), using 4 representative collabora- tive filtering models (reason at D.4j. Surprisingly, we observe two novel phenomena: double-penk and logarithmic (Figure D
0.51
Double-Penk
Logarithmic
0.47
NDCG@20
apart from the common Singlepeakk. Additionally, we have two observations, for the samne model, some datasets exhibit double-peak trend, while others show a logarithmic increase. More interestingly, dataset, the phenomena are not always consistent. For exam- ple, BPR (Rendle et al., 2009) exhibits a double-peak pattern, while SGL [Wu et al. 2021) follows a logarithmic trend.
0.43
0.39
also find that even within the same
0.35
2 2
Embedding Size
Figure Double-peak: first rising, then falling, followed by another rise and decline; logarithmic, performance follows logarith- mic increase.
Clearly, we prefer the logarithmic scaling for its expected stable performance, while the emergence of the double-peak phenomenon indicates limitation in scalability. Since both phenomena appear across different datasets within the same model, we boldly speculate that this is due to certain negalive factors in the interactions. Inspired by He et al. (2022), we concluded that the issue originates from the noise present in the data, and we conducted a thorough analysis of the causes of noise at each stage. We also conducted an in-depth analysis of the noise resistance and robustness various collaborative filtering models and the results are generally consistent with the empirical evidence.
Our contributions are summarized follows.
To the best of our knowledge, our work is the first report of the double-peak and logarithmic phenomenon. In the realm of collaborative filtering, there has long been perplexing question: why does expanding em- bedding dimensions not always result in performance enhancements? Our research make further additions to it.
Through in-depth analysis, identified interaction noise, in addition to overfitting, the key factors causing the double-peak phenomenon. To test this idea, we propose simple yet effective sample drop stratogy that helps collaborative filtering models scale more effectively. For instance, the embedding dimensions can be expanded up to 32,768 without significant risk of collapse.
We theoretically analyzed the noise robustness of BPR, NeuMF, LightGCN, and SGL, with results largely aligning with empirical observations, indirectly validating our analysis. For instance, BPR shows poor noise robustness and more frequently exhibits the double-peak phenomenon, while SGL more robust and tends to follow a logarithmic pattern.
The implications for future research.
Transformer-bant based LLMs have garnered significant attention. Most of researches (Zang al. 2024 Xu et. al.] 2024) aim to bring the success of the Transformer architecture into recommendation models to
2

---

Published in Transactions on Machine Learning Research (12/2025)
achieve superior scaling performance. However, in this paper, we explore different path: finding the 'Transformer' within collaborative filtering models. We discovered that SGL possesses structural advan- tages in this regard. Therefore, future research directions may focus on better filtering clean interactions and further scaling up SGL for enhanced performance.
2 Background: Collaborative Filtering
Notations. We use lowercase letters for scalars. Denole the set of real numbers by R. We use bolded lowercase letters p to represent the entire embedding, and subscripts to refer to specific embedding Pa- We use calligraphic letters c.g., V to denote the value spaces (other notions in Appendix A
User and Item Embedding. In collaborative filtering, the core components involve representing users and items using embedding vectors. The user embedding contains the user's preference information, which can better capture the similarity between different users. Similarly, the item embedding contains the item's 's feature information. Let us define: Pa E Rn as the embedding vector for user ", where = 1,2...., m. qi E Ih as the embedding vector for item i, where = 1,2,..... n. m is the dimensionality of the embedding.
Collaborative Filtering. To compute the preference score fig of user u for item i, many powerful and rep- resentative collaborative filtering models have been proposed, such BPR (Rendle al. 2009 NeuMF (He 2017 LightGCN (He et al. 2020) and SGL (Wu et al. 2021
et al.
BPR (Renlle et al. 2009 calculates the preference score the inner product of user and item embeddings, or employs a neural network to capture nonlinear interactions:
fiz = T q
Σ 1=1
un Ujt, Pig = /MLP(Pu q:),
where ua and ey are the 1-th components of pa and q, respectively.
2 NeuMF He et al. 2017 is representative deep learning- based collaborative filtering model with classic two-lower architecture. It fuses Generalized Matrix Faclorization (GMF) and Multi-Layer Perceptron (MLP) to capture both linear and nonlinear user-item interactions:
GMF f = P. qi:
MLP Pig
fMLP((Pulla)),
where [-II-] denotes vector concatenation. The final prodiction Pig is computed by combining the outputs from GMF and MLP, typically through a fully connected layer.
LightGCN (He al. 2020) simplifies GCN by removing feature transformation and nonlinear activation to better capture high-order connectivity in user-item graphs. It adopts symmetric normalization in the graph convolution process, aggregating embeddings from neighboring nodes using degree-aware weighting scheme:
(+1)
Σ
(I+1)
Σ NEA,
q
P
SEN. VWN
This symmetric normalization allows the model to more accurately propagate information and better capture user interests.
1 SGL. (wu et al. 2021 rntrodeces sel self-supervised learning by augmenting graph structure and node features, incorporating contrastive objectives to improve robustness and alleviate sparsity:
L =
BPP
+y
Loont contrastive
collaborative filtering
where CBR the Bayesian Personalized Ranking loss EmwijeeD In (P..q: q P._ q.) and Loontrestive the contrastive loss encouraging consistency between augmented views. In addition, ? controls the contrastive task's contribution.
3

---

Published in Transactions on Machine Learning Research (12/2025)
3 Analysis of Scaling: Scale it up! Scale it up again!
We focus on three points: First, what strategy we use to increase embedding size. Second, what phenomenon we observe in experiments, Third, why these phenomena happen.
3.1 Observation Settings
We select four popular collaborative filtering models: BPR (Rendle et al.) 2009). NeuMF (He et al. 2017 LightGCN (He et al. 2020) and SGL (Wu et al. p0 along with ten classic datasets, covering different level of scale and sparsity. These datasets are described in the Appendix 3] To ensure a thorough investigation, our experiments were conducted on a
Our experiments are based on the RecBole
(Xu et al. 2023
setup equipped with 8*80G GPUs. Other settings regarding all model parameters, optimizer, learning rate,
etc. are in the Appendix D.1
3.2 Double-peat and Logarithmic Phenomenon
In the past understanding, when the embedding dimension increases, the performance shows single-peak phenomenon of first. increasing and then decreasing. However, we believe that this understanding lacks microscopic and extensive observations. Throngh large-scale experiments (Figure we find that on the ML. 100K dataset, when the embedding dimension increases, the overall performance of the BPR model is indeed consistent with previous studies, showing an overall trend of first increasing and then decreasing. However, discovere more details for the first time: the embedding dimension increases, the recommendation performance initially increases significantly, then decreases, then increases slightly, and finally decreases again. We call this new trend double-peak: phenomenon (as shown in Figure 2 (b, c, e, ()). Moreover, we noticed that the most of peak performance occurs at 29, which are different from the traditional settings of 32, 64, and 128. But unexpectedly, in LightGCN and SGL, this phenomenon alleviated a lot, and SGL alleviates it belter than GCN (as shown in Figure 2 (k, h)). addition, our experiments show that on the Modcloth dataset, BPR, NeuMF, LightGCN, and SGL all show a continuous upward trend. For example, even when the dimension is set to 2 the performance of LightGCN is still rising. We call this ideal trend the logarithmic phenomenon (as shown in Figure 2 (a)). that is, as the embedding dimension increases, the performance continues to improve in stable and predictable manner, while the marginal gains gradually diminish. Such a logarithmic trend is particularly desirable, as it provides consistent performance improvements with incrcasing embedding size. The logarithmic phenomenon completely exceeds the past understanding. Compared with the NDCG@20 at 128 dimensions, the performance has increased by 25.57%. On the contrary, we only set the dimension to 128, we will lose 34.36% of the potential performance improvement. Currently, most research in the recommendation field, such as debiasing (Chen al.] 2021 2023) and denoising (Wang et al., 2021 He et al., 2024), believes that 10% performance improvement is meaningful. However, in observations, simply increasing the dimension can achieve performance improvement far excceding that of claborately designed methods.
To ensure the generality of the phenomenon, we design a series of experiments, conducting observations across other datasets (E.1) and also analysis the results in detail. Empirical results confirm that the phenomenon is indoed widespread.
The same model exhibits different phenomenons across different. datasets. As shown in Figure 2 (a, d), BPR exhibits double-peak phenomenon the ML-100K dataset, while showing logarithmic growth on the Modcloth dataset. This seems to be due to some negative factor in ML-100K that causes scaling to fail. @ The same dataset exhibits different phenomena across different. models. As shown in Figure 2 (a. b, c), we observe that BPR exhibits a double-peak phenomenon, while LightGCN and SGL show logarithmic growth. Moreover, we believe this might be because the structures LightGCN and SGL incorporate mechanisms that mitigate negative factors. Moreover, the logarithmic growth of SGL appears to be more ideal.
Two Observations.
4

---

Published in Transactions on Machine Learning Research (12/2025)
0.65
0.43
0.4
NDC
0.39
NIX
0.64
NDCGG620
0.7
0.62
0.45
0.35
0.31
0.42
0.27
2
2
215
2
2
2 2
2
2 2 2
2
Embedding Size
Embedding Size
Emhedding Size
(a) ModCloth BPR
(b) Douban BPR
(c) ML-100k BPR
0.47
0.67
0.49
NDC0@220
0.42
NDC0G200
0.65
NDCGG20
0.47
0.37
0.32
0.62
0.44
0.43
0.27
0.60
22
212
213
2
2 2 213
2
2
2
2
Embedding Size
Embedding Size
Emhedding Size
(d) ModCloth NeuMF
(e) Douban NeuMF
(f) ML-1000 NeuMF
0.47
0.68
ND00@200
0.43
NDC0G200
0.66
NDCGG220
0.45
0.39
0.64
0.39
0.35
0.62
0.31
0.60
0.29
2a
2
2
2
29
2 2
2
2 2
2
2
Embedding Size
Emhedding Size
Emhodding Size
(g) ModCloth LightGCN
(h) Douban LightGCN
(i) ML-100k LightG
0.50
0.50
0.46
ND008820
NDX0G220
0.42
NDCG@200
0.42
0.38
0.34
0.34
0.34
0.25
0.25
0.30
2a
2
2
2
213
2
2
2.2
2
2
Embedding Size
Embedding Size
Emhedding Size
(i) ModCloth SGL
(k) Douban SGL
ML-100k SGL
Figure 2: Scale the embedding dimension exponentially by factor of across different collaborative filtering models and datasets. Each row corresponds to model (BPR, LightGCN, SGL, NeuMF), and each column represents dataset (Modcloth, Douban, ML-100k).
3.3 Why Two Phenomena Occur
Based on the previous two observations, believe this is due to the presence of certain negative factors in the datasct that BPR and NeuMF cannot resist, while LightGCN and SGL can. When these negative factors are mitigated, a logarithmic trend emerges, whereas if influenced by these factors, a double-peak pattern observed. In addition, the
aspect the double-peak phenomenon lies in the third stage, where performance rises again. If this stage were absont, with performance simply rising and then falling, it could easily be explained as overfitting. However, the appearance of the third stage prevents from attributing it to overfitting, and we believe this is the key to understanding this phenomenon. For
5

---

Published in Transactions on Machine Learning Research (12/2025)
0.66
0.50
NDCG@20
NDCG@200
0.64
0.47
0.45
0.62
0.43
0.60
BPR Drop
RPR Drop
0.40
BPR
BPR
0.38
29
212
213
2
0.58
2
2
2
2
Embedding Size
Embedding Size
(a) MI-100K
(b) Douban
Figure 3: Comparing the standard BPR with the denoising strategy-based BPR Drop in ML-100K and Douban, we can clearly observe that the performance degradation has significantly improved.
logarithmic phenomenon, this is an ideal trend. If every model can exhibit this trend on every dataset, it would be wonderful.
5 we observe that SGL tends to exhibit such a logarithmic trend. We attribute this behavior to the contrastive learning framework in SGL, which provides both regularization and robustness enhancement. In contrast, other methods lack such a mechanism, which may explain their deviation from this desirable pattern.
Upon close examination of Figures 2
Noisy Interactions Induces the Double-peak Phenomenon. For the double-peak phenomenon, this is novel phenomenon. Our approach to analyzing this phenomenon was inspired by two phenomena in machine lcarning: the memorization effect on noisy labels (Zhang et al. 2017) and the sparse double descent (He et al. 2022 which found that the sparsity of model paramelers and the level of label noise jointly affect the model's performance (more about noisy interactions at B.3
Definition 1 (Sparse Double Descent) He el al. 2022 is phenomenon that occurs during model pruning. It is characterized by the model's s test performance first decreasing due to overfitting sparsity increases, then improving once overfitting is alleviated after reaching a certain level of sparsity, and finally decreasing again as sparsity increases, due to the loss of useful information.
We think that scaling the embedding dimension related the sparsification of neural networks in recom- mendation tasks, as increasing the dimension effectively reduces the sparsity of the neural network. Overall, this process can be summarized into four phases: the model first tends learn clean samples (Phase improves), then memorizes noisy samples (Phase 2, declines). As the embedding size increases, the enlarged space dilutes the negative impact of noise (Phase 3, rebounds); finally, excessively large embedding size leads to overfitting (Phase 4, drops). Notably, the resulting double-pest peake curve resembles the phenomenon of sparse double descent (He et al.] (022), since both are driven by sparsity and data noise. Nevertheless, there are important distinctions: sparse double descent typically studied in general machine learning settings involving label noise and model pruming, whereas our double-peak phenomenon arises specifically collab- orative filtering when scaling the embedding dimension under noisy interactions. Specifically, in the first stage, the clean learning stage, the model initially learns from clean interactions. As the dimensionality of the embedding increases, the model has improvements. In the second stage, the noise memorization stage, the presence of interaction noisc, combined with a dimensionality large enough to accommodale noisy interactions, causes the model to begin learning from these noisy interactions, resulting in performance degradation. In the third stage, the sweet stage, the embedding space enlarges to adapt to interaction noise, thereby enhancing the model's robustness. In the fourth stage, the overfitting stage, the continuous increase in embedding dimensions leads to higher fit the training dataset, which subsequently reduces generalization performance.
space to accommodate user interests, leading to performance

---

Published in Transactions on Machine Learning Research (12/2025)
4 One of the Keys to Scaling: Improve Robustness to Noisy Interactions
In this section, we conduct a detailed analysis based on the models and explore the charactoristies of cach model, focusing on three main questions: Why are BPR and NeuMF casily affected by noise? Why does graph collaborative filtering exhibit noise resistance? Why does graph self-supervised learning perform better than graph collaborative filtering?
4.1 Analysis of the Resistance to Noise of BPR
We first analyze why the BPR model is particularly sensitive to noisy interactions. Our analysis is based on the degradation representation quality under noisy preference distributions.
Definition 2 (Representation Quality) Let O {p ) denote the ideal latent representations learned from clean data Do, and θ {Pu: qib be any learned parameters. The represenlation quality is defined as:
Q(0) = ||0 0² Σ -IpP--PHH PulB+ Σ 11q -f
(1)
Theorem (Representation Perturbation Under Noisy Interactions) Let Oy be the minimizer of the clean empirical loss (e<r) æ)], and consider a noisy interaction distribution DD == (1 5YDg &N, where N noise distribution and 8 E [0, 1] is the noise rutio. Define the noise gradient shift as Anoise := (ae=N[VVe((nng
Ez-D, (Ve=o((Ve((0o =)]. Assume the loss ((O;x) is twice differentiable and the expected clean loss is locally strongly conver around Oo. Then, for sufficiently small ö, the representation quality satisfies:
Q(0s) = ||0g - Ooll 7 s 8². | A noise 0
(2)
where Og is the minimizer under Dj. and == v BD (e(O) x ee is the Hessian at the clean optimum. The complete proof can be found in C.1
Remark 1 Theorem 7 reveals that the degradation in representation quality under noisy interactions is quadratically proportional to the noise ratio 8, and linearly controlled by ||Ancisell, which is dictated by the model architecture.
Lemma 1 (High Gradient Sensitivity in BPR) In BPR, the scoring function is defoned [(u, i,j) P.T (q: 10-g) q;), which is linear in its porameters. Therefore, the per-sample gradient is:
vol(O:2) (q: q: Par -Pw),
with unbounded norm when embeddings grow. As a result, || || tends be large, making BPR more vulnerable. to noisy somples.
Remark 2 This explains the double-peak phenomenon observed in BPR training, especially under large embedding dimensions, due to overfiling to noise.
Experimental Evidence. To validate the robustness of BPR, we adopted a simple strategy: we applied demoising collaborative filtering methods to observe whether there would be an improvement in performance, particularly when the embedding dimensions are high (Code at D.2.
r++ + 0 (/BRR (0': {a. C ( < <T)
This equation illustrates dynamic sample-dropping stratogy Wang et al. 2021 to enhance BPR's noise robustness. During gradient updates, only samples with losses below threshold 7 are retained, filtering out high-loss interactions likely to be noise. Experiments show this approach effectively mitigates noise high- dimensional embedding spaces while improving rocommendation performance (as shown in Figure 3). The threshold generally depends on the dropout rate: higher dropout leads a lower threshold, and vice versa. We conducted extra experiments Figure 7 with different dropout rates for further evidence.
7

---

Published in Transactions on Machine Learning Research (12/2025)
4.2 Analysis of the Resistance to Noise of NeuMF
We now analyze the noise sensitivity of NeuMF from a Jacobian-based perspective. Despite the inclusion of nonlinear components, the gradient propagation through multiple nonlinear layers can amplify noise, depending on the Jacobian of the model output with respect to the parameters.
Lemma 2 (Noise-Induced Gradient Amplification) Let f'u, i) be the NeuMF prediction function, and let @ E RRd denole all model parameters. Define the Jacobian malrix of the scoring funclion with respect to parameters as J2 Vef(u, E ggxd for input z = (u, Then for noisy interaction _N, the norm of the loss gradient is bounded by:
[o((O;;)) Le 11
(5)
where La is the Lipschitz constant of the loss function. Hence, the gradient norm is directly governed by the spectral norm ||l of the Jacobian.
Theorem (Gradient Instability in NeuMF) Let f(u, i) = MLP((p..: 9:)) + q be the NeuMF scor- ing function, where MLP has L layers with weight matrices W()),... W each with spectral norm [ Then the Jocobian solisfies:
Of(u.)) 00
I (II
||
([V,,(((,,
(6)
which grows multiplicatively with depth, implying NeuMF can have exponentially amplified gradient sensitivity under noisy interactions. The proof precess are at C.2
Corollary 1 (Comparative Noise Sensitivity: NeuMF vs. BPR) Although BPR has unbounded in- ner product gradients w.r.t. embeddings, its Jacobian norm is linear in embedding size, i.e., [| x !|p'll + ||lll-- In contrast, NeuMF's Jacobian grows with the product of layer spectral norms, potentially leading to
NeuMFF [BBR
in deep or poorly regularized MLPs. Hence, NeuMF may not exhibit superior robustness, and may in fact be more susceptible to gradient perturbations from noise.
4.3 Analysis of the Resistance Noise of Graph Collaborative Filtering
LightGCN exhibits a relatively expected logarithmic growth pattern. We believe this is likely due to the inherent noise-resistant structure of LightGCN. We think that the process of aggregating information from user's neighboring nodes in graph convolution (He et al. 2020 is highly consistent with Mixup (Zhang
et al. 2018
1
(ù,
V
W
KEN:
graph convolutiio
(8)
(ū,
Σ Auu Σ
s.t.
Σ =
Mizuup
Specifically, first-order neighbors aggregate information about items, while second-order neighbors aggregate information from other users. Since Mixup is regarded as noise-resistant, we hypothesize that graph- based collaborative filtering possesses similar noise-resistant. capabilities. Moreover, this idea aligns with findings
8

---

Published in Transactions on Machine Learning Research (12/2025)
in Han et al. 2024
(ū,v) v)
Uk Vi
Wi
KEN:
Uk Σ
(9)
Wil
Wil
KEN;
KEN:
A==
Avvi
KEM
REA;
This makes it easy to understand why LCg noise-resistant, as Mixup, being a classic and popular method for domain generalization, is effective in mitigating noise disturbances. Next, we analyze from the spectral ive
Definition 3 (Spectral Representation of Graph Convolution) Let the adjacency matrir of the user- ilem interuction graph G be A & (m++))x (m=+)) and the degree malrix be D. The normalized adjacency matria A = D 11212216 AD/2A--1p Performing eigendecomposition on A:
A= UAU", A diag(A> A2...., Amta),
(10)
where 1A12 12 Ammnal, and U is the orthogonal eigenveclor malris.
Definition 4 (Low-Rank Property of Graph Convolution) The embedding propagation in LightGCN (Eq.2) can be represented as a spectral filleing process. Afler L layers of propagation, the final user /ilem embeddings are:
Pu = Σ P.
10)
q = Σ 3AA'q.") (0)
(11)
1=
1=0
where === 2... In the spectral domain, this is equivalent applying low-pass filter to the initial embed- dings:
I Pa = U E 3ua
EaN' 1=0
U D.D
(12)
The filler coefficient matrix Et 3rA allenuates high-ffequency comporents (small (Ael) while preserving low-free fow-frequnccy components (large Del), leading to low-rank embeddings. The complete proof can be found
in C.4
4.4 Analysis of the Resistance Noise of Self-supervised Graph Collaborative Filtering
Self-supervised graph collaborative filtering (SGL) is subset of graph-based collaborative filtering. Natu- rally, we can also infer that sele-super graph collaborative filtering inherently possesses the ability to resist interaction noise due to the graph convolution mechanism. Moreover, in most cases, SGL outperforms LightGCN. So, what additional module plays crucial role in this difference?
SGL adopts self-supervised contrastive learning. SGL generates multi-view representations via stochastic graph perturbations and optimizes contrastive loss. For view generation, create augmented graphs g' and g" by: edge dropout: Remove edges with probability and feature masking: zero out embedding dimensions with probability p. For contrastive loss, we maximize consistency between views for the same node:
exp (sim 22,,, z²)/T) Eee exp (sim (Z/, 2()/T)²
Lcont
Σ log uetd
(13)
where sim(-) cosine similarity, J is a temperature hyperparameter, and z z", are augmented embeddings
of u.

---

Published in Transactions on Machine Learning Research (12/2025)
Definition 5 (Mutual Information Objective in Contrastive Learning) SGL marimizes the matual
information T(2y3 z") between augmented views zyy and 2y via the contrustive loss Leont- By the information bottleneck principle:
1<2y; = H( - H(=)l="),
(14)
where H.) denoles entropy. Minimiiing Leont forces zy and z lo share noise-inuarianl information.
Theorem 3 (Embedding Space Constraint via Contrastive Learning) Let the noise perturbalion 8n. satisfy 6n 1 S, where S is the subspace of clean interaction signals. The contrastive loss in SGL implicitly constrains the embedding vectors Zu to lie within S:
Yu, Projs (zu)ll2 e,
(15)
where is constant related to noise level, and - is the orthogonal complement of S. The complete proof can be found in
C.S.
Corollary 2 (Noise Filtering in SGL) As demonstrated in Theorem 3, SGL achieves noise resistance by emploging low-poss fülering in graph convolution to oltenuale high-frequency noise while relaining essential low-frequency components, and by imposing a subspace constraint in contrastive learning to further reduce noise variance and improve noise robustness. Mathematically, SGL embeddings are:
Zu (rojs(z + Projs (8m),
(16)
low-rank signal
noise
where Projj (dw)lla S and decreases with contrastive learning strength
5 Discussion
The quality of the recommendation dataset is more important than we thought. The quality of the dataset, especially noisy interactions, yields a substantial influence on the scaling recommendation performance. When the dataset contains less noise, the model can scale more smoothly, with algorithms adapting and expanding more efficiently to handle larger volumes of data. Conversely, high level of noise data muddles the underlying patterns, undermining the model's ability to scale effectively and leading to suboptimal performance as the scale incrcases.
Collaborative filtering models with noise-resistant structures have more advantages when it comes to scaling. We can clearly see that, compared to BPR, NeuMF and LightGCN, SGL demonstrates stronger scaling capabilities. Even when the embodding dimensions are significantly increased, SGL main- (ains stable performance stays close to peak performance, unlike BPR, which experiences a sharp decline. This advantage stems from the inherent noise resistance of SGL's structure. We believe that further explo- ration of this structure could potentially lead to the emergence of a "Transformer' in the field of collaborative filtering.
6 Conclusion and Limitations
Conclusion. Our contributions lie in the discovery of two new phenomena (double-peak and logarithmic), the interpretation of their causes, and the theoretical analysis of noise robustness in representative col- laborative filtering models. Specifically, to investigate whether scaling embedding dimensions can improve collaborative filtering models, we conducted extensive observational experiments across 10 datasets and 4 representative models. Compared to the previously recognized single-peak phenomenon, identified two novel phenomena: double-peak and logarithmic. We conducted detailed analysis of the underlying cause of the double-peak effect and found it to result from noise interactions. To empirically validate our hypot we introduced a denoising loss function, which supported our explanation. Furthermore, we provided a theoretical analysis of the models' robustness to noise with results aligned with our empirical observations.
Limitations. Due to limitations in available computational resources, we were unable to conduct ex- periments broader range of collaborative filtering models. Additionally, our observations were based
10

---

Published in Transactions on Machine Learning Research (12/2025)
primarily the Top metric with K set to 20. With sufficient resources, future work could explore addi- tional values of K (e.g., 5, 10) and incorporate more fine-grained evaluation metrics to further validate the generality of the observed phenomena.
References
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Xingzhuo Guo, Junwei Pan, Ximei Wang, Baixu Chen, Jic Jiang, and Mingsheng Long. On the embedding collapse when scaling up recommendation models. In Forty-first International Conference on Machine Learning, 2024.
Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. Understanding scaling laws for recommendation models. orXiv preprint Xrxi:22208. 08489, 2022.
Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Shen Li, Yanli Zhao, Yuchen Hao, Yantao Yao, Ellie Dingqiao Wen, et al. Wukong: Towards a scaling law for large-scale recommendation. In Forty-first International Conference on Machine Learning, 2024.
Yufei Ye, Wei Guo, Jin Yao Chin, Hao Wang, Hong Zhu, Xi Lin, Yuyang Ye, Yong Liu, Ruiming Tang, Defu Lian, al. Fuxi-a: Scaling recommendation model with feature interaction enhanced transformer. arviv preprint arXfe:250.000053 2025.
Yi Xu, Zhiyuan Lu, Xiaochen Li, Jinxin Hu, Hong Wen, Zulong Chen, Yu Zhang, and Jing Zhang. Addressing information loss and interaction collapse: dual enhanced attention framework for feature interaction. arXiv preprint arXi::2503.1133 2025.
Shaowen Peng, Kazunari Sugiyama, Xin Liu, and Tsunenori Mine. Balancing embedding spectrum for recommendation. ACM Transactions on Recommender Systems, 3(4):1-25, 2025.
Haochen Liu, Xiangyu Zhao, Chong Wang, Xiaobing Liu, and Jiliang Tang, Automated embedding size search in deep recommender systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2307-2316, 2316, 2020.
Yunke Qu, Tong Chon, Xiangyu Zhao, Lizhen Cui, Kai Zheng, and Hongzhi Yin. Continuous input embed- ding size search for recommender systems. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 708-77 717, 2023.
Shuyao Wang, Yongduo Sui, Jiancan Wu, Zhi Zheng, and Hui Xiong. Dynamic sparse learning: A novel paradigm for efficient recommendation. In Proceedings of the 1 17th ACM International Conference on Web Search and Data Mining, pages 740 749, 2024.
Qinyi Luo, Penghan Wang, Wei Zhang, Fan Lai, Jiachen Mao, Xiaohan Wei, Jun Song, Wei-Yu Yu Tsai, Shuai Yang, Yuxi Hu, et al. Fine-grainee grained embedding dimension optimization during training for recommender systems. arXiv preprint ar arXiv:241.044000 2024.
Siyi Liu, Chen Gao, Yihong Chen, Depeng Jin, and Yong Li. Learnable embedding sizes for recommender systems. In International Conference on Learning Representations, 2021.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thiees Bpr: Bayesian person- alized ranking from implicit feedback. UAI '09, 2009.
Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised graph learning for recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 726-733, 2021.
Zheng He, Zeke Xie, Quanzhi Zhu, and Zengchang Qin. Sparse double descent: Where network pruming aggravates overfitting. In International Conference on Machine Learning, pages 8635 8659. PMLR, 2022.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Rengan Xu, Junjie Yang, Yifan Xu, Hong Li, Xing Liu, Devashish Shankar, Haoci Zhang, Meng Liu, Boyang Li, Yuxi Hu, et al. Enhancing performance and scalability of large-scale recommendation systems with jagged flash attention. arXiv preprint arXiv:2409.15377, 2024.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pages 173-182, 2017.
Xiangman He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 639-648, 2020.
Lanling Xu, Zhen Tian, Gaowei Zhang, Junjie Zhang, Lei Wang, Bowen Zheng, Yifan Li, Jiakai Tang, Zeyu Zhang, Yupeng Hou, Xingyu Pan, Wayne Xin Zhao, Xu Chen, and Ji-Rong Wen. Towards more friendly and casy-to-use benchmark library for recommender systems. In SIGIR, pages 2837-2847. ACM, 2023.
Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang, Autodebias: Learning to debias for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 21-30, 2021.
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3): 1-39, 2023.
Wenjic Wang, Fuli Fong, Xiangnan He, Liqiang Nic, and Tat-Seng Chua. Denoising implicit foedback for recommendation. In Proceedings of the 14th ACM international conference web search and dala mining, pages 373-33333333 183-381,44 2021.
Zhuangzhuang He, Yifan Wang, Yonghui Yang, Peijic Sun, Le Wu, Haoyue Bai, Jinqi Gong, Richang Hong, and Min Zhang. Double correction framework for denoising recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1062- 1072, 2024.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
Xiaotian Han, Hanqing Zeng, Yu Chen, Shaoliang Nic, Jingzhou Liu, Kanika Narang, Zahra Shakeri, Karthik Abinav Sankararaman, Song Jiang, Madian Khabsa, Qifan Wang, and Xia Hu. On the equivalence of graph convolution and mixup. Transactions on Machine Learning Research, 2024.
Gaowei Zhang, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen. Scaling law of large sequential recommendation models. Xi preprint arXiv:2311.11331 2023.
Donald Loveland, Xinyi Wu, Tong Zhao, Danai Koutra, Neil Shah, and Mingxuan Ju. Understanding and scaling collaborative filtering optimization from the perspective of matrix rank. In Proceedings of the ACM on Web Conference 2025, pages 436 449, 2025.
Hung Vinh Tran, Tong Chen, Nguyen Quoc Viet Hung, Zi Huang, Lizhen Cui, and Hongzhi Yin. A thorough performance benchmarking on lightweight embedding based recommender systems. ACM Transactions on Information Systems, 2025.
Antonio Ginart, Maxim Naumov, Dheevatsa Mudigere, Jiyan Yang, and James Zou. Mixed dimension embeddings with application to memory-efficient recommendation systems. In 2021 IEEE International Symposium on Information Theory (ISIT), pages 2786 2791. IEFF, 2021.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Weipu Chen, Zhuangzhuang He, and Fei Liu. When sparsemoe meets noisy interactions: An ensemble view on denoising recommendation. In TCASSP 2025-2025 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1 5. IEEE, 2025.
Yonghui Yang, Le Wu, Zihan Wang, Zhuangzhuang He, Richang Hong, and Meng Wang. Graph bottlenecked social recommendation. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Dala Mining, pages 3853 3862, 2024.
Yonghui Yang, Le Wu, Yuxin Liao, Zhuangzhuang Hc, Pengyang Shao, Richang Hong, and Meng Wang. Invariance matters: Empowwring social recommendation via graph invariant learning. In Proceedings of the 48th International ACM SIGIR Conference Reseorch and Development in Information Retrieval, pages 2038 2047, 2025.
Weilin Lin, Xiangyu Zhao, Yejing Wang, Yuanshao Zhu, and Wanyu Wang. Autodenoisc: Automatic data instance denoising for recommendations. In Proceedings of the ACM Web Conference 2023, pages 1003 1011, 2023.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Appendix of "Understanding Embedding Scaling"
Contents
Notions
15
B Related Work
15
B.1 Scaling Recommendation,
15
B.2 Efficient Recommendation.
15
B.3 Denoising Recommendation
16
C Theoretical Discussions
16
Proof of Theorem
16
Proof of Theorem 2
C.3 Proof of Theorem 3
18
Derivation of Definition
18
D Experimental Settings
19
D.1
Hyperparameters
19
D.2 Pytorch Code of BPR Drop
19
D.3 Delailed description the dalasets
D.4 Reason for Choosing Models
20
E
Experiments
20
All Observations
20
E.2 Effect of Varies Dropping Ratio
21
14

---

Published in Transactions on Machine Learning Research (12/2025)
A
Notions
Table 1: Symbol Definitions
Symbol
Description
Indices representing users.
Indices representing items.
E R
Latent representation vector for user Latent representation vector for item Set of all user and item latent representations.
qi E 1t*
+ {Pu, q.)
O = (pa: q:)
Ideal set of user and item latent representations satisfying true preference relations. Optimal representations by minimizing the loss function without noise. Optimal representations by minimizing the loss function with noise proportion f/- Set of authentic (noise-free) interaction triples (u, Set of noisy interaction triples ju,
Oo
Do
Dn
D Do UD,
Combined set of authentic and noisy interaction triples. Proportion of noisy interactions in the dataset.
E (0
A>0
Regularization parameter in loss function. Sigmoid function.
(=) Ite
())
Loss function without noise.
(0)
Loss function component due to noisy interactions. Total loss function with noise = Lo(l) + (O). Representation quality metric. Frobenius norm.
((0)
Q())
1
12
Euclidean (I.2) norm.
#2
Strong convexity parameter.
c> O
Lower bound constant related to noisy interactions.
B
Related Work
B.1
Scaling Recommendation.
Large language models (LLM) have shone brightly in today's era, and one of the key reasons is that the performance of the foundational Transformer [Vaswani] 2017) model scale with parameter expansion.
Recently, few efforts (Ardalani et al. 2022
Zhang et al. 2023 2024 Guo et al. 2024 Xu et al. 2024 Ye
ct al.
2025 Xu et al. 2025 Peng et al. 2025
Loveland et al. 2025 in the rocommendation field have explored
8 similar idea: whether the performance of recommendation models can increase with parameter scaling,
(Ardalani et al. 2022) explores the scaling properties of recommendation models across data, compute, and parameters, revealing that parameter scaling has limited impact on performance compared data scaling, and highlights the need for future advancements in model architecture and system design. (Guo al., 2024) identifies the embedding collapse phenomenon the inhibition of scalability, where the embedding matrix leans towards occupying a low-dimensional subspace. It demonstrates a two-sided effect of feature interaction in recommendation modcls, with collapsed embeddings restricting learning and worsening the collapse.
B.2 Efficient Recommendation.
How to set the dimension of embeddings is a common question for almost everyone studying collaborative filtering. This is because one of the core aspects of collaborative filtering is representing users or items with embeddings, and the quality of these embeddings directly determines the performance the recommendation system. Reflecting on the efforts recent years, to determine the optimal embedding dimension, many
rescarchers have learned the embodding dimension automatically from the perspectives of neural architecture
search (Luu al.
2020:
Qu et al.] 2023), weight pruning (Wang et al. 2024 Luo et al. 20044] Liu et al., exploring mixed dimension GGnart et al. 2021 These methods seem to have
2021
Tran al.
2025
15

---

Published in Transactions on Machine Learning Research (12/2025)
yielded notable results. The implicit assumption of these methods is that there exists optimal dimension, meaning the performance exhibits single-peak phenomenon.
B.3 Denoising Recommendation
In recommendation systems based on implicit feedback, user behaviors (e.g,, clicks, views) often contain noisy interactions--igee that do not truly reflect user preferences, such as clicks driven by item position or popularity rather than genuine interest, Chen et al. (2025); Yang et al. (2024 2025). These biases degrade model generalization and make learning reliable user preferences challenging. One the simplest and most practical approaches the drop-based method, which identifies noisy samples based on their loss values: samples with higher loss are more likely be noisy, while those with lower loss are considered clean. drop- based method filters out noisy interactions before training. For example, T-CE (Wang al. 2021 finds that noisy samples tend to have higher loss values, which can be used denoising signal. AutoDenoise {Lin scarch-and-deciste process and applics reinforcement learning to automate it. DCF (He et. al. 2024) further improves performance by distinguishing between noisy interact ions and hard samples, and by relabeling the noisy ones.
ct al. 2023) models denoising
Theoretical Discussions
C.1 Proof of Theorem
Proof. Since Oy minimizes the clean objective, we have the first-order optimality condition:
VelEaD,, (((0:: æ)] 0.
(17)
Let us denote the noisy objective
(s(9) == Ex-D, = (1 E) [((9;))] + (E
(18)
Then the gradient of the noisy objective at the clean optimum On is:
VeLs(Oo) = 1 tV
8 A noise-
Assume the loss twice continuously differentiable and the clean loss locally μ-strongly convex. Let ((0;:=)l be the positive definite Hessian. Then, using Taylor expansion:
VeLo(Os) = VeLs(O,) + H(O, Oo) + R.
(19)
where o(I@, -ol12).
At the noisy optimum, the gradient vanishes:
0 = VeLs(Os) + + - - Anoise).
(20)
Now compute:
Q(0,) 110, Ool
RJ 118 A "noissa 11
8² 11 A Anoisa A +
C.2 Proof of Theorem 2
Proof. Let iu, = ( +p) q, and define the loss ((0x)) )= ((f(u, i),u) for sample x = (u, The gradient of the loss with respect to model parameters & satisfies the chain rule:
Vol(O;z) V=l(f(u,i).)) (ol(u,
(21)
16

---

Published in Transactions on Machine Learning Research (12/2025)
where : f(u, R is the scalar output score.
Let 1 == (u, denote the Jacobian of the scoring function. Taking norms on both sides yields:
[|Ve((;;))] < [VVe(((( (), || -
(22)
We now analyze ||2|| The scoring function can be decomposed as:
f(u, MLP((pp:qu) q:l) plq.
(23)
The Jacobian of the inner product term is linear in Pa and qi and contributes only bounded term proportional to | + ll We focus on the MIP term, which dominates in depth.
Let the MLP be defined recursively as:
h0)) = [Pai qil- h ol' (a()) for 1 = I....,,,
where (00) is a pointwise activation function (e.g., ReLU or sigmoid), and W E Rdxdd- the weight matrix layer I. Then the gradient of the output with respect parameters is:
0f(u,i) 00
(MLP(h(0)
a(PT.)
(24)
de
00
Focusing on the MLP term, its Jacobian w.r.t. (0)) is:
(MLP(h(0)
II Dw
(25)
(h(O)
where D(O == diag(o" (W())(1)) diagonal matrix of activation derivatives.
Taking operator norms, we get:
OMLP(h(0) (h
$ II 11 ||W0)|2.
(26)
Since ||D00|2 & max, (x)), which is bounded for standard activations like ReLU (bounded by 1) or sigmoid (bounded by 1/4), we conclude:
05 (u,
L < CC II ||
(27)
de
for some constant C, > 0 depending activation functions. Combining this with the earlier chain rule, we obtain:
L Ve S (V:((((,)) Co. II 11
(28)
Hence, the gradient norm can grow multiplicatively with the spectral norms of the MLP layers, implying exponential sensitivity in depth.
17

---

Published in Transactions on Machine Learning Research (12/2025)
C.3 Proof of Theorem 3
Proof. Decompose the embedding 2/3 + ow: where =/ E S is the clean signal and dn E S+ is noise. The contrastive loss requires:
(zn + " + 8")
cos(2, (())
= 1.
(29)
22 + 8ulll + 81l
Expanding this approcimation:
18 1
|18 8 = 0. Since 8 and 6 are i..d. the expected variance is:
=
(30)
2 2
E - 8M | [1 2 = 2Vr
(31)
Thus, Var(6.)) S E,, constraining noise to low variance and ensuring embeddings primarily reside in S.
C.4 Derivation of Definition 4
Derivation. We begin by recursively applying the propagation rule described in Eq. 12 The embeddings of the I-th layer are given by:
p() APP!-1)
(32)
where ((0) represents the initial embeddings, and Á is the normalized adjacency matrix. This recurrence relation reflects the embedding propagation at cach layer of the graph convolutional network.
The final embeddings after L layers of propagation are obtained by laking the weighted sum across all layers:
P
(33)
1=0
where A== =tr is the weight for the 1-th layer.
Substituting the spectral decomposition of À into this expression, we gel:
UAUT
(34)
where is the matrix of eigenvectors and A is the diagonal matrix of eigenvalues of Å. Therefore, we can rewrite the embeddings as:
P Σ B
(35)
Using the associalivily of matrix multiplication, this simplifies to:
P U
Σ BA 1 p'' 1-0
(36)
Now, let us analyze the behavior of the filter coefficients EL BrA in the spectral domain. For each
eigenvalue d of Å, the correspouding coefficient behaves as follows:
For A 1 (low-frequency components), have:
Σ B E.H.
144
-1
(37)
1=0
This means that low-frequency components preserved.
18

---

Published in Transactions on Machine Learning Research (12/2025)
For A x 0 (high-frequency components), we have:
1 0' 0.
Σ B Σ
2141
(38)
L+1 t=0
This means that high-frequency components are attenuated, effectively suppressing noisc.
Thus, the filter Σi=o ArA' acts as a low-pass filter that suppresses high-frequency components and retains the low-frequency components of the embeddings.
As result, the final embedding matrix P dominated by the low-free frequenquncy components, and the rank of P is bounded by the number of significant eigenvalues (those corresponding to low-frequency components). Specifically, if r is the number of significant cigenvalues, can conclude that the rank of P is bounded by r:
rank P) Sr.
(39)
This establishes the low-rank property of the embeddings after L layers of propagation in the Light GCN model.
D Experimental Settings
D.1 Hyperparameters
We pick the most commonly used Top-k metries NDCG@K measure performance, as well for com- prehensive view of the phenomenon,
set to be 20. We set the embedding dimensions powers of 2, starting from 2 and going up to 2 We train the model for 300 epochs with batch size of 2,048 using the Adam optimizer. The learning rate is fixed at 0.001 without weight decay (L2 regularization), and we disable gradient norm clipping. For negative sampling during training, we adopt a static strategy with negative item per positive interaction, sampled uniformly from the item space. We evaluate the model after every training epoch and implement early stopping validation performance does not improve for consecutive epochs. All models operate in essentially the same way; apart from differences in architecture, aspects like training loss nearly identical.
D.2 Pytorch Code of BPo_Drop
Code of BPR Drop
class BPRdropLoss (nn. :Module """BPRLosS, based on Bayesian Personalized Ranking
Args:
gamma (float); Small value to avoid division by
Shape:
Pos_score: (1)
Neg_score: (N), same shape the Pos_score
Dutput: scalar.
Examples::
>>> loss BPRLoss ()
>>> pos_ _o__e torch. randn (3, requires_grad-True)
>>> neg_score score
requires_grad-True) >>> output loss (pos_acore, neg_score)
>>> output. backward
19

---

Published in Transactions on Machine Learning Research (12/2025)
def ___iitt__.. __init__(self, rave_ratio, get_lov, gamma=1e-10) self).__ind ___
self gamma gamma
ratio = save_ratio
self. self.get_low get_low
forward(self,, ves_score, neg_score).
loss -torch. log (self gamma + torch. www.oiddppssssseeocooeeeieosseee (as__scorr)mmeeee
per par__ample__ossee losses -torch. (self ganma torch. torch.sigmoid balamooddppsssccoo neg_score).
num num_samples per_sample__ @aample_loseee lossessssssss. s0)
int iiittnnmm faat(n___aamn samplasslleesseeeeeeeeeemmeeeese * self save_ratio)
max 1)
if self self.get__te low 1:
else: - indices = torch. paaer__smmee__ssess k, largest=False)
indices = tocchttopppppe_ssste losses, k, largest=True)
selected_lossee - per_sample_len sosseee___ssseeeee [osses meaa)))
return selected selected_losser losses
D.3 Detailed description of the datasets
In our research, we utilized four distinct datasets to ensure the compreeeensiveness and robustness our anal- ysis. Each dataset contributes unique characteristics our study. Table summarizes the statistics of the 10 datasets used in our experiments, demonstrating the comprehensiveness of our evaluation. The datasets span wide range of interaction scales, from Small (e.g., ML-100K) to X-Large (e-g., Amazon Books), and cover various sparsity levels, from relatively dense (e.g. ML-1M) to extremely sparse (e-g., Amazon Beauty). Notably, the majority of datasets fall into the Ultra-sparse category, reflecting the real-world challenge of data sparsity in recommendation. This diversity ensures a thorough and robust evaluation of model per- formance across different data regimes. The color-coded annotations provide inluitive insights into dalaset characteristics.
D.4 Reason for Choosing Models
BPR RRndle et al. 2009 is a classic collaborative filtering model known for its simplicity and ease of use. Many subsequent influential methods have been developed based on NeuMF (He et al. 2017). in contrast, is a highly representative deep learning- based model that adopts classic two-tower architecture, combining linear modeling via matrix factorization with nonlinear interaction modeling through neural networks. As and items be naturally mod-
research progressed, it became clear that the relationships between eled as graph structures. This insight led to the emergence of numerous graph-based collaborative filtering approaches, such as LightGCN (He et al. 2020). Building upon graph structures, graph self-supervised mod- further enhance performance by leveraging both feature and structural
els-such as SGL (Wu et al., 2021] 2021)-furtte -urtherr information, attracting signilicant atteution. These are the reasons why we chose focus on these three categories of models. The more specific information is Table 2
E
Experiments
E.1
All Observations
we present additional experimental results on various datasets. As shown in Figures and SGL demonstrates slower performance degradation trend compared to LightGCN, which in turn outperforms NeuMF, while BPR performs similarly to SGL. These results are consistent with theoretical analysis,
20

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: collaborative filtering models and their publication details
Model
Paper Title
Year
Venue
Citation
BPR
BPR: Bayesian Personalized Ranking from Implicit Feed-
2009
UAI
8,232
back
NeuMF
Neural Collaborative Filtering
2017
WWW
8,052
LightGCN
LightGCN: Simplifying and Powering Craph Convolution
2020
SIGIR
4,358
Network for Recommendation
SGL
Self-supervised Graph Learning for Recommendation
2021
SIGIR
1,423
Table 3: Statistics of the 10 benchmark datasets used in our experiments, highlighting both the dataset scale and sparsity level. Scale is categorized based on the mumber of interactions: Small (<IM), Medium (1M-5M), Large (5M-10M), and X-Large (> 10M). Sparsity Level is categorized based on the sparsity ratio: Dense (<0.6), Sernissparse (0.6 0.95), Sparse (0.95 0.99), and Ultra-sparse (20.99). Colored backgrounds provide intuitive cues for comparison across datasets.
Dataset
# Users
#Items
#Interactions
Sparsity
Scale
Sparsity Level
MI-100K
943
1,682
100,000
0.93695
Small
Semi-sparse
Modeloth
9,000
10,000
500,000
0.99945
Small
Ultra-sparse
Amazon Baby
531,890
64,426
915,446
0.99997
Small
Ultra-sparse
ML-1M
6,040
3,706
1,000,209
0.97570
Medium
Sparse
Douban
11,257
2,281
1,330,000
0.99945
Medium
Ultra-sparse
Pinterest
55,187
9,911
1,445,622
0.99736
Medium
Ultra-sparse
Amazon Beauty
1,210,271
249,274
2,023,070
0.99999
Medium
Ultra-sparse
Yelp
1,326,101
174,567
5,261,669
0.99998
Large
Ultra-sparse
Gowalla
107,092
1,280,969
6,442,892
0.99990
Large
Ultra-sparse
Amazon Books
Bookks
8,026,324
2,330,066
22,507,155
0.99990
X- X-Large
Ultra-sparse
confirming the validity of our investigation into the noise robustness of different models. Additionally, we observed double-peak phenomenon in other datasets as well (e.g., Figure 4 (b, e, h)), along with an logarithm phenomenon (e.g., Figure (i, k, 1)). Moreover, the single-peak phenomenon widely discussed prior work is also clearly present in our experiments (e.g., Figure (a, b, c, etc.)). We argue that SGL's ability to maintain or even improve performance as the embedding dimension increases is largely due to the regularization and robustness enhancement effects introduced by contrastive learning. In contrast, GiN lacks auxiliary training signals and regularization methods beyond supervised learning, making it more prone to overfitling or performance degradation in high-dimensional settings. Overall, these experimental results further validate the generality of the newly observed phenomena and confirm that the observed performance trends are well aligned with our theoretical analysis.
E.2 Effect of Varies Dropping Ratio
The Figures 7 show the NDCG@20 of BPR Drop with different save-ratios and standard BPR on ML 100K and Douban datasets as embedding size varies. In Figure (a) for MI-100K, BPR Drop with save ratios 0.95, 0.9, 0.85, 0.8 and BPR are compared as embedding size ranges from 2 to 2 BPR BPR_Drop exhibits distinct trends, indicating sensitivity to scaling. In Figure (b) for Douban, similar comparisons are made. BPR__rooo performance curves show unique patterns with embedding size changes, suggesting better scalability than standard BPR. These plots offer insights into BPR Drop's scaling behavior for optimization in different datasets.
21

---

Published in Transactions on Machine Learning Research (12/2025)
0.36
0.26
0.4
NDC
0.33
NDX066220
0.25
NDCG@20
0.7
0.24
0.30
0.32
0.23
0,22
0.27
0.24
0.22
2.
2/4
2
2
2
2
2
2
2'
2
2
2
Embedding Size
Embedding Size
Emhedding Size
(a) Amazon Beauty- BPR
(b) Amazon Baby BPR
(c) Amazon Books BPR.
0.28
0.40
0.36
ND006820
NDX0G1800
0.27
NDCG@20
0.36
0.32
0.26
0.33
0.25
0.30
0.29
0.28
2.0
2/4
2
2
210
2.4
2
2
2
2
2
Embedding Size
Embedding Size
Emhedding Size
Amazon Beauty NeuMF
Amazon Baby NeuMF
(f) Amazon Books NeuMF
0.42
ND00000
0.34
NDC0G220
0.29
NDCG1220
0.37
0.28
0.30
0.32
0.27
0.27
0.27
0.23
0.25
0.22
22.12
244
2
2
2
2
2
2
2
Embedding Size
Embedding Size
Emhedding Size
(g) Amazon Beauty LightGCN
(h) Amazon Baby LightGCN
(1)
Amazon Books LightGCN
0.50
0.50
NDCCGG200
NDCGG000
0.42
0.34
0.34
0.34
0.25
0.25
0.17
0.17
2a 2
2
232
29
2
233
2
29
2.2
2
2
Embedding Size
Embedding Size
Emhedding
(i) Amazon Beauty SGL
(k) Amazon Baby SGL
(I) Amazon Books SGL
Figure 4: Scale the embedding dimension exponentially by factor of 2 across different collaborative filtering models and datasets. Each row corresponds to a model (BPR, LightGCN, SGL, NeuMF), and each column represents a dataset (Amazon Beauty, Amazon Baby, Amazon Books).
22

---

Published in Transactions on Machine Learning Research (12/2025)
0.29
0.37
NDC06:20
NDX0G6200
0.47
NDCG@20
0.24
0.31
0.21
0.28
0.19
2
2.4
2
2
29
2
2
2
2 2
2
2
Embedding Size
Embedding Size
Emhedding Size
Yelp BPR
(b) Gowalla BPR
(c) Pinterest -BRR
0.29
0,52
0.37
N0000800
0.27
NDXOG
0.49
NDCG@20
0.3
0.26
0.46
0.24
0.43
0.33
0.23
0.32
0.40
29
2
2
29
2
2
2
2
2
2
2
Embedding Size
Embedding Size
Emhedding Size
(d) Yelp NeuMF
Gowalla NeuMF
(f) Pinterest
NeuMF
0.29
0.54
0.39
NDC0@20
0.26
SDCGG220
0.49
SDCGG220
0.36
0.44
0.33
0.22
0.19
0.30
0.15
0.34
0.27
2
2
2
2
2
2
2
2
Embedding Size
Embedding Size
Emhedding Size
(g) Yelp LightGCN
(h) Gowalla LightGCN
(i) Pinterest LightGCN
0.40
0.30
VDC06:00
NICCG@20
0.40
NDCGG000
0.37
0.27
0.24
0.28
0.21
0.16
0.31
0.18
0.04
0.28
29
2
2.5
24
29
2
2
2
2
2
2
2
Embedding Size
Embedding Size
Fmheeding
(i) Yelp SGL
(k) Gowalla SGL
(1) Pinterest SGL
Figure 5: Scale the embedding dimension exponentially by factor of 2 across different collaborative filtering models and datasets. Each row corresponds to a model (BPR, LightGCN, SGL, NeuMF), and each column represents a dataset (Yelp, Gowalla, Pinterest)).
23

---

Published in Transactions on Machine Learning Research (12/2025)
0.45
0.48
0.6
VIX001800
0.1
NDCG6800
0.42
NDCG6620
0.62
0.37
0.59
0.57
0.31
0.29
2a 2
2
2
2' 212
2
2
27
2
2
2
Embedding Size
Embedding Size
Fiheeddin Size
(a) ML-1M BPR
(b) ML-1M LightGCN
(c) ML-1M SGL
Figure 6: Scale the embedding dimension exponentially by factor of 2 across different collaborative filtering models and datasets.
0.50
0.65
0.46
NDCG
NDCG@20
0.63
0.38
save ratio-0.95
save
0.61
save matio-0.9
save ratio-0.85
0.34
save ratio=0,85
save ratio=0.8
0.30
save ratio-0.8
0.59
save ratio-0.95
BPR
BPR
0.26
0.57
29
222
2
2
29
212
213
2
Embedding Size
Embedding Size
(a) ML-100K
(b) Douban
Figure Comparing the standard BPR with the denoising strategy-based BPR BPR_Drop in MI-100K and Douban, we can clearly observe that the performance degradation has significantly improved.
24


Reviews:
[{"review_id": "39zFUPQIwR", "reviewer": "Reviewer_BBUR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates how embedding dimension scaling affects the performance of collaborative filtering models. Through experiments on 10 popular datasets and 4 classical models (BPR, NeuMF, LightGCN, SGL), the authors discover two new phenomena: the double-peak effect, where performance swing with increasing dimensions, and the logarithmic effect, where performance grows in a logarithmic manner. They attribute these effects to differences in noise robustness. Theoretical analysis reveals that BPR and NeuMF suffer from gradient instability, while graph-based models remain robust through structural mechanisms like spectral filtering and contrastive regularization. Empirical results and a denoising strategy futher show that noisy interactions cause the double-peak phenomenon.\n\nStrength: \n1) This work is written with great clarity, rigor, and interest. It builds insightful observations from newly discovered phenomena and tightly connects theoretical analysis with experimental validation. The overall logic is coherent and well-structured, leaving little room for criticism.\n2) This work provides a deeper understanding of embedding scalability and offering insights into noise robustness and provides formal proofs to link noise robustness to model design. \n3) This work includes a comprehensive set of experiments, demonstrating substantial implementation effort. This makes the work particularly solid and convincing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The important new phenomena discovered in this work will serve as one of the key contributions to the future development of generative recommendation. It shifts the scaling perspective from improving Transformer architecture to emphasizing the intrinsic characteristics of collaborative filtering—an insight of substantial significance.", "audience_yn": "Yes", "audience_explanation": "I think the observations in this paper will capture the interest of people in this field.", "requested_changes": "Weakness:\n1) While the proposed Double-peak and Logarithmic Phenomena offer useful insights, they may not fully explain all experimental observations. For example, in Figures 5 and 6, some cases still show a single-peak behavior, which the current framework does not entirely account for.\n2) Some typos, such as incorrect singular or plural usage of phenomenon in the abstract.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "60rzfrQ9aO", "reviewer": "Reviewer_GSnC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contributions of this work include (1) two new phenomena i.e., \"double peaks\" and \"logarithmic\" discovered from extensive experiments; (2) uncovering that interacting noise is the main reason for double peaks; (3) analysis of the model robustness of  four typical collaborative filtering.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This work is important because it disgrees with the old understanding that \"there is a unique optimal embedding dimension\". It for the first time show two new findings (double peaks and logarithmic), and shows us the cause (interactive noises). It also theoretically explain why noise-resilient structures like SGL has better scalability.", "audience_yn": "Yes", "audience_explanation": "This work may be interesting to a large number of potential readers including theoretical researchers and engineers in recommandation systems. It's empirical findings (double peaks and logarithmic) and theoretical analysis are new.", "requested_changes": "This paper is well motivated and very interesting. I only have very minor comments as follows.\n\n1. The writing is not always satisfactory. There are typos like \"two novel phenomenon\".\n\n2. I think it is important for the experiments to be repeated. I suggest the authors opensource the code for reproductible evaluation.", "broader_impact_concerns": "To the best of my knowledge, there are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yhRqydoDzG", "reviewer": "Reviewer_nKt4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on giving reason to a popular question in collaborative filtering \"Why does expanding embedding dimensions not always result in performance enhancements\". The paper evaluates 4 different models across 10 different datasets of varying sparsity levels. The observe two novel phenomenons double-peak and logarithmic. For the former, as the embedding dimension increases the performance improves, then declines, rises again and eventually drops. For the latter, it follows a perfect logarithmic curve. The paper also tries to explain the cause of double-peak phenomenon, links it to noise and sparsity and finally theoretically analyze the noise robustness of collaborative filtering models.\n\n## Strengths\n- The paper tackles a relevant question of \"why expanding embedding dimension not always results in performance enhancements in collaborative filtering\".\n- They use sufficient datasets and do sufficient experiments.\n- Testing and comparison of models BPR and BPR_Drop justifies the dependence of double peak phenomenon on noise in the datasets, which is a nice addition.\n- The theoretical analysis of noise does gives some intuition as to why some models consistently perform better than others.\n\n## Weakness\nI am mentioning the weakness and my questions in the *Requested Changes* section, please see that.", "claims_and_evidence_yn": "No", "claims_explanation": "I have some doubts about the claims and need more convincing, I am posting my queries in requested changes.", "audience_yn": "Yes", "audience_explanation": "Yes the paper target relevant questions and some people will definitely be interested in it.", "requested_changes": "## Weakness/Questions\nI have marked question as [Critical] which are critical to secure my recommendation for acceptance.\n\n1. [Critical] First a basic question, I see authors call the logarithmic plot ideal and would want to achieve it, can you convince me? Based on figure 1 it looks like double-peak gives better results using fewer embedding dimensions and if one finds the first peak it's a better result as it gives better performance using smaller embedding dimension.\n2. [Critical] Extending to the first question, I understand that for a logarithmic curve finding the embedding dimension will be much easier, however I see that in double-peak second peak is always as lower accuracy than the first peak, so my objective will be to find the first peak, which should not be that hard. Is there an example where the second peak is higher than the first one?\n3. [Critical] I fail to understand the reason why increasing the embedding dimension will shift the model's learning from phase 2 to phase 3, i.e., learning noise which degrades performance to adapting to the noise thereby improving the performance? This is very core question for this paper and is not clear to me after reading the draft, I request the authors to explain it clearly.\n4. [Critical] Authors compare the phenomenon of double-peak with sparse double descent by linking change in embedding dimension with sparsity, does it mean that double-peak is just sparse double descent with changing embedding dimension? How is it different from sparse double descent? \n5. [Critical] In the experiments authors use early stopping, which won't allow the model to overfit to the training data, so the numbers in plot are not numbers for overfitted model. From my understanding the models should be trained till training error reduces so that the plots will also show overfitting as the embedding dimensions blows up.\n6. In first paragraph of page 3, line 2-3 there seem to be incorrect notations use, for $P_u \\in \\mathbb{R}^k$ there is no $i$ and similarly for $q_i \\in \\mathbb{R}^k$ there is no $j$ used.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)