Adaptive Regularization for Large-Scale Sparse Feature Embedding Models
Authors: Mang Li
Wei Lyu
Venue: ICLR 2026 Poster (2026)
Keywords: adaptive regularization
CTR estimation
large-scale sparse feature
optimization
one-epoch overfitting
Primary area: optimization

Abstract: The one-epoch overfitting problem has drawn widespread attention, especially in CTR and CVR estimation models in search, advertising, and recommendation domains. These models which rely heavily on large-scale sparse categorical features, often suffer a significant decline in performance when trained for multiple epochs. Although recent studies have proposed heuristic solutions, the fundamental cause of this phenomenon remains unclear. In this work, we present a theoretical explanation grounded in Rademacher complexity, supported by empirical experiments, to explain why overfitting occurs in models with large-scale sparse categorical features. Based on this analysis, we propose a  regularization method that constrains the norm budget of embedding layers adaptively. Our approach not only prevents the severe performance degradation observed during multi-epoch training, but also improves model performance within a single epoch. This method has already been deployed in online production systems.

Published as a conference paper at ICLR 2026
ADAPTIVE REGULARIZATION FOR LARGE-SCALE SPARSE FEATURE EMBEDDING MODELS
Mang Li Wei Lyu
Institute of Intelligent Technology
Alibaba International Digital Commerce Group Hang Zhou, China
{mang. 11, (a186934))alibaaaae -ineaaccaiiccommm
ABSTRACT
The one-epoch overfitting problem has drawn widespread attention, especially in CTR and CVR estimation models in search, advertising. and recommendation do- mains. These models which rely heavily large-scale sparse categorical features, often suffer a significant decline in performance when trained for multiple epochs. Although recent studies have proposed heuristic solutions, the fundamental cause of this phenomenon remains unclear. In this work, we present a theoretical expla- nation grounded in Rademacher complexity, supported by empirical experiments, to explain why overfitting occurs in models with large-scale sparse categorical features. Based this analysis, propose regularization method that con- strains the norm budget of embedding layers adaptively. Our approach not only prevents the severe performance degradation observed during multi-epoch train- ing, but also improves model performance within single epoch. This method has already been deployed in online production systems.
INTRODUCTION
Click-through rate (CTR) and conversion rate (CVR) estimation are critical for advertising, search and recommendation (ASR) applications. E-commerce platforms like Amazon and Taobao rely on optimizing CTR and CVR estimation to boost gross merchandise volume (GMV), while advertis- ing platforms Google and Meta depend on to drive revenue growth. In the past decade, as deep learning has been widely adopted in ASR applications, most estimation models have been built on deep learning frameworks and rely on large-scale, sparse categorical features (Cheng et al. (2016); Guo et al. (2017); Lian et al. (2018)). Recent work (Zhang et al. (2022b)) demonstrates, through extensive experiments, that such models commonly suffer from the one-epoch overfitting phenomenon, where model performance drops sharply after the first epoch of training. It empiri- cally suggests that the optimal practice is to train the models for only one epoch. At the same time, their empirical analysis also reveals strong connection between feature sparsity and the one-epoch phenomenon. In ASR scenarios, categorical features can easily reach the scale of billions, and most feature values occur extremely infrequently (Xie et al. (2020); Zhang al. (2022a)). which makes models especially susceptible to overfitting after the first epoch. Additionally, Ouyang et al. (2022) reported a similar one-epoch phenomenon during supervised fine-tuning (SFT) of large language model (LLM), though they argue that moderate overfitting can actually be beneficial. We leave it in future work. In this paper, we focus on addressing the multi-epoch overfitting problem for models with large-scale sparse features in ASR applications.
Liu et al. (2023) was the first to address the one-epoch problem using a heuristic approach. They introduced a multi-epoch data augmentation method (MEDA) that can be easily applied to esti- mation models with large-scale sparse categorical features. Later, they extended to a continual learning paradigm (Fan et al. (2024)). To mitigate overfitting during multi-epoch training, MEDA reinitializes the embeddings of categorical features and their corresponding optimizer states at the beginning of each epoch. Although this approach effectively alleviates multi-epoch overfitting, it is
"Equal contribution.

---

Published as a conference paper at ICLR 2026
fundamentally heuristic because the embedding parameters are only reinitialized epoch bound- aries. It neither ensures optimal convergence nor explains the root cause of overfitting. Moreover, this embedding reinitialization approach may result in the loss of substantial information, potentially resulting in suboptimal model performance. Nevertheless, it provides important inspiration for sub- sequent research, Another method, proposed by Wang et al. (2025), adopts L.L.-style paradigm. In this approach, generative pretraining is used to obtain embeddings for categorical features, which remain frozen during subsequent training. This strategy also helps to mitigate overfitting. However, it requires significant training resources to obtain the pretrained embeddings, and the performance cannot be fairly compared to single-epoch training because the parameter budget used during the pretraining phase is not included. Therefore, it cannot be regarded as solution to the one-epoch problem, although it does highlight the pivotal role of the embedding layer this phenomenon.
There are several well-established and general-purpose methods to mitigate overfitting, such as dropout (Srivastava et (2014)), I, and L22 regularization (Schmidt et al. (2009); Moore & DeNero 1)), and weight decay (Hanson & Pratt (1988)). However, in real-world industrial applications, data is typically high-dimensional and sparse, with different features exhibiting varying degrees of sparsity. Consequently, methods like AdamW (Loshchilov & Hutter (2017)). which apply the same weight decay across all parameters, tend to be suboptimal for addressing overfitting. Specifically, such approaches may reduce the fitting accuracy for relatively dense features while failing to effec- tively control overfitting in sparse features.
This paper presents foundational theoretical analysis of the root cause of one-epoch overfitting, and proposes a method that adaptively determines regularization coefficients according to the effective update interval of each categorical feature value, and integrates these coefficients into the optimizers update rules. The proposed approach is particularly well-suited for models with large-scale sparse features. reduces regularization for relatively dense features and neural network parameters, while assigning appropriate regularization strengths to individual categorical features. As a result, this method not only prevents the sharp performance degradation observed in multi-epoch training, but also enhances performance within a single epoch.
2 PRELIMINARY
In the industrial ASR domain, mainstream estimation models typically combine embedding layers with a variety of dense multi-layer perceptron (MLP) backbones, such as Gu et al. (2022). Taking an e-commerce platform estimation model as an example, these models process large-scale sparse inputs, including item IDs, brand IDs, and seller IDs, where the scale of categorical features can range from millions billions. Before being processed by the estimation model, these sparse fea- tures are first mapped to low-dimensional dense representations via embedding layers. The one-hot categorical feature x, (t) E R.N. is defined as (t)(t) [0, o,....,,........................ O , E [S] E [T], which can be represented by embedding vector e; (t) E Rd shown below
e; (1) E. (1)
where E, E Xcd is the embedding matrix for feature and de is the embedding size. T e N is the number of training samples. S E N denotes the number of categorical features. N; E N, Vi E [S] denotes the number of distinct values for feature i. At each update step, an embedding lookup performed to retrieve the dense representations for the sparse features via equation 1, which are then fed into the MLP layers. For simplicity, we separate the model into the embedding component and the MLP component denoted as f. In our theoretical analysis, use basic DNN backbone and provide experiments in section 4 to demonstrate the generalizability of our method with other backbone architectures.
2.1 NEURAL NETWORK DEF
The general form of basic DNN function can be defined
(2) where L denotes the number of MLP layers, af.) is the ReLU function (Agarap (2018)), W; is the linear projection matrix of layer 1, [7.]. In this paper, only discuss the impact of embedding
f-) = WioL--(Wiee (WL-10L-2 (Wi-))
2

---

Published as a conference paper at ICLR 2026
layer can define the t-th sample output of DNN
y(t f [[ x1 (t); E (2(1);.... (t))
(3)
where y(t) is the logit output for CTR or CVR estimation model.
2.2 RADEMACHER COMPLEXITY BOUND
Rademacher complexity is a standard tool to control the uniform convergence of given classes of predictors (Koltchinskii & Panchenko (2002),Zhang et al. (2016)). Formally, given real-valued
function class H. with the imput 2, we define the empirical Rademacher complexity RT (H)
RT (H) = Ee sup Ee h(zi)
1
(4)
KEH T
where E (Ei....,ETT) ET) is a vector whose entries are independent and uniformly distributed in +-1,+11. Using standard arguments, such bounds, as long the norm of is bounded, can be converted to bounds on the generalization error, assuming access to sample of iidd training samples.
Based on Theorem of Golowich et al. (2018), and noting that the embedding layer can be regarded linear projection, equation 3 as
obtain the following upper bound on the Rademacher complexity of
(TH)) - 1 (1IM(U))
Σ M
(V210g(2)L+1)
ΣΣ
(t)l2 (5)
where || denotes l2 norm, and each matrix Wi in function / has Frobenius norm most MF(1), E [T.]. ME; is the Frobenius norm of embedding matrix E, and the activation functions assumed to be I-Lipschitz and positive-homogeneone Furthermore, because Vi, (t)]( this upper bound can rewritten as
I. (TH)) = V V M )
S
N ΣΣΑm TV i=1==1
S
2 (og +1)
(6)
where Tiy the squared Eg norm j-th row of embedding matrix E, ASR applications, the ma- jority of parameters are concentrated in the embedding layers. Thus, we can see that Σ Σn= Tip has significant impact on the upper bound of Rademacher complexity, and consequently, on the generalization error bound. In appendix D, we provide an upper bound on the Rademacher com- plexity for an FM-like model, which indicates that the embedding layers also have significantly impact on the bound.
THE PROPOSED APPROACH
As discussed in section nigh-dimensional embedding matrices lead increased generalization error bounds if the training loss remains constant. On the other hand, strictly constraining the norms of the embedding vectors to reduce Rademacher complexity may increase the training error, thus po- tentially degrading overall performance. To identify the optimal regularization factor, we formulate this trade-off as a constrained optimization problem, described as follows.
S
N
S
N
min 20
migi((TT))
s.t.
ΣV
T SC
where (P(rt) min le., Ps,,, C(e.)) with E [S] and j E [N:]. Here, Giy denotes the j-th row of embedding matrix E, and L(e,)) represents the average cross-entropy (CE) loss evaluated on the DNN output when the training sample activates eij. For the CE loss function. L(e,y) is lower semi- continuous and bounded below bounded sets (Goodfellow al. (2016)). Consequently, sp(Tt)
3

---

Published as a conference paper at ICLR 2026
is well-defined (Ok (2011)) and monotonically non-increasing in Tij. since the feasible set expands with Tij. Here, Tij is the squared norm budget of embedding vector eij. According to DiBenedetto (2016), (TT,)) is differentiable almost everywhere, and we assume that (p(T)) ) is differentiable' at the optimal points 7ij for analysis. The coefficient mij denotes the sample frequeney with which embedding vector Cij appears in the training dataset and C' a constant that defines the global norm upper bound for embedding layers. As shown in equation 6, the value of C directly determines the Rademacher complexity upper bound.
Proposition 1 A necessary condition for the optimal regularization multiplier Xig associated with the ||e 2 S Ti is given by Xig poo/osp where poo is the Lagrange multiplier corresponding to E L1=1 2j=1 T C.
Proof. Based on the above assumption, at points of differentiability where a standard constraint qual- ification holds for the inner optimal solution, the envelope theorem and Lagrangian decomposition (Shapiro (1979)) yield
of ('(rty) - -
(8)
where X/j- 0 is the Lagrange multiplier corresponding to ||ey|| T... and Tiÿ is the optimal solution of equation 7. Here, (T(,)) denotes the derivative of ((rtt) with respect to T.J- We re- formulate the optimization problem defined in equation 7 with the Lagrange multipliers Pij and pos corresponding to the nonnegativity constraints of Tij 2 and Σ E Téj S C' respectively,
where May and Po are restricted be non-negative.
S
N
S
N
min T
m.)((T))) + +40
ΣV
T4
PiyTaj
(9)
s.t. p4
paj 0, Vi E [S]. [N:]
Based on the KKT condition (Boyd & Vandenberghe (2004)), we have msysp (TT) + 40 Mú = 0. Applying the complementary slackness condition, we obtain (4yTy = O. This implies the optimality condition for Tiz > 0 can be simplified as
majid' (TTT)) + 4o
(10)
By substituting equation 8 into equation 10, we obtain a necessary condition for the optimal regu- larization multiplier X;₃ Po/ma..
3.1 ADAPTIVE REGULARIZATION METHOD
As shown in equation 6, embedding layers in ASR applications typically take the majority of the model parameters and have a substantial impact on the generalization error bound. Proposition suggests that the norm budget for each embedding vector should be allocated according its sample frequency. However, it is not easy to use the frequency directly during the training process. We can estimate my via the stochastic occurrence interval Zıj of eij for i [S] and [N:]. Specifically, given that (t) is sampled from an i.i.d. distribution (a common assumption in deep learning), we have E[m,y] = T //[[ (Papoulis (1965)). It enables us incorporate frequency estimation into the norm budget allocation strategy during training.
Based on the analysis above, we propose an adaptive method that assigns the regularization strength based on the occurrence interval of each embedding vector. Specifically, at each training step k we define last valid update step (LVS) for the embedding vector Gi₃ as siy- If the gradient norm of eij satisfies || > 0. we update the LVS by setting s = k. Otherwise, sty retains its previous value. Therefore, st serves lazy-uddate variable. We define the update interval of step k as The = k k The adaptive regularization coefficient X is then dynamically computed as below
X/ = min (1,11;) i [S]. [N.]
(11)
In appendix C, give discussion for the non-smooth case.
4

---

Published as a conference paper at ICLR 2026
Here, E 1) denotes the base regularization coefficient. Following the decoupled weight de- cay approach in AdamW (Loshchilov & Hutter (2017)). we incorporate the dynamically computed regularization into each optimizer update step. Algorithm 1 outlines the procedure of Adam with adaptive regularization (AdamAR).
For clarity and practical implementation, where p E [P] and P is the total number of parameters. At each update step k we first compute the adaptive regularization XΦ according equation 11 and use it to update the corresponding model parameter og. We then identify the parameters whose gradient norm ||9 is greater than zero in current step and update their last update step state si for use in the next iteration.
0, to denote parameters in the estimation model,
In fact, our method optimizers which have weight decay factor, such as Adagrad (Duchi et al. (2011)). We have imple- mented an adaptive version for Adagrad in algorithm 2 of the appendix G. In section 4, we conducted comparative experiments evaluating performance across both Adam and Adagrad.
only suitable for Adam, but also compatible with various gradient-based
Moreover, the AdamAR algorithm requires additional storage record the last valid update step. Further details of the computation and memory analysis are provided in appendix F.
Algorithm Adam with Adaptive Regularization (AdamAR)
given 31 0.9, 32 0.999, & 10 a, learning rate 7) initialize time step A < O, parameter 0/=0, first moment mp k=0 + 0, second moment og + 0,
2:
last update step state sh=0
3: repeat
4:
h: ++1
5:
-- v (0/-1) (Get gradients w.r.t. stochastic objective at timestep t)
6:
m
+ Bimb-
k-1
(11331).55 (Update biased first moment estimate)
c + B2rl-1 + +(1 (1-B2) (of) (Update biased second raw moment estimate)
7:
8:
m t m (1 (1-39) (14 (Compute bias-corrected first moment estimate)
9:
ak + upl (1- (5) (Compute bias corrected second raw moment estimate)
p
10:
t min (1, (k - sh-1 1) a) (Compute adaptive regularization through equation 1I)
11:
k if 1l than zero)
else sp (Update the last update step when the gradient norm is greater
p 05
gk N 17. m (v +E) (Update parameters)
13:
until stopping criterion is met
14:
return optimized parameters 6r
3.2 DISCUSSION ON THE MECHANISM OF REGULARIZATTIN
As shown in equation 7, if we do not constrain the embedding norm, the global norm will continue to grow until further increases no longer yield improvements in the objective value since (TT,))) non-increasing. In other words, the embedding norms will continue grow during training. resulting in looser upper bound on the Rademacher complexity. The sharper drop in performance during multi-epoch training can be attributable to the increased Rademacher complexity resulting from unconstrained norm growth as demonstrated in the experimental section 4.4.
Then we discuss how the adaptive regularization takes effect, and also provides guidance for select- ing the regularization coefficient
When adaptive regularization is applied according to equation 11, the update rule for parameters satisfy ||0|1|| I a²²² 11eh- +1 || -l h ( +
Proposition 2
Proof. Let P E Zz0 n |0, 1/a), and a is typically chosen such that a E 10. By applying Bernoulli's inequality to (1 a) t² we obtain - a 2 al In the case where 13 Zzo O [1/a, 2o), it follows that a > 0 since a 1 and 1 > 0. Combining both cases,

---

Published as a conference paper at ICLR 2026
have
(1 - o0)* 2 min (1, a.
(12)
We substitute equation 12 into 0--1 -2 -1 - -n-mp/ (v Bp +
then we can have
|0l11 S (1 a² 1100-111 +1 |1 m (V it + e)ll
(13)
From proposition 2, the update rule for of can be interpreted intuitively. If the interval 7 is large for the corresponding sparse categorical features, the term (1 -a)² | becomes negligible, and 0g effectively determined by the latest gradient. In the ASR domain, certain categorical feature values may not appear in every mini-batch, resulting in their embedding parameters being updated infrequently, while the MLP parameters are updated in every batch. After the MLP component has nearly converged, the embedding parameters associated with sparse features will have received far fewer updates and may become misaligned with the current state of the MLP parameters, making
the previous value 6 less informative. Proposition demonstrates that the factor (1 (1-a)* aa)² ex- ponentially attenuates the previous value 08-1, making the embedding parameters of sparse features depend more heavily on the current gradient. Meanwhile, since the update interval T for MLP parameters identically zero (as they are updated in every batch), the regularization primarily fects the embedding parameters corresponding to very low-frequency feature values. Moreover, observe that the method proposed in Liu et al. (2023) is a special case of our approach when zero reinitialization is applied to the embedding and I is specified
if mod = [1/a, = Th P 10, (0
(14)
otherwise
Where B is the batch size. It can be seen that this heuristic is primarily effective at epoch boundaries. However, for features that have already become overfitted within single epoch, its performance remains suboptimal.
3.3 MINIMUM CONVERGENCE
We analyze the convergence of adaptive regularization methods the non-convex setting using the minimum convergence rules proposed by Khaled & Richtárik (2020). The assumptions underlying our convergence analysis are listed below.
The function f is differentiable and its gradient is Lipschitz continuous, i.e., there exists Lo > 0 such that [V - S Tol 1b11 vk: 2 1, and f lower bounded at the optimal solution, i.e., f -00.
Assumptions 1.
Assumptions g* is an unbiased estimator of the full gradient, i.e., E [g = ( with M > 0, and the algorithm accesses a bounded stochastic gradient, i.e., ||g S M a.s.
Proposition 3 The adaptive regularization method preserves the minimum convergence bound of the Adam optimizer with stochastic conditions, which can be expressed as
[|\)((0*)))] C+ 1hk C3 Σ 1k
min ISKS
(15)
ΣK
where C₁ O2 and C3 are constants, denotes the step size at iteration k out of K total iterations.
The proof, along with the definitions the constants C1, C2 and C3. can be found appendix B. Proposition 3 shows that neither the random noise the deterministic regularization parameter affects the minimum convergence of Adam, and it only changes the constant term in equation 15.
4 EXPERIMENTAL VALIDATION
To evaluate the effectiveness of our proposed method, we conduct several experiments on different public datasets and industrial dataset. The experimental setup is described in section 4.1. The

---

Published as a conference paper at ICLR 2026
training framework proposed by Zhu al. (2022) is used for training on public datasets, while XDL (Jiang et al. (2019)) is employed for training our proprietary industrial dataset. We show the learning curve the Avazu dataset in section 4.2 to demonstrate the better generalization of our methods over multi-epoch training. In section 4.3, we compare multiple backbones and datasets to prove the generalization of our method. We give detailed example show that the low-frequency em- beddings are the primary contributors of the one-epoch problem in section 4.4, which demonstrates should allocate a smaller norm budget embeddings with lower sample frequency. Finally, in section 4.5, we conduct an ablation study to further clarify the contribution of occurrence in- terval estimation. For reproduction, the code? is available, and we use the same seed for identical experimental settings.
4.1 EXPERIMENTAL SETUP
Datasets. In our experiments, we use three public datasets You', Amazon', Avazu along with LZD, a proprietary online business dataset from sponsored search. Dataset details are provided in appendix L.
MLP Backbones. We evaluate our method using four MLP backbones, namely DNN, WDL (Cheng et al. (2016)), XDeepFM (Lian al. (2018)) and WuKong (Zhang et al. (2024))
Methods, To evaluate the one-epoch overfitting phenomenon, we train the models for epochs and compare 4 benchmarks. 1) Baseline optimizer: Adam and Adagrad serve as the baseline optimizers. 2) MEDA: MEDA applied using the same optimizer settings as baseline. 3) Adam and AdagradW: The baseline optimizers are enhanced with weight decay only embedding layers. The baseline optimizers are combined with SAM (Foret et al. (2020)). 5) AdamAR and AdagradAR: The baseline optimizers are combined with our method.
4) SAM:
Hyperparameters. The embedding dimension is set to 32, with zero initialization. The learning rate 0.001 for Adam and 0.01 for Adagrad, respectively. The batch size is 2048. Both and the weight decay parameter are selected via grid search over values of the form 10" where ranges from -6.5 to -1 with step size of 0.5. The optimal hyperparameters are selected based on validation performance (see appendix I for details). Network architectures are configured with minor adjustments across datasets. For detailed configurations, please refer to table in appendix H. All other hyperparameters are set to their default values. Area under the curve (AUC) and binary cross-entropy loss are used as evaluation metrics. Experiments are conducted on a single machine with NVIDIA L20 GPU.
4.2 LEARNING CURVE AND GENERALIZATION RESULTS
In this section, present the training loss and test AUC across multiple epochs the Avazu dataset, using a DNN backbone and the Adam optimizer. The learning curves in figure compare method with several baselines. For the native optimizer, training loss decreases rapidly after the first epoch. However, test AUC drops sharply with additional epochs, indicating clear overfitting. In contrast, MEDA and weight decay alleviate this problem, yielding training. Our method achieves the best overall performance. Figures 1(b) and I(c) reveal an inverse correlation between the & norm of the embedding vectors and test AUC. It demonstrates that con- straining the embedding norm improves generalization ability of models. Among all methods, our proposed AdamAR achieves the lowest cumulative & norm for the embedding vectors, highlighting its effectiveness in controlling regularization strength.
stable test AUC throughout
4.3 PERFORMANCE OVER DIFIERENT DATASETS AND MLP BACKBONES
In this section, we compare test AUC across different datasets and MLP backbones demonstrate the generalization capability of our method. Each experiment is repeated three times with different seed. We report the average test AUC using Adam and Adagrad optimizers in table and table
x1

---

Published as a conference paper at ICLR 2026
3.00
0.70
0.70
1.30
0.77
Saprdd
Adea
0.70
2.24
1.32
9134
Aixell
0.68
5632
AdeeAR
0:90
Epude
Ejaacts
Equacix
(a) Training Loss
(b) Test AUC
(c) 82 norm of embedding vectors
Figure 1: Performance of four methods on Avazu dataset with DNN backbone. (a) shows the training loss curves. (b) presents the test AUC. (c) illustrates the cumulative lg norm of embedding vectors.
2, while the standard deviation and detailed scalability comparison are presented in appendix J. The results show that our methods consistently outperform MEDA and weight decay method on all datasets and architectures in single-epoch training. except for the Amazon dataset which has less features and samples where SAM achieves superior performance. Notably, in multi-epoch training, our method achieves the highest AUC, surpassing all other approaches across every dataset and ar- chitecture. Furthermore, the gains are consistent across diverse model architectures. ranging from basic DNN to more sophisticated designs like WuKong, which capture complex feature interac- tions. Overall, these results demonstrate the robustness and versatility of our adaptive regularization approach across various settings.
Table I: Comparison of average test AUC across different datasets and models using Adam op- timizer. EI-E4 denote results after 1-4 epochs, respectively. The best results are highlighted in bold.
DNN
WTM.
DDeePPM
WulKong
Datsee
Method
Adum
07515
0.7304
0.7061
0.7014
0.7619
0.7370
0,7028
0.6087
0,7500
0.7391
0.6969
0.6844
0,7611
0.7442
0,7082
0.6915
iPinYou
MEDA
0.7515
0.7644
0.7644
0.7717
0,7619
0.7599
0,7505
0.7551
0.7590
0.7584
0,7575
0.7397
0.7611
07763
0,7662
0.7706
0,7510
0.7993
0,7455
0.7256
0,7610
0,7404
0.7248
0.7568
0.3517
0,7413
0.7274
0,7033
0.7455
0.7465
0.7306
0,7623
0.7568
0.755)
0.7656
0,7607
0.765)
0.7574
0,7579
AlhmAR
0.7566
0.7692
0,7688
0.7724
0.7655
0.7729
0.7670
0.7668
0,7725
0.7733
0,7711
0.7678
0.7653
0.7736
0.7748
0.7736
Adum
0.8482
0.8548
0.8335
0.8180
0.8474
0.8510
0.8261
0.8156
0.8660
0.8535
0.8287
0.8163
0,8590
0.5600
0.8348
0.8032
MEDA
03482
0.8544
0.855
0.8573
0.8474
0.8506
0.856
018566
0.8460
0.8519
0.8551
0,8562
0,8500
0.5611
0.8021
018626
Amazon
0.8417
0.8349
0,8516
0,8505
0.8587
0,8390
0,8332
0.8588
0.8276
0.8461
0.8557
0.8381
0.5240
0.8564
0.8632
AdamAR
0.8708
0.9666
0.8496
0,8654
0.8689
0.8483
0.9676
0.8687
0.9675
0.8583
0.869
Adam
0.7461
0.7305
0.7014
0.6883
0.7483
0.7221
0.6982
0.6886
0.7488
0.7217
0,7019
0.6869
0.7514
0.7360
0.714)
0.7079
MFDA
0.7461
0.7498
0,7489
0.7485
0.7483
0.7488
0.7480
0.7494
0.7488
0.7489
0,7505
0.7506
0.7514
0.7548
0,7553
0.7571
0.2451
0.7194
0,7013
0.6899
0,7477
0.6902
0.7484
0.7190
0,7010
0.6902
0.7155
0.7156
AdamW
0,1572
0.7582
0,7582
0.7583
0.7585
0.7581
0.7563
0.7583
0.7382
0.7589
0.7393
0.7542
0.7547
AlhmAR
0.7617
0.7631
0.7629
0.7629
0.7629
0.7629
0.7627
0.7626
0.7628
0.7633
0.7638
0.7636
0.7624
0.7612
0.7623
0.7624
Adum
0.7118
0.6613
0.6252
0.6065
0.7155
0.6726
0.6308
0.6079
0.7164
0.6787
0.6321
0.6050
0,7101
0.6645
0,6102
0.6044
MEDA
0.7118
0.7105
0,7081
0.7176
0.7155
0.7162
0.7177
0.7162
0.7164
0.7170
0,7152
0.7139
0,7101
0.7170
0,7129
0.7128
0.6666
0.6341
0.6155
0.7161
0,6750
0.6344
0.6134
0.7146
0.6713
0.6443
0.6212
AdamW
0.7322
0.7135
0.7140
0.7139
0.7145
0.7138
0.7142
0.7131
0.7142
0.7151
0.7139
0.7139
0.7115
0.7135
0.7152
0.7122
AdamAR
0.7229
0.7235
0.7241
0.7234
0.7233
0.7240
0.7246
0.7240
0.7244
0.7256
0.7242
0.7238
0.7227
0.7215
0.7208
0.7202
4.4 EXAMPLE THE RoOT CAUSE OF OVERFIITING
In this section, we give an example to show the root cause of multi-epoch overfitting. We use the iPin You dataset with DNN backbone and Adam to illustrate this issue by single experiment. The detailed feature statistics are listed in table 7 in appendix K. We can observe that most features on iPin dataset are relatively dense. To investigate the root cause of overfitting, we select the feature "IP", which is the most sparse feature in IPinYou dataset with 704,966 unique IDs. We apply a filtering procedure to reduce its feature sparsity. Given a ratio r, we only retain the top-r fraction (by frequency) of IDs and replace other IDs with default ID)
8

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of average test AUC across different datasets and models using Adagrad op- timizer. E1-E4 denote results after 1-4 epochs, respectively. The best results are highlighted in bold.
DNN
WDL.
<DeopFM
WoKong
Docaset
Method
81
E1
B2
Adagmad
0.7393
0.6507
0.6231
0.5065
0,7616
0,6653
0.6321
0.6241
0.7674
0.6843
0.6505
0.6527
0.7661
0.6900
0.6497
0.6320
MEDA
0.7599
0.7686
0.771
0.772
0,7646
0.7651
0,7685
0,7715
0.7674
0.7722
0.7728
0.7740
0.7661
0.7705
0.7729
0.7895
You
SAM
0.7576
0.7661
0,7487
0,7303
0.7034
0.7076
0,7518
0.7509
0.7637
0.7661
0.7499
0.736
0.7409
0.7678
0,7525
0,7427
AdagradW
0.7558
0.7651
0.7667
0.795
0,7593
0. 17765
0,7635
0.7593
0,7665
0.7673
0.7666
0.7652
0.7578
0.7661
0.7619
0,7600
AdagadAR
0.76881
0.7744
0.7731
0.7720
0.7760
0.7768
0.7745
0.7718
Adagrad
0.5438
0.8402
0,8042
0.8406
0,7962
0.8405
0.8374
0.7996
0.8491
0.5472
0.8156
MEDA
0.5438
0.8481
0.8495
0.8505
0.8406
0.8470
0.8497
0,8510
0,8405
0.8463
0.8476
0.8500
0.5491
0.8369
0.8587
0.8009
Amazon
SAM
0,8500
0.8527
0.8361
0,8193
0.8490
0.7495
0,8325
0.8155
0.84K3
0.8521
0.5325
0.8170
0.8556
0.8367
018556
0.8174
AdggradW
0.8428
0.8578
0.8563
0.8535
0,8424
0,8553
0.8531
0,8498
0,8410
0.8565
0.8522
0.8508
0.8513
0,8630
0,5617
0,8606
AdagnalAR
0,8479
0,8659
0,8711
0,8712
0,8453
0.8631
0,8687
0,8707
0,8444
0.8641
0.8681
0,5703
0,8536
0,9690
0,5708
0,8700
Adagrad
0.7541
0.7074
0.7543
0,7550
0.7319
0.7548
0.7311
0.7541
0,7545
0.7543
0.7547
0.7551
0,7550
0.7538
Avepa
SAM
0.7553
0.7333
0.7199
0.7094
0.7557
0.7528
0.7178
0,7077
0.7564
0.7332
0.7198
0.7094
0.7558
0.7353
0.7211
0.7110
AdagradW
0.7578
0,7580
0.7515
0,7566
0.7594
0.7581
0.7567
0,7583
0,7583
0.7581
0.7580
0.7585
0.7524
0.7555
0.7555
0.7563
AdaggadAR
0.7628
0.7620
0.7630
0.7624
0.7631
0.7634
0.7623
0,7626
0.7636
0.7633
0.7638
0.7633
0.7628
0.7626
0.7627
0.7620
Adagnad
0.7222
0.6658
0.6433
0.6376
0.7344
0.6695
0.6389
0.6386
0.7222
0.6675
0.6400
0.6322
0.7247
0.6776
0.6412
0.6339
12D
MEDA
0077226
0.7244
0.7237
0.7247
0.7237
0.7234
0.72
0.7229
0.7229
0.6854
0.6802
AdagradW
017144
0.7132
0.7135
0.7154
0.7138
0.7149
0.7137
0.7145
0.7155
0.7135
0.7117
0.7127
0.7111
0.7147
0.7136
AdagradAR
0.7253
0.7247
0.7134
0.7341
0.7269
0.7258
0.7251
0.7257
6.7273
0.7268
0.7264
0.7263
0.7269
0.7260
0.7252
0.7239
Figure 2(a) demonstrates that as decreases, the test AUC remains stable across multiple epochs, indicating that the one-epoch overfitting phenomenon effectively alleviated. It suggests that low- frequency IDs in the "IP" feature are the primary cause of one-epoch overfiuting in the iPin You dataset. However, as shown by the case r = O, removing the "IP" feature results in a substantial test AUC drop (from 0.7498 0.7429) the end of the first epoch.
Although removing sparse features can mitigate the one-epoch overfitting issue, often leads to diminished model performance. In contrast, our proposed method dynamically adjusts the regular- ization strength, effectively preventing overfitting while maintaining strong predictive performance.
10
0.75
0.7:29
Test AUC
0.70
E: norm
10a
lel
16-1
le2
le-3
0.00
10-1
1e.3
le-1
0.35
10
2 Epochs
2
Epochs
(a) Test AUC
(b) & norm of embedding vectors
Figure 2:
Performance comparison using various filter ratios for the "IP" feature on the iPinYou dataset. (a) shows the test AUC results. (b) presents the cumulative la norm of embedding vectors.
4.5 ABLATION STUDY AND BUCKET ANALYSIS
We use the You dataset for analysis because it contains a single feature causing the one-epoch issue as described in section 4.4. Since "IP" can be interpreted proxy for user, create

---

Published as a conference paper at ICLR 2026
buckets based the frequency of the "IP" feature perform user-based bucket analysis, examining AUC gains, regularization strength, and &g norm of our method. A smaller bucket index indicates lower occurrence frequency. Figure shows that the bucket norms can be controlled via adaptive regularization strength while preserving the AUC gains across all buckets, and our method achieves particularly strong performance in the high-frequency bucket due to the larger norm budget avail- able. Then, we conduct an ablation study on occurrence interval estimation with a DNN backbone. In the AdamW baseline, we apply constant weight decay to the embedding layers only. We then gradually apply our method to different buckets from to 5. Table 3 shows that, compared with using a constant weight decay, decreasing the weight decay for feghffreuency features and increas- ing it for low-frequency features can further improve performance while alleviating the one-epoch issue.
1805
Mone HIDA SAld
Toegsommattnn Stroeh
1,80
84
2.30
1.25
(a) Test AUC
(b) Average regularization strength
(c) E norm
Figure 3: Bucket analysis of "IP" feature on the Youn dataset with DNN backbone and Adam optimizer. (a) shows the test AUC results over different feature frequency buckets at the end of
epoch 2. (b) presents the regularization strength. (c) shows the cumulative le norm of "IP" feature at the end of epoch 2.
Table 3: Comparison of test AUC for ablation study using Adam optimizer.
Experiment Setting
E1
E2
E3
E4
AdamW
0.7486
0.7595
0.7628
0.7500
AdamAR-Bucker & AdamW-Bucket 2-5
0.7457
0.7607
0.7646
0.7520
AddamAR-Bucket 1-2 &
0.7496
0.7622
0.7655
0.7556
AdamAR- Bucket 1-3 & AdamW- B-W-kk-BBt444 4-5
0.7510
0.7646
0.7657
0.7614
AdamAR Bucket & AdamW- 5 5
0.7470
0.7656
0.7660
0.7635
AdamAR
0.7549
0.7728
0.7730
0.7725
5 CONCLUSION
We propose an adaptive regularization method to address the one-epoch problem in estimation mod- els for the ASR domain. Experimental results demonstrate that our approach effectively mitigates the one-epoch issue and improves estimation performance. Additionally, we provide theoretical explanation for the one-epoch phenomenon and illustrate how the proposed method takes effect through analytical derivations and experiments. This approach has already been fully deployed in the production environment of sponsored search our company.
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (relu).
arXiv preprint
1rXiv:1803.08375. 2018.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexitie: Risk bounds and structural results. Journal of machine learning research, 3(Nov):463 482, 2002.
Stephen Boyd and Lieven Vandenherghe. Convex optimization. Cambridge university press, 2004.
10

---

Published as a conference paper at ICLR 2026
Heng-Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Frank H Clarke. Generalized gradients and applications. Trunsactions of the American Mathemati- cal Society, 205:247-262, 1975.
Emmanuele DiBenedetto. Real analysis. Springer, 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang, Shuang Li, and Kun Gai. Multi-epoch learning with data augmentation for deep click-through rate prediction. arXiv preprint arXiv:2407.0000 2024.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXin:2010.01412, 2020.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pp. 297-299. PMLR, 2018.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang. Wei Zhang, Xin Jiang, al. Wukong: 100 million large-scale chinese cross-modal pre-training benchmark. Advances in Neural Information Processing Systems, 35:26418 264431,,,,,, 2022.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization- machine based neural network for ctr prediction. arXiv preprint arXiv: 7707.042411 2017.
Xingzhuo Guo, Junwei Pan, Ximei Wang, Baixu Chen, Jie Jiang, and Mingsheng Long. On the embedding collapse when scaling up recommendation models. arXiv preprint arXiv:2310.04400, 2023.
Stephen Hanson and Lorien Pratt. Comparing biases for minimal network construction with back- propagation. Advances neural information processing systems, 1, 1988.
Meixuan He, Yuqing Liang, Jinlan Liu, and Dongpo Xu. Convergence of adam for non-convex objectives: Relaxed hyperparameters and non-ergodic case. arXiv preprint arX1v.230..11782, 2023.
Biye Jiang, Chao Deng, Huimin Yi, Zelin Hu, Guorui Zhou. Yang Zheng, Sui Huang, Xinyang Guo, Dongyue Wang, Yue Song, et al. Xdl: an industrial deep learning framework for high- dimensional sparse data. In Proceedings of the 1st International Workshop on Deep Learning Practice for High- Diiigh-DDmnsinnall Sparse Data, pp. 1-9, 2019.
Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329 2020.
Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. The Annals of Statistics, 30(1):1-50, 2002.
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1754-1763. 2018
Zhaocheng Liu. Zhongxiang Fan, Jian Liang, Dongying Kong, and Han Li. Multi-epoch learning for deep click-through rate prediction models. arXiv preprint arXiv:2305. 19531, 2023.
11

---

Published as a conference paper at ICLR 2026
Ily Loshchilov and Frank Hutter. arXiv:1711.0S101, 2017.
Decoupled weight decay regularization. arXiv preprint
Robert Moore and John DeNero. LI and 12 regularization for multiclass hinge loss models. In MLSLP, pp. 1-5, 2011.
Yurii Nesterov. Introductory lectures on convex optimization: basic course, volume 87. Springer Science & Business Media, 2013.
Efc A Ok. Real analysis with economic applications. Princeton University Press, 2011.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Athanasios Papoulis. Random variables and stochastic processes. McGraw Hill, 1965.
Mark Schmidt, Glenn Fung, and Romer Rosales. Optimization methods for I1-regularization. Uni- versity of British Columbia, Technical Report TR-2009-19, 2009.
Jeremy F Shapiro. survey of lagrangean techniques for discrete optimization. In Annals of Discrete Mathematics, volume 5, pp. 113-138. Elsevier, 1979.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research, 15()):1929-1988 2014.
Chunqi Wang, Bingchao Wu, Zheng Chen, Lei Shen, Bing Wang, and Xiaoyi Zeng. Scaling trans- formers for discriminative recommendation via generative pretraining. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining 2. pp. 2893-2903, 2025.
Minhui Xie, Kai Ren, Youyou Lu, Guangxu Yang. Qingxing Xu, Bihai Wu, Jiazhen Lin, Hongbo Ao, Wanhong Xu, and Jiwu Shu. Kraken: memory-efficient continual learning for large-scale real- time recommendations. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-17. IEEE, 2020.
Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, et al. Wukong: Towards scaling law for large-scale recommendation. arXiv preprint arXiv:2403..m 02545, 2024.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 161.0350 2016.
Yuanxing Zhang, Langshi Chen, Siran Yang, Man Yuan, Huimin Yi, Jie Zhang, Jiamang Wang, Jianbo Dong, Yunlong Xu, Yue Song, et al. Picasso: Unleashing the potential of gpu-centric training for wide-and-den recommender systems. 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 3453-3466, IEEE, 2022a.
Zhao-Yu Yu Zhang, Xiang-Rona Sheng, Yujing Zhang, Biye Jiang, Shuguang Han, Hongbo Deng, and Bo Zheng. Towards understanding the overfitting phenomenon deep click-through rate models. In Proceedings of the 31 ACM international conference information knowledge management, pp. 2671 12800, 2022b.
Jieming Zhu, Quanyu Dai, Liangcai Su, Rong Ma. Jinyang Liu, Guohao Cai, Xi Xiao, and Rui Zhang, Bars: Towards open benchmarking for recommender systems. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2912-2923, 2022.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A GENAI USAGE DISCLOSURE
During the preparation of this work, the author used ChatGPT to improve the language. After using this tool, the author reviewed and edited the content necessary and takes full responsibility for the final publication.
B MINIMUM CONVERGENCE ANALYSIS
B. PROOF OF MINIMUM CONVERGENCE
Based on the assumptions outlined in section 3.3, and following the analytical framework of He al. (2023), we apply the descent lemma (Nesterov (2013)) to derive
Lo
S (0k+t) 5 (0*) + (VS(0*), 0 000(0*)),++1000 -0")
110ht1
2
@ f(0%) NL (VS(O)).
m
+ tgion
Tong
m
1/00
+
Vot +e
Virk +
(16)
5((0)) 9M (VS(O)).
m
Loni
m k
Vo + LonE
2 m
v 2/ + E
7h (VS(0*). X5Oh) 2
+ Lovi
| ||001/
V +
and 7) be specified either constant according to schedule. Let AØ= min(1, /////////
where 776
then construct X ? so that (a) is satisfied. Based on proposition 2 and Lemma 16 in He et al. (2023), we can derive E |||0*| < W, 2 1 by mathematical induction, where W R a constant. Furthermore, according to appendix B.2, E [[[[ + 1l] < | 122 and m7 is lower bounded by Tmins
derive that
E
|
[I] < (1) + v2) /Tmmin
(17)
Let a (I + v2) Mmin = vo. and applying the Cauchy- Schwarz inequality yields
E [(Vf(0*). X/ N [IV
WMro
(18)
Then we invoke the equation 23 in He al. (2023), under the conditions 0 < B1 and <
B2 <
- 81
[f [0*+))] < E [f (0t)]
MEE
|V (o*)
(0*)
2 +E
B1 M2
VPM1
& m Σ
s (2)
(19)
E
53/2
M
Wo.WTH Th + Loug
Th
Upon rearranging the above equation and summing both sides over & from 1 to T, we have
1 B1
(0+) < ss(0)) 1 B
mh
E
[VS(())))
1
M/=1
2
+E
k=1
(20)
VPM
K
M Too
K Σ
K
K
g//2
1
g
B2) +
ni + MR
W Σ
mk + 2
Σ
2
1=1
1==
Am
13

---

Published as a conference paper at ICLR 2026
Suppose that {y h221 is non-inereasing real sequence. Assume that there exist positive constants Co and Co such that Coru I ny s Co By applying the equation 25 and 26 in He et al. (2023), we obtain
V +E k=1
1-
Σ
TEE
|
||
= ( - f LD
$ c 31)
K Σ
T
k=1
(21)
CoVPM -2)
K
2 Lo
K
Σ
+
Σ
th
+ Mr W Σ
The + Lovi W22
Σ
2
3/2C6(1 - B1) k=1
1=1
k=1
k=1
multiplying both sides of the above equation by
M 1-3 B1
we finally obtain
K
K
V
Σ
7k
C3
(22)
where C1, C2 and C3 in equation 22 are given by
C 2 + E (f(()) )
1- 31
C1 B2) P(M2 +8) MuoW V
/M² +
C2 =
(23)
c - -11
1- B1
2 Lov M M2 +
Lo M +E c(1 -31)
Long W2
M2 +
C3 E c0 -31)²
31
because of
K
min IskSK
VJ (8*) ||"]
Σ 7E S Σ T E. (0*)
[ [|a]
(24)
h=1
we can derive that
E Ci C (0)))
Ak + C3 Σ K
k= 1
min ISRSK
(25)
2k=1 1/
It can be observed that the incorporation of adaptive regularization modifies only the constant term in the upper bound of the minimum convergence rate, while leaving the inherent convergence prop- erties of the Adam algorithm unaffected.
B.2 BOUND OF OCCURRENCE INTERVAL. NOISE
We assume the occurrence interval 1 is subject to additive random noise 8² and it holds that E I + 21, where E [8*] Mı. I e RR" is finite constants vector. Let I I + 8F. Since 1 O, the random noise 8* has lower bound of -I. We rewrite the vector form of equation 11
as
A min (1, a (I + +))) a(I min , 1/a-1a/11))) I))
(26)
Let 8hh min (8*, 1/a I). then it is straightforward to show that the actual occurrence interval noise is bounded, i.e., 8¹ E [--,, 1, 1/a ПI. So we can see that the actual noise which impact * has been bounded, i.e., E [1]8*113 < 2i and 2y finite constant.
C NON-SMOOTH CONDITION DISCUSSION
We analyze the non-smooth condition under the assumption that the function p(TTt) is locally Lips- chitz continuous. Based on the Clarke subdifferential (Clarke (1975)) and the generalized stationar- ity condition for the optimal solution of the inner problem, obtain
Nr a0(TT)
(27)
14

---

Published as a conference paper at ICLR 2026
where 0Ø(TT) denotes the set of subgradients at r". Based generalized KKT framework, there exists Pij E ào(T") such that
MajPij + Mm Assuming that 0Ø(7*) n {ol majp μn 0} * Ø, we can select consistent subgradient from 0o(r*) such that Pij = - -X Substituting this choice into equation 28 yields Nig Maj- Therefore, the necessary condition remains valid in the non-smooth case.
(28)
D RADEMACHER COMPLEXITY BOUND OF FM MODEL
We analyze the Rademacher complexity of bias-free factorization machine (FM) model. Let fFM E F be defined as
8
(t)
= Σ w e,(1) + + Σ i=1 j=1i=+11
e;(1)
e()
(29)
where F denotes the class of real-valued FM like functions, Let w; RR, Vi denote the linear weight vector and let et(t) be the embedding vector for feature i at t. We decompose the Rademacher complexity into linear part (TFi and interaction part (F,),, where Ji define the linear function class and Fg define the interaction function class. Therefore, the overall upper bound of (F) satisfy
RT(F) S RT(FF) (Fa For the linear part, the upper bound of Rx (Fi can be obtained as
(30)
T
8
(F)
w
Σ
(0)()))
w Me.
(31)
According to Bartlett & Mendelson (2002), the upper bound of Rr is given by
RT(F,)
Σ
Σ M
Mr,Mr,
(32)
VT
Finally, we obtain the upper bound the Rademacher complexity for the FM-like function class as follows
$
S
(F) <
Σ
nu ME,
ME. ME,
(33)
j=++1
From equation 33, observe that the embedding norm plays the most important role in determining the upper bound the Rademacher complexity and ultimately affects the generalization error.
E ANALYSIS OF EMBEDDING SIZE
We examine the impact of varying embedding sizes the xDeepFMM backbone using the iPin You dataset. The embedding size ranged from 16 128, and the multi-epoch performance is compared across five methods. As shown table 4, our method performs consistently well across all embed- ding sizes. Furthermore, we examine the singular spectrum of the embedding matrix for the "IP" feature. As shown in figure 4(a), training with the basic Adam optimizer leads rapid increase in the sum of singular values (SS) across multiple epochs. In contrast, figure 4(b) demonstrates that the information abundance (IA), as defined in Guo et al. (2023), decreases correspondingly. This inverse relationship reveals the occurrence of an embedding-collaps phenomenon. Notably, our method effectively controls SS growth and enhances IA, which may explain its superior perfor- mance in mitigating overfitting.
F COMPUTAIONT AND MEMORY ANALYSIS
We compare the computation and memory costs of AdamAR and Adam, excluding gradient com- putation. P is number of parameters. The total costs summarized in the table below.
15

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of test AUC with xDeepFM backbone and iPin You dataset using Adam opti- mizer across different embedding sizes. The best results are highlighted in bold.
Embedding
Embeddiing Size
Embedding
Embeiding
Mothod
133
Adam
0.7593
0.7373
0.6914
0.6980
0.7607
0.7379
0.6928
0.6831
0.7579
0.7396
0.6867
0.7630
0.7430
0,7076
0.6315
MEDA
0.7593
0,7003
0.7548
0.7628
0.7607
0.7592
0.7544
0.7595
0.7579
0.7596
0.7624
0.7994
0.7630
0.7500
0.7633
0.7627
SAM
0.7602
0,7567
0.7405
0.7396
0.7644
0.7514
0.7401
0.7265
0.7582
0.7516
0.735
0.7197
0.7680
0.7536
0.7994
0.7200
Adsmn
0.7622
0,7683
0.7653
0.7632
0.7611
0.7655
0,7652
0.7588
0.7615
0.7633
0.7638
0.7274
0.7701
0.7891
0.7091
0.3475
AdamAR
0,7688
0.7757
0.7710
0.7700
0.7744
0.7732
0,7708
0.7081
0.7608
0,7733
0.7698
0.7653
0.7770
0,7719
0,7721
0,7646
2000
20.0
Adam
Adaas
17.5
MEDDA
Embeoding $
1500
SAM
SAM
AduMNW
IP Embesding
15.0
AdsmmAt
AdaanAR
12.5
10.0
500-
7.5
5.0
0-
Epochs
Epocls
(a) Sum of singular values
(b) Information Abundance
Figure 4: Singular spectrum analysis of "IP" feature embedding on the iPin dataset. (a) shows the sum of singular values. (b) presents the information abundance.
G ADAGRAD WITH ADAPTIVE REGULARIZATION
Adagrad, like Adam, is widely used in the ASR domain. Our method extends readily to Adagrad, as detailed in Algorithm 2.
Algorithm Adagrad with Adaptive Regularization (AdagradAR)
1: given 10-5 a, learning !
2: initialize time step t + 0, parameter 0:=0, squared gradient accumulator o/ +- 0, last update step state + <- 0
3: repeat
4:
1+1
5:
S Vogf (0t-1)
6:
+ v
1- + (g/)
7:
A t min (1, op
1-1 a)
8:
if |ل
0 else 1
9:
0 0 0 n. s (o [op+ +5)
10:
until stopping criterion is met
return optimized parameters p
H NETWORK ARCHITECTURE CONF GURATIONS
We adjust the network architectures based on the feature and sample sizes of each dataset. As summarized in table 6, larger and deeper networks utilized for datasets with features and samples, such as iPin You, Avazu, and LZD. contrast, smaller and shallower networks are adopted for the Amazon dataset mitigate the risk of overfitting.
16

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of per-iteration computation and memory between Adam and AdamAR.
Optimizer
Mul. / iter
Add. / iter
Memory cost
Adam
9P
3P
3P
AdamAR
11P
se 5P
4P
Table 6: Network architecture configurations for different datasets
DDepPFM
WaKong
Dataset
ML. Hiddons
MLP Hiddens
CIN Hiddens
MLP Hiddens
Layers
LCBRFMB Eimhs
FMB Hiddens
MLP Hiddens
You
[512, 256, 128]
[512, 256, 128)
[16, 16, 16]
[512, 256, 128]
[64, 321
[512, 256, 128)
Amazon
[512,
[812,
[8,
[512,
Avazu
[512, 256, 128)
[512, 256, 128)
[16, 16, 16]
256,, 128]
[64, 321
[312, 256, 128)
LZD
[512, 256, 128]
[512, 256, 128)
[16, 16, 16]
[512, 256, 128]
12
[64. 321
[512, 256. 128)
I
GRID SEARCH FOR THE WEIGHT DECAY COFFFCIENT
The weight decay coefficient a crucial hyperparameter that can significantly impact model perfor- mance. In our adaptive regularization method, this coefficient is denoted by in equation 11. To ensure fair comparisons across different methods, we perform grid search to identify the optimal weight decay value for each dataset. Figures 5, 6, 7, and illustrate the effect of varying the weight decay coefficient on test AUC across datasets using DNN backbone at the end of epoch 2. Al- though test AUC varies across weight decay coefficient settings, proposed method consistently surpasses the performance of AdamW and AdagradW. Moreover, whereas AdamW and AdagradW exhibit high sensitivity the weight decay coefficient, method maintains stable performance, thereby reducing the complexity of hyperparameter tuming in practice. Notably, the Avazu and LZD datasets, which contain larger number of sparse features and samples, require smaller optimal weight decay coefficient than the iPin You and Amazon datasets. This observation offers practical guidance for coefficient selection in real-world industrial scenarios.
0.78
0.78
0.7728
0:7706
0.77
#-2614
0.7805
0.78
Test AUC
0.76
Test AUC
0.75
0.74
0.74
0.73
Adami AdamAR
0.72
AdaggaddW AdagradAR
0.72
10
10-5
10-4
10-31
10-7
10-1
10
10-5
10
10-3
10-2
10-1
Weight Decay
Weight Decay
(a) iPin You dataset Adam
(b) iPin You dataset with Adagrad
Figure 5: Performance of different weight decay coefficient on iPin You dataset with DNN backbone at the end of epoch 2. (a) shows the performance with Adam. (b) shows the performance with Adagrad.
J DETAILED EXPERIMENTAL RESULTS
Here present detailed experimental results with estimated standard deviation in section 4.3.
17

---

Published as a conference paper at ICLR 2026
0.88
0.88
01,0009
0,8635
0.85
0.8554
0.86
Test AUC
0.84
0.84
Test.
0.82
0.82
0.80
AduntW AdamAR
AdagradW
0.80
AdagradAR
0.78
10
10-h
10-4
10-3
10-2
10-1
10-5
10-3
10-1
10-5
10-2
Weight Decay
Wright Devay
(a) Amazon dataset with Adam
(b) Amazon dataset with Adagrad
Figure 6: Performance of different weight decay coefficient Amazon dataset with DNN backbone at the end of epoch (a) shows the performance with Adam. (b) shows the performance with Adagrad.
0.77
0,7640
01.7229
0.76
0.750
0.70
0.7586
Test AUC
Test AUC
0.75
0.75
0.74
AdagradW
0.73
Adaan AdaanAl
0.73
AdlaradAR
0.72
0.72
10
10
10-6
10
10
Weight Decay
Weight Deray
(a) Avazu dataset with Adam
(h) Avazu dataset with Adagrad
Figure 7: Performance of different weight decay coefficient on Avazu dataset with DNN backbone the end of epoch 2. (a) shows the performance with Adam. (b) shows the performance with Adagrad.
K FEATURE STATISTICS ON YOU DATASET
The iPin You dataset contains total of 16 features, among which only few sparse. Table lists the top six sparse features along with their statistical indicators.
L DATASET DETAILS
You. The training dataset includes processed bidding, impression, click, and conversion logs from PPinYou DSP. It contains 19.5 million records and 16 categorical features, providing a suitable context for exemplifying the overfitting phenomenon.
Amazon. This is a widely used dataset from Amazon for evaluating CTR estimation models. In study, we use the electronics category of the Amazon dataset, which contains approximately million records and 3 categorical features.
Avazu. This dataset comprises approximately 10 days of labeled click-through data from mobile advertisements, consisting 40 million records and 22 categorical features spanning both user attributes and advertisement characteristics.
18

---

Published as a conference paper at ICLR 2026
0.74
0.74
0.7256
0.7269
0.72
0:71:32
0.72
0:73
Test AUC
Test. ATC
0.70
0.70
0.68
0.88
AdaanW AdamAR
ArlagndW
0.08
AdagraiAR
10
10
10-4
10-3
10-2
10-1
10-9
10-0
10-1
10-3
10
10-1
Weight Decay
Wright Devay
(a) LZD dataset with Adam
(b) LZD dataset with Adagrad
Figure 8: Performance of different weight decay coefficient LZD dataset with DNN backbone at the end of epoch 2. (a) shows the performance with Adam. (b) shows the performance with Adagrad.
Table 7: For the top six sparse features in the You dataset, the table below presents the number of unique IDs, the average occurrences of each ID, and the mean update interval for each ID with batch size of 2048.
Feature
IP
slotid
domain
city
creative
useragent
Unique IDs
704,966
180,696
51,322
372
133
42
Mean Occurrences
27.7
107.9
379.9
52,408.5
146,586.3
464, 189.9
Mean Update Intervals
344.222
88.230
25.060
0.182
0.065
0.021
LZD. The dataset is sampled from our production environment and consists of real-time bidding logs from sponsored search. The dataset contains 25 million records and 13 categorical features.
19

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of average test AUC with DNN and WDL models using Adam optimizer.
DNN
WDL
Dataset
Method
Bi
F2
E3
E4
El
B2
E3
E4
Adam
0.7616
0.700l
0.7320 0.7619450000 noes
0:6087 0.7028 .0.0
MEDA
(.7555 1o90m
0.768119000
ioces
0.7589 0.7619.00mm
0.7551 pa427
iPin
SAM
0.7510 0.7593
0.7
a.7256.ucoes
0.7581 0.7610a000t
0.7 M.
0.7208
AdamW
0.7 0.7477snn00s
0.7592
0.7623
0. 7.7688aco61
0.
0. 0.7656 0.761
0. 0.7666429036
0. 0.76332000019
AdamAR
0.7724.nsee
0.7655..wmm
0.7729 aq.mas
0.7670.nnase
0.7668 a.001
Adam
0.8482 1000
0.85835.000m
o.8
0.8474400ms
0.8261 0005
120000
0.8156
MEDA
(.8812.mm0rr 0.85:111200m
0.85366
0.85733nc00
0.8506 0.81711oons
0.8366.12400
0.8566
Amazon
SAM
am8507 0.8587
0.8117.aumn
0.82192ucezz
0.8516
0.8567 S1L00R:
0.8396220000
0.8213
AdamW
0.8476..0.0e
0.8426 0.8571
0018
0.8276.a.c00
0,85333 0.8461
0015
0.8380
0.82
AdamAR
0,8507 0.8683 insess
0.8708
0.8686.
0.8654 0.8496...007
0.8689
0.8659
Adam
0.
0.7211 pu0392
0.6853.pcom
0.7483.500
0.7221 vo00e
0.6982 100309
0,68866 o000
MEDA
0.7
119000
0.7d89..c.mm
0.7485.ncom2
0,7488 L.7183
0.7480.2axx
1.719
Avazu
SAM
0.7451g0.00 0.7194-b02s
0.7013-0.00t
0.68992ac000
0. 0.7205 0.7477=3001
0 0.6093
0.690220.003
Adamm
0.7572.00011
0.7582 0.75824b0031
0006
0.7
0.7581 0.758522,005
0014
0.7563
0.7570.00011
AdamAR
0,7617 .ux01
0.7e31.nnase
.auwns
0.7629 .ueee
0.7629, sus902
0,7629
auxee
0.7627-nva1z
0.7626 auwmas
Adam
0.7118 oocc
0.6613
0,6252 (0001
0.6065
0.6726 0.7155
0,6308
0.60/99.oms
MEDA
0.7118en
0.7106=0000s
0.708129:003
0.71 0.7176saco0p
0.7155g30015
0.7162gntmet
0.7177=00M7
0.7162
LZD
0.7130ga.oc Dres
0.6696.202224
0.634
0.615520c007
0.7061
0.6111 0.6380424084
Adamm
0.7132
0.7135
0.
0.7139..
0.7143..0001
0.7138
0.7131 0.7142.22000
AdamAR
0.7229 s0
0.7235:n081
U.7241=a5om
0.7234.a1001
0.7233
0.7240
0.7246.n0953
0.7240
Table 9: Comparison of average test AUC with xDeepFM and WuKong models using Adam opti-
mizer.
xDeepIMM
WuKong
Dataset
Method
Adam
0.7380
0.7391 Liv00s7
0.6969 (owws
0.681L
0.76
0.7412
0.7082.1x020
0.6915..awc
MEDA
0.7590
0,75842pa0
0,7575
0.7597
0.7663 0.7611
0,7662.2w9
0.7706
You
0.7565...com non
0.7517400019
0.74
0.72744ocon
0.7033
0.7485
0.7306
AdamW
0.7607 to
2.7000
0.7651.0.0m
0.7574.o
0.7579-.001
0.7634 0006
0.7511.0.0w 0.7805
AdamAR
0.7725
0.7733
rueses
0,7711
0.7678 asess
0.7653
0.7736 azezz
0.7748
0.7736
Adam 0.8160.nmmm
0.8535_10009
0,8287
100000
0.8163.ncess
0.85801oomms
0.8600
0.8318.1saar
0.8232
Amazon
MEDA
0.8460.0.00
0.651945000
0.8551
0.856240c004
0.8580-2.005
0.861
0001
0.8621
0.8626
SAM
0.8005.a.000
0.8587_200317
000000000000
0.8212,0co1s
0,8588 .001
018639
0.84044200e
0.82
Adamm
0.8446.pmm
0.8557.1209e
0.8881.0004
0.8240..com
0.8364..007
0.8478.20557
0.8349.
AdamAR
0. 0.8483
0.8676100059
0.8687230015
0,8675.0001
0.8682-2000
0.8696
0.8693
0.8664_3001
Adam
0.7 188
0.7217=000s
0.701920.002
0.686940:046
0.7514420007
0.7360sa
0.7141=20126
0.707920000
MIIDA
0.7 0.7488 noon
0.74884002ss
0.7505
0.75062aco16
0.7514g50097
0.7548 n014
0.7553420x4
0.757 g00013
Avazu
0.7484.00017
0.719044000s
0.7010...003
0.69012.cco77
0.7513-p0m
0.7333 ooc0
0.7155.2a3as
0.7156.
AdamW
0.75883anous
0.75822300x6
0.7588)a0mm
0.73833anceu
0.754223022
0.75472xum0
0.7358220xx
0.756420
AdamAR
0.7628
0.7633-na006
0.7638-4008
0.7636241066
0.7624-0om
0.7612:90005
0.7623-nam4
0.7624
0.7 0.7164.n0017
0.6797400225
0.6321 g0.000
0.605044ac00
0.7101
0.6845gn.012
0.6102-30004
0.604440m033
MEDA
0.716440.007
0.71704200ss
0.7152
0.71394accco
0.7101
0.7170
0.7129
0.7128
LZD
SAM
0.7166..
0.63444...om 0.67504
0.6134..ccmm
(.6713. 0.7146420ux
0.64433.200e
0.6212.0.000
Adam
0.7 14242mo
0.7151
g.7130gumm
0.7 17130gnco0e
0.71 0.71 15g20222
0.7 185
0.7 0.715222asus
0.7142gamz2
AdamAR
0.7244-2.000
0.7256.0a943
0.7242
0.7238
0.7227
0.7215
0.7208
0.7202
20

---

Published as a conference paper at ICLR 2026
Table 10: Comparison of average test AUC with DNN and WDL models using Adagrad optimizer.
DNN
WDL
Dataset
Method
Ei
E3
Adagrad
0. 0.7593
0.6231 0.6307
0.6099520.044
0.7816
0.6653e0
0.682lancces
0.6211
MEDA
0. 007502
0.7710 0. g.70
0.764 0.7729
0.768 go
D. 0.7885
0.7715
iPin You
SAM
0.7576
0.7187 0.708l
2.7308
0.7076
0.7369
AdagradW
0. 0.7651 0.7558
0.7667
0.7595
0. 0.750
0.7675 0000
0. 0.7635
0. 0.7593.
AdagradAR
209954
0.7754
0.7760
0.7723 0.7744
0.7772_au000
0.7748
0.7720
Adagrad
0.8438
0.8402atumus
0880422
0.8406_co001
0.834520002
0.80852n0044
0.798220000
MEDA
0.8 inc000
0.8195 0.8081
0.85051naecs
0.8/206..com .c00t1
0.8170
0.8 1ncons
Amazon
SAM
0.8500-0007 0.8361 0.8527
CoGG.
0.81534asc15
0.8490.
0.8325
0.8155200017
AdagradW
0 118820
0.8578acam
0.8533 110
0
2,8003
D
0.8196
AdugradAR
0.8479.0cc0
0.8659.eama
0.8711
0.8712
0.84533..0000
0.8631 ax00
0.8687
0.8707
Adugrad
0. Joc0ss
c0.066
0.7164. cono
0.7074:0scm
0.7543..oom
0.7305 o000
0.7073.
MEDA
0.7541
0. 0.7549
0.
0.7545g00806
0.7548__000s
0.7047200m2
D. 0.76402n:00o
2.7007
Avazu
SAM
0.7333.c.00
0.7199
0.7357 1.7
1.7
0.7178
0.7079.0003
AdagmddW
0. 0.7578
0.7575 0.7580
0.75662nac16
0.7584_60000
0.7581
0.7567
0.7585g0000
AdagradAR
0,7628 suuses
0.7629:0amm
0.7630
0.7624.avess
0.7631
0.7634
0,7623
0.7626
Adagrad
0.7226
0.0058aamms
0.6433
0.63762nxxs
0.724420ww7
0.0005:21m0
0 anc077
0.63862100:
MEDA
0.7150 0.7183.c.0m5
0.7296.
0.7244-p007
0.7239.orom 0,7241.0006
0.7219.
LZD
SAM
0.721 3anveses
1.6761gause
0.6446gnmis
0.622220.x55
0.7220
0.6830g0mms
0.648 Enncess
0. 0.6206200025
AdagradW
0.71 stueser
0.7132 c0.00
0.7135 mo
0.7154
0.7138 .c0022
0.7149
in00n2
0.7145
AdagmdARR
0.7253-aa021
0.7247
0.7234_a
0.7241
0.7269:eamm
0.7258_a.m03
0.7251 =4003
0.7257.00m
Table I1: Comparison of average test AUC with xDeepFM and WuKong models using Adagrad
optimizer.
DDeepEM
WaKong
Dataset
Method
EI
E2
E3
E4
E1
E2
E3
E4
Adagrad
Aaseoss
0.68-B3auss
0.652722xx0s
0.7661 0.6900
0.6-977nnoes
0.6320gav220
YoinYon
MEDA
0.7728 0.7722.c.o0z 0.767440c0s
0014
ooro
0.7740.
0.7705 0.7861-po0m mco
0.7729.0.0m3
0.7696.
SAM
0.7637ane
0.7061 20.004
0.7490
0.73632nxxa7
0. 0.7400
D. 0.7625 0.767820m0
0.7427
Adagradd
0. (o956
0.7673
0,7686 mons
0.7652 10.00s
0.7578
0.7661 1017
0. 0.7619
ioren
0.76000 pooss:
AdagradAR
0.7760-am0ss 0.776828002
0.7762-a0019
0.7745
0.7718
0.7776_ax027
0.7782
0.7774_00229
Adagrad
0.8406400004
0.8374g0.000
0.8103
0. 0.780040.044
0.840 180001200022
0.8472.0
0.81562o.000
0.8046220033
MEDA
0.84062nues4 0.8463
0.8476
0.85002n.0000
0.8:91 0.80991-com22
0. 0.80002mmm
0.8587
0. 0.86092m0000
Amuzon
SAM
0,8483 nases
0.8521 10.05
0.8325
0.81
0.8556 se.022
0.8567.0
0.8356 nets
AdagradW
0.8410anesss
0,8006
0.86222n000
0.85082n.c34
0.8513
0. 0.8630gamms
D. 0.8617
0 0.86002100m
AdagradAR
euoses
0.8681 axser
0,8703 ru0004
0.8538 .o00to
0.8690
aumns
aaess
0,8700, nowm
Adagrad
i.7550.nvsss 0.7160 0.7319
0.75 0.7069
0.7311
n.7160
0.7011.nom
MEDA
0.
0.7550400082 0.7550 0.7508
0.7548-p.00ss 0.75532nxc08
0.7551
10 0006
D. 0.7556
0.7584-20015
SAM
0.7561 0.7332
0.71982mmes
0.7094
0.7358_camm
0.
0.735820m00
Lanoer
0.7110gpom
AdagradW
(.758 0.7581
0,7580).o
0.7524 0.75
0.7555
0. 0.7555.ocom coce
0.75
AdagradAR
0.7636_0890
0.7633:80m3
0,7635
0.7633-asee
0.7628:sa99
0,7626_ax00
0.7627
0.7620
Adagrad
0.72224ncw0 0.0676guu022
0.6400_a c04
D.633244ux31
0.7247=00947
0.64124n.0es 0.6726
0.6330
MEDA
0.7222,new01 0,7258.mvezzz 0.7237
0.72561nxcoe
0.7217
0,7237
0. 0.723L
0.7239.po91
SAM
0.7229+04043
0.6480)aa 0.685442.002 coc.
0.62684n.c4
0.7248-000i
0.6802 mo0
0.647940.0144
0.62894p00
AdagradW
0.7155aneesss
0.7135acam4
0.7117
0.7150auscor
0.7127
0.7111
0.7147ansees
0.7136230012
AdagradAR
0.7273 iusees
0.7268 eaams
0,7264. amms
0.7263
0.7269
0.7260 a.mo
0.7252 aseen
0.7239. nus22
21


Reviews:
[{"review_id": "QxPzbpNaj0", "reviewer": "Reviewer_amMd", "summary": "This paper analyzes the one-epoch overfitting phenomenon in large-scale CTR/CVR models with sparse embeddings. Using Rademacher complexity, the authors show that overfitting arises from unconstrained embedding norm growth for infrequent features. They propose an adaptive regularization method that dynamically adjusts regularization strength by feature update intervals, forming optimizers. Experiments on public and industrial datasets demonstrate consistent performance gains, improved generalization, and easy integration into existing training pipelines.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results.\n\n1. Sound and well-motivated analysis linking embedding sparsity and overfitting.\n2. Adaptive rule is intuitive and easy to implement.\n3. Multi-dataset, multi-optimizer validation with consistent gains.\n4. Addresses a critical and under-theorized issue in real-world recommender systems.\n5.Provides a clear explanation for why rare features require stronger regularization.", "weaknesses": "1. The analysis assumes i.i.d. sampling and bounded feature norms, which may not hold in practice for long-tail industrial data.\n2. The derivation of adaptive coefficients relies on approximating feature frequency by occurrence intervals; this estimation may introduce stochastic noise not formally analyzed.\n3. The differentiability of \\phi(\\tau_{ij}) and the KKT-based derivation rest on idealized smoothness assumptions that may not hold in real deep networks.\n4. The paper doesn’t rigorously study how the adaptive coefficient \\lambda_{ij} affects optimizer dynamics or convergence stability, especially in non-convex regimes.\n5. A theoretical guarantee on convergence rate or bias induced by adaptive decay would strengthen the work.\n6. Tracking the “last valid update step” for every embedding vector may introduce memory overhead in billion-scale models. The paper does not discuss memory/time complexity or optimization strategies for this in production.\n7. While the baselines are relevant, comparisons with more modern adaptive regularization or implicit bias control methods  e.g., SAM, AdaNorm, or adaptive dropout  would enhance credibility.\n8. No statistical significance testing is reported for AUC gains, though the improvements are small (≈0.002–0.01).\n9. The paper briefly mentions online deployment but does not analyze potential trade-offs e.g., regularization lag under distribution shift, interpretability implications, or fairness across infrequent users/items.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lTuA0UZwIO", "reviewer": "Reviewer_s7gA", "summary": "The paper targets a common phenomenon in large-scale CTR/recommender models with highly sparse features: performance is often best after the first epoch, but further training leads to overfitting and degradation. The authors provide an explanation from the perspective of generalization bounds through Rademacher complexity as follows:\n\n* most of the model capacity is concentrated in the huge embedding matrix, and in particular, in the very infrequent IDs’ embedding rows. \n* the embedding rows with very infrequent IDs are updated on a much slower schedule than the dense part / MLP, so their norms can grow uncontrollably and thus inflate the overall capacity. \n\nBased on this, the authors propose an adaptive regularization scheme that assigns stronger weight decay to infrequent embedding rows . They integrate this row-wise adaptive weight decay into Adam/Adagrad (termed AdamAR/AdagradAR), so that the sparser and less frequently updated a row is, the stronger the regularization it receives. \n\nExperiments on multiple datasets and CTR architectures show that this approach can mitigate or even eliminate the “multi-epoch performance drop,” while slightly improving AUC.", "questions": "1. Does applying strong L2 to low-frequency rows in adamW reproduces the main gains? If so, that would validate the causal story more strongly.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Problem importance. The paper tackles an industry-recognized issue in large-scale CTR models—a core component of nearly all advertising systems.\n* Theoretical grounding. It links the empirical “train only one epoch” practice to a capacity-control perspective, giving the phenomenon a clearer theoretical justification.\n* Practical simplicity. The proposed technique is easy to integrate: it only requires tracking the last valid update step for each embedding row and can be directly plugged into existing optimizers.", "weaknesses": "1. The connection to existing “frequency-aware” or “epoch-level reset” approaches (e.g., MEDA-style methods) is not elaborated systematically.\n2. The experiments mainly report global metrics such as AUC/Logloss, but lack fine-grained bucket analyses (by feature frequency, long-tail IDs, cold-start features) to directly demonstrate that “low-frequency rows were actually controlled.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yGgiJiABQe", "reviewer": "Reviewer_fK43", "summary": "This paper investigates the prevalent phenomenon of \"one-epoch overfitting\" in large-scale sparse feature embedding models, especially pertinent to click-through rate and conversion rate estimation in online advertising and recommendation systems. The authors provide a theoretical analysis rooted in Rademacher complexity, pinpointing the growth of embedding norms as the root cause of deteriorating generalization when training persists beyond a single epoch. To address this, they propose an adaptive regularization technique that dynamically adjusts regularization strength per embedding based on occurrence intervals, integrating seamlessly with standard optimizers like Adam and Adagrad. Comprehensive experiments across public and industrial datasets demonstrate that the proposed method outperforms existing baselines in mitigating overfitting and promoting stable, improved performance during multi-epoch training.", "questions": "* How Sensitive is the Proposed Method to Hyperparameter $\\alpha$ Across Datasets?\nFigure 3 provides some evidence, but would the authors provide additional experimental data on extreme values of $\\alpha$, and practical guidelines for selection especially in large-scale real-world datasets?\n* Handling Never-seen or Extremely Rare Features:\nDoes the adaptive regularization approach systematically bias against tail classes? How does it perform in cold-start (new feature value) scenarios, and is there risk of underfitting for features with vanishing sample frequency? Can the method adapt dynamically if sparsity levels change after deployment?\n* Ablation on Update Interval Estimation:\nWould the authors run ablation studies to decouple the effect of interval-based regularization vs. embedding-based regularization overall?\n* Comparison with Other Adaptive/Hybrid Regularization Approaches:\nHow does the proposed method perform compared to thresholding or pruning-based strategies, as well as non-uniform L1/L2 variants found in the sparse modeling literature? Would Table 1/2 results meaningfully shift under such baselines?\n* Computational Overhead and Scalability:\nAre there notable compute or memory implications to tracking per-embedding update intervals, especially as feature cardinality scales beyond the reported datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Theoretical Insights: The paper offers a coherent theoretical framework using Rademacher complexity to explain one-epoch overfitting in sparse embeddings, clarifying a widely observed yet under-theorized phenomenon in deep CTR/CVR models (see Section 2.2 and 3).\n* Principled Adaptive Method: Instead of relying on heuristics, the proposed adaptive regularization derives directly from the theoretical analysis, offering an intuitive and practical approach that is compatible with common optimizers (Section 3.1, Algorithm 1).\n* Comprehensive Empirical Evaluation: The method is validated across a diverse set of datasets (iPinYou, Amazon, Avazu, proprietary LZD) and backbone architectures (DNN, WDL, DeepFM, WuKong), demonstrating strong and consistent improvements over multiple baselines (Tables 1, 2).", "weaknesses": "* Limited Positioning Relative to Broader Regularization Literature: While the paper covers baseline regularization approaches (L1, L2, weight decay), it does not sufficiently engage with recent or foundational work on sparsity-driven regularization at both the architectural and optimization levels. For instance, no discussion is provided relating their adaptive method to pruning/growth strategies or thresholding-based approaches, even though such techniques are central to the sparsity literature.\n* Lack of Ablations: There is little to no ablation study dissecting critical components of the proposed method, such as the sensitivity to the base regularization coefficient $\\alpha$ beyond Figure 3, or the explicit contribution of occurrence interval estimation. It remains unclear how much each modeling choice contributes to the performance gains.\n* Hyperparameter Selection Transparency: Although hyperparameters are said to be chosen by grid search (Section 4.1.3), the details are sequestered in the appendix and visualization (Figure 3), limiting transparency and reproducibility. The main text should provide a concise summary of grid search procedures and best practice recommendations.\n* Ambiguity in Some Notation and Algorithm Details: Certain notations, particularly around frequency and interval estimation for embeddings (Section 3.1), could be more precise. For instance, the paper uses $I_{ij}$ as both a stochastic occurrence interval and an empirical quantity with little justification for the estimation method used during optimizer steps.\n* Insufficient Theoretical Analysis of Method Limitations: The theoretical section does not address possible drawbacks or edge cases (such as whether the adaptive regularization could lead to underfitting for extremely rare or never-seen values), nor the impact of frequency misestimation on convergence or generalization.\n* Lack of Comparison to other LLM-based models: There are some new algorithms which are LLM-based recommendation algorithm, and this paper does not consider them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LZBSJ9SynP", "reviewer": "Reviewer_oGeC", "summary": "The authors study the one-epoch overfitting phenomenon in pCTR and pCVR models. They first conduct a theoretical analysis using Rademacher complexity, indicating that the norm of sparse embeddings leads to poor generalization. The authors propose an adaptive regularization method (AdamAR / AdagradAR)  with a dynamic strength of the embedding regularization terms. Experiments on several public datasets demonstrate the effectiveness of the proposed method.", "questions": "1. Can the complexity analysis be extended to explicit feature interaction models like FM? It would be great to include such analysis since most modern recommendation systems heavily rely on explicit interactions.\n\n2. How does the rewriting from Eq. 5 to Eq. 6 happen?\n\n3. I'm curious about the effect of the proposed regularization method on the dimensional robustness of the learned embeddings. There are several recent works discussing the dimensional collapse (DC) issue in recommendation models [1,2], and I wonder whether there is a connection between the one-epoch phenomenon and DC. Can you provide an analysis of the singular spectrum of sparse embeddings?\n\n[1]. On the embedding collapse when scaling up recommendation models. ICML 2024.\n\n[2]. Balancing Embedding Spectrum for Recommendation. 2025.\n\n[3]. Towards Mitigating Dimensional Collapse of Representations in Collaborative Filtering. WSDM 2024.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Motivation. The paper studies a well-known and critical problem in pCXR prediction - the \"one-epoch\" overfitting problem.\n\n2. Theoretical Grounding. The analysis links overfitting to Rademacher complexity, showing that embedding norm growth drives generalization error, giving a principled basis for adaptive regularization.\n\n3. Technical Novelty. The proposed method is well-derived and elegant, with a limited impact on the model training.", "weaknesses": "1. Extension to Explicit Interaction Models. The analysis is only in DNNs, whose effectiveness is limited. It would be great to extend the theoretical discussion to explicit interaction models such as FM or CrossNet in DCN V2.\n\n2. Experiments. Besides Wukong (BTW, the citation of the Wukong paper is wrong), other models in the experiment section are a bit out of date. It would be great to add experiments on the new SOTAs, e.g., DCN V2, xDeepFM.\n\n3. Writing. The abstract contains limited concrete content. It would be great to be a bit more specific on the proposed theoretical analysis and method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)