Adaptive Regularization for Large-Scale Sparse Feature Embedding Models

## Abstract

Abstract The one-epoch overfitting problem has drawn widespread attention, especially in CTR and CVR estimation models in search, advertising, and recommendation domains. These models which rely heavily on large-scale sparse categorical features, often suffer a significant decline in performance when trained for multiple epochs. Although recent studies have proposed heuristic solutions, they have not clearly identified the fundamental cause of this phenomenon. In this work, we provide a theoretical analysis that explains why overfitting occurs in models that use large-scale sparse categorical features. Based on this analysis, we propose an adaptive regularization method to address it. Our approach not only prevents the severe performance degradation observed during multi-epoch training, but also improves model performance within a single epoch. This method has already been deployed in online production systems.

## 1 Introduction

Click-through rate (CTR) and conversion rate (CVR) estimation are critical for advertising, search and recommendation (ASR) applications. E-commerce platforms like Amazon and Taobao rely on optimizing CTR and CVR estimation to boost gross merchandise volume (GMV), while advertising platforms at Google and Meta depend on it to drive revenue growth. In the past decade, as deep learning has been widely adopted in ASR applications, most estimation models have been built on deep learning frameworks and rely on large-scale, sparse categorical features (cheng2016wide; guo2017deepfm; lian2018xdeepfm). Recent work (zhang2022towards) demonstrates, through extensive experiments, that such models commonly suffer from the one-epoch overfitting phenomenon, where model performance drops sharply after the first epoch of training. It empirically suggests that the optimal practice is to train the models for only one epoch. At the same time, their empirical analysis also reveals a strong connection between feature sparsity and the one-epoch phenomenon. In ASR scenarios, categorical features can easily reach the scale of billions, and most feature values occur extremely infrequently (xie2020kraken; zhang2022picasso), which makes models especially susceptible to overfitting after the first epoch. Additionally, ouyang2022training reported a similar one-epoch phenomenon during supervised fine-tuning (SFT) of large language model (LLM), though they argue that moderate overfitting can actually be beneficial. We leave it in future work. In this paper, we focus on addressing the multi-epoch overfitting problem for models with large-scale sparse features in ASR applications.

liu2023multi was the first to address the one-epoch problem using a heuristic approach. They introduced a multi-epoch data augmentation method (MEDA) that can be easily applied to estimation models with large-scale sparse categorical features. Later, they extended it to a continual learning paradigm (fan2024multi). To mitigate overfitting during multi-epoch training, MEDA reinitializes the embeddings of categorical features and their corresponding optimizer states at the beginning of each epoch. Although this approach effectively alleviates multi-epoch overfitting, it is fundamentally heuristic because the embedding parameters are only reinitialized at epoch boundaries. It neither ensures optimal convergence nor explains the root cause of overfitting. Moreover, this embedding reinitialization approach may result in the loss of substantial information, potentially resulting in suboptimal model performance. Nevertheless, it provides important inspiration for subsequent research. Another method, proposed by wang2025scaling, adopts an LLM-style paradigm. In this approach, generative pretraining is used to obtain embeddings for categorical features, which remain frozen during subsequent training. This strategy also helps to mitigate overfitting. However, it requires significant training resources to obtain the pretrained embeddings, and the performance cannot be fairly compared to single-epoch training because the parameter budget used during the pretraining phase is not included. Therefore, it cannot be regarded as a solution to the one-epoch problem, although it does highlight the pivotal role of the embedding layer in this phenomenon.

There are several well-established and general-purpose methods to mitigate overfitting, such as dropout (srivastava2014dropout), $L_{1}$ and $L_{2}$ regularization (schmidt2009optimization; moore2011l1), and weight decay (hanson1988comparing). However, in real-world industrial applications, data is typically high-dimensional and sparse, with different features exhibiting varying degrees of sparsity. Consequently, methods like AdamW (loshchilov2017decoupled), which apply the same weight decay across all parameters, tend to be suboptimal for addressing overfitting. Specifically, such approaches may reduce the fitting accuracy for relatively dense features while failing to effectively control overfitting in sparse features.

This paper presents a foundational theoretical analysis of the root cause of one-epoch overfitting, and proposes a method that adaptively determines regularization coefficients according to the effective update interval of each categorical feature value, and integrates these coefficients into the optimizer’s update rules. The proposed approach is particularly well-suited for models with large-scale sparse features. It reduces regularization for relatively dense features and neural network parameters, while assigning appropriate regularization strengths to individual categorical features. As a result, this method not only prevents the sharp performance degradation observed in multi-epoch training, but also enhances performance within a single epoch.

## 2 Preliminary

In the industrial ASR domain, mainstream estimation models typically combine embedding layers with a variety of dense multi-layer perceptron (MLP) backbones, such as gu2022wukong. Taking an e-commerce platform estimation model as an example, these models process large-scale sparse inputs, including item IDs, brand IDs, and seller IDs, where the scale of categorical features can range from millions to billions. Before being processed by the estimation model, these sparse features are first mapped to low-dimensional dense representations via embedding layers. The one-hot categorical feature ${\bm{x}}_{i}(t)\in\mathbb{R}^{N_{i}}$ is defined as ${\bm{x}}_{i}(t)=\begin{bmatrix}0,0,\ldots,1,\ldots,0\end{bmatrix}^{\top},i\in[S],t\in[T]$, which can be represented by an embedding vector ${\bm{e}}_{i}\left(t\right)\in\mathbb{R}^{d_{i}}$ shown below

$$ ${\bm{e}}_{i}\left(t\right)={\bm{E}}_{i}^{\top}{\bm{x}}_{i}(t)$ (1) $$

where ${\bm{E}}_{i}\in\mathbb{R}^{N_{i}\times d_{i}}$ is the embedding matrix for feature $i$ and $d_{i}$ is the embedding size. $T\in\mathbb{N}$ is the number of training samples. $S\in\mathbb{N}$ denotes the number of categorical features. $N_{i}\in\mathbb{N},\forall i\in[S]$ denotes the number of distinct values for feature $i$. At each update step, an embedding lookup is performed to retrieve the dense representations for the sparse features via equation [1](https://arxiv.org/html/2511.06374v1#S2.E1), which are then fed into the MLP layers. For simplicity, we separate the model into the embedding component and the MLP component denoted as $f$. In our theoretical analysis, we use a basic DNN backbone and provide experiments in section [4](https://arxiv.org/html/2511.06374v1#S4) to demonstrate the generalizability of our method with other backbone architectures.

### 2.1 Neural Network Definition

The general form of basic DNN function can be defined as

$$ $f\left(\cdot\right)={\bm{W}}_{L}\sigma_{L-1}\left({\bm{W}}_{L-1}\sigma_{L-2}\left(\dots\sigma_{1}\left({\bm{W}}_{1}\cdot\right)\right)\right)$ (2) $$

where $L$ denotes the number of MLP layers, $\sigma(\cdot)$ is the ReLU function (agarap2018deep), ${\bm{W}}_{l}$ is the linear projection matrix of layer $l,l\in[L]$. In this paper, we only discuss the impact of embedding layer so we can define the $t$-th sample output of DNN as

$$ $y(t)=f\left(\begin{bmatrix}{\bm{E}}_{1}^{\top}{\bm{x}}_{1}(t);{\bm{E}}_{2}^{\top}{\bm{x}}_{2}(t);\dots;{\bm{E}}_{S}^{\top}{\bm{x}}_{S}(t)\end{bmatrix}\right)$ (3) $$

where $y(t)$ is the logit output for CTR or CVR estimation model.

### 2.2 Rademacher Complexity Bound

Rademacher complexity is a standard tool to control the uniform convergence of given classes of predictors (koltchinskii2002empirical,zhang2016understanding). Formally,
given a real-valued function class $\mathcal{H}$ with the input ${\bm{z}}$, we define the empirical Rademacher complexity $\widehat{\mathcal{R}}_{T}\left(\mathcal{H}\right)$ as

$$ $\widehat{\mathcal{R}}_{T}\left(\mathcal{H}\right)=\mathbb{E}_{\bm{\epsilon}}\left[\sup_{h\in\mathcal{H}}\frac{1}{T}\sum_{t=1}^{T}\epsilon_{t}\,h({\bm{z}}_{t})\right]$ (4) $$

where $\bm{\epsilon}=(\epsilon_{1},\dots,\epsilon_{T})$ is a vector whose entries $\epsilon_{t}$ are independent and uniformly distributed in $\{-1,+1\}$. Using standard arguments, such bounds, as long as the norm of ${\bm{z}}_{t}$ is bounded, can be converted to bounds on the generalization error, assuming access to a sample of $T$ i.i.d. training samples.

Based on Theorem 1 of golowich2018size, and noting that the embedding layer can be regarded as a linear projection, we can obtain the following upper bound on the Rademacher complexity of equation [3](https://arxiv.org/html/2511.06374v1#S2.E3) as

$$ $\widehat{\mathcal{R}}_{T}(\mathcal{H}_{L})\leq\frac{1}{T}\left(\prod_{l=1}^{L}M_{F}(l)\right)\left(\sqrt{\sum_{i=1}^{S}M^{2}_{E_{i}}}\right)\left(\sqrt{2\log(2)L}+1\right)\sqrt{\sum_{t=1}^{T}\sum_{i=1}^{S}\|{\bm{x}}_{i}(t)\|^{2}}$ (5) $$

where $\|\cdot\|$ denotes $\ell_{2}$ norm, and each matrix ${\bm{W}}_{l}$ in function $f$ has Frobenius norm at most $M_{F}(l),l\in[L]$. $M_{E_{i}}$ is the Frobenius norm of embedding matrix ${\bm{E}}_{i}$ and the activation functions are assumed to be 1-Lipschitz and positive-homogeneous. Furthermore, this upper bound can be rewritten as

$$ $\widehat{\mathcal{R}}_{T}(\mathcal{H}_{L})\leq\frac{\left(\prod_{l=1}^{L}M_{F}(l)\right)\sqrt{S}}{\sqrt{T}}\left(\sqrt{\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}\tau_{ij}}\right)\left(\sqrt{2\log(2)L}+1\right)$ (6) $$

where $\tau_{ij}$ is the squared $\ell_{2}$ norm of $j$-th row of embedding matrix ${\bm{E}}_{i}$. In ASR applications, the majority of parameters are concentrated in the embedding layers. Thus, we can see that $\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}\tau_{ij}$ has a significant impact on the upper bound of Rademacher complexity, and consequently, on the generalization error bound.

## 3 The Proposed Approach

As discussed in section [2](https://arxiv.org/html/2511.06374v1#S2), high-dimensional embedding matrices lead to increased generalization error bounds if the training loss remains constant. On the other hand, strictly constraining the norms of the embedding vectors to reduce Rademacher complexity may increase the training error, thus potentially degrading overall performance. To identify the optimal regularization factor, we formulate this trade-off as a constrained optimization problem, described as follows.

$$ $\displaystyle\min_{\tau_{ij}>0}\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}m_{ij}\varphi(\tau_{ij})\quad\text{s.t.}\quad\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}\tau_{ij}\leq C$ (7) $$

where $\varphi(\tau_{ij})=\min_{\|{\bm{e}}_{ij}\|^{2}\leq\tau_{ij}}\mathcal{L}({\bm{e}}_{ij})$ with $i\in[S]$ and $j\in[N_{i}]$. Here, ${\bm{e}}_{ij}$ denotes the $j$-th row of embedding matrix ${\bm{E}}_{i}$, and $\mathcal{L}({\bm{e}}_{ij})$ represents the average cross-entropy (CE) loss evaluated on the DNN output when the training sample activates ${\bm{e}}_{ij}$. For the CE loss function, $\mathcal{L}({\bm{e}}_{ij})$ is lower semicontinuous and bounded below on bounded sets (goodfellow2016deep). Consequently, $\varphi(\tau_{ij})$ is well-defined (ok2011real) and monotonically non-increasing in $\tau_{ij}$, since the feasible set expands with $\tau_{ij}$. Here, $\tau_{ij}$ is the squared norm budget of embedding vector ${\bm{e}}_{ij}$. According to dibenedetto2016real, $\varphi(\tau_{ij})$ is differentiable almost everywhere, and we assume that $\varphi(\tau_{ij})$ is differentiable at the optimal points $\tau^{*}_{ij}$ for analysis. The coefficient $m_{ij}$ denotes the sample frequency with which embedding vector ${\bm{e}}_{ij}$ appears in the training dataset and $C$ is a constant that defines the global norm upper bound for embedding layers. As shown in equation [6](https://arxiv.org/html/2511.06374v1#S2.E6), the value of $C$ directly determines the Rademacher complexity upper bound.

###### Proposition 1

A necessary condition for the optimal regularization multiplier $\lambda^{*}_{ij}$ associated with the $\|{\bm{e}}_{ij}\|^{2}\leq\tau^{*}_{ij}$ is given by $\lambda^{*}_{ij}=\mu_{0}/m_{ij}$, where $\mu_{0}$ is the Lagrange multiplier corresponding to $\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}\tau^{*}_{ij}\leq C$.

Proof. Based on the above assumption, at points of differentiability where a standard constraint qualification holds for the inner optimal solution, the envelope theorem and Lagrangian decomposition (shapiro1979survey) yield

$$ $\varphi^{\prime}(\tau^{*}_{ij})=-\lambda^{*}_{ij}$ (8) $$

where $\lambda^{*}_{ij}\geq 0$ is the Lagrange multiplier corresponding to $\|{\bm{e}}_{ij}\|^{2}\leq\tau^{*}_{ij}$, and $\tau^{*}_{ij}$ is the optimal solution of equation [7](https://arxiv.org/html/2511.06374v1#S3.E7). Here, $\varphi^{\prime}(\tau^{*}_{ij})$ denotes the derivative of $\varphi(\tau^{*}_{ij})$ with respect to $\tau^{*}_{ij}$. We reformulate the optimization problem defined in equation [7](https://arxiv.org/html/2511.06374v1#S3.E7) with the Lagrange multipliers $\mu_{ij}$ and $\mu_{0}$, corresponding to the nonnegativity constraints of $\tau_{ij}\geq 0$ and $\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}\tau_{ij}\leq C$ respectively, where $\mu_{ij}$ and $\mu_{0}$ are restricted to be non-negative.

$$ $\displaystyle\min_{\tau_{ij}>0}$ $\displaystyle\left(\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}m_{ij}\varphi(\tau_{ij})+\mu_{0}\left(\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}\tau_{ij}-C\right)-\sum_{i=1}^{S}\sum_{j=1}^{N_{i}}\mu_{ij}\tau_{ij}\right)$ (9) s.t. $\displaystyle\mu_{0}\geq 0,\mu_{ij}\geq 0,\forall i\in[S],\forall j\in[N_{i}]$ $$

Based on the KKT condition (boyd2004convex), we have $m_{ij}\varphi^{\prime}(\tau^{*}_{ij})+\mu_{0}-\mu_{ij}=0$. Applying the complementary slackness condition, we obtain $\mu_{ij}\tau^{*}_{ij}=0$. This implies the optimality condition for $\tau_{ij}>0$ can be simplified as

$$ $m_{ij}\varphi^{\prime}(\tau^{*}_{ij})+\mu_{0}=0$ (10) $$

By substituting equation [8](https://arxiv.org/html/2511.06374v1#S3.E8) into equation [10](https://arxiv.org/html/2511.06374v1#S3.E10), we obtain a necessary condition for the optimal regularization multiplier $\lambda^{*}_{ij}=\mu_{0}/m_{ij}$.

###### Proposition 1

### 3.1 Adaptive Regularization Method

As shown in equation [6](https://arxiv.org/html/2511.06374v1#S2.E6), embedding layers in ASR applications typically take the majority of the model parameters and have a substantial impact on the generalization error bound. Proposition [1](https://arxiv.org/html/2511.06374v1#Thmtheorem1) suggests that the norm budget for each embedding vector should be allocated according to its sample frequency. However, it is not easy to use the frequency directly during the training process. We can estimate $m_{ij}$ via the stochastic occurrence interval $I_{ij}$ of ${\bm{e}}_{ij}$ for $i\in[S]$ and $j\in[N_{i}]$. Specifically, given that ${\bm{x}}_{i}(t)$ is sampled from an i.i.d. distribution (a common assumption in deep learning), we have $\mathbb{E}[m_{ij}]=T/\mathbb{E}[I_{ij}]$ (papoulis1965random). It enables us to incorporate frequency estimation into the norm budget allocation strategy during training.

Based on the analysis above, we propose an adaptive method that assigns the regularization strength based on the occurrence interval of each embedding vector. Specifically, at each training step $k$, we define last valid update step (LVS) for the embedding vector ${\bm{e}}_{ij}$ as $s_{ij}$. If the gradient norm of ${\bm{e}}_{ij}$ satisfies $\|{\bm{g}}_{ij}\|>0$, we update the LVS by setting $s_{ij}=k$. Otherwise, $s_{ij}$ retains its previous value. Therefore, $s_{ij}$ serves as a lazy-update variable. We define the stochastic update interval as $I_{ij}=k-s_{ij}-1$. The feature-wise regularization coefficient $\lambda_{ij}$ is then dynamically computed as below

$$ $\lambda_{ij}=\min\left(1,\alpha I_{ij}\right),i\in[S],j\in[N_{i}]$ (11) $$

Here, $\alpha\in[0,1)$ denotes the base regularization coefficient. Following the decoupled weight decay approach in AdamW (loshchilov2017decoupled), we incorporate the dynamically computed regularization into each optimizer update step. Algorithm [1](https://arxiv.org/html/2511.06374v1#alg1) outlines the procedure of Adam with adaptive regularization (AdamAR).

For clarity and practical implementation, we use $\theta_{p}$ to denote parameters in the estimation model, where $p\in[P]$ and $P$ is the total number of parameters. At each update step $k$, we first compute the adaptive regularization $\lambda^{k}_{p}$ according to equation [11](https://arxiv.org/html/2511.06374v1#S3.E11) and use it to update the corresponding model parameter $\theta^{k}_{p}$. We then identify the parameters whose gradient norm $\|g^{k}_{p}\|$ is greater than zero in current step and update their last update step state $s^{k}_{p}$ for use in the next iteration.

In fact, our method is not only suitable for Adam, but also compatible with various gradient-based optimizers which have weight decay factor, such as Adagrad (duchi2011adaptive). We have implemented an adaptive version for Adagrad in algorithm [2](https://arxiv.org/html/2511.06374v1#alg2) of the appendix [B](https://arxiv.org/html/2511.06374v1#A2). In section [4](https://arxiv.org/html/2511.06374v1#S4), we conducted comparative experiments evaluating performance across both Adam and Adagrad.

Figure: Algorithm 1 Adam with Adaptive Regularization (AdamAR)

### 3.2 Discussion on the Mechanism of Regularization

As shown in equation [7](https://arxiv.org/html/2511.06374v1#S3.E7), if we do not constrain the embedding norm, the global norm will continue to grow until further increases no longer yield improvements in the objective value since $\varphi(\tau_{ij})$ is non-increasing. In other words, the embedding norms will continue to grow during training, resulting in a looser upper bound on the Rademacher complexity. The sharper drop in performance during multi-epoch training can be attributable to the increased Rademacher complexity resulting from unconstrained norm growth as demonstrated in the experimental section [4.4](https://arxiv.org/html/2511.06374v1#S4.SS4).

Then we discuss how the adaptive regularization takes effect, and also provides guidance for selecting the regularization coefficient $\alpha$.

###### Proposition 2

When adaptive regularization is applied according to equation [11](https://arxiv.org/html/2511.06374v1#S3.E11), the update rule for parameters satisfy $\|\theta^{k-1}_{p}-\lambda^{t}_{p}\theta^{k-1}_{p}-\eta\cdot\hat{m}^{k}_{p}/(\sqrt{\hat{v}^{k}_{p}}+\varepsilon)\|\leq\left(1-\alpha\right)^{I^{k}_{p}}\|\theta^{k-1}_{p}\|+\|\eta\cdot\hat{m}^{k}_{p}/(\sqrt{\hat{v}^{k}_{p}}+\varepsilon)\|$

Proof. Let $I^{k}_{p}\in\mathbb{Z}_{\geq 0}\cap[0,1/\alpha)$, and $\alpha$ is typically chosen such that $\alpha\in[0,1)$. By applying Bernoulli’s inequality to $\left(1-\alpha\right)^{I^{k}_{p}}$, we obtain $\left(1-\alpha\right)^{I^{k}_{p}}\geq 1-\alpha I^{k}_{p}$. In the case where $I^{k}_{p}\in\mathbb{Z}_{\geq 0}\cap[1/\alpha,\infty)$, it follows that $\left(1-\alpha\right)^{I^{k}_{p}}>0$ since $\alpha<1$ and $I^{k}_{p}\geq 0$. Combining both cases, we have

$$ $\left(1-\alpha\right)^{I^{k}_{p}}\geq 1-\min\left(1,\alpha I^{k}_{p}\right)$ (12) $$

We substitute equation [12](https://arxiv.org/html/2511.06374v1#S3.E12) into $\theta^{k-1}_{p}-\lambda^{k}_{p}\theta^{k-1}_{p}-\eta\cdot\hat{m}^{k}_{p}/\left(\sqrt{\hat{v}^{k}_{p}}+\varepsilon\right)$, then we can have

$$ $\|\theta^{k-1}_{p}-\lambda^{k}_{p}\theta^{k-1}_{p}-\eta\cdot\hat{m}^{k}_{p}/(\sqrt{\hat{v}^{k}_{p}}+\varepsilon)\|\leq\left(1-\alpha\right)^{I^{k}_{p}}\|\theta^{k-1}_{p}\|+\|\eta\cdot\hat{m}^{k}_{p}/(\sqrt{\hat{v}^{k}_{p}}+\varepsilon)\|$ (13) $$

From proposition [2](https://arxiv.org/html/2511.06374v1#Thmtheorem2), the update rule for $\theta^{k}_{p}$ can be interpreted intuitively. If the interval $I^{k}_{p}$ is large for the corresponding sparse categorical features, the term $\left(1-\alpha\right)^{I^{k}_{p}}\|\theta^{k-1}_{p}\|$ becomes negligible, and $\theta^{k}_{p}$ is effectively determined by the latest gradient. In the ASR domain, certain categorical feature values may not appear in every mini-batch, resulting in their embedding parameters being updated infrequently, while the MLP parameters are updated in every batch. After the MLP component has nearly converged, the embedding parameters associated with sparse features will have received far fewer updates and may become misaligned with the current state of the MLP parameters, making the previous value $\theta^{k-1}_{p}$ less informative. Proposition [2](https://arxiv.org/html/2511.06374v1#Thmtheorem2) demonstrates that the factor $\left(1-\alpha\right)^{I^{k}_{p}}$ exponentially attenuates the previous value $\theta^{k-1}_{p}$, making the embedding parameters of sparse features depend more heavily on the current gradient. Meanwhile, since the update interval $I^{k}_{p}$ for MLP parameters is identically zero (as they are updated in every batch), the regularization primarily affects the embedding parameters corresponding to very low-frequency feature values.

Moreover, we observe that the method proposed in liu2023multi is a special case of our approach when zero reinitialization is applied to the embedding and $I^{k}_{p}$ is specified as

$$ $I^{k}_{p}=\begin{cases}1/\alpha,&\text{if }kB\bmod T=0\\ 0,&\text{otherwise}\end{cases}$ (14) $$

Where $B$ is the batch size. It can be seen that this heuristic is primarily effective at epoch boundaries. However, for features that have already become overfitted within a single epoch, its performance remains suboptimal.

###### Proposition 2

## 4 Experimental Validation

To evaluate the effectiveness of our proposed method, we conduct several experiments on different public datasets and our industrial dataset. The experimental setup is described in section [4.1](https://arxiv.org/html/2511.06374v1#S4.SS1). The training framework proposed by zhu2022bars is used for training on public datasets, while XDL (jiang2019xdl) is employed for training our proprietary industrial dataset. We show the learning curve on the Avazu dataset in section [4.2](https://arxiv.org/html/2511.06374v1#S4.SS2) to demonstrate the better generalization of our methods over multi-epoch training. In section [4.3](https://arxiv.org/html/2511.06374v1#S4.SS3), we compare multiple backbones and datasets to prove the generalization of our method. Finally, we give a detailed example to show that the low-frequency embeddings are the primary contributors of one-epoch problem, which demonstrates we should allocate a smaller norm budget to embeddings with lower sample frequency. For reproduction, the code(^1^11[https://anonymous.4open.science/r/AdaptiveRegularization-AIDC/](https://anonymous.4open.science/r/AdaptiveRegularization-AIDC/)) is available.

### 4.1 Experimental Setup

#### 4.1.1 Datasets

In our experiments, we use three public datasets as well as an online business dataset from sponsored search. Details of these datasets are provided below.

iPinYou(^2^22[https://huggingface.co/datasets/reczoo/iPinYou_x1](https://huggingface.co/datasets/reczoo/iPinYou_x1)). The training dataset includes processed bidding, impression, click, and conversion logs from iPinYou DSP. It contains 19.5 million records and 16 categorical features, providing a suitable context for exemplifying the overfitting phenomenon.

Amazon(^3^33[https://huggingface.co/datasets/reczoo/AmazonElectronics_x1](https://huggingface.co/datasets/reczoo/AmazonElectronics_x1)). This is a widely used dataset from Amazon for evaluating CTR estimation models. In our study, we use the electronics category of the Amazon dataset, which contains approximately 3 million records and 3 categorical features. Popular models such as DIN (zhou2018deep) has used this dataset for performance evaluation.

Avazu(^4^44[https://huggingface.co/datasets/reczoo/Avazu_x1](https://huggingface.co/datasets/reczoo/Avazu_x1)). This dataset comprises approximately 10 days of labeled click-through data from mobile advertisements, consisting of 40 million records and 22 categorical features spanning both user attributes and advertisement characteristics. Many of these features are highly dimensional, making this dataset a common benchmark for CTR estimation.

LZD. The dataset is sampled from our production environment and consists of real-time bidding logs from sponsored search. The dataset contains 25 million records and 13 categorical features.

#### 4.1.2 MLP Backbone

We test our method with four backbones as follows:

DNN. It is a basic deep learning model defined in equation [2](https://arxiv.org/html/2511.06374v1#S2.E2).

WDL (cheng2016wide). It combines a linear model and a deep neural network to capture both memorization and generalization for recommender systems.

DeepFM (guo2017deepfm). It integrates factorization machines for learning low-order feature interactions and deep neural networks for high-order feature interactions.

WuKong (gu2022wukong). It proposes a specific model designed to explore these scaling laws. Its architecture is constructed with deeper and wider layers to capture high-order feature interactions.

#### 4.1.3 Methods and Hyperparameters

To evaluate the one-epoch overfitting phenomenon, we set the number of training epochs to 4 and compare the performance of four benchmarks as follows:

Baseline optimizer. Adam and Adagrad serve as the baseline optimizers.

MEDA. MEDA is applied using the same optimizer settings as in baseline.

AdamW and AdagradW. The baseline optimizers are enhanced with weight decay.

AdamAR and AdagradAR. The baseline optimizers are combined with our method.

The hyperparameter settings are as follows. The embedding dimension is set to 32, with zero initialization. The learning rate is set to 0.001 for Adam and 0.01 for Adagrad, respectively. The batch size is 2048. Both $\alpha$ and the weight decay parameter are selected via grid search over the range $10^{-7}$ to $10^{-1}$, with the best hyperparameter settings determined according to validation performance (see appendix [D](https://arxiv.org/html/2511.06374v1#A4) for details). Network architectures are configured with minor adjustments across datasets. For detailed configurations, please refer to table [3](https://arxiv.org/html/2511.06374v1#A3.T3) in appendix [C](https://arxiv.org/html/2511.06374v1#A3). All other hyperparameters are set to their default values. Area under the curve (AUC) and binary cross-entropy loss are used as evaluation metrics. Experiments are conducted on a single machine with an NVIDIA L20 GPU.

### 4.2 Learning Curve and Generalization Results

In this section, we present the training loss and test AUC across multiple epochs on the Avazu dataset, using a DNN backbone and the Adam optimizer. The learning curves in figure [1](https://arxiv.org/html/2511.06374v1#S4.F1) compare our method with several baselines. For the native optimizer, training loss decreases rapidly after the first epoch. However, test AUC drops sharply with additional epochs, indicating clear overfitting. In contrast, MEDA and weight decay alleviate this problem, yielding more stable test AUC throughout training. Our method achieves the best overall performance. Figures [1(b)](https://arxiv.org/html/2511.06374v1#S4.F1.sf2) and [1(c)](https://arxiv.org/html/2511.06374v1#S4.F1.sf3) reveal an inverse correlation between the $\ell_{2}$ norm of the embedding vectors and test AUC. It demonstrates that constraining the embedding norm improves generalization ability of models. Among all methods, our proposed AdamAR achieves the lowest cumulative $\ell_{2}$ norm for the embedding vectors, highlighting its effectiveness in controlling regularization strength.

Figure: (a) Training Loss
Refer to caption: x1.png

### 4.3 Performance over Different Datasets and MLP Backbones

In this section, we compare test AUC across different datasets and MLP backbones to demonstrate the generalization capability of our method. The results, summarized in table [1](https://arxiv.org/html/2511.06374v1#S4.T1) and table [2](https://arxiv.org/html/2511.06374v1#S4.T2), show that our methods consistently outperform other approaches on all datasets and architectures, achieving superior AUC in both single-epoch and multi-epoch training. Furthermore, the gains are consistent across diverse model architectures, ranging from basic DNN to more sophisticated designs like WuKong, which capture complex feature interactions. Overall, these results demonstrate the robustness and versatility of our adaptive regularization approach across various settings.

**Table 1: Comparison of test AUC across different datasets and models with the Adam optimizer. E1-E4 denote results after 1-4 epochs, respectively. The best results are highlighted in bold.**
| Dataset | Method | DNN | WDL | DeepFM | WuKong |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| E1 | E2 | E3 | E4 | E1 | E2 | E3 | E4 | E1 | E2 | E3 | E4 | E1 | E2 | E3 | E4 |  |  |
| iPinYou | Adam | 0.7498 | 0.7258 | 0.7174 | 0.7026 | 0.7622 | 0.7322 | 0.6972 | 0.6978 | 0.7595 | 0.7343 | 0.7057 | 0.6983 | 0.7644 | 0.7437 | 0.7137 | 0.6955 |
| MEDA | 0.7498 | 0.7631 | 0.7713 | 0.7692 | 0.7622 | 0.7582 | 0.7574 | 0.7506 | 0.7595 | 0.7582 | 0.7546 | 0.7518 | 0.7644 | 0.7671 | 0.7686 | 0.7714 |  |
| AdamW | 0.7486 | 0.7595 | 0.7628 | 0.7500 | 0.7609 | 0.7615 | 0.7692 | 0.7644 | 0.7600 | 0.7584 | 0.7486 | 0.7367 | 0.7593 | 0.7687 | 0.7586 | 0.7549 |  |
| AdamAR | 0.7549 | 0.7728 | 0.7730 | 0.7725 | 0.7675 | 0.7701 | 0.7692 | 0.7650 | 0.7677 | 0.7683 | 0.7637 | 0.7590 | 0.7668 | 0.7737 | 0.7722 | 0.7738 |  |
| Amazon | Adam | 0.8482 | 0.8532 | 0.8314 | 0.8179 | 0.8489 | 0.8500 | 0.8255 | 0.8123 | 0.8496 | 0.8510 | 0.8236 | 0.8117 | 0.8574 | 0.8577 | 0.8355 | 0.8219 |
| MEDA | 0.8482 | 0.8537 | 0.8561 | 0.8574 | 0.8489 | 0.8506 | 0.8565 | 0.8557 | 0.8496 | 0.8511 | 0.8560 | 0.8557 | 0.8574 | 0.8577 | 0.8601 | 0.8627 |  |
| AdamW | 0.8472 | 0.8554 | 0.8409 | 0.8268 | 0.8473 | 0.8524 | 0.8377 | 0.8195 | 0.8472 | 0.8523 | 0.8380 | 0.8192 | 0.8544 | 0.8615 | 0.8442 | 0.8315 |  |
| AdamAR | 0.8506 | 0.8669 | 0.8709 | 0.8683 | 0.8513 | 0.8656 | 0.8690 | 0.8645 | 0.8513 | 0.8657 | 0.8690 | 0.8644 | 0.8576 | 0.8687 | 0.8702 | 0.8664 |  |
| Avazu | Adam | 0.7483 | 0.7229 | 0.6973 | 0.6868 | 0.7485 | 0.7226 | 0.6949 | 0.6895 | 0.7488 | 0.7196 | 0.6918 | 0.6821 | 0.7479 | 0.7322 | 0.7000 | 0.7147 |
| MEDA | 0.7483 | 0.7508 | 0.7479 | 0.7479 | 0.7485 | 0.7475 | 0.7472 | 0.7516 | 0.7488 | 0.7478 | 0.7441 | 0.7523 | 0.7479 | 0.7538 | 0.7555 | 0.7559 |  |
| AdamW | 0.7584 | 0.7594 | 0.7579 | 0.7576 | 0.7586 | 0.7590 | 0.7544 | 0.7580 | 0.7590 | 0.7586 | 0.7546 | 0.7580 | 0.7511 | 0.7533 | 0.7565 | 0.7553 |  |
| AdamAR | 0.7621 | 0.7640 | 0.7636 | 0.7625 | 0.7630 | 0.7622 | 0.7617 | 0.7630 | 0.7629 | 0.7623 | 0.7613 | 0.7636 | 0.7619 | 0.7607 | 0.7624 | 0.7620 |  |
| LZD | Adam | 0.7148 | 0.6661 | 0.6250 | 0.6105 | 0.7147 | 0.6719 | 0.6305 | 0.6053 | 0.7131 | 0.6706 | 0.6237 | 0.6105 | 0.7156 | 0.6629 | 0.6165 | 0.6069 |
| MEDA | 0.7148 | 0.7110 | 0.7113 | 0.7177 | 0.7147 | 0.7183 | 0.7169 | 0.7157 | 0.7131 | 0.7125 | 0.7157 | 0.7149 | 0.7156 | 0.7181 | 0.7182 | 0.7185 |  |
| AdamW | 0.7146 | 0.7132 | 0.7107 | 0.7138 | 0.7142 | 0.7136 | 0.7143 | 0.7129 | 0.7121 | 0.7107 | 0.7140 | 0.7132 | 0.7126 | 0.7149 | 0.7163 | 0.7115 |  |
| AdamAR | 0.7238 | 0.7256 | 0.7238 | 0.7234 | 0.7226 | 0.7233 | 0.7236 | 0.7242 | 0.7227 | 0.7199 | 0.7218 | 0.7221 | 0.7239 | 0.7218 | 0.7201 | 0.7185 |  |

**Table 2: Comparison of test AUC across different datasets and models with the Adagrad optimizer. E1-E4 denote results after 1-4 epochs, respectively. The best results are highlighted in bold.**
| Dataset | Method | DNN | WDL | DeepFM | WuKong |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| E1 | E2 | E3 | E4 | E1 | E2 | E3 | E4 | E1 | E2 | E3 | E4 | E1 | E2 | E3 | E4 |  |  |
| iPinYou | Adagrad | 0.7626 | 0.6496 | 0.6366 | 0.6079 | 0.7623 | 0.6697 | 0.6424 | 0.6190 | 0.7626 | 0.6745 | 0.6440 | 0.6202 | 0.7651 | 0.6962 | 0.6489 | 0.5994 |
| MEDA | 0.7626 | 0.7673 | 0.7733 | 0.7689 | 0.7623 | 0.7725 | 0.7650 | 0.7717 | 0.7626 | 0.7722 | 0.7660 | 0.7728 | 0.7651 | 0.7756 | 0.7730 | 0.7712 |  |
| AdagradW | 0.7586 | 0.7614 | 0.7651 | 0.7570 | 0.7538 | 0.7652 | 0.7656 | 0.7598 | 0.7511 | 0.7662 | 0.7631 | 0.7632 | 0.7544 | 0.7704 | 0.7657 | 0.7645 |  |
| AdagradAR | 0.7717 | 0.7706 | 0.7759 | 0.7709 | 0.7705 | 0.7763 | 0.7758 | 0.7725 | 0.7703 | 0.7767 | 0.7741 | 0.7739 | 0.7689 | 0.7803 | 0.7792 | 0.7774 |  |
| Amazon | Adagrad | 0.8435 | 0.8392 | 0.8127 | 0.8019 | 0.8427 | 0.8359 | 0.8113 | 0.7985 | 0.8425 | 0.8359 | 0.8102 | 0.8005 | 0.8487 | 0.8465 | 0.8176 | 0.8050 |
| MEDA | 0.8435 | 0.8477 | 0.8506 | 0.8521 | 0.8427 | 0.8480 | 0.8500 | 0.8503 | 0.8425 | 0.8493 | 0.8471 | 0.8522 | 0.8487 | 0.8568 | 0.8584 | 0.8613 |  |
| AdagradW | 0.8429 | 0.8564 | 0.8553 | 0.8533 | 0.8432 | 0.8554 | 0.8531 | 0.8492 | 0.8436 | 0.8556 | 0.8533 | 0.8492 | 0.8513 | 0.8616 | 0.8590 | 0.8584 |  |
| AdagradAR | 0.8474 | 0.8652 | 0.8709 | 0.8716 | 0.8462 | 0.8641 | 0.8696 | 0.8704 | 0.8462 | 0.8645 | 0.8697 | 0.8707 | 0.8539 | 0.8675 | 0.8708 | 0.8697 |  |
| Avazu | Adagrad | 0.7547 | 0.7324 | 0.7154 | 0.7082 | 0.7555 | 0.7321 | 0.7175 | 0.7080 | 0.7553 | 0.7317 | 0.7165 | 0.7070 | 0.7531 | 0.7324 | 0.7170 | 0.7039 |
| MEDA | 0.7547 | 0.7544 | 0.7532 | 0.7548 | 0.7555 | 0.7547 | 0.7544 | 0.7556 | 0.7553 | 0.7545 | 0.7545 | 0.7550 | 0.7531 | 0.7543 | 0.7561 | 0.7547 |  |
| AdagradW | 0.7579 | 0.7586 | 0.7569 | 0.7563 | 0.7588 | 0.7572 | 0.7573 | 0.7593 | 0.7592 | 0.7586 | 0.7570 | 0.7586 | 0.7482 | 0.7554 | 0.7549 | 0.7551 |  |
| AdagradAR | 0.7626 | 0.7629 | 0.7629 | 0.7622 | 0.7629 | 0.7640 | 0.7632 | 0.7634 | 0.7628 | 0.7633 | 0.7626 | 0.7620 | 0.7621 | 0.7617 | 0.7620 | 0.7612 |  |
| LZD | Adagrad | 0.7232 | 0.6666 | 0.6417 | 0.6400 | 0.7252 | 0.6696 | 0.6374 | 0.6388 | 0.7254 | 0.6677 | 0.6409 | 0.6399 | 0.7280 | 0.6753 | 0.6449 | 0.6365 |
| MEDA | 0.7232 | 0.7206 | 0.7137 | 0.7237 | 0.7252 | 0.7241 | 0.7253 | 0.7202 | 0.7254 | 0.7256 | 0.7245 | 0.7223 | 0.7280 | 0.7254 | 0.7252 | 0.7213 |  |
| AdagradW | 0.7190 | 0.7057 | 0.7071 | 0.7119 | 0.7210 | 0.7139 | 0.7142 | 0.7100 | 0.7209 | 0.7110 | 0.7098 | 0.7105 | 0.7179 | 0.7076 | 0.6834 | 0.6869 |  |
| AdagradAR | 0.7232 | 0.7261 | 0.7204 | 0.7265 | 0.7284 | 0.7261 | 0.7256 | 0.7269 | 0.7280 | 0.7267 | 0.7258 | 0.7263 | 0.7281 | 0.7258 | 0.7254 | 0.7213 |  |

### 4.4 Example of the Root Cause of Overfitting

In this section, we give an example to show the root cause of multi-epoch overfitting. We use the iPinYou dataset with DNN backbone and Adam to illustrate this issue. The detailed feature statistics are listed in table [4](https://arxiv.org/html/2511.06374v1#A5.T4) in appendix [E](https://arxiv.org/html/2511.06374v1#A5). We can observe that most features on iPinYou dataset are relatively dense, and their mean update intervals during training are less than 1. To investigate the root cause of overfitting, we select the feature ”IP”, which is the most sparse feature in iPinYou dataset with 704,966 unique IDs. We apply a filtering procedure to reduce its feature sparsity. Given a ratio $r$, we only retain the top-$r$ fraction (by frequency) of IDs and replace other IDs with a default ID.

Figure [2(a)](https://arxiv.org/html/2511.06374v1#S4.F2.sf1) demonstrates that as $r$ decreases, the test AUC remains stable across multiple epochs, indicating that the one-epoch overfitting phenomenon is effectively alleviated. It suggests that low-frequency IDs in the ”IP” feature are the primary cause of one-epoch overfitting in the iPinYou dataset. Based on this observation, a straightforward approach to mitigate this issue is to exclude the ”IP” feature. However, as shown by the case $r=0$, removing the ”IP” feature results in a substantial test AUC drop (from 0.7498 to 0.7429) at the end of the first epoch.

Although removing sparse features can mitigate the one-epoch overfitting issue, it often leads to diminished model performance. In contrast, our proposed method dynamically adjusts the regularization strength, effectively preventing overfitting while maintaining strong predictive performance.

Figure: (a) Test AUC
Refer to caption: x4.png

## 5 Conclusion

We propose an adaptive regularization method to address the one-epoch problem in estimation models for the ASR domain. Experimental results demonstrate that our approach effectively mitigates the one-epoch issue and improves estimation performance. Additionally, we provide a theoretical explanation for the one-epoch phenomenon and illustrate how the proposed method takes effect through analytical derivations and experiments. This approach has already been fully deployed in the production environment of sponsored search in our company.

## Appendix

## Appendix A GenAI Usage Disclosure

During the preparation of this work, the author used ChatGPT to improve the language. After using this tool, the author reviewed and edited the content as necessary and takes full responsibility for the final publication.

## Appendix B Adagrad with Adaptive Regularization

Adagrad, like Adam, is widely used in the ASR domain. Our method extends readily to Adagrad, as detailed in Algorithm [2](https://arxiv.org/html/2511.06374v1#alg2).

Figure: Algorithm 2 Adagrad with Adaptive Regularization (AdagradAR)

## Appendix C Network Architecture Configurations

We adjust the network architectures based on the feature and sample sizes of each dataset. As summarized in table [3](https://arxiv.org/html/2511.06374v1#A3.T3), larger and deeper networks are utilized for datasets with more features and samples, such as iPinYou, Avazu, and LZD. In contrast, smaller and shallower networks are adopted for the Amazon dataset to mitigate the risk of overfitting.

**Table 3: Network architecture configurations for different datasets**
| Dataset | DNN | WDL | DeepFM | WuKong |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MLP Hidden Sizes | MLP Hidden Sizes | MLP Hidden Sizes | Layers | LCB Emb | FMB Emb | FMB Hiddens | MLP Hidden Sizes |  |
| iPinYou | [512, 256, 128] | [512, 256, 128] | [512, 256, 128] | 5 | 12 | 12 | [64, 32] | [512, 256, 128] |
| Amazon | [512, 256, 128] | [512, 256] | [512, 256] | 1 | 8 | 8 | [32, 32] | [512, 256] |
| Avazu | [512, 256, 128] | [512, 256, 128] | [512, 256, 128] | 5 | 12 | 12 | [64, 32] | [512, 256, 128] |
| LZD | [512, 256, 128] | [512, 256, 128] | [512, 256, 128] | 5 | 12 | 12 | [64, 32] | [512, 256, 128] |

## Appendix D Grid Search for the Weight Decay Coefficient

The weight decay coefficient is a crucial hyperparameter that significantly affects model performance. In our adaptive regularization method, the weight decay coefficient is denoted as $\alpha$ in equation [11](https://arxiv.org/html/2511.06374v1#S3.E11). To ensure a fair comparison across methods, we perform a grid search to select the optimal weight decay value for each dataset and model combination. Figure [3](https://arxiv.org/html/2511.06374v1#A4.F3) illustrates the impact of varying the weight decay coefficient on test AUC for the Avazu dataset using a DNN backbone at the end of epoch 2. Although test AUC fluctuates with different values of the weight decay coefficient, our proposed method consistently achieves superior performance compared to AdamW and AdagradW. Furthermore, the performance of AdamW and AdagradW varies dramatically with the weight decay coefficient, while our method demonstrates greater stability. This indicates that our method is less sensitive to the choice of weight decay coefficient, thereby simplifying hyperparameter tuning in practice.

Figure: (a) Adam
Refer to caption: x6.png

## Appendix E Feature Statistics on iPinYou Dataset

The iPinYou dataset contains a total of 16 features, among which only a few are sparse. Table [4](https://arxiv.org/html/2511.06374v1#A5.T4) lists the top six sparse features along with their statistical indicators.

**Table 4: For the top six sparse features in the iPinYou dataset, the table below presents the number of unique IDs, the average occurrences of each ID, and the mean update interval for each ID with a batch size of 2048.**
| Feature | IP | slotid | domain | city | creative | useragent |
| --- | --- | --- | --- | --- | --- | --- |
| Unique IDs | 704,966 | 180,696 | 51,322 | 372 | 133 | 42 |
| Mean Occurrences | 27.7 | 107.9 | 379.9 | 52,408.5 | 146,586.3 | 464,189.9 |
| Mean Update Intervals | 344.222 | 88.230 | 25.060 | 0.182 | 0.065 | 0.021 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses.
soundness: 4
strengths: A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results.

1. Sound and well-motivated analysis linking embedding sparsity and overfitting.
2. Adaptive rule is intuitive and easy to implement.
3. Multi-dataset, multi-optimizer validation with consistent gains.
4. Addresses a critical and under-theorized issue in real-world recommender systems.
5.Provides a clear explanation for why rare features require stronger regularization.
summary: This paper analyzes the one-epoch overfitting phenomenon in large-scale CTR/CVR models with sparse embeddings. Using Rademacher complexity, the authors show that overfitting arises from unconstrained embedding norm growth for infrequent features. They propose an adaptive regularization method that dynamically adjusts regularization strength by feature update intervals, forming optimizers. Experiments on public and industrial datasets demonstrate consistent performance gains, improved generalization, and easy integration into existing training pipelines.
weaknesses: 1. The analysis assumes i.i.d. sampling and bounded feature norms, which may not hold in practice for long-tail industrial data.
2. The derivation of adaptive coefficients relies on approximating feature frequency by occurrence intervals; this estimation may introduce stochastic noise not formally analyzed.
3. The differentiability of \phi(\tau_{ij}) and the KKT-based derivation rest on idealized smoothness assumptions that may not hold in real deep networks.
4. The paper doesn’t rigorously study how the adaptive coefficient \lambda_{ij} affects optimizer dynamics or convergence stability, especially in non-convex regimes.
5. A theoretical guarantee on convergence rate or bias induced by adaptive decay would strengthen the work.
6. Tracking the “last valid update step” for every embedding vector may introduce memory overhead in billion-scale models. The paper does not discuss memory/time complexity or optimization strategies for this in production.
7. While the baselines are relevant, comparisons with more modern adaptive regularization or implicit bias control methods  e.g., SAM, AdaNorm, or adaptive dropout  would enhance credibility.
8. No statistical significance testing is reported for AUC gains, though the improvements are small (≈0.002–0.01).
9. The paper briefly mentions online deployment but does not analyze potential trade-offs e.g., regularization lag under distribution shift, interpretability implications, or fairness across infrequent users/items.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Does applying strong L2 to low-frequency rows in adamW reproduces the main gains? If so, that would validate the causal story more strongly.
soundness: 4
strengths: * Problem importance. The paper tackles an industry-recognized issue in large-scale CTR models—a core component of nearly all advertising systems.
* Theoretical grounding. It links the empirical “train only one epoch” practice to a capacity-control perspective, giving the phenomenon a clearer theoretical justification.
* Practical simplicity. The proposed technique is easy to integrate: it only requires tracking the last valid update step for each embedding row and can be directly plugged into existing optimizers.
summary: The paper targets a common phenomenon in large-scale CTR/recommender models with highly sparse features: performance is often best after the first epoch, but further training leads to overfitting and degradation. The authors provide an explanation from the perspective of generalization bounds through Rademacher complexity as follows:

* most of the model capacity is concentrated in the huge embedding matrix, and in particular, in the very infrequent IDs’ embedding rows. 
* the embedding rows with very infrequent IDs are updated on a much slower schedule than the dense part / MLP, so their norms can grow uncontrollably and thus inflate the overall capacity. 

Based on this, the authors propose an adaptive regularization scheme that assigns stronger weight decay to infrequent embedding rows . They integrate this row-wise adaptive weight decay into Adam/Adagrad (termed AdamAR/AdagradAR), so that the sparser and less frequently updated a row is, the stronger the regularization it receives. 

Experiments on multiple datasets and CTR architectures show that this approach can mitigate or even eliminate the “multi-epoch performance drop,” while slightly improving AUC.
weaknesses: 1. The connection to existing “frequency-aware” or “epoch-level reset” approaches (e.g., MEDA-style methods) is not elaborated systematically.
2. The experiments mainly report global metrics such as AUC/Logloss, but lack fine-grained bucket analyses (by feature frequency, long-tail IDs, cold-start features) to directly demonstrate that “low-frequency rows were actually controlled.”

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: * How Sensitive is the Proposed Method to Hyperparameter $\alpha$ Across Datasets?
Figure 3 provides some evidence, but would the authors provide additional experimental data on extreme values of $\alpha$, and practical guidelines for selection especially in large-scale real-world datasets?
* Handling Never-seen or Extremely Rare Features:
Does the adaptive regularization approach systematically bias against tail classes? How does it perform in cold-start (new feature value) scenarios, and is there risk of underfitting for features with vanishing sample frequency? Can the method adapt dynamically if sparsity levels change after deployment?
* Ablation on Update Interval Estimation:
Would the authors run ablation studies to decouple the effect of interval-based regularization vs. embedding-based regularization overall?
* Comparison with Other Adaptive/Hybrid Regularization Approaches:
How does the proposed method perform compared to thresholding or pruning-based strategies, as well as non-uniform L1/L2 variants found in the sparse modeling literature? Would Table 1/2 results meaningfully shift under such baselines?
* Computational Overhead and Scalability:
Are there notable compute or memory implications to tracking per-embedding update intervals, especially as feature cardinality scales beyond the reported datasets?
soundness: 3
strengths: * Theoretical Insights: The paper offers a coherent theoretical framework using Rademacher complexity to explain one-epoch overfitting in sparse embeddings, clarifying a widely observed yet under-theorized phenomenon in deep CTR/CVR models (see Section 2.2 and 3).
* Principled Adaptive Method: Instead of relying on heuristics, the proposed adaptive regularization derives directly from the theoretical analysis, offering an intuitive and practical approach that is compatible with common optimizers (Section 3.1, Algorithm 1).
* Comprehensive Empirical Evaluation: The method is validated across a diverse set of datasets (iPinYou, Amazon, Avazu, proprietary LZD) and backbone architectures (DNN, WDL, DeepFM, WuKong), demonstrating strong and consistent improvements over multiple baselines (Tables 1, 2).
summary: This paper investigates the prevalent phenomenon of "one-epoch overfitting" in large-scale sparse feature embedding models, especially pertinent to click-through rate and conversion rate estimation in online advertising and recommendation systems. The authors provide a theoretical analysis rooted in Rademacher complexity, pinpointing the growth of embedding norms as the root cause of deteriorating generalization when training persists beyond a single epoch. To address this, they propose an adaptive regularization technique that dynamically adjusts regularization strength per embedding based on occurrence intervals, integrating seamlessly with standard optimizers like Adam and Adagrad. Comprehensive experiments across public and industrial datasets demonstrate that the proposed method outperforms existing baselines in mitigating overfitting and promoting stable, improved performance during multi-epoch training.
weaknesses: * Limited Positioning Relative to Broader Regularization Literature: While the paper covers baseline regularization approaches (L1, L2, weight decay), it does not sufficiently engage with recent or foundational work on sparsity-driven regularization at both the architectural and optimization levels. For instance, no discussion is provided relating their adaptive method to pruning/growth strategies or thresholding-based approaches, even though such techniques are central to the sparsity literature.
* Lack of Ablations: There is little to no ablation study dissecting critical components of the proposed method, such as the sensitivity to the base regularization coefficient $\alpha$ beyond Figure 3, or the explicit contribution of occurrence interval estimation. It remains unclear how much each modeling choice contributes to the performance gains.
* Hyperparameter Selection Transparency: Although hyperparameters are said to be chosen by grid search (Section 4.1.3), the details are sequestered in the appendix and visualization (Figure 3), limiting transparency and reproducibility. The main text should provide a concise summary of grid search procedures and best practice recommendations.
* Ambiguity in Some Notation and Algorithm Details: Certain notations, particularly around frequency and interval estimation for embeddings (Section 3.1), could be more precise. For instance, the paper uses $I_{ij}$ as both a stochastic occurrence interval and an empirical quantity with little justification for the estimation method used during optimizer steps.
* Insufficient Theoretical Analysis of Method Limitations: The theoretical section does not address possible drawbacks or edge cases (such as whether the adaptive regularization could lead to underfitting for extremely rare or never-seen values), nor the impact of frequency misestimation on convergence or generalization.
* Lack of Comparison to other LLM-based models: There are some new algorithms which are LLM-based recommendation algorithm, and this paper does not consider them.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Can the complexity analysis be extended to explicit feature interaction models like FM? It would be great to include such analysis since most modern recommendation systems heavily rely on explicit interactions.

2. How does the rewriting from Eq. 5 to Eq. 6 happen?

3. I'm curious about the effect of the proposed regularization method on the dimensional robustness of the learned embeddings. There are several recent works discussing the dimensional collapse (DC) issue in recommendation models [1,2], and I wonder whether there is a connection between the one-epoch phenomenon and DC. Can you provide an analysis of the singular spectrum of sparse embeddings?

[1]. On the embedding collapse when scaling up recommendation models. ICML 2024.

[2]. Balancing Embedding Spectrum for Recommendation. 2025.

[3]. Towards Mitigating Dimensional Collapse of Representations in Collaborative Filtering. WSDM 2024.
soundness: 3
strengths: 1. Motivation. The paper studies a well-known and critical problem in pCXR prediction - the "one-epoch" overfitting problem.

2. Theoretical Grounding. The analysis links overfitting to Rademacher complexity, showing that embedding norm growth drives generalization error, giving a principled basis for adaptive regularization.

3. Technical Novelty. The proposed method is well-derived and elegant, with a limited impact on the model training.
summary: The authors study the one-epoch overfitting phenomenon in pCTR and pCVR models. They first conduct a theoretical analysis using Rademacher complexity, indicating that the norm of sparse embeddings leads to poor generalization. The authors propose an adaptive regularization method (AdamAR / AdagradAR)  with a dynamic strength of the embedding regularization terms. Experiments on several public datasets demonstrate the effectiveness of the proposed method.
weaknesses: 1. Extension to Explicit Interaction Models. The analysis is only in DNNs, whose effectiveness is limited. It would be great to extend the theoretical discussion to explicit interaction models such as FM or CrossNet in DCN V2.

2. Experiments. Besides Wukong (BTW, the citation of the Wukong paper is wrong), other models in the experiment section are a bit out of date. It would be great to add experiments on the new SOTAs, e.g., DCN V2, xDeepFM.

3. Writing. The abstract contains limited concrete content. It would be great to be a bit more specific on the proposed theoretical analysis and method.


Decision: Accept (Poster)