Title: Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad
Venue: NeurIPS 2024
Accepted: yes
Keywords: Scale-invariant methods

AdaGrad

non-convex optimization

adaptive methods
Abstract: Adaptive methods are extremely popular in machine learning as they make learning rate tuning less expensive. This paper introduces a novel optimization algorithm named KATE, which presents a scale-invariant adaptation of the well-known AdaGrad algorithm. We prove the scale-invariance of KATE for the case of Generalized Linear Models. Moreover, for general smooth non-convex problems, we establish a convergence rate of  $O((\log T)/\sqrt{T})$ for KATE, matching the best-known ones for AdaGrad and Adam. We also compare KATE to other state-of-the-art adaptive algorithms Adam and AdaGrad in numerical experiments with different problems, including complex machine learning tasks like image classification and text classification on real data. The results indicate that KATE consistently outperforms AdaGrad and matches/surpasses the performance of Adam in all considered scenarios.

Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad

Abstract

Adaptive methods are extremely popular in machine learning as they make learning rate tuning less expensive. This paper introduces a novel optimization algorithm named KATE , which presents a scale-invariant adaptation of the well-known AdaGrad algorithm. We prove the scale-invariance of KATE for the case of Generalized Linear Models. Moreover, for general smooth non-convex problems, we establish a convergence rate of O ( log T / √ T ) for KATE , matching the best-known ones for AdaGrad and Adam . We also compare KATE to other state-of-the-art adaptive algorithms Adam and AdaGrad in numerical experiments with different problems, including complex machine learning tasks like image classification and text classification on real data. The results indicate that KATE consistently outperforms AdaGrad and matches/surpasses the performance of Adam in all considered scenarios.

1 Introduction

In this work, we consider the following unconstrained optimization problem:

where f : R d → R is a L -smooth and generally non-convex function. In particular, we are interested in the situations when the objective has either expectation f ( w ) = E ξ ∼D [ f ξ ( w )] or finite-sum f ( w ) = 1 n ∑ n i =1 f i ( w ) form. Such minimization problems are crucial in machine learning, where w corresponds to the model parameters. Solving these problems with stochastic gradient-based optimizers has gained much interest owing to their wider applicability and low computational cost. Stochastic Gradient Descent ( SGD ) (Robbins and Monro, 1951) and similar algorithms require the knowledge of parameters like L for convergence and are very sensitive to the choice of the stepsize in general. Therefore, SGD requires hyperparameter tuning, which can be computationally expensive. To address these issues, it is common practice to use adaptive variants of stochastic gradient-based methods that can converge without knowing the function's structure.

There exist many adaptive algorithms such as AdaGrad (Duchi et al., 2011), Adam (Kingma and Ba, 2014), AMSGrad (Reddi et al., 2019), D-Adaptation (Defazio and Mishchenko, 2023), Prodigy (Mishchenko and Defazio, 2023), AI-SARAH (Shi et al., 2023) and their variants. These adaptive techniques are capable of updating their step sizes on the fly. For instance, the AdaGrad method determines its step sizes using a cumulative sum of the coordinate-wise squared (stochastic) gradient

of all the previous iterates:

where g t represents an unbiased estimator of ∇ f ( w t ) , i.e., E [ g t | w t ] = ∇ f ( w t ) , diag ( M ) ∈ R d is a vector of diagonal elements of matrix M ∈ R d × d , ∆ > 0 , and the division by vector is done component-wise. Ward et al. (2020) has shown that this method achieves a convergence rate of O ( log T / √ T ) for smooth functions, similar to SGD , without prior knowledge of the functions' parameters. However, the performance of AdaGrad deteriorates when applied to data that may exhibit poor scaling or ill-conditioning. In this work, we propose a novel algorithm, KATE , to address the issues of poor data scaling. KATE is also a stochastic adaptive algorithm that can achieve a convergence rate of O ( log T / √ T ) for smooth non-convex functions in terms of min t ∈ [ T ] E [ ∥∇ f ( w t ) ∥ ] 2 .

1.1 Related Work

A significant amount of research has been done on adaptive methods over the years, including AdaGrad (Duchi et al., 2011; McMahan and Streeter, 2010), AMSGrad (Reddi et al., 2019), RMSProp (Tieleman and Hinton, 2012), AI-SARAH (Shi et al., 2023), and Adam (Kingma and Ba, 2014). However, all these works assume that the optimization problem is contained in a bounded set. To address this issue, Li and Orabona (2019) proposes a variant of the AdaGrad algorithm, which does not use the gradient of the last iterate (this makes the step sizes of t -th iteration conditionally independent of g t ) for computing the step sizes and proves convergence for the unbounded domain.

Each of these works considers a vector of step sizes for each coefficient. Duchi et al. (2011) and McMahan and Streeter (2010) simultaneously proposed the original AdaGrad algorithm. However, McMahan and Streeter (2010) was the first to consider the vanilla scalar form of AdaGrad , known as

Later, Ward et al. (2020) analyzed AdaGradNorm for minimizing smooth non-convex functions. In a follow-up study, Xie et al. (2020) proves a linear convergence of AdaGradNorm for strongly convex functions. Recently, Liu et al. (2022) analyzed AdaGradNorm for solving smooth convex functions without the bounded domain assumption. Moreover, Liu et al. (2022) extends the convergence guarantees of AdaGradNorm to quasar-convex functions 2 using the function value gap. Orabona et al. (2015) introduce the notion of scale-invariance, which is a special case of affine invariance (Nesterov and Nemirovskii, 1994; Nesterov, 2018; d'Aspremont et al., 2018), propose a scaleinvariant version of AdaGrad for online convex optimization for generalized linear models, and prove O ( √ T ) regret bounds in this setup.

Recently, Defazio and Mishchenko (2023) introduced the D-Adaptation method, which has gathered considerable attention due to its promising empirical performances. In order to choose the adaptive step size optimally, one requires knowledge of the initial distance from the solution, i.e., D := ∥ w 0 -w ∗ ∥ where w ∗ ∈ argmin w ∈ R d f ( w ) . The D-Adaptation method works by maintaining an estimate of D and the stepsize choice in this case is d t / √ ∑ t τ =0 ∥ g τ ∥ 2 for the t -th iteration (here d t is an estimate of D ). Mishchenko and Defazio (2023) further modifies the algorithm in a follow-up work and introduces Prodigy (with stepsize choice d 2 t / √ ∑ t τ =0 d 2 τ ∥ g τ ∥ 2 ) to improve the convergence speed.

Another exciting line of work on adaptive methods is Polyak stepsizes. Polyak (1969) first proposed Polyak stepsizes for subgradient methods, and recently, the stochastic version (also known as SPS ) was introduced by Oberman and Prazeres (2019); Loizou et al. (2021); Abdukhakimov et al. (2024, 2023); Li et al. (2023) and Gower et al. (2021). For a finite sum problem min w ∈ R d f ( w ) := 1 n ∑ n i =1 f i ( w ) , Loizou et al. (2021) uses f i ( w t ) -f ∗ i c ∥∇ f i ( w t ) ∥ 2 as their stepsize choices (here f ∗ i := min w ∈ R d f i ( w ) ), while Oberman and Prazeres (2019) uses 2( f ( w t ) -f ∗ ) E [ ∥∇ f i ( w t ) ∥ 2 ] for k -th iteration. However, these methods are impractical when f ∗ or f ∗ i is unknown. Following its introduction,

several variants of the SPS algorithm emerged (Li et al., 2023; D'Orazio et al., 2021). Lately, Orvieto et al. (2022) tackled the issues with unknown f ∗ i and developed a truly adaptive variant. In practice, the SPS method shows excellent empirical performance on overparameterized deep learning models (which satisfy the interpolation condition i.e. f ∗ i = 0 , ∀ i ∈ [ n ] ) (Loizou et al., 2021).

1.2 Main Contribution

Our main contributions are summarized below.

KATE : new scale-invariant version of AdaGrad . We propose a new method called KATE that can be seen as a version of AdaGrad , which does not use a square root in the denominator of the stepsize. To compensate for this change, we introduce a new sequence defining the numerator of the stepsize. We prove that KATE is scale-invariant for generalized linear models: if the starting point is zero, then the loss values (and training and test accuracies in the case of classification) at points generated by KATE are independent of the data scaling (Proposition 2.1), meaning that the speed of convergence of KATE is the same as for the best scaling of the data.

Convergence for smooth non-convex problems. We prove that for smooth non-convex problems with noise having bounded variance KATE has O ( log( T ) / √ T ) convergence rate (Theorem 3.4), matching the best-known rates for AdaGrad and Adam (Défossez et al., 2020).

Numerical experiments. We empirically illustrate the scale-invariance of KATE on the logistic regression task and test its performance on logistic regression (see Section 4.1), image classification, and text classification problems (see Section 4.2). In all the considered scenarios, KATE outperforms AdaGrad and works either better or comparable to Adam .

1.3 Notation

We denote the set { 1 , 2 , · · · , n } as [ n ] . For a vector a ∈ R d , a [ k ] is the k -th coordinate of a and a 2 represents the element-wise suqare of a , i.e., a 2 [ k ] = ( a [ k ]) 2 . For two vectors a and b , a b stands for element-wise division of a and b , i.e., k -th coordinate of a b is a [ k ] b [ k ] . Given a function h : R d → R , we use ∇ h ∈ R d to denote its gradient and ∇ k h to indicate the k -th component of ∇ h . Throughout the paper ∥ · ∥ represents the ℓ 2 -norm and f ∗ = inf w ∈ R d f ( w ) . Moreover, we use ∥ w ∥ A for a positivedefinite matrix A to define ∥ w ∥ A := √ w ⊤ Aw . Furthermore, E [ · ] denotes the total expectation while E t [ · ] denotes the conditional expectation conditioned on all iterates up to step t i.e. w 0 , w 1 , . . . , w t .

2 Motivation and Algorithm Design

We focus on solving the minimization problem (1) using a variant of AdaGrad . We aim to design an algorithm that performs well, irrespective of how poorly the data is scaled.

Generalized linear models. Here, we consider the parameter estimation problem in generalized linear models (GLMs) (Nelder and Wedderburn, 1972; Agresti, 2015) using maximum likelihood estimation. GLMs are an extension of linear models and encompass several other valuable models, such as logistic (Hosmer Jr et al., 2013) and Poisson regression (Frome, 1983), as special cases. The parameter estimation to fit GLM on dataset { x i , y i } n i =1 (where x i ∈ R d are feature vectors and y i are response variables) can be reformulated as

for differentiable functions φ i : R → R (Shalev-Shwartz and Ben-David, 2014; Nguyen et al., 2017b; Takáˇ c et al., 2013; He et al., 2018; Chezhegov et al., 2024). For example, the linear regression on data { x i , y i } n i =1 is equivalent to solving (4) with φ i ( z ) = ( z -y i ) 2 . Next, the choice of φ i for logistic regression is φ i ( z ) = log (1 + exp ( -y i z )) .

Scale-invariance. Now consider the instances of fitting GLMs on two datasets { x i , y i } n i =1 and { V x i , y i } n i =1 , where V ∈ R d × d is a diagonal matrix with positive entries. Note that the second dataset is a scaled version of the first one where the k -th component of feature vectors x i are multiplied by a scalar V kk . Then, the minimization problems corresponding to datasets { x i , y i } n i =1 and { V x i , y i } n i =1 are (4) and

respectively, for functions φ i . In this work, we want to design an algorithm with equivalent performance for the problems (4) and (5). If we can do that, the new algorithm's performance will not deteriorate for poorly scaled data, i.e., the method will be scale-invariant (Orabona et al., 2015), which is a special case of affine-invariance, see (Nesterov and Nemirovskii, 1994; Nesterov, 2018; d'Aspremont et al., 2018). To develop such an algorithm, we replace the denominator of AdaGrad step size with its square (remove the square root from the denominator), i.e., ∀ k ∈ [ d ]

for some m t ∈ R d . 3 The following proposition shows that this method (6) satisfies a scale-invariance property with respect to functional value.

Proposition 2.1 (Scale invariance) . Suppose we solve problems (4) and (5) using algorithm (6). Then, the iterates ˆ w t and ˆ w V t corresponding to (4) and (5) follow: ∀ k ∈ [ d ]

with g τ = φ ′ i τ ( x ⊤ i τ ˆ w τ ) x i τ and g V τ = φ ′ i τ ( x ⊤ i τ V ˆ w τ ) V x i τ for i τ chosen uniformly from [ n ] , τ = 0 , 1 , . . . , t , t ≥ 0 . Moreover, updates (7) and (8) satisfy

for all t ≥ 0 when ˆ w 0 = ˆ w V 0 = 0 ∈ R d . Furthermore we have

The Proposition 2.1 highlights that the update rule of the form (6) satisfies a scale-invariance property for GLMs. In contrast, AdaGrad does not satisfy (9) and (10). In Appendix C, we illustrate numerically the scale-invariance of KATE and the lack of the scale-invariance of AdaGrad . We also emphasize that AdaGrad with ∆ = 0 is known to be a scale-free method 4 .

Algorithm 1 KATE

Require: Initial point w 0 ∈ R d , step size β > 0 , η ∈ R d + and b -1 , m -1 = 0

1: for t = 0 , 1 , ..., T do

2: Compute g t ∈ R d such that E [ g t ] = ∇ f ( w t ) .

3: b 2 t = b 2 t -1 + g 2 t

5: w t +1 = w t -βm t b 2 t g t

Design of KATE . In order to construct an algorithm following the update rule (6), one may choose m t [ k ] = 1 ∀ k ∈ [ d ] . However, the step size from (6) in this case may decrease very fast, and the resulting method does not necessarily converge. Therefore, we need a more aggressive choice of m t , which grows with t . It motivates the construction of our algorithm KATE (Algorithm 1), 5 where we choose m t [ k ] = √ η [ k ] b 2 t [ k ] + ∑ t τ =0 g 2 τ [ k ] b 2 τ [ k ] . Note that the term ∑ t τ =0 g 2 τ [ k ] b 2 τ [ k ] is scale-invariant for GLMs (follows from Proposition 2.1). To make m t scale-invariant, we choose η ∈ R d in the following way:

η → 0 : When η is very small, m t is also approximately scale-invariant for GLMs.

η = 1 / ( ∇ f ( w 0 )) 2 : In this case ηb 2 t = b 2 t / ( ∇ f ( w 0 )) 2 is scale-invariant for GLMs (follows from Proposition 2.1) as well as m t .

KATE can be rewritten in the following coordinate form

where g t is an unbiased estimator of ∇ f ( w t ) and the per-coefficient step size ν t [ k ] is defined as

Note that the numerator of the steps ν t [ k ] is increasing with iterations t . However, one of the crucial properties of this step size choice is that the steps always decrease with t , which we rely on in our convergence analysis.

Lemma 2.2 (Decreasing step size) . For ν t [ k ] defined in (11) we have

Comparison with the scale-invariant version of AdaGrad by Orabona et al. (2015). In the special case of GLMs, Orabona et al. (2015) propose a different version of AdaGrad . The method is proposed for the case of online convex optimization, and in the case of standard optimization with GLMs (4), it has the following form

where { i τ } t τ =0 are arbitrary indices from [ n ] (e.g., selected uniformly at random), functions f i : R d → R are defined as f i ( w ) := φ i ( x ⊤ i w ) for i ∈ [ n ] , and γ is such that f i ( w ) is γ -Lipschitz for i ∈ [ n ] . In this setup, the update rule of KATE with w 0 = 0 can be written as follows:

introduced by Cesa-Bianchi et al. (2005, 2007) in the context of learning with expert advice and extended to the context of generic online convex optimization by Orabona and Pál (2015, 2018). We emphasize that scale-freeness and scale-invariance are completely different concepts.

.

where b t := √ ∑ t τ =0 ( ∇ f i τ ( w τ )) 2 , { i τ } t τ =0 are sampled from [ n ] uniformly at random. Although both methods can be seen as variations of AdaGrad due to the terms ∑ t τ =0 ( ∇ f i τ ( w τ / a τ ) 2 and ∑ t τ =0 ( ∇ f i τ ( w τ )) 2 respectively, the scale-invariance is achieved quite differently in these methods. The method from (14) uses the feature vectors explicitly in the update rule to ensure scale-invariance: indeed, the square root in the definition of w t +1 is independent of scaling, and a 2 t in the denominator ensures that ˆ w t +1 = V ˆ w V t +1 if we define them similarly to KATE (see equations (7)-(8)). In contrast, KATE achieves the scale-invariance by removing the square root from the denominator (as explained earlier). Moreover, unlike the method from (14), KATE does not use the feature vectors explicitly in its update rule (only in the gradients of f i τ ) and, thus, can be used for general stochastic optimization (not necessarily for the case of GLMs).

3 Convergence Analysis

In this section, we present and discuss the convergence guarantees of KATE . In the first subsection, we list the assumptions made about the problem.

3.1 Assumptions

In all our theoretical results, we assume that f is smooth as defined below.

Assumption 3.1 ( L -smooth) . Function f is L -smooth, i.e. for all w,w ′ ∈ R d

This assumption is standard in the literature of adaptive methods (Li and Orabona, 2019; Ward et al., 2020; Liu et al., 2022; Nguyen et al., 2018, 2021, 2017a; Beznosikov and Takáˇ c, 2021). Moreover, we assume that at any iteration t of KATE , we can access g t -anoisy and unbiased estimate of ∇ f ( w t ) . We also make the following assumption on the noise of the gradient estimate g t .

Assumption 3.2 (Bounded Variance) . For fixed constant σ > 0 , the variance of the stochastic gradient g t ( unbiased estimate of ∇ f ( w t )) at any time t satisfies

Bounded variance is a common assumption to study the convergence of stochastic gradient-based methods. Several assumptions on stochastic gradients are used in the literature to explore the adaptive methods. Ward et al. (2020) used the BV , while Liu et al. (2022) assumed the sub-Weibull noise, i.e. E [ exp( ∥ g t -∇ f ( w t ) ∥ / σ ) 1 / θ ] ≤ exp(1) for some θ > 0 , to prove the convergence of AdaGradNorm . Li and Orabona (2019) assumes sub-Gaussian ( θ = 1 / 2 in sub-Weibull condition) noise to study a variant of AdaGrad . However, sub-Gaussian noise is strictly stronger than BV . Recently, Faw et al. (2022) analyzed AdaGradNorm under a more relaxed condition known as affine variance ( i.e. E t [ ∥ g t -∇ f ( w t ) ∥ 2 ] ≤ σ 2 0 + σ 2 1 ∥∇ f ( w t ) ∥ 2 ) .

3.2 Main Results

In this section, we cover the main convergence guarantees of KATE for both deterministic and stochastic setups.

Deterministic setting. We first present our results for the deterministic setting. In this setting, we consider the gradient estimate to have no noise (i.e. σ 2 = 0 ) and g t = ∇ f ( w t ) . The main result in this setting is summarized below.

Theorem 3.3. Suppose f satisfy Assumption 3.1 and g t = ∇ f ( w t ) . Moreover, β > 0 and η [ k ] > 0 are chosen such that ν 0 [ k ] ≤ 1 L for all k ∈ [ d ] . Then the iterates of KATE satisfies

where η 0 := min k ∈ [ d ] η [ k ] .

Discussion on Theorem 3.3. Theorem 3.3 establishes an O ( 1 / T ) convergence rate for KATE , which is optimal for finding a first-order stationary point of a non-convex problem (Carmon et al., 2020). However, this result is not parameter-free. To prove the convergence, we assume that ν 0 [ k ] ≤ 1 L , ∀ k ∈ [ d ] in Theorem 3.3, which is equivalent to β √ 1 + η 0 ( ∇ k f ( w 0 )) 2 ≤ ( ∇ k f ( w 0 )) 2 / L , ∀ k ∈ [ d ] . Note that the later condition holds for sufficiently small (dependent on L ) values of β, η 0 > 0 .

However, it is possible to derive a parameter-free version of Theorem 3.3. Indeed, Lemma 2.2 implies that the step sizes are decreasing. Therefore, we can break down the analysis of KATE into two phases: Phase I when ν 0 [ k ] > 1 / L and Phase II when ν 0 [ k ] ≤ 1 / L , when the current analysis works, and then follow the proof techniques of Ward et al. (2020) and Xie et al. (2020). We leave this extension as a possible future direction of our work.

Stochastic setting. Next, we present the convergence guarantees for KATE in the stochastic case, when we can access an unbiased gradient estimate g t with non-zero noise.

Theorem 3.4. Suppose f satisfy Assumption 3.1 and g t is an unbiased estimator of ∇ f ( w t ) such that BV holds. Moreover, we assume ∥∇ f ( w t ) ∥ 2 ≤ γ 2 for all t . Then the iterates of KATE satisfy

where η 0 := min k ∈ [ d ] η [ k ] and

Comparison with prior work. Theorem 3.4 shows an O ( log 1 / 2 T / T 1 / 4 ) convergence rate for KATE with respect to the metric min t ≤ T E [ ∥∇ f ( w t ) ∥ ] for the stochastic setting. Note that, in the stochastic setting, KATE achieves a slower rate than Theorem 3.3 due to noise accumulation. Up to the logarithmic factor, this rate is optimal (Arjevani et al., 2023). Similar rates for the same metric follow from the results 6 of (Défossez et al., 2020) for AdaGrad and Adam .

Finally, Li and Orabona (2019) considers a variant of AdaGrad closely related to KATE :

for some ε ∈ [0 , 1 / 2 ) and ∆ > 0 . It differs from AdaGrad in two key aspects: the denominator of the stepsize does not contain the last stochastic gradient, and also, instead of the square root of the sum of squared gradients, this sum is taken in the power of 1 / 2 + ε . However, the results from Li and Orabona (2019) do not imply convergence for the case of ε = 1 / 2 , which is expected since, in this case, the stepsize converges to zero too quickly in general. To compensate for such a rapid decrease, in KATE , we introduce an increasing sequence m t in the numerator of the stepsize.

Proof technique. Compared to the AdaGrad , KATE uses more aggressive steps (the larger numerator of KATE due to the extra term ∑ t τ =0 g 2 τ [ k ] / b 2 τ [ k ] ). Therefore, we expect KATE to have better empirical performance. However, introducing ∑ t τ =0 g 2 τ [ k ] / b 2 τ [ k ] in the numerator raises additional technical difficulties in the proof technique. Fortunately, as we rigorously show, the KATE steps ν t [ k ] retain some of the critical properties of AdaGrad steps. For instance, they (i) are lower bounded by AdaGrad steps up to a constant, (ii) decrease with iteration t (Lemma 2.2), and (iii) have closed-form upper bounds for ∑ T t =0 ν 2 t [ k ] g 2 t [ k ] . These are indeed the primary building blocks of our proof technique.

4 Numerical Experiments

In this section, we implement KATE in several machine learning tasks to evaluate its performance. To ensure transparency and facilitate reproducibility, we provide an access to the source code for all of our experiments at https://github.com/nazya/KATE .

4.1 Logistic Regression

In this section, we consider the logistic regression model

to elaborate on the scale-invariance and robustness of KATE for various initializations. For the experiments of this Section 4.1, we used Mac mini (M1, 2020), RAM 8 GB and storage 256 GB. Each of these plots took about 20 minutes to run.

4.1.1 Robustness of KATE

To conduct this experiment, we set the total number of samples to 1000 (i.e. n = 1000 ). Here, we simulate the independent vectors x i ∈ R 20 such that each entry is from N (0 , 1) . Moreover, we generate a diagonal matrix V ∈ R 20 × 20 such that log V kk iid ∼ Unif ( -10 , 10) , ∀ k ∈ [20] . Similarly, we generate w ∗ ∈ R 20 with each component from N (0 , 1) and set the labels

We compare KATE 's performance with four other algorithms: AdaGrad , AdaGradNorm , SGDdecay and SGD-constant , similar to the section 5.1 of Ward et al. (2020). For each algorithm, we initialize with w 0 = 0 ∈ R 20 and independently draw a sample of mini-batch size 10 to update the weight vector w t . We compare the algorithms · AdaGrad with stepsize β √ ∆+ ∑ t τ =0 g 2 τ , · AdaGradNorm with step size β √ ∆+ ∑ t τ =0 ∥ g τ ∥ 2 , · SGD-decay with stepsize β / ∆ √ t +1 , and · SGD-constant with step size β / ∆ . Similarly, for KATE we use stepsize βm t b 2 t where m 2 t = ηb 2 t + ∑ t τ =0 g 2 τ / b 2 τ and b 2 t = ∆ + ∑ t τ =0 g 2 τ . Here, we choose β = f ( w 0 ) -f ( w ∗ ) and vary ∆ in { 10 -8 , 10 -6 , 10 -4 , 10 -2 , 1 , 10 2 , 10 4 , 10 6 , 10 8 } .

In Figures 1a, 1b, and 1c, we plot the functional value f ( w t ) (on the y -axis) after 10 4 , 5 × 10 4 , and 10 5 iterations, respectively. In theory, the convergence of SGD requires the knowledge of smoothness constant L . Therefore, when the ∆ is small (hence the stepsize is large), SGD-decay and SGD-constant diverge. However, the adaptive algorithms KATE , AdaGrad , and AdaGradNorm can auto-tune themselves and converge for a wide range of ∆ s (even when the ∆ is too small). As we observe in Figure 1, when the ∆ is small, KATE outperforms all other algorithms. For instance, when ∆ = 10 -8 , KATE achieves a functional value of 10 -3 after only 10 4 iterations (see Figure 1a), while other algorithms fail to achieve this even after 10 5 iterations (see Figure 1c). Furthermore, KATE performs as well as AdaGrad and better than other algorithms when the ∆ is large. In particular, this experiment highlights that KATE is robust to initialization ∆ .

4.1.2 Peformance of KATE on Real Data

In this section, we examine KATE 's performance on real data. We test KATE on three datasets: heart, australian, and splice from the LIBSVM library (Chang and Lin, 2011). The response variables y i of

10

7

10

5

10

3

10

1

10 1

10 3

10 5

10 7

10

3

10

2

10

1

f

(

)

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(a) Plot of

f

(

w

t

)

10

7

10

5

10

3

10

1

10 1

10 3

10 5

10 7

10

3

10

2

10

1

f

(

)

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(b) Plot of Accuracy

10

7

10

5

10

3

10

1

10 1

10 3

10 5

10 7

10

3

10

2

10

1

f

(

)

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(c) Plot of Gradient Norm

0

1000

2000

3000

4000

iterations

0.40

0.45

0.50

0.55

0.60

0.65

0.70

f

(

)

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(a) Dataset: heart

0

1000

2000

3000

4000

iterations

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

f

(

)

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(b) Dataset: australian

0

1000

2000

3000

4000

iterations

0.4

0.5

0.6

0.7

0.8

f

(

)

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(c) Dataset: splice

0

1000

2000

3000

4000

iterations

0.0

0.2

0.4

0.6

0.8

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(d) Dataset: heart

0

1000

2000

3000

4000

iterations

0.0

0.2

0.4

0.6

0.8

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(e) Dataset: australian

0

1000

2000

3000

4000

iterations

0.0

0.2

0.4

0.6

0.8

KATE

AdaGrad

AdaGradNorm

SGD-constant

SGD-decay

(f) Dataset: splice

each of these datasets contain two classes, and we use them for binary classification tasks using a logistic regression model (17). We take η = 1 / ( ∇ f ( w 0 )) 2 for KATE and tune β in all the experiments. For tuning β , we do a grid search on the list { 10 -10 , 10 -8 , 10 -6 , 10 -4 , 10 -2 , 1 } . Similarly, we tune stepsizes for other algorithms. We take 5 trials for each of these algorithms and plot the mean of their trajectories.

We plot the functional value f ( w t ) (i.e. loss function) in Figures 2a, 2b and 2c, whereas Figures 2d, 2e and 2f plot the corresponding accuracy of the weight vector w t on the y -axis for 5 , 000 iterations. We observe that KATE performs superior to all other algorithms, even on real datasets.

4.2 Training of Neural Networks

In this section, we compare the performance of KATE , AdaGrad and Adam on two tasks, i.e. training ResNet18 (He et al., 2016) on the CIFAR10 dataset (Krizhevsky and Hinton, 2009) and BERT (Devlin et al., 2018) fine-tuning on the emotions dataset (Saravia et al., 2018) from the Hugging Face Hub. We use internal cluster with the following hardware: AMD EPYC 7552 48-Core Processor, 512GiB RAM, NVIDIA A100 40GB GPU, 200gb user storage space.

General comparison. We choose standard parameters for Adam ( β 1 = 0 . 9 and β 2 = 0 . 999 ) that are default values in PyTorch and select the learning rate of 10 -5 for all considered methods. We run KATE with different values of η ∈ { 0 , 10 -1 , 10 -2 } . For the image classification task, we normalize the images (similar to Horváth and Richtárik (2020)) and use a mini-batch size of 500. For the BERT fine-tuning, we use a mini-batch size 160 for all methods.

Figures 3-8 report the evolution of top-1 accuracy and cross-entropy loss (on the y -axis) calculated on the test data. For the image classification task, we observe that KATE with different choices of

0

1000

2000

3000

4000

5000

Iterations

10

20

30

40

50

60

70

ADAM

=1.0e-04

ADAM

=1.0e-06

ADAM

=1.0e-08

ADAM

=1.0e-10

AdaGrad

=1.0e-04

AdaGrad

=1.0e-06

AdaGrad

=1.0e-08

AdaGrad

=1.0e-10

KATE

=1.0e-06

KATE

=1.0e-08

KATE

=1.0e-10

0

1000

2000

3000

4000

5000

Iterations

10 0

CrossEntropyLoss

0

1000

2000

3000

4000

5000

Iterations

10

20

30

40

50

60

70

0

1000

2000

3000

4000

5000

Iterations

10 0

CrossEntropyLoss

Figure 4:

CIFAR10:

η

=

0

.

001

0

1000

2000

3000

4000

5000

Iterations

10

20

30

40

50

60

70

0

1000

2000

3000

4000

5000

Iterations

10 0

CrossEntropyLoss

Figure 5:

CIFAR10:

η

=

0

.

1

0

250

500

750

1000

1250

1500

1750

2000

Iterations

30

40

50

60

70

80

90

ADAM

=1.0e-04

ADAM

=1.0e-06

ADAM

=1.0e-08

ADAM

=1.0e-10

AdaGrad

=1.0e-04

AdaGrad

=1.0e-06

AdaGrad

=1.0e-08

AdaGrad

=1.0e-10

KATE

=1.0e-04

KATE

=1.0e-05

0

250

500

750

1000

1250

1500

1750

2000

Iterations

10 0

CrossEntropyLoss

Figure 6: Emotion:

η

= 0

0

250

500

750

1000

1250

1500

1750

2000

Iterations

30

40

50

60

70

80

90

0

250

500

750

1000

1250

1500

1750

2000

Iterations

10 0

CrossEntropyLoss

Figure

7:

Emotion:

η

=

0

250

500

750

1000

1250

1500

1750

2000

Iterations

30

40

50

60

70

80

90

0

250

500

750

1000

1250

1500

1750

2000

Iterations

10 0

CrossEntropyLoss

Figure 8: Emotion:

η

= 0

.

1

0

.

001

Hyper-parameters tuning. Next, we compare baselines presented in Saravia et al. (2018) for emotions classification and Zhang et al. (2019) for image classification. These papers provide efficient setups for learning rates and learning rate schedulers that are reasonable to compare with. Saravia et al. (2018) performs a search of efficient learning rate and uses a linear learning rate scheduler with warmup for Adam optimizer. A different learning rate (1e-5), ∆ =1e-5 and the same scheduler applied for KATE lead to the same performance, see Figure 9. We would like to point out that it is challenging to find a reference for hyper-parameters for a certain setup. Thus, to fairly compare with Saravia et al. (2018) we use distilrobertabase model. Zhang et al. (2019) did a grid search for an efficient learning rate and used a multi-step scheduler for Adam optimizer, decaying the learning rate by a factor of 5 at the 60th, 120th, and 160th epochs. Zhang et al. (2019) refers to DeVries and Taylor (2017) for the code implementing special techniques, namely data augmentation and cutout to achieve higher accuracy. A different learning rate (1e-3), the same scheduler and ∆ =1e-3 applied for KATE demonstrates comparable performance, see Figure 10. For

0

2500

5000

7500

10000

12500

15000

17500

20000

Iterations

10

20

30

40

50

60

70

80

90

Test accuracy

Cifar10+ResNet18

ADAM

KATE

0

2

4

6

8

10

Epochs

30

40

50

60

70

80

90

Test accuracy

Emotion+RoBERTa

ADAM

KATE

Figure 10: Emotion: η = 0 . 001 BERT each run takes about 20 minutes, while 100 minutes for ResNet.

Acknowledgments and Disclosure of Funding

We thank Francesco Orabona and Dmitry Kamzolov for the pointers to the related works that we missed while preparing the first version of this paper. We also thank anonymous reviewers for their useful feedback and suggestions.

References

Abdukhakimov, F., Xiang, C., Kamzolov, D., Gower, R., and Takáˇ c, M. (2023). Sania: Polyak-type optimization framework leads to scale invariant stochastic algorithms. arXiv preprint arXiv:2312.17369 .

Abdukhakimov, F., Xiang, C., Kamzolov, D., and Takáˇ c, M. (2024). Stochastic gradient descent with preconditioned polyak step-size. Computational Mathematics and Mathematical Physics , 64(4):621-634.

Agresti, A. (2015). Foundations of linear and generalized linear models . John Wiley & Sons.

Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. (2023). Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199(1-2):165-214.

Beznosikov, A. and Takáˇ c, M. (2021). Random-reshuffled sarah does not need a full gradient computations. In Optimization for Machine Learning Workshop @ NeurIPS 2021 .

Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points i. Mathematical Programming , 184(1-2):71-120.

Cesa-Bianchi, N., Mansour, Y., and Stoltz, G. (2005). Improved second-order bounds for prediction with expert advice. In International Conference on Computational Learning Theory , pages 217-232. Springer.

Cesa-Bianchi, N., Mansour, Y., and Stoltz, G. (2007). Improved second-order bounds for prediction with expert advice. Machine Learning , 66:321-352.

Chang, C.-C. and Lin, C.-J. (2011). Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27.

Chezhegov, S., Skorik, S., Khachaturov, N., Shalagin, D., Avetisyan, A., Beznosikov, A., Takáˇ c, M., Kholodov, Y., and Gasnikov, A. (2024). Local methods with adaptivity via scaling. arXiv preprint arXiv:2406.00846 .

d'Aspremont, A., Guzman, C., and Jaggi, M. (2018). Optimal affine-invariant smooth minimization algorithms. SIAM Journal on Optimization , 28(3):2384-2405.

Defazio, A. and Mishchenko, K. (2023). Learning-rate-free learning by D-adaptation. arXiv preprint arXiv:2301.07733 .

Défossez, A., Bottou, L., Bach, F., and Usunier, N. (2020). A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395 .

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .

DeVries, T. and Taylor, G. W. (2017). Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 .

D'Orazio, R., Loizou, N., Laradji, I., and Mitliagkas, I. (2021). Stochastic mirror descent: Convergence analysis and adaptive variants via the mirror stochastic polyak stepsize. arXiv preprint arXiv:2110.15412 .

Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7).

Faw, M., Tziotis, I., Caramanis, C., Mokhtari, A., Shakkottai, S., and Ward, R. (2022). The power of adaptivity in sgd: Self-tuning step sizes with unbounded gradients and affine variance. In Conference on Learning Theory , pages 313-355. PMLR.

Frome, E. L. (1983). The analysis of rates using poisson regression models. Biometrics , pages 665-674.

Gower, R. M., Defazio, A., and Rabbat, M. (2021). Stochastic polyak stepsize with a moving target. arXiv preprint arXiv:2106.11851 .

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778.

He, X., Tappenden, R., and Takac, M. (2018). Dual free adaptive minibatch sdca for empirical risk minimization. Frontiers in Applied Mathematics and Statistics , 4:33.

Horváth, S. and Richtárik, P. (2020). A better alternative to error feedback for communication-efficient distributed learning. arXiv preprint arXiv:2006.11077 .

Hosmer Jr, D. W., Lemeshow, S., and Sturdivant, R. X. (2013). Applied logistic regression , volume 398. John Wiley & Sons.

Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images.

Li, S., Swartworth, W. J., Takáˇ c, M., Needell, D., and Gower, R. M. (2023). Sp2: A second order stochastic polyak method. ICLR .

Li, X. and Orabona, F. (2019). On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd international conference on artificial intelligence and statistics , pages 983-992. PMLR.

Liu, Z., Nguyen, T. D., Ene, A., and Nguyen, H. L. (2022). On the convergence of adagrad on R d : Beyond convexity, non-asymptotic rate and acceleration. arXiv preprint arXiv:2209.14827 .

Loizou, N., Vaswani, S., Laradji, I. H., and Lacoste-Julien, S. (2021). Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics , pages 1306-1314. PMLR.

McMahan, H. B. and Streeter, M. (2010). Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908 .

Mishchenko, K. and Defazio, A. (2023). Prodigy: An expeditiously adaptive parameter-free learner. arXiv preprint arXiv:2306.06101 .

Nelder, J. A. and Wedderburn, R. W. (1972). Generalized linear models. Journal of the Royal Statistical Society Series A: Statistics in Society , 135(3):370-384.

Nesterov, Y. (2018). Lectures on convex optimization , volume 137. Springer.

Nesterov, Y. and Nemirovskii, A. (1994). Interior-point polynomial algorithms in convex programming . SIAM.

Nguyen, L., Liu, J., Scheinberg, K., and Takáˇ c, M. (2017a). Sarah: A novel method for machine learning problems using stochastic recursive gradient. In In 34th International Conference on Machine Learning, ICML 2017 .

Nguyen, L. M., Liu, J., Scheinberg, K., and Takáˇ c, M. (2017b). Stochastic recursive gradient algorithm for nonconvex optimization. arXiv preprint arXiv:1705.07261 .

Nguyen, L. M., Nguyen, P. H., van Dijk, M., Richtárik, P., Scheinberg, K., and Takáˇ c, M. (2018). Sgd and hogwild! convergence without the bounded gradients assumption. In In 34th International Conference on Machine Learning, ICML 2018 .

Nguyen, L. M., Scheinberg, K., and Takáˇ c, M. (2021). Inexact sarah algorithm for stochastic optimization. Optimization Methods and Software , 36(1):237-258.

Oberman, A. M. and Prazeres, M. (2019). Stochastic gradient descent with polyak's learning rate. arXiv preprint arXiv:1903.08688 .

Orabona, F., Crammer, K., and Cesa-Bianchi, N. (2015). A generalized online mirror descent with applications to classification and regression. Machine Learning , 99:411-435.

Orabona, F. and Pál, D. (2015). Scale-free algorithms for online linear optimization. In International Conference on Algorithmic Learning Theory , pages 287-301. Springer.

Orabona, F. and Pál, D. (2018). Scale-free online learning. Theoretical Computer Science , 716:50-69.

Orvieto, A., Lacoste-Julien, S., and Loizou, N. (2022). Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. Advances in Neural Information Processing Systems , 35:26943-26954.

Polyak, B. T. (1969). Minimization of unsmooth functionals. USSR Computational Mathematics and Mathematical Physics , 9(3):14-29.

Reddi, S. J., Kale, S., and Kumar, S. (2019). On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237 .

Robbins, H. and Monro, S. (1951). A stochastic approximation method. The annals of mathematical statistics , pages 400-407.

Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., and Chen, Y.-S. (2018). CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3687-3697, Brussels, Belgium. Association for Computational Linguistics.

Shalev-Shwartz, S. and Ben-David, S. (2014). Understanding machine learning: From theory to algorithms . Cambridge university press.

Shi, Z., Sadiev, A., Loizou, N., Richtárik, P., and Takáˇ c, M. (2023). AI-SARAH: Adaptive and implicit stochastic recursive gradient methods. Transactions on Machine Learning Research .

Takáˇ c, M., Bijral, A., Richtárik, P., and Srebro, N. (2013). Mini-batch primal and dual methods for svms. In In 30th International Conference on Machine Learning, ICML 2013 .

Tieleman, T. and Hinton, G. (2012). Rmsprop: Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. COURSERA Neural Networks Mach. Learn , 17.

Ward, R., Wu, X., and Bottou, L. (2020). Adagrad stepsizes: Sharp convergence over nonconvex landscapes. The Journal of Machine Learning Research , 21(1):9047-9076.

Xie, Y., Wu, X., and Ward, R. (2020). Linear convergence of adaptive stochastic gradient descent. In International conference on artificial intelligence and statistics , pages 1475-1485. PMLR.

Zhang, M., Lucas, J., Ba, J., and Hinton, G. E. (2019). Lookahead optimizer: k steps forward, 1 step back. Advances in neural information processing systems , 32.

Supplementary Material

Contents

A Technical Lemmas

Lemma A.1 (AM-GM) . For λ > 0 we have

Lemma A.2 (Cauchy-Schwarz Inequality) . For a 1 , · · · , a n , b 1 , · · · , b n ∈ R we have

Lemma A.3 (Holder's Inequality) . Suppose X,Y are two random variables and p, q > 1 satisfy 1 p + 1 q = 1 .

Then

Lemma A.4 (Jensen's Inequality) . For a convex function g : R d → R and a random variable X such that E (Ψ( X )) and Ψ( E ( X )) are finite, we have

Lemma A.5. For a 1 , a 2 , · · · , a n ≥ 0 and b 1 , b 2 , · · · , b n > 0 we have

Proof. Expanding the LHS of (22) we get

̸

The last inequality follows from a i √ b i ≥ 0 for all i ∈ [ n ] . Now, using Cauchy-Schwarz Inequality (19), we have

Then combining (23) and (24), we get

Finally dividing both sides by ∑ n b i and taking square root we get the desired result.

i =1

Proof. Note that, using ν t [ k ] ≥ β √ η [ k ] b t [ k ] we have

Note that the second last inequality follows from the use of triangle inequality in the following way

while the last inequality follows from

Then from (26) we have

For term I in (27), we use Lemma A.1 with

to get

The last inequality follows from BV . Similarly, we again use Lemma A.1 with

Therefore using (28) and (29) in (28) we get

This completes the proof of this Lemma.

Lemma A.7.

Proof. Using b 2 t [ k ] = ∑ t τ =0 g 2 τ [ k ] we have

The inequality follows from the fact 1 b 2 t [ k ] ≤ 1 z when b 2 t -1 [ k ] ≤ z ≤ b 2 t [ k ] . This completes the proof of the Lemma.

B Proof of Main Results

B.1 Proof of Proposition 2.1

Proposition B.1 (Scale invariance) . Suppose we solve problems (4) and (5) using algorithm (6). Then, the iterates ˆ w t and ˆ w V t corresponding to (4) and (5) follow: ∀ k ∈ [ d ]

with g τ = φ ′ i τ ( x ⊤ i τ ˆ w τ ) x i τ and g V τ = φ ′ i τ ( x ⊤ i τ V ˆ w τ ) V x i τ for i τ chosen uniformly from [ n ] , τ = 0 , 1 , . . . , t , t ≥ 0 . Moreover, updates (31) and (32) satisfy

for all t ≥ 0 when ˆ w 0 = ˆ w V 0 = 0 ∈ R d . Furthermore we have

Proof. First, we will show ˆ w t = V ˆ w V t and V g t = g V t using induction. Note that for τ = 1 and k ∈ [ d ] , we get

as ˆ w 0 = ˆ w V 0 = 0 . Therefore, we have ∀ k ∈ [ d ] , ˆ w 1 [ k ] = V kk ˆ w V 1 [ k ] . This can be equivalently written as ˆ w 1 = V ˆ w V 1 , as V is a diagonal matrix. Then it is easy to check

where the second equality follows from ˆ w 1 = V ˆ w V 1 . Now, we assume the proposition holds for τ = 1 , · · · , t . Then, we need to prove this proposition for τ = t +1 . Note that, from (7) we have

Here, the second last equality follows from ˆ w τ = V ˆ w V τ and V g τ = g V τ ∀ τ ∈ [ t ] , while the last equality holds due to (32). Therefore, we have ˆ w t +1 = V ˆ w V t +1 . Then similar to (34) we get V g t +1 = g V t +1 using ˆ w t +1 = V ˆ w V t +1 . Again, using ˆ w t = V ˆ w V t , we can rewrite f ( ˆ w t ) as follow

The last equality follows from (5). This proves f ( ˆ w t ) = f V ( ˆ w V t ) . Finally using V g t = g V t we get

This completes the proof of Proposition 2.1.

B.2 Proof of Lemma 2.2

Lemma B.2 (Decreasing step size) . For ν t [ k ] defined in (11) we have

Proof. We want to show that ν t +1 [ k ] ≤ ν t [ k ] . Taking square and rearranging the terms (13) is equivalent to proving

Using the expansion of m 2 t +1 [ k ] , b 2 t +1 [ k ] , LHS of (35) can be expanded as follow

Similarly, the RHS of (35) can be expanded to

Therefore using (36) and (37), inequality (35) is equivalent to

Now subtracting m 2 t [ k ] b 4 t [ k ] from both sides of (38) and then multiplying both sides by b 2 t [ k ] + g 2 t +1 [ k ] , (38) is equivalent to

Therefore, proving (13) is equivalent to proving (39). Note that, from the expansion m 2 t [ k ] = η [ k ] b 2 t [ k ] + ∑ t τ =0 g 2 t [ k ] b 2 t [ k ] , we have m 2 t [ k ] ≥ g 2 0 [ k ] b 2 0 [ k ] = 1 and m 2 t [ k ] ≥ η [ k ] b 2 t [ k ] . Then using m 2 t [ k ] ≥ 1 we get

Again, using m 2 t [ k ] ≥ η [ k ] b 2 t [ k ] , we have

Then adding (40) and (41) we get

η [ k ] g 2 t +1 [ k ] b 6 t [ k ] + η [ k ] g 4 t +1 [ k ] b 4 t [ k ] + g 2 t +1 [ k ] b 4 t [ k ] ≤ m 2 t [ k ] g 4 t +1 [ k ] b 2 t [ k ] + 2 m 2 t [ k ] g 2 t +1 [ k ] b 4 t [ k ] . (42) Therefore, (39) is true due to (42) and m 2 t [ k ] g 6 t +1 [ k ] + 2 m 2 t [ k ] g 4 t +1 [ k ] b 2 t [ k ] ≥ 0 . This completes the proof of the Lemma.

B.3 Proof of Theorem 3.3

Theorem B.3. Suppose f is L -smooth, g t = ∇ f ( w t ) and η, β are chosen such that ν 0 [ k ] ≤ 1 L for all k ∈ [ d ] . Then for (11) we have

Proof. Suppose g t = ∇ f ( w t ) . Then using the smoothness of f we get

Then using this bound recursively we get

Note that, we initialized KATE such that ν 0 [ k ] ≤ 1 L ∀ k ∈ [ d ] . Therefore using Lemma 2.2 we have ν t [ k ] ≤ 1 L , which is equivalent to 1 -ν t [ k ] L 2 ≥ 1 2 for all k ∈ [ d ] . Hence from (43) we have

Then rearranging the terms and using f ( w T +1 ) ≥ f ∗ we get

Then from (43) and m t [ k ] ≥ √ η 0 b t [ k ] we get

Now from the definition of b 2 t [ k ] , we have b 2 t [ k ] = b 2 t -1 [ k ] + g 2 t [ k ] . This can be rearranged to get

Here the last inequality (46) follows from recursive use of (45). Then, taking squares on both sides and summing over k ∈ [ d ] we get

The second inequality follows from b 0 [ k ] + ∑ T t =0 g 2 t [ k ] b t [ k ] ≥ 0 for all k ∈ [ d ] and the last inequality from (44). Now note that ∑ T t =0 ∥ g t ∥ 2 = ∑ T t =0 ∑ d k =1 g 2 t [ k ] = ∑ d k =1 b 2 t [ k ] . Therefore dividing both sides of (47) by T +1 , we get

This completes the proof of the theorem.

B.4 Proof of Theorem 3.4

Theorem B.4. Suppose f is a L -smooth function and g t is an unbiased estimator of ∇ f ( w t ) such that BV holds. Moreover, we assume ∥∇ f ( w t ) ∥ 2 ≤ γ 2 for all t . Then KATE satisfies

where

Proof. Using smoothness, we have

Then, taking the expectation conditioned on w t , we have

The second last equality follows from E t [ ∇ k f ( w t ) ( ∇ k f ( w t ) -g t [ k ])] = ∇ k f ( w t ) ( ∇ k f ( w t ) -E t [ g t [ k ]]) = 0 . Now we use (25) to get

Then rearranging the terms we have

Now we take the total expectations to derive

The above inequality holds for any t . Therefore summing up from t = 0 to t = T and using f ( w T +1 ) ≥ f ∗ we get

Note that, using the expansion of ν 2 t [ k ] we have

Here (49) follows from b 2 j [ k ] ≥ b 2 0 [ k ] and (50) from b 2 t [ k ] = ∑ t j =0 g 2 j [ k ] . Then using (50) in (48) we derive

Here the last inequality follows from (30). Now using Jensen's Inequality (21) with Ψ( z ) = log( z ) we have

Here the RHS is exactly C f . Using (22) we have

Therefore using (52) in (51) we arrive at

Now we use Holder's Inequality (20) E ( XY ) ( E | Y | 3 ) 1 3 ≤ ( E | X | 3 2 ) 2 3 with

to get a lower bound on LHS of (53):

Therefore from (53) and (54) we get

Then multiplying both sides by ∥ b 0 ∥ + 2 T ( γ + dσ ) T we have

Here we use E [ ∥∇ f ( w t ) ∥ ] 4 3 ≤ E [ ∥∇ f ( w t ) ∥ 4 3 ] (follows from Jensen's Inequality (21) with Ψ( z ) = z 4 / 3 ) in the above equation to get

This completes the proof of the Theorem.

C Additional Experiments: Scale-Invariance Verification

In this experiment, we implement KATE on problems (4) (for unscaled data) and (5) (for scaled data) with

We generate the data similar to Section 4.1.1. We run KATE for 10 , 000 iterations with mini-batch size 10 , η = 1 / ( ∇ f ( w 0 )) 2 and plot functional value f ( w t ) and accuracy in Figures 11a and 11b. We use the proportion of correctly classified samples to compute accuracy, i.e. 1 n ∑ n i =1 1 { y i x ⊤ i w t ≥ 0 } .

In plots 11a and 11b, the functional value and accuracy of KATE coincide, which aligns with our theoretical findings (Proposition 2.1). Figure 11c plots ∥∇ f ( w t ) ∥ 2 and ∥∇ f ( w t ) ∥ 2 V -2 for unscaled and scaled data respectively. Here, (10) explains the identical values taken by the corresponding gradient norms of KATE iterates for the scaled and unscaled data. Similarly, in Figure 12, we compare the performance of AdaGrad on scaled and un-scaled data. This figure illustrates the lack of the scale-invariance for AdaGrad .

0

2000

4000

6000

8000

10000

iterations

0.0

0.5

1.0

1.5

2.0

2.5

3.0

f

(

)

Dataset: (

xi

,

yi

)

Dataset: (

Vxi

,

yi

)

(a) Plot of

f

(

w

t

)

0

2000

4000

6000

8000

10000

iterations

0.5

0.6

0.7

0.8

0.9

1.0

Dataset: (

xi

,

yi

)

Dataset: (

Vxi

,

yi

)

(b) Plot of Accuracy

0

2000

4000

6000

8000

10000

iterations

10

4

10

3

10

2

f

(

)

2 ; Dataset: (

xi

,

yi

)

f

(

)

2

V

2

; Dataset: (

Vxi

,

yi

)

(c) Plot of Gradient Norm

(a) Plot of

f

(

w

t

)

(b) Plot of Accuracy

(c) Plot of Gradient Norm

AdaGrad

AdaGrad

AdaGrad

0.7

1.0

10-2

0.6

0.9

0.5

103

(M）

0.4

Dataset:(xi,yi)

0.8

Vf(w)²;Dataset:(xi,yi)

0.3

Dataset:(Vxi,yi)

10-4

Vf(wt)-2;Dataset:(Vxi，yi)

0.7

0.2

Dataset:(x,yi)

0.1

0.6

Dataset:(Vx,y)

105

0

2000

4000

6000

8000

10000

0

2000

4000

6000

8000

10000

0

2000

4000

6000

8000

10000

iterations

iterations

iterations

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: the new method and its scale invariance property are introduced in Section 2, main convergence results are provided in Section 3, and the numerical results are provided in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: see Section 3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: see Section 3 and the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: see Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: see Section 4.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: see Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: the results are consistent for different runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have added all the details in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: the paper follows NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: the paper is mostly theoretical and does not have a direct societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: we do not release data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: see Section 4.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: not applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper presents a novel algorithm, KATE, which demonstrates impressive scale-invariance properties for Generalized Linear Models.

STRENGTHS:
This paper presents a novel algorithm, KATE, which demonstrates impressive scale-invariance properties for Generalized Linear Models. The thorough theoretical analysis provided in the paper, along with the experimental results, showcases the effectiveness of KATE in various machine learning tasks. The comparison with existing algorithms like AdaGrad and Adam highlights the superior performance of KATE in terms of convergence rate and efficiency.

WEAKNESSES:
While the paper excels in presenting the theoretical foundations and empirical results of the KATE algorithm, there are a few areas that could be further strengthened. Firstly, the paper could benefit from a more detailed discussion on the limitations of the proposed algorithm, especially in scenarios where certain assumptions may not hold. Additionally, providing insights into the computational efficiency and scalability of KATE with larger datasets could enhance the practical applicability of the algorithm.

QUESTIONS:
1 How does the scale-invariance property of KATE impact its performance in real-world applications compared to traditional adaptive algorithms?

2 Can the authors elaborate on the computational complexity of KATE and how it scales with the size of the dataset?

3 Are there any specific scenarios or types of machine learning tasks where KATE may not perform as effectively, and if so, how does the algorithm address these limitations?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an optimizer that achieves the optimal convergence guarantee for smooth nonconvex settings and more importantly the scale-invariance property.\n\nSTRENGTHS:\nThe paper is very cleanly written. It was very easy to follow.\nThe main results look sound, and the experimental results are well presented as well.\n\nWEAKNESSES:\nOverall, the paper presents the main scope and the results very well.\nIn other words, I do not have much concern for this paper by itself. \n\nMy only concern is the importance of the problem it tackles. Scale-invariance is definitely a desirable property, but I'm not sure what advances it brings about for the ML optimization. What I mean is, in general, the important question in the community is whether one can design an optimizer that has a noticeable advantage over the previous popular ones. I'm not sure whether resolving scale-invariance will drastically improve our current technology for optimizing ML models. The experiments presented in this paper doesn't seem to justify this in a compelling way. (I think results are convex models have limited practical impact.)\n\nQUESTIONS:\n- I think since a lot of baselines this paper considers is originally designed for online convex optimization, it is important to also do an extensive analysis of KATE for the corresponding OCO settings and compare. In particular, some regret analysis as well as the convergence analysis for nonsmooth convex setting might be helpful.\n\n\n- The main motivation this paper argues for the need of scale-invariance under the data scaling is that previous approaches could be brittle when data has poor-scaling or ill-conditioning. In order to make a case that this is an important question to solve, I'd like to see some practically relevant scenarios where the lack of good scaling of data leads to failure of non-scale-invariance approaches like Adam and Adagrad. I think for this paper to be have a bigger impact, a compelling set of experiments along this line seems to be necessary.\n\n- Do you expect poor data scaling to be the major issue in the training large AI models such as LLMs? If that's the case, I think this paper might have a bigger impact.\n\nLIMITATIONS:\nAs I said, the reason for the current score is mainly due to the main scope of this paper.\nIn my opinion, unless the authors have compelling experimental results or arguments, tacking the scale-invarance seems to have limited practical impact in the community.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel optimization algorithm which demonstrate scale-invariance property for generlized linear models unlike Adagrad. The authors analyzed KATE for smooth and non-convex functions and on generalized linear models to obtain the same convergence upper bounds (asymptotically) as Adagrad and Adam. However the scale invariant algorithm implies, the speed of the algorithm would be the same as for the best scaling of data. The authors also empirically verify this via numerical experiments on logistic regression and deep learning tasks.\n\nSTRENGTHS:\n1) The authors come up with KATE which is scale invariant, and yet admit a convergence upper bound of $\\mathcal{O}(\\log(T)/\\sqrt{T})$, which is an interesting algorithmic finding.\n2) The paper is easy to read and understand. \n3) Comprehensive appendix.\n\nWEAKNESSES:\nI think authors need to emphasize the first point in Strengths by comparing with the diagonal online-newton method (diag-SONew) [1] which is also scale-invariant algorithm and what part of the analysis can fail for diag-SONew. I wrote the algorithm without the EMA here, which is essentially adagrad without the square root but with varying $\\eta_t$:\n\n$$\nw_t := w_{t-1} -\\eta_t g_t /({\\sum_{s=1}^t g_s* g_s})\n$$\n\n schedules one can try is $\\eta_t=1$ or $\\eta_t=\\sqrt{t}$. I think comparing with the latter schedule will give a good understanding of the novelty behind KATE, as the schedule simulates the square root in Adagrad while being scale-invariant. Similarly empirical comparisons (if possible) with this simple algorithm in logistic-regression can help understand which algorithm is better.\n\n2) Comparison with Adam doesn’t make sense in neural networks as KATE lacks momentum and EMA in second-moments (which are key features of Adam). Devising KATE with these features similar to [2] would help the empirical performance in neural networks.\n\n[1] Devvrit, Fnu, Sai Surya Duvvuri, Rohan Anil, Vineet Gupta, Cho-Jui Hsieh, and Inderjit Dhillon. \"A computationally efficient sparsified online newton method.\" Advances in Neural Information Processing Systems 36 (2024).\n[2] Defazio, A., & Mishchenko, K. (2023, July). Learning-rate-free learning by d-adaptation. In International Conference on Machine Learning (pp. 7449-7479). PMLR.\n\nQUESTIONS:\nIn line 80-81  “meaning that the speed of convergence of KATE is the same as for the best scaling of the data.”, is this reflected in the convergence bound? i.e., whether scale dependence of Adagrad bound vs scale independence of KATE bound is emphasized. The Table 1 only mention asymptotic bounds for Adagrad. It would help the paper if the convergence bounds are analyzed for the generalized linear models for Adagrad and KATE to understand the affect of scale on the final upperbounds.\n\nLIMITATIONS:\nAuthors have adequately addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes a scale-invariant variant of AdaGrad, called KATE, particularly for generalized linear models. Theoretically, the authors proved a convergence rate of $\mathcal{O}(\log T/\sqrt{T})$ for KATE, matching the best known rates for AdaGrad and Adam. Numerical experiments are used to illustrate KATE on several machine learning tasks, which outperforms AdaGrad consistently and matches/outperforms Adam.

STRENGTHS:
This work studies a crucial problem of developing a scale-invariant variant of AdaGrad which is particularly useful whenever data exhibit poor scaling or ill-conditioning. Convergence rates of KATE similar to those of AdaGrad and Adam under both deterministic and stochastic settings are established, with comprehensive numerical experiments to justify the efficacy of KATE compared to AdaGrad and Adam.

WEAKNESSES:
As the motivation of KATE is to develop a scale-invariant optimizer, the experiments (even the one with simulated data) do not seem to have demonstrated this.

QUESTIONS:
While it could be harder to demonstrate the scale-invariant property of KATE with real data experiments, is it possible to demonstrate the scale invariant property of KATE compared to AdaGrad and Adam with the logistic regression using simulated data? This could better help readers understand why the scale-invariant property of KATE is plausible compared to other adaptive optimizers.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-19