Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper extends AdaBelief by making two modifications: (1) the 2nd momentum s_t is computed by taking the EMA of (m_t/(1-beta_1^t)-m_{t-1}/(1-beta_1^{t-1}) )^2 in comparison to the EMA of (m_t-g_t)^2 in AdaBelief; (2) the parameter delta (or epsilon  in AdaBelief) is introduced in a slightly different manner to avoid division by zero.  The new optimiser is referred to as ADADQH. Experiments on different DNN tasks indicates that ADADQH produce either better and competitive performance. 
strength_and_weaknesses: Strength: 
1. The paper propose new types of modifications to AdaBelief, aiming to improve its performance, which I haven't seen from literature before. 
2. Theoretical analysis is provided for the new method. 

Weekness: 
(1) As I mentioned earlier, ADADQH made two modifications to AdaBelief. It is not clear which modification plays a crucial role for the performance gain. I am guessing the first modification for computing s_t makes a difference in performance improvement. The reason is that m_t/(1-beta_1^t)-m_{t-1}/(1-beta_1^{t-1}) = (1-beta_1)/(1-beta_1^t) * [g_t - m_{t-1}/(1-beta_1^{t-1})]. The scalar  (1-beta_1)/(1-beta_1^t) essentially reduce the variance of the adaptive stepsizes in comparison to AdaBelief which computes the EMA of (g_t-m_t)^2.  In a recent paper "On Exploiting Layerwise Gradient Statistics for Effective Training of Deep Neural Networks", another method named Aida was proposed to reduce the variance of the adaptive stepsizes of AdaBelief by performing vector projections between m_t and g_t before being used for the computing the 2nd momentum. I suggest the authors to conduct performance also in comparison to Aida. In the abstract, the authors state that ADADQH is compared to SOTA optimizers. Both Aida and ADADQH are closely related to AdaBelief and should be compared.  

(2) In AdaBelief paper, the validation accuracy of AdaBelief for training ResNet34 over CIFAR10 is above 95%. In the current paper, the validation accuracy of AdaBelief for training ResNet32 over CIFAR10 is below 93%. Even though ResNet 34 is slightly complicated than ResNet32, I think the performance gap is too large for AdaBelief.  I suggest the authors take the opensource from AdaBelief to make a direct comparison. 

(3) My personal experience is the selection of parameter epsilon in Adam and AdaBelief plays a very important role in the performance. In order to make a fair comparison, the parameter epsilon needs to searched for each optimizer. In Figure 1, I don't see that the parameter epsilon  is being searched for Adam and AdaBelief. Inproper epsilon value in Adam and AdaBelief would slow down their convergence speed.  
clarity,_quality,_novelty_and_reproducibility: 1. The proposed method ADADQH is in fact an extension of AdaBelief. As I mentioned earlier, The EMA of (m_t-g_t)^2 in AdaBelief can also be viewed to approximate the diagonal Hessian of a function, which is very similar to the EMA of ( m_t/(1-beta_1^t)-m_{t-1}/(1-beta_1^{t-1})  )^2= ( (1-beta_1)/(1-beta_1^t) * [g_t - m_{t-1}/(1-beta_1^{t-1})] )^2 in ADADQH.   In the paper, I don't see the study or discussion on the connection between AdaBelief and ADADQH. The authors only mention AdaBelief without detailed information.   
2. I don't get how (3) is obtained by solving (2). Note that m_t does not appear in (2).  There is a gap between (2) and (3). 

summary_of_the_review: The paper proposes two new modifications to AdaBelief, and demonstrates performance gain on both synthetic and real-data experiments. 1. The connection between the proposed method  ADADQH and the two existing methods AdaBelief and Aida should be studied, or properly described in the introduction in the paper so that the readers better understand the reasoning behind it. 2. The parameter epsilon needs to be searched for each optimizer in order to make a fair comparison in all experiments. 3. The paper needs to conduct ablation study to find out which modification in ADADQH is more important to improve performance in AdaBelief.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies an Adam type optimizer, adaDQH, by designing a new rule to update $v$. The idea is i) using gradient difference to approximate (diagnalized) Hessian, and ii) using a max in the denominator of to switch AdaDQH to SGD when necessary. Theoretical results and numerical tests are carried out to support the convergence of proposed AdaDQH.
strength_and_weaknesses: **Strength:**
- S1. The idea to approximate Hessian is interesting.
- S2. On tested tasks, AdaDQH exhibits improved performance. 

**Weakness:**

- W1. (Novelty.) Two of key ideas, Hessian approximation, and auto switch have been used in the literatures but not well cited. For example, the idea of using gradient difference to estimate Hessian appeared in http://proceedings.mlr.press/v70/zheng17b.html, and the auto switch shares similar idea to https://arxiv.org/abs/1902.09843. In addition, although the idea of approximating Hessian is interesting, the motivating equation in two lines after Alg. 1 is less convincing. First, the value of $\theta$ is taken as 1, however, the reason is unclear. If it is taken from mean value theorem, the choice of $\theta$ can be different in different cases. Second, the reason to set $\Delta w$ to $e_i$ is also unclear, and it looks like this is not the case when updating the model weights. It can be more reasonable if the authors can plot the difference between the proposed Hessian estimation and real Hessian. Another question is that how well does EMA approximates the true gradient. The lack of justification makes the claim "better estimated Hessian" less convincing. 

- W2. (Numerical experiments.) It looks like the models for evaluating the proposed algorithm are not well chosen. For example, it is standard to use ResNet50 rather than ResNet18 on ImageNet. And ResNet18 can easily achieve 93 - 94\% test accuracy on CIFAR10, while the tested ResNet20 and ResNet32 struggle on the same task. The choice of testing models makes the numerical results less competitive.
And why are some results in Table 4 are training on P100 and some on V100? 


(Minor 1) Does the experiment in Figure 1 take stochastic gradient into account?

(Minor 2) The theory uses diminishing $\beta_{1,t}$ while the main algorithm adopts fixed $\beta_1$.

clarity,_quality,_novelty_and_reproducibility: The idea is easy to follow and well presented. 

Novelty can be found in W1. 

Code is provided for reproducibility.
summary_of_the_review: The proposed AdaDQH is interesting in how the Hessian is estimated. However, this paper does not provide convincing explanation to their Hessian estimate, and does not compare with existing works with similar ideas. Some of experiments are not carried out on standard models making their numerical improvement less impressive.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The research is on developing an adaptive optimization algorithm called AdaDQH, which proposes a new technique to approximate the Hessian matrix in the second-order gradient descent scheme in a computationally efficient manner. The authors demonstrate the optimizer’s performance on datasets from Natural Language Processing, Computer Vision, and Recommendation Systems to show faster convergence and better generalization. The paper also provides theoretical proofs on bounds on regret and time complexity to explain the trade-off between generalization, time and computational resources. Weights in a deep-network at iteration are updated as:
   w_(t+1) = w_t − α_t B_t^(−1t) m_t
where,α_t is the step-size, represents a linear combination of the gradients ∇_wL(w_t) , and B_t, called as the precondition matrix is an approximation of the Hessian H_t . Several previous second-order optimization algorithms like AdamW, AdaBelief, AdaHessian have developed methods to approximate . The current algorithm, AdaDQH, selects B_t as: B_t^2 = diag(EMA(s_1s_1^T, s_2s_2^T, s_ts_t^T ))/(1 − β_2^t))
where, EMA represents an exponential moving average, and is given by a recurrent relation in terms of s_t.
The structure of is adopted for bias-correction is similar to Adam. In addition to approximating the Hessian, the research also proposes a way to switch between an adaptive version and EMA version for the gradients.


strength_and_weaknesses: 1. The paper provides mathematical details to support time-complexity and computational analysis, and bounds for establishing convergence.
2. Results for the paper cover problems from different areas in Machine Learning and Optimization, adding to credibility and usefulness in the future.
3. An essential point covered in the paper is: Trade-off of computations, time and generalization.
4. It would be interesting to note the performance of AdaDQH on highly non-convex optimization problems  (i.e. functions with several local minima) like Ackley and Schwefel. This would help test the idea of whether AdaDQH is able to converge to global minima or not. Challenge problems from CEC 2019 would also help.
5. An important question arises: Is AdaDQH able to converge to global minima of few functions wherein the current state-of-the-art algorithms are only able to make it to local minima? This would make it a significant improvement over the current state-of-the-art algorithms.
clarity,_quality,_novelty_and_reproducibility: 1. The research is a novelty since it showcases a different mechanism to estimate the Hessian matrix.
2. The paper is well-presented to include the algorithm, mathematical details and proofs for convergence, and
lastly, empirical results on different datasets.
3. It is clear in its approach and covers most of the essential details for completion o
4. The paper quality is good since it has come up with a optimizer, with potential applications in all sub-fields of Machine Learning.
5. Seems reproducible

summary_of_the_review: Research paper is well-presented and produces a novel technique to approximate the Hessian for optimization. Overall, it’s an important paper to be considered in Machine Learning.

Additional Comments:

What is s_t, b_t and beta_2?
Could different notations be used for vector and scalar?
The paper updates B_t with max(B_t , delta*I). Why is this so?
How was equation 3 obtained from equation 2?
Is it possible to give mathematical guarantees on autoswitch? Say when it switches and in which conditions does it occur.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new optimizer based of Adam with a few modifications. The main question for this type of works is about the true goal: is it to provide a faster optimizer when all the hyperparameters are carefully tuned? Is it to provide an optimizer that is competitive across many different settings with the same value for the hyperparameters?

Because the reviewers barely engaged in a discussion with the authors, I had a closer look at the paper. As it is, I do not feel the theoretical part or the empirical part make a convincing enough argument. The Hessian approximation lacks validation, not in the sense that it does not work, but rather that many preconditioners end up giving good performance and I became wary of explanations.
As for the experimental part, most of the results are in the same ballpark as existing methods. I have yet to see cases where differences of that magnitude were enough to determine the choice of algorithm.

Ultimately, I could not be convinced of the impact and decided to side with both reviewers who gave a 5.
justification_for_why_not_higher_score: This is yet another optimization method as there are so many out there. Differences with existing ones are marginal and I don't see the publication of this work changing anyone's habit.
Decision: Reject