# Combining Optimization Methods Using an Adaptive Meta Optimizer
## 1 Introduction  
Stochastic Gradient Descent [1] (SGD) is the dominant method for solving optimization problems. SGD iteratively updates the model parameters by moving them in the direction of the negative gradient calculated on a mini-batch scaled by the step length, typically referred to as the learning rate. It is necessary to decay this learning rate as the algorithm proceeds to ensure convergence. Manually adjusting the learning rate decay in SGD is not easy. To address this problem, several methods have been proposed that automatically reduce the learning rate. The basic intuition behind these approaches is to adaptively tune the learning rate based on only recent gradients, therefore limiting the reliance on the update to only a few past gradients. ADAptive Moment estimation [2] (ADAM) is one of several methods based on this update mechanism [3]. On the other hand, adaptive optimization methods such as ADAM, even though they have been proposed to achieve a rapid training process, are observed to generalize poorly with respect to SGD or even fail to converge due to unstable and extreme learning rates [4]. To try to overcome the problems of both of these types of optimizers and at the same time try to exploit their advantages, we propose an optimizer that combines them in a new meta-optimizer.  
As depicted in Figure 1, the basic idea of the ATMO optimizer proposed here is to combine two different known optimizers and automatically go quickly towards the direction of both on the surface of the loss function when the two optimizers agree (see geometric example in Figure 1(a)). When the two optimizers used in the combination do not agree, our solution always goes towards the predominant direction between the two but slowing down the speed (see example of Figure 1(b)).  
In the literature, there are many papers that compare neural models trained with the use of different optimizers [5; 6; 7; 8] or that propose modifications for existing optimizers [4; 9; 10], always aimed at improving the results on a subset of problems. Each paper demonstrates that an optimizer is better than the others, but as the problem changes, this type of result is no longer valid and we have to start from scratch. Our method can be combined with other methods like Genetically Trained DNN [11], which combines learning using gradient descent with genetic algorithms. The genetic part, after a selected number of epochs, selects a new population through three states called selection, crossover, and manipulation. In general, a Genetically Trained DNN is very different from our proposal, which combines two gradient descent methods together. However, the genetic method can also be used with ATMO.  
In our paper, we propose combining two different optimizers like SGD and ADAM to overcome the performances of the single optimizers in very different problems.  
Figure 1: Intuitive representation of the idea behind the proposed ATMO that Mix ADAM and SGD optimizers: the weights are modified simultaneously by both the optimizers.  
Figure 2: Graphical representation of the basic idea for the proposed ATMO optimizer. In (**a**), if the two translations \(\vec{w}_{1}\) and \(\vec{w}_{2}\) obtained from two different optimizers are similar, then the resulting translation \(\vec{w}_{1}+\vec{w}_{2}\) is boosted. In (**b**), if the translations \(\vec{w}_{1}\) and \(\vec{w}_{2}\) go in two different directions, then the resulting translation is smaller. We also use two hyper-parameters \(\lambda_{1}\) and \(\lambda_{2}\) to weigh the contribution of the two optimizers.  
Below are the main contributions of this paper:  
* We show experimentally that the combination of two different optimizers in a new meta-optimizer leads to a better generalization capacity in different contexts.
* We describe ATMO using Adam and SGD but show experimentally that other types of optimizers can be profitably combined.
* We release the source code and setups of the experiments [12].