SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Meta-AdaM: A Meta-Learned Adaptive Optimizer with Momentum for Few-Shot Learning
Siyuan Sun
Ames, IA 50011
sxs14473@iastate.edu
Hongyang Gao
hygao@iastate.edu
We introduce Meta-AdaM, a meta-learned adaptive optimizer with momentum, designed for few-shot learning tasks that pose significant challenges to deep learning models due to the limited number of labeled examples. Meta-learning has been successfully employed to address these challenges by transferring meta-learned prior knowledge to new tasks. Most existing works focus on meta-learning an optimal model initialization or an adaptive learning rate learner for rapid convergence. However, these approaches either neglect to consider weight-update history for the adaptive learning rate learner or fail to effectively integrate momentum for fast convergence, as seen in many-shot learning settings. To tackle these limitations, we propose a meta-learned learning rate learner that utilizes weight-update history as input to predict more appropriate learning rates for rapid convergence. Furthermore, for the first time, our approach incorporates momentum into the optimization process of few-shot learning via a double look-ahead mechanism, enabling rapid convergence similar to many-shot settings. Extensive experimental results on benchmark datasets demonstrate the effectiveness of the proposed Meta-AdaM.
Deep learning has demonstrated its capability in solving various challenging tasks in many-shot-learning settings, where each classification class contains sufficient training examples [12]. However, its performance is hindered on small datasets or under few-shot-learning (FSL) settings [37]. In FSL, unseen tasks with limited training examples pose significant challenges for optimizing deep learning models [45]. With limited training data, deep learning models may suffer from severe underfitting or overfitting problems [28]. Numerous meta-learning methods have been proposed to address this challenge by identifying and encoding shared knowledge among different data distributions within the model. With prior generalized knowledge, a meta-learning model can be rapidly transferred to a new task by fine-tuning it with limited unseen data [13].
Existing optimization-based meta-learning methods for FSL facilitate this transfer process by fine-tuning a meta-learned model on new few-shot tasks. There are two primary directions for using meta-learning in FSL tasks: learning an optimal initialization and generating high-quality gradient updates for rapid convergence. In the first direction, most methods [7, 8, 29, 24, 17, 18] employ an inner loop to fine-tune the meta-model on specific tasks and iteratively optimize the meta-model in an outer loop based on the adapted model from the inner loop. The resulting initial model weights enable the adapted model to converge more quickly to an optimal solution for a new task. In the second direction, some works [38, 2, 46] focus on generating high-quality and task-adaptive gradient updates, allowing the optimizer to rapidly converge to an appropriate solution.
Our work focuses on the second direction. Most existing works in this direction aim to use adaptive learning rates by considering weights and gradients [18, 30, 3]. However, it has been shown that the weight-update history is more important than the weights themselves for determining an appropriate learning rate. Additionally, existing works often fail to incorporate momentum in FSL, which has been shown to be effective in accelerating the optimization process in many-shot settings. This is due to the fact that momentum typically exhibits high fluctuations during the initial updates [19]. Another challenge for the meta-trained base model is the incorporation of prior knowledge. Certain classes exhibit features that align more closely with prior knowledge than others, leading to an imbalance in predictions generated by the model.
To address these challenges, we propose a meta-learned adaptive optimizer with momentum (Meta-AdaM). To learn a more accurate adaptive learning rate, our approach meta-learns a learning rate learner based on gradient update history instead of weights. The learned learning rate learner, along with the meta-learned model initialization, is applied when adapting to new tasks. Moreover, we introduce a dynamic class weight scheme for classification tasks, aimed at effectively balancing the loss distribution among various classes, which can help the model generate unbiased results.
In this section, we introduce related works to our proposed approach.
2.1 Few-Shot Learning and Meta Learning
A formal definition of machine learning can be described as follows: a computer program can learn some experience  $E$  from a set of classes that belongs to task  $T$  with performance metric  $P$  if  $P$  increases with  $E$  on  $T$  measured by  $P$  [22]. Few-shot-learning problems are a subset of machine learning problems in which  $E$  only contains a limited number of samples [45]. Under this setting, training deep neural networks can be challenging. The empirical risk  $R$  generated from the limited experience  $E$  can differ greatly from the expected empirical risk. Meta-learning enhances the performance  $P$  by learning shared features from other data. The metric-based [36] and model-based meta-learning methods [35, 23, 21] rely on extra features or models to improve the few-shot learning capabilities. Recently, optimization-based meta-learning methods have obtained more attention for their strong generalization ability. The optimization-based methods reduce the meta-Learning problem into a bi-level optimization problem. The inner loop optimizes the base model on a certain task, and the outer loop optimizes the base model across several tasks to adjust the initial weight for quick adaption. Without introducing new elements, such a structure has the potential to adapt better to unseen data. The most representative optimization-based method is the MAML method [7]. Subsequent MAML variants [8, 29, 24, 17, 18, 34, 9] focus on optimizing the optimization process.
2.2 Gradient-Descent Based Optimizer
The optimizer plays a crucial role in enhancing the efficiency of the training process in deep learning. Most popular optimizers are based on the gradient descent algorithm [1]. To further improve training efficiency and increase convergence speed, existing optimizers, such as AdaM [14], often utilize the first and second orders of momentum. The first order of momentum, also known as momentum, is the accumulated sum of historical gradients during the training process, which can be considered a more effective update direction compared to gradients. The second order of momentum is used to control the step size for each parameter by estimating the update velocity for each parameter. Moreover, based on AdaM, other optimizers [14, 6, 19] introduce additional elements to better estimate the update direction and step size. Learned optimizers [2, 1, 43] utilize a trainable model to estimate the update direction and step size, which can better adapt to new tasks.
2.2.1 Optimizer for Few-Shot Learning
Meta-learning has shown a strong ability to solve few-shot learning problems. So, researchers also use meta-learning methods to train optimizers for few-shot learning problems. [18] propose an optimizer that works as the stochastic gradient descent. [30, 3] propose to use linear models to carefully estimate the learning rate for each parameter. [25] consider estimating curvature instead of gradients. However, these works only consider the gradient change in one step and ignore the
A. Regular initialization with random optimization
B. Meta-learned initialization
Figure 1: Illustration of different few-shot-learning optimizing strategies.  $\hat{\theta}$  is the global optimum solution.  $\theta^{*}$  is the best approximation in the hypothesis space  $\mathcal{H}$  determined by the FSL model. (A) shows the optimization trajectory of naive optimizer with random initialization  $\theta_{s}$  and regular weights updates, leading to a poor solution  $\theta_{A}$ . (B) illustrates using a regular weights update with meta-learned initialization  $\theta_{s}^{\prime}$ , resulting in a better approximation  $\theta_{B}$  than  $\theta_{A}$ . In (C), a meta-learned optimizer is used with the meta-learned initialization  $\theta_{s}^{\prime}$ , leading to the best approximation  $\theta_{C}$ .
C. Meta-learned Optimizer With Meta-learned Initialization
gradient change history in the fine-tuning process, which is not optimal. In this work, we propose an optimizer called Meta-AdaM, specially designed for the inner loop in meta-learning. We design an LSTM-based meta-learner to adaptively estimate the learning rate. Also, we make wiser use of momentum to dynamically determine the optimal updating direction. This can help the base model to converge to the optimum in fewer steps.
3 Meta-AdaM: A Meta-Learned Adaptive Optimizer with Momentum
We propose a Meta-learned Adaptive optimizer with Momentum (Meta-AdaM) for FSL.
3.1 Motivations and Challenges
Few-shot learning (FSL) is a formidable problem in machine learning, where models are expected to generalize effectively to new tasks with only a handful of labeled examples. The limited amount of training data, however, poses a significant challenge for traditional optimizers, as they tend to overfit and yield subpar solutions [28]. As demonstrated in [45], the empirical risk minimizer or optimizer in FSL is unreliable due to its propensity to overfit a limited number of training examples, resulting in unsatisfactory solutions, as illustrated in Figure 1 (A). Consequently, optimization steps are often restricted to prevent overfitting, which makes it difficult for an optimizer to discover suitable solutions for FSL tasks. Generally, two primary strategies exist to tackle this issue. One involves identifying an optimal initialization for model parameters, as depicted in Figure 1 (B). The other focuses on generating high-quality gradient updates for rapid convergence to an appropriate solution, as displayed in Figure 1 (C). These strategies are orthogonal and can be combined. The objective of this work is to develop an optimizer capable of producing high-quality gradient updates, allowing the model to achieve a satisfactory optimum with a limited number of updates.
The most commonly used optimization algorithms for training deep learning models are variants of gradient descent, which utilize the following update formula:
$$
\theta_ {i} ^ {t} = \theta_ {i} ^ {t - 1} - \eta_ {i} ^ {t} \times \left(\alpha_ {i} ^ {t} \times g _ {i} ^ {t} + \beta_ {i} ^ {t} \times m _ {i} ^ {t - 1}\right), \tag {1}
$$
$$
m _ {i} ^ {t} = \alpha_ {i} ^ {t} \times g _ {i} ^ {t} + \beta_ {i} ^ {t} \times m _ {i} ^ {t - 1}. \tag {2}
$$
In the update formula,  $\theta_{i}^{t}$  represents the value of a trainable parameter  $\theta_{i}$  of the model after undergoing  $t$  updates. The variables  $\eta_{i}^{t}, g_{i}^{t}$ , and  $m_{i}^{t}$  refer to the learning rate, gradient, and first-order momentum for  $\theta_{i}$  at the  $t^{th}$  update, respectively. Additionally,  $\alpha_{i}^{t}$  and  $\beta_{i}^{t}$  are coefficients to balance the gradients and momentum, where momentum is an exponentially weighted moving average of the gradients.
In many-shot learning scenarios, where each learning task has abundant labeled data, several optimizers [41, 14, 32], such as Adam [14] and SGD with momentum [27], have been developed to speed up
the convergence speed by incorporating adaptive learning rates [5] and momentum. However, these optimization algorithms cannot be directly applied to FSL tasks as the variance of momentum in the initial optimization steps tends to be high [19], leading to unstable gradient updates with limited optimization steps and resulting in poor local optima.
Meta-learning has recently demonstrated a robust capacity to address FSL problems. Specifically, a meta-learner is trained on similar tasks and employed to learn a learner for any new task [38, 2, 46]. While several approaches [18, 30, 3] have been proposed to meta-learn adaptive learning rates in FSL by incorporating gradients and weights, the relationship between learning rates and weight-changing history, rather than just weight values, has been shown to be crucial [11, 15]. Furthermore, effectively utilizing momentum has been challenging due to its high variance in the initial updates [19]. In light of these challenges, we propose a novel approach, called Meta-AdaM, which meta-learns an adaptive optimizer with momentum. Our approach leverages the power of meta-learning to learn a model that predicts adaptive learning rates based on gradient update history, and effectively integrates momentum to speed up the convergence speed. By doing so, Meta-AdaM enables a learner to quickly adapt to new few-shot learning tasks with only a limited number of update steps.
3.2 Update History-Based Adaptive Learning Rates through Meta-Learning
In this section, we propose utilizing meta-learning to develop a learner that generates adaptive learning rates based on gradient update history. Existing methods, such as [30, 3], depend on a meta-trained multi-layer perceptron to predict learning rates using model parameter values and gradients. However, recent studies indicate that the relationship between learning rates and weight-changing history is more critical than between learning rates and weight values [11, 15]. As a result, it is essential to predict adaptive learning rates based on weight-changing history, specifically gradient  $g_{i}^{t}$  and momentum  $m^{t - 1}i$ . To achieve this, we treat the trajectory of  $g_{i}^{t}$  and  $m_{i}^{t}$  as sequence data and employ a long short-term memory network (LSTM) [39]. The LSTM accepts momentum and new gradients as inputs and outputs an adaptive learning rate for each trainable parameter  $\theta_{i}$ . Formally, the update formula for  $\theta_{i}$  with the proposed meta-learner is defined as follows:
Algorithm 1 MetaMomentumInner
1: Input:  $\theta$ : meta model, learning rate learner: LSTM,  $S_{T}$ : data of a task.
2: Require:  $\eta$ : base step size,  $K$ : number of inner loops,  $f(\cdot)$ : loss function.
3: Set  $\theta^0 = \theta, m^0 = 0$
4: for all  $t = 1$  to  $K$  do
5:  $\pmb{g}^{t} = \nabla_{\pmb{\theta}^{t - 1}}f(S_{\mathcal{T}},\pmb{\theta}_{-}^{t - 1})$
6:  $\pmb{\theta}_{m}^{t} = \pmb{\theta}^{t - 1} - \eta \pmb{m}^{t - 1}$
7:  $\pmb{\theta}_{g}^{t} = \pmb{\theta}^{t - 1} - \eta \pmb{g}^{t}$
8:  $\Delta \mathcal{L}_m^t = f(S_\mathcal{T},\pmb {\theta}_m^t) - f(S_\mathcal{T},\pmb{\theta}^{t - 1})$
9:  $\Delta \mathcal{L}_g^t = f(S_\mathcal{T},\pmb{\theta}_g^t) - f(S_\mathcal{T},\pmb{\theta}^{t - 1})$
10:  $\alpha^t, \beta^t = \sigma (\Delta \mathcal{L}_m^t, \Delta \mathcal{L}_g^t)$
11:  $\pmb{\eta}^{t} = \mathrm{LSTM}(\alpha^{t}\pmb{g}^{t},\beta^{t}\tilde{\pmb{m}}^{t - 1})$
12:  $\pmb{m}^{t} = \alpha^{t}\pmb{g}^{t} + \beta^{t}\pmb{m}^{t - 1}$
13:  $\pmb{\theta}^{t} = \pmb{\theta}^{t - 1} - \pmb{\eta}^{t}\times \pmb{m}^{t}$
14: end for
15: return  $\theta^K$
$$
\eta_ {i} ^ {t} = \operatorname {L S T M} \left(\alpha_ {i} ^ {t} \times g _ {i} ^ {t}, \beta_ {i} ^ {t} \times m _ {i} ^ {t - 1}\right), \tag {3}
$$
$$
\theta_ {i} ^ {t} = \theta_ {i} ^ {t - 1} - \eta_ {i} ^ {t} \times \left(\alpha_ {i} ^ {t} \times g _ {i} ^ {t} + \beta_ {i} ^ {t} \times m _ {i} ^ {t - 1}\right), \tag {4}
$$
where  $\eta_i^t$  is the predicted learning rate for the trainable parameter  $\theta_{i}$  in the  $t^{th}$  update. Compared to previous methods that primarily employ multi-layer perceptrons to predict learning rates based on model parameter values and gradients, our approach naturally considers weight-changing history, leading to more accurate learning rate predictions. For now, we set  $\alpha_{i}^{t} = 1$  and  $\beta_{i}^{t} = 0$  temporarily to exclude momentum from the update. This is because the variance of momentum is exceptionally high in the initial update steps, rendering it unsuitable for direct application in FSL tasks. In the next section, we propose an effective method to incorporate momentum, accelerating convergence.
It is crucial to emphasize that our approach fundamentally differs from Meta-LSTM [30]. MetaLSTM primarily employs a multi-layer perceptron to predict learning rates and weight decay, taking current weights and gradients as input without considering the weight-update history.
3.3 Double Lookahead for Effective Momentum Integration
In this section, we propose to effectively integrate momentum in the FSL optimizer to accelerate the convergence speed by using double lookahead. To generate more effective gradients updates during the training process for few-shot-learning tasks, we compute the adaptive model updates by applying lookahead to both momentum and new gradients, leading to a more stable learning
trajectory and better generalization. Lookahead is an effective optimization strategy by looking ahead at a sequence of fast weights [47] for fast converge and a corresponding slow gradient to smooth oscillating. However, in few-shot-learning context, the fast weight will easily trigger the overfitting problem. And the limited update steps number make the slow gradient hard to smooth the oscillating.
We propose addressing this challenge by judiciously selecting the coefficients of the momentum and new gradients using one-step lookahead. Specifically, before performing the accumulation process in Eq. (2), we apply lookahead to both the momentum and the new gradients, resulting in two losses. In this context, the aggregation of weight updates should prioritize the update direction with lower loss. To achieve this, we employ a softmax operator to obtain the coefficients by normalizing the loss decreases. Formally, the proposed approach is defined as in Algorithm 1. In this algorithm,  $S_{\mathcal{T}}$  is a batch of support data,  $\eta$  is a fixed learning rate,  $\sigma(\cdot)$  is a softmax operator, and  $f(\cdot)$  is a loss function that computes loss for a batch data  $S_{\mathcal{T}}$  with a given model  $\theta$ . We
Algorithm 2 Meta-AdaM
1: Input: Task distribution  $p(\mathcal{T})$
2:Require:  $\alpha ,\beta$  :step size hyper-parameters
3: Randomly initialize meta model  $\theta$
4: Randomly initialize weights in LSTM:  $\pmb{\theta}_{l}$
5: while not done do
6: Sample a batch of tasks  $\mathcal{T}_i\sim p(\mathcal{T})$
7: for all  $\mathcal{T}_i$  do
8: Finetune  $\pmb{\theta}$  with  $K$  examples:
9:  $\pmb{\theta}_{i}^{\prime} =$  MetaMomentumInner(  $\pmb{\theta}$  ,LSTM,  $\mathcal{T}_i)$
0: end for
1: Update meta model and LSTM:
2:  $\pmb{\theta} \gets \pmb{\theta} - \alpha \nabla_{\pmb{\theta}} \sum_{\mathcal{T}_i} \mathcal{L}_{\mathcal{T}_i}(f_{\pmb{\theta}_i'})$
3:  $\pmb{\theta}_{l} \gets \pmb{\theta}_{l} - \beta \nabla_{\pmb{\theta}_{l}} \sum_{\mathcal{T}_{i}} \mathcal{L}_{\mathcal{T}_{i}}(f_{\pmb{\theta}_{l}^{\prime}})$
4: end while
first generate two lookahead models by updating the current model parameters  $\pmb{\theta}^{t-1}$  using momentum  $\pmb{m}^{t-1}$  and new gradients  $\pmb{g}^t$  separately in line (6) and line (7), which results in  $\pmb{\theta}_m^{t-1}$  and  $\pmb{\theta}_g^t$ . Then  $f(\cdot)$  is applied to both models with  $S_{\mathcal{T}}$ , yielding loss changes for both lookahead models:  $\Delta \mathcal{L}_m^{t-1}$  and  $\Delta \mathcal{L}_g^t$ . In line (10), the softmax operator is used to normalize two loss changes so that the aggregation coefficients sum to 1. The resulting coefficients are used to predict adaptive learning rates in line (11), and aggregate the momentum and new gradients in line (12). Finally, line (13) updates the model parameters using accumulated momentum  $\pmb{m}^t$ .
With dynamic coefficients generated through lookahead, if either the momentum or new gradients lead to an ineffective optimization direction, the small coefficient in line (12) can mitigate the negative impact and place greater emphasis on more effective optimization directions. As a result, the double-lookahead strategy can stabilize the momentum update and accelerate the convergence speed.
3.4 Dynamic Class Weighting Scheme for Classification Tasks
In FSL classification tasks employing meta-learning, the meta-trained base model is significantly influenced by prior knowledge. As a result, certain classes with features closer to prior knowledge become easier to classify, leading to an imbalance that compromises the model's performance. To address this issue, we introduce a dynamic class weighting schema, which emphasizes classes contributing more to model optimization. We estimate a class's contribution by observing the changes in its loss. If a class has been well-learned, the loss will be stable, and minor loss changes will be observed. In contrast, if there is a substantial loss drop for a particular class, it may contribute more to future updates. Based on the loss changes for each class, we use a softmax operator to obtain aggregation weights for every class. The accumulated loss is the weighted summation of all class losses. This process is defined as follows:
$$
\mathcal {L} _ {1} ^ {t}, \dots , \mathcal {L} _ {C} ^ {t} = f _ {2} \left(S _ {\mathcal {T}}, \boldsymbol {\theta} ^ {t - 1}\right), \tag {5}
$$
$$
w _ {1}, \dots , w _ {C} = \sigma \left(\left(\mathcal {L} _ {t} ^ {1}, \dots , \mathcal {L} _ {t} ^ {C}\right) / T\right), \tag {6}
$$
$$
\mathcal {L} ^ {t} = \sum_ {k = 1} ^ {C} w _ {k} \mathcal {L} _ {k} ^ {t}, \tag {7}
$$
where  $f_{2}(\cdot)$  is a loss function that computes losses for each class based on support data  $S_{\mathcal{T}}$ ,  $C$  is the number of classes in task  $\mathcal{T}$ , and  $T$  is the temperature used in the softmax operator  $\sigma(\cdot)$ . In the above schema, if the class  $k$  has a larger loss  $\mathcal{L}_{k}^{t}$ , its corresponding aggregation weight  $w_{k}$  will be larger, leading to a stronger optimization signal from class  $k$ . Thus, the class that needs to be more well-trained will obtain more attention, leading to more effective optimization.
Table 1: Comparison results using Convnet4 on Mini-ImageNet, TieredImageNet, and Cifar100 datasets. We report performances in terms of accuracy (\%) with standard deviation.* Baseline MC uses a larger Convnet4 backbone network, and uses extra data regularization method.
Dataset	Method	5-way-1-shot	5-way-5-shot
Mini-ImageNet	Meta-LSTM [30]	43.44 ± 0.77	60.60 ± 0.71
MAML [7]	48.70 ± 1.75	63.11 ± 0.92
Meta-SGD [18]	50.47 ± 1.87	64.03 ± 0.94
MAML+ALFA [3]	50.58 ± 0.51	69.12 ± 0.47
e3bm [20]	53.2	65.1
Sparse-MAML [44]	51.04 ± 0.59	68.05 ± 0.84
MAML+SiMT [42]	51.49 ± 0.18	68.74 ± 0.12
MeTAL [40]	52.63 ± 0.37	70.52 ± 0.29
Meta-AdaM (ours)	52.00 ± 0.49	70.70 ± 0.49
MC* [25]	55.73 ± 0.94	70.33 ± 0.72
TieredImageNet	MAML [7]	49.06 ± 0.50	67.48 ± 0.47
MAML+ALFA [3]	53.16 ± 0.51	70.54 ± 0.46
e3bm [20]	52.1	70.2
MAML+SiMT [42]	52.51 ± 0.21	69.58 ± 0.11
MeTAL [40]	54.34 ± 0.31	70.40 ± 0.21
Meta-AdaM (ours)	53.93 ± 0.49	72.66 ± 0.49
Cifar100	e3bm [20]	39.9	52.6
MAML [7]	38.20 ± 0.48	49.94 ± 0.49
MAML+ALFA [3]	39.77 ± 0.48	53.39 ± 0.49
Meta-AdaM (ours)	41.11 ± 0.49	56.32 ± 0.49
3.5 Meta-AdaM
We present our proposed Meta-AdaM in Algorithm 2. Given a task distribution  $p(\mathcal{T})$ , our goal is to meta-learn an optimal model initialization  $\theta$  and an adaptive learning learner  $\theta_{l}$ . In each iteration, we sample a batch of tasks from  $p(\mathcal{T})$ . For each sampled task, we fine-tune an adapted model  $\theta^{\prime}$  and update  $\theta$  and  $\theta_{l}$  using Algorithm 1. During testing, we directly utilize  $\theta$  and  $\theta_{l}$  for a new task.
We conduct experiments to evaluate the proposed Meta-AdaM using three benchmark datasets.
This section describes datasets and backbone models used to evaluate our Meta-AdaM.
Datasets. We evaluate the proposed methods using three datasets: Mini-ImageNet [10], Tiered-ImageNet [33], and Cifar100 [16] datasets. MiniImageNet dataset contains 100 classes with 600 samples per class. Each data sample is an  $84\times 84$  colored image. Following previous works [30], we split 100