Meta-AdaM: An Meta-Learned Adaptive Optimizer with Momentum for Few-Shot Learning
Authors: Siyuan Sun
Hongyang Gao
Venue: NeurIPS 2023 poster (2023)
Keywords: Few shot learning
Meta Learning
TL;DR: An optimizer specially designed for few-shot-learning problems

Abstract: We introduce Meta-AdaM, a meta-learned adaptive optimizer with momentum, designed for few-shot learning tasks that pose significant challenges to deep learning models due to the limited number of labeled examples. Meta-learning has been successfully employed to address these challenges by transferring meta-learned prior knowledge to new tasks. Most existing works focus on meta-learning an optimal model initialization or an adaptive learning rate learner for rapid convergence. However, these approaches either neglect to consider weight-update history for the adaptive learning rate learner or fail to effectively integrate momentum for fast convergence, as seen in many-shot learning settings. To tackle these limitations, we propose a meta-learned learning rate learner that utilizes weight-update history as input to predict more appropriate learning rates for rapid convergence. Furthermore, for the first time, our approach incorporates momentum into the optimization process of few-shot learning via a double look-ahead mechanism, enabling rapid convergence similar to many-shot settings. Extensive experimental results on benchmark datasets demonstrate the effectiveness of the proposed Meta-AdaM.

Meta-AdaM: A Meta-Learned Adaptive Optimizer with Momentum for Few-Shot Learning
Siyuan Sun
Hongyang Gao
Department of Computer Science Iowa State University
Department of Computer Science
Iowa State University
Ames, IA 50011 sxs14473@iastate.
Ames, IA 50011
hygao@iastate.edd
Abstract
We introduce Meta-AdaM, meta-learned adaptive optimizer with momentum, de- signed for few-shot learning tasks that pose significant challenges to deep learning models due the limited number of labeled examples. Meta-learning has been successfully employed to address these challenges by transferring meta-learned prior knowledge to new tasks. Most existing works focus on meta-learning an opti- mal model initialization or an adaptive learning rate learner for rapid convergence. However, these approaches either neglect to consider weight-update history for the adaptive learning rate learner or fail to effectively integrate momentum for fast convergence, as seen many-shot learning settings. To tackle these limitations, we propose a meta-learned learning rate learner that utilizes weight-update history as input to predict more appropriate learning rates for rapid convergence. Further- more, for the first time, our approach incorporates momentum into the optimization process of few-shot learning via a double look-ahead mechanism, enabling rapid convergence similar to many-shot settings. Extensive experimental results on benchmark datasets demonstrate the effectiveness of the proposed Meta-AdaM.
1 Introduction
Deep learning has demonstrated its capability in solving various challenging tasks in many-shot- learning settings, where each classification class contains sufficient training examples [12). However, its performance is hindered on small datasets or under few-shot-learning (FSL) settings [37]. FSL, unseen tasks with limited training examples pose significant challenges for optimizing deep learning models [45). With limited training data, deep learning models may suffer from severe underfitting overfitting problems 28 Numerous meta-learning methods have been proposed to address this challenge by identifying and encoding shared knowledge among different data distributions within the model. With prior generalized knowledge, a meta-learning model can be rapidly transferred to new task by fine-tuning it with limited unseen data 13
Existing optimization- baaad meta-lenrning methods for FSL facilitate this transfer process by fine- tuning a meta-lcarned model on new few-shot tasks. There are two primary directions for using meta-learning in FSL tasks: learning optimal initialization and generating
gradient
updates for rapid convergence. In the first direction, most methods 7 g 29 24 17 18] employ an
inner loop to fine-tune the meta-model on specifie tasks and iteratively optimize the meta-model in an outer loop based on the adapted model from the inner loop. The resulting initial model weights enable the adapted model converge more quickly to an optimal solution for new task. In the second direction, some works [38] 46) focus on generating high-quality and task-adaptive gradient updates, allowing the optimizer to rapidly converge an appropriate solution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Our work focuses on the second direction. Most existing works in this direction aim to use adaptive learning rates by considering weights and gradients [18 30 31 However, it has been shown that the weight-update history is more important than the weights themselves for determining appropriate learning rate. Additionally, existing works often fail to incorporate momentum in FSL., which has been shown to be effective in accelerating the optimization process in many-shot settings. This is due to the fact that momentum typically exhibits high fluctuations during the initial updates [19]. Another challenge for the meta-trained base model is the incorporation of prior knowledge. Certain classes exhibit features that align more closely with prior knowledge than others, leading to an imbalance in predictions generated by the model.
To address these challenges, we propose a meta-learned adaptive optimizer with momentum (Meta- AdaM). To learn a more accurate adaptive learning rate, our approach meta-learms a learning rate learner based on gradient update history instead of weights. The learned learning rate learner, along with the meta-learned model initialization, is applied when adapting to new tasks. Moreover, we introduce a dynamic class weight scheme for classification tasks, aimed at effectively balancing the loss distribution among various classes, which can help the model generate unbiased results.
2 Related Works
In this section, we introduce related works to our proposed approach.
2.1 Few-Shot Learning and Meta Learning
A formal definition of machine learning can be described as follows: a computer program can learn some experience E' from set of classes that belongs to task T' with performance metric P if P increases with E on T' measured by P [22]. Few-shot-learning problems are a subset of machine learning problems in which E only contains limited number of samples (45). Under this setting, training deep neural networks can be challenging. The empirical risk R generated from the limited experience E can differ greatly from the expected empirical risk. Meta-learning enhances the performance P by learning shared features from other data. The metric-based (36) and model-based meta-learning methods [35 23 21 rely extra features or models improve the few-shot learning meta-learning methods have obtained more attention
capabilities. Recently, optimization-bant for their strong generalization ability. The optimization-baseed methods reduce the meta-Learning problem into bi-level optimization problem. The inner loop optimizes the base model on a certain task, and the outer loop optimizes the base model across several tasks to adjust the initial weight for quick adaption. Without introducing new elements, such a structure has the potential to adapt better to unseen data. The most representative optimization-basee
method is the MAML method M. focus on optimizing the optimization process. 18
Subsequent MAML variants 8 29 24 17
34 9
2.2 Gradient-Descent Based Optimizer
The optimizer plays a crucial role in enhancing the efficiency of the training process in deep learning. Most popular optimizers are based on the gradient descent algorithm []] To further improve training efficiency and increase convergence speed, existing optimizers, such as AdaM [1), often utilize the first and second orders of momentum. The first order of momentum, also known as momentum, is the accumulated sum of historical gradients during the training process, which can be considered a more effective update direction compared to gradients. The second order of momentum is used to control the step size for each parameter by estimating the update velocity for each parameter. Moreover, based AdaM,, other optimizers [14, 6 19 introduce additional elements to better estimate the update direction and step size. Learned optimizers 2 43) utilize a trainable model to estimate the update direction and step size, which can better adapt to new tasks.
2.2.1 Optimizer for Few-Shot Learning
Meta-learning has shown a strong ability to solve few-shot learning problems. So, researchers also use meta-learning methods to train optimizers for few-shot learning problems. [18] propose an
optimizer that works the stochastic gradient descent. (30 3 propose to use linear models to
carefully estimate the learning rate for each parameter. (25) consider estimating curvature instead
of gradients. However, these works only consider the gradient change in one step and ignore the

---

i
i
i
0
0
0
0 B
A
H
O' H
0 H
0*
A. Regular initilization with random optimization
B. Meta-learned initializstion
Meta-learned Optimizer
With Meta-learned Initialization
Figure 1: Illustration of different few-shot-learing optimizing strategies. Ô is the global optimum solution. 0 is the best approximation in the hypothesis space H determined by the FSL model. (A) shows the optimization trajectory of naive optimizer with random initialization O, and regular weights updates, leading to poor solution A (B) illustrates using a regular weights update with meta-learned initialization 0' resulting in hetter approximation On than A In (C), meta-learned optimizer is used with the meta-learned initialization 0' leading to the best approximation Ic.
gradient change history in the fine-tuning process, which is not optimal. In this work, we propose an optimizer called Meta-AdaM, specially designed for the inner loop in meta-learning. We design an LSTM- meta-learner to adaptively estimate the learning rate. Also, we make wiser use of momentum to dynamically determine the optimal updating direction. This can help the base model to converge to the optimum fewer steps.
3 Meta-AdaM: A Meta-Learned Adaptive Optimizer with Momentum
We propose a Meta-lcamned Adaptive optimizer with Momentum (Meta-AdaM) for FSL.
3.1 Motivations and Challenges
Few-shot learning (FSL) is a formidable problem in machine learning, where models are expected to generalize effectively to new tasks with only a handful of labeled examples. The limited amount of training data, however, poses significant challenge for traditional optimizers, as they tend to overfit and yield subpar solutions (28). As demonstrated
[45), the empirical risk minimizer or optimizer in FSL. is unreliable due to its propensity to overfit limited number of training examples, resulting in unsatisfactory solutions, as illustrated in Figure [ (A). Consequently, optimization steps often restricted to prevent overfitting, which makes it difficult for an optimizer to discover suitable solutions for FSL tasks. Generally, two primary strategies exist to tackle this issue. One involves identifying optimal initialization for model parameters, depicted in Figure T (B). The other focuses on generating high-quality gradient updates for rapid convergence to an appropriate solution, displayed in Figure (C.. These strategies are orthogonal and can be combined. The objective of this work is to develop an optimizer capable of producing high-quality gradient updates, allowing the model achieve satisfactory optimum with limited number of updates.
The most commonly used optimization algorithms for training deep learning models are variants of gradient descent, which utilize the following update formula:
0 0f of x (af of Bf m)
(I)
mf of of 8 m 1
In the update formula, 0 represents the value of trainable parameter 0, of the model after undergoing t updates. The variables nf. gf. and m² refer to the learning rate, gradient, and first-order momentum for 6, at the tth update, respectively. Additionally, af and 8 are coefficients to balance the gradients and momentum, where momentum is an exponentially weighted moving average of the gradients.
In many-shot learning scenarios, where each learning task has abundant labeled data, several optimiz- ers 41 14 32), such as Adam [14] and SGD with momentum (27], have been developed to speed up
3

---

the convergence speed by incorporating adaptive learning rates [S] and momentum. However, these optimization algorithms cannot be directly applied FSL tasks as the variance of momentum in the initial optimization steps tends to be high [9), leading to unstable gradient updates with limited optimization steps and resulting poor local optima.
Meta-learning has recently demonstrated a robust capacity to address FSL problems. Specifically, a meta-learner trained on similar tasks and employed learn learner for any new task [38] 2 46)
While several approaches [8 30 3 have been proposed to meta-learn adaptive learning rates in FSL
by incorporating gradients and weights, the relationship between learning rates and
history, rather than just weight values, has been shown to be crucial [TT] [5]. Furthermore, effectively utilizing momentum has been challenging due to its high variance in the initial updates [19]. In light of these challenges, we propose novel approach, called Meta-AdaM, which meta-learn an adaptive optimizer with momentum. Our approach leverages the power of meta-learning to learn a model that predicts adaptive learning rates based on gradient update history, and effectively integrates momentum to speed up the convergence speed. By doing so, Meta-AdaM enables a learner to quickly adapt to new few-shot learning tasks with only a limited number of update steps.
3.2 Update History-Based Adaptive Learning Rates through Meta-Learning
In this section, we propose utilizing meta-learning to develop learner that generates adaptive learn-
Algorithm 1 MetaMomennummnnee
ing rates based on gradient update history. Existing
1: Input: 0: meta model, learning rate learner:
methods, such as 13 B depend on meta-trained multi-layer perceptron to predict learning rates us-
LSTM,, Sy: data a task.
2: Require:
base step size, K: number of inner loops, f(-): loss function.
ing model parameter values and gradients. However,
3: Set 0 0. m 0
recent studies indicate that the relationship between
4: all = to do
learning rates and weight-changing history is more critical than between learning rates and weight val- ues [T [5]. As a result, it is essential to predict adaptive learning rates based on weight-changing his-
5:
g' = /
6:
e.. = 6 'mmm"
7:
o = 0--1 nngg
8:
AC = =f fST, 0'm)) - (ST, 0--1)
tory, specifically gradient of and momentum m² 1 To achieve this, treat the trajectory of of and m as sequence data and employ a long short-term mem-
9:
AC = (ST, -f (ST, 0*11
10:
at, ge o(AC AC
11:
n' = LSTM( q 3'm ml
ory network (LSTM) 39] The LSTM accepts mentum and new gradients as inputs and outputs an adaptive learning rate for each trainable parameter 0,- Formally, the update formula for 0g with the proposed meta-learner is defined as follows:
12:
m =o'g" + B m
13:
===00- n m
14: end for
15: return gx
nf LSSM((a) x 9h.0, x m
0: = 0.-1 n/ x (a) x of + 3 x m
where nf is the predicted learning rate for the trainable parameter 0 in the th update. Compared to previous methods that primarily employ multi-layer perceptrons to predict learning rates based on model parameter values and gradients, our approach naturally considers weight-changing history, leading to more accurate learning rate predictions. For now, we set of = 1 and 3 = 0 temporarily to exclude momentum from the update. This is because the variance of momentum exceptionally high in the initial update steps, rendering it unsuitable for direct application in FSL tasks. In the next section, we propose an effective method to incorporate momentum, accelerating convergence.
is crucial to emphasize that our approach fundamentally differs from Meta-LSTM [30] Meta- LSTM primarily employs multi multi-layer perceptron to predict learning rates and weight decay, taking current weights and gradients as input without considering the weight-update history.
3.3 Double Lookahead for Effective Momentum Integration
In this section, we propose to effectively integrate momentum the FSL, optimizer to accelerate the convergence speed by using double lookahead. To generate more effective gradients updates during the training process for few-shot-leaming tasks, we compute the adaptive model updates by applying lookahead to both momentum and new gradients, leading to more stable learning

---

trajectory and better generalization. Lookahead is an effective optimization strategy by looking ahead sequence of fast weights [47] for fast converge and a corresponding slow gradient smooth oscillating. However, in few-shot-learning context, the fast weight will easily trigger the overfitring problem. And the limited update steps number make the slow gradient hard to smooth the oscillating.
We propose addressing this challenge by judi- ciously selecting the coefficients of the mo- mentum and new gradients using one-step lookahead. Specifically, before performing the accumulation process in Eq. 2 we apply lookahead both the momentum and the new gradients, resulting in two losses. In this con- text, the aggregation of weight updates should prioritize the update direction with lower loss. To achieve this, we employ a softmax opera-
Algorithm 2 Meta-AdaM
1: Input: Task distribution ((T)
2: Require: a, 8, step size hyper- hyper-paramene parameters
3: Randomly initialize meta model
Randomly initialize weights in LSTM: O₃
5: while not done do Sample batch of tasks To ppT)
7:
for all T; do
Finetune O with K examples:
tor to obtain the coefficients by normalizing
9:
0 = MctaMomentumlmner(( LSTM, TD)
the loss decreases. Formally, the proposed ap- proach is defined algorithm, Sy is a batch of support data, 17 is a fixed learning rate, (-) is softmax operator, and f(.) is a loss function that computes loss for batch data Sy with a given model 0. We
10:
end for
in Algorithm [ In this
11:
Update meta model and LSTM:
12:
< Vo Σr, LT, (fo))
13:
0 - 0, Vo. ΣT, CT. (fo;)
14:
end while
first generate two lookahead models by updating the current model parameters 0² using momentum m/-1 and new gradients g6 separately in line 6 and line 7 which results in 0r and 0 Then f() is applied to both models with S7, yielding loss changes for both lookahead models: A and AC In line (0). the softmax operator is used to normalize two loss changes so that the aggregation coefficients sum 1. The resulting coefficients are used to predict adaptive learning rates in line [I]) and aggregate the momentum and new gradients in line (12). Finally, line (13) updates the model parameters using accumulated momentum mn
With dynamic coefficients generated through lookahead, if either the momentum or new gradients lead to ineffective optimization direction, the small coefficient in line (12) can mitigate the negative impact and place greater emphasis on more effective optimization directions. As a result, the double-lookahead strategy can stabilize the momentum update and accelerate the convergence speed.
3.4 Dynamic Class Weighting Scheme for Classification Tasks
In FSL classification tasks employing meta-learning, the meta-trained base model is significantly influenced by prior knowledge. As result, certain classes with features closer to prior knowledge become easier to classify, leading to an imbalance that compromises the model's performance. To address this issue, we introduce a dynamic class weighting schema, which emphasizes classes contributing more to model optimization. We estimate a class's 's contribution by observing the changes in its loss. If a class has been well-learned, the loss will be stable, and minor loss changes will be observed. In contrast, if there is a substantial loss drop for particular class, it may contribute more to future updates. Based on the loss changes for each class, we use softmax operator to obtain aggregation weights for every class. The accumulated loss is the weighted summation of all class losses. This process is defined as follows:
C. C. = [2(S, 0'-1),
(5)
8
Wc = o oC( (TT/T),
(6)
L. wh Ch
k=
where fz(-) is loss function that computes losses for each class based on support data Sy, C' is the number classes task T, and T is the temperature used in the softmax operator (-)). In the above schema, if the class & has a larger loss L)., its corresponding aggregation weight wy will be larger, leading to a stronger optimization signal from class k. Thus, the class that needs to be more well-trained will obtain more attention, leading to more effective optimization.
5

---

Table 1: Comparison results using Convnet4 on Mini-ImageNet, mageed and Cifar 100
datasets. We report performances terms of accuracy (%) with standard deviation. BBsseline MC uses larger Convnet4 backbone network, and uses extra data regularization method.
Dataset
Method
5-way-1-shot
5-way-5-shot
Meta-I.STM
30
43.44 0.77
60.60 0.71
MAML 7
48.70 = 1.75
63.11 + 0.92
Meta-SGD 18
50.47 1.87
64.03 0.94
MAML+ALFA
3
50.58 = 0.51
69.12 = 0.47
e3bm 20
53.2
65.1
Mini-ImageNet
Sparse-MAML
51.04 = 0.59
68.05 + 0.84
MAML+SiMT
51.49 = 0.18
68.74 = 0.12
McTAL 40
52.63 = 0.37
70.52 = 0.29
Meta-AdaM (ours)
52.00 = 0.49
70.70 = -0.49
MC 25
55.73 = 0.94
70.33 = 0.72
MAML 7
49,06 = 0.50
67,48 = 0.47
MAML+ALFA
3
53.16 0.51
70.54 = 0.46
e3bm (00
52.1
70.2
TieredlmageNet
SIM
42
52.51 = 0.21
69.58 0.11
MeTAAL 40
54.34 = 0.31
70,40 = 0.21
Meta-AdaM (ours)
53.93 0.49
72.66 0.49
e3bm (20]
39.9 38.20 = 0.48
52.6 49.94 = 0.49
MAML 7
Cifar100
MAML+ALFA 3
39.77 = 0.48
53.39 = 0.49
Meta-AdaM (ours)
41.11 = 0.49
56.32 + 0.49
3.5 Meta-AdaM
Given a task distribution p(T), our goal is to meta-learn an optimal model initialization @ and an adaptive learning learner 0, In each iteration, we sample a batch of tasks from p(7). For each sampled task, we fine-tune an adapted model O' and update 0 and 0₁ using Algorithm During testing, we directly utilize 0 and O; for new task.
present our proposed Meta-AdaM in Algorithm 2
4
Experiments
We conduct experiments to evaluate the proposed Meta-AdaM using three benchmark datasets.
4.1
Experimental Settings
This section describes datasets and backbone models used to evaluate our Meta-AdaM.
Table Hyperparameters for experiment. Hyperpameter tasks batch size inner learning rate 7) outer learning rate a, B inner fine-tune step training epochs
Value
2
Datasets. We evaluate the proposed methods us- ing three datasets: Mini-ImageNet [10]. Tiered- ImageNet (33. and Cifar100 [16] datasets. Mini- ImageNer dataset contains 100 classes with 600 sam- ples per class. Each data sample is an 84 84 colored image. Following previous works [30, we split 100 classes into 64, 16, and 20 class groups for training, validation, and testing. The image size of Tiered-ImageNet is also 84 84. This dataset consists of 608 classes with 779,165 images. These 608 classes are further combined into 34 high-level classes. These high-level classes are divided into 20, 6, and 8 class groups for training, validation, and testing [31), respectively. considers the class similarity when divide the datatset to ensure that the distribution of train and test data is very different. Cifarl00 dataset for few-shot learning contains 60,000 images with 100 classes. The image size is 32 x 32. The 100 classes are divided into 60, 20, and 20 groups for training, validation, and testing. These datasets are bench-marking datasets in the FSL domain.
0.01
0.001
5
100
outer steps in each epoch
500
Seered-ImageNet
Backbone models. We evaluate the proposed Meta-Adal using two backbone models: Convnet4 and R2 12]. Convnet4 is 4-layer Convolutional Neural Network with 32.9 thousand trainable

---

Table 3: Comparison results using ResNet12 on Mini-ImageNet, TieredImageNet, and Cifar 100
datasets. We report accuracy (%) with standard deviation. "Baseline MC uses WRN-28-10 as backbone and extra data regularization method.
Dataset
Method MAML
5-way-1-shot
5-way-5-shot
58.37 0.49
69.76 0.46
MAML+ALA
3
59.74 = 0.49
77.96 + 0.47
MAML+SiMT
56.28 0.63
72.01 0.26
Mini-ImageNet
Sparse-MAML
44
56.39 = 0.38
73.01 = 0.24
MeTAL 40
59.64 = 0.38
76.20 = 0.19
Meta-AdaM (ours)
59.89 = 0.49
77.92 1 0.43
MC 25
64.40 = 0.10
80.21 = 0.10
MAML 7
58.58 0.49
71.24 0.43
MAML+ALFA
3
64.62 + 0.49
82.48 + 0.39
MAML+SIMT 42
59.72 = 0.22
74.40 = 0.90
Tiered- -raee-e
McTAL 40)
63.89 = 0.43
80.14 = 0.40
Meta-AdaM(ours)
65.31 0.48
85.24 = 0.35
MC 25
67.21 = 0.10
82.61 = 0.10
MAML 7
38.79 0.48
51.65 0.49
C0
MAML+ALFA 3
40.88 = 0.48
54.54 1 0.49
Meta-AdaM (ours)
41.12 0.49
56.14 0.49
parameters, while Resnet12 is an 18-layer Convolutional Neural Network featuring 4 million trainable parameters. By employing these two models, we demonstrate that our proposed optimizer is effective on both small and large networks.
Hyperparameters. We show the hyperparamters in Table which applies to each dataset and each backbone. To ensure fair comparison, we follow the setting of (3] and report the test performance of the ensemble of top-5 performing models on the validation set.
4.2 Results Using A Small Network
We first evaluate our Meta-AdaM using the Convnet4 network three datasets. The experiments conducted under two settings: 5-way- I-shot and 5-way-5-shot, where a-way-b-shot means each task contains classes and there are b training examples for each class. We compare our Meta-AdaM as main baseline. summary of the comparison results be found in Table ! From the results,
with Meta-L.STM [30), MAMI. 77], Meta-SGD [T8], and MAML+ALFA
observe that our method achieves promising results on all three datasets under two settings. Under the 5-way- 1-shot setting, our Meta-adaM outperforms MAML and MAML+ALFA by margins of 3 3.30% and 1.42% on Mini-ImageNet, 4.87% and 0.77% on TieredImageNet, and 2.91% and 1.34% on Cifar100, respectively. Under the 5- way-5-shot setting, our method outperforms MAML and MAML+ALFA by margins of 7.59% and 1.58% on Mini- Imageemma,Net,, 5.18% and 2.12% on TieredImageNet, and 6.38% and 2.93% Cifar100, respectively.
These promising results demonstrate that our Meta-AdaM can consistently yield more effective optimization results on various benchmark datasets. Notably, the performance gains are even larger in the 5-way-5-shot setting. Given that both the momentum and the new gradients estimated from few training examples could be noisy with only training example per class, it is difficult to produce effective gradient updates from them. The power Meta-AdaMa is better reflected by slightly increasing the number of training examples. We also compare methods with other recent research
works, including e3bm (20), Sparse-MAML. [44) MAML+SiMY 42 and MeTAL [40] We can observe that our methods outperform the other methods in most of the settings.
4.3 Results Using A Large Network
In the previous section, we evaluated our Meta-AdaM using a small network, Convnet4, as backbone model. However, optimizing a large model is more challenging due to the risk of overfitting. In this section, we conduct experiments on larger backbone model, Resnet1 further demonstrate the effectiveness of our Meta-AdaM. The evaluation is conducted under 5-way- 1-soot and 5-way-5-

---

Table 4: Abalation study results using the Mini-ImageNet dataset on the Convnet4 backbone network.
Momentum
Look-ahead
Adaptive Ir
5-way-1-shot
5-way-5-shot
51.64:0.49
68.80=0046
X
49.30:1.49
64,09=t0.47
X
49.02++0.50
67.98++0.66
X
X
48.28+0049
63.27+0048
X
48.70-11.75
63.11++0992
Table 5: Comparison result LSTM and MLP using Convnet4 on Mini-ImageNer, TieeedImageNet,
and Cifar100 100 datasets. We report performances in terms of accuracy (%) with standard deviation.
Dataset
Method
5-way-1-shot
5-way-5-shot
Meta-Adam with MLP
52.271 0.49
70.48 + 0.49
Mini-ImageNNt
Meta-AdaM with LSTM
52.00 + 0.49
70.70 + 0.49
Meta-Adam with MLP
53.48 + 0.49
72.60 + 0.49
Nieeet
Meta-AdaM with LSTM
53.93 + 0.49
72.66 + 0.49
Meta-Adam with MLP
40.28 + 0.49
54.96 0.49
Cifar100
Meta-AdaM with LSTM
41.11 + 0.49
56.32 + 0.49
shot settings using the Mini-ImageNet,, Tiered- and Cifar100 datasets. The comparison results with MAML and MAML+ALFA summarized in Table 3
Under the 5-way-1-shen setting, we observe margins of 0.15%, 0.69%, and 0.24% improvements compared to previous state-of-the-ar models the Mini-ImageNee, TieredImageNet, and Ci datasets, respectively. Under the 5-way-5-shot setting, our Meta- Aeta--AaaM outperforms ALFA by margins of 2.76%, and 1.60% mageed,,,eeNet, and Cifar lffar100 datasets respectively. On Mini-ImageNer our method gives equal performance compared to MAML+ALFA. Our method also outperforms other methods, including MAML+SiMT [42] and MeTAL 40) The consistently promising results on the three benchmark datasets demonstrate that our Meta-AdaM is effective when used to optimize larger backbone model.
4.4 Ablation Study
In this study, we introduce three novel components in our Meta-AdaM: momentum, double lookahead and an adaptive learning rate via a meta-leamedd learner. Also, we introduce a dynamic blass-weighhing scheme for loss functions specially designed for classification tasks. In this section, we evaluate the individual contributions of each component. accomplish this, we conduct experiments in both 5- --way--shho and 5-way-5-shot settings, utilizing Convnet4 as the backbone model on the Mini-ImageNet dataset.
4.4.1 Momentum, Double lookahead and Adaptive learning rate
Specifically, we test varying combinations of the first three components and assess the performance of the resulting optimizers. These comparative results can be summarized in Table The results in Table demonstrate that each component contributes the overall performance. Specifically, heuristic aggregation of momentum does not significantly enhance the performance, with a 0.42% decrease observed in the 1-shot and modest 161 increase in the 5-shot setting compared the original MAMI.. However, incorporating momentum with double look-ahead strategy leads to superior results over those from the momentum-on/e setting. This enhanced optimizer surpasses the original MAML by 0.6% in the 1-shot and 0.98% in the 5-shot settings. The adaptive learning rate also plays crucial role. By solely applying this adaptive learning rate the inner loop, the resultant optimizer outperforms the original MAML by 0.32% in the 1-shot and 4.87% in the 5-shot settings. When these three components are combined, the performance improvement is even more significant. The momentum with look-ahead results in more stable and efficient training optimization process, thereby aiding the meta-learner better estimating the adaptive learning rates. From these results, we see a performance boost of 2.94% in the 1-shot and 5.69% in the 5-shot settings. To further evaluate the effect of weight update history, we conduct additional experiments that substitute the LSTM network with MLP network with same inputs. The results summarized in Table 5 From
8

---

Table 6: Ablation Study for class dynamic weight using Convnet4 on Mini-ImageNet,, TieredIma- geNet, and Cifar100 datasets
Dataset
Method
5-way-I-shot 50.58 = 0.5l
5-way-5-shot
ALFA
69.12 = 0.47
ALFA+DW
50.65 0.49
70.02 0.45
Mini-ImageNet
Meta-AdaM w/o DW
51.64 = 0.49
68.80 + 0.46
Meta-AdaM (ours)
52.00 0.49
70.70 0.49
ALEA
53.16 = 0.51
70.54 = 0.46
ALFA+DW
53.54 = 0.49
72.36+ 0.19
Neeee
Meta-AdaM w/o DW
53.62 = 0.50
71.57 = 0.49
Meta-AdaM (ours)
53,93 0.49
72.66 + 0.49
ALFA
39.77 = 0.48
53.39 0.49
ALFA+DW
40.79 = 0.19
54.34 = 0.48
10
Meta-AdaM w/o DW
40.14 0.49
54.36 0.50
Meta-AdaM (ours)
41.11 = 0.49
56.32 = 0.49
change for tack Cifart00
Loss change for fask Teeeeet-mageeNe
Loss change test task Mini-ImageNNet
Mado-AAd MMMM
89-
Mata-AAdMM
MaMM ALFA
ALFA
ALFA
4-
Step
Figure 2:
Visualization of loss changes for each step. The left, middle, and right figures are the average loss changes and 95% confidence interval for each step on Mini-ImageNet, NeageN,, and Cifar100, respectively.
the result, LSTM outperform MI. network in 5 out of experiment setting, which indicates that the weight update history is important.
4.4.2 dynamic class-weighting scheme
To demonstrate the effect of dynamic class, conduct additional experiments four configurations: ALFA, ALFA with Dynamic Class Weighting, Meta-AdaM excluding class weighting. and full Meta-AdaM and summarize in result in Table 6 The result shows that Meta-AdaM without Dynamic Class Weighting outperforms ALFA out of 6 evaluated settings. Noticeably, Dynamic Class Weighting provides a tangible enhancement when integrated with ALFA.
4.5 Visualization of Optimization Process
In this section, we visualize the optimization process of our Meta-AdaM and compare it with ALFA. We randomly select 900 testing tasks from three datasets: Mini-ImageNNe, 100. The experiment setting is 5-way-1-shot using Convnet4 as the backbone model. We use the checkpoints with the highest validation accuracy for Meta-AdaM and ALFA. We calculate the average loss change for all 900 tasks in each update step and plot the results in Figure 2 From the plot, our Meta- AdaM helps the model find a better optimum than ALFA.
Imageee, and
Additionally, We conducted experiments on 300 5-way- 1-shot test tasks on Cifar100 with Convnet4 as the backbone model. We include the change inner learning rate n and a and in the "Double look ahead strategy." in Figure As illustrated in Figure on the left, our optimizer tends to initially predict smaller learning rates due to the substantial conffict between momentum and gradient. smaller learning rate will lead to slow convergence rate in the first few steps, providing spaces for accumulating high-quality momentum. Therefore, from Figure 2 the loss from our method decreases slower than ALFA. Once the momentum quality is good after several steps, the predicted learning

---

Predicted learning rate for test fask Cifar100
Mean and variance alpha and beta for test fask Cifar100
0.60
Alpha
Prodleted Iearning rate
0,175
Beta
0.55
6.125
Value
0.50
0.45
0.075
0.050
0.40
Step
Step
Figure 3: Left: Visualization of predicted learning rates in 5 steps with variance bonds. learning rates tend to increase along with the optimization process. Right: Visualization of computed coefficients a and 3 of 5 steps with variance bonds in the fine-tuning process. The coefficients for momentum small the beginning due their high variances.
The predicted
rates become larger, leading to a better optimum than ALFA. This also highlights our contribution to using double-look-ahead to handle highly variable momentum.
Figure 3 on the right shows that the gradients give lower losses until the first three steps, so the gradient a coefficient is larger than for momentum 3. However, after accumulating for few steps, the accumulated momentum can generate lower loss and a larger coefficient. Such results support the intuition that the momentum quality improves over the trajectory.
4.6 Efficiency Study of Meta-AdaM
In this section, assess the efficiency of Meta- AdaM. In Meta-AdaM, we employ double looka- head strategy to evaluate the momentum and new gra- dients, which may introduce additional computational costs. To evaluate these added costs, we compare the total training time of MAML,, MAML+ALFA, and Meta-AdaM. We use the experimental setting of way-1-shot on the Mini-ImageNNe dataset and record the running time on an NVIDIA RTX A4000 GPU. The results are summarized Table The results
Table 7: Average running time of one epoch.
Method
Running time (s)
MAML
252.04
MAML+ALFA
560.63
Meta-AdaM
595.91
indicate that our AdaM takes slightly longer running time than ALFA. However, considering the performance improvements brought by Meta-AdaM, the additional computational resources required by Meta-AdaM are justifiable.
4.7 Predicted Negative Learning Rate
During our experiments, the predicted learning rate our adaptive learning rate component could occasionally be negative. This phenomenon has been described in [4] and observed in ALFA (3] which employs a two-layer network to predict learning rates. Interestingly, applying negative learning rates to trainable parameters in some update steps can lead to better optimization results. However, determining which parameters are suitable for a negative learning rate and when apply the negative learning rate remains be clarified. Further research exploring the role of negative learning rates in meta-lcarning may contribute to improving the optimization of meta-nnn models.
Conclusion
In this work, propose a pettallarrinn-specifff optimizer, Meta-AdaM, which incorporates three novel optimization components that can hetter utilize update signals: momentum and new gradients. Specifically, we propose to meta-learn an adaptive learning rate learner using an LSTM network by looking back at the weight update history. We also introduce double lookahead algorithm to dynamically accumulate momentum, leading to more stable and effective momentum update and gradient update. Moreover, we propose a dynamic class weighting scheme to balance the losses across different classes. Experimental results demonstrate the effectiveness of our proposed Meta-AdaM.
10

---

Acknowledgement
This work has been supported in part by National Science Foundation grant II12104797..
References
[1] S.-i. Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5):
185-196, 1993.
[2] M. Andrychowiez, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
[3] Baik, M. Choi, J. Choi, H. Kim, and K. M. Lee. Meta-learning with adaptive hyperparameters. Advances in neural information processing systems, 33:20755-2007 2020.
[4] A. Bernacchia. Meta-learning with negative learning rates. arXiv preprint 10X09:2102.00940
2021.
[5] M. Donini, L. Franceschi, O. Majumder, M. Pontil, and P. Frasconi. Marthe: scheduling the learning rate via online hypergradients. Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 21 1122, 2021.
[6] T. Dozat. Incorporating nesterov momentum into adam. The Eleventh International Conference on Learning Representations, 2016.
[7] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
[8] C. Finn, K. Xu, and Levine. Probabilistic model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
[9] S. Flennerhag, A. A. Rusu, R. Pascanu, F. Visin, H. Yin, and R. Hadsell. Meta-learning with warped gradient descent. arXiv preprint arXiv: vrXiv:/909.0000 1909,0005, 2019.
[10] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4367- 4375, 2018.
[11] I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[13] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning in neural networks: A survey, 2020.
[14] D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint 182 2014.
[15] M. J. Kochenderfer and T. A. Wheeler. Algorithms for optimization. Mit Press, 2019.
[16] A. Krizhevsky, Hinton, Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[17] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/VV conference on computer vision and pattern recognition, pages 10657-10665. 2019.
[18] 7. Li, Zhou, Chen, and H. Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.098355 2017.

---

[19] L. Liu, H. Jiang, P. He. W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond. arXiv preprint Sa0i:::9000.03625.. 2019.
[20] Y. Liu, B. Schiele, and Q. Sun. An ensemble of epoch-wise empirical bayes for few-shot learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XVI 16, pages 404 4214 Springer, 2020.
[21] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-learner. arXiv preprint arXivv 2017.
[22] M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning. MIT press, 2018.
[23] T. Munkhdalai and H. Yu. Meta networks. In International conference on machine learning. pages 2563. PMLR, 2017.
[24] A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 2018.
[25] E. Park and J. B. Oliva. Meta-curvature. Advances in Neural Information Processing Systems, 32, 2019.
[26] A. Phaphuangwittayakuu Y. Guo, and F. Ying. Fast adaptive meta-learning for few-shot image generation. IEEE Transactions on Multimedia, 14:2205-22111 2021.
[27] B. T. Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1-17, 1964.
[28] J. Rajendran, A. Irpan, and E. Jang. Meta-learning requires meta-auggeenattion. Advances in Neural Information Processing Systems, 33:5705-5715, 2020.
[29] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-lcarning with implicit gradients. Advances in neural information processing systems, 32, 2019.
[30] S. Ravi and H. Larochelle. Optimization as model for few-shot learning. In International conference on learning representations, 2017.
[31] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:1803.00676 2018.
[32] S. Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv: 1609.04777 2016.
[33] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):111-25,, 2015.
[34] A. A. Rusu, D. Rao, Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960. 2018.
[35] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.
[36] Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[37] Y. Song, T. Wang, S. K. Mondal, and J. P. Sahoo. A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities. arXiv preprint arXiv:2205.0674. 2022.
[38] Q. Sun, Y. Liu, T.-S. Chua, and B. Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 403-412, 2019.
12

---

[39] M. Sundermeyer, R. Schlüter, and H. Ney. Lstm neural networks for language modeling. In Thirteenth anmual conference of the international speech communication association, 2012.
[40] H. K. D. C. J. M. K. M. L. Sungyong Baik, Janghoon Choi. Meta-learning with task-adaptive loss function for few-shot learning. In International Conference Computer Vision (ICCV), 2021.
[41] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance initialization and momentum in deep learning. In International conference machine learning, pages 1139- 1147. PMLR, 2013.
[42] J. Tack, J. Park, H. Lee, J. Lee, and J. Shin. Meta-learning with self-improving momentum target. In Advances Neural Information Processing Systems, 2022.
[43] Vicol, I.. Metz, and J. Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In International Conference on Machine Learning. pages 10553-10563. PMLR, 2021.
[44] von Oswald, D. Zhao, Kobayashi, Schug, M. Caccia, N. Zucchet, and Sacramento. Learning where to learn: Gradient sparsity in meta and continual learning, 2021.
[45] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3)11-34 2020.
[46] O. Wichrowska, N. Maheswaranathan, M. W. Hoffman, S. G. Colmenarejo, M. Denil, N. Freitas, and Sohl-Dickstein. Learned optimizers that scale and generalize. In International conference on machine learning. pages 3751-3760. PMLR, 2017.
[47] M. Zhang, J. Lucas, J. Ba, and G. E. Hinton. Lookahead optimizer: k steps forward, 1 step back. Advances neural information processing systems, 32, 2019.
13

---

Table 8: Comparison results on few-shot regression tasks. The target function for regression is sine curve y(x) sin(wx). We report MSE error with a 95% confidence interval.
Method
Training
5-shot Testing
10-shot Testing
20-shot Testing
MAML
5-shot
1.13 = 0.08
0,85 = 0.14 0.63 0.12
0.71 = 0.12
Meta-SGD 18
5-shot
0.90 0.16
0.50 0.10
ALFA
5-shot
0.60 + 0.04
0,41 + 0.02
0.25 + 0.22
Meta-AdaM
5-shot
0.52 0.04
0.35 0.04
0.23 0.01
MAMLE
10-shot
1.17 = 0.16
0.77
0.56 0.08
Meta-SGD 18
10-shot
0.88 0.14
0.09
0.06
ALFAB
10-shot
0.72 = 0.16
0,45 0.03
0.26 0.02
Meta-AdaM
10-shot
0.66 0.22
0.34 + 0.05
0.23 + 0.01
MAML
20-shot
1.29
0.76
0.48 +
Meta-SGD[I8
20-shot
1.01 0.17
0.54 0.08
0.31 0.05
ALFA
20-shot
1.01 0.18
0,48 0.09
0.25 0.03
Meta-AdaM
20-shot
0.95 + 0.11
0.43 + 0.06
0.23 + 0.03
Loss for each class-No dynamic class weight
Loss for each class-dynamic class weight
35
classl
class]
class2
class2
class3
class3
classA
classé
class5
class5
LOSS
2
Step
Step
Figure 4: Visualization of dynamic class weighting scheme. The left and right figures show the optimization process without and with the dynamic class weighting scheme, respectively.
Appendix
7.1 Result on regression task
To further prove the efficiency of our method, we conduct additional experiments on few-shot regression tasks and show the result in Table g We follow the experiment setting of the Meta- SGD[I8] The regression task is to map the underlying function based on the input. The target function y(x) A sin(w where amplitude A, frequency w, and phase all follow a uniform distribution with the interval [0.1, 5.0]. [0.8, 1.2]. and [0, л). We train three few-shot regression learners on 5-shot, 10-shot, and 20-shot tasks separately. Each learner will test on 100 5-shot, 10-shot, and 20-shot testing tasks. We report MSE errors with 95% confidence interval.
We compare our approach with MAML Meta-SGD Meta-SGD+ALFAN and Meta-SGD + META-AadM. The result has shown that our method outperforms the baselines, Meta-SGD and Meta-SGD+ALFA all settings. The promising results indicate that our proposed method is also effective on few-shot regression tasks.
7.2 Study of the Dynamic Class Weighting Scheme
In previous section, we present quantitative results to demonstrate the effectiveness class weighting scheme. In this section, we visualize the loss changes for each class when using this scheme. We compare the loss function using our scheme and the one without class weighting scheme. Each optimization process consists of five update steps. The left figure in Figure 4 displays
dynamic
14

---

the loss changes of each class without employing class weighting scheme. Class 2 has significantly lower loss than the other classes. For the last four steps, only the loss of class 2 optimized, while the other four classes remain under-optimized. Such situations can lead to model collapse problems in meta-learning [26], which means the model tends to classify all testing examples into one or two classes. The right figure in Figure&illustren illustrates the loss changes when using the dynamic class weighting scheme. Class has a lower loss the first step compared to the other classes. The optimizer focuses more the other four classes by emphasizing those with larger losses. Ultimately, all class losses converge towards the optimization endpoint, effectively mitigating the risk of model collapse problems.
15


Reviews:
[{"review_id": "jfNIn8Wirb", "reviewer": "Reviewer_xuqU", "summary": "In this paper, the authors aim to solve the few-shot learning problem by proposing a meta-learned learning rate learner. This learner utilizes weight-update history as input to predict more appropriate learning rates for rapid convergence. Furthermore, they incorporate momentum into the optimization process of few-shot learning via a double look-ahead mechanism, enabling rapid convergence similar to many-shot settings. Extensive experimental results are provided to show the effectiveness of the proposed method.", "questions": "Please address my concerns in the \"weaknesses\"", "limitations": "See my concerns in the \"weaknesses\"", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and easy to read. A lot of figures are included to explain the proposed method.\n\n&nbsp;\n\n- The proposed method is technically sound. It is reasonable to use meta-learning to adjust the learning rates. \n\n&nbsp;\n\n- Extensive experimental results are provided. \n", "weaknesses": "- The proposed idea of using a model to output the learning rates is not novel. A similar thing has been done in [a], which uses a meta-learned model to output a series of hyperparameters, including learning rates.\n\n&nbsp;\n\n- The performance of the proposed method is much lower compared to recent few-shot learning methods. For example, the 1-shot accuracy on miniImageNet is only 59.89% in Table 3. Existing methods, e.g., [b] and [c], achieve more than 68% accuracy in the same setting. It is not necessary that the proposed method beats all existing methods. However, it should not be much lower than other methods. At least, the performance should be comparable. Besides, it is necessary to indicate the proposed method doesn’t achieve the SOTA performance and compare the SOTA in the tables. \n\n&nbsp;\n\n- All important related works are missing, e.g., [a-d]. They should be compared and discussed. \n\n&nbsp;\n\nOverall, I think the quality of this paper is not satisfactory. The idea is a little bit incremental. The overall performance is not significant. Many important related works are missing. Therefore, I recommend rejection. The authors need to show the proposed method can be applied to recent popular baselines, e.g., [b, c, d]. I don’t think the current version is ready to be presented at NeurIPS.\n\n&nbsp;\n\n[a] An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning, ECCV 2020. \n\n[b] Partner-Assisted Learning for Few-Shot Image Classification, ICCV 2021.\n\n[c] DeepEMD: Differentiable Earth Mover’s Distance for Few-Shot Learning, TPAMI. \n\n[d] Rectifying the Shortcut Learning of Background for Few-Shot Learning, NeurIPS 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Table 1:** Additional Comparison results using Convnet4 on Mini-ImageNet, TieredImageNet, and\nCifar100 datasets. We report performances in terms of accuracy (%) with standard deviation.\n\n|Dataset|Method| 5-way-1-shot|5-way-5-shot |\n|:-----------------:|:-------------------------:|:------------------------:|:------------------------:|\n| Mini-Imagenet |     e3bm[3]        |53.2| 65.1|\n|Mini-Imagenet |Meta-AdaM (ours)| **52.00** ± 0.49| **70.70** ± 0.49|\n|TieredImageNet |e3bm[3]| 52.1| 70.2|\n|TieredImageNet |Meta-AdaM(ours)| **53.93** ± 0.49| **72.66** ± 0.49|\n|Cifar100|e3bm[3]| 39.9 |52.6|\n|Cifar100|Meta-AdaM (ours) |**41.11** ± 0.49| **56.32** ± 0.49|\n________________________________________________________________________________________________________________________\n**Question 1**: *The proposed idea of using a model to output the learning rates is not novel. A similar thing has been done in [a], which uses a meta-learned model to output a series of hyper-parameters, including learning rates.*\n\n**Answer**:\n\n It seems there’s been a misunderstanding regarding the originality of our methods. We wish to emphasize our pioneering contributions in two aspects: \n* Integrating momentum into a meta-learned adaptive optimizer.\n* Utilizing the history of weight changes for adaptive learning rate prediction\n\n Existing methods have been hindered from employing momentum for weight updates due to pronounced variances in momentum during initial optimization steps. To overcome this limitation, we introduce a double-look-ahead strategy. Furthermore, our approach involves leveraging momentum for adaptive learning rate prediction, which more accurately reflects weight change compared to other adaptive learning rate prediction methods. The key difference between our work and [1] and other related works is that our learning rate prediction is based on the update history. Our LSTM network takes accumulated momentum and current gradients to grasp the weight update history, which is more critical than gradients and weights [1, 2].\nIn contrast, [3 ] and other related works mostly use average loss, input features or weight, and current gradient as their inputs to determine the learning rate. They focus more on the current update rather than the update history. The experiment studies in section 4.2 and section 4.3 and the additional results presented in Table 1 show that our methods are more effective than other related works.\n\n__________________________________________________________________________________________________________________________\n**Question 2**: *The performance of the proposed method is much lower compared to recent few-shot learning methods. For example, the 1-shot accuracy on miniImageNet is only 59.89 % in Table 3. Existing methods, e.g., [b] and [c], achieve more than 68% accuracy in the same setting. It is not necessary that the proposed method beats all existing methods. However, it should not be much lower than other methods. At least, the performance should be comparable. Besides, it is necessary to indicate the proposed method doesn’t achieve the SOTA performance and compare the SOTA in the tables.*\n\n**Answer**:  \n\nThank you for highlighting these studies. We want to note that the methods presented in [4,5,6] (b,c,d in your review) use pre-trained models. In contrast, our primary focus centers around the proposed Meta-Adam optimizer. Consequently, the reported performance results do not stem from pre-trained models, ensuring minimal interference from other factors. While time constraints have prevented us from providing performance data using pre-trained models, it’s crucial to understand that our techniques complement them. As such, our methods can seamlessly be integrated with gradient-based meta-learning strategies that employ pre-trained models.\n\n_________________________________________________________________________________________________________________________\nReferences\n\n[1] I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT Press, 2016.\n\n[2] M. J. Kochenderfer and T. A. Wheeler. Algorithms for optimization. MIT Press, 2019.\n\n[3] Y. Liu, B. Schiele, and Q. Sun. An ensemble of epoch-wise empirical bayes for few-shot learning.In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pages 404–421. Springer, 2020.\n\n[4] X. Luo, L. Wei, L. Wen, J. Yang, L. Xie, Z. Xu, and Q. Tian. Rectifying the shortcut learning of background for few-shot learning. Advances in Neural Information Processing Systems, 34:13073–13085, 2021.\n\n[5] J. Ma, H. Xie, G. Han, S.-F. Chang, A. Galstyan, and W. Abd-Almageed. Partner-assisted learning for few-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10573–10582, 2021\n\n[6] C. Zhang, Y. Cai, G. Lin, and C. Shen. Deepemd: Differentiable earth mover’s distance for few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5632–5648, 2022.50"}, {"review_id": "0eUkoFzus1", "reviewer": "Reviewer_cqvx", "summary": "This paper introduces a novel meta-learned optimizer with momentum for few-shot learning. The proposed approach incorporates the parameter-update history by leveraging LSTMs to adaptively adjust the learning rate. To enhance convergence speed,  the momentum is integrated into the meta-optimizer by using a double look-ahead mechanism. Furthermore, to address imbalanced learning, a dynamic class weighing schema is introduced, encouraging optimal model optimization by assigning more weight to influential classes. The effectiveness of the proposed method is evaluated on multiple benchmarks, showcasing its performance in few-shot learning scenarios.", "questions": "* I would appreciate it if the authors and other reviewers could carefully verify the inclusion of all relevant previous works in this area within the paper. This thorough examination is crucial for evaluating the novelty of the paper, and it will significantly influence my final rating.\n\n* I would expect the author could improve the conciseness of the writing.\n\n* The dynamic loss contributes to the accuracy of the proposed method, but the paper does not mention such a technique in the introduction. I expect the author could provide an explanation. ", "limitations": "The paper includes the limitation in section 4.8 but fails to mention the negative social impact. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed meta-optimizer is novel, as it considers the history of weight updates and momenta when adapting the learning rate, which distinguishes it from previous works in the field. However, as my research expertise lies outside of the meta-optimizer domain, I cannot ascertain the paper's coverage of previous literature. Thus, I eagerly anticipate feedback from other reviewers with expertise in this topic.\n\n* The paper is written in a clear and concise manner, facilitating readers' comprehension of the ideas and technical contributions. Each sub-problem is precisely defined, including its limitations, followed by the proposal of techniques to address those issues. This clarity enhances the paper's accessibility and aids in conveying the research concepts effectively.", "weaknesses": "* The related work section may have missed some recent works or baseline methods on meta-optimizer for few-shot learning (FSL). All the existing works mentioned in Section 2.2.1 were published before 2020, which raises the question of whether there have been significant developments in this topic between 2021 and 2023. This could lead to confusion and a potential gap in the literature review.\n\n* The writing in the paper lacks conciseness. The author repeats high-level ideas and the limitations of previous works using very similar expressions multiple times throughout the entire paper. For instance, the description provided in Line 118, Line 129, and Line 139 are identical, which can be redundant and could benefit from consolidation.\n\n* The method section contains some content that should be included in the related works section. For example, in Line 145, the comparison between the proposed idea and a previous work should be presented in the related works section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Table 1:** Additional Comparison results using Convnet4 on Mini-ImageNet, TieredImageNet, and\nCifar100 datasets. We report performances in terms of accuracy (%) with standard deviation.\n\n|     Dataset        |        Method                 |  5-way-1-shot  |  5-way-5-shot    |\n|:-----------------:|:-------------------------:|:------------------------:|:------------------------:|\n| Mini-Imagenet |     e3bm(2020)[1]        |53.2| 65.1\n| Mini-Imagenet | Sparse-MAML(2021)[4] |51.04 ± 0.59| 68.05 ± 0.84|\n|Mini-Imagenet |  MAML+SiMT (2022)[3| 51.49 ± 0.18| 68.74 ± 0.12|\n|Mini-Imagenet    |MeTAL(2022)[2]|52.62 ± 0.37 |70.52 ± 0.29|\n|Mini-Imagenet|Meta-AdaM (ours)| 52.00 ± 0.49| **70.70 ± 0.49**|\n\n|     Dataset        |        Method                 |  5-way-1-shot  |  5-way-5-shot    |\n|:-----------------:|:-------------------------:|:------------------------:|:------------------------:|\n|TieredImageNet |e3bm(2020)[1]| 52.1| 70.2|\n|TieredImageNet |Sparse-MAML(2021)[4]|56.39 ± 0.38|73.01 ± 0.24|\n|TieredImageNet |MAML+SiMT(2022)[3]|52.51 ± 0.21| 69.58 ± 0.11|\n|TieredImageNet |MeTAL(2022)[2]| 54.34 ± 0.31| 70.40 ± 0.21|\n|TieredImageNet |Meta-AdaM(ours)| 53.93 ± 0.49| **72.66** ± 0.49|\n\n|     Dataset        |        Method                 |  5-way-1-shot  |  5-way-5-shot    |\n|:-----------------:|:-------------------------:|:------------------------:|:------------------------:|\n|Cifar100|e3bm(2020)[1]| 39.9 |52.6|\n|Cifar100|Meta-AdaM (ours) |**41.11** ± 0.49| **56.32** ± 0.49|\n___________________________________________________________\n\n\nTable 2: Additional Comparison results using Resnet12 on Mini-ImageNet and TieredImageNet. We\nreport performances in terms of accuracy (%) with standard deviation.\n| Dataset| Method  | 5-way-1-shot|  5-way-5-shot |\n| :-----------------: | :-------------------------:| :------------------------:|:------------------------:|\n|Mini-ImageNet|MAML+SiMT(2022)[3]| 56.28 ± 0.63| 72.01 ± 0.21\n|Mini-ImageNet|MeTAL(2022)[2]| 59.64 ± 0.38 |76.20 ± 0.19|\n|Mini-ImageNet|Meta-AdaM(ours)| **59.89** ± 0.49| **77.92** ± 0.43|\n\n| Dataset| Method  | 5-way-1-shot|  5-way-5-shot |\n| :-----------------: | :-------------------------:| :------------------------:|:------------------------:|\n|TieredImageNet|MAML+SiMT(2022)[3]|59.72 ± 0.22 |74.40 ± 0.90|\n|TieredImageNet|MeTAL](2022)[2] |63.89 ± 0.43| 80.14 ± 0.40|\n|TieredImageNet|Meta-AdaM(ours)| **65.31** ± 0.48| **85.24** ± 0.35|\n\n________________________________________________________________________________________________________________________\n**Weakness 1 & Question 1**:   *I would appreciate it if the authors and other reviewers could carefully verify the inclusion of all relevant previous works in this area within the paper. This thorough examination is crucial for evaluating the paper's novelty and will significantly influence my final rating.*\n\n\n**Answer**:\n\nThanks for your suggestion. We have carefully examined the related works, and the novelty of our work is integrating momentum into a meta-learned adaptive optimizer. Existing methods cannot utilize momentum for weights update due to the high variances of momen at the initial update steps. We address this challenge by double look-ahead. Also, we propose to utilize momentum for adaptive learning rate prediction.\n\nTo address your concern, we present the result of recent works in Table 1 and Table 2. The experiment section in our paper already presents sparse-MAML (2021)[4]. In addition, we include e3bm (2020)[1], MAML+SiMT (2022)[3], and MeTAL (2022)[2], which also work on improving the fine-tuning process. We can observe that our methods outperform the other methods in most of the settings.\n________________________________________________________________________________________________________________________\n\n**Weakness 2 & Question 2**: *The writing in the paper lacks conciseness. The author repeats high-level ideas and the limitations of previous works using similar expressions multiple times throughout the paper. For instance, the description provided in Line 118, Line 129, and Line 139 are identical, which can be redundant and could benefit from consolidation.*\n\n**Answer**: \n\nThanks for your suggestion. We will revise and reduce the redundancy in the final version.\n\n_________________________________________________________________________________________________________________________\n**Weakness 3**:  *The method section contains some content that should be included in the related works section. For example, in Line 145, the comparison between the proposed idea and previous work should be presented in the related works section.*\n\n**Answer**: \n\nThanks for your suggestion. We will add the missing baseline in the related work section of our final paper.\n\n_____\n\n**Question 3**:  *The dynamic loss contributes to the accuracy of the proposed method, but the paper does not mention such a technique in the introduction. I expect the author could provide an explanation.*\n\n**Answer**: \n\nThanks for pointing out this. We will add a discussion of dynamic loss in the introduction.\n_______\n\n**References**\n\n[1] Y. Liu, B. Schiele, and Q. Sun. An ensemble of epoch-wise empirical bayes for few-shot learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pages 404–421. Springer, 2020.29\n\n[2] H. K. D. C. J. M. K. M. L. Sungyong Baik, Janghoon Choi. Meta-learning with task-adaptive loss function for few-shot learning. In International Conference on Computer Vision (ICCV),2021\n\n[3] J. Tack, J. Park, H. Lee, J. Lee, and J. Shin. Meta-learning with self-improving momentum target. In Advances in Neural Information Processing Systems, 2022.34\n\n[4] J. von Oswald, D. Zhao, S. Kobayashi, S. Schug, M. Caccia, N. Zucchet, and J. Sacramento.\nLearning where to learn: Gradient sparsity in meta and continual learning, 2021.\n\n"}, {"review_id": "1QkG0YVJ3i", "reviewer": "Reviewer_W1we", "summary": "The paper proposes Meta-Adam, a meta-learned adaptive optimizer that employs momentum to rapidly adapt a meta-learned initial model to a few-shot task. The proposed framework consists of four components. First, an LSTM that predicts a learning rate for each weight inside the model using the weight update history in previous iterations. Furthermore, the optimizer employs momentum and additionally develops a look-ahead procedure for identifying environments where momentum hurts optimization and adjusting the parameters accordingly. Lastly, the loss function is weighted class-wise based on the class-specific loss during each gradient step to ensure that all class losses are optimized collectively. Experiments on mini-, tiered-ImageNet and Cifar100 benchmarks demonstrate the efficacy of the proposed method.", "questions": "See weaknesses for questions. Happy to improve my rating once the authors have addressed the concerns above.", "limitations": "There is no explicit discussion of limitations or potential negative societal impact of the work, although a brief discussion of some directions for future work, in particular with respect to negative learning rates, is provided. I strongly encourage the authors to provide a discussion of other potential limitations of the work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The method proposed explores interesting ideas around better-optimizing learning rates and momentum in few-shot adaptive meta-learned models. The components of the method are well-motivated, derived to address specific problems, and empirically effective.\n- Experiments demonstrate improvements in performance across various benchmarks compared to MAML-based benchmarks.\n- Ablation studies provided demonstrate the empirical efficacy of each component of the proposed method and how much they contribute to the overall improvements in few-shot image classification accuracy.\n- The paper is written clearly with reasonable notation and is easy to follow.", "weaknesses": "- All baselines that are compared are MAML-based. Although it's understandable given the focus of the paper on gradient adaptive few-shot image classification models, the method would still be compared against other groups (for instance, metric-based [1, 2, 3, 4]) when deciding to use a few-shot learner in an applied setting. It's useful to provide a discussion of this and potentially include a comparison to other non-MAML-based baselines.\n- Figure 2 / Section 4.5 - it would be interesting to see what the variance bounds are for each optimization method as averages can be uninformative as to how much loss reduction can vary per task. Furthermore, ALFA seems to achieve much faster convergence although a worse optimum compared to Meta-AdaM. Why is that the case?\n- 4.6 Dynamic class loss discussion lacks details as to the task/benchmark used in Figure 3. Without said details, it's difficult to evaluate whether the trends observed are statistically significant to support claims on reducing loss across all classes.\n\n[1] Prototypical Networks for Few-shot Learning\n[2] Improved Few-Shot Visual Classification\n[3] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes\n[4] Enhancing Few-Shot Image Classification with Unlabelled Examples", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Table 1:** Additional Comparison results using Convnet4 on Mini-ImageNet. We report performances in terms of accuracy (%) with standard deviation.\n|Dataset|Method| 5-way-1-shot|5-way-5-shot |\n|:-----------------:|:-------------------------:|:------------------------:|:------------------------:|\n|Mini-ImageNet|ProtoNet[1]|49.42 ± 0.78|68.20 ± 0.66|\n|Mini-ImageNet|Meta-AdaM(ours)|52.00 ± 0.49|70.70 ± 0.49|\n___\n\n**Question 1**\n\n*All baselines that are compared are MAML-based. Although it's understandable given the focus of the paper on gradient adaptive few-shot image classification models, the method would still be compared against other groups (for instance, metric-based [1, 2, 3, 4]) when deciding to use a few-shot learner in an applied setting. It's useful to provide a discussion of this and potentially include a comparison to other non-MAML-based baselines.*\n\n\n**Answer**\n\nThank you for highlighting these studies. In Table 1 above, we compare our result with ProtoNet [1] on Mini-Imagenet using Convnet4 as the backbone. The results show that our method outperforms the ProtoNet [1] in both settings.  For the other three methods, we want to note that the methods presented in [2,3,4] use pre-trained models. In contrast, our primary focus centers around the proposed Meta-Adam optimizer. Consequently, the performance results we have reported do not stem from pre-trained models, ensuring minimal interference from other factors. While time constraints have prevented us from providing performance data using pre-trained models, it's crucial to understand that our techniques complement them. As such, our methods can be integrated with gradient-based meta-learning strategies that employ pre-trained models as long as they employ an inner fine-tuning process.\n___\n\n**Question 2:** \n\n*Figure 2 / Section 4.5 - it would be interesting to see what the variance bounds are for each optimization method, as averages can be uninformative as to how much loss reduction can vary per task. Furthermore, ALFA seems to achieve much faster convergence, although a worse optimum compared to Meta-AdaM. Why is that the case?*\n\n**Answer:**  \n\nThank you for your recommendation. In Figure 1 of the global rebuttal, we present the variance bounds of both our method and ALFA across 500 test tasks. From the figure, we can see that after the third step, the losses associated with the test tasks closely align with the mean loss. The variances after step 3 are around $1 \\times 10^{−4}$. Therefore, the loss reduction works for the majority of\ntasks.\nFor the convergence rate of the first several steps, we empirically show the predicted learning rates across test tasks in Figure 2 in the global rebuttal. The results show that our optimizer tends to predict a lower learning rate at the first few steps since the conflict between momentum and gradient is large. This can be reflected in Figure 3 in the global rebuttal, where momentum coefficients are smaller than\ngradients. Using a smaller learning rate will lead to a slow convergence rate in the first few steps, which provides spaces for accumulating high-quality momentum. Once the momentum quality is good after several steps, the predicted learning rates become larger and lead to a better optimum than ALFA. This also highlights our contribution that using double-look-ahead to handle highly variable momentum.\n___\n\n**Question 3:** \n\n4.6 Dynamic class loss discussion lacks details regarding the task/benchmark used in Figure 3. Without said details, it’s difficult to evaluate whether the trends observed are statistically significant to support claims on reducing loss across all classes.\n\n**Answer:**\n\nThank you for drawing attention to this. In Section 4.6, we used a 5-way-1-shot task on the mini-Imagenet dataset. For the finalized version, we intend to broaden our evaluation to include additional tasks and benchmark datasets.\n___\n\n**References:**\n\n[1] Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical networks for few-shot learning. Advances in neural information processing systems\n\n[2] Bateni, P., Goyal, R., Masrani, V., Wood, F., & Sigal, L. (2020). Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14493-14502).\n\n[3] Requeima, J., Gordon, J., Bronskill, J., Nowozin, S., & Turner, R. E. (2019). Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems\n\n[4] Bateni, P., Barber, J., Van de Meent, J. W., & Wood, F. (2022). Enhancing few-shot image classification with unlabelled examples. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2796-2805).\n"}, {"review_id": "pgUgSVwHnC", "reviewer": "Reviewer_7eNt", "summary": "The authors introduce an adaptive optimizer for meta-learning that uses double lookahead to incorporate momentum in scenarios where ‘conventional’ approaches of using momentum usually fall short (few-shot inner-loop, due to high momentum/gradient fluctuations during initial steps). The use of an LSTM to predict the adapted learning rates allows for consideration of the weight-update history.  To further boost the obtained results, the authors additionally introduce dynamic weighting of the class-specific losses to aid a more balanced optimization / loss minimization.", "questions": "The three main points & questions are listed in the weaknesses section. However, I have several additional suggestions/questions I'd like comments from the authors on:\n\nSeveral experiments / ablation studies would imo strengthen this work: \n* What are the actual learning rates predicted within the inner loop? (Is there some ‘commonality’ within the same step across different test episodes?)\n* What is the ‘average’ mix of gradient and momentum contribution across inner-loop update steps (i.e. Algorithm 1, l.10)? Does this reflect  ‘intuitions’ of growing momentum along the trajectory, or maybe defy these? (visualization/comment would be interesting)\n* How big is the influence of the base learning rate in Algorithm 1? You are using it in lines 6-9 to predict the effect of taking a step of this size into the gradient or momentum direction – the step-size (and direction) of the actual step then taken however differs and is according to your comments ‘not limited’ (not even sign-wise) -> Is the lookahead still a ‘good and robust’ measure, or did you encounter difficulties? Does it change when choosing larger / smaller base-lrs, or is it rather robust (and why)? \n\nAdditional questions:\n* What is the size of the LSTM, and how much complexity does it add?\n* How is the temperature chosen for the dynamic class weighting, and how much does it matter & potentially differ across datasets?\n* I’m slightly confused about the message of Figure 2: While Meta-AdaM does converge to a better minimum/lower loss at the final step, ALFA seems to converge faster – It would be helpful for the authors to comment on this.  (In its current state, I am not convinced about this part of the paper in general, since we already see in the tables that Meta-Adam seems to converge to a better minimum, and the only fact the figure shows me is that it does so in a slower way than ALFA?) \n* Algorithm 2 ln 12/13 indicate SGD for outer loop – is this a simplification for conciseness, or did you choose SGD for the outer loop?\n\n\nSome remarks to further improve presentation:\n* Stating where the results for MAML are taken from would be helpful, since some of the datasets and architectures have not been evaluated in the original paper\n* Line 36/37: “[…] has been shown that weight-update history is more important than the weights themselves […]” -> Reference would be important! I am aware that some are given in line 131, but two entire books without any sub-indication are a rather large search space for the reader! -> more recent publication or chapter/sub-chapter of book would be appreciated\n* Sorting the references in ascending order (easier for reader to cross-check)\n* Typos/Layout: Fig. 1: space after “A”; mix of capitalized and non-capital words within sub-captions inconsistent\n\n**Note**: I do think the idea presented in this paper is interesting! Depending on the response from the authors, I am happy to reconsider my current rating.\n\n> UPDATE: raised rating to weak accept after rebuttal.\n", "limitations": "The authors provide the runtime in comparison to related approaches, which adequately addresses the main limitation I see for this approach.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "* Well written paper, easy to read and follow, good introduction of method and required background. Placement within related work well accomplished, and approach well motivated.\n* Generally interesting idea and approach: Incorporating momentum with the double look-ahead and predicting the learning rate based on the history are to me the main contributions.\n* Results of the overall approach seem solid.\n* For the most part, interesting evaluations including run-time.\n", "weaknesses": "Major: \n\n1. **Entanglement of ‘dynamic class weighting scheme’ and ‘optimiser approach’**: \n While an interesting contribution in itself, the “dynamic class weighting scheme” is highly entangled with the actual meta-adam approach throughout the reported results. While I do understand the pressure to ‘outperform’ other works, the reported results in its current state do in my opinion not allow for an insightful comparison of approaches & limits the reader’s insights in an unnecessary manner. I would highly encourage the authors to: \n   - Add the Meta-AdaM (w/o dyn-cl-w) to the tables\n   - Evaluate MAML or others together with their proposed dyn-cl-w. \n\n   This would allow the reader to better judge the influence of the individual contributions and their influence beyond this particular work!\n\n   Note: I am aware that Table 4 shows the result w/o dynamic class loss, but since the *paper’s main selling point and TITLE focuses on the optimiser*, this should be individually demonstrated and be part of an honest discussion.\n\n2. **Usefulness of the LSTM / history**: \n  How essential is the fact that the adaptive learning rate being predicted by an LSTM? How important is the history, really – which is one of the main claims of the paper! -> How does it compare to e.g. using an MLP with similar complexity and the same input (prev. momentum and current gradient)?\n\n\nMinor:\n\n3. **Dataset inconsistency** (Description & References):\n  The authors state 'Cifar100' as few-shot learning benchmark – however, there two inconsistencies that should be improved:\n   * The FSL community uses two different versions: Cifar-FS and FC100, that yield very different results due to their different complexities. The provided reference in the paper links neither of them, but the original non-FSL Cifar  -> clarification here is required.\n   * The description of the dataset (sec 4.1) states “*with 100 different labels of birds*”, which seems more like a description for the popular CUB dataset than Cifar -- I assume some mixup has happened here?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Table 1:** Ablation Study for class dynamic weight using Convnet4.\n| Dataset|Method|5-way-1-shot| 5-way-5-shot |\n| :----: | :-----:|:--------------:|:------------:|\n|Mini-ImageNet|ALFA |50.58 ± 0.51|69.12 ± 0.47|\n|Mini-ImageNet|ALFA+DW| 50.65 ± 0.49|70.02 ± 0.45|\n|Mini-ImageNet|Meta-AdaM w/o DW| 51.64 ± 0.49| 68.80 ± 0.46|\n|Mini-ImageNet|Meta-AdaM| 52.00 ± 0.49| 70.70 ± 0.49|\n|TieredImageNet|ALFA |53.16 ± 0.51| 70.54 ± 0.46|\n|TieredImageNet|ALFA+DW| 53.54 ± 0.9|72.36± 0.19|\n|TieredImageNet|Meta-AdaM w/o DW| 53.62 ± 0.50|71.57 ± 0.49|\n|TieredImageNet|Meta-AdaM| 53.93 ± 0.49|72.66 ± 0.49|\n|Cifar100|ALFA|39.77 ± 0.48|53.39 ± 0.49|\n|Cifar100|ALFA+DW|40.79 ± 0.19|54.34 ± 0.48|\n|Cifar100|Meta-AdaM w/o DW|40.14 ± 0.49|54.36 ± 0.50|\n|Cifar100|Meta-AdaM (ours)|41.11 ± 0.49|56.32 ± 0.49|\n\n**Table 2**: Comparison result LSTM and MLP using Convnet4.\n| Dataset|Method|5-way-1-shot| 5-way-5-shot |\n| :----: | :-----:|:--------------:|:------------:|\n|Mini-ImageNet|Meta-Adam with MLP| 52.27± 0.49| 70.48 ± 0.49|\n|Mini-ImageNet|Meta-AdaM with LSTM|52.00 ± 0.49| 70.70 ± 0.49|\n|TieredImageNet| Meta-Adam with MLP|53.48 ± 0.49|72.60 ± 0.49|\n|TieredImageNet|Meta-AdaM with LSTM|53.93 ± 0.49|72.66 ± 0.49|\n|Cifar100|Meta-Adam with MLP |40.28 ± 0.49 |54.96 ± 0.49|\n|Cifar100| Meta-AdaM with LSTM| 41.11 ± 0.49 |56.32 ± 0.49|\n\n**Weakness 1:**  \n*Add the Meta-AdaM (w/o dyn-cl-w) to the tables. Evaluate MAML or others together with proposed dyn-cl-w.*\n\n**Answer**\nWe conduct additional experiments on four configurations: ALFA [1], ALFA with Dynamic Class Weighting, Meta-AdaM excluding class weighting, and Meta-AdaM in Table 1. The results show that Meta-AdaM without Dynamic Class Weighting outperforms ALFA in 5 out of 6 evaluated settings. Notably, Dynamic Class Weighting can enhance other meta-learners like ALFA. \n___\n\n**Weakness 2** \n*How essential the adaptive learning rate is predicted by LSTM? How important is history-> How does it compare to e.g. using an MLP with similar complexity and the same input (prev. momentum and current gradient)?*\n\n**Answer**\nWe conduct additional experiments on LSTM and an MLP with the same inputs: accumulated momentum and gradients. The result is presented in Table 2. From the results, we can observe that LSTM outperforms MLP in most settings. This indicates the importance of weight update history.\n___\n**Weakness 3** \n*Dataset inconsistency*\n\n**Answer**\nThanks for pointing out that. We will change to the right description in our final paper.\n___\n\n**Question 1:**\n*What are the actual learning rates predicted within the inner loop? Is there some commonality within the same step across different test episodes*\n\n**Answer**\n We conduct experiments on test tasks on Cifar100 with Convnet4 as the backbone model and present the result in Figure 2 in the global rebuttal. From the result, we can observe the predicted learning rate is small at the beginning since the conflict between momentum and gradient is large in the first few steps. The learning rate becomes larger along with the update steps. This\nbehavior indicates the conflict between momentum and gradient is smaller, and we get a smoother update direction.\n___\n\n**Question 2**\n*What is the average mix of gradient and momentum across inner-loop update steps? Does this reflect intuitions of growing momentum*\n\n**Answer**\nWe conduct experiments on test tasks on Cifar100 with Convnet4 as the backbone and present the results in Figure 2. The result shows that the gradients give lower losses until the first three steps. So the coefficient for gradient α is larger than for momentum β. However, after accumulating for a few steps, the accumulated momentum can generate a lower loss and a larger coefficient. Such results support the intuition that the momentum quality improves over the trajectory.\n___\n\n**Question3**\n*How big is the influence of the base learning rate in Algorithm 1*\n\n**Answer**\nIn our experiments, we set the base learning rate as 0.01. We do some additional evaluation on base learning rates 0.001 and 0.0001. Under the 5-way-1-shot setting on Mini-Imagenet using Convnet4 as the backbone, the accuracy for learning rates of 0.001 and 0.0001 are similar, which indicates that LSTM can adjust the predicted learning rate accordingly. The base learning rate has a small impact on the overall performance.\n___\n**Question 4**\n*What is the size of LSTM, and how much complexity does it add*\n\n**Answer**\nThe LSTM used in our paper contains two layers with 30 and 50 hidden sizes for 5-way-1-shot and 5-way-5-shot tasks, respectively. The complexity will increase. Table 5 in our paper shows that our method has an extra 35-second running time compared to ALFA for one epoch.\n___\n\n**Question 5**\n*How is temperature chosen*\n\n**Answer**\nIn our experiment, we choose 15 for 5-way-1-shot tasks and 25 for 5-way-5-shot tasks during the dynamic class weighting in all our settings and datasets.\n___\n\n**Question 6**\n*Why ALFA seems to converge faster*\n\n**Answer**\nFor the convergence rate of the first several steps, we empirically show the predicted learning rates across test tasks in Figure 2 in the global rebuttal. The results show that our optimizer tends to predict a smaller learning rate at the first few steps since the conflict between momentum and gradient is large. This can be reflected in Figure 3 in the global rebuttal,\nwhere momentum coefficients are smaller than gradients. A smaller learning rate will lead to a slow convergence rate in the first few steps, providing spaces for accumulating high-quality momentum. Once the momentum quality is good after several steps, the predicted learning rates become larger, leading to a better optimum than ALFA.\n___\n\n**Question 7**\n*did you choose SGD for the outer loop?*\n\n**Answer**\nIn our experiment, we use Adam as our outer loop optimizer. In Algorithm 2 ln 12/13, we use a general update form.\n___\n**References**\n\n[1] S. Baik, M. Choi, J. Choi, H. Kim, and K. M. Lee. Meta-learning with adaptive hyperparameters,2020"}, {"review_id": "rtLNthNtLK", "reviewer": "Reviewer_n9ad", "summary": "The paper presents Meta-AdaM, a meta-learned adaptive optimizer that also includes momentum. The parameter of the optimizer in the inner loop, i.e. inner learning rate and momentum, are predicted by a LSTM network, that takes into account the previous gradient and momentum steps. The optimization process also includes a double look-ahead strategy for the prediction of the coefficients for the gradient and the momentum. Lastly, the optimization also includes a dynamic class weighting scheme, using a softmax to assign weights to each class according to the loss changes for each class. \nThe optimizer is then combined with MAML and evaluated on three benchmarks datasets (mini-Imagenet, tiered-Imagenet and CIFAR100) with two different backbones (conv4 and Resnet12).", "questions": "- From the paper and the equations, it seems that the optimization process does not include weight decay. This difference from the ALFA [1] paper is not really explained in the related work section, what is the motivation behind this change ?\n- Why $\\alpha$ and $\\beta$ need to sum to 1 ? Why the learning rate is included in the update coming from the momentum ?\n- How the dynamic class weighting scheme is included in the optimization process ? Does it impact both the loss of the LSTM and the base learner ?\n- What are the results without the class weighting ? And in different tasks than classification ?", "limitations": "The authors did not include any discussion of the limitations of their work, nor of potential negative impacts. An important limitation is the limitation to few-shot classification tasks as mentioned above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "### Originality\nThe paper builds upon ALFA [1] by replacing the learned network that predicts the *inner learning rate* and *weight decay* parameters, by a LSTM network, that predicts *inner learning rate* and *momentum* parameters. The LSTM allows taking into account previous update steps. The dynamic class weighting scheme is also an interesting regularization approach to reduce overfitting to easy classes in the inner loop.\n### Significance\nThe proposed method shows strong results on the different benchmarks considered.  \n\n[1] : Baik, S., Choi, M., Choi, J., Kim, H., & Lee, K. M. (2023). Learning to learn task-adaptive hyperparameters for few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.", "weaknesses": "### Clarity\n\n- The paper is difficult to follow as there is a lot of text to describe the method, but it is missing a formal definition of the process. Thankfully, the two algorithms help to clarify the text, but the equations in the algorithms could be directly introduced and described in the text. There is also several concepts vaguely defined such as the loss function $f_2$ in Equation 5 and the temperature $T$ in Equation 6 that is never mentioned in the hyperparameters. \n- The discussion around the dynamic class weighting scheme seems unrelated to the original idea described throughout the paper. The way it is introduced in the optimization is also not really explained, neither in the text nor in the algorithms. \n- The ablation study mentions four different components, but it is not clear what the *momentum* component refers to.  \n### Quality\n- The gradient descent formula described in Equation 1 and 2 is not the commonly used one, described in [2] for instance. Here, the learning rate is impacting the update coming from the momentum. \n- In line 175, it is said that the coefficients $\\alpha$ and $\\beta$ should sum to 1, hence the use of a softmax function, but it is not clear to me why it should be the case, and it is never explained.\n### Significance\nThe introduction of the dynamic class weighting scheme, while being an interesting addition, makes the work specific to the problem of few-shot classification, even though this component does not seem necessary in the approach. From Table 4 in the ablation study, we can see that it has a strong impact on the performance, mainly in the 5-shot setting, and makes the method more effective than others in the mini-imagenet benchmark. Results without this weighting and in other tasks, such as few-shot regression, would make the paper and the evaluation of the approach more solid. \n\n[1] : Baik, S., Choi, M., Choi, J., Kim, H., & Lee, K. M. (2023). Learning to learn task-adaptive hyperparameters for few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.\n\n[2] : Sutskever, I., Martens, J., Dahl, G., & Hinton, G. (2013, May). On the importance of initialization and momentum in deep learning. In International conference on machine learning (pp. 1139-1147). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Table 1:** Ablation Study for class dynamic weight using Convnet4 on Mini-ImageNet, TieredIma- geNet, and Cifar100 datasets\n| Dataset|Method|5-way-1-shot| 5-way-5-shot |\n| :----: | :-----:|:--------------:|:-------------:|\n|Mini-ImageNet|ALFA |50.58 ± 0.51|69.12 ± 0.47|\n|Mini-ImageNet|ALFA+DW| 50.65 ± 0.49|70.02 ± 0.45|\n|Mini-ImageNet|Meta-AdaM w/o DW| 51.64 ± 0.49| 68.80 ± 0.46|\n|Mini-ImageNet|Meta-AdaM| 52.00 ± 0.49| 70.70 ± 0.49|\n\n| Dataset|Method|5-way-1-shot| 5-way-5-shot |\n| :----: | :-----:|:--------------:|:-------------:|\n|TieredImageNet|ALFA |53.16 ± 0.51| 70.54 ± 0.46|\n|TieredImageNet|ALFA+DW| 53.54 ± 0.9|72.36± 0.19|\n|TieredImageNet|Meta-AdaM w/o DW| 53.62 ± 0.50|71.57 ± 0.49|\n|TieredImageNet|Meta-AdaM| 53.93 ± 0.49|72.66 ± 0.49|\n\n| Dataset|Method|5-way-1-shot| 5-way-5-shot |\n| :----: | :-----:|:--------------:|:-------------:|\n|Cifar100|ALFA|39.77 ± 0.48|53.39 ± 0.49|\n|Cifar100|ALFA+DW|40.79 ± 0.19|54.34 ± 0.48|\n|Cifar100|Meta-AdaM w/o DW|40.14 ± 0.49|54.36 ± 0.50|\n|Cifar100|Meta-AdaM (ours)|41.11 ± 0.49|56.32 ± 0.49|\n____\n**Concern 1 (Clarity 1):**  \n\n*The paper is difficult to follow as there is a lot of text to describe the method, but it is missing a formal definition of the process.*\n\n**Answer:** \n\nThank you for the suggestions. We will improve the clarity of our work and provide comprehensive formal definitions for variables used in the work.\n\n___\n**Concern 2 (Clarity 2):** \n\n*The discussion around the dynamic class weighting scheme seems unrelated to the original idea described throughout the paper. The way it is introduced in the optimization is also not really explained, neither in the text nor in the algorithms.*\n\n**Answer:** \n\nThe dynamic class weighting aims to provide a more balanced training signal. The loss computed by it is used to train both LSTM and the base learner.\n\n___\n\n**Concern 3 (Clarity 3):** \n\n*The ablation study mentions four different components, but it is not clear what themomentum component refers to.*\n\n**Answer:** \n\nSorry for the confusion. The momentum component indicates whether we keep an accumulated momentum for the inner loop for weights update and adaptive learning rate prediction.\n\n___\n**Concern 4 (Quality 1):** \n\n*The gradient descent formula described in Equation 1 and 2 is not the commonly used one, described in [2] for instance. Here, the learning rate is impacting the update coming from the  momentum.*\n\n**Answer:**\n\nThanks for pointing out this. We want to note that Equation 1 represents the gradient descent. We substitute the gradient $g_{i}^{t}$ in SGD with accumulated first-order momentum. Equation 2 represents the update $\\alpha_i^t \\times g_{i}^{t} + \\beta_i^t \\times m_{i}^{t-1}$ process of accumulated first-order momentum. This form is widely used in popular optimizers for deep learning models, such as AdaM [3], AdaMax [3], and RmsDrop [2].\n___\n\n**Concern 5 (Question 2):**  \n\n*Why α and β need to sum to 1? Why the learning rate is included in the update coming24\nfrom the momentum?*\n\n**Answer:**\n\n The primary rationale behind setting the sum of α and β to 1 is to ensure that the resultant gradient aligns in scale with both the momentum and the original gradient. This approach follows the same strategies employed by popular optimizers frequently used by deep learning models, such as AdaM [3], AdaMax [3], and RmsDrop [2]. Incorporating the learning rate with the momentum arises because momentum isn’t directly applied to weight updates. As observed in algorithms like Adam, the learning rate and momentum\nare combined.\n____\n\n**Concern 6 (Significance & Question 4):**  \n\n*Contribution of dynamic class weighting scheme*\n\n**Answer:** \n\nThank you for your insightful recommendation. We conduct experiments and provide more comparative results with ALFA [1] in Table 1. Our comparison spans four configurations: ALFA, ALFA with Dynamic Class Weighting, Meta-AdaM excluding class weighting, and the full Meta-AdaM. Based on the results, it’s shown that our Meta-AdaM without dynamic weighting surpasses ALFA in 5 out of the 6 evaluated settings. Moreover, Dynamic Class Weighting offers a tangible enhancement when integrated with ALFA.\n\nFor few-shot regression tasks, the dynamic class weight scheme cannot be used since it is used to balance the performances of different classes. While in regression tasks, there is no explicit class information.\n\n___\n\n**Question 1:** \n\n*From the paper and the equations, it seems that the optimization process does not include weight decay. This difference from the ALFA [1] paper is not really explained in the related work section, what is the motivation behind this change?*\n\n**Answer:**\n\n Thank you for drawing attention to this distinction. Notably, our method can predict weight decay, as illustrated in Eq. (3). This potential enhancement could positively influence the inner optimization process owing to its inherent regularization effects. In the current context, we emphasize the influences of adaptive learning rates and momentum. By sidelining the impact of weight decay, we can observe their contributions more clearly. Nonetheless, we are considering the inclusion of weight decay prediction from Eq. (3) in our finalized version.\n____\n\n**Question  3:** \n\n*How the dynamic class weighting scheme is included in the optimization process? Does it impact both the loss of the LSTM and the base learner?*\n\n**Answer:**\n\n Thank you for highlighting that. Indeed, you’re right. The LSTM and the base learner will be trained using the dynamic loss.\n___\n\n**References:**\n\n[1] S. Baik, M. Choi, J. Choi, H. Kim, and K. M. Lee. Meta-learning with adaptive hyperparameters,2020.\n\n[2] I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.\n\n[3] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.\n"}]

Author rebuttal:
Dear Reviewers,

Firstly, we'd like to express our deep gratitude for the comprehensive review and insightful feedback on our paper. We have carefully addressed each of your comments, and our individual responses to each query can be found in the subsequent sections.

Incorporating your feedback, we've endeavored to enhance the overall clarity and substance of our paper. To this end, we've conducted supplementary experiments for dynamic class weighting, the predicted learning rates, and computed coefficients for momentum. Some figures are now integrated into the attached PDF.

Your diligent review and valuable suggestions have been important in improving the quality of our manuscript. We deeply value your expertise and the time you invested in assessing our work.

Best regards,

The Authors

Meta-review: This paper introduces a novel meta-learned optimizer with momentum for few-shot learning. The majority of reviewers awarded positive scores, recognizing the innovation and robust technical contribution of this study. This positive assessment was particularly reinforced after the authors provided additional technical details and experimental validation in their rebuttal. Consequently, the AC concurs with the majority of reviewers in recommending the acceptance of this manuscript.
Decision: Accept (poster)