SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Hyperparameters for experiment.
Hyperpameter	Value
tasks batch size	2
inner learning rate η	0.01
outer learning rate α, β	0.001
# inner fine-tune step	5
# training epochs	100
# outer steps in each epoch	500
classes into 64, 16, and 20 class groups for training, validation, and testing. The image size of Tiered-ImageNet is also  $84 \times 84$ . This dataset consists of 608 classes with 779,165 images. These 608 classes are further combined into 34 high-level classes. These high-level classes are divided into 20, 6, and 8 class groups for training, validation, and testing [31], respectively. Tiered-ImageNet considers the class similarity when divide the dataset to ensure that the distribution of train and test data is very different. Cifar100 dataset for few-shot learning contains 60,000 images with 100 classes. The image size is  $32 \times 32$ . The 100 classes are divided into 60, 20, and 20 groups for training, validation, and testing. These datasets are bench-marking datasets in the FSL domain.
Backbone models. We evaluate the proposed Meta-AdaM using two backbone models: Convnet4 and Resnet12 [12]. Convnet4 is a 4-layer Convolutional Neural Network with 32.9 thousand trainable
Table 3: Comparison results using ResNet12 on Mini-ImageNet, TieredImageNet, and Cifar100 datasets. We report accuracy (%) with standard deviation. *Baseline MC uses WRN-28-10 as backbone and extra data regularization method.
Dataset	Method	5-way-1-shot	5-way-5-shot
Mini-ImageNet	MAML [7]	58.37 ± 0.49	69.76 ± 0.46
MAML+ALFA [3]	59.74 ± 0.49	77.96 ± 0.47
MAML+SiMT [42]	56.28 ± 0.63	72.01 ± 0.26
Sparse-MAML [44]	56.39 ± 0.38	73.01 ± 0.24
MeTAL[40]	59.64 ± 0.38	76.20 ± 0.19
Meta-AdaM (ours)	59.89 ± 0.49	77.92 ± 0.43
MC* [25]	64.40 ± 0.10	80.21 ± 0.10
Tiered-ImageNet	MAML [7]	58.58 ± 0.49	71.24 ± 0.43
MAML+ALFA [3]	64.62 ± 0.49	82.48 ± 0.39
MAML+SiMT [42]	59.72 ± 0.22	74.40 ± 0.90
MeTAL[40]	63.89 ± 0.43	80.14 ± 0.40
Meta-AdaM(ours)	65.31 ± 0.48	85.24 ± 0.35
MC* [25]	67.21 ± 0.10	82.61 ± 0.10
Cifar100	MAML [7]	38.79 ± 0.48	51.65 ± 0.49
MAML+ALFA [3]	40.88 ± 0.48	54.54 ± 0.49
Meta-AdaM (ours)	41.12 ± 0.49	56.14 ± 0.49
parameters, while Resnet12 is an 18-layer Convolutional Neural Network featuring 4 million trainable parameters. By employing these two models, we demonstrate that our proposed optimizer is effective on both small and large networks.
Hyperparameters. We show the hyperparameters in Table 2, which applies to each dataset and each backbone. To ensure a fair comparison, we follow the setting of [3] and report the test performance of the ensemble of top-5 performing models on the validation set.
4.2 Results Using A Small Network
We first evaluate our Meta-AdaM using the Convnet4 network on three datasets. The experiments are conducted under two settings: 5-way-1-shot and 5-way-5-shot, where  $a$ -way- $b$ -shot means each task contains  $a$  classes and there are  $b$  training examples for each class. We compare our Meta-AdaM with Meta-LSTM [30], MAML [7], Meta-SGD [18], and MAML+ALFA [3] as main baseline. A summary of the comparison results can be found in Table 1. From the results, we can observe that our method achieves promising results on all three datasets under two settings. Under the 5-way-1-shot setting, our Meta-adaM outperforms MAML and MAML+ALFA by margins of  $3.30\%$  and  $1.42\%$  on Mini-[ImageNet,  $4.87\%$  and  $0.77\%$  on TieredImageNet, and  $2.91\%$  and  $1.34\%$  on CIFar100, respectively]. Under the 5-way-5-shot setting, our method outperforms MAML and MAML+ALFA by margins of  $7.59\%$  and  $1.58\%$  on Mini-[ImageNet,  $5.18\%$  and  $2.12\%$  on TieredImageNet, and  $6.38\%$  and  $2.93\%$  on CIFar100, respectively.
These promising results demonstrate that our Meta-AdaM can consistently yield more effective optimization results on various benchmark datasets. Notably, the performance gains are even larger in the 5-way-5-shot setting. Given that both the momentum and the new gradients estimated from a few training examples could be noisy with only one training example per class, it is difficult to produce effective gradient updates from them. The power of Meta-AdaM is better reflected by slightly increasing the number of training examples. We also compare our methods with other recent research works, including e3bm [20], Sparse-MAML [44], MAML+SiMT [42] and MeTAL [40]. We can observe that our methods outperform the other methods in most of the settings.
4.3 Results Using A Large Network
In the previous section, we evaluated our Meta-AdaM using a small network, Convnet4, as the backbone model. However, optimizing a large model is more challenging due to the risk of overfitting. In this section, we conduct experiments on a larger backbone model, Resnet12, to further demonstrate the effectiveness of our Meta-AdaM. The evaluation is conducted under 5-way-1-shot and 5-way-5-
Table 4: Abalation study results using the Mini-ImageNet dataset on the Convnet4 backbone network.
Momentum	Look-ahead	Adaptive lr	5-way-1-shot	5-way-5-shot
✓	✓	✓	51.64±0.49	68.80±0.46
✓	✓	X	49.30±0.49	64.09±0.47
X	X	✓	49.02±0.50	67.98±0.46
✓	X	X	48.28±0.49	63.27±0.48
X	X	X	48.70±1.75	63.11±0.92
Table 5: Comparison result LSTM and MLP using Convnet4 on Mini-ImageNet, TieredImageNet, and Cifar100 datasets. We report performances in terms of accuracy  $(\%)$  with standard deviation.
Dataset	Method	5-way-1-shot	5-way-5-shot
Mini-[ImageNet]	Meta-Adam with MLP	52.27±0.49	70.48±0.49
Meta-AdaM with LSTM	52.00±0.49	70.70±0.49
TieredImageNet	Meta-Adam with MLP	53.48±0.49	72.60±0.49
Meta-AdaM with LSTM	53.93±0.49	72.66±0.49
Cifar100	Meta-Adam with MLP	40.28±0.49	54.96±0.49
Meta-AdaM with LSTM	41.11±0.49	56.32±0.49
shot settings using the Mini-ImageNet, Tiered-ImageNet, and CIFar100 datasets. The comparison results with MAML and MAML+ALFA are summarized in Table 3.
Under the 5-way-1-shot setting, we observe margins of  $0.15\%$ ,  $0.69\%$ , and  $0.24\%$  improvements compared to previous state-of-the-art models on the Mini-[ImageNet, TieredImageNet, and Cifar100 datasets, respectively. Under the 5-way-5-shot setting, our Meta-AdaM outperforms ALFA by margins of  $2.76\%$ , and  $1.60\%$  on TieredImageNet, and Cifar100 datasets respectively. On Mini-[ImageNet, our method gives a equal performance compared to MAML+ALFA. Our method also outperforms other methods, including MAML+SiMT [42] and MeTAL [40]. The consistently promising results on the three benchmark datasets demonstrate that our Meta-AdaM is effective when used to optimize a larger backbone model.
In this study, we introduce three novel components in our Meta-AdaM: momentum, double lookahead and an adaptive learning rate via a meta-learned learner. Also, we introduce a dynamic class-weighting scheme for loss functions specially designed for classification tasks. In this section, we evaluate the individual contributions of each component. To accomplish this, we conduct experiments in both 5-way-1-shot and 5-way-5-shot settings, utilizing Convnet4 as the backbone model on the Mini-ImageNet dataset.
4.4.1 Momentum, Double lookahead and Adaptive learning rate
Specifically, we test varying combinations of the first three components and assess the performance of the resulting optimizers. These comparative results can be summarized in Table 4. The results in Table 4 demonstrate that each component contributes to the overall performance. Specifically, heuristic aggregation of momentum does not significantly enhance the performance, with a  $0.42\%$  decrease observed in the 1-shot and a modest  $0.16\%$  increase in the 5-shot setting compared to the original MAML. However, incorporating momentum with a double look-ahead strategy leads to superior results over those from the momentum-only setting. This enhanced optimizer surpasses the original MAML by  $0.6\%$  in the 1-shot and  $0.98\%$  in the 5-shot settings. The adaptive learning rate also plays a crucial role. By solely applying this adaptive learning rate in the inner loop, the resultant optimizer outperforms the original MAML by  $0.32\%$  in the 1-shot and  $4.87\%$  in the 5-shot settings. When these three components are combined, the performance improvement is even more significant. The momentum with look-ahead results in a more stable and efficient training optimization process, thereby aiding the meta-learner in better estimating the adaptive learning rates. From these results, we see a performance boost of  $2.94\%$  in the 1-shot and  $5.69\%$  in the 5-shot settings. To further evaluate the effect of weight update history, we conduct additional experiments that substitute the LSTM network with a MLP network with same inputs. The results are summarized in Table 5. From
Table 6: Ablation Study for class dynamic weight using Convnet4 on Mini-ImageNet, TieredImageNet, and Cifar100 datasets
Dataset	Method	5-way-1-shot	5-way-5-shot
Mini-ImageNet	ALFA	50.58 ± 0.51	69.12 ± 0.47
ALFA+DW	50.65 ± 0.49	70.02 ± 0.45
Meta-AdaM w/o DW	51.64 ± 0.49	68.80 ± 0.46
Meta-AdaM (ours)	52.00 ± 0.49	70.70 ± 0.49
TieredImageNet	ALFA	53.16 ± 0.51	70.54 ± 0.46
ALFA+DW	53.54 ± 0.49	72.36± 0.19
Meta-AdaM w/o DW	53.62 ± 0.50	71.57 ± 0.49
Meta-AdaM (ours)	53.93 ± 0.49	72.66 ± 0.49
Cifar100	ALFA	39.77 ± 0.48	53.39 ± 0.49
ALFA+DW	40.79 ± 0.19	54.34 ± 0.48
Meta-AdaM w/o DW	40.14 ± 0.49	54.36 ± 0.50
Meta-AdaM (ours)	41.11 ± 0.49	56.32 ± 0.49
Figure 2: Visualization of loss changes for each step. The left, middle, and right figures are the average loss changes and  $95\%$  confidence interval for each step on Mini-ImageNet, Tiered-ImageNet, and Cifar100, respectively.
the result, LSTM outperform MLP network in 5 out of 6 experiment setting, which indicates that the weight update history is important.
4.4.2 dynamic class-weighting scheme
To demonstrate the effect of dynamic class, we conduct additional experiments on four configurations: ALFA, ALFA with Dynamic Class Weighting, Meta-AdaM excluding class weighting, and full Meta-AdaM and summarize in result in Table 6. The result shows that Meta-AdaM without Dynamic Class Weighting outperforms ALFA in 5 out of 6 evaluated settings. Noticeably, Dynamic Class Weighting provides a tangible enhancement when integrated with ALFA.
4.5 Visualization of Optimization Process
In this section, we visualize the optimization process of our Meta-AdaM and compare it with ALFA. We randomly select 900 testing tasks from three datasets: Mini-ImageNet, Tiered-ImageNet, and Cifar100. The experiment setting is 5-way-1-shot using Convnet4 as the backbone model. We use the checkpoints with the highest validation accuracy for Meta-AdaM and ALFA. We calculate the average loss change for all 900 tasks in each update step and plot the results in Figure 2. From the plot, our Meta-AdaM helps the model find a better optimum than ALFA.
Additionally, We conducted experiments on 300 5-way-1-shot test tasks on Cifar100 with Convnet4 as the backbone model. We include the change of inner learning rate  $\eta$  and  $\alpha$  and  $\beta$  in the "Double look ahead strategy." in Figure 3. As illustrated in Figure 3 on the left, our optimizer tends to initially predict smaller learning rates due to the substantial conflict between momentum and gradient. A smaller learning rate will lead to a slow convergence rate in the first few steps, providing spaces for accumulating high-quality momentum. Therefore, from Figure 2, the loss from our method decreases slower than ALFA. Once the momentum quality is good after several steps, the predicted learning
Predicted learning rate for test task on Cifar100 Mean and variance of alpha and beta for test task on Cifar100
Figure 3: Left: Visualization of predicted learning rates in 5 steps with variance bonds. The predicted learning rates tend to increase along with the optimization process. Right: Visualization of computed coefficients  $\alpha$  and  $\beta$  of 5 steps with variance bonds in the fine-tuning process. The coefficients for momentum are small at the beginning due to their high variances.
rates become larger, leading to a better optimum than ALFA. This also highlights our contribution to using double-look-ahead to handle highly variable momentum.
Figure 3 on the right shows that the gradients give lower losses until the first three steps, so the gradient  $\alpha$  coefficient is larger than for momentum  $\beta$ . However, after accumulating for a few steps, the accumulated momentum can generate a lower loss and a larger coefficient. Such results support the intuition that the momentum quality improves over the trajectory.
4.6 Efficiency Study of Meta-AdaM
In this section, we assess the efficiency of our Meta-AdaM. In Meta-AdaM, we employ a double lookahead strategy to evaluate the momentum and new gradients, which may introduce additional computational costs. To evaluate these added costs, we compare the total training time of MAML, MAML+ALFA, and Meta-AdaM. We use the experimental setting of 5-way-1-shot on the Mini-ImageNet dataset and record
the running time on an NVIDIA RTX A4000 GPU. The results are summarized in Table 7. The results indicate that our Meta-AdaM takes slightly longer running time than ALFA. However, considering the performance improvements brought by Meta-AdaM, the additional computational resources required by Meta-AdaM are justifiable.
Table 7: Average running time of one epoch.
Method	Running time (s)
MAML	252.04
MAML+ALFA	560.63
Meta-AdaM	595.91
4.7 Predicted Negative Learning Rate
During our experiments, the predicted learning rate in our adaptive learning rate component could occasionally be negative. This phenomenon has been described in [4] and observed in ALFA [3], which employs a two-layer network to predict learning rates. Interestingly, applying negative learning rates to trainable parameters in some update steps can lead to better optimization results. However, determining which parameters are suitable for a negative learning rate and when to apply the negative learning rate remains to be clarified. Further research exploring the role of negative learning rates in meta-learning may contribute to improving the optimization of meta-learning models.
In this work, we propose a meta-learning-specific optimizer, Meta-AdaM, which incorporates three novel optimization components that can better utilize update signals: momentum and new gradients. Specifically, we propose to meta-learn an adaptive learning rate learner using an LSTM network by looking back at the weight update history. We also introduce a double lookahead algorithm to dynamically accumulate momentum, leading to a more stable and effective momentum update and gradient update. Moreover, we propose a dynamic class weighting scheme to balance the losses across different classes. Experimental results demonstrate the effectiveness of our proposed Meta-AdaM.
This work has been supported in part by National Science Foundation grant III-2104797.
[1] S.-i. Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5): 185-196, 1993.
[2] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
[3] S. Baik, M. Choi, J. Choi, H. Kim, and K. M. Lee. Meta-learning with adaptive hyperparameters. Advances in neural information processing systems, 33:20755-20765, 2020.
[4] A. Bernacchia. Meta-learning with negative learning rates. arXiv preprint arXiv:2102.00940, 2021.
[5] M. Donini, L. Franceschi, O. Majumder, M. Pontil, and P. Frasconi. Marthe: scheduling the learning rate via online hypergradients. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 2119-2125, 2021.
[6] T. Dozat. Incorporating nesterov momentum into adam. The Eleventh International Conference on Learning Representations, 2016.
[7] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017.
[8] C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
[9] S. Flennerhag, A. A. Rusu, R. Pascanu, F. Visin, H. Yin, and R. Hadsell. Meta-learning with warped gradient descent. arXiv preprint arXiv:1909.00025, 2019.
[10] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4367-4375, 2018.
[11] I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning in neural networks: A survey, 2020.
[14] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[15] M. J. Kochenderfer and T. A. Wheeler. Algorithms for optimization. Mit Press, 2019.
[16] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[17] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10657-10665, 2019.
[18] Z. Li, F. Zhou, F. Chen, and H. Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835, 2017.
[39] M. Sundermeyer, R. Schlüter, and H. Ney. LSTM neural networks for language modeling. In Thirteenth annual conference of the international speech communication association, 2012.
[40] H. K. D. C. J. M. K. M. L. Sungyong Baik, Janghoon Choi. Meta-learning with task-adaptive loss function for few-shot learning. In International Conference on Computer Vision (ICCV), 2021.
[41] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147. PMLR, 2013.
[42] J. Tack, J. Park, H. Lee, J. Lee, and J. Shin. Meta-learning with self-improving momentum target. In Advances in Neural Information Processing Systems, 2022.
[43] P. Vicol, L. Metz, and J. Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In International Conference on Machine Learning, pages 10553-10563. PMLR, 2021.
[44] J. von Oswald, D. Zhao, S. Kobayashi, S. Schug, M. Caccia, N. Zucchet, and J. Sacramento. Learning where to learn: Gradient sparsity in meta and continual learning, 2021.
[45] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
[46] O. Wichrowska, N. Maheswaranathan, M. W. Hoffman, S. G. Colmenarejo, M. Denil, N. Freitas, and J. Sohl-Dickstein. Learned optimizers that scale and generalize. In International conference on machine learning, pages 3751-3760. PMLR, 2017.
[47] M. Zhang, J. Lucas, J. Ba, and G. E. Hinton. Lookahead optimizer: k steps forward, 1 step back. Advances in neural information processing systems, 32, 2019.
Table 8: Comparison results on few-shot regression tasks. The target function for regression is a sine curve  $y(x) = A \sin(\omega x)$ . We report MSE error with a 95% confidence interval.
Method	Training	5-shot Testing	10-shot Testing	20-shot Testing
MAML[7]	5-shot	1.13 ± 0.08	0.85 ± 0.14	0.71 ± 0.12
Meta-SGD[18]	5-shot	0.90 ± 0.16	0.63 ± 0.12	0.50 ± 0.10
ALFA[3]	5-shot	0.60 ± 0.04	0.41 ± 0.02	0.25 ± 0.22
Meta-AdaM	5-shot	0.52 ± 0.04	0.35 ± 0.04	0.23 ± 0.01
MAML[7]	10-shot	1.17 ± 0.16	0.77 ± 0.11	0.56 ± 0.08
Meta-SGD[18]	10-shot	0.88 ± 0.14	0.53 ± 0.09	0.35 ± 0.06
ALFA[3]	10-shot	0.72 ± 0.16	0.45± 0.03	0.26 ± 0.02
Meta-AdaM	10-shot	0.66 ± 0.22	0.34 ± 0.05	0.23 ± 0.01
MAML[7]	20-shot	1.29 ± 0.20	0.76 ± 0.12	0.48 ± 0.08
Meta-SGD[18]	20-shot	1.01 ± 0.17	0.54 ± 0.08	0.31 ± 0.05
ALFA[3]	20-shot	1.01± 0.18	0.48± 0.09	0.25 ± 0.03
Meta-AdaM	20-shot	0.95 ± 0.11	0.43 ± 0.06	0.23 ± 0.03
Figure 4: Visualization of dynamic class weighting scheme. The left and right figures show the optimization process without and with the dynamic class weighting scheme, respectively.
7.1 Result on regression task
To further prove the efficiency of our method, we conduct additional experiments on few-shot regression tasks and show the result in Table 8. We follow the experiment setting of the Meta-SGD[18]. The regression task is to map the underlying function based on the input. The target function is  $y(x) = A\sin (\omega x)$ , where amplitude  $A$ , frequency  $\omega$ , and phase  $b$  all follow a uniform distribution with the interval [0.1, 5.0], [0.8, 1.2], and [0,  $\pi$ ]. We train three few-shot regression learners on 5-shot, 10-shot, and 20-shot tasks separately. Each learner will test on 100 5-shot, 10-shot, and 20-shot testing tasks. We report MSE errors with a  $95\%$  confidence interval.
We compare our approach with MAML[7], Meta-SGD[18], Meta-SGD+ALFA[3], and Meta-SGD + META-AadM. The result has shown that our method outperforms the baselines, Meta-SGD and Meta-SGD+ALFA, in all settings. The promising results indicate that our proposed method is also effective on few-shot regression tasks.
7.2 Study of the Dynamic Class Weighting Scheme
In previous section, we present quantitative results to demonstrate the effectiveness of the dynamic class weighting scheme. In this section, we visualize the loss changes for each class when using this scheme. We compare the loss function using our scheme and the one without a class weighting scheme. Each optimization process consists of five update steps. The left figure in Figure 4 displays
the loss changes of each class without employing a class weighting scheme. Class 2 has a significantly lower loss than the other classes. For the last four steps, only the loss of class 2 is optimized, while the other four classes remain under-optimized. Such situations can lead to model collapse problems in meta-learning [26], which means the model tends to classify all testing examples into one or two classes. The right figure in Figure 4 illustrates the loss changes when using the dynamic class weighting scheme. Class 4 has a lower loss in the first step compared to the other classes. The optimizer focuses more on the other four classes by emphasizing those with larger losses. Ultimately, all class losses converge towards the optimization endpoint, effectively mitigating the risk of model collapse problems.