SlowGuess/NeurIPS_2023_OCR huggingface dataset:
JunHoo Lee, Jayeon Yoo, and Nojun Kwak* Seoul National University {mrjunoo, jayeon.yoo, nojunk}@snu.ac.kr
In this paper, we hypothesize that gradient-based meta-learning (GBML) implicitly suppresses the Hessian along the optimization trajectory in the inner loop. Based on this hypothesis, we introduce an algorithm called SHOT (Suppressing the Hessian along the Optimization Trajectory) that minimizes the distance between the parameters of the target and reference models to suppress the Hessian in the inner loop. Despite dealing with high-order terms, SHOT does not increase the computational complexity of the baseline model much. It is agnostic to both the algorithm and architecture used in GBML, making it highly versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT, we conduct empirical tests on standard few-shot learning tasks and qualitatively analyze its dynamics. We confirm our hypothesis empirically and demonstrate that SHOT outperforms the corresponding baseline. Code is available at: https://github.com/JunHoo-Lee/SHOT
With the advent of deep learning revolution in the 2010s, machine learning has widened its application to many areas and outperformed humans in some areas. Nevertheless, machines are still far behind humans in the ability to learn by themselves, in that humans can learn concepts of new data with only a few samples, while machines cannot. Meta-Learning deals with this problem of learning to learn. An approach to address this issue is the metric-based methods Chen and He [2021], Snell et al. [2017], Bromley et al. [1993], Grill et al. [2020], Chen et al. [2020], Sung et al. [2018], Radford et al. [2021], which aims to learn 'good' kernels in order to project given data into a well-defined feature space. Another popular line of research is optimization-based methods Finn et al. [2017], Ravi and Larochelle [2017], Rusu et al. [2018a], so-called Gradient-Based Meta-Learning (GBML), which aims to achieve the goal of meta-learning by gradient descent.
As the most representative GBML methods applicable to any model trained with the gradient descent process, model-agnostic meta-learning (MAML) Finn et al. [2017] and its variations Raghu et al. [2019], Oh et al. [2020], Rusu et al. [2018b] exploit nested loops to find a good meta-initialization point from which new tasks can be quickly optimized. To do so, it divides the problem into two loops: the inner loop (task-specific loop) and the outer loop (meta loop). The former tests the meta-learning property of fast learning, and the latter moves the meta-initialization point by observing the dynamics of the inner loop. The general training is performed by alternating the two loops. In this paper, by focusing on the inner loop of GBML methods, we propose a new loss term for the outer loop optimization.
To properly evaluate the meta-learning property in the inner loop, GBML samples a new problem for each inner loop. Fig. 1 shows how the problem is formulated. For every inner loop, the task is
$\tau_{3}$
Figure 1: For the  $i$ -th inner loop in metarning, the task,  $\tau_{i}$ , is sampled from the  $\kappa$  distribution:  $N$  (e.g. 3) classes consisting each task are randomly sampled with a dom configuration order, meaning that the model does not have any information before it enters an unseen task.
sampled from the task distribution. The classes constituting each task are randomly sampled, and the configuration order (class order) is also randomized. This means that the model tackles an entirely new task at the beginning of each inner loop. Since the task is unknown and the model has to solve the problem within a few gradient steps (some extreme algorithms Raghu et al. [2019], Oh et al. [2020] solve the problem in one gradient step), the model typically exploits large learning rates to adapt quickly. For example, MAML Finn et al. [2017] and ANIL Raghu et al. [2019] use significantly larger learning rates of 0.4 and 0.5 respectively, for the inner loop compared to that of the outer loop  $(10^{-3})$ . These rates are much larger than those used in modern deep learning techniques such as ViT Dosovitskiy et al. [2020], which typically use rates in the order of  $10^{-6}$ . However, using such large learning rates can lead to instability due to high-order terms like the Hessian LeCun et al. [2015]. The issue of instability caused by high-order terms is particularly evident during the initial training phase. To mitigate this problem, some deep-learning methodologies employ a heuristic that utilizes a smaller learning rate at the beginning Dosovitskiy et al. [2020], He et al. [2018]. Given that GBML goes through only several optimizations in the entire inner loop, GBML's inner loop corresponds to the initial training phase. Therefore, we anticipate that the problem caused by high-order terms may be severe with a large learning rate in each inner loop. However, it is noteworthy that GBML appears to perform well despite these potential issues.
In this paper, we investigate the inner loop of GBML and its relationship with the implicit risk of using a large learning rate with a few optimization steps in the inner loop. We observe that the gradient of a GBML model remains relatively constant within an inner loop, indicating that the Hessian along the optimization trajectory has a minimal impact on the model's dynamics as learning progresses. We hypothesize that this phenomenon is a key property that enables GBML to mitigate the risk of using a large learning rate in the inner loop.
Our hypothesis that GBML implicitly suppresses the Hessian along the optimization trajectory suggests a desirable property for a GBML algorithm. By explicitly enforcing this property instead of relying on implicit suppression, we anticipate that the model will achieve faster convergence and superior performance. However, enforcing this property is not trivial as it involves dealing with high-order terms, which naturally require a significant amount of computation. In this paper, we propose an algorithm called Suppressing the Hessian along the Optimization Trajectory (SHOT) that suppresses the Hessian in the inner loop by minimizing the distance of the target model from a reference model which is less influenced by the Hessian. Our algorithm can enforce the Hessian to have desirable properties along the inner loop without much increasing the computational complexity of a baseline model. More specifically, SHOT only requires one additional forward pass without any additional backward pass in general cases.
Our algorithm, SHOT, is agnostic to both the algorithm and architecture used in GBML, making it highly versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT, we conducted empirical tests on standard few-shot learning tasks, including miniImagenet, tiered-Imagenet, Cars, and CUB Vinyals et al. [2016], Ren et al. [2018], Krause et al. [2013], Welinder et al. [2010]. Furthermore, we tested SHOT's cross-domain ability by evaluating its performance on different benchmarks. Also, SHOT can be applied to one-step or Hessian-free baselines, acting as a regularizer. Our results demonstrate that SHOT can significantly boost GBML in an algorithm- and architecture-independent manner. Finally, we analyzed the dynamics of SHOT and confirmed that it behaves as expected.
GBML, or optimization-based meta-learning, is a type of methodology which wants to mimic people's rapid learning skill. Thus, a model should learn with a small number of samples and gradient
steps. To achieve this property, Ravi and Larochelle [2017] tried to save meta-data to recurrent networks. Recently, MAML-based methods Finn et al. [2017], Rajeswaran et al. [2019], Rusu et al. [2018b], Baik et al. [2020], Bernacchia [2021] are typically considered as a synonym of GBML consisting of nested loops: outer loop (meta-loop) and inner loop (task-specific loop). They sample few-shot classification tasks to test the meta-learning property in the inner loop and update the meta-initialization parameters in the outer loop with SGD. To learn rapidly, they exploit a more-than-thousand times larger learning rate in the inner loop compared to modern deep learning settings such as ViT Dosovitskiy et al. [2020] and CLIP Radford et al. [2021]. Although some researches have proved their convergence property Fallah et al. [2020], Wang et al. [2020], they set infinitesimal size of learning rate to prove convergence. Fallah et al. [2020] set the learning rate considering the Hessian i.e., setting the learning rate small enough so that the higher-order terms do not affect the convergence and Wang et al. [2020] proved the convergence with the assumption that the learning rate goes to zero. Although some lines of research have shown algorithms dealing with learning rates Baik et al. [2020], Bernacchia [2021], they did not explain the success of a large learning rate. Baik et al. [2020] used an additional model to predict a proper task-specific learning rate. ANIL (Almost No Inner Loop) Raghu et al. [2019] and BOIL (Body Only Inner Loop) Oh et al. [2020] tried to explain GBML with the dynamics of the encoder's output features. Although both used the same feature-level perspective, they argued in exactly different ways. ANIL Raghu et al. [2019] argued feature reuse is an essential component, while BOIL Oh et al. [2020] said that feature adaption is crucial. Both algorithms can be considered as a variant of preconditioning meta-learning since they freeze most layers in the inner loop, thereby frozen layers can be considered as warp-parameters Flennerhag et al. [2019]. Our SHOT can be viewed as a variant of preconditioning, as we hypothesize that there exists a suitable condition of meta-parameter that can adapt well to a given meta-learning task. However, unlike previous work, we define the condition of the meta-parameter based on the curvature of the loss surface. Additionally, our algorithm does not require any specific architecture such as warp-parameters, making it applicable to any architecture and any GBML algorithm. Finally, there exists some paper which delts with curvature, those all required some special architecture Simon et al. [2020], or explicit calculation of the Hessian Hiller et al. [2022], Park and Oliva [2019]. In contrast, we propose much weaker condition and does not require specific architecture. We only suppress the Hessian along the inner loop. In addition, we discuss in the Appendix the possibility of integrating the hypotheses proposed in ANIL Raghu et al. [2019] and BOIL Oh et al. [2020] into our SHOT framework.
Optimization methods based on second-order or first-order derivatives that take curvature into account is a vast area of research. Although they incur a higher computational cost than vanilla SGD, they have been demonstrated to exhibit superior convergence properties. Several studies have investigated the curvature of the loss surface while maintaining the first-order computational costs. Natural gradient descent Bonnabel [2013], a type of Riemannian gradient descent employing the Fisher information metric as a measure, is coordinate-invariant, and exhibits better convergence properties than vanilla gradient descent. Given that GBML utilizes relatively fewer samples, incorporating second-order terms is computationally more feasible. As a result, there is a growing body of research that employs natural gradient descent in the inner loop. Additionally, the SAM optimizer Foret et al. [2020] implicitly leverages curvature by iteratively searching for flat minima during training, leading to better generalization properties than SGD. This means they find a point where the Hessian is close to zero around a local minimum. Our method can be interpreted as the variation of Knowledge Distillation Hinton et al. [2015]. We reduced the distance between models, slow model and fast model. However, we do not need any pretrained model. Those models share same meta-initialization parameter.
As our goal is to find a proper meta-initialization point that is not affected by the Hessian along the inner loop, our proposed algorithm, SHOT, also deals with the Hessian of the loss surface. However, unlike SAM optimizer Foret et al. [2020], we do not seek a point where the Hessian is zero in any direction. We rather seek a point where the Hessian does not take effect only along the optimization trajectory in the inner loop. By using this restriction, we give a looser constraint. Also, while the SAM optimizer has a nested loop to find a proper point, SHOT does not need any iteration in a single optimization step. Also, although natural gradient descent regards curvature like our algorithm, our SHOT does not calculate the exact Hessian during inference time as we precondition the Hessian and only consider the Hessian's effect along the optimization trajectory. In this paper, we aim to shed light on how GBML's inner loop works. Through an analysis of the curvature of the loss surface in the inner loop, we hypothesize that the primary role of the outer loop is to implicitly regulate the
Figure 2: This figure provides the concept of the proposed SHOT. Specifically, a reference model  $(\hat{\theta}^{\star})$  is initialized with the same support set and starting point as our model  $(\theta^{\star})$ , but undergoes significantly more optimization steps. The outer loop then minimizes the distance between the two models.
curvature of the inner loop's loss surface. To reinforce this property by explicitly incorporating a corresponding loss term in the inner loop, we propose an algorithm called SHOT, which takes into account the curvature of the loss surface in the inner loop and aims to enhance its generalization properties and dynamics by controlling the curvature itself. We provide empirical evidence supporting our hypothesis through the success of SHOT. Additionally, we offer a new interpretation of GBML as a variant of MBML in the Appendix.
3 Preliminaries: GBML
We consider the few-shot classification task of  $N$ -way  $K$ -shot, which requires learning  $K$  samples per class to distinguish  $N$  classes in each inner loop. In few-shot GBML, we can apply gradient descent for only a limited number of steps for fast adaptation.
Let  $f(x|\theta) \in \mathbb{R}^N$  be the output of the classifier parameterized by  $\theta$  for a given input  $x$ . In this setting,  $f(\cdot)$  can be considered as the logit before the softmax operation. The model parameters  $\theta$  are updated by the loss,  $L(x,y|\theta) = D(s(f(x|\theta)),y)$ , where  $x$  and  $y$  are the input and the corresponding label,  $s(\cdot)$  is the softmax operation and  $D(\cdot,\cdot)$  is some distance metric.
As shown in Fig. 1, at each inner loop, we sample a task  $\tau \sim p(\mathcal{T})$ , an  $N$ -way classification problem consisting of two sets of labeled data:  $\mathcal{X}_s^\tau$  (support set) and  $\mathcal{X}_t^\tau$  (target set). In the inner loop, the goal is to find the task-specific parameter  $\theta_{\tau}^{\star}$  which minimizes the loss  $L$  for the support set  $\mathcal{X}_s^\tau$  given the initial parameter  $\theta_0$  as follows:
$$
\text {I n n e r L o o p :} \quad \theta_ {\tau} ^ {\star} = \underset {\theta} {\arg \min } \sum_ {(x, y) \in \mathcal {X} _ {s} ^ {\tau}} L (x, y | \theta ; \theta_ {0}). \tag {1}
$$
Normally,  $\theta_{\tau}^{\star}$  is obtained by SGD [Schmidhuber, 1987, Rusu et al., 2018a, Oh et al., 2020, Raghu et al., 2019]:  $\theta^{k + 1} = \theta^k -\alpha \nabla_\theta L(\theta^k)$ , where  $k$  indicates an inner loop optimization step and  $\alpha$  is a learning rate. We use  $T$  steps in the inner loop.
Then, for the outer loop, we optimize the initialization parameters  $\theta_0$  which improve the performance of  $\theta_{\tau}^{\star}$  on  $\mathcal{X}_t$  as follows:
$$
\text {O u t e r L o o p :} \quad \theta_ {0} ^ {\star} = \underset {\theta_ {0}} {\arg \min } \sum_ {\tau} \sum_ {(x, y) \in \mathcal {X} _ {t} ^ {\tau}} L (x, y | \theta_ {\tau} ^ {\star}; \theta_ {0}). \tag {2}
$$
Although GBML gained its success with this setting, this success is quite curious: since  $N$  classes constituting each task are randomly sampled and the configuration (class) order is also random, 'What characteristics of meta-initialization  $\theta_0$  make GBML work?' remains quite uncertain.
4 Optimization in GBML
4.1 Hypothesis: Outer loop implicitly forces inner loop's loss surface linear
The perspective of gradient flow provides a way to understand the dynamics of optimization. In this view, the evolution of model parameters is seen as a discretization of continuous time evolution. Specifically, the equation governing the model parameters is  $\dot{\theta} = -\nabla_{\theta}L(\theta)$ , where  $\theta$  represents the model parameter and  $L(\theta)$  is the loss function. With gradient flow, we can characterize our models into two principal types: the Fast Model and the Slow Model. These definitions emerge from our observation of the models' convergence timelines. Considering the terminal point in time as 1 and the initialization as 0, we deduce the timestamp of a singular learning step as  $\frac{1}{N}$ , where N signifies the number of steps.
Figure 3: (left) Averaged cosine similarity between the overall parameter difference  $(\theta_{\tau}^{\star} - \theta_0)$  and each step's gradient  $(\nabla_{\theta^k}L(\theta^k))$  in the inner loop over the training (meta) epochs. The cosine similarity increases as training progresses, indicating that the gradient does not vary much during an inner loop. (right) Validation accuracy vs. training epoch. To ensure a fair comparison, we shifted  $\mathrm{SHOT}_p$  for 30 epochs as it pretrains for 30 epochs with  $L_{SHOT}$  only. For this figure, we trained  $\mathrm{SHOT}_r$  for 300 epochs to compare it at a glance. Our results show that SHOT converges much faster than the baseline. Additionally, we observed a strong correlation between cosine similarity and validation accuracy.
Fast Model: A model with a relatively small value of  $\mathbf{N}$  is should converge rapidly. A salient exemplar is GBML's inner loop, typically evolving within a minuscule span of 1 to 5 optimization steps. Such rapid convergence necessitates an elevated learning rate.
Slow Model: In contrast, models with a larger value of N take longer to converge. Traditional deep learning models fall into this category. They require many gradient steps and a lower learning rate.
This separation makes clear the differences between typical deep learning models and GBML's inner loop. However, we should note that GBML's high learning rate approach can arise some issues.
$$
\int_ {0} ^ {1} \nabla L (\theta (t)) \cdot \nabla L (\theta^ {k}) d t \approx \int_ {0} ^ {1} \| \nabla L (\theta^ {k}) \| _ {2} ^ {2} - \alpha t \nabla L (\theta^ {k}) ^ {T} H (\theta^ {k}) \nabla L (\theta^ {k}) d t > 0, \qquad (3)
$$
Gradient descent is the notion of treating the model as a (locally) linear approximation, adjusting its parameters in a direction where the loss decreases most rapidly. If a model adheres to Eq. 3, we can expect the gradient descent effective in that context. (Refer to Appendix A for the detailed proof.) However, the inner loop of GBML can be characterized as a fast model. This implies a pronounced influence of the Hessian in the optimization dynamics. Due to this, we cannot guarantee loss decreases in the inner loop. In this paper, We argue that the outer loop's primary role in GBML is to counteract this influence of the Hessian. In more specific terms, it aims to attenuate the impact of  $H(\theta^k)\nabla L(\theta^k)$  on the optimization trajectory.
Fig. 3 supports our claim. As mentioned before, the optimization steps are a discrete representation of the gradient over time. This allows us to track how the gradient changes as time progresses. In meta-learning, the gradient's direction and magnitude do not change significantly as time progresses. This is because meta-learning algorithms are designed to learn how to quickly adapt to new tasks. By learning how to adapt to new tasks, meta-learning algorithms can avoid the need to learn from scratch on each new task. In contrast, in conventional deep learning, the gradient's direction and magnitude change significantly as time progresses. This is because conventional deep learning algorithms are not designed to learn how to quickly adapt to new tasks. Instead, conventional deep learning algorithms are designed to learn a general model that can perform well on a wide variety of tasks. The difference in how the gradient changes over time between meta-learning and conventional deep learning is one of the key factors that explains why meta-learning algorithms are able to learn new tasks so quickly.
It seems clear that Eq. 3 plays a crucial role in GBML, suggesting that we can enhance its performance and dynamics by explicitly inducing the model to meet this condition. However, the challenge remains: how can we directly reduce the Hessian?
Table 1: Comparison of the test accuracies  $(\%)$  of a 4-convolutional network with random initialization and SHOT. SHOT does not receive label information from the outer loop. (i.e.,  $\mathrm{SHOT}_r$  without supervision) The numbers in parentheses indicate the number of shots.
meta-train	miniImageNet	Cars
meta-test	miniImageNet	tieredImageNet	Cars	Cars	CUB	miniImageNet
Random-init (1)	21.36 ± 0.01	21.20 ± 0.01	21.10 ± 0.93	21.20 ± 0.93	21.62 ± 0.02	21.36 ± 0.01
SHOT-init (1)	24.62 ± 0.04	23.80 ± 0.06	24.63 ± 0.04	25.84 ± 0.06	25.86 ± 0.03	26.48 ± 0.04
Random-init (5)	21.29 ± 0.09	21.72 ± 0.19	21.58 ± 0.37	21.58 ± 0.04	22.31 ± 0.06	21.28 ± 0.09
SHOT-init (5)	27.32 ± 0.59	31.28 ± 1.68	29.10 ± 1.81	28.28 ± 0.79	30.34 ± 2.69	26.61 ± 1.05
4.2 SHOT (Suppressing Hessian along the Optimization Trajectory)
In this part, we present our novel algorithm, Suppressing Hessian along the Optimization Trajectory (SHOT), which is the main contribution of our paper. SHOT is designed to obviate the effect of the Hessian along the inner loop while maintaining the order of computation cost.
As previously discussed, directly reducing the Hessian is impractical. However, we can reduce the effect of the Hessian along the optimization trajectory, specifically by minimizing the product  $H(\theta_t)\nabla L(\theta_t)$ , which is in line with the condition expressed in Eq. 3 and computationally feasible. To achieve this, we define a measure that quantifies the effect of the Hessian along the optimization trajectory.
To do so, we first create a reference model that is not influenced by the Hessian. We obtain this reference model by increasing the number of optimization steps. The intuition behind this is that as we increase the number of optimization steps, the smaller the discretized step becomes, which means that the model with many optimization steps is less influenced by the Hessian. We then define the measure as the distance between the target model (a model with few optimization steps,  $\theta_t$ ) and the reference model (a model with many optimization steps,  $\theta_r$ ). This distance metric, denoted as  $D(\theta_t, \theta_r)$ , quantifies the distortion along the optimization path and evaluates the deviation from the ideal optimization trajectory. By minimizing this measure, we can achieve our goal of reducing the effect of the Hessian along the optimization trajectory. We will call this method as SHOT (Suppressing Hessian along the Optimization Trajectory) which adds the following loss term in the outer loop of a conventional GBML algorithm:
$$
L _ {S H O T} = D \left(\theta_ {t} ^ {T}, \theta_ {r} ^ {R}\right). \tag {4}
$$