Title: SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: meta learning

Hessian

Gradient-Based meta learning

Feature Reuse

Implicit Prior
Abstract: In this paper, we hypothesize that gradient-based meta-learning (GBML) implicitly suppresses the Hessian along the optimization
  trajectory in the inner loop. Based on this hypothesis, we introduce an algorithm called
  SHOT (Suppressing the Hessian along the Optimization Trajectory) that minimizes the distance between the parameters of the target and reference models to suppress the Hessian in the inner loop. Despite dealing with
  high-order terms, SHOT does not increase the computational complexity of the baseline model much.
  It is agnostic to both the algorithm and architecture used in GBML, making it highly
  versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT,
  we conduct empirical tests on standard few-shot learning tasks and qualitatively
  analyze its dynamics. We confirm our hypothesis empirically and demonstrate that SHOT
  outperforms the corresponding baseline.

SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning

Abstract

In this paper, we hypothesize that gradient-based meta-learning (GBML) implicitly suppresses the Hessian along the optimization trajectory in the inner loop. Based on this hypothesis, we introduce an algorithm called SHOT (Suppressing the Hessian along the Optimization Trajectory) that minimizes the distance between the parameters of the target and reference models to suppress the Hessian in the inner loop. Despite dealing with high-order terms, SHOT does not increase the computational complexity of the baseline model much. It is agnostic to both the algorithm and architecture used in GBML, making it highly versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT, we conduct empirical tests on standard few-shot learning tasks and qualitatively analyze its dynamics. We confirm our hypothesis empirically and demonstrate that SHOT outperforms the corresponding baseline. Code is available at: https://github.com/JunHoo-Lee/SHOT

1 Introduction

With the advent of deep learning revolution in the 2010s, machine learning has widened its application to many areas and outperformed humans in some areas. Nevertheless, machines are still far behind humans in the ability to learn by themselves, in that humans can learn concepts of new data with only a few samples, while machines cannot. Meta-Learning deals with this problem of learning to learn. An approach to address this issue is the metric-based methods Chen and He [2021], Snell et al. [2017], Bromley et al. [1993], Grill et al. [2020], Chen et al. [2020], Sung et al. [2018], Radford et al. [2021], which aims to learn 'good' kernels in order to project given data into a well-defined feature space. Another popular line of research is optimization-based methods Finn et al. [2017], Ravi and Larochelle [2017], Rusu et al. [2018a], so-called Gradient-Based Meta-Learning (GBML), which aims to achieve the goal of meta-learning by gradient descent.

As the most representative GBML methods applicable to any model trained with the gradient descent process, model-agnostic meta-learning (MAML) Finn et al. [2017] and its variations Raghu et al. [2019], Oh et al. [2020], Rusu et al. [2018b] exploit nested loops to find a good meta-initialization point from which new tasks can be quickly optimized. To do so, it divides the problem into two loops: the inner loop (task-specific loop) and the outer loop (meta loop). The former tests the meta-learning property of fast learning, and the latter moves the meta-initialization point by observing the dynamics of the inner loop. The general training is performed by alternating the two loops. In this paper, by focusing on the inner loop of GBML methods, we propose a new loss term for the outer loop optimization.

To properly evaluate the meta-learning property in the inner loop, GBML samples a new problem for each inner loop. Fig. 1 shows how the problem is formulated. For every inner loop, the task is

˜

θ

1

⋆

'Well learned'

˜

θ

⋆

θ

1

2

⋆

θ

⋆

2

̂

θ 0 θ ⋆ θ ⋆ sampled from the task distribution. The classes constituting each task are randomly sampled, and the configuration order (class order) is also randomized. This means that the model tackles an entirely new task at the beginning of each inner loop. Since the task is unknown and the model has to solve the problem within a few gradient steps (some extreme algorithms Raghu et al. [2019], Oh et al. [2020] solve the problem in one gradient step), the model typically exploits large learning rates to adapt quickly. For example, MAML Finn et al. [2017] and ANIL Raghu et al. [2019] use significantly larger learning rates of 0.4 and 0.5 respectively, for the inner loop compared to that of the outer loop ( 10 -3 ). These rates are much larger than those used in modern deep learning techniques such as ViT Dosovitskiy et al. [2020], which typically use rates in the order of 10 -6 . However, using such large learning rates can lead to instability due to high-order terms like the Hessian LeCun et al. [2015]. The issue of instability caused by high-order terms is particularly evident during the initial training phase. To mitigate this problem, some deep-learning methodologies employ a heuristic that utilizes a smaller learning rate at the beginning Dosovitskiy et al. [2020], He et al. [2018]. Given that GBML goes through only several optimizations in the entire inner loop, GBML's inner loop corresponds to the initial training phase. Therefore, we anticipate that the problem caused by high-order terms may be severe with a large learning rate in each inner loop. However, it is noteworthy that GBML appears to perform well despite these potential issues.

θ 0 θ ⋆ In this paper, we investigate the inner loop of GBML and its relationship with the implicit risk of using a large learning rate with a few optimization steps in the inner loop. We observe that the gradient of a GBML model remains relatively constant within an inner loop, indicating that the Hessian along the optimization trajectory has a minimal impact on the model's dynamics as learning progresses. We hypothesize that this phenomenon is a key property that enables GBML to mitigate the risk of using a large learning rate in the inner loop.

Our hypothesis that GBML implicitly suppresses the Hessian along the optimization trajectory suggests a desirable property for a GBML algorithm. By explicitly enforcing this property instead of relying on implicit suppression, we anticipate that the model will achieve faster convergence and superior performance. However, enforcing this property is not trivial as it involves dealing with high-order terms, which naturally require a significant amount of computation. In this paper, we propose an algorithm called Suppressing the Hessian along the Optimization Trajectory (SHOT) that suppresses the Hessian in the inner loop by minimizing the distance of the target model from a reference model which is less influenced by the Hessian. Our algorithm can enforce the Hessian to have desirable properties along the inner loop without much increasing the computational complexity of a baseline model. More specifically, SHOT only requires one additional forward pass without any additional backward pass in general cases.

Our algorithm, SHOT, is agnostic to both the algorithm and architecture used in GBML, making it highly versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT, we conducted empirical tests on standard few-shot learning tasks, including miniImagenet, tieredImagenet, Cars, and CUB Vinyals et al. [2016], Ren et al. [2018], Krause et al. [2013], Welinder et al. [2010]. Furthermore, we tested SHOT's cross-domain ability by evaluating its performance on different benchmarks. Also, SHOT can be applied to one-step or Hessian-free baselines, acting as a regularizer. Our results demonstrate that SHOT can significantly boost GBML in an algorithm- and architecture-independent manner. Finally, we analyzed the dynamics of SHOT and confirmed that it behaves as expected.

2 Related Works

GBML, or optimization-based meta-learning, is a type of methodology which wants to mimic people's rapid learning skill. Thus, a model should learn with a small number of samples and gradient

steps. To achieve this property, Ravi and Larochelle [2017] tried to save meta-data to recurrent networks. Recently, MAML-based methods Finn et al. [2017], Rajeswaran et al. [2019], Rusu et al. [2018b], Baik et al. [2020], Bernacchia [2021] are typically considered as a synonym of GBML consisting of nested loops: outer loop (meta-loop) and inner loop (task-specific loop). They sample few-shot classification tasks to test the meta-learning property in the inner loop and update the meta-initialization parameters in the outer loop with SGD. To learn rapidly, they exploit a more-thanthousand times larger learning rate in the inner loop compared to modern deep learning settings such as ViT Dosovitskiy et al. [2020] and CLIP Radford et al. [2021]. Although some researches have proved their convergence property Fallah et al. [2020], Wang et al. [2020], they set infinitesimal size of learning rate to prove convergence. Fallah et al. [2020] set the learning rate considering the Hessian i . e ., setting the learning rate small enough so that the higher-order terms do not affect the convergence and Wang et al. [2020] proved the convergence with the assumption that the learning rate goes to zero. Although some lines of research have shown algorithms dealing with learning rates Baik et al. [2020], Bernacchia [2021], they did not explain the success of a large learning rate. Baik et al. [2020] used an additional model to predict a proper task-specific learning rate. ANIL (Almost No Inner Loop) Raghu et al. [2019] and BOIL (Body Only Inner Loop) Oh et al. [2020] tried to explain GBML with the dynamics of the encoder's output features. Although both used the same feature-level perspective, they argued in exactly different ways. ANIL Raghu et al. [2019] argued feature reuse is an essential component, while BOIL Oh et al. [2020] said that feature adaption is crucial. Both algorithms can be considered as a variant of preconditioning meta-learning since they freeze most layers in the inner loop, thereby frozen layers can be considered as warp-parameters Flennerhag et al. [2019]. Our SHOT can be viewed as a variant of preconditioning, as we hypothesize that there exists a suitable condition of meta-parameter that can adapt well to a given meta-learning task. However, unlike previous work, we define the condition of the meta-parameter based on the curvature of the loss surface. Additionally, our algorithm does not require any specific architecture such as warp-parameters, making it applicable to any architecture and any GBML algorithm. Finally, there exists some paper which delt with curvature, those all required some special architecture Simon et al. [2020], or explicit calculation of the Hessian Hiller et al. [2022], Park and Oliva [2019]. In contrast, we propose much weaker condition and does not require specific architecture. We only suppress the Hessian along the inner loop. In addition, we discuss in the Appendix the possibility of integrating the hypotheses proposed in ANIL Raghu et al. [2019] and BOIL Oh et al. [2020] into our SHOT framework.

Optimization methods based on second-order or first-order derivatives that take curvature into account is a vast area of research. Although they incur a higher computational cost than vanilla SGD, they have been demonstrated to exhibit superior convergence properties. Several studies have investigated the curvature of the loss surface while maintaining the first-order computational costs. Natural gradient descent Bonnabel [2013], a type of Riemannian gradient descent employing the Fisher information metric as a measure, is coordinate-invariant, and exhibits better convergence properties than vanilla gradient descent. Given that GBML utilizes relatively fewer samples, incorporating second-order terms is computationally more feasible. As a result, there is a growing body of research that employs natural gradient descent in the inner loop. Additionally, the SAM optimizer Foret et al. [2020] implicitly leverages curvature by iteratively searching for flat minima during training, leading to better generalization properties than SGD. This means they find a point where the Hessian is close to zero around a local minimum. Our method can be interpreted as the variation of Knowledge Distillation Hinton et al. [2015]. We reduced the distance between models, slow model and fast model. However, we do not need any pretrained model. Those models share same meta-initialization parameter.

As our goal is to find a proper meta-initialization point that is not affected by the Hessian along the inner loop, our proposed algorithm, SHOT, also deals with the Hessian of the loss surface. However, unlike SAM optimizer Foret et al. [2020], we do not seek a point where the Hessian is zero in any direction. We rather seek a point where the Hessian does not take effect only along the optimization trajectory in the inner loop. By using this restriction, we give a looser constraint. Also, while the SAM optimizer has a nested loop to find a proper point, SHOT does not need any iteration in a single optimization step. Also, although natural gradient descent regards curvature like our algorithm, our SHOT does not calculate the exact Hessian during inference time as we precondition the Hessian and only consider the Hessian's effect along the optimization trajectory. In this paper, we aim to shed light on how GBML's inner loop works. Through an analysis of the curvature of the loss surface in the inner loop, we hypothesize that the primary role of the outer loop is to implicitly regulate the

curvature of the inner loop's loss surface. To reinforce this property by explicitly incorporating a corresponding loss term in the inner loop, we propose an algorithm called SHOT, which takes into account the curvature of the loss surface in the inner loop and aims to enhance its generalization properties and dynamics by controlling the curvature itself. We provide empirical evidence supporting our hypothesis through the success of SHOT. Additionally, we offer a new interpretation of GBML as a variant of MBML in the Appendix.

3 Preliminaries: GBML

We consider the few-shot classification task of N -way K -shot, which requires learning K samples per class to distinguish N classes in each inner loop. In few-shot GBML, we can apply gradient descent for only a limited number of steps for fast adaptation.

Let f ( x | θ ) ∈ R N be the output of the classifier parameterized by θ for a given input x . In this setting, f ( · ) can be considered as the logit before the softmax operation. The model parameters θ are updated by the loss, L ( x, y | θ ) = D ( s ( f ( x | θ )) , y ) , where x and y are the input and the corresponding label, s ( · ) is the softmax operation and D ( · , · ) is some distance metric.

As shown in Fig. 1, at each inner loop, we sample a task τ ∼ p ( T ) , an N -way classification problem consisting of two sets of labeled data: X τ s (support set) and X τ t (target set). In the inner loop, the goal is to find the task-specific parameter θ glyph[star] τ which minimizes the loss L for the support set X τ s given the initial parameter θ 0 as follows:

Normally, θ glyph[star] τ is obtained by SGD [Schmidhuber, 1987, Rusu et al., 2018a, Oh et al., 2020, Raghu et al., 2019]: θ k +1 = θ k -α ∇ θ L ( θ k ) , where k indicates an inner loop optimization step and α is a learning rate. We use T steps in the inner loop.

Then, for the outer loop, we optimize the initialization parameters θ 0 which improve the performance of θ glyph[star] τ on X t as follows:

Although GBML gained its success with this setting, this success is quite curious: since N classes constituting each task are randomly sampled and the configuration (class) order is also random, 'What characteristics of meta-initialization θ 0 make GBML work?' remains quite uncertain.

4 Optimization in GBML

4.1 Hypothesis: Outer loop implicitly forces inner loop's loss surface linear

The perspective of gradient flow provides a way to understand the dynamics of optimization. In this view, the evolution of model parameters is seen as a discretization of continuous time evolution. Specifically, the equation governing the model parameters is ˙ θ = -∇ θ L ( θ ) , where θ represents the model parameter and L ( θ ) is the loss function. With gradient flow, we can characterize our models into two principal types: the Fast Model and the Slow Model . These definitions emerge from our observation of the models' convergence timelines. Considering the terminal point in time as 1 and the initialization as 0, we deduce the timestamp of a singular learning step as 1 N , where N signifies the number of steps.

Cosine Similarity

ValidationAccuracy

0.9

X

0.6

0.8

0.5

0.7

0.6

0.4

0.5

MAML+SHOT_r

MAML+SHOT_p

0.3

MAML+SHOT_p

MAML+SHOT_r

0.4

MAML

MAML

0.2

0

50

100

150

200

250

300

0

50

100

150

200

250

300

Fast Model : A model with a relatively small value of N is should converge rapidly. A salient exemplar is GBML's inner loop, typically evolving within a minuscule span of 1 to 5 optimization steps. Such rapid convergence necessitates an elevated learning rate.

Slow Model : In contrast, models with a larger value of N take longer to converge. Traditional deep learning models fall into this category. They require many gradient steps and a lower learning rate.

This separation makes clear the differences between typical deep learning models and GBML's inner loop. However, we should note that GBML's high learning rate approach can arise some issues.

Gradient descent is the notion of treating the model as a (locally) linear approximation, adjusting its parameters in a direction where the loss decreases most rapidly. If a model adheres to Eq. 3, we can expect the gradient descent effective in that context. (Refer to Appendix A for the detailed proof.) However, the inner loop of GBML can be characterized as a fast model. This implies a pronounced influence of the Hessian in the optimization dynamics. Due to this, we cannot guarantee loss decreases in the inner loop. In this paper, We argue that the outer loop's primary role in GBML is to counteract this influence of the Hessian. In more specific terms, it aims to attenuate the impact of H ( θ k ) ∇ L ( θ k ) on the optimization trajectory.

Fig. 3 supports our claim. As mentioned before, the optimization steps are a discrete representation of the gradient over time. This allows us to track how the gradient changes as time progresses. In meta-learning, the gradient's direction and magnitude do not change significantly as time progresses. This is because meta-learning algorithms are designed to learn how to quickly adapt to new tasks. By learning how to adapt to new tasks, meta-learning algorithms can avoid the need to learn from scratch on each new task. In contrast, in conventional deep learning, the gradient's direction and magnitude change significantly as time progresses. This is because conventional deep learning algorithms are not designed to learn how to quickly adapt to new tasks. Instead, conventional deep learning algorithms are designed to learn a general model that can perform well on a wide variety of tasks. The difference in how the gradient changes over time between meta-learning and conventional deep learning is one of the key factors that explains why meta-learning algorithms are able to learn new tasks so quickly.

It seems clear that Eq. 3 plays a crucial role in GBML, suggesting that we can enhance its performance and dynamics by explicitly inducing the model to meet this condition. However, the challenge remains: how can we directly reduce the Hessian?

4.2 SHOT (Suppressing Hessian along the Optimization Trajectory)

In this part, we present our novel algorithm, Suppressing Hessian along the Optimization Trajectory (SHOT), which is the main contribution of our paper. SHOT is designed to obviate the effect of the Hessian along the inner loop while maintaining the order of computation cost.

As previously discussed, directly reducing the Hessian is impractical. However, we can reduce the effect of the Hessian along the optimization trajectory, specifically by minimizing the product H ( θ t ) ∇ L ( θ t ) , which is in line with the condition expressed in Eq. 3 and computationally feasible. To achieve this, we define a measure that quantifies the effect of the Hessian along the optimization trajectory.

To do so, we first create a reference model that is not influenced by the Hessian. We obtain this reference model by increasing the number of optimization steps. The intuition behind this is that as we increase the number of optimization steps, the smaller the discretized step becomes, which means that the model with many optimization steps is less influenced by the Hessian. We then define the measure as the distance between the target model (a model with few optimization steps, θ t ) and the reference model (a model with many optimization steps, θ r ). This distance metric, denoted as D ( θ t , θ r ) , quantifies the distortion along the optimization path and evaluates the deviation from the ideal optimization trajectory. By minimizing this measure, we can achieve our goal of reducing the effect of the Hessian along the optimization trajectory. We will call this method as SHOT (Suppressing Hessian along the Optimization Trajectory) which adds the following loss term in the outer loop of a conventional GBML algorithm:

Here, T and R denote the number of steps in an inner loop used to obtain the target and reference model respectively. In this paper, we used KL divergence ( KL ( s ( f ( θ T t )) | s ( f ( θ R r )) ) as a distance metric D , but we also conducted experiments with other distance metrics such as L 2 ( || θ T t -θ R r || 2 2 ) and cross-entropy ( -∑ c s ( f ( θ T t )) c log s ( f ( θ R r )) c , as shown in Table. 6. We set T < R because our intention is to measure distortion caused by the Hessian in the inner loop. As we increase the number of optimization steps in the inner loop, the effect of distortion by the Hessian decreases. Therefore, we could use SHOT as a pseudo-measure of distortion in the inner loop caused by the Hessian. To match the scale of distribution between θ T t and θ R r , we set α in Eq. 1 as α r = T R α t .

Reducing the effect of Hessian along the optimization in the inner loop, as measured by Eq. 4, can lead to faster convergence and better performance. By minimizing the deviation from the ideal optimization trajectory, SHOT can effectively enforce the condition expressed in Eq. 3, which is critical for the success of GBML. As shot is the measure of distortion in the whole trajectory of an inner loop. We use SHOT at the outer loop, where scoring of each optimization trajectory is done.

Table 1 highlights the importance of distortion in GBML. We compared the performance of a randomly-initialized model with that of a lower-distortion-initialized model, where we only reduced the distortion introduced in Eq. 4 in the outer loop. As our SHOT loss is self-supervisory that can be applied without target class labels, no target labels are given here. The results demonstrate that the lower-distortion-initialized model outperforms the randomly-initialized model, indicating that distortion is a crucial factor in GBML. This also suggests that we can generally improve GBML by incorporating our loss term, SHOT.

4.3 SHOT can be used as a regularizer or meta-meta-initializer in the outer loop.

One potential approach to incorporating SHOT is to use it as a regularizer in GBML. This approach involves adding SHOT to the loss function of the outer loop during training, thereby leveraging it as a regularizer reducing the effect of the Hessian along the optimization trajectory. Alternatively, SHOT can be utilized as a meta-meta-initializer as is used to obtain Table 1. In this approach, the entire model is pre-trained with SHOT, i . e ., The outer loss is substituted with L SHOT without utilizing any labeling information. Then the model is trained with the original loss function. However, there is a vulnerability to homogeneous corruption, where the loss goes to zero only if θ t and θ r produce the same output of all inputs. As this phenomenon is trivial in the self-supervised learning area, we drew inspiration from BYOL Grill et al. [2020]. To address this issue, we added a projector to the reference model, while the target remained the same. We also augmented X τ t . After pre-training with SHOT, we started training by loading the best model with the highest validation accuracy. To differentiate between SHOT utilized as a regularizer and SHOT utilized as a meta-meta-initializer, we denote them as SHOT r and SHOT p , respectively.

4.4 SHOT does not require any computation burden

Several meta-learning algorithms have already assumed linearity in the inner loop. For example, FoMAMLand Reptile Finn et al. [2017], Nichol et al. [2018] proposed Hessian-Free GBML algorithms that both assumed linearity in the inner loop. Additionally, ANIL Raghu et al. [2019] proposed a feature reuse concept that allows for translation to zero distortion in the inner loop. However, none of these studies explicitly argued that GBML implicitly suppresses the Hessian in the inner loop as an inductive prior, nor did they propose an algorithm that controls the Hessian. Our algorithm is similar to the 'SAM' optimizer Foret et al. [2020] or natural gradient descent Bonnabel [2013]. Like SAM and natural gradient descent, our algorithm searches for a good initialization point based on its curvature. However, SHOT finds it without much increasing the computational cost compared to normal gradient descent, which is a notable advantage over SAM or natural gradient descent. Although SAM uses a first-order computation term, it searches for flat minima by iteratively adding noise to each point. Natural gradient descent exploits the Fisher metric, which is a second-order computation cost.

On the other hand, we can make SHOT not give any computation burden. When we set the reference model θ r to exploit the same number of optimization steps in the inner loop as the baseline model and the target model θ t exploits one optimization step in the inner loop, because both models share the same initialization point θ 0 , i . e ., θ 0 t = θ 0 r . θ t can be obtained by calculating the first optimization step of θ r . Therefore, SHOT does not require any additional backpropagation. What we require is only one more inference at the endpoint θ T t for SHOT r . Also, even if we set θ r to have more optimization steps than the baseline model and set θ t to have the same number of optimization steps as the baseline model, the computation cost at test time is unchanged as it is only used in the training phase.

5 Experiments

Datasets Our method is evaluated on miniImageNet [Vinyals et al., 2016], tieredImageNet Ren et al. [2018], Cars Krause et al. [2013], and CUB Welinder et al. [2010] datasets. miniImageNet and tieredImageNet are subsets of ImageNet Russakovsky et al. [2015] with 100 classes each and can be considered as general datasets. Cars and CUB are widely used benchmarks for fine-grained image classification tasks as they contain similar objects with subtle differences. By conducting experiments on these datasets, we can evaluate the performance of our method across a range of tasks.

Architectures We use 4-Conv from Finn et al. [2017] and ResNet-12 from Oreshkin et al. [2018] as our backbone architecture. We trained 4-Conv for 30k iterations and ResNet-12 for 10k iterations.

Experimental Setup To ensure a fair comparison, we fine-tuned MAML Finn et al. [2017] first, and our baselines outperformed the original paper. We used Adam Kingma and Ba [2014] as our optimizer and fixed the outer loop learning rate to 10 -3 and inner loop learning rate to 0.5 for 4-Conv, and 10 -5 and 0.01 for ResNet-12. We restricted the total number of steps in the inner loop to 3, as argued in Finn et al. [2017]. We set the number of steps to 1 for the target model θ t and 3 for the reference model θ r in our SHOT. With this setting, we can exploit SHOT without any additional computation burden as discussed in Sec 4.4. For SHOT r , we fixed λ to 0.1 when we set the original

model as θ r . For one-step algorithms, we set the number of optimization steps to 2 for θ r and 1 for θ t . So actual optimization step in inference time is 1, which is the same as the baseline. We set λ for 10 -6 in that case. For the Hessian-Free algorithm FoMAML, we fine-tuned λ to meet the best performance. We set the learning rate of the outer and inner loops to 10 -4 and 0.01, respectively, for 4-Conv. To maintain conciseness, we only report the results of SHOT r except for the comparison with MAML baseline. We conducted our experiments on a single A100 GPU. We implemented our method using torchmeta Deleu et al. [2019] library except for the implementation of LEO Rusu et al. [2018a], where we used the official implementation and embeddings.

5.1 Characteristics of SHOT

SHOT Outperforms the Baselines on Benchmarks and Cross-Benchmark Settings Our method, SHOT, outperforms the corresponding baseline on all benchmarks, as shown in Table 2. Both SHOT r and SHOT p outperform the baseline, regardless of whether the benchmark is general or fine-grained. Additionally, SHOT generalizes well to cross-benchmark settings, as demonstrated in Table 3, where it improves the performance of the baseline. Also, we can see that this result is independent of the backbone architecture, as shown in Table 4.

Table 4: Test accuracy % of ResNet-12. The values in parentheses indicate the number of shots. The better accuracy between the baseline and SHOT is bold-faced.

SHOTworks with Hessian-Free and One-Step algorithms As many GBML algorithms are based on Hessian-Free algorithms which only exploit gradients and One-Step algorithms which only exploit one step in the inner loop Finn et al. [2017], Nichol et al. [2018], Oh et al. [2020], Raghu et al. [2019], it is important to show that our method also works with these algorithms. As shown in Table 5, our method improves the performance of the baseline on both Hessian-Free and One-Step algorithms. For One-Step algorithms, we set θ t as the original model and θ r as the 2-step optimized model. This is practically useful since many algorithms exploit only one step in the inner loop. Notably, our method is applied only in the training phase, so it has the same inference cost as the baseline. Also, we only used first-order optimization for the meta-optimizer to ensure a fair comparison when adding SHOT to FoMAML. For ANIL Raghu et al. [2019] and BOIL Oh et al. [2020], we conducted experiments with the same settings from each baseline paper, and we bring the results from Oh et al. [2020] for comparison.

SHOT enables faster convergence . Our method, SHOT, achieves faster convergence compared to the baseline, as demonstrated in Figure 3, where we plot the validation accuracy against the training

epoch. To understand why this occurs, we plotted averaged cosine similarity between the overall parameter difference with each inner gradient step in the inner loop, as shown in Figure 3. We found that overall similarity increases in both cases of with and without SHOT, and the similarity is highly correlated with the validation accuracy. SHOT enforces high similarity from the beginning, resulting in faster convergence with better generalization ability.

Applying other distance metrics . We used KL divergence as a distance metric in SHOT, but it is possible to use other metrics as well. For example, we could use cross-entropy as a distance metric from a probability perspective, or we could use L2 distance between parameters as a distance metric. We conducted experiments with different distance metrics, and for L2 distance we divided the L2 distance by the square root of the number of parameters to account for the initialization scheme Kumar [2017]. For the cross-entropy loss, we used the same λ (balancing term for Eq. 4) settings as for KL divergence. The results are shown in Table 6. We observed that SHOT outperformed the baseline for every distance metric. This supports our hypothesis that we can improve performance and dynamics by suppressing the effect of the Hessian along the optimization trajectory.

6 Conclusion

In this paper, we claim that GBML implicitly suppresses the Hessian along the optimization path. We also demonstrate that we can improve the performance of GBML by explicitly enforcing this implicit prior. To achieve this, we propose a novel method, SHOT (Suppressing the Hessian along Optimization Trajectory). Although it deals with curvature of the loss surface, it has the same computational order compared to other GBML methods. More specifically, it adds a negligible computational overhead to the baseline and costs the same at inference time. We show that SHOT outperforms the baseline and is algorithm- and architecture-independent.

We believe that our method can be applied to other GBML methods in a plug-and-play manner.

References

Baik, M. Choi, J. Choi, H. Kim, and K. M. Lee. Meta-learning with adaptive hyperparameters. Advances in Neural Information Processing Systems , 33:20755-20765, 2020.

Bernacchia. Meta-learning with negative learning rates. arXiv preprint arXiv:2102.00940 , 2021.

Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control , 58(9):2217-2229, 2013.

Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems , 6, 1993.

Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15750-15758, 2021.

Deleu, T. Würfl, M. Samiei, J. P. Cohen, and Y. Bengio. Torchmeta: A Meta-Learning library for PyTorch, 2019. URL https://arxiv.org/abs/1909.06576 . Available at: https://github.com/tristandeleu/pytorch-meta.

Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR , abs/2010.11929, 2020. URL https: //arxiv.org/abs/2010.11929 .

Fallah, A. Mokhtari, and A. Ozdaglar. On the convergence theory of gradient-based modelagnostic meta-learning algorithms. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 1082-1092. PMLR, 26-28 Aug 2020. URL https://proceedings.mlr.press/v108/fallah20a.html .

Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Flennerhag, A. A. Rusu, R. Pascanu, F. Visin, H. Yin, and R. Hadsell. Meta-learning with warped gradient descent. arXiv preprint arXiv:1909.00025 , 2019.

Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently improving generalization. CoRR , abs/2010.01412, 2020. URL https://arxiv.org/abs/2010. 01412 .

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, and M. Li. Bag of tricks for image classification with convolutional neural networks, 2018. URL https://arxiv.org/abs/1812.01187 .

Hiller, M. Harandi, and T. Drummond. On enforcing better conditioned meta-learning for rapid few-shot adaptation. In NeurIPS , 2022.

Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops , 2013.

K. Kumar. On weight initialization in deep neural networks. arXiv preprint arXiv:1704.08863 , 2017.

LeCun, Y. Bengio, and G. Hinton. Deep learning. nature , 521(7553):436-444, 2015.

Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 , 2018.

Oh, H. Yoo, C. Kim, and S.-Y. Yun. Boil: Towards representation change for few-shot learning. arXiv preprint arXiv:2008.08882 , 2020.

N. Oreshkin, P. R. López, and A. Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. CoRR , abs/1805.10123, 2018. URL http://arxiv.org/abs/1805.10123 .

Park and J. B. Oliva. Meta-curvature. In NeurIPS , 2019. Published 9 February 2019.

Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR , abs/2103.00020, 2021. URL https://arxiv.org/abs/2103.00020 .

Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157 , 2019.

Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. Advances in neural information processing systems , 32, 2019.

Ravi and H. Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=rJY0-Kcll .

Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. In the Sixth International Conference on Learning Representations , 2018.

Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. CoRR , abs/1807.05960, 2018a. URL http://arxiv.org/ abs/1807.05960 .

A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 , 2018b.

Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook . PhD thesis, Technische Universität München, 1987.

Simon, P. Koniusz, R. Nock, and M. Harandi. On modulating the gradient for meta-learning. In ECCV , 2020.

Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1199-1208, 2018.

Vinyals, C. Blundell, T. Lillicrap, and D. Wierstra. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Wang, R. Sun, and B. Li. Global convergence and induced kernels of gradient-based meta-learning with neural nets. CoRR , abs/2006.14606, 2020. URL https://arxiv.org/abs/2006.14606 .

Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-ucsd birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology , 2010.

A Proof of (3)

Due to fundamental theorem of calculus,

where C is a trajectory whose start and end points are θ k and θ k +1 . In GD setting, because θ k +1 = θ k -α ∇ L ( θ k ) for some learning rate α > 0 , we can think of the straight line trajectory joining θ k and θ k +1 . In this case, the velocity vector becomes v ( t ) = -α ∇ L ( θ k ) and

where θ (0) = θ k , θ (1) = θ k +1 and θ ( t ) = (1 -t ) θ (0) + tθ (1) .

By Taylor series expansion it becomes

and combining Eq.(6) and Eq.(7) proves Eq.(3).

B SHOT is robust to hyperparameter settings

Table 7: Test accuracy % of 4-conv using SHOT with different λ values. The values in parentheses indicate the number of shots. The better accuracy between the baseline and SHOT is bold-faced.

Table 7 shows the performance of SHOT with various hyperparameter settings. We conducted experiments with different learning rates of λ , and in all cases, SHOT improved the performance of the baseline. This suggests that SHOT is robust to hyperparameter settings.

C Another viewpoint of ANIL and BOIL

In this section, we reconcile the opposite opinions of feature reuse versus feature adaptation in gradient-based meta-learning (GBML) with our hypothesis that reducing the impact of the Hessian in the inner loop can improve performance.

ANIL, proposed in Raghu et al. [2019], argues that feature reuse is key in the inner loop, where the feature remains invariant while only the decision boundary is adapted. On the other hand, BOIL, proposed in Oh et al. [2020], argues that feature adaptation is key in the inner loop, where the feature is adapted while the decision boundary remains invariant.

To test their arguments, ANIL and BOIL proposed two algorithms. ANIL freezes the encoder and only updates the head in the inner loop, while BOIL freezes the head and only updates the encoder in the inner loop. The problem is that both algorithm shown good performance thereby both arguments look persuasive despite they argue exactly in the opposite ways.

Our hypothesis, which suggests that the outer loop implicitly suppresses the Hessian along the optimization trajectory, can reconcile the arguments of both ANIL and BOIL. This is because our hypothesis implies that the model acts linearly in the inner loop. ANIL and BOIL can be interpreted as algorithms that enforce linearity in the inner loop by restricting parameters and reducing the number of non-linear components between layers.

ANIL freezes the encoder and only updates the head in the inner loop, reducing the number of nonlinear components in the inner loop. This enforces linearity in the inner loop, as the only non-linearity is the loss function. ANIL achieves better performance than MAML in 1-step optimization, as it is more powerful at 1-step optimization, which views the model as linear.

BOIL freezes the head and only updates the encoder in the inner loop, reducing the number of non-linear components in the inner loop. By applying BOIL, the gradient norm is predominant in the last layer of the encoder, making it a variant of ANIL that updates only the penultimate layer. This layer has much stronger performance, as it can change the feature while maintaining the linearity of the model. Table 14 of Oh et al. [2020] shows a boosted performance when all but the penultimate layer is not frozen. By explicitly enforcing linearity in the inner loop, BOIL achieves improved performance.

D GBMLis a variant of Prototype Vector method

In this section, we provide a novel viewpoint of GBML, that GBML (Gradient-Based Meta Learning) is a varient of MBML (Metric-Based Meta Learning). This viewpoint relies on the linearity assumption . i . e ., the effect of the Hessian along the optimization trajectory is zero, thereby the model act as linear in the inner loop.

Suppose there exists a meta-learning model that satisfies the linearity assumption in the inner loop, then classifying a new classification task with a task-specific function f ( ·| θ glyph[star] ) after an inner loop is equivalent to creating a prototype vector for each class on a specific feature map and classifying the input as the class of the most similar prototype vector.

The proof starts by defining the prototype vector at first.

Prototype Vector We define a prototype vector V c for class c in an N -way K -shot classification task formally as

where X ij is the j -th input sample for the i -th class, ϕ c ( · ) ∈ H is a class-specific feature map and β ij indicates the importance of X ij for constituting the prototype vector V c .

In other words, there exists a feature map ϕ c for each class c , and the support set is mapped to the corresponding feature map and then weighted-averaged to constitute the prototype vector of the corresponding class. At inference time, the classification of a given query X is done by taking the class of the most similar prototype vector as follows:

Here, ϕ : X → H is a non-class-specific mapping. We can also rewrite the prototype vector using ϕ and by defining a projection P c : X → X as glyph[negationslash]

where y ( X ) is the ground truth class of X and N is the null space of ϕ i . e ., ϕ ( ν ) = 0 .

Then by defining ϕ c glyph[defines] ϕ ◦ P c and β ij glyph[defines] 1 K , it becomes

SGD in the inner loop If GBML satisfies the hypothesis of linearity in the inner loop, f is locally linear in θ in an inner loop. More specifically, there exists an equivalent feature map ϕ c : X → H

which satisfies f c ( ·| θ c ) = 〈 θ c , ϕ c ( · ) 〉 for every x ∈ X where f ( ·| θ ) = [ f 1 ( ·| θ 1 ) , · · · , f N ( ·| θ N )] T . With the loss function L ( x, y | θ ) = D ( s ( f ( x | θ )) , y ) for some distance measure D such as cross entropy, we can formulate the inner loop of N -way K -shot meta learning by SGD as

since all samples in the support set are inputted in a batch of an inner loop.

Because the model is linear in the inner loop, the batch gradient does not change. Let β ij = -∂D ∂f c | θ 0 c ,X ij . Then after t steps, by (8), the model becomes

At the initialization step of an inner loop, there is no information about the class, even the configuration order of the class, because the task is randomly sampled. If so, the problem is solved in the inner loop. For example, if a class Dog is allocated to a specific index such as Class 3 . There is no guarantee that it will have the identical index the next time the class Dog comes in. Thus, at a meta-initialization point θ 0 , the scores for different classes would not be much different, i . e ., f i ( x | θ 0 i ) glyph[similarequal] f j ( x | θ 0 j ) for i, j ∈ [1 , · · · N ] .

Considering the goal of classification is achieved through relative values between f i ( X ) 's, the value at the initialization point does not need to be considered significantly. Therefore

So inner loop in GBML can be interpreted as making proptotype vector with given support set. glyph[square]

Table 8: Test accuracy % of 4-conv network on benchmark data sets. The values in parentheses indicate the number of shots. The best accuracy among different methods is bold-faced. To differentiate the notation, we have denoted SHOT 3 as a model that uses 3 optimization steps in f r and 1 optimization step in f t , and SHOT 6 as a model that uses 6 optimization steps in f r and 3 optimization steps in f t .

E SHOT with more optimization step in the inner loop

In the main paper, we used only one step for the target model to improve computation efficiency. However, it's also important to test if SHOT works with more optimization steps in the inner loop. As a reference model, we set the number of optimization steps to 6, which is different from the main paper where we only used 3 steps (same as the baseline). As shown in Table 9, SHOT still performs better than the baseline even with more optimization steps in the inner loop.

F SHOT can act as a regularlizer

Regularlization techniques are widely used in many GBML algorithms Rajeswaran et al. [2019], Rusu et al. [2018a] and SHOT can also act as a regularizer. We replaced the regularization term of LEO Rusu et al. [2018a], a popular GBML algorithm that uses regularization in the inner loop, with

SHOT. The results, shown in Table 9, demonstrate that SHOT slightly enhances LEO's performance. Although the results are marginal, there is still room for improvement considering LEO works with highly-fine-tuned hyperparameters. This suggests the potential of SHOT as a plug-and-play regularization technique for GBML algorithms, whether it is highly-fine-tuned or not. Also, the role of SHOT as a regularizer explains why regularization techniques are effective in GBML. If we minimize the total transport distance in the inner loop, parameters do not move much even with a large learning rate, thereby the effect of the Hessian reduces.

G Algorithm

Random direction

Gradient direction

Randomly-Initialized Model

SHOTp Pretrained Model

700

200

200

600

600

150

500

500

150

400

400

100

300

50

100

200

300

100

200

50

1.00

1.00

100

0.75

0.75

0.50

0.50

-1.00.750.500.25.00.250.500.751.00

0.25

1.00.750.50.25.00.250.500.751.00

0.25

0.00

0.00

0.25

-0.50

0.25

-1.00

0.75

0.50

-1.00

0.75

H Loss surface actually gets linear along the optimization trajectory

Fig. 4 shows that our algorithm actually do what we expected. We compared between randomlyinitialized model and SHOT p initialized model. Randomly-initialized model shows rough loss surface. The characteristic doesn't differ whether it is gradient's direction or not. However, SHOT p pretrained models shows smooth surface along the gradient direction, while other direction remains rough.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors raise the hypothesis that gradient-based meta-learning implicitly suppresses the Hessian along the inner-loop’s optimization trajectory, and introduce a method (‘SHOT’) to leverage the trajectory of an already trained reference model to guide their target model’s optimization path based on this hypothesis.\n\nSTRENGTHS:\n* Interesting avenue of research and investigation\n* Good motivation of the underlying mechanics and basis of the posed hypothesis, especially the importance of the Hessian in FSL settings in contrast to conventional many-shot deep learning\n* Demonstrated versatility of proposed method by incorporating it into a range of different FSL-approaches, and tested across a range of few-shot setting and datasets\n\nWEAKNESSES:\n**TLDR**: While I do think the idea presented here is very interesting, the current form of this manuscript has severe weaknesses that should (and need to) be improved by the authors! \n\nMajor weaknesses:\n\n**W1: Validity of some of the comparisons (to baselines)**\n - The main message of Table 1: What is the underlying reference model that the loss is applied to? (And is it applied to its initialization set of parameters?)   \nWhy is the comparison done to a ‘randomly initialized’ model? I feel that adding a comparison to a meta-learnt baseline as well as the reference model would be more appropriate and insightful to see what ‘SHOT’ actually causes, and what is achieved already by any meta-trained (reference) model.\n - Evaluation for e.g. the one-step method is done with “baselines using 1 training step”, while SHOT uses a 2-step reference model and distills this (potentially much more powerful) information into a 1-step method. While I am not questioning the contribution of the authors to use such distillation, it would be important to state:\n   - How well the underlying 2-step method actually performs (i.e. baseline with 2-opt steps);\n   - How well does training the baseline with 2-steps of learning rate alpha, and evaluating it on 1-step with 2*alpha (no SHOT) perform?   \n- L.338: ‘faster convergence’ of the training procedure -> Isn’t the ‘basis of comparison’ misleading, since your model is distilling information from an already converged model for its optimization path and initialization point,  and thus has a much stronger training signal than conventional meta-training? (so the overall convergence to enable SHOT is actually the ‘original’ model + the SHOT-constraint model, isn’t it?)\n\n**W2: Insufficient discussion of approach w.r.t to other works**\n-  Relationship to distillation methods must be (briefly) discussed, since method very alike in terms of how objective is posed\n-  Preconditioning methods in FSL/Meta-Learning must be discussed in more detail:\n    - Lines108ff: “ *SHOT can be viewed as a variant of preconditioning […]* ” -> Yet, many recent methods are neither mentioned nor contrasted to/discussed -> See [e.g. 1,2,3]\n    - Lines 109/110 state: “*unlike previous work, we define the condition of the meta-parameter based on the curvature of the loss surface*”; Note that the authors of [1] also use the curvature of the loss surface via an approximation of the Hessian for their preconditioning method    \n-> I’d like to see a discussion of:    \nWhat is the relation of the findings of this work to the findings in [1] (both are different in their perspective, [1] encouraging low condition number vs. your work encouraging low magnitude like SAM) – you nicely discuss the difference to SAM, so I feel this additional discussion would be a valuable and necessary addition to this paper\n\n[1] Hiller et al., 'On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptation', NeurIPS 2022   \n[2] Simon et al., 'On modulating the gradient for meta-learning', ECCV 2020   \n[3] Park and Oliva, 'Meta-curvature', NeurIPS2019\n\n**Further claims/points that need clarification**: \n-> Please see the \"Questions\" section.\n\nQUESTIONS:\nIn addition to the two main points raised in the weaknesses section, I'd like further clarification on the following points:\n \n- Main method: Equation (4) to me essentially is a distillation loss, distilling a set of parameters for the student (‘target’) from a teacher (‘reference’) model; \n   - How does the motivation of suppressing the Hessian here really apply? The loss is enforcing similarity between the parameters, how is it applied in practice? If the target model takes 3 steps and the reference model 6, is the set of target-parameters at step 1 compared to reference at step 2? Or how is this performed? (the stated lr (alpha) in line 226 somewhat indicates this via the fractionally-adjusted alpha, but it would be advantageous for the reader to point this out explicitly)\n    - Also, does the Hessian really play a major role in this? If you are providing ‘optimal endpoints’ or at least ‘optimal directions’ for a large step-size via the other model, only gradient-direction (highly local) really matters, doesn’t it? Is the Hessian then really suppressed, or rather ignored?\n\n- \"Suppressing the Hessian\" (name of algorithm) -> Is it possibility to actually compute the Hessian (potentially approximated version) for e.g. the small Conv4 network? It would be interesting to see whether it’s actually the case that the magnitude is small!  (It’s one of the main claims & motivations of the approach after all)\n\n\nSome additional notes/hints:   \n - Sec 5.1: “SHOT outperforms on all benchmarks” – might be wise to tone this down a bit, since the confidence intervals of almost all results are highly overlapping! (e.g. something like “performs better or on-par” might be more suited) \n - Heading 4.4 (and line 267) is an overstatement: Since shot is inflicting additional parameters and forward steps, stating that it “does not require ANY computational burden” is incorrect and should be modified appropriately (maybe akin to the wording in the introduction); It might be helpful to put further emphasis the difference between training overhead and inference time overhead (or potential saving) as done towards the end of this paragraph\n - Line 290: Lambda has not been introduced previously (to the best of my knowledge)\n - Formulation l.40 – 42: “instability caused by Hessian” -> I feel this is misleading and think the authors should clarify this -> The Hessian itself is simply a representation/measure of the curvature of the loss surface, but doesn’t itself cause anything. Instability will be due to highly non-smooth or curved loss-surface of the initial parameter space (which can be quantified via the Hessian) \n - I understand that this is purely for motivation, but the statement of ViT using 1e-6  is incorrect -> If you check the ViT paper, they state lr in the range of 1e-4 – 1e-3, and note that this is for ImageNet where significantly more parameter updates are performed due to dataset size\n - Conclusion section: I suggest reformulating the first sentence and removing the “we claim”, since at this point of the paper, you (want to) have proven/substantiated your claim.\n\n> **UPDATE**: Updated rating to weak accept after clarification regarding the use of the term 'reference model' during rebuttal phase.\n\nLIMITATIONS:\nLimitations are not clearly stated, and the authors should improve upon this (include potential limitations that the authors are aware of).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hypothesis: _gradient-based meta-learning (GBML) implicitly suppresses the Hessian along the optimization trajectory in the inner loop_. Based on the hypothesis, proposes an algorithm that **implicitly suppress the hessian** by minimizing the distance between the target (trained with fewer steps within inner-loop) and reference model (trained with more steps within inner-loop). The idea is that **a model trained with smaller learning rate will have less distortion due to the hessian**. The proposed algorithm is algorithm- and model- agnostic and improves performance.\n\nSTRENGTHS:\n- Interesting idea that suppressing hessian. \n- Impressive performance improvements.\n\nWEAKNESSES:\n- In Sec 4. some explanation is missing or confusing. Text can be more clearly written.\n    - In my understanding, reference model is \"less\" (stated as \"not\" in L208) influenced by the Hessian, because of smaller learning rate, not increased the number of optimization steps. L97-99 states smaller learning rate makes hessian less affect the convergence.\n    - $D(\\theta_t, \\theta_r)$ is a distance metric, but using KL divergence as a distance metric is confusing. I guess $D$ is not necessarily a distance metric.\n    - How gradient from $L_{SHOT}$ is defined? Is $\\theta_r^R$ detached or does it participate in gradient computation?\n    - In table 1. how SHOT-init (or lower-distortion-initialization) is obtained? Are Random/SHOT-init meaning that they are trained w/o or w/ SHOT loss? Or is this relevant to $SHOT_p$?\n    - What's the motivation of using model trained only with SHOT loss as meta-meta-initializer? If it starts with the initialization, then suppressed hessian property is maintained without SHOT loss?\n    - Which parameter, $\\theta_r$ or $\\theta_t$, is used for MAML loss?\n    - \"$\\theta_t$ does not require additional backprop.\" means that it reuses the gradient at the 1st step of $\\theta_r$ but using different lr?\n\n- $\\lambda$ appears in Sec. 5, but not defined. (Maybe weights of SHOT loss?)\n\nQUESTIONS:\n- How gradient from $L_{SHOT}$ is defined? Is $\\theta_r^R$ detached or does it participate in gradient computation?\n - In table 1. how SHOT-init (or lower-distortion-initialization) is obtained? Are Random/SHOT-init meaning that they are trained w/o or w/ SHOT loss? Or is this relevant to $SHOT_p$?\n- What's the motivation of using model trained only with SHOT loss as meta-meta-initializer? If it starts with the initialization, then suppressed hessian property is maintained without SHOT loss?\n- Which parameter, $\\theta_r$ or $\\theta_t$, is used for MAML loss?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
Authors consider meta learning task and formulate the hypothesis regarding property, common for gradient based approaches: they suppress Hessian implicitly due to the big step sizes in inner loop which works for functions, which are linear on a path to task-specific optimum. Proposed method does this suppression explicitly by changing loss function or adding regulariser to it, which computation is cheap and which can be incorporated in other GBML methods

STRENGTHS:
The hypothesis formulated is natural, proposed approach is easy to combine with other approaches, empirical study' methodology is proper, experiments on several practically important tasks are presented.

WEAKNESSES:
In fact, I do not see significant advantage in using the proposed approach, because most of scores in tables, which should indicate the advantage of the approach, lie within ± std interval of other scores, which makes advantage statistically insignificant. If this impression doesn't correspond to authors' observations, they probably should present them in more friendly way, emphasizing the cases where advantage is statistically significant together with percentage of this advantage.

QUESTIONS:
No questions in addition to the recommendation given above.

LIMITATIONS:
Everything is okay.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an algorithm called SHOT (Suppressing the Hessian along the Optimization Trajectory) for gradient-based meta-learning (GBML). The authors hypothesize that GBML implicitly suppresses the Hessian along the optimization trajectory in the inner loop. Based on this hypothesis, they propose the SHOT that minimizes the distance between the parameters of the target and reference models to suppress the Hessian in the inner loop. This versatile approach, while without additional computation cost, proves to be effective in few-shot classification tasks.\n\nSTRENGTHS:\n1. The paper introduces a novel perspective on GBML by highlighting the potential benefits of suppressing the Hessian along the optimization trajectory, which can lead to performance improvements in meta-learning models.\n\n2. The proposed SHOT algorithm is simple, effective, and versatile, with the potential to be applied to any GBML baseline, irrespective of the specific algorithm and architecture in use. This broad applicability increases its potential usefulness in the field of meta-learning.\n\nWEAKNESSES:\n1. While the authors provide empirical evidence to support their claims, it would be beneficial to supplement this with theoretical guarantees or additional concrete evidence to lend further weight to their arguments.\n\n2. The paper would benefit from a wider range of experimental results, specifically those pertaining to few-shot regression tasks or reinforcement learning, in order to validate the effectiveness of SHOT across various tasks.\n\nQUESTIONS:\nCan the authors provide visualizations of the landscape to support their claims about the suppression of the Hessian along the optimization trajectory?\n\nLIMITATIONS:\nThe paper lacks a discussion regarding the limitations and potential adverse societal impacts of the proposed model in the main paper. I strongly suggest adhering to the conference rules by addressing this crucial aspect.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the inner loop of GBML. Firstly, authors hypothesize that  the outer loop of GBML suppresses the Hessians during the gradient steps of inner loop. Secondly, authors propose a novel algorithm designed to reduce the Hessian along optimization trajectory in the inner loop.\n\nSTRENGTHS:\n- The paper presents an interesting observation that the loss function in the inner loop of Gradient-Based Meta-Learning (GBML) is linear. This linearity property allows for the use of larger learning rates, which can be beneficial for the optimization process.\n- The experiments conducted in the paper show promising results. The proposed algorithm outperforms the baseline methods, indicating that the approach has practical value and can potentially lead to improved performance in real-world applications.\n\nWEAKNESSES:\n-  Section 4 is hard to follow and lacks the description of the proposed methods. The paper should include clear descriptions of the proposed methods, including algorithms or pseudocode for both $SHOT_p$ and $SHOT_r$. \n- After reading the paper I got intuition why linearity of loss allows for larger learning rates but it is still not clear for me why it leads for better generalization. As authors investigate the inner loop of GBLM I think authors should also cover this question.\n\nQUESTIONS:\n- Eq. 4 lacks the dependence on $\\theta$. Does $\\theta_t^T$ depend on $\\ theta $ (or $\\theta_r^R$ depend on $\\ theta$) when we optimize the outer loop loss function w.r.t. $\\theta$? The authors should clarify its role and include it in the equation.\n- In experiments setup authors mention \"...fixed inner loop learning rate $0.5$...\", \"...total number of steps in inner loop $3$...\" and \"...number of steps $1$ for target model $\\tau_t$ and $3$ for reference model $\\tau_r$\". Does this mean that $\\alpha_r = 0.5$ and $\\alpha_\\tau = 1/6$? If so, for me it seems unclear why the baseline makes $3$ steps on inner loop with learning rate $0.4$ and target model makes 1 step with learning rate $0.4$. Is the comparison fair, as baseline makes more steps with same learning rate? Why there is no comparision of SHOT with  the baseline with reference model in inner loop?\n- In Eq. 1 its written that normally $\\theta_\\tau^*$ is obtained by SGD, but the equation that follows uses GD, there seems to be an inconsistency\n- In Eq. 1, 2 $L(x, y \\vert \\theta; \\theta_0)$ and $L(x, y \\vert \\theta_\\tau^*; \\theta_0)$ are undefined. Do $\\theta$ and $\\theta_\\tau^*$ depend on $\\theta_0$?\n- Eq. 3 seems to be wrong. It should be $\\nabla L(\\theta(t))^T$.\n- $\\lambda$ seems to be undefined in experimental section.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-31"}