SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning
Authors: JunHoo Lee
Jayeon Yoo
Nojun Kwak
Venue: NeurIPS 2023 poster (2023)
Keywords: meta learning
Hessian
Gradient-Based meta learning
Feature Reuse
Implicit Prior
TL;DR: We propose our algorithm plug-and-play algorithm SHOT (Suppressing the Hessian along the Optimization Trajectory) in meta learning . SHOT works both algorithm-architecture agnostic with maintaining computation complexity.

Abstract: In this paper, we hypothesize that gradient-based meta-learning (GBML) implicitly suppresses the Hessian along the optimization
  trajectory in the inner loop. Based on this hypothesis, we introduce an algorithm called
  SHOT (Suppressing the Hessian along the Optimization Trajectory) that minimizes the distance between the parameters of the target and reference models to suppress the Hessian in the inner loop. Despite dealing with
  high-order terms, SHOT does not increase the computational complexity of the baseline model much.
  It is agnostic to both the algorithm and architecture used in GBML, making it highly
  versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT,
  we conduct empirical tests on standard few-shot learning tasks and qualitatively
  analyze its dynamics. We confirm our hypothesis empirically and demonstrate that SHOT
  outperforms the corresponding baseline.

SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning
JunHoo Lee, Jayeon Yoo, and Nojun Kwak
Scoul National University jmrjunoo, jayeon. nojunk)Osnu. ac kr
Abstract
In this paper, we hypothesize that gradient-based meta-leaminng (GBML) implicitly suppresses the Hessian along the optimization trajectory in the inner loop. Based on this hypothesis, we introduce an algorithm called SHOT (Suppressing the Hessian along the Optimization Trajectory) that minimizes the distance between the param- eters of the target and reference models to suppress the Hessian the inner loop. Despite dealing with high-order terms, SHOT does not increase the computational complexity of the baseline model much. It is agnostic to both the algorithm and architecture used in GBML, making it highly versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT, we conduct empirical tests on stan- dard few-shot learning tasks and qualitatively analyze its dynamics. We confirm our hypothesis empirically and demonstrate that SHOT outperforms the corresponding baseline. Code is available at: https://github.co https://ittubbcoo//uuHooo com/JJnHoo-Le//SHOO
Introduction
With the advent of deep learning revolution in the 2010s, machine learning has widened its application to many areas and outperformed humans in some areas. Nevertheless, machines are still far behind humans in the ability learn by themselves, in that humans can learn concepts of new data with only few samples, while machines cannot. Meta-Learning deals with this problem of learning to learn. An approach to address this issue is the srctric-based methods Chen and He [2021] Snell et al. 2017 Bromley al. [1993]. Grill et al. 2220)] Chen al. 22020) [Sugg et al. [2018 1018, Radford et al. 2021. which aims to learn "good" kernels in order to project given data into a well-defined feature
space. Another popular line of research is aptimization-basee methods Finn al. [2017]. Ravi and
Larochelle (2017). Rusu et al. [2018a], so-called Gradient-Bart Meta-Learning (GBML), which
aims achieve the goal of meta-learning by gradient descent.
As the most representative GBML methods applicable to any model trained with the gradient descent process, model-agnostic meta-learning (MAML) Finn et al. [2017] and its variations Raghu et al. [2099), Oh et al. 2220), Rusu et al. [2018b exploit nested loops to find good meta-imitializztion point from which
Oh
tasks can be quickly optimized. To do so. it divides the problem into two loops: the inner loop (task-specific loop) and the outer loop (meta loop). The former tests the meta-lcarning property of fast learning, and the latter moves the meta-initialization point by observing the dynamics of the inner loop. The general training is performed by alternating the two loops. In this paper, by focusing the inner loop of GBML methods, propose optimization.
loss term for the outer loop
To properly evaluate the meta-learning property in the inner loop, GBML samples new problem for each inner loop. Fig. shows how the problem formulated. For every inner loop, the task
"Coreespondigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

p(T)
For the i-th inner loop in meta- learning, the task, 7, is sampled from the task distribution: N (e.g. 3) classes consti- tuting each task are randomly sampled with a random configuration order, meaning that the model does not have any information before it retrieves unseen task.
Figure 1:
T1
henl
T2
MON
T3
sampled from the task distribution. The classes constituting each task randomly sampled, and the configuration order (class order) is also randomized. This means that the model tackles an entirely new task at the beginning of each inner loop. Since the task is unknown and the model has solve
the problem within a few gradient steps (some extreme algorithms Raghu et al. [2019). Oh et al.] (2020) solve the problem in one gradient step), the model typically exploits large learning rates to adapt quickly. For example, MAML et al. [017 and ANIL Raghu et al. [2019) use significantly larger learning rates of 0.4 and 0.5 respectively, y, for the inner loop compared that of the outer loop (10-3). These rates are much larger than those used in modern deep learning techniques such as ViT Dosovitskiy et al. 2020). which typically use rates in the order of 10 However, using such large learning rates can lead to instability due to high-order terms like the Hessian LeCun et al. (015]. The issue of instability caused by high-order terms is particularly evident during the initial training phase. To mitigate this problem, some deep-learning methodologies employ heuristic that utilizes a smaller learning rate at the beginning Dosovitskiy et al. [2020]. He et al.] 2018). Given that GBML goes through only several optimizations in the entire inner loop. GBML's inner loop corresponds to the initial training phase. Therefore, we anticipate that the problem caused by high-order terms may be severe with a large learning rate in each inner loop. However, it is noteworthy that GBML appears to perform well despite these potential issues.
In this paper, we investigate the inner loop of GBML and its relationship with the implicit risk of using large learning rate with a few optimization steps in the inner loop. We observe that the gradient of a GBML model remains relatively constant within an inner loop. indicating that the Hessian along the optimization trajectory has minimal impact on the model's s dynamics as learning progresses. We hypothesize that this phenomenon is key property that enables GBML to mitigate the risk of using a large learning rate in the inner loop.
Our hypothesis that GBML implicitly suppresses the Hessian along the optimization trajectory suggests a desirable property for GBML algorithm. By explicitly enforcing this property instead of relying on implicit suppression, we anticipate that the model will achieve faster convergence and superior performance. However, enforeing this property is not trivial as it involves dealing with high-order terms, which naturally require a significant amount of computation. In this paper, we propose an algorithm called Suppressing the Hessian along the Optimization Trajectory (SHOT) that suppresses the Hessian in the inner loop by minimizing the distance of the target model from a reference model which is less influenced by the Hessian. Our algorithm can enforce the Hessian to have desirable properties along the inner loop without much increasing the computational complexity of a baseline model. More specifically, SHOT only requires one additional forward pass without any additional backward pass in general cases.
Our algorithm, SHOT, is agnostic both the algorithm and architecture used in GBML, making it highly versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT, we conducted empirical tests on standard few-shot learning tasks, including minilmagenet, ticred- Imagenet, Cars, and CUB Vinyals et al. [2016), Ren et al. [2018), Krause et al. [2013), [Welinder al. 2010). Furthermore, we tested SHOT's s domain ability by evaluating its performance on different benchmarks. Also, SHOT can be applied to one-step or Hessian- free baselines, acting as regularizer. Our results demonstrate that SHOT can significantly boost GBML in an algorithm- and archiecttre-indeeeedent manner. Finally, we analyzed the dynamics of SHOT and confirmed that it behaves as expected.
2
Related Works
GBML,, or optimization-basse meta-learning, is a type of methodology which wants to mimic people's 's rapid learning skill. Thus, model should learn with a small number of samples and gradient

---

steps. To achieve this property, Ravi and Larochelle 2017 tried to save meta-data to recurrent
networks. Recently, MAML-based methods Finn et al. 2217 Rajeswaran et al. [2019. Rusu et al.)
[2018)) Baik et al. 2020]. Bernacchia [2021] are typically considered as synonym of GBML. consisting of nested loops: outer loop (meta-loop) and inner loop (task-specific loop). They sample few-shot classification tasks to test the meta-learning property in the inner loop and update the meta-initialization parameters in the outer loop with SGD. To learn rapidly. they exploit a more-than- thousand times larger learning rate in the inner loop compared to modern deep learning settings such as ViT Dosovitssy et al. [020] and CLIP Radford et al. [202T]. Although some researches have proved their convergence property Fallah et al. [2020], Wang et al. [2020) , they set infinitesimal size of learning rate to prove convergence, Fallah et al. [0200 set the learning rate considering the Hessian i.e., setting the learning rate small enough so that the higher-order terms do not affect the convergence and Wang et al. [0200 proved the convergence with the assumption that the learning rate goes to zero. Although some lines of research have shown algorithms dealing with learning rates Baik et al. 2020], [eenacchia 2021], they did not explain the success of a large learning rate. [Baik]
et al.] (2200) used an additional model to predict a proper task-specific learning rate. ANIL (Almost
No Inner Loop) Raghu et al. [2019] and BOIL (Body Only Inner Loop) Oh et al.
[2020) tried to
explain GBML with the dynamics of the encoder's output features. Although both used the same feature-level perspective, they argued in exactly different ways, ANIL Raghu et al. [0099] argued
feature reuse is an essential component, while BOIL Oh et al. (2020) said that feature adaption is crucial. Both algorithms can be considered as a variant of preconditioning meta-learning since they freeze most layers in the inner loop, thereby frozen layers can be considered as warp-paran warp-parameters Flennerhag et al. (019). Our SHOT can be viewed as a variant of preconditioning as we hypothesize that there exists a suitable condition of meta-parameter that can adapt well to a given meta-learning task. However, unlike previous work, we define the condition of the meta- parameter based on the curvature of the loss surface. Additionally, our algorithm does not require any specific architecture such as warp-parameters, making it applicable to any architecture and any GBML algorithm. Finally, there exists some paper which delt with curvature, those all required some special architecture Simon et al. [200]]. or explicit calculation of the Hessian Hiller et al. [022]] Park and Oliva 2009 In contrast, we propose much weaker condition and does not require specific architecture. We only suppress the Hessian along the inner loop. In addition, we discuss in the Appendix the possibility of integrating the hypotheses proposed in ANIL Raghu et al. [209]] and BOIL Oh et al. 2020 into our SHOT framework.
Optimization methods based on second-order or first-order derivatives that take curvature into account is a vast area of research. Although they incur a higher computational cost than vanilla SGD, they have been demonstrated to exhibit superior convergence properties. Several studies have investigated the curvature of the loss surface while maintaining the first-order computational costs. Natural gradient descent [Bonnabel [2013), a type of Riemannian gradient descent employing the Fisher information metrie as measure. is coordinate- invariant, and exhibits better convergence properties than vanilla gradient descent. Given that GBML utilizes relatively fewer samples, incorporating second-order terms is computationally more feasible. As result, there is growing body of research that employs natural gradient descent in the inner loop. Additionally, the SAM optimizer Foret et al. 2020) implicitly leverages curvature by iteratively searching for flat minima during training. leading to better generalization properties than SGD. This means they find a point where the Hessian is close to zero around local minimum. Our method can be interpreted the variation of Knowledge Distillation Hinton et al. [2015). We reduced the distance between models, slow model and fast model. However, we do not need any pretrained model. Those models share same meta-initialization parameter.
poorddnate-invariant,
As our goal is to find proper meta- mettainitializzatonn point that is not affected by the Hessian along the inner loop, our proposed algorithm, SHOT, also deals with the Hessian of the loss surface. However, unlike SAM optimizer Foret et al. 2020 we do not seek a point where the Hessian is zero in any ForrntterrFootttttte
direction. We rather seek a point where the Hessian does not take effect only along the optimization trajectory in the inner loop. By using this restriction, we give a looser constraint. Also, while the SAM optimizer has a nested loop to find a proper point, SHOT does not need any iteration in a single optimization step. Also, although natural gradient descent regards curvature like our algorithm, our SHOT does not calculate the exact Hessian during inference time as we precondition the Hessian and only consider the Hessian's isss effect along the optimization trajectory. In this paper, we aim to shed light on how GBML's inner loop works. Through an analysis of the curvature of the loss surface in the inner loop, we hypothesize that the primary role of the outer loop is to implicitly regulate the
3

---

Figure 2: This figure provides the concept of the proposed SHOT. Specifically, a reference model (0*) is initialized with the same support set and starting point as our model (f*), but undergoes sig- nificantly more optimization steps. The outer loop then minimizes the distance between the two models.
O+ 0
00
curvature of the inner loop's loss surface. To reinforce this property by explicitly incorporating corresponding loss term in the inner loop, we propose an algorithm called SHOT, which takes into account the curvature of the loss surface in the inner loop and aims to enhance its generalization properties and dynamics by controlling the curvature itself. We provide empirical evidence supporting our hypothesis through the success of SHOT. Additionally, we offer new interpretation of GBML as a variant of MBML the Appendix.
3 Preliminaries: GBML
We consider the few-shot classification task of N-way K-shot, which requires learning K samples per class to distinguish N classes in each inner loop. In few-shot GBML,, we can apply gradient descent for only limited number of steps for fast adaptation.
Let (=]00) E RR-V be the output of the classifier parameterized by 0 for given input In this setting, f(-) can be considered as the logit before the softmax operation. The model parameters 0 are updated by the loss, y|x, D(s(f (z(0)), y), where x and y the input and the corresponding label, s(.) is the softmax operation and ((,,) ) is some distance metric.
As shown in Fig. | at each inner loop, we sample a task T p(T), an N-way classification problem consisting of two sets of labeled data: X" (support set) and X? (target set). In the inner loop, the goal is to find the task-specific parameter 6 which minimizes the loss L for the support set X given the initial parameter 8g as follows:
Inner Loop: 07 = arg min
Σ
(1;0)))
(a)))
Normally, 0% is obtained by SGD [Schmidhuber, 1987 Rusu et al. 2018a Sa Oh et al. 2020 Raghu
al. 2019] gk+1 0 a V ((**,, where & indicates an inner loop optimization step and a is learning rate. We use 7 steps in the inner loop.
Then, for the outer loop, we optimize the initialization parameters Og which improve the performance of 07 on X₂ as follows:
Outer Loop: 0 = arg min
Σ (..U)EX7
L(x, vl0,)
Although GBML gained its success with this setting. this success is quite curious: since N classes constituting each task are randomly sampled and the configuration (class) order is also random, 'What characteristics of meta-initialization Og make GBML work?' remains quite uncertain.
4 Optimization in GBML
4.1 Hypothesis: Outer loop implicitly forces inner loop's 's loss surface linear
The perspective of gradient flow provides way to understand the dynamics of optimization. In this view, the evolution of model parameters is seen as discretization of continuous time evolution. Specifically, the equation governing the model parameters is á (() where 0 represents the model parameter and L(#) is the loss function. With gradient flow, we can characterize our models into two principal types: the Fast Model and the Slow Model. These definitions emerge from our observation of the models" convergence timelines. Considering the terminal point in time as I and the initialization as O, we deduce the timestamp of a singular learning step as N where signifies the number of steps.

---

Cosine Similarity
Validation Accuracy
0.9
0.6
0.7
0.6
0.4
SHOT
MAML SHOT P
MAML SHOTT
MAML SHOT
0.4
MAML
MAML
0.2
250
300
50
100
150
200
250
300
Figure 3: (left) Averaged cosine similarity between the overall parameter difference (0) - 0)) and each step's gradient (V [((h)) in the inner loop over the training (meta) epochs. The cosine similarity increases training progresses, indicating that the gradient does not vary much during an inner loop. (right) Validation accuracy vs. training epoch. To ensure a fair comparison, we shifted SHOT, for 30 epochs it pretrains for 30 epochs with LSнoT only. For this figure, we trained SHOT, for 300 epochs to compare it at a glance. Our results show that SHOT converges much faster than the baseline. Additionally, observed a strong correlation between cosine similarity and validation accuracy.
Fast Model: A model with relatively small value of N is should converge rapidly. A salient exemplar is GBML's inner loop. typically evolving within a minuscule span of to 5 optimization steps. Such rapid convergence necessitates elevated learning rate.
Slow Model: In contrast, models with a larger value of take longer to converge. Traditional deep learning models fall into this category. They require many gradient steps and a lower learning rate.
This separation makes clear the differences between typical deep learning models and GBML's inner loop. However, we should note that GBML's high learning rate approach can arise some issues.
L' VL(0()) VL(O S
- A ydt > 0,
(3)
Gradient descent the notion of treating the model as a (locally) linear approximation, adjusting its parameters in direction where the loss decreases most rapidly. If model adheres Eq, 3 we can expect the gradient descent effective in that context. (Refer to Appendix [ the detailed proof.) However, the inner loop of GBML can be characterized as fast model. This implies pronounced influence of the Hessian in the optimization dynamics. Due to this, we cannot guarantee loss decreases in the inner loop. In this paper, We argue that the outer loop's 's primary role in GBML to counteract this influence of the Hessian. In more specific terms, aims to attenuate the impact of ((*)) (@*) the optimization trajectory.
Fig. supports our claim. As mentioned before, the optimization steps are a discrete representation of the gradient over time. This allows us to track how the gradient changes as time progresses. In meta-learning, the gradient's direction and magnitude do not change significantly as time progresses. This is because meta-learning algorithms are designed to learn how to quickly adapt to new tasks. By learning how to adapt to new tasks, meta-lcarning algorithms can avoid the need to learn from scratch on each new task. In contrast, in conventional deep learning, the gradient's 's direction and magnitude change significantly as time progresses. This is because conventional deep learning algorithms are not designed to learn how to quickly adapt to new tasks. Instead, conventional deep learning algorithms are designed to learn a general model that can perform well on a wide variety of tasks. The difference in how the gradient changes over time between meta-learnin and conventional deep learning is one of the key factors that explains why meta-learning algorithms are able to learn new tasks so quickly.
It seems clear that Eq. plays crucial role in GBML, suggesting that we can enhance its performance and dynamics by explicitly inducing the model to meet this condition. However, the challenge remains: how can we directly reduce the Hessian?
5

---

Table 1: Comparison of the test accuracies (%) of a 4-convolutional network with random initialization and SHOT. SHOT does not receive label information from the outer loop. (i.e., SHOT, without supervision The numbers in parentheses indicate the number of shots.
meta-train
minilmageNet
Cars
meta-test
minilmageNet
tieredImageNet
Cars
Cars
CUB
minilmageNet
Random-init (1)
21.36 1: 0.01
21.20 0.01
21.10 1: 0.93
21.20 0.93
21.62 0.02
21.36 0.01
SHOTTinii (1)
24.62 1 0.04
23.80 0.06
24.63 1 0.04
25.84 1 0.06
25.86 1 0.03
26.48 0.04
Random-init (5)
21.29 + 0.09
21.72
0.19
21.58 + 0.37
21.58
0.04
22.31
0.06
21.28 0.09
SHOT-init (5)
27.32 + 0.59
31.28 1.68
29.10 # 1.81
28.28 + 0.79
30.34
2.69
26.61 1.05
4.2 SHOT (Suppressing Hessian along the Optimization Trajectory)
In this part, we present our novel algorithm, Suppressing Hessian along the Optimization Trajectory (SHOT), which is the main contribution of our paper. SHOT is designed to obviate the effect of the Hessian along the inner loop while maintaining the order of computation cost.
As previously discussed, directly reducing the Hessian is impractical. However, we can reduce the effect of the Hessian along the optimization trajectory, specifically by minimizing the product ((,,) V ((+).. which is in line with the condition expressed in Eq. 3 and computationally feasible. To achieve this, we define measure that quantifies the effect of the Hessian along the optimization trajectory.
To do so, we first create reference model that is not influenced by the Hessian. We obtain this reference model by increasing the number of optimization steps. The intuition behind this that as we increase the number of optimization steps, the smaller the discretized step becomes, which means that the model with many optimization steps is less influenced by the Hessian. We then define the measure as the distance between the target model (a model with few optimization steps, 0,) and the reference model (a model with many optimization steps, 0,). This distance metric, denoted D(0,0,), quantifies the distortion along the optimization path and evaluates the deviation from the ideal optimization trajectory. By minimizing this measure, we can achieve our goal of reducing the effect of the Hessian along the optimization trajectory. We will call this method as SHOT (Suppressing Hessian along the Optimization Trajectory) which adds the following loss term in the outer loop of conventional GBML algorithm:
LSHOT = D(0T,0..
(4)
Here, and R denote the number of steps in an inner loop used to obtain the target and reference
model respectively. In this paper, we used KL divergence (K (KL(s(f(O) (( ))) as a distance
metric D, but we also conducted experiments with other distance metrics such as L2(1 - 0/113) and cross-entropy (- E. s(f (( )) log sff()) as shown in Table. 6 We set T < R because intention to measure distortion caused by the Hessian in the inner loop. As we increase the number of optimization steps in the inner loop, the effect of distortion by the Hessian decreases. Therefore, we could use SHOT as a pseudo-measure of distortion in the inner loop caused by the Hessian. To match the scale of distribution between 07¹ and 6,R, we set a in Eq. 1 as a, F 0++
Reducing the effect of Hessian along the optimization in the inner loop, as measured by Eq, can lead to faster convergence and better performance. By minimizing the deviation from the ideal optimization trajectory, SHOT can effectively enforce the condition expressed in Eq. 3 which is critical for the success of GBML. As shot is the measure of distortion in the whole trajectory of an inner loop. We use SHOT at the outer loop, where scoring of each optimization trajectory is done.
Table highlights the importance of distortion in GBML, We compared the performance of a randomly-initialian model with that of the distortion introduced in Eq. in the outer loop. As our SHOT loss is self-supervisory that can be applied without target class labels, no target labels are given here. The results demonstrate that model outperforms the randomly-initialize model, indicating that distortion is a crucial factor in GBMI.. This also suggests that we can generally improve GBML. by incorporating our loss term, SHOT.
wwwddiioootoonnniitiiizeeee model, where we only reduced
the lowee-distortion-iittaaaa

---

4.3 SHOT can be used as regularizer or metaamettainittaliier in the outer loop.
One potential approach to incorporating SHOT is to use it as a regularizer in GBML. This approach involves adding SHOT to the loss function of the outer loop during training, thereby leveraging it as regularizer reducing the effect of the Hessian along the optimization trajectory. Alternatively, SHOT can be utilized as a meta-meta-inie meta-initializer model is pre-trained with SHOT, i.e., The outer loss is substituted with HOT without utilizing any labeling information. Then the model is trained with the original loss function. However, there is a vulnerability to homogeneous corruption, where the loss goes to zero only if 0g and 8, produce the same output of all inputs. As this phenomenon is trivial in the celf-supervise learning area, drew inspiration from B BOL Grill et al. 2020]. To address this issue, we added a projector to the reference model, while the target remained the same. We also augmented XT. After pre-training with SHOT, we started training by loading the best model with the highest validation accuracy. To differentiate between SHOT utilized as a regularizer and SHOT utilized as meta--etaainnittalize,, we denote them as SHOT, and SHOTp. respectively.
is used to obtain Table I In this approach, the entire
4.4 SHOT does not require any computation burden
Several meta-learning algorithms have already assumed linearity in the inner loop. For example, Fo- MAMI. and Reptile Finn et al. (2177), Nichol et al. [2018) proposed Hessian-Free GBML algorithms that both assumed linearity in the inner loop. Additionally, ANIL Raghu et al. (019)) proposed a feature reuse concept that allows for translation to zero distortion in the inner loop. However, none of these studies explicitly argued that GBML implicitly suppresses the Hessian in the inner loop as an inductive prior, nor did they propose an algorithm that controls the Hessian. Our algorithm is similar to the 'SAM' optimizer Foret et al. [2020) or natural gradient descent [Bonnabe]] [2013). Like
SAM and natural gradient descent, our algorithm searches for a good initialization point based on its curvature. However, SHOT finds it without much increasing the computational cost compared to normal gradient descent, which is a notable advantage over SAM or natural gradient descent. Although SAM uses a first-order computation term, it searches for flat minima by iteratively adding noise to each point. Natural gradient descent exploits the Fisher metric, which is a second-order computation cost.
On the other hand, we can make SHOT not give any computation burden. When we the reference model 0, to exploit the same number of optimization steps in the inner loop as the baseline model and the target model Og exploits one optimization step in the inner loop, because both models share the same initialization point 0+, i.e., 0g = 0- Og can be obtained by calculating the first optimization step of 0,. Therefore, SHOT does not require any additional backpropagation. What we require is only one more inference at the endpoint 07, for SHOT,. Also, even if we set 6, to have more optimization steps than the baseline model and set 0, to have the same number of optimization steps as the baseline model, the computation cost at test time is unchanged as it is only used in the training phase.
5 Experiments
Datasets Our method is evaluated minilmageNet (Vinyals al., 2016]. tieredImage Ree et al. [2018), Cars Krause et al. [2013), and CUB Welinder et al. [2010] datasets. minilmageNet and magee are subsets of ImageNet (ussakkvskk et al. [2015] with 100 classes each and can be
considered as general datasets. Cars and CUB are widely used benchmarks for fine-grained image classification tasks as they contain similar objects with subtle differences. By conducting experiments on these datasets, we can evaluate the performance of our method across a range of tasks.
Architectures We use 4-Conv from Finn et al. 2017) and ResNet- 12 from Oreshkin et al. 2018 as our backbone architecture. We trained 4-Conv for 30k iterations and ResNet- 12 for 10k iterations.
Experimental Setup To ensure a fair comparison, we fine-tuned MAML Finn et al. (2017) first, and our baselines outperformed the original paper. We used Adam Kingma and Ba [2014] as our optimizer and fixed the outer loop learning rate to 10-3 and inner loop learning rate to 0.3 for 4-Conv, and 10-5 and 0.01 for ResNet- 12. We restricted the total number steps in the inner loop to 3, as argued in Finn et al. [2077). We set the number of steps to for the target model O, and 3 for the reference model 0, in our SHOT. With this setting, we can exploit SHOT without any additional computation burden as discussed in See 4.4 For SHOT,, fixed À to 0.1 when we set the original
7

---

Table 2: Test accuracy % of 4-conv network on benchmark data sets. The values in parentheses indicate the number of shots. The best accuracy among different methods is bold-faced.
Doman
General (Coaase grained)
Speciicc (Fine grained)
Dutuset
mimilmageNel
magedll
Cass
Cub
())
47.88
1 0.35
46.93
1,07
47.78 0.99
57.04 1: 1.42
MAML SHOT, (1)
0.71
47.53
50.44
0.64
MAML- + SHOT, (1)
48.11 2: 0.26
47,33
0.68
89.08
0.88
58,23
1: 0.39
MAML (5)
64.81 = 1.55
66.12 +: 1,10
02:34
+ 2.11
72.48
= 0.80
MAML SHOT, (5)
0.58
69.08
64.20 1.58
73.38
0.32
MAML SHOT, (5)
+: 0.27
69.94
64.84
2.87
73.27 + 0.40
Table 3: Test accuracy % of c-nv network cross-domain adaptation. The values in parentheses indicate the number of shots. The best accuracy among different methods is bold-faced.
Idaptation
Generaa In General
General to Specitic
Specticc to General
Spectfic To Specile
Dereill mogeNee
mimlmageeNe GeredlmageNNe
mimimageNN
No
Cara
CLR
meta
minlmagNN
minilmageNe
BereeImageNe
()
47.50 1.56
51.84
34.41
0.477
40.91 0.57
28.77 1.17
30.79
32.74
1.12
30.55
1.4]
MAML + SHOT, (1)
48.20 0.64
51.08 =: 0.08
34.03
1.81
41.58
= 0.56
30.19 0.30
31.96 0.50
33.11
0.41
51.21
# 0.64
MAMI + SHOT, (1)
48.45 + 0.13
51.97 + 0.25
34,84
0.16
41.65 0.70
29.02
0.75
31.26 +.544
33,69 + 0.45
31.45 + 0.30
MAML(()
66.36 = 1.38
67.5
= 1.22
46.57
= 0.53
56,32
= 1.17
37.23
1.35
41.00 126
44 2.25)
4134 1:25
MAML + SHOT,
70.70 0.29
69.48 0.17
48.42
1 0.72
58.40 0:48
40.79 0.93
42.73 0.332
44.47 0.25
43.46
1 0.89
MAML SHOT,
69.
47.99
0.44
37.96
41.98 0:23
43.83
40.64 1.25
model as 6,. For one-step algorithms, we set the number of optimization steps to for 8, and 1 for O,, So actual optimization step in inference time is I, which is the same as the baseline. We set X for 10 in that case. For the Ressian-Freee algorithm FoMAML, we fine-tuned À to meet the best performance. We set the learning rate of the outer and inner loops to 10 and 0.01, respectively, for 4-Conv. To maintain conciseness, we only report the results of SHOT, except for the comparison with MAML baseline. We conducted our experiments method using torchmeta Deleu et al. [2019) library except for the implementation of LEO Rusu et al. 2018a where used the official implementation and embeddings.
single A100 GPU. We implemented our
5.1 Characteristics of SHOT
SHOT Outperforms the Baselines on Bench- marks and Cross-Benchmark Settings Our method, SHOT, outperforms the corresponding baseline on all benchmarks, as shown in Table 2 Both SHOT, and SHOT, outperform the baseline, regardless of whether the benchmark general or fine-grained. Additionally, SHOT generalizes well to boos-beenchark settings, demonstrated in Table 3, where it improves the performance of the baseline. Also, we can see
Table 4: Test accuracy % of ResNet- 12. The values in parentheses indicate the number of shots. The better accuracy between the baseline and SHOT is bold-faced.
mntoo-train
minilntageeer
mecta-Isst
minimmageNer
heredimnggeer
Clrs
())
49,47
$4.88
32.74 0.48
MAML- SHOT, (1)
$1.11
0.32
$5.14 0.63
32.66 4: 0.27
MAML (5)
69.12 + 0.08
71.00 0.08
51.99
=: 0.25
MAML SHOT,
69.74
71.71 0.39
52.16 0.28
that this result is independent of the backbone architecture, shown in Table 4
SHOT works with Hessian-Free and One-Step algorithms As many GBML algorithms are based on Hessian- algorithms which only exploit gradients and One-Step al aggorithms which only exploit one step in the inner loop Finn et al. [2077) Nichol et al. [2018), Oh et al. [2020) Raghu et al. 2019). it is important to show that our method also works with these algorithms. As shown in Table our method improves the performance of the bascline on both Hessian- Free and One- One-Step algorithms. For One-Step algorithms, we set Oe as the original model and 0, as the 2-step optimized model. This is practically useful since many algorithms exploit only one step in the inner loop. Notably, our method applied only in the training phase, it has the same inference cost the baseline. Also, we only used first-order optimization for the meta-optimizer to ensure a fair comparison when adding SHOT to FoMAML. For ANIL Raghu et al. [009]] and BOIL et al. [2020]. we conducted experiments with the same settings from each baseline paper, and we bring the results from Oh et al. 2020 for comparison.
SHOT enables faster convergence. Our method, SHOT, achieves faster convergence compared to the baseline, demonstrated in Figure where we plot the validation accuracy against the training
8

---

Table 5: Test accuracy (%) of 4-conv network on benchmark data sets. The values in parentheses indicate the number of shots. The better accuracy between the baseline and SHOT is bold-faced. Baselines in the table corresponds to either one-step algorithm (which uses one step optimization in the inner loop) or Hessian- Free algorithm (an algorithm which only uses gradients in the outer loop).
Domain
ConerallCoaree (aaiingggedgdd)
Specifie (Pine- grainer)nnd
Dateset
minilmageNet
BeredlanggNNer
Cirs
Cub
(.MAML()
47.70 = 0.55
47.72
0.76
49,88
60.18
= 0045
RoMAML SHOT, (1)
47.91
(NIL())
17.82
0.20
49.35
0.26
46.81
0.24
57.03
0.41
ANIL SHOT,- (3)
48.02
0.38
18.29
0.33
47.82
1.48
57.31
0.89
BOIL (1)
0.16
18.58
56.82
61:00
0.57
BOIL SHOT,
0.42
50.36
56
0.30
R-MAMI.(S)
64.49
0.46
65.25
0.64
67.99 1: 1.20
73.18 = 0.18
RoMAMI + SHOT, (5)
64.58
0:06
66.92
0.22
68.02 + 1.00
73,89
0:18
ANIL
ANTL + SHOT,- (5)
0.75
66.19
61.00
0.90
71.71
1.34
ROIL (5)
66.45
0.37
69.37
0.12
75 18
4 0.21
75.96
= 0.17
BOIL+ SHOT, (5)
66.34
0.21
69.20
0.25
75.25
4 0.14
76,77
0.51
epoch. To understand why this occurs, we plotted averaged cosine similarity between the overall parameter difference with each inner gradient step in the inner loop. as shown in Figure 3 We found that overall similarity increases in both cases of with and without SHOT, and the similarity is highly correlated with the validation accuracy. SHOT enforces high similarity from the beginning, resulting in faster convergence with better generalization ability.
Applying other distance metrics. We used KL divergence as distance metric in SHOT, but it is possible to use other metrics as well. For example, we could use cross-entropy as a distance metric from a probability perspective, or we could use L2 distance between parameters We conducted experiments with different distance metrics, and for L2 distance we divided the L2 distance by the square root of the number of parameters account for the initialization scheme Kumar (017]. For the cross- cross-entropy loss, we used the same À (balancing term for Eq. 44) settings as for KL divergence. The results are shown in T We observed that SHOT outperformed the baseline for every distance metric. This supports our hypothesis that we can improve performance and dynamics by suppressing the effect of the Hessian along the optimization trajectory.
distance metric.
Conclusion
In this paper, we claim that GBML implicitly suppresses the Hessian along the optimization path. We also demonstrate that the performance of GBML by explicitly enfore- ing this implicit prior. To achieve this, pro- pose a novel method, SHOT (Suppressing the Hessian along Optimization Trajectory). Al- though it deals with curvature of the loss surface,
Table 6: Test accuracy % of 4-conv using SHOT with various distance metrics. The values in paren- theses indicate the number of shots. The better accuracy between the bascline and SHOT is bold- faced.
improve
meta train
MinimaaeeNat
meta sest
minilmageNe:
tinedlmageNeet
Cars
Baseline (1)
47,88 1: 0.55
51.84
021
34.41
0.47
Ki-DDvergence (1)
0.71
51.68
34.03
it has the same computational order compared to other GBML. methods. More specifically, it adds a negligible computational overhead to the baseline and costs the same at inference time. We show that SHOT outperforms the baseline and is algorithm- and architecture- innpeeeeurr-ddeeedddenetttt We believe that our method can be applied to other GBML methods in plug-and and-play manner.
Cross Eniropy (1)
19.07
0.14
53,83
1: 0.34
31.54
0.31
L2 Distance (1)
50.89 + 0.06
5463
0.25
35.75
0.29
Baseline
1.63
1:22
46.77
Kl-DDvergence (5)
0.58
71.77
0.72
Ccose-Entrorp Entropy (5)
66.34
1: 0.35
69.47
1: 0.35
4886
0.71
1.2 Distance (5)
66.85 2: 0.44
69.96
0.27
48
40:00:000 0.12
7 Acknowledgements
This work was supported by NRF grant 2022-0-00953), all of which were funded by Korean Govermment (MSIT).
and IITP grants (2021-0-01343,

---

References
S. Baik, M. Choi, J. Choi, H. Kim, and K. M. Lee. Meta-learning with adaptive hyperparameters. Advances in Neural Information Processing Systems, 33:20755- 20765, 2020.
A. Bernacchia. Meta-learning with negative learning rates. arXiv preprint arXiv:2102.00940. 2021.
S. Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 5809):2217-2229, 2013.
J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah. Signature verification using a" siamese" time delay neural network. Advances neural information processing systems, 6, 1993.
T. Chen, S. Kormblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
T. Deleu, T. Würfl, M. Samiei, P. Cohen, and Y. Bengio. Torchmeta: A Meta-Learning library for PyTorch, 2019. URL https://arxiv. org/ http:/abs1199..00000 06576
Available at:
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16
words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2020. URLhttpp
//arxiv.org/ abs///10..0000.. 11929
A. Fallah, A. Mokhtari, and A. Ozdaglar. On the convergence theory of gradient-based model- agnostic meta-learning algorithms. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1082-1092 1092. PMLR, 26-28 Aug 2020. URL https://prooceedings mlr. prress/v10//aalah.D0 html
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks.
In International conference on machine learning, pages 1126-1135. PMLR, 2017.
S. Flennerhag, A. A. Rusu, R. Pascanu. F. Visin, H. Yin, and R. Hadsell. Meta-learning with warped gradient descent. arXiv preprint
2019.
P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness- minimization for efficiently improving generalization. CORR, abs/2010.01412, 2020. URL httpp:::/aaivvv.org///aaaa/// 01412
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, Doersch, Avila Pires,
Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances neural information processing systems, 23:21271-21222 2020.
T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, and M. Li. Bag of tricks for image classification with convolutional neural networks, 2018. URL httph:tttt///rteeeee org abs///b2//11111111
M. Hiller, M. Harandi, and T. Drummond. On enforcing better conditioned meta-learning for rapid few-shot adaptation. In NeurlPS, 2022.
G. Hinton, O. Vinyals, and Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 15030233 2015.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Krause, Stark, Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference computer vision workshops, 2013.
10

---

S. K. Kumar. On weight initialization in deep neural networks. arXiv preprint 2017.
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning, nature, 521(7553):436-444, 2015.
A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint vrXiv:1803.02999 2018.
J. Oh, H. Yoo, C. Kim, and S.-Y. Yun. Boil: Towards representation change for few-shot learning. arXiv preprint 20ii::0008.888222 2020.
B. N. Oreshkin, P. R. López, and A. Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. CoRR, abs/1805. 0tss/1805.1023,,, 2018. URL http::/t///a/r hwtgg//aa/////a 1805. 10123
E. Park and J. Oliva. Meta-curvature. In NewrIPS, 2019. Published February 2019.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, 0bs/2103.000020 2021. htt/s:tp:://attt
org/ abs 00020
A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards under- standing the effectiveness of maml. arXiv preprint arXiv:1909.09157.02 2019, 2019.
A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
S. Ravi and H. Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations, 2017. URL httpp:://openreeeeeee ootpe//wwe.eeeooommmmaaoooooo---
M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, Larochelle, and R. Zemel. Meta-learning for semi-supervised few-shot classification. In the Sixth International Conference on Learning Representations, 2018
O. Russakovsky, J. Deng, H. Su, Krause, Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-253 2015.
A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals. R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. CoRR, abs/1807.00960 2018a. URL http://////axi org/ abs/1807.055 .59960
A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. arXiv preprini arXiv: 1807.09900 2018b.
Schmidhuber. Evolutionary principles in self-referenttal learning, or on learning how to learn: the meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
C. Simon, P. Koniusz, R. Nock, and M. Harandi. On modulating the gradient for meta-learning. In ECCV, 2020.
J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
F. Sung. Y. Yang. L. Zhang. T. Xiang, P. H. Torr, and T. M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018.
O. Vinyals, C. Blundell, T. Lillierap, and D. Wierstra. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
H. Wang, R. Sun, and B. Li. Global convergence and induced kernels of gradient based meta-ng with neural nets. CoRR, abs/2006. 14606, 2020. URL httt/t:tt:/:/aiiii arg/abs//006.... 14606
Welinder, S. Branson, Mita, C. Wah, Schroff, S. Belongie, and P. Perona. Caltech-uesd birds 200. Technical Report CNS-TR-2010-001 California Institute of Technology, 2010.
11

---

A Proof of (3)
Due to fundamental theorem of calculus,
L(Oh+1) 1(0++))-(00)) VI(0) d0 VL(O())) =())dl,
where C' is trajectory whose start and end points are oh and gh+1, In GD setting, because @k+1 0h ((*( for some learning rate > we can think of the straight line trajectory joining H and g In this case, the velocity vector becomes e(t) -VV and
L(Ohtz) -((0t) -a //" VL(O(()) VL(00)d
(6)
where 9(0) 8h, ((1) g and 0(t) = (1 -))(())))))))) + t0()).
By Taylor series expansion it becomes
VL(((t)) = VL((k) + H() (((()) - 8h) = (I -I
(7)
and combining Eq. 6 and Eq. proves Eq.
B SHOT is robust to hyperparameter settings
Table Test accuracy of 4-conv using SHOT with different values. The values in parentheses indicate the number of shots. The better accuracy between the baseline and SHOT is fold-faced.
size of A
minilmageNet (5)
(baseline, MAML)
64.81 = 1.63
0.1 (SHOT,)
66.86 + 0.58
10-2 2
65.97 + 0.93
10
66.85 0.47
10-1
66.08 + 0.36
10-5
66.81 + 0.28
Table 7 shows the performance of SHOT with various hyperparameter settings. We conducted experiments with different learning rates of and in all cases, SHOT improved the performance of the baseline. This suggests that SHOT is robust to hyperparameter settings.
C Another viewpoint of ANIL and BOIL
In this section, we reconcile the opposite opinions of feature reuse versus feature adaptation in gradient-based meta-learning (GBML) with our hypothesis that reducing the impact the Hessian in the inner loop can improve performance.
ANIL, proposed in Raghu et al. [2019], argues that feature reuse is key in the inner loop, where the feature remains invariant while only the decision boundary is adapted. On the other hand, BOIL, proposed in Oh et al. 20220) argues that feature adaptation is key in the inner loop, where the feature is adapted while the decision boundary remains invariant.
To test their arguments, ANIL and BOIL proposed two algorithms. ANIL freezes the encoder and only updates the head in the inner loop, while BOIL freezes the head and only updates the encoder in the inner loop. The problem is that both algorithm shown good performance thereby both arguments look persuasive despite they argue exactly in the opposite ways.
Our hypothesis, which suggests that the outer loop implicitly suppresses the Hessian along the optimization trajectory, can reconcile the arguments of both ANIL and BOIL. This is because our hypothesis implies that the model acts linearly in the inner loop. ANIL and BOIL can be interpreted as algorithms that enforce linearity the inner loop by restricting parameters and reducing the number of non-linear components between layers.
12

---

ANIL freezes the encoder and only updates the head in the inner loop, reducing the number of non- linear components the inner loop. This enforces linearity in the inner loop, as the only non-linearity is the loss function. ANII achieves better performance than MAMI. in 1-step optimization, as it is more powerful 1-step optimization, which views the model as linear.
BOIL freezes the head and only updates the encoder in the inner loop, reducing the number of non-linear components the inner loop. By applying BOIL, the gradient norm predominant the last layer of the encoder, making it a variant of ANIL that updates only the penultimate layer. This layer has much stronger performance, as it can change the feature while maintaining the linearity of the model. Table 14 of et al. (2020) shows boosted performance when all but the penultimate layer is not frozen. By explicitly enforcing linearity in the inner loop, BOIL achieves improved performance.
D GBML is variant of Prototype Vector method
In this section, we provide a novel viewpoint of GBML, that GBML (Gradient- Brradient-Based Meta Learning) varient of MBML (Metric-Based Meta Learning). This viewpoint relies on the linearity assump- tion. i.., the effect of the Hessian along the optimization trajectory is zero, thereby the model act as linear in the inner loop.
Suppose there exists meta-learning model that satisfies the linearity assumption in the inner loop, then classifying new classification task with task-specific function f(-10*) after an inner loop is equivalent to creating prototype vector for each class on a specific feature map and classifying the input as the class of the most similar prototype vector.
The proof starts by defining the prototype vector at first.
Prototype Vector We define prototype vector Ve for class e in an N-way K-shot classification task formally
Ve Σ Σ (ejpa(.,)), CE (1,",,N)
(8)
where X,y is the j-th input sample for the i-th class, (-()) E H. is a class-specific feature map and Bi indicates the importance of X;j for constituting the prototype vector V..
In other words, there exists feature map sr. for each class e, and the support set is mapped to the corresponding feature map and then weighted-averaged to constitute the prototype vector of the
corresponding class. At inference time, the classification of a given query X is done by taking the class of the most similar prototype vector as follows:
è = arg mar.max(VV.o (X)).
(9)
Here, up: : H a non-cless-specifie mapping. can also rewrite the prototype vector using y and by defining a projection P. X' X as
ify(X) = e, (VEN(O)) if v() $
X
Pa(X)
(10)
where x(X) is the ground truth class of X and N is the null space of so i.e., plv) = 0.
Then by defining Pe e y Pa and Big A 1 K, becomes
1 V.
Σ F=1
((xo))-
(11)
K
SGD in the inner loop GBML satisfies the hypothesis of linearity in the inner loop, / locally linear in 0 in inner loop. More specifically, there exists equivalent feature map (P X - H
13

---

which satisfies fc(-10.) - (fe. (-()) for every E X' where f(-10) [fi(-10),,.. fN((11N))'
With the loss function Lzz, y D( (x(()), y) for some distance measure such as cross entropy, we can formulate the inner loop of N-way K-shot meta learning by SGD as
(L u( =
N
K
aD
gh+ 0
(X()))
(12)
00.
Ofe
since samples in the support set are inputted in a batch of an inner loop.
Because the model is linear in the inner loop, the batch gradient does not change. Let Big = aD
8f. Ney Then after t steps, by (8 the model becomes
K 0º = 0 al ΣΣ ==
Bizga(Xiz) =
- aVV.
(13)
At the initialization step of inner loop, there is information about the class, the configuration order of the class, because the task is randomly sampled. If so, the problem is solved in the inner loop. For example, if a class Dog is allocated to specific index such Class 3. There is guarantee that it will have the identical index the next time the class Dog comes in. Thus, at a meta-initialization point (0,, the scores for different classes would not be much different, i.e., (e():(Op) [5(x]09) for i [1.... N].
Considering the goal of classification is achieved through relative values between fi(X the value at the initialization point does not need to be considered significantly. Therefore
arg max = arg (0 (xX)) = arg max - atV (xX)) arg max(V x(X)) (14)
So inner loop in GBML can be interpreted as making proptotype vector with given support set.
Table 8: Test accuracy % of 4-conv network on benchmark data sets. The values in parentheses indicate the number of shots. The best accuracy among different methods is bold-faced. To differ- entiate the notation, we have denoted SHOT, as model that uses 3 optimization steps fr and optimization step in St, and SHOT, model that uses 6 optimization steps in Sr and 3 optimization steps in fc.
meta-train
minilmageNet
Cars
meta-test
minilmageNet
mageedInageNet
Cars
Cars
minilmageNNa
CUB
MAML (1)
47.88 1 0.55
51.84
0.24
34.41
0.47
47.78 0.99
28.67 +: 1.17
30.95 :: 1.41
MAML SHOT; (I)
47.97 1 0.71
51.68 = 0.68
34.03 1.11
1.62
30.19 0.50
31.21 2.21 0.64
MAMI. SHOT,
48.15
0.31
0.73
34.79
0.70
49.89
0.37 28.39
30.83
0.26
(5)
64.81 = 1.63
67.96
1.22
46.37
0.33
6..24:2.011
37.23
41.84
MAML SHOT, (5)
66.86 1 0.58
69.48 = 0.17
48.42 0.72
69.08 0.31
40.79
43.46 :: 0.89
MAML + SHOT, (5)
66.27 = 0.23
69.60 = 0.31
45.83 4: 1.82
66.37 + 1.93
0.74
42.63 = 0.24
E SHOT with more optimization step in the inner loop
In the main paper, used only step for the target model to improve computation efficiency. However, it's also important to test if SHOT works with more optimization steps in the inner loop. As reference model, we set the number of optimization steps to 6, which is different from the main paper where we only used steps (same as the baseline). As shown in Table 9, SHOT still performs better than the baseline even with optimization steps in the inner loop.
F SHOT can act as regularlizer
Regularlization techniques are widely used in many GBML algorithms Rajeswaran et al. (2019). Rusu et al. [2018a] and SHOT can also act as a regularizer. We replaced the regularization term of LEO Rusu et al. [2018,, popular GBML algorithm that uses regularization in the inner loop, with
14

---

Table 9: Test accuracy % of LEO [Rusu et al. 2018a]. The values in parentheses are the number of shots. The better accuracy between baseline and SHOT is bold-faced.
Dataset
minilmageNet
tieredImageNet
LEO (1)
60.24 + 0.02
65.07 0.14
LEO SHOT, (1)
60.39 + 0.07
65.26 + 0.10
LEO (5)
75.27 + 0.13
79.87 + 0.06
LEO SHOT, (5)
75.32 + 0.17
79.84 + 0.14
SHOT. The results, shown in Table demonstrate that SHOT slightly enhances LEO's performance. Although the results are marginal, there is still room for improvement considering LEO works with highly-fine-tune hyperparameters. This suggests the potential of SHOT regularization technique for GBML algorithms, whether it is highly-finee role of SHOT regularizer explains why regularization techniques effective in GBML, If minimize the total transport distance the inner loop, parameters do not move much even with large learning rate, thereby the effect of the Hessian reduces.
soug-and-layy or not. Also, the
G Algorithm
Algorithm Training Algorithm for GBML Model
Initialize parameter theta (the model)
while not reach max number of epochs:
# Sample query set Q, support set
# Start of Inner loop
Initialize reference model and target model with theta theta_r theta theta_t theta
# Optimize theta_r with more (e.g. 3) inner loop steps step_r # and smaller learning rate alpha
for i in range(step_r).
theta_r theta_r alpha gradient (loss (theta_r, S), theta_r)
# Optimize theta_t with less (e.g. 1) inner loop steps step_t
# and a larger learning rate step_r / step_t * alpha
theta_t theta_t step_r step_t * alpha
gradient coosstthea__.. S), theta_t)
# End of Inner loop
Start Outer loop
# Calculate outer loss outer_loss loss (theta_r, Q)
# Reduce the distance betreen theta_r and theta_t using KL-divergence kl_s di_eeiveeeeece (tarrgee staaggt_diistribtiiin (theitantthht,,,, Q)
peference_distribbee (thetarr, Q) detach outer_loss
lambda kl_loss
# Use meta-optimizer to optimize parameter theta theta theta beta gradient (outer_lost theta)
# End of Outer loop # End of training 1oop
15

---

handomly-Inittalizn Model
SHOTp Pretrained Model
700
500
500
150
300
100
100
200
1.00
0.75
0.50
-100 -10..85.5 28. , s0,. 1.00
1.0 20 2922 a200
0.00
-0:25
50
-0.75
-055
Random direction
Gradient direction
Figure 4:
Comparison of loss surface between randomly initialized model and SHOT, initialized model. Showing that effect of the Hessian is suppressed is along the optimization trajectory.
H Loss surface actually gets linear along the optimization trajectory
Fig. shows that our algorithm actually do what we expected. We compared between randomly- initialized model and SHOT, initialized model. Randomly candolly-initiaiizedd model shows rough loss surface. The characteristic doesn't 't differ whether it is gradient's direction or not. However, SHOT, pretrained models shows smooth surface along the gradient direction, while other direction remains rough.
16


Reviews:
[{"review_id": "v9o1Hxxi9h", "reviewer": "Reviewer_Xqme", "summary": "The paper studies the inner loop of GBML. Firstly, authors hypothesize that  the outer loop of GBML suppresses the Hessians during the gradient steps of inner loop. Secondly, authors propose a novel algorithm designed to reduce the Hessian along optimization trajectory in the inner loop.", "questions": "- Eq. 4 lacks the dependence on $\\theta$. Does $\\theta_t^T$ depend on $\\ theta $ (or $\\theta_r^R$ depend on $\\ theta$) when we optimize the outer loop loss function w.r.t. $\\theta$? The authors should clarify its role and include it in the equation.\n- In experiments setup authors mention \"...fixed inner loop learning rate $0.5$...\", \"...total number of steps in inner loop $3$...\" and \"...number of steps $1$ for target model $\\tau_t$ and $3$ for reference model $\\tau_r$\". Does this mean that $\\alpha_r = 0.5$ and $\\alpha_\\tau = 1/6$? If so, for me it seems unclear why the baseline makes $3$ steps on inner loop with learning rate $0.4$ and target model makes 1 step with learning rate $0.4$. Is the comparison fair, as baseline makes more steps with same learning rate? Why there is no comparision of SHOT with  the baseline with reference model in inner loop?\n- In Eq. 1 its written that normally $\\theta_\\tau^*$ is obtained by SGD, but the equation that follows uses GD, there seems to be an inconsistency\n- In Eq. 1, 2 $L(x, y \\vert \\theta; \\theta_0)$ and $L(x, y \\vert \\theta_\\tau^*; \\theta_0)$ are undefined. Do $\\theta$ and $\\theta_\\tau^*$ depend on $\\theta_0$?\n- Eq. 3 seems to be wrong. It should be $\\nabla L(\\theta(t))^T$.\n- $\\lambda$ seems to be undefined in experimental section.", "limitations": "-", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The paper presents an interesting observation that the loss function in the inner loop of Gradient-Based Meta-Learning (GBML) is linear. This linearity property allows for the use of larger learning rates, which can be beneficial for the optimization process.\n- The experiments conducted in the paper show promising results. The proposed algorithm outperforms the baseline methods, indicating that the approach has practical value and can potentially lead to improved performance in real-world applications.", "weaknesses": "-  Section 4 is hard to follow and lacks the description of the proposed methods. The paper should include clear descriptions of the proposed methods, including algorithms or pseudocode for both $SHOT_p$ and $SHOT_r$. \n- After reading the paper I got intuition why linearity of loss allows for larger learning rates but it is still not clear for me why it leads for better generalization. As authors investigate the inner loop of GBLM I think authors should also cover this question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing us with your insightful feedback on our paper. We apologize for any confusion caused in Section 4. To enhance the understanding of our method, **we provide a pseudo code in common response**, which will also be included in the supplementary material. As shown in Fig. 2, our algorithm embodies a straightforward design. Both the target model and reference model share the same parameter as their meta-initialization point, after which they independently adapt to the support set through gradient descent. The distinctions between these models arise from the difference in the number of optimization steps and learning rates used. In the outer loop, we add the loss using the balancing weight $\\lambda$. **We also included a detailed explanation for your question in the common response.**\n\nRegarding the connection between generalization and linearity which is an important question, Sec. 4.1 hints why SHOT would help general learning ability. Because the outer loop involves multiple tasks and our SHOT loss is added to the outer loop, we are searching for a universal linear region for multiple task losses. Therefore, if trained well, the loss surface for a new task would also be linear around the learned initialization point and optimization would become easier with a large learning rate.   Also, we have already discussed the details in Appendix D, which states that when the model satisfies the Hessian-independence condition in the inner loop, it is equivalent to a prototype network at a high level. Considering MBML (metric-based meta-learning) is well-supported by theory and proven well in that environment, we can explain the success of GBML through the lens of MBML.\n\nTo answer your questions:\n1. As mentioned in L217-218, Eq. 4 is added to Eq. 2 to find a better initialization point $\\theta_0$. Therefore, $\\theta_r^R$ in Eq. 4 corresponds to $\\theta_\\tau^*$ in Eq. 2, which is optimized in the inner loop (Eq. 1). \n2. I apologize for any confusion that you encountered in reading the experimental setup section. We trained a baseline model with 3 steps in the inner loop, and 0.5 for learning rate in the inner loop. Also, for SHOT, the reference and target model share the same meta-parameter. Reference model adopts the same training procedure from the baseline. i.e., 3 steps and 0.5 learning rate in the inner loop. And we set the target’s step size to 1, (L227), then the learning rate becomes 1.5 as $\\alpha_t = 3\\alpha_r$.  We think provided pseudo code will help you clear some confusion.\n3. Here, $\\nabla_\\theta$ is a stochastic gradient obtained by a minibatch. It is a convention of many previous works to use $\\nabla_\\theta$ for stochastic gradients.\n4. Here, $L$ is typically the conventional cross entropy loss in classification. But for other tasks such as policy gradient method (REINFORCE), it can be different. $\\theta_0$ is the initial parameter for $\\theta$, indicating that all parameters stem from the meta-parameter $\\theta_0$.  In SHOT, $\\theta^\\star_\\tau$ and $\\theta$ depend on $\\theta_0$\n5. There is a dot that indicates the operation is the inner product. However, we will fix it using the transpose as you suggested.\n6. Thank you for pointing out this. $\\lambda$ serves as the balancing term for Eq.4, and the outer loop loss becomes Eq.2 + $\\lambda$ Eq.4. We will include this clarification in the paper. \n"}, {"review_id": "zngpZEROaK", "reviewer": "Reviewer_d83w", "summary": "This paper introduces an algorithm called SHOT (Suppressing the Hessian along the Optimization Trajectory) for gradient-based meta-learning (GBML). The authors hypothesize that GBML implicitly suppresses the Hessian along the optimization trajectory in the inner loop. Based on this hypothesis, they propose the SHOT that minimizes the distance between the parameters of the target and reference models to suppress the Hessian in the inner loop. This versatile approach, while without additional computation cost, proves to be effective in few-shot classification tasks. \n", "questions": "Can the authors provide visualizations of the landscape to support their claims about the suppression of the Hessian along the optimization trajectory?", "limitations": "The paper lacks a discussion regarding the limitations and potential adverse societal impacts of the proposed model in the main paper. I strongly suggest adhering to the conference rules by addressing this crucial aspect.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel perspective on GBML by highlighting the potential benefits of suppressing the Hessian along the optimization trajectory, which can lead to performance improvements in meta-learning models.\n\n2. The proposed SHOT algorithm is simple, effective, and versatile, with the potential to be applied to any GBML baseline, irrespective of the specific algorithm and architecture in use. This broad applicability increases its potential usefulness in the field of meta-learning.", "weaknesses": "1. While the authors provide empirical evidence to support their claims, it would be beneficial to supplement this with theoretical guarantees or additional concrete evidence to lend further weight to their arguments.\n\n2. The paper would benefit from a wider range of experimental results, specifically those pertaining to few-shot regression tasks or reinforcement learning, in order to validate the effectiveness of SHOT across various tasks.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and thoughtful questions regarding our paper. \n\nIn response to your request for additional theoretical backing, I think appendix C and D would give some theoretical background on your question. We have stated the connection between GBML and MBML (metric-based meta-learning) based on the linearity assumption of the loss surface in Appendix D. We have also provided another perspective of ANIL [1] and BOIL [2] in Appendix C. As explicitly stated in lines 511-514 of Appendix, a meta-learning model that satisfies the linearity assumption in the inner loop is able to classify a new classification task using a task-specific function $f(·|\\theta^*)$ after the inner loop. This process equates to creating a prototype vector for each class on a specific feature map and then classifying the input as the class of the most similar prototype vector.\n\nRegarding your suggestion to include a broader range of experimental results, we acknowledge that incorporating tasks such as few-shot regression and reinforcement learning would enhance the paper. While our current references primarily focus on few-shot classification, we also agree that our architecture-independent algorithm can effectively extend its applicability to broader tasks including the ones you suggested. However, due to limited rebuttal period, we are not able to provide additional experimental results here and plan to supplement our material to cover these additional areas after the rebuttal period.\n\nYour suggestions about providing visualizations to support our assertions about the suppression of the Hessian along the optimization trajectory is very pertinent. We have indeed performed such visualizations, which resulted in proving our claim. It only suppressed the Hessian along the inner loop. The figure can be found in a separate one-page PDF file attached during the rebuttal which will be included in the supplementary material.  Again, your feedback has been crucial in guiding the enhancement of our work, and we appreciate your constructive critique.\n\n[1] Raghu, Aniruddh, et al. \"Rapid learning or feature reuse? towards understanding the effectiveness of maml.\" arXiv preprint arXiv:1909.09157 (2019).\n\n[2] Oh, Jaehoon, et al. \"Boil: Towards representation change for few-shot learning.\" arXiv preprint arXiv:2008.08882 (2020)."}, {"review_id": "oE5otgTx60", "reviewer": "Reviewer_Tzy2", "summary": "Authors consider meta learning task and formulate the hypothesis regarding property, common for gradient based approaches: they suppress Hessian implicitly due to the big step sizes in inner loop which works for functions, which are linear on a path to task-specific optimum. Proposed method does this suppression explicitly by changing loss function or adding regulariser to it, which computation is cheap and which can be incorporated in other GBML methods", "questions": "No questions in addition to the recommendation given above. ", "limitations": "Everything is okay. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The hypothesis formulated is natural, proposed approach is easy to combine with other approaches, empirical study' methodology is proper, experiments on several practically important tasks are presented. ", "weaknesses": "In fact, I do not see significant advantage in using the proposed approach, because most of scores in tables, which should indicate the advantage of the approach, lie within ± std interval of other scores, which makes advantage statistically insignificant. If this impression doesn't correspond to authors' observations, they probably should present them in more friendly way, emphasizing the cases where advantage is statistically significant together with percentage of this advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comment. Your point about the statistical significance of our results is well taken. We want to clarify that what we reported in our tables after $\\pm$ is the standard deviation among three differently seeded experiments. You seem to be saying that the standard deviation margins overlap, but what we reported is a sample standard deviation. Therefore, if we are to perform a t-test, it should actually be divided by the square root of the number of seeds. \n\nFor instance, if we look at Table 2's mini-imagenet (5), when we divide the STD by the square root of 3, we observe a statistically significant performance improvement which does not overlap within the significance margin. We will clarify these details in our revised manuscript. Thank you again for your constructive feedback.\n"}, {"review_id": "5aYRgHyuHW", "reviewer": "Reviewer_GyHa", "summary": "This paper proposes a hypothesis: _gradient-based meta-learning (GBML) implicitly suppresses the Hessian along the optimization trajectory in the inner loop_. Based on the hypothesis, proposes an algorithm that **implicitly suppress the hessian** by minimizing the distance between the target (trained with fewer steps within inner-loop) and reference model (trained with more steps within inner-loop). The idea is that **a model trained with smaller learning rate will have less distortion due to the hessian**. The proposed algorithm is algorithm- and model- agnostic and improves performance.", "questions": " - How gradient from $L_{SHOT}$ is defined? Is $\\theta_r^R$ detached or does it participate in gradient computation?\n - In table 1. how SHOT-init (or lower-distortion-initialization) is obtained? Are Random/SHOT-init meaning that they are trained w/o or w/ SHOT loss? Or is this relevant to $SHOT_p$?\n- What's the motivation of using model trained only with SHOT loss as meta-meta-initializer? If it starts with the initialization, then suppressed hessian property is maintained without SHOT loss?\n- Which parameter, $\\theta_r$ or $\\theta_t$, is used for MAML loss?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Interesting idea that suppressing hessian. \n- Impressive performance improvements.", "weaknesses": "- In Sec 4. some explanation is missing or confusing. Text can be more clearly written.\n    - In my understanding, reference model is \"less\" (stated as \"not\" in L208) influenced by the Hessian, because of smaller learning rate, not increased the number of optimization steps. L97-99 states smaller learning rate makes hessian less affect the convergence.\n    - $D(\\theta_t, \\theta_r)$ is a distance metric, but using KL divergence as a distance metric is confusing. I guess $D$ is not necessarily a distance metric.\n    - How gradient from $L_{SHOT}$ is defined? Is $\\theta_r^R$ detached or does it participate in gradient computation?\n    - In table 1. how SHOT-init (or lower-distortion-initialization) is obtained? Are Random/SHOT-init meaning that they are trained w/o or w/ SHOT loss? Or is this relevant to $SHOT_p$?\n    - What's the motivation of using model trained only with SHOT loss as meta-meta-initializer? If it starts with the initialization, then suppressed hessian property is maintained without SHOT loss?\n    - Which parameter, $\\theta_r$ or $\\theta_t$, is used for MAML loss?\n    - \"$\\theta_t$ does not require additional backprop.\" means that it reuses the gradient at the 1st step of $\\theta_r$ but using different lr?\n\n- $\\lambda$ appears in Sec. 5, but not defined. (Maybe weights of SHOT loss?)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for pointing out areas that need improvement. I'll provide the following responses to your points:\nRegarding the motivation drawn from Gradient-Flow, we acknowledge that our motivation could be clearer. The motivation behind using fewer epochs to reduce the loss $L$ and increasing the learning rate is essentially the same, as we explain through gradient flow. To avoid confusion, we plan to modify the wording to terms like 'fast-adaptation' and 'slow-adaption', emphasizing that a slower model evolution implies less influence from the Hessian.\n\nAlso, you mentioned that D is not a necessary a distance metric. You are correct. KL divergence is not a distance metric in a strict sense since it's not symmetric. It can, however, serve as a pseudo-distance between models. We will replace the term `metric’ with a more appropriate term in the revised manuscript.\n\nWe appreciate your observation about $\\lambda$ in Sec. 5. $\\lambda$ indeed serves as the balancing term for Eq.4, resulting in the outer loop loss of Eq.2 + $\\lambda$ Eq.4. We will include this clarification in the revised manuscript.\n\nRegarding questions:\n1. In L(SHOT), $\\theta_r$ is indeed detached from gradient computation.\n2. In Table 1, we trained SHOT$_r$ without the outer loss that forces the model to achieve high performance on the query set. As stated in L247, this approach is vulnerable to homogeneous corruption, so we used early stopping with 3 epochs. However, it may achieve higher performance if trained with SHOT$_p$. More details are discussed in the common response.\n3. The motivation for using a model trained only with SHOT loss as the meta-meta-initializer is that around the initial point $\\theta_0$ obtained by SHOT$_p$, the loss surface tends to be more linear than the region near a randomly initialized point. This makes meta-learning easier with the learned meta-initialization point.\n4. For the MAML loss, we use $\\theta_r$. (which means we use 3 steps in the inner loop, This is consistent across all methods in 4conv for inference. i.e., Same architecture, same inner loop setting except one-step optimization scheme.)\n5. Yes, \"$\\theta_t$ does not require additional backprop\" means that it reuses the gradient at the first step of $\\theta_r$, but uses a different learning rate. Although it is not explicitly written in the code, functionally, it's equivalent.\n\nBesides, I also want to point out that the term ‘reference’ is used to indicate that it is less influenced by the Hessian ‘compared to’ the Target model. Both models share the same meta-parameter (starts from the same initialization $\\theta_0$).\n"}, {"review_id": "RTPHjMjkA0", "reviewer": "Reviewer_eLqV", "summary": "The authors raise the hypothesis that gradient-based meta-learning implicitly suppresses the Hessian along the inner-loop’s optimization trajectory, and introduce a method (‘SHOT’) to leverage the trajectory of an already trained reference model to guide their target model’s optimization path based on this hypothesis.", "questions": "In addition to the two main points raised in the weaknesses section, I'd like further clarification on the following points:\n \n- Main method: Equation (4) to me essentially is a distillation loss, distilling a set of parameters for the student (‘target’) from a teacher (‘reference’) model; \n   - How does the motivation of suppressing the Hessian here really apply? The loss is enforcing similarity between the parameters, how is it applied in practice? If the target model takes 3 steps and the reference model 6, is the set of target-parameters at step 1 compared to reference at step 2? Or how is this performed? (the stated lr (alpha) in line 226 somewhat indicates this via the fractionally-adjusted alpha, but it would be advantageous for the reader to point this out explicitly)\n    - Also, does the Hessian really play a major role in this? If you are providing ‘optimal endpoints’ or at least ‘optimal directions’ for a large step-size via the other model, only gradient-direction (highly local) really matters, doesn’t it? Is the Hessian then really suppressed, or rather ignored?\n\n- \"Suppressing the Hessian\" (name of algorithm) -> Is it possibility to actually compute the Hessian (potentially approximated version) for e.g. the small Conv4 network? It would be interesting to see whether it’s actually the case that the magnitude is small!  (It’s one of the main claims & motivations of the approach after all)\n\n\nSome additional notes/hints:   \n - Sec 5.1: “SHOT outperforms on all benchmarks” – might be wise to tone this down a bit, since the confidence intervals of almost all results are highly overlapping! (e.g. something like “performs better or on-par” might be more suited) \n - Heading 4.4 (and line 267) is an overstatement: Since shot is inflicting additional parameters and forward steps, stating that it “does not require ANY computational burden” is incorrect and should be modified appropriately (maybe akin to the wording in the introduction); It might be helpful to put further emphasis the difference between training overhead and inference time overhead (or potential saving) as done towards the end of this paragraph\n - Line 290: Lambda has not been introduced previously (to the best of my knowledge)\n - Formulation l.40 – 42: “instability caused by Hessian” -> I feel this is misleading and think the authors should clarify this -> The Hessian itself is simply a representation/measure of the curvature of the loss surface, but doesn’t itself cause anything. Instability will be due to highly non-smooth or curved loss-surface of the initial parameter space (which can be quantified via the Hessian) \n - I understand that this is purely for motivation, but the statement of ViT using 1e-6  is incorrect -> If you check the ViT paper, they state lr in the range of 1e-4 – 1e-3, and note that this is for ImageNet where significantly more parameter updates are performed due to dataset size\n - Conclusion section: I suggest reformulating the first sentence and removing the “we claim”, since at this point of the paper, you (want to) have proven/substantiated your claim.\n\n> **UPDATE**: Updated rating to weak accept after clarification regarding the use of the term 'reference model' during rebuttal phase.\n", "limitations": "Limitations are not clearly stated, and the authors should improve upon this (include potential limitations that the authors are aware of).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Interesting avenue of research and investigation\n* Good motivation of the underlying mechanics and basis of the posed hypothesis, especially the importance of the Hessian in FSL settings in contrast to conventional many-shot deep learning\n* Demonstrated versatility of proposed method by incorporating it into a range of different FSL-approaches, and tested across a range of few-shot setting and datasets\n", "weaknesses": "**TLDR**: While I do think the idea presented here is very interesting, the current form of this manuscript has severe weaknesses that should (and need to) be improved by the authors! \n\nMajor weaknesses:\n\n**W1: Validity of some of the comparisons (to baselines)**\n - The main message of Table 1: What is the underlying reference model that the loss is applied to? (And is it applied to its initialization set of parameters?)   \nWhy is the comparison done to a ‘randomly initialized’ model? I feel that adding a comparison to a meta-learnt baseline as well as the reference model would be more appropriate and insightful to see what ‘SHOT’ actually causes, and what is achieved already by any meta-trained (reference) model.\n - Evaluation for e.g. the one-step method is done with “baselines using 1 training step”, while SHOT uses a 2-step reference model and distills this (potentially much more powerful) information into a 1-step method. While I am not questioning the contribution of the authors to use such distillation, it would be important to state:\n   - How well the underlying 2-step method actually performs (i.e. baseline with 2-opt steps);\n   - How well does training the baseline with 2-steps of learning rate alpha, and evaluating it on 1-step with 2*alpha (no SHOT) perform?   \n- L.338: ‘faster convergence’ of the training procedure -> Isn’t the ‘basis of comparison’ misleading, since your model is distilling information from an already converged model for its optimization path and initialization point,  and thus has a much stronger training signal than conventional meta-training? (so the overall convergence to enable SHOT is actually the ‘original’ model + the SHOT-constraint model, isn’t it?)\n\n**W2: Insufficient discussion of approach w.r.t to other works**\n-  Relationship to distillation methods must be (briefly) discussed, since method very alike in terms of how objective is posed\n-  Preconditioning methods in FSL/Meta-Learning must be discussed in more detail:\n    - Lines108ff: “ *SHOT can be viewed as a variant of preconditioning […]* ” -> Yet, many recent methods are neither mentioned nor contrasted to/discussed -> See [e.g. 1,2,3]\n    - Lines 109/110 state: “*unlike previous work, we define the condition of the meta-parameter based on the curvature of the loss surface*”; Note that the authors of [1] also use the curvature of the loss surface via an approximation of the Hessian for their preconditioning method    \n-> I’d like to see a discussion of:    \nWhat is the relation of the findings of this work to the findings in [1] (both are different in their perspective, [1] encouraging low condition number vs. your work encouraging low magnitude like SAM) – you nicely discuss the difference to SAM, so I feel this additional discussion would be a valuable and necessary addition to this paper\n\n[1] Hiller et al., 'On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptation', NeurIPS 2022   \n[2] Simon et al., 'On modulating the gradient for meta-learning', ECCV 2020   \n[3] Park and Oliva, 'Meta-curvature', NeurIPS2019\n\n**Further claims/points that need clarification**: \n-> Please see the \"Questions\" section.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The main purpose of Table 1 is to highlight that 'Hessian-independence in the inner loop' is a crucial condition for GBML. We compared the performances of parameters that are randomly initialized and those pre-trained with SHOT (parameters trained 'ONLY' with the SHOT loss, which reinforces Hessian-independence in a self-supervised manner without explicit supervisory signals). This comparison aims to demonstrate that merely meeting the necessary condition significantly enhances the model's meta-learning ability.\n\nConcerning the evaluation of the one-step method, we consent on your opinion and experience on the baseline. The miniimagenet accuracies of SHOT are  62.65% for 5-shot and 47.9% for 1 shot in 4-conv setting(Without SHOT, it recorded 61.75% for 5 shot and 47.44% for 1 shot [1] (BOIL))\n\nWe chose one-step and three-step optimization step for inference. This is because the original paper [2] (MAML) claimed that three optimization step is enough, and one-step methods are widely researched practically. For two-step, we conducted experiments on miniimagenet, and the setting is the same for other optimization.  In this case, both 5 shot and 1 shot showed betterr performance than the baselines. For 1 shot, SHOT performed 49.01% whereas baseline showed 47.13%, also, for 5 shot, SHOT obtained 65.56% which is slightly better than 65.26% of the baseline.\n\nWe apologize for the confusion regarding the term 'reference' in L.338.) We meant it to refer to a model that is less influenced to the Hessian compared to the target model. Both the target model and the reference model share the same meta-parameter within the same task. The only difference lies in the number of optimization steps and learning rate in the inner loop. Which means, we do not use any pretrained reference model during training. We will provide a pseudo code to clarify this distinction. \n\nWe appreciate your mention of recent advancements in preconditioning methods in meta-learning and connection to distillation. We noticed that ‘reference model’ can be view as a teacher in broad range. We will include brief explanation on this. Also, our approach, SHOT, does employ a form of preconditioning, but it's fundamentally different from the episode-dependent methods discussed in [3][4].\nUnlike these methods, SHOT seeks a 'global' initialization point 'along the inner loop' where the optimization trajectory is independent of the Hessian. This contrasts with other methods that attempt to satisfy a specific condition in the entire Hessian, which involves wide exploration on the local surface.\n\nThis distinction allows SHOT to operate under conditions relaxed in dimensionality - we consider dimension of $p$ (the number of parameters), rather than a $p \\times p$ matrix, reducing the need for extensive computational resources [3][5] or specific architectural adaptations [4]. This highlights the novelty of our work.\n\nLastly, in relation to the suggested link between low condition numbers and our work, we acknowledge that while those methods deal with the entire Hessian, SHOT specifically targets the effectiveness of the Hessian along the optimization trajectory.\n\n[1] Oh, Jaehoon, et al. \"Boil: Towards representation change for few-shot learning.\" arXiv preprint arXiv:2008.08882 (2020).\n\n[2] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. \"Model-agnostic meta-learning for fast adaptation of deep networks.\" International conference on machine learning. PMLR, 2017.\n\n[3] Park and Oliva, 'Meta-curvature', NeurIPS 2019\n\n[4] Simon et al., 'On modulating the gradient for meta-learning', ECCV 2020\n\n[5] Hiller et al., 'On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptation', NeurIPS 2022\n\n\nRegarding the questions,\n\n1. As mentioned earlier, we do not require a 'pre-trained' reference model. Nonetheless, we agree that the structure of the loss itself mirrors that of distillation loss. Therefore, we would like to reference previous distillation works in the related works section of our paper.\nResponse to your question about the reference model taking 6 steps when the target model takes 3 steps as discussed in Sec E, the learning rate of the reference model becomes half that of the target model. Additionally, we detach the reference model’s logit in $L_{SHOT}$.\n2. We want to clarify that we do not advocate for 'Suppressing the Whole Hessian.' As described in Eq.3, our primary concern is the Hessian ALONG the inner loop, which presents a more relaxed condition. \n3. We agree with your observation that many values lie between the standard deviations. We consent that there is a need to tone down as you suggested. However, it is essential to note that we reported the sample standard deviation, not the mean standard deviation. Applying the sample std, most of the values exceed the variance margin.\n4. As previously stated, we do not require a pre-trained model. Therefore, in SHOT$_r$, no additional backpropagation is needed.\n5. We appreciate you pointing out the need for clarity regarding $\\lambda$. $\\lambda$ indeed serves as the balancing term for Eq.4, and the outer loop loss becomes Eq.2 + $\\lambda$ Eq.4. We will include this information in the paper.\n6. Thank you for pointing out the need to change the learning rate term in regard to ViT. We will make this adjustment.\n\nRegarding your final point, because we did not include a rigorous proof of our claim, we  keep the original sentence.  \n\nWe hope these responses and the pseudo code below provide the clarification you were seeking and appreciate your insightful questions.\n"}]

Author rebuttal:
### Pseudocode for the Meta-learning Algorithm
- Reference model $\( \theta_r \)$: A model which is less-influenced by the Hessian along the inner loop. (relatively-slow adaptation by a smaller learning rate / more steps)
- Target model $\( \theta_t \)$: A model which is more-influenced by the Hessian along the inner loop (Fast adaptation by a larger learning rate / less steps)
```python
# Initialize parameter theta (the model)
while not reach max number of epochs:
    # Sample query set Q, support set S
    
    # Start of Inner loop
    # Initialize reference model and target model with theta
    theta_r = theta
    theta_t = theta
    
    # Optimize theta_r with more (e.g. 3) inner loop steps and a smaller learning rate alpha
    for i in range(3):
        theta_r = theta_r - alpha * gradient(loss(theta_r, S), theta_r)
    
    # Optimize theta_t with less (e.g. 1) inner loop steps and a larger learning rate 3 * alpha
    theta_t = theta_t - 3 * alpha * gradient(loss(theta_t, S), theta_t)
    # End of Inner loop

    # Start of Outer loop
    # Calculate outer loss
    outer_loss = loss(theta_r, Q)     
    # Reduce the distance between theta_r and theta_t using KL-divergence
    kl_loss = KL_divergence(target_distribution(theta_t, Q), reference_distribution(theta_r, Q).detach())
    outer_loss += lambda * kl_loss
    # Use meta-optimizer to optimize parameter theta
    theta = theta - beta * gradient(outer_loss, theta)
    # End of Outer loop
# End of training loop
```

### We are only interested in the Hessian ‘Along the inner loop’, not total Hessian itself.
The figure attached in our pdf compares between randomly-initialized parameters and SHOT-pretrained parameters. As you can see, only the direction along the gradient becomes linearized while other ‘random’ directions remain noisy. It implies that our algorithm effectively captures our required condition, which is a far more relaxed condition compared to suppressing the whole Hessian.

### About table 1
We intended for Table 1 to offer readers a deeper insight into GBML. The training was conducted using SHOT$_r$ without any supervision loss and was early-stopped at three epochs due to homogeneous corruption. This means that the training satisfied the given conditions without any dataset supervision. The results further supports our algorithm SHOT$_p$

### Large learning rate and Few optimization steps are analogous in the condition of Gradient Flow
We appreciate the valuable feedback and recognize the need for clarity in this section. Since having few optimization steps is analogous to using a large learning rate within the gradient flow context, we alternated between them (two phrases). We think that we can provide a detailed explanation for the connection. We will add this interpretation in the revised manuscript. 

### Regarding Section 4: Explanation of SHOT and the Intuition behind SHOT$_r$ and SHOT$_p$
We apologize for any ambiguity in our initial presentation regarding our SHOT algorithm. Let's delve deeper to clarify its fundamentals.
At its core, SHOT involves two models in the inner loop: a "reference" model and a "target" model. Their names are derived from their interactions with the Hessian:
- **Reference Model**: It undergoes more optimization steps with a smaller learning rate, which makes it less influenced by the Hessian in the inner loop.
- **Target Model**: In contrast, the target model is more influenced by the Hessian due to its variations in its epochs and a larger learning rate within the inner loop episodes. 


both models **originate from identical meta parameters**.
A pivotal concept in SHOT is the utilization of  the pseudo-distance between the target and reference models, serving as a stabilizing term. This facilitates the distinction between SHOT$_r$ and SHOT$_p$.
We've posited that an optimal GBML enforces linearity within the inner loop, effectively implying that GBMLs should tackle constrained-optimization problems. From this standpoint:
- **SHOT$_r$** operates as a regularized optimization method, imposing penalties in cases where linearity conditions are violated. 
- **SHOT$_p$** functions as a projected gradient descent technique with one iteration. It first ensures that parameters align with the desired condition before embarking on an unconstrained optimization with the conventional loss. This, in essence, makes it a variant of preconditioned GBML.
We hope that this elucidation provides a clearer understanding of SHOT's design and objectives.

### Comparison with Other Preconditioned Methods Focusing on the Hessian
Our primary concern is not with the Hessian itself but with its effect 'along' the inner loop. This differentiates our method from established algorithms [1,2]. While addressing the Hessian directly is a familiar aspect of deep learning, we take a unique perspective.  As mentioned in L122, the SAM optimizer aims to find a locally flat local minimum, possibly implying  a zero Hessian point. **However, our algorithm's emphasis isn't on the general characteristics of the Hessian. Instead, we focus solely on the Hessian's independence 'along the inner loop', which doesn't necessitate extra computational steps.**
## Clarification of parameters.
- SHOT$_r$ is used for table 1 without supervision.
- $\lambda$ serves as the balancing term for Eq.4, and the outer loop loss becomes Eq.2 + $\lambda$ Eq.4.

[1] Simon et al., 'On modulating the gradient for meta-learning', ECCV 2020

[2] Hiller et al., 'On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptation', NeurIPS 2022


Meta-review: Dear authors,

thank you for submitting your paper on suppressing the Hessian along the optimization trajectory. The paper received some positive feedback. The reviewers appreciated the observations that led to your algorithm design, which allows for larger learning rates and encouraging numerical experiments, among others.
However, several weaknesses were pointed out that should definitely be incorporated into the final version.
Fortunately, during the rebuttal phase, you addressed most of the concerns raised by the reviewers, but after reading your paper, I also agree that you should improve the presentation of your work.

Best Regards,
AC
Decision: Accept (poster)