SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\theta_ {c} ^ {t} = \theta_ {c} ^ {0} + \alpha t \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {K} \beta_ {i j} \varphi_ {c} \left(X _ {i j}\right) = \theta_ {c} ^ {0} + \alpha t V _ {c}. \tag {13}
$$
At the initialization step of an inner loop, there is no information about the class, even the configuration order of the class, because the task is randomly sampled. If so, the problem is solved in the inner loop. For example, if a class  $Dog$  is allocated to a specific index such as Class 3. There is no guarantee that it will have the identical index the next time the class  $Dog$  comes in. Thus, at a meta-initialization point  $\theta^0$ , the scores for different classes would not be much different, i.e.,  $f_{i}(x|\theta_{i}^{0})\simeq f_{j}(x|\theta_{j}^{0})$  for  $i,j\in [1,\dots N]$ .
Considering the goal of classification is achieved through relative values between  $f_{i}(X)$ 's, the value at the initialization point does not need to be considered significantly. Therefore
$$
\arg \max  _ {c} f _ {c} (X) = \arg \max  _ {c} \left\langle \theta_ {c} ^ {t}, \varphi (X) \right\rangle = \arg \max  _ {c} \left\langle \theta_ {c} ^ {0} + \alpha t V _ {c}, \varphi (X) \right\rangle \sim \arg \max  _ {c} \left\langle V _ {c}, \varphi (X) \right\rangle \tag {14}
$$
So inner loop in GBML can be interpreted as making proptotype vector with given support set.
Table 8: Test accuracy % of 4-conv network on benchmark data sets. The values in parentheses indicate the number of shots. The best accuracy among different methods is bold-faced. To differentiate the notation, we have denoted  $\mathrm{SHOT}_3$  as a model that uses 3 optimization steps in  $f_r$  and 1 optimization step in  $f_t$ , and  $\mathrm{SHOT}_6$  as a model that uses 6 optimization steps in  $f_r$  and 3 optimization steps in  $f_t$ .
meta-train	miniImageNet	Cars
meta-test	miniImageNet	tieredImageNet	Cars	Cars	miniImageNet	CUB
MAML (1)	47.88 ± 0.55	51.84 ± 0.24	34.41 ± 0.47	47.78 ± 0.99	28.67 ± 1.17	30.95 ± 1.41
MAML + SHOT3(1)	47.97 ± 0.71	51.68 ± 0.68	34.03 ± 1.11	50.44 ± 0.62	30.19 ± 0.50	31.21 ± 0.64
MAML + SHOT6(1)	48.15 ± 0.31	51.67 ± 0.73	34.79 ± 0.70	49.89 ± 0.17	28.39 ± 0.37	30.83 ± 0.26
MAML (5)	64.81 ± 1.63	67.96 ± 1.22	46.57 ± 0.53	62.24 ± 2.01	37.23 ± 1.95	41.84 ± 1.25
MAML + SHOT3(5)	66.86 ± 0.58	69.48 ± 0.17	48.42 ± 0.72	69.08 ± 0.31	40.79 ± 0.93	43.46 ± 0.89
MAML + SHOT6(5)	66.27 ± 0.23	69.60 ± 0.31	45.83 ± 1.82	66.37 ± 1.93	39.35 ± 0.74	42.63 ± 0.24
E SHOT with more optimization step in the inner loop
In the main paper, we used only one step for the target model to improve computation efficiency. However, it's also important to test if SHOT works with more optimization steps in the inner loop. As a reference model, we set the number of optimization steps to 6, which is different from the main paper where we only used 3 steps (same as the baseline). As shown in Table 9, SHOT still performs better than the baseline even with more optimization steps in the inner loop.
F SHOT can act as a regularizer
Regularization techniques are widely used in many GBML algorithms Rajeswaran et al. [2019], Rusu et al. [2018a] and SHOT can also act as a regularizer. We replaced the regularization term of LEO Rusu et al. [2018a], a popular GBML algorithm that uses regularization in the inner loop, with
Table 9: Test accuracy % of LEO [Rusu et al., 2018a]. The values in parentheses are the number of shots. The better accuracy between the baseline and SHOT is bold-faced.
Dataset	miniImageNet	tieredImageNet
LEO (1)	60.24 ± 0.02	65.07 ± 0.14
LEO + SHOTr (1)	60.39 ± 0.07	65.26 ± 0.10
LEO (5)	75.27 ± 0.13	79.87 ± 0.06
LEO + SHOTr (5)	75.32 ± 0.17	79.84 ± 0.14
SHOT. The results, shown in Table 9, demonstrate that SHOT slightly enhances LEO's performance. Although the results are marginal, there is still room for improvement considering LEO works with highly-fine-tuned hyperparameters. This suggests the potential of SHOT as a plug-and-play regularization technique for GBML algorithms, whether it is highly-fine-tuned or not. Also, the role of SHOT as a regularizer explains why regularization techniques are effective in GBML. If we minimize the total transport distance in the inner loop, parameters do not move much even with a large learning rate, thereby the effect of the Hessian reduces.
Algorithm 1 Training Algorithm for GBML Model
Initialize parameter theta (the model)   
while not reach max number of epochs: # Sample query set Q, support set S # Start of Inner loop Initialize reference model and target model with theta theta_r = theta theta_t = theta # Optimize theta_r with more (e.g. 3) inner loop steps step_r # and a smaller learning rate alpha for i in range(step_r): theta_r = theta_r - alpha * gradient(loss(theta_r, S), theta_r) # Optimize theta_t with less (e.g. 1) inner loop steps step_t # and a larger learning rate step_r / step_t * alpha theta_t = theta_t - step_r / step_t * alpha * gradient(loss(theta_t, S), theta_t) # End of Inner loop # Start of Outer loop # Calculate outer loss outer_loss = loss(theta_r, Q) # Reduce the distance between theta_r and theta_t using KL-divergence kl_loss = KL_divergence(target_distribution(theta_t, Q) , reference_distribution(theta_r, Q).detach()) outer_loss += lambda * kl_loss # Use meta-optimizer to optimize parameter theta theta = theta - beta * gradient(outer_loss, theta) # End of Outer loop   
# End of training loop
Figure 4: Comparison of loss surface between randomly initialized model and  $\mathrm{SHOT}_p$  initialized model. Showing that effect of the Hessian is suppressed is along the optimization trajectory.
H Loss surface actually gets linear along the optimization trajectory
Fig. 4 shows that our algorithm actually do what we expected. We compared between randomly-initialized model and  $\mathrm{SHOT}_p$  initialized model. Randomly-initialized model shows rough loss surface. The characteristic doesn't differ whether it is gradient's direction or not. However,  $\mathrm{SHOT}_p$  pretrained models show smooth surface along the gradient direction, while other direction remains rough.