BOIL: Towards Representation Change for Few-shot Learning

1 INTRODUCTION . Meta-learning , also known as “ learning to learn , ” is a methodology that imitates human intelligence that can adapt quickly with even a small amount of previously unseen data through the use of previous learning experiences . To this aim , meta-learning with deep neural networks has mainly been studied using metric- and gradient-based approaches . Metric-based meta-learning ( Koch , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ) compares the distance between feature embeddings using models as a mapping function of data into an embedding space , whereas gradient-based meta-learning ( Ravi & Larochelle , 2016 ; Finn et al. , 2017 ; Nichol et al. , 2018 ) quickly learns the parameters to be optimized when the models encounter new tasks . Model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) is the most representative gradient-based meta-learning algorithm . MAML algorithm consists of two optimization loops : an inner loop and an outer loop . The inner loop learns task-specific knowledge , and the outer loop finds a universally good meta-initialized parameter allowing the inner loop to quickly learn any task from the initial point with only a few examples . This algorithm has been highly influential in the field of meta-learning , and numerous follow-up studies have been conducted ( Oreshkin et al. , 2018 ; Rusu et al. , 2018 ; Zintgraf et al. , 2018 ; Yoon et al. , 2018 ; Finn et al. , 2018 ; Triantafillou et al. , 2019 ; Sun et al. , 2019 ; Na et al. , 2019 ; Tseng et al. , 2020 ) . Very recent studies ( Raghu et al. , 2020 ; Arnold et al. , 2019 ) have attributed the success of MAML to high-quality features before the inner updates from the meta-initialized parameters . For instance , Raghu et al . ( 2020 ) claimed that MAML learns new tasks by updating the head ( the last fully connected layer ) with almost the same features ( the output of the penultimate layer ) from the metainitialized network . In this paper , we categorize the learning patterns as follows : A small change in the representations during task learning is named representation reuse , whereas a large change is named representation change.1 Thus , representation reuse was the common belief of MAML . ∗The authors contribute equally to this paper . 1In our paper , representation reuse and representation change correspond to feature reuse and rapid learning in ( Raghu et al. , 2020 ) , respectively . To prevent confusion from terminology , we re-express the terms . Herein , we pose an intriguing question : Is representation reuse sufficient for meta-learning ? We believe that the key to successful meta-learning is closer to representation change than to representation reuse . More importantly , representation change is crucial for cross-domain adaptation , which is considered the ultimate goal of meta-learning . By contrast , the MAML accomplished with representation reuse might be poorly trained for cross-domain adaptation since the success of representation reuse might rely heavily on the similarity between the source and the target domains . To answer this question , we propose a novel meta-learning algorithm that leverages representation change . Our contributions can be summarized as follows : • We emphasize the necessity of representation change for meta-learning through crossdomain adaptation experiments . • We propose a simple but effective meta-learning algorithm that learns the Body ( extractor ) of the model Only in the Inner Loop ( BOIL ) . We empirically show that BOIL improves the performance over most of benchmark data sets and that this improvement is particularly noticeable in fine-grained data sets or cross-domain adaptation . • We interpret the connection between BOIL and the algorithm using preconditioning gradients ( Flennerhag et al. , 2020 ) and show their compatibility , improving performance . • We demonstrate that the BOIL algorithm enjoys representation layer reuse on the low-/midlevel body and representation layer change on the high-level body using the cosine similarity and the Centered Kernel Alignment ( CKA ) . We visualize the features between before and after an adaptation , and empirically analyze the effectiveness of the body of BOIL through an ablation study on eliminating the head . • For ResNet architectures , we propose a disconnection trick that removes the backpropagation path of the last skip connection . The disconnection trick strengthens representation layer change on the high-level body . 2 PROBLEM SETTING . 2.1 META-LEARNING FRAMEWORK ( MAML ) . The MAML algorithm ( Finn et al. , 2017 ) attempts to meta-learn the best initialization of the parameters for a task-learner . It consists of two main optimization loops : an inner loop and an outer loop . First , we sample a batch of tasks within a data set distribution . Each task τi consists of a support set Sτi and a query set Qτi . When we sample a support set for each task , we first sample n labels from the label set and then sample k instances for each label . Thus , each support set contains n× k instances . For a query set , we sample instances from the same labels with the support set . With these tasks , the MAML algorithm conducts both meta-training and meta-testing . During metatraining , we first sample a meta-batch consisting of B tasks from the meta-training data set . In the inner loops , we update the meta-initialized parameters θ to task-specific parameters θτi using the task-specific loss LSτi ( fθ ) , where fθ is a neural network parameterized by θ , as follows : 2 θτi = θ − α∇θLSτi ( fθ ) ( 1 ) Using the query set of the corresponding task , we compute the loss LQτi ( fθτi ) based on each inner updated parameter . By summing all these losses , the meta-loss of each meta-batch , Lmeta ( θ ) , is computed . The meta-initialized parameters are then updated using the meta-loss in the outer loop through a gradient descent . θ′ = θ − β∇θLmeta ( θ ) , whereLmeta ( θ ) = B∑ i=1 LQτi ( fθτi ) ( 2 ) In meta-testing , the inner loop , which can be interpreted as task-specific learning , is the same as in meta-training . However , the outer loop only computes the accuracy using a query set of tasks and does not perform a gradient descent ; thus , it does not update the meta-initialization parameters . 2.2 EXPERIMENTAL SETUP . We used two backbone networks , 4conv network with 64 channels from Vinyals et al . ( 2016 ) and ResNet-12 starting with 64 channels and doubling them after every block from Oreshkin et al . ( 2018 ) . For the batch normalization , we used batch statistics instead of the running statistics during metatesting , following the original MAML ( Finn et al. , 2017 ) . We trained 4conv network and ResNet-12 for 30,000 and 10,000 epochs , respectively , and then used the model with the best accuracy on metavalidation data set to verify the performance . We applied an inner update once for both meta-training and meta-testing . The outer learning rate was set to 0.001 and 0.0006 and the inner learning rate was set to 0.5 and 0.3 for 4conv network and ResNet-12 , respectively . All results were reproduced by our group and reported as the average and standard deviation of the accuracies over 5 × 1,000 tasks , and the values in parentheses in the algorithm name column of the tables are the number of shots . We validated both MAML/ANIL and BOIL on two general data sets , miniImageNet ( Vinyals et al. , 2016 ) and tieredImageNet ( Ren et al. , 2018 ) , and two specific data sets , Cars ( Krause et al. , 2013 ) and CUB ( Welinder et al. , 2010 ) . Note that our algorithm is not for state-of-the-art performance but for a proposal of a new learning scheme for meta-learning . Full details on the implementation and data sets are described in Appendix A.3 In addition , the results of the other data sets at a size of 32 × 32 and using the 4conv network with 32 channels from Finn et al . ( 2017 ) ( i.e. , original setting ) are reported in Appendix C and Appendix D , respectively . 3 BOIL ( BODY ONLY UPDATE IN INNER LOOP ) . 3.1 THE ULTIMATE GOAL OF META-LEARNING : DOMAIN-AGNOSTIC ADAPTATION . Recently , Raghu et al . ( 2020 ) proposed two opposing hypotheses , representation reuse and representation change , and demonstrated that representation reuse is the dominant factor of MAML . We can discriminate two hypotheses according to which part of the neural network , body or head , is mostly updated through the inner loop . Here , the body indicates all convolutional layers , and the head indicates the remaining fully connected layer . In other words , the representation change hypothesis attributes the capability of MAML to the updates on the body , whereas the representation reuse hypothesis considers that the network body is already universal to various tasks before the inner loops . To demonstrate the representation reuse hypothesis of MAML , the authors proposed the ANIL ( Almost No Inner Loop ) algorithm , which only updates the head in the inner loops during training and testing , and showed that ANIL has a performance comparable to that of MAML . This implies that the representation trained by MAML/ANIL , even before updated task-specifically , is sufficient 2Although the inner loop ( s ) can be applied through one or more steps , for simplicity , we consider only the case of a single inner loop . 3All implementations are based on Torchmeta ( Deleu et al. , 2019 ) except for WarpGrad , and all results were reproduced according to our details . These results are not the highest for MAML/ANIL because our setting is more fitted to BOIL . However , under more suitable hyperparameters for each algorithm , the best performance of BOIL is better than that of MAML/ANIL . to achieve the desired performance . Furthermore , they proposed the NIL-testing ( No Inner Loop ) algorithm , which removes the head and performs unseen tasks using only the distance between the representations of a support set and those of a query set during testing to identify the capability of representation reuse . NIL-testing of MAML also achieves a performance comparable to MAML . Based on these results , it was claimed that the success of MAML is attributed to representation reuse . Here , we investigate the necessity of representation change . We believe that the meta-trained models should achieve a good performance in many other domains , which is referred to as domain-agnostic adaptation in this paper . To this end , representation reuse is not appropriate since representation reuse uses the similarity between the source and target domains . The higher the similarity , the higher the efficiency . Therefore , when there are no strong similarities between the source and target domains , good representations for the source domain could be imperfect representations for the target domain . Table 2 , which lists our experimental results on cross-domain tasks , shows that the MAML enjoying representation reuse is worse than BOIL leveraging representation change , which will be discussed in detail in the next section . 3.2 BOIL ALGORITHM . Inspired by the necessity , we design an algorithm that updates only the body of the model and freezes the head of the model during the task learning to enforce representation change through inner updates . Because the gradients must be back-propagated to update the body , we set the learning rate of the head to zero in the inner updates during both meta-training and meta-testing . Otherwise , the learning and evaluation procedures of BOIL are the same as those of MAML . Therefore , the computational overhead does not change . Formally speaking , with the notations used in Section 2.1 , the meta-initialized parameters θ can be separated into body parameters θb and head parameters θh , that is , θ = { θb , θh } . For a sample image x ∈ Ri , an output can be expressed as ŷ = fθ ( x ) = fθh ( fθb ( x ) ) ∈ Rn , where fθb ( x ) ∈ Rd . The task-specific body parameters θb , τi and head parameters θh , τi through an inner loop given task τi are thus as follows : θb , τi = θb − αb∇θbLSτi ( fθ ) & θh , τi = θh − αh∇θhLSτi ( fθ ) ( 3 ) where αb and αh are the inner loop learning rates corresponding to the body and head , respectively . MAML usually sets α = αb = αh ( 6= 0 ) , ANIL sets αb = 0 and αh 6= 0 , and BOIL sets αb 6= 0 and αh = 0 . These simple differences force the change in the dominant factor of task-specific updates , from the head to the body . Figure 1 shows the main difference in the inner updates between MAML/ANIL and BOIL . To solve new tasks , the head mainly or only changes in MAML/ANIL ( Raghu et al. , 2020 ) , whereas in BOIL , the body changes .

Summary: This paper proposes a variant of MAML, called body only in the inner loop (BOIL), in which the output layer is not trained (learning rate is set to 0, always) but gradients are still backpropagated to the remaining layers, which are trained as in MAML. In essence, this is the inverse of the ANIL method of Raghu et al., which showed that MAML mainly adapts only the output layer for few-shot image classification. This paper shows that never training the output layer of the network (simply leaving it at its randomly-initialized values) can provide much better performance across a number of few-shot image classification tasks, including those requiring domain transfer.