Learning to Learn with Conditional Class Dependencies

1 INTRODUCTION . In machine learning , the objective of classification is to train a model to categorize inputs into various classes . We usually assume a categorical distribution over the label space , and thus effectively ignore dependencies among them . However , class structure does exist in real world and is also present in most datasets . Although class structure can be implicitly obtained as a by-product during learning , it is not commonly exploited in an explicit manner to develop better learning systems . The use of label structure might not be of prime importance when having access to huge amounts of data , such the full ImageNet dataset . However , in the case of few-shot learning where little data is available , meta-information such as dependencies in the label space can be crucial . In recent years , few-shot learning—learning from few examples across many tasks—has received considerable attention ( Ravi & Larochelle , 2016 ; Snell et al. , 2017 ; Finn et al. , 2017 ; Vinyals et al. , 2016 ) . In particular , the concept of meta-learning has been shown to provide effective tools for few-shot learning tasks . In contrast to common transfer learning methods that aim to fine-tune a pre-trained model , meta-learning systems are trained by being exposed to a large number of tasks and evaluated in their ability to learn new tasks effectively . In meta-training , learning happens at two levels : a meta-learner that learns across many tasks , and a base-learner that optimizes for each task . Model-Agnostic Meta-Learning ( MAML ) is a gradient-based meta-learning algorithm that provides a mechanism for rapid adaptation by optimizing only for the initial parameters of the base-learner ( Finn et al. , 2017 ) . Our motivation stems from a core challenge in gradient-based meta-learning , wherein the quality of gradient information is key to fast generalization : it is known that gradient-based optimization fails to converge adequately when trained from only a few examples ( Ravi & Larochelle , 2016 ) , hampering the effectiveness of gradient-based meta-learning techniques . We hypothesize that under such circumstances , introducing a metric space trained to encode regularities of the label structure can impose global class dependencies on the model . This class structure can then provide a high-level view of the input examples , in turn leading to learning more disentangled representations . We propose a meta-learning framework taking advantage of this class structure information , which is available in a number of applications . The Conditional class-Aware Meta-Learning ( CAML ) model is tasked with producing activations in a manner similar to a standard neural network , but with the additional flexibility to shift and scale those activations conditioned on some auxiliary meta-information . While there are no restrictions on the nature of the conditioning factor , in this work we model class dependencies by means of a metric space . We aim to learn a function mapping inputs to a metric space where semantic distances between instances follow an Euclidean geometry—classes that are semantically close lie in close proximity in an ` p sense . The goal of the conditional class-aware transformation is to make explicit use of the label structure to inform the model to reshape the representation landscape in a manner that incorporates a global sense of class structure . The contributions of this work are threefold : ( i ) We provide a meta-learning framework that makes use of structured class information in the form of a metric space to modulate representations in few-shot learning tasks ; ( ii ) We introduce class-aware grouping to improve the statistical strength of few-shot learning tasks ; ( iii ) We show experimentally that our proposed algorithm learns more disentangled representation and achieves competitive results on the miniImageNet benchmark . 2 BACKGROUND . We start by describing the meta-learning formulation proposed by Vinyals et al . ( 2016 ) and Ravi & Larochelle ( 2016 ) , and review MAML ( Finn et al. , 2017 ) , of which CAML is an instance . 2.1 META-LEARNING PROBLEM FORMULATION . The goal of meta-learning is to learn from a distribution of tasks . The learning happens on two levels : ( i ) a meta-level model , or meta-learner , that learns across many tasks , and ( ii ) a base-level model , or base-learner , that operates within each specific task . Meta-learning happens in task space , where each task can be treated as one meta-example . In the meta-learning formulation , we define a collection of regular tasks as meta-sets D , and each task D∈D has its own Dtrain and Dtest split . Dtrain is often denoted as the “ support set ” andDtest the “ query set ” . The resulting meta-learner objective is to choose parameters θ that minimize the expected loss L ( · ; θ ) across all tasks in D , θ∗=argminθED∼D [ L ( D ; θ ) ] . At the meta-level , the meta-sets D can be further split into disjoint meta-training set Dmeta−train , meta-validation set Dmeta−valid and meta-test set Dmeta−test . The meta-learner is trained on Dmeta−train , validated on Dmeta−valid and finally evaluated on Dmeta−test . 2.2 MODEL-AGNOSTIC META-LEARNING . Model-Agnostic Meta-Learning ( Finn et al. , 2017 ) is a meta-learning algorithm that aims to learn representations that encourage fast adaptation across different tasks . The meta-learner and base-learner share the same network structure , and the parameters learned by the meta-learner are used to initialize the base-learner on a new task . To optimize the meta-learner , we first sample a set of tasks { D1 , D2 , ... , DS } from the meta-training set Dmeta−train . For a meta-learner parameterized by θ , we compute its adapted parameters θi for each sampled task Di . The adapted parameters θi are task-specific and tell us the effectiveness of θ as to whether it can achieve generalization through one or a few additional gradient steps . The objective of the meta-learner is to optimize the representation θ such that it leads to good task-specific adaptations θi with only a few gradient steps . The meta-learner performs slow learning at the meta-level across many tasks to support fast learning on new tasks . At meta-test time , we initialize the base-learner with the meta-learned representation θ∗ followed by gradient-based fine-tuning . 3 METHOD . 3.1 CONDITIONAL CLASS-AWARE META-LEARNING . As shown in Figure 1 , the proposed Conditional class-Aware Meta-Learning ( CAML ) is composed of four components : an embedding function fφ that maps inputs to a metric space , a base-learner fθ that learns each individual task , an adaptation function fc that conditionally modulates the representations of the base-learner , and a meta-learner that learns across different tasks . Figure 1 depicts a toy illustration of the task inference procedure where examples from three classes are mapped onto a metric space using fφ , which are further used to modulate the base-learner fθ through a conditional transformation function fc . The main contribution of this paper is to incorporate metric-based conditional transformations ( fc ) into the meta-learning framework at the instance level . A notable feature of the proposed method is that the model has a global sense of the label space through the embedding function fφ by mapping examples onto the semantically meaningful metric space . The embeddings on the metric space inform the base-learner fθ about the label structure which in turn helps disentangle representations from different classes . This structured information can also provide a global view of the input examples to improve gradient-based meta-learning . In a simplistic form , our proposed model makes predictions using ŷ=fθ ( x ; fc ( fφ ( x ) ) ) , where the base-learner fθ is conditioned on the embedding space fφ ( x ) through the conditional transformation fc . This is in contrast with a regular base-learner where ŷ=fθ ( x ) . In our framework , we use MAML to meta-learn fc and fθ . The metric space is pre-trained using distance-based loss function . 3.2 METRIC SPACE AS CONDITIONAL INFORMATION . We encode information of the label structure through fφ in the form of anM-dimensional metric space , where each input example is reduced to a point in the metric space . The goal of the metric learning step is to optimize parameter φ such that distances between examples in the metric space are semantically meaningful . Given the parameters of the metric space φ , which is represented by a convolutional network , we calculate a centroid ct for each class t , ct= 1 K ∑ ( xi , yi ) ∈Dtrain 1 { yi=t } fφ ( xi ) , whereK denotes the number of examples for class t , 1 { yi=t } denotes an indicator function of yi which takes value 1 when yi=t and 0 otherwise . The centroid ct is the sample mean among all instances from the same class which is treated as a prototype representation of the class t. The mapping function fφ is optimized to minimize the negative log-probability defined in Eq . ( 1 ) by minimizing the Euclidean distance d between an example and its corresponding class centroid ct while maximizing its Euclidean distance to other class centroids ct′ : argmin φ E [ d ( fφ ( xi ) , ct ) ) +log ∑ t′ exp ( −d ( fφ ( xi ) , ct′ ) ) ] . ( 1 ) In relation to prototypical networks ( Snell et al. , 2017 ) , we use the same loss function for metric learning . However , these frameworks differ in the test mode : we are not interested in example-centroid distances for label assignment , but rather in the projection fφ ( xi ) from the input space to the metric space that encapsulates inferred class regularities given the input example xi . In relation to other pre-training methods that use the meta-train classes to train a 64-way classifier , our use of the metric space imposes distance-based constraints to learn embeddings that follow semantically meaningful distance measures . We empirically find it difficult to optimize both the metric space and base-learner end-to-end . The metric space is pre-trained on the meta-train data and it is not updated during meta-learning . This also ensures the metric space is trained on a large number of classes to capture the global class dependencies . 3.3 CONDITIONALLY TRANSFORMED CONVOLUTIONAL BLOCK . We now turn to describing the conditionally transformed convolutional block , shown in Figure 2 , which uses the metric space described in Section 3.2 to inform the base-learner about the label structure of a task . The conditional transformation fc receives embeddings from the metric space and produces transformation operations to modulate convolutional representations of the base-learner fθ . Our conditional transformation has close relation to Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) that normalizes the input to every layer of a neural network . In order to conditionally modulate feature representations , we use Conditional Batch Normalization ( CBN ) ( Dumoulin et al. , 2017 ) to predict scale and shift operators from conditional input si : γ̂c=fc , γ ( si ) , β̂c=fc , β ( si ) , ( 2 ) where fc , γ and fc , β can be any differentiable function . This gives our model the flexibility to shift or scale the intermediate representations based on some source information in si . Since examples belonging to the same class are conceptually close , we exploit this inherent relationship in the metric space to modulate the feature maps at the example level in a way that encodes the label structure . Once we obtained the embedding function fφ , we use two auxiliary networks , learned end-to-end together with the meta-learner , to predict the shift and scale factors of the convolutional feature map : γ̂i , c=fc , γ ( fφ ( xi ) ) , β̂i , c=fc , β ( fφ ( xi ) ) . ( 3 ) Having computed γ̂i , c and β̂i , c , Conditional Batch Normalization ( CBN ) is applied as follows : CBN ( Ri , c|γ̂i , c , β̂i , c ) = γ̂i , c Ri , c−E [ Rc ] √ Var [ Rc ] + +β̂i , c , ( 4 ) where Ri , c refers to the cth feature map from the ith example , is a small constant , βc and γc are learnable parameters shared within a task . E [ Rc ] and Var [ Rc ] are batch mean and variance of Rc . It is worthwhile to note the effect of conditional transformation . The conditional bias transformation with β̂i , c is analogous to concatenation-based conditioning where the conditional information is concatenated to the feature maps ( Dumoulin et al. , 2018 ) . The conditional scaling factor provides multiplicative interactions between the metric space and the feature maps to aggregate information . Furthermore , the goal of the conditionally transformed convolutional block is to simultaneously capture the two views of a classification task : a global view that is aware of the relationships among all classes , and a local views of the current N-way K-shot classification task . The metric space , or the global view , is pre-trained in a way that is independent of the current N-way K-shot task ; while the base-learner , or the local view , attempts to develop representations for the current classification task . Although the metric space is never trained on the meta-test classes , we expect the learned metric space to generalize from the meta-train tasks to the meta-test tasks . We further describe parameter sharing for CBN learning in Section 3.3.1 , and class-aware grouping in Section 3.3.2 which provides more statistical strength for more effective few-shot learning .

Summary: This paper proposes a new few-shot learning method with class dependencies. To consider the structure in the label space, the authors propose to use conditional batch normalization to help change the embedding based on class-wise statistics. Based on which the final classifier can be learned by the gradient-based meta-learning method, i.e., MAML. Experiments on MiniImageNet show the proposed method can achieve high-performance, and the proposed part can be proved to be effective based on the ablation study.