Learning to Learn with Conditional Class Dependencies

1 INTRODUCTION . In machine learning , the objective of classification is to train a model to categorize inputs into various classes . We usually assume a categorical distribution over the label space , and thus effectively ignore dependencies among them . However , class structure does exist in real world and is also present in most datasets . Although class structure can be implicitly obtained as a by-product during learning , it is not commonly exploited in an explicit manner to develop better learning systems . The use of label structure might not be of prime importance when having access to huge amounts of data , such the full ImageNet dataset . However , in the case of few-shot learning where little data is available , meta-information such as dependencies in the label space can be crucial . In recent years , few-shot learning—learning from few examples across many tasks—has received considerable attention ( Ravi & Larochelle , 2016 ; Snell et al. , 2017 ; Finn et al. , 2017 ; Vinyals et al. , 2016 ) . In particular , the concept of meta-learning has been shown to provide effective tools for few-shot learning tasks . In contrast to common transfer learning methods that aim to fine-tune a pre-trained model , meta-learning systems are trained by being exposed to a large number of tasks and evaluated in their ability to learn new tasks effectively . In meta-training , learning happens at two levels : a meta-learner that learns across many tasks , and a base-learner that optimizes for each task . Model-Agnostic Meta-Learning ( MAML ) is a gradient-based meta-learning algorithm that provides a mechanism for rapid adaptation by optimizing only for the initial parameters of the base-learner ( Finn et al. , 2017 ) . Our motivation stems from a core challenge in gradient-based meta-learning , wherein the quality of gradient information is key to fast generalization : it is known that gradient-based optimization fails to converge adequately when trained from only a few examples ( Ravi & Larochelle , 2016 ) , hampering the effectiveness of gradient-based meta-learning techniques . We hypothesize that under such circumstances , introducing a metric space trained to encode regularities of the label structure can impose global class dependencies on the model . This class structure can then provide a high-level view of the input examples , in turn leading to learning more disentangled representations . We propose a meta-learning framework taking advantage of this class structure information , which is available in a number of applications . The Conditional class-Aware Meta-Learning ( CAML ) model is tasked with producing activations in a manner similar to a standard neural network , but with the additional flexibility to shift and scale those activations conditioned on some auxiliary meta-information . While there are no restrictions on the nature of the conditioning factor , in this work we model class dependencies by means of a metric space . We aim to learn a function mapping inputs to a metric space where semantic distances between instances follow an Euclidean geometry—classes that are semantically close lie in close proximity in an ` p sense . The goal of the conditional class-aware transformation is to make explicit use of the label structure to inform the model to reshape the representation landscape in a manner that incorporates a global sense of class structure . The contributions of this work are threefold : ( i ) We provide a meta-learning framework that makes use of structured class information in the form of a metric space to modulate representations in few-shot learning tasks ; ( ii ) We introduce class-aware grouping to improve the statistical strength of few-shot learning tasks ; ( iii ) We show experimentally that our proposed algorithm learns more disentangled representation and achieves competitive results on the miniImageNet benchmark . 2 BACKGROUND . We start by describing the meta-learning formulation proposed by Vinyals et al . ( 2016 ) and Ravi & Larochelle ( 2016 ) , and review MAML ( Finn et al. , 2017 ) , of which CAML is an instance . 2.1 META-LEARNING PROBLEM FORMULATION . The goal of meta-learning is to learn from a distribution of tasks . The learning happens on two levels : ( i ) a meta-level model , or meta-learner , that learns across many tasks , and ( ii ) a base-level model , or base-learner , that operates within each specific task . Meta-learning happens in task space , where each task can be treated as one meta-example . In the meta-learning formulation , we define a collection of regular tasks as meta-sets D , and each task D∈D has its own Dtrain and Dtest split . Dtrain is often denoted as the “ support set ” andDtest the “ query set ” . The resulting meta-learner objective is to choose parameters θ that minimize the expected loss L ( · ; θ ) across all tasks in D , θ∗=argminθED∼D [ L ( D ; θ ) ] . At the meta-level , the meta-sets D can be further split into disjoint meta-training set Dmeta−train , meta-validation set Dmeta−valid and meta-test set Dmeta−test . The meta-learner is trained on Dmeta−train , validated on Dmeta−valid and finally evaluated on Dmeta−test . 2.2 MODEL-AGNOSTIC META-LEARNING . Model-Agnostic Meta-Learning ( Finn et al. , 2017 ) is a meta-learning algorithm that aims to learn representations that encourage fast adaptation across different tasks . The meta-learner and base-learner share the same network structure , and the parameters learned by the meta-learner are used to initialize the base-learner on a new task . To optimize the meta-learner , we first sample a set of tasks { D1 , D2 , ... , DS } from the meta-training set Dmeta−train . For a meta-learner parameterized by θ , we compute its adapted parameters θi for each sampled task Di . The adapted parameters θi are task-specific and tell us the effectiveness of θ as to whether it can achieve generalization through one or a few additional gradient steps . The objective of the meta-learner is to optimize the representation θ such that it leads to good task-specific adaptations θi with only a few gradient steps . The meta-learner performs slow learning at the meta-level across many tasks to support fast learning on new tasks . At meta-test time , we initialize the base-learner with the meta-learned representation θ∗ followed by gradient-based fine-tuning . 3 METHOD . 3.1 CONDITIONAL CLASS-AWARE META-LEARNING . As shown in Figure 1 , the proposed Conditional class-Aware Meta-Learning ( CAML ) is composed of four components : an embedding function fφ that maps inputs to a metric space , a base-learner fθ that learns each individual task , an adaptation function fc that conditionally modulates the representations of the base-learner , and a meta-learner that learns across different tasks . Figure 1 depicts a toy illustration of the task inference procedure where examples from three classes are mapped onto a metric space using fφ , which are further used to modulate the base-learner fθ through a conditional transformation function fc . The main contribution of this paper is to incorporate metric-based conditional transformations ( fc ) into the meta-learning framework at the instance level . A notable feature of the proposed method is that the model has a global sense of the label space through the embedding function fφ by mapping examples onto the semantically meaningful metric space . The embeddings on the metric space inform the base-learner fθ about the label structure which in turn helps disentangle representations from different classes . This structured information can also provide a global view of the input examples to improve gradient-based meta-learning . In a simplistic form , our proposed model makes predictions using ŷ=fθ ( x ; fc ( fφ ( x ) ) ) , where the base-learner fθ is conditioned on the embedding space fφ ( x ) through the conditional transformation fc . This is in contrast with a regular base-learner where ŷ=fθ ( x ) . In our framework , we use MAML to meta-learn fc and fθ . The metric space is pre-trained using distance-based loss function . 3.2 METRIC SPACE AS CONDITIONAL INFORMATION . We encode information of the label structure through fφ in the form of anM-dimensional metric space , where each input example is reduced to a point in the metric space . The goal of the metric learning step is to optimize parameter φ such that distances between examples in the metric space are semantically meaningful . Given the parameters of the metric space φ , which is represented by a convolutional network , we calculate a centroid ct for each class t , ct= 1 K ∑ ( xi , yi ) ∈Dtrain 1 { yi=t } fφ ( xi ) , whereK denotes the number of examples for class t , 1 { yi=t } denotes an indicator function of yi which takes value 1 when yi=t and 0 otherwise . The centroid ct is the sample mean among all instances from the same class which is treated as a prototype representation of the class t. The mapping function fφ is optimized to minimize the negative log-probability defined in Eq . ( 1 ) by minimizing the Euclidean distance d between an example and its corresponding class centroid ct while maximizing its Euclidean distance to other class centroids ct′ : argmin φ E [ d ( fφ ( xi ) , ct ) ) +log ∑ t′ exp ( −d ( fφ ( xi ) , ct′ ) ) ] . ( 1 ) In relation to prototypical networks ( Snell et al. , 2017 ) , we use the same loss function for metric learning . However , these frameworks differ in the test mode : we are not interested in example-centroid distances for label assignment , but rather in the projection fφ ( xi ) from the input space to the metric space that encapsulates inferred class regularities given the input example xi . In relation to other pre-training methods that use the meta-train classes to train a 64-way classifier , our use of the metric space imposes distance-based constraints to learn embeddings that follow semantically meaningful distance measures . We empirically find it difficult to optimize both the metric space and base-learner end-to-end . The metric space is pre-trained on the meta-train data and it is not updated during meta-learning . This also ensures the metric space is trained on a large number of classes to capture the global class dependencies . 3.3 CONDITIONALLY TRANSFORMED CONVOLUTIONAL BLOCK . We now turn to describing the conditionally transformed convolutional block , shown in Figure 2 , which uses the metric space described in Section 3.2 to inform the base-learner about the label structure of a task . The conditional transformation fc receives embeddings from the metric space and produces transformation operations to modulate convolutional representations of the base-learner fθ . Our conditional transformation has close relation to Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) that normalizes the input to every layer of a neural network . In order to conditionally modulate feature representations , we use Conditional Batch Normalization ( CBN ) ( Dumoulin et al. , 2017 ) to predict scale and shift operators from conditional input si : γ̂c=fc , γ ( si ) , β̂c=fc , β ( si ) , ( 2 ) where fc , γ and fc , β can be any differentiable function . This gives our model the flexibility to shift or scale the intermediate representations based on some source information in si . Since examples belonging to the same class are conceptually close , we exploit this inherent relationship in the metric space to modulate the feature maps at the example level in a way that encodes the label structure . Once we obtained the embedding function fφ , we use two auxiliary networks , learned end-to-end together with the meta-learner , to predict the shift and scale factors of the convolutional feature map : γ̂i , c=fc , γ ( fφ ( xi ) ) , β̂i , c=fc , β ( fφ ( xi ) ) . ( 3 ) Having computed γ̂i , c and β̂i , c , Conditional Batch Normalization ( CBN ) is applied as follows : CBN ( Ri , c|γ̂i , c , β̂i , c ) = γ̂i , c Ri , c−E [ Rc ] √ Var [ Rc ] + +β̂i , c , ( 4 ) where Ri , c refers to the cth feature map from the ith example , is a small constant , βc and γc are learnable parameters shared within a task . E [ Rc ] and Var [ Rc ] are batch mean and variance of Rc . It is worthwhile to note the effect of conditional transformation . The conditional bias transformation with β̂i , c is analogous to concatenation-based conditioning where the conditional information is concatenated to the feature maps ( Dumoulin et al. , 2018 ) . The conditional scaling factor provides multiplicative interactions between the metric space and the feature maps to aggregate information . Furthermore , the goal of the conditionally transformed convolutional block is to simultaneously capture the two views of a classification task : a global view that is aware of the relationships among all classes , and a local views of the current N-way K-shot classification task . The metric space , or the global view , is pre-trained in a way that is independent of the current N-way K-shot task ; while the base-learner , or the local view , attempts to develop representations for the current classification task . Although the metric space is never trained on the meta-test classes , we expect the learned metric space to generalize from the meta-train tasks to the meta-test tasks . We further describe parameter sharing for CBN learning in Section 3.3.1 , and class-aware grouping in Section 3.3.2 which provides more statistical strength for more effective few-shot learning .

Summary: The paper presents an enhancement to the Model-Agnostic Meta-Learning (MAML) framework to integrate class dependency into the gradient-based meta-learning procedure. Specifically, the class dependency is encoded by embedding the training examples via a clustering network into a metric space where semantic similarity is preserved via affinity under Euclidean distance. Embedding of an example in this space is further employed to modulate (scale and shift) features of the example extracted by the base-learner via a transformation network, and the final prediction is made on top of the modulated features. Experiments on min-ImageNet shows that the proposed approach improves the baseline of MAML.