Few-Shot Attribute Learning

Semantic concepts are frequently defined by combinations of underlying attributes . As mappings from attributes to classes are often simple , attribute-based representations facilitate novel concept learning with zero or few examples . A significant limitation of existing attribute-based learning paradigms , such as zeroshot learning , is that the attributes are assumed to be known and fixed . In this work we study the rapid learning of attributes that were not previously labeled . Compared to standard few-shot learning of semantic classes , in which novel classes may be defined by attributes that were relevant at training time , learning new attributes imposes a stiffer challenge . We found that supervised learning with training attributes does not generalize well to new test attributes , whereas selfsupervised pre-training brings significant improvement . We further experimented with random splits of the attribute space and found that predictability of test attributes provides an informative estimate of a model ’ s generalization ability . 1 INTRODUCTION . Formation of class concepts is one of the most fundamental processes in machine perception . Although class concepts are often defined based on their attribute information , e.g. , birds are warmblooded vertebrates that lay eggs and have feathers , attributes are rarely considered in a typical machine perception system that directly maps from input signals to output classes . Humans also leverage similarity in the attribute space to recognize classes , which are “ information-rich bundles of attributes that form natural discontinuities ” ( Rosch & Mervis , 1975 ) . The acquisition of attribute knowledge therefore helps us build a more compact and efficient representation that is useful for the perception of classes . Another distinct advantage of using attribute information is that it facilitates learning new classes with few or even zero examples , which has been leveraged in studies of zero-shot learning ( ZSL ) ( Palatucci et al. , 2009 ; Lampert et al. , 2014 ; Misra et al. , 2017 ) . Other models use attributes as direct outputs before the classes for improved modularity and interpretability ( Farhadi et al. , 2009 ; Koh et al. , 2020 ) . However , all of these attribute-based models rely on a pre-defined set of attributes , that are shared among all classes . Consider a learning agent deployed in the wild . Although the agent may learn new classes by composing some of the existing attributes , its learning capability would be greatly improved if it can expand its attribute vocabulary . Motivated by this learning scenario , we are interested in the problem of learning new attributes that are previously not labeled in the dataset . This is a step towards continual learning ( Van de Ven & Tolias , 2019 ) , where the reward function evolves and the system must adapt if the reward becomes dependent on previously irrelevant input attributes . In particular , we focus on a few-shot learning ( FSL ) setup ( Lake et al. , 2011 ; Vinyals et al. , 2016 ) , where only a few positive and negative examples of the target attributes are available , to model the rapid adaptation task . Considering few-shot attribute learning has an extra benefit , in that it can provide manipulable factors to study generalization . In standard few-shot learning , object semantic classes are split into training and test ; however , there is still a lack of understanding of when models transfer their knowledge from training classes to test ones . Since semantic classes can often be defined with a set of attributes , a split in the attribute space therefore provides us a finer control on the degree to which training classes are related to test ones . By studying the transfer performance on novel attributes , we expect our work can generate insight into the generalization performance on semantic classes in standard few-shot learning . To study this challenging task of few-shot attribute learning ( FSAL ) , we contribute new benchmark datasets consisting of images of faces ( Celeb-A ) ( Liu et al. , 2015 ) , shoes ( Zappos50K ) ( Yu & Grauman , 2014 ) , and general objects ( ImageNet-with-Attributes ) ( Deng et al. , 2009 ) . Unlike in standard few-shot learning where supervised pre-training generally helps learning , surprisingly we found that directly supervising the model with a set of training attributes does not generalize well on the test attributes , whereas self-supervised pre-training brings significant improvement . We further ran experiments with random splits of the attribute space and discovered that the predictability of attributes provides an informative estimate of a model ’ s ability to generalize . The few-shot attribute learning paradigm proposed in this paper will facilitate more efficient and flexible continual learning and shed a light on the practical understanding on generalization of novel concepts . 2 FEW-SHOT ATTRIBUTE LEARNING . In this section , we define our few-shot attribute learning ( FSAL ) paradigm and highlight the additional challenges of FSAL compared to the standard few-shot learning of semantic classes . Similar to standard few-shot learning , at test time , the learner is presented with an episode of data . The support set consists of N positive and negative examples of the target attributes S = { ( xS+1 , 1 ) , . . . , ( x S+ N , 1 ) , ( x S− 1 , 0 ) , . . . , ( x S− N , 0 ) } , ( 1 ) where the + or − superscript suffix denotes whether the input is a positive or negative example . After rapid learning on the support set , the model is then evaluated on the binary classification performance of the query set : Q = { ( xQ+1 , 1 ) , . . . , ( x Q+ M , 1 ) , ( x Q− 1 , 0 ) , . . . , ( x Q− M , 0 ) } . ( 2 ) At test time , the target binary label may concern a novel attribute that was previously unlabeled in the training set . For example , in one test episode , a smiling face with eyeglasses is positively labeled alongside other faces with eyeglasses . The task here is to learn the attribute of “ wearing eyeglasses ” . However , while the learner might have seen training images with eyeglasses , it was never a relevant feature for the purpose of predicting the positively labeled instances . Furthermore , suppose that in another test episode , the same smiling face is positively labeled alongside other smiling faces . The target attribute here has now changed from “ wearing eyeglasses ” to “ smiling. ” This highlights a critical difference between few-shot attribute learning and standard few-shot learning of semantic classes : in standard FSL , each instance can belong to only one class regardless of the episode . In FSAL , due to the multi-label nature of the attribute space , one instance could have different labels depending on the context of the support set examples . Furthermore , there may be a large amount of ambiguity when the support set is small . Figure 1 shows a few examples of our attribute learning episodes . Note that in order to create task diversity , we allow both unary and binary attributes , where binary attributes are conjunctions of two unary attributes . In order to solve the FSAL task , the learner must correctly determine the context . Just like in zero-shot learning , one natural way to solve this problem would be to learn to predict the underlying attributes of each image . Given the attributes , you could then estimate the context in each episode ( Lampert et al. , 2014 ) . However , methods that accurately predict attributes relevant to training episodes may not generalize well , since at test time FSAL introduces novel attributes . Instead , we explore methods that allow more general representations to be learned . 3 METHODOLOGY . In this section , we present our proposed approach to tackle the problem of few-shot attribute learning . Two classes of approaches are frequently considered in standard few-shot learning . First , episodic approaches train methods directly from a set of few-shot episodes . This class of methods can be naturally applied to our learning setting . Second , pre-training approaches train a network to directly classify a set of training classes , and at test time , the embedding network is transferred to solve the test task by training another classifier on top . If absolute attribute IDs are provided to the learner , then one natural approach is to instead train an attribute classifier . After the attribute classifier network is learned , we can then transfer the representations to recognize test attributes . We explored both of these approaches in our experiments , but we found that they tend to learn good representations only for the training attributes but not for test ones . To address the generalization issues of these standard approaches , we propose a new algorithm that preserves more general features . Our proposed method follows the pre-training paradigm and contains three stages . The first stage is pre-training the representation network using an unsupervised ( contrastive ) learning objective . The second stage is fine-tuning using supervision based either on attribute or episode information . Lastly , we learn a sparse linear classifier at test-time to solve a new episode . We describe each stage of learning below . 3.1 STAGE I : UNSUPERVISED REPRESENTATION PRE-TRAINING . Our proposed few-shot attribute learning tasks require a strongly generalizable learner . We hypothesized that learning general-purpose features that capture varying aspects of objects would be helpful to enable this desired capability . We therefore considered unsupervised representation learning , and we hypothesize that it can learn general semantic features . We chose SimCLR ( Chen et al. , 2020 ) as a representative from this category due to its empirical success . In general , contrastive learning approaches aim to build invariant representations between a pair of inputs { x , x′ } that are produced by applying random data augmentations ( e.g . cropping ) to an input image . It is likely to preserve more general semantic features since all image attributes are useful towards identifying another random crop of the same image . We first obtain the embedding output h from the CNN , and then following SimCLR , we project h to z using a multi-layered perceptron ( MLP ) : h = CNN ( x ) , z = MLP1 ( h ) . With a batch of image pairs denoted by { xi } , { x′i } , we can obtain their features { zi } , { z′i } , and the contrastive loss function is defined similar to the cross entropy function : L1 = − ∑ i log exp ( zi · z′i/τ ) ∑ j exp ( zi · z′j/τ ) , ( 3 ) where τ is an extra temperature hyperparameter . 3.2 STAGE II : REPRESENTATION FINE-TUNING . In the second stage , we aim to utilize some labels from the training set to supervise the network . We consider using two different modes of supervision : 1 ) the FSAL binary episodic labels , or 2 ) the underlying binary attribute IDs . To prevent overwriting the representations and making them overly sensitive to training attributes , we add another projection MLP that learns more specific representations for finetuning on training attributes : g = MLP2 ( h ) . The fine-tuning objectives we consider are as follows . • Unsupervised-then-FineTune-on-Episodes ( UFTE ) . We adopt the Prototypical Networks ( Snell et al. , 2017 ) formulation , where the network solves a learning episode of N positive and negative support examples by using prototypes p : p+ = 1N ∑ i g + i ; p − = 1N ∑ i g − i . With query example gq , we can make a binary prediction : ŷq = exp ( −d ( gq , p+ ) ) exp ( −d ( gq , p+ ) ) + exp ( −d ( gq , p− ) ) , ( 4 ) where d is some dissimilarity score , e.g . Euclidean distance or cosine dissimilarity , and the training objective is to minimize the classification loss between the prediction ŷq and the label yq : L2E = ∑ j −yj log ŷqj − ( 1− y q j ) log ( 1− ŷ q j ) , ( 5 ) where j is the index of query examples . • Unsupervised-then-FineTune-on-Attributes ( UFTA ) . With persistent attribute information , we can train a linear classifier with sigmoid activation to directly predict the absolute attribute labels a : â =WAg + bA , with the loss being L2A = ∑ k −ak log âk − ( 1− ak ) log ( 1− âk ) , ( 6 ) where k is the index of attributes .

Summary: This paper proposes the problem of few-shot attribute learning (FSAL) that follows the standard few-shot/meta-learning paradigm but focuses on attributes instead of object classes. The authors argue in Section 2 that the multi-label nature of attributes (smiling & wearing eyeglasses can be present simultaneously) makes this problem different; the context of each positive example in each episode can drastically change what the model is asked to learn.   This paper also proposes three benchmark datasets to study this problem, based on Celeb-A, Zappos50K, and ImageNet-with-Attributes. The paper evaluates multiple existing approaches on these benchmarks as well as improves upon them further.