Few-Shot Attribute Learning

Semantic concepts are frequently defined by combinations of underlying attributes . As mappings from attributes to classes are often simple , attribute-based representations facilitate novel concept learning with zero or few examples . A significant limitation of existing attribute-based learning paradigms , such as zeroshot learning , is that the attributes are assumed to be known and fixed . In this work we study the rapid learning of attributes that were not previously labeled . Compared to standard few-shot learning of semantic classes , in which novel classes may be defined by attributes that were relevant at training time , learning new attributes imposes a stiffer challenge . We found that supervised learning with training attributes does not generalize well to new test attributes , whereas selfsupervised pre-training brings significant improvement . We further experimented with random splits of the attribute space and found that predictability of test attributes provides an informative estimate of a model ’ s generalization ability . 1 INTRODUCTION . Formation of class concepts is one of the most fundamental processes in machine perception . Although class concepts are often defined based on their attribute information , e.g. , birds are warmblooded vertebrates that lay eggs and have feathers , attributes are rarely considered in a typical machine perception system that directly maps from input signals to output classes . Humans also leverage similarity in the attribute space to recognize classes , which are “ information-rich bundles of attributes that form natural discontinuities ” ( Rosch & Mervis , 1975 ) . The acquisition of attribute knowledge therefore helps us build a more compact and efficient representation that is useful for the perception of classes . Another distinct advantage of using attribute information is that it facilitates learning new classes with few or even zero examples , which has been leveraged in studies of zero-shot learning ( ZSL ) ( Palatucci et al. , 2009 ; Lampert et al. , 2014 ; Misra et al. , 2017 ) . Other models use attributes as direct outputs before the classes for improved modularity and interpretability ( Farhadi et al. , 2009 ; Koh et al. , 2020 ) . However , all of these attribute-based models rely on a pre-defined set of attributes , that are shared among all classes . Consider a learning agent deployed in the wild . Although the agent may learn new classes by composing some of the existing attributes , its learning capability would be greatly improved if it can expand its attribute vocabulary . Motivated by this learning scenario , we are interested in the problem of learning new attributes that are previously not labeled in the dataset . This is a step towards continual learning ( Van de Ven & Tolias , 2019 ) , where the reward function evolves and the system must adapt if the reward becomes dependent on previously irrelevant input attributes . In particular , we focus on a few-shot learning ( FSL ) setup ( Lake et al. , 2011 ; Vinyals et al. , 2016 ) , where only a few positive and negative examples of the target attributes are available , to model the rapid adaptation task . Considering few-shot attribute learning has an extra benefit , in that it can provide manipulable factors to study generalization . In standard few-shot learning , object semantic classes are split into training and test ; however , there is still a lack of understanding of when models transfer their knowledge from training classes to test ones . Since semantic classes can often be defined with a set of attributes , a split in the attribute space therefore provides us a finer control on the degree to which training classes are related to test ones . By studying the transfer performance on novel attributes , we expect our work can generate insight into the generalization performance on semantic classes in standard few-shot learning . To study this challenging task of few-shot attribute learning ( FSAL ) , we contribute new benchmark datasets consisting of images of faces ( Celeb-A ) ( Liu et al. , 2015 ) , shoes ( Zappos50K ) ( Yu & Grauman , 2014 ) , and general objects ( ImageNet-with-Attributes ) ( Deng et al. , 2009 ) . Unlike in standard few-shot learning where supervised pre-training generally helps learning , surprisingly we found that directly supervising the model with a set of training attributes does not generalize well on the test attributes , whereas self-supervised pre-training brings significant improvement . We further ran experiments with random splits of the attribute space and discovered that the predictability of attributes provides an informative estimate of a model ’ s ability to generalize . The few-shot attribute learning paradigm proposed in this paper will facilitate more efficient and flexible continual learning and shed a light on the practical understanding on generalization of novel concepts . 2 FEW-SHOT ATTRIBUTE LEARNING . In this section , we define our few-shot attribute learning ( FSAL ) paradigm and highlight the additional challenges of FSAL compared to the standard few-shot learning of semantic classes . Similar to standard few-shot learning , at test time , the learner is presented with an episode of data . The support set consists of N positive and negative examples of the target attributes S = { ( xS+1 , 1 ) , . . . , ( x S+ N , 1 ) , ( x S− 1 , 0 ) , . . . , ( x S− N , 0 ) } , ( 1 ) where the + or − superscript suffix denotes whether the input is a positive or negative example . After rapid learning on the support set , the model is then evaluated on the binary classification performance of the query set : Q = { ( xQ+1 , 1 ) , . . . , ( x Q+ M , 1 ) , ( x Q− 1 , 0 ) , . . . , ( x Q− M , 0 ) } . ( 2 ) At test time , the target binary label may concern a novel attribute that was previously unlabeled in the training set . For example , in one test episode , a smiling face with eyeglasses is positively labeled alongside other faces with eyeglasses . The task here is to learn the attribute of “ wearing eyeglasses ” . However , while the learner might have seen training images with eyeglasses , it was never a relevant feature for the purpose of predicting the positively labeled instances . Furthermore , suppose that in another test episode , the same smiling face is positively labeled alongside other smiling faces . The target attribute here has now changed from “ wearing eyeglasses ” to “ smiling. ” This highlights a critical difference between few-shot attribute learning and standard few-shot learning of semantic classes : in standard FSL , each instance can belong to only one class regardless of the episode . In FSAL , due to the multi-label nature of the attribute space , one instance could have different labels depending on the context of the support set examples . Furthermore , there may be a large amount of ambiguity when the support set is small . Figure 1 shows a few examples of our attribute learning episodes . Note that in order to create task diversity , we allow both unary and binary attributes , where binary attributes are conjunctions of two unary attributes . In order to solve the FSAL task , the learner must correctly determine the context . Just like in zero-shot learning , one natural way to solve this problem would be to learn to predict the underlying attributes of each image . Given the attributes , you could then estimate the context in each episode ( Lampert et al. , 2014 ) . However , methods that accurately predict attributes relevant to training episodes may not generalize well , since at test time FSAL introduces novel attributes . Instead , we explore methods that allow more general representations to be learned . 3 METHODOLOGY . In this section , we present our proposed approach to tackle the problem of few-shot attribute learning . Two classes of approaches are frequently considered in standard few-shot learning . First , episodic approaches train methods directly from a set of few-shot episodes . This class of methods can be naturally applied to our learning setting . Second , pre-training approaches train a network to directly classify a set of training classes , and at test time , the embedding network is transferred to solve the test task by training another classifier on top . If absolute attribute IDs are provided to the learner , then one natural approach is to instead train an attribute classifier . After the attribute classifier network is learned , we can then transfer the representations to recognize test attributes . We explored both of these approaches in our experiments , but we found that they tend to learn good representations only for the training attributes but not for test ones . To address the generalization issues of these standard approaches , we propose a new algorithm that preserves more general features . Our proposed method follows the pre-training paradigm and contains three stages . The first stage is pre-training the representation network using an unsupervised ( contrastive ) learning objective . The second stage is fine-tuning using supervision based either on attribute or episode information . Lastly , we learn a sparse linear classifier at test-time to solve a new episode . We describe each stage of learning below . 3.1 STAGE I : UNSUPERVISED REPRESENTATION PRE-TRAINING . Our proposed few-shot attribute learning tasks require a strongly generalizable learner . We hypothesized that learning general-purpose features that capture varying aspects of objects would be helpful to enable this desired capability . We therefore considered unsupervised representation learning , and we hypothesize that it can learn general semantic features . We chose SimCLR ( Chen et al. , 2020 ) as a representative from this category due to its empirical success . In general , contrastive learning approaches aim to build invariant representations between a pair of inputs { x , x′ } that are produced by applying random data augmentations ( e.g . cropping ) to an input image . It is likely to preserve more general semantic features since all image attributes are useful towards identifying another random crop of the same image . We first obtain the embedding output h from the CNN , and then following SimCLR , we project h to z using a multi-layered perceptron ( MLP ) : h = CNN ( x ) , z = MLP1 ( h ) . With a batch of image pairs denoted by { xi } , { x′i } , we can obtain their features { zi } , { z′i } , and the contrastive loss function is defined similar to the cross entropy function : L1 = − ∑ i log exp ( zi · z′i/τ ) ∑ j exp ( zi · z′j/τ ) , ( 3 ) where τ is an extra temperature hyperparameter . 3.2 STAGE II : REPRESENTATION FINE-TUNING . In the second stage , we aim to utilize some labels from the training set to supervise the network . We consider using two different modes of supervision : 1 ) the FSAL binary episodic labels , or 2 ) the underlying binary attribute IDs . To prevent overwriting the representations and making them overly sensitive to training attributes , we add another projection MLP that learns more specific representations for finetuning on training attributes : g = MLP2 ( h ) . The fine-tuning objectives we consider are as follows . • Unsupervised-then-FineTune-on-Episodes ( UFTE ) . We adopt the Prototypical Networks ( Snell et al. , 2017 ) formulation , where the network solves a learning episode of N positive and negative support examples by using prototypes p : p+ = 1N ∑ i g + i ; p − = 1N ∑ i g − i . With query example gq , we can make a binary prediction : ŷq = exp ( −d ( gq , p+ ) ) exp ( −d ( gq , p+ ) ) + exp ( −d ( gq , p− ) ) , ( 4 ) where d is some dissimilarity score , e.g . Euclidean distance or cosine dissimilarity , and the training objective is to minimize the classification loss between the prediction ŷq and the label yq : L2E = ∑ j −yj log ŷqj − ( 1− y q j ) log ( 1− ŷ q j ) , ( 5 ) where j is the index of query examples . • Unsupervised-then-FineTune-on-Attributes ( UFTA ) . With persistent attribute information , we can train a linear classifier with sigmoid activation to directly predict the absolute attribute labels a : â =WAg + bA , with the loss being L2A = ∑ k −ak log âk − ( 1− ak ) log ( 1− âk ) , ( 6 ) where k is the index of attributes .

Summary: The article proposes to address attribute detection with a small data constraint in the training phase (few-shot approach). The experimental protocol is divided in a series of episodes, each defined by a support set that specifies the attributes to be predicted and a query set used to evaluate the prediction. The generic algorithmic structure to solve this problem is divided in two steps: a common pre-training phase that combines self-supervised representation learning and supervised fine-tuning, followed by a supervised learning phase to solve each episode task. A training dataset containing image samples with attributes is used to learn the pre-trained representation space. The pre-training phase is compared to other approaches on three benchmarks implementing few-shot attribute prediction episodes. An indicator computed from a logistic regression between attribute values is proposed as a way to predict transferability between the learning dataset and the few-shot problem.