Few-Shot Attribute Learning
Authors: Mengye Ren
Eleni Triantafillou
Kuan-Chieh Wang
James Lucas
Jake Snell
Xaq Pitkow
Andreas S. Tolias
Richard Zemel
Venue: ICLR 2022 Submitted (2022)
Keywords: Few-shot learning
transfer learning
attribute learning

Abstract: Semantic concepts are often defined by a combination of attributes. The use of attributes also facilitates learning of new concepts with zero or few examples. However, the zero-shot learning paradigm assumes that the set of attributes are known and fixed, which is a limitation if a test-time task depends on a previously irrelevant attribute. In this work we study rapid learning of attributes that are previously not labeled in the dataset. Compared to standard few-shot learning of semantic classes, learning new attributes imposes a stiffer challenge. We found that directly supervising the model with a set of training attributes does not generalize well on the test attributes, whereas self-supervised pre-training brings significant improvement. We further experimented with random splits of the attribute space and found that the predictability of attributes provides an informative estimate of a model's ability to generalize.

Under review as a conference paper at ICLR 2022
FEW-SHOT ATTRIBUTE LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Semantic concepts are frequently defined by combinations of underlying at- tributes. As mappings from attributes to classes are often simple, stribute-based representations facilitate novel concept learning with zero or few examples. significant limitation of existing attribute-based learning paradigms, such as zero- shot learning, is that the attributes are assumed to known and fixed. In this work we study the rapid learning of attributes that were not previously labeled. Com- pared to standard few-shot learning of semantic classes, in which novel classes may be defined by attributes that were relevant at training time, learning new attributes imposes a stiffer challenge. We found that supervised learning with training attributes does not generalize well to new test attributes, whereas self- supervised pre-training brings significant improvement. We further experimented with random splits of the attribute space and found that predictability of test at- tributes provides an informative estimate of model's generalization ability.
INTRODUCTION
Formation of class concepts is one of the most fundamental processes in machine perception. Al- though class concepts are often defined based on their attribute information, e.g., birds are warm- blooded vertebrates that lay eggs and have feathers, attributes are rarely considered in a typical machine perception system that directly maps from input signals to output classes. Humans also leverage similarity in the attribute space to recognize classes, which are "information-rich bundles of attributes that form natural discontinuities" (Rosch & Mervis, 1975). The acquisition of attribute knowledge therefore helps us build more compact and efficient representation that is useful for the perception of classes.
Another distinct advantage of using attribute information is that it facilitates learning new classes with few or even zero examples, which has been leveraged in studies of zero-shot learning (ZSL) (Palatucci et al., 2009; Lampert et al., 2014; Misra et al., 2017). Other models use attributes as direct outputs before the classes for improved modularity and interpretability (Farhadi et al., 2009; Koh et al., 2020). However, all of these attribute- baste models rely on pre-defined set of attributes, that are shared among all classes. Consider learning agent deployed in the wild. Although the agent may learn new classes by composing some of the existing attributes, its learning capability would be greatly improved if it can expand its attribute vocabulary.
Motivated by this learning scenario, are interested the problem of learning new attributes that are previously not labeled in the dataset. This is step towards continual learning (Van de Ven & Tolias, 2019), where the reward function evolves and the system must adapt if the reward becomes dependent on previously irrelevant input attributes. In particular, we focus on a few-shot learning (FSL) setup (Lake al., 2011: I; Vinyals al., 2016), where only few positive and negative examples of the target attributes are available, model the rapid adaptation task.
Considering few-shot attribute learning has an extra benefit, in that it can provide manipulable fac- tors to study generalization. In standard few-shot learning, object semantic classes are split into training and test; however, there is still a lack of understanding when models transfer their knowl- edge from training classes to test ones. Since semantic classes can often be defined with set of attributes, a split in the attribute space therefore provides us a finer control on the degree to which training classes are related to test ones. By studying the transfer performance on novel attributes, we expect our work can generate insight into the generalization performance on semantic classes in standard fho-shot learning.

---

Under review as a conference paper at ICLR 2022
Celeb-A A
Zappos-50K
Attribucc: Male Smilling
Artritutte: Cheekhoness Farrings
Att Womon Laco Up
Attributc popces & On
examples
eamples
Nagat diamples
Nogative snamples
Negativ cxamples
Negolive camp IS
Figure I:
Sample FSAL episodes using Celeb-A (left) and Zappos-50K (right), Positive and negative ex- amples are sampled according to attributes.
To study this challenging task of few-shot attribute learning (FSAL), we contribute new benchmark datasets consisting of images of faces (Celeb-A) (Liu al., 2015), shoes (Zappos50K) (Yu & Grau- man, 2014), and general objects (ImageNet-with- -it--ttttt---triuteees (Deng et al., 2009). Unlike in standard few-shot learning where supervised pre-training generally helps learning, surprisingly we found that directly supervising the model with set of training attributes does not generalize well on the test attributes, whereas self-supervised pre-training brings significant improvement. We further ran ex- periments with random splits of the attribute space and discovered that the predictability of attributes provides an informative estimate of a model's s ability to generalize. The few-shot attribute learning paradigm proposed in this paper will facilitate more efficient and flexible continual learning and shed a light on the practical understanding on generalization of novel concepts.
FEW-SHOT ATTRIBUTE LEARNING
In this section, we define our few-shot attribute learning (FSAL) paradigm and highlight the addi- tional challenges of FSAL compared to the standard few-shot learning of semantic classes.
Similar to standard few-shot learning, at test time, the learner is presented with an episode of data. The support set consists of N positive and negative examples of the target attributes
S {(x{+ 1), ......... (xx* 1), (xf= .),,, ...,..., (xx=, ,));
(1)
where the + or - superscript suffix denotes whether the input is a positive or negative example. After rapid learning on the support set, the model is then evaluated on the binary classification performance of the query set:
Q = (((pt 1,,,.,,, (x (x (x .x)....,
(x90 .0).
(2)
At test time, the target binary label may concern a novel attribute that was previously unlabeled in the training set. For example, in one test episode, smiling face with eyeglasses is positively labeled alongside other faces with eyeglasses. The task here to learn the attribute of "wearing eyeglasses". However, while the learner might have seen training images with eyeglasses, it was never a relevant feature for the purpose of predicting the positively labeled instances.
Furthermore, suppose that in another test episode, the same smiling face is positively labeled along- side other smiling faces. The target attribute here has now changed from "wearing eyeglasses" to "smiling." This highlights critical difference between few-shot attribute learning and standard few-shot learning of semantic classes: in standard FSL, each instance can belong to only one class regardless of the episode. In FSAL, due to the multi-label nature of the attribute space, one instance could have different labels depending on the context of the support set examples. Furthermore, there may be large amount of ambiguity when the support set is small. Figure/ 1 shows a few examples of our attribute learning episodes. Note that in order to create task diversity, we allow both unary and binary attributes, where binary attributes are conjunctions of two unary attributes.
In order to solve the FSAL task, the learner must correctly determine the context. Just like in zero-shot learning, one natural way solve this problem would be learn to predict the under- lying attributes of each image. Given the attributes, you could then estimate the context in each episode (Lampert et al., 2014). However, methods that accurately predict attributes relevant train- ing episodes may not generalize well, since at test time FSAL introduces novel attributes. Instead, we explore methods that allow more general representations to be learned.
METHODOLOGY
In this section, we present our proposed approach to tackle the problem of few-shot attribute learn- ing. Two classes of approaches are frequently considered in standard few-shot learning. First,
2

---

Under review as a conference paper at ICLR 2022
episodic approaches train methods directly from set of few-shot episodes. This class of methods can be naturally applied to our learning setting. Second, pre-training approaches train a network to directly classify set of training classes, and at test time, the embedding network is transferred to solve the test task by training another classifier on top. If absolute attribute IDs are provided to the learner, then one natural approach is to instead train attribute classifier. After the attribute classifier network is learned, we can then transfer the representations to recognize test attributes.
We explored both of these approaches in our experiments. but found that they tend to learn good representations only for the training attributes but not for test ones. To address the generalization is- sues of these standard approaches, we propose a new algorithm that preserves more general features.
Our proposed method follows the pre-training paradigm and contains three stages. The first stage pre-training the representation network using an unsupervised (contrastive) learning objective. The second stage is fine-tuning using supervision based either on attribute or episode information. Lastly, we learn sparse linear classifier at tis-timm to solve new episode. We describe each stage of learning below.
3.1 STAGE I: UNSUPERVISED REPRESENTATION PRE-TRAINING
Our proposed few-shot attribute learning tasks require a strongly generalizable learner. We hypothe- sized that learning purposs features that capture varying aspects of objects would be helpful to enable this desired capability. We therefore considered unsupervised representation learning, and we hypothesize that can learn general semantic features. We chose SimCLR (Chen et al., 2020) as representative from this category due to its empirical success. In general, contrastive learning ap- proaches aim to build invariant representations between a pair of inputs (x, x') that are produced by applying random data augmentations (e.g. cropping) to input image. It is likely to preserve more general semantic features since all image attributes are useful towards identifying another random crop of the same image.
We first obtain the embedding output h from the CNN, and then following SimCLR, we project h to z using multi-layered perceptron (MLP): h CNN(x), z MLP (h). With a batch of image pairs denoted by { {x]}, we can obtain their features {z:}, [z]], and the contrastive loss function is defined similar to the cross entropy function:
exp(z; 2//T)
==
Σ log Σ3 exp(z; z 2//T)
(3)
where T is an extra temperature hyperparameter.
3.2 STAGE II: REPRESENTATION -UNNNTGNNGG
In the second stage, aim utilize some labels from the training supervise the network. We consider using two different modes of supervision: 1) the FSAL binary episodic labels, or 2) the underlying binary attribute IDs. To prevent overwriting the representations and making them overly sensitive to training attributes, we add another projection MLP that learns more specific representations for finetuning on training attributes: g MLP2(h). The fine-tuning objectives we consider are as follows.
Unsupervised-theee -waee--iinTnnnn---eee al., 2017) formulation, where the network solves learning episode of N positive and negative support examples by using prototypes p: p n g::p g", we can make binary prediction:
(UFTE). We adopt the Prototypical Networks (Snell
E, With query example
exp( -(g p g- exp( (pp(-dg''.tt)) + exp( (gg)
(4)
where is some dissimilarity score, e.g. Euclidean distance cosine dissimilarity, and the training objective is to minimize the classification loss between the prediction j and the label y":
LE = Σ -リ log 0/ (1 - 33) log(1 - 03))
(5)
where is the index of query examples.
3

---

Under review as a conference paper at ICLR 2022
Paradigm
Test time task
Task specification
ZSL (Lampert ct al., 2014)
Novel semantic classes of cxisting attributes
Attribute IDs
CZSL (Misra ct al., 2017)
Novel combinations of existing attributes & classes
Attribute IDs
FSL (Lake et al., 2011)
Novel semantic classes
Support examples
FSAL (Ours)
Novel (previously unlabcled) attributes
Support examples
Table I:
Differences between zero-shot learning (ZSL), compositional ZSL (CZSL), few-shot learning (FSL), and our newly proposed few-shot attribute learning (FSAL). Our task requires the model to gener- alize new attributes.
Unsupervised-thent TimeeTTTn-on--ttriiiiii (UFTA). With persistent attribute information, we can train a linear classifier with sigmoid activation to directly predict the absolute attribute labels a: â = WAg + ba. with the loss being
C2A Σ
ag log åk (1 av.) log(1 ax),
(6)
K
where l is the index of attributes.
3.3 STAGE III: FEW-SHOT LEARNING
Once representations are learned, it remains to be decided how to use the small support set of each given test episode in order to make predictions for the associated query set. We consider three al- ternative approaches. MatchingNet (Vinyals et al., 2016) uses a nearest neighbor (NN) classifier, whereas ProtoNet (Snell et al., 2017) uses the nearest centroid (NC). Following Chen et al. (2019), propose directly learn logistic regression (L.R) top of the representation. This approach learns a weight coefficient for each feature dimension, thus performing some level of feature selec- tion, unlike the NC or NN variants. Still, the weights need be properly regularized to encourage high-fidelity selection. This is important since each episode only focuses on a new classification criteria using a subset of features. For this, we apply an L1 regularizer on the weights to encourage sparsity. In this way, the learning of a classifier is essentially done at the same time as the selection of feature dimensions. The overall objective of the classifier is:
arg argmin min (og(()) () (1 y) log(1 ú) A/|w/|1; w.b
where و - (w h + b). and h is the representation vector extracted from the CNN backbone. Note that in this stage we discard the projection MIPs that are defined in previous stages since they are trained towards training attributes and we found that they do not transfer well to novel attributes.
4 RELATED WORK
Few-shot learning (FSL) (Fei-Fei et al., 2006; Lake et al., 2011; Vinyals et al., 2016) entails learning new tasks with only few examples. With an abundance of training data, FSL is closely related to the general meta-learning or learning to learn paradigm (Thrun, 1998), as a few-shot learning algorithm can be developed In standard few-shot classification, each image only has a single unambiguous class label, whereas in our few-shot attribute learning, the target attributes can vary depending on how the support set is presented. We show in this paper that this is more challenging problem as it requires the model to be more flexible and generalizable. In early benchmarks, a set of semantic classes was randomly split into training and test set. We hypothesize that this often leads to common set of attributes that span (most of) the training and test classes, thus causing high transferability between these two sets, which allows simple solutions based on feature re-use (Chen et al., 2019; Raghu et al., 2020) to work well. Later benchmarks explicitly attempt to vary the separation between train and test classes, based on varying the distances in the underlying WordNer classes (fiered-ImageNNt (Ren et al., 2018)), or in different image domains "Meta-Datase (Triantafillou et al., 2020)). However, argue that reasoning about the underlying attributes directly offers more systematic framework to measure the relatedness and transferability between the train and test set. We expect our analysis to open the door to such studies in the future. Few-shot attribute learning is also related to multi- label few-shot learning (Alfassy et al., 2019; Li et al., 2021). These prior works emphasize on the compositional aspect, whereas we propose models that address the transferability of the learned representations. Additionally, Xiang et al. (2019) explored combining incremental few-shot learning and attribute learning for pedestrian images.
Few-shot learning:
training tasks and run novel tasks at test time.
Attribute learning: In the past, there have been a number of works that aim to predict attribute information from raw inputs Ferrari & Zisserman (2007); Farhadi et al. (2009; 2010); Wang &
4

---

Under review as a conference paper at ICLR 2022
Celeb-A
Zappos-50 500K
Sup.
5-shot
20-shot
5-shot
20-shot
Chance
50.00..0.00
50.00:00.00
50.00
50.00::..00
NetchingNet
68. 06
71.73=0.52
77.26=00:0
80,47:0.19
MAML/ANIL
E
71.24:10.74
73.35=00.53
77.05=0.50
81.10:0.43
TAFENet
E
69. 00
72.11=
79. 0:750:57
83,42::0.4
ProtoNet
E
12.72
75.27=00.51
77 51
83, 83,42++0,41
TADAM
73.54
76.06:10.53
81.45
86.23::0.40
C
69.95
77.53-10.53
U
73.47:1066
79.97=10.51
83.88:20444
90.92:40.32
UFTE (Ours)
76.69 1.0.6
82.83-10.48
85,50
02.20:1:0.3
SA
72.71
78.86:1048
82.17:10.48
88.24:10.37
UFTA (Ours)
78.98::0.09
84.14
84.61 +0.43
91.66:10.29
Oracles
SA
84.74
89.1520.35
88.11=0.59
93.00-1..8
GT
91.07
98. 98.1671.11
97.66=0.116
99.84+t0.04
Table 2: 5- and 20-shot attribute learning results on Celeb-A and 50ppos-50K. We compare standard FSL methods to variants of our approach. Representation learning based on class identity or attribute labels are out- performed by unsupervised methods that fine-tune using attribute episode labels. Methods can be supervised by 1) "ep=iddd binary labels, "A "aattribbutes, and 3) "C"=facc identity. The best is bolded and the second best underlined.
Mori (2010). A related model later proposed by Koh et al. (2020) to achieve better causal inter- pretability. There have also been number of datasets that have been collected with visual attributes annotated (Liu et al., 2015; Yu & Grauman, 2014; Welinder et al., 2010; Patterson & Hays, 2016: Pham et al., 2021). One key difference between our work and these attribute learning approaches is that at test time we aim to learn classifier on novel attributes that are previously not labeled in the training set, and this brings additional challenges of transfer learning and learning with limited labeled data.
Zero-shot learning: In zero-shot learning (ZSL) (Farhadi et al., 2009; Akata al., 2013; Xian et al., 2019; Lampert et al., 2014; Romera-Parede & Torr, 2015; Akata et al., 2015), a model is asked to recognize classes not present in the training set, supervised only by some auxiliary descrip- tion (Ba et al., 2015) or attribute values (Farhadi et al., 2009) (see Wang et al. (2019a) for survey). Lampert al. (2014) studied the direct attribute prediction method, similar the Supervised At- tribute baseline described in Section 5.2. Compositional 7SL. aims at learning classes (Misra et al., 2017; Purushwalkam al., 2019; Wang et al., 2019b; Yang et al., 2020) defined by novel composi- tion of labeled attributes and object classes. An important distinction between ZSL and our few-shot attribute learning task is that ZSL uses the same set of attributes for both training and testing; by contrast, our task asks the model to learn attributes for which there are labels during training, and they may not be relevant to any of the training attributes or episodes. We summarize the relationships between ZSL, FSL and our task in Table 1.
Generalization to novel tasks: One key component of our work an attempt to understand the generalization behavior of learning novel concepts at test time. Relevant theoretical studies consider novel task generalization, casting a transfer learning and learning to learn framework (Baxter, 2000; Ben-David & Borbely, 2008; Ben-David et al., 2010 Pentina & Lampert, 2014; Amit & Mcir, 2018; Lucas et al., 2021). A common theme in these studies is in characterizing task relatedness, and the role that it plays in generalization to novel tasks. Arnold & Sha (2021) studied task clustering for few-shot learning in the embedding space and found class splits that are of different difficulty levels. Sariyildiz et al. (2021) use the WordNet hierarchy compute semantic distances. In our paper, we instead split the data in the attribute space, and if we assume that semantic classes are combinations of attributes, then disjoint attribute split will imply further semantic distances. In our work, we investigate the role of task relatedness empirically by investigating generalization performance under different splits of the attribute space.
5 EXPERIMENTS
In this section, we evaluate our proposed UFTE and UFTA approaches on several FSAL tasks.
5.1 DATASETS
We consider the following three datasets:
Celeb-A (Liu et al., 2015) contains over 200K images of celebrities' faces. Each image is anno- tated with binary attributes, detailing hair color, facial expressions, and other descriptors. We split 14 attributes for training and 13 for test.
5

---

Under review as a conference paper at ICLR 2022
Celeb-A
Zappos-50K
NN
NC
NN
Meta
71.73::0.52
75.27:0.51
73.38::0.53
80.47:00.9
83.42:0.11
81.10:1443
75.72++0.48
77.78+0.52
79.97:10.51
85.17=00.4
88.63
90.92:0.32
UFTE
79.03=20.47
81.04::0.47
82. 82.833+0.48
86.23+0.54
90.61
92.20:10.28
75.33++0:47
77.24+00551
88
81.17=044
85.48:10.1
88.24:10.7
UFTA
77.30=t0:22
82.16::0.46
84.14:10..48
86.40
1:0.36
90.25
91.66::0.29
SA
78.84=0.11
84.61+00/22
89. 89.15-+:038
87.54
90.97:1:.31
93.00::0.28
Table 3: Combination of different representation few-shot learners on 20-shot attribute learning. Our
proposed representation learning plus a simple logistic regression (LR) consistently the best. Note: Meta-NN Net, Mcta-NC= ProtoNet, Meta-LR MAML/ANIL.
Train
Celeb-A
Test altr
Gap
Train aitr
Zappos-50K SUK
Test
Cap
L
D?
Val Acc.
UFTE
Val Acc. (A)
UFTA
Gap
ProtoNet
k7.12
U
79 59.444354
79.97a051
0.49
94.03-0:23
50.92=0032
-3.11
78.02 (-2.19)
-9.72
8281 (+2.60)
-4.00
UFTE
87 25
82.83
-.42
95.91
02.20-:038
-3.71
76.86 (-3.35)
-11.14
79.56 (-0.65)
-7.43
SA
88 25
78.86
9.39
95.11=009
88. 88.24
-0.87
82.01 (+1.80)
-5.63
(43.18)
-2.05
UFTA
85 43.534
84.14:04
-1.99
91.66-00.9
-2.55
2
76.32 (-3.89)
-11.58
79.71 (-0.50)
-7.23
87,881039
+1.27
95.59.0018
93.0020118
82.43(42.22)
-483
83,86
-190
Table 4: Comparison of representation learning methods Table 5: Number of projection layers (L) with respect to their ability to prediet training and test- during finetuning, and whether they are dis- ing attributes. Standard methods such ProtoNet and SA carded (D) during testing. Numbers are from perform well training attributes but do not transfer well to Celeb-A 20-shot. denotes changes compared novel ones (large training vs. test gaps in red).
to no finetuning.
Zappos-50K (Yu & Grauman, 2014) contains just under 50K images of shoes annotated with attribute values. We split these into 40 attribute values for training, and 39 for testing. mmageNet-with-Attriine
is a small subset of the ImageNet dataset (Deng et al., 2009) with attribute annotations. It has 9.6K images. We used 11 attributes for training and 10 for testing.
In all of the datasets above, there is overlap between training and test attributes. Additional split details can be found in the supplementary materials.
Episode construction: For each episode, randomly select
two attributes and look for
positive examples belonging to these attributes simultaneously. We also sample an equal number of negative examples that don't match the selected attributes. This will construct support set of positive and negative samples, and then we repeat the same process for the corresponding query set well. Sample episodes shown in Figure 1.
5.2 METHODS FOR COMPARISON
We first consider directly trained on FSAL episodes of training attributes.
of classic few-shot learning methods for comparison. These methods are
MatchingNet (Vinyals et al., 2016) is a soft version of l-nearest-neighbor, At test time, will retrieve the label of the support example that is the closest in the feature space.
MAML (Finn et al., 2017) performs several gradient descent steps in an episode and learns the parameter initialization. For simplicity, we used the ANIL variant (Raghu al., 2020) that only learns the last layer in the inner loop.
ProtoNet (Snell et al., 2017) computes an average "prototype" for each class and retrieves the closest one.
TAFENet (Wang et al., 2019c) learss a meta-network that can output cunkiconditioned classifier parameters.
TADAM (Oreshkin al., 2018) predicts the batch normalization parameters by using the average features of the episode. For our task we found that conditioning on the positive examples only works the best.
In addition to the approaches above, we consider the following representation learning baselines for comparison.
ID trains network to perform the auxiliary task of face identity classification (Celeb-A only). SA, or supervised attribute, resembles the "Baseline" approach in the FSL literature (Chen et al., 2019). The network learns representations by predicting the training attributes associated with each image.
denotes unsupervised representation learning (SimCLR). We train separate models on the Celeb-A and Zappos datasets. For ImageNet, utilize the off-the-shelf model checkpoint trained on the full ImageNet- Compared to our approach, this baseline skips the finetuning stage (Stage II).

---

Under review as a conference paper at ICLR 2022
x
S-shol
20-shot
NN
NC
UR
Thatn
Test
adir
Coep
Chance
50,00
50.00
Mesa
61.2 zos
97.45
2:39
2000
MAML
$7,90 0.75
37.46 2:000
71.12
71.25
26.36
71.25
-5.11
69.05
71.25
69.7 am
7284 au1
72:12 -46
5A
68.03
78.1
64.16 au
78.12
4.87
6.19
UFTE (Ours)
20.92 0.59
72.12 0.5
71.58
UPTA
77.08
72.91 108
4.17
70.4
7032
34
03.77
759 gm
230
UFTA (Ours)
71.12 0.55
72.91
0.6
Table and 20-shot attribute learning re- Table 7: 20-shot FSAL on Im- Table Training vs. test at- sults on ImageNet. Learners uses logistic re- ageNet with different few-shot tributes of 20-shot FSAL on gression (LR) at test time.
learners.
ImageNet.
1) Mustache
2) Brown Hair & Cheesbone
3) Poep Toe & Sipp On
Women
Heels
54
SA*
UFTA
Figure 2: Visualization of few-shot classifiers using CAM (Zhou et al., 2016), top of different repre- sentations. Left: Celeb-A: A; Right: Zappos-50K. Target attributes that define the episode shown above and images are from the query set of the positive class at test time.
We also provide two oracle approaches to study the upper bound to generalization to novel attributes.
Oracle SA* learns its representations by predicting all binary attributes including both training and test ones.
Oracle GT directly uses the ground-truth binary attribute values as input features, and the readout is performed by training logistic regression. It still needs to select the active attributes that used in each episode.
For representation learning baselines, we mainly use logistic regression (LR) in few-shot episodes, but we also report results using the nearest neighbor (NN) and nearest centroid (NC) classifiers.
Implementation details: For Celeb-A and Zappos, images were cropped and resized to 84x84. We used 12 (He et al., 2016; Oreshkin et al., 2018) as the CNN backbone. The projection MLPs have 512-512-122 units. We train SimCLR entirely on Geleb-A/Zappor images, i.e. not using pre-trained ImageNet checkpoints for fair comparison. For ImageNet-with with--ttttts we utilize the off-the--elff SimCLR model from ImageNet- 1k, which has access to more unlabeled images. The image dimensions are 224 x24. We include additional experimental details in the Appendix.
5.3 MAIN RESULTS
Table shows our main results on Celeb-A and Zappos-50K with 5- and 20-shot episodes. Ta- ble 3 explores different combinations of representations and few-shot learners. Overall, the standard episodic meta-learners performed relatively poorly. Also, supervised attribute (SA) learning and learning via the auxiliary task of class facial identification (ID) were helpful for representation learning either. Interestingly. U attained relatively better test performance, suggesting that the train- ing objective in contrastive learning indeed preserves more general features not just for semantic classification tasks as shown in prior work, but also for the flexibly-define attribute classes in our FSAL paradigm.
Our proposed UFTA and UFTE approaches obtained significant gains in performance, suggesting that a combination of unsupervised features with some supervised information indeed beneficial for this task. Lastly, methods able to reduce the generalization gap between SA and the oracle SA*, in fact almost closing it entirely on Zappos-50K.
Results on weiheNeee-wtt--tttiiie UFTA had access to additional unlabeled examples. As shown in Table 6, both UFTE and UFTA outperformed other methods substantially. Because of the additional unlabeled data available in this setting, even U achieved a substantially better accuracy than SA and MAML, Results in Table show that UFTE and UFTA work well when combined with different few-shot learners.
reported separately for clarity, because U, UFTE, and

---

Under review as a conference paper at ICLR 2022
Cdebb A
Zupora S0x
20
P.shors
IX
25
(a) How many examples needed for FSAL? Per- (b) Correlation between readout accuracy and few- formance increases with number of shots, even when shot accuracy using UFTA. Part the variance can given the binary ground-trumg attribute vector (GT), be explained by the challenge of predicting attributes, suggesting that there is greater natural ambiguity in the and the rest from the ambiguity of FSAL. More shots task than in standard FSL.
reduce variance and improve correlation.
Figure 3: Task ambiguity and the effect of number of shots in FSAL
To understand and interpret the decision made by few-shot lin- ear classifiers, we visualize the classifier weights by using CAM (Zhou et al., 2016), and plot the heatmap over the 11x1 11 spatial feature map in Figure 2. SA sometimes shows incorrect localization as it is not trained to classify those novel test attributes. SA* shows bigger but less precise heatmaps since the training objective encourages the propagation of attribute information spatially. In con- trast, our proposed method produces surprisingly accurate and localized heatmaps that pinpoint the location of the attributes (e.g. mustache or cheekbone); this is impressive since no labeled informa- tion concerning these attributes was available during representation learning. This result supports the hypothesis that local features can be good descriptors that match different views of the same instance during contrastive learning, and finetuning further establishes a positive transfer between training and test attributes.
Visualizing few-shot classifiers:
Number of shots and task ambiguity: Since we have flexible target attribute class in each episode, it could be the case that the support examples are ambiguous. For example, by presenting only smiling face with eyeglasses in the support set, it is unclear whether the positive set is deter- mined by "smiling" or "wearing eyeglasses". Figure 3a show several approaches evaluated using LR with varying numbers of support examples per class in Celeb-A and Zappos-50K episodes, respec- tively. The oracle GT gradually approached 100% accuracy as the number of shots approached 20. This demonstrates that FSAL tasks potentially require more support examples than standard FSL to resolve ambiguity. Again here, UFTA and UFTE consistently outperformed U, SA, and ID baselines across different number of shots. Figure 3b shows the correlation between readout performance of attributes and few-shot learning accuracy, using UFTA. With larger number of shots, there is a higher correlation between the two, but there is still large amount of variance that is due to the ambiguity of the task itself. More details are included in the Appendix.
Ablation studies: Table 5 studies the effect of the projection MLP for attribute classification fine- tuning. Adding MLP projection layers was found to be beneficial for unsupervised learning in prior work (Chen al., 2020). Here we found that adding MLP layers is also critical in our representation finetuning stage as well. Finetuning directly on the backbone (depth=O), and keeping the MLP dur- ing test (Discard=no) both led to significant drop in performance. In the Appendix, we also report on studies of the effect of adding the L1 regularizer on LR.
5.4 ANALYSIS ON FEW- SHOT GENERALIZATION
In Tables 4 and 8, we study the performance gap between training attributes and test attributes. Notably, SA performs very well on test episodes defined using training attributes, but there is large generalization gap between training and test attributes. Our methods show significant improvements in terms of reducing the generalization gap between training and test attributes. Moreover, we find that self-supervised pre-training generally preserves informative features and is more general than supervised pre-training.
Investigating the cause of generalization issues: We hypothesize that the weak performance of episodic learners and SA on our benchmarks is because their training objectives essentially encourage ignoring attributes that are not useful at training time, but may still be useful at test time. In Appendix G. we study a synthetic problem further analyze these generalization issues. We explore training a ProtoNet model on data from linear generative model, where each FSAL episode presents ambiguity in identifying the relevant attributes. In this setting, the network is forced to discard information that is useful for test tasks in order to solve the training tasks effectively, and thus fails to generalize.
Transferability score: Up to this point, we have only studied one particular split of training/test attributes. We would like to examine whether our conclusions generalize over different splits. More
8

---

Under review as a conference paper at ICLR 2022
A. Transferability Score (T-Score)
B. Few-Shot Episodes
Episodie
T-Score
AcC.
0.56
0.93
Logistc Regress on Label Space
0.71
Cheekbones & Earrings
0.86
Positive
T-Score
Avg. Acc.
Prodiction
TrainyTeus Spit
Negative
Law
Med.
High
All Attributes
Rew Shot Actr bute Leomnnng code
Tra ceferblbiylityy Jovel
C. Transferability Results
U
6.1%
1.8%-
CalobA
2.8%
0.9%
-67%
3.2%
12.2%
0.8
-5.7%
.1.24
-12%
Meckum
-2.9%
(6.7,
LeA
Mezum
ligh
-1.3%
Law
Medan
High
-62%
(a57.077
Medur
Zoopos
1.79
1.24
1.19
3.7%
-5.0%
4.5%
0.7%
1us2. 1872. 1.00
(0.53 Bel
(2.00
(a 18 (2.2)
1.1%
Imaget
0.6%
4.2%
2.3%
0.%
0.6%
6.1%
3.3%
1945.
6.50
0.5%
10.45
Lee Mo-i
Hith
Lw 0.0
14gh 10.55 1973.
Makke
1948..3 1245, Transtrabelly
Hockays
High
Tranalerabilty
Tranieratiily
Trarafsmundly
Figure 4: Few-shot performance vs. transferability across training and test attributes. A: Transferability
score (T-score) is computed based on the AUC of test attribute predicted by logistic regression model on a set of training attributes. 100 different random splits
train/test attributes per split used. B: Both episodic accuracy and T-scores are recorded on 60,000 episodes (600 episodes per split). Episodes are grouped into three bins in terms of their T-scores. C: Performance training or finetuning on training attributes correlates with T-score. Error bars are standard errors in each bin.
importantly, we aim to predict the transferability between training and test splits by analyzing the training vs. test attributes. Each image has complete attribute vector, describing the values of each attribute in the image. Some of these attributes in the training set, and others in the test set. To quantify the transferability. we leverage the idea of mutual information. More concretely, we learn a logistic regression model that takes the training attribute vector in particular image as input and predicts the value of one of the test attributes in that image. Each logistic regression model will generate an AUC score on held-out images, and we average them across the relevant test attributes in each episode, and define this AUC score the "transferability score." Our hypothesis is that more mutual information between the attribute label distributions will translate to higher transfer performance when the model is supervised on the training attributes and tested the test attributes.
In Figure 4, we ran experiments using 100 random splits of training and test attributes. The results verify hypothesis. We positive correlation between the transfer performance and our trans- ferability score: When subtracting U as a baseline, both UFTA and UFTE models get better when there is higher transferability (subtraction reduces the effect of per-episode variability). The same conclusion can be drawn when we subtract SA* from SA. By plotting the relation between and SA, we show that supervised learning is more helpful when there is higher transferability the label space whereas self-supervised learning is more flexible at adapting to novel target tasks.
CONCLUSION
Acquiring knowledge of new attributes is one of the most basic learning skills of an intelligent agent. In this paper, we propose few-shot attribute learning to enable this core functionality. We developed benchmarks using the Celeb-A, Zappos-50K, and ImageNet datasets to create learning episodes using existing attribute labels. This setting presents strong generalization challenge, since the split in attribute space can make the training and test tasks less similar. Consequently standard supervised representation learning performs poorly on the test set, unlike recent benchmark results in few-shot learning of semantic classes. We found that unsupervised contrastive learning preserved more general features, and further finetuning yielded strong performance. We also discovered that the similarity in the attribute label space can roughly predict the gain obtained by supervised training and finetuning, which could provide some insight into the generalization behavior of representation learning algorithms when dealing with novel tasks at test time.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for attribute-bass baseee-bbdddde classification. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2013.
Zeynep Akata, Scott E. Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In IEEE Conference Computer Vision and Pattern Recognition, CVPR, 2015. 5
Amit Alfassy, Leonid Karlinsky, Amit Aides, Joseph Shtok, Sivan Harary, Rogério Schmidt Feris, Raja Giryes, and Alexander M. Bronstein. Laso: Label-set operations networks for multi-label few-shot learning. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR, 2019. 4
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended pac-bayes theory. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICMI., 2018.
Sébastien M. R. Arnold and Fei Sha. Embedding adaptation is still needed for few-shot learning. CoRR, abs/2104.07255,, 2021.
Lei Jimmy Ba, Kevin Swersky, Sanja Fidler, and Ruslan Salakhutdinov. Predicting deep zero-shot convolutional neural networks using textual descriptions. In IEEE International Conference on Computer Vision, ICCV, 2015. 5
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000. 5
Shai Ben-David and Reha Schuller Borbely. notion of task relatedness yielding provable multiple- task learning guarantees. Machine learning, 73(3):273-287, 2008.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 1911-2) 2010. 5
Ting Chen, Simon Kormblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020. 3,
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang. and Jia-Bin Huang. closer look few-shot classification. In Proceedings of the 7th International Conference on Learning Representations, ICLR, 2019. 4,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2009. 2,
Ali Farhadi, Ian Endres, Derek Hoiem, and David A. Forsyth. Describing objects by their attributes. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR. 2009. 1, 4,
Ali Farhadi, Ian Endres, and Derek Hoiem. Attribue-centric recognition for cross-category gener- alization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2010.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE TPAMI, 28 (4):594-611, 2006.
Vittorio Ferrari and Andrew Zisserman. Learning visual attributes. In Advances in Neural Informa- tion Processing Systems 20, NIPS, 2007. 4
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, ICMI., 2017. 6
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016.7
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning, ICMI., 2020. 1,5
Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the 33th Annual Meeting of the Cognitive Science
Society, CogSci, 2011. 1,
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-aseed classification for zero-shot visual object categorization. IEEE Trans. Pattern Anal. Mach. Intell., 36(3))453-4666 2014. 2, 4,5
Zeqian Li, Michael Mozer, and Jacob Whitehill. Compositional embeddings for multi-label one-shot learning. In IEEE Winter Conference on Applications of Computer Vision, WACV, 2021. 4
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision, ICCV, 2015. 2,5 5
James Lucas, Mengye Ren, Irene Kameni, Toniann Pitassi, and Richard S. Zemel. Theoretical bounds on estimation error for meta-learning. 2021.
Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017. 1, 1,4,3 5
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems 31, NeurIPS, 2018. 6,
Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton, and Tom M. Mitchell. Zero-shot learning with semantic output codes. In Advances Neural Information Processing Systems 22, NIPS, 2009. 1
Genevieve Patterson and James Hays. COCO attributes: Attributes for people, animals, and objects. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), 14th European Conference Computer Vision ECCV, 2016. 5
Anastasia Pentina and Christoph H. Lampert. A pac-bayesian bound for lifelong learning. In Pro- ceedings of the 31th International Conference on Machine Learning. ICML, 2014.
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivas- tava. Learning to predict visual attributes in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Mare' Aurelio Ranzato. Task-driven modular networks for zero-shot compositional learning. In IEEE/CVF International Conference on Computer Vision, ICCV, 2019. 5
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of MAML. In 8th International Conference on Learning Representations, ICLR, 2020. 4,
Mengye Ren, Eleni Triantafiliou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classifica- tion. In Proceedings of 6th International Conference on Learning Representations, ICLR, 2018.
Bernardino Romera-Paredes Paredes and Philip H. S. Tom. An embarrassingly simple approach to zero-shot learning. In Proceedings of the 32nd International Conference on Machine Learning, ICML, 2015. 5
11

---

Under review as a conference paper at ICLR 2022
Eleanor Rosch and Carolyn B Mervis. Family resemblances: Studies in the internal structure of categories. Cognitive psychology, 7(4):573- 605, 1975.
Mert Bülent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept generaliza- tion in visual representation learning. 2021. 5
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30, NIPS, 2017. 3, 4,
Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, pp. 181-209 Springer, 1998.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Car- les Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In 8th International Conference on Learning Representations, ICLR, 2020.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint 2XXiv:904.00773: 2019. 1
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems 29, NIPS, 2016. 1, 4,
Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology, TIST, 10 (2):1-37, 2019a.
Xin Wang, Fisher Yu, Trevor Darrell, and Joseph E. Gonzalez. Task-aware feature generation for zero-shot compositional learning. CoRR, Ne211006008554 2019b. 5
Xin Wang, Fisher Yu, Ruth Wang, Trevor Darrell, and Joseph E. Gonzalez. Tafe-net: Task-aware feature embeddings for low shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019c. 6
Yang Wang and Greg Mori. A discriminative latent model of object classes and attributes. 11th European Conference on Computer Vision, ECCV, 2010. 4
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010. 5
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning A comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal. Mach. Intell., 41(9):2251-226; 2019.
Liuyu Xiang, Xiaoming Jin, Guiguang Ding, Jungong Han, and Leida Li. Incremental few-shot learning for pedestrian attribute recognition. In Proceedings of the Twenty-Eighe Eightth International Joint Conference on Artificial Intelligence, IJCAI, 2019. 4
Muli Yang, Cheng Deng, Junchi Yan, Xianglong Liu, and Dacheng Tao. Learning unseen concepts via hierarchical decomposition and composition. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2014. 2, 5,
Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016. 7,8
12

---

Under review as a conference paper at ICLR 2022
Mean AUC
RND
PN
ID
SA
UFTE
UFTA
SA*
All (40)
79.18
88.80
91.29
90.27
92.80
93.34
93.33
94.46
Train+ Test (27)
82.27
93.38
94.31
94.23
95.78
96.53
96.52
97.18
Train (14)
84.40
96.04
95.34
96.04
96.43
97.23
97.23
97.50
Test (13)
79.96
90.52
93.19
92.63
95.08
95.78
95.76
96.84
Table 9: Celeb-A attribute readout performance of different representations, measured in mean AUC, RND denotes using randomly initialized CNN; PN denotes ProtoNet.
SA
U
UFTE
UFTA
SA#
GT
LR
77.4
79.2
82.2
83.1
87.1
95.8
+L1 (lc-)
77.6 (+0.2)
(+1.2)
82.3 (40.1)
83.2 (40.1)
(4..)
96.1 (40.3)
(1c:3)
78.2 (+0.8)
80.2 (+1.0)
(+0.2)
83.8 (+0.7)
88.4 (+13)
(+1.3)
+11 (1e-2)
75.7 (-1.)
78.3 (-0.9)
78.8 (-3.5)
79.5 (336)
(+..))
98.2
Table 10: Effect of the L1 regularizer different representations for the validation set of Celeb-A 20-shot.
A ATTRIBUTE READOUT
In Tab. and 11, we provide attribute readout performance with different learned representations. This is a similar task that measures the generalizability, but it does not evaluate the rapid learning aspect brought by few-shot learning.
B ABLATION STUDIES
Table 10 studies the effect of the LI regularization. The benefit is especially noticeable on SA* and GT, since it allows the few-shot learner to have a sparse selection disentangled feature dimensions.
C ADDITIONAL HEATMAP VISUALIZATION
We provide additional visualization results in Figure 5, 6, and 7, and we plot the heat map to visualize the LR classifier weights. Figure 5 includes SA*, U, and UFTE which are omitted the main paper due space limitations. Figure 6 and 7 visualize more information including both support and query examples in the episode, and some the episodes are challenging to solve given just a few examples.
D ATTRIBUTE SPLITS OF CELEB-A
We include the attribute split for Celeb-A Table 12. There are 14 attributes in training and 13 attributes in val/test. We discarded the rest of the 13 attributes in the original datasets since they either hard to classify with the oracle classifier (e.g. big lips, oval face) or simply ambiguous (e.g. young, attractive).
ATTRIBUTE SPLITS OF ZAPPOS-50K
The 5appos-50K dataset annotates images with different values relating to following aspects of shoes: 'Category', 'Subcategory', 'HeelHeight', 'Insole', 'Closure', 'Gender', 'Material' and "oesttll'.
Mean AUC
SA
SA
UFTA
All (25 altributes)
72.01
73.02
81.08
82.49
Train+Test (21 attributes)
73.43
78.98
80.14
82.37
Train (11 attributes)
72.69
75.86
80.63
82.43
Test (10 attributes)
72.01
74.98
81.08
83.30
Table 11:
ImageNet-withe Autributes attribute readout binary prediction performance different representa-
tions, measured in mean AUC.
13

---

Under review as a conference paper at ICLR 2022
11 Mustache
Brown Haair & High Cheekbone
3) Poop Too & Slip On
4) Women Heels
S4
UTTE
UFTA
Figure 5: Additional visualization results, 20-shot episodes, including more methods for comparison.
Context: Bangs
support
Negative support
Positive query
Negativee query
26.0%
Context: Brown Hair Mouth Slightly Open
support
Negative suppor:
Positive query
Negative query
Figure 6: Visualization of Celeb-A 20-shot LR classifiers using CAM top of UFTA representations. Context attributes that define the episode are shown above. Classifier sigmoid confidence scores shown the bottom. Red numbers denote wrong classification and green denote correct.
discarded the 'Insole' values, since those refer to the inside part of the shoe which isn't visible in the images. We also discarded some 'Material' values that we deemed hard recognize visually. We also modified the values of 'HeelHeight' which originally was different ranges of cm of the height of the heel of each shoe. Instead, divided those values into only two groups: 'short heel' and 'high heel', to avoid having to perform very fine- ame-grained heel height recognition which we deemed was too difficult.
14

---

Under review as a conference paper at ICLR 2022
Context: Slip-On & Heels
Positive support
Negative support
Positvv query
Negative query
$3.05
67.55
52.5%
34.996
16.2%
2244
MAY
Context: Boots & Ankle
Positive support
Negalivee support
Positive query
Negalive query
70.3%
30.5%
60.4%
25.0%
2.4%
20.2%
Figure 7: Visualization of Zappos-50K 20-shot LR classifiers using CAM on top of UFTA representations. Context attributes that define the episode are shown above. Classifier sigmoid confidence scores are shown at the bottom. Red numbers denote wrong classification and green denote correct.
Black_Hair Eyeglasses
Blond_Hai Goatee Pale Skin
Chubby Gray Hair
Double Chin
Train
Male
No Beard
Receding Hairline
Rosy -Cheeks
Smiling
Bald
Bangs
Brown Hair
Hcavy_Makenu Makeup
High. Cheekbones
Mouth Mouth_Slightly Open
Mustache
Narrow Eyes
Val/Test
Sideburns
Wearing Earring
Wearing Hat
Wearing Lipstick
Cearing.Neckten
Table 12: Attribute Splits for Celeb-A
These modifications leave us with total 79 values (across all higher-level categories). Not all images tagged with value from each category, while some value from the same category (e.g. two different materials used in different parts of the shoe). split these values into 40 "training attributes' and 39 'val/test attributes'.
tagged with more than one
We include the complete list of attributes in Table 13. The format we use 'X-Y' where X stands for the category (c.g. 'Material') and Y stands for the value of that category (c.g. "Wool'). We do this to avoid ambiguity, since may happen that different categories have some value names in common, e.g. 'Short Heel' is value of both 'SubCategory' and 'HeelHeight'.
F ATTRIBUTE SPLITS OF IMACENET-WITH-ATTRIIUT
We include the attribute split for wiith-Neewwtt--ttriiie in Table 14. There are 11 attributes in training and 10 attributes val/test. We discarded the rest of the attributes in the "shape" category (long, round, rectangular and square), since they difficult predict from the images.
15

---

Under review as a conference paper at ICLR 2022
-alegory-SSiess
Culgony-Kandalls Sandals
Subt Cateeooy----ooonn
SuliCaiepory-He Heeel
SubCulegory- Boot
SubCalegory Slipper Flats
SubCategory Short heel
SubCategury Flats
SulCCieegry Slipper
SubCCaioooy-Aillle
SubCCalogory Kuve High
SubCancgury- Crib
SubCalegory- the Knee
HeelHeiihh-HHgg High heel
Closrr-Pull-on
ClosureAnkle Strap
Train
Closure Zipper
-lusure-Supp
Closure Elastic Gore
Closure-T-She T--rap
Closure Sling Back
Clesure Spai Strup
Closure Toggle
Genke-Menn
Gender-ayys
Material-Rubber Rubber
Material-We Mooriai-Woll
Material-Siik
Material- Material-Aluman Aluminum
Material-Plaste
Toestyle-aapp Toe
SquualeeSquure Toe
Tocslye-Smub Toe
Tocstyk-Bicycle Toe
Dboeyle-Open Toe
Roestyle-Poittel Toe
Toestyle-Alnonn
Toestyle-Apre Toe
Toestyle Stip Toe
Toestyle Medallion
Category-Boo Boots
Category-Siigerrs
Sub(Cuepoyy-Mid.ccm
SuhCategory-AAkee
SubCategory- Leafers
SubCategory -aatttt--Batttt Shoes
SuhCategory- Cloog and Mules
Sneakers and Athletic Shoes
SaboCayy Heels
SubCategory Prewalker
SubCategocy Prewalker Boots
SubC
BubCCatogoey-FFan Fisswaller
HeelHeight- Short heel
Closure-I
Closure- Closure-Ruckle Buakle
Closure-Hook Loop
Closure Slip
Closure- Dosure-Ankle Wrap
Closure-Bungee
ClasureAdjustabl
Val/Test
Cloeare-Rute Burton Loop
Cloure Menre-Mekkcaoo Strap
Closurp- Bell
Hender--Womes
Gender-Girts
Material-Suee
Saakeia-
MaterillCCnduroy
Material Hanesial-Horse Hair
Maserial Stingray
Toestyle Round Toe
Trestyle Closed Toe
Trestyle-M
Toessyle- Wingtip
Thesryle- CessryleContter Seam
Thessyle Algonquin
Toestyle- Bump The
Toestyle Wide Toe Box
Tesdyle-Wide
Thesryle Peep Toe
Table 13: Attribute splits for Zappos-50K
pink
spotted
wet
blue
shiny
rough
striped white
Train
metallic
wooden
gray
brown
green
violet
red
orange
yellow
furry
black
Val/Test
vegetation
smooth
Table 14: Attribute Splits for ImageNer-
G FEW-SHOT ATTRIBUTE LEARNING ToY PROBLEM
In this section, we present toy problem that illustrates the challenges introduced by the FSAL setting and the failures of existing approaches on this task. This simple problem captures the core elements of our FSAL tasks, including ambiguity, introducing novel attributes at test time, and the role of learning good representations. The primary limitation of this model is the fact that it is fully linear and the attribute values are independent-in a more realistic FSAL task recovering a good representation from the data is significantly more challenging, and the data points will have more complex relationship with the attributes as in our benchmark datasets.
Problem setup
We define FSAL problem where the data points E Rm generated from binary attribute strings, z E {0, 13dd with x = Az + Ç for some matrix A E Rmxd with full column rank and noise source Ç. Thus, each data point is sum of columns of A with some additive noise.
In each episode, examples are labelled as positive when two designated entries of the attribute strings both 1-valued, and negative otherwise. For the training episodes, the labels depend only the first d, < entries of z. At test time, the labels depend on the remaining d - d, attributes. The training and test episodes are generated by choosing two of the attributes in the respective sets. Then k data points are sampled with positive labels (the two attributes are 1-valued) and k with negative labels least one of the attributes is O-valued).
Linear prototypical network Now, consider training prototypical network on this data with linear embedding network, g(x) = Wx. Within each episode, the prototypical network computes the prototypes for the positive and negative examples,
C -
-(x) x,ES,
Σ x.ES,
Σ
za W a for j (0,
where S, is the set of data points in the episode with label j. and a, is the th column of the matrix A. Further, the prototypical network likelihood is given by.
exp {- |x Co 1 2]
p(y = 0|x) exp {- WX Co 1 + exp { [ Wx - c1 12
The goal of the prototypical network is thus to learn weights W that lead to small distances between data points in the same class and large distances otherwise. In the FSAL tasks, there is an addi-
16

---

Under review as a conference paper at ICLR 2022
(A) FSAL features
(B) FSL task learned features
Figure 8: Projecting data features into prototypical network embedding space (W A) for the linear toy problem. Values closer to darker in colour. On the FSAL task, the model destroys information from the test attributes to remove ambiguity at training time.
tional challenge in that class boundaries shift between episodes. The context (the choice of attribute entries) defining the boundary is unknown and must be inferred from the episode. However, with few shots (small k) there is ambiguity in the correct context with high probability that several possible contexts provide valid explanations for the observed data.
Fitting the prototypical network for j {0, 1} we have,
Notice that under our generative model, with x = Wz and
Wx W -W-WA 1 Σ x))+ + / WwG+ww
ES
Notice that if (2) = A - 2;) E Ker(W), the kernel of W. then the entire first term zero. Further, if z Sj (the same class the prototype) then there is contribution from the positive attribute features in this term. Otherwise, this term is guaranteed to have some contribution from the positive attribute features.
Therefore, if W projects to the linear space spanned by the positive attribute features then Wv, (z) when z E Sy and non-zero otherwise. This means that the model will be able solve the episode without contextual ambiguity. Then the optimal weights are those that project to the set of features used in the training set destroying all information about the attributes which would otherwise introduce ambiguity.
We observed this effect empirically in Figure 8, where we have plotted the matrix abs(W (WA). Each column of these plots represents column of A mapped to the prototypical network's 's embedding space. The first columns correspond to attributes used at training time, and the remaining 5 to those used at test time.
In the FS task described above, as our analysis suggests, the learned prototypical feature weights project out the features used at test time (the last columns). As result, the model achieved 100% training accuracy but only 51% test accuracy (chance is 50%).
We also compared against equivalent problem set up that resembles the standard few-shot learning setting. In the FSL problem, the binary attribute strings may have only a single non-zero entry and each episode is binary classification problem where the leamner must distinguish between two classes. Now the vector z is a one-hot encoding and the comparison to the prototypes occurs only single feature column of A, thus there is no benefit to projecting out the test features. As
17

---

Under review as a conference paper at ICLR 2022
expected, the model we learned (Figure B) is not forced throw away test-time information and achieves 100% training accuracy and 99% test accuracy.
Settings for Figure 8 We use 10 attributes. of which are used for training and for testing. We use uniformly random sampled A E IR 1R and the prototypical network learns W E R 10x30 We use additive Gaussian noise when sampling data points with standard deviation of 0.1. The models are trained with the Adam optimizer using default settings over a total of 30000 random episodes, and evaluated on an additional 1000 test episodes. We used 20 to produce these plots, but found that the result was consistent over different shot counts.
18


Reviews:
[{"review_id": "uiXXda0Cyny", "reviewer": "Reviewer_5A2w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*** Positive aspects:\n- The paper is clear, and well written. The contributions are also clear.\n- The proposed approach delivers good performance against a set of reasonable few-shot learning baselines. More generally, all aspects of the experiments seem good. In particular, I appreciate the effort put into ensuring that no outside data is used in part 1 (which is a problem often acknowledged in zero-shot learning) [1]\n- The problem setting is clearly motivated.\n- Ablation studies are performed, giving more details into the effect of the different components.\n\n*** Concerns:\nThere are no major concerns with the approach: the proposed method seems to work well. \n\nAmong the less important concerns, the most important one is that overall the proposed method is an assembly of previously trialed approaches. Combining unsupervised learning techniques with classical approaches to few-shot adaptation does not constitute a novel method.  In this review, I am not arguing for novelty for novelty's sake. I am rather just mentioning that the paper consists in introducing a variation on a classical learning problem (few shot strategies with attributes have been extensively studied in the literature), which is then solved using techniques that were designed for related problems. \n\n*** Questions\n- What motivated the choice of few-shot algorithm in Part 3 of the learning algorithm? Was there a theoretical motivation, or rather a practical one? \n- How are hyper-parameters for each of the 3 parts optimized? All together or separately for each?\n- What would be the effect of injecting attribute information directly in Part 1 (by an auxiliary loss e.g., which would combine part 1 and 2)?\n- I wanted to confirm the following with the authors. When running experiments on Imagenet-1K, the SimCLR model they use has never seen any of the the test images, is that correct?\n- Have the authors tried other self-supervised learning algorithms for part 1? Did other methods lead to additional insights, apart from performing worse? (i.e. is there a link between different families of self-supervised learning algorithms, and performance?).\n\n\n*** References\n[1] Xian, Yongqin, et al. \"Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.\" IEEE transactions on pattern analysis and machine intelligence 41.9 (2018): 2251-2265.", "summary_of_the_paper": "The authors consider the problem of few-shot attribute learning. Contrary to most past approaches on learning with attributes, including classical zero-shot learning, the authors deal with the case where the attributes seen at test-time are previously unseen. As a result, few samples must be used to quickly adapt the learning algorithm to new attributes.\nThe approach works in 3 stages. The first uses a classical representation learning algorithm to learn a visual representation, SimCLR. Following this, the network is given supervised training to fine-tune the representation. In the last step, the authors adapt a previously devised few-shot learning strategy that consists in training a linear classifier.\n", "summary_of_the_review": "The approach works, the components are well motivated. Despite this, in my opinion, the approach can be understood as a variation on more classical attribute-related few-shot learning tasks, solved by combining methods that work well on these. I would like to highlight that, despite the previous point, I feel the proposed idea has scientific merit, hence borderline score I have given currently.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hzqw-ZzBNwN", "reviewer": "Reviewer_tUmn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The problem setting, learning attributes in a few-shot setting, is novel. The problem may generalize to many realistic applications.\n2. The authors reveal an interesting observation that self-supervised training can help the backbone to focus on generalizable features, which might also benefit the general FSL tasks.\n3. The authors perform extensive experiments and ablation studies. The experiment results indicate that the proposed model leads to consistent improvement.\n\nWeaknesses:\n1. The three stages in the model are introduced in other papers before, which hinder the novelty of this paper. The self-supervised training is proposed in [1r], the fine-tuning stage adopts [2r], and the few-shot learning stage adopts logistic regression proposed in [3r].\n\n2. Some model configuration and experiment details are not clear. \n* If the LR outperforms NN and NC in the Few-shot learning stage, I'm curious about the performance of utilizing LR in the fine-tuning stage.\n\n* The episode construction: on page 6 paragraph 1, the author randomly selects one or two attributes and looks for positive examples belonging to these attributes simultaneously. From Figure 2 we can observe that mustache is learned alone while Brown Hair and High Cheekbone are learned together. When would the author select two attributes and why is this happening?\n\n*  In Table 3 the ablation study is performed under a 20-shot setting, however, nearly all methods achieve similar results using 20 samples. Why not use the results in the 5-shot setting? \n*  In the CAM visualization of different methods, the author claim that the proposed method produces accurate and localized heatmaps for each attribute, which is even better than the oracle setting SA*. Why is the proposed model producing localized attention? I do not see any training constraints for this. More explanation for this phenomenon is appreciated.\n* In related work the author claim that \"In this paper, we study learning novel contextual similarities in the form of attributes using only a few training examples\". What does context mean? And where is the evidence to support this claim?\n3. My other major concern lies in Section 5.4, the generalization analysis. The author defines a transferability score and reveals a positive correlation between transferability and generalization ability. However, is this real transferability? My point is that some attributes are heavily correlated with each other the model might take advantage of the correlation to predict novel attributes. For instance, earings and lipsticks co-occur in most of the images. If the network has seen lipsticks as the base attribute, then given a few images with those two attributes showing together, the model will naturally misunderstand our intuition and predict the lipsticks instead of earings. Some more qualitative results might help us to understand if the network successfully generalizes to novel attributes, or just take advantage of the attribute correlation and misunderstand the attributes.\n\n[1r] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020.\n\n[2r] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30, NIPS, 2017.\n\n[3r] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In Proceedings of the 7th International Conference on Learning Representations, ICLR, 2019.\n\n\n", "summary_of_the_paper": "This paper investigates a new and interesting topic, few-shot attribute learning (FSAL), where the model is trained with base attributes with abundant samples and tested with novel attributes with only a few samples. Experiments were performed on three benchmark datasets with attribute annotation. The paper found that self-supervised learning helps to train a backbone that generalizes well to novel attributes, compared to training the backbone with only base attributes. The author also compares several few-shot learning models, e.g., MatchingNEt, MAML, ProtoNet, in this FSAL setting and found that the logistic regression layer works the best.", "summary_of_the_review": "This paper introduces an interesting topic and performs extensive experiments to verify the conclusion. However, the technical novelty is hindered since most of the contributions are not new. Besides, there are some unclear points and claims that need to be clarified and discussed. I'm looking forward to the authors' responses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SL6dioGltEb", "reviewer": "Reviewer_ym8e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths\n\n- Introduces a formulation for the problem of few-shot attribute learning.\n\n- Proposes reasonable baselines and experimental results on three benchmarks.\n\n- Provides some insight about learning behavior (explainability heatmaps, transferability of attribute, influence of shot number).\n\n\n**Weaknesses\n\nBasically I found that the article addresses an interesting issue about attributes as an intermediate representation for transferability but that it fails to provide convincing answers to this question mostly due to lack of rigor and precision in the way it is presented and justified. Most of my remarks go in this direction.\n\n- The target scientific goal of the paper is unclear: Finding ways to better analyze transferability from attribute prediction? Justifying that supervised fine-tuning is needed for pre-training? Proposing a new algorithm to solve few-shot attribute learning?\n\n- The task definition is also unclear: I didn't understand whether the attribute prediction task is only a binary classification on a single attribute (or a conjunction) from the unseen set, or a multiple-attribute detection problem? Is each episode considered as a \"2-way/n shot\" problem, the two classes being the target attribute present or not? The notations used to define such a task are ambiguous or incomplete.\n\n- The real algorithmic novelty of the proposed approach is low: the main result seems to be that self-supervision yields better features for the problem of attribute detection (this is not a real surprise given the literature [1,2]) and that supervised fine-tuning also improves features. Indeed, the proposed approaches (UFTA & UFTE) are very elementary and should rather be considered as baselines.\n\n- The transferability analysis could be interesting but is not clearly conducted. For instance, what does \"subtracting U as a baseline\" mathematically mean, how is it done? Also, it is said that the transferability score is an average on attributes in each episode, but I understood an episode as testing a single attribute (see my question about task definition). The formula used to compute it should be made more explicit as it is a major contribution of the paper.\n\n- The evaluation protocol for FSAL only exploits one split on the attribute set: the study about transferability uses multiple random splits. Why not use such a protocol for the main task? It would give more consolidated results. \n\n- Is attribute prediction fundamentally different from multi-label classification, an already studied area [3,4]? The introduction also speaks of expanding the vocabulary but the paper does not seem to evaluate such incremental capacity.\n\n- Compared methods:  it is not clear how they are modified to take into account the attribute detection problem instead of a classification with exclusivity between classes. More details should be given, at least in the supplemental material. \n  \n- The idea of \"context\", presented as a key issue in the paper, is not clear: is it given by the training dataset or by the support set for each episode?\n\n- It is now well established that self-supervised learning (SSL) is able to provide more general and robust features than supervision: the impact of the SSL method and of the DNN backbones however are important for classification (see [1]), and the proposed study only studies one configuration for each dataset. How really general are the results provided? The difference between the U algorithm (i.e. that skips the supervised fine-tuning part) and the two proposed baselines is often small: the role of the supervised fine-tuning is not obvious. Also, one could use pre-trained features learned using a bigger external dataset. If the main objective of the paper is to justify that a supervised fine-tuning part is useful - a reasonable assumption - more competing unsupervised features need to be compared.\n\n- There are many other datasets with attributes or labels: COCO attribute, AWA, CUBS, Flower... which contain more attributes than the datasets used in the paper. Since the paper is to introduce a new problem, a justification of the benchmark is expected.\n\n- A missing reference [5] with a clearer presentation of a  few-shot attribute learning problem.\n\n- Writing style: Please avoid assertions such as \"our proposed method produces *surprisingly* accurate and localized heatmaps that pinpoint the location of the attributes (e.g. mustache or cheekbone); this is impressive since no labeled information concerning these attributes was available during representation learning\". It seems to imply that self-supervised representations magically locate attributes! The interesting scientific question would be to understand why such behavior happens, if it is a consequence of the self-supervision or supervised fine-tuning, for instance.\n\n- Formal description: please describe with better precision the mathematical spaces underlying symbols, and their meaning: it helps to fully understand the paper. \n\n\n[1] Sariyildiz, M. B., Kalantidis, Y., Larlus, D., & Alahari, K. (2021). Concept generalization in visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9629-9639).\n\n[2] Van Horn, G., Cole, E., Beery, S., Wilber, K., Belongie, S., & Mac Aodha, O. (2021). Benchmarking Representation Learning for Natural World Image Collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12884-12893).\n\n[3] Alfassy, A., Karlinsky, L., Aides, A., Shtok, J., Harary, S., Feris, R., ... & Bronstein, A. M. (2019). Laso: Label-set operations networks for multi-label few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6548-6557).\n\n[4] Li, Z., Mozer, M., & Whitehill, J. (2021). Compositional embeddings for multi-label one-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 296-304).\n\n[5] Xiang, L., Jin, X., Ding, G., Han, J., & Li, L. (2019) Incremental Few-Shot Learning for Pedestrian Attribute Recognition. IJCAI.\n", "summary_of_the_paper": "The article proposes to address attribute detection with a small data constraint in the training phase (few-shot approach). The experimental protocol is divided in a series of episodes, each defined by a support set that specifies the attributes to be predicted and a query set used to evaluate the prediction. The generic algorithmic structure to solve this problem is divided in two steps: a common pre-training phase that combines self-supervised representation learning and supervised fine-tuning, followed by a supervised learning phase to solve each episode task. A training dataset containing image samples with attributes is used to learn the pre-trained representation space. The pre-training phase is compared to other approaches on three benchmarks implementing few-shot attribute prediction episodes. An indicator computed from a logistic regression between attribute values is proposed as a way to predict transferability between the learning dataset and the few-shot problem.\n", "summary_of_the_review": "\nThe question of addressing transferability at attribute level is interesting. A \"few-shot\" formulation to do so, as proposed in the article, is a possible direction.\n\nHowever, the overall presentation of the problem is messy, superficial and lacks precision. The algorithm novelty is low and the experimental part relies on questionable protocols. \n\nThe article should be reworked with a clearer focus on the scientific hypothesis that is tested and better justified experimentations.\n\nFinal decision:\n\nMy conclusion about acceptance has not really evolved: the problem of attribute transferability addressed in the paper is interesting, but the way it is studied and presented must be improved and better justified (see detailed comment of rebuttal after author's response). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I4kwduYnaQy", "reviewer": "Reviewer_7SAf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Setting (FSAL)\nThe problem of rapid learning of attributes is well-motivated and practical. At a high level, Section 4 put the proposed setting in different contexts well.\n\n## Benchmark\nWhile the three proposed datasets are diverse in domains (faces, shoes, and general), I feel, however, that there is not enough concrete discussion about comparisons between proposed benchmarks and existing ones: FC100, tiered-ImageNet, Meta-Dataset, as well as Arnold and Sha 2021. My concern is that the proposed benchmarks seem to be very limited in the number of attributes (~20-80 in total each). Further, random splits are adopted without leveraging insights from previous work that split the semantic object classes. Would it be possible to put your task construction in the context of your goal of understanding generalization as well as in the context of related work that split the data in different ways?\n\n## Experiments\nExperiments are quite extensive with multiple baselines considered, with the proposed approaches improving upon existing ones further. However, it is unclear how the findings in Section 5.3 differ from previous findings when semantic object classes are considered. What would be our takeaway from the trend that SA performs poorly and U performs well?\n\n## More detailed comments\n- Setting: The claim that the setting with attributes is different from the standard few-shot setting on object classes is sound. I agree that the setting could be useful in understanding “the generalization performance semantic classes in standard few-shot learning” [Last sentence of page 1, last sentence of the intro]. However, I was wondering if the authors could say something more specific about what kind of insights we expect from exploring the setting? For example, how exactly would this offer “a more systematic framework to measure the relatedness and transferability between the traing and test set” [Last sentence of “Few-shot learning” in Section 4)? Do you expect this setting to be harder or easier than the one with object classes? Do you expect that the reuse of images for learning different attribute concepts in different stages might make the problem easier?\n- Related Work: Should Arnold and Sha 2021 fall into “Few-shot learning” instead of “Generalization to novel tasks”. More generally, perhaps more discussion on how you separate these two items.\n- Experiments: Have you tried “Baseline++” (Chen et al. 2019) for SA?\n\n", "summary_of_the_paper": "This paper proposes the problem of few-shot attribute learning (FSAL) that follows the standard few-shot/meta-learning paradigm but focuses on attributes instead of object classes. The authors argue in Section 2 that the multi-label nature of attributes (smiling & wearing eyeglasses can be present simultaneously) makes this problem different; the context of each positive example in each episode can drastically change what the model is asked to learn. \n\nThis paper also proposes three benchmark datasets to study this problem, based on Celeb-A, Zappos50K, and ImageNet-with-Attributes. The paper evaluates multiple existing approaches on these benchmarks as well as improves upon them further.\n", "summary_of_the_review": "Novel and potentially useful setting to study but some concerns on execution related to benchmarks and how we should interpret the results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been reviewed with four expert reviewers. The reviewers have reached the consensus that the paper is not yet ready for publication. The main concerns are related to novelty. All reviewers gave substantial and constructive feedback. Following the recommendation of the reviewers, the meta reviewer recommends rejection.
Decision: Reject