Make an Omelette with Breaking Eggs: Zero-Shot Learning for Novel Attribute Synthesis
Authors: Yu-Hsuan Li
Tzu-Yin Chao
Ching-Chun Huang
Pin-Yu Chen
Wei-Chen Chiu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Most of the existing algorithms for zero-shot classification problems typically rely on the attribute-based semantic relations among categories to realize the classification of novel categories without observing any of their instances. However, training the zero-shot classification models still requires attribute labeling for each class (or even instance) in the training dataset, which is also expensive. To this end, in this paper, we bring up a new problem scenario: ''Can we derive zero-shot learning for novel attribute detectors/classifiers and use them to automatically annotate the dataset for labeling efficiency?'' Basically, given only a small set of detectors that are learned to recognize some manually annotated attributes (i.e., the seen attributes), we aim to synthesize the detectors of novel attributes in a zero-shot learning manner. Our proposed method, Zero-Shot Learning for Attributes (ZSLA), which is the first of its kind to the best of our knowledge, tackles this new research problem by applying the set operations to first decompose the seen attributes into their basic attributes and then recombine these basic attributes into the novel ones. Extensive experiments are conducted to verify the capacity of our synthesized detectors for accurately capturing the semantics of the novel attributes and show their superior performance in terms of detection and localization compared to other baseline approaches. Moreover, we demonstrate the application of automatic annotation using our synthesized detectors on Caltech-UCSD Birds-200-2011 dataset. Various generalized zero-shot classification algorithms trained upon the dataset re-annotated by ZSLA shows comparable performance with those trained with the manual ground-truth annotations.

Make an Omelette with Breaking Eggs: Zero-Shot Learning for Novel Attribute Synthesis
Yu-Hsuan Li National Chiao Tung University eval18902270gmail com
Tzu- Vin Chao National Chiao Tung University
chaotzuyin@nctu. edu. tw
Ching-Chin Huang National Chiao Tung University chingchun@cs. nctu. actu.edu.et. tw
Pin-Yo Yu Chen
Wei-Chen Chiu
IBM Research
National Chiao Tung University walonêcs nctu, edu. tw
pin-yu. chen@ibm com
Abstract
Most of the existing algorithms for zero-shot classification problems typically rely on the attribute-based semantic relations among categories realize the classification novel categories without observing any of their instances. However, training the zero-shot classification models still requires attribute labeling for each class (or even instance) in the training dataset, which is also expensive. To this end, this paper, bring up new problem scenario: "Can derive zero-shot learning for novel attribute detectors/classifers and use them to automatically annotate the dataset for labeling efficiency?" Basically, given only small of detectors that are learned to recognize some manually annotated attributes (i.e., the seen attributes),
aim to synthesize the detectors of novel attributes in zero-sio zero-shot learning manner. Our proposed method, Zero-Shot Learning for Attributes (7SLAA), which the first of its kind to the best of our knowledge, tackles this new research problem by applying the set operations to first decompose the seen attributes into their basic attributes and then recombine these basic attributes into the novel ones. Extensive experiments are conducted to verify the capacity of our synthesized detectors for accurately capturing the semantics of the novel attributes and show their superior performance in terms of detection and localization compared other baseline approaches. Moreover, we demonstrate the application of automatic annotation using synthesized detectors Caltech-UCSD Birds-200-200 dataset. Various generalized zero-shot classification algorithms trained upon the dataset re-annotated by 7SLLA shows comparable performance with those trained with the manual ground-truth annotations. Please refer to our project page for source code: https://yuhsuanli .iciiiiggiii github.io/ZSI 1o//SLA/
Introduction
Zero-shot learning (ZSL) algorithms for classification aim to recognize novel categories without observing any of their instances during model training; thus, the cost of collecting training samples for the novel categories can be eliminated. Typically, the challenge behind zero-shot classification lies in associating novel categories with the seen ones during training. Various existing approaches leverage different auxiliary semantic information to construct such associations across categories,
thus being able to generalize the learned models for classifying novel categories 117
1
32
16
3
37
synthesize the training samples for each novel category [39 40
33
4B
26). Among different
types of auxiliary semantic information adopted for ZSL, defining a group of attributes shared among
"The authors contributed equally this work. 36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

red wing
red
labels
generated labels
Union
red
red wing
blue breaat
Intersection
green broast
green breast
breast
antotale ornner arth buttuut
Sean artributns
Boase Buten
Unaman Bautee
red broast
Figure 1: Given set of trained/seen attribute detectors (c.g. "red wing", "red head", "blue breast", and "green breast"), our ZSLA can synthesize a novel detector for the unseen attribute (e.g. "red breast") by the following process: (1) applying the intersection operation the subsets ("red wing". "red head") and ('blue breast", "green breast") respectively to extract the common semantics each subsets, i.e. "red" and "breast", as the base attributes; (2) combining the base attributes via the union operation to realize the novel/unseen attribute detector, i.e. "red breast". The novel attribute detectors can later be applied annotate the dataset.
categories becomes one of the most popular choices, where each category is described by multiple attributes (i.e., multi-labeled by the attributes), and the attribute-baze bastee--bedddd representations discrimina- tive across categories. However, it comes with the expensive cost of manually annotating the samples in the dataset their attribute labels at a much granular level. For example, CUB dataset (351, one of the most widely-used benchmarks for learning zero-shot classification, is built by spending a great deal of time and effort to label 312 attributes for 11 11788 images.
As motivated by the issue of annotation efficiency on attribute labels, this paper aims to develop ZSL. on known attributes to annotate novel attributes for dataset automatically That is, analogous to the zero-shot classification scenario, we now advance to annotate novel attributes for a dataset via utilizing the knowledge from few types of seen/given manual attributes, as illustrated in Figure I Specifically, we take the well-known CUB dataset (35] as our main test-bed and have a deep investigation its attributes. We discover that, many attributes in CUB dataset (e.g. "red head" or "blue belly") follow the form combinations over base attributes (e.g. "red", "bluc", "head" and "belly" respectively). Building upon such observation, given defined set of attributes in the form of the ones used in the CUB dataset and labels of a few seen attributes (where the number is far less than that of overall defined attributes). propose Zero-Shot Learning for Attributes (ZSLA), method of training the seen attribute detectors and then tackle the ZSL problem to synthesize unseen attribute detectors via aecompooe-anddreasseemn are firstly decomposed into base attribute representations, in which they are further reassembled with novel combinations into novel attribute detectors, illustrated in Figure TT Here, both the decomposition and reassembly steps are achieved via set operations (i.e., the interaction and union operators, respectively). Together with the seen ones, the novel attribute detectors can be utilized to annotate the attribute labels for the dataset automatically.
manner. In detail, the seen attribute detectors
To demonstrate the efficiency of ZSLA, we synthesize 207 novel attribute detectors by leveraging only 32 seen ones from the CUB dataset. These novel attribute detectors are shown to be effective in capturing their corresponding semantic information and benefit both the attribute detection and localization for the samples in CUB dataset. Besides, we also synthesize a-CLEVR dataset by [13] further discuss the influence of noisy seen attribute
for conducting the controlled experiments labels. The results show that ZSLA can provide more robust annotations than the other baseline methods under the noisy scenario. Below, we highlight the contributions of this paper:
To the best of our knowledge, are the first to propose 7SL. for attributes to automatically annotate attribute labels for the zero-shot classification datasets.
We propose novel decompose-and- -n---andsdrrebbbl approach to single out the base attribute represen- tations by applying intersection on the seen attribute detectors and synthesize the unseen ones by having the union operation
the base attributes representations.
we show on the CUB dataset that, given only 32 attributes with manual annotations, ZSLA can synthesize novel attribute detectors to provide high-quality annotations for the dataset. By using the auto-annotated attributes, generalized zero-shot classification algorithms can also achieve comparable or even better performance than that using 3 12 manually annotated attributes.
Section 4.2

---

2 Related Works
Zero-shot learning (ZSL) was originally proposed to tackle the specific classification problem, where the model is expected to be capable of classifying the samples belonging to the novel categories which are not seen previously during training. The problem setup has been extended to other applications
such as detection [1 9 and segmentation
Here we provide a brief review of the works 26]. Without loss of generality, the ZSL approaches rely on using the auxiliary information (such as attributes, word embeddings, or text descriptions) as the basis for describing the categories and building the semantic relation among seen and unseen categories; and the existing methods can be roughly categorized into two
6 45
of zero-shot classification [17 32 2
16
groups: the embedding-basee -based methods [ 16 33 41 and generative methods [39 40
3
43] 26].
The embedding- based methods basically aim to learn a latent space that connects between the feature representations of training samples and the embeddings of their corresponding auxiliary information (e.g., the visual features and the embeddings of attribute labels for the training images in the CUB dataset), such that the test samples can be classified as the novel categories once their feature representations are close to the embeddings of novel categories (which are defined upon auxiliary information without requiring any additional training samples). The generative methods instead utilize the deep-generative models (e.g., generative adversarial networks (II], variational autoencoder [15], or their hybrids/variants) for learning to synthesize the samples or features of the unseen categories based on their auxiliary semantic information. Though saving the effort of collecting the training samples to recognize novel categories via ZSL techniques, manually annotating the auxiliary semantic information for the samples in the zero-shot training dataset is still quite expensive and time-consuming. In turn, our proposed task of zero-shot learning for novel attributes helps to reduce such costs for the scenario of zero-shot classification where the auxiliary information defined on attributes.
Aside from the typical zero-shot classification problem, compositional zero-shot learning (CZSL) and blind source separation (BSS) are tasks that our work is conceptually related to. CZSL 23 25
4 [2 24], also as known as state-object compositionality problem, aims to recognize the novel compositions (e.g. "ripe tomato") given the seen visual primitives of states (e.g. "ripe", "rotten") and objects (e.g. "apple", "tomato") in the training dataset. Our proposed problem scenario ZSLA is distinct from CZSL under several perspectives: (1) An image our problem scenario would have multiple attributes while there usually exists only a single state-object composition for CZSL; (2) Our synthesized attribute detectors are able to provide labels of novel attributes (i.e. these novel attributes do not have any manually labeled samples in the training set) for the images thus leading to more detailed descriptions for all the categories, while CZSL typically aims to increase the number of categories (i.e. each novel composition is treated as a new ane-grrined class). We provide the extensive discussions and experiments with respect to CZSL in Appendix D.3
On the other hand, BSS 36.6 42 g aims to separate and distinguish the source of several signals saa-mpss--nndreasseeee procedure seems to be similar to blind source separation at first glance, there exists a significant distinction that differentiates our ZSLA from BSS in signal processing: Our intersection operation to perform decomposition on seen attributes is non-blind, in which it works by the guidance of logical and semantic constraints (i.e. two input attributes should have a common ground in one of the base attributes but not both); by contrast, there is no such constraint in the blind source separation (which instead typically adopts independent assumption or mutual information in its modeling).
from their mixture. Though our ZSLA framework composed of
3 ZSLA: Proposed Method
Given zero-shot classification dataset {X, Y, each image x E X has its class label E Y and the
labels 6² (xx). where (x) is binary vector with its each element denoting if has a certain attribute E A" ZSLA starts with using {X, A*} to train the detectors M for all the attributes in A which are treated as seen attributes, then it adopts the seen attribute detectors M to synthesize the detectors 2 for the unseen attributes A A via decompose-and-reasenes
procedure, where A O A 0. For ease of understanding, we use the most popular zero-shot classification dataset, CUB [35). to illustrate how these steps are realized as follows.
3

---

3.1 Training Seen Attribute Detectors
Our attribute detectors are built on top of the image feature space produced by the image feature extractor f. Given an input image z and its feature map f(x) E R xt where each C-dimensional feature vector at position (} of f(x), denoted as f())(i. is the feature representation of the corresponding image patch on æ, the attribute detectors M RCXN" (in which N denotes the number of attributes in A) aim to give high response on the image patches containing the visual appearance related to the attributes in A*. Specifically, each column in M is acting as the embedding of a certain attribute. We use m to indicate the k-th column of M The response of the corresponding k-th attribute in A with respect to the patch-wise feature vector [i is calculated by specific form of their cosine similarity cos( cos(Ing) [ [(r)((,, where |m=| denotes applying element-wise absolute- value operator on mg. We have [ma] in our cosine similarity computation due to the reason that: Each dimension along channels of f(x)) is considered to capture specific visual pattern. Our |m hence acts as to apply the weighted combination over these various visual patterns for representing the characteristies of the k-th attribute in A", A, and the absolute-value operator over m is to ensure the combination weights are non-negatin negative...
1. Training the seen attribute detectors
2. Learning to synthesize unseen attribute detectors
Gume
orange wing
red head
réd wing
F2
Intersection
feature extractor
Share wnight
X
e(FO))
(Kx)
Union
7i
eFp
red wing
blue wing
Intersecioon
locatin
Blue wing
green breast
green Wing
wing
Senn utvilaaees
Base attribatte
Seen attruuuess
Seen attribute detectors, defined the embeddings for each seen attribute, are built top of the image features and their training is guided by two objectives: Lice and Lume, where the former drives the trained detectors to perform binary classification for attributes on image patches (cf. Eq. 2) while the latter enforces the uni-modal constraint on the response map (x) of patch-wise image features with respect to each attribute, in order to make it compact and concentrated (cf. Eq. 3 (2) Learning to synthesize novel/unseen attribute detectors via a detectors of seen attributes, the intersection operation is firstly applied on them to extract base attributes, and then these base attributes are further combined by union operation to synthesize the novel/unseen attributes. The training of these operations is driven by the reconstruction loss Cree (cf. Eq. 5) once the synthesized attribute coincides with any of the seen ones.
Figure 2: Overview of Our ZSLA. (1) Training the seen attribute detectors:
procedure: Given the trained
We denote (x:) E WxxN' as the response map which has included the cosine similarities of all the seen attributes A at each position on fxx). Note that, as our feature extractor f adopts the ReLU activation function in its last layer (similar to most image feature extractors based on the convolutional networks), the values in fix) become non-negative. Furthermore, as both [m and f()) j] are non-negative vectors, all entries of R results to be within the range [o, 1]. Following the popular tricks for ZSL and deep learning pointed out in [34] where adopting scaled cosine similarity in logits computation is important to achieve better model training, we use the computation below to calibrate the value of elements in R Rxx):
Rxxx=>> (2--(())-)
(1)
where the calculation within brackets shifts and expands the values in R2 (a:) towards [-1, 1] to match the typical value range of cosine similarity, and the hyperparameter y set to as suggested by (34. Then, we perform the max- max--ooolng operation on it" (x) and obtain the image-wise attribute response F* (x) E R Such logits over attributes thus are able to drive the model training (i.e. optimization over M and f' via the error between the attribute detection results and the ground-truth attribute labels di (x). The objective function Lice to evaluate the error between the logits of attribute detection result F* (z) and the ground-trute truth attribute labels 6f (x)) is defined via the binary cross-entropy:

---

N' Σ (oa + - ([](() log(1 - (f1(2)))
Licce
(2)
k
where (x) and (z) denote the k-th elements in 6 (a) and f (z) respectively, and is the sigmoid function.
In addition to the Lhiee loss, we introduce another objective function Lume place the uni-modal constraint 44 the response map R (z)), which encourages the response map for certain attribute (e.g. (z), the k-tth channel of (x)) to be uni-modal and concentrated. other words, we regularize a detector to focus single location or a small region in the image z. We provide further discussion of relaxing this assumption in Appendix A.44
N
(1 i 1
Lume
(R((()))E
(6.)
where ie.
5 0k arg , (x)((,)))) and || denotes the Euclidean norm.
The overall objective to train the feature extractor f and the seen attribute detectors M is illustrated in the left portion of Figure 2 and summarized as: Lhee + ALumes where the hyperparameter À controls the balance between losses and set to 0.2 in our experiments.
Moreover, we are aware that in CUB dataset the additional annotations indicating the ground-trun locations for the attributes which an image z has are also available (e.g. we know where the attribute "brown wing" appears on an image of "gadwall""). Hence, in addition to max max-pooling the response map (xx) obtain the image-wise response r" (x) for attributes, we experiment another way to (a:) is true, the k-th element in r (ar), i.e. (x). is assigned by R i. where the centre of the ground -round-truth location for the k-th attribute in A is located on the patch related to the position (i, of R"; (2) If (x) is false, rī((1) is assigned by having the average pooling over the k-th channel of (x)) Appendix D. provides the analysis for the impact using such additional annotations of attribute location on the performance of ZSLA.
obtain r (ar): (1) If of
3.2 Decompose-and- -eanpooo--andRRRssembblee for Synthesizing Novel Attribute Detectors
After obtaining the seen attribute detectors A we now aim perform the decompose-ane reassemble procedure (as shown in the right-half of Figure 2 for generating the detectors MM E RCXN" of the novel attributes A" (where N is the number attributes A") by leveraging M
First, we observe that most of the attributes in CUB dataset (the most popular zero-shot classification dataset and also our test-bed in this work) follow the form of "adjective + object part", for instance: "black eye", "brown forehead", "red upper-tail", or "buff breast".. Starting from such observation, we define two disjoint sets of base attributes, B and B", representing the adjectives and object parts used in the seen attributes, respectively (c.g. "blue", "yellow", "solid", and "perching-liks" for B%; "leg", "beak", "belly", and "throat" for BP). Please note that the concepts behind adjectives B in CUB dataset include only color but also texture, shape, and others. Formally, given attribute we use (a) and (a) to denote its corresponding base attributes on the adjective and object part, respectively (i.e. (a) B and (a) B"), where (*).) and BP(-) are functions to indicate the base attributes in B and B" for an attribute a, respectively.
and (P)) }. if ag and aj have common ground in either the base attribute of adjectives (i.e. (ax)) (at) B9) or the of object parts (i.c. 3P (P(a)) = 3P BP(a)) E B²) but not both, then we can use the intersection operation to extract such common base attribute from ag and o;:
Now, given two seen attributes ag and or E A in which as = (,
[ (ax) if (ax)) (s()), (P(a)) + (Pt)
I(ax, a4)
(ax)))
if (ax) * (ar), (Px) = (Pt)
For instance, is able to extract the base attribute "red" from the attributes "red wing" and "red breast"; or the base attribute "tail" from the seen attributes "buff tail" and "black tail".
Once we obtain the base attributes via intersection over seen attributes, we further adopt the union operation to create novel attributes. Given two pairs of seen attributes (as, az and {ax, av in which 8o (ax) I1(aR, a)) and (P(a)) Ilay, ay)) i.c. {az, at} share the same base attribute
5

---

of adjective while {ay, ar) share the same base attribute of object part, a novel attribute à can be synthesized by combining (ax)) and 3 (a i.e. ā (ae) (" (P(a)))) In particular, if such combination of base attributes has been seen in A i.e. there exists an attribute E A where ga(a)= = (ā) and (a) = (ā), we say the seen attribute is reconstructed by â. Otherwise, none of the seen attributes has the identical combination as our synthesized û, we denote ä a novel attribute and ā E A". In summary, extracting base attributes from seen attributes via intersection, followed by combining the base attributes into novel attributes via union, holistically forms our procedure to synthesize the novel attributes.
In practice, our implementation of the intersection function I (please refer to its illustration provided in Appendix. A.2) built based on the encoder architecture of vision transformer [TO] (ViT), in which its input is the embeddings of the seen attributes. i.e. the model takes mi and my from M as input when performing T(as, ai). where ag, (a E A To be detailed, there are several modifications in our model for intersection II with respect to the original VIT: (1) We remove the position embedding
in order to fulfil the commutative property of intersection, i.e. I(aa; ar) = I(a a&): (2) We attach learnable token named "intersection head" to the input sequence of transformer, which is similar to the extra class embedding in ViT. The corresponding output of this intersection head after going through the transformer encoder represents the embedding of the resultant base attribute, where absolute-value operation on it to make it non-negative vector (being
we apply the element-wiss analogous to what we did for the seen attributes); (3) There is only single self-attention block in the transformer. Please note that, the embedding of a base attribute is also a C-dimensional vector. Regarding our union function U, we simply adopt the average operation for its implementation, that is: Given two base attributes 8 E B and ip E B", we obtain the embedding in of the synthesized attribute à U(b B²) by averaging the embeddings of o and b". Specifically, such C-dimensional embedding m. is also defined upon the image feature and acts as the detector for the synthesized attribute a.
The training of our decompose-and-reesssee based on the reconstruction loss of the seen attributes Lyee. Given a synthesized attribute o, if there exists a seen attribute ag E A with having (ax))) = (a) and (ax) = (r(ā), the embedding m of ā and the embedding mi of aj expected to be identical to each other, free is thus defined as:
procedure for synthesizing novel attributes is simply
Cree
mm mil
(5)
Note that, our union function U has no trainable parameters (since is simply average operation), the gradient of Cree is propagated to focus on learning the parameters of our transformer for the intersection function T. In other words, we expect that the transformer is powerful be capable of extracting the base attributes where their averages are informative enough to act as the detectors for the synthesized attributes. Note that in order to fully leverage the seen attributes for training our decompose-ande -edsss--aadrrrremmblee procedure, we have a particular training scheme where its algorithm and the implementation details are shown in the Appendix. A.I
4 Experimental Results
Dataset. Our experiments are mainly conducted on the Caltech-UCSD Birds-200-2011 dataset 35
(usually abbreviated as CUB) for zero-shot classification. CUB dataset collects 11,788 images of 200 bird categories, where each image is annotated with 312 attributes. We select 32 attributes as our seen attributes A which can be decomposed into 15 base attributes of adjective B and 16 base attributes of object part B", and we use these base attributes to synthesize 207 novel attributes A" We follow the setting proposed by (38) for the task of generalized zero-shot learning (GZSL) split the CUB dataset, where such training and testing sets are used to train and evaluate our proposed scenario of ZSL on attributes, respectively. Furthermore, we base on [13 to create a synthetic dataset, named a a-CLEVR, for providing extensive analysis on the robustness of our proposed ZSLA against the noisy seen attributes (which are related an issue hidden behind CUB dataset). a-CLEVR contains 24 attributes (i.e. types of toy bricks) which are the combinations of 8 colors (i.e., base attributes of adjective B*) and 3 shapes (i.e., base attributes of object part B²). Among them, 16 attributes, which can be decomposed into the 11 base attributes, are selected as seen attributes A for training our proposed ZSLA,, where we are then able to synthesize novel attribute detectors to annotate the images of a-CLEVR dataset. Noting that, classes in our c-CLEVR dataset are defined by the specific combinations of toy bricks (where toy bricks with different color-shape combinations are treated as different attributes) For performing the GZSL task on a- c-CLEVR and evaluate the annotation quality

---

Table 1: Evaluation of synthesized novel/unseen attributes attribute classification (mAUROC), retrieval (mAP@ 50), and localization (mLA). No is the number of seen attributes.
mAUROC
MAP@50
mLA
N
32
64
96
32
64
96
32
64
96
A-LAGO
0.600
0.612
0.627
0.173
0.180
0.222
0.782
0.787
0.795
A-ESZSL
0.626
0.614
0.632
0.223
0.200
0.234
0.756
0.769
0.756
Our ZSLA
0.689
0.704
0.717
0.320
0.327
0.329
0.846
0.860
0.867
of our synthesized attribute detectors, we create 160 classes in a-CLEVR, where each class has 30 images; 80 classes are set as seen data, and the other 80 are set as unseen data. For GZSL inference. testing images from both seen and unseen classes are used.
Baselines. As our task of ZSL on attributes for dataset annotation is novel, there is no prior work that can directly make comparison with. However, the task of ZSL attribute has hicrarchy between attributes and base attributes, we adapt two representative methods of zero-shot classification (which explicitly have the class attribute hierarchy behind their formulation) be our baselines, by using the analogy between two hierarchies (i.e. our attribute-bass attribute versus their class- attribute). These two baselines are ESZSL [32] and LAGO singleton [ (note that both of them realize classification with the help of attribute prediction), in which we particularly rename their adaptions our scenario of ZSL on attributes as A-ESZSL and A-LAGO respectively for avoiding confusion. Details of implementing baselines are provided in Appendix C.3
Note that, in the following experiments, both the baselines and ZSLA use the additional ground-truit of attribute locations (i.c. knowing where an attribute appears on the image) provided by CUB to train the seen attribute detectors, unless stated otherwise.
Limitation and discussion. Our proposed method (ZSLA) tackles zero-shot learning for unseen attributes and enables the efficient attribute annotations when constructing the new datasets (later shown in 4.c as long as the attribute format can re-factored (e.g. blue wing) for assembling novel attributes via the attribute format restricts the direct application of our ZSLA on some ZSL datasets (e.g. AWA2 and SUN datasets), the particular attribute format adopted in our ZSLA actually better fits the human's 's intuition for describing the discriminative attributes for the fline-grained" classes. Hence, we would not see this format requirement on attributes the limitation of our proposed method but the restriction of the mainstream ZSL datasets for not having the fine- nne-grained attributes.
approach. Although such requirement on the
4.1 Evaluation of Unseen Attributes
We design three schemes evaluate the quality of the synthesized novel attribute detectors learnt by ZSLA: (1) Attribute Classification. Based on ground-truth attribute annotation of the test images (note that each image typically has multiple attributes), we measure the performance of our synthesized attribute detectors on recognizing their corresponding attributes in the test images. We adopt the area under receiver operating characteristic (AUROC) as our metric for the classification accuracy of each attribute, and we report the average over AUROCs (denoted as mAUROC) of all synthesized attribute detectors; (2) Attribute Retrieval. We rank the test images according to their image-wise responses as to given attribute detector, to simulate the application scenario of retrieving the images which are most likely to own the target attribute from an image set. Note that the image-wise response is computed by max-pooling over the responses of patch-wise image features with respect to the attribute detector. For each attribute detector we compute the average precision (AP) of its top 50 retrieved images, and report the average AP (denoted as mAP@ (0) of all detectors as the metrie; (3) Attribute Localization. As in CUB the ground-truth locations that an attribute appears the test images are available, we introduce the localization accuracy (LA) to measure how well the location having the highest response to an attribute detector matches with the ground-trute ones (counted as correct if they are located on the same neighboring patches). We average over the LA of each attribute as the metric (denoted as mLA).
Table I summarizes the performance in terms of mAUROC, mAP@50, and mLA obtained by baselines and 7SLA, with the number N of seen attributes A set as (32, 64, 96). It is clear to see that ZSLA provides superior performance in comparison to the baselines all the settings of a
7

---

Queri Red Back
Gueny: Forchoad
Black
Queryy:
Gueny Drey Wing
Quary: Blae Uppprrarts
Quans Brown Upperital
White Dreatt
Query: Black Throat
Figure 3: Examples of attribute retrieval and localization. Each set shows the top-5 retrieved images and their response maps for a synthesized novel attribute, where the images marked with red borders are the false positives according to CUB ground-truth.
Base- Rese-ttributes:
Base Uppertas
Dase-strrbatr: Leg
Basc wwtrrtttttteee
Dase wttribtte:
Gsse-atribuut:: Tellow
Rase-attrtbute:
Base-atrrbutt:
Figure Examples showing the retrieval and localization ability base attributes. Each set shows the top-5 retrieved images and their corresponding response map for base attribute representation (extracted by applying our intersection operation on seen attributes detectors).
and evaluation schemes, particularly the localization accuracy. Moreover, by using merely 32 seen attributes to perform the synthesis of novel attribute detectors. ZSLA can achieve comparable results with the baselines of using 64 or 96 seen attributes. Qualitative examples for showing the results of attribute retrieval and attribute localization for the novel attributes synthesized by ZSLA are provided Figure 3 Besides these quantitative and qualitative results demonstrating the efficacy 7SSA on novel attributes, we also provide some qualitative examples in Figure to showcase the localization and retrieval ability of our base attribute representations extracted from the seen attribute detectors.
4.2 Automatic Annotations for Learning Generalized Zero-Shot Image Classification
To further access the quality of our synthesized attribute detectors, we adopt the 32 seen attribute detectors and the 207 novel attribute detectors (i.e. = _2=2007 learned by ZSLA to re-annotate the attribute labels for the whole CUB dataset to simulate the labeling process during constructing new dataset, and name the resultant new dataset "8-CUB". Then we adopt 5-CUB train and
evaluate four representative GZSL algorithms, i.e. ALE . ESZSL (32. CADAVAE [33], and
TFVAEGAN 26 using the settings proposed by (38 (i.e. for 8-CUB and CUB, training with samples from the 150 seen classes, then evaluating the performance on all 200 classes including the 50 unseen
ones). Note that, the class-attribute matrix, which shows the composition of attributes for each class and is needed for GZSL (i.c. the semantic information of classes), is computed by the statistics in

---

Table 2: Experiments results of training and evaluating four representative GZSL methods (i.e. CADAVAE, TFVAEGAN, ALE, ESZSL) on the datasets built upon different sources of attribute annotation (e.g. manual annotation given by original CUB dataset, and re-annotation provided by ZSLA or baselines). As for the columns, S and U represent the accuracy on seen and unseen classes respectively, while H represents the harmonic mean of S and U. The highest scores marked in bold red, while the second-highest ones are marked bold blue. Particularly, we encourage the readers observe the relative improvement/gain (in terms of harmonic mean) produced by various approaches/settings with respect to the results obtained by using 32 manually-labelled seen attributes for GZSL (i.e. the results on the blue-shaded row for CUB dataset), where the superior gain contributed by our propose ZSLA well demonstrates its practical value of automatically producing high-quality annotations on the novel attributes.
CADAVAE
33
TFVAEGAN
26
ALE
II
ESZSL
32
S
U
H
U
H
U
H
U
H
Manual
42.9
27.3
33.4
45.5
31.2
37.1
26.4
9.2
13.7
29.8
10.8
13.9
(N*=32 for CUB)
Manual (N*=312 for CUB)
53.5
51.6
52.4
64.7
52.8
58.1
62.8
23.7
34.4
63.8
12.6
21.0
A-LAGO
45.4
55.4
49.9
57.4
53.0
55.1
51.8
27.2
35.6
49.7
17.1
25.4
A-ESZSL
41.5
48.7
44.8
56.0
48.5
52.0
49.7
17.1
25.4
61.3
9.2
16.0
Our ZSLA
50.3
56.4
53.2
59.0
55.9
57.4
52.4
27.5
36.1
65.1
16.4
26.2
<N*= 2 for 5-CUB)
6-CUB. Similarly, we also use A-ESZSL and A-LAGO baselines to re-annotate CUB dataset and perform GZSL under the same aforementioned setting. The results related to ZSLA and baselines are summarized in the row shaded by the orange color of Table 2 Moreover, on training the four GZSL algorithms by using only 32 attributes or using all 312 attributes obtained from the original CUB dataset as the semantic information, where their results are summarized in the rows shaded by the blue and green color of Table 2 respectively.
additionally experiment
From the results, we observe that using 8-CUB for training, where our ZSLA automatically annotates all the attribute labels, can largely benefit the performance of GZSL algorithms. By treating the harmonic mean over the accuracy numbers on both seen and unseen categories as the metric for GZSL, 8-CUB is superior to those datasets annotated by baselines or even the one using manual annotations. Specifically, the gain obtained by using our d-CUB with respect to the setting of using 32 manually-labeles attributes (i.c., the blue-shaded row of Table 2 demonstrates the practical value of our proposed problem scenario of ZSL on attributes: Without additional cost for collecting annotation, we provide more attribute labels via synthesizing novel attribute detectors from the seen ones, and thus different categories can be better distinguished by more fine- grainnd/ddetaleed attribute-based representations. Moreover, regarding the results that our automatic re-annotation leads to better performance than the manual one (i.e., the green-shaded row of Table (2), believe that this mainly due to the biased semantic information caused by noisy labels stemming from the inconsistency between different human annotators when building CUB dataset. In comparison, our attribute detectors can produce consistent attribute annotations as we use the same set of attribute detectors for labeling all images; it eventually contributes more suitable semantic for learning zero-shot classification. We provide more discussions on such issues in Appendix C.I
4.3 Robustness Against Noisy Attribute Labels
Due to the preference bias among different annotators mentioned in Section 4.2, it is hard to obtain perfect seen attribute labels for training. Thus, is interesting to discuss the effect of the noisy level of seen attribute labels (used for training) on the final annotation quality produced by different auto- annotation methods. Thus, we conduct the controlled experiments using a-CLEVR to understand the effect of the noisy labels. In detail, we adjust the amount of noisy labels for the analysis. To measure the performance drop caused by noisy seen attribute labels, we define the wrong attribute label rate (abbreviated WALR) to represent the noisy level of attribute labels. For example, when WALR is set to 0.3, any toy brick in the training images has a 30% chance of inaccurately annotating (e.g., a blue cube is annotated
red sphere).
As the mAUROC curves shown in Figure ((a) we can observe that: (1) ZSLA outperforms the baselines in attribution classification, no matter how noisy the training data is; (2) the performance

---

MAUROC
Curver Average hermanic the GZSL algarshims
2.914
71.2%
0.9
0.869
07.155
0.818
0.034
80
6%
0.8
harmonicc mea
50
40. .5%
atcurass
64 44.4%
0.7
0.608
0.639
04424
32
13.7%
0.595
Manual annotation
0.6
Our ZSLA
0.500
Our ZSLA
A-LAGO
0.524
A-LAGO
13.9%
A-ES2SL
20
A-E5ZSL
0.5
0.1
0.3
0.5
0.5
WALR
WALR
Figure 5: (a) Left: Evaluation (in terms of attribute classification, with mAUROC as the metric) on the robustness against noisy attribute labels for various methods which learn to synthesize the novel attributes. (b) Right: Evaluation on the quality of automatic re-annotation produced by different methods, where the performance is based on the average harmonic mean of four GZSL algorithms using the re-annotated attributes (cf. the last paragraph of Sec. 4.3 for details). Noting that each curve reports the average performance over 5 runs of different noisy label sets (as the label noise is randomly injected), while the shaded bands around each curve represent the 95% confidence interval.
drop of ZSLA with respect to WALR is much smaller than that of the baselines; (3) baselines have a larger variance than ours, i.e., they even the noise level is the same. The observations prove the robustness of ZSLA against the noisy labels of training attributes. We also show in Figure 13 in Appendix that mAP@50 50 and mLA (for attribute retrieval and location) have a similar trend as mAUROC.
sensitive to different combinations of the noisy labels
Moreover, similar to the experimental setting Section 4.2 thesized by different methods under various WALR settings, to automatically re-annotate the dataset. The resultant dataset is used for learning four GZSL algorithmss (i.c., CADAVAE, TFVAEGAN, ALE, and ESZSL). The average of their harmonic means is reported in Figure 5(b) We can observe the superior quality in terms of automatic e--annttation produced by our ZSLA (i.e., the red curve) compared to the other baselines (i.e., the blue and green curves for A-ESZSL A-LAGO, respectively) under all WALR settings. Specifically, attribute labels for the dataset while maintaining the corresponding WALRS (i.e., the purple curve). It leads to a similar observation we find in the CUB dataset. Once WALR is high (i.e., quite noisy labeling), the performance of GZSL. algorithms trained with the semantic information provided by 7SLA (i.e., the red curve) becomes superior the trained with the noisy manual labels.
we use the novel attribute detectors, syn-
also simulate the situation where humans annotate all
5
Conclusion
This paper proposes new method of developing zero-shot learning novel attributes to reduce the attribute annotation cost for constructing a zero-shot classification dataset. By leveraging the trained detectors of seen attributes, our model learns to decompose them into base attributes to further synthesize novel unseen attributes by reassembling pairs of base attributes. Experimental results show that our method is able to exploit the information embedded in the seen attributes to generate high-quality unseen attributes, validated by various evaluation schemes for attribute classification, retrieval, and localization. We also demonstrate that the semantic information based on our automatic re-annotation beneficial for the GZSL task.
Acknowledgement. This project supported by NSTC (National Science and Technology Council, Taiwan) 111-2636-E-A49-00 1111F-000--12-MMY We are grateful to the National Center for High-performance Computing for computer time and facilities.
100-221-EE-A49-0666 and 109-
10

---

References
[1] Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for attribute-basse based classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
[2] Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-gres mee-grained image classification. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[3] Yuval Atzmon and Gal Chechik. Probabilistic and-or attribute grouping for zero-shot learning. In Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
[4] Yuval Atzmon, Felix Kreuk, Uni Shalit, and Gal Chechik. A causal view of compositional zero-shot recognition. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[5] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In European Conference Computer Vision (ECCV), 2018.
[6] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[7] Pierre Comon. Independent component analysis, a new concept? Signal processing, 1994.
[8] Nathalie Delfosse and Philippe Loubaton. Adaptive blind separation of independent sources: deflation approach. Signal processing, 1995.
[9] Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli dizler-Cinbis Zero-shot object detection by hybrid region embedding. arXiv preprint arXiv:1800 1e0..0011..000000 057.06667770007 2018
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[11] Ian Goodfellow, Jean Douget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
[12] Dat Huynh and Ehsan Elhamifar. Compositional zero-shot learning via fine-grained dense feature composition. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[13] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[14] Charles Kemp, Joshua B Tenenbaum, Thomas L Griffiths, Takeshi Yamada, and Naonori Ueda. Learning systems of concepts with an infinite relational model. In AAAI Conference on Artificial Intelligence (AAAI), 2006.
[15] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[16] Elyor Kodirov, Tao Xiang, and Shaogang Gong, Semantic autoencoder for zero-shot learning.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[17] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[18] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.

---

[19] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, Attribute-basse classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2013.
[20] Yonglu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-object compositions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11313-11322, 2020.
[21] Massimiliano Mancini, Muhammad Ferjad Naeem, Yonggin Xian, and Zeynep Akata. Open world compositional zero-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[22] Tomas Mikolov, Hya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NeurIPS), NIPS' 13, page 3111-3119, Red Hook, NY, USA,
2013. Curran Associates Inc.
[23] Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[24] Muhammad Ferjad Nacem, Yonggin Xian, Federico Tombari, and Zeynep Akata, Learning graph embeddings for compositional zero-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[25] Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attribute- object compositions. In European Conference on Computer Vision (ECCV), 2018.
[26] Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snoek, and Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. In European Conference Computer Vision (ECCV), 2020.
[27] Daniel N Osherson, Joshua Stern, Ormond Wilkie, Michael Stob, and Edward Smithh Default probability. Cognitive Science, 1991.
[28] G. Patterson and J. Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[29] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, 1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14111666 URLlattss://adaattollog org/
D14-1162
[30] Senthil Purushwalkam, Maximillian Nickel, Abhinav Gupta, and Marc'aurelio Ranzato. Task- driven modular networks for zero-shot compositional learning. In IEEE International Confer- ence Computer Vision (ICCV), pages 3592-3601, 10 2019.
[31] Shafin Rahman, Salman Khan, and Fatih Porikli. Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts. In Asian Conference on Computer Vision (ACCV), 2018
[32] Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach zero-shot learning. In International Conference on Machine Learning (ICML), 2015.
[33] Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. General- ized zero-and few-shot learning via aligned variational autoencoders. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[34] Ivan Skorokhodov and Mohamed Elhoseiny, Class normalization for (continual)? generalized zero-shot learning. arXiv preprint arXiv:2006. 1328, 2020.
[35] C. Wah, S. Branson, Welinder, P. Perona, and Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001 California Institute of Technology, 201
12

---

[36] DeLiang Wang. Time- freefreuy masking for speech separation and its potential for hearing aid design. Trends amplification, 2008.
[37] Yinduo Wang, Haofeng Zhang, Zheng Zhang, and Yang Long. Asymmetric graph based zero shot learning. Multimedia Tools and Applications, 2020.
[38] Yonggin Xian, H. Christoph Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning- a comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2018.
[39] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5542-5551, 2018.
[40] Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generating framework for any-shot learning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[41] Wenjia Xu, Yongqin Xian, Jiumiu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. arXiv preprint arXiv:2008. 08290, 2020.
[42] Ozgur Yilmaz and Scott Rickard. Blind separation of speech mixtures via time-frequency masking. IEEE Transactions on signal processing, 2004.
[43] Yunlong Yu, Zhong Ji, Jungong Han, and Zhongfei Zhang. Episode-based prototype generating network for zero-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[44] Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. Learning multi-attention convolutional neural network for fine-grained image recognition. IEEE International Conference on Computer Vision (ICCV), 2017.
[45] Ye Zheng, Jiahong Wu, Yonggiang Qin, Faen Zhang, and Li Cui. Zero-shot instance segmenta- tion. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
13


Reviews:
[{"review_id": "XiCbNtiISdv", "reviewer": "Reviewer_Vc5a", "summary": "The paper proposes an approach for estimating new detectors of visual attribute conjunctions on images by unmixing its parameters and recombining them. The attribute detector parameter is a prototype in deep feature space that localizes the attributes by cosine similarity. The recombination is produced by averaging the detector parameters. The new attribute synthesis can be used to solve Zero-Shot Classification (ZSC) problems when only few attribute annotations are available in the learning dataset by augmenting the annotations with new synthesized attribute detectors. The approach is compared to 4 other ZSC algorithms on the CUB dataset.\n", "questions": "- When reading in the supplementary (C.2) the way the unseen categories are described in $\\delta$-CUB dataset from a series of images, it seems to me that what is actually evaluated is more a “few-shot”  than a “zero-shot” learning scheme since images are needed to compute the description: isn’t there a bias when comparing to “pure” GZSL approaches which have to deal with noisy attribute description of unseen categories?  \n- I didn’t understand what is the input of the network computing the intersection : is it just an attribute label? If so, how is it encoded? Or is it the detector parameter $m$? \n- Exploiting a VIT architecture seems to me oversized given the complexity of the dataset. Why such a model? \n- The question of building atomic attributes from their conjunctions reminds me of the classical question of blind source separation in signal processing: is there any similarity to that problem?\n", "limitations": "Non applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Addressing knowledge transfer at the level of attribute description for classifier design is an interesting idea.\n- Rather detailed experiments (distributed b/w main paper and supplementary material) although on a single small dataset.\n\nWeaknesses\n- The approach is validated on a single dataset (CUB), limiting its potential usefulness.\n- A lot of material is provided in the main paper and supplementary material (17 pages!) but in a confusing organization. For instance, the main algorithm (“intersection”) is not fully described in the main paper, and we need to search for the information in the supplementary material. Wordy general discussions could be replaced fruitfully by more specific technical details.\n- The Zero-shot character of the proposed approach seems to me over-stated, and thus misleading: the new attribute synthesis looks like an unmixing/mixing problem, and the unseen category description with the new attributes in fact requires images, which breaks the “zero-shot” principle, as far as I understand it. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T7cIWpY4d-5", "reviewer": "Reviewer_2STe", "summary": "The paper proposed to train the model by decomposing overlapping concepts from labeled annotations and then combining them to create new concepts. After training, the model can be used to provide new annotations which can be further used for supervised training. Experimental analyses are provided to explain the results and show improvement over compared methods.", "questions": "1) My biggest concern is regarding the generalizability of the approach. Taking Figure 1 for example, the model is basically requesting the training data and testing data to be limited to a scope of linear combinations of the two concept sets \"red, blue, green\" and \"head, wing, breast\". Would the model totally fail when a new concept is introduced, like either \"yellow\" or \"leg\" is they have not been seen during training? That is, is the model \"simply\" decomposing the existing concepts without really generalizing to real unseen concepts. As the analogy in the title, if the model can really \"make an omelette with breaking eggs\", that would be generalizing to a relevant concept of omelette from egg, not just different combinations of \"brown, white, dark\" and \"shell, yolk, membrane\" under the realm of egg.\n\n2) How is the proposed model related to multi-task image-segmentation models? For example, if we have an encoder-decoder model with different prediction heads for colors and parts, that model should also be able to disentangle two types of attributes and predict accordingly for unseen combinations, and it can even interpolate between attributes by, e.g., providing soft predictions. How is the proposed model better in that sense?\n\n3) I'm not too familiar with the CUB dataset, so it might be helpful to provide SOTA results on CUB for direct comparisons.", "limitations": "The authors addressed the limitations in Section 4.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The intuition and motivation are well demonstrated and easy to understand. The results on given comparison settings are reasonable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q4oUV-13MPm", "reviewer": "Reviewer_2LT8", "summary": " This paper has proposed a method named zero-shot learning for attributes to deal with a research problem about novel attribute classification and attribute labeling. Specifically, this research problem is tackled by first decomposing the seen attributes into basic attributes, and then recombining these basic attributes into new ones. Experiments are conducted on the CUB and α-CLEVR (a synthetic dataset) datasets for empirical evaluation.", "questions": "1. In Sec. 4.2, the four representative GZSL algorithms are a bit archaic, with the earliest being 2013 and the latest being 2020.\n2. There is a problem with the case of the title in the references, for example, in Ln. 390, \"Clevr\" should be \"CLEVR\".\n3. Several NeurIPS conference papers are incorrectly used in the journal paper format, e.g., [6], [9] and [10].", "limitations": "In Ln. 257-265, the authors discuss the limitation of the proposed approach that the format of the attribute annotation must be \"adjective + object part\", e.g., blue wing. However, some ZSL datasets are not labeled with attributes in this form, such as AWA2 and SUN.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: This paper proposes a novel research problem about novel attribute classification and attributes labeling, which is tackled by zero-shot learning for attributes .\n\n- Quality: In the paper, Sec. 3 details the training of visible attribute detectors with the decomposition-recombination strategy used to synthesize new attribute detectors, providing theoretical support for ZSLA.\n\n- Clarity: The paper is easy to follow and the figures on the paper are straightforwards.\n\n- Significance: The zero-shot problem of attributes is presented and solved, while the proposed method can be used for automatic labeling of attributes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Hh4vaDi34TT", "reviewer": "Reviewer_RG1h", "summary": "Motivated by the issue of annotation efforts on attribute labels, which are required for zero-shot learning, this paper develops methods to automatically annotate novel attributes for a dataset. Given seen attributes, the proposed method can detect the unseen attributes via a decompose-and-reassemble manner. Results are demonstrated using the CUB dataset alone. As the task of ZSL on attributes is new, most experiments are specified by the authors.", "questions": "As discussed in the limitation of the paper, would the proposed method only work for fine-grained and specific datasets? If it does not work for general, coarse-grained ones like AWA2 and SUN, does it imply this approach has quite limited applications?", "limitations": "Yes, the authors have provided a paragraph in the beginning of the experimental section to discuss the limitation of the proposed method.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem addressed in this paper is new: automatically annotating unseen attributes for zero-shot learning. While the problem is interesting, is it important to drive the ZSL research? Attributes are among one type of semantic information that relate seen and unseen classes. Besides attributes, word embeddings of labels (via word2vec or BERT) can also be used, which can be learned via unsupervised learning and do not require manual annotation. The experiments do not compare the proposed semantic representations (which requires supervision though) to those unsupervised approaches.\nAlthough the proposed decompose-and reassemble approach is fairly sounded, it is not fully validated by the experiments. The experiments were conducted on only one dataset (CUB), which is only one of the five popular datasets in ZSL benchmarks. CUB is fine-grained and challenging, but others (e.g., AWA2, SUN) have different characteristics and are also important.   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept