Attribute Prediction as Multiple Instance Learning
Authors: Diego Marcos
Aike Potze
Wenjia Xu
Devis Tuia
Zeynep Akata
Venue: Accepted by TMLR ()

Abstract: Attribute-based representations help machine learning models perform tasks based on human understandable concepts, allowing a closer human-machine collaboration. However, learning attributes that accurately reflect the content of an image is not always straightforward, as per-image ground truth attributes are often not available. 
We propose applying the Multiple Instance Learning (MIL) paradigm to attribute learning (AMIL) while only using class-level labels. 
We allow the model to under-predict the positive attributes, which may be missing in a particular image due to occlusions or unfavorable pose, but not to over-predict the negative ones, which are almost certainly not present. We evaluate it in the zero-shot learning (ZSL) setting, where training and test classes are disjoint, 
and show that this also allows to profit from knowledge about the semantic relatedness of attributes. 
In addition, we apply the MIL assumption to ZSL classification and propose MIL-DAP, an attribute-based zero-shot classification method, based on Direct Attribute Prediction (DAP), to evaluate attribute prediction methods when no image-level data is available for evaluation.
Experiments on CUB-200-2011, SUN Attributes and AwA2 show improvements on attribute detection, attribute-based zero-shot classification and weakly supervised part localization.

Published in Transactions on Machine Learning Research (08/2022)
Attribute Prediction as Multiple Instance Learning
Diego Marcos
Ciego..marcosiinriio..
Inria, Pronce
Aike Potze
aike, poize Owur nl
Wageningen University, The Netherlands
Wenjia Xu Chinese Academy of Sciences, China
nuwerjjia Gimmalls www.:ad.dwww.s.ccc...
Devis Tuia EPFL, Switzerland
devis. devis.twiaee Seiaaepf..c
Zeynop Akata
University of Tübingen, Germony
Reviewed on OpenReview: https: opwp://veepeeeeee.e.mmmm net/ forum? id= id=nmFezdJtc2
Abstract
Attribute-baser representations help machine learning models perform tasks based on hu- man understandable concepts, allowing a closer human-machine collaboration. However, learning attributes that accurately reflect the content of an image not always straightfor- ward, as per-image ground truth attributes are often not available. We propose applying the Multiple Instance Learning (MIL) paradigm using class-level labels. We allow the model to under-predict the positive attributes, which may be missing in a particular image due to occlusions or unfavorable pose, but not to over-predict the negative ones, which are almost certainly not present. We evaluate it in the zero-shot learning (ZSL) setting, where training and test. classes are disjoint, and show that this also allows to profit from knowledge about the semantic relatedness attributes. In addition, we apply the MIL assumption ZSL classification and propose MIL-DAP, an altribute-based zero-shot classification method, based on Direct, Aitribute Prediction (DAP), to evaluate attribute prediction methods when no image-level data is available for evaluation. Experiments CUB-200-2011. SUN Attributes and Aw show improvements on attribute detection, attribute- based zero-shot classification and weakly supervised part localization.
attribute learning (AMIL) while only
1 Introduction
Semantic attributes provide a rich, human understandable representation that allows to connect the visual world with other sources of knowledge, such as text corpora and knowledge graphs (Rohrbach al.
2010
This enables automatic reasoning that uses set of pre-defined semantic elements (Hudson & Manning 2019
allowing much closer collaboration between these systems and humans (Branson et al. [2010). Attribute-
based representations have proven useful for domain adaptation (Han et al. 2014 Chen et al. 2015
Gebru
ct al. 2017). natural scene perception MMarcos et al. (020)), image-to-image translation (Wu et al. 2019
image retrieval (Siddiquie et al.) 2011 Felix el al.) 2012), and zero-shot learning (ZSL) (Lampert el al. 2013). Attribute annotations are often obtained by asking human annotators to either provide image-level attributes (Patterson & Hays] 2012 Wah et al.) 2011], which is labour intensive and requires all attributes, or least a few (Sun al.] 2015). be annotated in every image, class-level attribute labels (Lampert)
et al.] 2013), requiring many annotations there classes. An alternative approach that requires

---

Published in Transactions on Machine Learning Research (08/2022)
manual annotations is to use class ontologies such as WordNet (Al-Halah & Stiefelhagen 2015 corpora AAl-Halah & Stiefelhagen 2017 in order to mine class-level attributes.
text
No "red leg"
False positives not allowed
Standard BCE loss
"Red leg"
Noise robust loss
False negatives allowed
0
Predict score for "Red Leg"
We formulate the task of attribute prediction as a dataset-wide MIL framework, where some classes may contain given attribute (positive bag) and some must not contain it (negative bag). We suggest treating the positives differently during training by allowing false negatives using noise robust loss.
Figure 1:
These class-level attributes inform us about the attributes that are potentially present in an image of a given class, but do not help us in knowing whether the attribute is actually visible on specific image. For instance, we know that an image contains an Arctic tern or a Red-legged kittiwake (seen in Figure D. we can conclude that it possible, although not certain, that the image contains red legs, since they may be occluded, for instance. On the other hand, knowing the class also provides us with information about attributes that will surely not be present: if the image is labeled as Red-legged kittiwake, can almost certainly conclude that there will be no yellow legs visible. We propose to this asymmetry between positive and negative class-level attributes to improve image-level attribute prediction.
Our solution is based on the Multiple Instance Learning (MIL) DDetteriihh et al., 1997: Maron & Lozano- Pérez 1998) paradigm, i.e. weakly supervised learning in which samples are distributed in bags and labels are provided only the bag level. A bag labeled as nogative means that all the samples contains are negative; bag labeled as positive means that at least one sample in the bag is positive. We build our bags using: 1) images with single class labels, e.g. animal species, and 2) class-level attribute annotations in the form of a binary matrix representing which attribute is possibly present in each class. Using these two elements, we cast attribute learning as multi-label MIL problem, or MIML (Zhou et al. 2012). In contrast with previous approaches, in which the size of the bags is limited by their memory footprint, ours can handle bags of unlimited size by applying noise robust loss to the positive bags and a standard binary cross entropy loss the negatives (Figure D.. We show that this approach is also particularly suitable for injecting additional external knowledge the form of attribute relatedness.
Under the assumption that only class-level attribute annotations available, run into the problem of evaluating the quality of the attribute predictions without resorting to image-level attributes. To solve this issue, we propose MIL-DAP, an attribute- brasb ZSL method that allows to evaluate attribute prediction quality without image-level labels. We test the proposed Attribute MIL (AMIL) approach on three attributo datasets (CUB-200-2011 SUN Attributes and Aw and consistently show that we are able to improve the performance on attribute detection, seribuueebasser zero-shot classification and weakly supervised part localization.
2 Related works
MIL in computer vision. MIL approaches often build each bag using all the patches or region proposals stemming from an individual image: the label is provided at image-level but the desired solution is at patch-
level. This has boen applied to weakly supervised object detection WWang & Forsyth, 2009 Wan et al. 2019). semantic segmentation (Pathak al. 2015), attention-basee classification (Ilse et al. 2018)) and Shro-Shot
Learning (7ST.) (Rahman & Khan 2018). When accounting for multiple attributes, MIL. not only results
2

---

Published in Transactions on Machine Learning Research (08/2022)
in multiple overlapping bags, setting called Multiple Instance Multi-Label setting, MIML Zhou et al. 2012), but, in addition, each bag spans almost the entire dataset, since for each attribute the positive and negative bags jointly contain all classes and thus the whole dataset. This setting is not appropriate for traditional MIL approaches, since these tend to require performing computations over all instances within a bag, thus practically limiting the bag size. possible solution is to approximate the MIL function by building bags at the batch level (Huang et al. 2014; Feng & Zhou 2017), alleviating the memory issuc, or to use dynamic programming to make the problem more tractable (Pham et al.) 2017). Our proposed AMIL implicitly keeps track of dataset-wide statistics by applying noise robust loss to the positive attributes that penalizes false negatives less strictly than false positives, thus deciding on whether or not the positive labels should be enforced by comparing them to the current decision boundary while allowing the use of batch-based methods.
Learning with noisy labels. Some approaches use auxiliary information, such as a small set of clean labels, to model the relation between noisy and clean labels for class correction (Vahdat, 2017) or to learn a ro-weighting of the noisy samples minimizing a loss on the clean samples (Ren et al. 2018 Zhang et al. 2020). Other works use the current predictions of the model to find wrongly labeled samples. They aim at reducing the high loss imposed instances confidently misclassified by the model, either by weighing the loss with the confidence of the model (Mnih & Hinton 2012), lincarly combining the label with the current prediction (Reed et al.) 2015 or by making the penalization of large errors tunable to adapt to different noise levels (Zhang & Sabuncu 2018). In AMIL, we adapt the latter family of methods to the MII. setting by only applying them to the positive bags (i.e. in the positive bag the attribute can either be present or not, in the negative we know for sure that the attribute is absent).
Attribute-based ZSL. Zero-shot learning (ZSL) aims at classifying images into classes that are not seen during training [ampert et al. 2009). Attributes often act as class embeddings that capture visual sim- ilarities between classes for transferring knowledge learned from the seen classes to unseen classes. Many approaches map the visual feature into attribute space, then learn a compatibility function between the ground truth and predicted attributes (Romera- Paredes et al. 2015 Changpinyo et al. 2016 Akata et al. 2015 Zhang et al. 2017 Xu et al., 2022). However, these methods are optimized for classification and tend to perform poorly on attribute prediction (Akata et al. 2015). Most recent state-of-thhe-at ZSL methods are based on generative approaches that bridge the gap between seen and unseen classes by synthesizing visual samples of the unseen classes (Xian et al. 2018b Narayan et al. 2020). These methods do not output attribute scores and thus are unsuitable for attribute prediction. In order to overcome the issues stemming from the use of class-level altribule descriptions, such as the disregard for intra-class attribute variability and attribute co-occurrence, some methods propose to make the ZSL classification more robust (Jayaraman & Grauman 2014 Zhu et al., 2018) while others propose to directly improve attribute prediction by decor- relating attribute representations (Jayaraman et al. 2014 Chen et al., 2017). In this work, we show that AMIL can further benefit. from an auxiliary loss that correlates related attributes and decorrelates those that
are unrelated.
Direct Attribute Prediction (DAP) Lampert et al. 2013 and its variants (uuuki et al. 2014 distinguish themselves from the rest of 7SL. methods for only training the attribute prediction model, while the classifier is a fixed function of the class-attribute matrix. This makes it, in principle, suitable for assessing the quality of the predicted attributes, unlike more recent ZSL methods that do not include attribute prediction within their pipeline. However, (Lampert et al., 2013) shows that DAP performs less well higher quality image- level trained attributes than on those trained with llaes labels, suggesting that DAP is not a good proxy for attribute prediction quality. To solve this, we propose MIL-DAP, a modified version of DAP that takes the MIL assumption into account, in order to tackle the asymmetric noise induced by using class-level labels. We show that the ZSL results of MIL-DAP are strongly correlated with the quality of the predicted attributes, making it an alternative for evaluating attribute prediction when no image-level attributes are available at test time.
3

---

Published in Transactions on Machine Learning Research (08/2022)
yellow legi red leg black wing
whic syng
black Iataas
white breast
Clats
Clais
Class
Cou
Attribute semantic relatedness matrix, R
Class-atriiute matrix
yellow leg.
yellow log red leg
leg
0
black wing
Liimmm Σ2 Vi Rey)2
black wing
white wing
white wing
0
black
breast
white
f()) E kxxxx
Softmaz())
M red log
predicted attribures q
Class
Class has attribute?
f(x)
yellow log.
Negative bag: strict loss
Jt W niw 7
MI, m
0.29
leg
CBCB (Ams tm)
black
image encoder
Softmar(-)
white wing
Positive bag. noise robust loss
white breast
brown brcast
Lyz (Jms fm)
white
0.16
Input,
HW 2
Σ D2,
Figure For a given attribute, such as "red legs", we divide the whole dataset two bags: instances that never display the attribute (negative bag) and those of classes that may (positive bag). A CNN (left) is trained to predict the presence of the attribute. The images from the negative bag all enforced by strict binary cross-entropy loss (BCE), but those from the positive bag are learned with noise robust (NR) loss that is more lenient towards mistakes. In addition, we exploit the information about the semantic relatedness of the attributes by regularizing the attribute prototypes V such that their Gram matrix closely resembles the matrix of semantic relatedness, R.
3 Attribute Multiple Instance Learning
Following (Zaao et al. 2019) and as shown in Figure 2 (left), given the input image and the CNN encoder g()), we can get the feature tensor fxx E Hxi where H, W, and refer to the hoight, width and channel number. Then the attribute localization module takes image representation f(() and attribute classifiers E TRAx as input, and output the dot-product maps map D E xW encodes the location of the th attribute. Then we normalize D with spatial softmax, M softmaz(D), to derive the attention map M #x for attribute For instance, the attention map M for attribute "red leg" focuses on the leg area, and M for attribute "white breast" focuses on the breast.
f(x) VT RR where each
To predict the presence value for an attribute, we multiply the attention map M back to the dot-product maps D", resulting in an attention weighted feature map, and apply global average pooling as follows:
@o
Mr D.
(1)
HW
We evaluate the attribute prediction results in the ZSL setting, since ensures that class prediction is not being internally used as a proxy for attribute detection.
3.1 The MIL assumption and AMIL
In the weakly-supervised, binary classification-bat based MIL setting, the dataset is divided in bags, such that one bag X ((x1,...; xa): contains instances x, X'. Each instance xj has implicit and unknown
4

---

Published in Transactions on Machine Learning Research (08/2022)
target label ty {0, 1), such that the target label of the bag is
0 + - vj E (1,
(2)
otherwise,
meaning that positive bag must contain at least one positive instance. If the instance labels were known, the bag label could be inferred as t = max
In our AMIL setting, we want to learn the presence of specific attributes in images using only class-level labels, along with prior knowledge about relations between attributes. To accommodate this, we extend the MIML (Zhou al.) 2012) framework to aftribute bags with high overlap between attributes, such that bags can be arbitrarily large. We construct as many pairs of data bags (one positive and one negative) as there are attributes, and each pair of bags will cover the entire dataset. These bags are built using a binary allribule-class malrie indicating which attribule is present in which class. Each positive bag X. is composed by all images of all classes where an attribute can be observed (e.g. for = red leg, by all images of red-logged kittiwakes and arctic terns etc., irrespective of whether the red legs can be seen in a particular image or not, since we do not have this information). Negative bags contain all images of all species where the attribute is known not be present.
Attribute MIL (AMIL). MIL generally builds bags consisting of individual images, where instances are either patches pixels within the image (Carbonneau et al. 2018 Since the instance-level ground truth ij is not available and we have only access to the bag-level label, instance-wise predictions need to be pooled into a bag-level prediction. Handerafted MIL pooling functions, such as the maximum or mean prediction (Wang et al. 2018) have been used, but learnable pooling functions have been shown to perform better [[ise et al. 2018
For bag to accommodate many more instances than a single batch of data, we propose to frame the MIL problem one of asymmetric noise, as depicted in Figure 2 Instead of having MIL pooling function that assigns weight to each instance by directly comparing it to other instances in the bag, done in previous works, we allow the model to be more flexible when predicting positive altributes while making sure that all negative attributes are predicted as negative. This can be achieved by applying losses designed for noise robustncss (NR) to the positive bags while applying a stricter loss, such as binary cross entropy (BCE), to the negative bags. This allows for batch-based optimization without limiting the size of the bags.
Given a CNN f, q= (x) the vector of scores of the predicted attributes for input x belonging class y: such that 4m can be interpreted as p(a,, /x). Given the class-attribute target {0, 13M corresponding to class y, we propose to use a loss function
CAMIL(q.t)
Σ 1=
EMIL(gm: tm)
(3)
such that each element of the loss is
LNn (yms bm), if y has attribute a.m. CBCE (yms tm): otherwise,
Lxau (9m (m)
(4)
where LBCE is the binary cross entropy loss and LNR is some noise robust loss.
3.2 MIL-DAP Model
Direct Attribute Prediction (Lampert et al., 2013 (DAP). Given image and unseen class
y, want to estimate p(y)x) by making use of intermediate attribute representation E AA = {0, M
in two step manner by computing p(y[x) = p(yla)p(alxx By applying Bayes' rule we obtain:
p(yla) p(alu)p(v)
(5)
p(a)
5

---

Published in Transactions on Machine Learning Research (08/2022)
Note that p(a(x) can respectively be factorized as:
p(a(x) II p(am [),
(6)
m=l
where
p am (p(am/x)) (x),
if y has attribute 0m p(amlx), otherwise,
pmam = a dm/xx
=
(7)
and that the same can be done for p(aly) and p(a). By combining these elements into the first expression oblain:
plyxx) = p(y) II plam a (y) 0%1(x)
(8)
p(um am)
Since the class-attribute matrix binary, we have that p(am uniform class priors p(y), obtain the DAP MAP prediction in Lampert et al. 2013):
a% (y)
for all m. If we also assume
p(am a x) II 1=.....l m== p(am am)
(x) arg max
(9)
We propose a modified version of DAP that is consistent with the MIL assumption and that provides a better proxy for evaluating attribute prediction through ZSL, thus removing the need for image-level attributes for the evaluation.
DAP under the MIL assumption (MIL-DAP). We propose three modifications to DAP to make it suitable for estimating attribute quality. First, we observe that under the MIL assumption only the negative values in the class-altribute matrix are considered to be trustworthy, since the positives are interpreted as allowed and not as required. If My is the set of negative attributes associated to class y, the MIL-DAP class probability estimation using Eq.
8
becomes:
[1 - p(um/v)][ p(am|x))
plulx) p(u) II
(10)
--((mm)
mEMY
In addition, we propose to modify the way of estimating both [1 p(am)] and [] - p(amly)). In DAP [1 p(am)] computed using the class-attribute matrix. However, we observed substantially improved ZSL results, particularly with altributes learned with image-level labels, by computing [1 p(am) over the training images. In DAP p(am. a% ly) 1 for all the attributes. Even if we are assuming the negative attribute labels to be cleaner than the positive ones, we do not expect them to be perfectly clean. In our experiments, we allow for up to ~ 10% of noise within the negatives due to the threshold used in creating the class-attribute matrix. we are testing whether x belongs to class y, we can start by assuming that it is indoed the case and then use the observed score as prior:
[1 - p(amaly))) = [1 p(am x)].
(11)
If it turns out that indeed belongs class y, any class in which attribute is not allowed, then we would be using more informed class dependent prior. If the assumption does not hold and x belongs to class in which m is allowed, 1 - p(amlx) will tend to be smaller and thus the wrong class will be further penalized. This brings us to the final MAP formulation of MIL-DAP:
(L- p(am (x))
fMIL-DA(xX) arg max
II
(12)
1=,,, L
p(am)
mE

---

Published in Transactions on Machine Learning Research (08/2022)
Exploiting attribute relatedness. Another source of information that can be used to solve some of the ambignities brought by the weakly supervised setting is the semantic relatedness of the attributes. For instance, if two attributes relate to the same body part or if they describe the same property of different parts (e.g. colors), we may want our model to know that the vector prototype representing these attributes, i.e. the learned weights in the classifier corresponding to those altributes, should be similar. We use this information to build the matrix of semantic relatedness between all pairs of attributes, R E [0, 1]4x4 We propose to encourage the Gram matrix of the set of the normalized attribute protolypes to approach R:
CGram ΣE -
(13)
where V is the protolype, or linear classifier weights, of the jth attribute. This loss will make sure that semantically related attributes will have similar prototype vectors while semantically unrelated ones will tend to become orthogonal.
3.3 Attribute quality evaluation
To assess the performance of AMIL on attribute prediction, we evaluate it on image-level attribute prediction, attribute-basod zero-shot learning with MIL.-DAP and wcakly supervised part localization.
Zero-shot learning. Zero-shot learning aims at recognizing previously unseen classes by transfering knowl- edge from seen classes. Attributes are widely used as the class embodding for zero-shot classification. How- ever, existing ZSL methods (Ronera-Paredes et. al. 2015 Xian et al.] 201 Akata et al. 2015) make use of continuous annotations and thus treat equally the positive and negative class-level attributes for the novel classes. In order to adapt ZSL to the AMIL assumptions, we match the predicted attributes with the ground truth binary attribute of the unseen classes. To ensure that the ZSL results reflect the quality of the at- tributes, we do not use compatibility function to train the network in previous works (Akata et al. 2015 Xu et al. 2020), but simply train the above explained attribute prediction network. At test time, we use the predicted attribute scores and the binary class-level attribute annotation to obtain the class scores with the proposed MII-DAP,, Eq. 12 We also evaluate the altribute quality under a more realistic setting, i.e. generalized ZSL, which aims at recognizing both the seen and unseen classes during inference.
Part localization. Our attribute localization module outputs the attention map Ma E " for each attribute a, where the high-activated area indicates the location of that attribute. We turn this into a weakly supervised attribute localization task by comparing the predicted attribute location with the ground truth. We evaluate part localization in the ZSL setting where the train and test classes are disjoint.
4 Experiments and discussion
We evaluate AMIL using the image-level attribute annotations where available. Then, we evaluate the learned attributes on attribute-based downstream tasks: zero-shot classification and part localization.
Datasets. We use three datasets with attribute annotations: CUB-200-222 201-2011 (CUB), SUN Attribute Dataset (SUN) and Animals With Attributes (AwA2). The continuous valued class-attribute matrices of all datasets are binarized such that values lower than 0.1 are assigned to the negative bags, making sure that they will generally be cleaner than the positive bags. The value of 0.1 was chosen due to being close to the average values of the CUB and SUN matrices (0.114 for CUB, 0.098 for SUN), which are the values typically used in the literature. We observed no effect on the results on CUB and SUN stemming from this change. In all experiments we use the train-lest splits proposed for ZSL in (Xian et al. 2018a) such that the evaluation always performed on unseen classes. CUB (Wah et al. 2011) is a fine-graine birds dataset, with 11,788 images from 200 classes, and 312 attributes. An attribute consist of two parts: the name of an attribute (eleg color" or "tail texture" etc), and its value ("red" or "striped"). We consider two attributes and j to be related, Ra, = 1, if they share one of the parts, and unrelated, Ra, = 0, if they do not. CUB
7

---

Published in Transactions on Machine Learning Research (08/2022)
Image-level attribute ovaluation
Class-level attributc evaluation
CUB
SUN
CUB
SUN
AWA
Method
AMIL
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
u
Image-level
33.79
30.09
80.72
77.98
51.07
47.31
93.42
92.82
63.90
82.50
65.47
86.22
28.07
24.84
75.49
72.66
40.4
38.69
91.46
90.50
63.78
82.33
68.30
89.23
74.04
76.66
Class-level
28.35
25.31
76.22
73.10
40.79
39.00
91.62
90.73
65.30
83.76
68.98
89.45
73.94
76.81
28.01
24.84
75.43
72.75
40.40
38.24
91.49
90.52
63.90
82.42
68.37
89.27
74.17
76.61
Deep-MIML
28.39
25.17
76.44
73.03
40.72
38.95
91.59
90.71
65.31
83.77
68.86
89.29
73.94
76.81
27.49
23.99
74.96
72.19
39.
37.45
91.22
90.25
63.64
81.81
68.06
89.05
74.04
76.84
GCE
28.09
24.76
74.96
72.09
39.47
38.41
91.00
90.35
63.01
81.34
67.58
88.52
73.52
75.48
27.92
25.14
76.30
72.94
39.72
38.85 38.83
91.27
90.50
64.17
81.89
68.25
88.53
73.74
76.34
27.98
24.79
75.34
72.65
39.95
38.31
91.39
90.46
63.88
82.28
68.18
89.21
73.92
76.37
Bootstrap
28.43
25.19
76.13
72.76
41.01
38.81
91.59
90.71
64.55
83,02
68.41
89.13
74.81
76.85
28.68
25.62
77.33
74.43
40.50
38.92
91.50
90.87
65.97
84.33
68.68
89.29
74.79
76.78
Attribute prediction on CUB, SUN and AWA, evaluated with image-level labels and class-level labes. We report average precision (AP) and area under the ROC curve (AUC). AMIL indicates using LAMIL. and G using the Gram loss. GCE (Zhang & Sabuncu, 2018 and Bootstrap (Reed et al. 2015) represent the two LnR we used to test AMIL. Deep-MIME (Feng & Zhou 2017) is an alternative MIL approach.
Table 1:
is annotated with imago-level attributes and body part locations, which we use for evaluation but not for training. SUN (Patterson & Hays] 2012) consists of 14,340 images from 717 scene classes and annotated with 102 scene attributes. Aw (Xian et al. 2018a) contains 37,322 images of 50 animal classes with 85 per-class attribute labels, i.e. no per-inage attributes are available. Since the attributes of SUN and AwA2 do not follow the binomial structure of CUB, we simply apply a decorrelation loss between all attributes.
Noise-robust losses and baselines. In this work use two methods designed for dealing with noisy labels, i.c. Generalized Cross Entropy (GCE) (Zhang & Sabuncu 2018) and Bootstrap (Reed et al.] 2015). GCE is a generalization of of the Eotso-Entropy (CE) and Mean Absolute Error (MAE) losses that aims at finding a trade-off between the noise robustness of MAE and the implicit focus on hard samples of CE, while Bootstrap uses the current model predictions to dilute the ground truth, thus reducing the impact on the loss value of samples for which the model confidently diagrees with the ground truth. We found the truncated version of the GCE loss to work best. With the bootst loss, we obtained the best performance with the soft version (Reed et al. 2015), in which the label is averaged with the current prediction score. In addition we compare with DIML (Feng & Zhou| 2017) which operates in the traditional, batch-wise, MIL setting. We compare each noise robust loss in the standard setting (i.e. applied to all attributes) and the modified version of Bootstrap and GCE that is adapted to MIL. by applying Eq. (4) (GCE+MIT. and Mootstrap+MMLL. As a baseline, we use the standard approach of training directly on the class-level attributes, while the image-level attributes are used to train a model that serves as an upper bound. In order to test the effect of the Gram loss (Eq. 13 proposed appraoches (GCE+MIL+G and Bootstrap+ MIL- MIL++) and in the basclines (Class-level Attr+G and Mre)-MIML+G).
the different methods, apply it both the
Implementation details. In all the experiments use ResNet50 pretrained on ImageNet the image encoder. On all experiments trained, validated hyperparameters and tested the model using the ZSL train-val-test splits recommended in (Xian et al. 2018a). The attribute prediction results are thus evaluated only on unseen classes. All models are trained for three epochs with a multi-label binary cross-entropy loss or a noise robust loss using the binary class-level label with the exception of the image-level trained upper- bound models for CUB and SUN, which use only BCE. We use the Adam optimizer with learning rate of 0.0001 for the attribute prediction base model and 0.001 for learning the last linear layer of the attribute prediction model, with learning rate decay of 0.25 after each epoch.
8

---

Published in Transactions on Machine Learning Research (08/2022)
4.1 Attribute prediction results
We start by comparing the different approaches directly the task of attribute prediction. We report two metrics for multi-label classification: average precision or area under the precision-recall curve (AP) and area under the ROC curve (AUC), both computed per attribute and macro-averages averagede We first show the results of using the image-lovel attributes for training and use it as an upper bound for the llvesl trained models, including baselines, the AMIL approach (+MIL) and the Gram loss (+G).
We first evaluate at test time on the image-level attribute labels for CUB and SUN. All methods are trained on the attributes of the ZSL seen classes provided by (Xian et al., 2018a) and tested on the image-level labels of both seen and unseen classes. We observe from Table (left) that the image-level upper bound does indeed reach a substantially better result than any of the class-level methods. We can also see that the proposed scheme tends to improve the results in all metries, although a large gap to the image-level upper bound is still present. This improvement is much more consistent in CUB than in SUN, where AMIT, performs only marginally better than the class-level baseline (e.g. 38.92 AP versus 38.69 AP in the unseen classes). We suspect this is due to the fact that CUB attributes were created with class discrimination in mind, resulting in more distincl class-level attribute representations that lead to better attribute prediction. This can be seen in the fact that levesl AP results are closer to the image level results in CUB (around 5 AP difference) than in SUN (around 10 AP difference). Although this observation does not apply to AUC, this metric seems to be surprisingly saturated in the case of SUN, potentially obscuring the effect of the class-level attributes quality. The methods for learning with noisy labels (GCE and Bootstrap) perform similarly and close to the class-level and Deep-MIML methods. This may be due to the fact that these methods assume similar levels of noise in the positive and negative bags, when the positive bag are likely noisier. This is supported by the fact that, applying the noise robustness only to the positive bags, according to the proposed AMIL, tends to improve performances across the board, showing that the MIL assumptions are indeed met in class-level attribute prediction on these datasels.
We also test on the binarized class-level labels for all datasets, Table [ (right). Although AMIL also provides an improvement in CUB, results in SUN and AwA2 are more mixed. Most interestingly, the image-level upper bound performs substantially worse than most of the class-level methods, suggesting that evaluating on class-level attributes does not actually reflect the quality of the predicted attributes.
4.2 (Generalized) Zero-shot Learning
We evaluate the attribute prediction models using ZSL and generalized ZSL (GZSL) results as proxy metrios. This can be useful in cases where image-level attribute annotations are not available even at test time. We use MIL-DAP, the proposed hard-coded attribute-based classifier in Eq. {12 and show that it better reflects the quality of the predicted attributes. We would like to emphasize that use ZSL based attribute prediction as proxy for attribute quality and this paper does not aim at improving ZSL. performance over seate-of-the-ars ZSL methods, with most methods mentioned in Section 2 reaching substantially higher ZSL accuracy than what is presented here.
G7SL, results are shown in Table 2 (left). The first interesting observation in CUB and SUN is that all class-level methods perform better than the image-level upper bound on the seen classes, while the upper bound is best on the unseen classes. This suggests that the class-level models tend to learn the attribute signature of each class rather than the actual attributes, which indicates we should only use the unseen class performance, or the harmonic mean, to assess the quality of the attributes. Indeed, as for attribute prediction, AMIL obtains an improvement over the baselines on CUB and SUN, with mixed results on AwA2, e.g. MIL+G improves the harmonic mean of GCE and Bootstrap on CUB dataset by 5.43% and 5.14%.
ZSL results (Table 2 right) show similar trend GZSL on the unseen classes, with the imago-level models for CUB and SUN obtaining the best scores and with AMIL resulting in improved results in most cases, e.g. MII++G improves the accuracy on CUB (by 6.4%) and SUN (by 2.7%) when coupled with GCE.
9

---

Published in Transactions on Machine Learning Research (08/2022)
Generalized Zero-shot learning
Zero-shot learning
CUB
SUN
AWA
CUB SUN AWA
Method
AMIL
G
h
h
h
Image-level
37.73
22.78
28.41
10.06
8.54
9.24
49.68
36.04
59.02
8.70
15.16
13.34
5.63
7.91
93.25
1.00
1.99
36.77
33.61
36.82
Class-level
53.28
11.19
18.50
14.06
5.28
7.67
93.76
0.61
1.22
36.40
33.68
36.67
58.46
8.43
14.73
12.78
5.83
8.01
93.28
0.69
1.36
35.79
33.89
37.63
Deep-MIML
V
53.05
10.92
18.11
13.80
5.83
8.20
93.68
0.73
1.46
35.83
33.47
36.11
49.70
9.03
15.29
10.86
5.42
7.23
93.36
0.71
1.41
30.91
30.42
37.36
GCE
55.29
9.61
16.37
12.74
5.63
7.80
93.04
0.69
1.36
36.33
32.43
37.43
49.33
13.11
20.72
14.02
5.35
7.74
93.09
0.56
1.12
37.31
33.13
36.33
58.76
8.80
15.30
14.02
6.11
8.51
93.36
0.56
1.12
34.98
33.33
37.43
Bootstrap
58.02
10.31
17.51
14.10
5.69
8.11
92.29
0.51
1.02
39.60
33.89
37.68
48.59
12.94
20.44
14.78
6.39
8.92
92.80
0.32
0.63
41.52
33.89
38.27
Table 2: Generalized ZSL and ZSL results using MIL-DAP, on CUB, SUN and AWA. For GZSL report the accuracies the seen (s) and unseen classes (u), as well as the harmonic mean (h) of the two. For ZSL, we train the attribute prediction network with class-level binary attributes and test it on the unseen classes. Accuracy reported as percentage OA. AMIL indicates using LAMIL and G using the Gram loss.
GCE (Zhang & Sabuncu 2018) and Bootstrap (Reed et al.) 2015) represent the two CNR we used to test AMIL. Deep-MIML (Feng & Zhou 2017) is an alternative MIL approach.
CUB Part Localization
AMIL
G
Head Breast Belly Back Wing
Tail
Leg
Avg
Image-level
66.1
67.6
49.3
32.1
37.7
10.5
44.5
44.0
48.1
61.8
55.2
32.0
43.7
1.8
36.2
39.8
Class-level
v
56.1
62.7
62.1
35.5
59.0
2.0
45.5
46.1
48.4
64.2
53.8
33.4
43.5
1.6
35.3
40.0
Deep-MIML
v
56.2
62.1
62.1
34.7
57.2
2.1
46.6
45.9
46.8
59.1
59.1
29.4
36.2
1.5
38.2
38.6
GCE
47.7
56.8
56.5
30.9
46.0
2.0
40.1
40.0
51.0
59.0
62.3
33.4
60.4
2.3
49.4
45.4
49.3
63.7
54.2
32.6
42.8
1.7
34.9
39.9
Boot
46.9
57.9
54.7
31.5
49.5
1.5
33.7
39.4
56.4
56.5
63.2
32.1
55.2
2.9
50.1
45.2
Table 3: The Percentage of Correctly Localized Parts (PCP) in CUB. We report the PCP for seven bird body parts well as the average accuracy. AMIL indicates using LAMIL. and G using the Gram loss. GCE (Zhang & Sabuncu, 2018 and Bootstrap (Reed et al. 2015) represent the two LNR we used to test AMIL. Deep-Min MIML (Feng & Zhou 2017) is an alternative MIL approach.
4.3 Evaluating Part Localization
A better attribute prediction may also result in an improved learning signal for part localization. Since each attribute in CUB dataset is associated with body part ("belly", "wing". etc.), evaluate the part localization ability with the per-image part location annotations. We use directly the attribute attention maps M E ## (Figure 2 left) for localization. We calculate the Percentage of Correctly Localized Parts (PCP) following the approach of (Xu et al.) 2020). The quantitative results on part localization are shown Table 3 where we observe similar trends as the previous tasks. However, there are stark differences between the parts. All methods seem to perform well on "Breast" and "Belly", even reaching image-level performance. This is because breast and belly are large parts that usually occupy the center
10

---

wing Published in Transactions on Machine Learning Research (08/2022)
of the image, making those parts attended to by default, and do not require the model to perform part detection but merely bird detection. The class-level qualitative results in Fig. 3 illustrate the bias towards the center of the bird. We observe that peripheral parts tend to obtain much worse results for the state of the art methods, c.g. the failure at locating "tail", with the highest PCP of 2.9% using Bootstrap+MIL+G. Compared to the other settings, Bootstrap++MiI.. an impressive 50.1% PCP, 14 points higher than the baseline. We hypothesize that this increase likely associated the positive relations in the Gram loss due the synergy of learning multiple "leg" prototypes jointly that are able to guide each other towards the right direction.
able to provide usable results on *leg", obtaining
Orange leg
Yellow
Black Iridescent
Grey leg
Dagger bill
Boot+MIL Class-level Image-leval
Moot-MMI++G Boot+MiL Class-leval Image-level
Figure 3: Qualitative part localization on CUB image displaying the attention maps learned by cach model. The 1t row corresponds to the image-level upper bound results, the 2nd to the class-level baseline, the 3rd and 4h to the proposed AMIL, with and w/o the Gram loss.
4.4 Ablation Studies
Ablation results on the AMIL loss. The main assumption for our work, namely the relatively higher reliability of the negative class-level attributes, is captured by the definition of the asymmetric LAMIL loss in Eq. 44. Tf this assumption holds, we would expect the results of applying only INR or only LBCE to be worse than using LAMIL- The results of applying only LNR, instantiated as either GCE or Bootstrap (Tables D, and LACE, the class-level baseline, show that indeed LAMIL. always results in better image-level attributes prediction than using either LNR or LBCE- This behaviour is clearer when using the Bootstrap loss, and results are more mixed with the GCE loss. Interestingly, we see comparable results for both ablated settings (using only LnR or LBCE) and consistent improvement by applying them separately the positive and negative samples according LAMI. (AMIL column in the tables). A similar trend can be observed in the unseen GZSL results in Table 2 This confirms that the MIL assumption of clean negatives and noisy positives does hold in the studied cases.
A blation study on the interaction between AMIL and MIL-DAP components. We explore the contributions of cach component of the AMIL and MIL-DAP on CUB and SUN. In the first row of Table 4 we can see how each component of MIL-DAP (i.e. the empirical p(a) prior, the MIL assumption by only accounting for negative attributes and the empirical p(aly) prior) results in increase in generalized ZSL performance using the attributes trained with image-level labels. The following three rows show that this trend is absent in the models trained with class-level labels, even when using a noise robust loss (i.e. Bootstrap (Reed et al. 2015)) and the Gram loss, although the latter does result in substantially improved performances in all setting for CUB.
The last two rows show that AMIL, both with and without the Gram loss, results in generalized ZSL performance trends that closely match those of the image-level upper bound. This suggests that the improved
11

---

Published in Transactions on Machine Learning Research (08/2022)
results in attribute prediction obtained by AMIL do indeed result in learning attributes that more closely follow the structure of the image-level attributes. Once again, the application of the Gram loss results in improvements on all settings.
CUB
SUN
DAP
+p(a)
+IL
+p(alv)
DAP
+p(a)
+MIL.
+p(al)
Image-level
3.01
19.75
22.35
28.41
2.18
6.45
6.66
9.24
Class-leva
12.94
14.19
15.85
15.16
8.82
8.90
8.73
7.91
Boot.
Reed al.
2015
13.33
14.15
16.32
15.30
9.48
9.55
9.38
8.51
B
20.90
22.37
22.68
20.44
8.77
9.04
8.53
7.45
Boot.+-MIL
11.26
14.37
16.92
17.51
7.21
8.11
8.21
8.11
+MoL++
12.81
17.67
18.41
20.44
6.56
8.41
8.30
8.92
Table 4: Generalized zero-shot learning results, harmonic mean between seen and unseen classes, on CUB and SUN. Each row presents an additional modification to DAP (Lampert et al.) 2013), with the last row corresponding to MIL-DAP: substituting the attribute prior by an empirical estimate over the training samples (+p(a)), applying DAP only using the negative classes (+MIL) and using the attribute prediction of the (est. image as prior for the altribute given the class (+p(aly)))
4.5 Limitations
Our results show that AMIL and the Gram loss make a substantial difference in the quality of the altributes learned on CUB, where the attributes are all visual and their relatedness well defined, and that MIL-DAP is useful for detecting this improvement the absence of image-level labels. In SUN, where some attributes are subjective, the improvement less marked and in AwA2, where many of the attributes are not visual, the proposed approach seems to result only in a marginal improvement in 7ST and no improvement in GZSL. This suggests that AMIL and MIL-DAP are only appropriate when dealing with clearly visual attributes and that no improvement should be expected for attributes that are subjective or not visual in nature.
5 Conclusion
Despite the recent interest in attribute- based methods for applications such as zero-shot classification and explainability, we have observed a large gap in performance between attribute detection trained with weak class-level annotations, as is often done in the literature due its lower cost of acquisition, and the potential performance that could be obtained by using labour intensive image-level annotations. To close that gap, we propose to leverage the Multiple Instance Learning assumption. We show that applying it to existing noise robustness approaches results in better attribute detection results since it induces tolerance of positive attributes that are not present or occluded on a particular image. Via extensive experiments on three datasets, we show that the proposed approach helps to improve the performance on attribute prediction, and that this improvement can be also be measured via attribute-based tasks such as zero-shot classification and weakly supervised part localization.
References
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedddigg for image clas- sification. IEEE TPAMI, 38(7):1425-1438, 2015.
Ziad Al-Halah and Rainer Stiefelhagen. How to transfer? zero-shot object recognition via hierarchical transfer of semantic attributes. In WACV, 2015.
Ziad Al-Halah and Rainer Stiefelhagen. Automatic discovery, association estimation and learning of semantic attributes for a thousand categories. In CVPR, 2017.
Steve Branson, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pictro Perona, and Serge Belongie. Visual recognition with humans in the loop. In ECCV, 2010.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Mare-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristies and applications. Pallern Recogralion, 77:329-30 -0000-33--00 2018.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In CVPR, 2016.
Chao-Yell Chen, Dinesh Jayaraman, Fei Sha, and Kristen Grauman. Divide, share, and conquer: Multi-task attribute learning with selective sharing. In Visual attributes, pp. 49-85. Springer, 2017.
Qiang Chen, Junshi Huang, Rogerio Feris, Lisa M Brown, Jian Dong, and Shuicheng Yan. Deep domain adaptation for describing people based on fine-grained clothing attributes. In CVPR, 2015.
Thomas Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-paralle rectangles. Artificial intelligence, (911-):31--11, 1997.
X Yu Felix, Rongrong Ji, Ming- Hen Tsai, Guangnan Ye, and Shih-Fu Chang, Weak attributes for large-scale image retrieval. In CVPR, 2012.
Ji Feng and Zhi-Hua Zhou. Deep miml network. In AAAI, 2017.
Timnit Gebru, Judy Hoffiman, and Li Fei-Fei. Fine-grained recognition in the wild: multi-task domain adaptation approach. In ICCV, 2017.
Yahong Han, Yi Yang, Zhigang Ma, Haoquan Shen, Nicu Sebe, and Xiaofang Zhou. Image attribute adap- tation. IEEE Transactions Multimedio, 16(4):1115 1126, 2014.
Sheng-Jun Huang, Wei Gao, and Zhi-Hua Zhou. Fast multi-instance multi-label learning. In AAAI, 2014.
Drew Hudson and Christopher D Manning. Learning by abstraction: The neural state machine. In NewrIPS, 2019.
Maximilian ssse, Jakub M Tomezak, and Max Welling. Attention-basod deep multiple instance learning. In ICML, 2018.
Dinesh Jayaraman and Kristen Grauman. Zero-shot recognition with unreliable attributes. In NewrIPS, 2014.
Dinesh Jayaraman, Fei Sha, and Kristen Grauman. Decorrelating semantic visual attributes by resisting the urge to share. In CVPR, 2014.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In CVPR, 2009.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, Attribute- based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453-44) 465, 2013.
Diego Marcos, Sylvain Lobry, Ruth Fong, Nicolas Courty, Remi Flamary, and Devis Tuia. Contextual semantic interpretability. In Asian Conference Computer Vision (ACCV), 2020.
Odod Maron and Tomás Lozano-Pérez. A framowork for multiple-instance learning. In NewrIPS, 1998.
Volodymyr Mnih and Geoffrey E Hinton. Learning to label aerial images from noisy data. In JCML, 2012.
Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snoek, and Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. ECCV, 2020.
Deepak Pathak, Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional multi-class mul- tiple instance learning. ICLR, 2015.
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR, 2012.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Anh T Pham, Raviv Raich, and Xiaoli Z Fern. Dynamic programming for instance annotation in multi- instance multi-label learning. TPAMI, 39(12):2381 -2994, 2017.
Shafin Rahman and Salman Khan. Deep multiple instance learning for zero-shot image lagging. In ACCV, 2018.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping, ICLR (Workshop), 2015.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust doep learning. In ICML, 2018.
Marcus Rohrbach, Michael Stark, György Szarvas, Iryna Gurevych, and Bernt Schiele. What helps where- and why? semantic relatedness for knowledge transfer. In CVPR, 2010.
Bernardino Romere-Paredes, ENG OX, and Philip HS Torr. An embarrassingly simple approach to zero-shot learning. In ICML, 2015.
Behjat Siddiquie, Rogerio S Feris, and Larry Davis. Image ranking and retrieval based on multi-attribute queries. In CVPR, 2011.
Chen Sun, Chuang Gan, and Ram Nevatia. Automatic concept discovery from parallel text and visual corpora. In ICCV, 2015.
Masahiro Suzuki, Haruhiko Sato, Satoshi Oyama, and Masahito Kurihara. Transfer learning based on the observation probability of each attribute. In International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2014.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In NewrIPS, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The Maltech-UCSD Birds-200-2011 Datasel. Technical Report CNS-TR-2011-001, California Institule of Technology, 2011.
Fang Wan, Chang Liu, Wei Ke, Xiangyang Ji, Jianbin Jiao, and Qixiang Ye. C-MIL: Continuation multiple instance learning for weakly supervised object detection. In CVPR, 2019.
Gang Wang and David Forsyth. Joint learning of visual attributes, object classes and visual saliency. Tn ICCV, 2009.
Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition, 74:15 24, 2018
Po-Wei Wu, Yu-Jing Lin, Cho-Han Chang, Edward Y Chang, and Shih-Wei Liao. Relgan: Multi-domain image-to-inage translation via relative altributes. In ICCV, 2019.
Yongqin Xian, Christoph H Lampert, Bernt Schicle, and Zeynep Akata. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEFF TPAMI, 2018a.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akala. Feature generating networks for zero-shot learning. In CVPR, 2018b.
Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. In NewrIPS, 2020.
Wenjia Xu, Yongqin Xian, Jiunin Wang, Bernt Schicle, and Zeynep Akata. Attribute prototype network for any-shot learning. International Journal of Compuler Vision, pp. 1 19, 2022.
Li Zhang, Tao Xiang, and Shaogang Gong. Learning deep embedding model for zero-shot learning. CVPR, 2017.
14

---

Published in Transactions on Machine Learning Research (08/2022)
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NewrIPS, 2018.
Zizhao Zhang, Han Zhang, Sercan O Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. In CVPR, 2020.
Xiangyun Zhao, Yi Yang, Feng Zhou, Xiao Tan, Yuchen Yuan, Yingze Bao, and Ying Wu. Recognizing part attributes with insufficient data. ICCV, 2019.
Zhi-Hua Zhou, Min-Ling Zhang, Sheng-Jun Huang, and Yu-Feng Li. Multi-instance multi-label learning. Artificial Intelligence, 176(1):2291 2320, 2012.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zero-shot learning from noisy texts. In CVPR, 2018.
15


Reviews:
[{"review_id": "XFmNam5Wsg", "reviewer": "Reviewer_M5yj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. Exploiting the positive bag labels and differentiating positive and negative attributes for zero-shot learning is interesting.\n\n2. ZSL performance is improved by combining with the proposed loss function.\n\nWeakness:\n\n1. The proposed gram matrix loss does not seem to improve under all circumstances. For the method of Deep-MIML I am wondering why there is no comparison of with and without gram matrix loss?\n\n2. Performance improvement is not very obvious for SUN and AWA. Can the author provide some analysis?\n\n3. The motivation of introducing the gram loss is not clear. The semantic relatedness does not guarantee being visually similar. For example, yellow leg and red leg have high similarity, but encouraging the classifier for yellow leg and red leg to be the same does not make sense. Moreover, the gram loss will encourage the features of yellow leg and red leg to be close which essentially harms the generalization capability of the network to unseen classes which may have attributes violating the semantic relatedness calculated from seen classes only.\n\n4. From a technical perspective, alternative gram matrix loss design should be studied as well. Since $\\mathbf{V}$ is the classifier weights, the product $f(x)\\mathbf{V}^\\top$ returns the unbounded logit. Therefore, the scale of $\\mathbf{V}$ is unbounded and encouraging the inner product of $\\mathbf{V}$ to be 0 or 1 is not reasonable. A more reasonable design could be using the sigmoid cross-entropy loss, i.e. $L_{Gram}=-\\sum_i\\sum_j R_{ij}\\log \\mathbf{V}_i^\\top\\mathbf{V}_j$. Please explain why the regression loss is optimal or provide empirical evaluation by comparing alternative designs.\n\n5. Noise robust loss is the key element in the new formulation. However, it is only briefly mentioned in the experiment section. I am wondering what specific design makes this design better suited for positive bags of attributes.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed to modify the attribute learning loss by taking into consideration that positive attribute may not necessarily appear in all training samples from the seen classes. A gram loss is further introduced to exploit the attribute semantic relatedness by encouraging classifier weights of highly related attributes to be close. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. In particular, I would like to see the justification of gram loss and comparison with alternative designs. This is critical to the decision.\n\n2. It is also good to provide analysis of why performance is not improved much on SUN and AWA datasets.\n\n3. Please provide more analysis into the noise robust loss in the methodology section.\n\n4. Typos: \"ours can can handle bags of unlimited\" -> \"ours can handle bags of unlimited\". Please carefully check the caption of Table 4. I find it does not align with the contents in the table. For example, \"Each row presents an additional modification to DAP\". Is this correct? $p(a|y)$ appears in the table, but it is referred to as $p(a|z)$ in the caption.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BFVl9dnRSN", "reviewer": "Reviewer_Htzo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths are as follows:\n+ the idea of asymmetric class-level attributes sounds reasonable\n+ comprehensive evaluation on multiple datasets of different vision tasks\n+ easy to follow\n\nThe weaknesses are as follows:\n- the key contributions of this paper are not sufficiently novel; if I understand correctly,  only using independent losses on positive and negative bags in the context of MIL is the main contribution, while the remaining parts are adopted from existing methods\n- comparative methods are a bit outdated, more recent methods after 2018 are suggested to compare ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel MIL-DAP method for attribute prediction in the context of zero-shot learning, which shows its effectiveness on multiple popular benchmarks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The following items need to be addressed:\n1. clarify the main contributions of this paper, the authors are suggested to provide one paragraph at the end of the introduction\n2. more recent methods should be included to make the proposed methods more convincing\n3. more details of the noise robust loss should be provided in Sec. 3.1 and ablation studies on selection of such a loss are recommended", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kjG7gOdxBO", "reviewer": "Reviewer_1ko8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It is interesting to apply MIL to weakly-supervised attribute learning. \n2. For positive and negative attributes, the authors adopted two different loss functions, which solves the problem of limited bag size.\n3. The authors proposed the MIL-DAP model to apply the proposed method to zero-shot learning. \n4. Experiments on benchmark datasets justify its effectiveness. \n\nWeaknesses:\n1. It seems that most of the compared methods are proposed several years ago (around 2017). Therefore, I am wondering whether there are any new competitors proposed recently?\n\n2. Missing ablation:  the positive and negative losses in Equ (4) should be ablated individually. For example, what is the result if we only adopt the loss for positive attributes?\n\n3. It seems that the model trained with the image level attributes is the performance upbound. However, in Table 1 and Table 2, the model trained with weakly supervised supervision usually outperforms such upbound. Could explain it?\n\n4. There are some typos:\n    P2: '...memory footprint, ours can can'  --> '...memory footprint, ours can'\n    Equ (4): it is the definition of L_{MIL} instead of the L_{AMIL}, right?\n    In the caption of Table 1, the authors stated that the overall accuracy (OA) is reported. However, in Table 1, the OA results are missing. \n    P9: \"binarized class-level labels for for all datasets\" --> \"binarized class-level labels for all datasets\"\n    \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces multiple instance learning into attribute learning (AMIL). Specifically, given the asymmetric character of positive attributes and negative ones, the authors propose to under-predict the positive attributes and not to over-predict the negative ones. Based on the AMIL, the authors also proposed an attribute-based zero-shot classification method and verify its effectiveness on three standard benchmark datasets.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See weaknesses. ", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PjMp3CamLS", "reviewer": "Reviewer_Q5u8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n1.\tFor the improved part of the DAP model, the paper has a more complete theoretical derivation and introduction.\n2.\tThe paper presents the complete ablation studies and the proof of excellence for certain methods. Especially after adding their own innovative part, the effect can get some improvement.\n3.\tThis paper is fully structured and is able to explain their ideas effectively.\nWeaknesses:\n1.\tSome of the content is not presented comprehensively enough. For example, how is the attribute semantic relatedness matrix R constructed; what are the detailed formula definitions of BCE Loss and NR Loss? The authors just directly cite the relevant papers, but do not introduce them.\n2.\tIn the section on positive-negative bags, the authors only explain how to construct them, but do not reflect how they are handled during the training phase.\n3.\tThis paper performs attributes prediction before performing the ZSL task, so there is a discrepancy between the title and the main body of this paper.\n4.\tThe ResNet-101 is recommended as the backbone, as it is the most popular backbone in the ZSL and GZSL tasks.\n5.\tWhy the authors do not utilize generative ZSL methods to test the generated class-level attributes, because the DAP and some remaining linear optimization methods are obsolete in the ZSL field and they are difficult to clearly demonstrate the superiority of the authors' methods.\n6.\tThere are some typos in this paper, for example, in the last paragraph of Sect. Introduction, the authors misspell ‘ours can’ as ‘ours can can’.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the Attribute Multiple Instance Learning task based on Multiple Instance Learning task, and utilizes the attention mechanism and positive-negative loss to accomplish attribute prediction and localization. In addition, the superiority of constructive attributes is further explored by improving the DAP model in the ZSL task.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1.\tThe authors need to add clarification to what is missing in Disadvantages.\n2.\tThe authors need to supplement some experiments, the ZSL models are no longer limited to some outdated methods such as DAP, etc. \n3.\tRevision of the title to better fit the content of this paper.\n4.\tThe authors should strengthen the connection between ZSL and attribute prediction tasks, which are now somewhat fragmented in current manuscript.\n", "broader_impact_concerns": "I have no more concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies attribute learning under the setting of multiple instance learning, where only the image-level labels are available. The paper presents a few technical contributions and validates them in the problem of zero-shot learning. Initially some common weakness
points are raised by more than one reviewers, including the lack of comparisons with more recent ZSL methods, and the less comprehensive
ablation studies. The authors have successfully addressed most of these raised concerns. Two of three reviewers who have given final ratings are mostly convinced by the responses. Given the reviews and discussions, acceptance with mandatory revision is decided. The authors are expected to revise and improve the paper by addressing all the comments from the reviewers and including them in the final paper. More precisely, the authors are expected to improve the paper by (1) presenting comparisons with more recent ZSL methods, even though many of these methods may not involve attribute prediction, (2) including more ablation studies as requested by reviewers (e.g., those on positive and negative losses), (3) change of the paper title appropriately, (4) other typo issues pointed out by the reviewers. With the revised paper, the authors are required to include a "Summary of Change" file that explains clearly how the paper has been revised according to the above comments.
Decision: Accept (poster)