Min-Max Zero-Shot Multi-Label Classification
Authors: Sima Behpour
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transfer learning
zero-shot learning

Abstract: In many classification problems, acquiring labeled examples for many classes is difficult, resulting in high interest in zero-shot learning frameworks. Zero-shot learning (ZSL)  is a problem setup, where at test time, a learner observes samples from classes that were not observed/trained in the training phase and is required to predict the category they belong to. Zero-shot learning transfers knowledge from seen
classes (observed classes in the training phase) to unseen classes (unobserved classes in the training phase but present in the testing phase), reducing human labor of data annotation to build new classifiers. However, most zero-shot learning researches target single-label classification (multi-class setting). There are few studies on multi-label zero-shot learning due to the difficulty in modeling complex semantics conveyed by a set of labels. 
We propose a novel probabilistic model that incorporates more general feature representation (e.g., Word-Net hierarchy, word2vec features, convolutional neural network features (layer-wise), and co-occurrence statistics) and learns the knowledge transfer in terms of data structure and relations. We also investigate the effect of leveraging different CNN layers' features. Our experimental
results prove the efficacy of
our model in handling unseen labels. We run additional experiments to analyze the flat-sharp minima convergence of methods as a generalization factor. Our study suggests that our proposed method converges to flat minima resulting in strong generalization.

Under review as a conference paper at ICLR 2023
MIN-MAX ZERO-SHOT MULTI-LABEL CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
In many classification problems, acquiring labeled examples for many classes is difficult, resulting in high interest in zero-shot learning frameworks. Zero-shot learning (ZSL) is problem setup where at test time, learner observes samples from classes that were not observed/traieed in the training phase and is required to predict the category they belong to. Zero-shot learning transfers knowledge from seen classes (observed classes in the training phase) to unseen classes (unohserved classes in the training phase but present in the testing phase), reducing the human labor of data annotation to build new classifiers. However, most zero-shot learning researches target single-label classification (multi-class setting). There are few studics on multi-label zero-shot learning duc to the difficulty in modcling complex semantics conveyed by set of labels. We propose novel probabilistic model that incorporates more general feature representation (e.g., Word-Net hierarchy, word2vec features, convolutional neural network features (layer-wise), and co- occurrence statistics) and learns the knowledge transfer in terms of data structure and relations. We also investigate the effect of leveraging different CNN layers' features. Our experimental results prove the efficacy of our model in handling unseen labels. We run additional experiments to analyze the flat-sharp minima convergence of methods generalization factor. Our study suggests that our proposed method converges to flat minima resulting in strong generalization.
INTRODUCTION
Supervised learning methods require a large number of labeled instances for training purposes inducing high annotation cost. Developing classification algorithms that reduce the annotation cost has recently attracted more attention. In Zero-shot learning (ZSL), the classes that the classifier observes and is trained on are called seen classes. Conversely, the classes that the classifier does not observe in the training phase and just tests on them in the testing phase called unseen classes. This ability to classify instances of an unseen class leveraging the knowledge from seen classes is called scro-shot/l learning Socher et al. (2013).
The attribute description of the class labels is a primary source in ZSL to bridge the gap between seen and unseen classes Lampert al. (2009; 2014); Fu et al. (2015); Romera- Paredes & Torr (2015). The attributes can carry transferable information across classes since they are defined based on common and specific characteristics of different category concepts. Hence, many 7SL methods employ different attribute types and demonstrate impressive results. For instance, textual descriptions from Wikipedia articles Qiao al. (2016); Akata al. (2016), word embedding vectors trained from large text corpus using natural language processing (NLP) techniques Akata et al. (2015b); Frome et al. (2013); Xian al. (2016); Zhang & Saligrama (2015); Al-Halah al. (2016), co-occurrence statisties of hit-counts from search engine Rohrbach et al. (201 1); Mensink et al. (2014) and WordNet hierarchy information of the labels Rohrbach al. (2010; 2011), Li al. (2015) have been used. However, majority of these studies investigate multi-class ZSL, while the more challenging multi-label ZSL problem has received less attention Mensink et al. (2014); Zhang et al. (2016); Lee et al. (2017).
Multi-Label (ML) classification is applied in diverse domains like recommender systems Prabhu & Varma (2014) and computer vision Wang ct al. (2016). Most cxisting multi-label methods consider structural relations on binary label outputs leverage the label correlation in domains with large number of labels Yu et al. (2014): Rai et al. (2015). Multi-label classification is more challenging when all the labels not available at training time Zhang al. (2016); Mensink et al. (2014) 4). In many real-world applications (e.g., recommender systems and computer vision), the label set keeps increasing, and re-training the classifier might be time-consuming and expensive. Gaure proposed

---

Under review as a conference paper at ICLR 2023
probabilistic framework Gaure et al. (2017) leveraging the co-oceurrence statisties of the seen labels with both seen and unseen labels. It is an EM-based algorithm in which every parameter update requires solving weighted ridge-regression problem.
The relationship//ttucuurre between seen and unseen classes is ignored in most multi-label ZSL methods. For instance, seen and unseen class can share similar co-occurrence and taxonomy pattern resulting in classifying the unseen class correctly in the test phase. We propose a novel framework that relies on the data relationship among different classes (structured prediction). To account the pattern and structure among seen and unseen classes, we propose to model the classes (graph nodes) and their relationship (graph edges) as a complete graph. Transferring the relationships among classes is one of the main novelties of our method.
We conduct extensive experiments to investigate the performance of our proposed approach. The empirical results demonstrate the effectiveness of our proposed approach, considering different baselines and datasets. Finally, We run convergence analysis based on the flat/sharp minima aspect and demonstrate our proposed method's s convergence to flat minima. It presents the importance of convergence to flat minima for strong generalization support.
BACKGROUND AND RELATED WORK
In this section, we explain ZSL and the motivation behind our approach. We explain multi-label classification and related works regarding zero-shot learning in Appendix for the sake of space. Zero-Shot Learning (ZSL): In supervised classification, sufficient labeled samples for target classes required in the training phase Bishop (2006). Collecting and labeling large number of samples is expensive in many classification tasks. For example, it is hard/dangerous to take images of samples like wild animals in the oceans, plants in the high depth of oceans, or the high height of mountains. Moreover, many objects "in the wild" do not occur frequently enough to collect and label a large set of representative samples build the corresponding classifier Changpinyo al. (2016). Zero-shot learning (ZSL) addresses this problem by transferring knowledge from abundantly labeled source classes to help build classifiers for target classes with no available labeled samples. In more detail, there are labeled data for some classes in the training phase, and the learner is asked to predict those classes at test time. The primary assumption is that there should be some similarity between the unseen and seen classes. Although they are different classes but related via some features and auxiliary information.
ZSL builds classifier to predict the presence or absence of unseen classes for test data. The critical factor in ZSL is effectively transferring knowledge between seen and unseen classes. Our approach employs the Min-Max formulation Asif et al. (2015) and models the multi-label ZSL learning the structure and the relationships among the nodes in complete graph. In addition, we employ auxiliary resources like word2vee, WordNet hierarchy, and features from different convolutional neural network layers in the model fcature representation. However, the solid dependency of the model on these resources is controlled by modeling the problem as min-max zero-sum game Asif et al. (2015); Behpour et al. (2018).
3 MIN-MAX MULTI-LABEL ZSL
We propose a novel approach for multi-label zero-shot learning based on leveraging robust cuts Blum & Chawla (2001); Behpour et al. (2018). A labeled training dataset (seen classes) and test dataset with set of seen and unseen classes are given in this setting. The objective is to learn multi-label classifier from the training set (with seen elasses) and generalize it to the test set with unseen labels.
This generalization requires an intermediate feature representation shared between training and test classes. Hence, a model that can define the importance of features for the classification task is required. We propose robust ZSL classifier capturing the most effective and implicit feature similaritics leveraging graph representation to make better generalizations and predictions on unseen classes. Our approach builds on the min- min-max robust lcarning, which approximates the training data labels with a worst-case distribution while still resembling training data properties Topsøe (1979); Grünwald & Dawid (2004); Asif et al. (2015). The learning task in multi-label classification is modeled as a min-max zero-sum game between a maximizer and a minimizer based on minimum cost graph cuts. Furthermore, it deploys a joint representation of the features in its feature representation
2

---

Under review as a conference paper at ICLR 2023
to embed the label co-oceurrence statistics in the training phase. This graph representation and the min-max training process provide informative knowledge transfer in the ZSL framework.
3.1 PROBLEM DEFINITION AND NOTATIONS
We first describe how multi-label (ML) classification is modeled in method following Asif et al. (2015). Next, we present our proposed ML-ZSL framework and then explain our feature representation that plays significant role in method's outstanding performance and accuracy.
The MI annotation prediction is modeled as a Markov network with nodes indicating the labels/classes, edges representing the pairwise relationship among the nodes, and two extra nodes: "0" and "1". These two extra nodes are connected to all other nodes in the graph. In this framework, the parameters are learned by the classifier to make min-cut in the graph resulting in present nodes (connected to node "I"" and absent ones (connected to node "O"). For better illustration, please consider dataset with six labels: {Lion, Sheep, Panda, Cow, Dog, Dolphén) as our running example. We present an overview of the Markov network for our running example in figure 1.
As is shown in figure 1, Markov networks with nodes corresponding to the predicted variables, = (v'....,y") ))), chosen from fixed set of labels y' E y, Vi C [n], where [n] (1.....7)) m)). In our notation, we denote the corresponding random variables for these label variables by Y"), and present vectors and multivariate variables in bold. Given information variables are demonstrated using a single vector, x E X, with a corresponding random variable denoted as X.
In this setting, the task is to make predictions for unseen (presented "u") classes/labels (1.........) (1.....)) given an input and set of m training example pairs, (x, y jj) As indicated, there is an associated label vector y for training sample x corresponding to different classes of the dataset. For instance, Figure 2 demonstrates the label vector where labels sheep, cow and dog are present in the image. Here I() is an indicator function whose value only the an inner logical expression is true.
I Littion 1)
Mobeep
[Merr 1)
[[aig 1)
Figure 2:
image incloding sheep. cow,
Figure 1: Markov Network model Multi-Label prodiction.
and a dog is represented as label vector
3.2 FEATURE REPRESENTATION
Feature representation plays key role in finding an intermediate clue for generalization from seen classes to unseen ones. new/unseen class can be demonstrated by matching the similarities of classes. Thus, incorporating semantics into the classification process is crucial achicving high prediction accuracy. Four feature representations (joint, layer-wise, pairwise, and hierarchy) employed in approach learning the features representation and the features' relationship between different classes. Given a training sample x, we present its feature representation by Φz, which is employed to define our other features. For instance, the feature vector provided by Mulan datasel Tsoumakas et al. (2011) for figure 2 can be counted as feature vector á for this image.
We define joint features following Kim et al. (2012) based on the presence of the class instance in the sample as equation 1. In our equations, I() is an indicator function whose value is 1
10y/mm == 1)(dc] Iyy 1|[6=]
Ily Hin == (KC 1ly 1)CNNK(2)
Øjoint
blayer-
(2)
Mygom -- 1)(0=]
M6grriin - 1)NNNK(()
3

---

Under review as a conference paper at ICLR 2023
only if the inner logical expression is true. The layer-wise features represent the features of layer k from a convolutional neural network represented as CNNx(z) for the given image x. Recently, convolutional neural networks (CNN)
have shown outstanding performance in providing informative features for images. Every layer of CNN extracts different features. The lower layers represent the structure/texture of the image input, and the upper layers provide general/high- heaee//llhhleeeelllee representation. Existing ZSL frameworks that employ CNN features focus only on features from fully connected layers and ignore the significance of the lower convolutional layers. One of the novelties of our method is that extract the features from different layers and show that employing features from other layers of the convolutional layers in addition to the top layers results in better performance and accuracy in ML-ZSL. Different classes share the same similar structures, specifically when they are from the same ancestors. The CNN lower and top convolutional layers capture these structures and can be generalized well to unscen elasses. In our experiment, we consider different CNN layers' features and demonstrate the outstanding performance of these feature representations.
E:A
at died
d-d Wireled
t6 Parass
Ei. Lanreibbeil
Craetsheents
Prguedive
Narahik
BiCw
Crn
Carahera
Sherp
stitigg
Figure 3: The VOC2006 taxonomy.
We present the pairwise (edge) features by leveraging word2vee' features to present the relationship between two classes/ nodes i and j. The pairwise feature representation based on word2vec features is computed as follows:
wwildd) )) 1ly # (lt////)//wrlll worddee22))))) word2ver((]]
(3)
Here, Φw refers to pair-wise wise features based on word2vec feature representations for the edge between nodes j.d
Our last type of features, taxonomy features, detail the inner-class similarities and dependencies that improve classification performance. We deploy taxonomy features in addition joint, layer-wise, and pairwise features to provide a comprehensive feature representation for the classifier. In this setting, we are given a laxonomy T demonstrated by an arbitrary directed graph (V,, E, where Vi = (11, 22 .... 2)rr) represents the graph nodes set and E, presents the graph edges set. The set of nodes from the root node to a leave node is presented as π (y). We employ Tyy) to represent π (v)
structure as follows: Ty())= =
[1
Vi C iTy)
o otherwise
For instance, the taxonomy vector of figure is presented as T=[AlI, Nonlife, 2- 2-wheeled, Bcycle, Motorbike, 4-wheeled, Bus, Car, Life, Person, Laurasitheria, Cetartiodactyla, Cow, Sheep, Pegasofera, Carnivora, Cat, Dog, Horse]. Following this taxonomy vector, class sheep is presented as T(sheep) = [1, 0. O, O, 0, O, 0, O, 1, O, 1, 1. 0. 1, O, O, 0,0, 0J'. In T(sheep), the nodes All, Life. Laurasitheria, Cetartiodactyla, and sheep " presented the path from "root" to "sheep" " get value "1" and the other nodes get value "0".
We subsume the structural taxonomy information and encode the taxonomy paths by leveraging joint feature representation following Kim et al. (2012) that given by tensor product class features and class indicator [[M, E (v))) as follows:
d())[n e m(v) e(x)le e =tyil
(i, w) = o(x) T()= =
(4)
*())|up) +())]
3.3 MIN-MAX METHOD FORMULATION
Min-mat prediction methods Grünwald & Dawid (2004) leverage two players, maximizer and minimizer, and demonstrate a maximizer distribution approximation of the training data labels/classes, Pmaa ( (z), while seek minimizer distribution Pmima (ymini (z:).
The minimizer distribution over possible labeling of the samples minimizes the expected loss against the worst-case distribution chosen by the maximizer distribution (over possible labeling of the
hatps://codeegggggs..coooooooooo
4

---

Under review as a conference paper at ICLR 2023
samples):
mrax B- Patu- PaweetiiaaallaaPPPaaPaaiaa Pa [loss[Yaus, Youri) such Eg- D Ploord (o(X, Yruul) Pann
Here the label vector prediction by maximizer and minimizer players are shown as Ymuxi and Y/miii respectively. & is the feature representation explained in detail in section Feature Representation. The maximizer player, Mmaxi is constrained by empirical data statistics, č- using equality or inequality constraints. č is the empirical data statistics based on the feature function of training samples, and it computed follows:
F (+(X, Y)]
Σ (6)
(5)
PDtt presents the empirical distribution X [and Y] in the training data set. This Min-Max formulation provides better performance practice for classification Asif et al. (2015) due to aligning training objective to the loss measure instead of surrogate losses like hinge loss. important min-max method features that qualifies it for zero-shot learning. The knowledge transfer in ZSL gets more accurate when the learning and optimization of seen classes are done based on the exact-loss optimization method (not loss- approximation methods like SVM and logistic regression).
one of
Another strength point of this method lies in providing probability distribution over classes (y* presents the kth label/class in label vector y, < le S n.). This probability distribution is beneficial in the ZSL setting as represents the correlations between known and unknown label variables, P(y", yy) where y' E classes, and y² unseen classes. It plays key role in generalizing classes to unseen classes in the ZSL setting, Other probabilistic methods like conditional random fields Lafferty et al. (2001) are not computationally effective compared the Min-Max method. In the remainder of this section, we present the Min-Max method for the ZSL setting.
We present the structure of our method Figure As it is shown in Figure 4, the classificr is trained to make the right min-cut on training samples graphs (including only seen classes) in the training phase, and it predicts min-cut on testing samples graphs (including only unseen classes).
The number of nodes (classes) in the testing phase can differ from the training phase. When the test set includes both seen and unseen classes, it is called generalized ZSL.
Our Min-Max method provides strong performance in making the right min-cut in the modeled graph as it leverages exact-loss optimization, the structure of the data modeling the problem, and appropriate feature representation. We present Min-Max ZSL in the next section in full detail.
Training on sern classes.
Texting on unseem classes.
Figure 4: The structure Min-Max ML-ZSL applying graph cut transfer the knowlodge. The graph nodes are classes/labels, which disjoint sets. The edge weights identified with feature patentials.
3.4 MIN-MAX MULTI-LABEL ZERO- SHOT LEARNING
We model the ML-ZSL problem as a binary- valued Markov network with non non-negatin -egaatitivv pairwise potentials. Two additional nodes, source (0) and target (1), added the network to indicate present and absent nodes/classes in every sample. Therefore, all network nodes (class nodes) are connected to these two nodes. Edges from the source and the sink nodes weighted based unary (joint, layer-wise, and taxonomy) features potentials. The edges between predicted variables weighted based on the pairwise features potentials. provide the maximum value assignment of the network by dividing the nodes into two mutually exclusive sets connected to the nodes source or target (c.g., figure 4).
mnnimum-cuu/maximman
algorithm can
We consider Hamming loss as the loss function this study and experiments. Hamming loss computes the fraction of the correct predicted labels in the multivariate annotation task.
loss Yeasi Yrui) Σ (iiim o Vinn)
(6)
5

---

Under review as a conference paper at ICLR 2023
Our features presented using Φ including &taxonomy (or), joiint Playerwise (Dtw), and Ppairuise (ppw) in our model formulation. Min-Max ML-ZSL game formulation is presented as:
min Pasin
max mazs
Ex, Y PDoea. Ymtoi XX Fam
[loss( mints maass)
Ymaxi X ~ Pauii
such Ex--Peta Y maxi Puu ( x)] Exyy-paan (l(r x)] Vi E [w]]; Etasooommy
Ix PData .X masei aun
joint mani x)] = Hx Y- Paata BBosnelY x)] vi (n];
Ex- Pitta Y maxi Kmen
Bi X) = BX Di (V' x:] Vi [m]; bwokayerwwse
PDace (X-xxXX--Faan
Op (Kiia Vinan X) Bx.y-Poote Ao y x)]
vi.j puspaiiiis
(7)
In formulation 7, the maximizer aims make the most uncertain label distribution resulting in large amount of expected loss for the maximizer. However, leveraging the constraints defined on features and feature relations put the maximizer in predictable direction.
These constraints can be incorporated as lagrangian potentials with learning parameters 0 including Prazonomy (fr), Ojoint, Otayer- wise ). and 0 pairrwise (6pw) by deploying strong duality Boyd & Vandenberghe (2004) and the method of lagrangian multipliers:
min (0t), aastii/soo 105 Prata (Yau (x)) Prean (Youc (b))
min
max
Ex, ~ PDattor Y minil IX ~ Pani-
loss(Y Yous) + Lo; (o) x) 61(Y",X)
Y muxi XX
Possi
gjon (ono C Oriui x) Joini (X* x)) Low (0f (Yamn X) (x
E [2
ghu (0) (Y Yi X) Bia por,,Y?, xi
(8)
For a detailed proof on transforming Eq. 7 to Eq. 8, please see appendix, section "Eq. 2 to Eq. transformation proof".
Constructing this game formulation requires high computational complexity in time and space. The double oracle, constraint generation method, is deployed overcome this bottleneck.
3.5 MODEL LEARNING PARAMETERS
Optimizing the model parameters 0' and 0² is another major step in Min-Max ML-ZSL. We employ standard tools from convex optimization like AdaGrad Duchi et (2011). The gradients g and g') for single sample (x, y) are as follows:
s: [Bymnees mase Pass (5: (Youa' xi] Φi1y' x3) Byjonn {3x Paom Josn (Yrua " x]] - Dfomnsl' .x)) xx}
si.. (BYmaxi Paus (1 (Y " x]] Bi (u' (u'xxx) x3}
%. (Ex Pame yod YYmm' Youd x). dpd (v' x))
3.6 INFERENCE
Min-Max ML-ZSL model parameters are learned using the double oracle method McMahan et al. (2003) and AdaGrad Duchi et al. (2011) as optimization. In the inference step, the learned parameters are employed, and the algorithm is run. The probabilistic output of the min-max method as distribution over possible graph-cuts (label vectors) for the maximizer is possible support assignment set. The graph-cut (label vector) with the highest probability is chosen as the prediction.
The training phase is done using seen classes training data. The testing data are from unseen and seen classes in the testing phase. Hence, nodes (seen/unseen classes) connected to have the label meaning there is entity of the classes in the sample. The other nodes connected to 1 have the label 1 (i.e., there exist entities of these classes in the sample data) in our prediction.
In this model, nodes and edges play essential role in transferring knowledge from classes to unseen classes. We stimulate the similarities among seen and unseen classes using graph edges. When two nodes (seen and unseen) are similar, they would have similar/close weights and features values for their outgoing edges to other nodes. Therefore, the edges and nodes unseen classes' weights/values in the graph would have values to their similar classes in the seen classes. Hence,

---

Under review as a conference paper at ICLR 2023
the learning parameter and the inference process are directed to make the correct prediction (min-cut in our method). For instance, an unseen class "fish" has close feature values to the feature values of the seen class "dolphin" on the edges and the graph structure representation (figure 1). It directs the min-cut in the right direction, recognizing an unseen class "fish" though the classifier has not been trained for this class, We employ this key idea and leverage comprehensive feature representations. Overview of the Min-Max ZSL method: Since our min-max ZSL method and feature representation have been explained in previous sections, we present it through simple example for better illustration. Assume training classifier on seen classes like lion, sheep, dog, dolphin, ocean, cow, and panda presented as graph nodes.
The classifier is trained over the training samples (seen classes) to make the right min-cut to connect present labels to node and absent labels to node 0 a multi-label problem. The test sample includes classes like "fish" and "sea". Therefore, the classifier is tested on a testing sample, including classes on which the classifier has not been trained.
Our proposed method and feature representation help the classifier to 1) learn the co-occurrence of classes like "dolphin" " and "ocean" in the training phase. 2) recognizes the high hierarchy feature representation overlap of classes "fish" and "dolphin". 3) recognizes the high hierarchy feature representation overlap of classes "sea" and "ocean". 4) conclude the co-occurrence of classes "fish" and "sea" based on the edges and nodes values (f.((z:) and make right min-cut in the graph where connects classes "fish" and "sea" to the node "1" in the same side of the min-cut, In general, the co-occurrences, similarities, and hierarchy structure in our graph representation lead the classifier to behave the same on similar seen and unseen classes. Modeling a multi-label problem as graph allows the classifier to learn the structurefrelationshipp among the labels. This learned structure/ relationship knowledge exists among similar classes and is transferred in the ZSL setting.
EXPERIMENTS
Our experiments include two main analyses: performance and convergence to flat/sharp minima. We first present our experimental studies on ML-7SL, methods' performance small and large datasets. Next, we explain convergence to flat/Sharp minima as an important metric to evaluate the generalization of ZSL methods. We run an analysis and show that the methods with convergence to flat minima result in better performance in the ZSL setting.
We benchmark our method on different datasets and compare them with ofatc-of-the-ar methods. Finally, we present more experiments and explain our basclines in Appendix, section "More Experiments".
Performance Analysis:
In the first experiment, we consider two standard multi-label datasets: the PASCAL VOC2007 Everingham et al. (a) and VOC2012 Everingham et al. (b) datasets. The VOC2007 dataset includes 20 visual object classes. There exist 9963 images in total in this dataset, 5011 for training and 4952 for testing. The VOC2012 dataset contains 5717 and 5823 images for training and validation from 20 classes. We leverage the validation set for test evaluation. For each image, we extract 4096-dim visual features VGG19 Simonyan & Zisserman (2014) pre-trained on ImageNet. We split the seen classes into two disjoint subsets with an equal number of classes for training and validation to determine the syper-parameters. In this setting, we train the model on the training set and choose hyper- parem-rrreeeerss based on the test performance on the validation set. After parameter selection, the training and validation data are put back together to train the model for the final evaluation of unseen test data.
For our method, Min-Max, we employ the feature representations of CNN 6, 33 layers to present layer- wise and pairwise feature representation as provides better feature representation for knowledge transfer (based on experiments in Appendix, section "More Experiments"). The taxonomy features representation is computed using VOC 2006 taxonomy structure, and the seen-unseen standard split in Norouzi et al. (2013). We report our experimental results in table I using three different multi-label evaluation metrics: MiAP, micro-F1 (presented as mi-F1 in table 1), and macro-F1 (presented as ma-Fi in table 1). The Mean image Average Precision (MiAP) Li et al. (2016) evaluates how well the labels are ranked on a given image based on the prediction scores. Micro-Fl and macro-F1 measure how well the predicted labels match the test data's ground truth labels.
Discussion: In table 1, we present the experiment results of six ZSL methods (Conse Norouzi et al. (2013), LatEn-M Xian al. (2016), DMP Fu et al. (2015), FastoTag Zhang et al. (2016), and TAEP-C
7

---

Under review as a conference paper at ICLR 2023
VOC2007
VOC2012
Methods
MiAP
mi:
ma--F
MIAP
mi-FI
mar P1
CouSE
45.65
27.64
24.83
45.34
30.83
26.11
LatEm-M
47.37
28.39
27.16
46. 17
27.46
26.37
DMP
53.23
35.53
38.82
52.32
34.31
35,94
FastOTag
52.45
34.67
35.91
51.09
33.13
33.97
TAEP-C
54.64
36.56
37.87
$2.84
33.26
GEN
54.92
36.73
37.65
53.89
36.24
36.10
55.28
37.58
54.89
37.63
Min-Max
58.11
38.86
39.08
57.10
39.22
42.85
Table 1: Average comgarison results (%) over les runs un xeto shot multi label imaage classification.
NUS-WIDE
Open-Images
Aunseen 7186/400)
Method
Task
P
K 3
R
F1
P
K
R
5
F1
mAP
K -
F1
Km
R
211
F1
mAP
CONSE
ZSL
17.5
28.0
21.6
13.9
37.0
30.2
9.4
0.2
7.3
2.6
0.4
0.2
1.7
11.3
0.3
2.4
40.4
LabelEM
ZSL.
13.6
25.0
19.2
13.4
35.7
19.5
7.1
0.2
8.7
0.3
0.2
15.8
0.4
40.5
GZSL
15.5
6.8
9.5
13.4
9.8
11.3
2.2
4,8
5.6
5.2
3.7
8.5
5.1
45.2
PasefTag
0.6
GZSL
18.8
8.3
11.5
15.9
11.7
13.5
3.7
14,8
17.3
16.0
9.3
21.5
12.9
45.2
Attetton
ZSL.
20.0
31.9
24.6
15.7
41.9
22.9
12.9
0.6
22.9
1.2
0.4
32.4
13.5
0.9
40.7
LESA (M-18)
GZSL.
ZSL
23.6
25.7
41.1
10.4
31.6
14.4
19.7
19.8
52.5
14.6
28.7
16.8
19.4
5.6
16.2
0.7
25.6
18.9
17.4
1.4
10.2
0.5
23.9
37.4
14_3
1.0
41.7
45.4
2.2
43,0
GZSL
30,9
13.6
18.9
26.0
19.1
23.0
8.9
33.6
38.9
36,1
22.8
$2.8
31.9
49.7
ZSL
26.4
12.4
32.5
19.8
53.1
28.8
25.3
10.3
42.4
16.6
12.9
$2.1
20.7
43.7
DRM
GZSL
30.3
24.0
7.8
33,9
39.1
36.3
24.2
32.7
49.9
BIAM
GZSL
28.7
30.4
14.0
19.2
21.6
28.5
18.6
22.1
30.7
26.0
9.2
36.9
4.6
44.1
35.5
36.2
8.3
24.5
3.8
51.6
33.3
544
49.9
45.3
ZSL
29.4
31.7
30.5
20.2
29.25
26.4
34.1
33.7
33.9
23.1
44.38
30.5
47.3
23.7
24.3
18.5
31.6
23.3
22.1
27.6
Min-Max
ZSL.
29.3
44.7
35.4
22.7
57.4
31.8
30.9
5.3
44.5
9.5
4.8
54.4
8.8
48.3
CZSL
33,0
24.5
28.1
32.3
30.5
31,4
11.0
37,9
41,8
39,8
27.6
55.3
36.8
51.5
Table State-of-thheart comparison ZSL and GZSL tasks on the NUS-WIDE and Open Images datasets. report the results in terms mAP and F'l score at KE333, for NUS-WIDE and K (100 20} for Open Images. The best results in bold.
Ye & Guo (2019)) VOC2007 and VOC2012 datasets. As it is shown, the Min-Max provides the maximum MiAP compared with other methods on both datasets. Considering MiAP, Min-Max outperforms Conse and BiAm by 12.4% and 2.8% the VOC2007 dataset. The experiment results on VOC2012 present the strong performance Min-Max compared to other methods.
Table 2 shows the state-of-the-ard comparison for ZSL and generalized ZSL (GZSL) ML classification on NUS-WIDE WUS-WIDE and Open-Images datasets. The results are reported in terms mAP and Fl score at top K predictions (3, 5}). NUS-WIDE: For the ZSL task, SKG Lec et al. (2018) and BiAM approaches Narayan et al. (2021) achieve improved performance over GEN Gupta et al. (2021), LESA Huynh & Elhamifar (2020). DRM Ji et al. (2020) and FastOTag, with mAP scores of 26.4 and 26.0. Our approach (Min-Max) achieves state-of-thh-art results with an absolute gain of 4.5% in mAP over the best existing approach, SKG. Similarly, our approach obtains consistent improvement in classification performance over the state-of-the-art in terms of F1 score (K 5)).
For the GZSL task, the min-max approach achieves mAP score of 11.0, outperforming BiAM, GEN, and DRM with absolute gains of 1.8%, 2.1% 1,, and 3.2% respectively. Similarly, our approach achieves consistent improvement classification performance with absolute gains of 8.9% and 9.3% over BiAM in terms of Fl score at K=3 and K =5 respectively.
Open Images: The results are reported in terms of mAP and Fl score top-K predictions (KE(10, 20}). Compared to the JUS-WIDD dataset, Open Images has more labels. It makes ranking problem within an image more challenging, reflected by the lower scores in the table. For the ZSL task, the min-max method performs favorably against other baselines with F1 scores of 9.5 and 8.8 10 and K K-20, respectively. It is worth noting that this dataset has 400 unseen labels resulting in challenging ZSL problem. Our Min-max approach outperforms the state-of-the-ate in terms of both Fl and mAP, for the GZSL task. In Narayan et al. (2021), the mAP on OpenImage dataset is reported as 73.6 and 84.5 for ZSL and GZSL. We ran these experiments leveraging the code provided by the authors but could not replicate the reported mAP results.
4.1 SHARP AND FLAT MINIMA Hochreiter Hochreiter & Schmidhuber (1995) launched the discussion about flat and sharp local minima in deep neural networks. A flat minimizer (learning parameter) 0 is the one for that the
8

---

Under review as a conference paper at ICLR 2023
function varies slowly in relatively large neighborhood of & leading to low sensitivity. In sharp minimizer 0, the function decreases rapidly in small neighborhood of ê resulting in high sensitivity. The low sensitivity of the training function at a flat minimizer positively impacts the ability of the trained model to generalize on new data; Rissanen (1983) presents that statistical models that require fower bits to describe are of low complexity and generalize better Keskar et al. (2016). Keskar ct al. (2016) shows that learning systems that find sharp local minim lead poor generalization.
Sharpness Metric: The sharpness of an optimizer/ method can be measured by the magnitude of the eigenvalues of (a)) This computation requires high computational costs in deep neural networks. We use the sensitivity measure Keskar et al. (201 6) to avoid the infeasible high computational cost.
The sensitivity measure (sharpness measure) is the largest value that the function f can achieve in small neighborhood of the local minima of the function f. To make an accurate maximization process, the maximization process done both in the entire space R (where n is the feature space dimension) and in random manifolds. For this purpose, the matrix Symm. where m represents the manifold dimension, with random values is employed. We set constraint set C. is defined ensure invariance of sharpness to problem dimension and sparsity.
100 in our experiments. The
C, f= E R - =)1l +
=(I(s* z).1 + 1), Vi 21, mm))
Here St denotes the pseudo- inverse of S; hence, с controls the box size. Given x E e 0 and E IR "xxm, the measure of sharpness (or sensitivity) of function f at x is computed following:
b (r. (maxyec, f(x + Sy)) f(x) 100.
(I))
f(x)
The sharp minima have a different definition in convex optimization literature Ferris (1988). To better illustrate curvature changes in a ZSL setting, we run an experiment on different datasets and calculate the sharpness value for consecutive tasks following Keskar al. (2016) experiment setting. We report the result of experiments in table 3. In this experiment setting, we explore the full-space
(S = In) setting the value of e to (10-3). The higher the value of sharpness presents the more method sensitivity. We report more experiments considering as (5.10-1) in the appendix.
Method//Datastt
VOC2007
VOC 2012
NUS. 1000
10
ConSB
87.
7.87
5.61
4.52
LatEm
67
21.21-1
+ 4.32
97.83
5,88
FastOTag
53.86
5.53
43.65
6.34
90.59
7.26
DMP
51.96 = 1.90
40.44044444
8.31
6.54
80.73
4.62
TABP-C
49.87 + 3.66
39
79
9.51
88.94
8.49
LESA
45.31
2.23
37.51
3.48
83.24
5.49
GEN
41.34 + 4.49
34 78
3.84
80.27
5.43
BIAM
4.64
4.21
79.
Min-Max Max
35.7 + 3.94
31.24
+ 5.
75.67 + 6.78
Table 3: Sharpness value considering
10
Discussion As it is presented in table 3, Min-Max demonstrates much less sharpness compared with the other methods on three datasets. presents the effectiveness of different approaches leading the model to flat minima in the optimization process. The sharp minima in a multi-label setting happen when the loss function increases rapidly only along a small dimensional subspace. The function almost flat in other directions. The Min-Max method smooths this significant gradient update. This experiment demonstrates the importance of convergence to flat minima in the optimization process for ZSL frameworks. As presented in table the methods that provide better performance in the ZSL setting have less sensitivity (sharpness).
5 CONCLUSION
This paper proposes novel probabilistic framework for ZSL, including extensive feature representation and probabilistic cuts for ZSL. Our method relies on probabilistic graph cut output, effectively capturing the implicit features and finding the most important and closest attributes to unseen class attributes at testing time.
The Min-Max method significantly outperforms the existing ZSLL state-of-the-art methods well- known datasets. Besides, we run sharpness analysis considering different ZSL methods. Our study suggests the importance of low sharpness value in better generalizing multi-label ZSL methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence, 38(7):142511438 2015a.
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schicle. Evaluation of output embeddings for fine-grained image classification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1927-2936, 2015b.
Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bernt Schiele. Multi-cue zero-shot learning with strong supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 59-68, 2016.
Ziad Al-Halah, Makarand Tapaswi, and Rainer Stiefelhagen. Recovering the missing link: Predicting class-attribute associations for unsupervised zero-shot learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5975-5984, 2016.
Kaiser Asif, Wei Xing, Sima Behpour, and Brian Ziebart. Adversarial cost-sensitive classification. In UAL, pp. 92-101, 2015.
Sima Behpour, Wei Xing, and Brian D. Ziebart. Are: Adversarial robust cuts for semi-supervised and multi-label classification. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2704-2711. AAAI Press, 2018. URL https: /www. aaai. org/ ocs/index: php AAAI AAAII8 paper /tt///iewwwi//ee 173099.
Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateek Jain. Sparse local embeddings for extreme multi-label classification. In Advances in Neural Information Processing Systems, pp. 730-738, 2015.
Christopher M Bishop. Puttern recognition and machine learning. springer, 2006.
Avrim Blum and Shuchi Chawla. Learning from labeled and unlabeled data using graph mincuts. 2001.
Stephen Boyd and Lieven Vandenberghe. Convex Optimizaation. Cambridge University Press, 2004.
Serhat Sclcuk Bucak, Rong Jin, and Anil K Jain. Multi-label learning with incomplete class assignments. Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 2801-2808. IEEE, 2011.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5327-5336, 2016.
Soravit Changpinyo, Wei-Lun Chao, and Fei Sha. Predicting visual exemplars of unseen classes for zero-shot learning. In Proceedings of the IEEE international conference on computer vision, pp. 3476-3485, 2017.
Minmin Chen, Alice Zheng. and Kilian Weinberger. Fast image lagging, In International conference on machine learning, pp. 1274-1282, 2013.
Yao-Nan Chen and Hsuan-Tien Lin. Feature- label space dimension reduction for multi-label classification. In Advances Neural Information Processing Systems, pp. 1529-1537, 2012.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Mach. Learn. Res.. 12:2121-2159, July 2011. ISSN 1532-4435. URL bttp:://al. acm. org/ citation. cfm?id=195300 2021068.
M. Everingham, L. Van Gool, C. K. Williams. J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal-
10

---

Under review as a conference paper at ICLR 2023
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal. wtw.:p/awwaaaggssoo.../eeee//e/a
b.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1785. IEEE, 2009.
Michael Charles Ferris. PhD thesis, University of Cambridge, 1988.
Weak sharp minima and penalty functions in mathematical programming.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In Advances in neural information processing systems, pp. 2121-2129, 2013.
Yanwei Fu, Yongxin Yang, Tim Hospedales, Tao Xiang, and Shaogang Gong. Transductive multi-label zero-shot learning. arXiv preprint arXiv::50:: 07790, 2015.
Abhilash Gaure, Aishwarya Gupta, Vinay Kumar Verma, and Piyush Rai. A probabilistic framework for zero-shot multi-label learning. In The Conference Uncertainty in Artificial Intelligence (UAI), volume 1, pp. 3, 2017.
Peter D. Grünwald and A. Phillip Dawid. Game theory, maximum entropy, minimum discrepancy, and robust Bayesian decision theory. Annals of Statisties, 32:1367-1433, 2004.
Yuchen Guo, Guiguang Ding, Jungong Han, and Yue Gao. Zero-shot learning with transferred samples. IEEE Transactions on Image Processing, 26(7):3277-322 2017.
Yuhong Guo and Dale Schuurmans. Adaptive large margin training for multilabel classification. In AAAI, 2011.
Akshita Gupta, Sanath Narayan, Salman Khan, Fahad Shahbaz Khan, Ling Shao, and Joost van de Weijer. Generative multi-label zero-shot learning. arXiv preprint arXiv:2101./1666, 2021.
Bharath Hariharan, SVN Vishwanathan, and Manik Varma. Efficient max-margin multi-label classification with applications zero-shot learning. Machine learning. 88(1-2):127-155 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Steve Higgins, Chris Falzon, Ian Hall, David Moseley, Fay Smith, Heather Smith, and Katie Wall. Embedding ict in the literacy and numeracy strategies. 2005.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances in neural information processing systems, pp. 529-536, 1995.
Dat Huynh and Ehsan Elhamifar. A shared multi-attention framework for multi-label zero-shot learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8776-8788 8786, 2020.
Zhong Ji, Biying Cui, Huihui Li. Yu-Gang Jiang, Tao Xiang, Timothy Hospedales, and Yanwei Fu. Deep ranking for image zero-shot multi-label classification. IEEE Transactions on Image Processing, 29:6549 00600666020000 2020.
Ashish Kapoor, Raajay Viswanathan, and Prateek Jain. Multilabel classification using bayesian compressed sensing. In Advances in Neural Information Processing Systems, pp. 2645-2653, 2012.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On latch-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609 160.00669 2016.
Suna Kim, Suha Kwak, Jan Feyereisl, and Bohyung Han. Online multi-target tracking by large margin structured learning. In Asian Conference on Computer Vision, pp. 98-111. Springer, 2012.
11

---

Under review as a conference paper at ICLR 2023
John Lafterty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribbte transfer. Computer Vision and Pattern Recognition, 2009. CVPR 2009, IEEE Conference on, pp. 951-958. IEEE, 2009.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-bssed classification for zero-shot visual object catcgorization. IEEE Transactions on Puttern Analysis and Machine Intelligence, 36(3):453-44 465, 2014.
Chung-Wei Lec, Wei Fang, learning with structured knowledge graphs. arXiv preprint arXiv:1711.06526 2017.
Ych, and Yu-Chiang Frank Wang. Multi-label zero-shot
Chung-Wa Lee, Wei Fang, Chih-Kuan Ych, and Yu-Chiang Frank Wang. Multi-label zero-shot learning with structured knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1576-1585, 2018.
Xin Li, Yuhong Guo, and Dale Schuurmans. Semi-supervised zero-shot classification with label representation learning. In Proceedings of the IEEE International Conference on Computer Viston, pp. 1211-4219, 2015.
Xirong Li, Tiberio Uricchio, Lamberto Ballan, Marco Bertini, Cees GM Snoek, and Alberto Del Bimbo. Socializing the semantic gap: A comparative survey on image tag assignment, refinement, and retrieval. ACM Computing Surveys (CSUR), 49(1):1-39, 2016.
H Brendan McMahan, Geoffrey Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In Proceedings of the International Conference on Machine Learning, pp. 536 543, 2003.
Thomas Mensink, Efstratios Gavves, and Cees GM Snoek. Costa: Co-occurrence statistics for zero-shot classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2441-2448, 2014.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances neural information processing systems, pp. 3 11111111 11111199 2013.
Sanath Narayan, Akshita Gupta, Salman Khan, Fahad Shahbaz Khan, Ling Shao, and Mubarak Shah. Discriminative region-basee based multi-label zero-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8731 8740, 2021.
J von Neumann, Oskar Morgenstern, et al. Theory of games and economic behavior, 1944.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv: 13t::/312..00 13/2.5650, 2013.
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2751-2758. IEEE, 2012.
Yashoteja Prabhu and Manik Varma. Fastxml: A fast, accurate and stable tree-classifier for extreme multi-label learning. In Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 263-272. ACM, 2014.
Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. Less is more: zero-shot learning from online textual documents with noise suppression. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2249 22577, 2016.
Piyush Rai, Changwei Hu, Ricardo Henao, and Lawrence Carin. Large-scale bayesian multi-label learning via based label embeddings. In Advances in Neural Information Processing Systems, pp. 3222-3230, 2015.
12

---

Under review as a conference paper at ICLR 2023
Jorma Rissanen. A universal prior for integers and estimation by minimum description length. The Annals of statistics, pp. 416-431, 1983.
Marcus Rohrbach, Michael Stark, György Szarvas, Iryna Gurevych, and Bernt Schiele. What helps where-and why? semantic relatedness for knowledge transfer. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. pp. 910-917. IEEE, 2010.
Marcus Rohrbach, Michael Stark, and Bernt Schiele. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In Computer Vision and Pattern Recognition (CVPR), 2011 I IEEE Conference on, pp. 1641-1648. IEEE, 2011.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. International Conference on Machine Learning, pp. 2152-2161, 2015.
Ubai Sandouk and Ke Chen. Multi-label zero-shot learning via concept embedding. arXiv preprint (rXiv:/606.000 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15333 2014.
Maurice Sion. On general minimax theorems. Pacific Journal of mathematics, 8(1):111-776 1958.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In Advances in neural information processing systems, pp. 935 935-943, 2013.
Ben Taskar, Vassil Chatalbashev, Daphne Koller, and Carlos Guestrin. Learning structured prediction models: A large margin approach. In Proceedings of the International Conference Machine Learning, pp. 896-903. ACM, 2005.
Flemming Topsøe. Information thhooretcaa optimization techniques. Kybernetika, (5(1):8-27, 1979.
Yao-Hung Hubert Tsai, Liang-Kang Huang, and Ruslan Salakhutdinov. Learning robust visual- semantic embeddings. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 3591-3600. IEEE, 2017.
Grigorios Tsoumakas, Eleftherios Spyromitros-Xiautte Jozef Vilcek, and Ioannis Vlahavas. Mulan: java library for multi-label learning. Journal of Machine Learning Research, 12:2411-2414 2011.
A. Vedaldi. A MATLAB wrapper SVMstruct http://www.vlfeet wtw.feeett..og//eddaa./// code wete/vvn-seee st ruct- mmat 2011.
John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Bull. Amer. Math. Soc, 51 (1)4449855 1945.
Jiang Wang. Yi Yang. Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Can-rnn: A unified framework for lulti-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2285-2299 2016.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 6857-6866, 2018.
Jason Weston, Samy Bengio, and Philippe Hamel. Multi-tasking with joint semantic spaces for large-scale music annotation and retrieval. Journal of New Music Research, 40(4):337-348, 2011.
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 69-77, 2016.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4582-4591,
13

---

Under review as a conference paper at ICLR 2023
Yang Yang, Yadan Luo, Weilun Chen, Fumin Shen, Jie Shao, and Heng Tao Shen. Zero-shot hashing via transferring supervised knowledge. In Proceedings of the 2016 ACM on Multimedia Conference, pp. 1286-1295. ACM, 2016.
Chen Yc, Jian Wu, Victor S Sheng, Pengpeng Zhao, and Zhiming Cui. Multi-label active learning with label correlation for image classification. In Image Processing (ICIP), 2015 IEEE International Conference on, pp. 3437-3441. IEEE, 2015.
Meng Ye and Yuhong Guo. Multi-label zero-shot learning with transfer-aware label embedding projection, arXiv preprint arXiv:1808.022 02474, 2018.
Meng Ye and Yuhong Guo. Multi-label zero-shot learning with transfer-aware label embedding projection. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 3671-3675.
IEEE, 2019.
Hs Yu, Prateek Jain, Purushottam Kar, and Inderjit Dhillon. Large-scale multi-label learning with missing labels. In International conference on machine learning, pp. 593-6593, 2014.
Yang Zhang, Boqing Gong, and Mubarak Shah. Fast zero-shot image tagging. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5985-59944 IEEE, 2016.
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via semantic similarity embedding. In Proceedings of the IEEE international conference computer vision, pp. 4166-4174, 2015.
14


Reviews:
[{"review_id": "RSRmx9CZga", "reviewer": "Reviewer_TsJ7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper presents a novel method for zero-shot multi-label classification where the model is trained on seen classes and tested on unseen classes without further model tuning. The strategy uses min-max zer-sum game between the maximizer and the minimizer with minimum cost graph cuts. The proposed method has four additional losses besides the Hamming loss applied for each element of a target vector, namely taxonomy, joint, layer-wise, and pairwise losses.  The proposed method is proven superior compared to the state-of-the-art in multi-label ZSL on NUS-WIDE and Open Images.  \n", "summary_of_the_review": "This work has some potentials to tackle the problem of multi-label zero-shot learning looking at excellent performance compared to SOTA. However, the presentation is lacking of clarity, and there are also some claims that need to be tone-down to make the contributions clear.", "strength_and_weaknesses": "\nStrengths: \n- The proposed method is somewhat novel with a min-max graph cut strategy to tackle multi-label ZSL.\n- There are some rigorous experiments to show the effectiveness of the method and the generalization capability using the sharpness metric.\n- The performance is superior compared to some selected state-of-the-art methods for all experiments.\n\nWeaknesses:\n- The structured prediction for multi-label zero shot learning is not first proposed in this work, thus cannot be included as a contribution. Previous works by Lee et al. (“Multi-Label Zero-Shot Learning with Structured Knowledge Graphs”) and Huang et al., (“Multi-label Zero-shot Classification by Learning to Transfer from External Knowledge”) have proposed similar strategies using rstructured graph networks and relational graph convolutional networks, respectively.\n- This work has an Inconsistent term whether the problem is zero-shot multi-label classification or multilabel zero-shot learning.\n- As understood from the description in this work, there would be a drawback in representing the labels with low occurance and also create “blind” correlation without any causation between attributes and classes.\n- The notation and mathematical expressions of this paper are inconsistent, unstructured, and difficult to follow. Sometimes a label vector is denoted as $\\bold{y}$ and $\\bold{Y}$. Another thing is y^{entity} is written as y_{entity}. Variable x is sometimes bold.\n- The proposed method is not clear at all, the readability for this work is very low. After several times reading the mathematical expressions in Section 3, the proposed method is unclear. For instance, why does Eq. 3 uses abs(diff. between two embeddings from word vector) as the divisor? What are parameters to be optimized in Eq. 8 ($\\Theta^{i,j}$ is not clear)? \n- $P_{data}$, $P_{maxi}$ and $P_{mini}$ are not elaborated further in text.\n- Some abbreviations for comparing with all SOTA methods in all tables are not clear and referred in text (e.g., One Attention, LabelEM, and no explanation of M=10 for LESA). Please add complete references in each table.\n- There is no direct comparison with the methods with similar approaches e.g., Graph convolutional networks.\n- The “Discussion” paragraph in Page 7 has no discussion at all. There are only some references and the report of the results.\n- Some grammatical errors: by Mulan dataset, features values, 1) learn 2) recognizes 3) recognizes. Please review the manuscript in more details as there are notable mistakes in writing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NowO-TQBIZ3", "reviewer": "Reviewer_ZVzZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores a challenging task of zero-shot multi-label classification. To address this task, the authors propose to combine multiple types of feature representations to represent data, including Word-Net hierarchy, word2vec features, CNN layer-wise features and co-occurrence statistics. Based on these features, a new min-max zero-shot multi-label learning method is proposed to learn classifiers to predict test data for unseen classes. Experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "Overall, although there are merits in this paper, some issues affect its clarity and contributions. It is better for authors to address the concerns for further improving the quality of the paper.", "strength_and_weaknesses": "Strength:\n+ The explored task is highly challenging and beneficial to practical applications.\n+ Constructing class relations based on the graph structure is interesting.\n+ Empirical results seem good compared to the presented baselines.\n\nWeaknesses or questions:\n- All these feature representation manners are simply combined without in-depth analysis. Can you provide any insights on them? In addition, how to accurately construct the relationship between seen classes and unseen classes based on these features?\n- More related studies are welcomed. For example, visual-language models have recently shown highly promising performance for zero-shot learning. It would be better to discuss them.\n- Figure 4 is unclear. How to learn the classifier for making min-cut predictions for unseen classes during training?\n- It is better to give some visualization results for some images and their prediction probabilities for different unseen classes.\n In addition, please also provide some failure cases to discuss the limitations.\n- In Section 4.1, it is better to visualize the loss surface to see whether it is flatter than other baselines.\n \nMinor issues:\n- The format of in-text citations is not good and can be improved.\n- All figures are not formal and academic. The authors are suggested to make them more formal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-iHsdPDqEBa", "reviewer": "Reviewer_WnWm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method, Min-Max, for multi-label zero-shot learning (ML-ZSL). The proposed method is evaluated on multiple benchmarks, including ZSL, GZSL, and multi-label ZSL datasets, and shows compelling results.", "summary_of_the_review": "Summary: Given this paper's novelty and quality, I am inclined to accept it. However, I am not very familiar with this area, and I might be wrong regarding the novelty.\n", "strength_and_weaknesses": "Pros:\n1. The proposed method seems novel in the respective area\n2. The method achieves state-of-the-art performance on various benchmarks for ML-ZSL. It also performs very competitive results on conventional ZSL/GZSL. \n\nCons:\n1. The organization of the paper can be improved. I find myself having to go back and forth to understand certain parts when reading. The presentation of the formulae can also be improved to make them more straightforward.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject