# Practical Aspects of Zero-Shot Learning

Elie Saad, Marcin Paprzycki, Maria Ganzha

► To cite this version:

Elie Saad, Marcin Paprzycki, Maria Ganzha. Practical Aspects of Zero-Shot Learning. International Conference on Computational Science, Jun 2022, Londre, United Kingdom. pp.88-95, 10.1007/978-3-031-08754-7_12. hal-04454737

HAL Id: hal-04454737

https://hal.science/hal-04454737v1

Submitted on 27 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Practical Aspects of Zero-Shot Learning*

Elie Saad¹[0000-0002-5627-6351], Marcin Paprzycki²[0000-0002-8069-2152], and
Maria Ganzha¹[0000-0001-7714-4844]

¹ Warsaw University of Technology, Warsaw, Poland

elie.saad.stud@pw.edu.pl, maria.ganzha@pw.edu.pl

² Systems Research Institute Polish Academy of Sciences

marcin.paprzycki@ibspan.waw.pl

**Abstract.** One of important areas of machine learning research is zero-shot learning. It is applied when properly labeled training data set is not available. A number of zero-shot algorithms have been proposed and experimented with. However, none of them seems to be the “overall winner”. In situations like this, it may be possible to develop a meta-classifier that would combine “best aspects” of individual classifiers and outperform all of them. In this context, the goal of this contribution is twofold. First, multiple state-of-the-art zero-shot learning methods are compared for standard benchmark datasets. Second, multiple meta-classifiers are suggested and experimentally compared (for the same datasets).

**Keywords:** Zero-Shot Learning · Meta-Classifier · Performance analysis.

# 1 Introduction

Classic machine learning methods focus on classifying instances, when the classes have been “seen” during training. Whereas, many real world applications require classifying “entities” that have not been seen before, e.g., object recognition (where every object is a category), cross-lingual dictionary induction (where every word is a category), etc. Here, one of reasons for this situation is lack of resources to annotate the available (and, possibly, systematically growing) dataset. To solve this problem, zero-shot learning has been proposed. Hence, zero-shot learning can be seen as an example of “frugal AI”, where the frugality is caused by lack of data annotation, e.g. due to lack of resources.

While large number of zero-shot learning approaches have been proposed (see, for instance, [11,12,16,29,27,6,26]), they have been experimented with, mostly, in different contexts. Nevertheless, it is possible to conjecture that there is not a single “best” approach. In situations like this, meta-classifiers have been proposed. They receive suggestions from individual classifiers and “judge”, which

<footnote>* Research was funded in part by the Centre for Priority Research Area Artificial Intelligence and Robotics of Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme</footnote>
of them is the “best”. The assumption is that performance of a meta-classifier
will perform better than any of the individual ones.

In this context, the contribution of this work is twofold. First, a compre-
hensive comparison of five state-of-the-art zero-shot learning approaches applied
to five standard benchmarking datasets. Second, explorations into possibility of
developing a meta-classifier for zero-shot learning.

## 1.1 Problem Formulation

In zero-shot learning, instances in the feature space are represented as a vector
(usually, a real number vector). A subset of training instances is labeled, while
testing instances are not. The labeled training classes are referred to as seen
classes, while the unlabeled testing instances are referred to as unseen classes.
In this work, the seen and unseen datasets are assumed to be disjoint. However,
it is recognized that an interesting area of research covers also the case where
some (usually small) overlap between seen and unseen classes exists.

Let us now formulate the specific problem. Given a dataset of image em-
beddings $\mathcal{X} = \{(x_i, y_i) \in \mathcal{X} \times \mathcal{Y} | i = 1, ..., N_{tr} + N_{te}\}$, each image is a real
$D$-dimensional embedding vector comprised of features $x_i \in \mathbb{R}^D$, and each class
label is represented by an integer $y_i \in \mathcal{Y} \equiv \{1, ..., N_0, N_0 + 1, ..., N_0 + N_1\}$ giv-
ing $N_0 + N_1$ distinct classes. Here, for generality, it is assumed that $\mathcal{X} \stackrel{\text{def}}{=} \mathbb{R}^D$.
The dataset $\mathcal{X}$ is divided into two subsets: (1) training set and (2) test set.
The training set is given by $X^{tr} = \{(x_i^{tr}, y_i^{tr}) \in \mathcal{X} \times \mathcal{Y}_0 | i = 1, ..., N_{tr}\}$, where
$y_i^{tr} \in \mathcal{Y}_0 \equiv \{1, ..., N_0\}$, resulting in $N_0$ distinct training classes. The test set is
given by $X^{te} = \{(x_i^{te}, y_i^{te}) \in \mathcal{X} \times \mathcal{Y}_1 | i = N_{tr} + 1, ..., N_{te}\}$, where $y_i^{te} \in \mathcal{Y}_1 \equiv
\{N_0 + 1, ..., N_0 + N_1\}$ providing $N_1$ distinct test classes.

The goal of zero-shot learning is to train a model on the training set $X^{tr}$ that
performs “well” for the test set $X^{te}$. Obviously, zero-shot learning is infeasible
without auxiliary information associating labels from the training and the test
sets, when the training and test set classes are disjoint $\mathcal{Y}_0 \cap \mathcal{Y}_1 = \emptyset$. The proposed
solution is to represent each class label $y$ ($1 \le y \le N_0 + N_1$) by its prototype
$\pi(y) = p \in \mathcal{P} \stackrel{\text{def}}{=} \mathbb{R}^M$ (also known as its semantic embedding). Here, $\pi(\cdot) :$
$\mathcal{Y}_0 \cup \mathcal{Y}_1 \rightarrow \mathcal{P}$ is the prototyping function and $\mathcal{P}$ is the semantic embedding
space. The prototype vectors are such that any two class labels $y_0$ and $y_1$ are
similar if and only if their prototype representations $\pi(y_0) = p_0$ and $\pi(y_1) = p_1$
are close in the semantic embedding space $\mathcal{P}$. For example, their inner product
is large in $\mathcal{P}$, i.e, $\langle \pi(y_0), \pi(y_1) \rangle_{\mathcal{P}}$ is large. Prototyping all of class labels into a
joint semantic space, i.e., $\{\pi(y)|y \in \mathcal{Y}_0 \cup \mathcal{Y}_1\}$, results in labels becoming related.
This takes care of the problem of disjoint class sets, and the model can learn
from the labels in the training set, and predict labels from the test set.

## 2 Literature Review

Many approaches have been proposed for the zero-shot learning. Here, Deep Vi-
sual Semantic Embedding (DeViSE) [8], Attribute Label Embedding (ALE) [2],
and Structured Joint Embedding (SJE) [3] use a bilinear compatibility function. They follow the Stochastic Gradient Descent (SGD), where the SGD [19] is implicitly regularized by early stopping. The Embarrassingly Simple Approach to Zero-Shot Learning (ESZSL) [17] uses the square loss to learn the bilinear compatibility function and explicitly defines regularization with respect to the Frobenius norm. Kodirov et al. in [10] proposes a semantic encoder-decoder model, called Semantic AutoEncoder (SAE), where the training instances are projected into the semantic embedding space $\mathcal{P}$ with the projection matrix $W$, and then projected back into the feature space $\mathcal{X}$, with projection matrix $W^*$ – where $W^*$ is the conjugate transpose.

Another group of approaches adds a non-linearity component to the linear compatibility function. Here, Latent Embeddings (LATEM) [25] learns a piece-wise linear compatibility and uses the same ranking loss function as the DeViSE, as well as the SGD. LATEM learns multiple mappings corresponding to different visual characteristics of the data and uses a latent variable to select, which matrix mapping to use in predictions. Cross Modal Transfer (CMT) in [20] trains two-layer neural network using nonlinear mapping function tanh.

A third set of approaches uses probabilistic mapping [11]. Here, Direct Attribute Prediction (DAP) algorithm makes class prediction by combining scores of the learned attribute classifiers, whereas the Indirect Attribute Prediction (IAP) algorithm approximates the probabilities of the attributes associated with the training instances, by predicting probabilities of each training class value.

One more group of approaches expresses the input image features and the semantic embeddings as a mixture of seen classes. Here, Semantic Similarity Embedding (SSE) [30] connects class values within both semantic embedding space and feature embedding space of feature vectors. Convex Combination of Semantic Embeddings (ConSE) [14] learns the probability of a feature vector belonging to a specific class. Synthesized Classifiers (SynC) [5] first, learns a mapping from the semantic embedding space to the model space, which holds the classifiers associating real class values and phantom class values. The class values along with the set of phantom class values, which are introduced to connect the seen and unseen class values, form a weighted bipartite graph. The objective of SynC is to align the semantic embedding space and the model space, resulting in reducing the distortion error. The Generative Framework for Zero-Shot Learning (GFZSL) [23] applies generative modeling. Here, each class-conditional distribution is modeled as a multivariate Gaussian distribution.

In the fifth set of models, during training, both seen and unseen classes are included in the training data. The Discriminative Semantic Representation Learning (DSRL) [28] can be applied to any method that is based on a compatibility learning function. It learns the feature embedding vectors from the training instances with non-negative matrix factorization and aligns the feature embedding vectors with the semantic embedding vectors of their corresponding class values.
## 3 Selection of Methods and Experimental Setup

Based on analysis of the literature five zero-shot learning approaches have been selected: (1) DeViSE, (2) ALE, (3) SJE, (4) ESZSL, and (5) SAE. This was done, due to the fact that they have been found to be the “most competitive”.

In the literature, the following datasets have been found to be the most popular benchmarks for zero-shot learning: (a) Caltech-UCSD-Birds 200-2011 CUB [24], (b) Animals with Attributes 1 AWA1 [11], (c) Animals with Attributes 2 AWA2 [26], (d) Attribute Apascal&Ayahoo aPY [7], and (e) Scene UNderstanding SUN [15]. Hence, they were selected.

Due to lack of space, state-of-the-art in meta-classifiers has been omitted from the literature review. However, note that this work is focused on zero-shot learning. Hence, five standard meta-classifiers have been tried: (A) Meta-Decision Tree MDT [22], (B) deep neural network DNN [9], (C) Game Theory-based approach GT [1], (D) Auction-based model Auc [1], (E) Consensus-based approach Con [4], and (F) a simple majority voting MV [18]. Here, classifiers (C), (D), (E) and (F) have been implemented directly as in the cited literature. However, the implementation of (A) differs from the one described in [22] by not applying a third condition, which is the weight condition on the classifiers. This is due to the fact that all individual classifiers are applied simultaneously to the same datasets. Finally, for the DNN has been implemented a simple neural network with two hidden layers. The hidden layers and the output layer use the rectified linear activation function. The optimization function is the SGD algorithm, where the mean squared error loss function is used.

The code that has been used in experiments, and the list of hyperparameter values for both the individual classifiers and the meta-classifiers can be found in the Github repository<fnref n="3" />. The hyperparameter values were obtained through a series of ten experiments, after which, the hyperparameter values with the best resulting performance have been selected for the experimentation. We would like to emphasise that the hyperparameter values which were selected are not the best ones to be used, and better accuracy values can be obtained with different hyperparameter values.

The following standard measures have been used to measure the performance of explored approaches: (T1) Top-1, (T5) Top-5, (LogLoss) Logarithmic Loss, and (F1) F1 score. Their definitions can be found in [26,13,21]. The reason for the selection of these accuracy measurements is because of their usage abundance in the literature for studying models.

Finally, the issue of comparing the results of performed experiments with the state-of-the-art reported in [26] has to be addressed. To the best of our knowledge, the codes used there are not publicly available. Thus, the best effort was made to implement the methods using codes based on [8,2,3,17,10]. The known difference is that the feature vectors and the semantic embedding vectors, provided in the datasets were used, instead of calculated ones. Another difference between the code written for the experimentation and the original codes, is that

<footnote n="3">³ https://github.com/Inars/Developing MC for ZSL</footnote>
the dataset splits for the code written follow the proposed splits in [26], whereas the original codes use the standard splits. Nevertheless, we believe that the results, reported in what follows, fairly represent work reported in [26].

## 4 Experiments with Individual Classifiers

The first set of results was obtained using the five individual classifiers applied to the five benchmark datasets. Results displayed in Table 1 show the Top-1 accuracy. They include also those reported in [8,2,3,17,10] (the O column). The R column represents the replicated results based on [26]. The I columns represents the results of in-house implementations of the five approaches.

<table><caption>**Table 1.** Individual Classifier Performance for the Top-1 Accuracy</caption><thead><tr><th rowspan="2">CLF</th><th colspan="3">CUB</th><th colspan="3">AWA1</th><th colspan="3">AWA2</th><th colspan="3">aPY</th><th colspan="3">SUN</th></tr><tr><th>O</th><th>R</th><th>I</th><th>O</th><th>R</th><th>I</th><th>R</th><th>I</th><th>O</th><th>R</th><th>I</th><th>O</th><th>R</th><th>I</th></tr></thead><tbody><tr><td>DeViSE</td><td>-</td><td>52</td><td>46.82</td><td>-</td><td>54.2</td><td>53.97</td><td>59.7</td><td>57.43</td><td>-</td><td>37</td><td>32.55</td><td>-</td><td>56.5</td><td>55.42</td></tr><tr><td>ALE</td><td>26.3</td><td>54.9</td><td><strong>56.34</strong></td><td>47.9</td><td>59.9</td><td>56.34</td><td>62.5</td><td>51.89</td><td>-</td><td>39.7</td><td>33.4</td><td>-</td><td>58.1</td><td><strong>62.01</strong></td></tr><tr><td>SJE</td><td>50.1</td><td>53.9</td><td>49.17</td><td>66.7</td><td>65.6</td><td><strong>58.63</strong></td><td>61.9</td><td><strong>59.88</strong></td><td>-</td><td>31.7</td><td>31.32</td><td>-</td><td>52.7</td><td>52.64</td></tr><tr><td>ESZSL</td><td>-</td><td>51.9</td><td>53.91</td><td>49.3</td><td>58.2</td><td>56.19</td><td>58.6</td><td>54.5</td><td>15.1</td><td>38.3</td><td><strong>38.48</strong></td><td>65.8</td><td>54.5</td><td>55.63</td></tr><tr><td>SAE</td><td>-</td><td>33.3</td><td>39.13</td><td>84.7</td><td>53.0</td><td>51.5</td><td>54.1</td><td>51.77</td><td>-</td><td>8.3</td><td>15.92</td><td>-</td><td>40.3</td><td>52.71</td></tr></tbody></table>

Looking at column O, it can be seen that the SAE individual classifier is most accurate on the AWA1 dataset (84.7%; [10]). Moreover, results for EZSL for SUN dataset outperform all other classifiers (65.8%; [17]).

Since no other results, matching our setup, were available, we will now focus on these reported in column I, first. Here, the following observations can be made:

- DeViSE - got similar results for AWA1, AWA2, aPY, and SUN datasets, whereas there is a 5.18% decrease in accuracy for the CUB dataset.

- ALE - got similar results for CUB, AWA1, and SUN datasets, whereas there is a 10.61% and a 6.3% decrease in accuracy for AWA2 and aPY datasets.

- SJE - got similar results for CUB, AWA2, aPY, and SUN datasets, whereas there is a 6.97% decrease in accuracy for the AWA1 dataset.

- ESZSL - got similar results for all five datasets.

- SAE - got similar results for AWA1 and AWA2 datasets, whereas there is a 5.83%, 7.62% and a 12.41% increase in accuracy for CUB, aPY, and SUN datasets, respectively.

Moreover, ALE got the highest accuracy value for both the CUB and the SUN datasets (56.34% and 62.01%), SJE got the highest accuracy for both the AWA1 and the AWA2 datasets (58.63% and 59.88%), whereas ESZSL got the highest accuracy for the aPY dataset (38.48%). Note also that neither DeViSE nor SAE managed to outperform other classifiers for any dataset. All results
are around 50%, with the best of all being at 62.01%. Finally, it all classifiers performed “considerably worse” when applied to the *a*PY dataset.

Comparing results between columns *R* and *I* the first observation is that results are close to each other. Second, depending on the dataset and the method either of the two is slightly more accurate. Overall, out of 25 results, methods based on [26] are more accurate in fifteen cases, while implementations native to this work “win” in the remaining ten cases. This being the case, the remaining parts of this contribution are based on our implementation of individual classifiers. However, it should be noted that additional experiments have been performed and the same, general, pattern was observed.

While the Top-1 performance measure has its definite merits – usually one is interested in performance for the “key class” – other measures also have been used. Therefore, in Table 2 performance of individual classifiers measured using Top-5 accuracy have been presented.

<table><caption>**Table 2.** Individual Classifier Performance for the Top-5 Accuracy</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th></tr></thead><tbody><tr><td>DeViSE</td><td>51.45</td><td>65.34</td><td><strong>69.03</strong></td><td><strong>61.01</strong></td><td>57.78</td></tr><tr><td>ALE</td><td><strong>64.44</strong></td><td>63.44</td><td>63.01</td><td>60.88</td><td><strong>64.24</strong></td></tr><tr><td>SJE</td><td>54.27</td><td><strong>67.54</strong></td><td>68.54</td><td>60.82</td><td>55.14</td></tr><tr><td>ESZSL</td><td>58.25</td><td>64.91</td><td>64.52</td><td>57.93</td><td>57.92</td></tr><tr><td>SAE</td><td>44.97</td><td>65.56</td><td>66.98</td><td>49.29</td><td>56.18</td></tr></tbody></table>

Here, it can be observed that: (i) *DeViSE* obtained the highest accuracy for the *AWA2* and the *a*PY datasets (69.03% and 61.01%); (ii) *ALE* reached highest accuracy for *CUB* and *SUN* datasets (64.44% and 64.24%); whereas (iii) *SJE* obtained highest accuracy for *AWA1* dataset (67.54%). Results of Top-5 accuracy are about 10% higher than the results obtained for the Top-1 accuracy (Table 1). The best accuracy is close to 70%. Again, the *a*PY dataset remains the “hardest” also from the perspective of Top-5 measure.

The next set of results, reported in Table 3 presents that performance of the individual classifiers assessed from the perspective of Logarithmic Loss Accuracy.

<table><caption>**Table 3.** Individual Classifier Performance for the Logarithmic Loss Accuracy</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th></tr></thead><tbody><tr><td>DeViSE</td><td>3.36</td><td>2.2</td><td>1.67</td><td><strong>2.3</strong></td><td>3.69</td></tr><tr><td>ALE</td><td><strong>1.87</strong></td><td>1.88</td><td>1.92</td><td>2.53</td><td>3.68</td></tr><tr><td>SJE</td><td>2.93</td><td><strong>1.41</strong></td><td>1.49</td><td><strong>2.3</strong></td><td>3.4</td></tr><tr><td>ESZSL</td><td>2.36</td><td>1.49</td><td><strong>1.45</strong></td><td>7.04</td><td><strong>3.36</strong></td></tr><tr><td>SAE</td><td>3.6</td><td>2.06</td><td>2.03</td><td>2.51</td><td>3.92</td></tr></tbody></table>
Here, it can be noticed that: (a) SJE got the best accuracy for AWA1 and aPY datasets (1.41 and 2.3); (b) ESZSL obtained best accuracy for AWA2 and SUN datasets (1.45 and 3.36), whereas (c) DeViSE and SJE got the best accuracy for aPY dataset (2.3). Overall, the SUN and aPY datasets are the most difficult to form the perspective of Logarithmic Loss accuracy.

The final set of results, reported in Table 4, presents performance of individual classifiers assessed using F1 Score accuracy.

<table><caption>**Table 4. Individual Classifier Performance; F1 Score Accuracy**</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th></tr></thead><tbody><tr><td>DeViSE</td><td>0.47</td><td>0.48</td><td><strong>0.55</strong></td><td>0.27</td><td>0.55</td></tr><tr><td>ALE</td><td>0.53</td><td>0.53</td><td>0.42</td><td>0.3</td><td><strong>0.62</strong></td></tr><tr><td>SJE</td><td>0.49</td><td>0.55</td><td><strong>0.55</strong></td><td><strong>0.43</strong></td><td>0.53</td></tr><tr><td>ESZSL</td><td><strong>0.54</strong></td><td><strong>0.57</strong></td><td>0.53</td><td>0.25</td><td>0.56</td></tr><tr><td>SAE</td><td>0.39</td><td>0.44</td><td>0.45</td><td>0.08</td><td>0.53</td></tr></tbody></table>

In this case it can be seen that: (a) SJE is most accurate for AWA2 and aPY datasets (0.55 and 0.43); (b) ESZSL is the most accurate for CUB and AWA1 datasets (0.54 and 0.57); (c) ALE is the most accurate for SUN dataset (0.62); (d) DeViSE and SJE are the most accurate for AWA2 (0.55). Thus, aPY dataset, again, is the hardest to master.

Overall, it is clear that (A) different performance measures “promote” different zero-shot learning approaches; (B) aPY is most difficult to learn from; (C) as expected, none of individual classifiers can be seen as a clear winner. Therefore, a simplistic method has been proposed to gain better understanding of “overall strength” of individual classifiers. However, what follows “should be treated with a grain of salt”. There are many ways to “combine performance”.

In the proposed method, individual classifiers obtain points 5, 4, 3, 2, and 1 for best, second best, third best, fourth best, and fifth best accuracy obtained for each dataset. This process is repeated for all four accuracy measures. After points were added, they have been reported in Table 5.

<table><caption>**Table 5. Individual Classifier Combined Performance**</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th><th>Total</th></tr></thead><tbody><tr><td>DeViSE</td><td>8</td><td>8</td><td>17</td><td>16</td><td>11</td><td>60</td></tr><tr><td>ALE</td><td><strong>19</strong></td><td>11</td><td>7</td><td><strong>15</strong></td><td><strong>18</strong></td><td>70</td></tr><tr><td>SJE</td><td>12</td><td><strong>19</strong></td><td><strong>18</strong></td><td><strong>15</strong></td><td>8</td><td>72</td></tr><tr><td>ESZSL</td><td>17</td><td>14</td><td>14</td><td>11</td><td>17</td><td><strong>73</strong></td></tr><tr><td>SAE</td><td>4</td><td>8</td><td>8</td><td>7</td><td>6</td><td>33</td></tr></tbody></table>

Here, it can be observed that: (a) ALE performed the best overall on CUB and SUN datasets; (b) SJE performed best overall on AWA1 and AWA2 data-
sets; (c) SJE and ALE performed best on aPY dataset. The surprising result is that, while ESZSL is not a “winner” for a single dataset, it is the best overall performer, when combining results from all datasets (73 points). Finally, SAE is clearly the weakest method both within individual datasets and for combined performance. Combined performance of ALE, SJE, and EZSL is very similar.

Overall, reported experiments further confirm that not a single individual classifier outperforms others. Experimental results depend on the data and on used accuracy measure. These, in turn, are application dependent.

## 4.1 Analysis of the Datasets

Since the performance of the classifiers is directly related to the datasets, in this section, this relation is further elaborated. Specifically, for each dataset, its difficulty has been explored. Hence, for each instance, it is deemed to be: (a) *lvl 0* if it has been correctly identified by *no* individual classifier; (b) *lvl 1* if it has been identified by *one* classifiers; (c) *lvl 2* if it has been identified by *two* classifiers; (d) *lvl 3* if it has been identified by *three* classifiers; (e) *lvl 4* if it has been identified by *four* classifiers; (and f) *lvl 5* if it has been identified by *all* classifiers. The results in Table 6, show (in percents) how many instances belong to each category, for each dataset.

<table><caption>**Table 6.** Analysis of Instance Difficulty</caption><thead><tr><th>CLF</th><th>lvl 0</th><th>lvl 1</th><th>lvl 2</th><th>lvl 3</th><th>lvl 4</th><th>lvl 5</th></tr></thead><tbody><tr><th>CUB</th><td>23.86</td><td>15.98</td><td>13.11</td><td>12.4</td><td>15.23</td><td>19.41</td></tr><tr><th>AWA1</th><td>20.47</td><td>15.67</td><td>11.29</td><td>12.19</td><td><strong>19.33</strong></td><td>21.04</td></tr><tr><th>AWA2</th><td>21.88</td><td>12.95</td><td>13.46</td><td><strong>14.62</strong></td><td>12.26</td><td>22.84</td></tr><tr><th>aPY</th><td><strong>36.37</strong></td><td><strong>25.76</strong></td><td><strong>15.75</strong></td><td>11.75</td><td>9.53</td><td>0.85</td></tr><tr><th>SUN</th><td>19.31</td><td>12.78</td><td>10.69</td><td>12.5</td><td>16.88</td><td><strong>27.85</strong></td></tr></tbody></table>

Looking at Table 6, a number of observations can be made. (1) Difficulty of the *aPY* dataset is clear, as it has largest percents of instances that have not been recognized by a single classifier (36.37%), or by one or two classifiers only (jointly, 41.51%). At the same time, only 0.85% of its instances have been recognized by all classifiers. Overall only about 22% of its instances can be considered relatively easy to be recognized. These observations further augment analysis of the *aPY* dataset found in [26]. (2) On the other side of the spectrum is the *SUN* dataset, which not only has 27.85% of instances that have been correctly recognized by all classifiers. It also has about 55% of instances that can be considered as relatively easy. (3) Other than *aPY*, the remaining datasets have largest numbers of instances that are either most difficult (*lvl 0*) or easiest (*lvl 5*).

Approaching the issue from yet another side, “influence” of all attributes has been “scored”. For each correctly recognized instance, all its attributes have been given +1 “point”. For each incorrectly recognized instance, all its attributes
were given -1 “point”. Out of all attributes of correctly (incorrectly) recognized instances (treated separately), these with highest score have been marked as *easiest* (*hardest*). Results of this operation have been reported in Table 7. There, *total* refers to the result, in which all individual classifiers have been considered jointly; whereas *all* refers to all individual classifiers (and *total*), treating the same attribute *easiest* (*hardest*).

<table><caption>**Table 7.** Analysis of the Datasets</caption><thead><tr><th>CLF</th><th>Easiest Attribute</th><th>Individual Classifier</th><th>Hardest Attribute</th></tr></thead><tbody><tr><td>CUB</td><td>has eye color black</td><td>all</td><td>has eye color black</td></tr><tr><td rowspan="4">AWA1</td><td>old world</td><td>DeViSE; SAE</td><td>group</td></tr><tr><td>fast</td><td>ALE</td><td>ocean</td></tr><tr><td>old world</td><td>SJE; ESZSL</td><td>ocean</td></tr><tr><td>quadrupedal</td><td>total</td><td>swims</td></tr><tr><td rowspan="2">AWA2</td><td>old world</td><td>DeViSE; ALE; SJE; SAE; total</td><td>group</td></tr><tr><td>old world</td><td>ESZSL</td><td>ocean</td></tr><tr><td rowspan="3">aPY</td><td>metal</td><td>DeViSE; ALE; SJE; ESZSL</td><td>metal</td></tr><tr><td>head</td><td>SAE</td><td>metal</td></tr><tr><td>furry</td><td>total</td><td>metal</td></tr><tr><td>SUN</td><td>no horizon</td><td>all</td><td>man-made</td></tr></tbody></table>

Starting off with the *CUB* dataset: it can be seen that the attribute “has eye color black” is both the easiest and the hardest attribute. Inspecting images in the dataset, most (if not all) birds in the dataset seem to have that attribute in some form. This means that the “has eye color black” attribute will be associated with most instances that have been both correctly and incorrectly classified, regardless of which attribute is responsible for the result reported by individual classifiers. In this way, the “has eye color black” attribute can be seen as “useless” as it does not “help” in distinguishing between classes. Obviously, there may be also other attributes that are equally useless.

Looking at the *AWA1* dataset: it can be seen that the attribute “oldworld” is the easiest to classify, whereas the attribute that was easiest to classify by all individual classifiers combined “quadrupedal”. Similarly, the hardest attribute to classify is “ocean”, whereas the hardest to classify is “swims”.

Now, moving to the *AWA2* dataset. It is quite similar to the *AWA1* dataset. Here, again, the attribute “oldworld” is the easiest to classify. The hardest attribute to classify is “group”.

In the case of the *aPY* dataset, the easiest and hardest attribute to classify for individual classifiers is “metal”. The easiest attribute to be classified by DeViSE, ALE, SJE, and ESZSL is “metal”, by SAE is “head”, whereas for the five individual classifiers combined is “furry”. Judging by the images and their correlating prototypes, the attribute “metal” seems to be associated with the largest number of instances (similar to the attribute “has eye color black” from the *CUB* dataset), thus we come to the same conclusion that “metal” is classified
incorrectly regardless of whether it was the reason for the misclassification or not.

Finally, the *SUN* dataset, in which the easiest attribute to classify is “no horizon”, whereas the hardest attribute to classify “man-made”. Again, looking at the images from the dataset, it can be seen why that is since some instances under some classes such as “zen garden” are relatively hard to classify since their prototypes are similar – considering that they were engineered – making the attribute “man-made” the reason for the misclassification of a lot of instances.

In summary, results reported above lead to an interesting research question. What would happen if the most obvious/popular attributes were excluded from the dataset. However, exploring this question is out of scope of this contribution.

# 5 Meta-Classifiers

The last part of research concerns usability of basic meta-classifiers, as mechanisms to combine predictions made by individual classifiers. Before proceedings, let us note that the number of *hard* instances, found for each dataset and reported in Section 3, establishes the hard ceiling for: *DNN*, *MDT*, and *MV*. Specifically, here, if not a single individual classifier gave a correct answer, their combination will also not be able to do so. Therefore, for these meta-classifiers, for the 5 datasets, the performance limits are *CUB*: 76.13%, *AWA1*: 79.53%, *AWA2*: 78.13%, *aPY*: 63.63%, and *SUN*: 80.7%.

In Table 8, performance of the six meta-classifiers is compared using the Top-1 measure, where the **Best** row denotes the best result obtained from the “winning” individual classifier, for given dataset (see, Table 1).

<table><caption>**Table 8.** Meta-classifier performance; Top-1 accuracy</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th></tr></thead><tbody><tr><td>MV</td><td>53.43</td><td>58.71</td><td>56.56</td><td>32.72</td><td>61.94</td></tr><tr><td>MDT</td><td>47.89</td><td>56.43</td><td>51.89</td><td>33.40</td><td>62.08</td></tr><tr><td>DNN</td><td>48.63</td><td>57.56</td><td>54.72</td><td>34.89</td><td>60.63</td></tr><tr><td>GT</td><td>46.58</td><td>56.75</td><td>59</td><td>32.63</td><td>59.51</td></tr><tr><td>Con</td><td>46.82</td><td>53.97</td><td>57.43</td><td>32.55</td><td>55.42</td></tr><tr><td>Auc</td><td>47.89</td><td>56.34</td><td>51.89</td><td>33.40</td><td>62.01</td></tr><tr><td>Best</td><td>56.34</td><td>58.63</td><td>59.88</td><td>38.48</td><td>62.01</td></tr></tbody></table>

Here, *MV* achieved highest accuracy for *CUB* and *AWA1* datasets (53.43% and 58.71%); *MDT* achieved highest accuracy for *SUN* dataset (62.08%); *DNN* got the highest accuracy for *aPY* dataset (34.89%); finally, *GT* reached highest accuracy for *AWA2* dataset (59%). Moreover, neither *Con* nor *Auc* managed to get the best accuracy for a single dataset.

Comparing results obtained by meta-classifiers with the best individual classifiers, one can see that: (a) the best individual classifier performed better than
the best meta-classifier on CUB, AWA2, and aPY (2.91%, 0.88%, and 3.59% better); (b) best meta-classifier performed better than the best individual classifier on AWA1 and SUN datasets (0.08% and 0.08% better).

The second view of the results, for the meta-classifiers, was obtained when the F1 measure was applied. The results are summarized in Table 9.

<table><caption>**Table 9.** Meta-classifier Performance; F1 Score Accuracy</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th></tr></thead><tbody><tr><td>MV</td><td>0.54</td><td>0.55</td><td>0.52</td><td>0.34</td><td>0.62</td></tr><tr><td>MDT</td><td>0.48</td><td>0.53</td><td>0.42</td><td>0.31</td><td>0.62</td></tr><tr><td>DNN</td><td>0.49</td><td>0.53</td><td>0.46</td><td>0.28</td><td>0.6</td></tr><tr><td>GT</td><td>0.36</td><td>0.26</td><td>0.37</td><td>0.24</td><td>0.34</td></tr><tr><td>Con</td><td>0.47</td><td>0.26</td><td>0.36</td><td>0.27</td><td>0.55</td></tr><tr><td>Auc</td><td>0.48</td><td>0.53</td><td>0.42</td><td>0.31</td><td>0.62</td></tr><tr><td>Best</td><td>0.54</td><td>0.57</td><td>0.55</td><td>0.43</td><td>0.62</td></tr></tbody></table>

Here, MV achieved highest accuracy for CUB, AWA1, AWA2, and aPY datasets (0.54, 0.55, 0.52, and 0.34); MV, MDT, and Auc achieved highest accuracy for SUN dataset (0.62). Moreover, neither DNN nor GT nor Con managed to get the best accuracy for a single dataset.

Comparing results obtained by meta-classifiers with the best individual classifiers, one can see that: (a) the best individual classifier performed better than the best meta-classifier on AWA1, AWA2, and aPY (0.02, 0.03, and 0.09 better); (b) best meta-classifier performed equally to the best individual classifier, on CUB and SUN datasets (0.54 and 0.62).

Using the same method of point distribution, used for measuring the “combined performance” of the individual classifiers, results reported in Table 10 have been obtained (combining both performance measures – Top-1 and F1). In this case, the top scorer obtained 6 points, as there are six meta-classifiers.

<table><caption>**Table 10.** Meta-Classifier combined performance</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th><th>Total</th></tr></thead><tbody><tr><td>MV</td><td>12</td><td>12</td><td>10</td><td>10</td><td>10</td><td>54</td></tr><tr><td>MDT</td><td>8</td><td>8</td><td>6</td><td>10</td><td>12</td><td>44</td></tr><tr><td>DNN</td><td>10</td><td>10</td><td>8</td><td>10</td><td>8</td><td>46</td></tr><tr><td>GT</td><td>4</td><td>8</td><td>9</td><td>5</td><td>5</td><td>31</td></tr><tr><td>Con</td><td>6</td><td>5</td><td>7</td><td>5</td><td>5</td><td>28</td></tr><tr><td>Auc</td><td>8</td><td>7</td><td>6</td><td>10</td><td>11</td><td>42</td></tr></tbody></table>

Here it can be noticed that: (a) MV performed the best overall (54 points); (b) Con performed the worst overall (28 points); (c) MV, MDT, and DNN out
performed *GT*, *Con*, and *Auc*, while their performance was quite similar (42–46 points).

It should be noted that the “simplest” approaches from both the individual classifiers (ESZSL) and the meta-classifiers (MV) obtained the highest number of points using the “combined performance” method in both Table 10 and Table 5.

Finally, the “combination method” was applied jointly to both the meta-classifiers and the individual classifiers. Here, Top-1 and the F1 accuracy measures were taken into account. Moreover, since 11 classifiers were compared, the top score was 11 points. Table 11 displays the resulting total scores.

<table><caption>**Table 11.** Meta-Classifier and individual classifier combined performance</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th><th>Total</th></tr></thead><tbody><tr><td>DeViSE</td><td>13</td><td>12</td><td>20</td><td>12</td><td>13</td><td>70</td></tr><tr><td>ALE</td><td><b>21</b></td><td>15</td><td>11</td><td>17</td><td>21</td><td>85</td></tr><tr><td>SJE</td><td>17</td><td><b>20</b></td><td><b>22</b></td><td>16</td><td>10</td><td>85</td></tr><tr><td>ESZSL</td><td>21</td><td>16</td><td>16</td><td>16</td><td>15</td><td>84</td></tr><tr><td>SAE</td><td>10</td><td>10</td><td>11</td><td>7</td><td>11</td><td>49</td></tr><tr><td>MV</td><td>20</td><td>21</td><td>17</td><td><b>18</b></td><td>20</td><td><b>96</b></td></tr><tr><td>MDT</td><td>15</td><td>16</td><td>11</td><td><b>18</b></td><td><b>22</b></td><td>82</td></tr><tr><td>DNN</td><td>17</td><td>18</td><td>15</td><td>17</td><td>18</td><td>85</td></tr><tr><td>GT</td><td>10</td><td>14</td><td>15</td><td>11</td><td>13</td><td>63</td></tr><tr><td>Con</td><td>13</td><td>10</td><td>13</td><td>12</td><td>13</td><td>61</td></tr><tr><td>Auc</td><td>15</td><td>15</td><td>11</td><td><b>18</b></td><td>21</td><td>80</td></tr></tbody></table>

Here, it can be noticed that: (a) *MV* performed the best overall (96 points, better by 11 points than the next three classifiers: *ALE*, *SJE*, and *DNN*); (b) *SAE* performed the worst overall (28 points); (c) among competitive approaches, one can list also *ESZL* and *MDT*, and *Auc*; (d) meta-classifiers performed better overall than the individual classifiers (averaging 77.83 vs. 74.6 points). Obviously, since meta-classifiers combined results of individual classifiers, there is a lot of “methodological grey area”. Recognizing this we can still state that when results from multiple individual classifiers are available, then they can be combined using *simple majority voting* to improve overall performance of classification process. At the same time, use of other (more advanced) meta-classifiers is not straightforwardly leading to performance gains.

# 6 Concluding Remarks

In this contribution, performance of five zero-shot learning models has been studied (*DeViSE*, *ALE*, *SJE*, *ESZSL*, and *SAE*), when applied to standard benchmarking datasets. Moreover, the practical nature of the difficulties of these datasets has been explored. Finally, use of six standard meta-classifiers (MV, MDT, DNN, GT, Con, and Auc) has been experimented with.
The main findings were: (1) there is not a single best classifiers, and their
performance depends on the, application-specific, dataset and applied perfor-
mance measure; (2) the *aPY* dataset is the most difficult (of the five that have
been tried) to achieve good results of zero-shot learning; (3) use of standard
meta-classifiers may brings some overall benefits; (4) despite explicit perfor-
mance ceiling, the *MV*, *MDT*, and *DNN* classifiers, they performed better than
their counterparts; (5) simplest methods obtained best results (both the indi-
vidual classifier *ESZSL* and the meta-classifier *MV*).

Relatively low quality of obtainable results (accuracy of prediction of order of less than 70%) suggests that a lot of research is needed for both individual zero-shot learning approaches and, possibly, meta-classifiers. Moreover, work should be verified on datasets similar to *aPY*, which are particularly difficult to apply to. Here, some attention should be devoted to the role that individual attributes play in class (instance) recognition.

References

1. Abreu, M.d.C., Canuto, A.M.: Analyzing the benefits of using a fuzzy-neuro model in the accuracy of the neurage system: An agent-based system for classification tasks. In: The 2006 IEEE International Joint Conference on Neural Network Pro- ceedings. pp. 2959–2966. IEEE (2006)

2. Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for im- age classification. IEEE transactions on pattern analysis and machine intelligence 38(7), 1425–1438 (2015)

3. Akata, Z., Reed, S., Walter, D., Lee, H., Schiele, B.: Evaluation of output embed- dings for fine-grained image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2927–2936 (2015)

4. Alzubi, O.A., Alzubi, J.A.A., Tedmori, S., Rashaideh, H., Almomani, O.: Consensus-based combining method for classifier ensembles. Int. Arab J. Inf. Technol. 15(1), 76–86 (2018)

5. Changpinyo, S., Chao, W.L., Gong, B., Sha, F.: Synthesized classifiers for zero-shot learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5327–5336 (2016)

6. Ding, Z., Shao, M., Fu, Y.: Low-rank embedded ensemble semantic dictionary for zero-shot learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2050–2058 (2017)

7. Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their at- tributes. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 1778–1785. IEEE (2009)

8. Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., Mikolov, T.: Devise: A deep visual-semantic embedding model. Advances in neural informa- tion processing systems 26 (2013)

9. Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. MIT press (2016)

10. Kodirov, E., Xiang, T., Gong, S.: Semantic autoencoder for zero-shot learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3174–3183 (2017)

11. Lampert, C., Nickisch, H., Harmeling, S.: Attribute-based classification for zero-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence 36(3) (2013)
12. Larochelle, H., Erhan, D., Bengio, Y.: Zero-data learning of new tasks. In: AAAI. vol. 1, p. 3 (2008)

13. Mannor, S., Peleg, D., Rubinstein, R.: The cross entropy method for classification. In: Proceedings of the 22nd international conference on Machine learning. pp. 561–568 (2005)

14. Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S., Dean, J.: Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650 (2013)

15. Patterson, G., Hays, J.: Sun attribute database: Discovering, annotating, and recognizing scene attributes. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 2751–2758. IEEE (2012)

16. Rohrbach, M., Stark, M., Schiele, B.: Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In: CVPR 2011. pp. 1641–1648. IEEE (2011)

17. Romera-Paredes, B., Torr, P.: An embarrassingly simple approach to zero-shot learning. In: International conference on machine learning. pp. 2152–2161. PMLR (2015)

18. Ruta, D., Gabrys, B.: Classifier selection for majority voting. Information fusion 6(1), 63–81 (2005)

19. Shalev-Shwartz, S., Ben-David, S.: Understanding machine learning: From theory to algorithms. Cambridge university press (2014)

20. Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C.D., Ng, A.Y.: Zero-shot learning through cross-modal transfer. arXiv preprint arXiv:1301.3666 (2013)

21. Sokolova, M., Japkowicz, N., Szpakowicz, S.: Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation. In: Australasian joint conference on artificial intelligence. pp. 1015–1021. Springer (2006)

22. Todorovski, L., Džeroski, S.: Combining classifiers with meta decision trees. Machine learning 50(3), 223–249 (2003)

23. Verma, V.K., Rai, P.: A simple exponential family framework for zero-shot learning. In: Joint European conference on machine learning and knowledge discovery in databases. pp. 792–808. Springer (2017)

24. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-ucsd birds 200 (2010)

25. Xian, Y., Akata, Z., Sharma, G., Nguyen, Q., Hein, M., Schiele, B.: Latent embeddings for zero-shot classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 69–77 (2016)

26. Xian, Y., Lampert, C.H., Schiele, B., Akata, Z.: Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence 41(9), 2251–2265 (2018)

27. Xu, X., Shen, F., Yang, Y., Zhang, D., Tao Shen, H., Song, J.: Matrix tri-factorization with manifold regularizations for zero-shot learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3798–3807 (2017)

28. Ye, M., Guo, Y.: Zero-shot classification with discriminative semantic representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7140–7148 (2017)

29. Yu, X., Aloimonos, Y.: Attribute-based transfer learning for object categorization with zero/one training example. In: European conference on computer vision. pp. 127–140. Springer (2010)

30. Zhang, Z., Saligrama, V.: Zero-shot learning via semantic similarity embedding. In: Proceedings of the IEEE international conference on computer vision. pp. 4166–4174 (2015)