# Generalized Zero-Shot Learning for Image Classification-Comparing Performance of Popular Approaches

Elie Saad, Marcin Paprzycki, Maria Ganzha, Amelia Bădică, Costin Bădică, Stefka Fidanova, Ivan Lirkov, Mirjana Ivanović

► To cite this version:

Elie Saad, Marcin Paprzycki, Maria Ganzha, Amelia Bădică, Costin Bădică, et al. Generalized Zero-Shot Learning for Image Classification-Comparing Performance of Popular Approaches. Information, 2022, 13 (12), pp.561. 10.3390/info13120561. hal-04454711

HAL Id: hal-04454711

https://hal.science/hal-04454711v1

Submitted on 26 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Generalized Zero-shot Learning for Image Classification – comparing performance of popular approaches

Elie Saad <fnref n="1" />, Marcin Paprzycki <fnref n="2" />, Maria Ganzha <fnref n="1" />, Amelia Bădică<fnref n="3" />, Costin Bădică<fnref n="3" />, Stefka Fidanova<fnref n="4" />, Ivan Lirkov<fnref n="4" />, and Mirjana Ivanović<fnref n="5" />

**Abstract:** There are many areas, where conventional supervised machine learning does not work well. For instance, in the cases with large, or systematically increasing, number of countably infinite classes. This is where zero-shot learning has been proposed. Moreover, in generalized settings, the zero-shot learning problem represents real-world applications where some test instances are present during inference. Separately, recently, increase of interest in meta-classifiers, which "combine" results from individual ones, to improve overall classification quality, can be observed. In these contexts, the purpose of this contribution is twofold. First, performance of five state-of-the-art, generalized zero-shot learning methods is compared, for five popular benchmark datasets. Second, six standard meta-classification approaches are experimented with. In performed experiments, all meta-classifiers are applied to the same datasets, and their performance is compared to each other, and to the original classifiers.

**Keywords:** Zero-Shot Learning, Generalized Zero-Shot Learning, Meta-Classifier, Performance comparison

## 1. Introduction

Large number of machine learning methods focus on "classifying instances", and they are explored in the case when the classes have been "seen" during the model training phase. Here, "seen" means that at least a single (correctly labelled) instance of each existing class has been present in the training dataset. Obviously, best results of model training are obtained when available, labelled, datasets are "large" and when the number of instances in each class is "balanced".

However, realistic real-world applications may require classifying instances belonging to classes that have not been seen before. Here, the examples include, but are not limited to, object recognition (where every object is a class; [1]), cross-lingual dictionary induction (where every word is a class [2,3]), "mind reading" using magnetic resonance imaging scans of the brain (where every word is a class, and every instance is a brain scan [4]).

In this context, zero-shot learning (ZSL) has been proposed. It is a class of model training approaches that aims at dealing with the situation when the classes, covered by the training instances, and the classes to be classified, are disjoint [5–10].

In zero-shot learning, some instances, in the feature space, are given. In practice, these instances are represented as a vector (usually a real number vector), placed within a feature space (usually a real number space). The set of all instances consists of a subset of instances, which are labeled, and the second subset of instances, which are not labeled. The

**Citation:** Saad, E.; Paprzycki, M.; Ganzha, M.; Bădică, A.; Bădică, C.; Fidanova, S.; Lirkov, I.; Ivanović, M. Generalized Zero-shot Learning for Image Classification – Comparing Performance of Popular Approaches. AI 2022, 1, 1–16. https://doi.org/

Received:

Accepted:

Published:

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2024 by the authors. Submitted to AI for possible open access publication under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

<footnote n="1">¹ Department of Mathematics and Information Science, Warsaw University of Technology, Warsaw, Poland; elie.saad.stud@pw.edu.pl, maria.ganzha@pw.edu.pl</footnote>

<footnote n="2">² Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland; marcin.paprzycki@ibspan.waw.pl</footnote>

<footnote n="3">³ University of Craiova, Craiova, Romania; amelia.badica@edu.ucv.ro, costin.badica@edu.ucv.ro</footnote>

<footnote n="4">⁴ Institute of Information and Communication Technologies, Bulgarian Academy of Sciences, Sofia, Bulgaria; ivan.lirkov@iict.bas.bg, stefka.fidanova@iict.bas.bg</footnote>

<footnote n="5">5 University of Novi Sad, Faculty of Sciences, Novi Sad, Serbia; mira@dmi.uns.ac.rs</footnote>

<footnote>* Correspondence: elie.saad.stud@pw.edu.pl;</footnote>
classes corresponding to the labeled instances are referred to as seen classes, comprising the training set and a part of the testing set. On the other hand, classes corresponding to the unlabeled instances are referred to as unseen classes. They constitute the remaining part of the testing set. Note that the set of classes that correspond to the training instances and the set of classes that correspond to the testing instances may not be disjoint. Here, it is reasonably assumed that this represents the real-world circumstances more accurately. Zero-shot learning, where it is not assumed that training and testing classes are disjoint is called generalized zero-shot learning (GZSL). It is important to note that GZSL differs from ZSL only in the “relationships” between the training and the testing datasets.

This being the case, machine learning approaches that have been used to solve the ZSL problem have been also applied to the GZSL [11]. However, to the best of our knowledge, for both ZSL and GZSL, proposed model training methods have been evaluated “separately”. Moreover, even if they have been compared with each other (in a “sub-group”), different datasets have been used, in reported experiments. Finally, even if all datasets and classifiers were “the same”, only a single performance measure (e.g., Top-1) has been used. However, as shown in [12,13], in the case of ZSL solvers, different performance measures give different answers to the question: which ZSL approach is the “most effective”. Therefore, the first goal of this contribution is to experimentally compare performance of five different approaches to solving the GZSL problem, when the same (most popular in the literature) benchmark datasets are used. In these comparisons, four different performance measures are applied.

Separately, one can easily observe (see, for instance, [14]) that existence of large number of approaches that can be applied to the same machine learning task, results in attempts to develop meta-classifiers. Here, the term meta-classifier denotes software that takes, as its input, results from individual classifiers and “combines them” with a hope to obtain result(s) that is(are) better than the best individual one(s). Hence, the second goal of the reported experimental work is to apply standard (state-of-the-art) meta-classifiers and establish if either one of them can improve performance over the individual GZSL approaches.

In this context, we proceed as follows. First, precise problem formulation is provided in Section 2. Next, brief summary of pertinent literature is included (in Section 3). Comprehensive description of experimental setup is provided in Section 4. It is followed by description and analysis of obtained experimental results (Section 5). Section 6 summarizes the main findings and points to future research directions.

## 2. Problem formulation

Across the literature, the following mathematical formulation of the GZSL problem has been accepted. Note that this definition is geared towards image processing/classification task, which is the focus of the experimental work reported in this contribution. However, after only minimal modifications, it will hold also in more general settings. Assume the following:

* let $X^s = \{(x_i^s, y_i^s)_{i=1}^{N_s} \in X^s \times T^s\}$ be the set of seen instances

* and let $X^u = \{(x_i^u, y_i^u)_{i=1}^{N_u} \in X^u \times T^u\}$ be the set of unseen instances,

* where $T^s = \{1, ..., N^{Ts}\}$ is the set of all seen class labels

* and $T^u = \{N^{Ts} + 1, ..., N^{Ts} + N^{Tu}\}$ is the set of all unseen class labels,

* such that $\mathcal{X} = X^s \cup X^u$ is the set of all instances where $X^s \cap X^u = \emptyset$

* and $\mathcal{T} = T^s \cup T^u$ is the set of all class labels where $T^s \cap T^u = \emptyset$ resulting in $N^{Ts} + N^{Tu}$ distinct classes.

Given a dataset of image embeddings and their corresponding class labels $\mathcal{D} = \{(x_i, y_i) \in \mathcal{X} \times \mathcal{T} | i = 1, ..., N_{tr} + N_{te}\}$, each image is a real $D$-dimensional embedding vector comprised of features $x_i \in \mathbb{R}^D$, and each class label is represented by an integer $y_i \in \mathcal{T}$. The following is denoted $\mathcal{X} \stackrel{\text{def}}{=} \mathbb{R}^D$, for generality. The division of the dataset $\mathcal{D}$ results in two subsets: the training and the test sets.
* The training set – is defined as $X^{tr} = \{(x_i^{tr}, y_i^{tr}) \in \mathcal{X} \times T^{tr} | i = 1, ..., N_{tr}\}$, such that $y_i^{tr} \in T^{tr} \subset \mathcal{T}$. Let $Y^{tr} = \{y_i^{tr} \in T^{tr} | i = 1, ..., N_{tr}\}$ be the set of class labels corresponding to the training set $X^{tr}$.

* The test set – is defined as $X^{te} = \{(x_i^{te}, y_i^{te}) \in \mathcal{X} \times T^{te} | i = N_{tr} + 1, ..., N_{tr} + N_{te}\}$, such that $y_i^{te} \in T^{te} \subset \mathcal{T}$. Let $Y^{te} = \{y_i^{te} \in T^{te} | i = N_{tr} + 1, ..., N_{tr} + N_{te}\}$ be the set of class labels to be predicted, corresponding to the set $X^{te}$.

The goal of generalized zero-shot learning is to deliver a classifier that is trained on the training set $X^{tr}$ and that is able to deliver "reasonably good" classification performance on the test set $X^{te}$. It is quite apparent that the GZSL setting is nearly infeasible without any auxiliary information associating the labels of the classes from both the training set and the test set, since some of the training classes are used in conjunction with all of the test classes for testing.

One of the solutions to this problem is to represent each class label $y$ where $1 \le y \le N^{Ts} + N^{Tu}$ by its prototype $\pi(y) = p \in \mathcal{P} \stackrel{\text{def}}{=} \mathbb{R}^M$ – also known in the literature as its semantic embedding – where $\pi(\cdot)$ is the prototyping function, and $\mathcal{P}$ is the semantic space, or the semantic embedding space. The resulting prototype vectors, obtained from the prototyping function, are obtained in such a way that any two class labels $y_0$ and $y_1$ are considered similar if and only if their corresponding prototype representations $\pi(y_0) = p_0$ and $\pi(y_1) = p_1$ are relatively close in the semantic embedding space $\mathcal{P}$. To make this concept more specific, consider the following example: the inner product of the resulting similar prototype vectors in $\mathcal{P}$ is large, i.e., $\langle\pi(y_0), \pi(y_1)\rangle_{\mathcal{P}}$ is large. Prototyping all of the class labels into a joint semantic space i.e., $\{\pi(y) | y \in T^{tr} \cup T^{te}\}$, forces the labels to become related. By consequence, this alleviates prior problem of having nearly disjoint class sets, so that the model can learn from the training set, and test its predictions from the test set.

### 3. Related work

Multiple approaches have been proposed to solve the ZLS/GZSL problem. An extended state-of-the-art summary can be found in [13]. Here, we will focus attention mostly on work directly related to the GZSL problem. Moreover, since only standard state-of-the art meta-classifiers will be experimented with, their detailed description is omitted (it can be found in [13]). However, details concerning their implementation are summarized in Section 4.2.

In general, as of today, there are five classes of approaches to the solution of the ZSL/GZSL problem [11].

1. The first utilizes a linearity factor, i.e., is based on the learning function that is linear. Here, for instance, Deep Visual Semantic Embedding (DeViSE) [15], Attribute Label Embedding (ALE) [16], and Structured Joint Embedding (SJE) [17] algorithms use a bilinear compatibility function, i.e., a symmetrically bilinear mapping, where the Stochastic Gradient Descent [18] method is used for learning and is implicitly regularized by early stopping. The Embarrassingly Simple Approach to Zero-Shot Learning (ESZSL) [19] uses the square loss error, to learn the bilinear compatibility function between image embedding and class values, i.e, class labels, and explicitly defines regularization, which regularizes the unregularized risk minimization formulation with respect to the Frobenius norm. In the Semantic AutoEncoder (SAE) model, the training instances are projected into the semantic embedding space $\mathcal{P}$ with the projection matrix $\mathbf{W}$, and then projected back into the feature space $\mathcal{X}$, with projection matrix $\mathbf{W}^*$ – where $\mathbf{W}^*$ is the conjugate transpose.

2. Second group of approaches adds a non-linearity component to the linear compatibility function. Here, Latent Embeddings (LATEM) [20] learns a piecewise linear compatibility and uses the same ranking loss function as the DeViSE approach, combined with the stochastic gradient descent. LATEM learns multiple mappings corresponding to different visual characteristics of the data, and uses a latent variable to select, which matrix mapping to use in predictions. Cross Modal Transfer (CMT) in [21] trains two-layer neural network using nonlinear mapping function tanh.
3. The third group of approaches uses probabilistic embeddings mapping to the class values [5]. Here, Direct Attribute Prediction (DAP) algorithm makes class prediction by combining scores of the learned attribute classifiers, whereas the Indirect Attribute Prediction (IAP) algorithm approximates the probabilities of the attributes associated with the training instances, by predicting probabilities of each training class value.

4. The fourth group of approaches expresses the input image features and the semantic embeddings as a mixture of seen classes. Here, Semantic Similarity Embedding (SSE) [22] connects class values within both the semantic embedding space and the feature embedding space of the feature vectors. Convex Combination of Semantic Embeddings (ConSE) [23] learns the probability of a feature vector belonging to a specific class. Synthesized Classifiers (SynC) [24] first, learns a mapping from the semantic embedding space to the model space, which holds the classifiers associating real class values and, what is called, phantom class values. The class values, along with the set of phantom class values, which are introduced to connect the seen and unseen class values, form a weighted bipartite graph. The objective of SynC is to align the semantic embedding space and the model space, resulting in reducing the distortion error. The Generative Framework for Zero-Shot Learning (GFZSL) [25] applies generative modeling. Here, each class-conditional distribution is modeled as a multivariate Gaussian distribution.

5. Finally, in the fifth set of models, during training, both seen and unseen classes are included in the training data. The Discriminative Semantic Representation Learning (DSRL) [26] can be applied to any method that is based on a compatibility learning function, i.e., a mapping of the image embeddings to class labels. It learns the feature embedding vectors from the training instances with non-negative matrix factorization and aligns the feature embedding vectors with the semantic embedding vectors of their corresponding class values.

Taking into account the results of the literature review, and the goals of the project that this work is a part of, the following question had to be confronted: “Which classifiers should be experimented with?” While testing them all is tempting, it would require a lot of resources and would not fit into a single report. Upon further reflection, it was decided that testing representatives of all five groups of approaches (e.g. one per group) could result in a lot of questions concerning the selection process and the fairness of comparison (since each class of approaches the ZSL/GZSL problem from very different perspective). Therefore, it has been decided that a reasonable first step would be to comprehensively explore approaches belonging to the first class, where large number of methods has been already proposed. Hence, five individual classifiers, belonging to the first class of approaches, and implementing different linear factors, i.e., DeVISE, ALE, SJE, ESZSL, and SAE, have been selected. This decision was further supported by the fact that all these approaches have been explored in [11], which could be treated as a base, against which further (relatively fair) comparison of results is made.

## 4. Experimental setup

In the following sections, the key aspects of the experimental setup are discussed: (1) implementation details of the ZSL/GZSL algorithms, (2) implementation details of the meta-classifiers, (3) performance measures applied to the obtained results, and (4) datasets used in the experiments.

### 4.1. Individual classifier implementations

Individual classifiers were taken from the sources reported in the respective publications (see, Section 3). Details concerning their implementation have been reported in [12,13]. In the context of GZSL, before proceeding with the actual experiments, “light hyperparameter tuning” was performed. Specifically, for each classifier and for each dataset (see, Section 4.4) series of ten experiments has been completed. Each time, hyperparameters were slightly shifted, in the direction where better results have been anticipated
(see, also, [12,13]). In these experiments, the Top-1 accuracy measure has been used (see, Section 4.3) to assess performance. The final values of the hyperparameters, for each of the individual classifiers, for each dataset, have been summarized in Table 1. Here, it has to be stressed that, while these hyperparameter values delivered the “best accuracy” for the performed experiments, it is not claimed that they are the “optimal ones”. It is possible (and quite likely) that there exist hyperparameters that would result in a better performance of each individual classifier (for each dataset and, as it will be shown, for each separate performance measure). However, searching the “complete hyperparameter space” was out of scope of this work. Instead it was assumed that a “reasonably good” set of hyperparameters should be applied for each classifier and dataset, and used to compare classifier performance for considered performance measures.

<table><caption>**Table 1.** Hyperparameters used in experiments for each individual classifiers and datasets.</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th></tr></thead><tbody><tr><td rowspan="3">DeViSE</td><td>lr= 1</td><td>lr= 0.01</td><td>lr= 0.001</td><td>lr= 1</td><td>lr= 0.01</td></tr><tr><td>mr= 1</td><td>mr= 200</td><td>mr= 150</td><td>mr= 1</td><td>mr= 3</td></tr><tr><td>norm= L<sup>2</sup></td><td>norm=std</td><td>norm=std</td><td>norm= L<sup>2</sup></td><td>norm=None</td></tr><tr><td rowspan="2">ALE</td><td>lr= 0.3</td><td>lr= 0.01</td><td>lr= 0.01</td><td>lr= 0.04</td><td>lr= 0.1</td></tr><tr><td>norm= L<sup>2</sup></td><td>norm= L<sup>2</sup></td><td>norm= L<sup>2</sup></td><td>norm= L<sup>2</sup></td><td>norm= L<sup>2</sup></td></tr><tr><td rowspan="3">SJE</td><td>lr= 0.1</td><td>lr= 1</td><td>lr= 1</td><td>lr= 0.01</td><td>lr= 1</td></tr><tr><td>mr= 4.0</td><td>mr= 2.5</td><td>mr= 2.5</td><td>mr= 1.5</td><td>mr= 2</td></tr><tr><td>norm=std</td><td>norm= L<sup>2</sup></td><td>norm= L<sup>2</sup></td><td>norm=None</td><td>norm=std</td></tr><tr><td rowspan="2">ESZSL</td><td>γ = 3</td><td>γ = 3</td><td>γ = 3</td><td>γ = 2</td><td>γ = 3</td></tr><tr><td>λ = 0</td><td>λ = 0</td><td>λ = 0</td><td>λ = 0</td><td>λ = 2</td></tr><tr><td rowspan="2">SAE</td><td>λ<sub>1</sub> = 80</td><td>λ<sub>1</sub> = 3.2</td><td>λ<sub>1</sub> = 0.8</td><td>λ<sub>1</sub> = 0.16</td><td>λ<sub>1</sub> = 0.32</td></tr><tr><td>λ<sub>2</sub> = 0.2</td><td>λ<sub>2</sub> = 0.8</td><td>λ<sub>2</sub> = 0.2</td><td>λ<sub>2</sub> = 2.56</td><td>λ<sub>2</sub> = 0.08</td></tr></tbody></table>

In Table 1, $lr$ denotes the learning rate; $\lambda_1$ and $\lambda_2$ are the weighting coefficients of the Sylvester equation, for the encoder and the decoder, respectively. The image embedding normalization functions that have been used are: the Euclidean norm denoted as $L^2$, and standardization of features, which removes the mean and scales to unit variance. The latter, for the training instance vector $x$, is defined as follows:

$$||x||_{\text{std}} = \frac{x - \bar{x}}{s(x)}, \qquad (1)$$

where $\bar{x}$ is the mean of the training instances defined as

$$\bar{x} = \frac{\sum_{i=1}^{N_{tr}} x_i}{N_{tr}}, \qquad (2)$$

and $s(\cdot)$ is the standard deviation of the training instances, defined as follows:

$$s(x) = \sqrt{\frac{\sum_{i=1}^{N_{tr}} (x_i - \bar{x})^2}{N_{tr}}}. \qquad (3)$$

Finally, $mr$ is the margin value, which is used by each individual classifier. It applies to the stochastic gradient descent approach for optimization in DeViSE, ALE, and SJE.

## 4.2. Meta-classifier implementations

Six standard state-of-the-art meta-classifiers have been tested, namely Meta-Decision Tree (MDT) [27], Deep Neural Network (DNN) [28], Game Theory-based approach (GT) [29], an Auction-based model (Auc) [29], a Consensus-based approach (Con) [30], and a simple
Majority Voting (MV) [31]. Here, GT, Auc, Con and MV classifiers have been implemented exactly as described in the cited literature.

The implementation of the MDT differs from that described in [27], by not applying the weight condition on the individual classifiers. The weight condition is defined as a function, which returns the fraction of the training examples that have been used by the individual classifier to estimate the class distribution for a given training instance. Since, in the case of performed experiments, each individual classifier uses the entirety of the dataset, weighting is irrelevant.

Finally, a simple neural network has been implemented for the DNN. It had two hidden layers using the rectified linear activation function. The stochastic gradient descent algorithm is applied as an optimization function for the network, where the mean squared error loss function is used.

Again, as in the case of individual classifiers, the goal was not to create the "most powerful" meta-classifier for a given context, but to implement "standard" state-of-the-art meta-classifiers and reflect on their potential, in the context of GZSL. Therefore, the same approach that has been used to obtain the hyperparameters for the individual classifiers (see [12,13] and Section 4.1), was applied to obtain the MDT hyperparameters, which have been summarized in Table 2.

<table><caption>**Table 2.** Meta Decision Tree Hyperparameters.</caption><thead><tr><th>Hyperparameter</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th></tr></thead><tbody><tr><td>α</td><td>1</td><td>0.013</td><td>0.185</td><td>0.0101</td><td>3.21</td></tr><tr><td>β</td><td>1</td><td>0.16632</td><td>0.185</td><td>0.159171</td><td>3.21</td></tr></tbody></table>

The α hyperparameter is the maximum displacement value and the β hyperparameter is the entropy displacement value. The learning rate of the DNN meta-classifier has been set to $1 \times 10^{-4}$.

The implementation of both the individual and meta-classifiers has been done from scratch, where the resulting code that was used in the experiments, can be found in the Github repository<fnref n="1" />.

### 4.3. Performance measures

In all the performed experiments, the accuracy of the classifiers has been assessed using four accuracy measures: Top-1 (T1), Top-5 (T5), Logarithmic Loss (LogLoss), and the F1 score (F1).

The T1 accuracy, as described in [11], was measured for the single label image classification. In this case, the prediction is accurate whenever the predicted class corresponds to the ground truth. However, it is easy to realize that for the ZSL/GZSL one is interested in obtaining high performance, even on sparsely populated classes. Therefore, the T1 accuracy is measured by averaging the class accuracy of each class independently, in the following way:

$$ accy = \frac{1}{N^{Ts} + N^{Tu}} \sum_{c=1}^{N^{Ts} + N^{Tu}} \frac{\# \text{ correct predictions in } c}{\# \text{ samples in } c} \quad (4) $$

The T5 accuracy was measured in the same way as the T1 accuracy. Here the prediction is considered correct whenever one of the top five predicted classes is true. In the context of ZSL/GZSL, the T1 accuracy can be seen as a measure of "exact precision", while the T5 accuracy measures the "robustness" of the approach.

Next, the F1 score [32], which calculates the precision and robustness, i.e., recall, of the model, has been used. Finally, since this research focuses on the generalized zero-shot learning – where the search space, at evaluation time, is composed of training and test

<footnote n="1">¹ https://github.com/Inars/Zero-Shot_Learning_for Object_Recognition_and_Classification</footnote>
classes (where $c_i \in \mathcal{T}$), one more accuracy measure is typically applied. Specifically, the harmonic mean of the training accuracy and test accuracy is computed. The harmonic mean is selected, instead of the arithmetic mean, because the harmonic mean is not affected (significantly) by the potentially large difference(s) between the training accuracy and the testing accuracy [33]. The harmonic mean can be seen as a way to capture variance in the differences.

Note that most predictions of the models are not probabilistic, therefore the prediction values have been converted into a probabilistic distribution, by applying the normalized exponential function [34], also known as the softmax function. The normalized exponential function $\sigma$ is a function that normalizes the input vector $z = (z_1, ..., z_k)^T \in \mathbb{R}^k$ into a probability distribution consisting of $k$ probabilities.

### 4.4. *Datasets used in experiments*

Among the many datasets found in the literature, in the context of GZSL problem, five of the most widely used have been selected for this study, namely: Caltech-UCSD-Birds 200-2011 (CUB) [35], Scene UNderstanding (SUN) [36], Animals with Attributes 1 (AWA1) [5], Animals with Attributes 2 (AWA2) [11], and Attribute Apascal&Ayahoo (aPY) [37].

Each of these five datasets forms a continuous attribute space. All of the datasets consist of image embeddings. In the experimentation, the dataset splitting followed what was suggested in [11], and is represented in Table 3. Here the scalars $N_{lr}^{te}$ and $N_{le}^{te}$ denote the number of instances used during the testing phase, the type of dataset is displayed in the “detail” column, and the number of semantic attributes is displayed in the “M” column.

<table><caption>**Table 3.** Statistics for the Datasets.</caption><thead><tr><th>DS</th><th>Detail</th><th>M</th><th>N<sup>Ts</sup> + N<sup>Tu</sup></th><th>|T<sup>tr</sup>|</th><th>|T<sup>te</sup>|</th><th>N<sub>lr</sub> + N<sub>te</sub></th><th>N<sub>lr</sub></th><th>N<sub>lr</sub><sup>te</sup></th><th>N<sub>le</sub><sup>te</sup></th></tr></thead><tbody><tr><td>CUB</td><td>fine</td><td>312</td><td>200</td><td>100 + 50</td><td>50</td><td>11788</td><td>7057</td><td>1764</td><td>2967</td></tr><tr><td>AWA1</td><td>coarse</td><td>85</td><td>50</td><td>27 + 13</td><td>10</td><td>30475</td><td>19832</td><td>4958</td><td>5685</td></tr><tr><td>AWA2</td><td>coarse</td><td>85</td><td>50</td><td>27 + 13</td><td>10</td><td>37322</td><td>23527</td><td>5882</td><td>7913</td></tr><tr><td>aPY</td><td>coarse</td><td>64</td><td>32</td><td>15 + 5</td><td>12</td><td>15339</td><td>5932</td><td>1483</td><td>7924</td></tr><tr><td>SUN</td><td>fine</td><td>102</td><td>717</td><td>580 + 65</td><td>72</td><td>14340</td><td>10320</td><td>2580</td><td>1440</td></tr></tbody></table>

The aPY dataset, consists of two datasets – the Pascal dataset and the Yahoo dataset – merged into one. It is a coarse-grained dataset consisting of 32 classes. Of the 32 classes available, 20 classes provided from the Pascal dataset are used for training. The 12 classes, provided by the Yahoo dataset, are used for testing. Of the 20 classes from the Pascal dataset, 5 classes are selected randomly to serve as the validation classes. The aPY dataset consists of 64 attributes, describing the classes forming the semantic embedding space.

The AWA1 dataset is a coarse-grained dataset, consisting of 50 classes. Of these 50 classes, 40 classes are used for training, where 13 were selected randomly for validation, and the remaining 10 classes are used for testing. The AWA1 dataset consists of 85 attributes. In terms of number of image instances, the AWA1 dataset consists of a total of 30,475 image instances.

Similarly to AWA1, AWA2 was introduced in [11] for use as an alternative, since the images on the AWA1 dataset are not available for public use. The AWA2 dataset is publicly available. Here, recall that in this work the images themselves were not actually needed, as only features that were extracted from them are used. This is why wee were able to experiment with both AWA1 and AWA2 datasets. AWA2 is a coarse-grained dataset, consisting of 50 of the same classes used in the AWA1 dataset. The AWA2 dataset consists of the same 85 attributes, used in the AWA1 dataset. Unlike AWA1, the AWA2 dataset consists of 37,322 image instances from public web sources, protected under the free-use and redistribution licence, all of which do not overlap with the images from the AWA1 dataset. Similarly to how AWA1 is split, of the 50 available classes in AWA2, 40 classes
are used for training, where 13 are selected randomly for validation, and the remaining 10 classes are used for testing.

The CUB dataset is a fine-grained dataset in terms of both the number of classes and the number of images, consisting of 200 total classes of bird species and 11,788 bird images. Of the 200 available classes, 150 classes are used for training where 50 of which are selected randomly for validation, and the remaining 50 classes are used for testing. The classes in the CUB dataset are annotated with 312 attributes forming the continuous attribute space.

The SUN dataset is a fine-grained dataset, in terms of both the number of classes and the number of images available therein. It consists of 717 classes of different scenes, and a total of 14,340 images. Of the 717 available classes, 645 classes are used for training, where 65 are selected randomly for validation, and the remaining 72 classes are used for testing. The classes in the SUN dataset are annotated with 102 attributes forming the continuous attribute space.

All five datasets, used in the experimentation, already include the feature vectors, which contain the features extracted from the image instances provided within each dataset. Since the feature extraction, to form the feature space, is not the focus of this work, it is assumed that datasets provided in [11] are sufficient. Therefore, various feature extraction techniques, such as ResNet, have note been applied (e.g. as a preprocessing step) to the images provided in the datasets.

# 5. Experimental results and their analysis

As stated above, the first goal of the performed experiments was to evaluate and compare the behavior of the five individual classifiers. The second goal was to explore the performance of the six meta-classifiers, including comparison of their performance to the individual classifiers.

## 5.1. *Experiments with individual classifiers*

The first set of results represents the T1 accuracy, and is summarized in Table 4. Here, as defined in Section 2, $Y^{tr}$ represents the set of training class labels, $Y^{te}$ denotes the set of testing class labels, while $\hat{H}$ represents the harmonic mean of the two (as described in Section 4.3).

The first “group of rows” copies results found in [11] (where only the T1 measurement was used to assess the performance), whereas the second group of rows displays the results obtained during performed experiments (In-House Implementation). Moreover, the first row of the results reported for the SAE classifier (for the In-House Implementation), represents accuracy obtained from the encoder, whereas the second row represents accuracy obtained from the decoder. Finally, for each classifier, and each dataset, the “best results” are marked in a bold font.

The following observations can be formulated on the basis of results presented in Table 4, when experiments based on In-House Implementation are compared with results found in [11]. Note that if the difference between results is within ±5%, they are considered to be “close enough to be considered to be indistinguishable”. This is stipulated due to the fact that, as stated earlier, the aim of this work is not to fine-tune hyperparameters but to fairly compare performance of different approaches to GZSL.

* In-House Implementation of *DeViSE* – obtained a 7.29% increase in accuracy (over results reported in [11]), for the **AWA1** dataset.

* ALE – reached a 9.2% increase in accuracy for the **aPY** dataset.

* SJE – delivered an 8.98% increase in accuracy for the **aPY** dataset.

* ESZSL – got similar results on all five datasets.

* SAE – got an 8.09%, a 6.42%, a 7.22%, an 11.87%, and an 8.2% increase in accuracy for the **CUB**, **AWA1**, **AWA2**, **aPY**, and **SUN** datasets, respectively.

In summary, it can be stated that the In-House Implementation resulted in a slightly better performance than the results reported in [11]. However the results were relatively close (within 10%) and, most likely, the difference originated from differences in hyperpa-
<table><caption>**Table 4.** Individual Classifier Results for the Top-1 Accuracy.</caption><thead><tr><th>DS</th><th colspan="2">CUB</th><th colspan="2">AWA1</th><th colspan="2">AWA2</th><th colspan="2">aPY</th><th colspan="2">SUN</th></tr><tr><th>CLF</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th></tr></thead><tbody><tr><td colspan="14" style="text-align:center;">Results reported in [11]</td></tr><tr><th>DeViSE</th><td>23.8</td><td>52</td><td>32.8</td><td>13.4</td><td>68.7</td><td>22.4</td><td>17.1</td><td>74.7</td><td><strong>27.8</strong></td><td>3.5</td><td>78.4</td><td>6.7</td><td>16.9</td><td>27.4</td><td>20.9</td></tr><tr><th>ALE</th><td>23.7</td><td>62.8</td><td><strong>34.4</strong></td><td>16.8</td><td>76.1</td><td><strong>27.5</strong></td><td>14</td><td>81.8</td><td>23.9</td><td>4.6</td><td>73.7</td><td><strong>8.7</strong></td><td>21.8</td><td>33.1</td><td><strong>26.3</strong></td></tr><tr><th>SJE</th><td>23.5</td><td>59.2</td><td>33.6</td><td>11.3</td><td>74.6</td><td>19.6</td><td>8</td><td>73.9</td><td>14.4</td><td>1.3</td><td>71.4</td><td>2.6</td><td>14.4</td><td>29.7</td><td>19.4</td></tr><tr><th>ESZSL</th><td>14.7</td><td>56.5</td><td>23.3</td><td>6.6</td><td>75.6</td><td>12.1</td><td>5.9</td><td>77.8</td><td>11</td><td>2.4</td><td>70.1</td><td>4.6</td><td>11</td><td>27.9</td><td>15.8</td></tr><tr><th>SAE</th><td>7.8</td><td>54</td><td>13.6</td><td>1.8</td><td>77.1</td><td>3.5</td><td>1.1</td><td>82.2</td><td>2.2</td><td>0.4</td><td>80.9</td><td>0.9</td><td>8.8</td><td>18</td><td>11.8</td></tr><tr><td colspan="14" style="text-align:center;">In-House Implementation</td></tr><tr><th>DeViSE</th><td>23.41</td><td>61.96</td><td>33.98</td><td>18.01</td><td>84.41</td><td><strong>29.69</strong></td><td>17.3</td><td>71.78</td><td><strong>27.88</strong></td><td>4.12</td><td>76.28</td><td>7.8</td><td>18.54</td><td>32.75</td><td>23.68</td></tr><tr><th>ALE</th><td>26.07</td><td>62.74</td><td><strong>36.83</strong></td><td>13.46</td><td>80.14</td><td>23.05</td><td>12.15</td><td>77.59</td><td>21.01</td><td>10.26</td><td>69.85</td><td><strong>17.9</strong></td><td>23.68</td><td>37.13</td><td><strong>28.92</strong></td></tr><tr><th>SJE</th><td>22.82</td><td>60.8</td><td>33.19</td><td>9.93</td><td>79.29</td><td>17.64</td><td>10.48</td><td>78.82</td><td>18.5</td><td>6.26</td><td>73.31</td><td>11.54</td><td>18.75</td><td>33.37</td><td>24.01</td></tr><tr><th>ESZSL</th><td>14.7</td><td>56.53</td><td>23.34</td><td>5.29</td><td>86.84</td><td>9.98</td><td>4.04</td><td>88.83</td><td>7.72</td><td>2.25</td><td>81.07</td><td>4.39</td><td>13.75</td><td>28.41</td><td>18.53</td></tr><tr><th>SAE</th><td>13.86</td><td>49.88</td><td>21.69</td><td>5.29</td><td>80.52</td><td>9.92</td><td>5</td><td>81.42</td><td>9.42</td><td>8.28</td><td>27.97</td><td>12.77</td><td>16.81</td><td>24.69</td><td>20</td></tr><tr><th></th><td>15.72</td><td>57.02</td><td>24.64</td><td>14.72</td><td>82.93</td><td>25</td><td>13.86</td><td>87.2</td><td>22.41</td><td>9.48</td><td>56.62</td><td>16.24</td><td>19.03</td><td>31.2</td><td>23.64</td></tr></tbody></table>

parameter tuning. Overall, it can be stated that these results support each other as being fair estimate of current state-of-the-art accuracy of GZSL solvers for T1 performance measure.

Looking at the results originating from the In-House Implementation of individual classifiers (see, Section 4.1), reported in the “bottom part” of Table 4 one can see that: (i) ALE got the highest accuracy value on the CUB, SUN, and aPY datasets (38.83%, 17.9% and 28.92%), whereas DeViSE reached the highest accuracy values on both AWA1 and AWA2 datasets (29.69% and 27.88%). Another aspect, worthy pointing out, is that (ii) neither SJE, ESZSL, nor SAE managed to outperform the remaining classifiers, for either of the five datasets. Additionally, (iii) for the harmonic means results, the only ones that are above 30% are DeViSE, ALE, and SJE for the CUB dataset. Moreover, (iv) it is easy to notice that all of the individual classifiers performed considerably worse on the aPY dataset than on the other datasets. This result matches that reported in [12,13] for the ZSL settings. It also matches the pattern of results reported in [11].

Finally, (v) there is a significant drop in accuracy, across the board, compared to the results from the ZSL experiments, reported in [12,13]. The reason for this drop is the availability of the seen classes – taken from the source domain $T^{tr}$ – during the testing phase in the target domain $T^{te}$. In other words, the fact that $T^{tr} \cap T^{te} \neq \emptyset$. Here, the seen classes act as a “deterrence” for the individual classifiers, since they were trained to exclusively classify seen classes from the source domain, while no unseen class from the target domain was present during the training phase. This fact gives the seen classes “an edge” over their counterparts, i.e., an increased bias, during the prediction process. This results in the seen classes having a higher selection rate than their unseen counterparts. As a consequence, this reduces the overall accuracy of the individual classifiers.

The remaining experimental results, reported in what follows, could not have been compared to others, as – to the best of our knowledge – no (fully) comparable results (obtained for the same approaches, with the same datasets, with the same performance measures) exist in the literature. The first set of results has been obtained for the T5 accuracy, and is reported in Table 5. Here, again, the “best results” are marked in a bold font.

The following observations follow from results reported in Table 5. (A) DeViSE managed to obtain the highest performance on AWA1, AWA2, and aPY datasets (81.22%, 79.09% and 71.65%), whereas (B) ALE obtained best overall performance on both CUB and SUN datasets (75.71% and 62.57%). Additionally, (C) all of the reported results are less than 80%. (D) The SUN dataset seems to be the “hardest” dataset when the T5 performance measure is applied. Finally, (E) SAE performed the worst. Most of the values of SAE, for the T5 measure, are relatively similar to one another across all five datasets. This should
<table><caption>**Table 5. Individual Classifier Results for the T5 Accuracy.**</caption><thead><tr><th>DS</th><th colspan="2">CUB</th><th colspan="2">AWA1</th><th colspan="2">AWA2</th><th colspan="2">aPY</th><th colspan="2">SUN</th></tr><tr><th>CLF</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th></tr></thead><tbody><tr><td>DeViSE</td><td>59.6</td><td>87.6</td><td>70.9</td><td>69.3</td><td>98.0</td><td><strong>81.2</strong></td><td>66.9</td><td>96.8</td><td><strong>79.1</strong></td><td>58.3</td><td>93.0</td><td><strong>71.7</strong></td><td>47.3</td><td>62.8</td><td>54.0</td></tr><tr><td>ALE</td><td>65.6</td><td>89.5</td><td><strong>75.7</strong></td><td>23.1</td><td>80.1</td><td>60.0</td><td>62.7</td><td>97.8</td><td>76.4</td><td>54.2</td><td>98.1</td><td>69.8</td><td>57.1</td><td>69.2</td><td><strong>62.6</strong></td></tr><tr><td>SJE</td><td>62.8</td><td>86.7</td><td>72.9</td><td>66.0</td><td>96.6</td><td>78.4</td><td>66.3</td><td>97.0</td><td>78.8</td><td>51.0</td><td>93.3</td><td>65.9</td><td>48.3</td><td>64.1</td><td>55.1</td></tr><tr><td>ESZSL</td><td>58.7</td><td>86.5</td><td>69.9</td><td>63.9</td><td>97.8</td><td>77.3</td><td>63.0</td><td>98.4</td><td>76.8</td><td>40.9</td><td>97.8</td><td>57.7</td><td>42.5</td><td>58.4</td><td>49.2</td></tr><tr><td>SAE</td><td>13.9</td><td>51.9</td><td>21.8</td><td>15.1</td><td>83.2</td><td>25.5</td><td>14.6</td><td>84.0</td><td>24.8</td><td>16.6</td><td>36.5</td><td>22.8</td><td>19.0</td><td>25.0</td><td>21.6</td></tr><tr><td></td><td>15.7</td><td>57.8</td><td>24.7</td><td>23.1</td><td>84.6</td><td>36.3</td><td>21.3</td><td>88.4</td><td>34.3</td><td>17.8</td><td>57.9</td><td>27.2</td><td>21.0</td><td>31.5</td><td>25.2</td></tr></tbody></table>

be taken into account when comparing the values of SAE for the T5 measurement to the values for the T1 measure.

The next set of results reports performance of the five classifiers, measured using the LogLoss accuracy. The results are summarized in Table 6. Note that, here, the lowest value is the best (and the best results are is marked in bold font).

<table><caption>**Table 6. Individual Classifier Results for the LogLoss Accuracy.**</caption><thead><tr><th>DS</th><th colspan="2">CUB</th><th colspan="2">AWA1</th><th colspan="2">AWA2</th><th colspan="2">aPY</th><th colspan="2">SUN</th></tr><tr><th>CLF</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th></tr></thead><tbody><tr><td>DeViSE</td><td>5.24</td><td>5.23</td><td>5.23</td><td>3.48</td><td>3.00</td><td>3.22</td><td>3.38</td><td>2.95</td><td>3.15</td><td>3.47</td><td>3.31</td><td>3.39</td><td>6.53</td><td>6.53</td><td>6.53</td></tr><tr><td>ALE</td><td>5.29</td><td>5.29</td><td>5.29</td><td>3.90</td><td>3.90</td><td>3.90</td><td>3.91</td><td>3.91</td><td>3.91</td><td>3.56</td><td>3.44</td><td>3.50</td><td>6.57</td><td>6.57</td><td>6.57</td></tr><tr><td>SJE</td><td>4.90</td><td>4.40</td><td>4.62</td><td>3.78</td><td>3.78</td><td>3.78</td><td>3.81</td><td>3.77</td><td>3.79</td><td>3.47</td><td>3.47</td><td>3.47</td><td>6.88</td><td>6.48</td><td>6.67</td></tr><tr><td>ESZSL</td><td>3.78</td><td>3.43</td><td><strong>3.59</strong></td><td>2.93</td><td>2.18</td><td><strong>2.50</strong></td><td>2.92</td><td>2.13</td><td><strong>2.46</strong></td><td>4.67</td><td>1.79</td><td><strong>2.57</strong></td><td>5.63</td><td>5.57</td><td><strong>5.60</strong></td></tr><tr><td>SAE</td><td>4.98</td><td>4.92</td><td>4.95</td><td>3.69</td><td>3.51</td><td>3.6</td><td>3.67</td><td>3.46</td><td>3.56</td><td>3.38</td><td>3.03</td><td>3.20</td><td>6.21</td><td>6.18</td><td>6.19</td></tr><tr><td></td><td>5.05</td><td>5.01</td><td>5.03</td><td>3.66</td><td>3.51</td><td>3.58</td><td>3.63</td><td>3.43</td><td>3.52</td><td>3.5</td><td>2.87</td><td>3.15</td><td>6.23</td><td>6.19</td><td>6.21</td></tr></tbody></table>

Comparing the results in Table 6, the following observations can be made. (a) ESZSL got the lowest values on both AWA1 and aPY datasets (1.41 and 2.3), while (b) ESZSL managed to get the lowest values across the board for the CUB, AWA1, AWA2, aPY, and SUN datasets (3.59, 2.5, 2.46, 2.57, and 5.6). When evaluated from the logloss perspective the SUN dataset seems to be the most difficult to deal with, again.

Finally, in Table 7, the performance of the five individual classifiers is compared in terms of the F1 measure. Here, the best (highest) values are marked in bold font.

<table><caption>**Table 7. Individual Classifier Results for the F1 Accuracy.**</caption><thead><tr><th>DS</th><th colspan="2">CUB</th><th colspan="2">AWA1</th><th colspan="2">AWA2</th><th colspan="2">aPY</th><th colspan="2">SUN</th></tr><tr><th>CLF</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th></tr></thead><tbody><tr><td>DeViSE</td><td>0.24</td><td>0.62</td><td>0.34</td><td>0.15</td><td>0.88</td><td><strong>0.26</strong></td><td>0.09</td><td>0.80</td><td><strong>0.16</strong></td><td>0.03</td><td>0.76</td><td>0.06</td><td>0.19</td><td>0.33</td><td>0.24</td></tr><tr><td>ALE</td><td>0.26</td><td>0.62</td><td><strong>0.37</strong></td><td>0.12</td><td>0.87</td><td>0.22</td><td>0.07</td><td>0.86</td><td>0.13</td><td>0.05</td><td>0.74</td><td><strong>0.10</strong></td><td>0.24</td><td>0.37</td><td><strong>0.29</strong></td></tr><tr><td>SJE</td><td>0.23</td><td>0.6</td><td>0.33</td><td>0.1</td><td>0.86</td><td>0.18</td><td>0.07</td><td>0.86</td><td>0.13</td><td>0.03</td><td>0.74</td><td>0.06</td><td>0.19</td><td>0.33</td><td>0.24</td></tr><tr><td>ESZSL</td><td>0.15</td><td>0.56</td><td>0.23</td><td>0.07</td><td>0.9</td><td>0.13</td><td>0.04</td><td>0.92</td><td>0.08</td><td>0.01</td><td>0.83</td><td>0.02</td><td>0.14</td><td>0.28</td><td>0.19</td></tr><tr><td>SAE</td><td>0.26</td><td>0.50</td><td>0.22</td><td>0.06</td><td>0.83</td><td>0.11</td><td>0.05</td><td>0.85</td><td>0.09</td><td>0.05</td><td>0.23</td><td>0.09</td><td>0.17</td><td>0.25</td><td>0.20</td></tr><tr><td></td><td>0.26</td><td>0.59</td><td>0.36</td><td>0.28</td><td>0.86</td><td>0.43</td><td>0.21</td><td>0.90</td><td>0.35</td><td>0.05</td><td>0.70</td><td>0.10</td><td>0.31</td><td>0.32</td><td>0.31</td></tr></tbody></table>

The following observations can be made on the basis of results reported in Table 7. (1) ALE algorithm obtained the highest performance on CUB, aPY, and SUN datasets (0.37, 0.1, and 0.29), whereas (2) DeViSE obtained the highest values on AWA1 and AWA2 datasets (0.26 and 0.16). (3) The aPY dataset is the hardest to deal with when the F1 accuracy measure is applied.

Overall, on the basis of all experiments, similar conclusions can be made to the ones reported in [12,13]. (i) Different performance measures “promote” different GZSL
approaches. (ii) The aPY and the SUN datasets are the most difficult to classify; depending on the performance measure that is being used. This varies from the ZSL settings, where only the aPY dataset was the difficult one. (iii) None of the individual classifiers can be considered as “the best one”. Moreover, none of the classifiers delivers particularly good results, regardless of the dataset and performance measure used to evaluate it.

Interest in finding the best overall approach was addressed in [13], by introducing a “competitive scoring”. Specifically, each of the five classifiers was assigned scores from 5 to 1 for each dataset, and for each performance measure, depending on its result (the best performance gave 5 points, while the worst gave 1 point). Next, the results have been added. The same approach to representing “robustness” of all classifiers has been applied here and the results are displayed in Table 8. As before, best results for each dataset, and overall, are marked in bold.

<table><caption>**Table 8. Individual Classifier Combined Performance.**</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th><th>Total</th></tr></thead><tbody><tr><td>DeViSE</td><td>13</td><td><b>19</b></td><td><b>19</b></td><td>14</td><td>13</td><td><b>78</b></td></tr><tr><td>ALE</td><td><b>16</b></td><td>11</td><td>11</td><td><b>15</b></td><td><b>17</b></td><td>70</td></tr><tr><td>SJE</td><td>14</td><td>12</td><td>13</td><td>12</td><td>13</td><td>64</td></tr><tr><td>ESZSL</td><td>11</td><td>12</td><td>11</td><td>10</td><td>10</td><td>54</td></tr><tr><td>SAE</td><td>6</td><td>6</td><td>9</td><td>12</td><td>10</td><td>43</td></tr></tbody></table>

The observations from Table 8 are as follows. (A) *DeViSE* performed best for **AWA1** and **AWA2** datasets, and (B) it also obtained the best overall result (78 points). (C) The *ALE* performed the best for **CUB**, **aPY**, and **SUN** datasets. It is worth noting that (D) *SAE* performed the worst for almost all datasets, as well as overall.

These results differ from these reported in [12,13], where the best overall score was reported for the *ESZSL* classifier. Overall, for the GZSL problem, if specific characteristics of the dataset are not known beforehand, the *DeViSE* approach may be the one to try first. However, obtained results further strengthen the reflection that a lot more work is needed to develop deeper understanding of relations between datasets, approaches, and performance measures in the case of the GZSL problem.

## 5.2. Performance of Meta-classifiers

The second part of reported work concerns experimental evaluation of performance of meta-classifiers. Here, the T1 accuracy results have been summarized in Table 9. As previously, the best results have been marked in bold font.

<table><caption>**Table 9. Meta-Classifier Results for the T1 Accuracy.**</caption><thead><tr><th>DS</th><th colspan="3">CUB</th><th colspan="3">AWA1</th><th colspan="3">AWA2</th><th colspan="3">aPY</th><th colspan="3">SUN</th></tr><tr><th>CLF</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th></tr></thead><tbody><tr><td>MV</td><td>21.98</td><td>66.83</td><td>33.08</td><td>8.76</td><td>84.01</td><td>15.87</td><td>7.34</td><td>81.61</td><td>13.48</td><td>3.24</td><td>80.36</td><td>6.23</td><td>22.64</td><td>37.6</td><td><b>28.26</b></td></tr><tr><td>MDT</td><td>25.96</td><td>75.89</td><td>38.69</td><td>20.21</td><td>80.01</td><td>32.27</td><td>20.4</td><td>80</td><td>32.51</td><td>38.53</td><td>62.65</td><td>47.72</td><td>11.02</td><td>90.01</td><td>19.63</td></tr><tr><td>DNN</td><td>25.9</td><td>75.89</td><td>38.62</td><td>20.18</td><td>80.05</td><td>32.23</td><td>20.46</td><td>80.02</td><td>32.59</td><td>38.53</td><td>64.17</td><td>48.15</td><td>11.16</td><td>90.06</td><td>19.86</td></tr><tr><td>GT</td><td>26.45</td><td>75.49</td><td><b>39.17</b></td><td>20.86</td><td>80.59</td><td><b>33.14</b></td><td>21.03</td><td>80.24</td><td>33.33</td><td>38.62</td><td>64</td><td><b>48.17</b></td><td>10.36</td><td>90.06</td><td>18.58</td></tr><tr><td>Con</td><td>25.97</td><td>75.71</td><td>38.67</td><td>20.18</td><td>80.05</td><td>32.23</td><td>21.42</td><td>80.3</td><td><b>33.82</b></td><td>37.92</td><td>64.84</td><td>47.85</td><td>10.91</td><td>90.08</td><td>19.47</td></tr><tr><td>Auc</td><td>25.96</td><td>75.89</td><td>38.69</td><td>20.18</td><td>80.05</td><td>32.23</td><td>20.46</td><td>80.03</td><td>32.59</td><td>38.53</td><td>62.65</td><td>47.72</td><td>10.77</td><td>90.03</td><td>19.25</td></tr></tbody></table>

The following key information can be derived from the results reported in Table 9. (i) The *GT* reached highest performance for **CUB**, **AWA1**, and **aPY** datasets (39.17%, 33.14%, and 48.17%), whereas (ii) *Con* reached the highest overall T1 accuracy for **AWA2** dataset (33.82%). Finally, (iii) *MV* got the highest T1 accuracy value for **SUN** dataset (28.26%). These results mean that (iv) neither *MDT*, *DNN*, nor *Auc* managed to get the best scores for any single dataset. Additionally, (v) all results are below 50% accuracy. It is also worth
noting that (vi) all obtained results are higher than these reported in Table 4. Furthermore, (vii) the largest difference is for ESZSL classifier and AWA2 dataset (69.07%), while the smallest is for SAE classifier and CUB dataset (0.12%). Finally, (viii) the SUN dataset turned out to be the most difficult when performance of meta-classifiers was measured in terms of T1 accuracy measure.

It is important to point out that the T5 accuracy will not be reported due to the fact that some meta-classifiers, e.g., DNN, return the individual classifier result as an output. Hence, since the experimentation is being done on five individual classifiers, the resulting accuracy of the meta-classifier using the T5 measure will always result in a 100% regardless of the correctness of the output.

When the F1 accuracy was used to measure performance of meta-classifiers, the results have been summarized in Table 10 (bold font marked results are “the best”).

<table><caption>**Table 10.** Meta-Classifier Results for the F1 Accuracy.</caption><thead><tr><th>DS</th><th colspan="3">CUB</th><th colspan="3">AWA1</th><th colspan="3">AWA2</th><th colspan="3">aPY</th><th colspan="3">SUN</th></tr><tr><th>CLF</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th><th>f(Y<sup>tr</sup>)</th><th>f(Y<sup>te</sup>)</th><th>ŷ</th></tr></thead><tbody><tr><td>MV</td><td>0.66</td><td>0.22</td><td><strong>0.33</strong></td><td>0.89</td><td>0.09</td><td>0.17</td><td>0.88</td><td>0.06</td><td>0.1</td><td>0.81</td><td>0.02</td><td>0.03</td><td>0.38</td><td>0.23</td><td><strong>0.28</strong></td></tr><tr><td>MDT</td><td>0.12</td><td>0.36</td><td>0.19</td><td>0.01</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.1</td><td>0.02</td><td>0.03</td><td>0.11</td><td>0.1</td><td>0.1</td></tr><tr><td>DNN</td><td>0.12</td><td>0.36</td><td>0.18</td><td>0.02</td><td>0.01</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.11</td><td>0.67</td><td><strong>0.19</strong></td><td>0.12</td><td>0.1</td><td>0.11</td></tr><tr><td>GT</td><td>0.16</td><td>0.2</td><td>0.18</td><td>0.1</td><td>0.3</td><td>0.2</td><td>0.12</td><td>0.01</td><td>0.1</td><td>0.11</td><td>0.12</td><td>0.11</td><td>0.03</td><td>0.1</td><td>0.05</td></tr><tr><td>Con</td><td>0.12</td><td>0.29</td><td>0.17</td><td>0.28</td><td>0.39</td><td><strong>0.32</strong></td><td>0.16</td><td>0.11</td><td><strong>0.13</strong></td><td>0.05</td><td>0.17</td><td>0.07</td><td>0.1</td><td>0.12</td><td>0.11</td></tr><tr><td>Auc</td><td>0.19</td><td>0.36</td><td>0.19</td><td>0.01</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.03</td><td>0.02</td><td>0.11</td><td>0.02</td><td>0.03</td><td>0.12</td><td>0.1</td><td>0.11</td></tr></tbody></table>

The following observations can be derived from Table 10. (a) MV got the highest values for CUB and SUN datasets (0.33 and 0.28), whereas (b) Con reached the highest performance for AWA1 and AWA2 datasets (0.32 and 0.13), and (c) DNN obtained the highest accuracy score for aPY dataset (0.19). Moreover, (d) neither MDT, GT, nor Auc managed to obtain the best accuracy score for any dataset. Additionally, (d) all of the reported results are below 0.4. It is also worth noting that (e) the obtained results are comparable but, somewhat worse than the results reported in Table 7. Finally, (f) the AWA2 dataset turned out to be the most difficult from the perspective of the F1 accuracy measure.

Comparing the results obtained when applying the F1 accuracy score to the meta-classifiers, with these obtained by the individual classifiers, one can see that: the meta-classifiers performed better than the individual classifiers on AWA1 and aPY datasets (0.32 and 0.28 compared to 0.26 and 0.1). At the same time, the individual classifiers obtained better results on CUB, AWA2, and SUN datasets (0.37, 0.17, and 0.29 compared to 0.33, 0.13, and 0.28).

Using the, above described, method of competitive point distribution, used for measuring the combined performance of the individual classifiers, results reported in Table 11 have been calculated for the meta-classifiers (best results reported in bold font). Both performance measures – T1 and F1 – were combined into forming the meta-results. The top scorer, in a given category, obtained 6 points, since there are six meta-classifiers.

<table><caption>**Table 11.** Meta-Classifier Combined Performance.</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th><th>Total</th></tr></thead><tbody><tr><td>MV</td><td>8</td><td>7</td><td>7</td><td>5</td><td><strong>12</strong></td><td>39</td></tr><tr><td>MDT</td><td>9</td><td>8</td><td>7</td><td>6</td><td>8</td><td>38</td></tr><tr><td>DNN</td><td>7</td><td>7</td><td>8</td><td><strong>11</strong></td><td>10</td><td>43</td></tr><tr><td>GT</td><td><strong>10</strong></td><td><strong>11</strong></td><td>10</td><td><strong>11</strong></td><td>4</td><td><strong>46</strong></td></tr><tr><td>Con</td><td>8</td><td>10</td><td><strong>12</strong></td><td>8</td><td>8</td><td><strong>46</strong></td></tr><tr><td>Auc</td><td>9</td><td>7</td><td>8</td><td>6</td><td>7</td><td>37</td></tr></tbody></table>

The following can be noted on the basis of results reported in Table 11. (A) GT performed the best for CUB and AWA1 datasets, and also obtained one of the best overall
result, (B) alongside Con (total score for both equal to 46 points), which (C) got the best score for **AWA2** dataset. (D) **GT** and **DNN** performed the best for **aPY** dataset, (E) while **MV** performed the best for **SUN** dataset. It is worth noting that (F) **Auc** performed the worst overall (on individual datasets and in the case of combined performance).

Finally, the same competitive score combination method was applied jointly to both the meta-classifiers and the individual classifiers. For obvious reasons, only the T1 and the F1 accuracy measures were taken into account. Moreover, since 11 classifiers were compared, the top score was 11 points. The results are displayed in Table 12 (bold font marks the best results).

<table><caption>**Table 12.** Individual Classifier and Meta-Classifier Combined Performance.</caption><thead><tr><th>CLF</th><th>CUB</th><th>AWA1</th><th>AWA2</th><th>aPY</th><th>SUN</th><th>Total</th></tr></thead><tbody><tr><td>DeViSE</td><td>16</td><td>18</td><td>18</td><td>11</td><td>17</td><td>80</td></tr><tr><td>ALE</td><td><b>18</b></td><td>16</td><td>16</td><td>16</td><td><b>22</b></td><td><b>88</b></td></tr><tr><td>SJE</td><td>14</td><td>13</td><td>15</td><td>12</td><td>18</td><td>72</td></tr><tr><td>ESZSL</td><td>11</td><td>9</td><td>9</td><td>7</td><td>8</td><td>44</td></tr><tr><td>SAE</td><td>9</td><td>8</td><td>11</td><td>14</td><td>15</td><td>57</td></tr><tr><td>MV</td><td>13</td><td>11</td><td>13</td><td>9</td><td>20</td><td>66</td></tr><tr><td>MDT</td><td>16</td><td>13</td><td>14</td><td>14</td><td>10</td><td>67</td></tr><tr><td>DNN</td><td>13</td><td>12</td><td>15</td><td><b>21</b></td><td>12</td><td>73</td></tr><tr><td>GT</td><td>16</td><td>17</td><td>19</td><td><b>21</b></td><td>6</td><td>79</td></tr><tr><td>Con</td><td>13</td><td><b>20</b></td><td><b>21</b></td><td>17</td><td>10</td><td>81</td></tr><tr><td>Auc</td><td>16</td><td>12</td><td>15</td><td>14</td><td>9</td><td>66</td></tr></tbody></table>

Following are the observations that can be obtained from results reported in Table 12.

* For the individual classifiers – ALE obtained the highest score on both CUB and SUN datasets (18 and 22), as well as the highest overall score (88). Only ALE managed to get a score of 20 or higher on any given dataset. DeViSE, ALE, and SJE managed to obtain an overall score of 60 or higher, whereas ESZSL and SAE scored below this threshold.

* For the meta-classifiers – Con obtained the highest score on both AWA1 and AWA2 datasets (20 and 21); DNN and GT both obtained the highest score for aPY dataset. MV, DNN, GT, and Con managed to get a score of 20 or higher on any given dataset. All of the meta-classifiers managed to get an overall score of 60 or higher, where half of them reached overall scores of above 70.

* When comparing individual and meta classifiers – half of the meta-classifiers obtained total scores higher than 70, whereas less than half of the individual classifiers did.

* None of the meta-classifiers obtained total scores of less than 60, while three of the five individual classifiers did.

* Only ALE was able to obtain a score of 20 or above on an individual dataset, whereas four of the six (meaning two thirds) of the meta-classifiers did reach this level.

* Unlike the conclusions made in [12,13] for the ZSL problem setting, where the simpler meta-classifiers (e.g. MV), gave better results; in the case of the GZSL problem, more complex meta-classifiers (e.g. DNN), delivered better results.

Overall, one can easily conclude that also in the GZSL settings, selection of “best approach” is very much context dependent. When one has “inside knowledge” of the characteristics of the dataset and/or aims at obtaining fine-tuned results for a given performance measure, this can be achieved using one of individual classifiers. On the other hand, when the goal is to solve the problem, while avoiding the “worst case performance” then use of meta-classifiers is desirable.

## 6. Concluding remarks and future research directions

The aim of this contribution was to experimentally study two different aspects of the generalized zero-shot learning problem. The first was a comprehensive study of five
different approaches (DeViSE, ALE, SJE, ESZSL, and SAE) for solving the GZSL problem, on five standard benchmarking datasets (CUB, AWA1, AWA2, aPY, and SUN), while applying four performance measures. The second was an introductory study of the performance of six standard state-of-the-art meta-classifiers (MV, MDT, DNN, GT, Con, and Auc) applied to the results obtained by the five individual GZSL classifiers.

Similar to the conclusions summarized in [12,13], at this stage of research, there is no single best overall classifier for the GZSL problem, since the performance of the existing ones depends on the dataset and the applied performance measure. Thus, establishing which classifier to use will depend on the data and the goals. For instance, a classifier may be applied when the single best result is important and a different one when the total performance for the five best results matters.

Another important finding is that the meta-classifiers seem to be valuable as an approach geared towards avoiding performance pitfalls, when it is not clear which solver should be tried. Here, despite their performance ceiling, MV, MDT, and DNN, seemed to have performed comparably well. However, the non-constrained models (GT and Con) obtained the best performances when applied to the GZSL problem.

As far as future work is concerned, first, and foremost, it is clear that a lot more work is needed to propose novel approaches to obtain satisfactory results for the GZSL problem. To do so, more work is needed to develop better understanding of relations between characteristics of the dataset, the method used to solve the GZSL problem, and the performance measure that is being used. Here, there is an obvious need to introduce additional datasets that can be used to gain more knowledge about such relationships. Moreover, study similar to this one should be extended to include also methods belonging to other categories of ZSL/GZSL solvers (see, Section 3). Finally, similar to the research reported, for instance, in [38], more refined meta-classifiers or meta-learning systems can be proposed and studied for the ZSL/GZSL problem. Combining knowledge obtained following these research directions may result in better understanding which approaches have a chance to bring the quality of available solutions to the next level.

**Author Contributions:** This contribution is a result of a team effort, supported by multiple funding and cooperation agreements. While the lead author was responsible for coding and running the experiments, the remaining authors participated equally in all remaining aspects of the project.

**Funding:** This work is funded in part by the Centre for Priority Research Area Artificial Intelligence and Robotics of Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme. Moreover it was supported by and realized within the framework of bilateral project between Polish Academy of Sciences and Romania Academy, entitled "Semantic foundation of the Internet of Things"; bilateral project IC-PL/01/2022-2023 between Polish Academy of Sciences and Bulgarian Academy of Sciences, entitled "Practical aspects of scientific computing"; as well as a collaboration agreement between University of Novi Sad, University of Craiova, SRI PAS and Warsaw University of Technology.

**Conflicts of Interest:** The authors declare no conflict of interest.

**Abbreviations**

The following abbreviations are used in this manuscript:
<table><tr><td>MDPI</td><td>Multidisciplinary Digital Publishing Institute</td></tr><tr><td>ZSL</td><td>Zero-Shot Learning</td></tr><tr><td>GZSL</td><td>Generalized Zero-Shot Learning</td></tr><tr><td>DeViSE</td><td>Deep Visual Semantic Embedding</td></tr><tr><td>ALE</td><td>Attribute Label Embedding</td></tr><tr><td>SJE</td><td>Structured Joint Embedding</td></tr><tr><td>ESZSL</td><td>Embarrassingly Simple Approach to Zero-Shot Learning</td></tr><tr><td>SAE</td><td>Semantic AutoEncoder</td></tr><tr><td>MDT</td><td>Meta-Decision Tree</td></tr><tr><td>DNN</td><td>Deep Neural Network</td></tr><tr><td>GT</td><td>Game Theory-based approach</td></tr><tr><td>Auc</td><td>Auction-based model</td></tr><tr><td>Con</td><td>Consensus-based approach</td></tr><tr><td>T1</td><td>Top-1</td></tr><tr><td>T5</td><td>Top-5</td></tr><tr><td>LogLoss</td><td>Logarithmic Loss</td></tr><tr><td>F1</td><td>F1 Score</td></tr><tr><td>CUB</td><td>Caltech-UCSD-Birds 200-2011</td></tr><tr><td>SUN</td><td>Scene Understanding</td></tr><tr><td>AWA1</td><td>Animals with Attributes 1</td></tr><tr><td>AWA2</td><td>Animals with Attributes 2</td></tr><tr><td>aPY</td><td>Attribute Apascal&amp;Ayahoo</td></tr></table>

550

## References

551

1. Chao, W.L.; Changpinyo, S.; Gong, B.; Sha, F. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In Proceedings of the European conference on computer vision. Springer, 2016, pp. 52–68.

552

553

2. Joachims, T. Transductive learning via spectral graph partitioning. In Proceedings of the 20th international conference on machine learning (ICML-03), 2003, pp. 290–297.

554

555

3. Arnold, A.; Nallapati, R.; Cohen, W.W. A comparative study of methods for transductive transfer learning. In Proceedings of the Seventh IEEE international conference on data mining workshops (ICDMW 2007). IEEE, 2007, pp. 77–82.

557

559

4. Palatucci, M.M. Thought recognition: predicting and decoding brain activity using the zero-shot learning model; Carnegie Mellon University, 2011.

560

561

5. Lampert, C.; Nickisch, H.; Harmeling, S. Attribute-based classification for zero-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence 2013, 36, 453–465. https://doi.org/10.1109/TPAMI.2013.140.

562

6. Larochelle, H.; Erhan, D.; Bengio, Y. Zero-data learning of new tasks. In Proceedings of the AAAI, 2008, Vol. 1, p. 3.

563

7. Rohrbach, M.; Stark, M.; Schiele, B. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In Proceedings of the CVPR 2011. IEEE, 2011, pp. 1641–1648.

564

565

8. Yu, X.; Aloimonos, Y. Attribute-based transfer learning for object categorization with zero/one training example. In Proceedings of the European conference on computer vision. Springer, 2010, pp. 127–140.

566

9. Xu, X.; Shen, F.; Yang, Y.; Zhang, D.; Tao Shen, H.; Song, J. Matrix tri-factorization with manifold regularizations for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3798–3807.

567

568

10. Ding, Z.; Shao, M.; Fu, Y. Low-rank embedded ensemble semantic dictionary for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2050–2058.

569

570

11. Xian, Y.; Lampert, C.H.; Schiele, B.; Akata, Z. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence 2018, 41, 2251–2265.

571

572

12. Saad, E.; Paprzycki, M.; Ganzha, M. Practical Aspects of Zero-Shot Learning. In Proceedings of the Computational Science – ICCS 2022; Groen, D.; de Mulatier, C.; Paszynski, M.; Krzhizhanovskaya, V.V.; Dongarra, J.J.; Sloot, P.M.A., Eds.; Springer International Publishing: Cham, 2022, pp. 88–95.

573

574

575

13. Saad, E.; Paprzycki, M.; Ganzha, M. Practical Aspects of Zero-Shot Learning. arXiv preprint arXiv:2203.15158 2022.

576

577

14. Jain, S.; Kotsampasakou, E.; Ecker, G.F. Comparing the performance of meta-classifiers—a case study on selected imbalanced data sets relevant for prediction of liver toxicity. Journal of computer-aided molecular design 2018, 32, 583–590.

578

15. Frome, A.; Corrado, G.S.; Shlens, J.; Bengio, S.; Dean, J.; Ranzato, M.; Mikolov, T. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems 2013, 26.

579

580

16. Akata, Z.; Perronnin, F.; Harchaoui, Z.; Schmid, C. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence 2015, 38, 1425–1438.

581

582

17. Akata, Z.; Reed, S.; Walter, D.; Lee, H.; Schiele, B. Evaluation of output embeddings for fine-grained image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2927–2936.

583

584

18. Shalev-Shwartz, S.; Ben-David, S. Understanding machine learning: From theory to algorithms; Cambridge university press, 2014.

585
19. Romera-Paredes, B.; Torr, P. An embarrassingly simple approach to zero-shot learning. In Proceedings of the International conference on machine learning. PMLR, 2015, pp. 2152-2161.

586

587

20. Xian, Y.; Akata, Z.; Sharma, G.; Nguyen, Q.; Hein, M.; Schiele, B. Latent embeddings for zero-shot classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 69-77.

588

589

21. Socher, R.; Ganjoo, M.; Sridhar, H.; Bastani, O.; Manning, C.D.; Ng, A.Y. Zero-shot learning through cross-modal transfer. *arXiv preprint arXiv:1301.3666* **2013**.

590

591

22. Zhang, Z.; Saligrama, V. Zero-shot learning via semantic similarity embedding. In Proceedings of the IEEE international conference on computer vision, 2015, pp. 4166-4174.

592

593

23. Norouzi, M.; Mikolov, T.; Bengio, S.; Singer, Y.; Shlens, J.; Frome, A.; Corrado, G.S.; Dean, J. Zero-shot learning by convex combination of semantic embeddings. *arXiv preprint arXiv:1312.5650* **2013**.

594

595

24. Changpinyo, S.; Chao, W.L.; Gong, B.; Sha, F. Synthesized classifiers for zero-shot learning. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5327-5336.

596

597

25. Verma, V.K.; Rai, P. A simple exponential family framework for zero-shot learning. In Proceedings of the Joint European conference on machine learning and knowledge discovery in databases. Springer, 2017, pp. 792-808.

598

599

26. Ye, M.; Guo, Y. Zero-shot classification with discriminative semantic representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 7140-7148.

600

601

27. Todorovski, L.; Džeroski, S. Combining classifiers with meta decision trees. *Machine learning* **2003**, *50*, 223-249.

602

603

28. Goodfellow, I.; Bengio, Y.; Courville, A. *Deep learning*; MIT press, 2016.

604

605

29. Abreu, M.d.C.; Canuto, A.M. Analyzing the benefits of using a fuzzy-neuro model in the accuracy of the neura ge system: An agent-based system for classification tasks. In Proceedings of the 2006 IEEE International Joint Conference on Neural Network. IEEE, 2006, pp. 2959-2966.

606

607

608

30. Alzubi, O.A.; Alzubi, J.A.A.; Tedmori, S.; Rashaideh, H.; Almomani, O. Consensus-based combining method for classifier ensembles. *Int. Arab J. Inf. Technol.* **2018**, *15*, 76-86.

609

610

31. Ruta, D.; Gabrys, B. Classifier selection for majority voting. *Information fusion* **2005**, *6*, 63-81.

611

612

32. Sokolova, M.; Japkowicz, N.; Szpakowicz, S. Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation. In Proceedings of the Australasian joint conference on artificial intelligence. Springer, 2006, pp. 1015-1021.

613

614

615

33. Ferger, W.F. The nature and use of the harmonic mean. *Journal of the American Statistical Association* **1931**, *26*, 36-40.

616

617

34. Nwankpa, C.; Ijomah, W.; Gachagan, A.; Marshall, S. Activation functions: Comparison of trends in practice and research for deep learning. *arXiv preprint arXiv:1811.03378* **2018**.

618

619

35. Welinder, P.; Branson, S.; Mita, T.; Wah, C.; Schroff, F.; Belongie, S.; Perona, P. Caltech-UCSD birds 200. Technical Report 2010-001, California Institute of Technology, 2010.

620

621

36. Patterson, G.; Hays, J. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012, pp. 2751-2758.

37. Farhadi, A.; Endres, I.; Hoiem, D.; Forsyth, D. Describing objects by their attributes. In Proceedings of the 2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009, pp. 1778-1785.

622

623

38. Demertzis, K.; Iliadis, L. GeoAI: A model-agnostic meta-ensemble zero-shot learning method for hyperspectral image analysis and classification. *Algorithms* **2020**, *13*, 61.