Title: LEARNING TO SEPARATE DOMAINS IN GENERALIZED ZERO-SHOT AND OPEN SET LEARNING: A PROBA-BILISTIC PERSPECTIVE

Abstract: This paper studies the problem of domain division which aims to segment instances drawn from different probabilistic distributions. This problem exists in many previous recognition tasks, such as Open Set Learning (OSL) and Generalized Zero-Shot Learning (G-ZSL), where the testing instances come from either seen or unseen/novel classes with different probabilistic distributions. Previous works only calibrate the confident prediction of classifiers of seen classes (W-SVM Scheirer et al. ( 2014)) or taking unseen classes as outliers Socher et al. (2013) . In contrast, this paper proposes a probabilistic way of directly estimating and fine-tuning the decision boundary between seen and unseen classes. In particular, we propose a domain division algorithm to split the testing instances into known, unknown and uncertain domains, and then conduct recognition tasks in each domain. Two statistical tools, namely, bootstrapping and Kolmogorov-Smirnov (K-S) Test, for the first time, are introduced to uncover and fine-tune the decision boundary of each domain. Critically, the uncertain domain is newly introduced in our framework to adopt those instances whose domain labels cannot be predicted confidently. Extensive experiments demonstrate that our approach achieved the state-of-the-art performance on OSL and G-ZSL benchmarks.

1. INTRODUCTION:

This paper discusses the problem of learning to separate two domains which include the instances sampled from different distributions. This is a typical and general research topic that can be potentially used in various recognition tasks, such as Open Set Learning (OSL) and Generalized Zero-Shot Learning (G-ZSL). Particularly, OSL can break the constraints of the closed set in supervised learning, and aim at recognizing the testing instances from one of the seen classes (i.e., known domain), and the novel class (i.e., unknown domain). The novel classes include the testing instances which have different distributions from that of the seen ones. In contrast, G-ZSL targets at distinguishing the labels of instances from the seen and unseen classes. Only the seen classes have the training instances, but unseen classes do not. Note that OSL does not explicitly give the class labels for those instances categorized as the novel class, but G-ZSL requires predicting the class labels of unseen classes. To address G-ZSL, semantic attributes or vectors are introduced as the intermediate representations; each (seen/unseen) class has one semantic prototype that contains class level information. Specifically, a reasonable solution of OSL and G-ZSL is via dividing the known and unknown domains. For training classes, the predictors are constructed to map visual features to the class label space (OSL), (or semantic space (G-ZSL)). Testing is performed on each separated domain to identify seen classes and the novel class (OSL), or both seen and unseen classes (G-ZSL).

The key question of OSL and ZSL is how to deal with the newly introduced novel class/unseen classes efficiently in the testing time. This is different from the conventional Zero-Shot Learning (ZSL) task which assumes that, in the testing stage, seen classes would not be misclassified as unseens, and vice versa; ZSL only uses the unseen classes for testing. Unfortunately, the predictors learned on training classes will inevitably make OSL or G-ZSL approaches tend to be biased towards the seen classes, and thus leading to very poor classification results for the novel class (OSL) or unseen classes (G-ZSL) Xian ). We categorize the SAE prediction as known or unknown domain labels and compare with the groundtruth in Fig. 1(c ). We show that a large portion of unseen instances being predicted as one of the known classes.

A natural recipe for addressing this problem is to learn to separate domains by the distributions of instances; and different classifiers can be directly applied in each domain. However, there are still two key problems. First, visual features alone are not discriminative enough to help to distinguish the seen and unseen/novel classes. As Fig. 2 (a), bicycle and motorbike, respectively, are one of the seen and unseen classes1 in aPY dataset (described in Sec. 6.1). We can observe that there is a large overlapping region between their t-SNE visualized feature distributions. That is, the visual features may not be representative enough to differentiate these two classes; the instances of motorbike (circled as the uncertain domain) may be taken as the bicycle, or vice versa; Second, the predictors trained on seen classes may be not trustworthy. A not well-trained predictor may negatively affect the recognition algorithms. Third and even worse, the performance of classifiers in each domain is still very sensitive to the results of domain separation: should the domain of one testing instance be wrongly divided, it would never be correctly categorized by the classifiers.

To tackle the aforementioned issues, our key insight (see Fig. 2 (a)) is to introduce a novel domain -uncertain domain that accounts for the overlapping regions of testing instances from seen or novel/unseen classes. Thus, the visual or semantic space can be learned to be divided into known, unknown and uncertain domains. The recognition algorithms will be directly employed in each domain. Nonetheless, how to divide the domains based on known information is also a non-trivial task. Though the supervised classifiers can learn the patterns of known classes, not all classes encountered during testing are known.

Formally, we propose exploiting the distribution information of seen and novel/unseen classes to efficiently learn to divide the domains from a probabilistic perspective. Our domain separation algorithm has two steps: the initial division of domains by bootstrapping, and fine-tuning by the Kolmogorov-Smirnov test. Specifically, according to extreme value theory Scheirer et al. ( 2014), the maximum/minimum confidence scores predicted by the classifier of each class can be taken as an extreme value distribution. Since we do not have the prior knowledge of the underlying data distributions of each class; bootstrapping is introduced here as an asymptotically consistent method in estimating an initial boundary of known classes. Nevertheless, the initial boundary estimated by bootstrapping is too relaxed to include novel testing instances as is illustrated in Fig. 2 (b). To finetune the boundary, we exploit the K-S Test to validate whether the learned predictors are trustworthy in a specific region. The uncertain domain introduced thus accounts for those testing instances whose labels are hard to be judged. Recognition models can be conducted in each domain.

Contributions::

The main contribution is to present a systematic framework of learning to separate domains by probabilistic distributions of instances, which is capable of addressing various recognition tasks, including OSL and G-ZSL. Towards this goal, two simple, most widely used, and very effective tools -bootstrapping and the Kolmogorov-Smirnov test, are employed to firstly initially estimate and then fine-tune the boundary. In particular, we introduce an uncertain domain, which encloses the instances which can hardly be classified into known or unknown with high confidence.

We extensively evaluate the importance of domain division on several zero-shot learning benchmarks and achieved significant improvement over existing ZSL approaches. However, ZSL usually assumes that the unseen classes cannot be mis-classified as seen classes and vice versa. This has greatly simplified the learning task.

2. RELATED WORKS:



One-Class Classification (OCC:

Generalized Zero-Shot Learning. 

= {x i , y i , l i } ns i=1 : x i ∈ R n is the feature of i th instance with the class label l i ∈ C s , where C s is the source class set; n c

s is the number of instances in seen class c. Analogous to standard ZSL setting, we introduce the target label classes C t with C s C t = ∅ and the total class label set C = C s ∪ C t . y i is the semantic attribute vector of instance x i . In general, the y i of instances in one class should be the same Lampert et al. (2014) . We simplify y c as the semantic prototype for all the instances in class c. Given one test instance x i , our goal is to predict its class label c i . We discuss two tasks: (1) Open set recognition:

c i ∈ {C s , novel class }; (2) Generalized zero-shot learning: c i ∈ {C s , C t }.

The semantic prototype is predefined for each class in C t . The 'novel class' is an umbrella term referring to any class not in C s .

3.2. EXTREME VALUE DISTRIBUTIONS:

We firstly introduce the background of modeling the extreme values (i.e., minimum/maximum scores) computed from one supervised classifier as the extreme value distributions. In particular, by using each source class c, we can train a binary predictor function, e.g. SVM, z c = f c (x) : R n → R where z is the confidence score of instance x belonging to the class c. In Extreme Value Theory (EVT) , the extreme values (i.e., maximum / minimum confidence scores) of the score distribution computed by the predictor function f c (•) can be modeled by an EVT distribution. Specifically, for instance set {x} that belong to class c; the minimum score z c min = minf c ({x}) follows the Weibull distribution,

EQUATION

where G (•) is the Cumulative Distribution Function (CDF) of Weibull distribution (z > ν): 

G(z c ; λ c , ν c , κ c ) = 1 -exp -z c -νc

EQUATION

where rG (•) is the CDF of reverse Weibull distribution: Particularly, given a test instance x i , the supervised binary algorithm can compute the confidence score z c i = f c (x i ) belonging to the class c. Here we introduce two events: E 1 : x i belonging to class c; E 2 : x i belonging to the other seen classes.

EQUATION

The distributions of extreme values defined in Eq (1) and Eq (2) can actually give us the boundary of each event above happened in a probabilistic perspective. Thus we can have the probability

P G (E 1 ) = 1 -exp - z c i -νc λc κc

; and P rG (¬E 2 ) = exp -

z c i -ν c λ c κ c

. Thus, to determine if one testing instance x i belong to class c, W-SVM Scheirer et al. ( 2014) computes the statistic m c (x i ) by,

EQUATION

W-SVM introduces a threshold δ c to determine whether the instance i belongs to the class c as,

c i = c m c (x i ) > δ c ¬c otherwise (5)

where δ c is a fixed value Scheirer et al. (2014). The instance x i rejected by all the seen classes by Eq (5) is labeled as the unknown domain. Generalizing to C s class is straightforward by training multiple prediction functions

{f c (x)}, c = 1, • • • , |C s |.

However, there are several key limitations in directly utilizing Eq (4) and Eq (5) of learning the division of domains: (1) Eq (4) directly multiplies two terms and this indicates that there is a potential hypothesis that no correlation exists between E 1 and ¬E 2 , which is generally not the case. ( 2 2013)) whose domains are wrongly labeled will never be correctly categorized.

4. PROPOSED DOMAIN DIVISION ALGORITHM:



4.1. DETERMINING THE INITIAL BOUNDARY BY BOOTSTRAPPING:

The W-SVM in Eq (5) and Eq (4) estimates the confidence scores by a fixed threshold empirically per-defined for any data distributions in the known domain. However, intrinsically, it can be taken as a model selection task in estimating the boundary by Eq (4) . In this paper, we tackle the question of constructing the boundary of the known domain via the bootstrap approach Efron (1979) . The bootstrapping2 is a strategy of estimating the standard errors and the confidence intervals of parameters when the underlying distributions are unknown. Its procedures are closely related to the other methods such as cross-validation, and jackknife. 3. The threshold of Eq (5) can thus be computed as the mean of these values, i.e., δc

= 1 n Σ n k=1 z c tr(k) .

its confidence score f c (x i ), c ∈ C s . To determine whether the class of a instance x i is seen or unseen, we calculate the statistic m c (x i ) in Eq (4) and Eq (5) with the threshold δ c estimated by the bootstrapping algorithm in Alg. 1. The instances computed in the known and unknown domain will be categorized by supervised, or zero-shot classifiers respectively.

There are still two difficulties in the above framework. ( 1) The whole framework relies on the classifier f c (•), c ∈ C s which is supposed to be robust and well-trained. However, empirically, we can not always train good classifiers for all classes. For example, some class has small number of labeled training instances which are insufficient in training the classifier; some outliers may affect the predictor; the hyper-parameters of the classifiers are wrongly tuned. (2) The naive bootstrapping in Alg. 1 generally provides the bad approximations of the distributions of empirical quantiles in practice Falk & Reiss (1989) . Practically, in our tasks, we observe that the estimated δ c may be consistently too relaxed to determine the boundary of the known domain. We illustrate such a phenomenon in Figure 2 (a): the low-density of seen class bicycle instances (blue points) in the northwest part extends the decision boundary. The relaxed boundary could inadvertently classify unseen instances (red points) as the false positives. Unfortunately, in the framework above, once one testing instance in unseen class is wrongly labeled as the known domain, this instance will never be correctly classified. To address these two problems, we suggest a shrinking step in updating the initial boundary of bootstrapping in the next subsection. (8)

The mapping function g (•) is learned on the known domain from features x i to its corresponding semantic attributes y i . Given one testing instance x i : if z i is very high, we can confidently predict x i belonging to one of seen classes; otherwise, the label of x i is either in the uncertain or unknown domain. We thus have,

c i =            argmax c∈Cs f c (x i ) seen domain argmin c∈Ct g (x i ) -y c unseen domain argmin c∈Ct∪{c } g (x i ) -y c uncertain domain (9)

where y c is semantic prototype of class c; c is the most likely known class to which x i belongs to. Note that in OSL, we only know the y c (c ∈ C s ) of seen classes; We can dynamically construct a C t set by randomly generating y i by making sure y i -y j > (∀y j ∈ C s ), and = min yi,yj ∈Cs;i =j y i -y j . The sample size is usually the same with the number of target classes.

5. RECOGNITION MODELS IN KNOWN/UNKNOWN/UNCERTAIN DOMAINS:

We can apply different recognition algorithms in each domain. In known domain, the standard supervised classifiers can be learned and applied. In unknown and uncertain domains, we propose a simple yet effective feature prototype embedding recognition algorithm as our plain implement.

Feature prototype embedding. Once the domain is well separated, we can use the ZSL algorithms to set up the mapping from feature space to semantic/attribute space. In order to confirm that our main contribution is the domain division part, we do not use very complicated ZSL algorithms. Only the simplest linear predictor is utilized here to recognize the unseen classes. Particularly, we use feature prototypes to replace all the instances of each class to avoid the unbalance sample size among classes. We learn a linear predictor to predict the attribute/word vector g (x) = w T • x. The feature prototype embedding is computed as,

EQUATION

where

x c = 1 n c s

Σ li=c x i is the feature prototype of class c; y c is the semantic prototype of class c.

When we tackle an instance in the unknown or uncertain domain, we need to embed features into semantic space with g, which can infer the class labels of instances:

EQUATION

where c is the most likely seen class for x i which is computed by the supervised classifier and y c is the semantic prototype.

Also, the experiment in each domain can be done with ANY other ZSL. For instance, we report the implement with f-CLSWGAN (f-C) Xian et al. ( 2018) so that G-ZSL can be done with f-CLSWGAN within a single domain. Experimental settings. Our model is validated in OSL and G-ZSL settings. OSL identifies whether an image belongs to the one of seen classes or the novel class. G-ZSL gives the class label of testing instances either from seen or unseen classes. We set the significance level α = 0.05 to tolerate 5% Type-I error. By default, we use SVM with RBF kernel with parameter cross-validated, unless otherwise specified.

6.2. RESULTS OF OPEN SET LEARNING:

We compare against the competitors, including Attribute Baseline (Attr-B), W-SVM Scheirer et al. . The attribute baseline is the variant of our task without using domain division algorithm. Particularly, the Attr-B uses the same semantic space and embedding as our model, but does not leverage domain division step, i.e., use negative samples and prototypes to identify projected instances directly (Fig. 1 (c) ).

We use the metric -F1-measure, which is defined as the harmonic mean of seen class accuracy (specific class) and unseen prediction accuracy (unnecessary to predict the specific class). The results are compared in Tab. 1. Significant performance gain over existing approaches has been observed, in particular for AwA, aPY and ImageNet. This validates the effectiveness of our framework. We attribute the improvement to the newly introduced uncertain domain which help better differentiate whether testing instances derive from known or unknown domain. Results. As seen in Tab. 2, our harmonic mean results are significantly better than all the competitors on almost all the datasets. This shows that ours can effectively address the G-ZSL tasks. Particularly,

(1) Our plain results can beat other competitors by a large margin on AwA and aPY dataset, due to the efficacy of our domain division algorithm. Also, thanks to the power of f-CLSWGAN (f-C), our implement with it on both CUB and AwA dataset are impressive. (2) The key advantage of the proposed framework is learning to better divide the testing instances into known, uncertain and unknown domains. In the known domain, we use the standard SVM classifier. In unknown/uncertain domains, we directly embed feature prototypes into semantic space and match the most likely class in the candidate pool. This is the most simple and straightforward recognition method. Thus our good harmonic mean performance on G-ZSL largely comes from the good domain division algorithm. Additionally, we also highlight that the other advanced supervised or zero-shot algorithms are orthogonal and potentially be useful in each domain if we want to further improve the performance of G-ZSL. (3) Our framework is also applied to large-scale datasets in Tab. 3. We compare several state-of-the-art methods that address G-ZSL on the large-scale dataset. We use the SVM with the linear kernel on this dataset, due to the large data scale. Our harmonic mean results surpass the other competitors with a very significant margin. We notice that other algorithms have very poor performance on U → T. This indicates the intrinsic difficulty of G-ZSL on large-scale dataset. In contrast, our domain division algorithm can better separate the testing instances into different domains; thus achieving better recognition performance. Additionally, we found that the prediction of ConSE Norouzi et al. ( 2013) is heavily biased towards known classes which is consistent with the results in small datasets. This is due to the probability of unseen classes are expressed as the convex combination of seen classes. Usually, there is no higher probability would be assigned to unseen classes than the most probable seen class, especially for large datasets.

6.4. ABLATION STUDY:

In the ablation study, we report the F1-measure and Harmonic mean for OSL and G-ZSL respectively with our plain implement. As is illustrated in Fig. 2 , we notice that although the distance statistic shows the different histogram patterns in feature space, the overlapping part is not negligible.

Importance of bootstrapping the initial threshold. We introduce a variant A of our framework by replacing bootstrapping step (Sec. 4.1) by using Eq (4) and Eq (5) to fix the threshold (i.e., W-SVM Scheirer et al. ( 2014)), i.e., K-S test ( √ ), and Bootstrap (×). As in Tab. 4, the results of variant A are significantly lower than our framework on all three datasets. This actually directly validates the importance of determining the initial threshold by bootstrapping.

Improvements of fine-tuning the threshold by K-S test. We define the variant B is to only use step without fine-tuning the boundary by K-S Test (in Sec. 4.2). Table 4 directly shows the improvement with/without fine-tuning the threshold, i.e., K-S test (×), and Bootstrap ( √ ). In particular, we note that variant B has significant lower results on OSL and G-ZSL than variant A and our framework. One reason is that our bootstrapping step actually learns to determine a very wide boundary of the known domain, to make sure the good results in labeling testing instances as unknown domain samples. The fine-tuning threshold step will further split the individual known domain into known/uncertain domain by shrinking the threshold. Without such a fine-tuning step, variant B may wrongly categorize many instances from unseen classes as one of the known classes. Thus, we can show that the two steps of our framework are very complementary to each other and they work as a whole to enable the good performance on OSL and G-ZSL. Finally, we introduce the variant C in Tab. 4, by using W-SVM to do OSL, and then use our ZSL model for G-ZSL, i.e., K-S test (×), and Bootstrap (×). The performance of variant C is again significantly lower than that of ours, and this demonstrates the efficacy of our model.

7. CONCLUSION:

This paper learns to divide the instances into known, unknown and uncertain domains for the recognition tasks from a probabilistic perspective. The domain division procedure consists of bootstrapping and K-S Test steps. The bootstrapping is used to set an initial threshold for each class; we further employ the K-S test to fine-tune the boundary. Such a domain division algorithm can be used for OSL and G-ZSL tasks, and achieves remarkable results.

:

Note that in OSL setting, we do not know the class label of motorbike, and its instances should be categorized as just the novel class.

It is different from "bootstrapping" (i.e., self-training) in computer vision Shrivastava et al. (2012).

Figures and Tables:

Figure 1: Feature and semantic spaces of seen and unseen classes. We can see that a large of portion of instances from unseen classes is wrongly labeled as seen classes (FP). TN: True Negative; TP: True Positive; FN: False Negative; FP: False Positive.

prototypes of unseen classes to further give the class label of those instances in the novel class. Both the OCC and OSL are able to divide the instances into known and unknown domains and recognize the known classes in the known domain. Intrinsically, their key difference lies in whether leveraging the information of different seen classes in building the classifiers. Specifically, OCC only utilizes the instances of one class to learn its class boundary, whilst OSL can use the instances of different seen classes. Zero-Shot Learning (ZSL). ZSL aims at recognizing the novel instances which have never been trained before. It transfers the knowledge learned from known source classes to recognize the testing instances from unknown target classes. The knowledge can be formulated as semantic attributes Farhadi et al. (2009); Lampert et al. (2009); Fu et al. (2015), semantic word vector Norouzi et al. (2013); Frome et al. (2013); Fu & Sigal (2016); Xu et al. (2017) or ontology Rohrbach et al. (2010).

Critically, Eq (1) models the lower boundary distribution of instance confidence scores belonging to class c. On the other hand, for instance set {x} NOT belonging to class c, the maximum score z c max = maxf c ({x}) should follow the reverse Weibull distribution Scheirer et al. (2011; 2012; 2014),

models the upper boundary distribution of confidence scores NOT belonging to class c. The scale parameters λ c , λ c , shape parameters κ c , κ c , and location parameters ν c , ν c are estimated by Maximum Likelihood Estimator fitted from the training data. Critically, Eq. (2) models the upper boundary distribution of instance confidence scores NOT belonging to class c.3.3 WEAKNESS OF CATEGORIZING INSTANCES INTO KNOWN/UNKNOWN DOMAINS One can conduct the OSL and G-ZSL by directly learning to separate the domains of seen and novel/unseen classes. The idea of learning to divide testing instances to known or unknown domains has been studied in W-SVM Scheirer et al. (2014) (OSL), and CMT Socher et al. (2013) (G-ZSL).

) In multiple seen classes, the instances may derive from many different classes. It is hard to determine a single fixed δ in Eq (5) for each class. (3) Furthermore, we give an illustration of the non-negligible overlapping between the intra-class and inter-class distances of each pair of instances on the aPY dataset (described in Sec. 6.1,Farhadi et al. (2009)). As shown in the feature space of Fig. 2 (b), we compute the pairwise L 2 distances over (1) the instances within the same classes (intra-class), and (2) the instance from different classes (inter-class) on aPY dataset Farhadi et al. (2009). We use the empirical density to show the results in Fig. 2 (c). Practically, it is hard to predict the class labels of instances of the overlapped region in the known/unknown domain. Due to the large overlapped region, the instances (e.g., CMT in Socher et al. (

Bootstrapping is the most widely used tool in approximating the sampling distributions of test statistics and estimators. To facilitate the discussion, we denote the training set of class c as {x c tr }; the testing set whose instances are mostly confidently predicted as class c, as {x c te }. Thus the corresponding confidence score set on training and testing data are {z c tr } = f c ({x c tr }) and {z c te } = f c ({x c te }) respectively. The whole algorithm is shown in Alg. 1. Till now, we had a sketch of our domain division algorithm. Specifically, the training instances of seen classes are utilized to learn the f c (•), c ∈ C s ; For any given testing instance x i , we compute Algorithm 1 Determining the initial threshold by bootstrapping Input: Confidence score set on training data {z c tr } Output: Threshold δc: 1. We sample from {z c tr } for n times (with replacement), producing a sampling set z c tr(k) n k=1 , where z c tr(k) indicates the k th sampled instance; 2. We also choose the significance level α, and generate the α quantile z c tr from z c tr(k) n k=1 . Particularly, we sort z c tr(k) with an ascending order and extract (max (Round [αn] , 1)) th value as z c tr . We repeat it for n times over {z c tr } to get z c tr(k) n k=1 .

SHRINKING THE BOUNDARY OF THE KNOWN DOMAIN BY KOLMOGOROV-SMIRNOV TEST The key idea of updating initial boundary of bootstrapping is to validate whether the learned classifier f c (•), c ∈ C s is trustworthy. Generally, assume the instances of class c independent and identically distributed and provided training samples sufficient, a ideal classifier f c (•) should produce the similar confidence score distributions of training and testing instances of class c.The Kolmogorov-Smirnov (K-S) test is an efficient, straightforward, and qualified choice method for comparing distributions Massey Jr (1951); Miller (1956);Wang et al. (2003). Remarkably, K-S test is a distribution free test, and the statistics of K-S test is effortless to compute. We define the null and alternative hypothesis as H 0 : {z c tr } and {z c te } are from the same distribution. H 1 : {z c tr } and {z c te } are from different distributions. (6) We introduce a distance measure K c = sup z F c tr (z) -F c te (z) where F c tr = ecdf({z c tr }) and F c te = ecdf({z c te }); and the ecdf (•) is the empirical distribution function. The null hypothesis would be rejected at the significant level α when, K c (α) > -|{z c tr }| + |{z c te }| 2 |{z c tr }| • |{z c te }| log α (7) When H 0 is accepted, it indicates that the f c (•) is trustworthy, and the confidence scores of training and testing instances in class c come from the same distribution. We are certain that a large portion of testing instances {z c te } = f c ({x c te }) should be indeed in the class c. On the other hand, when H 0 is rejected, we are not sure whether f c (•) is well learned; and the class labels of these testing instances are uncertain. To this end, we introduce a new domain -uncertain domain to include these instances. Uncertain Domain. The labels of instances in the uncertain domain should be labeled as the most likely seen class, or one of unseen classes. Specifically, we can compute the {z c = f c (x)} |Cs| c=1 over all C s classes; and we can obtain, {c , z } = argmax c∈Cs {z c } .

AND SETTINGS Datasets. Animal with Attribute (AwA) Dataset Lampert et al. (2014) has 50 classes and 30,475 images in total, with 85 class-level attributes annotated. We use 40 source training classes (including 13 classes as validation); the rest as testing. CUB Dataset Wah et al. (2011) includes 200 classes and 11,788 fine-grain images with 312 class-level attributes annotated. The training set has 150 classes (including 50 classes as validation). (3) aPY Dataset Farhadi et al. (2009) has 15,339 images in 32 classes with 64 class-level annotated attributes. We use 20 classes for training (including 5 validation classes). For the AwA, CUB and aPY, we use ResNet-101 features and the class split contributed by Xian et al. (2017). (4) ImageNet 2012/2010 dataset is proposed in Fu & Sigal (2016). As the large-scale dataset, we use the split as Fu & Sigal (2016): 1000 training classes with full training instances in ILSVRC 2012; and 360 testing classes in ILSVRC 2010, non-overlapped with ILSVRC 2012 classes. We address two recognition tasks: OSL and G-ZSL.

, One-class SVM Sch Älkopf et al. (2001), Binary SVM, OSDN Bendale & Boult (2016) and LOF Breunig et al. (2000)

U → T: Test instances from unseen classes, the prediction candidates include both seen and unseen classes.(3)  We employ the harmonic mean as the main evaluation metric to further combine the results of both S → T and U → T, asH = 2 • (Acc(U → T) × Acc(S → T)) / (Acc(U → T) + Acc(S → T)).Competitors. We compare several competitors. (1) DAP Lampert et al. (2014), trains a probabilistic attribute classifier and utilizes the joint probability to predict labels; (2) ConSE Norouzi et al. (2013), maps features into the semantic space by convex combination of attributes; (3) CMT Socher et al.

2013), projects features into unsupervised semantic space and uses LOF to detect novel classes; (4) SSE Zhang & Saligrama (2015), regards novel classes as mixtures of seen proportions to measure the instance similarity. (5) Latem Xian et al. (2016), is a novel latent embedding for ZSL and G-ZSL. (6) ALE Akata et al. (2016), embeds labels into the attribute space by learning a function to rank the likelihood of each class. (7) DeViSE Frome et al. (2013), uses both unsupervised information and annotated attributes to classify classes in an embedding model; (8) SJE Akata et al. (2015) is a hierarchical embedding to learn an inner product gram matrix between features and attributes. (9) ESZSL Romera-Paredes & Torr (2015), focuses on the regularization term in the projection from features to semantic space. (10) SYNC Changpinyo et al. (2016), aligns the semantic space to feature space by manifold learning. (11) SS-VOC Fu & Sigal (2016), optimizes the triplet loss to learn the projection from features to semantic space. (12) SAE Kodirov et al. (2017) is an auto-encoder to combine feature and semantic space. (13) SE-GZSL Verma et al. (2018) leverages VAE Kingma & MaxWelling (2014) as the generator of pseudo instances to train the mapping. (14-15) cycle-CLSWGAN (cycle-C) & f-CLSWGAN (f-C) Felix et al. (2018); Xian et al. (2018): both of them use W-GAN Arjovsky et al. (2017) to reconstruct features and cycle-C adds another regularizer. (16) PTMCA Long et al. (2018) uses transfer approach and embedding way to reduce bias and variance.

Chao et al. Changpinyo et al. (2016) realized that it is nontrivial to directly utilize the existing Zero-Shot Learning algorithms in a more general setting, i.e., G-ZSL. In such a setting, the testing instances can come from either the seen or unseen classes. A thorough evaluation of G-ZSL is further conducted inXian et al. (2017). Their results show that the existing ZSL algorithms do not perform well if directly applied to G-ZSL. The predicted results are inclined to be biased towards seen classes. 3 LEARNING TO CATEGORIZE INSTANCES INTO DIFFERENT DOMAINS 3.1 PROBLEM SETUP In learning tasks, we are given the training dataset, i.e., seen classes, of n s instances, D s

Table content: <table/>

Comparison of open set recognition algorithms

Table content: <table><tr><td>Method / Accuracy</td><td>AwA</td><td>CUB</td><td>aPY</td><td>ImageNet</td></tr><tr><td>Attr-B</td><td>33.8</td><td>18.7</td><td>5.1</td><td>3.7</td></tr><tr><td>Binary SVM</td><td>57.7</td><td>29.8</td><td>66.6</td><td>24.6</td></tr><tr><td>W-SVM</td><td>80.2</td><td>58.6</td><td>78.6</td><td>50.1</td></tr><tr><td>One-Class SVM</td><td>58.9</td><td>27.6</td><td>57.1</td><td>23.4</td></tr><tr><td>OSDN</td><td>49.9</td><td>36.7</td><td>41.5</td><td>-</td></tr><tr><td>LOF</td><td>60.0</td><td>54.5</td><td>49.1</td><td>38.0</td></tr><tr><td>Ours</td><td>93.7</td><td>59.5</td><td>94.3</td><td>67.6</td></tr><tr><td colspan="4">6.3 RESULTS OF GENERALIZED ZERO-SHOT LEARNING</td><td/></tr><tr><td colspan="5">Settings: We first compare the experiments on G-ZSL by using the settings in Xian et al. (2017).</td></tr><tr><td colspan="5">The results are summarized in Tab. 2. In particular, we further compare the separate settings;</td></tr><tr><td colspan="5">and top-1 accuracy in (%) is reported here: (1) S → T: Test instances from seen classes, the</td></tr><tr><td colspan="3">prediction candidates include both seen and unseen classes;</td><td/><td/></tr></table>

G-ZSL Results on AwA, CUB and aPY. ( Our implement; 1 Plain implement: prototype linear mapping; 2 Implement with f-C)

Table content: <table><tr><td/><td/><td>AwA</td><td/><td/><td>CUB</td><td/><td/><td>aPY</td><td/></tr><tr><td/><td>U → T</td><td>S → T</td><td>H</td><td>U → T</td><td>S → T</td><td>H</td><td>U → T</td><td>S → T</td><td>H</td></tr><tr><td>Chance</td><td>2.0</td><td>2.0</td><td>-</td><td>0.5</td><td>0.5</td><td>-</td><td>3.1</td><td>3.1</td><td>-</td></tr><tr><td>DAP</td><td>0.0</td><td>88.7</td><td>0.0</td><td>1.7</td><td>67.9</td><td>3.3</td><td>4.8</td><td>78.3</td><td>9.0</td></tr><tr><td>ConSE</td><td>0.4</td><td>88.6</td><td>0.8</td><td>1.6</td><td>72.2</td><td>3.1</td><td>0.0</td><td>91.2</td><td>0.0</td></tr><tr><td>CMT</td><td>8.4</td><td>86.9</td><td>15.3</td><td>4.7</td><td>60.1</td><td>8.7</td><td>10.9</td><td>74.2</td><td>19.0</td></tr><tr><td>SSE</td><td>7.0</td><td>80.6</td><td>12.9</td><td>8.5</td><td>46.9</td><td>14.4</td><td>0.2</td><td>78.9</td><td>0.4</td></tr><tr><td>Latem</td><td>7.3</td><td>71.7</td><td>13.3</td><td>15.2</td><td>57.3</td><td>24.0</td><td>0.1</td><td>73.0</td><td>0.2</td></tr><tr><td>ALE</td><td>16.8</td><td>76.1</td><td>27.5</td><td>23.7</td><td>62.8</td><td>34.4</td><td>4.6</td><td>73.7</td><td>8.7</td></tr><tr><td>DeViSE</td><td>13.4</td><td>68.7</td><td>22.4</td><td>23.8</td><td>53.0</td><td>32.8</td><td>4.9</td><td>76.9</td><td>9.2</td></tr><tr><td>SJE</td><td>11.3</td><td>74.6</td><td>19.6</td><td>23.5</td><td>59.2</td><td>33.6</td><td>3.7</td><td>55.7</td><td>6.9</td></tr><tr><td>ESZSL</td><td>6.6</td><td>75.6</td><td>12.1</td><td>12.6</td><td>63.8</td><td>21.0</td><td>2.4</td><td>70.1</td><td>4.6</td></tr><tr><td>SYNC</td><td>8.9</td><td>87.3</td><td>16.2</td><td>11.5</td><td>70.9</td><td>19.8</td><td>7.4</td><td>66.3</td><td>13.3</td></tr><tr><td>SAE</td><td>1.1</td><td>82.2</td><td>2.2</td><td>7.8</td><td>54.0</td><td>13.6</td><td>0.4</td><td>80.9</td><td>0.9</td></tr><tr><td>SE-G</td><td>56.3</td><td>67.8</td><td>61.5</td><td>41.5</td><td>53.3</td><td>46.7</td><td>-</td><td>-</td><td>-</td></tr><tr><td>cycle-C</td><td>56.9</td><td>64.0</td><td>60.2</td><td>45.7</td><td>61.0</td><td>52.3</td><td>-</td><td>-</td><td>-</td></tr><tr><td>f-C</td><td>57.9</td><td>61.4</td><td>59.6</td><td>43.7</td><td>57.7</td><td>49.7</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PTMCA</td><td>22.4</td><td>80.6</td><td>35.1</td><td>23.0</td><td>51.6</td><td>31.8</td><td>15.4</td><td>71.3</td><td>25.4</td></tr><tr><td>f-C</td><td>57.8</td><td>72.4</td><td>64.2</td><td>43.4</td><td>58.3</td><td>49.8</td><td>16.8</td><td>45.7</td><td>24.6</td></tr><tr><td>Ours 1</td><td>53.6</td><td>90.4</td><td>67.3</td><td>37.2</td><td>45.2</td><td>40.8</td><td>44.0</td><td>89.2</td><td>58.9</td></tr><tr><td>Ours 2</td><td>66.0</td><td>91.2</td><td>76.6</td><td>53.1</td><td>59.4</td><td>56.1</td><td>22.4</td><td>81.3</td><td>35.1</td></tr></table>

G-ZSL on the large-scale dataset -ImageNet 2012/2010. SS-Voc SAE ESZSL DeViSE ConSE Chance Ours

Table content: <table><tr><td>U → T</td><td>2.3</td><td>0.2</td><td>0.5</td><td>0.4</td><td>0.0</td><td>&lt;0.1</td><td>5.7</td></tr><tr><td>S → T</td><td>33.5</td><td>32.8</td><td>38.1</td><td>24.7</td><td>56.2</td><td>&lt;0.1</td><td>54.1</td></tr><tr><td>H</td><td>4.3</td><td>0.5</td><td>0.9</td><td>0.8</td><td>0.0</td><td>-</td><td>10.3</td></tr></table>

Ablation Study. √ /× indicate using/not using the corresponding step respectively.

Table content: <table><tr><td>×</td><td>√</td><td>×</td><td>√</td><td>×</td></tr></table>

Peer review (iclr2019):
 This paper deals with the difficult problem of novelty recognition which is the core issue in open set learning and generalized zero-shot learning. Indeed a method able to separate samples between known and unknown domains in these settings would clearly indicate the direction for their solution. The idea proposed here consists in starting from the extreme value theory and then using bootstrapping to model the confidence score for a sample of belonging or not to a certain class. Through a probabilistic evaluation (based on K-S test) on the trustworthy of each category classifiers, the domain separation is extended to consider also an uncertain domain and the separation threshold is progressively refined. Once the domains are separated, classification can be performed disjointly in each of them. + Having a way to define known/unknown and uncertain samples on the basis of which one can then proceed to solve OSL and GZSL sounds as a very effective strategy. Moreover, all the parts of the method are based on reliable probabilistic principles. - Unfortunately the text is not easy to read. There are several repetitions and disordered lists (same numbers used multiple times or mixing names and numbers for the items) which distract the reader. As a side note, it would be better to avoid mentioning dataset names without their description and definition (*aPY* appears out of the blue in the introduction). - The experiments extends over different datasets and the ablation study is valuable. However to understand how the proposed method advances over the current state of the art it is important to consider and discuss the most recent publications on OSL and GZSL. See for instance Open Set Learning with Counterfactual Images, ECCV 2018 Feature Generating Networks for Zero-Shot Learning, CVPR 2018