Adaptive Confidence Smoothing for Generalized Zero-Shot Learning ∗Yuval Atzmon
Bar-Ilan University, NVIDIA ResearchAbstract
Generalized zero-shot learning (GZSL) is the problem oflearning a classifier where some classes have samples andothers are learned from side information, like semantic attributes or text description, in a zero-shot learning fashion(ZSL). Training a single model that operates in these two
regimes simultaneously is challenging. Here we describe
a probabilistic approach that breaks the model into threemodular components, and then combines them in a consistent way. Specifically, our model consists of three classifiers: A “gating” model that makes soft decisions if a sample is from a “seen” class, and two experts: a ZSL expert,
and an expert model for seen classes. We address two maindifficulties in this approach: How to provide an accurate
estimate of the gating probability without any training samples for unseen classes; and how to use expert predictions
when it observes samples outside of its domain.
The key insight to our approach is to pass information
between the three models to improve each one’s accuracy,
while maintaining the modular structure. We test our approach, adaptive COnfidence SMOothing (COSMO ), on
four standard GZSL benchmark datasets and find that itlargely outperforms state-of-the-art GZSL models. COSMO
is also the first model that closes the gap and surpasses theperformance of generative models for GZSL, even-though it
is a light-weight model that is much easier to train and tune.

1. Introduction
People can easily learn to recognize visual entities basedon few semantic attributes. For example, we can recognize
a bird based on visual features (long beak, red crown), or
find a location based on a language description (a 2-stories
brick town house).
Taking into account the semantics of attributes becomes
crucial when no training samples are available. This learning setup, called Zero-Shot Learning (ZSL) is the task oflearning to recognize objects of classes without any training
samples [6]. Instead, learning is based on semantic knowledge about the classes [7, 1], a “class description”.
∗ An extended version of this work was published in the proceedings ofConference on Computer Vision and Pattern Recognition, 2019 [2]Gal Chechik
Bar-Ilan University, NVIDIA Research

Generalized zero-shot learning (GZSL) [3] is a realistic task that extends ZSL to make predictions when test data
has both seen and unseen classes. GZSL poses a unique
combination of hard challenges: First, the model has tolearn effectively for classes without samples (zero-shot), it
also needs to learn well for classes with plenty of samples,and finally, the two very different regimes should be combined in a consistent way in a single model. GZSL can beviewed as an extreme case of classification with unbalanced
classes, hence solving the last challenge can lead to betterways to address class imbalance, which is a key problem inlearning with real-world data.
These learning problems operate in different setups,
hence combining them into a single model is challenging.
Here we describe an architecture that combines three modules, each focusing on one problem [7, 10]. It uses two(Seen / Unseen) domain expert classifiers and a soft gating
mechanism that combines the expert predictions.
Unfortunately, softly combining expert predictions
raises several difficulties. First, when training a gating module, it is hard to provide an accurate estimate of the probability that a sample is from the ”unseen” classes, since bydefinition no samples have been observed from that class.
Second, in soft combination, each model also contributes
its beliefs to samples from the ”other” domain, typically
producing falsely confident spurious predictions that confuse the GZSL-mixture, because multi-class models tend toassign most of the softmax distribution mass to very few
classes, even when their input is random noise [4].
Our main contributions address the main difficulties inthis approach: (1) A novel confidence-based gating networkthat estimates the gating probability without any training
samples for unseen classes. (2) A novel adaptive prior applied during inference that smooths an expert prediction, if
it believes that an image is out of the expert domain.

2. The Zero-Shot Learning Setup
In Zero-shot learning, a training set has labeled samples from a set of seen classes S. At test time, a new setof samples is given from a set of unseen classes U. Ourgoal is to predict the class of each sample. As a supervision signal, each class y ∈ S ∪ U is accompanied with asemantic class description vector ay . ZSL Probabilistic ap-

0123315672232 $%&'"(#)
893  C@??,D33
73
7






!
"#
56722
3315672232
33
C@??,D6789:;<=To conclude, we clarify that S and U classes do not overlap. However, the unseen expert can also produce predictions about the seen classes which we use for estimating thegater probability. More details in the extended version [2].
Zero-Shot Expert: Here we use LAGO [1], which models
new classes as compositions of soft AND-OR expressions.
Seen Classes Expert: For seen classes, we train a logistic
regression classifier over pre-trained CNN [5] features.

>?@A-.B*+,-./01
3312012331
3 741
56722443

4. Experiments

Figure 1: Qualitative illustration our GZSL model.

proaches learn a compatibility score for samples and classdescriptions F (ay , x), that assigns a probability for eachclass p(Y=y|x)=F (ay , x). Y viewed as a random variable
for the label y of a sample x. In Generalized ZSL, samples
are drawn from either seen or unseen domains: Y ∈ S ∪ U.
Notation: For brevity, below we drop conditioning on x.

3. Our approach
By the law of total probability, we can derive a probabilistic method that divides the GZSL problem to modules:
p(y) = pS (y|S)pGate (S) + pZS (y|U)pGate (U).

(1)

Here, pS (y|S) is a model trained to classify seen Sclasses. Similarly, pZS (y|U) is a model classifying unseen
U classes, namely a ZSL model. Finally, pGate (S) is a gating classifier, that distinguishes seen from unseen domains
and combines the experts in a soft way.
Confidence-Based Gating Model: We train a network on
top of the softmax output of the two experts, with the goalof discriminating U from S images. Since no samples fromU are available, we create a hold-out set of classes from S,
which are not used for training the experts, and use themto estimate the output response of the expert classifiers overimages of unseen classes.
Adaptive Confidence Smoothing: When a classifier ispresented with a sample from an unfamiliar class, we would
intuitively wish that it outputs uniform low probability to
all classes, because they are all ”equally wrong”. However,classifiers tend to to assign most of the probability mass tofew (incorrect) classes. To address this, for the unseen expert classifier p(y|U), we apply the law of total probability
and weigh two terms: (1) The classifier predictions p(y|U);(2) A uniform smoothing prior π U . They are weighed by
the belief that the input image is from a class that is familiar
to the expert p(U) or unfamiliar (1 − p(U)).
p0 (y|U)= p(U)p(y|U) + (1−p(U))π U ,We tested our approach, adaptive COnfidence SMOothing (COSMO ), on four GZSL benchmarks (AWA, SUN,CUB, FLOWER), following standard evaluation protocols[9, 8]. We report AccH , the harmonic mean of accuracy
over seen classes, and accuracy over unseen classes. We
also compute a seen-unseen accuracy curve by sweeping
over decision threshold of the gating network. The curve
trades the performance of seen and unseen domains.
Results: Table 1 describes test accuracy of COSMO andcompared methods over the three benchmarks. Comparedwith non-generative models, COSMO improves the harmonic accuracy AccH by a large margin for all datasets.
Notably, accuracy of COSMO is comparable with state-ofthe-art generative models in spite of COSMO being mucheasier to train and tune than GAN-based methods.
METHOD / DATASETAWA SUN CUB FLOWERNON-GENERATIVE
DEM (CVPR 2017)
K ERNEL (CVPR 2018)
TRIPLE (TIP 2019)
RN (CVPR 2018)

47.3
29.8
38.6
46.7

23.6
28.1
-29.2
28.9
37.2
47-GENERATIVE
F -CLSWGAN (CVPR 2018)
CYCLE -WGAN (ECCV 2018)

59.6
59.8

39.4
39.4

49.7
53.0

65.6
65.2

COSMO AND BASELINESDCN (NIPS 2018)
LAGO (UAI 2018)
CS+LAGO (ECCV 2016)(O URS ) COSMO (CVPR 2019)

39.1
33.7
54.5
63.6

30.2
23.9
29.8
41

38.7
35.1
48.9
50.2

68.8

Table 1: Comparing AccH of COSMO with state-of-the-art GZSL models.

The seen-unseen plane (Figure 2): We provide a full SeenUnseen curve (blue dots) that shows how COSMO tradesoff the metrics. COSMO can be tuned to select any operation point along the curve, and achieves better or equivalent
performance at all regions.

(2)and similarly p0 (y|S) = p(S)p(y|S) + (1 − p(S))π S . The
rightmost term means that given that we know an image isunfamiliar for an expert, we assign a uniform low probability, which is weighed by the belief that the input image isindeed unfamiliar.

Figure 2: The Seen-Unseen curve for COSMO (blue dots). Squares: best
COSMO model and its LAGO-based baselines, Triangles: non-generative
approaches, ’X’: approaches based on generative models.

References[1] Y. Atzmon and G. Chechik. Probabilistic and-or attributegrouping for zero-shot learning. In Proceedings of theThirty-Forth Conference on Uncertainty in Artificial Intelligence, 2018.
[2] Y. Atzmon and G. Chechik. Adaptive confidence smoothingfor generalized zero-shot learning. In CVPR, 2019.
[3] R. Chao, S. Changpinyo, B. Gong, and S. F. An empirical
study and analysis of generalized zero-shot learning for object recognition in the wild. In ICCV, 2016.
[4] D. Hendrycks and K. Gimpel. A baseline for detectingmisclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
[5] K.He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385,2015.
[6] C. Lampert, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by between-class attribute transfer.
In CVPR. IEEE, 2009.
[7] R. Socher, M. Ganjoo, C. Manning, and A. Ng. Zero-shotlearning through cross-modal transfer. In NIPS, 2013.
[8] Y. Xian, C. Lampert, B. Schiele, and Z. Akata. Zero-shotlearning - A comprehensive evaluation of the good, the badand the ugly. arXiv preprint arXiv:1707.00600, 2017.
[9] Y. Xian, B. Schiele, and Z. Akata. Zero-shot learning - thegood, the bad and the ugly. In CVPR, 2017.
[10] H. Zhang and P. Koniusz. Model selection for generalized
zero-shot learning. In The European Conference on Computer Vision (ECCV) Workshops, September 2018.