## 3 Hierarchical Semantic Loss and Confidence Estimator for Generalized Zero-Shot Learning
### Zero-Shot Inference with Unseen Confidence Estimator  
The zero-shot architecture based on visual-semantic embedding classifies the input visual data mainly through cross-modal retrieval. Typical embedding-based zero-shot learning methods show performance degradation as the number of semantic candidates for inference increases. This is especially as most of the existing methodologies show higher performance degradation when the number of semantic classes for the observed data increases as compared to the case when the number of sematic classes for unseen data increases [7]. In other words, the existing zero-shot learning model tends to embed the unseen visual data into seen semantics. Therefore, a methodology is needed to mitigate the problem of embedding unseen visual data into the semantic of the seen data. A typical approach of this method is to design an estimator that estimates the confidence score to distinguish whether the input data are seen or unseen. Figure 3 shows the structure of the proposed generalized zero-shot  
Figure 2: Description of hierarchical semantic loss. inference architecture and the notations used in the proposed generalized zero-shot inference phase are as follows. * \(x\): visual data comprise unseen-classes not used in training phase. * \(y\): semantic data comprise both seen-classes used in training phase and unseen-classes. * \(x_{q}\): visual vector embedded from image embedding networks as query of cross-modal retrieval task. * \(y_{s}\): seen-class semantic vector embedded from text embedding networks as targets of cross-modal retrieval task. * \(y_{u}\): unseen-class semantic vector embedded from text embedding networks as targets of cross-modal retrieval task. * \(D_{s}\): distances between visual vector query and seen-class semantic vectors. * \(D_{u}\): distances between visual vector query and unseen-class semantic vectors. * \(s_{uc}\): unseen confidence score from unseen confidence estimator adjusting visual-unseen distance \(D_{u}\). The proposed generalized zero-shot inference architecture comprises trained image embedding networks, trained text embedding networks, and an unseen confidence estimator (UCE) based on pre-trained convolutional neural networks for the classification of seen class visual data. The proposed architecture embeds both seen-class and unseen-class semantic data into a common vector space through text embedding networks for creating the target seen-class semantic vector \(y_{s}\) and unseen-class semantic vector \(y_{u}\). We can perform zero-shot cross-modal retrieval using a visual vector \(x_{q}\) from image embedding networks as a query. However, as the embedding model is trained by the seen-classes, zero-shot cross-modal retrieval in this manner is not expected to achieve high performance. In this paper, we calculate an unseen confidence score \(s_{uc}\), which indicates whether the input visual data correspond to seen or unseen classes. We design \(s_{uc}\) to have a large value if the maximum value of softmax for the seen-class data is low. Then the \(s_{uc}\) adjusts the distance \(D_{u}\) between the query \(x_{q}\) and targets \(y_{u}\). Figure 3: Generalized zero-shot inference architecture with unseen confidence estimator. The confidence estimator includes pre-trained convolution neural networks, which perform classification on the seen-class training dataset. The classification result to the confidence estimator is expressed as a type of probability value for each class using the softmax function. The value of the largest softmax selected as the classification result is set as certainty for the seen-class. Using this, we calculate \(s_{uc}\) to apply it to \(D_{u}\) to make it smaller than \(D_{s}\) especially when the certainty for the seen class is very low. So, we can perform more efficient zero-shot inference using the unseen confidence score \(s_{uc}\) when unseen visual data are input.