## 3 Hierarchical Semantic Loss and Confidence Estimator for Generalized Zero-Shot Learning
### Zero-Shot Learning with Hierarchical Knowledge
It consists of hierarchical mean squared error (HMSE) and hierarchical triplet margin loss (HTL). HMSE is used to minimize the distance between a visual vector \(x_{s}\) and a seen super-class vector \(y_{s}^{cs}\) and the distance between a seen class semantic vector \(y_{s}^{+}\) and \(y_{s}^{cs}\). HTL allows the embedding networks to be trained such that the distance between the positive pairs becomes smaller than the distance between the negative pairs. Here, hierarchical knowledge is used in selecting the negative seen-class semantic vectors \(y_{s}^{-}\). HSL is based on the existing embedding-based zero-shot learning methodology that performs joint learning of image and text embedding networks. HSL is used in training each image and text embedding networks to embed semantically similar \(y_{s}^{+}\) in similar locations for \(x_{s}\). In this case, HMSE is the sum of distances from \(x_{s}\) and \(y_{s}^{+}\) to fixed \(y_{s}^{cs}\). However, using only HMSE makes it difficult to distinguish between visual vectors and other seen-class semantic vectors that have the same super-class, resulting in performance degradation of zero-shot inference. Therefore, HTL is applied to solve this problem. When selecting a negative sample of triplet loss, HTL selects \(y_{s}^{-}\) among the semantic vectors that have the same super-class as \(y_{s}^{+}\). This enables embedding more efficient so that the distinction between similar semantic vectors become clearer. The HMSE of the proposed HSL is given as:  
\[HMSE(x_{s},\ y_{s}^{+},\ y_{s}^{sc})\ =\ \parallel x_{s}-y_{s}^{sc}\parallel \frac{2}{2}\ +\ \parallel y_{s}^{+}-y_{s}^{sc}\parallel\frac{2}{2} \tag{3}\]  
In Equation (3), the distance is measured with l2 norm and the visual-semantic embedding networks are trained in the direction of minimizing it. In the above equation, because HMSE uses a fixed super class semantic vector \(y_{s}^{sc}\), all embedding networks can be trained faster and stably. The HTL of the proposed HSL is given as:  
\[HTL(x_{s},\ y_{s}^{+},\ y_{s}^{-},\alpha)=max(0,\ \alpha\ +\ \parallel x_{s}-y_{s}^{+} \parallel\frac{2}{2}\ -\ \parallel x_{s}-y_{s}^{-}\parallel\frac{2}{2}) \tag{4}\]  
In Equation (4), The HTL is defined by differences between distance of anchor \(x_{s}\) to positive pair \(y_{s}^{+}\) and distance of anchor \(x_{s}\) to negative pair \(y_{s}^{-}\). If the distance to the \(y_{s}^{-}\) from \(x_{s}\) is greater than the sum of margin \(\alpha\) and distance to \(y_{s}^{+}\) from \(x_{s}\), the HTL is set to 0 so that learning does not proceed anymore. In general, the triplet margin loss sets all seen-class semantic vectors as candidates of negative pair except the positive one. In contrast, the proposed method sets only the seen-class semantic vector that has the same super-class with positive visual vectors as candidates of negative pairs, reflecting the hierarchical knowledge of semantic class labels. Through this, the proposed HTL can train embedding networks more effectively for the input visual data. \[HSL(x_{s},y_{s}^{+},\ y_{s}^{-},\ y_{s}^{cs},\alpha,\ \lambda)=\lambda \ast HMSE(x_{s},\ y_{s}^{+},\ y_{s}^{cs})\ +\ (1-\lambda)\ast HTL(x_{s},\ y_{s}^{+},\ y_{s}^{-},\alpha) \tag{5}\] \[\underset{\theta_{l},\ \theta_{r}\in\mathbb{R}}{\text{argmin}}\ HSL(x_{s},y_{s} ^{+},\ y_{s}^{-},\ y_{s}^{cs},\alpha,\ \lambda) \tag{6}\]Equation (5) represents the proposed HSL adjusted by learning coefficient \(\lambda\). For example, the embedding networks might be accelerated faster than the relative differences between the child semantic labels if we assign them a relatively large learning coefficient \(\lambda\). On the contrary, when the learning coefficient \(\lambda\) is relatively small, we can expect detailed embedding results through hierarchical triplet margin loss. Finally, Equation (6) is a training objective function that minimizes HSL by adjusting weights of image embedding networks \(\theta_{l}\) and text embedding networks \(\theta_{T}\).