## 3 Hierarchical Semantic Loss and Confidence Estimator for Generalized Zero-Shot Learning  
### Zero-Shot Learning with Hierarchical Knowledge  
The proposed embedding-based generalized zero-shot learning model uses both visual data, e.g., images, and semantic data which is a class label of the visual data, as input data. Semantic data is a distributed representation vector obtained from a pre-trained language model. We can obtain visual and semantic vectors of the same dimension from the visual and semantic data through an embedding model based on deep neural networks. Unlike the general deep learning model which uses the same classes in both training and inference phase, the generalized zero-shot learning model distinguishesseen and unseen classes. The seen-classes are used in both the training and inference phase, but the unseen-classes are used only in the inference phase. There are no overlapping classes in the seen and unseen classes. The zero-shot architecture based on visual-semantic embedding classifies input visual data mainly through cross-modal retrieval, which involves finding a target class by retrieving a semantic vector of the closest distance by setting a visual vector as a query. At this time, zero-shot learning refers to the case where only the unseen-classes are used as the target semantic vectors for zero-shot cross-modal retrieval. In contrast, generalized zero-shot learning refers to the case where both seen and unseen classes are used as the target semantic vectors. That is, the difficulty of generalized zero-shot learning is higher than that of zero-shot learning. This paper proposes the zero-shot learning methodology using hierarchical semantic loss (HSL) for improving the performance of visual-semantic embedding. The proposed loss function uses hierarchical knowledge such as pre-defined WordNet hierarchy, hierarchy of label structure, or manually constructed hierarchy. Hierarchical knowledge represents the relation of semantic data and is used to construct HSL. All semantic data consist of distributed vectors obtained through a pre-trained language model. Figure 1 shows the structure of the proposed generalized zero-shot learning process and the notations used in the proposed zero-shot learning process are as follows. * \(x\): seen-class visual data such as an image in training dataset. * \(y\): seen-class semantic data such as a text label of \(x\) in training dataset. * \(x_{s}\): seen-class visual vector embedded from image embedding networks. * \(y_{s}^{+}\): positive seen-class semantic vector embedded from text embedding networks. * \(y_{s}^{-}\): negative seen-class semantic vector embedded from text embedding networks. It is selected by using hierarchical knowledge such that it has same super-class with \(y_{s}^{+}\). * \(y_{s}^{sc}\): seen super-class semantic vector obtained from pre-trained word vector and selected by using hierarchical knowledge. It is the super-class of \(y_{s}^{+}\) and \(y_{s}^{-}\). Figure 1: Zero-shot learning process with hierarchical semantic knowledge. In the training phase, only seen-class semantic data are used with visual data. At this time, the proposed methodology uses a more efficient loss function by using hierarchical knowledge about the seen-class semantic data. The seen super-class semantic vector is obtained from hierarchical knowledge, which is a fixed vector from a pre-trained language model. There are three examples presented as hierarchical knowledge sources: WordNet, label structure of training dataset, and manually constructed hierarchy [35; 36]. For example, a general ImageNet dataset consists of flat 1000 labels. In this situation, we can design the hierarchy by using WordNet, or manually construct the hierarchy directly based on the semantic similarity of the labels. In addition, if the hierarchical structure is already defined in datasets such as the CIFAR dataset, it can be used. HSL is a loss function using hierarchical knowledge for training each embedding model.