# Zero-Shot Image Classification with Rectified Embedding Vectors Using a Caption Generator
## 3 Proposed Method  
### Overall Structure  
We consider a set of training samples \(D_{tr}=\{(I_{i},T_{i},y_{i}),i=1,\ldots N_{tr}\}\) with an associated class label \(y_{i}\in\mathrm{T}_{tr}\), where \(I_{i}\) is the \(i\)th training image and \(T_{i}\) is a textual description of \(I_{i}\). The goal of the zero-shot classification task is to predict \(y_{j}\in\mathrm{T}_{ts}\) of test image \(I_{j}\)s under the condition that training and testing classes are mutually disjoint, such that \(\mathrm{T}_{tr}\cap\mathrm{T}_{ts}=\varnothing\).  
The overall structure of the proposed model is presented in Figure 1. In the training stage, when image \(I_{i}^{tr}\) and text \(T_{i}^{tr}\) are given, the features of each piece of information are extracted through the CNN and long short-term memory (LSTM) network, respectively. These features are mapped onto the joint-embedding space through two fully connected layers to obtain vector representations \(\varphi_{I}(I_{i}^{tr})\) and \(\varphi_{T}\big{(}T_{i}^{tr}\big{)}\). In the joint-embedding space, the semantic relationship of the image and text pair is represented by the distance of the embedding vector pair. For example, an embedding vector of an image of a bird with a white peak and the text-embedding feature of the description \"white head\" are trained to be located nearby in the joint-embedding space.  
In the testing stage, a given \(I_{j}^{ts}\) is assumed to belong to an unseen class \(\mathrm{T}_{ts}\) in the training stage. Conceptual information for the new class must be provided to recognize it, and it is represented as a vector in the learned joint-embedding space, defined as a prototype vector \(\mathbb{P}\). Given the prototype vectors \(\mathbb{P}_{k}(k\in\mathrm{T}_{tr})\), the classification of test image \(I_{j}^{ts}\) is performed based on the distance between the image-embedding vector \(\varphi_{I}\big{(}I_{j}^{ts}\big{)}\) and prototype vectors \(\mathbb{P}_{k}(k\in\mathrm{T}_{ts})\). This method is similar to the baseline approach [8] in that the joint-embedding model is constructed in an intermediate space. While previous studies [7; 8] have used a single image vector \(\varphi_{I}\big{(}I_{j}^{ls}\big{)}\) in the embedding space, we propose using additional textual information to obtain a better representation of the image. For example, when we examine a red bird, a human analyzes its visual appearance and simultaneously thinks, \"It is a red bird.\" By combining this kind of linguistic thinking with visual representation, we expect to better represent the object in the cognition system. To implement this procedure, we generate a textual description \(G\big{(}I_{j}^{ls}\big{)}\) expressing the image \(I_{j}\) linguistically by exploiting the image-caption generator and use this description to obtain a rectified embedding vector \(R_{j}\) complementing the image feature. In addition, the generated text description can also be used in the training stage. The image-text data pairs sometimes do not exist or are very limited in a real situation, which results in poor performance because the joint-embedding space is not well-formed. We deal with this problem of insufficient data using the generated texts as an augmented description.