# Zero-Shot Image Classification with Rectified Embedding Vectors Using a Caption Generator
## 2 Related Works  
**Zero-shot learning**: Early zero-shot image classification studies [19, 20, 21, 22] used a two-step strategy to predict the image class. First, when a new image comes in, the image attributes are predicted through the classifier obtained from the seen class in the learning phase. Second, the probability score of the attributes obtained in this way is combined with the prior attribute information of the unseen class and used for final class prediction. When combining the class attribute and attribute label in this two-stage approach, a domain shift problem occurs in which the attributes of two images from different classes overlap, and the performance deteriorates. For example, the shapes of a lion and a dog are very different, but the attribute called tail yields the same correct answer in the image-attribute classifier. This kind of information loss in the first stage leads to wrong predictions in the second stage. To solve this domain shift problem, some studies [7, 23, 24, 25, 26, 27, 28] have proposed locating the attribute and image in the same embedding space. When the input image and its corresponding attribute are given, their feature vectors in the joint-embedding space should be located closely. In the inference stage, the image class is assigned to the closest prototype by comparing the distance between the prototype composed of attributes and the input image in the same space. To construct the same joint-embedding space, a model embeds attribute data into the image space [7], the image into the attribute space [23, 26], and the image and attribute into the intermediate space [21, 24, 27, 28]. Some models [29, 30] that predict the class of the unseen image through the relationship between the image and semantic label using embedding are also presented. Although various approaches using predefined attributes have demonstrated good results, the fundamental problem is that these methods use attribute information. Predefined attributes rarely exist in real data and are expensive because humans must label them. The studies [7, 8] suggested using a free-style natural language description instead of an attribute to overcome this problem. Since such descriptions usually exist in image-text pair datasets that can be easily obtained through a web search, it is more natural and general to use the textual description as the conceptual definition of the test classes for zero-shot image classification. In this manner, the related works on image-text matching [17, 21, 22] have also been applied to zero-shot classification tasks. Recently, some research such as CLIP [27] and ALBEF [28] achieved outstanding performance for zero-shot recognition using transformer-based models [31, 32] with contrastive learning. Those models showed that a large number of image-text pairs can improve zero-shot recognition ability because they help to construct better joint-embedding space. However, those models are difficult to train because of high-computational costs. Though the embedding-based methods designed for general image-text matching have shown considerable success in zero-shot classification, they focus on the distance between individual image-text pairs and do not consider the class distribution. Since the class distributions of test images (unseen classes) are not observable while learning the embedding space, it is difficult to guarantee that the image features of a test class are well clustered around its corresponding prototype defined as textual features. In this paper, we try to obtain a better representation of test images by using additional textual information that is derived from images. **Image-caption generation**: Caption generation is a challenging artificial intelligence problem in which a text caption is generated from a given image. Recently, deep learning methods have achieved notable results [33; 34; 35; 36] for this problem with attention-based models. For the encoded image from a CNN, the image-caption-generation model predicts a sequence of words that describes the image through an RNN.