# Zero-Shot Image Classification with Rectified Embedding Vectors Using a Caption Generator
## 1 Introduction  
With the development of deep learning technology, the classification performance of image data has improved, and the field of application has expanded. This performance improvement was attributed to the developed feature extraction methodology and learning of a massive artificial neural network for numerous data designated by humans. However, the conventional methods for image classification are limited in that they do not work well when the classes in the testing dataset are not identical to those in the training set. Recently, zero-shot learning has been studied to overcome this problem. Zero-shot learning is the task of recognizing new patterns whose samples are unavailable during training [1, 2, 3, 4, 5, 6]. In zero-shot image classification, for example, a classifier that has learned using images of several bird species is asked to assign test images of new species to appropriate categories. Some conceptual information about the new species must be provided to the classifier in some form to perform this task. A representative form of the conceptual information is provided as predefined attributes such as the color of thebeak, but these kinds of predefined attributes are rarely given in real applications. As an alternative, more natural descriptions such as \"this bird has black feathers and a long beak.\" have also been used [7; 8; 9; 10; 11]. In this case, image-text matching plays an important role in assigning test images to an appropriate class defined by textual descriptions. Some researchers [7; 8] have introduced an embedding-based approach to solving this task, which creates a semantic embedding space containing representation vectors for both image and text. In the training stage, image and text features are extracted, respectively, and the two sets of features are mapped onto a common embedding space. This joint-embedding method trains the image and text features from a single pair to be close. In the test stage, some conceptual information about unseen classes is given as textual descriptions, and the trained model maps the descriptions to the embedding space to create a prototype vector for each unseen class. Then, the zero-shot classification of test images is performed by assigning the embedded feature of the test image to the class of the nearest prototype vector. The performance of this approach depends on whether the image features in the joint-embedding space are closely mapped to the corresponding class prototype. In other words, the image and text features belonging to the same category must be well-bound around the prototype in the embedding space. However, in zero-shot learning, test images are given from untrained classes, and their distributional properties can be different from those of training classes. This can lead to a large discrepancy between image features and class prototypes defined as textual features, which results in performance degradation. To deal with this problem, we propose rectifying the image-feature vector by using additional semantic information obtained from a caption generator [12; 13; 14]. A caption generator can generate a general description sentence that provides meaningful information for the given input images; thus, it can be used as a guide for enhancing the image feature. Therefore, by combining the two vectors, a better representation of the given input is expected than that from a single image feature. Figure 1 illustrates how the additional text feature given from the image caption can be combined with the original image feature to aid prediction and the comparison with prototype vectors. Furthermore, the generated description can also be used as augmented data in the training stage, improving the generalizability when the training data have insufficient text descriptions. Extensive experiments were conducted on the CUB-bird [15] and flower datasets [16] to verify the efficiency of the proposed model. Figure 1: An example of utilizing generated text as additional information for zero-shot image classification.