## 1 Introduction  
Image captioning is a classical task that aims at automatically describing an image using natural language. This research is at the intersection of two popular research fields, i.e., Computer Vision and Natural Language Processing. This task is simple for humans. However, it is a considerable challenge for computers, which not only requires understanding the content of the image but also generates sentences with human language habits. Image captioning has wide applications in image retrieval, image indexing, human-computer interaction, assisting visually impaired people, automatic video security monitoring, and journalism. Because of its importance, it is becoming a research hotspot. Typically, image captioning models are based on the encoder-decoder structure. Most models are trained on parallel data of image-sentence pairs. Some models [1; 2; 3; 4] even outperform humans on the MSCCO dataset in some metrics. Those models perform well when describing the objects that often appear in the training set, but fail to describe novel objects (rarely seen or never-seen). The most straightforward way to overcome this limitation is to collect more data [5; 6]. Such methods are effective, but the data collection process is highly expensive and time-consumption. In order to reduce consumption while still maintaining a competitive performance, most inexpensive approaches rely on the object detection model [7; 8; 9]. Modern object detection models can recognize a wide range of objects through incredibly large training sets [10] or zero-shot methods [11]. Those captioning models improve the noun-region matching ability by describing the nouns corresponding to the attended regions. Those models are good for describing rarely seen objects but are useless for never-seen objects. In particular, the number of never-seen objects is unknown while the dimension of the word embedding layer and final fully-connected layer in the captioning model is fixed. It is impossible to understand the meaning of the never-seen word and describe it. Even though the current novel captioning model can generate sentences containing selected tags through constrained beam search (CBS) [12], when the selected tag is semantically unclear novel words, it is harmful to the probability of subsequent words and affects the final sentence selection. The harm is especially severe when the selected tag is incorrect. However, the current novel object captioning model does not address this issue. To overcome those issues, we propose an external knowledge-based novel image captioning method that purposefully uses external knowledge to extend the word embedding casually, and makes the captioning model understand the meaning of the extended words. Specifically, the main contribution of our work is two blocks, i.e., the Semi-Fixed Word Embedding and Candidate Sentences Selection. The Semi-Fixed Word Embedding is a new embedding layer that can be extended to the never-seen word through external knowledge while assisting the captioning model in understanding the meaning of the extended never-seen word. The Candidate Sentences Selection block assists the captioning model in the selection of the candidate sentences to avoid the influence of the novel words in CBS. The study's contributions are as follows:  
1. We propose a novel prior-based word embedding block that incorporates semantic information to enable the model to understand various words, including never-seen words. It is beneficial for the captioning model to understand the meaning of the novel words and generate more accurate subsequent words. 2. We propose a new block to evaluate candidate sentences through the semantic similarity between generated sentences and the given image which is obtained by external knowledge so as to reduce the influence of never-seen words. 3. Proposed blocks can be attached to a detection-based captioning model to enable them to describe the rarely seen and never-seen objects. 4.