## 4 Proposed Model
### Semi-Fixed Word Embedding  
In the current novel image captioning task, some models initialize word embedding via glove [44]. The experiments in Nocaps [45] use glove for initial word embedding and demonstrate that glove is useful for rarely seen and never-seen words (out-of-domain), but harmful for common words (in-domain). For in-domain, the model with glove outperforms the model without glove by 0.4 in Spice but underperforms by 0.7 in CIDEr. According to the definitions of SPICE and CIDEr, CIDEr takes into account non-visual words, while SPICE only considers the relationship between visual words. This shows that glove works well for visual words, but slightly worse for non-visual words.  
Considering the CLIP predicts the correct image-text pair from the visual content. The non-visual words do not have visual information, while the novel object captioning task aims to describe the novel object. The word for the novel object must be a visual word. Therefore, we decide to fix the embedding of visual words through the CLIP feature and learn the embedding of non-visual words from the training set.  
Figure 4 shows the structure of our proposed Semi-Fixed Word Embedding. It mainly consists of three parts. The first part is a trainable non-visual word embedding \(E_{inv}\). The second part is fixed in-domain visual word embedding \(E_{inv}\), which enables the captioning model to understand the meaning of CLIP noun features. The third part is the never-seen visual word feature \(E_{ow}\) used in the inference phase, which allows the embedding layer to be extensible and can be used to understand the meaning of the novel words specified by the CBS.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & & & & & \\ \hline panda & 20.94 & 16.97 & 17.39 & 78.63 \\ bear & 15.62 & 18.31 & 24.72 & 100.13 \\ koala & 13.55 & 26.08 & 17.30 & 78.00 \\ animal & 15.48 & 18.63 & 19.50 & 82.69 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Similarity results of word with image.  
Figure 3: The structure of the CLIP [40].  
In the training phase and inference phase, word features can be calculated by Equation (2) and Equation (3), respectively.  
\[v_{w}=concat(E_{mw},E_{iw})\Pi_{t} \tag{2}\] \[v_{w}=concat(E_{mw},E_{iw},E_{ow})\Pi_{t} \tag{3}\]  
where \(\Pi_{t}\) is the one-hot encoding of the input word \(y_{t}\) at time step \(t\).  
The traditional captioning model calculates the probabilities of candidate words through the fully-connected layer and softmax layer as follows,  
\[p(y_{t}\mid y_{1:t-1})=\text{softmax}\big{(}W_{p}h_{t}+b_{p}\big{)} \tag{4}\]  
where \(\mathbf{h}_{t}\) is the hidden semantic feature. The size of the fully-connected layer is unchangeable in the inference stage.  
The traditional CBS predicts \(y_{t}\) through Equation (4) for each case based on \(p(y_{t}\mid y_{1:t-1})\). When the selected word is not included in the vocabulary, the probability and loss for the selected word are uncomputable. Hence, this manuscript calculates the loss for each word as follows:  
\[logp(y_{t}\mid y_{1:t-1})=\left\{\begin{array}{rl}&logp(y_{t}\mid y_{1:t-1}),\text{ if }y_{t}\in vocab\\ &l_{fix},\text{ if }y_{t}\
otin vocab\end{array}\right. \tag{5}\]  
where \(y_{t}\) can be a selected tag or generated word. \(l_{fix}\) is a fixed loss. This modification enables CBS to generate words beyond the dimension of the final fully-connected layer.