# Generalized Image Captioning for Multilingual Support
## 3 Filter Captioning Model Algorithm
### Image Caption Fine-Tuning  
In previous studies, as a fine-tuned learning Algorithm 1 for image captioning, image object tags and regional vectors for objects are received as inputs and learned. However, when learning in a conventional manner, the image caption model relies on image object tags. When the order of image object tags changes or some variations occur during inference after fine-tuned learning of image caption problems, normal captions for proper images may not be generated because they rely on image object tags. ``` Input: Object Detection Tag T, Vector V, Random Caption Word W Output: Image Caption weights load Pretraining BERT Model; for\(v_{i},t_{i},w_{i}\in V\), \(T\), \(W\)do \(r\gets uniform(0,1)\) if\(r\geq 0.9\)then \(t_{i}=random_{c}hoice(W)\)  end for for\(i\) in generate_lengthdo \(w\gets w_{i+1\ generate\_length}masking\) \(embedding\_words=Word\_Embedding(d_{i})\)  embedding_words += position_embedding + type_embedding  embedding_words = Dropout(Layer_norm(embedding_words)) \(embedding\_img=Dropout(Layer\_norm(linear(v_{i})))\)  embedding = cat(embedding_words, embedding_img) for\(j\) in range(12))do  embedding = self_attention(embedding)  embedding = Layer_norm(Dropout(dense(embedding + w)))  embedding = gelu(dense(embedding))  end for output = tanh(dense(embedding[-1]))  output = Layer_norm(Dropout(dense(output))) + bias \(w_{i}=output\)  end for  end for The filter caption model presented in this paper generates image captions by prioritizing the order of image object tags and area vectors by domain object dictionaries. For words in the domain object dictionary, it makes the model more focused on the domain object, leading to the creation of the desired image caption. At this time, when the model relies on the image object tag, a problem occurs in which normal tags cannot be generated. To address this problem, this paper proposes a filter captioning model, which is a method that can solve problems that depend on image caption tags of existing models and generate a variety of domain image caption information. For the implementation of image captioning during fine-tuned learning, as shown in Figure 1, if the image caption tag appears in the image caption, the order is changed with a 30% probability, and the segment embedding is changed to 2. To reduce image object tag dependence, we randomly extract image object tags with a probability of 5% and replace them in the dataset. By replacing the image object tag with one of the caption word data, the dependence on the image object tag is reduced, and the overall content of the image object tag can be described. ### Image Caption Inference

This section explains how to build a domain object dictionary corresponding to the domain to be applied among image object tags and induce image captioning using the built domain dictionary In Algorithm 2 and Figure 2, rather than building and learning image caption data for each domain to be applied. Figure 1: Image Caption It is an inference structure The Domain object dictionary re-aligns the location of the object to induce captions. for image captioning inference. Figure 2: Image embedding I and image object tags through the CNN-based backbone network are input to the multi-layer transformer. To avoid overfitting, 30% of the tags generated and 5% of regular tags randomly change segment embedding to 2. #### 3.4.1 Domain Object Dictionary

The domain object dictionary induces the generation of captions by rearranging the order of the input data, image object tags, and local information about the image object, according to the domain, when creating an image caption. It is a method of selecting and applying tags related to the domain to be applied among image object tags without having to build all words related to the domain when creating a domain object in advance. It is easy to build a dictionary without professional domain knowledge by selecting a tag related to the desired domain among image object tags and adding it to the dictionary.