# Leverage Boosting and Transformer on Text-Image Matching for Cheap Fakes Detection +
## 4 Proposed Method
### Methodology  
This paper proposes two approaches to measuring the correlation between image and caption: image captioning and image-caption matching. **Image Captioning:** For the image captioning approach, we aim to utilize [37] to generate the content description of an image. We can use a pre-trained large-scale dataset on the STS [38] task (Semantic Textual Similarity) to measure the correlation between caption and image by converting the image's content to textual form. **Image-Caption Matching:** For the image-caption matching approach, we utilized a trained model of image-text matching on the MSCOCO dataset [35] to measure the correlation between caption and image. In this paper, we used the Visual Semantic Reasoning [39] method to measure the similarity between image and caption. See Figure 4 for illustration. The VSRN (Visual Semantic Reasoning) [39] method utilizes margin ranking loss as the objective function. The margin ranking loss objective is the correlation measurement of the matching caption-image, which is higher than the non-matching caption-image and not  
Figure 4: Illustration of boosting with image captioning method. First, the image will be inferred self-critically [37] to obtain a description of the image in textual form. Next, RoBERTa(MNLI) is utilized to extract the correlation between caption\({}_{1}\), caption\({}_{2}\), and image (NLI(caption\({}_{1}\), caption\({}_{2}\)), NLI(caption\({}_{1}\), caption\({}_{\text{image}}\)), NLI(caption\({}_{2}\), caption\({}_{\text{image}}\))).To overcome the difference between training data and testing data issues and improve performance, we take advantage of the boosting algorithm on the part of the testing data to combine results from our proposal and the COSMOS baseline. trying to make matching caption-image have a matching score higher than the threshold. As shown in Figure 5, the matching caption image's matching score has a different range of values. It can have a lower value compared with different captions and images that do not match each other. However, compared to the same image with another caption that is not matching, the correlation measurement of the matching caption image is higher than that of the non-matching caption image. Based on this attribute of the VSRN method and margin ranking loss, we normalized the matching score using Equation (4) to overcome this issue. See Figure 6 for illustration. \[\hat{\mathsf{S}}(I,C)=S(I,C)-\frac{1}{2N}\sum_{r}^{N}[S(I,C_{r})+S(I_{r},C)], \tag{4}\]  
where \(\hat{\mathsf{S}}\) defines the normalize matching score, and \(r\) defines the random index that satisfies \(C_{r}\
eq C\) and \(I_{r}\
eq I\). By subtracting the mean of the matching score from the \(N\) sample, the result can express the correlation degree of the given matching image caption compared with other non-matching image captions. Hence, to estimate the correlation between two captions better, instead of using only cosine similarity measures from other methods trained on the STS task [38], we also used other methods on the NLI task (Natural Language Inference) [40] to express the semantic relation between two captions. We chose SBERT-WK [41] and RoBERTa [42] to extract semantic relations between two captions. One of the difficulties of the COSMOS dataset is that training/validation data have a different construct from testing data. In training data, each captioned image consists of only a not-out-of-context pair, and captions are always trustworthy news and match the image's context. While in testing data, data consist of out-of-context and not-out-of-context captioned images. The caption can be fake news, descriptions about the image, or match/mismatch with the image and other captions. Based on our experience, fine-tuning training data and evaluating directly on testing data gave poor results.