# Leverage Boosting and Transformer on Text-Image Matching for Cheap Fakes Detection +
## 4 Proposed Method
### COSMOS Baseline  
In prior research on image and news veracity classification, the method usually aims to utilize multi-modal by extracting features of text/captions and attached images through a pre-trained convolution neural network, LSTM [32] or BERT [33], layer and combine these features by concatenating or sum function with the appropriate objective function. This approach can take advantage of multiple datasets such as imagenet, MSCOCO, STS, and MNLL... for the basis of understanding and representing semantic information of data and fine-tuning other news datasets to improve performance.  
Besides the advantage of prior research, it is also limited in terms of the dataset's attributes. Most of the prior work uses fine-tuning on the new dataset, which makes it limited in many respects, such as in categories and characteristics of news, and cannot cover all subjects or situations not included in the dataset.  
In COSMOS, the author aims to match the caption with the most correlated object in the image by utilizing self-supervised learning. To do this, the author first uses Mask-RCNN [34] on MSCOCO [35] and selects the top 10 ROIs (Region of Interest) with the highest detection score and additional features of the entire image. For text pre-processing and processing, the author first makes use of NER (Named Entity Recognition) to generalize captions and then infers through USE (Universal Sentence Encoder) [36] to extract caption embedding. Next, the author infers the bounding box and caption embedding through a linear layer for mapping to the same dimension. The paper also uses max margin ranking loss [30] as objective/loss function using the equation:  
\[\mathcal{L}=\frac{1}{N}\sum_{i}^{N}\text{max}(0,(S_{IC}^{r}-S_{IC}^{m})+\alpha), \tag{1}\]  
where \(S_{IC}^{r}\), \(S_{IC}^{m}\) is the measure of similarity between a random caption-image pair and a matching caption-image pair, and \(\alpha\) is the margin parameter. This measure is calculated by the maximum dot function between \(11\) ROIs and matching/random caption. The similarity measure function is illustrated as Equation :  
\[S_{IC}=\max_{i}^{N}{(b_{i}^{T}c)}, \tag{2}\]  
where \(b_{i}\) is the features of the proposal bounding box and \(c\) is the features of the caption.  
At testing time, for each captioned image (caption\({}_{1}\),caption\({}_{2}\),image), the COSMOS method uses the simple if else rule to determine out-of-context captioned images:  
\[\begin{cases}OOC,&\text{If IoU}(B_{IC_{1}},B_{IC_{2}})>t_{i}\ \&\ S_{sim}(C_{1},C_{2})<t_{c}\\ \text{NOOC, otherwise},\end{cases} \tag{3}\]  
where IoU(\(B_{IC_{1}},B_{IC_{2}}\)) is the intersection-over-union of two bounding boxes having the largest value of similarity measure with the corresponding two captions; \(S_{sim}(C_{1},C_{2})\) is the similarity measure defined in cosine space, and \(t_{i}\), \(t_{c}\) is the fixed threshold of IoU(\(B_{IC_{1}},B_{IC_{2}}\)) and \(S_{sim}(C_{1},C_{2})\).  
By matching and comparing two captions with the corresponding object, the author can assess if two captions mention a related subject/object or not (determined by IoU(\(B_{IC_{1}},B_{IC_{2}}\)). If two captions mention a related subject/object and have uncorrelated semantic similarity (determined by \(S_{sim}(C_{1},C_{2})\)), then the given captioned image is out-of-context. The other situation is not-out-of-context.