Multimodal Attention is all you need
Authors: Marco Saioni, Cristina Giannone (Università Guglielmo Marconi, Almawave, 2024)
Topic: Fact Checking and Fake News Detection

Abstract
In this paper, we present a multimodal model for classifying fake news. The main peculiarity of the proposed model is the cross attention mechanism. Cross-attention is an evolution of the attention mechanism that allows the model to examine intermodal relationships to better understand information from different modalities, enabling it to simultaneously focus on the relevant parts of the data extracted from each. We tested the model using textit{MULTI-Fake-DetectiVE} data from Evalita 2023. The presented model is particularly effective in both the tasks of classifying fake news and evaluating the intermodal relationship.

Introduction
Internet has facilitated communication by enabling rapid, immersive information exchanges. However, it is also increasingly used to convey falsehoods, so today, more than ever, the rapid spread of fake news can have severe consequences, from inciting hatred to influencing financial markets or the progress of political elections to endangering world security. For this reason, mitigating the growing spread of fake news on the web has become a significant challenge. Fake news manifests itself on the internet through text, images, video, audio, or, in general, a combination of these modalities, which is a multimodal way. In this article, we took the two, text and image, components of news as it proposed, for instance, in a social network. In this work we proposed an approach to automatically and promptly identify fake news. We use the dataset MULTI-Fake-DetectiVE{https://sites.google.com/unipi.it/multi-fake-detective}} competition, proposed in EVALITA 2023{https://www.evalita.it} AUTHOR}. The competition aims to evaluate the truthfulness of news that combines text and images, an aim expressed through two tasks: the first, which carries out the identification of fake news (Multimodal Fake News Detection); the second, which seeks relationships between the two modalities text and image by observing the presence or absence of correlation or mutual implication (Cross-modal relations in Fake and Real News). Our approach proposes a Transformer-based model that focuses on relating the textual and visual embeddings of the input samples (i.e., the vector representations of the text and images it receives as input). The aim was to find a way to reconcile the two different representation embeddings because they are learned separately from two different corpora, such as text and images, trying to capture their mutual relationships through some interaction between the respective semantic spaces. The remainder of the paper is structured as follows: Section 2 presents a brief overview of related work, and section 3 describes the architecture of the proposed model. Section 4 discusses the dataset supplied by the Evalita tasks, followed by an overview of our experiments in section 5. Sections 6 and 7 present the final results and our conclusions, respectively.

Conclusion
The Internet has facilitated the multimodality of communication by enabling rapid information exchanges that are increasingly immersive but increasingly used to convey falsehoods. In this study, a multimodal model for identifying fake news was proposed which is based on the mechanism of cross attention between the representations of the features learned by the network on the textual component of the news and those learned on the visual component associated with it. Many multimodal models are based on the concatenation of features learned from distinct modalities which, despite having good performance, however, limit the potential of the interaction between the features themselves. From the experiments carried out, the use of cross-attention demonstrated significant improvements in the performance of the model proposed in this work compared to the first two models classified in the MULTI-Fake-DetectiVE competition for both tasks requested by the organizers, despite the dataset available for training is very small in size and unbalanced both with respect to the categories to be predicted and with respect to the source of the news. Despite the intrinsic complexity of the two tasks, the cross-layer of the proposed model manages to express the representations learned from the text and images of a news story in a harmonious, collaborative and synergistic way, balancing their contributions and preventing one from taking over the other. Future developments concern the components of the model which could use a Visual Transformer AUTHOR instead of the ResNet in order to relate textual embeddings and visuals both generated by training a Transformer network.