Towards a Multilingual System for Vaccine Hesitancy using a Data Mixture Approach
Authors: Oscar Araque, Maria Felipa Ledesma Corniel, Kyriaki Kalimeri (Universidad Politécnica de Madrid, ETSI Telecomunicación, Intelligent Systems Group, ISI Foundation, 2023)
Topic: Fact Checking and Fake News Detection

Abstract
Understanding public narratives on contentious topics like vaccination adherence is vital for promoting cooperative behaviors. During the COVID-19 pandemic, significant polarization arose from concerns about vaccines, with misinformation and conspiracy beliefs proliferating on social media. While many studies have analyzed these narratives, the focus has largely been on English-language content. This linguistic bias limits comprehensive global insights. Our study introduces a novel multilingual approach that addresses this gap. By integrating Italian examples into a primarily English dataset, we detect vaccine-hesitant language and demonstrate the model’s adaptability to diverse linguistic data. Our findings highlight the importance of incorporating varied linguistic datasets for a more holistic understanding of global narratives on vaccine hesitancy.

Introduction
Automatically understanding peoples’ narratives on controversial social issues is fundamental to efficiently address the real concerns as they occur fostering collaborative, prosocial behaviors. Vaccination adherence is an exemplar case where society witnessed a notable polarization concerning possible adverse reactions [1, 2]. Especially during the COVID-19 pandemic and despite vaccines being the most efficient and cost-effective intervention, the spread of misinformation [3], the skepticism around the scientific development of COVID-19 vaccines and the dissemination of conspiracy beliefs [4, 5], proliferated on social media platforms. Numerous studies analyzed user-generated text [6, 7, 8, 9, 10], almost exclusively focusing on the English language due to the availability of models and tools. Even if often English is universally spoken, it limits the analysis in specific sociodemographic groups. Lenti et al. [11] in a purely network-based approach showed the existence of a global misinformation network, calling for a multilingual analysis to further understand the drivers of vaccine hesitancy in the various languages. Here, in light of these issues, we propose a novel approach for multilingual language understanding able to deal with language imbalance. More specifically, here we progressively include Italian instances in a predominantly English dataset for the task of vaccine-hesitant language detection and demonstrate the ability of the model to generalize on previously unseen data. Often, researchers and practitioners have access to large English datasets, but data in other languages, such as Italian, are lacking. We show that including small datasets in different languages can improve overall performance when analyzing texts in several languages.

Conclusion
Here we design and evaluate a method that achieves multilingual vaccine hesitancy detection. The experimental design considers training a multilingual classification model on a mixture of English and Italian text excerpts. Progressively varying the combination of languages in the training data, we obtain a better understanding of the classification problem in the two languages. Additionally, we undertook a novel data collection effort on Twitter, manually annotating content based on vaccination stance. This curated dataset is now freely accessible to the scientific community, providing a valuable resource for further research. By adjusting the language composition in our training data, we gained deeper insights into the classification intricacies across both languages. Notably, our findings suggest that the model can effectively generalize to a different language even when its training set contains a minimal proportion (e.g., 10%) of instances from that language. This indicates the model’s robustness and adaptability in handling linguistic variations with limited data. Importantly, this approach is an important tool for researchers and practitioners who often have access to large datasets in English, but limited resources in other widely spoken languages such as Italian or Spanish. The evaluation shows that composing a mixture dataset can be effective in generating a model that classifies instances in two languages. In fact, the experimentation shows that this mixture is flexible, maintaining consistent performances across different ratios of language presence. This consistency suggests that the mixture approach is promising. Given its language-neutral nature, our technique holds promise for broader applications across multiple languages and diverse domains. As a next step, we aim to explore various multilingual models and languages to further ascertain the scalability and adaptability of our approach.