GenAI4ELab/papercli-papers huggingface dataset:
DRAGON at FIGNEWS 2024 Shared Task: a Dedicated RAG for October 7th conflict News
Authors: Sadegh Jafari; Mohsen Mahmoodzadeh; Vanooshe Nazari; Razieh Bahmanyar; Kathryn Burrows
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: In this study, we present a novel approach to annotating bias and propaganda in social media data by leveraging topic modeling techniques. Utilizing the BERTopic tool, we performed topic modeling on the FIGNEWS Shared-task dataset, which initially comprised 13,500 samples. From this dataset, we identified 35 distinct topics and selected approximately 50 representative samples from each topic, resulting in a subset of 1,812 samples. These selected samples were meticulously annotated for bias and propaganda labels. Subsequently, we employed multiple methods like KNN, SVC, XGBoost, and RAG to develop a classifier capable of detecting bias and propaganda within social media content. Our approach demonstrates the efficacy of using topic modeling for efficient data subset selection and provides a robust foundation for improving the accuracy of bias and propaganda detection in large-scale social media datasets.
LexiconLadies at FIGNEWS 2024 Shared Task: Identifying Keywords for Bias Annotation Guidelines of Facebook News Headlines on the Israel-Palestine 2023 War
Authors: Yousra El-Ghawi; Abeer Marzouk; Aya Khamis
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: News bias is difficult for humans to identify, but even more so for machines. This is largely due to the lack of linguistically appropriate annotated datasets suitable for use by classifier algorithms. The FIGNEWS Subtask 1: Bias Annotation involved classifying bias through manually annotated 1800 headlines from social media. Our proposed guidelines investigated which combinations of keywords available for classification, across sentence and token levels, may be used to detect possible bias in a conflict where neutrality is highly undesirable. Much of the headlines’ percentage required contextual knowledge of events to identify criteria that matched biased or targeted language. The final annotation guidelines paved the way for a theoretical system which uses keyword and hashtag significance to classify major instances of bias. Minor instances with bias undertones or clickbait may require advanced machine learning methods which learn context through scraping user engagements on social media.
Uot1 at FIGNEWS 2024 Shared Task: Labeling News Bias
Authors: Abdusalam Nwesri; Mai Elbaabaa; Fatima Lashihar; Fatma Alalos
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This paper outlines the University of Tripoli’s initiative in creating annotation guidelines to detect bias in news articles concerning the Palestinian-Israeli conflict. Our team participated in the Framing of Israeli Gaza News Media Narrative (FIGNEWS 2024) shared task. We developed annotation guidelines to label bias in news articles. Using those guidelines we managed to annotate 3,900 articles with the aid of our custom-developed annotation tool. Among 16 participating teams, we scored 48.7 on the macro F1 measure in the quality track in which we ranked 4th. In the centrality track we were ranked at the 6th position using the macro F1 avg measure, however, we achieved the 4th best kappa coefficient. Our bias annotation guidelines was ranked in the 9th position.
NLPColab at FigNews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media
Authors: Sadaf Abdul Rauf; Huda Sarfraz; Saadia Nauman; Arooj Fatima; SadafZiafat SadafZiafat; Momina Ishfaq; Alishba Suboor; Hammad Afzal; Seemab Latif
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: In this paper, we present our methodology and findings from participating in the FIGNEWS 2024 shared task on annotating news fragments on the Gaza-Israel war for bias and propaganda detection. The task aimed to refine the FIGNEWS 2024 annotation guidelines and to contribute to the creation of a comprehensive dataset to advance research in this field. Our team employed a multi-faceted approach to ensure high accuracy in data annotations. Our results highlight key challenges in detecting bias and propaganda, such as the need for more comprehensive guidelines. Our team ranked first in all tracks for propaganda annotation. For Bias, the team stood in first place for the Guidelines and IAA tracks, and in second place for the Quantity and Consistency tracks.
Bias Bluff Busters at FIGNEWS 2024 Shared Task: Developing Guidelines to Make Bias Conscious
Authors: Silvia Pareti; Jasmin Heierli; Serena Pareti; Tatiana Lando
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This paper details our participation in the FIGNEWS-2024 shared task on bias and propaganda annotation in Gaza conflict news. Our objectives were to develop robust guidelines and annotate a substantial dataset to enhance bias detection. We iteratively refined our guidelines and used examples for clarity. Key findings include the challenges in achieving high inter-annotator agreement and the importance of annotator awareness of their own biases. We also explored the integration of ChatGPT as an annotator to support consistency. This paper contributes to the field by providing detailed annotation guidelines, and offering insights into the subjectivity of bias annotation.
Ceasefire at FIGNEWS 2024 Shared Task: Automated Detection and Annotation of Media Bias Using Large Language Models
Authors: Noor Sadiah; Sara Al-Emadi; Sumaya Rahman
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: In this paper, we present our approach for FIGNEWS Subtask 1, which focuses on detecting bias in news media narratives about the Israel war on Gaza. We used a Large Language Model (LLM) and prompt engineering, using GPT-3.5 Turbo API, to create a model that automatically flags biased news media content with 99% accuracy. This approach provides Natural Language Processing (NLP) researchers with a robust and effective solution for automating bias detection in news media narratives using supervised learning algorithms. Additionally, this paper provides a detailed analysis of the labeled content, offering valuable insights into media bias in conflict reporting. Our work advances automated content analysis and enhances understanding of media bias.
Sahara Pioneers at FIGNEWS 2024 Shared Task: Data Annotation Guidelines for Propaganda Detection in News Items
Authors: Marwa Solla; Hassan Ebrahem; Alya Issa; Harmain Harmain; Abdusalam Nwesri
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: In today’s digital age, the spread of propaganda through news channels has become a pressing concern. To address this issue, the research community has organized a shared task on detecting propaganda in news posts. This paper aims to present the work carried out at the University of Tripoli for the development and implementation of data annotation guidelines by a team of five annotators. The guidelines were used to annotate 2600 news articles. Each article is labeled as “propaganda”, “Not propaganda”, “Not Applicable”, or “Not clear”. The shared task results put our efforts in the third position among 6 participating teams in the consistency track.
BiasGanda at FIGNEWS 2024 Shared Task: A Quest to Uncover Biased Views in News Coverage
Authors: Al Manar Al Wardi; Blqees Al Busaidi; Malath Al-Sibani; Hiba Salim Muhammad Al-Siyabi; Najma Al Zidjaly
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: In this study, we aimed to identify biased language in a dataset provided by the FIGNEWS 2024 committee on the Gaza-Israel war. We classified entries into seven categories: Unbiased, Biased against Palestine, Biased against Israel, Biased against Others, Biased against both Palestine and Israel, Unclear, and Not Applicable. Our team reviewed the literature to develop a codebook of terminologies and definitions. By coding each example, we sought to detect language tendencies used by media outlets when reporting on the same event. The primary finding was that most examples were classified as “Biased against Palestine,” as all examined language data used one-sided terms to describe the October 7 event. The least used category was “Not Applicable,” reserved for irrelevant examples or those lacking context. It is recommended to use neutral and balanced language when reporting volatile political news.
The CyberEquity Lab at FIGNEWS 2024 Shared Task: Annotating a Corpus of Facebook Posts to Label Bias and Propaganda in Gaza-Israel War Coverage in Five Languages
Authors: Mohammed Helal; Radi Jarrar; Mohammed Alkhanafseh; Abdallah Karakra; Ruba Awadallah
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This paper presents The_CyberEquity_Lab team’s participation in the FIGNEWS 2024 Shared Task (Zaghouani, et al., 2024). The task is to annotate a corpus of Facebook posts into bias and propaganda in covering the Gaza-Israel war. The posts represent news articles written in five different languages. The paper presents the guidelines of annotation that the team has adhered in identifying both bias and propaganda in coverage of this continuous conflict.
BSC-LANGTECH at FIGNEWS 2024 Shared Task: Exploring Semi-Automatic Bias Annotation using Frame Analysis
Authors: Valle Ruiz-Fernández; José Saiz; Aitor Gonzalez-Agirre
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This paper introduces the methodology of BSC-LANGTECH team for the FIGNEWS 2024 Shared Task on News Media Narratives. Following the bias annotation subtask, we apply the theory and methods of framing analysis to develop guidelines to annotate bias in the corpus provided by the task organizators. The manual annotation of a subset, with which a moderate IAA agreement has been achieved, is further used in Deep Learning techniques to explore automatic annotation and test the reliability of our framework.
GroningenAnnotatesGaza at the FIGNEWS 2024 Shared Task: Analyzing Bias in Conflict Narratives
Authors: Khalid Khatib; Sara Gemelli; Saskia Heisterborg; Pritha Majumdar; Gosse Minnema; Arianna Muti; Noa Solissa
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: In this paper we report the development of our annotation methodology for the shared task FIGNEWS 2024. The objective of the shared task is to look into the layers of bias in how the war on Gaza is represented in media narrative. Our methodology follows the prescriptive paradigm, in which guidelines are detailed and refined through an iterative process in which edge cases are discussed and converged. Our IAA score (Krippendorff’s 𝛼) is 0.420, highlighting the challenging and subjective nature of the task. Our results show that 52% of posts were unbiased, 42% biased against Palestine, 5% biased against Israel, and 3% biased against both. 16% were unclear or not applicable.
Sina at FigNews 2024: Multilingual Datasets Annotated with Bias and Propaganda.
Authors: Lina Duaibes; Areej Jaber; Mustafa Jarrar; Ahmad Qadi; Mais Qandeel
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: The proliferation of bias and propaganda onsocial media is an increasingly significant concern,leading to the development of techniquesfor automatic detection. This article presents amultilingual corpus of 12, 000 Facebook postsfully annotated for bias and propaganda. Thecorpus was created as part of the FigNews2024 Shared Task on News Media Narrativesfor framing the Israeli War on Gaza. It coversvarious events during the War from October7, 2023 to January 31, 2024. The corpuscomprises 12, 000 posts in five languages (Arabic,Hebrew, English, French, and Hindi), with2, 400 posts for each language. The annotationprocess involved 10 graduate students specializingin Law. The Inter-Annotator Agreement(IAA) was used to evaluate the annotationsof the corpus, with an average IAA of 80.8%for bias and 70.15% for propaganda annotations.Our team was ranked among the bestperformingteams in both Bias and Propagandasubtasks. The corpus is open-source and availableat https://sina.birzeit.edu/fada
SQUad at FIGNEWS 2024 Shared Task: Unmasking Bias in Social Media Through Data Analysis and Annotation
Authors: Asmahan Al-Mamari; Fatma Al-Farsi; Najma Zidjaly
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This paper is a part of the FIGNEWS 2024 Datathon Shared Task and it aims to investigate bias and double standards in media coverage of the Gaza-Israel 2023-2024 conflict through a comprehensive analysis of news articles. The methodology integrated both manual labeling as well as the application of a natural language processing (NLP) tool, which is the Facebook/BART-large-MNLI model. The annotation process involved categorizing the dataset based on identified biases, following a set of guidelines in which categories of bias were defined by the team. The findings revealed that most of the media texts provided for analysis included bias against Palestine, whether it was through the use of biased vocabulary or even tone. It was also found that texts written in Hebrew contained the most bias against Palestine. In addition, when comparing annotations done by AAI-1 and AAI-2, the results turned out to be very similar, which might be mainly due to the clear annotation guidelines set by the annotators themselves. Thus, we recommend the use of clear guidelines to facilitate the process of annotation by future researchers.
JusticeLeague at FIGNEWS 2024 Shared Task: Innovations in Bias Annotation
Authors: Amr Saleh; Huda Mohamed; Hager Sayed
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: In response to the evolving media representation of the Gaza-Israel conflict, this study aims to categorize news articles based on their bias towards specific entities. Our primary objective is to annotate news articles with labels that indicate their bias: “Unbiased”, “Biased against Palestine”, “Biased against Israel”, “Biased against both Palestine and Israel”, “Biased against others”, “Unclear”, or “Not Applicable”.The methodology involves a detailed annotation process where each article is carefully reviewed and labeled according to predefined guidelines. For instance, an article reporting factual events without derogatory language is labeled as “Unbiased”, while one using inflammatory language against Palestinians is marked as “Biased against Palestine”.Key findings include the identification of various degrees of bias in news articles, highlighting the importance of critical analysis in media consumption. This research contributes to the broader effort of understanding media bias and promoting unbiased journalism. Tools such as Google Drive and Google Sheets facilitated the annotation process, enabling efficient collaboration and data management among the annotators.Our work also includes comprehensive guidelines and examples to ensure consistent annotation, enhancing the reliability of the data.
Eagles at FIGNEWS 2024 Shared Task: A Context-informed Prescriptive Approach to Bias Detection in Contentious News Narratives
Authors: Amanda Chan; Mai A.Baddar; Sofien Baazaoui
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This research paper presents an in-depth examination of bias identification in media content related to the Israel-Palestine war. Focusing on the annotation guidelines and process developed by our team of researchers, the document outlines a systematic approach to discerning bias in articles. Through meticulous analysis, key indicators of bias such as emotive language, weasel words, and loaded comparisons are identified and discussed. The paper also explores the delineation between facts and opinions, emphasizing the importance of maintaining objectivity in annotation. Ethical considerations, including the handling of sensitive data and the promotion of multipartiality among annotators, are carefully addressed. The annotation guidelines also include other ethical considerations such as identifying rumors, false information, exercising prudence and selective quotations. The research paper offers insights into the annotation experience, highlighting common mistakes and providing valuable guidelines for future research in bias identification. By providing a comprehensive framework for evaluating bias in media coverage of the Israel-Palestine war, this study contributes to a deeper understanding of the complexities inherent in media discourse surrounding contentious geopolitical issues.
The Guidelines Specialists at FIGNEWS 2024 Shared Task: An annotation guideline to Unravel Bias in News Media Narratives Using a Linguistic Approach
Authors: Ghizlane Bourahouat; Samar Amer
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This article presents the participation of “The Guideline Specialists” in the FIGNEWS 2024 Shared Task, which aims to unravel bias and propaganda in news media narratives surrounding the Gaza-Israel 2023-2024 war. Leveraging innovative annotation methodologies and drawing on a diverse team of annotators, our approach focuses on meticulously annotating news articles using a linguistic approach to uncover the intricate nuances of bias. By incorporating detailed examples and drawing on related work that show how language structure represented in the use of passive voice or the use of nominalization and the choice of vocabulary carry bias, our findings provide valuable insights into the representation of the Gaza-Israel conflict across various languages and cultures. The guideline we developed detected the bias against Gaza, against Israel and others by setting keywords that are based on linguistic background tested by the AntConc concordance tool. The result was an annotation guideline that have a solid base. Through this collaborative effort, we developed a guideline that contributes to fostering a deeper understanding of media narratives during one of the most critical moments in recent history.
KSAA-CAD Shared Task: Contemporary Arabic Dictionary for Reverse Dictionary and Word Sense Disambiguation
Authors: Waad Alshammari; Amal Almazrua; Asma Al Wazrah; Rawan Almatham; Muneera Alhoshan; Abdulrahman Alosaimy
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This paper outlines the KSAA-CAD shared task, highlighting the Contemporary Arabic Language Dictionary within the scenario of developing a Reverse Dictionary (RD) system and enhancing Word Sense Disambiguation (WSD) capabilities. The first KSAA-RD (Al-Matham et al., 2023) highlighted significant gaps in the domain of RDs, which are designed to retrieve words by their meanings or definitions. This shared task comprises two tasks: RD and WSD. The RD task focuses on identifying word embeddings that most accurately match a given definition, termed a “gloss,” in Arabic. Conversely, the WSD task involves determining the specific meaning of a word in context, particularly when the word has multiple meanings. The winning team achieved the highest-ranking score of 0.0644 in RD using Electra embeddings. In this paper, we describe the methods employed by the participating teams and provide insights into the future direction of KSAA-CAD.
Cher at KSAA-CAD 2024: Compressing Words and Definitions into the Same Space for Arabic Reverse Dictionary
Authors: Pinzhen Chen; Zheng Zhao; Shun Shao
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: We present Team Cher’s submission to the ArabicNLP 2024 KSAA-CAD shared task on the reverse dictionary for Arabic—the retrieval of words using definitions as a query. Our approach is based on a multi-task learning framework that jointly learns reverse dictionary, definition generation, and reconstruction tasks. This work explores different tokenization strategies and compares retrieval performance for each embedding architecture. Evaluation using the KSAA-CAD benchmark demonstrates the effectiveness of our multi-task approach and provides insights into the reverse dictionary task for Arabic. It is worth highlighting that we achieve strong performance without using any external resources in addition to the provided training data.
MISSION at KSAA-CAD 2024: AraT5 with Arabic Reverse Dictionary
Authors: Thamer Alharbi
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: This research paper presents our approach for the KSAA-CAD 2024 competition, focusing on Arabic Reverse Dictionary (RD) task (Alshammari et al., 2024). Leveraging the functionalities of the Arabic Reverse Dictionary, our system allows users to input glosses and retrieve corresponding words. We provide all associated notebooks and developed models on GitHub and Hugging face, respectively. Our task entails working with a dataset comprising dictionary data and word embedding vectors, utilizing three different architectures of contextualized word embeddings: AraELECTRA, AraBERTv2, and camelBERT-MSA. We fine-tune the AraT5v2-base-1024 model for predicting each embedding, considering various hyperparameters for training and validation. Evaluation metrics include ranking accuracy, mean squared error (MSE), and cosine similarity. The results demonstrate the effectiveness of our approach on both development and test datasets, showcasing promising performance across different embedding types.
ASOS at KSAA-CAD 2024: One Embedding is All You Need for Your Dictionary
Authors: Serry Sibaee; Abdullah Alharbi; Samar Ahmad; Omer Nacar; Anis Koubaa; Lahouari Ghouti
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: Semantic search tasks have grown extremely fast following the advancements in large language models, including the Reverse Dictionary and Word Sense Disambiguation in Arabic. This paper describes our participation in the Contemporary Arabic Dictionary Shared Task. We propose two models that achieved first place in both tasks. We conducted comprehensive experiments on the latest five multilingual sentence transformers and the Arabic BERT model for semantic embedding extraction. We achieved a ranking score of 0.06 for the reverse dictionary task, which is double than last year’s winner. We had an accuracy score of 0.268 for the Word Sense Disambiguation task.
Baleegh at KSAA-CAD 2024: Towards Enhancing Arabic Reverse Dictionaries
Authors: Mais Alheraki; Souham Meshoul
Venue: ACL 2024 (Proceedings of the Second Arabic Natural Language Processing Conference)

Abstract: The domain of reverse dictionaries (RDs), while advancing in languages like English and Chinese, remains underdeveloped for Arabic. This study attempts to explore a data-driven approach to enhance word retrieval processes in Arabic RDs. The research focuses on the ArabicNLP 2024 Shared Task, named KSAA-CAD, which provides a dictionary dataset of 39,214 word-gloss pairs, each with a corresponding target word embedding. The proposed solution aims to surpass the baseline performance by employing SOTA deep learning models and innovative data expansion techniques. The methodology involves enriching the dataset with contextually relevant examples, training a T5 model to align the words to their glosses in the space, and evaluating the results on the shared task metrics. We find that our model is closely aligned with the baseline performance on bertseg and bertmsa targets, however does not perform well on electra target, suggesting the need for further exploration.