GenAI4ELab/papercli-papers huggingface dataset:
QFNU_CS at SemEval-2024 Task 3: A Hybrid Pre-trained Model based Approach for Multimodal Emotion-Cause Pair Extraction Task
Authors: Zining Wang; Yanchao Zhao; Guanghui Han; Yang Song
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This article presents the solution of Qufu Normal University for the Multimodal Sentiment Cause Analysis competition in SemEval2024 Task 3.The competition aims to extract emotion-cause pairs from dialogues containing text, audio, and video modalities. To cope with this task, we employ a hybrid pre-train model based approach. Specifically, we first extract and fusion features from dialogues based on BERT, BiLSTM, openSMILE and C3D. Then, we adopt BiLSTM and Transformer to extract the candidate emotion-cause pairs. Finally, we design a filter to identify the correct emotion-cause pairs. The evaluation results show that, we achieve a weighted average F1 score of 0.1786 and an F1 score of 0.1882 on CodaLab.
NewbieML at SemEval-2024 Task 8: Ensemble Approach for Multidomain Machine-Generated Text Detection
Authors: Bao Tran; Nhi Tran
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: Large Language Models (LLMs) are becoming popular and easily accessible, leading to a large growth of machine-generated content over various channels. Along with this popularity, the potential misuse is also a challenge for us. In this paper, we use SemEval 2024 task A monolingual dataset with comparative study between some machine learning model with feature extraction and develop an ensemble method for our system. Our system achieved 84.31% accuracy score in the test set, ranked 36th of 137 participants. Our code is available at: https://github.com/baoivy/SemEval-Task8
Hidetsune at SemEval-2024 Task 3: A Simple Textual Approach to Emotion Classification and Emotion Cause Analysis in Conversations Using Machine Learning and Next Sentence Prediction
Authors: Hidetsune Takahashi
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: In this system paper for SemEval-2024 Task3 subtask 2, I present my simple textual approach to emotion classification and emotion cause analysis in conversations using machine learning and next sentence prediction. I train a SpaCy model for emotion classification and use next sentence prediction with BERT for emotion cause analysis. While speaker names and audio-visual clips are given in addition to text of the conversations, my approach uses textual data only to test my methodology to combine machine learning with next sentence prediction. This paper reveals both strengths and weaknesses of my trial, suggesting a direction of future studies to improve my introductory solution.
CLTeam1 at SemEval-2024 Task 10: Large Language Model based ensemble for Emotion Detection in Hinglish
Authors: Ankit Vaidya; Aditya Gokhale; Arnav Desai; Ishaan Shukla; Sheetal Sonawane
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This paper outlines our approach for the ERC subtask of the SemEval 2024 EdiREF Shared Task. In this sub-task, an emotion had to be assigned to an utterance which was the part of a dialogue. The utterance had to be classified into one of the following classes- disgust, contempt, anger, neutral, joy, sadness, fear, surprise. Our proposed system makes use of an ensemble of language specific RoBERTA and BERT models to tackle the problem. A weighted F1-score of 44% was achieved by our system in this task. We conducted comprehensive ablations and suggested directions of future work. Our codebase is available publicly.
Hidetsune at SemEval-2024 Task 4: An Application of Machine Learning to Multilingual Propagandistic Memes Identification Using Machine Translation
Authors: Hidetsune Takahashi
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: In this system paper for SemEval-2024 Task4 subtask 2b, I present my approach to identifying propagandistic memes in multiple languages. I firstly establish a baseline for English and then implement the model into other languages (Bulgarian, North Macedonian and Arabic) by using machine translation. Data from other subtasks (subtask 1, subtask 2a) are also used in addition to data for this subtask, and additional data from Kaggle are concatenated to these in order to enhance the model. The results show a high reliability of my English baseline and a room for improvement of its implementation.
Hidetsune at SemEval-2024 Task 10: An English Based Approach to Emotion Recognition in Hindi-English code-mixed Conversations Using Machine Learning and Machine Translation
Authors: Hidetsune Takahashi
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: In this system paper for SemEval-2024 Task10 subtask 1 (ERC), I present my approach to recognizing emotions in Hindi-English codemixed conversations. I train a SpaCy model with English translated data and classify emotions behind Hindi-English code-mixed utterances by using the model and translating them into English. I use machine translation to translate all the data in Hindi-English mixed language into English due to an easy access to existing data for emotion recognition in English. Some additional data in English are used to enhance my model. This English based approach demonstrates a fundamental possibility and potential of simplifying code-mixed language into one major language for emotion recognition.
All-Mpnet at SemEval-2024 Task 1: Application of Mpnet for Evaluating Semantic Textual Relatedness
Authors: Marco Siino
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: In this study, we tackle the task of automatically discerning the level of semantic relatedness between pairs of sentences. Specifically, Task 1 at SemEval-2024 involves predicting the Semantic Textual Relatedness (STR) of sentence pairs. Participants are tasked with ranking sentence pairs based on their proximity in meaning, quantified by their degree of semantic relatedness, across 14 different languages. Each sentence pair is assigned manually determined relatedness scores ranging from 0 (indicating complete lack of relation) to 1 (denoting maximum relatedness). In our submitted approach on the official test set, focusing on Task 1 (a supervised task in English and Spanish), we achieve a Spearman rank correlation coefficient of 0.808 for the English language and 0.611 for the Spanish language.
0x.Yuan at SemEval-2024 Task 5: Enhancing Legal Argument Reasoning with Structured Prompts
Authors: Yu-an Lu; Hung-yu Kao
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: The intersection of legal reasoning and Natural Language Processing (NLP) technologies, particularly Large Language Models (LLMs), offers groundbreaking potential for augmenting human capabilities in the legal domain. This paper presents our approach and findings from participating in SemEval-2024 Task 5, focusing on the effect of argument reasoning in civil procedures using legal reasoning prompts. We investigated the impact of structured legal reasoning methodologies, including TREACC, IRAC, IRAAC, and MIRAC, on guiding LLMs to analyze and evaluate legal arguments systematically. Our experimental setup involved crafting specific prompts based on these methodologies to instruct the LLM to dissect and scrutinize legal cases, aiming to discern the cogency of argumentative solutions within a zero-shot learning framework. The performance of our approach, as measured by F1 score and accuracy, demonstrated the efficacy of integrating structured legal reasoning into LLMs for legal analysis. The findings underscore the promise of LLMs, when equipped with legal reasoning prompts, in enhancing their ability to process and reason through complex legal texts, thus contributing to the broader application of AI in legal studies and practice.
Groningen team D at SemEval-2024 Task 8: Exploring data generation and a combined model for fine-tuning LLMs for Multidomain Machine-Generated Text Detection
Authors: Thijs Brekhof; Xuanyi Liu; Joris Ruitenbeek; Niels Top; Yuwen Zhou
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: In this system description, we describe our process and the systems that we created for the subtasks A monolingual, A multilingual, and B forthe SemEval-2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box MachineGenerated Text Detection. This shared task aimsat detecting and differentiating between machinegenerated text and human-written text. SubtaskA is focused on detecting if a text is machinegenerated or human-written both in a monolingualand a multilingual setting. Subtask B is also focused on detecting if a text is human-written ormachine-generated, though it takes it one step further by also requiring the detection of the correct language model used for generating the text.For the monolingual aspects of this task, our approach is centered around fine-tuning a debertav3-large LM. For the multilingual setting, we created an ensemble model utilizing different monolingual models and a language identification toolto classify each text. We also experiment with thegeneration of extra training data. Our results showthat the generation of extra data aids our modelsand leads to an increase in accuracy.
Kathlalu at SemEval-2024 Task 8: A Comparative Analysis of Binary Classification Methods for Distinguishing Between Human and Machine-generated Text
Authors: Lujia Cao; Ece Lara Kilic; Katharina Will
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This paper investigates two methods for constructing a binary classifier to distinguish between human-generated and machine-generated text. The main emphasis is on a straightforward approach based on Zipf’s law, which, despite its simplicity, achieves a moderate level of performance. Additionally, the paper briefly discusses experimentation with the utilization of unigram word counts.
Team Unibuc - NLP at SemEval-2024 Task 8: Transformer and Hybrid Deep Learning Based Models for Machine-Generated Text Detection
Authors: Teodor-george Marchitan; Claudiu Creanga; Liviu P. Dinu
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This paper describes the approach of the UniBuc - NLP team in tackling the SemEval 2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection. We explored transformer-based and hybrid deep learning architectures. For subtask B, our transformer-based model achieved a strong second-place out of 77 teams with an accuracy of 86.95%, demonstrating the architecture’s suitability for this task. However, our models showed overfitting in subtask A which could potentially be fixed with less fine-tunning and increasing maximum sequence length. For subtask C (token-level classification), our hybrid model overfit during training, hindering its ability to detect transitions between human and machine-generated text.
LinguisTech at SemEval-2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversation
Authors: Mihaela Alexandru; Călina Ciocoiu; Ioana Măniga; Octavian Ungureanu; Daniela Gîfu; Diana Trandăbăț
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: The “Emotion Discovery and Reasoning Its Flip in Conversation” task at the SemEval 2024 competition focuses on the automatic recognition of emotion flips, triggered within multi-party textual conversations. This paper proposes a novel approach that draws a parallel between a mixed strategy and a comparative strategy, contrasting a Rule-Based Function with Named Entity Recognition (NER)—an approach that shows promise in understanding speaker-specific emotional dynamics. Furthermore, this method surpasses the performance of both DistilBERT and RoBERTa models, demonstrating competitive effectiveness in detecting emotion flips triggered in multi-party textual conversations, achieving a 70% F1-score. This system was ranked 6th in the SemEval 2024 competition for Subtask 3.
Text Mining at SemEval-2024 Task 1: Evaluating Semantic Textual Relatedness in Low-resource Languages using Various Embedding Methods and Machine Learning Regression Models
Authors: Ron Keinan
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: In this paper, I describe my submission to the SemEval-2024 contest. I tackled subtask 1 - “Semantic Textual Relatedness for African and Asian Languages”. To find the semantic relatedness of sentence pairs, I tackled this task by creating models for nine different languages. I then vectorized the text data using a variety of embedding techniques including doc2vec, tf-idf, Sentence-Transformers, Bert, Roberta, and more, and used 11 traditional machine learning techniques of the regression type for analysis and evaluation.
USMBA-NLP at SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials using Bert
Authors: Anass Fahfouh; Abdessamad Benlahbib; Jamal Riffi; Hamid Tairi
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This paper presents the application of BERT inSemEval 2024 Task 2, Safe Biomedical Natu-ral Language Inference for Clinical Trials. Themain objectives of this task were: First, to in-vestigate the consistency of BERT in its rep-resentation of semantic phenomena necessaryfor complex inference in clinical NLI settings.Second, to investigate the ability of BERT toperform faithful reasoning, i.e., make correctpredictions for the correct reasons. The submit-ted model is fine-tuned on the NLI4CT dataset,which is enhanced with a novel contrast set,using binary cross entropy loss.
CRCL at SemEval-2024 Task 2: Simple prompt optimizations
Authors: Clement Brutti-mairesse; Loic Verlingue
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: We present a baseline for the SemEval 2024 task 2 challenge, whose objective is to ascertain the inference relationship between pairs of clinical trial report sections and statements.We apply prompt optimization techniques with LLM Instruct models provided as a Language Model-as-a-Service (LMaaS).We observed, in line with recent findings, that synthetic CoT prompts significantly enhance manually crafted ones.The source code is available at this GitHub repository https://github.com/ClementBM-CLB/semeval-2024
SuteAlbastre at SemEval-2024 Task 4: Predicting Propaganda Techniques in Multilingual Memes using Joint Text and Vision Transformers
Authors: Ion Anghelina; Gabriel Buță; Alexandru Enache
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: The main goal of this year’s SemEval Task 4 isdetecting the presence of persuasion techniquesin various meme formats. While Subtask 1targets text-only posts, Subtask 2, subsectionsa and b tackle posts containing both imagesand captions. The first 2 subtasks consist ofmulti-class and multi-label classifications, inthe context of a hierarchical taxonomy of 22different persuasion techniques.This paper proposes a solution for persuasiondetection in both these scenarios and for vari-ous languages of the caption text. Our team’smain approach consists of a Multimodal Learn-ing Neural Network architecture, having Tex-tual and Vision Transformers as its backbone.The models that we have experimented with in-clude EfficientNet and ViT as visual encodersand BERT and GPT2 as textual encoders.
RFBES at SemEval-2024 Task 8: Investigating Syntactic and Semantic Features for Distinguishing AI-Generated and Human-Written Texts
Authors: Mohammad Heydari Rad; Farhan Farsi; Shayan Bali; Romina Etezadi; Mehrnoush Shamsfard
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: Nowadays, the usage of Large Language Models (LLMs) has increased, and LLMs have been used to generate texts in different languages and for different tasks. Additionally, due to the participation of remarkable companies such as Google and OpenAI, LLMs are now more accessible, and people can easily use them. However, an important issue is how we can detect AI-generated texts from human-written ones. In this article, we have investigated the problem of AI-generated text detection from two different aspects: semantics and syntax. Finally, we presented an AI model that can distinguish AI-generated texts from human-written ones with high accuracy on both multilingual and monolingual tasks using the M4 dataset. According to our results, using a semantic approach would be more helpful for detection. However, there is a lot of room for improvement in the syntactic approach, and it would be a good approach for future work.
BAMBAS at SemEval-2024 Task 4: How far can we get without looking at hierarchies?
Authors: Arthur Vasconcelos; Luiz Felipe De Melo; Eduardo Goncalves; Eduardo Bezerra; Aline Paes; Alexandre Plastino
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This paper describes the BAMBAS team’s participation in SemEval-2024 Task 4 Subtask 1, which focused on the multilabel classification of persuasion techniques in the textual content of Internet memes. We explored a lightweight approach that does not consider the hierarchy of labels. First, we get the text embeddings leveraging the multilingual tweets-based language model, Bernice. Next, we use those embeddings to train a separate binary classifier for each label, adopting independent oversampling strategies in each model in a binary-relevance style. We tested our approach over the English dataset, exceeding the baseline by 21 percentage points, while ranking in 23th in terms of hierarchical F1 and 11st in terms of hierarchical recall.
Team QUST at SemEval-2024 Task 8: A Comprehensive Study of Monolingual and Multilingual Approaches for Detecting AI-generated Text
Authors: Xiaoman Xu; Xiangrun Li; Taihang Wang; Jianxiang Tian; Ye Jiang
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This paper presents the participation of team QUST in Task 8 SemEval 2024. we first performed data augmentation and cleaning on the dataset to enhance model training efficiency and accuracy. In the monolingual task, we evaluated traditional deep-learning methods, multiscale positive-unlabeled framework (MPU), fine-tuning, adapters and ensemble methods. Then, we selected the top-performing models based on their accuracy from the monolingual models and evaluated them in subtasks A and B. The final model construction employed a stacking ensemble that combined fine-tuning with MPU. Our system achieved 6th (scored 6th in terms of accuracy, officially ranked 13th in order) place in the official test set in multilingual settings of subtask A. We release our system code at:https://github.com/warmth27/SemEval2024_QUST
YNU-HPCC at SemEval-2024 Task 9: Using Pre-trained Language Models with LoRA for Multiple-choice Answering Tasks
Authors: Jie Wang; Jin Wang; Xuejie Zhang
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This study describes the model built in Task 9: brainteaser in the SemEval-2024 competition, which is a multiple-choice task. As active participants in Task 9, our system strategically employs the decoding-enhanced BERT (DeBERTa) architecture enriched with disentangled attention mechanisms. Additionally, we fine-tuned our model using low-rank adaptation (LoRA) to optimize its performance further. Moreover, we integrate focal loss into our framework to address label imbalance issues. The systematic integration of these techniques has resulted in outstanding performance metrics. Upon evaluation using the provided test dataset, our system showcases commendable results, with a remarkable accuracy score of 0.9 for subtask 1, positioning us fifth among all participants. Similarly, for subtask 2, our system exhibits a substantial accuracy rate of 0.781, securing a commendable seventh-place ranking. The code for this paper is published at: https://github.com/123yunnandaxue/Semveal-2024_task9.
Team jelarson at SemEval 2024 Task 8: Predicting Boundary Line Between Human and Machine Generated Text
Authors: Joseph Larson; Francis Tyers
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: In this paper, we handle the task of building a system that, given a document written first by a human and then finished by an LLM, the system must determine the transition word i.e. where the machine begins to write. We built a system by examining the data for textual anomalies and combining a method of heuristic approaches with a linear regression model based on the text length of each document.
HU at SemEval-2024 Task 8A: Can Contrastive Learning Learn Embeddings to Detect Machine-Generated Text?
Authors: Shubhashis Roy Dipta; Sadat Shahriar
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This paper describes our system developed for SemEval-2024 Task 8, “Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection.” Machine-generated texts have been one of the main concerns due to the use of large language models (LLM) in fake text generation, phishing, cheating in exams, or even plagiarizing copyright materials. A lot of systems have been developed to detect machine-generated text. Nonetheless, the majority of these systems rely on the text-generating model. This limitation is impractical in real-world scenarios, as it’s often impossible to know which specific model the user has used for text generation. In this work, we propose a single model based on contrastive learning, which uses ~40% of the baseline’s parameters (149M vs. 355M) but shows a comparable performance on the test dataset (21st out of 137 participants). Our key finding is that even without an ensemble of multiple models, a single base model can have comparable performance with the help of data augmentation and contrastive learning.
Team AT at SemEval-2024 Task 8: Machine-Generated Text Detection with Semantic Embeddings
Authors: Yuchen Wei
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This study investigates the detection of machine-generated text using several semantic embedding techniques, a critical issue in the era of advanced language models. Different methodologies were examined: GloVe embeddings, N-gram embedding models, Sentence BERT, and a concatenated embedding approach, against a fine-tuned RoBERTa baseline. The research was conducted within the framework of SemEval-2024 Task 8, encompassing tasks for binary and multi-class classification of machine-generated text.
JN666 at SemEval-2024 Task 7: NumEval: Numeral-Aware Language Understanding and Generation
Authors: Xinyi Liu; Xintong Liu; Hengyang Lu
Venue: NAACL 2024 (Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024))

Abstract: This paper is submitted for SemEval-2027 task 7: Enhancing the Model’s Understanding and Generation of Numerical Values. The dataset for this task is NQuAD, which requires us to select the most suitable option number from four numerical options to fill in the blank in a news article based on the context. Based on the BertForMultipleChoice model, we proposed two new models, MC BERT and SSC BERT, and improved the model’s numerical understanding ability by pre-training the model on numerical comparison tasks. Ultimately, our best-performing model achieved an accuracy rate of 79.40%, which is 9.45% higher than the accuracy rate of NEMo.