Francis Wilde at SemEval-2023 Task 5: Clickbait Spoiler Type Identification with Transformers
Year: 2023

Abstract:
Clickbait is the text or a thumbnail image that entices the user to click the accompanying link. Clickbait posts hide the critical elements of the article and reveal partial information in the title, which arouses sufficient curiosity and motivates the user to click the link. In this work, we identify the kind of spoiler given a clickbait title. We formulate this as a text classification problem. We finetune pretrained transformer models on the title of the post and build models for the clickbait-spoiler classification. We achieve a balanced accuracy of 0.70 which is close to the baseline.

heading: 1 Introduction
text: Clickbait-spoiling (Hagen et al., 2022) aims to counteract the manipulative tactics used by clickbait creators and to help people avoid wasting their time on low-quality content. This is achieved by revealing the "suspense" that has been deliberately created in the title by the authors of the post. Without clickbait spoiling, the reader has to spend considerable time to satisfy his curiosity by finding the content which was promised in the title. By clickbait-spoiling, the user can easily skim through the key content without wasting valuable time.
The Clickbait Spoiling task at SemEval 2023 (Fröbe et al., 2023a) consists of two subtasks, Spoiler Type Classification and Spoiler Generation.
• Spoiler Type Classification: In this task, the goal is to classify a given clickbait post into one of the three kinds of spoiler types i.e phrase, passage or multi depending on the kind of text required to spoil the clickbait.
• Spoiler Generation: In this task, the goal is to generate the actual spoiler for the clickbait post.
Both tasks are presented in the English language. We attempted the first task which is Spoiler Type
Classification. We use pre-trained transformer models and finetune them on the postText to train a spoiler-type prediction model. We also experiment with pretrained transformer representations and train classical ML models on these representations for classifying the spoiler type. Our final model, a vanilla transformer model finetuned on this dataset, achieves a balanced accuracy metric of 0.7 which is close to the baseline.

heading: 2 Background
text: In the task for Clickbait Spoiler Type classification, the input is a tweet along with its associated metadata. Table 1 lists the various fields available from the dataset.
Table 2 lists few examples of clickbaits and their spoiler types.
We formulate the problem of spoiler type identification as a text classification task. We hypothesize that a human can easily predict the spoiler type of a clickbait by looking at the post of the clickbait i.e in our case, it is the postText associated with the tweet. We adopt the method followed by (Indurthi et al., 2020) for detecting the clickbait intensity where the authors use only the postText field to predict the intensity of the clickbait. We also limit ourselves to using only the postText field in the dataset.
We train models to predict the spoiler type given the postText of the tweet. We trained models with the combination of postText and targetParagraphs but they performed worse compared to the postText alone. Hence we settled to using only the postText for the task.

heading: 3 Transformers
text: Transfer learning is a popular approach in Natural Language Processing that involves leveraging knowledge gained from one task to improve performance on another related task. The transformer architecture introduced by (Vaswani et al., 2017)
1890
has become the backbone of many state-of-the-art NLP models. Transformers employ a self-attention mechanism to process input sequences and capture the relationships between different parts of the sequence. This architecture has proven highly effective in capturing complex patterns in text data, enabling models to achieve impressive performance on various NLP tasks (Wang et al., 2019b) and (Wang et al., 2019a). Pretrained models such as BERT (Devlin et al., 2018), DistilBERT (Sanh et al., 2019), RoBERTa (Liu et al., 2019), and DistilRoBERTa were trained on massive amounts of text data. They can be fine-tuned on specific tasks with relatively little additional data, resulting in significant performance gains. DistilBERT, a smaller and faster variant of BERT, has shown comparable performance on many tasks. In contrast, RoBERTa and DistilRoBERTa, trained with improved techniques, have achieved even higher performance on several benchmarks. These pre-trained transformer models have revolutionized the field of NLP and enabled researchers to achieve state-of-the-art results on various tasks.

heading: 4 System Overview
text: We follow the standard way of funetuning transformer models where pretrained transformer models are finetuned. In this method, all the layers of the transformer model and the weights in the classification layer are jointly learned during the finetuning process. We experiment with 4 transformer models i.e BERT, RoBERTa, DistilBERT and DistilRoBERTa. We use only the postText field to do the fine-tuning. We do not do any specific preprocessing. We finetune each model for 3 epochs. We use a learning rate of 2e-5. Default values provided by the huggingface library are used for the rest of the configurable parameters.

heading: 5 Experimental Setup
text: The organizers of the task have compiled a dataset of 5000 clickbaits. Out of this 1000 samples are re-
served as test data and their ground truths were kept hidden from the participants. The remaining 4000 samples were released for training the models. Out of these 4000 samples, we used 3200 samples for training our models and kept the other 800 samples for development/validation.
We use the huggingface transformers library 1 for training our models. Our metric for evaluation is the accuracy and the F1-score. Our training and evaluation was performed on Google Colab 2.
The clickbait evaluation happens on a platform called TIRA (Fröbe et al., 2023b) 3. The participants can either submit a docker image or upload code and models on a virtual machine hosted on tira, which can read the test data and produce the results. The TIRA platform is used so that the test data is kept confidential and to prevent participants overfitting to the test data. In addition, the participants can also upload the results file on the web and get the results evaluated.

heading: 6 Results
text: Table 4 show the performance of the models on the development/validation set of 800 tweets. We can observe that RoBERTa model has performed much better in terms of the Accuracy, Precision, Recall and F1 compared to all other models. RoBERTa achieves an accuracy of 0.72 and an F1 of 0.71 which is very close to the baseline performance. Hence we submitted the results produced by this model on the official test set for the final official evaluation. Table 5 shows the result of the model on the official test set. We can see that on the test set, RoBERTa model produces an accuracy of 0.70 and comparable F1 scores for all the classes.

heading: 7 Conclusion
text: The approach followed by us is a very simple method by just using the postTitle. In the future we can experiment with more fields and leverage semi-supervised learning techniques and improve the classification performance. We can explore more transformer models also.
1https://github.com/huggingface/transformers 2https://colab.google.com 3https://tira.io

heading: Future Work
text: In the future we can experiment with more fields and leverage semi-supervised learning techniques and improve the classification performance. We can explore more transformer models also. 1https://github. com/huggingface/transformers 2https://colab. google. com 3https://tira. io