Gallagher at SemEval-2023 Task 5: Tackling Clickbait with Seq2Seq Models
Year: 2023

Abstract:
This paper presents the systems and approaches of the Gallagher team for the SemEval-2023 Task 5: Clickbait Spoiling. We propose a method to classify the type of spoiler (phrase, passage, multi) and a question-answering method to generate spoilers that satisfy the curiosity caused by clickbait posts. We experiment with the state-of-the-art Seq2Seq model T5. To identify the spoiler types we used a finetuned T5 classifier (Subtask 1). A mixture of T5 and Flan-T5 was used to generate the spoilers for clickbait posts (Subtask 2). Our system officially ranks first in generating phrase type spoilers in Subtask 2, and achieves the highest precision score for passage type spoilers in Subtask 1.

heading: 1 Introduction
text: The goal of SemEval-2023 Task 5 is to spoil clickbait. Clickbait posts are texts that arouse curiosity not by providing informative summaries of articles, but by purposefully teasing and leaving out key information from an article to advertise a web page’s content. The aim of this shared task is to generate short texts (spoilers) that satisfy the curiosity induced by clickbait posts and provide more informative summaries of the linked articles. Although the first subtask may seem unnecessary, our results suggest that it could be highly helpful for the main purpose of spoiler generation. As clickbait posts are getting more common every day, this task holds more importance than ever. This task is only aimed towards English clickbait posts (Fröbe et al., 2023a).
Our strategy for this shared task was to approach it as a question-answering problem. Our system uses the state-of-the-art sequence-to-sequence (Seq2Seq) model T5 (Raffel et al., 2019). We experiment with different variants of T5, some of which were previously fine-tuned for question-answering tasks. We implement a different model for each
spoiler category type due to the contrast between the different types of output sequences.
Our system achieves competitive results in both subtasks compared to the baseline from the shared task organizers and to the other teams. It has the highest precision for classifying the passage type spoilers in the spoiler classification subtask, and ranks as the top submission for phrase type spoilers in the generation subtask. However, our system struggles with generating longer spoilers such as phrase and multi type spoilers.
We submitted our system as a docker image on the TIRA platform (Fröbe et al., 2023b) to increase reproducibility. We release our code 1 and the model checkpoints 2.

heading: 2 Background

heading: 2.1 Related Work
text: A limited number of studies have directly focused on the task of clickbait spoiling. The most extensive work comes from Hagen et al. (2022), the organizers of this shared task. In their study, they approach clickbait spoiling as a question-answering and passage retrieval task. They use the questionanswering method for phrase spoilers and the passage retrieval method for passage spoilers. The passage retrieval method is a relaxed questionanswering method, where it allows the answer to be a longer sequence of text. They experiment with a variety of Transformer models. However, their work does not include spoiler generation for multi type spoilers as they acknowledge that their methods would not work well with multi type spoilers.
The work of Heiervang (2022) focused on spoiling clickbait posts that they collected from the Reddit forum “Saved You a Click”, shortened as SYAC. They experiment by fine-tuning the T5 and the UnifiedQA (Khashabi et al., 2020) models on their
1https://www.github.com/tbilgis23/ clickbait-spoiling
2https://www.huggingface.co/Tugay
1650
Reddit SYAC dataset and introduced a new method called “Title Answering”, which achieves an overall BLEU-4 score of 0.1746 on the test set.

heading: 2.2 Data
text: The data provided is split into three sets; train, validation, and test. There are 3200 posts in the train set, 800 in the validation set, and 1000 in the test set, for a total of 5000 clickbait posts. These clickbait posts are collected from five different social media accounts and were manually spoiled (Hagen et al., 2022). The test set is hidden from the participants and only used by the organizers to evaluate the systems. Spoilers are categorized into 3 types: short phrase spoilers, longer passage spoilers, and multiple non-consecutive pieces of text. Table 1 provides the distribution of the spoiler types in the train and validation set.
The data for the shared task is provided in JSON format, where each clickbait post is a JSON object and has several fields describing the clickbait post. Descriptions of those fields can be found on the task description website3. The tags field contains the output for the spoiler classification subtask and the spoiler field contains the output for the spoiler generation subtask. These output fields are not provided in the test set.

heading: 3 System Overview
text: We used the T5 model (Raffel et al., 2019) for both the spoiler classification and spoiler generation tasks. T5 is a state-of-the-art Seq2Seq model and has shown successful performance for a variety of NLP tasks. It is a text-to-text framework, where the input is fed as a text and a target text is generated for the output.
Although T5 works for a variety of tasks out-ofthe-box, as it was trained on a large corpus with a mixture of supervised and unsupervised tasks, we did additional training to fine-tune it for the specific needs of the clickbait tasks. We also experimented
3https://pan.webis.de/semeval23/pan23-web/ clickbait-challenge.html
with variants of T5 and models that were fine-tuned for specific tasks such as question answering.

heading: 3.1 Subtask 1: Spoiler Classification
text: The task of spoiler classification is a first step towards the main goal of spoiler generation, allowing different approaches to be used in the generation of different spoiler types. We experimented with both T5 and LongT5 (Guo et al., 2021), as some of the passages exceeded the maximum sequence length of 512 and thus could potentially lose information needed to classify the spoiler type. We take advantage of the larger checkpoints of T5 and LongT5 with 770 Billion parameters (T5LARGE and LongT5LARGE).
For both models, we follow the approach of Hagen et al. (2022) of feeding the clickbait post and the article of the post as the input. Additionally, we add to each input a prefix of multi-class classification:, as T5 performs well with task prefixes (Raffel et al., 2019), and a suffix of 
, the string ending token. Figure 1 illustrates our approach for this subtask.
A potential challenge in this classification task is data imbalance. As seen in Table 1, the number of multi type spoilers is less than half that of phrase or passage type spoilers. However, the multi type spoilers usually contain a quantitative element in the clickbait post and in the article (as seen in Figure 1), which potentially could make it easier to classify the multi type spoilers.

heading: 3.2 Subtask 2: Spoiler Generation
text: Spoiler generation was the main focus of the shared task. We approach this as a question-answering
problem, similar to the approach of Hagen et al. (2022). Each type of spoiler differs in its structure, therefore we implemented different models for each. Along with the T5 model, we explore several other scaled and fine-tuned T5 models for each spoiler type.
For all the spoiler types, we give the clickbait post and the article as the input, in a format similar to the SQUAD dataset (Rajpurkar et al., 2016), where the clickbait post has the question: prefix, the article has the context: prefix, and 
, the string ending token, follows the article. We give the corresponding spoilers as the output. Figure 2 illustrates our approach to this subtask.

heading: 3.2.1 Phrase Spoiler Generation
text: Phrase type spoilers are spoilers for clickbait posts that can be spoiled in either a single word or a couple of words that occur together in the article. Hence, we approach it as an extractive questionanswering problem.
We experimented with fine-tuning the T5BASE , T5LARGE , UnifiedQALARGE , Flan-T5LARGE (Chung et al., 2022), and LongT5LARGE models. UnifiedQA is a T5 model that has been trained on various question-answering formats, combined into a single model.
The input is fed in the SQUAD format for T5, Flan-T5, and LongT5 models without any additional pre-processing of the data. However, for the UnifiedQA model, we follow the recommended pre-processing step for the input by giving no prefixes and separating the clickbait post and the article with the \n separator (Khashabi et al., 2020).

heading: 3.2.2 Passage Spoiler Generation
text: Passage type spoilers are slightly more complex than phrase spoilers, as the spoilers are full sen-
tences rather than words. We follow a similar approach to phrase spoiler generation and treat it as an extractive question-answering problem. However, we use a separate model from the model of the previous section since fine-tuning with both phrase and passage spoilers could confuse the model.
We experimented with fine-tuning the T5LARGE , UnifiedQALARGE , Flan-T5LARGE , and LongT5LARGE models. The input was given in the SQUAD format for T5, Flan-T5, and LongT5. For the UnifiedQA model, the special pre-processing described in Section 3.2.1 was applied to the input.

heading: 3.2.3 Multi Spoiler Generation
text: Multi type spoilers are spoilers that spoil multiple non-consecutive texts from the clickbait post’s article. It is more complex than the phrase or passage spoiler generation as it requires the model to pay attention to multiple parts of the article rather than a single part. The output should be the first five spoilers if there are more than 5 spoilers in the context, as the organizers assert that if 5 can be found correctly, others would be too. We approach this problem as a question-answering problem mixed with summarization for the multi-span extraction. This was the most challenging part of this shared task, as there hasn’t been a lot of previous work done in this area and the data for multi spoiler type was limited.
We only experimented with T5LARGE and UnifiedQALARGE models for the multi spoiler generation due to time and resource constraints. The input format was the same as other spoiler types: SQUAD format for the T5 model and the special preprocessing described in Section 3.2.1 was applied to the input of UnifiedQA.

heading: 4 Experimental Setup
text: We used the provided training data only to train our models, and the provided validation split only to evaluate the models. We did not use the validation split to further train our models. No preprocessing is done to the shared task data other than adding the subtask-specific prefixes mentioned in Sections 3.1 and 3.2. We used the Huggingface Transformers library (version 4.6.0) to train our models.
We used the AdamW optimizer with an initial learning rate of 5 × e−5 and with a weight decay of 0.01 during the training for all models. For the spoiler classification subtask, we trained our models for 10 epochs and with a batch size of 8,
and for the spoiler generation subtask, we trained our models for 5 epochs and with a batch size of 4. The maximum token length is 512 for all models except for the experiments with LongT5. We use a token length of 1024 with the LongT5 experiments.
After each epoch, we evaluated the output generated on the validation data. We used the sklearn.metrics.balanced_accuracy_score from the scikit-learn library (version 1.2.1) to evaluate the balanced accuracy score for the spoiler classification subtask and to choose the best-performing model. For the spoiler generation task, we evaluated the output by the BLEU-4 score using the Evaluate library (version 0.4.0) from Huggingface to choose the best-performing spoiler generation model for each spoiler category. We saved the model checkpoints after each epoch. We ran these experiments on LambdaLabs cloud instances, with an NVIDIA A100 GPU.

heading: 5 Results

heading: 5.1 Results on Validation
text: Table 2 shows our results on the validation split for the spoiler classification subtask. We only report the epoch that got the highest balanced accuracy score. The table also contains the baseline balanced accuracy score provided by the task organizers to show how our model compares. As can be seen, T5LARGE outperforms LongT5LARGE and gets a score that beats the baseline score of 0.734 (Fröbe et al., 2023a).
Table 3 shows our results on the validation split for the spoiler generation task. The models with the best BLEU-4 scores are highlighted for each spoiler category. We report the highest BLEU-4 score achieved in an epoch for each model. T5LARGE achieves the highest score for the phrase and multi spoiler types, while Flan-T5 achieves the bestperforming results for the passage spoilers. Across all models, the BLEU-4 scores are the highest for the phrase type spoilers, followed by the passage type spoilers, and lastly the multi type spoilers.
The results on the validation set guided our selection of models for our final system submission. We chose the fine-tuned T5LARGE as our model for the spoiler classification task since it performed the best, with the checkpoint that achieved the reported result. Similarly, we chose the T5LARGE for the phrase and multi type spoiler generation, and Flan-T5 for the passage spoiler generation.

heading: 5.2 Results on Test
text: The results on the test set come from our software submission on TIRA (Fröbe et al., 2023b). Table 4 shows the detailed results for the spoiler classification subtask. Our model achieves a balanced accuracy of 0.72 on the test set, which is lower than the score achieved on the validation set and than the baseline score provided for the evaluation set. However, our model gets the highest precision score for the passage type spoilers among all the submissions.
Detailed results for the spoiler generation task are provided in Table 5. Among all categories of spoilers, we achieve a BLEU-4 score of 0.41. The BLEU-4 score among all spoiler categories is
higher than the baseline score of 0.382 (Fröbe et al., 2023a). Our system achieves the highest scores for phrase spoilers among all other submissions. Similar to the results of the evaluation test, our system performs the best in phrase type spoilers, followed by the passage type spoilers, and finally the multi type spoilers.

heading: 5.3 Error Analysis
text: We investigated a sample of wrong predictions for Subtask 1 and wrongfully generated spoilers for Subtask 2 on the validation set.
For Subtask 1, the most common error is between classifying phrase and passage spoilers. The clickbait posts and articles for passage and phrase type spoilers can be very close in structure, especially since the data contains some short passage type spoilers and some longer phrase type spoilers that are similar to each other. Thus, our system is confused in those instances. One other common error is in the prediction of multi type spoilers. Our system reliably identifies the multi spoilers that have quantitative elements or indicators in the clickbait post or in the article but commonly misclassifies the multi posts and articles that don’t have quantitative elements or indicators.
The most common error pattern for Subtask 2 is incomplete spoilers for phrase and multi spoilers. For passage spoilers, our system often identifies where the spoiler is in the article but struggles with extracting the whole passage as the spoiler. Similarly, our system struggles with the same issue for multi type spoiler, but additionally, it often has a difficulty in generating spoilers that are spread out across the text compared to the multi spoilers that are close together.

heading: 6 Conclusion
text: We present a question-answering framework to spoil clickbait posts. We show that the state-ofthe-art Seq2Seq model T5 and its variants perform well in classifying the spoiler type and in generat-
ing the spoilers when fine-tuned on clickbait posts. We found that spoiling clickbait posts is easier for phrase type spoilers and gets progressively more difficult when the spoiler is longer, especially if it requires multi-span extraction like the multi type spoilers.
In future work, we plan to improve the performance of the spoiler generation for passage and multi type spoilers. We observed that a more complicated approach is needed for these longer spoiler types rather than a simple extractive questionanswering method.

heading: Future Work
text: In future work, we plan to improve the performance of the spoiler generation for passage and multi type spoilers. We observed that a more complicated approach is needed for these longer spoiler types rather than a simple extractive questionanswering method