TohokuNLP at SemEval-2023 Task 5: Clickbait Spoiling via Simple Seq2seq Generation and Ensembling
Year: 2023

Abstract:
This paper describes our system submitted to SemEval-2023 Task 5: Clickbait Spoiling. We work on spoiler generation of the subtask 2 and develop a system which comprises two parts: 1) simple seq2seq spoiler generation and 2) post-hoc model ensembling. Using this simple method, we address the challenge of generating multipart spoiler. In the test set, our submitted system1 outperformed the baseline by a large margin (approximately 10 points above on the BLEU score) for mixed types of spoilers. We also found that our system successfully handled the challenge of the multipart spoiler, confirming the effectiveness of our approach.

heading: 1 Introduction
text: Clickbait is a type of text on social media that is specifically designed to exploit users’ curiosity and lure them into clicking on a linked webpage. Due to the deceptive and misleading nature of the content often presented, clickbait is generally considered harmful (Hagen et al., 2022). Thus addressing the clickbait issue is a crucial concern for social media users.
Subtask 2 of SemEval-2023 Task 5 aims to generate spoilers of clickbait based on posted text and the linked webpage (Fröbe et al., 2023a). Hagen et al. (2022) proposed a method for generating spoilers via encoder-based question answering approach. However, their method has limitations in handling a specific type of spoiler where the answer is spread across several sections of the target article.
To tackle this problem, we develop a simple yet effective sequence-to-sequence (seq2seq) approach for the subtask 2: spoiler generation. In our
†equal contributions ‡Work done while at RIKEN AIP and Tohoku University. 1The docker image of our final submission on TIRA(Fröbe et al., 2023b) is registry.webis.de/code-research/ tira/tira-user-john-king/finalsub:0.0.4 and the implementation is available at https://github.com/ cl-tohoku/semeval2023-task5-seq2seq-ensemble.
approach, the target spoiler is directly generated from the seq2seq model, rather than predicting its spans in the context. This enables us to handle the challenge of multipart spoiler with ease without implementing more complex approaches such as multiple-span extraction (Segal et al., 2020). Experimental results demonstrate that our approach successfully addresses the multipart spoilers, leading to an overall score improvement without degrading the scores of single-part spoilers. Furthermore, we incorporate a post-hoc ensemble of models with multiple seeds in our system, leading to enhanced performance over a single model.
In analysis, we show that our system generates spoilers extractively rather than abstractively, even though we adapt the seq2seq generation approach. We also examine the errors made by our system and discuss the characteristics and potential difficulty of this task.
1756

heading: 2 Task Description
text: The SemEval-2023 Task 5 organizers offer the Webis Clickbait Spoiling Corpus 2022 dataset (Hagen et al., 2022). This dataset collects clickbait posts on social media (Facebook, Reddit, and Twitter), linked web pages in the posts, and various related information. The following information in the dataset is employed in this task (2023):
• postText: The clickbait posted on social media that entices a click.
• targetTitle: The title of the linked article on postText.
• targetParagraphs: The main content of the linked article on postText.
• spoiler: The human-extracted clickbait spoiler for postText either from targetTitle or targetParagraphs.
The task is to extract spoiler from targetTitle and targetParagraphs that spoils the clickbait post of postText (Figure 1). spoiler to be extracted are classified into the following three categories based on their structures (2023):
• phrase: Spoiler of single word or phrase.
• passage: Spoiler composed from continuous sentences.
• multi: Spoiler comprised of multiple discontinuous words or sentences.
The provided training data consist of 3200 samples. The counts for each type in the training data are 1367 (around 43%) for phrase, 1274 (around 40%) for passage, and 559 (around 17%) for multi. The agreement between the model’s predictions and the ground truths is evaluated based on the following three metrics: BLEU-4 (Papineni et al., 2002), BERTScore (Zhang* et al., 2020), and METEOR 1.5 (Denkowski and Lavie, 2014), an extended version of METEOR (Banerjee and Lavie, 2005).

heading: 3 Related Work
text: Hagen et al. (2022) employ an encoder-based question answering model for clickbait spoiling. They treat the target article as context and posted clickbait as question. Then the encoder-based question
answering model predicts the spans of the target spoiler in the context. However, their method only assumes one correct span; thus, it cannot handle the challenge of generating multipart spoilers where answers have multiple spans. Several studies in the field of question answering have tackled the issue of answering multi-span questions. For instance, Segal et al. (2020) have formulated the question answering task as a token classification problem, where each token is classified as either an answer or not. However, implementing these multiple-span extraction methods can be complex. In this work, we propose a simple seq2seq generation approach to address the challenge of multipart spoiler generations, avoiding complex implementations.

heading: 4 System Overview
text: This section presents an overview of the proposed system. Our system generates spoilers in two steps; first, multiseed seq2seq models generate spoilers, and then the generated outputs are postprocessively ensembled using a similarity-based ensemble approach.

heading: 4.1 Seq2seq Spoiler Generation
text: Seq2seq models take an input sequence and generate an output sequence directly. We use this seq2seq generation method for spoiler generation in a question answering manner. Given input sequence, which is concatenations of postText, targetTitle and targetParagraphs, the seq2seq model is expected to generate a sequence of target spoiler directly.
Input format. The input is provided in the form of "question : [postText] context : [targetTitle] - [targetParagraphs]". Since we use seq2seq models that have been partially trained with question answering during pre-training, incorporating question and context prefixes could help generate spoilers by using their inherent question answering abilities.
Output format. The model generates the sequence for the target spoilers directly. As for multipart spoiler, the model generates a concatenated sequence of several spans with white space since they have several answer spans.
With this simple seq2seq approach, we can address phrase, passage and multi spoilers simultaneously without implementing more complex methods (e.g., multiple-span extraction).
20XX/XX/XX Conference 8

heading: 4.2 Post-Hoc Model Ensembling
text: Some examples that cannot be successfully generated by a single model may be effectively generated by models trained with different seeds. Kobayashi (2018) has demonstrated that a simple post-hoc model ensembling approach inspired by majority voting in classification tasks also works effectively in generation tasks. We incorporate this posthoc ensembling method in our system to select a majority-like spoiler prediction among outputs from different seed models (Figure 2). Algorithm 1 outlines the adapted ensembling approach. First, the algorithm takes predicted spoilers from various seed models as inputs, calculates the edit distances of each predicted spoiler, and sums the computed distances over each model. The final output is chosen by the spoiler with the smallest summed edit distances. Kobayashi (2018) employ cosine as a similarity function, however, we use edit distance in our system for simplicity.

heading: 5 Experiment

heading: 5.1 Base Model Selection
text: To choose a seq2seq model for our system, we trained various seq2seq models on the given training data and evaluated their performance.
Algorithm 1 Post-hoc ensemble algorithm Input: Set X of predicted spoilers from different
seed models, function EditDist which calculates edit distance
Output: Selected spoiler y distances← {} for x ∈ X do
distSum←∑x′∈X,x̸=x′ EditDist(x, x′) distances[x]← distSum
end for y ← argmin(distances) return y
Settings. We used two pre-trained seq2seq models, T5 (Raffel et al., 2020) and Flan-T5 (Chung et al., 2022) both for base and large size. We selected these pre-trained models to leverage the knowledge and capabilities acquired during their multitask pre-training, including question answering. We fine-tuned these seq2seq models with the given training dataset. As a baseline, we used a large size of DeBERTa (He et al., 2021) fine-tuned for the question answering task, provided by the task organizers2. This model was first fine-tuned on SQuAD (Rajpurkar et al., 2018) and then further trained on the clickbait dataset. Training details for the seq2seq models are showed in Appendix A.
2https://github.com/pan-webis-de/ pan-code/tree/master/semeval23/baselines/ transformer-baseline-task-2
Results. Table 1 presents the performance of each model on the validation data. For all setting, the large-size seq2seq models consistently outperformed their base-size counterparts. Furthermore, when comparing models of the same size, FlanT5 consistently outperformed T5. Based on these findings, we selected the large-size of Flan-T5 as the base model for subsequent experiments. Comparing large-size of Flan-T5 and DeBERTa, FlanT5 outperformed DeBERTa on all and passage, but underperformed on phrase. As for multi, the Flan-T5 model showed competitive performance on all metrics, confirming that our approach successfully handled the multipart spoiler.

heading: 5.2 Post-Hoc Model Ensembling
text: In this experiment, we investigated the impact of the post-hoc model ensembling introduced in Section 4.2.
Settings. We prepared five Flan-T5 large models trained with different seeds. Then, we applied posthoc ensembling to the outputs from each model. Training details, including hyperparameters, are provided in Appendix A. We compared the following two settings:
• Single: The average scores of five Flan-T5 large models trained with various seeds.
• Ensemble: The scores of the post-hoc ensembling of five Flan-T5 large models trained with different seeds.
Results. Table 2 presents the performance of the Single and Ensemble on the test data. The findings demonstrated that the ensemble method outperformed the single model, confirming the effec-
tiveness of the ensemble. We submitted this ensemble of five Flan-T5 large models as our final system since we confirmed its effectiveness on the validation data.

heading: 6 Analysis
text: In this section, we examine our submitted system with the model ensembling.

heading: 6.1 Extractiveness of Seq2seq Model
text: Our system relies on the seq2seq models to generate spoilers. Therefore, our approach is not necessarily extractive. However, a natural question arises whether this seq2seq model is generating spoilers in an extractive or abstractive manner since our submitted system produces competitive results. We analyzed the proportion of the model’s outputs for phrase and passage included in targetTitle or targetParagraphs3. As a result, we discovered that 741 of 826 spoilers were extractively generated from targetTitle or targetParagraphs. This result demonstrates that the model generates spoilers extractively rather than abstractively, indicating that seq2seq models can learn to identify spoilers from given contexts.

heading: 6.2 Potential Usage of Spoiler Type Classifier
text: Hagen et al. (2022) have demonstrated that combining a high-performing spoiler type classifier can enhance the spoiler generation performance. To examine this hypothesis also holds for our system, we evaluated the model performance with and without the spoiler type information. First, we trained
3Since the output of the model for multi is generated as a single sentence consisting of multiple spoilers, it is difficult to verify whether each spoiler exists as it is in targetTitle or targetParagraphs. For this reason, we excluded multi from the analysis.
and evaluated three Flan-T5 large models with data of identical spoiler type each. This setting corresponds to the situation where we had a perfect spoiler type classifier and fully used the type information for the spoiler generation. We compared the above setting with our submitted system, which ignores the spoiler type information. Table 3 illustrates the model performances of the two settings on the test data. We discovered that the setting with the oracle classifier (Oracle) outperformed the setting without type information (Single, Ensemble) in all evaluation metrics. This indicates that even in the seq2seq method, spoiler type information can boost the performance of spoiler generation. In addition to combining high-performing spoiler type classifiers, incorporating multi-task learning, where the model learns spoiler classification and generation at the same time, would be future work.

heading: 7 Discussion
text: Table 2 illustrates that the performance of the passage is low compared to the other two types of spoilers. We examined the inclusion relationship between the model’s predictions and the gold spoilers on the passage to analyze how the model generates passage spoilers. We classified the errors into three categories: (1) the generated text includes a gold spoiler, (2) gold spoiler includes the generated text, (3) cases except (1) and (2). We had 67 cases for (1), 84 cases for (2), and 204 cases for (3) out of
355 error cases in the test set for passage. Table 4 shows the generation examples of the three types of passage spoiler errors. For (1), (2), and (3) exemplified in Table 4, although all of them cannot fully capture the answer span, none of these examples can be considered completely wrong, considering the meaning of postText and predicted spoilers. This demonstrates that it is challenging to uniquely determine the answer spoiler because of the characteristics of the task and the dataset. It is also assumed that annotations of these spoilers can vary even among the annotators. Therefore, it is important to consider the upper limit of the solvability of this task by measuring the inter-annotators agreement. Understanding where the current spoiler generation system has reached against the upper limit of the task and estimating the potential growth from the gap would be a big step forward for this task.

heading: 8 Conclusion
text: We presented our system submitted to the subtask 2 of SemEval-2023 Task 5: Clickbait Spoiling. Our system comprised two parts: 1) simple seq2seq spoiler generation and 2) post-hoc model ensembling. Experimental findings demonstrated that our system effectively addressed the multispan spoiler issue through seq2seq generation. Additionally, our system tended to generate sentences by identifying the appropriate position in the document, even
though it was not specifically trained to extract text. The analysis suggests that integrating reliable type classifiers can further improve the performance of our system. This will be future work.

heading: Acknowledgements
text: We would like to thank the task organizers for technical supports for the docker submission and the reviewers for helpful feedback. We also thank Keito Kudo for the technical advice. This work was supported by JSPS KAKENHI Grant Number JP22H00524, JST SPRING, Grant Number JPMJSP2114.

heading: A Training details
text: We implemented our experiment by Hugging Face (Wolf et al., 2020). During training, we used linear learning rate scheduler, with a maximum learning rate of 0.0001. We set max-update to 1000 and 2000 updates for base and large size, respectively, and selected checkpoints of 1000 and 1800 updates for each size. For the experiment in Section 5.1, we fixed the model seeds to 42. We used the model seeds of 43,45,46,47 and 48 for the experiment in Section 5.2.

heading: Future Work
text: This will be future work