Jack-Ryder at SemEval-2023 Task 5: Zero-Shot Clickbait Spoiling by Rephrasing Titles as Questions
Year: 2023

Abstract:
In this paper, we describe our approach to the clickbait spoiling task of SemEval 2023. The core idea behind our system is to leverage pretrained models capable of Question Answering (QA) to extract the spoiler from article texts based on the clickbait title without any taskspecific training. Since oftentimes, these titles are not phrased as questions, we automatically rephrase the clickbait titles as questions in order to better suit the pretraining task of the QAcapable models. Also, to fit as much relevant context into the model’s limited input size as possible, we propose to reorder the sentences by their relevance using a semantic similarity model. Finally, we evaluate QA as well as text generation models (via prompting) to extract the spoiler from the text. Based on the validation data, our final model selects each of these components depending on the spoiler type and achieves satisfactory zero-shot results. The ideas described in this paper can easily be applied in fine-tuning settings.

heading: 1 Introduction
text: The objective of the Clickbait Spoiling task (Fröbe et al., 2023a) is to find/extract the spoiler from a linked article that satisfies the reader’s curiosity that was induced by a clickbait title (e.g. “You won’t believe how well this SemEval clickbait spoiling system performs!”). The challenge differentiates between three types of spoilers: phrase (the spoiler is a short phrase), passage (a sentence or multiple sentences that span from a beginning position to an end position without any gap), and multi (the spoiler consists of multiple disjoint phrases/passages).
In this paper, we describe our system for this challenge that is based only on pre-trained models and no fine-tuning. We model spoiler extraction as a Question Answering (QA) task and explore four different pre-trained architectures: Two extractive QA using BERT models and two prompt-based text generation models (with two prompts each).
In order to make the clickbait title more similar to the questions the QA models have been trained on, we propose to automatically rephrase them into a question form. To ensure that the model receives the most relevant article text passages in its limited context, we propose to filter and sort the article text using a pre-trained semantic similarity model. We evaluate our pipeline on the validation dataset and combine the best components for each spoiler type as our final model. We find that rephrasing the clickbait title usually improves performance, while relevance-based context ordering does not. Our final model makes use of clickbait title rephrasing for phrase and multi-spoiler types, a DeBERTa QA model for phrase and passage-type spoilers, and a Flan-T5 model for multi-type spoilers. The results on the test set show satisfactory results, considering that our system is not fine-tuned in any way. Our code is publicly available1.

heading: 2 Background
text: Research on clickbait texts can be divided into mainly three categories: Clickbait detection, type classification, and spoiling. Clickbait detection tries to predict whether a news article’s headline is a clickbait title. For this, several feature-based machine learning models have been developed (Potthast et al., 2016; Cao et al., 2017; Genç and Surer, 2021). Clickbait type classification and spoiling are the tasks of this challenge, which try to detect the type of spoiler (phrase, passage, multi) and extract the spoiler from the text, respectively. For clickbait type classification, transformer-based models have shown the best performance (Hagen et al., 2022). The information about the type of spoiler can potentially be used in the spoiler extraction model. Clickbait spoiling is mainly done using fine-tuned extractive and abstractive Question Answering (QA) models (Hagen et al., 2022; Johnson
1https://github.com/lingsond/semeval-23-task-5
1090
et al., 2022; Heiervang, 2022), where extractive models are generally more suitable (Johnson et al., 2022). Zero-shot spoiling using pre-trained QA models is inferior to fine-tuned models (Heiervang, 2022). All of the available approaches use the clickbait title without modification as the “question” input, which might not be beneficial for the performance of the model, since the clickbait title of news articles is oftentimes not phrased as a question (e.g. “You won’t believe what happened after the event!”). In this paper, we show for the spoiling task that rephrasing the clickbait title as a question usually improves the performance of zero-shot QA models. While this still not achieves comparable performance to fine-tuned models, exploring easyto-implement steps to improve zero-shot performance can easily be adapted to fine-tuned settings.
For the task, a training and validation set are provided, with each example consisting of a clickbait title (postText; the text that is posted on social media), article text (targetTitle and targetParagraphs), the extracted spoiler (spoiler; not available during testing) with its positions (spoilerPositions; not available during testing), as well as the type of spoiler (tags). Additional meta-information is provided but not used by our system. All submissions are submitted to and evaluated on TIRA (Fröbe et al., 2023b).

heading: 3 System Overview
text: Our system is shown in Figure 1. It only leverages pre-trained models (question answering, text generation, and semantic similarity models) to extract the spoiler from a given text without the need for any training. While it is certainly beneficial to employ fine-tuning as done by previous work, we experiment with other ideas that have not been proposed in the clickbait spoiling literature, leading to new insights into what works and what does not
for this kind of task. The elimination of training enables faster model development and fast updates to the model pipeline when a better component is available. Note that our system can also be used in fine-tuning settings without any modifications, so our proposed additions to the pipeline could be beneficial to fine-tuned approaches.
Our pipeline consists of mainly three components that are introduced based on the following key ideas: 1. QA models are trained on questions, but clickbait titles often are not questions. Thus, we use a pre-trained model to rephrase clickbait titles into questions. 2. The article texts are oftentimes too long for the model input size. Therefore, we order each article’s sentences by relevance using semantic similarity to the title, selecting the most important sentences as model input. 3. Recently, zero-shot capabilities of large language models are getting massive research interest, but have not been investigated for clickbait spoiling. Therefore, we evaluate text generation-based models with different prompt templates in addition to QA models, calling them collectively “QA-capable models”. We now go through each of these three components (Clickbait Title Rephrasing, Relevance-based Context Ordering, and Pre-trained QA-capable Model) and describe how they work. We later select the model to use and whether to use clickbait title rephrasing and relevance-based context ordering based on the validation results for each spoiler type (phrase, passage, multi).

heading: 3.1 Clickbait Title Rephrasing
text: In order to use pre-trained Question Answering (QA) models to extract spoilers to a clickbait title from its corresponding article text, we hypothesize that rephrasing the clickbait title as a question overall improves performance. For this, we check whether the clickbait title is already a question by
looking for an ending question mark. If not, we let a pre-trained language model, i.e. Flan-T5 (Chung et al., 2022), rephrase the provided clickbait post text into a question using prompting (e.g. “The anytime snack you won’t feel guilty about eating” is rephrased to the question “What snack is the best for when you want to eat something without feeling guilty?”). We create the following prompt template: Change the headline to question form. Headline: {clickbait title}, with {clickbait title} being the article headline.
In order to ensure that the rephrased question is still similar to the original clickbait title, the output of the model is compared to the original clickbait title with a semantic similarity model. For this, we embed both texts as vectors using the all-MiniLM-L6-v2 model from SentenceBERT (Reimers and Gurevych, 2019) and compute the cosine similarity between them. If the similarity is above 0.7, we continue using the newly created question, otherwise, the original clickbait title is used. This procedure catches failed rephrasings, i.e. incorrect or unrelated questions derived from the clickbait title. For example “This popular soda could cure your hangovers scientists say” is rephrased to “What is the best way to cure hangovers?”, which does not capture the “soda” part of the original clickbait title and is thus not used. From now on, we refer to the output of this system component as the “question”, regardless of whether it is the clickbait title or a rephrased version.

heading: 3.2 Relevance-based Context Ordering
text: For the QA-capable models used, we need to provide a context that contains the answer to the question, i.e. the spoiler to the clickbait title. In general, we create such context by concatenating the article headline to the article text, if the headline is not already in the article. Note that the article headline and the clickbait title (which is often posted to social media to deceive people into clicking the link) might be different, so it is useful to consider the headline as part of the context. One issue with the application of QA-capable models to these texts is that the article text is usually longer than the supported input size. Consequently, we propose to select the most relevant sentences of the text using a semantic similarity model.
This process is again implemented via SentenceBERT: we split the article into sentences, embed all sentences separately, and sort them by embedding
similarity to the original clickbait title. The approach is based on the assumption that the sentence that contains the answer to a question is semantically more similar to the clickbait title than other possibly unrelated sentences in the article. We sort all sentences in decreasing order and concatenate them to create the new context. We then take as many tokens as possible from the beginning that fit into the context size of the QA-capable model.

heading: 3.3 Pre-trained QA-capable Model
text: Given the question and context of the previous components, we can apply pre-trained QA-capable models to extract the spoiler. Similarly to related work, we use Question Answering models that take the question and the context as input. In light of the recent rise of language models that are capable of showing impressive performance in many tasks in a zero-shot setting (Chung et al., 2022; Kojima et al., 2022), we also test such models. We call them Text Generation (TG) models from now on.
Question Answering Models We can apply pretrained QA models without any modifications since we mostly use questions as input. Concretely, we follow related work and experiment with DeBERTa2 (He et al., 2021) and RoBERTa3 (Liu et al., 2019) for question answering.
Text Generation Models with Prompting We also evaluate text generation models that take the question and context in a prompt template and generate the output according to their text completion objective. For the models, we experiment with Flan-T5 (Chung et al., 2022) and UnifiedQA (Heiervang, 2022). We also evaluate different prompt templates that are more focused on the spoiler extraction or the question-answering task. The spoiler-focused prompt template (Spoilprompt) is: From the following Context and Clickbait, extract the spoiler. Clickbait: {question} Context: {context} Spoiler: The question answering focused prompt template (QA-prompt) is: Answer the following question based on the given context. question: {question}
2https://hf.co/deepset/deberta-v3-large-squad2 3https://hf.co/deepset/roberta-large-squad2
context: {context} answer: Note that line breaks are used between parts of the prompt, following the work of UnifiedQA (Heiervang, 2022). The sorted article text from the previous step is at the end of the prompt, which leads to cutting off only context whenever the prompt is getting too large for the model to process.

heading: 4 Experiments and Results
text: We evaluate our proposed system on the validation data of the challenge and give the BLEU-4 score for all and spoiler-type specific examples in Table 1. Here, we observe multiple findings that we use to build the final model for our submission.
Clickbait Title Rephrasing improves results We propose to rephrase the clickbait title as a question in order to better reflect the type of texts the QA-capable models were trained on. In Table 1, each cell states the performance without and with the clickbait title rephrasing component, separated by a slash. We typeset the better performance of the two in bold. Overall, the clickbait title rephrasing component mostly improves performance on the dataset, especially for the phrase spoiler type.
Relevance-based Context Ordering usually does not help We propose to sort the sentences of the article text by the semantic similarity to the click-
bait title, presumably giving the most relevant context into the QA-capable models. As comparing the left and right parts of Table 1 suggest, this approach usually leads to worse performance than feeding the model the beginning of the article text. This can have multiple reasons: For one, the spoiler that needs to be extracted could not be present in the constructed context, which means that our approach does not select the most relevant sentences. Another reason might be that due to the reordering of the sentences, the content of the text is misleading or confusing. To analyze this, we compute the fraction of examples in which the desired spoiler text can be found in the context that we give to the QA-capable model. These fractions in Table 2 are lower for the relevance-based ordered context, meaning that the desired spoiler text is often found in the first few sentences of the article (around 90%) and that our approach does not find the most relevant sentences. We assume that the semantic similarity based on SentenceBERT is not a good measure for relevance, since the similarity between a question and its answer might often be low.
Question Answering models generally work better than Text Generation models While previous work exclusively uses QA models, we also experiment with more general text generation models that are instructed by prompts. Table 1 shows a clear tendency in this regard: Dedicated QA models generally work better than text generation models on this task. We argue that TG models are also capable of solving this task, however, they usually do not reach the performance of QA models. A reason for why this is the case is that QA models have exactly one task they were trained on. TG models are usually more general and can be instructed to perform tasks using prompts.
The choice of these prompts seems to have a large influence on the performance. We find that for the different models, different prompts work best:
the UnifiedQA model works best with the spoilingfocused prompt (Spoil-prompt), while Flan-T5 works better with the QA-focused prompt (QAprompt). We hypothesize that the better prompt better resembles prompts from training.

heading: 4.1 Final Hybrid Model
text: Based on our observations above, we build a hybrid model making use of the best model for every spoiler type (shaded green in Table 1). This means our final hybrid model uses clickbait title rephrasing for phrase and multi-spoiler types, no relevance-based context ordering, the DeBERTA QA model for phrase and passage-type spoilers, as well as the Flan-T5 model with QA-prompt for multi-type spoilers. Overall, we thus evaluate the DeBERTa model with and without clickbait title rephrasing as well as the Flan-T5 model with clickbait title rephrasing and QA-prompt as the prompt template on the test dataset of the challenge as well their combination depending on the spoiler type.

heading: 4.2 Challenge Results
text: Table 3 shows the BLEU-4 score, micro BERTScore, and METEOR score (Banerjee and Lavie, 2005) for all models we used to determine the final test results for our system. Overall, our system achieves satisfactory results, given the fact that no fine-tuning was conducted. For phrase and passage spoilers, the best validation models are also the models with the best test performance. For multi-type spoilers, however, the Flan-T5 model does not perform as well as the DeBERTa model with clickbait title rephrasing, indicating that pretrained QA models might be enough for this task.
Note that the shown models have been submitted to TIRA after the official deadline. The official submissions contain components derived from an incorrect interpretation of Table 1. Overall, however, the new results are very similar to the official submissions. According to the organizers, our official submissions were among the top ten teams for
most metrics and for phrase spoiling even among the top five teams for all three metrics.

heading: 5 Conclusion
text: In this paper, we have presented our approach to the clickbait spoiling challenge. We have explored multiple strategies to improve the performance of pre-trained models without any fine-tuning. The results are not as good as fine-tuned models (see other competitors and the challenge baseline), but we have found that clickbait title rephrasing usually improves performance, which can potentially be applied to fine-tuned models as well.
One strategy that has not worked was the relevance-based context ordering, which mostly decreases performance. We suspect that our measure of relevance is not reliable for question and answer pairs. Exploring more reliable and effective measures might be interesting. Another possible focus for future work might be to experiment with more prompts for text generation models. We have shown that different prompts can have a severe impact on the model’s performance, so more careful prompt engineering might lead to similar performance as for the QA models and can then get fine-tuned. Also, using larger and thus more powerful text generation models such as the hosted GPT-34 (Brown et al., 2020) or the open source OPT-175B (Zhang et al., 2022) are options that showed promising results in preliminary experiments, but were not further explored due to monetary and computational limitations.

heading: Acknowledgements
text: This work is partially supported by the MOTIV research project funded by the Bavarian Research Institute for Digital Transformation (bidt), an institute of the Bavarian Academy of Sciences and Humanities. The authors are responsible for the content of this publication.
4https://openai.com/

heading: Future Work
text: Another possible focus for future work might be to experiment with more prompts for text generation models. We have shown that different prompts can have a severe impact on the model’s performance, so more careful prompt engineering might lead to similar performance as for the QA models and can then get fine-tuned. Also, using larger and thus more powerful text generation models such as the hosted GPT-34 (Brown et al. , 2020) or the open source OPT-175B (Zhang et al. , 2022) are options that showed promising results in preliminary experiments, but were not further explored due to monetary and computational limitations