Year: 2023

Abstract:
In this overview paper, we report on the second PAN Clickbait Challenge hosted as Task 5 at SemEval 2023. The challenge’s focus is to better support social media users by automatically generating short spoilers that close the curiosity gap induced by a clickbait post. We organized two subtasks: (1) spoiler type classification to assess what kind of spoiler a clickbait post warrants (e.g., a phrase), and (2) spoiler generation to generate an actual spoiler for a clickbait post.

heading: 1 Introduction
text: Clickbait posts link to web pages and advertise their content by arousing curiosity instead of providing informative headlines or summaries. The PAN Clickbait Challenge at SemEval 2023 aims to develop techniques for automatic clickbait spoiling: generating a short text that can close the curiosity gap as exemplified in Figure 1. Despite manual clickbait spoiling being popular on social networks,1 the problem had not received much attention by the research community. We thus organized a respective challenge based on our previously published dataset and baseline approaches (Hagen et al., 2022). With approaches submitted by 30 teams, the challenge had a very good participation. To improve reproducibility and replicability, we asked the teams to submit their software via the TIRA platform (Fröbe et al., 2023)—already used in several shared task with an archive of more than 500 research prototypes. Still, for our Clickbait Challenge, we also allowed result submissions, but a majority of the teams (19 of the 30) actually submitted software in the form of Docker images.
The task of clickbait spoiling, as exemplified in Figure 1, specifically aims at completing the “life cycle” of clickbait posts. The creation (Xu et al., 2019) and detection (Potthast et al., 2016, 2018) of clickbait posts are already operationalized, so we
1Saved You a Click has 1.9 million members on Reddit.
now aim to cover the “end” by automating spoiling. If sufficiently many people adopt spoiling tools, a clear message is send to publishers and social media platforms. Still, spoiling clickbait instead of removing it gives publishers the benefit of the doubt, since there may also be people who enjoy clicking on links for these kinds of trivia.
The PAN Clickbait Challenge at SemEval 2023 has two subtasks: (1) spoiler type classification, and (2) spoiler generation. For spoiler type classification (i.e., should the spoiler be a phrase, a passage, or have multiple parts), the input consists of a clickbait post and the linked document. For spoiler generation, the input additionally includes the desired type of the spoiler. To motivate the teams to explore diverse approaches, the number of submissions of a team for either task was not limited. For spoiler generation, we focused on the evaluation of extractive spoilers (i.e., spoilers in the form of text from the linked document) and leave abstractive spoilers for future work (as conducting robust evaluations using generated texts would substantially increase the labeling efforts).
The datasets, results, and submitted software of the Clickbait Challenge 2023 are freely available.2
2github.com/pan-webis-de/SEMEVAL-2023
2275

heading: 2 Lab Overview and Statistics
text: For the clickbait spoiling task, 83 teams from 24 countries registered. The majority of registrations came from the United States (15 registered teams), followed by Germany and India (13 teams each), Bangladesh (8 teams), Romania (6 teams), China (4 teams), Poland (3 teams), Denmark, Israel, Mexico, Russia (2 teams each), Bulgaria, Canada, Colombia, France, Japan, Mexico, Morocco, Netherlands, Norway, Poland, Spain, Thailand, Turkey (1 team each). Overall, 26 of the 83 teams were undergrad students supervised by a senior researcher—most undergrad teams came from Germany (8)and India (7).
Of the 83 registered teams, 30 actively participated in the task, out of which 23 teams submitted notebook papers describing their approaches. We used TIRA (Fröbe et al., 2023) as our submission platform through which participants could either make software submissions or result submissions. Software submissions have advantages over result submissions in terms of reproducibility (i.e., software can be rerun on different data) but also allow for blinded experiments3 (i.e., ensuring that the test set is kept private). While we allowed result submissions, we strongly encouraged software submissions and made the test data only available upon request. Overall, 10 active teams submitted results only, 1 team submitted results and software, while a majority of 19 teams submitted their software and never had access to the test data.
To submit a software, a team implemented their approach in a Docker image that they then uploaded to their dedicated Docker registry in TIRA. Software submissions in TIRA are immutable, and after uploading the image, the teams specified the to-be-executed command—the same Docker image can thus be used for multiple software submissions (e.g., by changing some parameters). A team could upload as many Docker images or software submissions as they liked; only they and TIRA had access to their dedicated Docker image registry (i.e., the images were not public while the shared task was ongoing). Overall, we received 202 software submissions from 86 unique Docker images (the smallest image has a compressed size of 504 MB, the largest image has 45 GB).
The teams could execute their software on the validation and the test data. On the validation data, the results, the evaluation scores, and the standard
3wikipedia.org/wiki/Blinded_experiment
and error output were directly visible to be able to verify that a submitted software works as expected. On the test data, the evaluation scores and the outputs were blinded; the only available feedback was whether the software produced a valid output or not. To ensure that the test data is not leaked and to improve the overall reproducibility, TIRA executes software in a sandbox by removing the internet connection. This, for instance, helps to ensure that a software is fully installed in the Docker image which eases rerunning a software at some later point in time (all libraries and models must be contained in an image to be able to execute the software without internet connection). We assisted the teams in dockerizing their approaches (frequent issues were files that were not executable in an image, or resources like libraries or models that were not included in the image). Eventually, all teams who wanted to make a software submission actually submitted software that produced valid outputs. For the execution, participants could select resources out of four options: (1) 1 CPU core with 10 GB RAM, (2) 2 cores with 20 GB RAM, (3) 4 cores with 40 GB RAM, or (4) 1 CPU core with 10 GB RAM and 1 Nvidia GeForce GTX 1080 GPU with 8 GB RAM. Upon request, TIRA could even execute a software on a Kubernetes cluster with 1,620 CPU cores, 25.4 TB RAM, and 24 GeForce GTX 1080 GPUs. A team could run their software as often as they wanted using different resources to study the scalability and reproducibility (e.g., does a software run on a GPU yield the same results as on a CPU?).

heading: 3 Task Description and Datasets
text: In the PAN Clickbait Challenge at SemEval 2023, we follow our previous observation (Hagen et al., 2022) and distinguish three different types of spoilers: (1) short phrase spoilers, (2) longer passage spoilers, and (3) multi-part spoilers. Examples are given in Figure 1. Based on the hypothesis that tailored spoiler generators for the different spoiler types will be the most effective (i.e., identifying the required spoiler type before actually generating a spoiler might be beneficial), we organized two subtasks: (1) spoiler type classification, and (2) spoiler generation with given spoiler type information.
The input for spoiler type classification is the clickbait post and the linked document. The task is to identify whether the clickbait post requires either a phrase (the upper post in Figure 1), a passage (the
08. . . Twitter How to keep your workout How to Keep ["Sweaty clothes stink, but . . . ", . . . , Phrase [[[7, 276], [7, 283]]]clothes from stinking Your . . . ". . . consider washing your stuff . . . "]
15. . . Twitter Just how safe are NYC’s Just how safe ["The Post independently tested . . . ", Passage [[[0, 0], [0, 171]]]water fountains? are . . . . . . , "Still worried? For a cleaner . . . "]
42. . . Twitter
A Harvard nutritionist and A Harvard nu- ["No matter how old you are . . . ", . . . ,
Multi [[[3, 0], [3, 14]], brain expert says she avoids tritionist and "1. Added sugar, as the brain . . . ", . . . , [[6,0] [6,14]], these 5 foods that "weaken brain expert "2. Fried foods like French . . . ", . . . , [[10,0] [10,35]], memory and focus." says . . . "3. High-glycemic-load carb . . . ", . . . ] . . . ]
middle post in Figure 1), or a multi-part spoiler (the bottom post in Figure 1). For an input post (with a UUID), an output in the form of
{"uuid":"<UUID>","spoilerType":"<TYPE>"}
has to be generated, where <TYPE> is either phrase, passage, or multi.
The input for spoiler generation is the clickbait post, the linked document, and the required spoiler type (phrase, passage, or multi). For an input post (with a UUID), an output in the form of
{"uuid":"<UUID>","spoiler":"<SPOILER>"}
has to be generated, where <SPOILER> is the spoiler for the clickbait post.
Webis Clickbait Spoiling Corpus 2022 As the data for the tasks, we use our Webis Clickbait Spoiling Corpus 2022 (5,000 posts with spoilers, fixed random 64/16/20 train/validation/test split for both subtasks). The training and validation sets were publicly available while the test dataset was made available upon request only. After the second Clickbait Challenge has ended, the complete corpus will be accessible under a research-friendly Creative Commons Attribution license in JSONL format.4
Table 1 showcases the main corpus fields for the posts from Figure 1. Since no main content extraction method worked reliably for all the linked web pages, we manually extracted titles and paragraphs while annotating the spoiler positions and types. In total, we have spent about 560 hours of in-house work on manual post selection, main content extraction, and spoiler identification to ensure high data quality (details on the corpus: (Hagen et al., 2022)). Most spoilers are phrases (42.5%) or passages (40%)—our annotation guidelines asked for a spoiler to be as short as possible (i.e., if one word is enough, not a whole sentence should be chosen).
4doi.org/10.5281/zenodo.6362726
A spoiler’s exact position in the linked page is part of the corpus, so that an automatic assessment of extractive spoilers using BLEU-4 (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and BERTScore (Zhang et al., 2020) is possible.

heading: 4 Task 1: Spoiler Type Classification
text: The goal of the first task of the PAN Clickbait Challenge at SemEval 2023 was to classify which spoiler type a given clickbait post requires. We explored a wide range of approaches during a pilot study to provide a competitive baseline as the basis for the shared task. We evaluated the effectiveness of spoiler type classification using the balanced accuracy over all three classes as the main effectiveness measure that we complement by the precision, recall, and F1 score in detecting all three classes. Overall, 25 teams submitted results to task 1.

heading: 4.1 Baseline and Pilot Experiments
text: For spoiler classification, we experimented with the classic feature-based models Naïve Bayes, Logistic Regression, and SVM, as well as the neural models BERT (Devlin et al., 2019), DeBERTa (He et al., 2021), and RoBERTa (Liu et al., 2019). The classic models used tf - and tf ·idf -weighted word and POS tag uni- and bigrams from the clickbait post and the linked web page (chi-square feature selection). For the neural models, we concatenated the post and the main content of the linked page to predict the spoiler type. Logistic regression as the best classic model was less effective than the best neural model RoBERTa (61.10 vs. 73.40 balanced accuracy on the validation dataset). Hence, we used RoBERTa as the baseline for task 1, as it was the most effective model but still leaves much room for further improvements. We released the code and the model of the baseline for task 1 as well as the resulting Docker image to simplify the
onboarding for participants.5

heading: 4.2 Participating Systems
text: Team Alexander Knox (Woźny and Lango, 2023) explored multiple prompts, and augmented spoilers for classification approaches using few-shot learning with large language models or finetuning of transformers. The large language model BLOOM (Scao et al., 2022) was used to augment a hypothetical spoiler for a given clickbait post and article, where BLOOM generated a phrase spoiler, a passage spoiler, and a multi-part spoiler in a fewshot setting. The generated spoilers were then used as additional input for the spoiler classification task, and pilot experiments showed that few-shot classification with BLOOM was substantially less effective than finetuned transformers DistilBERTbase (Sanh et al., 2019), ALBERT, and RoBERTabase were used where RoBERTa achieved the highest effectiveness.
Billie Newman (Kruff and Tran, 2023) used RoBERTa and additional features extracted by specifically designed rules. Regular expressions were used to extract the number of enumerations, lists, named entities, and a ratio that compares the length of the clickbait tweet to the length of the linked article, which was then added as additional textual context to the clickbait post and linked article before they were passed to the RoBERTa model.
Team Billy Batson (Sharma et al., 2023) used the clickbait post and the linked article as input to a DeBERTa-base to classify the spoiler type where paragraphs were ranked by their relevance to the clickbait post using a pairwise retrieval model. The retrieval model scored all passages of the linked article using a RoBERTa model that was trained in a contrastive learning setup (telling a passage containing a spoiler apart from a passage without a spoiler), retaining only the top-k passages as input for the DeBERTa based type classification.
Team Chick Adams (Pan et al., 2023) used different inputs to RoBERTa and DeBERTa models. Using either only the clickbait post, the linked article, or both as input to either RoBERTa respectively DeBERTa, they found that DeBERTa was more effective than RoBERTa and that the concatenation of the clickbait post and the linked article were more effective then the clickbait post or the linked article alone, and also that the clickbait post and
5github.com/pan-webis-de/pan-code/tree/master/ semeval23/baselines
the linked article do not complement each other as ensembles of both were less effective in their pilot experiments. The most effective model was DeBERTa using the concatenation of the clickbait post and the linked article as input.
Team Clark Kent (Mihalcea and Nisioi, 2023) experimented with the Pixel-based Encoder of Language (PIXEL) model (Rust et al., 2022) for task 1 to inspect if visual properties of clickbait posts and articles help in spoiler classification. PIXEL transforms its input text into RGB images and is a pre-trained vision transformer masked autoencoder with 112M parameters. The PIXEL model was finetuned for multiple epochs on the training data with early stopping using the clickbait post and the linked article as input.
Team Mr. Fosdick (Falkenberg et al., 2023) experimented with data augmentation and expansion techniques, deriving new datasets from the original training dataset by paraphrasing the clibkbait posts and the clickbait articles with GPT3. The augmented datasets where then used to train a transformer model and a random forrest model, where a DeBERTa model that used the clcikbait post and the linked article as input was the most effective approach trained on the augmented dataset.
Team Francis Wilde (Indurthi and Varma, 2023) used RoBERTa for spoiler-type classification in task 1. The RoBERTa model used the clickbait post as input (the linked article is ommitted) and was finetuned on the official training dataset.
Team Gallagher (Bilgis et al., 2023) experimented with T5, Long-T5, and Flan-T5, finding that a T5-Large model was more effective than Long-T5 in classifying the spoiler type by predicting the probability that the clickbait post and linked article as input generate the token phrase, passage, respectively multi.
Jack Flood (Kumar et al., 2023) used a BiLSTM model that used the length of the clickbait post and the linked article, the count of overlapping terms between the post and the article, and RoBERTa representations of the post and article as input.
Team John Boy Walton (Shmalts, 2023) used an ensemble of five models (variants of DistilBERT, BERT-base, and DeBERTa), averaging the logits of the five models to obtain the final classification.
Team Machamp trained a multi-task-learning model using the Machamp framework (van der Goot et al., 2021) on all SemEval-2023 tasks following the idea that training a single model on di-
verse tasks enables synergies between the different training tasks.
Team Matt Bai (Tailor and Mamidi, 2023) used BERT-base to classify the spoiler type using the clickbait post and linked article as input to the model (a second submission used only the clickbait post as input but this was less effective).
Team Mr. Wallace (Saravanan and Wilson, 2023) used a DistilBERT-base model using the clickbait post and linked article as input.
Team Monique Marmelstein (Sterz et al., 2023) used RoBERTa in a multi-task learning setup to train the model in parallel on task 1 and task 2 following the hypothesis that improvements in one task transfer to the other task as well.
Team Morbo the Annihilator experimented with Naíve Bayes, Logistic Regression, and Support Vector Machines comparing them under different input representations and preprocessing steps. The best approach used logistic regression using the term frequency and TF-IDF on the targetTitle, targetParagraphs, postText, topic modeling using matrix factorization on the concatenation of "targetTitle" and "postText", calculating similarity between targetTitle and postText, counting nouns and punctuation signs from targetTitle and.
Team Nancy Hicks Gribble (Keller et al., 2023) used the clickbait post and the linked article as input to RoBERTa models, comparing a multiclass classification model with one-vs-rest models, finding that the single multiclass RoBERTa model was overall more effective then combinations of three standalone models.
Team Perry White tested BERT, RoBERTa, and DeBERTa for spoiler type classification, exploring three types of data presentations: (1) using the article’s contents only, (2) concatenating the beginning of the post, title of article and the articles contents, and truncating the rest, (3) were using posts text and only 200 first tokens from the article, skipping the title and first paragraph. Despite this method producing inputs shorter than the other two methods, it scores several p.p. higher. (this is inspired by the F-shape reading pattern theory6). The DeBERTa model was the most effective approach for the task (Post text + 200 first tokens yielded 76% accuracy after 3 epochs).
Team Sam Miller (Störmer et al., 2023) finetuned XLNET (Yang et al., 2019) using the click-
6www.nngroup.com/articles/f-shaped-pattern-readingweb-content/
bait post as input for spoiler classification. Team Stephen Colbert (Spreitzer and Tran, 2023) used the newest version of DeBERTa-Large (version 3) while exploring different representations of the input dataset. The main idea was to use markup information from the linked article as additional input following the hypothesis that different types of clickbait posts use different techniques for search engine optimization that are manifested in the underlying markup of the linked article.
Team Walter Burns (Villa Cueva et al., 2023) uses the clickbait post and the title of the linked article as input to to an ensemble of 5 RoBERTa models that were trained from different random initializations. The class probabilities of the 5 RoBERTa models are then used as input for a neural network layer that predicts the spoiler type.
Team Frankly Unctuous, Miles Clarks, Paul Morgan, Spider Jerusalem, and Trinity Wells did not submit a notebook paper. We are in contact with the team to get a accurate high-level description of their approach. (Miles Clarks, Spider Jerusalem, and Trinity Wells made software submissions so that their submission is reproducible and could be inspected.)

heading: 4.3 Results
text: Table 2 compares each team’s most effective submission for spoiler type prediction measured as balanced accuracy (the main measure that we employ; also the most effective submission for each team was selected via balanced accuracy) over all three spoiler types and the precision, recall, and F1 score for phrase, passage, and multi spoilers on the test set. Our baseline was very effective, and only team Billy Batson outperformed the baseline in the balanced accuracy but only in the third decimal place (that we do not report in the table to not overstate the precision of observations). We leave out all submissions that resubmitted the baseline to only report the distinct set of submissions. While the overall best accuracy of 0.74 shows that there is still much performance, inspecting the precision and recall of different approaches for the different spoiler types shows that the different approaches might complement each other well, which might be suitable directions for future work. For instance, the approach by Billy Batson achieves very high precision for detecting phrase and passage spoilers, while only achieving a precision of 0.58 for multi spoilers, while other approaches achieve substan-
tially higher precision for multipart spoilers. Furthermore, the precision and recall scores per spoiler type in Table 2 are particularly important to guide real implementations of spoiling algorithms. First practical implementations might focus specifically on precision to ensure that wrong spoilers do not increase users efforts, e.g., by only spoiling clickbait posts requiring a certain type of easy-tospoil clickbait posts.

heading: 5 Task 2: Spoiler Generation
text: The goal of the second task of the PAN Clickbait Challenge at SemEval 2023 was to generate the actual spoiler that could be shown to users to satisfy the curiosity introduced by a clickbait post. We explored a wide range of approaches during a pilot study to provide a competitive baseline as the basis for the shared task. We evaluated the effectiveness of spoiler generation by contrasting BLEU-4, METEOR, and BERTScore scores (generated vs. ground-truth spoiler). Overall, 17 teams submitted results to task 2.

heading: 5.1 Baseline and Pilot Experiments
text: For spoiler generation, we experimented with the question answering approaches BERT, DeBERTa, and RoBERTa, as well as the passage retrieval approaches BM25, MonoBERT (Nogueira and Cho, 2019; Nogueira et al., 2019), and MonoT5 (Nogueira et al., 2020). The question answering approaches used the clickbait post as question aiming to find the spoiler as answer in the linked article. We found that question answering approaches are substantially more effective than passage retrieval approaches and that fine-tuning the question answering models on SQuAD before fine-tuning on our clickbait spoiling task improves their effectiveness. We also compared models simply trained on all training posts to two-step approaches that first classify the spoiler type to then select a spoiler generator only trained on the respective spoiler type. For the two-step approaches, we compare variants using a perfect “oracle” classification or the RoBERTa classifier that achieved the best accuracy in the classification pilot study. Overall, DeBERTa achieves the highest spoiling effectiveness (according to BLEU-4, METEOR, and BERTScore) in all
scenarios. We released the code and the model of the baseline for task 2 as well as the resulting Docker image to simplify the onboarding for participants.7

heading: 5.2 Participating Systems
text: Team Billie Newman (Kruff and Tran, 2023) participated used a mixture of RoBERTa models enriched with a rule-based approach following manually designed rules. Two RoBERTa models were first fine-tuned on SQuAD version 2 for extractive question answering and subsequently one model was finetuned for phrase spoiling, and one model for passage spoiling, while a rule-based approach was used to spoil multi-type spoilers.
Team Brooke English (Tang, 2023) used four DeBERTa models for spoiler generation that were selected using the given spoiler type and the confidence score of the DeBERTa model. Three dedicated DeBERTa models were trained for phrase, passage, respectively multi-part spoilers, and one DeBERTa model for all three types. Each model used the clickbait post concatenated to the linked article as input, and for each input, two spoiler candidates were generated, one with the dedicated DeBERTa model for the spoiler type and one with the general-purpose DeBERTa model, and the candidate with the higher probability as estimated by the generating model was returned as spoiler.
Team Diane Simmons (Krog and Agirrezabal, 2023) experimented with DeBERTa in a finetuning setup and the large language model BLOOM in a zero-shot setup. DeBERTa used the clickbait post and the linked article as input to extract the spoiler from the linked article, while the clickbait post and the linked article where embedded in diverse prompts for abstractive spoiler generation using BLOOM. The DeBERTa model was substantially more effective then BLOOM.
Team Gallagher (Bilgis et al., 2023) experimented with T5 (Raffel et al., 2020), Long-T5 (Guo et al.), and Flan-T5 (Chung et al., 2022) building an ensemble: T5-Large was used for phrase and multi spoilers (was more effective then Long-T5 and Flan-T5) while Flan-T5-Large was used for passage spoilers.
Team Jack Flood (Kumar et al., 2023) used passage retrieval appraoches to select relevant passages from the linked article that are subsequently
7github.com/pan-webis-de/pan-code/tree/master/ semeval23/baselines
used as input to a RoBERTa model for spoiling. The linked article is segmented into sentences that are ranked with reciprocal rank fusion combining monoT5 and BM25 scores. The top-5 sentences are subsequently passed to dedicated RoBERTa models for phrase, passage, respectively multipart spoilers.
Team Jack Ryder (Wangsadirdja et al., 2023) use multiple zero-shot question-answering models without fine-tuning as ensemble, reformulating the clickbait posts as questions to imitate the expected input of pre-trained question-answering models showing the sentences from the linked article sorted by their SBERT similarity. They experiment with RoBERTa, DeBERTa, Flan-T5, and UnifiedQA, finding in pilot studies that rephrasing the clickbait post as questions usually improves the effectiveness while sorting sentences by their SBERT similarity does not. The final model combines the best configuration for each spoiler type used in a zero-shot setup.
Team John Boy Walton (Shmalts, 2023) ensembled five models (variants of DistilBERT, BERTbase, and DeBERTa), averaging the logits of the five models to obtain the extractive spoiler (extracting the position of the start and end of the spoiler).
Team John King (Kurita et al., 2023) used an ensemble of five sequence-to-sequence models (T5base, T5-large, Flan-T5-base, flan-t5-large, and DeBERTa-large) for clickbait spoiling following the idea that those models can generate multipart spoilers without adaption (e.g., span detection approaches would require more complex adoptions for multi-part spoilers). Each sequence-tosequence model generated spoiler candidates independent of each other, returning the candidate with the minimum edit distance to all other candidates as the spoiler.
Team Morbo the Annihilator used RoBERTabase for spoiling, starting from Roberta-base finetuned on SQUAD version 2. Three models were trained, for phrase, passage, and multi-part spoilers that used the clickbait post concatenated to the linked article as input.
Team Monique Marmelstein (Sterz et al., 2023) used RoBERTa in a multi-task learning setup to train the model in parallel on task 1 and task 2 following the hypothesis that improvements in one task transfer to the other task as well.
Team Nancy Hicks Gribble (Keller et al., 2023) used a RoBERTa model that was pre-trained on
the SQuAD (Rajpurkar et al., 2018) dataset for question answering (roberta-base-squad28). Three different models were trained for each spoiler type, using the clickbait post and the linked article as input.
Team Sabrina Spellman (Birkenheuer et al., 2023) trained a DeBERTa-base model using an additional dataset constructed from Huffington Post articles to better learn the structure of news articles. The first distant supervision dataset for training was inspired by cloze-style questions, in that a passage was removed from an article to fill a gap in a given sentence. The second distant supervision dataset for training retrieved enumerations from the text. After fine-tuning DeBERTa on both distantsupervision tasks, three models were trained specifically for phrase, passage, and multi spoilers.
Team Walter Burns (Villa Cueva et al., 2023) used an ensemble of RoBERTa and DeBERTa by averaging the predicted logits of both models. For multi-part spoiling, the top-5 extractive, nonoverlapping spoilers were extracted from the averaged logits.

heading: 5.3 Results
text: Table 3 compares each teams most effective submission for spoiler generation measured as BLEU-4 (BL4), BERTScore (BSc.), and METEOR (MET) accross all clickbait posts and across the three spoiler types. In contrast to task 1, much more teams outperform our strong baseline (6 teams have submissions with a higher BLEU-4 score). Team John King achieves the highest overall effectiveness (BLEU-4 of 0.48). Remarkably, the best approach of John King uses a generative sequence-tosequence approach, although our evaluation should favor extractive spoiling approaches as we calculate the scores in comparison to the ground-truth spoiler extracted from the linked article. Comparing the effectiveness accross the different spoiler types reveals that there is still substantial room for further improvement: While John King clearly achieves the highest effectiveness for multi spoilers, other submissions by Sabrina Spellman and Walter Burns achieve comparable effectiveness for phrase spoilers and the approach by Sabrina Spellmann substantially outperforms the submission of John King for passage spoilers. Interestingly, the intention of team John King to use a sequence-tosequence model for multipart spoiling because they
8huggingface.co/deepset/roberta-base-squad2
tira.run( ’clickbait-spoiling/<team-name>/<software>’, dataset=’<dataset>’ )
Listing 1: Local re-execution of the software <software> by team <team-name> that was submitted to the clickbait spoiling task.
dont need complex adoptions in comparison to span based extractive approaches proved to be correct because those models are substantially more effective then all other models (e.g., team John King achieves a BLEU-4 score of 0.44 that is followed 0.30 of team Sabrina Spellman).
Combining the results from Table 2 and Table 3 reveals that first user-facing implementations of clickbait spoiling might focus on only spoiling phrase spoilers, as the precision oriented detection of phrase spoilers is possible (maximum precision of 0.79) and spoiling phrase spoilers is also very accurate (the approach of team gallagher achieves an BLEU-4 score of 0.69, a BERTScore of 0.96, and a METEOR score of 0.71).

heading: 6 Post-Hoc Reproducibility Experiments
text: We used TIRA (Fröbe et al., 2023) as submission system so that we can publish all artifacts resulting from the shared task to improve reproducibility and re-usability. During the shared task, TIRA maintained all data in a Git repository, where each software execution and evaluation were triggered by commits in continuous integration and continuous deployment pipelines. We publish this repository, including all datasets, results, evaluations, logs, metadata, and software snapshots.9
Listing 1 exemplifies how software submissions can be re-executed on the same or new data (the API allows the re-use of existing datasets but also new data of the same structure, requiring only Docker and Python 3 as dependencies). Consequently, the shared task repository that we published after the completion of the shared task now serves as a entry point for follow-up studies. All researchers can fork this repository and contribute.

heading: 7 Conclusion
text: The second PAN Clickbait Challenge hosted as Task 5 at SemEval 2023 received submission from 30 active teams on two subtasks: (1) spoiler type
9github.com/pan-webis-de/SEMEVAL-2023
John King burning-desert 02-06-07-29-13 0.48 0.93 0.50 0.66 0.95 0.40 0.32 0.91 0.43 0.44 0.93 0.65 S. Spellman cerulean-r. 01-24-00-32-15 0.47 0.93 0.50 0.64 0.95 0.69 0.36 0.91 0.51 0.30 0.90 0.45 Walter Burns result upload 01-26-01-01-19 0.44 0.92 0.49 0.68 0.96 0.58 0.28 0.90 0.47 0.21 0.90 0.48 Mr. Fosdick Squad QA 01-10-21-21-07 0.42 0.92 0.48 0.65 0.95 0.46 0.32 0.91 0.50 0.12 0.89 0.43 Brooke English congruent-eel 01-31-05-27-54 0.42 0.92 0.42 0.66 0.95 0.41 0.29 0.90 0.44 0.15 0.89 0.39 Gallegher task2-gpu 02-01-21-35-34 0.41 0.92 0.44 0.69 0.96 0.71 0.24 0.90 0.42 0.12 0.88 0.35 Baseline legato-cinema 01-01-10-57-35 0.40 0.92 0.43 0.65 0.95 0.60 0.24 0.90 0.44 0.12 0.87 0.30 M. Marmelstein beige-cordon 01-31-23-01-02 0.36 0.91 0.43 0.62 0.95 0.55 0.19 0.89 0.41 0.11 0.89 0.38 Morbo achromatic-f. 02-01-08-16-39 0.36 0.91 0.39 0.63 0.95 0.65 0.19 0.89 0.36 0.09 0.88 0.30 Billie Newman poky-sprite 01-24-17-37-56 0.32 0.90 0.30 0.49 0.92 0.23 0.20 0.87 0.31 0.20 0.88 0.34 Sam Miller equidistant-m. 02-01-04-47-19 0.31 0.66 0.38 0.60 0.95 0.54 0.11 0.55 0.27 0.10 0.22 0.46 Jack Ryder joint-trap 02-01-12-59-21 0.27 0.89 0.27 0.48 0.92 0.37 0.16 0.88 0.28 0.03 0.85 0.18 N. Gribble short-screw 01-28-17-53-10 0.27 0.88 0.26 0.48 0.93 0.44 0.14 0.84 0.25 0.05 0.85 0.23 Diane Simmons result upload 01-24-14-49-38 0.25 0.89 0.26 0.48 0.93 0.44 0.07 0.86 0.23 0.08 0.85 0.18 Jack Flood result upload 02-12-16-12-01 0.18 0.88 0.18 0.32 0.89 0.14 0.08 0.87 0.21 0.05 0.85 0.16 J. Walton felt 02-13-16-20-23 0.08 0.86 0.14 0.16 0.87 0.17 0.03 0.85 0.14 0.01 0.83 0.10 Miles Clarkson black-cadet 01-24-09-32-18 0.04 0.84 0.10 0.04 0.84 0.04 0.02 0.84 0.12 0.07 0.85 0.17 Machamp result upload 01-28-15-24-37 0.00 0.83 0.01 0.00 0.85 0.00 0.00 0.82 0.01 0.00 0.81 0.01
classification to assess what kind of spoiler a clickbait post requires, and (2) spoiler generation to generate an actual spoiler for a clickbait post. The most effective spoiler type classifier uses sentence retrieval to run a DeBERTa-based classifier on the most relevant sentences of the linked document. The most effective spoiler generator uses five independently run generative sequence-to-sequence spoiler generators, from whose outputs the most promising spoiler is selected via majority voting based on edit distances.
Our results show some substantial advancements, but there is still much room for further improvements. First, as no spoiler generator achieves the highest effectiveness across all three spoiler types, the best generators still complement each other and combining the underlying ideas can probably further increase the effectiveness. Second, as the most effective spoiler generator is ensemble-based and comes as a 45 GB Docker image, distilling a model with a better efficiency–effectiveness tradeoff also is an interesting direction for future work.

heading: Acknowledgements
text: We want to thank all participants of the shared task for their valuable submissions. Furthermore, we thank all reviewers: Alexander Bondarenko, Anubhav Sharma, Dirk Wangsadirdja, Dragos Stefan Mihalcea, Ferdinand Schlatt, Hiroto Kurita, Ikumi Ito, Jüri Keller, Kim Bürgl, Maksim Shmalts, Manex
Aguirrezabal Zabaleta, Mateusz Woźny, Sebastian Günther, Sujit Kumar, Vijayasaradhi Indurthi, and Vineet Saravanan.
This work has been partially supported by the OpenWebSearch.eu project (funded by the EU; GA 101070014).

heading: Future Work
text: Second, as the most effective spoiler generator is ensemble-based and comes as a 45 GB Docker image, distilling a model with a better efficiency–effectiveness tradeoff also is an interesting direction for future work