Stephen Colbert at SemEval-2023 Task 5: Using Markup for Classifying Clickbait
Year: 2023

Abstract:
For SemEval-2023 Task 5, we have submitted three DeBERTaV3LARGE models to tackle the first subtask, classifying spoiler types (passage, phrase, multi) of clickbait web articles. The choice of basic parameters like sequence length with BERTBASE uncased and further approaches were then tested with DeBERTaV3BASE only moving the most promising ones to DeBERTaV3LARGE. Our research showed that information-placement on webpages is often optimized regarding e.g. adplacement. Those informations are usually described within the webpages markup which is why we conducted an approach that takes this into account. Overall we could not manage to beat the baseline, which we lead down to three reasons: First we only crawled markup for Huffington Post articles, extracting only <p>and <a>-tags which will not cover enough aspects of a webpages design. Second Huffington Post articles are overrepresented in the given dataset, which, third, shows an imbalance towards the spoiler tags. We highly suggest re-annotating the given dataset to use markupoptimized models like MarkupLM or TIE and to clear it from embedded articles like "Yahoo" or archives like "archive.is" or "web.archive" to avoid noise. Also, the imbalance should be tackled by adding articles from sources other than Huffington Post, considering that also multi-tagged entries should be balanced towards passageand phrase-tagged ones.

heading: 1 Introduction
text: The shared task "Clickbait Challenge at SemEval 2023 - Clickbait Spoiling" on PAN was about classifying and predicting spoilers from English (gossip) articles (Fröbe et al., 2023a) provided via TIRA (Fröbe et al., 2023b). It poses several technical challenges, especially handling text data with a high degree of diversity in genre, writing style, and structure. The task aims to close the curiosity gap clickbait posts cause within their readers by providing an informative summary. Solving this problem
would benefit the use of social media since click baits are an annoying phenomenon aiming to manipulate one into visiting a page which may affect the user’s credibility and quality perception in a negative way (Hagen et al., 2022). The task consists of two subtasks:
1. classifying spoilers into three categories: "passage," "phrase," or "multi"
2. predicting spoilers for an article based on those categories
Our system focuses on solving the classification task with a fine-tuned DeBERTaV3LARGE model trained on the given Webis Clickbait 2022 Corpus, which we enriched with information from markup crawled from the original websites. We have chosen this approach since we discovered that (gossip) articles, in most cases, as webpages, are subject to some professional style guide aiming to make visitors stay on the page, which is essential, especially when it comes to advertising revenues (Nielsen and Pernice, 2009). The results fell short of expectations since we only tested with markup from Huffington Post articles and only from the training dataset. Nevertheless, out of 31 submitting teams for task 1, our team, for most measures, got among the top 10 teams and the top 3 with the highest F1 score in classifying passage spoilers.

heading: 2 Background
text: With the Webis Clickbait 2022 Corpus, the dataset contains 4000 clickbait articles, split into training and validation sets. For solving the classification task, we focused on the following:
• the articles cleared from the advertisement or other web-page-related-noise, e.g., markup (column "targetParagraphs")
• article titles (column "targetTitle")
1844
• post text (the text that was initially posted on social media when providing a link to the article) (column "postText")
• the article URLs (column "targetUrl")
• spoiler classification (column "tags")
We utilized the state-of-the-art transformerbased architectures BERT (Devlin et al., 2019) and DeBERTaV3 (He et al., 2021), which, by adding a disentangled attention mechanism and an enhanced mask decoder, significantly improves performance on various tasks compared to BERT (He et al., 2021):
• BERTBASE uncased for initial parameter evaluation
• DeBERTaV3BASE for fast evaluation, especially on our balancing approaches
• DeBERTaV3LARGE for the most promising approaches evaluated from the base model and for our markup-based approach
Our experiments show that taking markup into account regarding webpage analysis is quite common in visually rich document understanding. However, since the given dataset does not meet the requirements to utilize models like MarkupLM (Li et al., 2022) or TIE (Zhao et al., 2022), we had to come up with our approach. Due to the small size of the training dataset, containing only 3200 rows, we also used the validation dataset for finetuning, which consists of 800 rows, to improve accuracy on every run.

heading: 3 System Overview
text: We initially started with fine-tuning BERTBASE uncased to gain fast insights into how parameters influence its accuracy.
Because of the different hardware setups available to us (see Table 1), we used BERTBASE and DeBERTaV3BASE on the smaller setup for pretesting purposes and DeBERTaV3LARGE to check for any improvements over the pre-tests on the most promising ones. Please mind that the values reported are the results achieved during training.
Runs with BERTBASE uncased First, we tested different sequence lengths on the article text (column "targetParagraphs").
With best performing on a sequence length of 512 (see Table 2), we moved on to combine the “targetParagraphs” column with other columns.
After conducting a thorough evaluation of the different combinations, we found that combining the "targetParagraphs" and "postText" columns gave us the best results (see Table 3).
Moving from BERTBASE to DeBERTaV3 The high level of performance achieved by our first approach demonstrated the effectiveness of fine-tuning BERT and encouraged us to continue exploring the potential of deep learning models for natural language processing. So we re-evaluated our findings with DeBERTaV3.
Both models achieved better results than our preevaluated ones (see Table 4).
After we noticed that articles from Huffington Post are by far the most significant cluster with over 774 (19.35%) articles compared to the next highest domain (see Table 5), we also checked the occurrences of tags in the dataset without Huffington Post articles as well as in the Huffington Post articles cluster (see Table 6).
We figured out two attempts at balancing the dataset: proportional balancing and tag-oriented balancing, which will be described in the following.
Proportional balancing For the balanced approach, the final portion was gathered by brute-forcing: running the model on the data with a sampled portion on half of all Huffington Post articles and further reducing this amount (see Table 7).
Our model performed best on a sixteenth with a balanced accuracy of 0.661 and a macro F1 score of 0.648, also better than our initial DeBERTaV3BASE run (Table 4). Therefore we tried this approach also with DeBERTaV3LARGE (Table 8). Unlike the improvements this adjustment made to our base model, the large one performed worse.
Tag-oriented balancing We recognized that in the cleared dataset (no Huffington Post articles), while passage- and phrasetagged articles make up 80.5%, only 19.5% are multi-tagged ones, with passage and phrase not diverging that much (6.9% compared to 23.1% between passage and multi and 17.2% between phrase and multi) (see Table 8). Due to this observation, we ran our DeBERTaV3BASE model on a combination of the cleared dataset and the dataset containing only Huffington Post articles limited to multi-tagged articles. For the record, we also ran on
adding the phrase- and passage-limited Huffington Post dataset (see Table 9). This apporach was not further tested with DeBERTaV3LARGE due to the lower results compared to the achieved balanced accuracy with 116 of Huffington Post.

heading: 3.1 Dataset enrichment with markup
text: While checking the domains and URLs, we found that 483 entities had no URL (see Table 5). Also, 689 were embedded articles from websites like "Yahoo" or archived versions of the original articles (which influences the markup structure making it noisier). Furthermore, while Huffington Post was overrepresented, it was the domain with the best reachable articles. So for fast evaluation of our markup hypothesis, we focused on crawling the source code for Huffington Post articles in the training and validation dataset. To make it utilizable within machine learning, we cleared the HTML code to take items in <p>- and <a>-tags only into account. This approach was directly tested with DeBERTaV3LARGE (see Table 10).

heading: 4 Experimental Setup
text: We evaluated three transformer-based models (for specifications, see Table 11), combining train and validation datasets. Utilizing BERTBASE to find the best working specifications, DeBERTaV3BASE to fast evaluate our balancing approaches (see Sys-
tem Overview and DeBERTaV3LARGE as final submission model on the most promising runs from DeBERTaV3BASE and our markup approach. We obtained the models from the Hugging Faces transformers library for PyTorch.
For our markup-based approach, we crawled the original articles using BeautifulSoup and cleared the results with a simple RegEx pattern (see section Dataset enrichment with markup for a more detailed description)
We submitted three DeBERTaV3LARGE models to TIRA, fine-tuned on the validation set, the validation set containing only 116 of all Huffington Post articles and the validation set enriched with markup for Huffington Post articles.

heading: 5 Results
text: As shown in Table 12, we could not surpass the baseline accuracy of 0.74 with any of our approaches achieving the highest accuracy with the combined approach. Comparing more specifically based on the three tags, we achieved a higher precision on multi-tagged articles with our fullcombined dataset model and the 116 HuffPost one. With 0.54 precision, the markup-enriched model does worse on a multi-tag prediction but achieves the highest precision on phrase-tagged ones. Finally, we see the highest recall of 0.79 for the fullcombined dataset and the 116 HuffPost run.
Compared to other submissions, we achieved the highest recall in detecting multi-part spoilers and got among the top 3 with the highest F1 score in classifying passage spoilers.

heading: 6 Discussion and Future Work
text: Since we had many mixed results, we recommend further balancing the dataset, especially to compensate for the overrepresentation of Huffington Post articles and the underrepresentation of multitagged articles. Since our markup-based approach resulted in the lowest accuracy, we assume this is because we only considered articles from Huffington Post. Also we only managed to focus on <p>and <a>-tag extraction which won’t cover all relevant aspects within webpage design. Since with MarkupLM and TIE, there are high-performing models to use for domains like webpages to solve prediction and classification tasks, we strongly recommend taking this into account for annotating clickbait articles in the future, always saving the pages’ entire markup within the dataset and avoiding embedded sources like "Yahoo" or archives like "archive.is" since they build markup around the embedded article and might leave out content like an advertisement which could hint to the position of the spoiler in the text.

heading: 7 Conclusion
text: We submitted several approaches, utilizing DeBERTaV3LARGE to solve the classification task of SemEval-2023’s Task 5. Although we could not quite reach the baseline’s accuracy we could point out weaknesses within the given dataset caused by the overrepresentation of Huffington Post articles and the underrepresentation of multi-tagged entries. Also, it misses providing the original markup, which we tried to crawl retroactively after we figured out that design is one of the critical parts of information placement on the website. This approach was not very successful, which might be because
we only crawled the markup for Huffington Post articles since they were best reachable at this time, only extracting <p>- and <a>-tags and therefore only covering a small aspect of markup design possibilities. After finding models like MarkupLM and TIE, which promise to perform better on markupbased presentations like webpages, we recommend re-annotating the dataset to use those.

heading: Future Work
text: Since with MarkupLM and TIE, there are high-performing models to use for domains like webpages to solve prediction and classification tasks, we strongly recommend taking this into account for annotating clickbait articles in the future, always saving the pages’ entire markup within the dataset and avoiding embedded sources like "Yahoo" or archives like "archive. is" since they build markup around the embedded article and might leave out content like an advertisement which could hint to the position of the spoiler in the text