Mr-Fosdick at SemEval-2023 Task 5: Comparing Dataset Expansion Techniques for Non-Transformer and Transformer Models: Improving Model Performance through Data Augmentation
Year: 2023

Abstract:
In supervised learning, a significant amount of data is essential. To achieve this, we generated and evaluated datasets based on a provided dataset using transformer and non-transformer models. By utilizing these generated datasets during the training of new models, we attain a higher balanced accuracy during validation compared to using only the original dataset.

heading: 1 Introduction
text: Titles of posts are increasingly utilizing exaggerations and other persuasive techniques to entice clicks. Such titles like the following are known as clickbait:
’Stranger Knocked Angrily on Her Door – What He Left Made Her Run to the Phone’
The Clickbait Challenge, hosted on Tira (Fröbe et al., 2023b), is a shared task that aims to address and satisfy the curiosity generated by this phenomenon. The challenge comprises two subtasks that utilize the post titles featuring clickbait and their corresponding articles as input. In the first task, three types of spoilers are to be distinguished, while the second task involves retrieving the spoiler from the article. For a detailed description of the tasks, please refer to the Overview Paper (Fröbe et al., 2023a) or the official Clickbait Challenge website (Clickbait).
We concentrated on optimising the given dataset to provide better training data for the models we created. In the following, we will describe our approaches to obtaining new datasets. We then evaluate our generated datasets and compare them to the original datasets. Our approaches for task 1, clickbait classification, using a non-transformer and a transformer-based approach are then described. After that we present our model for task 2 and show how we retrieve the spoilers for the clickbait titles.

heading: 2 Expanding the Dataset

heading: 2.1 Dataset Generation
text: The original dataset provided to us (Hagen et al., 2022) contained 3200 examples of training data. All examples consisted of a clickbait post, the article text, the spoiler type and the corresponding spoiler. Table 1 presents an overview of all utilized and created datasets, including their corresponding source file sizes.

heading: 2.1.1 QA Dataset
text: Our initial approach was to use existing data and format it for our purposes. For example, question answering datasets (QA) are appropriate for the task at hand. In QA datasets, the question can be understood as PostText and the answer as a spoiler. The length of the spoiler then in turn determines the spoiler type. To create our own suitable MixQA (MixQA Dataset) dataset, several QA datasets 1 were combined. The largest and best known dataset in this series is the SQuAD2.0 (Rajpurkar et al., 2018) dataset.

heading: 2.1.2 Pegasus Dataset
text: Data generation through paraphrasing is a process where new, diverse and meaningful samples are generated from existing data by changing the wording or sentence structure. One approach we tried to generate new data was Pegasus (Zhang et al., 2019), which is a transformerbased language generation model developed by Google. We used a pretrained Pegasus model (Pegasus Paraphrasing) fine-tuned on the paraphrasing task. The model is given a post title and generates a different but semantically similar title based on that. 16000 samples were generated, which together form the Pegasus (Pegasus Dataset) dataset.
1(Joshi et al., 2017) (Su et al., 2016) (Talmor and Berant, 2018) (Rajpurkar et al., 2018)
88

heading: 2.1.3 T5 Parrot Dataset
text: Like Pegasus, T5 (Text-to-Text Transfer Transformer) is also a deep learning based natural language processing model developed by Google Research. Through its text-to-text architecture, T5 can be easily fine-tuned for specific tasks without having to adjust the input or output layers. The Parrot model (Damodaran, 2021) we used is based on T5 and fine-tuned for paraphrasing short sentences with less than 32 tokens. These around 6600 samples give us the Parrot (Parrot Dataset) dataset.

heading: 2.1.4 Clickbait Generation Dataset
text: In the given training dataset, 84% of samples contained a so called "humanSpoiler". These are spoilers that were not extracted but handwritten. We tried using them to generate completely new titles from the paragraphs and spoilers with a question generation model. Those kind of models take in a context and the corresponding answer and generate a question from that. With this model (Romero, 2021) we were able to generate new clickbait titles based on the context and the spoiler. These around 6300 generated samples form the Q_Gen (Q-Gen Dataset) dataset.

heading: 2.1.5 GPT Dataset
text: Another approach for generating data was the utilization of GPT-3 (Brown et al., 2020), one of the most powerful language models available. More specifically, OpenAI’s Babbage GPT-3 model was used with the following prompt: ’Reformulate the following sentence as a clickbait headline:’. The dataset generated was named GPT (GPT Dataset). A sample of 1000 data points were examined and selected by human reviewers. The re-formulated post title would be selected if it keeps most of the features of the original (numbers, key nouns etc.) and still points in the general direction of the desired answer. This resulted in a final dataset of 234 instances, named GPT234_S (GPT234_S Dataset).

heading: 2.2 Dataset Evaluation
text: Our aim was to identify and provide the final model with the best dataset. To evaluate the performance of the various datasets for transformers, the DeBERTa-Large (He et al., 2021) model was fine-tuned using a single output layer. The results are presented in Table 2 and visualized in Appendix 2, which shows the difference from the baseline of the datasets. Due to the class imbalance present in all datasets, we focused on balanced accuracy as it provides a more accurate representation of the model’s performance. The results indicate that excessive amounts of artificially generated or augmented data can be detrimental to the model accuracy, as shown by the GPT dataset where a limited number of data points had a positive effect, but as the number of data points increased, the effect became negative. This suggests that the transformer approach is highly susceptible to changes in the dataset composition, even when generated data constitutes only a small portion of the complete training data. This implies that it is crucial to be very selective while incorporating generated data and to carefully consider the proportion of generated data in comparison to the original data. Overall, the GPT and Parrot datasets performed well, with GPT achieving the best results and being further improved through the manually selected GPT234_S dataset. Notably, a lot of improvement was observed without the need for human post-selection.

heading: 2.3 Outliers and Uncertainty
text: After initial analysis, the provided dataset appeared to have outliers that we aimed to remove. Outliers have a correlation to uncertainty and can often be identified with uncertainty quantification meth-
ods (Gonsior et al., 2022). We decided to use an Ensemble with the same model architecture as the dataset evaluation, trained on the provided validation data only, to evaluate the various training data. Uncertainty was measured by the logits deviation of each data point produced by the Ensemble. The uncertainty was normalised on the provided train split and visualised with the other datasets in Figure 1.
The data with the largest uncertainty, i.e. the suspected potential outliers, were removed and not used for training. This approach had no positive effect on the provided training split. There was either no negative effect of the outliers or they were not correctly detected by the uncertainty detection. Nevertheless, the uncertainty distributions showed how different the datasets are from the original and that similar datasets to the original like GPT234_S and more distant ones like Parrot can both work.

heading: 3 System Overview and Results

heading: 3.1 Task 1 Random Forest
text: We initially avoided using a transformer model and chose a Random Forest using the framework scikit-learn (Pedregosa et al., 2011). The extracted features for the Random Forest can be divided into 4 groups: general features, the TfIdf matrix of the PostText and the targetTitle, information from the Natural language processing with SpaCy (Montani et al., 2022) and regex patterns.
The general features were values like the largest number in the postText or the number of words. The Natural Language Processing features were based on the tagger labels, the parser labels and the Named Entity Recognition labels. We counted the number of tokens with the respective labels as well as the number of combinations of the labels. We also extracted the tense of the verbs as a feature. With the regex patterns we extracted word combinations like a number followed by "things" or “will ... you” as a feature.
With all these features we train 3 Complement Naive Bayes Classifiers (Rennie et al., 2003), each of which classifies a post into phrase or not, passage or not and multi or not. Each input feature is scaled by its maximum absolute value. The results of these 3 classifiers also serve as input for the forest.
We feed the 18011 features as input to a Random Forest Classifier. The Random Forest consists of 100 decision trees and measures the quality of a split using the Gini impurity.
We utilized this method to evaluate our datasets by training 100 Random Forests for each dataset and determining the average balanced accuracy score. By exclusively utilizing the provided train split, we achieved an average balanced accuracy of 58.6% during validation. We observed that in-
corporating the provided train split, Pegasus, Parrot, and GPT243_S datasets resulted in an improved average validation balanced accuracy score of 59.5%. On the validation dataset, our most successful model obtained a balanced accuracy of 61.6%, while on the test dataset, it achieved a balanced accuracy of 57.8%.
Interesting to note is that compared to the transformer approach, the Random Forest model exhibited significantly more robustness when dealing with new data and did not exhibit negative effects from the incorporation of a large amount of new data.
Moreover, Random Forest has a notable advantage in terms of transparency, as it enables us to extract feature importance or output the tree structure. In addition, compared to transformer-based approaches, Random Forest is significantly less computationally expensive.

heading: 3.2 Task 1 Transformer
text: For our final transformer model two dropout layers, one normalization layer and two linear layers were added to the pretrained DeBERTa-Large (He et al., 2021) model. As a result of the dataset evaluation, this model employed our best performing GPT234_S dataset. By utilizing the provided train split alone, we were able to attain a validation accuracy of 73.25%. After incorporating the GPT234_S dataset, we achieved a better validation accuracy of 75.125%. Our dataset contributed to achieving a balanced test accuracy of 69.1% in the end.

heading: 3.3 Task 2 Transformer
text: Deepset’s deberta-v3-large-squad2 (Deepset, 2022) model was fine-tuned using the extractive Question Answering Trainer architecture developed by HuggingFace (Wolf et al., 2020) with a slight modification in the post-processing hyper-parameter, extending the maximum acceptable answer length. In the example code available on Colab provided by HuggingFace, the value of max_answer_length was modified from 30 to 50 in the postprocess_qa_predictions function. The aim, similar to task 1, was to enhance results by experimenting with different datasets. However, the evaluation presented in Table 3 revealed that all datasets tested had a negative impact on overall performance, including those that were successful in task 1 such as GPT234_S and Parrot. As a result, the final model was trained on the original provided dataset without using the tag attribute, leading to an improvement
in validation BLEU-score from the given transformer task 2 baseline of 0.382 to 0.448. The positive effect arises only from the Deepset transformer and the hyper-parameter settings. The reason for lower performance with data augmentation is uncertain and requires further investigation. One potential explanation is that the strategies used may not be appropriate for task 2, resulting in poor quality training data due to their dissimilarity from the original clickbait.

heading: 4 Conclusion
text: Throughout this project, we have experimented with various methods to retrieve spoilers from clickbait titles and post texts in order to close the curiosity gap. These methods included a non-transformer based Random Forest, a custom DeBERTa-based model for task 1, and another transformer model for task 2. However, our primary focus was not on the individual models themselves, but on generating datasets to enhance model robustness and improve the accuracy of our results. By utilizing datasets generated from converting existing question-answering datasets, paraphrasing, question generation, and GPT reformulation, we were able to demonstrate significant improvements in the proposed approaches for task 1. Unfortunately, for unknown reasons, we did not observe any improvements in task 2 with the generated data. Instead, our task 2 results improved through fine-tuning hyperparameters in the transformer-based approach we described.
We demonstrated that the transformer approaches are extremely sensitive to even small amounts of newly generated data, whereas the Random Forest model could handle large amounts of new data with ease while still reaping benefits. Furthermore, our uncertainty analysis demonstrated that both more conservative and imaginative approaches to data generation can prove to be effective.
We are of the opinion that our results can be further enhanced by integrating more diverse data
generation techniques and better post-selection of the generated data, perhaps even through automatic means.
Limitations
First, we acknowledge that our dataset expansion efforts focused exclusively on a single dataset, which may have inherent biases due to its limited data sources. Although some of our expansion techniques yielded improved model performance on this dataset, it remains unclear whether these gains will generalize to other datasets. Moreover, the size of the expanded dataset was determined empirically and was only tested on this one dataset. It is possible that additional data augmentation could further improve model performance or that certain expansion methods are particularly suited to this dataset. For example, it is conceivable that GPT3 may be better suited for clickbait augmentation due to its training on internet data. Additionally, we cannot account for the differential performance between the two tasks employed in this study. While task 1 classification performed well, task 2 retrieval did not, and the reasons for this discrepancy are unclear. Future studies should examine the suitability of our dataset expansion techniques on a variety of sources, and consider the composition of a representative final training set for specific tasks.
Ethics Statement
In this paper, we augmented the original dataset with the intention of improving the performance of the models trained on it. The use of these models is outside of our control, but our purpose for augmenting the data was solely to enhance model performance and did not have any malicious intent. Our trained models were only used to address two tasks. Spoiling clickbait or news posts in general may discourage users from engaging with or seeking out diverse perspectives and sources of information. While some posts have short and definitive answers, some are ambivalent and need context, which can get lost in the retrieved spoilers. The datasets generated and used in this paper may contain inaccuracies or harmful content that have not been reviewed by humans. Training models on this data may result in incorrect understanding and incorrect outputs.

heading: Acknowledgements
text: The authors gratefully acknowledge the GWK support for funding this project by providing computing time through the Center for Information Services and HPC (ZIH) at TU Dresden.

heading: A Appendix

heading: Future Work
text: Future studies should examine the suitability of our dataset expansion techniques on a variety of sources, and consider the composition of a representative final training set for specific tasks