jack-flood at SemEval-2023 Task 5: Hierarchical Encoding and Reciprocal Rank Fusion-Based System for Spoiler Classification and Generation
Year: 2023

Abstract:
The rise of social media has exponentially witnessed the use of clickbait posts that grab users’ attention. This paper presents our approach, which use different encoding techniques. We propose hierarchical encoding with count and document length feature-based model for spoiler type classification, which uses Recurrence over Pretrained Encoding. We also propose combining multiple ranking with reciprocal rank fusion for passage spoiler retrieval and question-answering approach for phrase spoiler retrieval. For multipart spoiler retrieval, we combine the above two spoiler retrieval methods. Based on the benchmark, the experimental results indicate that our proposed spoiler retrieval methods are highly effective in retrieving spoilers that are semantically very close to the ground truth spoilers.

heading: 1 Introduction
text: The rapid adoption of social media among the masses has pushed online publishers to stay relevant by making their content prominent on these platforms (Teixeira, 2014). This is mostly achieved through clickbait. This paper presents our proposed approach and detailed analysis for two subtasks of the clickbait spoiling challenge (Task 5) (Fröbe et al., 2023) Subtask 1: clickbait spoiler type classification, b) Subtask 2: clickbait spoiler generation. This contest is organized in the 17th International Workshop on Semantic Evaluation (SemEval 2023). This challenge addresses the issue of generating spoilers for clickbait. The importance of this task can be gauged from the fact that spoilers would preemptively warn the readers about the content of the clickbait posts, enabling the users to make an informed decision. This would also mitigate the spread of misinformation (Silverman, 2015) and the over-dependence of content publishers on sensational posts for prominence.
∗* Equal contributions
Our proposed methods for spoiler-type classification and spoiler generation rely on strength of pretrained language models such as BERT (Devlin et al., 2018), and its different variants like DeBERTa (He et al., 2021). We propose a hierarchical encoding-based model for spoiler-type classifications (subtask-1), which encodes different aspects of hierarchical information presented in the target paragraph along with document length and count overlap-based features. Considering the unique challenges and characteristics posed by passage, phrase and multipart spoiler generation, we adopt three different approaches for spoiler generation. We combine several ranking models using reciprocal rank fusion (RRF) for passage retrieval, a pretrained RoBERTa model over question answering task for phrase retrieval and combine methods of passage retrieval and phrase retrieval for multipart spoiler retrieval. Our code repository is available at1https://github.com/ thesujitkumar/jack-flood-at-SemEval-2023-Task-5_ Spoiler-Classification-and-Generation to recreate and reproduce the results presented in this paper.

heading: 2 Background
text: Initial studies (Blom and Hansen, 2015; Schaffer, 1995; Rony et al., 2017; Anand et al., 2017; Biyani et al., 2016; Agrawal, 2016; Chakraborty et al., 2016) in the literature focus on the detection of clickbait, which deals with identifying whether a post is clickbait or not. But such detection of clickbait does not impede the curiosity induced by clickbait. To fill this gap and overcome such limitations, the recent study(Hagen et al., 2022) curated a dataset for spoiler generation and proposed information retrieval and question-answering methods for spoiler generation, which helps to impede the curiosity induced by clickbait. Task 5 of SemEval 2023, Clickbait Spoiling (Hagen et al., 2022) aims
1Jack-flood code repository
1906
to generate spoilers for clickbait posts. Generating effective spoilers poses unique challenges, such as understanding the context of the title and its relevance with the post body. We propose novel approaches to solving this problem by harnessing the power of pretrained language models and the Reciprocal Rank Fusion technique to generate phrase, passage, and multipart-type spoilers.

heading: 3 Methodology
text: This section presents our proposed method for subtask 1: spoiler type classification, and subtask 2: spoiler generation.

heading: 3.1 Subtask 1: Spoiler Type Classification:
text: Given a pair of post text C and target paragraphs P , the task is to identify the spoiler type Y where Y ∈ {phrase, passage, multipart}.
Inspired by the superiority of hierarchical encoding in literature for large document classification (Yoon et al., 2019; Sun et al., 2021), we exploit the hierarchical structure of target paragraphs P by following the hierarchical structure of the target paragraph while encoding. We propose Recurrence over the pretrained Encoding RPE model. RPE captures the following key properties of hierarchical encoding (i) encoding of a sentence using a pretrained language model which includes BERT2 (Kenton and Toutanova, 2019), RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2021), which captures long-term dependencies between words within a sentence through multi-head attention component in the encoder of above-mentioned pretrained language models. (ii) target paragraph P consists of a sequence of sentences in which every sentence is contextually related to the previous and next sentence and conveys the sequential information present within the target paragraphs P . To capture the sequential dependencies among the sentences of the target paragraphs P . We apply BILSTM (Hochreiter and Schmidhuber, 1997) over encodings of sentences, which effectively capture the context of the sentence from left and right. By encoding the sequence of sentences in P from left to right, RPE captures the context of sentences on the previous sentence, while encoding from right to left, RPE captures the context of a sentence from the next sentence. (iii) The task of spoiler-type classification is heavily influenced by the word count in the spoiler; for example, if a pair
2BERT
of post text C and target paragraph P is classified in passage type spoiler then the spoiler is going to be few sentences in length, if it is classified in a phrase type spoiler then spoiler will be few words and in case of multipart it is going to list of sentences from multiple segments of the document. Considering such properties of the spoiler type classification, RPE also considers document length and word count overlap features. Figure 1 presents a block diagram of Recurrence over a pretrained Encoding RPE model. Given a pair of post text C and target paragraphs, P the RPE model first splits the P into set sentences P = {S1,S2, ...Sn}. Next, encoded representation of post text C and sentences Si are obtained using a pretrained language model as follows:
si= ( PLM ( Si ))
(1)
c= ( PLM ( C ))
(2)
Ideally, any pretrained language model can be used to encode Si and C but in this study, we use BERT (Kenton and Toutanova, 2019), RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2021). Considering the limited size of the available training dataset, we did not finetune BERT (Kenton and Toutanova, 2019), RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2021) for spoiler-type classification. Next, we apply BiLSTM (Hochreiter and Schmidhuber, 1997) over the encoded representation of sentences to obtain the encoded representation p of target paragraphs P using equation 3. The main motivation behind applying BiLSTM over the encoded representation of sentences is to capture the sequential dependencies among the sentences of P . Here, sequential dependencies essentially mean that every sentence in a target paragraph is contextually related to its previous and next sentence.
p=BiLSTM ( s1, s2, ..., sn ) (3)
Given a pair of post text C and target paragraphs P , document length and count overlap features are estimated as follows: (i) Document length features : We extract the count of unigrams, the number of sentences, punctuation marks, and paragraphs present in P . We also extract the same features for post text C. We form a feature vector d by concatenating the document length feature as discussed above. (ii) Count overlap features : To extract the count overlap feature vector o, we first
apply preprocessing over C and P to remove all stop words and special symbols, etc. Then we extract the frequency of each unigram from C in P . Count overlap features essentially measure the similarity between C and P in terms of the number of unigram overlap and their overlap frequency. Subsequently, we adopted two different approaches for spoiler classification, namely (i) document classification and (ii) similarity-based approach. In the document classification approach, we simply concatenate the encoded representation c and p of post text C and target paragraphs, P respectively, along with feature vector d and o using the equation defined below.
f= ( p ⊕ c⊕ o ⊕ d ) (4)
In contrast, similarity-based approach, we form a feature vector by estimating the angle and difference between c and p. This estimates how similar post text C and target paragraphs P are.
f= ( p ⊕ c ⊕ (p− c)⊕ (p⊙ c) o ⊕ d ) (5)
Where ⊕ represents the concatenation of two vectors and ⊙ represents the element-wise multiplication operation between two vectors. Next, we pass the feature vector f to a two-layer fully connected neural network for spoiler type classification. We use a cross entropy loss function to learn the parameters.

heading: 3.2 Subtask 2: Spoiler Generation:
text: The objective of this subtask is to generate or retrieve text from target paragraphs P , which satisfies
the curiosity persuaded by a clickbait post text C. Based on the length and type of text that needs to be generated or retrieved from P , the spoiler is classified into three categories, namely phrase spoiler, passage spoiler, and multipart spoiler. Each of these spoiler categories poses unique characteristics and challenges (Fröbe et al., 2023; Hagen et al., 2022). Considering such challenges, we adopt different approaches for phrase spoiler, passage spoiler, and multipart spoiler generations.

heading: 3.2.1 Passage Spoiler Retrieval
text: To extract the passage spoiler from target paragraphs P , we adopt an information retrieval approach. Initially, we apply a probabilistic model BM25 (Robertson et al., 1994; Trotman et al., 2014) by considering post text C as a query and different sentences of target paragraphs P = {S1,S2, ...Sn} as a set of documents. The objective is to retrieve the most relevant sentences as passage spoilers. Though BM25 is relatively fast and accurate, it is a bag-of-word-based model; hence it does not consider the sequential and contextual information present in P and C. It also fails to capture semantic similarity. To overcome such limitations of BM25, we consider pretrained language models such Sentence-BERT (S-BERT) (Reimers and Gurevych, 2019, 2020) and DeBERTa (He et al., 2021) to encode post text C and sentences of target paragraphs P and then estimate the semantic similarity between encoded vector representations of post text C and encoded vector representation of sentences of target paragraphs P . Finally, we select the top similar sentences of target paragraphs
P as passage spoilers. Though the above-discussed approach based on a pretrained language model and semantic similarity improved the performance of passage spoiler retrieval over the BM25 model, this approach lacks training and fine-tuning on information retrieval tasks. To overcome these limitations, we consider the pretrained model MonoT5 3 (Nogueira et al., 2020) on document ranking information retrieval task. Studies (Benham and Culpepper, 2017; Clipa and Di Nunzio, 2020) have shown that reciprocal rank fusion (RRF) (Cormack et al., 2009) is an effective method for combining document ranking obtained from different information retrieval systems. Studies (Benham and Culpepper, 2017; Clipa and Di Nunzio, 2020; Cormack et al., 2009) in the literature also suggest that combining ranks obtained from different information retrieval systems outperform any individual information retrieval system. Motivated by such observations, we combine the rank of sentences obtained using BM25, the pretrained language models with semantic similarity and MonoT5 using reciprocal rank fusion (RRF) (Cormack et al., 2009). Subsequently, we select the top-rank sentence from RRF as a passage spoiler. Figure 2 presents the working of the passage spoiler retrieval system. It first splits the target paragraph P into set sentences P = {S1,S2, ...Sn}. Next, encoded representation c and si of post text C and sentences Si of
3MonoT5 Source
target paragraphs are obtained using Equations 2 and 1 respectively. Here two different pretrained language models have been used, and based on that; we obtain two different rankings. We obtain the first ranking vector α of sentences in P with S −BERT and as pretrained language model in Equation 1 and 2 and Equation defined below.
α= ( c P⊺ ) (6)
Where P is a matrix of sentences encoding si, and ⊺ is a matrix, transpose operator. Similarly, we also obtain a ranking vector β by considering DeBERTa as a pretrained language model in Equation 1, 2 and following Equation 6. Next, we obtain the ranking of sentences γ using BM25 as defined in the equation below.
γ = ( BM25 ( C,S1, ...,S1 )) (7)
Similarly, we further obtain the ranking of sentences δ using Mono-T5 as defined in the equation below.
δ = ( MonoT5 ( C,S1, ...,S1 )) (8)
Next, we combine the ranking of sentences α, β, γ and δ using RRF as defined in Equation 9. Our RRF ranking Equation is similar to the equation defined in study (Cormack et al., 2009).
xi=RRF ( Si ∈ P ) = (∑
r∈R 1 k + r ( Si ) ) (9)
We set k=60 and number of ranking R= 4 namely α, β, γ and δ. We select the top sentence ranked by RRF as passage spoiler.

heading: 3.2.2 Phrase Spoiler Retrieval
text: The key objective in phrase spoiler generation is to extract the group of words from target paragraph P , which satisfy the curiosity induced by a clickbait post text C. Considering such challenging nature of phrase spoiler generation, we first apply the word-level information retrieval system discussed in subsection 3.2.1 to retrieve words from target paragraphs. But we observe that such a system retrieves words from different parts of target paragraphs P , which lacks correlations and context among them. Considering such limitations of a word-level information retrieval system, we apply a question-answering approach that considers post text C as a query and target paragraphs P as a corpus from where answers need to be extracted. We consider two pretrained language models trained on question-answering tasks, RoBERTA4 (Liu et al., 2019) and DeBERTa5 (He et al., 2021). We also tried a system by combining (concatenation, union or intersection) the output of RoBERTA (Liu et al., 2019) and DeBERTa (He et al., 2021) for phrase spoiler generation, but we observed that RoBERTA (Liu et al., 2019) alone outperforms DeBERTa (He et al., 2021) and combinations of RoBERTA (Liu et al., 2019) and DeBERTa (He et al., 2021).

heading: 3.2.3 Multipart Spoiler Retrieval
text: The main objective of multipart spoiler retrieval is to extract more than one passage or phrase from target paragraphs P to satisfy the curiosity persuaded by a clickbait post text C. These extracted passages or phrases could be non-consecutive or from different segments of target paragraphs P . Considering the characteristics of a multipart spoiler, that multipart spoiler could be either passages or phrases, or both. We combine our approach proposed in subsection 3.2.1 for passage spoiler retrieval and the phrase retrieval approach discussed in section 3.2.2 for multipart spoiler retrieval. Given a post text C and target paragraphs P first we apply our passage spoiler retrieval system presented in section 3.2.1 and apply the sorting over the rank generated by RRF and select the top k sentences. If any numeric figure is mentioned in post text C, then we
4Pretrained RoBERTa model on question answering task 5Pretrained DeBERTa model on question answering task
consider that numeric figure as the value of k; otherwise, we set the value of k to 5. Next, we feed each pair of post text C and sentence Si from top k selected sentences to pretrained RoBERTa model trained for question answering task as defined in blow Equations.
M1= ( RoBERT ( C,S1 )) (10)
Mk= ( RoBERT ( C,Sk )) (11)
Subsequently, we combine answers M1 to Mk generated by RoBERTa model as defined in Equations 10 and 11 for multipart spoiler generation.

heading: 4 Experimental Setup
text: Table 3 presents the details of hyperparameters used to produce the results presented in this paper. This study uses the dataset provided by (Hagen et al., 2022) to analyze the performance of proposed systems. Further details of experimental setup in presented in section A.2.

heading: 5 Results and discussion
text: Table 6 present the performance comparison between the different setups of the proposed model Recurrence over Pretrained Encoding RPE over the validation set. Extensive study and analysis of experimental results over validation set are presented in section A.3. Comparing the performance of different setups of RPE from Table 6 following observations can be made: (i) performance of the RPE model with the similarity-based approach is superior compared to the RPE model with document classification approach. This indicates spoiler types also rely on the similarity between post text and the target paragraph. (ii) performance RPE model with RoBERTa as the encoder is superior to RPE model with BERT and DeBERTa. (iii) adding feature F (count overlap and document length) boost the performance of the RPE model. This validates our intuition behind considering document length features and counts overlap features that spoiler types heavily rely on length. However, the RPE model with a similarity-based approach and features RPE(RoBERT, S, F ) outperform all other setups of the RPE model. Accordingly, we submitted RPE(RoBERT, S, F ) for evaluation over the test set. Table 1 presents the performance of the RPE(RoBERT, S, F ) model over the test dataset for spoiler-type classification.
Table 5 presents the performance of the different combinations of systems for passage, phrase, and multipart spoiler generation. As mentioned in 3.2.1 S − BERT has been used to encode a sentence for generating ranking score α, DeBERTa has been used to encode a sentence for generating ranking score β, ranking score γ is obtained using BM25 and ranking score δ obtained using MonoT5. Accordingly, in Table 3.2.1 S−BERT , and DeBERTa, BM25 and MonoT5 indicated that passage spoiler are obtained based on α, β, γ and δ rank respectively. Similarly, RRF indicates our proposed system for passage spoiler retrieval in Figure 2. RoBERTa and DeBERTa in phrase columns indicate that pretrained RoBERTa and DeBERTa models, respectively, over question answering task is considered for phrase retrieval. Considering the BSc. score of systems from Table 5, it is apparent that our proposed system is able to retrieve the passage, phrase, and multipart spoiler, which is semantically very close to ground truth. Considering the performance measures BL4 and MET from Table 5, it is evident that the performance of our proposed system is average. BL4 and MET, which relies on the token overlap between generated text and ground truth(Zhang et al., 2019), and BSc, which relies on semantic similarity between generated text and ground truth. Study (Zhang et al., 2019) also suggest token overlap based evaluation methods fail to consider meaningpreserving tokens and compositional diversity. Our error analysis for different performance measures for the spoiler generation task also suggests that the BERTScore is more suitable for comparing ground truth spoilers and generated spoilers. Section A.1 presents the details of our error analysis over different performance measures in spoiler generation tasks. It is apparent that though our system performance is average over BL4 and MET, our system is able to retrieve text which is semantically very close to or similar to spoiler ground truth. Table 2 presents the performance of our system (RRF for passage, RoBERTa for phrase and RRF+RoBERTa for multipart spoiler) over the test set. Considering the BSc score in Table 2 it is apparent that our system is able to generate a spoiler that is semantically very close to ground truth.

heading: 6 Conclusion
text: We propose a hierarchical encoding-based model recurrence over Pretrained Encoding RPE for spoiler-type classification. This paper adopts reciprocal rank fusion (RRF) for passage retrieval and adopt question answering based approach for phrase spoiler retrieval, and combine the passage and phrase retrieval model for multipart spoiler retrieval.Our results suggest that the proposed system effectively identifies spoiler types and retrieves spoilers from the target paragraph, which are semantically very close to the ground truth spoiler. In future work, we plan to explore extractive summarization methods for spoiler generation.

heading: A Appendix
text: A.1 Error Analysis of BERTScore vs BLEU-4 vs Meteor
Figure 3 presents a comparison between the performance of the BLEU-4 score and the BERTScore for comparing the relation between ground truth spoiler and generated spoiler. The following observations can be made from Figure 3. If we consider example 4, it is evident that the ground truth spoiler and the generated spoiler are exactly similar but the BLEU-4 score is low, and the BERTScore is 1. Examples 2, 3, and 5 show that generated spoilers convey the same meaning by describing and elaborating the phrase in ground truth spoiler but even in this case, the BLEU-4 score is low, and the BERTScore is high. Similarly, for example 1, it is evident that both ground truth spoilers and generated spoilers are semantically similar, but due to a lack of lexical overlap between them, BLEU-4 score is low, but the BERTScore is high. From Figure 3 and the above discussion, it is established that the BERTScore is more suitable for the evaluation of spoiler generation rather than other BLEU and Meteor scores.
A.2 Details of Experimental Setups, Experimental Datasets and Hyperparameters
Table 3 presents the details of hyperparameters used to produce the results presented in this paper. We use a cross-entropy loss function with a learning rate of 0.01 to learn the parameters. We consider at most 35 sentences in the target paragraph P . We discard the sentences if it P has more than 35 sentences and apply padded random vector if it P has more than 35 sentences. This study uses the dataset provided by (Hagen et al., 2022) to analyze the performance of proposed systems. The no of posts for each training, validation and test split are shown in Table 4. From Table 4, it is apparent that the dataset has an imbalance where the multipart class is over-represented by phrase and passage class. It is to be noted that no extra data was used for training our models. We used the validation
split to choose the most effective model for both tasks.
A.3 Analysis of Experimental Results on Validation Set
Table 6 present the performance comparison between the different setups of the proposed model Recurrence over Pretrained Encoding RPE over the validation set. Our proposed RPE model differs in three parameters: (i) encoding of post text C and sentences Si of target paragraph using BERT (Kenton and Toutanova, 2019), RoBERTa (Liu et al., 2019) or DeBERTa (He et al., 2021), (ii) document classification D or similaritybased approach S, (iii) F denotes that count overlap and document overlap feature are considered in the model. For example, RPE(BERT,D, F ): indicate that C, Si is encoded using BERT (Kenton and Toutanova, 2019), D indicates that document classification approach and F denote that count overlap and document length features are considered. Comparing the performance of different setups of RPE from Table 6 following observations can be made: (i) performance of the RPE model with the similarity-based approach is superior compared to the RPE model with document classification approach. This indicates spoiler types also rely on the similarity between post text and the target paragraph. (ii) performance RPE
model with RoBERTa as the encoder is superior to RPE model with BERT and DeBERTa. (iii) adding feature F (count overlap and document length) boost the performance of the RPE model. This validates our intuition behind considering document length features and counts overlap features that spoiler types heavily rely on length. However, the RPE model with a similarity-based approach and features RPE(RoBERT, S, F ) outperform all other setups of the RPE model.

heading: Future Work
text: In future work, we plan to explore extractive summarization methods for spoiler generation