Brooke-English at SemEval-2023 Task 5: Clickbait Spoiling
Year: 2023

Abstract:
The task of clickbait spoiling is: generating a short text that satisfies the curiosity induced by a clickbait post. Clickbait links to a web page and advertises its contents by arousing curiosity instead of providing an informative summary. Previous studies on clickbait spoiling has shown the approach that classifing the type of spoilers is needed, then generating the appropriate spoilers is more effective on the Webis Clickbait Spoiling Corpus 2022 dataset. Our contribution focused on study of the three classes (phrase, passage and multi) and finding appropriate models to generate spoilers foreach class. Results were analysed in each type of spoilers, revealed some reasons of having diversed results in different spoiler types. "passage" type spoiler was identified as the most difficult and the most valuable type of spoiler.

heading: 1 Introduction
text: According to the clickbait spoiling generate research paper (Hagen et al., 2022) (Fröbe et al., 2023a) (Fröbe et al., 2023b), clickbait spoiling generation is similar to the task of machine reading comprehension (MRC), where context is the full article/advertises with its title waiting for a reader. Question of MRC corresponds to the clickbait, which is a short text that arouses curiosity without providing much informative summary, aiming at attracting people to click into the link towards the context article. And the answer is a spoiling text that could satisfy people’s curiosity raised by the clickbait, therefore people could save the time of reading.
Detailed research about the task and its dataset has been conducted in the referring paper. In order to spoil its content, we are listing a couple of key points already verified by the paper and its dataset, and they would be considered as important priors in our experiments.
• All the answers can be located as a span in
the context, using extraction models could produce the correct answers.
• Classifying the spoiler into tree types "phrase", "passage" and "multi" is helpful to the span prediction, under the condition of having a good classifier, otherwise using an unclassified model is a better option.
• DeBERTa (He et al., 2020) model showed a general good performance for most of the spoiler-type and score-metric combinations, it is also a good spoiler-type classifier. Although "multi" type was not evaluated, DeBERTa QA model can be regarded as a good initial model for the task fine tune.
During data processing some flaw on target span positions were found, target fixing procedure was applied, details would be described in Section3.1. Due to this target fix procedure, direct result comparison between this paper and the previous research paper may not be appropriate. On the other hand, "multi" type spoilers are included in our experiment. Within a multi example, multiple text spans that are often locate in different paragraph lines of context. Considering the differences in dataset, training tasks and the starting pretrained model, we designed different features, module layers, losses and postprocesses for each type of spoiler.
Experiment results verified that three types of spoilers shall be trained separately with it’s own parameters. Using a classifier with accuracy 0.76, the overall benefit of separation training overcomes the drawback of misclassifing the spoiler and send it to a wrong spoiler extractor. There existed a distribution shift between validation and test dataset, due to a small dataset size and the random dataset splitting. Consistant gap of the measures between validation and test dataset were observed. The reason of observed test performance being much better
64
than validation would be discussed. Evidences and error analysis suggested "passage" type spoiler and BLEU-n may be the causes of troubles.
Experiment results showed mixing all types of spoiler in a training dataset would give lower measures on both BLEU-n and meteor than firstly predicting the spoiler type, then use separate models to extrace spoilers. Replacing the classifier with a hypothetical perfect classifier, would produce the best measures. Separating the spoilers would improve on the signal-noise ratio, which would be shown as a very important aspect of explaining performance differences between models.
The DeBERTa QA model had been pretrained for SQuAD, and was used as initial parameters of my models. Relative paragraph id was used as an additional feature for classifier and "multi" model. Contrastive learning losses were used for "passage", "multi" and "mix" models, because their span token lengths were larger in both mean and variance. Learning the start and end boundary of a longer spoiler, is less similar to a typical SQuAD answer, whereas boundary of a short spoiler (i.e. phrase) typically encloses an entity, which is more similar to a SQuAD answer. More modifications were applied to "passage" and "multi" models due to their smaller similarity answers towards SQuAD.
Inference and training codes are released at 1

heading: 2 Background
text: Table1 shows a clickbait-spoiler data example, context text is a concatenated string of title and paragraphs, clickbait is a question-like sentence, and its answer is the spoiler. The length of each paragraph is at regular sentence length, but the joined context could contain over 2k tokens, which is a length exceeding the typical max length of transformer encoders. Spoilers could appear anywhere in the context, the position of one spoiler span can be expressed in two ways:
1. as shown in the example, [[span start paragraph id, start offset within the paragraph], [span end paragraph id, end offset within the paragraph]]
2. span position within contxts, [start offset within joined context, end offset within joined context]
1https://github.com/ttssrr423/SemEval2023-Task5clickbait-spoiling
For "phrase" and "passage" spoiler types, the answer simply includes 1 span, their difference is mainly on token length. For "multi" spoiler, the answer consists of multiple spans, which then can be concatenated to make the answer after extracted. Most "multi" spoilers are extracted from different lines, but there are a few examples of nested spans.
Dataset paragraph spans has to be converted into context spans, then the context would be splited into shorter texts, called feature texts. Each feature text has a fixed number of tokens overlapping with its neighbour feature texts, in order to reduce the likelihood of having a ground truth span that starts and ends across two consecutive feature texts. The feature texts belonging to an example, are then appended after the example clickbait text, forming the complete input texts, like shown in Fig1
The pretrained DeBERTa QA model predicts the start and end position logits separately and by adding the logit scores sij = gi+gj , the span score which starts at token position i and ends at token position j can be calculated. Selecting the span with the maximum score gives the answer span. If the feature does not contain any answers, the [CLS] token position shall have the maximum answer, i.e. s00 > sij∀i > 0, j > 0.
SQuAD dataset was used to pretrain the baseline model, and my models are all fine tuned starting with this baseline version.2
In order to merge the results of all features belonging to a single example, a beam search mechanism was used in the baseline postprocess. Pseudocode can be used to describe the postprocess alg1: This postprocess works fine for single span results, but it need to be modified for "multi" spoilers.
For the spoiler type classification task, only the first feature of each example is used as the input to calculate the [CLS] feature, then use a FC classifier to predict the spoiler class.
2https://github.com/pan-webis-de/pancode/tree/master/semeval23/baselines/transformer-baselinetask-2
Algorithm 1: baseline postprocess Data: [(gstm, gedm ),...]: list of feature logits pair
1 empty reranking list ; 2 for mth feature in example do 3 sort gstm, g ed m in descending order ; 4 foreach (n, i = posn) in topN of gstm do 5 foreach (n, j = posn) in topN of gedm do 6 check for j > i > 0 and span text is valid; 7 calculate and adding score sij to reranking list; 8 end 9 end
10 end 11 return best scored span in reranking list;

heading: 3 System Overview
text: In our experiment, dataset repair was conducted before model training, then a single model extracting all types of spoilers, called the mixed model was trained. The mix model was tried with various combinations of input features and loss setups, but the overall result was not satisfactory.
Table2 is the results of the mix model on validation dataset after trained, with detailed bleu-n and meteor on each spoiler type. The performances indicates phrase spoilers are the most similar texts towards a typical QA answer, whereas the multi spoilers have the least similarity. Multi spoilers cannot be produced in the mixed model, it only predicts one answer span.
A classifier was trained, then grid search of confidence threshold of classifier was performed. By comparing the classifier’s predicted spoiler type confidence with the thresholds, we could decide whether a type-specified model should be used, and those examples with low type confidences goes into the mixed model. 5 models were trained, three for each individual type, one for mixed, and one classification model.
The three type-specific models all output a score matrix sij , storing the reranking scores of spans start at token position i and ends at token position j. Their input features, prediction heads, and postprocess are different, depending on their ground truth similarity with the original QA task, more modifications were applied if similarity is low. Multitask training setup was considered but not experimented due to the GPU memory was limited within 24G, so encoder layers do not share parameters.

heading: 3.1 Dataset repair
text: The dataset provided sometimes make mistakes between context span offsets and paragraph offsets, and it may also mismatch with the ground truth text. A fix process on the offsets and the token positions was performed, aligning them with the ground truth spoiler text. If the paragraph id and its spoiler span offsets extracts different text with the ground truth spoiler text, searching around the initial mistaken positions could often fix the mistake. But there were cases that global search had to be done in order to find a match. Matching the paragraph offsets to context offsets while keeping the track of paragraph ids and their absolute position ids within the paragraph requires much carefulness, mistakes can be easily made when the clickbait tokens and
the overlap tokens are introduced.

heading: 3.2 Spoiler type classification
text: For the token type classification task, adding the position embeddings after transformer layers gives a better accuracy, like suggested by the DeBERTa paper. A DeBERTa model accepts the first feature of an example, concatenate it with clickbait as the inputs, and trained with cross entropy is the classifier being used for the later extraction experiments. Absolute position ids of input text (xpos = [0, 1, ...Lcb, 0, 1, ..., Lt, 0, ..., Lp1 , ..., LpN ], where Lcb=length of clickbait, Lt=length of title, Lpn=length of n
th paragraph) was embedded, and then added after transformers’ last layer hidden out. henc(x) = DeBERTa(x) + eabs where eabs = Embedding(xpos) Accuracy of the classifier was around 0.76.

heading: 3.3 Phrase extractor
text: Phrase extractor is the most similar task with the pretrained QA task, so minimum adjustment in the model was made, score metric was identically calculated as sij = gsti + g ed j where g st i = logitsst(x) ∈ RT , and x = (xw, xr), representing sequence token words and token types. Unlike the classifier model, absolute position ids was not used, gst = Linearst(DeBERTa(x)), subscript st indicates the score of start of a span was calculated. Probabilities are pst = softmax(gst) and cross entropy loss is the phrase model objective. Lossphrase = CE(p
st, tgtst) + CE(ped, tgted) Phrase spoilers typically has only a few tokens, therefore relative position in DeBERTa could well capture the n-gram relations if n is not large. Reranking and select the top 1 non-empty span to obtain the answer in postprocess is identical with alg1.

heading: 3.4 Passage extractor
text: The answer span has a larger variance in terms of the span token length, lengths of passage spoilers could vary from as short as phrases to as long as a 200 token paragraph. Score metric was sij = g st i + g ed j , the absolute position embeddings were not used. Softmax supresses non-target logits with a strength proportional to the exponential of logits, this involves all the possible relative positions. However the useful supressions are the "top but wrong" logits, their gradients would be more important than the gradient of supressing the long tail logits, because the long tail logits would not
threaten the gound truth logit. A contrastive loss was added to the cross entropy loss, in order to emphasis supression of the important negative logits. Total loss is a weighted sum of cross entropy loss same as phrase model plus the contrastive loss.
Contrastive loss Jc = 1K ∑
i,j /∈C δKReLU(sij −
s(+)), where s(+) is a positive span’s score. Phrase type only has on positive span, the span score of ground truth sij=gt. For those span positions not in the whitelist C, choose the top K scores which are greater than ground truth score, then calculate their mean advantages over the ground truth. We hope the ground truth score would have the maximum score, hence achieving Jc = 0. δK = 1 if (i, j) ∈ {TopK Span ij set} else 0. Whitelist C was defined as fig2, it contains the ground truth span position as well as any substring span positions within the ground truth. Also it may include a safty zone/margin, which is 2 token positions neighbouring the start or the end of ground truth span. This margin forbids negative sampling from substrings of ground truth including neighbouring positions xstpos − 2 to xedpos + 2. For longer spans, the information extracted would not change much if we add an extra 2 tokens, and we should focus on supressing those spans which share little IOU with ground truth span, meanwhile keeping high sij scores. The top K negative samples being sampled are from "-ve sample" and "invalid" region. Any spans within the whitelist predicts a span that may not be the ground truth but still gives non-zero BLEU, meteor and BertScore during evaluation, so avoiding negative sampling from whitelist would reduces the noise.
The overall loss for passage model is a
weighted sum of the contrastive loss and the cross entropy loss, with weights of 1.0 in both terms. Losspassage = Jc + CE(pst, tgtst) + CE(ped, tgted)
Postprocess of passage model is similar as alg1, the topN loop was based on score {i(N)} = topN(argsorti(maxj(sij))) and then {j(N,{i})} = topN(argsortj(sij | i ∈ {i(N)})). It selects the topN starting i’s based on their endings with max span score, and then loop foreach selected i’s, select topN endings. Hence we would have N × N top score candidates in the reranking list. A final reranking of selecting top N from the list would produce N spans. This is a much quicker approximation of ranking all combinations of ij and then choosing top N results. When N=1, the process is identical as i∗, j∗ = argmaxi,j(sij), which is our desired passage span.

heading: 3.5 Multi extractor
text: Multi spoiler has the least number of examples in the dataset, but the most modifications was applied on the multi model. Many of the multi spoilers are enumerated items that extracted from different paragraphs, therefore paragraph id information should be incoporated into the model. Paragraph ids were defined as the following:
xp(t) =    −1 if 0 ≤ t < Lcb 0 if Lcb ≤ t < Lcb + Lt d if Lcb + Lt + ∑
1≤e<d Lp(e) ≤ t
(1)
Tokens in the clickbait with token length Lcb were labeled with -1, tokens in title with length Lt were labeled as 0, and tokens in the dth paragraph were labeled with d, starting from paragraph index 1. Clickbait, title and paragraphs are concatenated into a single sequence, whose sequence position was labeled as t.
Relative paragraph id positions between title and paragraphs were calculated, then their embeddings were looked up as e(ctx)mn , representing the relative paragraph id embedding of query token in mth paragraph of context, and key token in nth paragraph of context. Context included title and paragraphs, but no clickbait. the embedding for clickbait was a different trainable embedding e(q).
The relative attention score bias between query
position i and key position j was calculated as:
sattn(i, j) = 1√
(DH)
[ q(i)TWqkk(j)+
(Wstq(i)) T ep(i, j) + (Wedk(j)) T ep(i, j) ] (2)
where query, key vectors are slices from DeBERTa out hidden plus absolute position embeddings same as type-classify model, q(t) = k(t) = h(t) = DeBERTa(x)+eabs ∈ RDH . Wqk, Wst and Wed are linear layers with biases. The relative paragraph embeddings looked up is:
ep(i, j) =
{ e(q) if xp(i) < 0 OR xp(j) < 0
e(ctx)mn if xp(i) = m AND xp(j) = n (3)
The score modifier was made antisymmetric as: mij = sattn(i, j) − sattn(j, i). The output score was sij = gsti + g ed j +mij . The logits g st and ged are also calculated with absolute position embeddings added after deberta out, like in the spoilertype classifier.
A token would attendent to tokens not only in its neighbour positions, but also the tokens in its neighbour paragraphs, it would be benefitial for those spoilers locatd in a series of consecutive paragraphs, which is quite oftenly the case.
Cross entropy loss for "multi" was different, because there were multiple (start,end) spans, a regular softmax would mistakenly supress some of the ground truth logits. Hence a "multi" version of softmax was used:
Z− = ∑
t|tgt(t)=0 exp{gt} (4)
Z = ∑
t
exp{gt} (5)
Z− is the partition function for all negative logits, Z is the normal version of partition function. The multi-softmax probability is:
p(t) =    exp{gt}/Z if tgt(t) = 0 exp{gt}
Z− + exp{gt} if tgt(t) = 1
(6)
The multi-softmax ensures any positive logit does not supress the logit of any other positive positions. Also the probabilities are close to the the single-positive softmax probabilities if we ignore the other positive positions temporarily as their probabilities are not being evaluated. Although
multi version has a sum probability may not be equal to 1, the logp losses would not be affected much and therefore gradients would still descend at same direction. When there’s only one positive logit, the multi-softmax is identical as regular softmax. Cross entropy loss of span start becomes Lossstce = L∑ l=0 −δtgtlt log(p st(t)), where the "multihot" delta is 1 when the lth ground truth span starts at position t (or ends at position t for the ed version loss).
Contrastive loss is also different as we need a threshold to know how many spans are produced as the multi answers. The span score at CLS, s00 was chosen as the threshold. Negative samples are those span position pos−(i, j) = {(i, j) /∈ C} not in whitelist defined same as the "passage" model, and their contrastive loss is:
Jc,− = 0.5 ∗ ( max(ReLU(stopK− − s00))+
mean(ReLU(stopK− − s00)) ) (7)
TopK negative sampling is performed to focus on learning the difficult spans. Positive samples produces contrastive losses as:
Jc,+ = 0.5 ∗ (max(ELU(s00 − s+))+ mean(ELU(s00 − s+)))
(8)
A balanced contribution between max and mean losses was considered for the both positive and negative samplings, only considering max would be a more accurate design but more difficult to train.
The overall loss of multi model is Lossmulti = (Jc,+ + Jc,−) + 0.5 ∗ (Lossstce + Lossedce), the loss gives less weight on the multi-CE loss and more on contrastive loss compare with "passage".
Postprocess of multi spoiler is also quite different, top1 search cannot be used. If we use topN search, there would be another problem. The n candidate spans with highest scores may overlap to each other due to they have close start and end positions. We want to group similar answer spans, and choose the best answer foreach group, so that the final answers are texts with "orthogonal" contents.
A greedy clustering of candidate answer was designed as alg2
IOU of token count between current and previous spans were used as a measure of deciding whether a new answer line (or a new group) should be added. The complete postprocess for multi
Algorithm 2: multi span orthogonize Data: [(sp1, sp2),...]: list of topN spans in a
feature 1 empty groups list G; 2 for spn ∈ topN span list do 3 foreach group id ig ∈ G do 4 if ∃spo<n ∈ G[ig] and IOU(spo, spn)>0.6 then 5 append spn after group list
G[ig]
6 end 7 end 8 if spn not added to any previous group then 9 create a new group in G
10 end 11 end 12 set result list as empty ; 13 foreach group id ig ∈ G do 14 sptop1ig = maxsp.score(G[ig]) ; 15 add sptop1ig to result list ; 16 end 17 return result list ;
spoiler of an example is alg3. The input is a list of features belong to a same example like shown in Fig1, with subscript v representing its id.
The output of "multi" postprocess keeps at least 2 answer lines and at most 6 lines. Statistic showed most of the ground truth multi spoilers contains 2 to 6 answer lines. Spans belonging to any feature of the example are sorted with their scores srij , r is a loop index of O which is the example’s output of alg3. We try to keep those spans which are allowed to be slightly lower (the 0.9 factor) than the threshold of its feature score sv|r00 , in the hope of increasing the recall. The final Group∗ process is slightly different from the feature Group process in terms of Group∗’s IOU is calculated on text offsets instead of token position ids, this makes calculation easier outside the feature loop.

heading: 3.6 Mixed extractor
text: Mixed model is similar to the "phrase" model, but its loss is a combination of CE with multisoftmax (CEM) and contrastive loss. Lossmix = LossCEM (g st, tgtst) + LossCEM (g ed, tgted) +
0.5 ∗ Jc where Jc = 1K ∑
i,j /∈C′ δKReLU(sij −
max(s(+))). Noticing the difference of contrastive
Algorithm 3: postprocess for multi spoilers
Data: E=[(feat1, feat2),...]: list of features in an example
1 define Group(list) as alg2 ; 2 empty candidate list D; 3 for featv ∈ E do 4 calculate scores and spans (sv, spv) = modelmulti(featv) ; 5 calculate Fcand ={sp1:N}= topN list of "passage" postprocess ; 6 calculate Fgp=Group(Fcand) ; 7 foreach sp ∈ Fgp do 8 append (sp, s(v)00 ) to D 9 end
10 end 11 calculate O = Group∗(D) ; 12 sort O with span score ssp in descending ; 13 select top 6 answers from O := O(top6) ; 14 delete any spr ∈ O(top6) if sspr < 0.9sv|r00 until |O| ≤ 2 ; 15 rearrange O with ascending paragraph id; 16 return O
loss here and in the "passage" model, the positive scores are wrapped with a max, meaning we have abandomed predicting multi answers, in order to avoid the distruption of learning second best span. Only one best span would be produced in the postprocess. The score metric was sij = gsti + g ed j , where absolute position embedding was not used after DeBERTa output. The safty margin in whitelist defination was set to 0, to make the span boundary clearer for short "phrase" spoilers. For the short phrase spoilers, extra 2 tokens would make a great difference in BLEU and meteor, so whitelist only contains substrings from token start i = xstpos to end j = xedpos of ground truth.

heading: 4 Experimental Setup
text: 800 examples were splited into 400 for validation and 400 for test, whereas the training example has a number of 9138. Train batch size was 4 and gradient accumulation steps was 5. Max token sequence length was set to 384. Negative sample in contrastive loss chose topK as K = 20, postprocess in multi answer grouping chose topN as N = 30. During the training of all the models, the following techniques based on experience were used to improve training performance:
1. Lookahead optimizer was being used, with AdamW being the base optimizer, lr was 1e-5, weight decay was 1e-2.
2. Learning rates for different transformer layers were grouped and separately set. Layer 1 to 8 have a lower learning rate of LR/1.6, layer 9 to 16 have an unchanged LR, and layer 17 to 24 have a higher LR ∗ 1.6.
3. Linear schedule with warmup were used, with warmup step=60.
4. Adversarial training was applied after the first epoch, AWP had an attack lr of 1e-4.
5. Epoch parameter set with the best evaluation metric was chosen, with max train epoch num=10.
Differences of setups for training all the models are listed in Table3. Rows are the model options, including: 1) absolute token positions in each paragraph; 2) paragraph ids used to calculate relative paragraph embedding and the antisymmetric modifier; 3) multi-softmax in cross entropy; 4&5) weights of cross entropy loss and contrastive loss; 6) width of margin in terms of number of tokens when defining the whitelist; 7) max number of tokens of a single answer span allowed; 8) top N number during postprocess; 9) number of answer sapans after grouping similar answers.

heading: 5 Results
text: Type classification result is shown in Table5. Reason of such a big difference is due to the imbalanced "passage" examples in the two dataset, and we would discuss it later with more details. We would show test and validation results together as the evidences.
If the spoiler type can be perfectly predicted, which is just a hypothetical scenario, the performance of models for different types can be evaluated separately.
Metric B is BLEU-n, n is in range [1,4] and not exceeding the length of number of words in ground truth text. Metric M is meteor. Since there were only 400 in validation and 400 in test dataset, the metrics evaluated had a large variance. Results indicated that test dataset may contain less hard examples than validation dataset, therefore their results were better except for the phrase spoilers. There were not enough time for me to k-fold all the models to make a more convincing result, but the overall BLEU-n and meteor for "all" spoiler types (combined result for all 3 spoiler types using the classifier model) are larger in the test dataset, so overfitting did not occure. The column "mix" showed result of training all types of spoilers with one single model. Clearly separating the spoiler types and train a model for each type would increase the "all" performance in both datasets.
Using a classifier already trained, confidences of each spoiler type can be obtained. A grid search on
validation dataset of thresholds for "phrase", "passage" and "multi" was performed. If an example has its top1 type probability above the confidence threshold of the corresponding type, then the typespecified model would extract the answer, otherwise the "mix" model would extract the answer instead.
Thresholds of phrase=0.75, passage=0.45, multi=0.7 were chosen because it had a balanced good performance in validation dataset. Results of two-stage prediction using such a threshold were shown in Table7
Spoiler type counts and their type prediction confusion matrices using the [0.75, 0.45, 0.7] thresholds are shown in Table8 and Table9.
The large difference of performance between validation and test datasets was mainly due to the distribution shift. Validation dataset has a larger proportion of "passage" spoilers, therefore increasing the difficulty of making accurate type predictions. If we remove the thresholds and analyse
the confusion matrix without the "mix" model, we could discover that both "phrase" and "multi" are most likely to be misclassfied to "passage". Both "phrase" and "multi" examples had a proportion of 23% which were predicted as false "passage", whereas only 12% passages were predicted as false "phrase" or "multi". This implied "passage" data is naturally noisy in the sence that some of its examples showed moderate similarities with the other two types. Unlike the "phrase" and "multi" examles which showed good orthogonality in terms of type similarity, "passage" is more like a "mixture" of all types of spoilers. Some "passage" had a short spoiler text which is shorter than the mean length of "phrase" spoilers, therefore it is reasonable for the model to misclassify "phrase" and "multi" to a "passage" if confidence level is low. That explains why we need high confidence thresholds for "phrase"(0.75) and "multi"(0.7), these two models show much better meteor performance compare with "passage" model when they are extracting their own type of spoilers.
In Table6 we could conclude the "passage" had a similar noise level as the "mix", because "passage" had performances which were closest to "mix" model. Low confidence threshold of "passage"(0.45) reflected the fact that "passage" model was accepting the "noisy" (or hard) examples, and it was still a slightly better model compre to the "mix" in most cases.
Comparing the number of all types of spoilers in validation and test dataset, we could exaplain why test results are better than validation results in nearly all the tables, because validation has 170 passage spoilers out of 400, whereas test has a fewer number of 152. Even though best snapshots of models on the validation dataset was chosen, the deterioration of performances due to the distribution shift of having more "passages", is still prominent. Spoiler type prediction accuracy confirms the fact that validation dataset was harder to be correctly classified due to these extra "passage" examples.
We have explained the causes of bad performances of "passage" model on validation dataset, so bad case analysis was done on the errors. Fig3 showed the detailed distribution of measure metrics of examples predicted as "passage" type. The type labels in the figure are the ground truth types, the x-axis are measure metrics, y-axis are the example counts. Hence the primary reason of bad
performance of "passage" predictions in Table7 can be identified as wrongly classifying "phrase" and "multi" examples to a false "passage".
Table10 showed the bleu-n and meteor of predicted spoilers with the "passage" model for each ground truth spoiler type on validation dataset. When ground truth type is "passage", the results were close to the corresponding results in Table6, which are the correctly classified results. However if the ground truth type is "phrase" or "multi", their results would be significantly lower than the results in Table6. Summing the true and false "passages", the predicted bleu-n and meteor in Table7 would be produced, giving the worst performance model which was the passage type model.
Fig3 also revealed the fact that BLEU-n is not as smooth as meteor. The term "smooth" means the histogram of the metric is a more even distribution, rather than a polarized distribution, which has concentrated measure values at the low end [0,0.1] or the high end (0.9, 1.0]. This means BLEU-n is encoding less information of similarity between ground truth and predicted spoiler, because it has a lower entropy.
Human analysis of differences between BLEUn and meteor was done by sorting ∆ = abs (meteor −BleuN) in descending order for the "passage" predictions of validation dataset. Table4 showed top5 examples that the two metrics showed a largest divergence. The predicted passages tend to include the ground truth as a substring, and it is reasonable to penalize the tedious unwanted texts, but there are cases like "talk about their money" and "handling their money and talking about it with their partners", which convey essentially the same information only with slightly different way of saying. Silly mistakes for both metrics happens in short phrase spoilers. Meteor failed with single-token predicted and target spoiler, for an example textprd = textgt="ottoman", has bleu-n 1.0 and meteor 0.5. This directly caused blue-n larger than meteor value in phrase ex-
amples. Bleu-n failed with substrings such as texttgt="Mitzi Gaynor Beverly Hills, California" and textprd="Mitzi Gaynor", which has bleu-n 0.0 and meteor 0.4. Non-english names also failed for both measures, such as texttgt="Carrer Avinyo" and textprd="Carrer Avinyó", producing bleu-n 0.0 and meteor 0.25.
These are common mistakes made by metrics, the predictions are not absolutely wrong. Extracting a short phrase or passage from a long article would always encounter such a problem that a spoiler could have "isotops" which bring the same information but expressed differently. Largest common string was used during dataset repair, maybe it is also a good metric for short texts. Bert score may also be able to reduce these pseudo mistakes. It would be interesting to make a further study on designing a suitable metric.
Because of the deadline time and the GPU memory limitation, multi-task setup was considered but not experimented, it is reasonable to hypothesize that sharing the first few layers of encoder could be benifitial, further study would also be focused on this approach.

heading: 6 Conclusion
text: Clickbait spoiling generation can be regarded as an MRC problem, its model accepts similar input features as SQuAD. The spoiler answer types and the article paragraph layout information were important because these are newly added components towards a typical QA problem. Designs considering multi-paragraph answers produced the largest improvement in the "multi" model results using the mix as a baseline, this can be concluded by comparing Table2 and Table6, revealing improvements on bleu-n for phrase, passage and multi are 0.022, 0.115 and 0.285 respectively, and the improvements on meteor are 0.052, 0.093 and 0.159. Separating different types of models brings improvements in all models. Despite having the least similarity with the starting QA task and the least number of training examples, multi type spoiler extractor achieved the largest improvement.
Phrase type spoiler extraction is the easiest task, it has little improvement after the type splitting, and in most cases the absolute performance is the best among the all types. It is the most similar task as the starting pretrained task, experiments of adding extra input features, loss design or postprocess showed little influence on its performance.
The influences are too small to be distinguished within a 400 example 1-fold validation, k-fold experiments shall be done to make a convincing weak improvement, making phrase type spoiler task not as attracted as the other types.
Passage type spoiler is the most difficult in the sence that its data is the noisiest. There is not a clear feature indicating the spoiler should be a "passage" type instead of the other two types, and there is not a good general description of the passage ground truth texts. It is not a named entity, not a list of enumeration items, not a complete sentence, but it is any possible substring of the context. Therefore we encounter the most unknown inputs and outputs, results of passage model is the closest towards mixtype results, confirming the randomness of passage spoilers.
Since passage type has a relative large proportion in the dataset and a relative poor performance, more study effort was put on passage model. Fig3 revealed the most important drawback for the passage model is misclassifing spoiler type and use it to predict a "phrase" or "multi" example. Tuning the thresholds showed a trade-off between task generalization and specialization. If a model is good at a specific type of task, then as shown in Table10, its performance in a general task (where passage is the closest to the general mix) would be bad, hence we need a high threshold. Designing the training for a general task like passage spoiler extraction, is challenging and valuable, further study would be focus on this.
Bad case study also revealed some fails for both bleu-n and meteor for short spoilers with only a few tokens. For longer spoiler texts, changing the way of expressing and difference in degree of penalty of substring, are the two major causes of diversified results between the two measures. A measuring metric to bring the gaps closer shall be studied in future.

heading: Future Work
text: A measuring metric to bring the gaps closer shall be studied in future