Spoiler Detection as Semantic Text Matching
Authors: Ryan Tran
Canwen Xu
Julian McAuley
Venue: EMNLP 2023 Main (2023)
Keywords: dataset
spoiler detection
semantic text matching
TL;DR: We propose a new task that formulates spoiler detection as semantic text matching.

Abstract: Engaging with discussion of TV shows online often requires individuals to refrain from consuming show-related content for extended periods to avoid spoilers. While existing research on spoiler detection shows promising results in safeguarding viewers from general spoilers, it fails to address the issue of users abstaining from show-related content during their watch. This is primarily because the definition of a spoiler varies depending on the viewer's progress in the show, and conventional spoiler detection methods lack the granularity to capture this complexity. To tackle this challenge, we propose the task of spoiler matching, which involves assigning an episode number to a spoiler given a specific TV show. We frame this task as semantic text matching and introduce a dataset comprised of comments and episode summaries to evaluate  model performance. Given the length of each example, our dataset can also serve as a benchmark for long-range language models.

Spoiler Detection as Semantic Text Matching
Ryan Tran', Canwen Xu', Julian McAuley
University of California, San Diego (rhtran, cxu, j euccdd. edu
Abstract
Online community comments
Juliet actually kissed him??
Engaging with discussion of TV shows on- line often requires individuals refrain from consuming show-related content for extended periods to avoid spoilers. While existing re- search spoiler detection shows promising results in safeguarding viewers from general spoilers, fails to address the issue of users abstaining from show-related content during their watch. This is primarily because the def- inition of a spoiler varies depending on the viewer's progress in the show, and conventional spoiler detection methods lack the granularity capture this complexity. To tackle this chal- lenge, we propose the task of spoiler match- ing, which involves assigning an episode num- ber to spoiler given specific TV show. We frame this task as semantic text matching and introduce dataset comprised comments and episode summaries to evaluate model perfor- mance. Given the length of each example, our dataset can also serve as benchmark for long- range language models."
The music in ept is besutiful 5 replies
Ep.
replies
Spollar
Julist actually kiseed him??
can discuss without worrying about spoilere!
The music ept is beautiful 5 replies
Inslewant
[HIDDEN SPOILER] replies
12
Predictions
The task of spoiler matching. The model matches the comments that discuss plot to the corre- sponding episode summary.
Figure I:
spoiler. However, for long shows with large number of episodes, this proves unsatisfactory: A user could be halfway through a show but will still be afraid to click on spoiler-tagged content for fear that it might contain spoilers for events later in the show when in reality, it might pertain to events that the viewer has already seen and with which they can engage. Again, some websites allow users to tag spoilers with more granularity, but it is far from guaranteed that users will be both accurate and con- sistent in tagging their content. This highlights the need for automatic spoiler matching. Unlike tradi- tional spoiler detection (Boyd-Graher al., 2013; Wan et al., 2019; Chang et al., 2021; Wröblewska et al., 202 1), which determines whether a comment is spoiler, spoiler matching aims to match given spoiler to episode number. A spoiler matching model working hand-in-han with spoiler detec- tion model could provide much more fine-grained protection from spoilers.
1 Introduction
Many online social platforms (e.g., Reddit, Dis- cord) provide opportunities for fans to discuss a particular TV show and share their thoughts about details in episodes. However, engaging in such dis- cussions comes with the risk of spoilers, which may lead to unsatisfactory viewing experiences (John- and Rosenbaum, 2015). As result, many viewers avoid these communities altogether until they have caught up with the latest episode of the show.
As an attempt to resolve this problem, some plat- forms (such as Reddit) have built-in functionality allowing users to tag their content containing
In this work, we consider the setting where the show is known, and would like to determine the episode which comment is referring, shown in Figure 1. Specifically, we pose the problem as a semantic text matching (Cer et al., 2018) task between comments and episode summaries. We ob- tain a high-quality evaluation dataset by manually
"Equal contribution.
'Code and model weights are publicly available at https: //github. ///gtthu.ccmm/bb/// wewpttrreeeeeermneeeee
http///huggggaace.../////// co/ "The data is available at https://

---

Summary
After thinking back Yor's training, Anya her "killer move" and throws the ball at Bill. However, the ball hits the ground and bounces loward Bill, who throws the ball right back and hits her Bill and his team were excited, thinking he was going to get Stella Star. However, Henry informs them that they do give out Stella Stars for simple P.E. game...
Comments
Relevant- Same episode; "Anya: 'Finisher strike: Star Catch Arrow!" Ball: 'nah, i don't 't really feel like it'
Relevant - Different Episode: "The dog finally has a name. Borf!"
Irrelevant: "This episode was fun. Just joy from start to end."
Irrelevant: "Haven't 'tt been this hyped over dodgehall game since Hunter x Hunter."
Table I: Example comments with example summary from the show Spy x Family. The first two comments are relevant; The first one corresponds to the same episode as the summary while the second does not, so they are examples of positive and negative example (respectively) during matching training. The third and fourth comments are irrelevant and are filtered out of the dataset during the auto-labeling step.
labeling the posts with the corresponding episode. To obtain a larger training set, we use prompt learn- ing to efficiently fine-tune an auto-labeler model to automatically label another 200k examples, in ad- dition to 5.9k manually labeled examples. The task is to match the comments to the correct episodes, with pairs of (episode summary, comment) as the input. As the number of episodes in a show is often limited, this task form is practically feasible. Also, as the median length of the concatenated summary and comment is 1,538 tokens, can be a good dataset for benchmarking long-range Transformer models (Beltagy et al.. 2020; Zaheer et al., 2020; Xiong et al., 2021).
Method
F1
Recall
Precision
AUC
Fine-Tuning LM-BIF (2021)
79.75
78.45
81.09
90.23
81.09
84.55
77.90
90.92
LM-BFF Ensemble
81.62
86.18
77.51
91.56
Table 2: Test set performance of the relevant/irelevant auto-labeler. For each model, the threshold that resulted the highest score on validation set chosen and used to compute the test set F1, recall, and precision.
based the discussion thread from which they are scraped, but we do not yet consider them labeled at this step. This is because we find that about half of the comments scraped this way are irrelevant. We take this time to note the subtle difference labels at this step compared to traditional spoiler detection.
2 Spoiler Matching Dataset
Data Collection We crawl 522,991 comments across 13 TV shows from discussion threads on Reddit. We also crawl 496 episode summaries from their respective episode pages on the website Fan- dom.3 Reddit threads have a hierarchical discus- sion format where users reply to each other's posts, and each reply increases the indentation level. Top- level comments comments that are not made in reply to any other comment. Threads focused around discussion of a particular episode are hand- picked by the human annotators, and the top-level comments are scraped. We take only top-level com- ments to minimize collecting comments that are incomplete thoughts continued from another part of conversation. We then remove links and markup clements and apply other cleaning preprocessing before proceeding.
While traditional spoiler detection classifies text as spoiler or non-spoiler, we at this step look separate the irrelevant comments we have scraped from the relevant ones. We define relevant com- ment as one that describes events from the episode (discussion thread) from which it is scraped. Exam- ples of irrelevant comments are discussions about music, acting quality, personal feelings about the episode, etc. Table shows some examples. While the first irrelevant comment is straightforward, the second is a more nuanced example: The episode in question is about dodgeball, but the comment does not discuss any events that occurred in the episode, so it is not considered relevant. Finally, we manually label 11,032 comments, reserved for the validation and test set.
Automatic Labeling To obtain a larger-scale
Manual Labeling After data cleaning, we group the comments by show name and episode number
training set, we split the 11 11,032 manually labeled examples into a small training, validation, and test set by the ratio of 7:2:1, 1, to train an auto-
Phttps: cottp:://fandom.co

---

labeler. To maximize data efficiency with this small dataset, we perform prompt-base basee fine-tuning to train RoBERTa (Liu et al., 2019). Specifically, we use R.M-FF (Gao et al., 2021) to fine-tune the model to predict if the comment is relevant.
#Tokens
#Examples
25%
50%
75%
Summary
496
1136.5
1537.5
2035.25
Comment
210,367
18
28
51
We evaluate LM-BFF under two settings. For the
Table Statistics the Spoiler Matching Dataset.
first, perform prompt-based fine-tuning on set of hand-crafted templates, then select the best one according to validation set AUC. For the second, we use the automatic template search (auto-7) to generate and rank 40 templates based on validation AUC. Under both settings, we do not perform the auto-L search and instead fix the label words as "relevant" and "irrelevant". From this ranking, the top 20 templates selected, and the logits from the corresponding models averaged. Table shows the evaluation of various auto-labelers. We find LM-BFF with template ensembling to be ef- fective for our use case. Critically, a higher recall score means more irrelevant comments caught and filtered, resulting in cleaner downstream training set for matching.
Human Labels Only
w/ Auto-labels
Method
dev
test
dev
test
BM25
38.90
36.80
40.67
36.80
RoBERTa
54.63
31.03
50.40
35.19
Nyströmformer
56.74
33.83
49.23
40.16
BigBird
62.75
33.28
54,25
46.22
MaxP-RoBERTS
63.42
40.44
58.63
54.09
Longformer 65.57
42.71
64.40
61.09
Table 4: MRR of the baseline methods on Spoiler Matching Dataset. Note that the development sets of the settings are different thus the numbers on the development sets are not comparable.
ther in the validation nor the training set. The remaining 2,787 hand-labeled comments are used for validation. The statistics of the resulting dataset are shown in Table 3.
is important note that this step, summaries are not concatenated with comments before being fed to the auto-labeler; the auto-labeler predicts rel- evant/irrelevant based on the words the comment alone. This is because the job of the auto-labeler is to filter out generic irrelevant comments; unlike matching, episode-speciff context is not required to perform this task.
3 Experiments
Task Formulation Our task is to match the com- ment to a certain episode in the show. Since there are only limited number of episodes in a show, we iterate through all episode summaries of the show and concatenate each episode summary with the comment with special token (e.g., [SEP]) in- serted. After inference for each summary-commen pair, we rank the episodes by the predicted match- ing scores. We use Mean Reciprocal Rank (MRR, Craswell, 2009) as the metric for evaluation.
Using our LM-BFF Ensemble auto-labeler, the 511,959 unlabeled comments are auto-labeled, sep- arating 204,47, 445 relevant comments from 307,484 irrelevant ones. Using our test set, we estimate that about 12% of the relevant comments are actually ir- relevant. As we will demonstrate later, this number is low enough that the auto-labeled comments still serve effective training set for fine-tuning spoiler matching model.
Baselines We select 6 baselines, including BM25 as lexical matching baseline, RoBERTa (Liu al., 2019) as a pretrained language model baseline, and three long-range pretrained Transformers: Big- Bird (Zaheer et al., 2020), LongFormer (Beltagy et al., 2020), and Nystromformer (Xiong et al., 2021). We truncate the summary if the input exceeds the maximum tokens allowed for each model. For RoBERTa, we also experiment with the MaxP passage aggregation strategy (Dai and Callan, 2019). All models base size. We imple- ment the training and evaluation pipeline with Hug- ging Face Transformers (Wolf et al., 2020). The modely are fine-tuned with the Adam W optimizer,
Dataset Construction To recap, we have 496 episode summaries, 511,959 auto-labeled com- ments and 1 11,032 hand-labeled comments. Among the auto-labeled comments, we have 204,475 rel- evant comments and among the hand-labeled we have 5,892. Relevant comments are converted to the spoiler matching dataset format by assigning them the episode number of the discussion thread from which they were scraped. To test the ability of the matching models to generalize to unseen shows, the test set is constructed such that it contains 3,505 land-ld comments from shows that are nei-

---

Rank
Correct
Prediction
Matching Score
Comment
23
21
0.1031
Great santa still alive
2
1
0.5822
Kars the ultimate lifeform is released from his stone inprisonment !!!! really really want Anya to have a dog she can communicate with. If she has a pupner with the ability to predict the future.......
11
13
0.8801
Table 5:
Longformer predictions three validation set examples. For each given comment, the first column represents the rank of the correct episode; the second is the correct episode number; the third is the episode with the highest matching score; the fourth this highest matching score, and the fifth the text of the comment. The score is the caaiss confidence after softmax.
a batch size of 32 and learning rate of 2c-5.
Family understand behaviors of the models." 4 Table lists several challenging comments along with Longformer's output. The first example refers to scene where Santa very briefly flies across the sky in the background. It is treated as unim- portant, fantastical garmish episode: None of the characters acknowledge it, so is understood to have not actually occurred. Thus, the episode 21 summary does not mention at all. This represents a class of comments that references relevant but obscure events, which only recent viewer of the episode might remember. In- terestingly, the score is relatively low, suggesting that the model understands that it is outputting low-quality prediction.
Settings We experiment with two settings to ify the effectiveness of the auto-labeling. In the human labels-only setting, we re-split the valida- tion set to a small training set of 2,229 comments and validation set of 558 comments, The second setting uses 204,475 automatically labeled training examples and 2,787 manually labeled examples for validation. The two settings share the same test set, which contains 3,105 manually labeled examples from 4 unseen shows.
event from the
Experimental Results Experimental results are shown in Table 4. In the human labels only setting, BM25 outperforms language models except MaxP- RoBERTa and Longformer. This finding is con- sistent with prior studies on text retrieval (Thakur et al., 2021) that BM25 can be a strong baseline when the training examples are insufficient. Com- pared with the human labels only setting, auto- labeling successfully improves model performance, verifying the effectiveness of our auto-labeling pipeline. Also, observe that RoBERTa suf- fers from short context length and underperforms BM25 under both settings. This especially evi- dent considering the large improvement resulting from the adoption of the passage aggregation strat- egy. Indeed, MaxP-RoBERTa is competitive with the long-range models. Among the long-range models, Longformer (Beltagy et al., 2020) achieves the best performance and outperforms the other models by large margin. We hope our dataset can also serve as benchmark for long-range pretrained language models, as the task requires interactions with long sentences.
The second comment describes an event from the show but references character from an en- tirely different show; it is drawing a comparison between two characters, one of them in the show, based on physical likeness. The reference is fairly well-known within the community, but without ad- ditional external information, would be difficult for the model understand this comment beyond just context clues.
The third comment is challenging because it con- cerns predictions. Comments ongoing shows often contain predictions, and if they happen to be correct, will likely match better lexically to the future episode when the events occur than the cur- rent episode when they are foreshadowed/prediitte This is not necessarily a bad thing for the end user, but poses a challenge for training and evaluating our models. For this example, it is visually hinted in episode 11 that the dog has the ability to see the future but not confirmed until episode 13, so the summary for episode 11 does not mention this ability explicitly but the summary for episode 13 does, posing possible explanation for the model's behavior.
4 Case Study
this section, we analyze comments from two shows in the validation set, Dr. Stone and Spy x
"This section contuins spoilers from these two shows.

---

Taken together, these examples give a glimpse into the challenges posed by spoiler matching. The hope is that this analysis motivates new lines of work into the study.
Zhuyun Dai and Jamie Callan. 2019. Deeper text un- derstanding for IR with contextual neural language modeling. In SIGIR, pages 985-988. ACM.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACI/UCNLLP (1), pages 38 3816-3830. As- sociation for Computational Linguistics.
5 Conclusion
In this work, we define new task of spoiler match- ing that formulates spoiler detection seman- tic text matching task and construct large scale dataset with 223k comments and 496 episode sum- maries by mixing human and automatic labeling. We benchmark the performance of BM25 and four language models on the proposed dataset.
Benjamin K Johnson and Judith E Rosenbaum. 2015. Spoiler alert: Consequences of narrative spoilers for dimensions of enjoyment, appreciation, and trans- portation. Communication Research, 42(8):1068- 1088.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1907.1 1692.
Limitations
One limitation of our work is that due to resource restrictions, we only benchmark four language models on our dataset, leaving many other long- range language models untested. Also, our dataset only covers 13 shows and the comments annotated by human annotators are relatively limited.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark zero-shot evalua- tion of information retrieval models. arXiv preprint arXiv:2104.088 00660666..6..
Mengting Wan, Rishabh Misra, Ndapa Nakashole, and Julian J. McAuley. 2019. Fine-grained spoiler tection from large-scale review corpora. ACL (I). pages Linguistics.
Acknowledgement
Association for Computational
The authors would like to thank Ching Tan and Minh Nguyen for their help with annotating the validation set.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-ther -hee------he-rttt- natural language processing. In EMNLP (Demos), pages 38-45 45. Association for Com- putational Linguistics.
References
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The onn-ddocumen transformer. arXiv preprint 2000:20000.05500.
Jordan L. Boyd-Graber, Kimberly Glasgow, and Jackie Sauter Zajac. 2013. Spoiler alert: Machine learning approaches to detect social media posts with revelatory information. In ASIST, volume 50 of Proc. Assoc. Inf. Sci. Technol., pages 1-9. Wiley.
Anna Wróblewska, Pawel Rzepiński, and Sylwia Sysko- Romańczuk. 2021. Spoiler in textstack: How much transformers help? arXiv preprint arXiv:2112.12913.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespeddes Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint 1rXiv:1803.11177
Yunyang Xiong, Zhampeng Zeng. Rudrasis Chakraborty. Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021. Nyströmformer: A nyström-based algorithm for approximating self-attention. In AAAI, pages 14138-14148. AAAI Press.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, and Amr Abmed. 2020. Big bird: Trans- formers for longer sequences. In NewrPPS.
Buru Chang, Inggeol Lee, Hyunjae Kim, and Jaewoo Kang. 2021. "killing me" is not spoiler: Spoiler detection model using graph neural networks with dependency relation-aware attention mechanism. In EACL, pages 3613-3617. Association for Computa- tional Linguistics.
Nick Craswell. 2009. Mean reciprocal rank. In Ency- clopedia of Dutabase Systems, page 1703. Springer US.


Reviews:
[{"review_id": "9L9E0tk1Kd", "reviewer": "Reviewer_SNpa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments.\n\n1. **Re. Unbalanced dataset splits**: We intended to manually label all data for the test set to ensure the quality of data for evaluating the models. For shows in the training set, we only label a small portion of them and use auto-labeler to complete the rest. As few-shot and zero-shot methods are more and more common, we believe that our decision to allocate more effort to test set annotation aligns with this line of research.\n2. **Re. Summarization for RoBERTa**: Thanks for your suggestion. Truncation is a simple strategy and default behavior for libraries including Hugging Face Transformer. We agree that a more complex strategy can improve the performance over truncation. We used MaxP [Dai et al., 2019] to aggregate the scores over passages from the episode summary. This aggregation strategy improves the performance by a large margin. We believe such methods can also be competitive to long text Transformers and our benchmark can be a testbed for these methods.\n\n| Method       | Human Labels Only (dev) | Human Labels Only (test) | w/ Auto Labels (dev) | w/ Auto Labels (test) |\n|--------------|-------------------------|--------------------------|----------------------|-----------------------|\n| RoBERTa      | 54.63                   | 31.03                    | 45.14                | 34.99                 |\n| MaxP-RoBERTa | **63.42**               | **40.44**                | **57.71**            | **53.69**             |\n\n3. **Re. Precision &  recall**: Thanks for your suggestion. We’d like to note that since there is only one relevant item, MRR is equal to mean average precision (MAP). We will update our table to include separate precision and recall numbers.\n\n----\n**[Dai et al., 2019]** Deeper Text Understanding for IR with Contextual Neural Language Modeling", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Spoiler often leads to unpleasant experiences during shows-related discussions on social media, spoiler detection studied in this paper is an interesting yet complex problem yet to be properly addressed.\n- A dataset with a mix of hand-labels and auto-labels is constructed, and will benefit the related studies in the community.\n- It sounds reasonable to formalize spoiler detection tasks into semantic text matching problems. ", "reasons_to_reject": "- The test set contains 3105 hand-labeled relevant comments from 4 unseen shows, yet only 5,892 hand-labeled relevant comments available for the 13 shows in total. Does this mean the data set is highly biased or imbalanced at least? If yes, this would make the evaluation results less convincible, wouldn't it?\n- Summary truncation could lose some key facts making the matching results incomplete for RoBERTa, but some passage aggregation strategy (e.g., rrf) could easily mitigate this in table 4.\n- Although the problem has been formalized into a text matching problem, precision and recall still sound more meaningful than MRR as evaluation metrics, I'd be more interested in seeing those numbers in Table 4.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes to formalize the spoiler detection task into a semantic text matching problem, and demonstrated the feasibility of applying off-the-shelf LM-based text matching models on this tasks. The major contribution is the problem formalization and the corresponding dataset construction. A dataset of 5,892 hand-labeled and 271,881 auto-labeled relevant comments to 496 episodes of 13 tv shows is collected, and the authors indicate this dataset, together will the code and model will be made public.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oFgfR6MXlP", "reviewer": "Reviewer_W1pv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We would like to clarify our design choices and hope this can address your concerns.\n\n1. **Re. Ranking setting**: We agree that adding “irrelevant” as a label instead of filtering them out is feasible. However, the decision to filter out the irrelevant posts is to align our dataset with a classic text matching/ranking setting where each text example is matched to a passage (i.e., episode summary in our paper). \n2. **Re. Episode context for auto-labeler**: Thanks for your suggestion. We did try to add episode context when training the auto-labeler and it has a negative impact on performance. (1) One possible reason is as shown in Table 4, adding episode context will make the sequence too long and introduce noise. (2) Adding the episode context could lead to overfitting the seen episodes. \nThe results of an auto-labeler trained with episode context are shown in the table below. As recall is the metric of interest for filtering out irrelevant examples, adding episode context does not improve the quality of data.\n\n| Method                 | F1    | Recall | Precision | AUC   |\n|------------------------|-------|--------|-----------|-------|\n| LM-BFF                 | 81.09 | 84.55  | 77.90     | 90.92 |\n| LM-BFF+Episode Context | 80.82 | 80.08  | 81.57     | 91.84 |\n\n3. **Re. Poor performance of the auto-labeler**: (1) Our auto-labeler can indeed improve performance of the matching model by a large margin, as shown in Table 4, which verifies its effectiveness. (2) We will release our raw data and annotations clearly marked as “auto label” or “manual label” so researchers can develop their own auto-labeler. Meanwhile, our test set stays the same for consistent evaluation, as all data are manually labeled. \\\n\tWe’ve also run an experiment to ensemble the auto-labeler models. The results of the ensemble of 20 models are shown in the table below. Ensemble can slightly improve the auto-labeler’s performance.\n\n| Method            | F1    | Recall | Precision | AUC   |\n|-------------------|-------|--------|-----------|-------|\n| LM-BFF            | 81.09 | 84.55  | 77.90     | 90.92 |\n| LM-BFF (ensemble) | 81.62 | 86.18  | 77.51     | 91.56 |\n\n4. **Re. Novelty concerns**: We would like to highlight the contribution of our dataset, which is suitable for long-form text matching. We would also like to point out that in production, using generative LLMs (e.g., ChatGPT or GPT-4) for text matching is financially infeasible, as every comment-summary pair requires one call of the API. That being said, as all test examples are manually labeled, our dataset could be a benchmark to evaluate few-shot learning for LLM research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well explained. \n2. Case study identifies good challenges in the task.\n\nSince this is a dataset paper, I am recommending an accept.", "reasons_to_reject": "Due to the following concerns I am proposing a rejection for this paper.\n\n1: There is no need to make this a two step solution - first identifying irrelevant / relevant and then identifying episode numbers. One could either solve it as a multiclass problem with \"irrelevant\" or \"other\" being one class if we were doing this in traditional ML ways. in the LLM world, same prompt can address both these cases (identifying whether comment is relevant and matching comment to episode).\n\n2: Their claim that auto-labeler identifying irrelevant comments does not need episode specific context is invalid. In fact, an argument can be made that their performance (F1 80.09) can be improved by adding this information. \n\n3: Poor performance of auto-labeler - Given the scope of the problem, I believe 77% precision and 81% recall are too low specially since this is being used as auto-labeler. Techniques like ensemble, prompt engineering, active learning all could have been applied to improve this. \n\n4: I am concerned about the novelty of the approach itself - in essence this is a multiclass semantic matching problem easily solvable through few shot prompt engineering. \n\nUpdate: \nthe authors have addressed the comments in their rebuttal, I encourage them to add these explanations in the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces the problem of identifying spoilers in discussion platforms like Reddit. The problem posed here is not one of detecting a spoiler but matching a spoiler to the episodes. Their key contributions are:\n\n1: Creating a spoiler matching dataset.\n2: Create a two step algorithm for identifying irrelevant comments first and then identifying spoilers from among them. \n3: Benchmark performance of BM25 and four other models.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I suggest a re-write of the abstract. Personally it sounded a bit convoluted and hard to follow.", "Justification_for_Ethical_Concerns": "N.A."}, "extra_text": {}}, {"review_id": "0jlpWWIBKw", "reviewer": "Reviewer_qhzf", "summary": "", "questions": "I like the paper very much! The descriptions are bright and clear to read. I have no questions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your endorsements!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- novel and interesting NLP task and dataset\n- a smart procedure for preparing the dataset with a very interesting description\n- also, the dataset has another usage for long-range language models, which was tested and described in the paper\n- very interesting and convincing result analysis with examples", "reasons_to_reject": "I have not found any :-)\n", "questions_to_authors": "I like the paper very much! The descriptions are bright and clear to read. I have no questions.", "paper_topic_and_main_contributions": "The paper defines a novel NLP task of spoiler matching, which links a particular spoiler to a related specific episode from a TV show; moreover, it presents a dataset for this task (text episodes' summaries and related user comments). ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I have not found any.", "Typos_Grammar_Style_and_Presentation_Improvements": "I have not found any."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes the creation of a spoiler-to-episode matching dataset. Different from previous datasets which focused on spoiler detection as a binary task, this dataset offers the possibility to match datasets with specific episodes. This makes a novel and valuable contribution to the task of spoiler detection.

I would suggest fleshing out the paper with extra details as suggested by reviewers, adding more evaluation metrics (precision and recall) and more details on how the data was sampled.

I would also suggest to make the contribution of a novel dataset (not just a new task) more explicit in the introduction.

Does the current paper title really convey the new task / dataset that it proposes?
Decision: Accept-Main