A Novel Contrastive Learning Method for Clickbait Detection on RoCliCo: A Romanian Clickbait Corpus of News Articles
Authors: Daria Mihaela Broscoteanu
Radu Tudor Ionescu
Venue: EMNLP 2023 Findings (2023)
Keywords: clickbait detection
low-resource language
Romanian corpus
contrastive learning
TL;DR: We introduce a novel Romanian clickbait corpus and a new contrastive learning approach based on title and content pairs.

Abstract: To increase revenue, news websites often resort to using deceptive news titles, luring users into clicking on the title and reading the full news. Clickbait detection is the task that aims to automatically detect this form of false advertisement and avoid wasting the precious time of online users. Despite the importance of the task, to the best of our knowledge, there is no publicly available clickbait corpus for the Romanian language. To this end, we introduce a novel Romanian Clickbait Corpus (RoCliCo) comprising 8,313 news samples which are manually annotated with clickbait and non-clickbait labels. Furthermore, we conduct experiments with four machine learning methods, ranging from handcrafted models to recurrent and transformer-based neural networks, to establish a line-up of competitive baselines. We also carry out experiments with a weighted voting ensemble. Among the considered baselines, we propose a novel BERT-based contrastive learning model that learns to encode news titles and contents into a deep metric space such that titles and contents of non-clickbait news have high cosine similarity, while titles and contents of clickbait news have low cosine similarity. Our data set and code to reproduce the baselines are publicly available for download at https://github.com/dariabroscoteanu/RoCliCo.

A Novel Contrastive Learning Method for Clickbait Detection on RoCliCo: A Romanian Clickbait Corpus of News Articles
Daria- Broscofeanu
Radu Tudor Ionescu
Department Computer Science
Department Computer Science
University of Bucharest
University of Bucharest Romania raducu. ionescu@gmail com
Romania
dariaäroscoteenuggmaa..cc com
Abstract
Potthast et al., 2016, 2018: Pujahari and Sisodia, 2021; Wu et al., 2020; Yi al., 2022), Chinese (Liu et al., 2022), Hindi (Kaushal and Vemuri, 2020), and Turkish (Genç and Surer, 2023), among others. However, for low-resource languages, such as Ro- manian, there is comparatively lower amount of research studying clickbait detection (Fakhruzza- man et al., 2021; Vineze and Szabó, 2020), primar- ily due to the lack of annotated resources. In this context, we introduce novel Romanian Clickbait Corpus (RoCliCo) comprising 8,313 news samples which are manually annotated with clickbait and non-clickbait labels. To the best of our knowledge, RoCliCo is the first publicly available corpus for Romanian clickbait detection. The news articles were collected from six publicly available news websites from Romania, while avoiding overlap- ping publication sources between our training and test splits. This ensures that the reported perfor- mance levels are not artificially increased because of models relying on factors such as authorship or publication source identification.
To increase revenue, news websites often re- sort to using deceptive news titles, luring users into clicking on the title and reading the full news. Clickbait detection is the task that aims to automatically detect this form of false adver- tisement and avoid wasting the precious time of online task, to the best of our knowledge, there is no publicly available clickbait corpus for Ro- manian language. To this end, we introduce novel Romanian Clickbait Corpus (RoCliCo) comprising 8,313 news samples which are man- ually annotated with clickbait and non-clickbait labels. Furthermore, we conduct experiments with four machine learning methods, rang- ing from handcrafted models to recurrent and transformer-base neural networks, to establish a line-up of competitive baselines. We also carry out experiments with a weighted voting ensemble. Among the considered baselines, we propose novel BERT-based contrastive learn- ing model that learns to encode news titles and contents into deep metric space such that titles and contents of non-clickbait news have high cosine similarity, while titles and contents of clickbait news have low cosine similarity. Our data set and code to reproduce the baselines are publicly available for download at https:// gi thub...cc wtw/dao wtw.arrbbbbroootee/rc.....o
Despite the importance of the
It is important to clearly separate between satiri- cal, fake and clickbait news. Satirical news repre- sents a type of fake news, where the intention of the author is to make the readers laugh. In satiri- cal news, it is clear to the readers that the reported events are fictitious. Fake news reports fake events or facts, where the intention of the author is to de- ceive the readers into belicving that the fake events or facts are true. Clickbait news represents a dis- tinct category where the news title is deceptive with respect to the content. In some clickbait news, the content can present real events and true facts. In other clickbait news, the reported events facts can be fake. Hence, these three categories (satirical, fake, clickbait) are distinct, although their intersec- tion not necessarily empty.
1 Introduction
Clickbait detection (Biyani al., 2016; Potthast al., 2016), the task of automatically identifying misleading news titles, has received a raising inter- est in recent years (Dong et al., 2019; Fakhruzza- man et al., 2021; 1; Indurthi et al., 2020; Kumar et al., 2018; Liu et al., 2022; Pujahari and Sisodia, 2021; Rony et al., 2018; Vorakitphan et al., 2019; Wu et al., 2020; Yi et al., 2022) due to its importance in addressing concerns regarding the deceptive behav- ior of online media platforms aiming to increase their readership and revenue an unethical man- ner. The task has been widely explored in mul- tiple languages, such as English (Agrawal, 2016;
We perform experiments with several machine learning models. Because some recent challenge reports (Chakravarthi et al., 2021) showed that shal-

---

low and handcrafted models can sometimes sur- pass deep models, we include both handcrafted and deep models as baselines for RoCliCo. Hence, two baselines are based on handcrafted features that are commonly used for clickbait detection, while the other three are based on deep learning architectures, such as Long Short-Term Mem- ory (LSTM) network (Hochreiter and Schmidhu- ber, 1997), and two versions of Bidirectional En- coder Representations from Transformers (BERT) (Devlin al., 2019) for Romanian (Dumitrescu et al., 2020). Among the considered baselines, we propose a novel BERT-basce contrastive learning model that learns to jointly encode news titles and contents into deep metric space. The embedding space is constructed such that titles and contents of non-clickbait news have high cosine similarity, while titles and contents of clickbait news have low cosine similarity. Thus, the similarity between the title and the content of an article leveraged as a clickbait indicator. Finally, the list of models eval- uated on RoCliCo is completed by an ensemble of the five individual baselines, which is based on weighted voting.
data samples for Hindi (Kaushal and Vemuri, 2020). There are three corpora containing more than 30K samples, but these comprise only tweets. Corpora formed of full news articles usually have less than 10K samples. With over 8K samples, RoCliCo is similar in size with the resources available for other languages. However, most of the other cor- pora are based on distant supervision, labeling the news as clickbait based on the source platforms they belong to. In contrast, our corpus is based on manual annotations. We note that news articles from a certain source platform cannot be placed in the same bucket, although it is true that differ- ent platforms use clickbait in different percentages, For RoCliCo, these percentages vary between 17% and 71 %. supporting our previous statement.
To the best of our knowledge, there are no pub- licly available data sets for Romanian clickbait de- tection. We thus consider RoCliCo as a useful resource for the low-resource Romanian language.
Related methods. To date, researchers proposed various clickbait detection methods, ranging from models based handcrafted features (Potthast et al., 2016) deep neural networks (Fakhruz- zaman et al., 2021; Liu et al., 2022; Wu et al., 2020). However, most of these approaches are based on standard learning objectives, little atten- tion being dedicated to alternative optimization cri- teria, such as those based on contrastive learning (i et al., 2022; López-Sánchez et al., 2018). Yi et al. (2022) trained a Variational Auto-Encoder jointly predict labels from text and generate text from labels using a mixed contrastive loss, while López-Sánchez et al. (2018) employed contrastive learning to minimize the distances between head- lines from the same category and maximize the distances between headlines from different cate- gories. Our model learns a metric space between titles and contents, while existing contrastive learn- ing methods learn metric space between titles, which is fundamentally different. To the best of our knowledge, we are the first to propose a con- trastive learning objective based on title and content pairs. The novelty of our approach consists of di- rectly transforming each news article into pair, essentially eliminating the need of performing hard example mining (Suh et al., 2019).
In summary, our contribution is twofold:
We introduce the first publicly available corpus of Romanian news articles compris- ing manually annotated clickbait and non- clickbait samples.
We propose novel clickbait detection model based on learning deep metric space where both news titles and contents are jointly repre- sented. In the shared embedding space, titles and contents of non-clickbait news are sup- posed to be neighbors, while titles and con- tents of clickbait news should be far apart.
2 Related Work
Related corpora. Clickbait detection was first studied for the English language, several English resources being available to date (Potthast et al., 2016, 2018). With the rising interest towards solv- ing the task, researchers started to compile and publish clickbait detection corpora for various lan- guages, including Chinese (Liiu et al., 2022), Hindi (Kaushal and Vemuri, 2020), Hungarian (Vincze and Szabó, 2020), Indonesian (Fakhruzzaman et al., 2021), Thai (Klairith and Tanachutiwat, 2018) and Turkish (Genç and Surer, 2023). These resources vary in size, ranging from 180 data samples for Hungarian (Vincze and Szabó, 2020) to 52,000
Related tasks. Although fake news and clickbait
detection are sometimes studied jointly (Bourgonje et al., 2017), we would like to underline once again that fake news detection and clickbait detection are

---

distinct tasks, i.e. clickbait title does not neces- sarily imply that the respective news article is fake. For example, an incomplete title might be sufficient to attract readers, only to reveal the full (and less at- tractive) story in the content, while presenting only true facts. Still, we would like to acknowledge that related tasks, namely fake (Busioc et al., 2020) and satirical (Rogoz et al., 2021) news detection, have been studied for Romanian.
note that the title's inclusion in one of the following genres may also represent an indicator for its mem- bership to the clickbait category: political, finance, celebrity. Consequently, we instructed the annota- tors to be extra careful when they have to annotate a news article from the politics, finance, or celebrity categories. These genres are picked by studying Romanian news websites and finding that the most common themes likely to contain clickbait are the ones above. To assign the non-clickbait category, even one of the above criteria fits for the title, the body and its similarity with the headline determine the final separation line between the two classes. The final label for a sample is decided based on the majority vote of three annotators, which reduces la- beling errors by a large extent. The Cohen's kappa coefficient among annotators is 0.73, indicating substantial agreement between annotators. We confirm the consistency of the labels by checking the assigned labels for a subset of 200 samples. Af- ter completing the labeling process, we found that the clickbait ratio per source ranges between 17% and 71 %, indicating that all the considered news sources seem to use clickbait titles to some extent. This is likely because the chosen sources do not use a subscription-based publication format, and the news publishers have to rely on advertisement to obtain earnings.
3 Corpus
RoCliCo gathers news from six of the most promi- nent Romanian news websites (Cancan, Digi24, Libertatea, ProTV, WowBiz, Viva), while only con- taining articles from the public web domain, be- ing free of subscription. The corpus consists of 8,313 samples distributed over two classes: 4,593 non-clickbait and 3,720 clickbait. As the corpus is manually labeled, each sample contains a title, a body, and a label that deems it clickbait or not by taking into consideration the characteristics of both text and title. There are 3,942,790 tokens in our data set, the average number of tokens per article being roughly 474. The average title length is 21 tokens, while the average content length is 454 to- kens. As expected, the titles are much shorter than the contents. In terms of the number of sentences, the news articles vary between 3 and 448. The average number of sentences per news article is 28.
We separate the news platforms between the training and test sets, keeping the news from Can- can, ProTV and WowBiz for training and those from Libertatea, Viva and Digi24 for testing. As such, the training set consists of 6,806 articles, while the test set comprises total of 1,507 articles. The distribution of classes per subset is presented in Table 1. We underline that the training set in- corporates more even data distribution across the two classes, providing sufficient examples to repre- sent both classes and to ensure smooth learning process. contrast, the test data represents a more natural distribution, with the non-clickbait class being dominant at a ratio of 2.4:1.
All data samples were labeled by annotators who received detailed instructions on the factors that should be taken into account when labeling news as clickbait or non-clickbait. The annotators are native Romanian speakers, who graduated from their bachelor studies at Romanian universities. We checked their reading comprehension skills prior to enrollment. Each annotator was asked to read an instructive document written in Romanian, where we provided definitions, guidelines and examples of clickbait and non-clickbait news articles. The labeling process, completed by three annotators, consists of filtering each sample through a set of criteria by identifying the properties of the title and the relationship between the body and the headline (title). As clickbait titles tend to misguide or at- tract readers by overemphasizing certain aspects, we recommended annotators to take into consid- erations the following characteristics: containing questions for the reader, teasers, emotionally evoca- tive structures, suggesting the existence a photo or video, being controversial, or being racist. We
Without source separation between training and test, model that overfits to certain source-specific aspects that are not related to clickbait (e.g. author style, preference towards certain subjects, and on) will reach high scores on the test. However, these scores are unlikely represent the actual performance the model in real-world scenario, where news can come from different sources with- out annotated samples to be used for training. Thus,

---

Subset
Clickbait
Non-clickbait
Total
The /CLS/ tokens returned by the last transformer block are used as sample embeddings in our classifi- cation task. The Ro-BERT embeddings are passed through a dropout layer, a fully-connected layer, and a classification layer. Finally, the labels are predicted by applying argmnxx to the probabilities produced by the classification layer.
Train
3,279
3,527
6,806
Test
441
1,066
1,507
Total
3,720
4,593
8,313
Table I: Data distribution of RoCliCo.
we consider that a more realistic evaluation is to separate the platforms. In the proposed setting, models that learn patterns related to author style, preferred themes and such will not be able capi- talize on features unrelated to clickbait detection.
Contrastive Ro-BERT. As a novel technique solve the clickbait detection problem, our con- trastive learning approach employs Siamese net- works (Chicco, 2021) to learn the similarity be- tween news headlines and contents. This is achieved by harnessing the distance between head- lines and contents in loss function that penal- izes the clickbait pairs that too close or the non-clickbait pairs that are too far apart. Let X {ai, 22, Fn. and Y {yı, y2, ... "e.... Yn represent our training set of news samples and the set of corresponding labels, where x; = (f4, Gi) a pair given by title ( and content G₁ and C {0. 1) is the class label (0 for clickbait and 1 for non- clickbait), for all i {1,2, .... n). Our approach applics Siamese Ro-BERT model to generate the normalized embeddings Vı, and Ve, for each title t and body a, respectively. We further compute the proposed contrastive loss, as follows:
4 Baselines
RF and SVM based on handcrafted features. Our first baseline embodies morphological, syn- tactical and pattern analysis model, similar to the recent approach of Coste et al. (2020), which uses a set of engineered features based on part-of-speech tagging, scores (CLScore, LIX, and RIX), and punctuation patterns. First, we lowercase the ti- tles and strip down special characters. Next, we ex- tract features such as the number of question words, punctuation patterns in the headline, the LIX and RIX indexes (Anderson, 1983), the CLSccree (Cole- man and Liau, 1975). part-of-speech tags based on Stanza (Qi et al., 2020), the number of common nouns, and the number of proper nouns. These features are then passed to two separate models, a Random Forest (RF) model and Support Vector Machines (SVM) classifier.
LCL. = - Σ (Us (Vi, vo,)+ 1=1
(1)
+ (1 vi) max (0, 5 (Vi,, Vo.)),
where m is margin that limits the optimization for the clickbait pairs, and 8 the cosine dissimilarity between Vs, and Ve. i.e.:
BiLSTM network. Our second baseline ap- proaches the problem via a recurrent architecture based on bidirectional LSTM (BiLSTM) layers (Graves et al., 2005). There are two BiLSTM branches, one for titles and one for contents. Each input is first preprocessed by the Keras tokenizer. Next, word embeddings are extracted using Fast- Text (Bojanowski et al., 2017). The resulting em- beddings passed as input to the BiLSTM neural network. The outputs of the separate BiLSTM branches are pooled, concatenated, and further passed through two fully-connected layers and a softmax classification layer.
Vt.
Ve:
5 (Veg Vc,)
(2)
Iv, Iv
We set m = in our experiments. Unlike our approach, related contrastive methods use (4, (z) pairs, i.c. a pair is composed of two titles of dif- ferent news articles. Such methods require hard example mining, resulting in expensive training process. In contrast, training complexity as the standard BERT. Ensemble. As our baselines consist of relatively distinct approaches to the problem, we consider the idea of combining the results into an ensemble of classifiers worthy of investigation. We gather the soft labels predicted by the baselines presented so far and use weighted voting mechanism predict the label of each sample. The weights are determined by computing the number of correct predictions generated by each model on a separate
approach has the same
Fine-tuned Ro-BERT. Our third baseline in- volves the fine-tuning of the Romanian BERT (Du- mitrescu et al., 2020), a transformer-based model trained exclusively on comprehensive collection of Romanian corpora. To fine-tune the model, the ti- tle and body of each news sample are concatenated using the separation token /SEP] as a delimiter, and the result is encoded using the Ro-BERT encoder.

---

Clickbait
Non- Non-Clickhait
Macro
Model
Precision
Recall
F1 Score
Precision
Recall
Fj Score
F1 Score
Random Forest
0.8666
0.6190
0.7222
0.8590
0.9606
0.9070
0.8146
SVM
0.8607
0.7006
0.7725
0.8850
0.9531
0.9178
0.8451
BiLSTM
0.7485
0.8367
0.7901
0,9290
0.8837
0.9058
0.8495
Fine-tuned Ro-BERT
0.9436
0.7210
0.8174
0.8948
0.9821
0.9364
0.8769
Contrastive Ro-BERT
0,9153
0,8571
0.8852
0.9424
0.9672
0.9546
0,9199
Ensemble
0.8981
0.8390
0.8675
0.9352
0.9606
0.9477
0.9076
Table Precision, recall F1 scores the proposed baselines on official RoCliCo test set. The scores are independently reported for each class to enable detailed class-based assessment of detection performance.
Ro-BERT, the latter model reaching the highest Fi score of 0.9546.
We evaluate the statistical significance of the pro- posed contrastive Ro-BERT with respect to the fine- tuned Ro-BERT using a paired McNemar's test. The test indicates that the performance improve- ment brought by our contrastive learning method is statistically significant, with p-value of 0.001.
0.8
Precision
0.7
Fine-tuned Ro-BERT (AP 0.90)
0.5
Contrastive Ro
0.85)
BiLSTM (AP 0.85)
In Figure we illustrate the precision-recall (PR) curves of the individual models on the test set. Each curve is summarized via the Average Precision (AP) measure, which represents the area under the PR curve. In terms of AP, it looks like the best model the fine-tuned Ro-BERT. Although the AP score of our contrastive Ro-BERT model is lower, our approach generates the sweetest spot on the graph, i.c. the closest point to the top-right corner. This point corresponds the best trade-off between precision and recall.
0.4
SVM (AP = 0.84)
Random Forest (AP
0.85)
0.0
0.2
0.4
0.6
Recall
Figure 1: Precision-recall (PR) curves of the individual models evaluated RoCliCo. The Average Precision (AP) of cach model is reported in the legend, next to each model's name. Best viewed color.
validation set, which is distinct from the test data. The weight of model is equal to the ratio between the number of correct labels and the number of validation samples.
The overall results of the models evaluated on the clickbait detection task suggest that predict- ing non-clickbait articles is fairly casy. However, clickbait articles are comparatively harder to detect, regardless of the employed model. This indicates that Romanian clickbait detection is challenging task, which remains to be solved in the future with the help our new corpus.
Reproducibility note. Appendix A provides de- tails about optimal hyperparameter values.
5 Results
In our empirical study, we perform binary classi- fication experiments aimed at predicting whether news is clickbait non-clickbait. As evaluation metrics, we report the precision, recall and Fi score for each of the classes, on the official test
Conclusion
In this work, we introduced RoCliCo, the first pub- lic corpus for Romanian clickbait detection. We trained and evaluated four models and an ensemble of these models to establish competitive baselines for future research on our corpus. Among these models, proposed a novel approach, namely BERT model based on contrastive learning that uses the cosine similarity between title and content to detect clickbait articles. The empirical results indicate that there is sufficient room for future re- search on the Romanian clickbait detection task.
The results obtained by the individual baselines, as well as their weighted ensemble, are shown in Table 2. Among the individual baselines, the LSTM network and our contrastive learning ap- proach achieve the highest recall rates clickbait news. The highest precision is obtained by the fine-tuned Ro-BERT, but our contrastive version reaches the highest F1 score (0.8852) on clickbait articles. On non-clickbait news, the best models are the fine-tuned Ro-BERT and the contrastive

---

7 Limitations
nia. Following GDPR regulations, we will remove all references person, upon receiving removal requests via an email to any of the authors.
This work is fully focused on our Romanian click- bait detection data set, but the performance levels of the considered approaches might be different on other languages. Due to our specific focus on the Romanian language, we did not test the general- ization capacity of the considered models across multiple languages. This could be addressed in future work, in an article with a broader focus.
References
Amol Agrawal, 2016. Clickbait detection using deep learning. In Proceedings of 2nd International Con- ference on Next Generation Computing Technologies (NGCT), pages 268-272.
Another limitation of our work is the rather lim- ited number of samples included in our corpus. Our examples are collected from six of the most common Romanian news websites, but there are several other websites of Romanian news that have not been considered. Extending the list of news sources should definitely result in collecting more news articles. However, the limitation is not caused by the low number of chosen websites, but by the la- borious annotation process involving human effort. Our annotation process is based on native Roma- nian volunteers who understood the importance of studying and solving the clickbait detection task, and agreed to label the collected articles for free. Labeling more samples would require the involve- ment of more annotators, but we were unable to find them due to the lack of funding for this project.
Jonathan Anderson. 1983. LIX and RIX:
Variations Liute-known Readability Index. Journal of Reading. 2606):400-496
Prakhar Biyani, Kostas Tsioutsiouliklis. and John Black- mer. 2016. "8 amazing secrets for getting more clicks": Detecting clickbaits in news streams using article informality. In Proceedings of the AAAI Con- ference on Artificial Intelligence (AAAI), volume 30.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- rion for Computational Linguistics, 5:135-146.
Peter Bourgonje. Julian Moreno Schneider, and Georg Rehm. 2017. From clickbait to fake news detection: An approach based detecting the stance of head- lines articles. In Proceedings of the 2017 EMNLP Workshop: Natural Language Procesing meets Jour- nalism, pages 84-89, Copenhagen, Denmark, Asso- ciation for Computational Linguistics.
8 Ethics Statement
Costin Busioc, Stefan Ruseti, and Mihai Dascalu. 2020. A Literature Review of NLP Approaches to Fake News Detection and Their Applicability to Romanian Language News Analysis. Revista Transilvania, 10.
The data was collected from six Romamian news websites. The news articles are freely accessible to the public without any type of subscription. As the data was collected from public websites, we adhere to the European regulations' that allow re- searchers to use data in the public web domain for non-commercial research purposes. We thus release our corpus as open-source under a non- commercial share-alike license agreement.
Bharathi Raja Chakravarthi, Gaman Mihaela, Radu Tu- dor Ionescu, Heidi Jauhiainen, Tommi Jauhiainen, Krister Lindén, Nikola Ljubešić, Niko Partanen, Ruba Priyadharshini. Christoph Purschke, Eswari Ra- jagopal. Yves Scherrer, and Marcos Zampieri. 2021. Findings of the VarDial evaluation campaign 2021. In Proceedings of the Eighth Workshop on NI. for Similar Languages, Varieties and Dialects (VarDial), pages 1-11.
The manual labeling was carried out by volun- teers who agreed to annotate the news articles for free. Prior to the annotation, they also agreed to let publish the labels along with the data set. Our corpus is collected from five different web- sites, so it may include linguistic variations, such as different styles, terminology, and so on. Infor- mation about the authors, such as gender, age or cultural background, is not included in our corpus to preserve the anonymity of the authors.
Davide Chicco. 2021. Siamese neural networks: An overview. In Hugh Cartwright, editor, Artificial Neu- ral Networks, pages 73 94. Springer US, New York. NY.
Meri Coleman and T. L. Liau. 1975. A computer read- ability formula designed for machine scoring. Jour- nal of Applied Psychology, 6002):283-244,
Claudia Ioana Coste, Darius Bufnea, and Virginia Niculescu. 2020. new language independent strat- egy for blic detection. In Proceedings of 2020 International Conference on Software, Telecommuni- cations and Computer Networks (SofiCOM), pages 1-6.
We acknowledge that some news samples could refer to certain people, e.g. public figures in Roma-
'https://eur- http:///urreeeeeeerr.aeeeeeeeeee europa..u/ell///r///a3.
oj

---

Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACI--HI.T), pages 4171-4186.
processing with machine learning techniques. In Pro- ceedings of 2018 International Conference on Engi- neering, Applied Sciences, and Technology (ICFAST), pages 1-4. IEEE.
Vaibhav Kumar, Dhruv Khattar, Siddhartha Gairola, Yash Kumar Lal, and Vasudeva Varma. 2018. Iden- tifying clickbait: A multi-strategy approach using neural networks. In Proceedings of the 41st Interna- tional ACM SIGIR Conference on Research & De- velopment in Information Retrieval (SIGIR), pages 1228-1288.
Manqing Dong. Lina Yao, Xianzhi Wang, Boualem Benatallah, and Chaoran Huang. 2019. Similarity- aware deep attentive model for clickbait detection. In Advances in Knowledge Discovery and Data Mining: 23rd Pacific-Asia Conference (PAKDD), pages 56-69. Springer.
Tong Liu, Ke Yu, Lu Wang, Xuanyu Zhang, Hao Zhou, and Xiaofei Wu. 2022. Clickbait detection on wechat: A deep model integrating semantic and syntactic in- formation. Knowledge-Band Based Systems, 245:108605.
Stefan Daniel Dumitrescu, Andrei-Marius Avram, and Sampo Pyysalo. 2020. The birth of Romanian BERT. In Findings of the Association for Computational Linguistics (EMNLP), pages 4324 4328.
Daniel López-Sánchez, Jorge Revuelta Herrero, Angélica González Arrieta, and Juan M Corchado. 2018. Hybridizing metric learning and case-based reasoning for adaptable clickbait detection. Applied Intelligence, 48:2967-2982.
Muhammad N. Fakhruzzaman, Saidah Z. Jannah, Ratih A. Ningrum, and Indah Fahmiyah. 2021. Click- bait Headline Detection Indonesian News Sites using Multilingual Bidirectional Encoder Representa- tions from Transformers (M-BERT). arXiv preprint arXiv:2102.0144
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint urXiv:1711.05101
Sura Genç and Elif Surer. 2023, ClickbaitTR: Dataset for clickbait detection from Turkish news sites and social media with comparative analysis via machine learning algorithms. Journal of Information Science, 49(2):480-49) 499..
Martin Potthast, Tim Gollub, Kristof Komlossy, Se- bastian Schuster, Matti Wiegmamn, Erika Patricia Garces Fernandez, Matthias Hagen, and Benno Stein. 2018. Crowdsourcing a large corpus of clickbait Twitter. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 1498-1507, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
Alex Graves, Santiago Fernández, and Jürgen Schmid- huber. 2005. Bidirectional LSTM Networks for Im- proved Phoneme Classification and Recognition. In Proceedings of 15th International Conference on Ar- rificial Neural Networks (ICANN), pages 799-804, Berlin, Heidelberg. Springer Berlin Heidelberg.
Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016. Clickbait detection. In Ad- vances in Information Retrieval (ECIR), pages 8 $10- 817.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(3):1335-1880
Abinash Pujahari and Dilip Singh Sisodia. 2021. Click- bait detection using multiple categorisation tech- niques. Journal of Information Science, 47(1):118- 128.
Vijayasaradhi Indurthi, Bakhtiyar Syed, Manish Gupta, and Vasudeva Varma. 2020. Predicting clickbait strength in online social media. In Proceedings of the 28th International Conference on Computational Linguistics (C (COLING), pages 3845, Barcelona, Spain (Online).
Peng Qi. Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics (ACL): System Demonstrations.
Vivek Kaushal and Kavita Vemuri. 2020. Clickbait in
Hindi News Media: A Preliminary Study. In Pro- ceedings of the 17th International Conference on Natural Language Processing (ICON), pages 89, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NIPAT).
Ana-Cristina Rogoz, Gaman Mihacla, and Radu Tudor Ionescu. 2021. SaRoCo: Detecting satire in novel Romanian corpus of news articles. Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties (ACL). pages Online. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980.
Md Main Uddin Rony, Nacemul Hassan, and Moham- mad Yousuf. 2018. BaitBuster: clickbait identifica- tion framework. Proceedings of the AAAI Confer- ence on Artificial Intelligence (AAAT), volume 32.
Praphan Klairith and Sansiri Tanachutiwat. 2018 Thai clickbait detection algorithms using natural language

---

Yumin Suh, Bohyung Han. Wonsik Kim, and Ky- oung Mu Lee. 2019. Stochastic Class-Bass Based Hard Example Mining for Deep Metric Learning. In Pro- ceedings of IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 7244- 7252. IEEE
Hyperparameter
Value
Number of estimators
150
Criterion
Entropy
Class weights
Balanced
Out-of-bag score
True
Veronika Vineze and Martina Katalin Szabó. 2020. Au- tomatic detection of Hungarian clickbait and enter- taining fake news. In Proceedings of the 3rd Interna- tional Workshop on Rumours and Deception in Social Media (RDSM), pages 58 69, Barcelona, Spain (On- line). Association for Computational Linguistics.
Table 3: Optimal hyperparameter configuration for the Random Forest classifier implemented in scikit-learn. Other hyperparameters are left to their default values.
Hyperparameter Value
Kernel
linear
Vorakit Vorakitphan, Fang- Yie Leu, and Yao-Chung Fan. 2019. Clickbait detection based word bedding models. Proceedings of the 12th Interna- tional Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), pages 557- 564. Springer.
C
Probability
True
Table 4: Optimal hyperparameter configuration for the SVM classifier implemented in scikit-learn, Other hy- perparameters are left to their default values.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020. Clickbait detection with style-aware title modeling and co-attention. In Proceedings of the 19th Chinese National Conference Compulational Linguistics (CCL.,, pages I 1143-1154 154, Haikou, China. Chinese Information Processing Society of China.
shared features: common noun counts, proper noun counts.
The resulting feature vectors standardized, be- fore being passed the classifiers. The Random Forest classifier and the SVM classifier are based on an optimal set of hyperparameters established through validation. These hyperparameters are pre- sented in Table 3 and Table 4, respectively. BiLSTM network. This model uses the Keras tokenizer to extract tokens from titles and bodies. We keep the most frequent 12,000 tokens from the headlines and 25,000 tokens from the contents, for which we extract 300-dimensional FastText embed- dings. The titles and contents are processed by two separate BiLSTM branches, and the resulting em- beddings are subsequently merged. Each branch is formed of two BiLSTM layers, with 32 units for titles and 64 units for contents. Each branch ends with global max-pooling layer. The result- ing feature vectors concatenated and passed through two fully-comnected layers with 128 and 64 neurons, respectively. Both fully-connected layers have a dropout rate 0.6. The final layer performs softmax classification. The model is trained for 10 epochs with a mini-batch size of 32 samples using the Adam optimizer (Kingma and Ba, 2017).
Xiaoyuan Yi, Jiarui Zhang, Wenhao Li, Xiting Wang, and Xing Xie, 2022. Clickbait detection via con- trastive variational modelling of text and label. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI), pages
A Appendix: Models and Parameters
This appendix serves as a comprehensive overview of the architcctural design and the specific hy- perparameters used for each model, to improve reproducibility and transparency. This appendix also contains supplementary information about the choices made during the experimental setup. Aside from the details given below, we release the code to reproduce the baselines along with our corpus. RF and SVM based on handerafted features. As presented in the main article, the RF and SVM mod- els rely on handcrafted features based on morpho- logical and syntactical attributes. As some features are extracted from only of the components of an article, while other features depend on both components, we provide a more detailed list of the extracted features below:
Fine-tuned Ro-BERT. This approach is based on fine-tuning the Romanian BERT model to generate embeddings of the input text. In our case, the text is given by the concatenated title and body, which are separated by the special separation token. We attach a dropout layer with a rate of 0.2, dense layer of 128 neurons, and a softmax classification
title features: part-of-speech tags, LIX. RIX, question word counts, punctuation character counts;
body features: LIX, RIX, CLScore;

---

layer. The modified Ro-BERT model is trained for 10 epochs with mini-batch size of samples, while using the A optimizer (Loshchilov and Hutter, 2017) 17 with a learning rate of 2 10 Contrastive Ro-BERT. We employ the Ro-BERT tokenizer to generate the embeddings for each sam- ple, while separately processing the title and the body. The maximum length of each sequence is set 256. Before using them in later stages, these em- beddings are normalized by subtracting the mean token embedding of the non-padding tokens. Each text and body pair is then used in the proposed loss function. The model is trained for 5 epochs with mini-batch size of 4 samples, using the Adam op- timizer with a fixed learning rate of 10 During inference, we compute the cosine similarity be- tween the title and the body forming test sample. The resulting similarity value used to separate each test article into non-clickbait and clickbait, using a threshold of 0.75.
detection very difficult task. The tricks involved range from simple punctuation tricks incomplete phrases suggesting different topic. For instance, the title "The trip to India made Dan Negru very rich" suggests that Dan Negru went to India for some business and made lot of money, but the content might reveal that Dan Negru became "spir- itually rich", which is completely different thing. In this example, model would have to figure out that the word "rich" is used with different meanings in the title and the content, respectively. This ex- ample illustrates that detecting clickbait is indeed a challenging task.
We consider that all the above aspects combined justify the study of clickbait detection for Roma- nian.
Ensemble of the five baselines. We upply weighted voting system determine the label of a sample, based on the outputs of the five individ- ual approaches. Based on preliminary validation experiments, we set the weights as follows: 0.19 for the Random Forest classifier, 0.19 for the SVM classifier, 0.22 for the BiLSTM network, 0.19 for fine-tuned Ro-BERT, and 0.21 for the contrastive Ro-BERT. For the ensemble, we set the clickbait threshold 0.5.
B Appendix: Discussion
Romanian an Eastern Romance language, being part linguistic group that evolved from several dialects of Vulgar Latin which separated from the Western Romance languages between the 5th and the 8th centuries AD. Being surrounded by Slavic neighbors, it has a strong Slavic influence, which makes Romanian unique language. Moreover, clickbait is often identified via subtle (semantic) differences between title and content, which can casily get lost in translation to more popular lan- guages. To test the ability of multilingual models in Romanian clickbait detection, we fine-tune the mul- tilingual BERT on our dataset, obtaining macro Fi of 0.6188, which is well below the macro Fi of Ro- BERT, indicating that pre-training on other languages does not bring any benefits to Romanian clickbait detection.
To create clickbait, news writers put their well- trained language skills to use, making clickbait


Reviews:
[{"review_id": "JGjbEFECoK", "reviewer": "Reviewer_DCj2", "summary": "", "questions": "Can you provide more details on the criteria used for selecting the five Romanian news websites included in the corpus? Were there any specific reasons for choosing these websites over others?\n\nHow did you ensure the quality and consistency of the annotations provided by the volunteers? Did you conduct any inter-annotator agreement analysis or provide any guidelines for the annotators?\n\nAs the clickbait detection task is quite similar to the natural language inference task (NLI), why do the authors not pick some existing NLI models for comparison?\n\nSection 3: In the last paragraph, “… the test data represents a more natural distribution, …”, what is “natural distribution”? Is imbalance data the same as “natural distribution”?\n\nSection 4: In the 1st paragraph, “… based on part-of-speech, scores, and punctuation patterns.”, what do the scores represent?\n\nSection 4: In the 2nd paragraph, “Each input is first preprocessed by a tokenizer.”, which tokenizer do the authors employ? NLTK, Gensim, or StanfordCoreNLP?\n\nSection 4: In the 2nd paragraph, “The resulting embeddings are passed as input to the neural network.”, what neural network is chosen? MLP, CNN, or Transformer?\n\nSection 4: In the 3rd paragraph, “The generated Ro-BERT-based embeddings are passed through a dropout layer, a fully-connected layer, and a classification layer.”, as the former sentence mentioned [SEP], does here the embedding denote [CLS] embedding?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. _The number and type of baseline models in the Experiments section are too few, only 4._\n\nReply: To address this point, we conducted additional experiments with a shallow model (SVM) based on the same features as the Random Forest, and a multilingual BERT that is first fine-tuned on XNLI, then fine-tuned on RoCliCo. The results of these models are presented below: \n\n| Model | Precision (clickbait) | Recall (clickbait) | F1 (clickbait) | Precision (non-clickbait) | Recall (non-clickbait) | F1 (non-clickbait) | Macro F1 |\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n| SVM | 0.7142 | 0.5780 | 0.6389 | 0.9245 | 0.9571 | 0.94056 | 0.7897 |\n| Multilingual BERT | 0.4731 | 0.2528 | 0.3295 | 0.8717 | 0.9474 | 0.9080 | 0.6188 |\n\nWe will include these additional models in Table 2 from the camera ready. We thank the reviewer for the constructive feedback.\n\n2. _As the clickbait detection task is quite similar to the natural language inference task (NLI), why do the authors not pick some existing NLI models for comparison?_\n\nReply: To the best of our knowledge, there is no NLI dataset and model for the Romanian language. Hence, the only option is to take a multilingual model pre-trained on XNLI (although XNLI does not contain sample translations to Romanian), and fine-tune it on our dataset. As earlier mentioned, a fine-tuned multilingual BERT obtains a macro F1 of 0.6188, which is below the macro F1 of Ro-BERT (please see Table 2). This confirms that the language-specific Ro-BERT is able to obtain better results. We thank the reviewer for the constructive comment.\n\n3. _The size of RoCliCo is relatively small since the majority of non-clickbait samples can be collected from some authoritative news website._\n\nReply: When we selected our sources, we initially considered protv.ro and digi24.ro as credible news sources. However, after the labeling process, we found clickbait news even from these sources (the minimum clickbait ratio per source is 17%). Unfortunately, all Romanian news sources seem to use clickbait titles to some extent. This is likely because there are no subscription-based publications in Romania, and the news publishers have to rely on advertisement to obtain earnings. We will clarify this in the camera ready.\n\n4. _Though Cohen’s kappa coefficient among annotators is 0.73, the labeling process lack of cross-check, which can introduce biases and errors in the labeling results._\n\nReply: The authors checked the assigned labels for a subset of 200 samples and confirmed the consistency of the labels. The final label for a sample is decided based on the majority vote of the three annotators, which reduces labeling errors by a large extent.\n\n5. _Can you provide more details on the criteria used for selecting the five Romanian news websites included in the corpus? Were there any specific reasons for choosing these websites over others?_\n\nReply: The chosen five Romanian news websites were selected based on their popularity among Romanians, as well as their reputation as credible or less credible sources. As protv.ro and digi24.ro sources cover a wide range of news categories and are considered as credible news sources by the population. In contrast, wowbiz.ro and cancan.ro tend to concentrate more on celebrity news, and use clickbait to attract readers. However, our annotation process revealed that these websites contain both clickbait and non-clickbait articles, though the percentage of clickbait news per source can differ significantly, between 17% and 71%.\n\n6. _How did you ensure the quality and consistency of the annotations provided by the volunteers? Did you conduct any inter-annotator agreement analysis or provide any guidelines for the annotators?_\n\nReply: Yes, we measured the agreement among annotators and obtained a Kappa coefficient of 0.73, which confirms that the inter-annotator agreement is substantial. Each annotator was asked to read a instructive document written in Romanian, where the authors provided definitions, guidelines and 5 examples of clickbait and non-clickbait news articles.\n\n7. _Section 3: In the last paragraph, “… the test data represents a more natural distribution, …”, what is “natural distribution”? Is imbalance data the same as “natural distribution”?_\n\nReply: We used two news sources, Cancan and WowBiz, that are likely to have clickbait news and only one source that is likely to have credible news, essentially skewing the distribution towards having more clickbait news. This led to a roughly balanced training set. For the test set, we used one credible and one less credible source, and the resulting distribution is indeed imbalanced. We consider the latter distribution to be closer to the natural one, since there was no intervention from our side to collect more clickbait news for the test set.\n\n8. _Section 4: In the 1st paragraph, “… based on part-of-speech, scores, and punctuation patterns.”, what do the scores represent?_\n\nReply: As mentioned on line 227, the features are taken from Coste et al. (2020). In this paper, the authors use the CLScore [1], LIX index [2] and RIX index [2]. To make our paper more self-contained, we will clarify this point in the camera ready version. We thank the reviewer for this suggestion.\n\n9. _Section 4: In the 2nd paragraph, “Each input is first preprocessed by a tokenizer.”, which tokenizer do the authors employ? NLTK, Gensim, or StanfordCoreNLP?_\n\nReply: We use the same tokenizer mentioned on line 234, namely Stanza (Qi 234 et al., 2020). We will add this mention in the camera ready to improve clarity. We thank the reviewer for the constructive feedback.\n\n10. _Section 4: In the 2nd paragraph, “The resulting embeddings are passed as input to the neural network.”, what neural network is chosen? MLP, CNN, or Transformer?_\n\nReply: This paragraph, from lines 239 to 250, is for the BiLSTM network. Hence, the neural network is a BiLSTM (Bidirectional Long Short-Term Memory) network. We will clarify this aspect in the camera ready. We thank the reviewer for the constructive comment.\n\n11. _Section 4: In the 3rd paragraph, “The generated Ro-BERT-based embeddings are passed through a dropout layer, a fully-connected layer, and a classification layer.”, as the former sentence mentioned [SEP], does the embedding denote [CLS] embedding?_\n\nReply: Yes, this paragraph refers to the [CLS] embeddings. We will clarify this point in the paper. Thank you for the clarity concerns mentioned above. These will definitely improve the clarity of our paper.\n\n**References:**\n\n[1] M. Coleman and T.L. Liau, “A computer readability formula designed for machine scoring, in Journal of Applied Psychology,” 1975, Vol. 60. pp. 283-284.\n\n[2] J. Anderson, “Lix and Rix: Variations on a Little-known Readability Index,” in Journal of Reading 26 (6), pp. 490-496, 1983.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is really well-motivated and well-written.\n\nThe RoCliCo enriches the low-resource natural language corpora and clickbait detection.\n\nThe selected baseline models show that Romanian clickbait detection is a challenging task.", "reasons_to_reject": "The number and type of baseline models in the Experiments section are too few, only 4. \n\nThe size of RoCliCo is relatively small since the majority of non-clickbait samples can be collected from some authoritative news website.\n\nThough Cohen’s kappa coefficient among annotators is 0.73, the labeling process lack of cross-check, which can introduce biases and errors in the labeling results.", "questions_to_authors": "Can you provide more details on the criteria used for selecting the five Romanian news websites included in the corpus? Were there any specific reasons for choosing these websites over others?\n\nHow did you ensure the quality and consistency of the annotations provided by the volunteers? Did you conduct any inter-annotator agreement analysis or provide any guidelines for the annotators?\n\nAs the clickbait detection task is quite similar to the natural language inference task (NLI), why do the authors not pick some existing NLI models for comparison?\n\nSection 3: In the last paragraph, “… the test data represents a more natural distribution, …”, what is “natural distribution”? Is imbalance data the same as “natural distribution”?\n\nSection 4: In the 1st paragraph, “… based on part-of-speech, scores, and punctuation patterns.”, what do the scores represent?\n\nSection 4: In the 2nd paragraph, “Each input is first preprocessed by a tokenizer.”, which tokenizer do the authors employ? NLTK, Gensim, or StanfordCoreNLP?\n\nSection 4: In the 2nd paragraph, “The resulting embeddings are passed as input to the neural network.”, what neural network is chosen? MLP, CNN, or Transformer?\n\nSection 4: In the 3rd paragraph, “The generated Ro-BERT-based embeddings are passed through a dropout layer, a fully-connected layer, and a classification layer.”, as the former sentence mentioned [SEP], does here the embedding denote [CLS] embedding?", "paper_topic_and_main_contributions": "The authors present RoCliCo, a corpus that is a collection of over 7,900 news samples in Romanian that are manually annotated with clickbait and non-clickbait labels. It is the first publicly available corpus for clickbait detection in Romanian news, and it provides a valuable resource for researchers and practitioners to develop and evaluate machine learning models for this task. Experiments with a few machine learning methods to establish competitive baselines, which can serve as a starting point for future research on clickbait detection in Romanian.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iEFS0KB6hu", "reviewer": "Reviewer_dgex", "summary": "", "questions": "Why is it important to have this dataset in Romanian? How is it different than other languages?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. _For classification tasks it would be good to see a PR curve to evaluate overall performance rather than just mean metrics._\n\nReply: As suggested, we computed the PR curve for a number of models. We added the figure at: https://postlmg.cc/w1qcXYR8. We will include the PR curves of all models in the final camera ready version. We thank the reviewer for the constructive feedback.\n\n2. _Why is it important to have this dataset in Romanian? How is it different from other languages?_\n\nReply: From our evaluation of multiple news sources (beyond those included in RoClico), all Romanian news sources seem to use clickbait titles to some extent. This is likely because there are no subscription based publications in Romania, and the news publishers have to rely on advertisement to obtain earnings. However, clickbait is often identified via subtle (semantic) differences between title and content, which can easily get lost in translation. Romanian is an Eastern Romance language, being part of a linguistic group that evolved from several dialects of Vulgar Latin which separated from the Western Romance languages between the 5th and the 8th centuries AD. Being surrounded by Slavic neighbors, it has a strong Slavic influence, which makes Romanian a unique language. We fine-tuned multilingual BERT on our dataset and obtained a macro F1 of 0.6188, which is well below the macro F1 of Ro-BERT, indicating that pre-training on other languages does not bring any benefits to Romanian clickbait detection. We consider that all these aspects combined justify the study of clickbait detection for Romanian.\n\n3. _Other click bait datasets across languages and a side by side comparison._\n\nReply: In lines 106 to 129, we relate to clickbait datasets in other languages, indicating that our corpus has a similar size to existing corpora. For the camera ready, we will add a table with comparative statistics between clickbait datasets. We thank the reviewer for this recommendation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It's a novel dataset and has well established labeling, reliability metrics, and ", "reasons_to_reject": "For classification tasks it would be good to see a PR curve to evaluate overall performance rather than just mean metrics.", "questions_to_authors": "Why is it important to have this dataset in Romanian? How is it different than other languages?", "paper_topic_and_main_contributions": "Gather qnd curate a click bait classification dataset in Romanian.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Other click bait datasets across languages and a side by side comparison.", "Typos_Grammar_Style_and_Presentation_Improvements": "Some simple edits are necessary but nothing unusual."}, "extra_text": {}}, {"review_id": "hJSHfiA7Ao", "reviewer": "Reviewer_X7XQ", "summary": "", "questions": "- Is there any related work for Clickbait Detection? How are their methods different from your work?\n\n- Please provide detailed information on the qualifications of the annotators and the payment process for the labeling task to ensure transparency and reliability.\n\n- The rationale behind separating the news platforms between the training and test sets (as discussed in Section 3) should be clearly explained to justify the authors' approach.\n\n- Table 1 should be appropriately placed in the text, precisely where it is mentioned, to enhance readability and facilitate easy reference.\n\n- To demonstrate the dataset's utility, it is essential to compare it with popular datasets to highlight the unique dimensions it can contribute to various research areas.\n\n- While the authors have presented a comprehensive overview of the Clickbait Detection task, including additional information on the linguistic aspects of the task would be valuable. This addition would provide readers with a deeper understanding of the study's theoretical foundations and the methodologies used to achieve the results.\n\n- To ascertain the significance of the reported results, the authors should provide statistical analysis. This will offer readers a more accurate understanding of the effectiveness of the methods employed.\n\n- The manuscript lacks mention of hyperparameters or other model parameters. It is crucial to explain how these parameters are optimized to ensure reproducibility and enable others to apply the methodology effectively.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. _Please provide detailed information on the qualifications of the annotators and the payment process for the labeling task to ensure transparency and reliability._\n\nReply: The annotators are native Romanian speakers, who graduated from their bachelor studies at Romanian universities. We checked their reading comprehension skills prior to enrolment. Then, they received instructions on how to perform the annotations. As mentioned in the paper, the annotators are volunteers who agreed to assign the labels for free.\n\n2. _The rationale behind separating the news platforms between the training and test sets (as discussed in Section 3) should be clearly explained to justify the authors' approach._\n\nReply: Without source separation between training and test, a model that overfits to certain source-specific aspects that are not related to clickbait (e.g. author style, preference towards certain subjects, etc.) will reach high scores on the test. However, these scores are unlikely to represent the actual performance of the model in a real-world scenario, where news can come from different sources without annotated samples to be used for training. We consider that a more realistic evaluation is to separate the platforms. In the proposed setting, models that learn patterns related to author style, preferred themes, and such will not be able to capitalize on features unrelated to clickbait detection. We will add these clarifying comments in the camera ready. We thank the reviewer for the constructive feedback.\n\n3. _Table 1 should be appropriately placed in the text, precisely where it is mentioned, to enhance readability and facilitate easy reference._\n\nReply: We will move the table just before the paragraph, as suggested by the review. We thank the reviewer for this observation.\n\n4. _To demonstrate the dataset's utility, it is essential to compare it with popular datasets to highlight the unique dimensions it can contribute to various research areas._\n\nReply: The relation to existing clickbait corpora in other languages is detailed in Section 2 (please see lines 106-129). In terms of size, our corpus is in the same range as the other corpora for low-resource languages. However, most of the other corpora are based on distant supervision, labeling the news as clickbait based on the source platforms. In contrast, our corpus is based on manual annotations. We observed that news articles from a certain source platform cannot be placed in the same bucket, although it is true that different platforms use clickbait in different percentages (between 17% and 71%). To address this concern, we will add a table with comparative statistics between related corpora in the camera ready. We thank the reviewer for the observation.\n\n5. _While the authors have presented a comprehensive overview of the Clickbait Detection task, including additional information on the linguistic aspects of the task would be valuable. This addition would provide readers with a deeper understanding of the study's theoretical foundations and the methodologies used to achieve the results._\n\nReply: Certainly, news writers use their language skills to create clickbait. The tricks involved range from simple punctuation tricks (e.g. the title is “Who is the person that killed three people on Thursday” without a question mark, which indicates that the content is going to reveal this person’s name. However, the content mentions that the chief of police declared that “we need to find out who is the person that killed three people on Thursday and ensure this will not happen again”, without actually naming the person) to incomplete phrases suggesting a different topic (e.g. the title is “The trip to India made Dan Negru very rich”, suggesting that Dan Negru went to India for some business and made a lot of money, but the content reveals that Dan Negru became “spiritually rich”, which is completely different). In the latter example, a model would have to figure out that the word “rich” was used with different meanings in the title and the content, respectively. This example shows that detecting clickbait is indeed a challenging task. We will add this discussion in the paper.\n\n6. _To ascertain the significance of the reported results, the authors should provide statistical analysis. This will offer readers a more accurate understanding of the effectiveness of the methods employed._\n\nReply: Since our main methodological contribution is the contrastive learning approach, we evaluated the statistical significance of the proposed contrastive Ro-BERT with respect to the fine-tuned Ro-BERT using a paired McNemar’s test. The test indicates that the performance improvement brought by our contrastive learning method is statistically significant, with a p-value of 0.05. We will add this result in the camera ready. We thank the reviewer for the constructive comment.\n\n7. _The manuscript lacks mention of hyperparameters or other model parameters. It is crucial to explain how these parameters are optimized to ensure reproducibility and enable others to apply the methodology effectively._\n\nReply: Please note that the collected data and the code to reproduce the baselines are attached as supplementary materials. We will publicly release the code and data to ensure the full reproducibility of our results by readers. We believe this should alleviate all concerns on reproducibility.\n\n8. _Is there any related work for Clickbait Detection? How are their methods different from your work?_\n\nReply: To the best of our knowledge, there is no clickbait detection corpus for the Romanian language. As mentioned in Section 2 (lines 106-129), there are related corpora for other low-resource languages. In terms of the proposed approach based on contrastive learning, there are only two studies based on contrastive learning for clickbait news detection, as discussed in lines 135-152 for Section 2. Related approaches applied contrastive learning on (title_i, title_j) pairs, i.e. they consider titles from different data samples. This approach requires hard example mining that results in an expensive training process. Different from these works, we use (title_i, content_i) pairs, where the title_i and the content_i represent the i-th data sample from the training set. In our approach, clickbait samples are regarded as negative pairs, and non-clickbait samples are regarded as positive pairs. Hence, there is no hard sample mining involved. Our model learns a metric space between titles and contents, while existing contrastive learning methods learn a metric space between titles, which is fundamentally different. We will add these comments in the paper to better explain how our approach is different.\n\n9. _To make the analysis of the results more transparent, authors should present their results in several metrics and as a 4-digit decimal (ex: 0.81xx)._\n\nReply: We thank the reviewer for this recommendation. We will update all results in Table 2 accordingly. Please note that we reported results with 4-digit decimals for two extra baselines in the rebuttal to Reviewer DCj2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The process of creating the dataset is thoroughly explained.\n- A dataset of Clickbait in Romanian with comprehensive annotations holds potential value for researchers.\n- The authors conduct multiple experiments to validate the efficacy of both handcrafted and deep models as baseline methods on the dataset.\n- Reporting the performance of various commonly used models on this dataset serves as a valuable reference for future benchmarking efforts.", "reasons_to_reject": "- Please provide detailed information on the qualifications of the annotators and the payment process for the labeling task to ensure transparency and reliability.\n\n- The rationale behind separating the news platforms between the training and test sets (as discussed in Section 3) should be clearly explained to justify the authors' approach.\n\n- Table 1 should be appropriately placed in the text, precisely where it is mentioned, to enhance readability and facilitate easy reference.\n\n- To demonstrate the dataset's utility, it is essential to compare it with popular datasets to highlight the unique dimensions it can contribute to various research areas.\n\n- While the authors have presented a comprehensive overview of the Clickbait Detection task, including additional information on the linguistic aspects of the task would be valuable. This addition would provide readers with a deeper understanding of the study's theoretical foundations and the methodologies used to achieve the results.\n\n- To ascertain the significance of the reported results, the authors should provide statistical analysis. This will offer readers a more accurate understanding of the effectiveness of the methods employed.\n\n- The manuscript lacks mention of hyperparameters or other model parameters. It is crucial to explain how these parameters are optimized to ensure reproducibility and enable others to apply the methodology effectively.", "questions_to_authors": "- Is there any related work for Clickbait Detection? How are their methods different from your work?\n\n- Please provide detailed information on the qualifications of the annotators and the payment process for the labeling task to ensure transparency and reliability.\n\n- The rationale behind separating the news platforms between the training and test sets (as discussed in Section 3) should be clearly explained to justify the authors' approach.\n\n- Table 1 should be appropriately placed in the text, precisely where it is mentioned, to enhance readability and facilitate easy reference.\n\n- To demonstrate the dataset's utility, it is essential to compare it with popular datasets to highlight the unique dimensions it can contribute to various research areas.\n\n- While the authors have presented a comprehensive overview of the Clickbait Detection task, including additional information on the linguistic aspects of the task would be valuable. This addition would provide readers with a deeper understanding of the study's theoretical foundations and the methodologies used to achieve the results.\n\n- To ascertain the significance of the reported results, the authors should provide statistical analysis. This will offer readers a more accurate understanding of the effectiveness of the methods employed.\n\n- The manuscript lacks mention of hyperparameters or other model parameters. It is crucial to explain how these parameters are optimized to ensure reproducibility and enable others to apply the methodology effectively.", "paper_topic_and_main_contributions": "This paper presents the authors' experience in Clickbait Detection, introducing RoCliCo, the first public corpus for Romanian clickbait detection. They train and evaluate four models and an ensemble of these models to establish competitive baselines for future research on their corpus. The authors propose a novel approach using a BERT model based on contrastive learning, which leverages cosine similarity between the title and content to detect clickbait articles. The empirical results indicate significant potential for future research on the Romanian clickbait detection task. The availability of RoCliCo opens up opportunities for collaboration and advancement in clickbait detection methods for the Romanian language.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- To make the analysis of the results more transparent, authors should present their results in several metrics and as a 4-digit decimal (ex: 0.81xx).\n"}, "extra_text": {}}, {"review_id": "vKTY8jqFyT", "reviewer": "Reviewer_T8m7", "summary": "", "questions": "A. (Line 200-202) Is this phenomenon announced by the website itself or by the author's study? If it is a study, please give an argument, otherwise I think this conclusion is a strong hypothesis lack of justification.\nB. The conclusion does not give insight and line332 mentioned that \"non-clickbait articles is fairly easy. However, clickbait articles are comparatively\",  is it possible that the data set is biased (4,460 non-clickbait samples and 3,453 clickbait samples)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. _The description of the dataset is not specific enough, such as the domains, the average number of sentences of these articles, how many sentences are in the longest article, and how many sentences are in the shortest article._\n\nReply: Regarding the domains, we do not impose any restrictions. We collected samples from multiple domains, including culture, politics, finance, science, technology, sports, celebrities, etc. The distribution of samples across domains is roughly uniform. In lines 177-181, we provided statistics about the number of tokens. As suggested by the reviewer, we also computed the average number of sentences per article, as well as the minimum number of sentences and the maximum number of sentences. The numbers are given below: \n\n| Average | Minimum | Maximum |\n|-----------|-----------|-----------|\n| 28 | 3 | 448|\n\nWe will add these statistics in the paper. We thank the reviewer for this recommendation.\n\n2. _The data collection method, the annotation process lack of details, for example, what is the annotator agreement score?_\n\nReply: We detailed the annotation process in lines 182-208. At the end of that paragraph, we reported the inter-annotator Kappa agreement score of 0.73. In lines 167-181, we also clearly mentioned the five websites from where the news articles are collected.\n\n3. _Do not describe the difference between their proposed method and Siamese networks._\n\nReply: In lines 266-267, we clearly mentioned that we used a Siamese BERT. As mentioned in lines 147-152, the novelty of our approach consists of applying the Siamese BERT on (title, content) pairs, where the title and the content represent a single data sample from our corpus. Related approaches applied siamese networks on (title_i, title_j) pairs, i.e. they consider titles from different data samples. This approach requires hard example mining that results in an expensive training process. In contrast, our approach has the same training complexity as the standard BERT.\n\n4. _Do not clarify the difference between fake news detection, satirical news detection and clickbait detection. Maybe the author can explain it in the intro section._\n\nReply: Satirical news form a type of fake news, where the intention of the author is to make the readers laugh. In satirical news, it is clear for the readers that the reported events are fictitious. Fake news represent news that report fake events or facts, where the intention of the author is to deceive the readers into believing that the fake events or facts are true. Clickbait news is a distinct category where the title of news is deceptive with respect to the content. In some clickbait news, the content can present real events and true facts. In other clickbait news, the reported events or facts can be fake. Hence, these three categories (satirical, fake, clickbait) are distinct, although their intersection is clearly not empty. We will add these clarifications in the introduction. We thank the reviewer for the constructive feedback.\n\n5. _(Line 200-202) Is this phenomenon announced by the website itself or by the author's study? If it is a study, please give an argument, otherwise I think this conclusion is a strong hypothesis lack of justification._\n\nReply: This is an observation of the authors, definitely not a conclusion. It is just provided as a hint for the annotators. More precisely, in the instructions provided to the annotators, we told the annotators to be extra careful when they have to annotate a news article from the politics, finance, or celebrity categories. However, they were free to judge and assign the clickbait or non-clickbait labels on a case by case basis, as mentioned in lines 202-206. We will reformulate the sentence to clarify this point.\n\n6. _The conclusion does not give insight and line 332 mentioned that \"non-clickbait articles is fairly easy. However, clickbait articles are comparatively\", is it possible that the data set is biased (4,460 non-clickbait samples and 3,453 clickbait samples)?_\n\nReply: We agree that the imbalance nature of the dataset can be the reason behind the reported results. However, testing models on intentionally balanced data does solve the problem in the real world. Instead, models should rather be developed to better cope with the imbalanced task. We will add this clarifying comment in the camera ready.\n\n7. _Line 278: for all i = ***, may be more understandable to use \"for all i ∈ [1, n]\" (Latex)_\n\nReply: We will make this change, as suggested by the reviewer.\n\n8. _Reproducibility concerns._\n\nReply: Please note that the collected data and the code to reproduce the baselines are attached as supplementary materials. We will publicly release the code and data to ensure the full reproducibility of our results by readers. We believe this should alleviate all concerns on reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main contributions consist of collecting a new Romanian clickbait dataset and adopting contrastive learning into clickbait detection task.", "reasons_to_reject": "1. The description of the dataset is not specific enough, such as the domains, the average number of sentences of these articles, how many sentences are in the longest article, and how many sentences are in the shortest article.\n2. The data collection method, the annotation process lack of details, for example, what is the annotator agreement score?\n2. Do not describe the difference between their proposed method and Siamese networks.\n3. Do not clarify the difference between fake news detection, satirical news detection and clickbait detection. Maybe the author can explain it in the intro section.", "questions_to_authors": "A. (Line 200-202) Is this phenomenon announced by the website itself or by the author's study? If it is a study, please give an argument, otherwise I think this conclusion is a strong hypothesis lack of justification.\nB. The conclusion does not give insight and line332 mentioned that \"non-clickbait articles is fairly easy. However, clickbait articles are comparatively\",  is it possible that the data set is biased (4,460 non-clickbait samples and 3,453 clickbait samples)?", "paper_topic_and_main_contributions": "This paper focuses on the clickbait detection task and collects a new Romanian clickbait dataset (low-resource languages) for future research.\nThe main contributions consist of collecting a new Romanian clickbait dataset and adopting contrastive learning into clickbait detection task.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 278: for all i = ***, may be more understandable to use \"for all $i \\in [1,n]$\" (Latex)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This study proposed clickbait detection dataset in Romanian, which is first of its kind. The dataset is comprised with 7,913 news samples, which are manually annotated with clickbait and non-clickbait labels.   

Reviewers provided insightful feedback. For instance, Reviewer DCj2 pointed out an imbalanced distribution in the dataset (e.g., a majority of non-clickbait samples). This should be clarified to improve the quality of the paper. This reviewer also expressed concerns about the baseline experiments. Since this study aims to serve as a baseline for future work, it is important to present additional results using different models. The selection of these models could be based on those that are widely used and demonstrate superior performance in related studies. 

Reviewer X7XQ suggested comparing the dataset with other popular datasets. It will help readers understand the utility and significance of the presented dataset.
Decision: Accept-Findings