Clickbait1

This model is a fine-tuned version of microsoft/Multilingual-MiniLM-L12-H384 on the Webis-Clickbait-17 dataset. It achieves the following results on the evaluation set:

• Loss: 0.0257

Model description

MiniLM is a distilled model from the paper "MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers".

We fine tune this model to evaluate (regression) the clickbait level of title news.

Intended uses & limitations

Model looks like the model described in the paper Predicting Clickbait Strength in Online Social Media by Indurthi Vijayasaradhi, Syed Bakhtiyar, Gupta Manish, Varma Vasudeva.

The model was trained with english titles.

Training and evaluation data

We trained the model with the official training data for the chalenge (clickbait17-train-170630.zip (894 MiB, 19538 posts), plus another set that was just available after the end of the challenge (clickbait17-train-170331.zip (157 MiB, 2459 posts).

Training procedure

Code can be find in Github.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.05	50	0.0571
No log	0.09	100	0.0448
No log	0.14	150	0.0391
No log	0.18	200	0.0326
No log	0.23	250	0.0343
No log	0.27	300	0.0343
No log	0.32	350	0.0343
No log	0.36	400	0.0346
No log	0.41	450	0.0343
0.0388	0.46	500	0.0297
0.0388	0.5	550	0.0293
0.0388	0.55	600	0.0301
0.0388	0.59	650	0.0290
0.0388	0.64	700	0.0326
0.0388	0.68	750	0.0285
0.0388	0.73	800	0.0285
0.0388	0.77	850	0.0275
0.0388	0.82	900	0.0314
0.0388	0.87	950	0.0309
0.0297	0.91	1000	0.0277
0.0297	0.96	1050	0.0281
0.0297	1.0	1100	0.0273
0.0297	1.05	1150	0.0270
0.0297	1.09	1200	0.0291
0.0297	1.14	1250	0.0293
0.0297	1.18	1300	0.0269
0.0297	1.23	1350	0.0276
0.0297	1.28	1400	0.0279
0.0297	1.32	1450	0.0267
0.0265	1.37	1500	0.0270
0.0265	1.41	1550	0.0300
0.0265	1.46	1600	0.0274
0.0265	1.5	1650	0.0274
0.0265	1.55	1700	0.0266
0.0265	1.59	1750	0.0267
0.0265	1.64	1800	0.0267
0.0265	1.68	1850	0.0280
0.0265	1.73	1900	0.0274
0.0265	1.78	1950	0.0272
0.025	1.82	2000	0.0261
0.025	1.87	2050	0.0268
0.025	1.91	2100	0.0268
0.025	1.96	2150	0.0259
0.025	2.0	2200	0.0257
0.025	2.05	2250	0.0260
0.025	2.09	2300	0.0263
0.025	2.14	2350	0.0262
0.025	2.19	2400	0.0269
0.025	2.23	2450	0.0262
0.0223	2.28	2500	0.0262
0.0223	2.32	2550	0.0267
0.0223	2.37	2600	0.0260
0.0223	2.41	2650	0.0260
0.0223	2.46	2700	0.0259

Framework versions

• Transformers 4.18.0
• Pytorch 1.11.0a0+17540c5
• Datasets 2.1.0
• Tokenizers 0.12.1