SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The ease of loading and using more than 6M sentiment annotated texts is as simple as the listings above. All data will be downloaded automatically; you must only appropriately filter it to your use case.
1 import datasets  
2  
3 mms_dataset = datasets.load_dataset("Brand24/mms")  
4 slavic = mms_dataset.filter( lambda row: row["Genus"] == "Slavic" and row["Polar questions"] == "interrogative word order")
Listing 1: Loading and filtering sentiment examples for Slavic genus with polar questions formed using interrogative word order.
1 import datasets  
2  
3 mms_dataset = datasets.load_dataset("Brand24/mms")  
4 afro_asiatic = mms_dataset.filter( lambda row: row["Family"] == "Afro-Asiatic" and row["Number of cases"] == "no morphological case-making")
Listing 2: Loading and filtering sentiment examples for Afro-Asiatic Language Family with no morphological case-making.
import datasets
mms_dataset = datasets.load_dataset("Brand24/mms")
pl_only_social_media = mms_dataset.filter( lambda row: row['language']) == 'pl' and row['domain'] == "social_media")
Listing 3: Loading and filtering sentiment examples for specific language and domain
1 import datasets  
2  
3 CLEANLAB_THRESHOLD = 0.7  
4  
5 mms_dataset = datasets.load_dataset("Brand24/mms")  
6 afro_asiatic = mms_dataset.filter( lambda row: row["cleanlab_self_confidence"] >= CLEANLAB_THRESHOLD)
Listing 4: Loading data based on cleanlab data quality score
D Error analysis
We present the error analysis of sentiment model results. The Tables 6 and 7 could be a little bit small, so we advise checking the editable version<sup>3</sup> of results with the option to check ours or create your pivot tables of the results.
A couple of patterns could be spotted in Table 6.
- Japanese neutral class is consistently low performing across all models, not exceeding even  $50\%$  f1 score. Interestingly, even well-resourced languages like English still have problems with the neutral class. only one English model FT_XLM-R achieved 0.7 F1 score.
- The positive class is the easiest class to learn for the models, but the performance of neutrals and negatives varies very much, and it always needs to be verified, and potentially there could be a need to annotate more data for these classes.
A couple of patterns could be spotted in Table 7.
- The first interesting pattern is a hard-to-learn neutral class for 'poems'. It could be related to a tiny dataset, only with approximately 1000 English examples (693 positive, 193 negative, and 166 neutral examples).
- The review texts are easy to train for positive and negative; however, they are tough to work decently for neutral class. The reason for it could be mixing positive and negative sentiment in many neutral annotated reviews, mentioning both advantages and disadvantages of products or services.
- News domain shows interestingly high performance for the neutral class; however, it could be related to skewed distribution in this domain for the neutral class. We have neutral as many positive and negative examples summed together.
There are many other angles on how we can use our benchmarking results. We advise you to check the editable results and create your pivot tables for your own purposes and needs.
3https://drive.google.com/drive/folders/1JKB1slTXXiTVu9cHhQL7Fo4GdhUmdInx?usp=sharing
Table 7: F1 scores across domains. Legend: ar - Arabic, bg - Bulgarian, bs - Bosnian, cs - Czech, de - German, en - English, es - Spanish, fa - Persian, fr - French, he - Hebrew, hi - Hindi, hr - Croatian, hu - Hungarian, it - Italian, ja - Japanese, lv - Latvian, pl - Polish, pt - Portuguese, ru - Russian, sk - Slovak, sl - Slovenian, sq - Albanian, sr - Serbian, sv - Swedish, th - Thai, ur - Urdu, zh - Chinese.
chats	mixed	news	poems	reviews	social_media
fl_pos	fl_neu	fl_neg	fl_pos	fl_neu	fl_neg	fl_pos	fl_neu	fl_neg	fl_pos	fl_neu	fl_neg	fl_pos	fl_neu	fl_neg	fl_pos	fl_neu	fl_neg
FT_DistilmbERT	0.70	0.73	0.48	0.75	0.72	0.65	0.61	0.86	0.50	0.73	0.00	0.67	0.94	0.47	0.86	0.74	0.68	0.70
FT_LaBSE	0.66	0.59	0.65	0.80	0.73	0.73	0.76	0.82	0.77	0.60	0.00	0.67	0.96	0.50	0.89	0.81	0.66	0.76
FT_mBERT	0.66	0.69	0.55	0.67	0.80	0.69	0.66	0.78	0.63	0.83	0.00	0.67	0.94	0.50	0.86	0.74	0.68	0.73
FT_MPNet	0.72	0.62	0.67	0.78	0.74	0.74	0.61	0.78	0.80	0.92	0.00	1.00	0.96	0.48	0.89	0.78	0.68	0.76
FT_mTS	0.70	0.47	0.70	0.79	0.73	0.73	0.74	0.87	0.57	0.92	0.00	0.67	0.95	0.51	0.89	0.78	0.66	0.71
FT_mUSE-dist	0.66	0.66	0.55	0.69	0.73	0.70	0.41	0.79	0.70	0.73	0.67	1.00	0.94	0.43	0.88	0.73	0.64	0.75
FT_XLM-R	0.68	0.71	0.63	0.78	0.77	0.68	0.64	0.86	0.70	0.73	0.00	1.00	0.93	0.67	0.86	0.80	0.72	0.69
FT_XLM-R-dist	0.70	0.77	0.70	0.77	0.77	0.71	0.51	0.82	0.74	0.60	0.00	1.00	0.96	0.47	0.89	0.78	0.69	0.74
HB_DistilmbERT	0.80	0.42	0.34	0.74	0.68	0.42	0.59	0.57	0.52	0.73	0.00	0.67	0.95	0.29	0.78	0.73	0.69	0.64
HB_LaBSE	0.70	0.67	0.67	0.81	0.72	0.59	0.75	0.82	0.45	0.73	0.00	1.00	0.96	0.30	0.84	0.75	0.64	0.73
HB_mBERT	0.84	0.36	0.48	0.71	0.68	0.56	0.55	0.78	0.59	0.73	0.00	0.67	0.94	0.34	0.81	0.73	0.68	0.63
HB_MPNet	0.66	0.67	0.57	0.76	0.70	0.67	0.57	0.74	0.67	0.83	0.00	1.00	0.95	0.40	0.87	0.73	0.65	0.73
HB_mTS	0.78	0.46	0.57	0.76	0.72	0.58	0.78	0.85	0.43	0.92	0.00	0.67	0.96	0.43	0.86	0.78	0.71	0.62
HB_mUSE-dist	0.76	0.63	0.45	0.74	0.74	0.58	0.63	0.84	0.45	0.60	0.00	0.67	0.95	0.39	0.82	0.76	0.64	0.65
HB_XLM-R	0.74	0.66	0.57	0.68	0.82	0.59	0.57	0.83	0.55	0.92	0.00	0.00	0.96	0.47	0.84	0.73	0.77	0.65
HB_XLM-R-dist	0.63	0.61	0.65	0.70	0.73	0.68	0.53	0.78	0.52	0.60	0.00	1.00	0.95	0.43	0.82	0.71	0.67	0.69
HL_DistilmbERT	0.84	0.34	0.52	0.78	0.57	0.43	0.64	0.76	0.29	0.92	0.00	0.00	0.93	0.17	0.74	0.61	0.70	0.63
HL_LaBSE	0.66	0.58	0.72	0.81	0.64	0.68	0.61	0.88	0.43	0.73	0.00	1.00	0.94	0.38	0.83	0.70	0.68	0.72
HL_mBERT	0.85	0.32	0.43	0.76	0.59	0.55	0.59	0.75	0.37	1.00	0.00	0.67	0.92	0.21	0.79	0.62	0.69	0.66
HL_MPNet	0.68	0.58	0.65	0.76	0.74	0.60	0.80	0.82	0.43	0.73	0.00	1.00	0.96	0.36	0.82	0.68	0.74	0.63
HL_mTS	0.78	0.29	0.50	0.80	0.69	0.53	0.79	0.76	0.55	0.92	0.00	0.67	0.96	0.29	0.77	0.62	0.73	0.61
HL_mUSE-dist	0.78	0.67	0.52	0.81	0.67	0.48	0.75	0.73	0.48	0.83	0.00	1.00	0.96	0.26	0.76	0.73	0.72	0.56
HL_XLM-R	0.84	0.46	0.52	0.80	0.71	0.54	0.74	0.82	0.50	0.92	0.00	0.00	0.96	0.41	0.75	0.65	0.78	0.60
HL_XLM-R-dist	0.74	0.51	0.78	0.80	0.63	0.65	0.71	0.70	0.57	0.60	0.00	1.00	0.96	0.29	0.78	0.75	0.63	0.67