ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing
Authors: Nam Quoc Nguyen
Thang Chau Phan
Duc-Vu Nguyen
Kiet Van Nguyen
Venue: EMNLP 2023 Main (2023)
Keywords: Language Models
Social Media Processing
Low resource

Abstract: English and Chinese, known as resource-rich languages, have witnessed the strong development of transformer-based language models for natural language processing tasks. Although Vietnam has approximately 100M people speaking Vietnamese, several pre-trained models, e.g., PhoBERT, ViBERT, and vELECTRA, performed well on general Vietnamese NLP tasks, including POS tagging and named entity recognition. These pre-trained language models are still limited to Vietnamese social media tasks. In this paper, we present the first monolingual pre-trained language model for Vietnamese social media texts, ViSoBERT, which is pre-trained on a large-scale corpus of high-quality and diverse Vietnamese social media texts using XLM-R architecture. Moreover, we explored our pre-trained model on five important natural language downstream tasks on Vietnamese social media texts: emotion recognition, hate speech detection, sentiment analysis, spam reviews detection, and hate speech spans detection. Our experiments demonstrate that ViSoBERT, with far fewer parameters, surpasses the previous state-of-the-art models on multiple Vietnamese social media tasks. Our ViSoBERT model is available only for research purposes. Disclaimer: This paper contains actual comments on social networks that might be construed as abusive, offensive, or obscene.

ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing
Quoc-Nam Nguyen 1,3," Thang Chau Phan 1.3,* Duc-Vu Nguyen 2.3 Kiet Van Nguyen
'Faculty of Information Science and Engineering, University of Information Technology, Ho Chi Minh City, Vietnam
Multimedia Communications Laboratory, University of Information Technology,
Ho Chi Minh City, Vietnam
Vietnam National University, Ho Chi Minh City, Vietnam {20520644, 20520929 @gm. uit.edu.vv {vund, kietav) Guit.edu.
Abstract
particular, BERT (Devlin et al., 2019) and its vari- ants (Liu et al., 2019; Conneau et al., 2020) have achieved state-of-the-arr performance on wide range of downstream NLP tasks, including but not limited to text classification, sentiment analysis, question answering, and machine translation. En- glish is moving for the rapid development of lan- guage models across specific domains such as med- ical (Lee et al., 2019; Rasmy et al., 2021), scientific (Beltagy et al., 2019), legal (Chalkidis et al., 2020), political conflict and violence (Hu al., 2022), and especially social media (Nguyen et al., 2020; DeLucia et al., 2022; Pérez et al., 2022: Zhang et al., 2022).
English and Chinese, known as resource-rich languages, have witnessed the strong devel- opment of transformer-based language mod- els for natural language processing tasks. Al- though Vietnam has approximately 100M peo- ple speaking Vietnamese, several pre-trained models, c.g., PhoBERT, ViBERT, and VELEC- TRA, performed well on general Vietnamese NLP tasks, including POS tagging and named entity recognition. These pre preiteained language models are still limited to Vietnamese social media tasks. In this paper, present the first monolingual pre-trained language model for Vietnamese social media texts, ViSoBERT. which is pre-trained high-quality and diverse Vietnamese social me- dia texts using XLM-R architecture. Moreover, we explored our pre-trained model on five im- portant natural language downstream tasks on Vietnamese social media texts: emotion recog- nition, hate speech detection, sentiment anal- ysis, spam reviews detection, and hate speech spans detection. Our experiments demonstrate that ViSOBERT, with far fewer parameters, sur- passes the previous state-of-the-an models on multiple Vietnamese social media tasks. Our ViSoBERT model is available" only for search purposes.
large-scale corpus of
Vietnamese is the eighth largest language used the internet, with around 85 million users across the world.. Despite a large amount of Viet- namese data available over the Internet, the ad- vancement NLP research in Vietnamese is still slow-moving. This can be attributed to several fac- tors, to name a few: the scattered nature of avail- able datasets, limited documentation, and minimal community engagement. Moreover, most existing pre-trained models for Vietnamese were primarily trained on large-scale corpora sourced from general texts (Tran al., 2020; Nguyen and Tuan Nguyen, 2020; Tran et al., 2023). While these sources pro- vide broad language coverage, they may not fully represent the sociolinguistic phenomena in Viet- namese social media texts. Social media texts often exhibit different linguistic patterns, informal lan- guage usage, non-standard vocabulary, lacking dia- crities and emoticons that are not prevalent in for- mal written texts. The limitations of using language models pre-trained on general corpora become ap- parent when processing Vietnamese social media texts. The models can struggle to accurately un-
Disclaimer: This paper contains actual com- ments social networks that might be strued as abusive, offensive, or obscene.
1 Introduction
Language models based on transformer architec- ture (Vaswani et al., 2017) pre-trained on large- scale datasets have brought about paradigm shift in natural language processing (NLP), reshaping how analyze, understand, and generate text. In
'Equal contribution.
"nttpps//hhngggingfacee.. wwwp://oo/oo/ttiiioiioooooooo

---

derstand and interpret the informal language, using emoji, leencode, and diacrities used in social media discussions. This can lead to suboptimal perfor- mance in Vietnamese social media tasks, including emotion recognition, hate speech detection, sen- timent analysis, spam reviews delection, and hate speech spans detection.
2.1 Pre-trained Language Models for Vietnamese
Several PLMs have recently been developed for processing Vietnamese texts. These models have varied in their architectures, training data, and eval- uation metrics. PhoBERT, developed by Nguyen and Tuan Nguyen (2020), is the first general pre- trained language model (PLM) created for the Viet- namese language. The model employs the same architecture as BERT (Devlin et al., 2019) and the same pre-training technique as RoBERTa (Liu et al.. 2019) to ensure robust and reliable perfor- mance. PhoBERT was trained on 20GB word- level Vietnamese Wikipedia corpus, which pro- duces SOTA performances stream tasks of POS tagging, dependency parsing, NER, and NLI.
We present ViSoBERT, a pre-trained language model designed explicitly for Vietnamese social media texts address these challenges. ViSoBERT is based on the transformer architecture and trained on a large-scale dataset of Vietnamese posts and comments extracted from well-known social me- dia networks, including Facebook, Tiktok, and Youtube. Our model outperforms existing pre- trained models on various downstream tasks, in- cluding emotion recognition, hate speech detec- tion, sentiment analysis, spam reviews detection, and hate speech spans detection, demonstrating its effectiveness in capturing the unique characteristics of Vietnamese social media texts. Our contribu- tions are summarized as follows.
range of down-
Following the success of PhoBERT, viBERT (Tran et al., 2020) and VELECTRA (Tran et al., 2020), both monolingual pre-trained language models based on the BERT and ELECTRA ar- chitectures, were introduced. They were trained on substantial datasets, with ViBERT using 10GB corpus and VELECTRA utilizing an even larger 60GB collection of uncompressed Vietnamese text. viBERT4news was published by NIpHUST, a Vietnamese version of BERT trained on more than 20 GB of news datasets. For Vietnamese text sum- marization, BARTpho (Tran et al., 2022) is pre- sented as the first large-scale monolingual seq2seq models pre-trained for Vietnamese, based on the seq2seq denoising autoencoder BART. Moreover, ViT5 (Phan et al., 2022) follows the encoder- decoder architecture proposed by Vaswani et al. (2017) and the T5 framework proposed by Raffel et al. (2020). Many language models are designed for general use, while the availability of strong baseline models for domain-specific applications remains limited. Since then, Minh al. (2022) in- troduced ViHealthBERT, the first domain- speciffc PLM for Vietnamese healthcare.
We presented ViSoBERT, the first PLM based on the XLM-R architecture and pre-training procedure for Vietnamese social media text processing. ViSoBERT is available publicly for research purposes in Vietnamese social media mining. ViSoBERT can be a strong baseline for Vietnamese social media text pro- cessing tasks and their applications.
ViSoBERT produces SOTA performances on multiple Vietnamese downstream social me- dia tasks, thus illustrating the effectiveness of our PLM on Vietnamese social media texts.
To understand our pre-trained language model deeply, analyze experimental results on the masking rate, examining social media char- acteristics, including emojis, teencode, and diacrities, and implementing feature- ex- traction for task-specific models.
2 Fundamental of Pre-trained Language Models for Social Media Texts
2.2 Pre-trained Language Models for Social Media Texts
Multiple PLMs were introduced for social me- dia for multilingual and monolinguals. BERTweet (Nguyen et al., 2020) was presented the first pub- lic large-scale PLM for English Tweets. BERTweet has the same architecture as BERT Bose (Devlin et al., 2019) and trained using the RoBERTa pre-
Pre-trained Language Models (PLMss) based on transformers (Vaswani et al., 2017) have become crucial element in cutting-edge NLP tasks, includ- ing text classification and natural language genera- tion. Since then, language models based on trans- formers related to our study have been reviewed, including PLMs for Vietnamese social media texts.
htttp://ggthab..om/bnaaat

---

training procedure (Liu al., 2019). Koto et al. (2021) proposed IndoBERTweet, the first large- scale pre-trained model for Indonesian Twitter. doBERTweet is trained by extending a monolin- gually trained Indonesian BERT model with an additive domain-specific vocabulary. RoBERTu- ito, presented in Pérez et al. (2022), is a robust transformer model trained on 500 million Span- ish tweets. RoBERTuito excels in various lan- guage contexts, including multilingual and code- switching scenarios, such as Spanish and English. TWilBert (Angel González et al., 2021) is proposed as specialization of BERT architecture both for the Spanish language and the Twitter domain to address text classification tasks in Spanish Twitter.
We crawled comments from Vietnamese-verified pages by Facebook posts via the Facebook Graph API between Jan- wary 2016 and December 2022.
Facebook:
2. TikTok: We collected comments from Vietnamese-verffie channels by TikTok through TikTok Research API between Jan- uary 2020 and December 2022.
We scrapped comments from Vietnam-verified channels' videos by YouTube via YouTube Data APP between January 2016 and December 2022.
YouTube
Bernice, introduced by DeLucia et al. (2022), is the first multilingual pre-trained encoder designed exclusively for Twitter data. This model uses cus- tomized tokenizer trained solely on Twitter data and incorporates a larger volume of Twitter data (2.58 tweets) than most BERT-style models. Zhang et al. (2022) introduced TwHIN-BERT, multilin- gual language model trained on 7 billion Twitter tweets in more than 100 different languages. It is designed to handle short, noisy, user-generated text effectively. Previously, (Barbieri al., 2022) ex- tended the training of the XLM-R (Conneau et al., 2020) checkpoint using a data set comprising 198 million multilingual tweets. As a result, XLM-T is adapted to the Twitter domain and was not exclu- sively trained on data from within that domain.
Pre-processing Data: Pre-processing is vital for models consuming social media data, which is mas- sively noisy, and has user handles (@ username), hashtags, emojis, misspellings. hyperlinks, and other noncanonical texts. We perform the follow- ing steps to clean the dataset: removing noncanon- ical texts, removing comments including links, re- moving excessively repeated spam and meaning- less comments, removing comments including only user handles (@ (@username). and keeping emojis in training data.
As a result, our pretraining data after crawling and preprocessing contains 1GB of uncompressed text. Our pretraining data is available only for re- search purposes.
3.2 Model Architecture
3 ViSoBERT
Transformers (Vaswani et al., 2017) have signifi- cantly advanced NLP research using trained mod- els in recent years. Although language models (Nguyen and Tuan Nguyen, 2020; Nguyen and Nguyen, 2021) have also proven effective on range of Vietnamese NLP tasks, their results on Vietnamese social media tasks (Nguyen et al., 2022) need be significantly improved. To ad- dress this issue, taking into account successful hyperparameters from XLM-R (Conneau et al., 2020), we proposed ViSOBERT, a transformer- based model in the style of XLM-R architecture with 768 hidden units, 12 self-attention layers, and 12 attention heads, and used masked language objective (the same Conneau al. (2020)).
This section presents the architecture, pre-training data, and our custom tokenizer on Vietnamese so- cial media texts for ViSoBERT.
3.1 Pre-training Data
We crawled textual data from Vietnamese public social networks such Facebook', Tiktok and YouTube" which are the three most well known social networks in Victnam, with 52.65, 49.86, and 63.00 million users" 10 respectively, in early 2023.
To effectively gather data from these platforms, we harnessed the capabilities of specialized tools provided by each platform.
'https:/ www.wffcebook.ccooo
"https: Bottps::/wwwr /www///wt.... wiw.ttt.ttoooo......
I "ntrpss//deveooperssaaooe wwwp:eoo....acoo.ooooooooo Phtpps:/ tttps:://ddeveloprsssiii tiktok com httpiittok..oooooooutstttts research-api/ 13 /nttpe://devaoooers.goo wow.se..oooooloooooooooo hwt.p///reeroooo..ooooooootaeeee
/https: fitttp://wwwwyouteee /ttp://wwtwb...ootttttotot///ttt
10 "https:/ http://daaaaapptaalllo//r//rrppp
digital-211 20233

---

3.3 The Vietnamese Social Media Tokenizer To the best of our knowledge. ViSoBERT is the first PLM with custom tokenizer for Vietnamese social media texts. Bernice (DeLucia et al., 2022) was the first multilingual model trained from scratch on Twitter'" data with a custom tokenizer; however, Bernice's s tokenizer doesn't t handle Vietnamese so- cial media text effectively. Moreover, existing Viet- namese pre-trained models' tokenizers perform poorly on social media text because of different domain data training. Therefore, developed the first custom tokenizer for Vietnamese social media texts.
epochs. We trained our model for about three days on 2xRTX4090 GPUs (24GB). Each sen- tence is tokenized and masked dynamically with a probability equal to 30% (which is extensively experimented on Section 5.1 to explore the opti- mal value). Further details on hyperparameters and training can be found in Table 6 of Appendix B,
Downstream tasks. To evaluate ViSoBERT, we used five Vietnamese social media datasets avail- able for research purposes, as summarized Table 2. The downstream tasks include emotion recogni- tion (UIT-VSMEC) (Ho et al., 2020), hate speech detection (UIT-ViHSD) (Luu et al., 2021), sen- timent analysis (SA-VLSP2016) (Nguyen et al., 201 spam reviews detection (ViSpamReviews) (Dinh et al., 2022), and hate speech spans detection (UIT-ViHOS) (Hoang al., 2023).
Owing to the ability to handle raw texts of Sen- tencePiece (Kudo and Richardson, 2018) without any loss compared to Byte-Pair Encoding (Con- neau et al., 2020), we built custom tokenizer on Vietnamese social media by SentencePiece on the whole training dataset. A model has better cov- erage of data than another when fewer subwords are needed represent the text, and the subwords are longer (DeLucia et al., 2022). Figure 2 (in Ap- pendix A) displays the mean token length for each considered model and group of tasks. ViSoBERT achieves the shortest representations for all Viet- namese social media downstream tasks compared to other PLMs.
Fine-tuning. We conducted empirical fine- tuning for all pre-trained language models using the simpletramsformerr"" 15 Our fine-tuning process followed standard procedures, most of which are outlined in (Devlin et al., 2019). For all tasks men- tioned above, we use a batch size of 40, maxi- mum token length of 128, a learning rate of 2e-5, and AdamW optimizer (Loshchilov and Hutter, 2019) with an epsilon of 1e-8. We executed a 10- epoch training process and evaluated downstream tasks using the best-performing model from those epochs. Furthermore, none of the pre- gre-processing techniques is applied in all datasets to evaluate our PLM's s ability to handle raw texts.
Emojis and teencode are essential to the "lan- guage" on Vietnamese social media platforms. Our custom tokenizer's s capability to decode emojis and teencode ensure that their semantic meaning and contextual significance are accurately captured and incorporated into the language representation, thus enhancing the overall quality and comprehensive- ness of text analysis and understanding.
Baseline models. To establish the main baseline models, we utilized several well-known PLMs, in- cluding monolingual and multilingual, to support Vietnamese NLP social media tasks. The details of each model are shown in Table 3.
To assess the tokenized ability of Vietnamese so- cial media textual data, we conducted an analysis of several data samples. Table shows several actual social comments and their tokenizations with the tokenizers of the two pre-trained language models, ViSOBERT and PhoBERT, the best strong base- line. The results show that our custom tokenizer performed better compared to others.
Monolingual language models: viBERT (Tran et al., 2020) and VELECTRA (Tran et al., 2020) are PLMs for Vietnamese based on BERT and ELECTRA architecture, respec- tively. PhoBERT, which is based BERT architecture and RoBERTa pre-training proce- dure, (Nguyen and Tuan Nguyen, 2020) is the first large-scale monolingual language model pre-trained for Vietnamese; PhoBERT obtains state-of-the-an art performances on range of Vietnamese NLP tasks.
4 Experiments and Results
4.1 Experimental Settings
We accumulate gradients over one step to simu- late batch size of 128. When pretraining from scratch, train the model for 1.2M steps in 12
Multilingual language models: Addition- ally, we incorporated two multilingual PLMs,
16 htttps: http:///wwiitttot///
15 "htres//simpleeraasfooeerss (ver 0.63.11)

---

Comments
ViSOBERT
PhoBERT
concãc cáilón giday
"conc", 'ác", "cai", "òn", "gl". day".
"oon @ "c #", "ic..
'il é @", "ôn". 's
English: Wut is dis fuckingdlck
cunks, sunko, <unko, 

cámdio onh
"c", "eá @", 'm @ @ "".""""". "anb'.
<b, "e", "cim", "din", "unh",
Englisk: Thankyou
cunk> <unko, </>
d4y 14 vj du oko mot cau teenende English: This Is teencade 33mt3nc3
<s,, "d", 14", "y", "T", 14", "du", "cko", "mat".
e T. "du", tek
"cmn", "teen", "code", <ld>
"te @: "on @ @: "oodo". 

Table Actual social comments and their tokenizations with the tokenizers of the two pre-trained language models, ViSoBERT and PhoBERT.
Dataset
Train
Dev
Test
Task
Evaluation Metrics
Classes
UTT-VSMEC
5,548
686
693
Emotion Recognition (ER)
7
UTT-HSD
24,048
2,672
6,680
Hate Speech Detection (HSD)
SA-VLSP2016
5,100
1,050
Sentiment Analysis (SA)
Acc. WFI, MFI (%)
3
ViSpamReviews
14,306
1,590
3,974
Spam Reviews Detection (SRD)
4
ViHOS
8,844
1,106
1,106
Hate Speech Spans Detection (HSSD)
3
Table 2: Statistics and descriptions of Vietnamese social media processing tasks, Acc, WF1, and MF1 denoted Accuracy, weighted Fl-score, and macro Fl-score metries, respectively.
mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020), which were previously shown to have competitive performances to monolingual Vietnamese models. XLM-R, cross-lingual PLM introduced by Conneau et al. (2020), has been trained in 100 lan- guages, among them Vietnamese, utilizing vast 2.5TB Clean CommonCrawl dataset. XLM-R presents notable improvements in various downstream tasks, surpassing the per- formance of previously released multilingual models such mBERT (Devlin al., 2019) and XLM (Lample and Comneau, 2019).
and 65.88% of Acc, WF1, and MF1, respec- tively, significantly higher than these PhoBERT and TwHIN- models.
Hate Speech Detection Task: ViSoBERT
achieves significant improvements over previous state-of-the-art models, PhoBERT and TwHIN- BERT, with scores of 88.51% 1, 88.31%, and 68. 77% in Acc, WF1, and MF1, respectively. Notably, these achievements are made despite the presence of bias within the dataset
Sentiment Analysis Task: XLM-R archived SOTA performance on three evaluation metrics. However, there no significant increase in per- formance on this downstream task, for 0.45%, 0.46%, and 0.46% higher on Acc. WF1, and MF1 compared to our pre-trained language model, PhoBERT Large- The SA-VISP2016 dataset do- main is technical article reviews, including Tin- hTe? and VnExpress 18 which are often used as Vietnamese standard data. The reviews or com- ments in these newspapers are in proper form. While most of the dataset is sourced from ar- ticles, it also includes data from Facebook 19 Vietnamese social media platform that accounts for only 12.21% % of the dataset. Therefore, the dataset does not fully capture Vietnamese social media platforms' diverse characteristics and infor-
Multilingual social media language models: To ensure a fair comparison with our PLM, we integrated multiple multilingual social me- dia PLMs, including XLM-T (Barbieri et al., 2022), TWHIN-BERY (Zhang et al., 2022), and Bernice (DeLucia et al., 2022).
4.2 Main Results
Table 4 shows ViSoBERT's scores with the previ- ous highest reported results on other PLMs using the same experimental setup. It is clear that our Vi- SoBERT produces new SOTA performance results for multiple Vietnamese downstream social media tasks without any pre-processing technique.
16 "UTT-HS is massively imbalanced, included 19,886; 1,606; and 2,556 of CLEEAN, OFFENSIVE, and HATE class.
Emotion Recognition Task: PhoBERT and TWHIN-BERY archive the previous SOTA perfor- mances on monolingual and multilingual models, respectively. ViSoBERT obtains 68. 10%, 68.37%,
17 "Ilpps:/ /tintt/////tt/////// "hupps:/ htlpss//vveeprrsssuuu "https://wwer httpe://www..fccbooo.comm

---

Model
ALayers
#Heads
#Steps
ABatch
Domain Data
4Params
AVocab
#MILL
CSMT
ViBERT (Tam ot al., 2020)
12
12
16
Victnamese News
30K
256
No
VELECTRA (Tran nl. 2020)
3
16
NewsCorpus OscarCorpus
30K
256
No
PowRERTau Sux (Ngayen and Than Nguyen, 2020)
12.
S40K
1024
WiWiki ViNows
135M
64K
256
No
PsoRERT (Nguyen and Tuan Ngayen, 2020)
24
16
1,08M
512
ViWiki ViNews
370M
64K
256
No
IBERT (Dovlin e1 al. 2019)
12
12
IM
256
BookCorpux FaWiki
110M
30K
512
No
XLM-Raux (Conneau al., 2020)
12
12.
1.5M
8192
CommonCruel Wiki
270M
250K
512
No
X (Conneau et 2020)
24
16
1.5M
8192
CommooCCnot Wiki
550M
250K
512
No
XLM-T (Barbicri et al., 2022)
12
12
8192
Muliilingual Tweets
250k
512
No
TwHIN-BERTR>> (Zhang al., 2022)
12
500K
6K
Multilingual Tweets
to
250K
No
TwHIN BERTTay (Zhing et al., 2022)
24
16
500K
9K
Mustilingual Tweets
550M
250K
128
No
Bernice (DeLocia 2023)
405K-
8192
Multilingual Tieett
270M
Yet
ViSOBERT (Ouns)
12
12
1.2M
128
Vieinamese social media
97M
15K
312
Yes
Table 3: Detailed information about baselines and our PLM. #Layers, #Heads, #Batch, #Params, #Vocab, #MSL, and CSMT indicate the number of hidden layers, number of attention heads, batch size value, domain training data, number of total parameters, vocabulary size, max sequence length, and custom social media tokenizer, respectively.
Model
Avg
Acc
Emotion
WF1
Regeonition
MF1
Hate Speech Detection
Acc
WF1
MF1
Acc
Sentiment Analysis
WF1
MF1
Spom
Acc
Reviews Detection
WF1
MF1
Hate Speech Spans Detection
Are
WFT
MF1
viBERT
71.57
61.91
61.98
59.70
85.34
85.01
62.07
74.85
74.73
74.73
89.93
59.79
76.50
90:12
90.45
84.55
VELECTRA
72.43
61.79
61.71
61.95
86.95
36.37
63.95
74.95
74.88
74.88
89.83
39:63
76.23
90.59
90.58
85.12
PRoBERTs
72.81
63.69
63.36
61.41
87.12
36.81
65.01
75,72
75.52
75.52
89.83
59.75
76.18
91.32
91.38
85.92
PboBERT
23.47
64.71
64.66
62.55
87.32
36.98
65.14
76.52
76.36
76.22
90.12
90.00
76.83
91.44
91.45
36.56
IRRRT (cavoall
68.07
56.17
53.48
83.55
83.90
60.62
67.14
67.16
67.16
89:05
55.80
74.52
89.87
84.57
IBERT (uncavol)
67.66
56.23
56:11
53.32
83.38
81.27
58.
67.25
67.2 27.22
67.22
88.92
53.72
74.32
89.84
89.82
84.51
XLM-RRuu
58.67
86.36
$6.08
63.35
76.
76.38
90.16
89.96
76.55
90.74
XLM-Ruage
73.40
62.44
61.37
60.25
87.15
$6.86
65.13
78.28
78.21
7N.21
90.36
90.31
76.75
91.52
91.50
$6.66
XLM-T
72.23
61.37
$6.12
63.48
75
75.60
90.07
91188
53.53
TWHIN-BERT.a
71.60
61.49
60.88
$7.97
86.63
86.23
63.67
73.76
73.72
73.72
90.25
90:35
76.98
900.99
90.900
$5.67
TWEIN-BERT.sa
73:42
61.29
61.12
86.78
65.23
76.83
76.83
90.47
36.65
Bernice
72.49
64.21
64.27
60.58
86.12
$6.18
64.32
74.57
74.90
74,90
90.22
90:21
76.89
90.48
90.06
85.67
VISeBERT
75.65
68.37
88.51
$8.31
68.77
77.83
77.75
77.75
90.99
90.92
79.06'
91.62
91.57
$6.80
Table 4: Performances on downstream Vietnamese social media tasks of previous of monolingual and multilingual PLMs without pre-processing techniques. Avg denoted the average MFI score of each PLM. denotes that the highest result is statistically significant at < 0.01 compared to the second best, using paired t-test.
mal language. However, ViSOBERT still surpassed other baselines by obtaining 131%%0.919 % Ace, 1 1.39% WF1, and 1. 1.53% 90.%% MF1 com- pared to PHOBERT/TWHIN BERT.
XLM-T and Bernice in five Vietnamese social me- dia tasks. It's worth noting that XLM-T, TwHIN- BERT, and Bernice were all exclusively trained on data from the Twitter platform. However, this approach has limitations when applied to the Viet- namese context. The training data from this source may not capture the intricate linguistic and contex- tual nuances prevalent in Vietnamese social media because Twitter is not widely used in Vietnam.
Spam Reviews Detection Task: ViSoBERT performed better than the top two baseline mod- els. PhoBERT and TwHIN-BERT Specifically, achieved 0.8%, 0.9%, and 2.18% higher scores in accuracy (Acc), weighted F1 (WF1), and micro Fl (MF1) compared to PhoBERT. When compared to TWHIN-BERT, ViSOBERT outperformed it with 0.52%, 0.50%, and 1.78% higher scores in Acc, WF1, and MF1, respectively.
Result Analysis and Discussion
In this section, we consider the improvement of our PLM more compared to powerful others, in- cluding PhoBERT and TWHIN-BERT, in terms of different aspects. Firstly, we investigate the effects of masking rate on our pre-trained model perfor- mance (see Section 5.1). Additionally, we examine the influence of social media characteristics on the model's ability to process and understand the lan- guage used in these social contexts (see Section 5.2). Lastly, we employed feature-based extraction techniques on task-specific models to verify the potential of leveraging social media textual data to enhance word representations (see Section 5.3).
Hate Speech Spans Detection Task20: Our pre-trained ViSoBERT boosted the results up to 91.62%. 91.57%, and 86.80% on Acc, WF1, and MF1, respectively. While the difference is insignif- icant, ViSoBERT indicates an outstanding ability to capture Vietnamese social media information compared to other PLMS (see Section 5.3).
Multilingual social media PLMs: The results show that ViSoBERT consistently outperforms
70 For the Hate Speech Spans Detection task, we evaluate the total of spans on each comment rather than spans of each word in Hoang et al. (2023) to retain the context of each comment.

---

5.1 Impact of Masking Rate on Vietnamese Social Media PLM
5.2 Impact of Vietnamese Social Media
Characteristics
For the first time presenting the Masked Language Model, Devlin et al. (2019) consciously utilized a random masking rate of 15%. The authors believed masking too many tokens could lead to losing cru- cial contextual information required to decode them accurately. Additionally, the authors felt that mask- ing too few tokens would harm the training process and make it less effective. However, according to Wettig et al. (2023), 15% not universally optimal for model and training data.
Emojis, teencode, and diacritics are essential fea- tures of social media, especially Vietnamese so- cial media. The ability of the tokenizer to de- code emojis and the ability of the model to un- derstand the context of teencode and diacritics are crucial. Hence, to evaluate the performance of Vi- SoBERT on social media characteristics, compre- hensive experiments were conducted among sev- eral strong PLMs: PM4ViSMT, PhoBERT, and TWIIN-BERT.
We experiment with masking rates ranging from 10% to 50% and evaluate the model's performance on five downstream Vietnamese social media tasks. Figure illustrates the results obtained from our experiments with six different masking rates. Inter- estingly, our pre-trained ViSoBERT achieved the highest performance when using masking rate of 30%. This suggests delicate balance between the amount of contextual information retained and the efficiency of the training process, and optimal masking rate can be found within this range.
Impact of Emoji on PLMs: We conducted two experimental procedures comprehensively inves- tigate the importance of emojis, including convert- ing emojis general text and removing emojis.
Table 5 shows our detailed setting and experi- mental results on downstream tasks and pre-trained models. The results indicate moderate reduc- tion in performance across all downstream tasks when emojis are removed or converted to text in our pre-trained ViSoBERT model. Our pre-trained de- creases 0.62% Acc, 0.55% WF1, and 0.78% % MF1 on Average for downstream tasks while converting emojis to text. In addition, an average reduction of 1.33% Acc, 1 1.32% WF1, and I 1.42% MFI can be seen in all emojis in each comment. This is because when emojis are converted to text, the context of the comment is preserved, while removing all emojis results in the loss of that context.
However, the optimal masking rate also depends on the specific task. For instance, in the hate speech detection task, we found that masking rate of 50% yielded the best results, sur surpassing other masking rate values. This implies that the optimal masking rate may vary depending on the nature and require- ments of different tasks.
pre-trained model while removing
Considering the overall performance across mul- tiple tasks, we determined that masking rate of 30% produced the optimal balance for our pre-trained ViSOBERT model. Consequently, we adopted this masking rate for ViSoBERT, ensur- ing efficient and effective utilization of contextual information during training.
This trend is also observed in the TWHIN-BERT model, specifically designed for social media pro- cessing. However, TWHIN-BERY slightly improves emotion recognition and spam reviews detection tasks compared to its competitors when operat- ing on raw texts. Nevertheless, this improvement is marginal and insignificant, as indicated by the small increments of 0.61 %, 0. 0.13%, and 0.21% in Acc, WF1, and MF1 on the emotion recognition task, respectively, and 0.08% Acc, 0.05% WF1, and 0.04% MF1 on spam reviews detection task. One potential reason for this phenomenon is that TWHIN-BERT and ViSoBERT are PLMM trained on emojis datasets. Consequently, these models can comprehend the contextual meaning conveyed by emojis. This finding underscores the importance of emojis in social media texts.
moroo Scares by Tick and Macking Rate
28
V006000
VEd50
VIOS
Teun
Figure 1: Impact of masking rate on our pre-trained ViSoBERT in terms of MF1.
In contrast, there is a general trend of improved performance across a range of downstream tasks when removing or converting emojis to text on

---

Vodel
Emostio
Regconition
Hale Speech Detection
Sentiment Analysis
Spam Reviews Detection
Hate Speech Spans Detection
Acc
MF1
Ace
WFT
WFT
Acc
WFF
MF1
Arc
MPF
Converting cmojis to fest
PsoBERT
66.08
66.15
63.35
87.43
87.22
65.32
76.73
76.18
76.48
90.35
90.11
77.02
92.16
91.98
86.72
A
1.37
1.49
T 0.80
0.31
0.24
0.18
0.31
0.012
0.12
0.23
T 0.08
1 0.14
T 1,7.2
0.52
1. 0.16
TWHIN-BERT..g
64.82
64.42
61.33
$603
85.52
63.52
75.42
75.95
75.95
90.55
90.47
77,32
92.21
92.01
86.84
A
[ 0.13
0.21
1.20
1,26
1.21
1.50
0.88
0.88
0.08
T 0.05
11.04
T. 0.76
1 0.54
1 0.39
ViSOBERT
67:33
67393
65.42
87.88
07.25
76.96
76335
76385
90.22
90.18
78.25
5/.42
9E:
0.57
0.44
1 0:46
0.00
0:41
1.49
1 0.88
0,90
0.90
0.77
4 0.74
0081
0.80
0.54
0.21
Removing emagle
PROBERT
65.21
65.14
6281
87.25
86,72
64.85
76.72
76.48
76.48
90.21
90.00
77.02
91.53
91.51
86.62
0.50
T 0.26
0.07
0.29
0.12
0.12
0.10
0.09
0.05
0.09
TWHIN-BERTr.y ag
62.03
62.14
19.25
86.98
86.32
64,22
75.00
75.11
75.11
89.83
89.25
76,85
91.32
91,33
86.42
2.18
1
0.25
0:46
1.92
1.72
1.72
1 0.67
0.43
0.14
0.23
VISOBERT [+]
66:2"
62002
64.55
67.32
87'1"
66.98
76.25
75.98
75.09
89.92
89.09
77.95
91.58
91,33
86.72
1.58
1.33
1.19
1.77
1.23
0.08
ViSeBERT )
68.10
68.37
65.88
58.51
88.31
68.77
77.83
77.75
77.75
90.99
90.92
79.06
91.62
91.37
$6,90
Table 5: Performances of pre-trained models downstream Vietnamese social media tasks by applying two emojis pre-processing techniques. [. ]. and denoted our pre-trained language model ViSoBERT converting emoji to text, removing emojis and without applying any pre-processing techniques, respectively. A denoted the increase (T) and the decrease (L) in performances of the PLMs compared to their competitors without applying any pre-processing techniques.
PhoBERT, the Vietnamese SOTA pre-trained lan- guage model. PhoBERT is PLM on general text (Vietnamese Wikipedia) dataset containing no emojis; therefore, when PhoBERT encounters an emoji, it treats as an unknown token (see Table Appendix B). Therefore, while applying emoji pre- processing techniques, including converting emoijs to text and removing emojis, PhoBERT produces better performances compared to raw text.
the ViSpamReviews dataset contains samples in which users use the word with duplicated characters to improve the comment length while standardizing tcencodes leads misunderstanding.
Experimental results strongly suggest that the improvement achieved by applying complex pre- processing techniques to pre-trained models in the context of Vietnamese social media text is rela- tively insignificant. Despite the considerable time and effort invested in designing and implementing these techniques, the actual gains in PLMs perfor- mance are not substantial and unstable.
Our pre-trained model ViSoBERT on raw texts outperformed PhoBERT and TwHIN-BERY even when applying two pre-processing emojis tech- niques. This claims our pre-trained model's ability to handle Vietnamese social media raw texts.
Impact of Vietnamese Diacritics on PLMs:
Victnamese words are created from 29 letters, in- cluding seven letters using four diacritics (ä, â-è-ô, d-tf, and d) and five diacritics used to designate tone (as in å, á, å, å, and a) (Ngo, 2020). These diacritics create meaningful words by combining syllables (Le-Hong, 2021). For instance, the syl- lable "ngu" can be combined with five different diacritic marks, resulting in five distinct syllables: "ngú", "ngû"" "ngu", "ngü", and "ngū". Each of these syllables functions as a standalone word.
Due to infor- mal and casual communication, social media texts often lead to common linguistic errors, such as mis- spellings and teencode. For example, the phrase "äng kommmmm" should be "än com" ("Eat rice" in English), and "ko" should be "không" ("No" in English). To address this challenge, Nguyen and Van Nguyen (2020) presented several rules to standardize social media texts. Building upon the previous work, Quoe Tran et al. (2023) proposed a strict and efficient pre-processing technique to clean comments on Vietnamese social media.
Impact of Teencode PLMs:
However, social media text does not always ad- here to proper writing conventions. Due to var- ious reasons, many users write text without dia- critic marks when commenting on social media platforms. Consequently, effectively handling dia- crities in Vietnamese social media becomes crit- ical challenge. To evaluate the PLMs capability to address this challenge, we experimented by re- moving all diacritic marks from the datasets of five downstream tasks. This experiment aimed to assess the model's 's performance in processing text without
Table 7 (in Appendix C) shows the results with and without standardizing teencode on so- cial media texts. There is an uptrend across PhoBERT, TWHIN-BERT, and ViSoBERT while applying standardized pre-processing techniques. ViSoBERT, with standardized pre-processing tech- niques, outperforms almost downstream tasks but spam reviews detection. The possible reason is that

---

diacrities and determine its ability understand Vietnamese social media content in such cases.
ing the first subword of each word token, which is similar to Devlin et al. (2019).
Table 8 (in Appendix C) presents the results of the two best baselines compared to our pre-trained diacrities experiments. The experimental results reveal that the performance of all pre-traines mod- els, including ours, exhibited a significant decrease when dealing with social media comments lack- ing diacritics. This decline in performance can be attributed to the loss of contextual information caused by the removal of diacritics. The lower the percentage of diacritic removal in each comment, the more significant the performance improvement in all PLMs. However, our ViSoBERT demon- strated relatively minor reduction in performance across all downstream tasks. This suggests that our model possesses a certain level of robustness and adaptability in comprehending and analyzing Vietnamese social media content without diacritics. We attribute this to the efficiency of the in-domain pre-training data of ViSoBERT.
Our experiment results (see Table 9 in Ap- pendix C) demonstrate that the word embeddings generated by our pre-trained language model Vi- SoBERT outperform other pre-trained embeddings when utilized with BiLSTM and BiGRU for all downstream tasks. The experimental results in- dicate the significant impact of leveraging social media text data for enriching word embeddings. Furthermore, this finding underscores the effec- tiveness of our model in capturing the linguistic characteristics prevalent in Vietnamese social me- dia texts.
Figure 3 (in Appendix D) presents the perfor- mances of the PLMs as input features to BiLSTM and BiGRU the dev set per epoch in terms of MFI. The results demonstrate that ViSoBERT reaches its peak MF1 score in only 1 to 3 epochs, whereas other PLMs typically require an average of to 10 epochs to achieve on-par performance. This suggests that ViSoBERT has a superior capabil- ity to extract Vietnamese social media information compared to other models.
In contrast, PhoBERT and TWHIN-BERT expe- rienced a substantial drop in performance across the benchmark datasets. These PLMs struggled to cope with the absence of diacritics in Vietnamese social media comments. The main reason is that the tokenizer of PhoBERT can not encode non- diacritics comments due to not including those in pre-training data. Several tokenized examples of the three best PLMs are presented in Table 10 (in Appendix F). Thus, the significant decrease in its performance highlights the challenge of handling diacritics on Vietnamese social media. While han- dling diacrities remains challenging, ViSoBERT demonstrates promising performance, suggesting the potential for specialized language models tai- lored for Vietnamese social media analysis.
6 Conclusion and Future Work
We presented ViSoBERT, a novel large-scale monolingual pre-trained language model on Viet- namese social media texts. We illustrated that Vi- SoBERT with fewer parameters outperforms re- cent strong pre-trained language models such viBERT, VELECTRA, PhoBERT, XLM-R, XLM- T, TwHIN-BERT, and Bernice and achieves state- of-the-art performances for multiple downstream Vietnamese social media tasks, including emotion recognition, hate speech detection, spam reviews detection, and hate speech spans detection. We conducted extensive analyses to demonstrate the ef- ficiency of ViSoBERT on various Vietnamese so- cial media characteristics, including emojis, teen- codes, and diacritics. Furthermore, pre-trained language model ViSoBERT also shows the poten- tial of leveraging Vietnamese social media text to enhance word representations compared to other PLMs. We hope the widespread use of our open- source ViSoBERT pre-trained language model will advance current NLP social media tasks and ap- plications for Vietnamese. Other low-resource lan- guages can adopt how to create PLMs for enhanc- ing their current NL.P social media tasks and rele- vant applications.
5.3 Impact of Feature-bass based Extraction to Task-Specific Models
task-specific models, the contextualized word embeddings from PLMs are typically employed as input features. We aim to assess the quality of contextualized word embeddings generated by PhoBERT, TWHIN-BERT, and ViSoBERT to verify whether social media data can enhance word repre- sentation. These contextualized word embeddings are applied embedding features BiLSTM, and BiGRU is randomly initialized before the classifi- cation layer. We append a linear prediction layer to the last transformer layer of each PLM regard-

---

Limitations
References
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XI.MT: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 258, Marseille, France. European Language Re- sources Association.
While we have demonstrated that ViSoBERT can perform state-of-the-art on range of NLP social media tasks for Vietnamese, we think additional analyses and experiments comprehend what aspects of ViSoBERT were re- sponsible for its success and what understanding of Vietnamese social media texts ViSoBERT cap- tures. We leave these additional investigations to future research. Moreover, future work aims to ex- plore broader range of Vietnamese social media downstream tasks that this paper may not cover. Also, we chose to train the base-size transformer model instead of the Large variant because base models are more accessible due to their lower com- putational requirements. For PhoBERT, XLM-R, and TWHIN-BERY we implemented two versions Base and Large for all Vietnamese social media downstream tasks. However, it is not a fair compar- ison due to their significantly larger model configu- rations. Moreover, regular updates and expansions of the pre-training data are essential to keep up with the rapid evolution of social media. This allows the pre-trained model to adapt effectively to the dy- namic linguistic patterns and trends in Vietnamese social media.
necessary fully
Iz Beltagy, Kyle Lo, and Arman Cohan, 2019. SciB- ERT: A Pretrained Language Model for Scientific Text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 1620-
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and lon Androutsopoulos. 2020. LEGAL-BERT The Muppets straight out of Law School. In Findings the Association for Com- putational Linguistics: EMNLP 2020, pages 2898- 2904, Online. Association for Computational Lin- guistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised Cross Cross-lingus lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association Computational Lin- guistics.
Ethics Statement
Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Philip Resnik, and Mark Dredze. 2022. Ber- nice: A Multilingual Pre-trained Encoder for Twitter. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 6191-6205, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
The authors introduce ViSoBERT, pre-trained language model for investigating social language phenomena in social media in Vietnamese. Vi- SoBERT is based on pre-training an existing pre- trained language model (i.e., XLM-R), which lessens the influence of its construction on the en- vironment. ViSoBERT makes use of large-scale corpus of posts and comments from social commu- nities that have been found to express harassment, bullying, incitement of violence, hate, offense, and abuse, as defined by the content policies of social media platforms, including Facebook, YouTube, and TikTok.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186,
Co Van Dinh, Son T. Luu, and Anh Gia-Tuan Nguyen. 2022. Detecting Spam Reviews on Vietnamese E- Commerce Wehsites. In Inteillientt Information and Database Systems, pages 995-607. Springer Interna- tional Publishing.
Acknowledgement
This research was supported by The VNUHCM- University of Information Technology's Scientific Research Support Fund. We thank the anonymous EMNLP reviewers for their time and helpful sug- gestions that improved the quality of the paper.
Vong Anh Ho, Duong Huynh-Cong Nguyen, Danh Hoang Nguyen, Linh Thi-Van Pham, Duc-Vu Nguyen, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen. 2020. Emotion Recognition for Vietnamese Social Media Text. In Computational Linguistics: 16th International Conference of the Pucifie Associa- tion for Computational Linguistics, PACLING 2019,

---

Hanoi, Vietnam, October 11-13, 2019, Revised Selected Papers 16. pages 319-333. Springer.
Artificial Intelligence Practices, pages 415-426.
Springer International Publishing.
Phu Gia Hoang. Canh Luu, Khanh Tran, Kiet Nguyen, and Ngan Nguyen. 2023. ViHOS: Hate Speech Spans Detection for Vietnamese. In Proceedings of the 17th Conference of the European Chapter of the Associa- tion for Compitational Linguisties, pages 022-669.
Nguyen Minh, Vu Hoang Tran, Vu Hoang, Huy Duc Ta. Trung Huu Bui, and Steven Quoe Hung Truong. 2022. ViHealthBERT. Pre-trained language models for Vietnamese in health text mining. In Proceedings of the Thirteenth Language Resources and Evalua- lion Conference, pages 328-337, Marseille, France. European Language Resources Association.
Yibo Hu, MohammadSaleh Hosseini, Erick Sko- rupa Parolin, Javier Osorio, Latifur Khan, Patrick Brandt, and Vito D' o'Orazio. 2022. ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 5469-5482, Seattle, United States. Association for Computational Linguisties.
B. Ngo. 2020. Vietnamese: An Essential Grammar. Es- sential grammar. Routledge, Taylor & Francis Group.
Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Viet- namese. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 1037-1042, Online. Association for Computational Linguisties.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Twwet::Tweet: A pretrained language model for Indonesian Twitter with effective domain-specific vocabulary initialization. Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages Punta Cana, Dominican Republic. Association for Computational Linguistics.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, On- line. Association for Computational Linguisties.
Online and
Huyen TM Nguyen, Hung V Nguyen, Quyen T Ngo. Lu- ong X Vu, Vu Mai Tran, Bach X Ngo, and Cuong A Le. 2018. VLSP Shared Task: Sentiment Analy- sis. Journal of Computer Science and Cybernetics. 34(4):295-33 -310.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Khang Phuoe-Quy Nguyen and Kiet Van Nguyen. 2020. Exploiting Vietnamese social media characteristics for textual emotion recognition in Vietnamese. In 2020 International Conference Asian Language Processing (IALP), pages 276-281. IEEE
Guillaume Lample and Alexis Conneau. 2019. Cross- lingual Language Model Pretraining. Advances in Neural Information Processing Systems (NeurIPS).
Linh The Nguyen and Dat Quoe Nguyen. 2021. PhoNLP: A joint multi-task learning model for Viet- namese part-of of opeepeech tagging, named entity recog- nition and dependency parsing. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Demonstrations, pages 1-7.
Phuong Le-Hong. 2021. Diacritics generation and application in hate speech detection on Viet- namese social networks. Knowledge-Baad -Based Systems, 107:00
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang- 2019. BioBERT: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1234-1240..
Luan Nguyen, Kiet Nguyen, and Ngan Nguyen. 2022. SMTCE: A social media text classification evaluation benchmark and BERTology models for Vietnamese. In Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation, pages 282- 291, Manila, Philippines. De La Salle University.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach.
Juan Manuel Pérez, Damián Ariel Furman, Laura Alonso Alemany. and Franco M. Luque. 2022. RoBERTuito: pre-trained language model for social media text in Spanish. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 7235-7243, Marseille, France. European Language Resources Association.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In Proceedings of the International Conference on Learning Representa-
tions.
Long Phan, Hieu Tran, Hieu Nguyen, and Trieu H. Trinh. 2022. ViT5: Pretrained text-to-text trans- former for Vietnamese language generation. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational
Son T. Luu, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen. 2021. A Scage Dataset for Hate Speech Detection on Vietnamese Social Media Texts. In Advances and Trends in Artificial Intelligence.

---

Linguistics: Human Language Technologies: Student Research Workshop, pages 136-142. Association for Computational Linguisties.
Khanh Quoe Tran, An Trong Nguyen, Phu Gia Hoang, Canh Duc Luu, Tro Do. and Kiet Van Nguyen. 2023, Vietnamese hate and offensive detection using PhoBERT-CNN and social media streaming data, Neural Computing and Applications, 35(1):573-594.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(5:55855-551.
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. 2021. Med-BERT: pretrained contextual- ized embeddings health records for disease prediction. NPJ digital medicine, 4( :1).
large-scale structured electronic
Cong Dao Tran, Nhut Huy Pham, Anh-Tuan Nguyen, Truong Son Hy, and Tu Vu. 2023. ViDeBERTa: A powerful pre-trained language model for Vietnamese. Findings of the Association for Computational Linguistics: EACL 2023, pages 1041-1048.
Nguyen Luong Tran, Duong Minh Le, and Dat Quoc Nguyen. 2022. BARTpho: Pre-trained Sequence-to- Sequence Models for Vietnamese. In Proceedings of the 23rd Annual Conference of the International Speech Communication Association.
Thi Oanh Tran, Phuong Le Hong. et al. 2020. Im-
proving sequence tagging for Vietnamese text using transformer-basee neural models. In Proceedings of the 34th Pucific Asia conference on language, infor- mation and computation, pages 13-20.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processing systems, 30.
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. Should You Mask 15% in Masked Language Modeling? In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 2985- 3000, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, and Ahmed EI- Kishky. 2022. TWHIN-BERT: A Socially-Enrian -arriched Pre-trained Language Model for Multilingual Tweet Representations, arXiv preprint XrXiv:209..00 07560
José Ángel González, Lluís-F. Hurtado, and Ferran Pla. 2021. TWilBert: Pre-trained deep bidirectional transformers for Spanish Twitter, Newrocomputing, 426:58-69.

---

A Tokenizations of the PLMs Social Comments
We conducted an analysis of average token length by tasks of Pre-trained Language Models to provide insights into how different PLMs perform regarding token length across various Vietnamese social media
downstream tasks. Figure 2 shows the average token length by Vietnamese social media downstream tasks of baseline PLMs and ours.
Average token length by Task
PLMs
viBERT
XLM-R
VELECTRA
XLM-T
40
PhoBERT
TWHIN-BRRT
mBEKT (cased)
Bericce
mBERT (uncused)
ViSoBERT (ours)
lengeh
Averge
20
Regconition
Speech Detection
Sentiment Amalysis
Spum Reviews Detection
Speech Spams Dotection
Tasks
Figure 2: Average token length by tasks of PLMs.
B Experimental Settings
Following the hyperparameters in Table 6, we train our pre-trained language model ViSoBERT for Vietnamese social media texts.
Algorithm
Adam
Learning rate
5e-5
Epsilon
le-8
Optimizer
LR scheduler
linear decay and warmup
Warmup steps
1000
Betas
0.9 and 0.99
Weight decay
0.01
Sequence length
128
Batch
Batch size
128
Vocab size
15002
Dropout
0.1
Misc
Attention dropout
0.1
Table 6: All hyperparameters established for training ViSoBERT.

---

C PLMs with Pre-processing Techniques
For an in-depth understanding of the impact of social media texts on PLMs, we conducted an analysis of the test results on various processing aspects. Table 7 presents performances of the pre-trained language models on downstream Vietnamese social media tasks by applying word standardizing pre- pre-processing techniques, while Table 8 presents performances of the pre-trained language models on downstream Vietnamese social media tasks by removing diacritics in all datasets.
Emotion Recognition
Hate Speech Detection
Sentiment Analysi
Spam Reviews Detection
Hate Speech Spans Detection
Model
Acc
WFT
MF1
Acc
WF1
MFT
Acc
WF1
MF1
Acc
WFF
MF1
Are
WFT
MF1
PROBERTEap
64.94
64.55
62.71
87.68
87.25
65.41
76,80
36.61
76.61
39.47
89.41
76.12
91.73
91.62
56.59
A
0.23
0.19
1 0.16
0.36
0.27
1 0.37
1 0.28
0.25
0.25
0.65
1 0.62
1 1.7.6
T. 1.229
1 00116
1 0.03
TWHIN-BEKTray
64.42
64.46
61.28
87.82
87.28
06.68
77.17
76994
76.94
89.49
89.43
76.35
91.74
91.64
à
0.11
0.17
T 0.16
0.59
0.50
0.45
T 0.25
0.01
0.11
1 0.98
4 0.99
0,93
0.29
1 0.117
0.02
ViSOBERT(
6825
6833"
65.94
8633
06.82
78.00
75.88
9083
90.75
98.77
91.89
91.8
à
0.15
T 0.15
T 0.06
0.02
0.02
0.08
T 0.18
0.13
0.13
0.16
1 0.17
0.29
0.27
1 0.25
0.13
VISoBERT[+]
60.10
60.37
65.88
58.51
31.31
69.74
77.83
73.75
77.75
90.99
90.92
79.06
91.62
91.57
86.00
Table 7: Performances of the pre-trained language models on downstream Vietnamese social media tasks by applying word standardizing pre-processing techniques. [ [+] and [ [+] denoted with and without standardizing word technique, respectively. A denoted the increase (f) and the decrease (L) in performances of the pre-trained language models compared to its competitors without normalizing teencodes.
To emphasize the essentials of diacritics, we conducted an analysis on several data samples by removing 100%, 75%, 50%, and 25% diacritics of total words that included diacritics in each comment of five downstream tasks. Table 8 presents performances of the pre-trained language models on downstream Vietnamese social media tasks by removing diacrities in all datasets.
Emotion Regconition WF1
Speech Detection
Sentiment Analais MF1
Spam Reviews Detection
Hate Speech Spans Detection WF1
Model
Acc
MF1
Acc
WF1
MFI
Acc
WFI
Acc
WF1
MF1
Acc
MF1
Removing 100%
discritics in commeent
PsoBERT; wyr
49.35
49.18
43.95
$1.25
81.42
55.43
62.38
62.36
62.36
87.68
87.56
71.89
91.32
91.37
86.43
115.36
115:48
118.60
6.07
5,56
9.71
114.14
114.00
113.86
2:47
0.1.2
0.13
TWHIN-BERT.
49.32
49:15
43.52
$4.25
78.48
51.32
66.66
66.68
66.68
89:45
89.26
74.59
91.12
91.22
86.35
11480
115.14
117.60
2.08
8:30
112.99
110.15
1.16
2.69
0,33
0.25
0.32
61.86
R208
38.46
57.50
64.87
72.05
92.1
72.0
W0.55
56.12"
01.48
91.42"
76.00
7:40
1.22
1.55
4.88
4.84
1.20
2.94
0.0.4
0:15
0.11
Removing 75%
dioertticss in euch commenw
PoOBERT
51.94
51.79
47.79
84.03
58.37
88.23
90.35
90.23
85.42
A
112.77
112.87
114.76
2.58
2.95
1 6.77
110.52
110.38
110.24
1.89
1 1..1
1 1.16
1.06
1.23
1.34
WWHIN-BHRT..g
51.17
44.63
83.22
81.42
52.24
67.23
67.32
67.32
89.12
88.95
75.20
90.62
89.93
112.99
113.13
116.49
4.00
1.5. 5.36
112.99
1. 9.60
9:51
1.9.1 9.51
1:35
1 1.47
1 2.08
0x3
1 1.55
0.84
[
6234
38.T7"
55:35
R8.12"
73.56
FK39
96.37
96.37
56.77"
97.68
61.77
5.76
6.11
L 7.35
1.16
1.43
3.65
1 3.93
3.78
3.78
0.58
1 0.61
2.99
0.60
1 0.40
0.78
Remoring 50%
diacrifics cack
PooBERT
57.28
57.36
54.02
85.29
84.71
59.40
66.57
66.46
66.
89.02
88.81
73.10
90:42
90.47
85.62
A
7.43
7:30
8:53
2.03
2.27
9.95
9..0
1.22
3.78
1.002
0.99
0.94
TWHIN-BERT,
53.70
53.39
49.55
83.41
83.31
55,22
70.42
70.53
70.53
39.33
89.05
75.32
90.73
90.12
85.92
A
110.51
110.90
111.57
3.82
3.47
110.01
6.50
6.30
6.30
1.14
1.37
1.96
0.72
1.35
0.73
ViSoBERT )
62.96
62.87
60.55
$7.44
87.10
65.25
74.76
74.72
74.72
90.41
90.35
77.31
91.12
91.24
86.22
A
5.14
5.50
5.33
1.07
1,21
3.52
3.07
3.03
3.03
0.58
0.57
1.75
0.50
0.33
0.58
Removing 25%
diacritics each commeent
PRoBERT wgr
61.03
00.80
57.87
85.51
61.96
73.42
73.23
73.28
89.80
89.39
75.53
90063
90.69
85,76
A
3.68
3.86
L |.68
1.35
1.47
3.18
1 3.10
3.08
1.944
0.332
1 0.11
|..35
0.81
0.77
0.50
TWEIN BERT
61.18
60:98
57:42
36.85
86.13
63.14
73.21
73.11
73.11
39.91
89:43
76.32
91.09
90.72
86.02
3.03
3:31
L 3.70
0.38
0.65
2.09
1 3.71
3.72
3.72
0.56
1 0.99
0.96
0.36
0.75
0.63
VISoBERT [+)
64.64
64.53
61.29
$7.85
87.56
66.54
75.42
75.44
75.44
90.76
90.64
78.15
91.22
91.24
86.47
3.43
3.84
1 4.59
0.66
0.75
2.23
2.41
2.31
2.31
0.23
1 0.28
0.91
0.40
0.33
0.33
VISoBERT +
68.10
68.37
65.88
88.51
68.31
68.77
77.83
77.75
77.75
90.99
96.92
79.06
91.62
91,57
$6,50
Table 8: Performances of the pre-trained language models on downstream Vietnamese social media tasks by removing diacritics in all datasets. [ [+]. [ [+] [
and [+] denoted the performances of our pre-trained on removing 100%, 75%, 50%, 25% each comment, respectively, and not removing diacrities marks dataset, respectively. A denoted the increase (T) and the decrease (J) in performances of the pre- pre-trained language models compared to its competitors without removing diacritics marks.

---

D PLM-based Features for BiLSTM and BiGRU
We conduct experiments with various models, including BiLSTM and BiGRU, to understand better the word embedding feature extracted from the pre-trained language models. Table 9 shows performances of the pre-trained language model as input features to BiLSTM and BiGRU on downstream Vietnamese social media tasks.
Emottion Regconition
Hate Speech Delection
Sentiment Analvsis
Spun Reviems Deteciion
Hale Speech Spans Delection
Model
Ace
BILSTM
PhoBERT
57.58
56.65
50.55
86.11
84.04
56.03
69.71
69.70
69.70
87.80
87.10
68.95
84.01
80.70
74.35
TWHIN-BERT
61.47
61.31
56.73
83.14
82.72
55.84
64.76
64.82
64.82
88.73
88.23
72.18
85.92
84.43
78.28
ViSoBEKT
62.36
39.16
87.62
73.53
73.50
90.11
75.71
88.37
87.87
82.18
BIGRU
PhoBERT Tw
55.12
54.53
49.59
85.21
83.23
$4.59
70.01
70.01
70.01
86.06
84.89
62.54
84.23
81.01
74.57
TWHIN-BERT
60.46
60.30
55.23
85.73
83.45
54.74
63.11
61,39
61.39
ViSORERT
63.20
63.25
60.73
87.02
86.25
63.36
70.48
70.53
70.53
89.33
88.96
76.57
68.88
88.19
82.63
Table Performances of the pre-trained language models as imput features BiL.STM and BiGRU on downstream Vietnamese social media tasks.
We implemented various PLMs when used as input features in combination with BiLSTM and BiGRU models to verify the ability to extract Vietnamese social media texts. The evaluation is conducted on the dev set, and the performance is measured per epoch for downstream tasks. Table 9 shows performances of the PLMs input features BiLSTM and BiGRU the dev set per epoch.
30
60-
40
MF1
50-
MI11
30
30
10
2
10
2
10
Epoch
Epoch
Epoch
Emotion Regconition
(b) Hate Speech Detection
(c) Sentiment Analysis
70
ViSoBERT-BIL.S
60
PhoBERT BILSTM
MFF
MF11
TWIIN-DALSTM
VISGBERT-BAGRU
75
PhOBERT-BIGRU
70-
2
10
10
Epoch
Epoch
(d) Spam Reviews Detection
(c) Hate Speech Spans Detection
Figure Performances the PLMs as input features BiLSTM BiGRU on dev set per epoch on Vietnamese social media downstream tasks, Large versions of PhoBERT and TwHIN-BER BERT are implemented for these experiments.

---

E Updating New Spans of Hate Speech Span Detection Samples with Pre- Pre-processing Techniques
Due to performing pre-processing techniques, the span positions on the data samples can be changed. Therefore, we present Algorithm I, which shows how to update new span positions of samples applied with pre-processing techniques in the Hate Speech Spans Detection task (UIT-ViHOS dataset). This algorithm takes input comment and its spans and returns the pre-processed comment and its span along with pre-processing techniques.
Algorithm 1 Updating new spans of samples applied with pre-processing techniques in Hate Speech Spans Detection task (UIT-ViHOS dataset).
1: procedure ALGGORITHD(Coommen, label, delete)
2: assert len(comment) == len(label)
3: new_comment + []. new new_label - []
4: for i 0 to len(comment) do
5: check
6: if comment[i] in emoji_to, to srooji_tt_wood.ooo werd.keys() then
7:
if delete then
continue
9:
for j 0 to len(emoji to_ word[comment(iilsspittt" ")) do
10:
if label[*] == 'B-T" then
11:
if check == 0 then
12:
check + check + 1, new www.label.aaper (abeed(lppen((l
13:
else
www_label.apppen((11)
15:
else
16:
new new_Label.appenn((abclii]]
17:
oow.yooommmraappppmeeeeoooooooom ')(j))
18:
else
19:
new wew_eentercccappenccommmnnttllli
20:
new new__labelapppen(((bbelli))
21:
assert (en(new_commen)) == len(new_labe)
22:
return new_comment, new new_label
F
Tokenizations of the PLMs on Removing Diacritics Social Comments
We analyze several data samples to see the tokenized ability of Vietnamese social media textual data while removing diacritics on comments. Table 10 shows several non-diacritics Vietnamese social comments and their tokenizations with the tokenizers of the three best pre-trained language models, ViSoBERT (ours), PhoBERT, and TwHIN-BERT

---

Model
Example
Example 2
cái con dó choi do mua dāu ahi cubi déo ahăt dude mám
Öi cái thanh niča håm lol. Dep måt quá
Raw comment
English: where did you buy that toy LMAO 808
English: Oh my god damn teenagers, lol. So deserved
Removing 100% diacritics comment
cai con do choi do mua nhi, cuoi deo nhat duoc mom.
Comment
Oi bo thanh nien ham Dep mut qua

, "cai". c'on,, "do", "cho é @ "do'. "maa".
"O G",, "V" "bo", "eai", i "Lu", ht "hanh" hi,, "ni dr". "en", "ham", "To @r, "T""", ", "D @ @* "mat", "qua", sunks, cunks, </c>
"o", "d @", "au", "nhi", ".. "cu @ @". "o.". "d @ "eo". "nh @ @", "at". "d @ @0, "oc".
PhoBERT
"m m". <unks, <unk>, <unk>, 
 
, "cai", "con". "do". "cho", " 'I", "do". "mmua" "o", 'dau". "nhi", " " "cu", "oi", "de", "o", "nha". "t", "du", 'oc", "mom",

> "Oi". "bo", "cai". "lu", "thanh". "nic", " "a", "ham". "lol". *. "De", "p", "mat", "qua", "h, r
TWHIN-BERS
</>>
<os, "cai", "con", "do", "chai", "do", "mua", "n", "dau", "nhi", "ou". 'ol", . "d", "co", "ahat", "duoc". "m". "om"."
"mua",
"bo", "cai", "lu", "thanh", "ni", ten",, "t",, "am". "Iol", r., "D", "ep", "mat", "qua",
ViSoBERT
</s
Removing 75% diacrities in each comment
cai di chiái do duu nhi, cuiói deo nhat danc
Comment
Ōi ho cui lu thanh niën häm lol. Dep mat qua
mom.
<xs, "eat", "can", "da", "chos", "do", "do" "mua", "b". "dāu", "n, "ouii" @ @", "eo", "nb @*, "art, "d @0", "oc", "m m", ".7, "enun,,, cunks.
"OF" "bo". "eai", "lu". "thanh ničn', n". m'. "lo @ @", "T.".. "De @ @" <unks, </o
PhoBERT
"""""". "p". "mat", "qua", sunks,
<unks, </o>

, "cai", "con", "do". "chai", "do", "mus", "o", "dau". "nhi", " "." "cudli" "de", "o", "nha", "I", "du", 'oc",
<sp, "O", "bo", "cai", "lu", "thanh", "ničn", "", "hä", "m", 'lol", ". "De", "p", "mat", "qua", ""
TWHIN-BERT
<dss

 <xs, "cai", "con", "do", "chai", "do"", "mua", "b", "din". "nhi", "cubi", 'd". "co". "nhat", "duoc". "m", "om". 

"mom", "

 'Oi.. "bo", "eai". "lu", "thanh". "n". "iča", "ha", "m".
ViSoBERT
"lol". "D", "op". "mar", "qua".
</>
Removing 50% diacrities in each comment
cai con do choi do mua dāu nhi, cuti deo nhăt duoc
Comment
Oi bo cai lu thanh ničn häm lol. Dep mät quá
mom.

, "eai", "con", 'do", "choi",
"mua"." "D". "dåu". "n",", "n "eudi" "d @ @". "éo", 'uhár'.
Of. "bo", "eai", "Tu", "thanh niên', "ham", "lo @", "I", "De @", "tt," "qud", sunks, <unk>, </o
PhoBERT
"du @ 6 "bre", "."" "momm" m".""" "." sunks, sunks, <unks, </>>
"cai", "con", "do", "chdi", "do", "mua", "o", "diu", "nhi", "cubi". "d". "6",""", "nh". "ăr". "du". 'oc". "mom".
<i. "O", 3", "bo", "cai", "lu", "thanh", "miën", "ha", "m"".
TWHIN-BERR BERT
<dss "lol". "De", "p", "måt". "qud", ""
" 


, "cai". "con', 'do", "choi", "do", "mua", "chu". "cutf", "déo", "nh", "it", "duoc", "m", "om", <e
<s, "Oi", "bo", "cai", "lu", "thanh", "n",, "iên", "hä", "m", "ool" ".,, "D", "ep", "mar", "qui",
VISoBERT
Removing 25% diacrities in each comment
cai con do choi dó mua à däu nhi. cuti déo nhăt duoc
Comment
môm. $60
bo cai lu thank ničn häm lol, Dep mutt quá
"cai", "can", 'do", " "choi", "d'6", "mua".""" "8', "dān", "nhi" "", "iiiii "d @ @0,, "én", I", "du @ "oc", "m m², sunko, sunko, sunks, 
 
, "cai", "con", "do". "choi", "do". "mus", "o", "dau". "nhi", ","," "cadii" "d", "8", to", "nh", "ät", "du", "oc", "mom", on on "
"O "bo", "caf", "Iu", "th Thannn niën", n", i'm" "Ia @", "T", "Dep. at", "m 31"", "qu cunko, sunks, 

PhoBERT

. "O", 5", "bo", "cai", "lu", "thanh", "niên", "", "hä", "m", "lol", ". "Dep", "mar", "quá", "",
TWHIN-BERT
<xs, "cai", "can", "do", "chat", "d6". "mua", "8", "dhu". "nhi". +.. "cubf", 'doo", "ab", "ar". "duoe". "môm". 
>

> "Oi", "bo", feai", "lu", "thanh", "n",, "ien", "ba", "m". 'lol". "Dep", "mar", "quá".
ViSoBERT


Table 10: Actual social comments and their tokenizations with the tokenizers of the three pre-trained language mod- els, including PhoBERT, TWHIN-BERS and ViSoBERT, on removing diacritics of social comments. Disclaimer: This table contains actual comments on social networks that might be construed as abusive, offensive, or obscene.


Reviews:
[{"review_id": "cI3XMniihG", "reviewer": "Reviewer_iNVu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ${\\color{Blue}iNVu}$,\n\nWe thank the reviewer ${\\color{Blue}iNVu}$ for taking the time to read this paper and suggest several valuable comments. Please find our detailed responses below. Please find our detailed response below.\n\n**Q: There is less information regarding the data scrapped for pretraining.**\n\nAs mentioned in the paper, our pretraining data is collected from the three most popular Vietnamese social media platforms, including Facebook, TikTok, and YouTube. To effectively gather data from these platforms, we harnessed the capabilities of specialized tools provided by each platform:\n\n1. Facebook: We crawled comments from Vietnamese verified pages by Facebook (including government pages, celebrity pages, organization pages, etc.) posts via the *Facebook Graph API* between January 2016 and December 2022. \n2. Tiktok: We collected comments from Vietnamese-verified channels via *Tiktok Research API* between January 2020 and December 2022.\n3. Youtube: We scrapped comments from Vietnam-verified channels' videos via *Youtube Data API* between January 2016 and December 2022. \n\nAs a result, our pretraining data after crawling and preprocessing contains 1GB of uncompressed text. Our pretraining data is available only for research purposes.\n\nNote that a verified page means the platform has confirmed that the Page or profile is the authentic presence of the individual, public figure, or brand it represents and verified channels.\n\n**Q: The paper didn't specify if the data is going to be open sourced.**\n\nThank you for your insightful suggestion. In the paper, lines 27-28, we noted that \"Our PLM4ViSMT model is available only for research purposes.\" we noted that pretraining data would also be open-sourced.\n\n**Q: The PLM proposed in the paper should have been compared with other LM trained specifically for Social Media tasks.**\n\nIn the paper, we implemented TwHIN-BERT, a multilingual PLM for social media tasks (including Vietnamese), and other SOTA pre-trained language models for Vietnamese such as PhoBERT, XLM-R, etc. However, we agree with the reviewer that more LMs, specifically for Social Media tasks, must be implemented and compared to our PLM4ViSMT. We conducted additional experiments regarding your concerns by comparing our pre-trained model with XLM-T [1] and Bernice [2] (multilingual PLMs for social media, including Vietnamese). \n\nOur results indicate that our PLM4ViSMT continues to exhibit superior performance compared to both XLM-T and Bernice across 5 Vietnamese social media tasks. XLM-T, TwHIN-BERT, and Bernice, like TwHIN-BERT, were exclusively trained on pretraining data from the Twitter platform. However, this limitation hinders these models from effectively capturing the intricate nuances of the Vietnamese social media context. The primary reason is that Twitter isn't a widely utilized social media platform in Vietnam, and therefore, the training data from this source may not accurately encapsulate the linguistic intricacies and contextual subtleties prevalent within the Vietnamese social media landscape. See the table below for the full results.\n\n| **Model**      | **Avg** | **Emotion Recognition** |          |          | **Hate Speech Detection** |          |          | **Sentiment Analysis** |        |        | **Spam Reviews Detection** |          |          | **Hate Speech Spans Detection** |          |          |\n| :------------- | :-----: | :---------------------: | :------: | :------: | :-----------------------: | :------: | :------: | :--------------------: | :----: | :----: | :------------------------: | :------: | :------: | :-----------------------------: | :------: | :------: |\n|                |         | Acc                     | WF1      | MF1      | Acc                       | WF1      | MF1      | Acc                    | WF1    | MF1    | Acc                        | WF1      | MF1      | Acc                             | WF1      | MF1      |\n| XLM-T          | 72\\.23  | 64\\.64                  | 64\\.37   | 59\\.86   | 86\\.22                    | 86\\.12   | 63\\.48   | 75\\.66                 | 75\\.6  | 75\\.6  | 90\\.07                     | 90\\.11   | 76\\.66   | 90\\.88                          | 90\\.88   | 85\\.53   |\n| Bernice        | 72\\.49  | 64\\.21                  | 64\\.27   | 60\\.68   | 86\\.12                    | 86\\.48   | 64\\.32   | 74\\.57                 | 74\\.90 | 74\\.90 | 90\\.22                     | 90\\.21   | 76\\.89   | 90\\.48                          | 90\\.06   | 85\\.67   |\n| **PLM4ViSMT** | **75\\.65**  | 68\\.10                | 68\\.37 | 65\\.88 | 88\\.51                  | 88\\.31 | 68\\.77 | 77\\.83                 | 77\\.75 | 77\\.75 | 90\\.99                   | 90\\.92 | 79\\.06 | 91\\.62                        | 91\\.57 | 86\\.80 |\n\nSince Bernice is the only multilingual model trained from scratch on Twitter data with a custom tokenizer, we also conducted experiments to verify its tokenizer on Vietnamese social media text. Unsurprisingly, our experiments revealed that Bernice's tokenizer performs poorly compared to most of our established baseline models, excluding viBERT and mBERT. This outcome further emphasizes the significance of our PLM4ViSMT's specialized tokenizer in effectively handling Vietnamese social media text (which is showcased in Section 3.3, \"The Vietnamese Social Media Tokenizer\" of the paper). See the table below for the results.\n\n| **Model**      | **Avg** | **Emotion Recognition** | **Hate Speech Detection** | **Sentiment Analysis** | **Spam Reviews Detection** | Hate Speech Spans Detection  |\n| :------------- | :-----: | :---------------------: | :-----------------------: | :--------------------: | :------------------------: | :------------------------------: |\n| XLM-T          | 26\\.77  | 18\\.59                  | 17\\.74                    | 39\\.45                 | 39\\.21                     | 18\\.87                           |\n| Bernice        | 28\\.41  | 19\\.87                  | 18\\.82                    | 41\\.41                 | 41\\.74                     | 20\\.20                           |\n| **PLM4ViSMT** | **23\\.98**  | 16\\.23                  | 15\\.30                    | 37\\.30                 | 34\\.81                     | 16\\.26                           |\n\nUpon publication, we will move these results forward to the main experimental results using the extra page of the camera-ready version.\n\n**References**\n\n[1] Barbieri et al., \"XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond\" LREC 2022\n\n[2] DeLucia et al., \"Bernice: A Multilingual Pre-trained Encoder for Twitter\" EMNLP 2022\n\n**Q: Line 215: large corpus of 01 GB of uncompressed texts. Is this a typo?**\n\nWe thank you for pointing this typo out. We noted that our pretraining data contains only 1GB of uncompressed text. We have revised it.\n\nWe believe these changes improved the quality of the article and would like to thank the reviewers again for their valuable comments and suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper focuses on building PLM for social media task in Vietnamese. Although the language is spoken by 100M people, not a lot of progress has been done in the field of social media tasks.\n2. This is the first monolingual PLM focusing on social media task in Vietnamese and have achieved SOTA results for Emotion Recognition, Hate Speech Detection, Spam Reviews Detection, Hate Speech Spans Detection. Although it doesn't achieve SOTA results for Sentiment Analysis task, it outperforms the existing general Vietnamese LM.\n3. Detailed ablation studies is carried out to show importance of different aspects of model like including emojis, removing emojis, converting emojis to text.", "reasons_to_reject": "1. There is less information regarding the data scrapped for pretraining. \n2. The paper didn't specify if the data is going to be open sourced.\n3. The PLM proposed in the paper should have been compared with other LM trained specifically for Social Media tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a PLM for Vietnamese Social Media. It is the first model based on the XLM-R architecture. They achieve SOTA results across different social media downstream tasks. Their model is the first one with custom tokeniser for Vietnamese social media. They have reported an extensive study to analyse their model which include different variations of masking, including emojis, teencode, and diacritics etc.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 215: large corpus of 01 GB of uncompressed texts.  Is this a typo?"}, "extra_text": {}}, {"review_id": "b4FSVCtyAj", "reviewer": "Reviewer_mcGf", "summary": "", "questions": "- Sec 3.1 : How were Vietnamese social media text identified?\n- Sec 4.2 : Were the results computed based on single runs or average of multple runs? \n- How did training loss compare with continued pre-training of other Vietnamese Models? Instead of training from sratch continued pretraining of previous models would achieve better results?\n- Further the baseline models when continued pre-trained usiing the same data can still perform better , with lesser training data. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ${\\color{Orange}mcGf}$,\n\nWe thank the reviewer ${\\color{Orange}mcGf}$ for taking the time to read this paper and suggest several valuable comments. Please find our detailed responses below.\n\n**Q: Insufficient details on how Vietnamese social media text were identified and collected.**\n\nAs mentioned in the paper, our pretraining data is collected from the three most popular Vietnamese social media platforms, including Facebook, TikTok, and YouTube. To effectively gather data from these platforms, we harnessed the capabilities of specialized tools provided by each platform:\n\n1. Facebook: We crawled comments from Vietnamese verified pages by Facebook (including government pages, celebrity pages, organization pages, etc.) posts via the *Facebook Graph API* between January 2016 and December 2022. \n2. Tiktok: We collected comments from Vietnamese-verified channels via *Tiktok Research API* between January 2020 and December 2022.\n3. Youtube: We scrapped comments from Vietnam-verified channels' videos via *Youtube Data API* between January 2016 and December 2022. \n\nAs a result, our pretraining data after crawling and preprocessing contains 1GB of uncompressed text. Our pretraining data is available only for research purposes.\n\nNote that a verified page means the platform has confirmed that the Page or profile is the authentic presence of the individual, public figure, or brand it represents and verified channels. \n\n**Q: Details regarding the behavior during training would be useful in assessing the relative improvements in the model due to training.**\n                    \nThanks a lot for your helpful suggestion! We're excited about trying out these extra experiments in our future work. Your insightful suggestion has given us a great new direction to explore.               \n\n**Q: Baselines : Comparing other multilingual pre-trained models trained on social media text would be better benchmarks - like BERNICE.**\n\nIn the paper, we implemented TwHIN-BERT, a multilingual PLM for social media tasks (including Vietnamese), and other SOTA pre-trained language models for Vietnamese such as PhoBERT, XLM-R, etc. However, we agree with the reviewer that more LMs, specifically for Social Media tasks, must be implemented and compared to our PLM4ViSMT. We conducted additional experiments regarding your concerns by comparing our pre-trained model with XLM-T [1] and Bernice [2] (multilingual PLMs for social media, including Vietnamese). \n\nOur results indicate that our PLM4ViSMT continues to exhibit superior performance compared to both XLM-T and Bernice across 5 Vietnamese social media tasks. XLM-T, TwHIN-BERT, and Bernice, like TwHIN-BERT, were exclusively trained on pretraining data from the Twitter platform. However, this limitation hinders these models from effectively capturing the intricate nuances of the Vietnamese social media context. The primary reason is that Twitter isn't a widely utilized social media platform in Vietnam, and therefore, the training data from this source may not accurately encapsulate the linguistic intricacies and contextual subtleties prevalent within the Vietnamese social media landscape. See the table below for the full results.\n\n| **Model**      | **Avg** | **Emotion Recognition** |        |        | **Hate Speech Detection** |        |        | **Sentiment Analysis** |        |        | **Spam Reviews Detection** |        |        | **Hate Speech Spans Detection** |        |        |\n| :------------- | :-----: | :---------------------: | :----: | :----: | :-----------------------: | :----: | :----: | :--------------------: | :----: | :----: | :------------------------: | :----: | :----: | :-----------------------------: | :----: | :----: |\n|                |         | Acc                     | WF1    | MF1    | Acc                       | WF1    | MF1    | Acc                    | WF1    | MF1    | Acc                        | WF1    | MF1    | Acc                             | WF1    | MF1    |\n| XLM-T          | 72\\.23  | 64\\.64                  | 64\\.37 | 59\\.86 | 86\\.22                    | 86\\.12 | 63\\.48 | 75\\.66                 | 75\\.6  | 75\\.6  | 90\\.07                     | 90\\.11 | 76\\.66 | 90\\.88                          | 90\\.88 | 85\\.53 |\n| Bernice        | 72\\.49  | 64\\.21                  | 64\\.27 | 60\\.68 | 86\\.12                    | 86\\.48 | 64\\.32 | 74\\.57                 | 74\\.90 | 74\\.90 | 90\\.22                     | 90\\.21 | 76\\.89 | 90\\.48                          | 90\\.06 | 85\\.67 |\n| **PLM4ViSMT** | **75\\.65**  | 68\\.10                  | 68\\.37 | 65\\.88 | 88\\.51                    | 88\\.31 | 68\\.77 | 77\\.83                 | 77\\.75 | 77\\.75 | 90\\.99                     | 90\\.92 | 79\\.06 | 91\\.62                          | 91\\.57 | 86\\.80 |\n\nSince Bernice is the only multilingual model trained from scratch on Twitter data with a custom tokenizer, we also conducted experiments to verify its tokenizer on Vietnamese social media text. Unsurprisingly, our experiments revealed that Bernice's tokenizer performs poorly compared to most of our established baseline models, excluding viBERT and mBERT. This outcome further emphasizes the significance of our PLM4ViSMT's specialized tokenizer in effectively handling Vietnamese social media text (which is showcased in Section 3.3, \"The Vietnamese Social Media Tokenizer\" of the paper). See the table below for the results.\n\n| **Model**      | **Avg** | **Emotion Recognition** | **Hate Speech Detection** | **Sentiment Analysis** | **Spam Reviews Detection** | Hate Speech Spans Detection |\n| :------------- | :-----: | :---------------------: | :-----------------------: | :--------------------: | :------------------------: | :------------------------------: |\n| XLM-T          | 26\\.77  | 18\\.59                  | 17\\.74                    | 39\\.45                 | 39\\.21                     | 18\\.87                           |\n| Bernice        | 28\\.41  | 19\\.87                  | 18\\.82                    | 41\\.41                 | 41\\.74                     | 20\\.20                           |\n| **PLM4ViSMT** | **23\\.98**  | 16\\.23                  | 15\\.30                    | 37\\.30                 | 34\\.81                     | 16\\.26                           |\n\nUpon publication, we will move these results forward to the main experimental results using the extra page of the camera-ready version.\n\n**References**\n\n[1] Barbieri et al., \"XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond\" LREC 2022\n\n[2] DeLucia et al., \"Bernice: A Multilingual Pre-trained Encoder for Twitter\" EMNLP 2022\n\n**Q: Sec 4.2 : Were the results computed based on single runs or average of multple runs?**\n\nAll of our results were computed based on a single run using the same seed for consistency across all our experiments.  However, it's important to note that we also undertook an extensive series of ablation studies to further validate the exceptional performance of our pre-trained model in comparison to other approaches. These ablation studies were meticulously designed to isolate and analyze the impact of different factors. \n\nBy carrying out these ablation studies, we could showcase not only the outstanding performance of our pre-trained model but also provide insights into its inner workings. The results of these studies collectively support our claim of superiority over other existing models and underscore the comprehensive nature of our pre-trained model.\n\n**Q: L215: 01 GB of uncompressed texts? I am assuming this is a typo.**\n\nWe thank you for pointing this typo out. We noted that our pretraining data contains only 1GB of uncompressed text. We have revised it.\n\nWe are grateful to the reviewers for your valuable comments and recommendations. These comments helped us to improve the quality of the article better. We hope the reviewers will be satisfied with our revisions according to your comments and recommendations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Authors propose a new model suited for Vietnamese social media text processing needs. Model presented outperforms other models specific to Vietnamese language. \n- Authors conduct extensive experiments on downstream tasks to showcase improvements on downstream tasks. ", "reasons_to_reject": "- Insufficient details on how Vietnamese social media text were identified and collected. \n- Details regarding the behavior during training would be useful in assessing the relative improvements in the model due to training. \n- Baselines : Comparing other multilingual pre-trained models trained on social media text would be better benchmarks -  like BERNICE. \n", "questions_to_authors": "- Sec 3.1 : How were Vietnamese social media text identified?\n- Sec 4.2 : Were the results computed based on single runs or average of multple runs? \n- How did training loss compare with continued pre-training of other Vietnamese Models? Instead of training from sratch continued pretraining of previous models would achieve better results?\n- Further the baseline models when continued pre-trained usiing the same data can still perform better , with lesser training data. ", "paper_topic_and_main_contributions": "Authors present a new pretrained LM, PLM4ViSMT, trained on Vietnamese social media text extracted from social media networks - Facebook, TikTok and YouTube. Authors compare previous pre-trained PLMs performance on downstream task with that of PLM4ViSMT. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L215: 01 GB of uncompressed texts? I am assuming this is a typo. \n- Including some more details on the training performance and plots would help in following up on the research presented in the paper. "}}, {"review_id": "rtqXdgLfX0", "reviewer": "Reviewer_117r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ${\\color{Green}117r}$,\n\nWe would like to thank reviewer ${\\color{Green}117r}$ for your really positive reviews. We firmly believe that conducting additional experiments and adding supplementary information based on the comments and suggestions from reviewers ${\\color{Orange}mcGf}$ and ${\\color{Blue}iNVu}$ would significantly enhance the quality of our paper. We're committed to making these improvements to create a better version of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They introduce the initial PLM that utilizes the XLM-R architecture and pretraining method specifically designed for handling Vietnamese social media text.", "reasons_to_reject": "-", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors introduce PLM4ViSMT, the initial monolingual pre-trained language model designed specifically for Vietnamese social media texts. This model, based on the XLM-R architecture, is trained on a vast and varied collection of high-quality Vietnamese social media texts. They tested the pre-existing model on five significant natural language tasks related to Vietnamese social media content. These tasks included identifying emotions, detecting hate speech, analyzing sentiment, detecting spam reviews, and identifying hate speech segments.\n", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper trains a monolingual MLM (XLM-R architecture) from scratch specifically for Vietnamese social media text. The data was scraped from popular Vietnamese social media sites and results on downstream tasks show that it outperforms other available models. 

The reviewers agree that the paper, data and models can be impactful for Vietnamese social media language understanding. The authors have provided some missing experimental results during the response period while other concerns regarding further training existing models on the newly collected data have been punted to future work.
Decision: Accept-Main