<figure><img src="image_1.png" /></figure>

HAL
open science

# Multi-feature Transformer for Multiclass Cyberbullying Detection in Bangla

Zaman Wahid, Abdullah Al Imran

► To cite this version:

Zaman Wahid, Abdullah Al Imran. Multi-feature Transformer for Multiclass Cyberbullying Detection in Bangla. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.439-451, 10.1007/978-3-031-34111-3_37. hal-05449778

HAL Id: hal-05449778

https://inria.hal.science/hal-05449778v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Multi-feature Transformer for Multiclass Cyberbullying Detection in Bangla

Zaman Wahid¹(✉) and Abdullah Al Imran²

¹ University of Calgary, Calgary, AB, Canada
zaman.wahid@ucalgary.ca

² University of Liverpool, Liverpool, UK
a.al-imran@liverpool.ac.uk

**Abstract.** Cyberbullying detection is a global issue that must be addressed to improve the cyberspace for millions of online users, services, and organizations. Online harassment of the general public and celebrities is now commonplace on social media, particularly in Bangladesh. In this paper, we present a novel multi-feature transformer followed by a deep neural network for multiple-dimensional cyberbullying detection. Using online Bangla textual data, we introduce the user's social profile, the lexical features, the contextual embedding, and the semantic similarities among word associations in Bangla in order to develop an effective and robust cyberbullying detection system. Our proposed method can detect cyberbullying in Bangla with a 98% detection accuracy for threats and a 90% detection accuracy for sarcastic comments. The aggregate accuracy of all six multiclass labels is 86.3%. In addition, the experimental results find that the proposed technique outperforms the state-of-the-art methods for detecting cyberbully in Bangla.

**Keywords:** Cyberbullying Detection from Text · Online Harassment Detection in Bangla · Natural Language Processing

## 1 Introduction

In the context of social media in Bangladesh, cyberbullying has become a daily occurrence for individuals and celebrities. Women are more vulnerable to cyberbullying, which can be motivated by body shaming, sexuality, religious fanatism, and other factors. While it may seem typical to experience smaller impacts on a daily basis, the repercussions for those who do so routinely are dangerous. Despite Bangladesh's claims to be a digitally advanced nation, it is cautioned in recent national news [26] that there are increasingly more instances of cyberbullying nowadays. Defamation and victim shaming are the two most common

<footnote>Z. Wahid and A. Al Imran—Both authors contributed to this work equally.</footnote>
forms of cyberbullying. The connection between a close-knit communal culture and anonymous harassment, as well as the lack of victim support infrastructure, are two of the most important factors for the increasing prevalence of cyberbullying in Bangladesh [23]. Computer-based Artificial Intelligence (AI) systems to aid in the identification of these potential cyberbullying activities have become essential, as it is nearly impossible to detect cyberbullying and take appropriate action using manual labor, especially given the massive volume of cyberspace and social network profiles in Bangladesh. Policymakers, scientists, social activists-even social media-based companies offering a variety of services in Bangla-are realizing and adapting to AI technologies that can combat online abuse in Bangla and identify simple countermeasures. In recent years, machine learning-based solutions have exhibited promising results in detecting online rumors and fake news [4,33], sexual predators [34], assessing trustworthiness [19], etc. However, Bangla is a low-resource language, which increases the difficulty of dealing with Bangla textual data and developing machine learning models in general [29]. While there has been recent research on online harassment (cyberbullying) in Bangla, the quantity of state-of-the-art works is yet minimal. Therefore, to contribute to this critical and contemporary problem domain specific to Bangla, we introduce a multi-feature transformer approach for cyberbullying detection, in this study. These are the research questions posed in this paper:

- Can we design a multi-feature transformer-based deep learning architecture for cyberbullying detection in Bangla?

- Can a multi-feature transformer-based deep learning architecture perform similarly or better than the state-of-the-art cyberbullying detection methods in Bangla?

By pursuing the above-mentioned research questions in this paper, we add the following contributions to the problem domain of cyberbullying detection in Bangla.

- A novel multi-feature transformer-based deep learning architecture is proposed and implemented for cyberbullying detection in Bangla.

- A set of experimentations is performed using a bunch of different unimodal and multi-feature architectures, followed by a thorough analysis of the performance of the models for cyberbullying detection in Bangla.

- It is demonstrated that the proposed method outperforms the state-of-the-art methods in this problem domain.

In the sections that follow, the related works, dataset description, proposed methodology, experimental result analysis, etc. are discussed in detail.

## 2 Related Works

Online cyberbullying detection has been a research domain for some time, since the field of study has attracted substantial interest from researchers and
government-crime-handling agencies worldwide. However, cyberbullying detection research in the context of Bangla language-specific domains still requires improvement, as Natural Language Processing (NLP) is still evolving its technologies for dealing with non-English languages, notably Bangla. A recent research [11] has introduced a transformer-based technique for detecting online harassment, expressing it as a multilabel classification problem. To better recognize abusive and hateful remarks from Online Social Network (OSN) users, the labels are divided into five categories: threat, sexual, troll, religious, and not bully. Three transformer models have been utilized: Bangla Bidirectional Encoders Representations from Transformers (BERT) [16], Bengali DistilBERT [28], and XLM-RoBERTa [36]. The best accurate model was the XML-RoBERTa model, with a maximum accuracy of 85%. A binary and multiclass classification model employing a Hybrid Neural Network [25] is proposed in [2]. The authors of [2] collected a large sample of Bangla cyberbullying data-Facebook user comments-and curated a large dataset with 44001 records and five annotated class labels. Their initial multiclass classification accuracy was 79.29%. However, by employing ensemble-based classical Machine Learning (ML) techniques such as Support Vector Machine (SVM) [31], K-nearest Neighbors (KNN) [24], Random Forest (RF) [8], and Naive Bayes (NB) [35], the performance was somewhat improved. Another recent study [7] on Bangla cyberbullying detection developed a novel approach that employs Efficiency Learning, an Encoder that Classifies Token Replacements Accurately (ELECTRA) [22], and BERT multi-lingual [10] pre-trained architectures. Using the same dataset as [2,11], the experiment was conducted. With ELECTRA, the performance of the model with the highest accuracy was 84.92%. The authors of [6] proposed Convolutional Neural Network (CNN) [5] and Long Short Term Memory (LSTM) [12] based techniques for online detection of abusive Bangla comments. Their models can recognize offensive Bangla comments based on religious, personal, geopolitical, and political motivations. Using networks such as CNN, Bi-LSTM [14], Conv-LSTM [20], and SVM models, the model with the highest reported performance is 78% accurate. Regarding the current and future trends of cyberbullying detection in all languages, Talpur et al. [32] provides a summary of the obstacles that must be surmounted in order to design a more effective cyberbullying detection system. Jahan et al. [15] created a Bangla cyberbullying detection system comprised of unigrams, bigrams, emojis, and sentiment words, among others. This study utilizes both transliterated and code-mixed Bangla and English text, in addition to the Bangla language. On a proprietary dataset, the authors trained SVM, Random Forest, and Adaboost algorithms and reported an accuracy of 72.14%. The paper [27] proposes a method for identifying abusive Bangla comment emanating from a radio broadcast gateway in real-time. The system uses Multinomial Naive Bayes (MNB) [35], RF, and Logistic Regression (LR) [17] to detect whether a radio broadcast gateway message is abusive. By filtering offensive comments, the proposed method can compare and map real-time communications to the dataset and isolate the positive comments or messages. The maximum reported accuracy for this proposed approach is 75.50%. A comprehensive comparative
study [3] of methods used for cyberbullying detection in Bangla revealed that the number of works is limited when comparing classical machine learning-based algorithms to deep learning-based algorithms, and the performance of the models ranges from 75% to 85% for both binary and multiclass classification. Almost all systems used unimodal approaches for cyberbullying detection in Bangla; however, multi-feature transformer-based approaches were never investigated. Multi-feature transformers can capture a more comprehensive understanding of the input data by utilizing multiple modalities that contribute to extracting more useful information and making more accurate predictions compared to unimodal approaches. Therefore, to develop a superior, more robust, and more effective system for cyberbullying detection in Bangla, we propose a multi-feature transformer approach for this research problem. To the best of our knowledge, we propose and investigate a multi-feature transformer approach for detecting cyberbully in Bangla for the first time.

# 3 Methodology

The proposed methodology introduce a multi-feature transformer based approach for cyberbullying detection in Bangla. The process of feature engineering includes contextual sentence embedding, baseline profiles, word embedding, and lexical profiles. Figure 1 illustrates the overall process of the proposed architecture for detecting cyberbully in Bangla.

<figure><img src="image_4.png" /><figcaption>Fig. 1. Workflow of the proposed methodology for cyberbullying detection in Bangla</figcaption></figure>

## 3.1 Data Preprocessing

Data preprocessing is a crucial component for improving the development of machine learning models, particularly in the field of Natural Language Processing. It is essential when dealing with languages with limited resources, such as
Bangla. Since the majority of training samples are informal Bangla Facebook comments, the following three textual data preparation techniques are used. In this step, three preprocessing tasks-denoising, normalizing, and stemming-have been performed. When data is extracted from many web sites, HTML tags, extra spaces, character encoding, and escape codes often appear. To make the data useable, these excessive noises must be removed, a process known as “denoising” [18]. A hard-coded program is implemented to automate the denoising process by removing noises, HTML tags, and extra white spaces. Text data should be normalized in various ways depending on the language, including case conversion, elimination of punctuation, accent marks, diacritics, stop words, and sparse terms. The Bangla text data were therefore normalized. In particular, a Bengali rule-based stemmer [21] is utilized for stemming.

## 3.2 Feature Engineering

In a machine learning pipeline, feature engineering plays a key role. From Bangla textual data, four dimensions of features are extracted: contextual sentence embedding, baseline profile, word embedding vectors, and lexical profile.

**Contextual Embedding:** In contrast to conventional word embedding vector representation techniques, we extract contextual information from online Bangla comments. The context is essential, particularly when it comes to comprehending and identifying cyberbully in Bangla. The Bi-directional Encoder Representations from Transformers (BERT) [16] pre-trained model is used to extract context from Bangla text. A multi-layer, bidirectional transformer designed for this model's architecture enables it to learn the context from both the left and right sides of unlabeled text. Due to its unified architecture, BERT distinguishes itself from other traditional algorithms in that the pre-trained BERT model may be improved to produce cutting-edge models for a variety of NLP applications with only one additional output layer. The unsupervised learning techniques Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) are used to pre-train BERT. From the sequence output layer, the contextual embedding information was extracted to be used for cyberbullying detection in Bangla.

**Social Profile:** OSN users' social profile is considered a feature set for cyberbullying detection in Bangla because the baseline information is found relevant for whether a person has the potential to be harassed online or not. Gender, user category, and reaction counts are the baseline profile of users. In the user's category, there are categories like Singer, Politician, Actor, etc. The number of reactions to a particular comment is also counted.

**Word Embedding:** For word embedding, we utilized the Word2vec [9] algorithm. It is a two-layer neural network technique that “vectorizes” words by
employing a neural network model to learn word relationships from a large corpus of text. It accepts a text corpus as input and generates a collection of 300-length feature vectors that collectively describe the corpus' words. Word2Vec keeps track of the sequence in which words appear in a sentence; this additional context helps to clarify the meaning of certain terms. It is a compilation of model architectures and optimizations that have proven successful for a number of subsequent NLP tasks. Using this Word2Vec word embedding technique, the Bangla texts' corresponding features are extracted.

**Lexical Features:** The lexical characteristics of Bangla comments of OSN users are extracted. In addition to word length, frequency, and high-frequency terms, additional lexical properties are incorporated. Language, genre, and other factors can influence the lexical features and collocation of a language. A total of 33 lexical characteristics are extracted and employed.

## 3.3 Experimental Design

This section describes the different types of multi-feature transformer architectures utilized for cyberbullying detection in Bangla. A multi-feature transformer is a deep learning architecture that operates on multiple input features to generate an output. Recent empirical studies have demonstrated the efficacy of multi-feature transformers for NLP tasks, leading to more accurate outputs [13]. Figure 2 depicts the workflow of the multi-feature transformer architecture used in this study.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Workflow of the multi-feature transformer architecture</figcaption></figure>

We proposed to utilize multi-feature transformer architecture for cyberbullying detection in Bangla because, typically, these models include an encoder that processes each modality and a decoder that combines the encoder's representations to produce the final output. Subsequently, multiple modalities can offer the
model with a more comprehensive grasp of the input data, resulting in enhanced performance. In this research problem, the online Bangla comments are the input for the BERT transformer module that processes the textual data and extract the corresponding contextual features. On the other hand, the non-textual features, including social profile and lexical features, directly pass through the combining module of the multi-feature transformer architecture. It should be noted that additional modules can be incorporated as well to handle textual data as necessary. For instance, alongside the BERT module, we utilize the Word2Vec module to generate the features of the Bangla textual data, followed by passing the features to the combining module altogether. For the BERT module, we use three distinct architecture of BERT to produce better deep features. The architectures of the BERT module are: BERT with individual MLPs on categorical and numerical features then concat, BERT with attention on categorical and numerical features, and BERT with gating on categorical and numerical features then sum.

**BERT with Individual MLPs Then Concat:** In this architecture, BERT is used to encode the text data into a vector representation and the Multilayer Perceptrons (MLPs) are used to encode the categorical and numerical features into different vector representations. Finally, these representations are concatenated to form a combined representations, which is used as input to a final layer to predict the target variable. This architecture is useful when the data contains both text and non-text information, and the goal is to utilize both types of information in a unified model. To refer this architecture, we use the term MF01.

**BERT with Attention Mechanism:** The attention mechanism in BERT allows the model to focus on the most relevant features and assign different weights to different features based on their importance. The attention-weighted representation of the categorical and numerical features is then combined with the representation from BERT to make the final classification. This architecture is effective when the objective is to combine both forms of features in a unified model while providing greater weight to the relevant characteristics. To refer this architecture, we use the term MF02.

**BERT with Gating Mechanism Then Sum:** The gating mechanism in BERT helps to control the flow of information in the network and decide which information to pass on to the next layer. The final representations produced by the BERT model, with the added gating mechanism, are then summed to produce the final output. This is quite useful in this research as the Bangla comments are converted as sequence of tokens to produce a representation that captures the semantic meaning of the input sequence as well. To refer this architecture, we use the term MF03.

Besides the multi-feature transformer architectures, we additionally designed a unimodal architecture with a simple BERT and DNN only. A total of 10 distinct
<table><caption>**Table 1.** Experimental Design of the Proposed Methodology</caption><thead><tr><th>Experiment Name</th><th>Architecture</th><th>Feature Set</th></tr></thead><tbody><tr><td>UM00</td><td>Simple BERT + DNN</td><td>*BC</td></tr><tr><td>MF01</td><td>BERT with **iMLPs + concat</td><td>*BC + *SP</td></tr><tr><td>MF02</td><td>BERT with **ATT</td><td>*BC + *SP</td></tr><tr><td>MF03</td><td>BERT with **GCM + sum</td><td>*BC + *SP</td></tr><tr><td>MF01-Ext</td><td>BERT with **iMLPs + concat</td><td>*BC + *SP + *LP</td></tr><tr><td>MF02-Ext</td><td>BERT with **ATT</td><td>*BC + *SP + *LP</td></tr><tr><td>MF03-Ext</td><td>BERT with **GCM + sum</td><td>*BC + *SP + *LP</td></tr><tr><td>MF01-Ext-W2V</td><td>BERT with **iMLPs + concat</td><td>*BC + *SP + *LP + Word2Vec</td></tr><tr><td>MF02-Ext-W2V</td><td>BERT with **ATT</td><td>*BC + *SP + *LP + Word2Vec</td></tr><tr><td>MF03-Ext-W2V</td><td>BERT with **GCM + sum</td><td>*BC + *SP + *LP + Word2Vec</td></tr></tbody></table>

*BC = Bangla Comments, *SP = Social Profile, *LP = Lexical Profile

**iMLPs = Individual MLPs on categorical and numerical features, **ATT = Attention mechanism on categorical and numerical features, **GCM = Gating mechanism on categorical and numerical features

architectures is designed and experimented combining these architectures with different feature sets. Table 1 demonstrates the experimental design of the study.

# 4 Experimental Results and Analysis

This section thoroughly presents the experimental results produced by the different sets of architectures experimented with, in this study. Primarily, the result analysis is divided into two parts. The first one discusses the performances of all the different experiments conducted with several multi-feature transformer architectures. The latter demonstrates a comparative analysis of the performance of the proposed method with the state-of-the-art methods.

## 4.1 Dataset

We have used a publicly available benchmark dataset [1] for cyberbullying detection in Bangla. The size of the dataset is 44001 consisting of online users' comments on Facebook, labeled in five different harassment and threat-wise classes. In addition to having users' comments in Bangla language, the baseline profile information of users is recorded. The dataset is split into 80:10:10 with a stratified sampling technique [30] for training, validating, and testing the models. The optimal parameters of the best performing architecture are: learning rate = 5 × 10⁻⁵, epochs = 500, activation function = Gaussian Error Linear Unit (GELU) with Adam optimizer.
## 4.2 Performance Analysis of the Proposed Method

Each model with a unique collection of architectures is validated with a validation set prior to being tested on unobserved original data. Table 2 depicts both the validation and test performance on validation and test data.

<table><caption>**Table 2.** Validation and Test Performance on different set of multi-feature architectures</caption><thead><tr><th rowspan="2">Experiment Name</th><th colspan="3">Validation Performance</th><th colspan="3">Test Performance</th></tr><tr><th>Accuracy</th><th>F1 Micro</th><th>F1 Macro</th><th>Accuracy</th><th>F1 Micro</th><th>F1 Macro</th></tr></thead><tbody><tr><td>UM00</td><td>0.839286</td><td>0.839286</td><td>0.825081</td><td>0.840076</td><td>0.840076</td><td>0.836686</td></tr><tr><td>MF01</td><td>0.841340</td><td>0.841340</td><td>0.824784</td><td>0.839760</td><td>0.839760</td><td>0.834561</td></tr><tr><td>MF02</td><td>0.843394</td><td>0.843394</td><td>0.826053</td><td>0.840392</td><td>0.840392</td><td>0.836593</td></tr><tr><td>MF03</td><td>0.842446</td><td>0.842446</td><td>0.829549</td><td>0.838654</td><td>0.838654</td><td>0.830090</td></tr><tr><td>MF01-Ext</td><td>0.839286</td><td>0.839286</td><td>0.825081</td><td>0.840076</td><td>0.840076</td><td>0.836686</td></tr><tr><td>MF02-Ext</td><td>0.837705</td><td>0.837705</td><td>0.825023</td><td>0.836915</td><td>0.836915</td><td>0.833955</td></tr><tr><td>MF03-Ext</td><td>0.839602</td><td>0.839602</td><td>0.826920</td><td>0.836125</td><td>0.836125</td><td>0.829774</td></tr><tr><td>MF01-Ext-W2V</td><td>0.848205</td><td>0.848205</td><td>0.833927</td><td>0.861839</td><td>0.861839</td><td>0.846454</td></tr><tr><td>MF02-Ext-W2V</td><td>0.850174</td><td>0.850174</td><td>0.834462</td><td>0.860021</td><td>0.860021</td><td>0.846516</td></tr><tr><td>MF03-Ext-W2V</td><td>0.851841</td><td>0.851841</td><td>0.836650</td><td><strong>0.863051</strong></td><td><strong>0.863051</strong></td><td><strong>0.850946</strong></td></tr></tbody></table>

The table demonstrates that the models are well-fitted, with little chance of overfitting or underfitting, as the performance of validation and test data is nearly identical. The experiment UM00, which has a simple BERT architecture with simply comments as a feature, scores the worst with an F1 Macro score of 84% and an accuracy of 84% on the test. In contrast, the performance of multi-feature techniques with a variety of BERT architectures and features appears to be superior. Interestingly, the performance of experiments MF01, MF02, and MF03 with both user comments and social profiles as the feature set, but with different BERT architectures, is similar to when only the feature 'user comments' is employed. It indicates that the social-profile feature set of individuals do not necessarily contribute significantly to cyberbullying detection performance. It implies that the textual comments of users are essential for detecting harassment. In experiments MF02-Ext and MF03-Ext, which follow the following architectures: BERT with attention mechanism on categorical and numerical features, and BERT with gating mechanism on categorical and numerical features then sum, the lexical characteristics of users' Bangla comments demonstrate even slightly poorer performance with an accuracy of 83%. The range of lexical features in the Bangla language is considerably fewer than in the English language, which may be a contributing factor to the inferior performance with lexical features. Moreover, the lexical features do not contribute considerably to the model improvement. Focusing more on the Bangla
comments and employing both the contextual embedding and word embedding vectors collected from the Bangla comments has a substantial impact on the performance of models with identical topologies. In the architectures of MF01-Ext-W2V, MF02-Ext-W2V, and MF03-Ext-W2V, the accuracy is improved to 86.2%, 86.0%, and 86.3%, respectively. Nonetheless, the experiment MF03-Ext-W2V with the architecture BERT with gating followed by sum performs even better, as the F1 micro and F1 macro scores are superior to those of the other experiments. Table 3 provides more information about the class-wise harassment detection performance of MF03-Ext-W2V.

<table><caption>**Table 3.** Class-wise performance of the proposed method for cyberbullying detection</caption><thead><tr><th></th><th>Not Bully</th><th>Religious</th><th>Sexual</th><th>Troll</th><th>Threat</th></tr></thead><tbody><tr><th>Accuracy</th><td>0.92</td><td>0.98</td><td>0.94</td><td>0.91</td><td>0.98</td></tr><tr><th>Precision</th><td>0.88</td><td>0.93</td><td>0.86</td><td>0.80</td><td>0.75</td></tr><tr><th>Recall</th><td><strong>0.94</strong></td><td><strong>0.99</strong></td><td><strong>0.97</strong></td><td><strong>0.94</strong></td><td><strong>0.99</strong></td></tr><tr><th>F1-Score</th><td>0.88</td><td>0.93</td><td>0.86</td><td>0.80</td><td>0.78</td></tr><tr><th>AUC Score</th><td>0.91</td><td>0.97</td><td>0.92</td><td>0.87</td><td>0.88</td></tr></tbody></table>

The performance of harassment-wise detection is greatly improved, with a detection accuracy of 98% for potential threats. With an accuracy of 91%, the identification rate for sarcastic comments is the lowest. The AUC Score exceeds 90% for nearly all classes except threat and troll, who have respective scores of 0.88 and 0.87. The religious class has the highest performance scores for accuracy and AUC, with 0.98 and 0.97, respectively. Overall, the performance table by class suggests that the proposed multi-feature transformer technique for detecting Bangla cyberbully performs significantly well.

<table><caption>**Table 4.** Performance comparison of the proposed method with state-of-the-art</caption><thead><tr><th>Method</th><th>Precision</th><th>Recall</th><th>F1 Score</th><th>Accuracy</th></tr></thead><tbody><tr><td>RoBERT [11]</td><td>0.84</td><td>0.84</td><td>0.86</td><td>85%</td></tr><tr><td>SVM [2]</td><td>0.85</td><td>0.85</td><td>0.84</td><td>85%</td></tr><tr><td>ELECTRA [7]</td><td>-</td><td>-</td><td>-</td><td>84.92%</td></tr><tr><td>BERT [7]</td><td>-</td><td>-</td><td>-</td><td>85%</td></tr><tr><td><strong>Proposed*</strong></td><td><strong>0.86</strong></td><td><strong>0.96</strong></td><td><strong>0.86</strong></td><td><strong>86.3%</strong></td></tr></tbody></table>

Now, consider the most recent state-of-the-art methods for detecting cyberbully in Bangla. Classical machine learning and deep learning techniques, particularly with unimodal approaches, are among the recently proposed methodologies. Table 4 depicts the comparison result of the proposed method with state-of-the-art method. In [2,7,11], the maximum rate of cyberbullying detection
accuracy is 85%. ELECTRA [22] performs relatively poorly, with an accuracy of 84.92%. Our proposed method, on the other hand, outperforms the state-of-the-art methods with an accuracy of 86.3%. Moreover, when comparing the precision, recall, and F1 score of all approaches, the maximum score is 0.86, whereas the proposed method has the lowest precision score of 0.863 and the highest recall score of 0.96. Consequently, based on all contemporary evaluation parameters, our proposed technique performs much better than the state-of-the-art with a 12.94% improvement in recall.

# 5 Conclusion

Cyberbullying detection is a worldwide open problem that is required to be dealt with to provide a better cyber world to millions of online users, services, and entities. In this research, a novel multi-feature transformer-based technique for detecting multiclass cyberbully in Bangla is proposed. The proposed multi-feature transformer-based deep learning architecture uses the extracted contextual meaning, lexical attributes, user's social profile, and semantic similarity of the words from the raw Bangla comments on social media as inputs. With an overall accuracy of 86.3% and class-wise cyberbullying detection accuracy ranging from 92% to 98%, troll to threat, the proposed method outperforms existing methods. Additionally, the performance of cyberbullying detection in Bangla is improved across all performance evaluation-metrics, at least by 12.94% improvement in recall and 1.17% improvement in accuracy. The proposed method is superior to all recently published state-of-the-art techniques. In the future, the deep linguistic profiles of users' Bangla comments will be extracted to investigate the effect of deep linguistic profiles in designing an cyberbullying detection system. Another new avenue of research can be the study of clustering the potential type of users prone to be involved in cyberbullying online in Bangla.

# References

1. Ahmed, M.F., Mahmud, Z., Biash, Z.T., Ryen, A.A.N., Hossain, A., Ashraf, F.B.: Bangla online comments dataset. Mendeley Data 1 (2021)

2. Ahmed, M.F., Mahmud, Z., Biash, Z.T., Ryen, A.A.N., Hossain, A., Ashraf, F.B.: Cyberbullying detection using deep neural network from social media comments in bangla language. arXiv preprint arXiv:2106.04506 (2021)

3. Ahmed, M.T., Rahman, M., Nur, S., Islam, A., Das, D.: Deployment of machine learning and deep learning algorithms in detecting cyberbullying in bangla and romanized bangla text: a comparative study. In: 2021 International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT), pp. 1–10. IEEE (2021)

4. Al Imran, A., Wahid, Z., Ahmed, T.: BNnet: a deep neural network for the identification of satire and fake bangla news. In: Chellappan, S., Choo, K.-K.R., Phan, N.H. (eds.) CSoNet 2020. LNCS, vol. 12575, pp. 464–475. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-66046-8_38
5. Albawi, S., Mohammed, T.A., Al-Zawi, S.: Understanding of a convolutional neural network. In: 2017 International Conference on Engineering and Technology (ICET), pp. 1–6. IEEE (2017)

6. Aporna, A.A., Azad, I., Amlan, N.S., Mehedi, M.H.K., Mahbub, M.J.A., Rasel, A.A.: Classifying offensive speech of bangla text and analysis using explainable AI. In: Advances in Computing and Data Sciences: 6th International Conference, ICACDS 2022, Kurnool, India, 22–23 April 2022, Revised Selected Papers, Part I. pp. 133–144. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-12638-3_12

7. Aurpa, T.T., Sadik, R., Ahmed, M.S.: Abusive bangla comments detection on facebook using transformer-based deep learning models. Social Netw. Anal. Min. 12(1), 24 (2022)

8. Biau, G., Scornet, E.: A random forest guided tour. TEST 25(2), 197–227 (2016). https://doi.org/10.1007/s11749-016-0481-7

9. Church, K.W.: Word2vec. Nat. Lang. Eng. 23(1), 155–162 (2017)

10. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

11. Emon, M.I.H., Iqbal, K.N., Mehedi, M.H.K., Mahbub, M.J.A., Rasel, A.A.: Detection of bangla hate comments and cyberbullying in social media using nlp and transformer models. In: Advances in Computing and Data Sciences: 6th International Conference, ICACDS 2022, Kurnool, India, 22–23 April 2022, Revised Selected Papers, Part I, pp. 86–96. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-12638-3_8

12. Graves, A., Graves, A.: Long short-term memory. In: Supervised Sequence Labelling with Recurrent Neural Networks, pp. 37–45 (2012)

13. Han, X., Yue, Q., Chu, J., Han, Z., Shi, Y., Wang, C.: Multi-feature fusion transformer for chinese named entity recognition. In: 2022 41st Chinese Control Conference (CCC), pp. 4227–4232. IEEE (2022)

14. Huang, Z., Xu, W., Yu, K.: Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)

15. Jahan, M., Ahamed, I., Bishwas, M.R., Shatabda, S.: Abusive comments detection in bangla-english code-mixed and transliterated text. In: 2019 2nd International Conference on Innovation in Engineering and Technology (ICIET), pp. 1–6. IEEE (2019)

16. Kowsher, M., Sami, A.A., Prottasha, N.J., Arefin, M.S., Dhar, P.K., Koshiba, T.: Bangla-bert: transformer-based efficient model for transfer learning and language understanding. IEEE Access 10, 91855–91870 (2022)

17. LaValley, M.P.: Logistic regression. Circulation 117(18), 2395–2399 (2008)

18. Liu, B., et al.: Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data, vol. 1. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19460-3

19. Liu, G., Li, C., Yang, Q.: Neuralwalk: trust assessment in online social networks with neural networks. In: IEEE INFOCOM 2019-IEEE Conference on Computer Communications, pp. 1999–2007. IEEE (2019)

20. Liu, Y., Zheng, H., Feng, X., Chen, Z.: Short-term traffic flow prediction with conv-lstm. In: 2017 9th International Conference on Wireless Communications and Signal Processing (WCSP), pp. 1–6. IEEE (2017)

21. Mahmud, M.R., Afrin, M., Razzaque, M.A., Miller, E., Iwashige, J.: A rule based bengali stemmer. In: 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 2750–2756. IEEE (2014)
22. Meng, Y., et al.: Pretraining text encoders with adversarial mixture of training signal generators. arXiv preprint arXiv:2204.03243 (2022)

23. Nova, F.F., Rifat, M.R., Saha, P., Ahmed, S.I., Guha, S.: Online sexual harassment over anonymous social media in Bangladesh. In: Proceedings of the Tenth International Conference on Information and Communication Technologies and Development, pp. 1–12 (2019)

24. Peterson, L.E.: K-nearest neighbor. Scholarpedia 4(2), 1883 (2009)

25. Psichogios, D.C., Ungar, L.H.: A hybrid neural network-first principles approach to process modeling. AIChE J. 38(10), 1499–1511 (1992)

26. Rezwana Rashid, T.T.: Laws protecting victims from cyber harassment (2021). https://www.thedailystar.net/law-our-rights/news/laws-protecting-victims-cyber-harassment-2196491

27. Ritu, S.S., Mondal, J., Mia, M.M., Al Marouf, A.: Bangla abusive language detection using machine learning on radio message gateway. In: 2021 6th International Conference on Communication and Electronics Systems (ICCES), pp. 1725–1729. IEEE (2021)

28. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

29. Sen, O., et al.: Bangla natural language processing: a comprehensive analysis of classical, machine learning, and deep learning based methods. IEEE Access 10, 38999–39044 (2022)

30. Sharma, G.: Pros and cons of different sampling techniques. Int. J. Appl. Res. 3(7), 749–752 (2017)

31. Suthaharan, S.: Support vector machine. In: Machine Learning Models and Algorithms for Big Data Classification. ISIS, vol. 36, pp. 207–235. Springer, Boston, MA (2016). https://doi.org/10.1007/978-1-4899-7641-3_9

32. Talpur, K.R., Yuhaniz, S.S., Amir, N.: Cyberbullying detection: current trends and future directions. J. Theor. Appl. Inf. Technol. 98, 3197–3208 (2020)

33. Wahid, Z., Imran, A.A., Rifat, M.R.I.: BNnetXtreme: an enhanced methodology for Bangla fake news detection online. In: Computational Data and Social Networks: 11th International Conference, CSoNet 2022, 5–7 December 2022, Proceedings, pp. 157–166. Springer, Heidelberg (2023). https://doi.org/10.1007/978-3-031-26303-3_14

34. Wani, M.A., Agarwal, N., Bours, P.: Sexual-predator detection system based on social behavior biometric (ssb) features. Procedia Comput. Sci. 189, 116–127 (2021)

35. Webb, G.I., Keogh, E., Miikkulainen, R.: Naive bayes. Encycl. Mach. Learn. 15, 713–714 (2010)

36. Zhao, Y., Tao, X.: ZYJ123@ DravidianLangTech-EACL2021: offensive language identification based on xlm-roberta with dpcnn. In: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, pp. 216–221 (2021)