## 3 Materials and Methods
### CustomBERT--Ensemble Model Design  
We developed the CustomBERT model, which combines different BERT versions and a CNN, inspired by the way CNNs recognize similarities in images, as investigated by [PERSON] and [PERSON] [42]. This approach is motivated by the need to capture diverse linguistic patterns from different transformer models. Each BERT variant offers distinct strengths: BERT multilingual excels in handling various aspect of language, XLM-RoBERTa is particularly adept at cross tasks, and DistilBERT provides efficiency with minimal loss in performance. By using these models, our ensemble aims to capture a wider range of semantic features. This method identifies similarities between various pre-trained transformer models using text input, similar to how CNNs identify similarities in images. The combination of transformer models and CNN introduces additional layers of representation  
Figure 2: Research methodology. learning, allowing the system to extract more nuanced textual features that a single model might miss. Figure 3 shows the detailed architecture of our approach. By using the advantages of various transformer models, this structure aims to provide an improved and flexible method to evaluate new text. We used BERT's bidirectional nature to fully understand the context of words in sentences [43]. After the pre-trained model processes the input, its output undergoes two subsequent processes. First, we perform concatenation by merging the outputs from the pre-trained model. Then, we pass this concatenated input through a CNN layer, which is designed to identify important features from both the text data and the token scores. CNN is chosen here to effectively capture local dependencies in the concatenated transformer outputs, enabling the model to recognize intricate relationships within the text that are critical for tasks like sexism detection. The pseudocode of Algorithm 1 is as follows:  
```
1:Input sentence
2:Output classes
3:functionPre-train Language Models(sentence)
4:\(bert\_output\leftarrow\) BERT Model(sentence)
5:\(x\_lmorebrate\_output\leftarrow\) XlmRoBERT Model(sentence)
6:\(distilbert\_output\leftarrow\) DistBERT Model(sentence)
7:return\(bert\_output\_xlmforebrate\_output\_,distilbert\_output\)
8:endfunction
9:functionCustomBERT(tokens)
10:\(bert\_output\_xlmbrobrate\_output\_,distilbert\_output\leftarrow\)
11:Pre-train Language Models Outputs(sentence)
12:\(concatenated\_outputs\leftarrow\) concatenate(\(bert\_output\_xlmbrobrate\_output\_,distilbert\_output\))
13:\(cnn\_output\leftarrow\) apply Conv1D layer, filters=64, kernel=3, activation='relu'(concatenated\_outputs)
14:\(flattened\_cnn\_output\leftarrow\) flatten(\(cnn\_output\))
15:\(output\leftarrow\) apply Dense layer with 1 unit, activation='sigmoid'(flattened\_cnn\_output)
16:return class
17:endfunction
18:finaloutput\(\leftarrow\)CustomBERT(input sentence)
```  
**Algorithm 1** CustomBERT model for sexism detection. Figure 3: Architecture of our CustomBERT model. As shown in Algorithm 1, the CustomBERT model combines the outputs of various pre-trained transformer models with token scores before classification. It consists of BERT multilingual [43], XLM-RoBERTa [44], and DistilBERT, a smaller but efficient version of BERT [45]. The main task for these transformer models is to encode the input sentence into a dense vector representation, capturing contextualized meaning and semantic nuances.