## 4 Results
### Model Training and Optimization
We used early stopping based on validation losses to prevent overfitting [49]. The binary cross-entropy loss function ([[https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy](https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy)]([https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy](https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy)), (accessed on 10 June 2023)) and the Adam optimizer ([[https://keras.io/api/optimizers/adam/](https://keras.io/api/optimizers/adam/)]([https://keras.io/api/optimizers/adam/](https://keras.io/api/optimizers/adam/)), (accessed on 10 June 2023)) were used for training, employing the'mixed float16' precision training policy ([[https://keras.io/api/mixed_precision/](https://keras.io/api/mixed_precision/)]([https://keras.io/api/mixed_precision/](https://keras.io/api/mixed_precision/)), (accessed on 10 June 2023)). A custom function was developed for model structure, employing various transformers like bert-base-multilingual-uncased, xlm-roberta-base, and distilbert-base-multilingual-cased. These transformers' outputs were amalgamated for binary classification with L2 regularization ([[https://www.tensorflow.org/api_docs/python/tf/keras/regularizers/L2](https://www.tensorflow.org/api_docs/python/tf/keras/regularizers/L2)]([https://www.tensorflow.org/api_docs/python/tf/keras/regularizers/L2](https://www.tensorflow.org/api_docs/python/tf/keras/regularizers/L2)), (accessed on 10 June 2023)). A comprehensive summary of the hyperparameters we examined is presented in Table 5.  
In evaluating the performance of various models, we considered several key metrics, including accuracy, precision, recall, and F1 score across multiple tasks. Table 6 summarizes the results of our experiments, comparing the performance of different models on Tasks A (binary classification of posts as sexist or non-sexist), B (four-class classification of sexist posts), and C (11-class system for more specific labels of sexism). According to Table 6, the CustomBERT model consistently outperforms the individual models across all tasks. For Task A, the CustomBERT model achieves the highest accuracy of 0.79, precision of 0.77, recall of 0.79, and F1 score of 0.76. Similarly, for Task B, the CustomBERT model leads with an accuracy of 0.71, precision of 0.69, recall of 0.71, and F1 score of 0.68. For Task C, the CustomBERT model again surpasses the others, showing superior performance with an accuracy of 0.67, precision of 0.65, recall of 0.67, and F1 score of 0.64. The consistent performance improvements observed with the CustomBERT model highlight the benefits of an ensemble approach, effectively combining the strengths of various models to achieve better overall results. This ensemble strategy, therefore, was selected as the final model for our application due to its robust performance across multiple evaluation metrics and tasks. \begin{table}
\begin{tabular}{l l} \hline
**Parameter** & **Description** \\ \hline Tokenization Max Length & 512 tokens \\ Learning Rate Range & 1 \(\times 10^{-5}\) to 1 \(\times 10^{-4}\) (Default: 3 \(\times 10^{-5}\)) \\ Batch Sizes & 32, 64, 128 \\ Learning Rate Scheduler & Cosine decay schedule \\ Warm up Steps & 200 steps \\ Early Stopping Patience & 5 epochs \\ Loss Function & Binary cross-entropy \\ Optimizer & Adam \\ Precision Training Policy & Mixed float16 \\ \hline \end{tabular}
\end{table} 
Table 5: Summary of model parameters and hyperparameters.