## 3 Results
the unidirectional LSTM layer is more prevalent in the decoder block of the architecture after generation four. Regarding the self-attention layers, there was a peak in generation three that indicates a temporal tendency, which was replaced by a new tendency in the next generation. In generation three, several replications of the encoder block were very common but they were rapidly reduced. The number of self-attention layers before the recurrent decoder started dropping from generation three, while the number of self-attention layers after the LSTM layers became the norm in later generations. The dimensionality of the architecture is of great concern, as it decides the number of trainable parameters of the model. Figure 7 shows the preferred dimensionality over the last three generations of the GA, regarding only the top five chromosomes from each, revealing that \(D=1024\) is the most common size among the populations, with 32 attention heads of 32 dimensions each. The output neural network ended up disappearing from the model's architecture. Moreover, the best chromosome from the last generation features all of these design trend characteristics. Figure 6: Evolution of the numbers of LSTM layers and self-attention layers over time. ‘All chromosomes show the average of all chromosomes within the generation, while the top 5 chromosomes show how the most relevant chromosomes are built. In the last generations, the eLU activation layer and the WordPiece tokenizer disappeared. This is a relevant result, as it makes it possible to disregard these features for future models. On the other hand, the ReLU and TanH activation functions were the most common cases. Typical dropout percentages are below 20%. The encoder block is normally replicated less than twice, as more replications lead to bigger models that may not have sufficient data to learn language relationships. In addition, the most common is to be replicated once (two encoder blocks), possibly because it is the simpler architecture that includes normalization and residual connections, which greatly improve the performance during training. Based on the previous analysis, the best architecture was considered to be the best chromosome from the seventh population (Figure 5). The best chromosomes typically feature Bi-LSTM layers followed by self-attention mechanisms, where this block is replicated twice. The final model architecture was trained from scratch under similar conditions to extract more information about its sensitivity and precision, as well as its overall accuracy, employing the FFCV. Figure 8 compares the accuracy, recall, precision, and ROC-AUC of the newer architecture against the BaseModel [27]. The increases in the ROC-AUC and the accuracy are notable, which indicate that the model correctly classified more instances. On the other hand, the decreases in precision and recall suggest that the F1-score decreased, mainly due to its precision. This reduction implies that the model became less conservative in predicting the minority classes (negative and neutral), resulting in more false positives but fewer false negatives. This is beneficial, in this case, due to the data imbalance scenario, where the positive class is overrepresented. Only the class weights in the loss function were beneficial for partially surpassing the bias from class skewness. The difference in the number of trainable parameters between the BaseModel and the optimized architecture is a decrease of approximately 2.7 times. Figure 7: Distribution of the dimensions found during the GA. In addition to the average metrics, the micro- (per class) metrics of the ROC-AUC and F1-score can be analyzed to understand how the data imbalance affects the model behavior on each class. The ROC-AUC values of the negative and positive classes are 89.6% and 83.9%, respectively, indicating the model has a greater affinity for the negative class, which is desirable since the majority class is the positive, and training tends to be biased towards this class.