# Enhancing Software Quality with AI: A Transformer-Based Approach for Code Smell Detection
## 3 Methodology
### Model Architectures
#### 3.2.5 Relational Embeddings for Code Smell Detection
22:r_proj = project_relation(R[i, j])
23:relation_score = dot(Q[b, i], r_proj)
24:# Combine the scores and apply scaling
25:scores[b, i, j] = (standard_score + relation_score)/Scale
26:# Compute attention weights using softmax
27:attention_weights = softmax(scores, axis = -1)
28:# Calculate the final output as a weighted sum of the value vectors
29:output = zeros((batch_size, seq_length, V.shape[\(-\)1]))
30:for b in range(batch_size):
31:for i in range(seq_length):
32:for j in range(seq_length):
33:output[b, i] += attention_weights[b, i, j] * V[b, j]
34:return output

#### 3.2.6 Discussion

By integrating the relational embedding \(\tau_{\rm ij}\) into the attention mechanism, the model can account for explicit dependencies among features. This is particularly beneficial in code smell detection where relationships (e.g., between code length and complexity metrics) can inform more accurate predictions. The pseudocode above outlines the process and can be adapted as needed for efficient implementation (e.g., using parallelized tensor operations in deep-learning frameworks such as PyTorch 2.6 or TensorFlow 2). ### Training Configuration

The following configuration was used for training all transformer-based models:

* Batch Size: 16;
* Sequence Length: 128 tokens;
* Learning Rate: 2 \(\times\) 10\({}^{-5}\), optimized using the AdamW optimizer [44];
* Epochs: 10, with early stopping based on validation performance. ### Evaluation Metrics Used in Model Architecture

Performance was evaluated using the following metrics [39]:

* Accuracy: accuracy is defined as the ratio of all correct predictions (both true positives and true negatives) to the total number of predictions made. * Precision: precision focuses specifically on the model's performance on the predicted positive instances. * Recall: a fraction of true positives among all actual positives, reflecting the model's sensitivity to identifying all instances of the target class. * F1-Score: the harmonic mean of precision and recall, balancing these two metrics. ## 4 Experimental Setup

The dataset was selected from GitHub repositories to ensure diversity in coding styles and complexity levels, making it a more generalizable benchmark. Stratified sampling was used to maintain class balance, preventing bias toward majority classes. The hyperparameters, including a learning rate of 2 \(\times\) 10\({}^{-5}\) and a batch size of 16, were optimized based on preliminary experiments and prior research findings in transformer fine-tuning for code analysis. Figure 1 shows overall Data preparation and model architecture workflow. Figure 1: Data preparation and model architecture workflow. ### Dataset Overview

The dataset used in this study contains 20 features and 1000 samples annotated with the large class code smell, representing various software metrics indicative of code complexity, maintainability, and quality issues. The target label, LargeClass, is a binary indicator of whether a code class is considered large. The researchers extracted these classes from Java-based open-source projects located on GitHub [42]. The project selection process aimed to include samples from web applications, enterprise systems, and utility libraries to obtain an adequate representation of large class occurrences. The dataset comprises features such as

Code Metrics: loc, lloc, scloc, comments, and blanks. Cognitive Metrics: effort, difficulty, and time. Halstead Metrics: volume, length, and bugs.