# Enhancing Software Quality with AI: A Transformer-Based Approach for Code Smell Detection
## 3 Methodology
### Model Architectures
#### 3.2.5 Relational Embeddings for Code Smell Detection

Table 9: Comparative analysis of the different feature selection methods. SHAP values quantify the contribution of each feature towards the model's prediction. Our SHAP-based analysis identified lines of code (LOC), Halstead volume, effort, and difficulty as the most influential features responsible for flagging a class as a large class. Specifically, classes with higher LOC and Halstead volume consistently exhibited higher SHAP values, indicating their significant role in driving the prediction. Moreover, cognitive metrics such as effort and difficulty also contributed notably to the final decision. LIME was employed to generate local explanations for individual predictions. The LIME results corroborated the SHAP findings, confirming that LOC and effort were the most critical features influencing the classification decision. Additionally, comment density and Halstead length emerged as supportive features in certain cases, reflecting their contextual importance in specific code instances. Table 10 presents the Summary of explainability results. ### Lightweight Transformers and Model Quantization

To address the practical limitations of computational overhead and high inference time observed in the RABERT model, we explored lightweight transformer architectures and model quantization techniques. These approaches aim to balance performance with reduced resource consumption, enabling more efficient deployment in real-time or resource-constrained environments. ### DistilBERT: Lightweight Transformer Model

DistilBERT is a distilled version of the original BERT architecture, designed to retain most of BERT's performance while reducing the model size and computational complexity. We employed DistilBERT for code smell detection to assess its applicability in comparison with the full RABERT model. The results showed that DistilBERT achieved an accuracy of 88.5% and an F1 score of 65.8%, with a significantly reduced inference time of 6.2 ms per sample--making it highly suitable for real-time or edge device applications. However, a slight performance trade-off was observed compared to the full RABERT model. ### Model Quantization of RABERT

Model quantization is a compression technique that reduces the precision of model weights (e.g., from float32 to int8), thereby decreasing model size and inference time without substantial loss in performance. We applied post-training static quantization to the RABERT model. The quantized RABERT model achieved an accuracy of 89.2% and an F1 score of 66.5% while reducing the inference time to 8.5 ms per sample. This result demonstrates that quantization provides a balanced solution between model performance and efficiency. Table 11 presents the Model Quantization of RABERT. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Explanation Method** & **Top Contributing Features** & **Observation** \\ \hline SHAP & Lines of Code (LOC), Halstead Volume, Effort, Difficulty & SHAP values indicate that LOC and Halstead volume are the primary contributors to identifying a class as a large class, supported by cognitive metrics like effort and difficulty. \\  & LIME & LOC, Effort, Comment Density, Halstead Length & LIME analysis highlights that LOC and effort are the dominant factors influencing the model’s decision, with comment density and Halstead length playing supportive roles. \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Summary of explainability results. These results highlight that both DistilBERT and model quantization strategies offer viable solutions for efficient deployment of transformer-based models in software quality assurance tasks. Future work can explore hybrid models or optimization techniques, such as pruning or knowledge distillation, to further enhance efficiency without sacrificing performance. ## 6 Findings and Their Implications

The introduction of relation-aware embeddings in transformers for software analysis aligns with broader trends in AI-driven software engineering.