# Enhancing Software Quality with AI: A Transformer-Based Approach for Code Smell Detection
## 3 Methodology
### Model Architectures
#### 3.2.5 Relational Embeddings for Code Smell Detection
While this method eliminates noise, it might exclude useful features that contribute to detecting large class smells. Data preprocessing choices: the normalization of numerical features using min-max scaling and handling of missing values using median imputation could introduce biases in the feature distribution, affecting model performance. Hyperparameter tuning: The transformer-based models were fine-tuned with specific hyperparameters (learning rate, batch size, etc. ), which may not be optimal for all datasets. While early stopping was used, a more exhaustive hyperparameter search could further improve results. #### 6.4.2 External Validity

Generalizability to other code smells: This study focuses on detecting large class smells. The model's effectiveness on other code smells (e.g., long method, God class, feature envy) remains untested. Future work should extend the approach to multiple code smell types. Programming language dependency: The dataset includes only Java classes. While Java is widely used in software engineering, the model's performance on other languages like Python, C++, or JavaScript remains uncertain. Future studies should evaluate cross-language performance. Dataset source bias: we collected the dataset from open-source projects on GitHub, which may not represent proprietary or industrial codebases with different coding styles and complexity patterns. #### 6.4.3 Construct Validity

Code smell labeling methodology: The dataset was labeled based on predefined thresholds for large class (e.g., lines of code, Halstead complexity). These thresholds may not align with developer perceptions of what constitutes a \"large class\" in different contexts. Evaluation metrics: While accuracy, precision, recall, and F1 score were used, they do not capture the interpretability of model predictions. Incorporating explainable AI (XAI) techniques could enhance trust in the model's decisions. #### 6.4.4 Conclusion Validity

Class imbalance effects: This study reports that Relation-Aware BERT (RABERT) achieved high precision (91.0%) but low recall (53.0%), indicating an imbalance in detecting positive cases. Future work should explore oversampling, cost-sensitive learning, or hybrid approaches to improve recall. Comparison with classical models: While transformer models outperformed classical models (e.g., gradient boosting), the computational overhead of transformers could make them impractical for real-time or large-scale systems. Investigating lightweight transformer variants such as DistilBERT or MobileBERT could address this limitation. #### 6.4.5 Future Work to Address These Threats

* Expanding the dataset to include multiple programming languages. * Incorporating multiple types of code smells for a broader evaluation. * Implementing explainable AI (XAI) techniques for interpretability. * Evaluating hybrid models combining transformers with classical approaches for better recall and computational efficiency. #### 6.4.6 Comparison with Recent Transformer Models

It is worthwhile to compare our approach with recent transformer models such as CodeBERT and GraphCodeBERT, which have been widely recognized for their strengths in code representation tasks. CodeBERT, with its pre-training on both natural language and code, and GraphCodeBERT, which enriches code embeddings by incorporating data flow information, have set new benchmarks in capturing code semantics and structural information. However, unlike these models, our RABERT leverages relational embeddings that explicitly model the dependencies between software metrics, thereby achieving high precision in detecting code smells. ### Future Directions

Domain-specific pretraining: adapting pretraining tasks to incorporate software-specific semantics, such as code structure and dependencies, could improve model performance further. Hybrid approaches: combining transformer-based models with classical approaches, such as gradient boosting, may yield complementary strengths, particularly for imbalanced datasets.