# Enhancing Software Quality with AI: A Transformer-Based Approach for Code Smell Detection
## 3 Methodology
### Model Architectures
#### 3.2.5 Relational Embeddings for Code Smell Detection  
To enhance the self-attention mechanism in the transformer architecture, we incorporate relational embeddings that explicitly capture the relationships between pairs of features. These embeddings allow the model to weight interactions between code metrics (e.g., LOC and Halstead volume) according to their intrinsic relationships. Mathematical Formulation  
In a standard transformer, the self-attention mechanism is computed as follows:  
\[\text{Attention (Q, K, V)}=\text{softmax(QK^{T}/\surd{k})}\,\text{V}, \tag{1}\]  
where  
Q = XWQ (query matrix)  
K = XWK (key matrix)  
V = XWV (value matrix)  
X is the input representation, and WQ, WK, and WV are learned projection matrices. d\({}_{\text{k}}\) is the dimensionality of the key vectors. To integrate relational information, let rij denote the learned relational embedding between feature i and feature j. The modified attention score between positions i and j is then defined as follows:  
\[\text{S}_{\text{ij}}=\text{Q}_{\text{i}}\,\text{(K}_{\text{j}}+\text{r}_{ \text{ij}})^{\text{T}}/\surd\text{d}_{\text{k}} \tag{2}\]  
This can also be expressed by separating the standard dot product and the relational term:  
\[\text{S}_{\text{ij}}=\text{Q}_{\text{i}}\text{K}^{\text{T}}_{\text{j}}/\surd \text{d}_{\text{k}}+\text{Q}_{\text{i}}\text{T}^{\text{T}}_{\text{ij}}/\surd \text{d}_{\text{k}} \tag{3}\]  
The attention weight for each pair is then computed by applying the softmax function to these scores:  
\[\text{a}_{\text{ij}}=\exp{\text{(S}_{\text{ij}})}/\sum_{j^{\prime}}\exp{ \text{(S}_{\text{ij^{\prime}}})}, \tag{4}\]Finally, the output for each token is calculated using the weighted sum of the value vectors:  
\[\text{Output}_{\text{i}}=\sum_{\text{j}}\text{a}_{\text{ij}}\text{V}_{\text{j}} \tag{5}\]  
This formulation ensures that each token's representation is influenced not only by the standard interactions (via \(\text{Q}_{\text{i}}\text{K}^{\text{T}}_{\text{j}}\)) but also by a learned measure of their relational relevance (via \(\text{Q}_{\text{i}}\text{K}^{\text{T}}_{\text{ij}}\)). Pseudocode for Relation-Aware Self-Attention  
Below is Algorithm 1 illustrating how the relational embeddings are integrated into the transformer's self-attention layer:  
```
1:#Input:
2:#X: Input feature matrix (batch_size x seq_length x d_model)
3:#W_Q_W_K, W_V: Learned projection matrices for Q, K, V respectively
4:#R: Relation embedding tensor (seq_length x seq_length x d_relation)
5:#Scale: Scaling factor = sqrt(d_k)
6:def relation_aware_self_attention(X, W_Q, W_K, W_V, R):
7:#Project the input matrix X to queries, keys, and values
8:Q = X @W_Q# shape: (batch_size, seq_length, d_k)
9:K = X @W_K# shape: (batch_size, seq_length, d_k)
10:V = X @W_V# shape: (batch_size, seq_length, d_v)
11:# Initialize the attention score matrix
12:batch_size, seq_length, = Q_shape
13:scores = zeros((batch_size, seq_length, seq_length))
14:# Compute attention scores with relational embeddings
15:for b in range(batch_size):
16:for i in range(seq_length):
17:for j in range(seq_length):
18:# Standard dot-product attention term
19:standard_score = dot(Q[b, i], K[b, j])
20:# Relational term: Assume a projection function (e.g., linear mapping) is applied to r_ij
21:# Here, r_proj could be a learned projection of the relation embedding to the same dimension as d_k.