## 4 Materials and Methods
### Modeling and Evaluation
#### 4.4.2 Fine-Tuning the BERT Model for Classifying Infringing Sites
```
1:Inputs: dataset \(X\), BERT\({}_{\textit{uncased}}\) model
2:Output: BestModel (BERT + DNN)
3: Load pre-trained BERT\({}_{\textit{uncased}}\) model
4: Unfreeze all 768 weights in BERT\({}_{\textit{uncased}}\) model
5: Pre-process textual features \(X_{\text{text}}\) via BERT\({}_{\textit{uncased}}\)(tokenization, padding) model
6: Preprocess numerical features \(X_{\text{num}}\) via Standard Scaling
7:for each data point \((X_{\text{text}},X_{\text{num}_{i}})\in X_{T}\)do
8: Generate vector \(\mathbf{x}_{i}\) from \(X_{\text{text}}\) using BERT\({}_{\textit{uncased}}\) model
9: Concatenate \(\mathbf{x}_{i}\) with \(X_{\text{num}_{i}}\) to form \(\mathbf{x}_{\text{concat}_{i}}\)
10:endfor
11: Construct the concatenated training subset \(X_{\text{concat}}\leftarrow\mathbf{x}_{\text{concat}_{i}^{\text{pr}}}\)
12: Partition dataset \(X_{\text{concat}}\) into training \(X_{T_{\text{concat}}}\in X_{\text{concat}}\) (80%) subset and validation subset \(X_{V_{\text{concat}}}\in X_{T_{\text{concat}}}\) (20%)
13: Initialize the DNN with Input size \(I=776\)
14: Initialize the DNN with output size \(O=5\)
15: Define the subset \(\mathcal{P}\) hyperparameters an values to optimize: learning rate \(\eta\), epochs \(\mathcal{E}\), batch size \(B\) and number of hidden layers \(H_{M}\) with their respecipve units \(U\)
16: Initialize BestModel (BERT + DNN), BestPerformance \(\leftarrow\) null, 0
17:for each subsetset of hyperparameters (\(\mathcal{P}\))do
18: Compute the Excepted Improvement function \(EI(\mathcal{P})\) on BERT + DNN using (\(X_{T_{\text{concat}}},y_{T}\))
19: Compute Sparse Categorical Cross-Entropy loss function (\(\mathcal{L}\))
20: Backpropagation to update BERT + DNN weights
21:if performance of \(EI(\mathcal{P})\) on \((X_{V_{\text{concat}}},y_{V})\) > BestPerformance then
22: BestModel \(\leftarrow\) BERT + DNN
23: BestPerformance \(\leftarrow\) performance of BERT + DNN
24:else Select the next subset of hyperparameters \(\mathcal{P}_{\text{next}}\) to improve \(EI(\mathcal{P})\)
25:endif
26:endfor
27:return BestModel (BERT + DNN) with the best subset of hyperparameters \(\mathcal{P}^{+}\)
```  
**Algorithm 1** Fine-tuning and optimization of the BERT + DNN architecture  
The numerical samples \(X_{\text{num}}\in X\) which include variables such as _Presence of URL shorteners_, _Presence of out-of-domain Javascript content_, _Presence of CAPTCHAS_, _Number of outbound download links and iframes_, and _Access level_, undergo to a Standard Scaling (SC) operation. This process involves removing the mean and adjusting each of them to a unit variance, a process detailed in Equation (5). \[X_{\text{num}_{i}}=\frac{X_{\text{num}_{i}}-\mu_{i}}{\sigma_{i}}, \tag{5}\]  
where \(X_{\text{num}_{i}}\) represents each of the numerical samples, \(\mu_{i}\) is the mean, and \(\sigma_{i}\) is the variance. The textual embedded vectors \(\mathbf{x}_{i}\) from \(X_{\text{text}}\) corresponding to the \(i\)-th position are merged with their respective standardized numerical vectors \(X_{\text{num}_{i}}\), culminating in the creation of a final set \(X_{\text{concat}}\) that harmoniously integrates both characteristics, as detailed in Equation (6). \[X_{\text{concat}}=\sum_{i=1}^{n}(\mathbf{x}_{i}\oplus X_{\text{num}_{i}}) \tag{6}\]  
\(X_{\text{concat}}\) is then conformed by \(\mathbf{x}_{i}\in\mathbb{R}^{i\times 76}\) samples, composed of fixed-size vectors of 768 elements from the embedding obtained from BERT\({}_{\textit{uncased}}\) and the eight remaining standardized numerical features. Afterwards, \(X_{\text{concat}}\) is divided into three different subsets: \(X_{T_{\text{concat}}}\), which is used to train the BERT + DNN algorithm; \(X_{R_{\text{concat}}}\), intended for performance testing; and \(X_{V_{\text{concat}}}\in X_{T_{\text{concat}}}\), which is used for accuracy verification during training. This last step is essential for adjusting the weights of BERT and the hyperparameters of the DNN. These subsets represent, respectively, 70%, 30%, and 20% of the total and are selected randomly and without replacement.