# Improved Anomaly Detection by Using the Attention-Based Isolation Forest
## 6 Concluding Remarks  
A new modification of iForest by using an attention mechanism has been proposed. Let us focus on the advantages and disadvantages of the modification. **Advantages:**  
1. ABIForest is very simple from the point of view of computation because, in contrast to an attention-based neural network, the attention weights in ABIForest are trained by solving a standard quadratic optimization problem. This modification avoids the use of gradient-based algorithms to compute the optimal learnable attention parameters. 2. ABIForest is a flexible model that can be simply modified. There are several components of ABIForest that can be changed to improve the model's performance. First, different kernels can be used instead of the Gaussian kernel considered above. Second, there are statistical models [58] that are different from [PERSON]'s \(\epsilon\)-contamination model that can also be used in ABIForest. Third, the attention weights can be associated with some subsets of trees, including intersecting subsets. In this case, the number of trainable parameters can be reduced to avoid overfitting. Fourth, the paths in trees can be also attended, for example, by assigning attention weights to each branch in every path. Fifth, multi-head attention can be applied to iForest in order to improve the model--for example, by changing the hyperparameter \(\omega\) of the softmax. Sixth, the distance between the instance **x** and all instances that fall in the same leaf as **x** can be differently defined. The above improvements can be regarded as directions for further research. 3. The attention model is trained after building the forest. This implies that we do not need to rebuild iForest to achieve higher accuracy. The hyperparameters are tuned without rebuilding iForest. Moreover, we can apply various modifications and extensions of iForest and incorporate the attention mechanism in the same way as in the original iForest. 4. ABIForest allows us to obtain an interpretation that answers the question of why an instance is anomalous. This can be done by analyzing the isolation trees with the largest attention weights. 5. ABIForest deals perfectly with tabular data. 6. It follows from the numerical experiments that ABIForest improves the performance of iForest for many datasets. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**The Ionosphere Dataset** & & & & \\ \hline \(n\) & 80 & 100 & 200 & 300 \\ \hline iForest & 0.488 & 0.492 & 0.588 & 0.677 \\ \hline ABIForest & 0.459 & 0.602 & 0.674 & 0.690 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: F1-score measures of the original iForest and ABIForest as functions of the number of instances \(n\) in the Ionosphere dataset under the condition that the ratio of normal and anomalous instances was not changed. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**The Ionosphere Dataset** & & & & & \\ \hline \(n_{anom}\) & 10 & 20 & 40 & 50 & 60 \\ \hline iForest & 0.655 & 0.690 & 0.694 & 0.687 & 0.681 \\ \hline ABIForest & 0.692 & 0.709 & 0.711 & 0.695 & 0.687 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: F1-score measures of the original iForest and ABIForest as functions of the number of anomalous instances \(n_{anom}\) in the Ionosphere dataset under the condition that the number \(n_{norm}\) of normal instances was not changed. **Disadvantages:**  
1. The main disadvantage is that ABIForest has three additional hyperparameters: the contamination parameter \(\epsilon\), the hyperparameter of the softmax operation \(\omega\), and the regularization hyperparameter \(\lambda\). We do not include the threshold \(\tau\), which is also used in iForest. Additional hyperparameters lead to significant increases in the validation time. 2. Some additional time is required to solve the optimization problem (14). 3. In contrast to iForest, ABIForest is a supervised model. It requires one to have labels of data (normal or anomalous) in order to determine the criteria of optimization, that is, to construct the optimization problem (14).