# Finsformer: A Novel Approach to Detecting Financial Attacks Using Transformer and Cluster-Attention
## 3 Materials and Methods
### Proposed Method
#### 3.3.3 Cluster-Attention Mechanism  
A key innovation in the Finsformer model, as presented in this paper, is the introduction of the cluster-attention mechanism, as shown in Figure 4. The following sections provide an in-depth explanation of the design details of the cluster-attention mechanism, its mathematical formulation, and the advantages it offers in detecting financial attacks. Figure 3: Illustration of the Finsformer. Differences from multi-head attention mechanism. The primary distinction between the cluster-attention mechanism and the traditional multi-head attention mechanism lies in the method of processing input data. While the multi-head attention mechanism focuses on capturing relationships between different positions within a sequence, cluster-attention emphasizes identifying inherent groups or patterns in the data. This involves two main steps:  
1. Data clustering: The cluster-attention mechanism initially clusters the input data, aiming to identify potential groups or patterns within it. This approach allows the model to concentrate on transactions with similar features, thereby more effectively identifying anomalous behavior. 2. Attention weight calculation: Following the clustering, attention weights are calculated within each cluster. This method results in a more concentrated distribution of attention, helping to highlight significant transaction patterns. Design of clustering algorithms. The choice of clustering algorithm is crucial to the effectiveness of the cluster-attention mechanism. In this model, clustering methods based on feature similarity, specifically k-means and DBSCAN, are employed to ensure the effective identification of inherent patterns in financial transaction data. K-means clustering, one of the most common clustering algorithms, aims to minimize the sum of squared distances of each point to the center of its cluster. The objective of k-means clustering can be expressed as minimizing the sum of squared distances between data points and their respective cluster centers:  
\[\min_{S}\sum_{i=1}^{k}\sum_{\mathbf{x}\in S_{i}}|\mathbf{x}-\mathbf{\mu}_{i}|^{2} \tag{9}\]  
where \(k\) is the number of clusters, \(S_{i}\) is the \(i\)-th cluster, \(\mathbf{\mu}_{i}\) is the center of cluster \(S_{i}\), and \(\mathbf{x}\) is a data point. The algorithmic process is described in Algorithm 1. The specific logic of the algorithm is as follows: (1) Input and initialization: The inputs to the algorithm include a set of data points \(X\) and the number of clusters \(k\) to be formed. Initially, \(k\) data points are randomly selected from the dataset \(X\) to serve as the initial cluster centers \(\mu_{1},\mu_{2},\ldots,\mu_{k}\). (2) Cluster assignment: The algorithm then enters a repetitive looping process. In each loop, for each data point \(x\) in the dataset \(X\), the nearest cluster center is identified based on a distance measure (here, the square of the Euclidean distance), and the point is assigned to the corresponding cluster. Specifically, for each point \(x\), the distance between it and each cluster center \(\mu_{i}\) is calculated, and the cluster center with the minimum distance is selected as the category \(c(x)\) to which \(x\) belongs. (3) Updating cluster centers: After all data points have been assigned to their respective clusters, the centers of each cluster are updated. Specifically, for each cluster \(i\), the new cluster center \(\mu_{i}\) is the mean of all points \(x\) in that cluster. (4) Termination condition: The above steps of assignment and updating of cluster centers are repeated until the cluster centers no longer change, indicating that the algorithm has converged. (5) Output: Finally, the algorithm outputs the updated cluster centers \(\mu\) and the cluster labels \(c(x)\) for each point \(x\) in the dataset \(X\). Figure 4: Illustration of the cluster-attention mechanism. K-means clustering is effective in identifying spherical or convex clusters.