# Digital Currency Illegal Behavior Detection Based on Mutual Information Prior Loss

Authors: Feng Yang; Guixin Dong; Chaoran Cui; Xiaojie Li; Yaxi Su; Yilong Yin
Journal: Scientific Programming
 (2021)

---

## Abstract

In recent years, with the rapid development of digital currency, digital currency brings us convenience and wealth, but also breeds some illegal and criminal behaviors. Different from traditional currencies, digital currency provides concealment to criminals while also exposing their behavior. The analysis of their behavior can be used to detect whether the current digital currency transaction is legal. There is a problem that most digital currency transactions are in compliance with laws and regulations, and only a small part of them uses digital currency to conduct illegal activities. It belongs to the problem of sample imbalance. It is quite challenging to accurately distinguish which transactions are legal and which are illegal in the massive digital currency transactions. For this reason, this study combines the mutual information and the traditional cross-entropy loss function and obtains the loss function based on the mutual information prior. The loss function based on the mutual information prior is that the bias of the category prior distribution is added after the output of the model (before the softmax), which makes the model consider category prior information to a certain extent when predicting. The experimental results show that the use of the loss function based on mutual information prior to the detection of digital currency illegal behavior has a good effect in SVM, DNN, GCN, and GAT methods.


---

## Body

## 1. Introduction

As Bitcoin exceeded the $50,000 mark, individuals’ interests in digital currency increased on February 16, 2020. Bitcoin is one of many peer-to-peer virtual cryptocurrencies. The biggest feature of digital currency is the peer-to-peer transmission, which shows that it is a decentralized payment system. Unlike traditional currencies, digital currencies cannot depend on specific organizations to issue them. It is in line with proprietary algorithms, which is generated by computer calculations over a long period of time. Digital currencies which use a distributed database of nodes throughout the P2P network can record and confirm all transactions, such as Bitcoin, using cryptography design to ensure the security of all aspects of currency circulation. Generally speaking, digital currency can be seen as a set of quasicurrency issuing system, as well as a set of currency circulation and settlement network. It can not only solve the problem of overissuance of existing credit currency but also realize low-cost, quasi-real-time currency settlement.Money laundering is regarded as a major illegal activity in digital currency. According to statistics, there are 1.6 trillion dollars of money laundering in the global market every year. The money that needs to be laundered is often the illegal income from terrorist activities, fraud, drug trade, corruption, and other criminal action. As one of the internationally recognized illegal economic activities, the behavior of money laundering will not only impede our country’s economic development but also affect our country’s political security and social stability. The traditional way of money laundering is to avoid the supervision of the domestic anti-money laundering department. In general, criminals would transfer a large amount of illegal money to countries whose supervision is less strict or take advantage of policy loopholes between different countries so as to evade supervision.Criminals are looking for a new type of vector-financial currency, mostly Bitcoin, while fiat money has become increasingly debased. Digital currency has been a major force in the money-laundering market out of its anonymity, decentralization, strong cross-border liquidity, lack of effective regulations, and opaque transaction information. Furthermore, it is also used in the dark net illegal trading and network extortion and other criminal activities. In 2017, a computer virus called WannaCry began to wreak havoc on computers all over the world, which can encrypt victims’ computer files and threat victims of computer virus attacks to pay $600 in Bitcoin to WannaCry’s Bitcoin address. The antiterrorist group discovered that about $140,000 worth of Bitcoins had been collected from the address WannaCry which is provided for the redemption money on August 3, 2017. Therefore, illegal money laundering has been a cancer and bomb in the process of economic development and social progress. Hence, it is extremely urgent to crack down on illegal money laundering.The analysis of illegal detection of digital currency is a detection of abnormal phenomena. The majority of digital currency holders are using digital currency legally, while only a small number of criminals are using digital currency for illegal transactions, such as extortion and money laundering [1–5]. As a result, it is difficult to accurately sort out a small amount of illegal transactions in the vast data set of digital currency transactions. It can be seen as a typical problem of data imbalance. Previous research considered using classical methods to deal with data imbalance, such as modifying the weight of legal use and illegal use categories and taking focal loss [6] to replace the original loss function.As for related materials of machine learning, mutual information is often used to measure the correlation between two things, which is regarded as a measure of the interdependence of variables. Different from the correlation coefficient, mutual information is not limited to the actual value of the random variable, while it determines the similarity between the product of two edge distributions and the joint distribution. It can be said that mutual information essentially reflects the relationship between input information and output information. As a result, if mutual information can be directly fitted, effective knowledge will be studied by individuals. In this article, cross-entropy fitting mutual information is considered to enhance the loss function, so that the problem of data sample imbalance can be relieved.Digital currency transactions are ubiquitous. Graph Neural Network, which is very popular in recent years, can be seen as the key tool for processing graph data. There are two methods that are commonly used in Graph Neural Network. One is Graph Convolution Network in accordance with space, which represents graph convolution by aggregating the information of adjacent nodes. The other is spectral on the basis of Graph Convolution Network, which interprets graph convolution as removing noise from the image signal.Because the trading network of digital currency is a graph structure, this article will take Graph Neural Network and traditional machine learning algorithm (DNN and support vector machine) as models to test the ordinary cross-entropy loss function, focal loss function, and mutual information priority-based loss function, respectively. The results show that the loss function based on mutual information prior proposed in this article has better property than the cross-entropy loss function and focal loss function.Innovation:(1)
Use mutual information-based loss to alleviate the problem of data imbalance(2)
Use Graph Neural Network method(3)
Experiment on real data sets


## 2. Related Work

Illegal behavior detection towards digital currency has great significance, playing an important role in keeping the security of network finance, governing the chaos of network finance, and preventing the risk of network finance. At present, the illegal behavior detection of digital currency has been involved in many fields such as finance, economy, law, and even politics. Illegal behavior detection towards digital currency has wide application and important research significance no matter in financial background or technology background. However, there are serious data imbalance problems in illegal transactions of digital currency, and how to effectively alleviate the data imbalance problem has been the major work goal of researchers. Therefore, this article will briefly introduce the existing digital currency illegal behavior detection, so as to improve the loss function to alleviate the data imbalance.


### 2.1. Detection of Digital Currency Violations

Akcora [7] put forward a solution in accordance with topology data analysis to divide the Bitcoin network into 24 hour long windows. We need to extract the following six characteristics for the address in each window: income, neighbor, weight, length, count, and loop. The new address of known ransomware family is detected, and the appearance of unknown ransomware family address is predicted. New tools on the basis of TDA have largely improved the accuracy of ransomware detection. Chen et al. [8] put forward a method of cascading feature extraction based on transaction graph for phishing account identification and a dual sampling integration algorithm on the basis of lightGBM, so as to solve the problem of class imbalance in phishing fraud identification. It targets the entire blockchain ecosystem, providing users with early warning against phishing scams.Weber et al. [9] put forward some cryptocurrency transaction authentication laws (especially for Bitcoin) to combat criminal activities, providing the anti-money laundering community with a data-rich, tagged set of transaction data that has never been publicly available before. It also provided a prototype for the visualization of such data and a model for enhancing human analysis and interpretation. The goal is to inspire others to tackle the big challenge of anti-money laundering and make the world’s financial system safer and more inclusive. These methods do not use the neural network method for detection. Neural network can be used for complex nonlinear modeling.


### 2.2. Improvement of the Loss Function to Alleviate the Data Imbalance

Cao et al. [10] put forward a theoretically principled label distribution perception of marginal LDAM losses, which was motivated by minimization of edge-based generalization limits. This loss replaces the standard cross-entropy goal during training and can be applied to previous class-imbalance training strategies, such as reweighting or resampling. Then, a simple but effective training program is put forward. This plan postpones reweighting until after the initial phase, which allows the model to learn the initial representation while avoiding some of the complexities associated with reweighting or resampling.Buda et al. [11] attached great importance to the influence of sample imbalance on the classification neural network and studied the influence of sample imbalance on the classification network in three general data sets of different sizes: MNIST/CIFAR-10/ImageNet as well as relevant ideas to solve the sample imbalance problems: oversampling/subsampling/two-stage training/threshold method. It is concluded that the sample imbalance is not conducive to the classification network. At present, the main method is oversampling. It is concluded that oversampling in general networks does not lead to overfitting. Focal loss was put forward by Lin et al. [6], and category imbalance was regarded as the reason that prevented the one-stage method from surpassing the two-stage method of top-performing. We need to focus on hard examples and reduce the weight of a large number of easy negatives, so as to use focal loss as well as an adjustment item in the cross entropy.The loss function method based on mutual information prior which is put forward in this article plays an important role in promoting the research on the detection of digital currency illegal behaviors, effectively improving the serious imbalance of samples in the Bitcoin data set and improving the accuracy and recall rate of illegal categories.


## 2.1. Detection of Digital Currency Violations

Akcora [7] put forward a solution in accordance with topology data analysis to divide the Bitcoin network into 24 hour long windows. We need to extract the following six characteristics for the address in each window: income, neighbor, weight, length, count, and loop. The new address of known ransomware family is detected, and the appearance of unknown ransomware family address is predicted. New tools on the basis of TDA have largely improved the accuracy of ransomware detection. Chen et al. [8] put forward a method of cascading feature extraction based on transaction graph for phishing account identification and a dual sampling integration algorithm on the basis of lightGBM, so as to solve the problem of class imbalance in phishing fraud identification. It targets the entire blockchain ecosystem, providing users with early warning against phishing scams.Weber et al. [9] put forward some cryptocurrency transaction authentication laws (especially for Bitcoin) to combat criminal activities, providing the anti-money laundering community with a data-rich, tagged set of transaction data that has never been publicly available before. It also provided a prototype for the visualization of such data and a model for enhancing human analysis and interpretation. The goal is to inspire others to tackle the big challenge of anti-money laundering and make the world’s financial system safer and more inclusive. These methods do not use the neural network method for detection. Neural network can be used for complex nonlinear modeling.


## 2.2. Improvement of the Loss Function to Alleviate the Data Imbalance

Cao et al. [10] put forward a theoretically principled label distribution perception of marginal LDAM losses, which was motivated by minimization of edge-based generalization limits. This loss replaces the standard cross-entropy goal during training and can be applied to previous class-imbalance training strategies, such as reweighting or resampling. Then, a simple but effective training program is put forward. This plan postpones reweighting until after the initial phase, which allows the model to learn the initial representation while avoiding some of the complexities associated with reweighting or resampling.Buda et al. [11] attached great importance to the influence of sample imbalance on the classification neural network and studied the influence of sample imbalance on the classification network in three general data sets of different sizes: MNIST/CIFAR-10/ImageNet as well as relevant ideas to solve the sample imbalance problems: oversampling/subsampling/two-stage training/threshold method. It is concluded that the sample imbalance is not conducive to the classification network. At present, the main method is oversampling. It is concluded that oversampling in general networks does not lead to overfitting. Focal loss was put forward by Lin et al. [6], and category imbalance was regarded as the reason that prevented the one-stage method from surpassing the two-stage method of top-performing. We need to focus on hard examples and reduce the weight of a large number of easy negatives, so as to use focal loss as well as an adjustment item in the cross entropy.The loss function method based on mutual information prior which is put forward in this article plays an important role in promoting the research on the detection of digital currency illegal behaviors, effectively improving the serious imbalance of samples in the Bitcoin data set and improving the accuracy and recall rate of illegal categories.


## 3. Method

### 3.1. Question Raising

To detect illegal digital currency transactions, we need to formalize digital currency transactions into a typical machine learning problem. We can regard detecting illegal digital currency transactions as a binary classification. The data set that is used in this article is provided by researchers from MIT, IBM, and Elliptic, a blockchain analytics company. It contains a data set of more than 200,000 Bitcoin transactions with a small number of tokens. It is by far the most cryptocurrency-related and tagged data set [9]. There are two categories in this data set. One is the legal category, which includes a series of legal services such as exchanges, wallet providers, and miners. The other category is illegal, which includes fraud, malware, terrorist groups, ransomware, Ponzi schemes, and other illegal transactions with only a few classified as illegal transactions. More important for detecting illegal transactions of digital currency is the few categories (i.e., illegal categories).This article formally describes the problem as follows:Input:X∈RV∗F,whereV is the number of nodes and F is the characteristic dimensionOutput:Y∈0,1, where 0 is the legal category and 1 is the illegal categoryBecause there is serious data imbalance between legal category and illegal category, in order to solve this problem, this article uses the loss function in terms of the mutual information prior for mitigation.


### 3.2. Illegal Transaction Monitoring of Digital Currency Based on Graph Neural Network

In general, the transactions of digital currency are generally modeled with a graph structure. During recent years, Graph Neural Network (GNN) [12], which is very popular in the field of deep learning, has excellent effect on the processing of graph structure. Therefore, this article considers Graph Convolutional Neural Network (GCN) [13–16] and Graph Attention Network (GAT) [17], the two pillars of the Graph Neural Network, as models for monitoring illegal transactions of digital currency.


#### 3.2.1. Graph Convolutional Network (GCN)

Graph Convolutional Network (GCN) is similar to Cognition Evolutionary Algorithm. It usually uses spectral convolution to drive neighborhood aggregation, which consists of multilayer graph convolution. In this article, the digital currency transaction graph is assumed from the data setG = (N, E), where N represents the features of nodes and E represents the edge features of the graph. In the Graph Convolutional Network (GCN), the input of each layer is the node feature matrix H and the adjacency matrix A, and then the node feature matrix H is updated as the output through the weight matrix W. The node characteristic matrix H and the weight matrix W are not necessarily the same in each layer, while the adjacency matrix A is always the same. Equation (1) defines the mathematical formula of Graph Convolutional Network:(1)Hl+1=σD˜−1/2A˜D˜−1/2HlWl,where l represents the current number of layers and σ is usually taken as a ReLU activation function. The input feature matrix comes from the features of the graph nodes, such as H0=X.Formula (2) defines A˜ and D˜:(2)A˜=A+I,D˜=diag∑jAij˜,where A˜ represents adjacency matrix plus self-linking and D˜ represents degree matrix.We can assume that the Graph Neural Network is used for node classification, the graph convolution layer is set, the activation function is ReLU, and the final output layer is softmax output. First, the characteristicH0 and adjacency matrix A of the node are input, convolved through the layer graph l, and finally output Hl through softmax, which is composed of the predicted probability. The graph convolution layer of each layer differs from its feed forward layer only by the front multiplication A^, which can be understood as a set of transformation embedding of adjacent nodes. It is driven by a spectral filter and is the result of a linear function of the Laplace matrix. The parameter to be updated is Wl. Equation (3) defines the mathematical formula of a 2-layer Graph Convolutional Network (GCN):(3)H2=softmaxA^⋅ReLUA^XW0⋅W1.


#### 3.2.2. Graph Attention Network (GAT)

As another representative Graph Convolutional Network, Graph Attention Network (GAT) adds attention mechanism to achieve better neighbor aggregation compared with the GCN. By learning the weight of neighbors, the Graph Attention Network can achieve the weighted aggregation of neighbors. Therefore, GAT is not only robust to noisy neighbors but the use of the attention mechanism also makes the model more interpretable.Equation (4) gives a linear transformation formula for node embedding of the l layer, where Wl is the parameter to be trained in the network and H is the node feature matrix:(4)zil=WlHil.Equation (5) gives the formula for calculating the original attention score between pairs of nodes. First, the nodes zi and zj are embedded and spliced, where  represents splicing, and then the dot product of the spliced embed and weight vector a⟶ is carried out, where a⟶ is the learnable weight vector and finally uses LeakyReLU as the activation function. This form of attention mechanism is often referred to as additive attention.(5)eijl=LeakyReLUa⟶lTzilzjl.Equation (6) gives the formula of attention weight. Softmax is used to calculate the original attention score of node entry edge, where Ni represents the neighbor node.(6)aijl=expeijl∑k∈Niexpeikl.Equation (7) gives the node feature update formula and carries out the weighted summation based on attention on the features of all neighbor nodes:(7)Hil+1=σ∑j∈Niaijlzjl.In essence, GAT transforms the normalized constant used in GCN for first-order neighbor nodes into a neighbor node feature aggregation function using attention weights.


### 3.3. Mutual Information Prior Loss

In statistics, the loss function is a measure of the degree of system error. In the supervised learning model, the loss function measures the difference between the predicted value of the sample produced by the model and the true label of the sample. In this section, we mainly introduce how to improve the loss function through the idea of mutual information, so as to alleviate the problem of class imbalance.


#### 3.3.1. Cross Entropy

The cross entropy loss function is the most widely used loss function in the field of machine learning today. Because the weight gradient of the last layer of the cross entropy loss function is not related to the derivative of the activation function but is only proportional to the difference between the real value and the output value, it converges quickly. Also, the back propagation is in the form of multiplication, so the whole weight matrix is updated quickly. Equation (8) gives the formula of cross entropy loss function:(8)Hp,q=−∑ipcilogqci,where qci is the output of the network, which is equivalent to the output result after the sample is input to the neural network, and pci is the expected sample distribution, equivalent to the label of real data.In this article, we mainly focus on the classification of labels. It is assumed that there arek categories, the training data are x,y∼D, and the distribution of the modeling is pθy|x. So, the optimization goal is to maximize likelihood estimation or to minimize the cross entropy. Equation (9) gives the formula for minimizing cross entropy:(9)argminθEx,y∼D−logpθy|x.


#### 3.3.2. Mutual Information

Mutual information [18] measures the dependence between two random variables, that is, how much less uncertainty y is given an x. If there is no relationship between x and y, then the value of mutual information is the minimum of 0. If given x can completely eliminate the uncertainty of y, then the value of mutual information is the maximum entropy of x.For any discrete random variable, its entropy is defined as formula (8).For the joint entropy of a group of random variables,x,y is similar to the entropy definition of a single discrete random variable. Equation (10) gives the joint entropy HX;Y formula:(10)HX;Y=−∑y∈Y∑x∈Xpx,ylogqx,y,where px,y is the joint probability distribution function of X and Y and pxandpy are the edge probability distribution function of X and Y, respectively. Equation (11) gives the mutual information formula:(11)IX;Y=HX+HY−HX,Y,where HX,HY,andHX,Y > 0 and IX;Y > 0.Mutual information can essentially express the relationship between input and output. Compared with fitting conditional probability, if the model can directly fit mutual information, more essential knowledge will be learned because mutual information is the indicator that reveals the core relationship. However, the training of mutual information fitting is very difficult, and the conditional probability is generally trained. The most commonly used method is to use cross entropy−logpθy|x directly for training. Therefore, how to make the model use cross entropy loss to fit the mutual information has become the focus of research [19].Softmax is often used as the activation function of the last layer of neural network in binary classification or multiple classification problems and is used to represent the predicted output of neural network due to its normalization function and easy derivation. Equation (12) gives the formula of softmax generalization:(12)qcj=ezj∑i=1nezi.In a neural network,zj is the output of the upper layer, qcj is the distribution form of the output of this layer, and ezj is a sum of ezi within a batch.


#### 3.3.3. Improved Loss Function Based on Mutual Information

First, this study assumes that logits isfx;θ, the output result of the network model. Substituting it into softmax, equation (13) can be obtained:(13)pθy|x=efyx;θ∑i=1Kefix;θ.Equation (14) gives the loss function form of equation (13):(14)−logpθy|x=−logefyx;θ∑i=1Kefix;θ=log1+∑i≠yefix;θ−fyx;θ.Equation (14) is the conventional softmax cross entropy. Equation (15) is the modeling of mutual information:(15)logpθy|xpy∼fyx;θ⇔logpθy|x∼fyx;θ+logpy.Equation (16) is the result after the softmax normalization is repeated in the form of the right end:(16)pθy|x=efyx;θ+logpy∑i=1Kefix;θ+logpi.Equation (17) is the loss function form of equation (16):(17)−logpθy|x=−logefyx;θ+logpy∑i=1Kefix;θ+logpi=log1+∑i≠ypipyefix;θ−fyx;θ.If it is more general, the formula of equation (18) after adding regulatory factor τ is as follows:(18)−logpθy|x=−logefyx;θ+logpy∑i=1Kefix;θ+logpi=log1+∑i≠ypipyτefix;θ−fyx;θ.In this way, the model still uses cross entropy as the loss function, but in essence it is fitting mutual information. An offset associated with the label prior is applied to each logarithmic output (i.e., the result before softmax activation) [20]. The training process combined with a neural network is shown in Figure 1.Figure 1
The process of training in a neural network.According to the above process, a neural network is first determined, datax are input into the neural network, and a coded logits is obtained through the neural network, and then logits is subtracted from one Inpy, in which py is the prior information of the label. This is then fed into the softmax activation layer, resulting in the predicted output y.


## 3.1. Question Raising

To detect illegal digital currency transactions, we need to formalize digital currency transactions into a typical machine learning problem. We can regard detecting illegal digital currency transactions as a binary classification. The data set that is used in this article is provided by researchers from MIT, IBM, and Elliptic, a blockchain analytics company. It contains a data set of more than 200,000 Bitcoin transactions with a small number of tokens. It is by far the most cryptocurrency-related and tagged data set [9]. There are two categories in this data set. One is the legal category, which includes a series of legal services such as exchanges, wallet providers, and miners. The other category is illegal, which includes fraud, malware, terrorist groups, ransomware, Ponzi schemes, and other illegal transactions with only a few classified as illegal transactions. More important for detecting illegal transactions of digital currency is the few categories (i.e., illegal categories).This article formally describes the problem as follows:Input:X∈RV∗F,whereV is the number of nodes and F is the characteristic dimensionOutput:Y∈0,1, where 0 is the legal category and 1 is the illegal categoryBecause there is serious data imbalance between legal category and illegal category, in order to solve this problem, this article uses the loss function in terms of the mutual information prior for mitigation.


## 3.2. Illegal Transaction Monitoring of Digital Currency Based on Graph Neural Network

In general, the transactions of digital currency are generally modeled with a graph structure. During recent years, Graph Neural Network (GNN) [12], which is very popular in the field of deep learning, has excellent effect on the processing of graph structure. Therefore, this article considers Graph Convolutional Neural Network (GCN) [13–16] and Graph Attention Network (GAT) [17], the two pillars of the Graph Neural Network, as models for monitoring illegal transactions of digital currency.


### 3.2.1. Graph Convolutional Network (GCN)

Graph Convolutional Network (GCN) is similar to Cognition Evolutionary Algorithm. It usually uses spectral convolution to drive neighborhood aggregation, which consists of multilayer graph convolution. In this article, the digital currency transaction graph is assumed from the data setG = (N, E), where N represents the features of nodes and E represents the edge features of the graph. In the Graph Convolutional Network (GCN), the input of each layer is the node feature matrix H and the adjacency matrix A, and then the node feature matrix H is updated as the output through the weight matrix W. The node characteristic matrix H and the weight matrix W are not necessarily the same in each layer, while the adjacency matrix A is always the same. Equation (1) defines the mathematical formula of Graph Convolutional Network:(1)Hl+1=σD˜−1/2A˜D˜−1/2HlWl,where l represents the current number of layers and σ is usually taken as a ReLU activation function. The input feature matrix comes from the features of the graph nodes, such as H0=X.Formula (2) defines A˜ and D˜:(2)A˜=A+I,D˜=diag∑jAij˜,where A˜ represents adjacency matrix plus self-linking and D˜ represents degree matrix.We can assume that the Graph Neural Network is used for node classification, the graph convolution layer is set, the activation function is ReLU, and the final output layer is softmax output. First, the characteristicH0 and adjacency matrix A of the node are input, convolved through the layer graph l, and finally output Hl through softmax, which is composed of the predicted probability. The graph convolution layer of each layer differs from its feed forward layer only by the front multiplication A^, which can be understood as a set of transformation embedding of adjacent nodes. It is driven by a spectral filter and is the result of a linear function of the Laplace matrix. The parameter to be updated is Wl. Equation (3) defines the mathematical formula of a 2-layer Graph Convolutional Network (GCN):(3)H2=softmaxA^⋅ReLUA^XW0⋅W1.


### 3.2.2. Graph Attention Network (GAT)

As another representative Graph Convolutional Network, Graph Attention Network (GAT) adds attention mechanism to achieve better neighbor aggregation compared with the GCN. By learning the weight of neighbors, the Graph Attention Network can achieve the weighted aggregation of neighbors. Therefore, GAT is not only robust to noisy neighbors but the use of the attention mechanism also makes the model more interpretable.Equation (4) gives a linear transformation formula for node embedding of the l layer, where Wl is the parameter to be trained in the network and H is the node feature matrix:(4)zil=WlHil.Equation (5) gives the formula for calculating the original attention score between pairs of nodes. First, the nodes zi and zj are embedded and spliced, where  represents splicing, and then the dot product of the spliced embed and weight vector a⟶ is carried out, where a⟶ is the learnable weight vector and finally uses LeakyReLU as the activation function. This form of attention mechanism is often referred to as additive attention.(5)eijl=LeakyReLUa⟶lTzilzjl.Equation (6) gives the formula of attention weight. Softmax is used to calculate the original attention score of node entry edge, where Ni represents the neighbor node.(6)aijl=expeijl∑k∈Niexpeikl.Equation (7) gives the node feature update formula and carries out the weighted summation based on attention on the features of all neighbor nodes:(7)Hil+1=σ∑j∈Niaijlzjl.In essence, GAT transforms the normalized constant used in GCN for first-order neighbor nodes into a neighbor node feature aggregation function using attention weights.


## 3.2.1. Graph Convolutional Network (GCN)

Graph Convolutional Network (GCN) is similar to Cognition Evolutionary Algorithm. It usually uses spectral convolution to drive neighborhood aggregation, which consists of multilayer graph convolution. In this article, the digital currency transaction graph is assumed from the data setG = (N, E), where N represents the features of nodes and E represents the edge features of the graph. In the Graph Convolutional Network (GCN), the input of each layer is the node feature matrix H and the adjacency matrix A, and then the node feature matrix H is updated as the output through the weight matrix W. The node characteristic matrix H and the weight matrix W are not necessarily the same in each layer, while the adjacency matrix A is always the same. Equation (1) defines the mathematical formula of Graph Convolutional Network:(1)Hl+1=σD˜−1/2A˜D˜−1/2HlWl,where l represents the current number of layers and σ is usually taken as a ReLU activation function. The input feature matrix comes from the features of the graph nodes, such as H0=X.Formula (2) defines A˜ and D˜:(2)A˜=A+I,D˜=diag∑jAij˜,where A˜ represents adjacency matrix plus self-linking and D˜ represents degree matrix.We can assume that the Graph Neural Network is used for node classification, the graph convolution layer is set, the activation function is ReLU, and the final output layer is softmax output. First, the characteristicH0 and adjacency matrix A of the node are input, convolved through the layer graph l, and finally output Hl through softmax, which is composed of the predicted probability. The graph convolution layer of each layer differs from its feed forward layer only by the front multiplication A^, which can be understood as a set of transformation embedding of adjacent nodes. It is driven by a spectral filter and is the result of a linear function of the Laplace matrix. The parameter to be updated is Wl. Equation (3) defines the mathematical formula of a 2-layer Graph Convolutional Network (GCN):(3)H2=softmaxA^⋅ReLUA^XW0⋅W1.


## 3.2.2. Graph Attention Network (GAT)

As another representative Graph Convolutional Network, Graph Attention Network (GAT) adds attention mechanism to achieve better neighbor aggregation compared with the GCN. By learning the weight of neighbors, the Graph Attention Network can achieve the weighted aggregation of neighbors. Therefore, GAT is not only robust to noisy neighbors but the use of the attention mechanism also makes the model more interpretable.Equation (4) gives a linear transformation formula for node embedding of the l layer, where Wl is the parameter to be trained in the network and H is the node feature matrix:(4)zil=WlHil.Equation (5) gives the formula for calculating the original attention score between pairs of nodes. First, the nodes zi and zj are embedded and spliced, where  represents splicing, and then the dot product of the spliced embed and weight vector a⟶ is carried out, where a⟶ is the learnable weight vector and finally uses LeakyReLU as the activation function. This form of attention mechanism is often referred to as additive attention.(5)eijl=LeakyReLUa⟶lTzilzjl.Equation (6) gives the formula of attention weight. Softmax is used to calculate the original attention score of node entry edge, where Ni represents the neighbor node.(6)aijl=expeijl∑k∈Niexpeikl.Equation (7) gives the node feature update formula and carries out the weighted summation based on attention on the features of all neighbor nodes:(7)Hil+1=σ∑j∈Niaijlzjl.In essence, GAT transforms the normalized constant used in GCN for first-order neighbor nodes into a neighbor node feature aggregation function using attention weights.


## 3.3. Mutual Information Prior Loss

In statistics, the loss function is a measure of the degree of system error. In the supervised learning model, the loss function measures the difference between the predicted value of the sample produced by the model and the true label of the sample. In this section, we mainly introduce how to improve the loss function through the idea of mutual information, so as to alleviate the problem of class imbalance.


### 3.3.1. Cross Entropy

The cross entropy loss function is the most widely used loss function in the field of machine learning today. Because the weight gradient of the last layer of the cross entropy loss function is not related to the derivative of the activation function but is only proportional to the difference between the real value and the output value, it converges quickly. Also, the back propagation is in the form of multiplication, so the whole weight matrix is updated quickly. Equation (8) gives the formula of cross entropy loss function:(8)Hp,q=−∑ipcilogqci,where qci is the output of the network, which is equivalent to the output result after the sample is input to the neural network, and pci is the expected sample distribution, equivalent to the label of real data.In this article, we mainly focus on the classification of labels. It is assumed that there arek categories, the training data are x,y∼D, and the distribution of the modeling is pθy|x. So, the optimization goal is to maximize likelihood estimation or to minimize the cross entropy. Equation (9) gives the formula for minimizing cross entropy:(9)argminθEx,y∼D−logpθy|x.


### 3.3.2. Mutual Information

Mutual information [18] measures the dependence between two random variables, that is, how much less uncertainty y is given an x. If there is no relationship between x and y, then the value of mutual information is the minimum of 0. If given x can completely eliminate the uncertainty of y, then the value of mutual information is the maximum entropy of x.For any discrete random variable, its entropy is defined as formula (8).For the joint entropy of a group of random variables,x,y is similar to the entropy definition of a single discrete random variable. Equation (10) gives the joint entropy HX;Y formula:(10)HX;Y=−∑y∈Y∑x∈Xpx,ylogqx,y,where px,y is the joint probability distribution function of X and Y and pxandpy are the edge probability distribution function of X and Y, respectively. Equation (11) gives the mutual information formula:(11)IX;Y=HX+HY−HX,Y,where HX,HY,andHX,Y > 0 and IX;Y > 0.Mutual information can essentially express the relationship between input and output. Compared with fitting conditional probability, if the model can directly fit mutual information, more essential knowledge will be learned because mutual information is the indicator that reveals the core relationship. However, the training of mutual information fitting is very difficult, and the conditional probability is generally trained. The most commonly used method is to use cross entropy−logpθy|x directly for training. Therefore, how to make the model use cross entropy loss to fit the mutual information has become the focus of research [19].Softmax is often used as the activation function of the last layer of neural network in binary classification or multiple classification problems and is used to represent the predicted output of neural network due to its normalization function and easy derivation. Equation (12) gives the formula of softmax generalization:(12)qcj=ezj∑i=1nezi.In a neural network,zj is the output of the upper layer, qcj is the distribution form of the output of this layer, and ezj is a sum of ezi within a batch.


### 3.3.3. Improved Loss Function Based on Mutual Information

First, this study assumes that logits isfx;θ, the output result of the network model. Substituting it into softmax, equation (13) can be obtained:(13)pθy|x=efyx;θ∑i=1Kefix;θ.Equation (14) gives the loss function form of equation (13):(14)−logpθy|x=−logefyx;θ∑i=1Kefix;θ=log1+∑i≠yefix;θ−fyx;θ.Equation (14) is the conventional softmax cross entropy. Equation (15) is the modeling of mutual information:(15)logpθy|xpy∼fyx;θ⇔logpθy|x∼fyx;θ+logpy.Equation (16) is the result after the softmax normalization is repeated in the form of the right end:(16)pθy|x=efyx;θ+logpy∑i=1Kefix;θ+logpi.Equation (17) is the loss function form of equation (16):(17)−logpθy|x=−logefyx;θ+logpy∑i=1Kefix;θ+logpi=log1+∑i≠ypipyefix;θ−fyx;θ.If it is more general, the formula of equation (18) after adding regulatory factor τ is as follows:(18)−logpθy|x=−logefyx;θ+logpy∑i=1Kefix;θ+logpi=log1+∑i≠ypipyτefix;θ−fyx;θ.In this way, the model still uses cross entropy as the loss function, but in essence it is fitting mutual information. An offset associated with the label prior is applied to each logarithmic output (i.e., the result before softmax activation) [20]. The training process combined with a neural network is shown in Figure 1.Figure 1
The process of training in a neural network.According to the above process, a neural network is first determined, datax are input into the neural network, and a coded logits is obtained through the neural network, and then logits is subtracted from one Inpy, in which py is the prior information of the label. This is then fed into the softmax activation layer, resulting in the predicted output y.


## 3.3.1. Cross Entropy

The cross entropy loss function is the most widely used loss function in the field of machine learning today. Because the weight gradient of the last layer of the cross entropy loss function is not related to the derivative of the activation function but is only proportional to the difference between the real value and the output value, it converges quickly. Also, the back propagation is in the form of multiplication, so the whole weight matrix is updated quickly. Equation (8) gives the formula of cross entropy loss function:(8)Hp,q=−∑ipcilogqci,where qci is the output of the network, which is equivalent to the output result after the sample is input to the neural network, and pci is the expected sample distribution, equivalent to the label of real data.In this article, we mainly focus on the classification of labels. It is assumed that there arek categories, the training data are x,y∼D, and the distribution of the modeling is pθy|x. So, the optimization goal is to maximize likelihood estimation or to minimize the cross entropy. Equation (9) gives the formula for minimizing cross entropy:(9)argminθEx,y∼D−logpθy|x.


## 3.3.2. Mutual Information

Mutual information [18] measures the dependence between two random variables, that is, how much less uncertainty y is given an x. If there is no relationship between x and y, then the value of mutual information is the minimum of 0. If given x can completely eliminate the uncertainty of y, then the value of mutual information is the maximum entropy of x.For any discrete random variable, its entropy is defined as formula (8).For the joint entropy of a group of random variables,x,y is similar to the entropy definition of a single discrete random variable. Equation (10) gives the joint entropy HX;Y formula:(10)HX;Y=−∑y∈Y∑x∈Xpx,ylogqx,y,where px,y is the joint probability distribution function of X and Y and pxandpy are the edge probability distribution function of X and Y, respectively. Equation (11) gives the mutual information formula:(11)IX;Y=HX+HY−HX,Y,where HX,HY,andHX,Y > 0 and IX;Y > 0.Mutual information can essentially express the relationship between input and output. Compared with fitting conditional probability, if the model can directly fit mutual information, more essential knowledge will be learned because mutual information is the indicator that reveals the core relationship. However, the training of mutual information fitting is very difficult, and the conditional probability is generally trained. The most commonly used method is to use cross entropy−logpθy|x directly for training. Therefore, how to make the model use cross entropy loss to fit the mutual information has become the focus of research [19].Softmax is often used as the activation function of the last layer of neural network in binary classification or multiple classification problems and is used to represent the predicted output of neural network due to its normalization function and easy derivation. Equation (12) gives the formula of softmax generalization:(12)qcj=ezj∑i=1nezi.In a neural network,zj is the output of the upper layer, qcj is the distribution form of the output of this layer, and ezj is a sum of ezi within a batch.


## 3.3.3. Improved Loss Function Based on Mutual Information

First, this study assumes that logits isfx;θ, the output result of the network model. Substituting it into softmax, equation (13) can be obtained:(13)pθy|x=efyx;θ∑i=1Kefix;θ.Equation (14) gives the loss function form of equation (13):(14)−logpθy|x=−logefyx;θ∑i=1Kefix;θ=log1+∑i≠yefix;θ−fyx;θ.Equation (14) is the conventional softmax cross entropy. Equation (15) is the modeling of mutual information:(15)logpθy|xpy∼fyx;θ⇔logpθy|x∼fyx;θ+logpy.Equation (16) is the result after the softmax normalization is repeated in the form of the right end:(16)pθy|x=efyx;θ+logpy∑i=1Kefix;θ+logpi.Equation (17) is the loss function form of equation (16):(17)−logpθy|x=−logefyx;θ+logpy∑i=1Kefix;θ+logpi=log1+∑i≠ypipyefix;θ−fyx;θ.If it is more general, the formula of equation (18) after adding regulatory factor τ is as follows:(18)−logpθy|x=−logefyx;θ+logpy∑i=1Kefix;θ+logpi=log1+∑i≠ypipyτefix;θ−fyx;θ.In this way, the model still uses cross entropy as the loss function, but in essence it is fitting mutual information. An offset associated with the label prior is applied to each logarithmic output (i.e., the result before softmax activation) [20]. The training process combined with a neural network is shown in Figure 1.Figure 1
The process of training in a neural network.According to the above process, a neural network is first determined, datax are input into the neural network, and a coded logits is obtained through the neural network, and then logits is subtracted from one Inpy, in which py is the prior information of the label. This is then fed into the softmax activation layer, resulting in the predicted output y.


## 4. Experiments and Results

### 4.1. Data Set

The data set used in this article was provided by Elliptic, Inc., with 203,769 node transactions and 234,355 edges. About 21 percent (42,019) of the transactions were marked as legal, 2 percent (4,545) were marked as illegal, and the rest were marked as unknown but with other characteristics. There are 49 different graphs, each of which is unrelated to each other. A node represents a transaction, and an edge represents the flow of Bitcoins.There are two problems with this data set at present: a large number of unmarked nodes and a serious data imbalance due to the fact that illegal categories only account for 10% of the nodes with tags compared to legal categories (Figure2).Figure 2
Proportion of node labels in the data set.


### 4.2. Measurement Standard

In order to verify the advantages of the method in this article more objectively, we need to evaluate the effect of the model after the machine learning system completes the modeling. The evaluation indexes were Precision, Recall, and F1-Score for comparative analysis. This is a dichotomous task, with a serious imbalance between legal and illegal categories. For the detection of illegal digital currency, it is more important for a few categories (i.e., illegal categories), so this article will focus on the F1-Score of illegal categories. We can take illegal as positive and legal as negative.The calculation formula of Precision is as follows:(19)Precision=TPTP+FP.The Recall formula is as follows:(20)Recall=TPTP+FN.The F1-Score formula is as follows:(21)F1−Score=2⋅Precision⋅RecallPrecision+Recall.TP is truly indicating that the number is correctly divided into illegal. FP is false positive, indicating that legal numbers are classified as illegal. TN is true negative, which means the number of valid ones. FN is false negative, which means illegal is classified as a legal number.


### 4.3. Experiment and Result Analysis

The hardware environment of the experiment in this article is as follows: operating system Ubuntu 18.04.5LTS, CPU model Intel Xeon E5-2643, memory 318G, graphics card NVIDIA GeForce RTX 1080 8G ∗3, and GPU acceleration library CUDA10.0. The deep learning framework is PyTorch.In this article, the Elliptic data set is divided into training set and test set according to the ratio of 6 : 1 : 3. That is to say, the first 31 graphs are used for training the model, and the 5 graphs from 31 to 36 are verified, and the 13 graphs are used for testing the model. The following are the experimental results obtained on the Elliptic data set in this article.Since all the models lost their effectiveness after graph 43, the survey found that the United States had a serious crack down on digital currency crimes during that period, with only 1 or 2 of each chart flagged as illegal activities from graphs 43 to 49. Therefore, this article extracted data of different models and different loss functions from graphs 37 to 42 for comparison.In this article, the Adam optimizer was used to set the learning rate to 0.001 to train the GCN and GAT models with 1000 epoch. Two hidden layers were used in the experiment, and the number of nodes in the hidden layer was set as 100. The prediction results and F1-Score of GCN, GAT, and two benchmark methods (DNN [21] (PyTorch implementation) and SVM [22] (PyTorch implementation)) are summarized in Figure 3.Figure 3
Experimental results of different methods on different loss.According to the data in Figure3, focal loss function on DNN and SVM did not improve but decreased compared with the ordinary cross-entropy loss function, but the loss function based on mutual information prior had a large improvement compared with it. On GCN, focal loss has a certain improvement compared with ordinary cross-entropy loss, and the loss based on mutual information prior has a certain improvement compared with focal loss. In GAT, it is found that although the loss based on mutual information prior is improved compared with the traditional cross-entropy loss, it is not as good as focal loss.The average results of the indicators from graphs 37 to 42 are summarized in Table1.Table 1
Graph 37–42 experimental results of different methods on different loss.
ModelLossPrecisionRecallF1SVMCross0.770.770.77Focal0.840.700.76Prior0.860.720.78DNNCross0.430.520.47Focal0.380.430.40Prior0.410.670.50GCNCross0.700.520.58Focal0.680.550.60Prior0.690.580.62GATCross0.580.710.63Focal0.750.590.66Prior0.750.580.65The bold values in the loss column are the methods we proposed, and the bold values in the prior are the best ones under the same evaluation index.The following conclusions can be drawn from Table1:(i)
Graph neural network is less effective than SVM; the reason may be that the node in the data set is not an account but a transaction. In normal graphs, nodes represent addresses and edges represent transactions, so the information learned using Graph Neural Network may be wrong.(ii)
The loss function based on mutual information prior in SVM improved by 1% compared with the traditional cross entropy loss F1-Score and 2% compared with focal loss F1-Score; in DNN, the cross entropy loss F1-Score increased by 3% compared with the traditional cross entropy loss F1-Score, and the F1-Score increased by 10% compared with focal loss; compared with the traditional cross entropy loss F1-Score in GCN, the F1-Score improved by 4%; compared with the focal loss F1-Score, the F1-Score improved by 2%. Compared with the traditional cross entropy loss F1-Score, GAT increased by 2%, but compared with focal loss F1-Score, it decreased by 1%. The reason may be the conflict between the attention mechanism of GAT and the priori based on mutual information.(iii)
GAT has an overall improvement over GCN, indicating that introducing the attention mechanism into the graphics network is an effective operation.It can be intuitively seen from the above chart that the loss function based on mutual information prior can effectively improve the problem of sample imbalance in the detection of digital currency illegal behaviors.


## 4.1. Data Set

The data set used in this article was provided by Elliptic, Inc., with 203,769 node transactions and 234,355 edges. About 21 percent (42,019) of the transactions were marked as legal, 2 percent (4,545) were marked as illegal, and the rest were marked as unknown but with other characteristics. There are 49 different graphs, each of which is unrelated to each other. A node represents a transaction, and an edge represents the flow of Bitcoins.There are two problems with this data set at present: a large number of unmarked nodes and a serious data imbalance due to the fact that illegal categories only account for 10% of the nodes with tags compared to legal categories (Figure2).Figure 2
Proportion of node labels in the data set.


## 4.2. Measurement Standard

In order to verify the advantages of the method in this article more objectively, we need to evaluate the effect of the model after the machine learning system completes the modeling. The evaluation indexes were Precision, Recall, and F1-Score for comparative analysis. This is a dichotomous task, with a serious imbalance between legal and illegal categories. For the detection of illegal digital currency, it is more important for a few categories (i.e., illegal categories), so this article will focus on the F1-Score of illegal categories. We can take illegal as positive and legal as negative.The calculation formula of Precision is as follows:(19)Precision=TPTP+FP.The Recall formula is as follows:(20)Recall=TPTP+FN.The F1-Score formula is as follows:(21)F1−Score=2⋅Precision⋅RecallPrecision+Recall.TP is truly indicating that the number is correctly divided into illegal. FP is false positive, indicating that legal numbers are classified as illegal. TN is true negative, which means the number of valid ones. FN is false negative, which means illegal is classified as a legal number.


## 4.3. Experiment and Result Analysis

The hardware environment of the experiment in this article is as follows: operating system Ubuntu 18.04.5LTS, CPU model Intel Xeon E5-2643, memory 318G, graphics card NVIDIA GeForce RTX 1080 8G ∗3, and GPU acceleration library CUDA10.0. The deep learning framework is PyTorch.In this article, the Elliptic data set is divided into training set and test set according to the ratio of 6 : 1 : 3. That is to say, the first 31 graphs are used for training the model, and the 5 graphs from 31 to 36 are verified, and the 13 graphs are used for testing the model. The following are the experimental results obtained on the Elliptic data set in this article.Since all the models lost their effectiveness after graph 43, the survey found that the United States had a serious crack down on digital currency crimes during that period, with only 1 or 2 of each chart flagged as illegal activities from graphs 43 to 49. Therefore, this article extracted data of different models and different loss functions from graphs 37 to 42 for comparison.In this article, the Adam optimizer was used to set the learning rate to 0.001 to train the GCN and GAT models with 1000 epoch. Two hidden layers were used in the experiment, and the number of nodes in the hidden layer was set as 100. The prediction results and F1-Score of GCN, GAT, and two benchmark methods (DNN [21] (PyTorch implementation) and SVM [22] (PyTorch implementation)) are summarized in Figure 3.Figure 3
Experimental results of different methods on different loss.According to the data in Figure3, focal loss function on DNN and SVM did not improve but decreased compared with the ordinary cross-entropy loss function, but the loss function based on mutual information prior had a large improvement compared with it. On GCN, focal loss has a certain improvement compared with ordinary cross-entropy loss, and the loss based on mutual information prior has a certain improvement compared with focal loss. In GAT, it is found that although the loss based on mutual information prior is improved compared with the traditional cross-entropy loss, it is not as good as focal loss.The average results of the indicators from graphs 37 to 42 are summarized in Table1.Table 1
Graph 37–42 experimental results of different methods on different loss.
ModelLossPrecisionRecallF1SVMCross0.770.770.77Focal0.840.700.76Prior0.860.720.78DNNCross0.430.520.47Focal0.380.430.40Prior0.410.670.50GCNCross0.700.520.58Focal0.680.550.60Prior0.690.580.62GATCross0.580.710.63Focal0.750.590.66Prior0.750.580.65The bold values in the loss column are the methods we proposed, and the bold values in the prior are the best ones under the same evaluation index.The following conclusions can be drawn from Table1:(i)
Graph neural network is less effective than SVM; the reason may be that the node in the data set is not an account but a transaction. In normal graphs, nodes represent addresses and edges represent transactions, so the information learned using Graph Neural Network may be wrong.(ii)
The loss function based on mutual information prior in SVM improved by 1% compared with the traditional cross entropy loss F1-Score and 2% compared with focal loss F1-Score; in DNN, the cross entropy loss F1-Score increased by 3% compared with the traditional cross entropy loss F1-Score, and the F1-Score increased by 10% compared with focal loss; compared with the traditional cross entropy loss F1-Score in GCN, the F1-Score improved by 4%; compared with the focal loss F1-Score, the F1-Score improved by 2%. Compared with the traditional cross entropy loss F1-Score, GAT increased by 2%, but compared with focal loss F1-Score, it decreased by 1%. The reason may be the conflict between the attention mechanism of GAT and the priori based on mutual information.(iii)
GAT has an overall improvement over GCN, indicating that introducing the attention mechanism into the graphics network is an effective operation.It can be intuitively seen from the above chart that the loss function based on mutual information prior can effectively improve the problem of sample imbalance in the detection of digital currency illegal behaviors.


## 5. Conclusion

As for the serious imbalance of positive and negative samples in the detection of digital currency illegal behaviors, this study integrates mutual information and cross entropy loss function and obtains a loss function based on mutual information prior. On the Elliptic data set, a comparative experiment will be conducted whether to use the method of mutual information prior loss function. The results show that, using this loss function, the F1 value of each method is improved to varying degrees compared with the original cross-entropy loss function and focal loss function. Among them, GCN and DNN were significantly improved, and SVM also had a certain improvement. Although the improvement in GAT is not as good as focal loss, it also has a large improvement compared with the ordinary cross entropy loss. Therefore, the loss function based on mutual information prior proposed in this article plays a significant role in the detection of digital currency illegal behaviors.



---

*Source: 9954204-2021-08-31.xml*