# Non-Intrusive Load Disaggregation Based on a Multi-Scale Attention Residual Network
## 2 Multi-Scale Attention Residual Network
### Multi-Scale Attention Resnet Based NILD  
From the point of view of neural networks, the problem of NILM can be interpreted in this way. Assuming that \(Y(t)\) is the sum of all the active power consumption of appliances in the household, it can be expressed as the following formula:  
\[Y(t)=\sum_{i}^{I}X_{i}(t)+e(t). \tag{9}\]  
In the formula, \(X_{i}(t)\) represents the power data of electrical equipment \(i\) at time \(t\), \(I\) represents the number of electrical equipment, and \(e\) represents the model noise. Therefore, there is a pair of data \((X,Y)\), a model can be trained to represent the relationship between \(X\) and \(Y\). \(X=f(Y)\) is a non-linear regression problem, and the neural network is an effective method to learn the function \(f\).  
The overall network structure of the multi-scale attention residual network (MSA-Resnet) proposed in this paper is shown in Figure 3.  
Figure 2: The attention block.  
(a) The multi-scale module is composed of convolution kernels with sizes of \(3\times 1\), \(5\times 1\), and \(1\times 1\) and the pooling layer [33]. By combining (b) the attention block and the residual structure, (c) the multi-scale attention residual block is formed. The structure of (a) is shown in Figure 4.  
All convolution cores of original residual elements are \(3\times 1\) in size, which makes it impossible for convolution layers to observe load data from multiple scales, and difficult to obtain more abundant load features. The multi-scale module first goes though a \(3\times 1\) convolution, followed by four branches. The first branch uses a \(1\times 1\) convolution to increase the load characteristics transformation [34], and a \(3\times 1\) convolution is then applied to obtain a feature map (map1). The second branch is convolved at \(1\times 1\), and a \(5\times 1\) convolution is then added to obtain map2. The third branch is pooled at \(3\times 1\) to obtain map3. The fourth branch uses a \(1\times 1\) convolution to obtain map4. Finally, these feature maps are concatenated to input vectors for the attention module. Because the actual load power has a large number of different gear positions, switch starts and stops, and operating characteristics, the multi-scale feature method can improve the network's ability to extract load characteristics and increase the diversity of different scales of the network, thus improving the accuracy of non-intrusive load disaggregation.  
In the network structure of MSA-Resnet, nine multi-scale attention residual blocks are used. Forty convolution cores are used in the first three blocks, 60 convolution cores are used in the forth to sixth blocks, and 80 convolution cores are allocated in the last three. The first convolution layer and each output part of multi-scale attention residual block is activated by an activation function Leaky-Relu. Relu and Leaky-Relu [35] are shown in Figure 5:  
Figure 4: Multi-scale block.  
Figure 3: The MSA-Resnet overall structure.  
The Relu activation function represents the \"modified linear element,\" which could accelerate the convergence speed of the network. Its equation is  
\[f(x)=max(0,x). \tag{10}\]  
When the input is positive, the derivative is not zero, so the learning is based on the gradient. However, when the input is negative, the learning speed of Relu slows down and can even inactivate the neurons, such that it cannot follow new weights. Equation (11) represents the Leaky-Relu activation function, where \(\lambda\in(0,1)\) modifies the data distribution and retains the value of the negative axis. As a result, the information retention ability is improved without losing more load characteristics, and the gradient is guaranteed not to appear.  
\[f(x)=\begin{cases}x&\text{if }x>0\\ \lambda x&\text{if }x<0\end{cases}. \tag{11}\]