# Non-Intrusive Load Disaggregation Based on a Multi-Scale Attention Residual Network
## 2 Multi-Scale Attention Residual Network  
### Deep Residual Network  
The depth of the neural network [28] is crucial to the performance of the model. Increasing the depth of the neural network is helpful to extract more complex features and improve the accuracy of the algorithm. However, at the same time, when the network reaches a certain depth, further deepening would make the training accuracy saturated or even reduced. Traditional methods dealing with gradient disappearance and gradient explosion, e.g., the activation function Relu and batch normalization, are able to alleviate these problems to a certain extent, but not fundamentally solve the problems. The deep residual network (Resnet) [29] uses the idea of cross-layer connection. If those behind the deep network are identity mapping, the model would degenerate directly into a shallow network, and it is difficult to use the neural network stacked with hidden layers to fit into a potential identity mapping function \(H(x)=x\). However, if the network is designed to be \(H(x)=F(x)+x\), the procedure can be translated into learning a residual function \(F(x)=H(x)-x\). When \(F(x)=0\), it presents an identity map \(F(x)=H(x)\). The residual network is a structure that outputs the features of front layers to back layers. By introducing this structure, a neural network is able to perform well. Assuming that the input of a forward neural network is \(x\), the dimension is \(H\times W\times C\), and the expected output of the network is \(H(x)\). The residual structure \(F(x)+x\) can be realized by a forward neural network and a skip connection. When the number of input and output channels is the same, the dimension of \(H(x)\) is also \(H\times W\times C\). A skip connection means that one or more layers of the network are bypassed. A skip connection performs only one identity mapping and adds its outputs to the outputs of stack layers. The residual structure is shown in Figure 1. Each structure has two layers, where \(W_{1}\) and \(W_{2}\) are the weight matrixes. The input is sequentially multiplied by the \(W_{1}\) matrix of the first layer, activated by the relu function, and then multiplied by the \(W_{2}\) matrix of the second layer to obtain the forward output. The forward neural network of the residual structure could be expressed as  
\[F(x)=W_{2}\sigma(W_{1}x), \tag{1}\]  
where \(\sigma\) is the activation function. The right hand is the skip connection \(x\) of the residual structure, and the final output is obtained after a summation operation. The formula is as follows:  
\[y=F(x,W_{i})+x, \tag{2}\]  
where \(F(x,W_{i})\) is the residual mapping function that needs to be studied. \(W_{i}\) represents the weight matrix of the hidden layer. When the input and output dimensions need to be changed, a linear transformation \(W\cdot x\) could be performed through the input at the skip connection residual structure. Thus, the same figure of input and output characteristics could be expressed as  
\[H(x)=F(x,W_{i})+W_{s}x. \tag{3}\]  
According to Equation (3), it could be noted that for a deeper layer \(L\), its relationship with the \(l\) layer could be expressed as \[x_{L}=x_{l}+\sum_{i-1}^{L-1}F(x_{l},W_{l}), \tag{4}\]  
where layer \(x_{L}\) and layer \(x_{l}\) are the residual unit inputs of layer \(L\) and layer \(l\). According to Equation (4), inputs of the residual unit in layer \(L\) could be expressed as the sum of inputs of a shallow residual unit and all the complex mappings. The calculation power needed of the sum is much less than that of the quadratics. With a loss function, and according to the chain rule of back propagation, we can obtain  
\[\frac{\partial\epsilon}{\partial x_{l}}=\frac{\partial\epsilon}{\partial x_{L }}\frac{\partial x_{L}}{\partial x_{l}}=\frac{\partial\epsilon}{\partial x_{ L}}(1+\frac{\partial}{\partial x_{l}}\sum_{L-1}^{i=1}F(x_{i},W_{l})). \tag{5}\]  
This means that continuous multiplications generated in the network are replaced by plus operations, and the problem of gradient disappearance is well solved.