# A Lightweight Graph Neural Network Algorithm for Action Recognition Based on Self-Distillation
## 3 Algorithm
### ST-GCN Compression Based on Self-Distillation
#### 3.3.2 Self-Distillation Compression  
If \(i<j\), any ST-GCN block \(f_{h}^{b_{i}}\) can be defined as a light network compressed based on the heavy network \(f_{h}^{b_{i}}\). Based on this idea, the light compressed network \(f_{l}\) is formed as  
\[f_{l} = \arg\min_{f_{h}^{b_{i}}}f_{c}(f_{h}^{b_{i}}(\cdot),f_{h}(\cdot)),\] (6) s.t. \[i<N \tag{7}\]  
where \(N\) is the maximum index of blocks in \(f_{h}\), \(f_{c}\) denotes the compressing model which forces \(f_{h}^{b_{i}}\) to be close to the performance of the whole uncompressed model \(f_{h}\), s.t. denotes \"subject to\" and \(b_{i}\) denotes block \(i\). The compressing model \(f_{c}\) adopted here is BYOT; its framework is shown in Figure 5. The basic idea of BYOT is taking the knowledge from the deeper blocks and squeezing this knowledge to the shallower blocks; therefore, the shallower blocks will have similar performance to the deeper blocks. This idea is also named as self-distillation. The benefit of the self-distillation structure was confirmed in [4]. The authors showed that it can help convergence to flat minima, which improves generalization inherently and can prevent models from having the vanishing gradient problem. Here we first use it to compress skeleton graph-based HAR backbones. Figure 4: The blocks division of each backbone. The green layers denoted by ‘Preprocess’ represent preprocessing such as batch normalization. The number denotes the dimensions of the output features from each block. The b1, b2, b3 indicate ST Blocks, and each layer (ST-GCN unit, marked by red, purple or blue) contains two units: graph spatial unit and temporal unit. Therefore, for ST-GCN 20, the number of layers in b1, b2, b3 are 18 layers in total, and counting the preprocessing layer and output fully connection (FC) layer, that would be 20 layers. Figure 3: The spatial partition to build new adjacency matrices. The black cross is the center of gravity of the whole skeleton, usually summarized as the average of all joints. The green nodes are the selected root joints and the purple nodes are centripetal nodes, while the orange nodes are centrifugal joints. The distance between the centripetal nodes and the center of gravity is shorter than the distance between the centrifugal nodes and the center of gravity. Since \(f_{h}\) is trained and distilled simultaneously, a supervised loss purely from labels is not sufficient to distill the knowledge from the deeper block to the shallower block. To deal with this requirement, the loss functions are built from three resources: labels to have supervision, the prediction just after the Softmax layer to limit the predicted distribution from each block and the features after each block to force feature similarity. The action labels capture the prior knowledge of actions; therefore, the cross entropy loss aims to approximate the predicted labels in order to make the predictions close to this prior knowledge. The predicted distributions from different blocks are expected to be similar, so their distances are measured by KL divergence and are forced to be small enough. Furthermore, the features from each blocks hold better performance if kept close together, and they are controlled by the \(\ell_{2}\) norm. These losses are marked as \(L_{s}\), \(L_{d}\) and \(L_{f}\), respectively, with \(L_{s}\) being the cross entropy loss from action labels under supervision, \(L_{d}\) being the distribution loss and \(L_{f}\) being the feature distance loss.