# A Lightweight Graph Neural Network Algorithm for Action Recognition Based on Self-Distillation
## 1 Introduction  
Human activity recognition (HAR) is aimed at developing methodologies to recognize different actions from a sequence of observations. It has been widely used in many applications, such as health monitoring, intelligent surveillance, virtual reality and human-computer interaction. HAR can help detect a baby's dangerous behaviors, detect crowd activities to prevent crowd crush and detect the actions of athletes to help the jury in sports competitions. It can also observe actions while playing virtual reality games in order to provide better feedback to users. Robots which interact with humans also need to recognize human actions.  
Recently, graph neural networks (GNNs) have come into the spotlight to recognize human actions. GNNs regard human skeletons as graphs, which emphasize the topology of graph joints, and can protect personal information such as facial identities and home settings. The earliest milestone is spatial-temporal graph convolutional networks (ST-GCN) [1] (Figure 1). Thereafter, multiple works based on ST-GCN have been proposed. Among them, the two-stream adaptive graph convolutional network (2s-AGCN) [2] is another typical work which adopts attention mechanisms. As GNNs are specialized in discovering the intrinsic relations between joints, GNN-based HAR methods have achieved a new state-of-the-art (SOTA).  
In real-life, a popular real-time action recognition algorithm should be light enough so as to quickly recognize actions without decreasing accuracy severely. However, popular GNN models such as ST-GCN usually prefer more layers to guarantee the model capacity and thereafter the model performance, leading to heavyweight models. Heavyweight models are computationally expensive and memory intensive while real-time recognition needs lightweight models. Given a heavyweight model, algorithm compression is one practical way for generating a lightweight algorithm while preserving the performance.  
There are many ways to compress neural networks (NNs). Among them, knowledge distillation (KD) is well-known [3] because the structure of the light algorithm that it candiscover is more flexible. Self-distillation, a KD model, was proposed to train and compress heavy models simultaneously. It does not need to carefully redesign the architecture, but instead simply takes part of the heavy model as the light model. During training, the knowledge from the deeper layers is squeezed to guide the shallower layers in order to make them achieve a similar performance to the deeper layers. BYOT [4] is one well-known self-distillation model. It was first proposed for ResNet [5] to perform object detection. The authors argue that the lightweight ResNet by BYOT contributes to comparable accuracy compared with the heavyweight ResNet. They did not analyze BYOT's capability for compressing GNNs thoroughly, nor did they analyze the extracted representations concretely. As far as we know, we are the first to use BYOT to generate a lightweight GNN backbone for HAR tasks. The heavyweight GNN backbone can be compressed by up to 80% without losing much performance.  
This paper is organized as follows: Section 2 gives a literature review on GNNs and self-distillation. Then, Section 3 demonstrates the methodology, leading to the following Section 4, which analyzes the performance of the proposed methods and the characteristics of learned representations, and Section 5, which provides the summary.