# Human Behavior Analysis: A Survey on Action Recognition
## 2 Human Action Recognition
### Global Representations
Nevertheless, commonly using object detectors and trackers as its foundation, one of the most promising human representations is the extraction of multi-person pose estimation [61; 62; 63; 64; 65; 66], as shown in Figure 5. Human skeleton sequences have three distinguishing characteristics: Starting with the existence of strong correlations between each node and adjacent nodes, consequently, skeleton frames are rich in body structural information. Second, its temporal continuity exists across frames within the same joints and also in the body structure, and, last but not least, a co-occurrence relationship between spatial and temporal domains is present in that kind of data. Furthermore, this technique overcomes all appearance noises that human region proposals can contain, being modular, semantically rich and very descriptive, and consequently, driving the learning process of the model exclusively on human behaviour. One of the earliest methods that explored skeleton data for action recognition was the work by [PERSON] et al. [67], where they explored the self-similarity matrix (SSM), which is computed by the distances between action representations of all pairs of time frames. They claimed that the SSMs are approximately invariant under viewpoint changes, as illustrated in Figure 6. Applying different types of features to compute the SSM, they concluded that between the same feature type, the pattern similarity was effectively similar. Figure 5: Multi-person pose estimation example using DensePose [63] from the Detectron2 framework [64]. Figure 6: The first row illustrates two actions performed by different subjects and under different poses (jump up in the first two columns and throw in the two rightmost columns). The bottom row provides the corresponding SSM yielding from the corresponding skeleton sequences (adapted from the work in [68]). As a structured data type, some methods employed LSTM networks [29] to model the time-series. Exploring this algorithm, [PERSON] et al. [69] proposed to convert the pose estimated to a tree structure in order to be unfolded as a sequence. Then, each LSTM unit is fed with a skeletal joint, which also takes into account the neighbouring joints and previous frames of the same joint. When analysing the human pose performing some actions in the real world, usually some skeleton joints have more importance than others, paying different attention to different regions of the scene [70]. [PERSON] et al. [71], in the same field of LSTM networks, proposed to model skeleton joints in a selective way as an attention mechanism. Composed of two attention networks, the spatial one assumes the weight of a joint (its importance) as the resulting activations from the network, and the temporal attention one uses the input gate of the LSTM network for learning to control the amount of information (its importance) to be used, in each frame, for the final classification decision. In a similar way, [PERSON] et al. [72] also proposed a recurrent neural network [73] with LSTM, but this time they take into account the translation from global body movement (the whole body dynamics in the scene) to local body posture (skeleton configuration upon the body centre in the first frame). This way, it is possible to adapt its viewpoint in order to be a more suitable observation for orientation alignment normalisation. Even though skeleton pose estimation is a structured data type, several methods approached the problem with 2D ConvNets [74; 75; 76; 77]. [PERSON] et al. [77] proposed a two-stream 2D ConvNet: one to extract features from spatial coordinates of the pose in a 3D manner (position, joints and frames) through a skeleton transformer module, which extracts weighted interpolated joints matrix. On the other stream, they extract the skeleton motion through computed distances between frames. [PERSON] et al. [75] presented a new representation for skeleton data, employing cylindrical coordinates generating a collection of clips which are used as input to a CNN.