## 4 Proposed Methods
### System Architecture
#### 4.2.1 Spatio-Temporal Image Formation (STIF)  
Human action consists of a sequence of frames in which the spatial and temporal information changes occur over time while performing an action. The sequence of frames can be RGB, depth, skeleton, or any other format. In the proposed method, we consider only the 3D skeleton joints information for human action recognition. Skeleton joints information is usually encoded in a 3-dimensional coordinate system. Most of the devices invented for capturing skeleton joints information of the human body consider only twenty joints, specifically Microsoft Kinect captures skeleton data with twenty joint coordinates values along \(X\), \(Y\), and \(Z\) axes in each frame. The positions of the joints change from frame to frame as the action is rendered. We observe the changes of both the spatial and temporal information of the joint positions and represent them into spatio-temporal image. To generate the spatio-temporal image first, we map all the twenty joints in a frame with the same color in the jet color map [17] and then change the colors as the time step passed. Finally, the STIF is created by connecting lines between joints in adjacent frames subsequently. Let us consider two joints, for example, \(A\left(X_{i,j},Y_{i,j}\right)\) and \(B\left(X_{(i+1),j},Y_{(i+1),j}\right)\) along \(XY\) plane at two adjacent frames in an action where \(i\) indicates the index of frame and \(j\) represents index of joints (in our case \(j=1\), \(2\),..., \(20\)). The spatio-temporal image representation of an action along \(XY\) plane (front view) can be obtained by joining line between \(A\) and \(B\) using Equation (1). \[Y-Y_{i,j}=m\big{(}X-X_{i,j}\big{)} \tag{1}\]  
Figure 5: Architecture of the proposed system. where \(m\) is the slope of the line passing through the points \(A\) and \(B\) given by Equation (2). \[m=\frac{Y_{(i+1),j}-Y_{i,j}}{X_{(i+1),j}-X_{ij}} \tag{2}\]  
Similarly, we get the spatio-temporal image representation of an action along \(YZ\) plane (side view) and \(ZX\) plane (top view). To map the spatial and temporal changes of joints information in spatial format such as image format for an action, we use the jet color map. First, we generate the jet colors information with length equal to the number of frames in an action as given in Equation (3). \[Colors=JET(length(action)) \tag{3}\]  
Figure 6 shows the jet colors in 3D space with bar chart that represents the variation of colors starting from blue to red. The generated colors are mapped by putting pixels for each joint as well as for the line passing through current and next frames. Equations (4) and (5) indicate the joints and line mapping with different colors to maintain spatial and temporal changes. \[jointsMapping=~{}putPixel(A,~{}B,~{}Color) \tag{4}\]  
\[lineMapping=~{}drawLine(A,~{}B,~{}Color) \tag{5}\]  
The final spatio-temporal image is obtained by the mapping combination of joints and line as given in Equation (6). \[spatioTemporalImage=concateMapping(jointsMapping,~{}lineMapping) \tag{6}\]  
The detailed of the STIF representation of human skeleton joints is summarized in Algorithm 1. The inputs to the algorithm are a sequence of frames containing twenty joint values along \(X\), \(Y\), and \(Z\) axes and different colors with the same length as the number of frames minus 1. The proposed method first computes the spatio-temporal image between two adjacent frames subsequently and finally combined all of them to generate the ultimate image. Figure 7a-c shows the spatio-temporal image representation of '_Clap_' action in UTD-MHAD dataset along \(XY\), \(YZ\), and \(ZX\) planes (front, side, and top views) in which 1 st frame indicates the mapping between frames at positions 1 and 2 in the action. The mapping of frames from positions 1 to (\(i+1\)) be shown as \(i\)th frame and similarly (\(i+k\))th frame  
Figure 6: Jet colors generation process. is the combined mapping from frames at positions 1 to (\(i+k+1\)).