## 2 Related Work
### Spatio-Temporal Data Representation  
To the best of our knowledge, no studies have used spatiotemporal information based on pure video taken without sensor information in the field of dog behavior recognition and classification. Therefore, in this sub-section, we investigate the spatiotemporal data representation methods and behavior recognition models commonly used in human behavior recognition studies, which have frequently been conducted among behavior recognition studies. Studies that obtain depth information from RGB images without depth sensors such as a Kinect have recently emerged. Furthermore, a joint-based behavior recognition method is gaining popularity, which extracts 2D joint position information based on the depth information obtained from RGB images [17]. Such 2D joint position information is important because it contains not only the joint position information but also the time sequence information. Most joint-based behavior recognition methods use recurrent neural networks (RNNs) that are suitable for time-series data processing, such as an RNN [18], an LSTM [19], and a GRU [20], to extract behaviors. For example, a hierarchical RNN [21] has been proposed, which classifies human motions based on joint position information containing spatiotemporal information; however, RNN-based methods have a limitation of overemphasizing the temporal information during training. As a method of sufficiently learning both spatial and temporal information, a learning method of adding z-values corresponding to the depth information has been proposed. Figure 1a shows a learning method based on a data representation method that flattens the 3D joint position information over time into 1D information [11]. The late fusion method shown in Figure 1b is a learning approach that processes a variety of spatiotemporal information and joint information and fuses the resulting values during the last stage. For example, as shown in Figure 1b [12; 13], tensor fusion networks that fuse the joint positions of each skeleton have been proposed. The authors proposed methods that apply various operations such as a Kronecker product and a Hadamard product, respectively, to implicitly represent spatiotemporal information and in the end fuse each feature value obtained through the operations. As indicated above, the simple concatenation [11] and late fusion [12; 13] methods have the following limitations. In Figure 1b, a new tensor called f1 is generated by fusing data through simple calculations using tensors (x1, y1, and z1) showing spatial information of each joint. However, /1 has a limitation in that it does not sufficiently represent the  
Figure 1: Three typical multi-modality fusion strategies. (**a**) Simple concatenation [11], (**b**) late fusion [12; 13], (**c**) AV-TFN [14], (**d**) MTLN [16], and (**e**) the proposed method (TRT-Net). correlation of spatiotemporal information per frame because the feature vectors of x1, y1, and z1 before fusion are unknown. To resolve this problem, a color-based data representation method that can maintain certain vector information was suggested. For an audio-visual tensor fusion network (AV-TFN) [14], a tensor fusion method was proposed, which represents the position information (x- and y-axes) of each joint of the skeleton and the audio information in the gray- and color scales, respectively, and fuses their colors to classify piano playing postures. The tensor fusion method proposed for AV-TFN is a color-based data representation method, which can be used to infer the pre-fusion vector information even after tensor fusion (Figure 1c). This helps the learning model learn meaningful information, such as the relationship between posture and sound. However, because AV-TFN is optimized for audio-visual data, it is unsuitable for representing the 3D poses suggested in this paper.