Class structure‐aware adversarial loss for cross‐domain human action recognition

IET Image ProcessingVolume 15, Issue 14 p. 3425-3432 ORIGINAL RESEARCH PAPEROpen Access Class structure-aware adversarial loss for cross-domain human action recognition Wanjun Chen, Wanjun Chen orcid.org/0000-0002-4937-1692 Department of Information Science, Xi'an University of Technology, Xi'an, Shaanxi, ChinaSearch for more papers by this authorLong Liu, Corresponding Author Long Liu liulong@xaut.edu.cn School of Automation and Information Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China Correspondence Long Liu, Department of Information Science, Xi'an University of Technology, No. 5 Jinhua South Road, Xi'an, Shaanxi, China. Email: liulong@xaut.edu.cnSearch for more papers by this authorGuangfeng Lin, Guangfeng Lin Department of Information Science, Xi'an University of Technology, Xi'an, Shaanxi, ChinaSearch for more papers by this authorYajun Chen, Yajun Chen Department of Information Science, Xi'an University of Technology, Xi'an, Shaanxi, ChinaSearch for more papers by this authorJing Wang, Jing Wang School of Automation and Information Engineering, Xi'an University of Technology, Xi'an, Shaanxi, ChinaSearch for more papers by this author Wanjun Chen, Wanjun Chen orcid.org/0000-0002-4937-1692 Department of Information Science, Xi'an University of Technology, Xi'an, Shaanxi, ChinaSearch for more papers by this authorLong Liu, Corresponding Author Long Liu liulong@xaut.edu.cn School of Automation and Information Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China Correspondence Long Liu, Department of Information Science, Xi'an University of Technology, No. 5 Jinhua South Road, Xi'an, Shaanxi, China. Email: liulong@xaut.edu.cnSearch for more papers by this authorGuangfeng Lin, Guangfeng Lin Department of Information Science, Xi'an University of Technology, Xi'an, Shaanxi, ChinaSearch for more papers by this authorYajun Chen, Yajun Chen Department of Information Science, Xi'an University of Technology, Xi'an, Shaanxi, ChinaSearch for more papers by this authorJing Wang, Jing Wang School of Automation and Information Engineering, Xi'an University of Technology, Xi'an, Shaanxi, ChinaSearch for more papers by this author First published: 10 July 2021 https://doi.org/10.1049/ipr2.12309AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Cross-domain action recognition is a challenging vision task due to the domain shift and the absence of labeled data in the target domain. With only labelled source domain and unlabelled target domain data during training, some existing methods rely on an adversarial framework to align the features from different domains to a common latent space. However, the existing adversarial-based approaches have a major limitation of only attempting to perform the alignment from a holistic view, ignoring the underlying coherence of class structure across domains. A class structure-aware adversarial loss (CSCAL) is presented to address this issue. The CSCAL incorporates the category information into the adversarial learning branch to capture the fine-grained alignment of each class, effectively avoiding the false mixup of samples from different categories in the embedding space. Experiments on HMDB51, UCF101 and Olympic Sports datasets show significant improvement compared to the baseline. Code and trained model can be found at https://github.com/bregmangh/CSCAL. 1 INTRODUCTION The success of human action recognition in recent years has been mostly driven by a large amount of accessible labelled data [1-3]. Supervised-based action recognition approaches [4, 5] assume that training data (a source domain) and testing data (a target domain) are sampled independently and identically distributed (i.i.d.) from the same distribution. While in practice, the training and testing data are usually collected from the related domain but under different distributions, which is called a domain shift. The performance of models degenerates significantly when applied to a new test domain with different distributions due to domain shift. This greatly limits the application of current action recognition models. To avoid the cost of annotating each new test domain, unsupervised domain adaptation (UDA) [6, 7] tackles domain shift by aligning the feature distribution of the source domain with that of the target domain. The current work in UDA-based cross-domain action recognition with adversarial training shares a common framework [8], as illustrated in Figure 1(a). First, rather than using a raw video, they use a sequence of features extracted by deep networks, such as ResNet [9], where the features are much smaller than the raw video in size, which is beneficial for downstream tasks. Then, they use a generator, usually composed of fully connected layers, to embed common features into the task-specific feature space. Finally, after defining a classifier and a binary domain discriminator, they train the classifier and generator in a supervised manner to minimize the classification errors in the source domain, thereby prompting the generator to extract features with class discrimination. At the same time, the generator and the domain discriminator are optimized so that the feature extracted by the generator cannot be distinguished by the domain discriminator from which domain the feature comes from. When the two players reach a balance, the generator can output domain-invariant features to satisfy the cross-domain action recognition of the model. FIGURE 1Open in figure viewerPowerPoint An illustration of domain alignment with traditional adversarial loss and our CSCAL. Traditional adversarial loss just blindly align the global feature distribution without paying attention to the potential structural context information between classes, leading to mix samples with different class labels together. Our CSCAL could not only align the features globally, but also consider the fine-grained alignment on the category at the same time Although the methods based on the above framework have achieved remarkable performance in cross-domain action recognition, they just blindly align the global feature distribution without paying attention to the potential structural context information between classes. This "one size fits all" and rough alignment does not guarantee a small prediction error in the target domain [10, 11], as shown on the left of Figure 1(b). Inspired by this, we propose a Class StruCture-aware Adversarial Loss function, called class structure-aware adversarial loss (CSCAL), to preserve the class structure while aligning features. CSCAL could not only align the features globally, but also consider the fine-grained alignment of the category at the same time, so as to better eliminate the falsely mixed-up samples in the blind global aligning process, as shown on the right of Figure 1(b). The experimental results verified that the generator trained based on CSCAL can further improve the recognition accuracy of the model in the target domain. 2 RELATED WORK 2.1 Action recognition With the successful application of deep neural networks in the field of images and video [2, 48], action recognition has developed rapidly in recent years. These methods can be summarized into three aspects. One branch is based on the 2D convolutional neural network (CNN), such as the two-stream network [12], temporal segment network [13], temporal relation network [14]. To model temporal relationships by using 2D CNN, Lin et al. [36] introduced a new method, termed temporal shift module (TSM), by extending the shift operation [37] to video understanding. Similar to TSM, Tin [38] introduced a temporal interlacing module to model the temporal convolution. Another branch leverages 3D CNNS to learn spatio-temporal features. For instance, I3D [16] inflates 2D convolutions of two-stream network to 3D. C3D [15] directly extends the 2D convolution operation to 3D. Similarly, ResNet3D [39] directly takes 2D ResNet [9] and replaces all the 2D convolutional filters with 3D kernels. In order to perform long-range temporal modelling, Wang et al. [40] designed a new building block, termed non-local, to capture the long-range dependence in both space and temporal domain. Recently, inspired by the biological Parvo- and magnocellular cells in the primate visual systems, Feichtenhofer et al. [41] proposed SlowFast, an efficient network with a slow pathway and a fast pathway. Since a video is essentially a temporal sequence, the third branch has explored recurrent neural networks (RNNs) for temporal modelling inside a video, particularly the usage of long short-term memory (LSTM). long-term recurrent convolutional network [42] and beyond-short-snippets [43] are the pioneer works that use LSTM for video action recognition in the two-stream network setting. In order to accurately model long-term and complex motions, Lattice-LSTM [44] extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. Recently, [45] proposed VideoLSTM to incorporate correlation-based spatial and motion-based attention mechanisms into the LSTM. Although these works have achieved remarkable performance on action recognition tasks, most of the works still rely on labelled target domain data to fine-tune their models. In contrast, we focus on unsupervised domain adaptation, where we do not have access to target labels during training. 2.2 Domain adaptation Domain adaptation aims to tackle the cross-domain learning problem, and has been widely used for image classification [17-25] and semantic segmentation [47]. These methods can be roughly classified into two categories. The first category is discrepancy-based methods [17-20]. Its main idea is to reduce the discrepancy between two domains such that a model learned from the source domain can be directly applied to the target domain. Under this assumption, variations of maximum mean discrepancy (MMD) [17, 18, 25], correlation alignment (CORAL) [19], and Wasserstein distance [20] have been proposed to align the distributions of the two domains. The secondary category exploits adversarial learning technologies to bring the source and target distributions closer. The representative work is to optimize the feature embeddings from both domains by using the standard minimax objective [21, 22], the inverted label objective [23], or margin disparity discrepancy [24]. 2.3 Cross-domain action recognition Unlike image-related domain adaptation recognition tasks, there only a few works on video-based cross-domain action recognition. The earliest research can be traced back to the work of Bian et al. [26]. They formulate human actions as a topic model, and use a regularization term constructed from topic pairs of both domains to generalize the topic learning in the target domain. However, their method needs partially labelled target data. Waqas et al. [27] promotes the domain generalizability by reducing the disturbances caused by the background. The deep adversarial action adaptation (DAAA) [28] is the first work to exploit adversarial learning to address this task. Very recently, Chen et al. [8] proposed temporal attentive adversarial adaptation network (TA3N) to attentively adapt video segments that contribute the most to the overall domain shift. Unlike TA3N aligning all video clips, Choi et al. [29] only align those discriminative non-background clips, and introduces a self-supervised learning task that performs clip order prediction to learn stronger domain-invariant video representations. Recently, [46] proposed a new generic multimodality domain adaptation framework, called progressive modality cooperation (PMC), to jointly capture both modality-complementary and modality-shared information for better cooperation among multiple modalities. During the training process, PMC progressively selects highly-reliable target samples in an easy-to-hard fashion to reduce the domain discrepancy. However, the above works are to align the features as a whole, that is, to match the marginal distributions of the two domains. It could lead to samples from different categories are mixed together in the common embedding space. Different from these works, our CSCAL, in another aspect, presents a fine-grained feature alignment that conditions the adversarial adaptation models on the high-level semantic structure conveyed in the categories to which the features belong. 3 METHOD We work in an unsupervised video domain adaptation setting. We have access to the labelled source data ( x s , y s ) from source domain { X S , Y S } , and target data x t from unlabelled target domain X T , where X ( • ) is the set of videos extracted from ResNet [9] pre-trained on ImageNet, and Y S is the actions label set. The task is to design methods to adapt a model to work well on the target domain with both labelled source data and unlabelled target data. 3.1 Revisit baseline model Figure 1(a) gives an illustration of the UDA-based cross-domain action recognition framework. TA3N [8] is a typical representative of this type, and is chosen as a state-of-the-art baseline in this work. Under this framework, the model learns domain invariant representation by a generator with a combination of cross-entropy loss for classification in the source domain alone with a domain adaptation adversarial loss, that is binary cross-entropy loss, between data of source and target domains. Formally, denoting the generator, the classifier, and the discriminator by f G , f C , and f D with parameters θ G , θ C , and θ D , respectively. Then the loss is defined as L = L c e ( θ G , θ C ) − λ L a d v ( θ G , θ D ) , (1) L c e ( θ G , θ C ) = − ∑ s ∈ S ∑ k = 1 K y s k log f C ( f G ( x s ) ) , (2) L a d v ( θ G , θ D ) = − ∑ s ∈ S log f D ( f G ( x s ) ) − ∑ t ∈ T log ( 1 − f D ( f G ( x t ) ) ) , (3)where K is the total number of classes and λ is a hyper-parameter for the trade-off between the classification and DA loss. Then, the optimal parameter θ ∗ is given by the following optimization problem ( θ G ∗ , θ C ∗ ) = arg min θ G , θ C L ( θ D ∗ ) , (4) θ D ∗ = arg min θ D L ( θ G ∗ , θ C ∗ ) . (5) 3.2 Class structure-aware adversarial loss The above baseline framework tries to align the marginal distributions of features across domains. This could falsely mix up some samples which are from different categories, and weaken the structural context between classes, leading to degenerated performance for the model on the target domain. In order to perceive the class structures while the discriminator strives to distinguish domains, we present a CSCAL to incorporate the category information into the adversarial branch. CSCAL not only enables the discriminator to align features globally, but also pays attention to the underlying structures between classes. Formally, CSCAL is formulated as L C S C A L ( θ G , θ D ) = − ∑ s ∈ S ∑ k = 1 K y ̂ s k log f D ( c = k | f G ( x s ) ) − ∑ t ∈ T ∑ k = 1 K y ̂ t k log ( 1 − f D ( c = k | f G ( x t ) ) ) . (6)where y ̂ ( • ) k is the kth entry of domain label y ̂ ( • ) ∈ R 2 K for sample x ( • ) . Specifically, y ̂ s is represented as a vector [ p s ; 0 ] , where p s ∈ R K is derived from the output of the classifier for x s , 0 is an all-zero K-dimension vector. A similar expression is for y ̂ t = [ 0 ; p t ] . It is noted that the domain labels for CSCAL are different with baselines, where are 2D vector [ 1 ; 0 ] and [ 0 ; 1 ] . The role of the vector p ( • ) can be viewed as a pseudo-label [30] to construct the latent class structures. Trusting p ( • ) as "ground truth" by encoding it as a hard label can lead to overconfident mistakes and the misleading effect brought by ambiguous or incorrect supervision. A natural idea is to generate a soft pseudo-label that balances a certain amount of confidence to other classes. Then we define the p ( • ) k as follows: p ( • ) k = exp o k T ∑ i = 1 K exp o i T , (7)where o k is kth entry of f C ( f G ( x ( • ) ) ) , T is a temperature to attenuate soft probability distribution over classes. How CSCAL could achieve global alignment and fine-grained alignment. In Equation (6), if y ̂ s k ≡ 1 , ∀ k ∈ K , then Equation (6) will degenerate to Equation (3), the discriminator only discriminates the samples at the domain-level, that is aligning the marginal distributions of the two domains globally. When y ̂ s k is specified as the classifier output, Equation (6) will not only discriminate the samples at the domain-level, but also at the fine-grained category-level, making the alignment of features from different domains more refined and accurate. The proposed CSCAL can be freely embedded into the existing adversarial-based cross-domain action recognition framework. To this end, we only need to replace the original adversarial loss function in (3) with (6) without changing the existing network architecture. 4 EXPERIMENTS 4.1 Datasets We conducted experiments on publicly available benchmarks built on HMDB51 [31], UCF101 [32], and Olympic Sports [33] datasets. In the following, the direction of the arrow denotes the source domain to the target domain. UCF-Olympic. Jamal et al. [28] released this dataset for studying cross domain human action recognition. It has 1145 videos with 6 overlapping action classes subsampled from UCF101 [32] and Olympic Sports [33] datasets. We show results in two settings, one is UCF → Olympic, the other is Olympic →UCF. UCF-HMDB. This dataset was proposed by Chen et al. [8]. It includes 12 overlapping classes and has a larger domain shift. The total number of videos is 3209 and is around three times larger than UCF-Olympic. There are also two settings on it: UCF → HMBD and HMDB → UCF. Some example snapshots of the overlapping human action 'shoot_ball' or 'Basketball' from the two datasets are shown in Figure 2, which depicts a huge domain shift caused by appearance, lighting, camera viewpoint etc. FIGURE 2Open in figure viewerPowerPoint Some example snapshots of the overlapping human action "shoot_ball" or "Basketball" from HMDB51, UCF101 and Olympic datasets, depicting a huge domain shift 4.2 Implementation details Our implementation is based on PyTorch [34] framework. We use the pre-extracted ResNet-101 [9] as inputs to our model. For a fair comparison, we adopt the same architecture design choice of TA3N [8]. In order to obtain a relatively accurate class structure information, we employ the "pre-train then adapt" training procedure. The feature extractor and classifier are first trained on source data for 30 epochs in all settings. Then, we further train it on the labeled source and unlabeled target dataset with the domain discriminator attached. For optimization, we utilize stochastic gradient descent (SGD) with the momentum and weight decay as 0.9 and 1 × 10 − 4 , respectively. The learning rate is initialized to η 0 = 0.03 and adjusted following the schedule [21]: η p = η 0 ( 1 + a p ) b , where p linearly increases from 0 to 1, a = 10 and b = 0.75 . The hyper-parameter T in (7) is set to 6.0, 8.0, 5.0, and 4.0 for different adaptation settings UCF → HMBD, HMDB → UCF, UCF → Olympic and Olympic → UCF by grid search strategy, respectively. And we will give a more detailed analysis on T in the following subsection. 4.3 Experimental results We compared our proposed CSCAL with DANN [22], JAN [25], TA3N [8] and PMC [46]. The 'source only' means the model is trained only with source labelled data (a lower bound for adaptation methods). The 'target only' indicates the model is trained with supervised target data (an upper bound for adaptation methods). The classification accuracies on three datasets are shown in Table 1. We can observe that the proposed CSCAL outperforms all other methods on all datasets. For small-scale benchmark UCF-Olympic, CSCAL could reach, or get very close to the upper bound. In particular, CSCAL achieves an accuracy of 100% on UCF→Olympic. For the larger-scale benchmark UCF-HMDB, CSCAL could also obtain a significant improvement of up to 2.78% compared to the baseline TA3N. The higher accuracy of CSCAL demonstrates the importance of aligning the fine-scale class structure in distribution matching. TABLE 1. Accuracy (%) of CSCAL compared with other methods on three datasets Method UCF→HMDB HMDB→UCF UCF→Olympic Olympic→UCF Target only 82.78 94.92 100.0 96.25 Source only 71.67 73.91 96.30 85.42 DANN [22] 75.28 76.36 91.60 89.96 JAN[25] 74.72 79.69 — — TA3N [8] 78.33 81.79 98.15 92.92 PMC [46] — — 98.40 94.80 This work (CSCAL) 81.11 83.54 100.0 94.58 4.4 Impact of T The effect of the proposed CSCAL is controlled by the hyper-parameter T. In this section, we study the impact of this parameter and see how it affects the performance of the proposed model. Figure 3 shows the accuracy w.r.t. T on UCF→HMDB task. We can see that when T is 1.0, it means no softening effect applied to the pseudo label, the accuracy is 73.89% and has a great potential for improvement. The reason is that there are a large number of false labels with high confidence, the parameters of the model are updated to sub-optimal with the guidance by these incorrect decisions. As the value of T increases, the influence of noisy labels with high confidence is gradually weakened. When T equals to 6.0, the model achieves the optimal performance of 81.11%, which is 7.22% higher than that of T = 1.0 . When T > 6.0 , the performance of the model will decreases significantly, indicating that too much softening effect will lead to a weaker discriminability for pseudo-labels and degenerate the construction of the class structure. FIGURE 3Open in figure viewerPowerPoint The impact of hyper-parameter T on accuracy 4.5 Visualization of distribution We visualize the feature distributions of both domains using t-SNE [35] to investigate how our CSCAL draws the target action distribution closer to the source. From Figure 4(a), we can observe that, with TA3N, most data from both domainsis clustered and aligned with each other, but it has a loose cluster structure. Opposite to this, for CSCAL shown in Figure 4(b), we can see that it has a more compact cluster structure and a larger inter-class margin, which is a prerequisite for CSCAL to be able to better eliminate the falsely mixed-up samples than the blind global aligning process. The distributions of source and target features are nearly indistinguishable, demonstrating the effectiveness of our CSCAL. However, we can also observe that there are three classes that are poorly aligned and mixed up together. The main reason is probably that there are a lot of similar action patterns between these categories. For this reason, the 'fencing' action is confused with 'punch' and 'riding_horse' at 17.0% and 20.0%, respectively, as the confusion matrix shown in Figure 5. FIGURE 4Open in figure viewerPowerPoint The comparison of t-SNE [35] visualization with source and target distributions on UCF→HMDB task FIGURE 5Open in figure viewerPowerPoint Confusion matrix of CSCAL on UCF→HMDB task 5 CONCLUSION This paper proposes CSCAL to address cross-domain human action recognition. We designed a class structure-aware adversarial loss, which guides the model to pay more attention to fine-scale class structure information during the process of distribution matching of action features. This mechanism can effectively reduce the falsely mixed-up data which are from different classes. Extensive experiments on three datasets verify that our proposed CSCAL can significantly improve the performance. In future work, we will investigate the application of CSCAL in other related computer vision areas, such as image recognition, action segmentation and person re-identification with the case of domain shift. ACKNOWLEDGEMENTS This work was supported by the National Natural Science Foundation of China under Grants No. 61673318, No. 61671374, No. 62076200, the Key Industrial Chain Project of Shaanxi Province under Grant No. 2020ZDLGY04-04, the Science and Technology Planning Project of Xi'an City under Grant No. GXYD14.1, Shaanxi Provincial Key Research and Development Program under Grant No. 2020SF-359, Natural Science Basic Research Program of Shaanxi Province under Grant No. 2020JM-468, and the PhD research startup foundation of Xi'an University of Technology under Grant No. 108-451118004. REFERENCES 1Liu, J., et al.: Ntu Rgb+D 120: A large-scale benchmark for 3D human activity understanding. IEEE Trans. Pattern Anal. Mach. Intell. 42(10), 2684– 2701 (2019) CrossrefPubMedWeb of Science®Google Scholar 2Monfort, M., et al.: Moments in time dataset: One million videos for event understanding. IEEE Trans. Pattern Anal. Mach. Intell. 42(2), 502– 508 (2019) CrossrefPubMedWeb of Science®Google Scholar 3Kay, W., et al.: The kinetics human action video dataset. arXiv:1705.06950 (2017). Google Scholar 4Liu, B., et al.: Rgb-D sensing based human action and interaction analysis: A survey. Pattern Recognit. 94, 1– 12 (2019) CrossrefCASWeb of Science®Google Scholar 5Wang, L., et al.: A comparative review of recent kinect-based action recognition algorithms. IEEE Trans. Image Process. 29, 15– 28 (2020) CrossrefPubMedWeb of Science®Google Scholar 6Wilson, G., Cook, D.J.: A survey of unsupervised deep domain adaptation. ACM Trans. Intell. Syst. Technol. 11(5), 1– 46 (2020) CrossrefCASPubMedWeb of Science®Google Scholar 7Kouw, W.M., Loog, M.: A review of domain adaptation without target labels. IEEE Trans. Pattern Anal. Mach. Intell. 43(3), 766– 785 (2019) CrossrefWeb of Science®Google Scholar 8Chen, M-H., et al.: Temporal attentive alignment for large-scale video domain adaptation. In: International Conference on Computer Vision, Seoul (2019) Google Scholar 9He, K., et al.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas (2016) Google Scholar 10Kumar, A., et al.: Co-regularized alignment for unsupervised domain adaptation. Adv. Neural Inf. Process. Syst. 31, 9345– 9356 (2018) Google Scholar 11Chen, C., et al.: Progressive feature alignment for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition, Long Beach (2019) Google Scholar 12Feichtenhofer, C., et al.: Convolutional two-stream network fusion for video action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City (2018) Google Scholar 13Wang, L., et al.: Temporal segment networks: Towards good practices for deep action recognition. In: European Conference on Computer Vision, Amsterdam (2016) Google Scholar 14Zhou, B., et al.: Temporal relational reasoning in videos. In: European Conference on Computer Vision, Munich (2018) Google Scholar 15Tran, D., et al.: Learning spatiotemporal features with 3D convolutional networks. In: IEEE International Conference on Computer Vision, Santiago (2015) Google Scholar 16Carreira, J., Zisserman, A.: Quo Vadis, action recognition? a new model and the kinetics dataset. In: IEEE Conference on Computer Vision and Pattern Recognition, Honolulu (2017) Google Scholar 17Borgwardt, K.M., et al.: Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14), e49– e57 (2006) CrossrefCASPubMedWeb of Science®Google Scholar 18Yan, H., et al.: Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition, Honolulu (2017) Google Scholar 19Sun, B., Feng, J.:Saenko, K.: Return of frustratingly easy domain adaptation. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence Phoenix, Arizona (2016) Google Scholar 20Shen, J., et al.: Wasserstein distance guided representation learning for domain adaptation. arXiv:1707.01217 (2017) Google Scholar 21Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning. Lille, France (2015) Google Scholar 22Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17(1), 2096– 2030 (2016) Google Scholar 23Tzeng, E., et al.: Adversarial discriminative domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition, Honolulu (2017) Google Scholar 24Zhang, Y., et al.: Bridging theory and algorithm for domain adaptation. In: Proceedings of the 36th International Conference on Machine Learning. Virtual Only. (2019) Google Scholar 25Long, M., et al.: Deep transfer learning with joint adaptation networks. In: International Conference on Machine Learning, Sydney (2017) Google Scholar 26Bian, W., Tao, D., Rui, Y.: Cross-domain human action recognition. IEEE Trans. Syst., Man, Cybern., Part B (Cybern.) 42(2), 298– 307 (2012) CrossrefPubMedGoogle Scholar 27Sultani, W., Saleemi, I.: Human action recognition across datasets by foreground-weighted histogram decomposition. In: IEEE Conference on Computer Vision and Pattern Recognition, Columbus (2014) Google Scholar 28Jamal, A., et al.: Deep domain adaptation in action space. In: Proceedings of British Machine Vision Conference(BMVC), Newcastle (2018) Google Scholar 29Choi, J., et al.: Shuffle and attend: Video domain adaptation. In: European Conference on Computer Vision, Glasgow (2020) Google Scholar 30Zou, Y., et al.: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: European Conference on Computer Vision, Munich (2018) Google Scholar 31Kuehne, H., et al.: HMDB: A large video database for human motion recognition. In: International Conference on Computer Vision, Barcelona (2011) Google Scholar 32Soomro, K., et al.: UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv:1212.0402 (2012) Google Scholar 33Niebles, J.C., et al.: Modeling temporal structure of decomposable motion segments for activity classification. In: European Conference on Computer Vision, Heraklion (2010) Google Scholar 34Paszke, A., et al.: PyTorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32, 8024– 8035 (2019) Google Scholar 35Maaten, L.v.d., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579– 2605 (2008) Web of Science®Google Scholar 36Lin, J., et al.: TSM: Temporal shift module for efficient video understanding. In: International Conference on Computer Vision, Seoul (2019) Google Scholar 37Wu, B., et al.: Shift: A zero flop, zero parameter alternative to spatial convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, (2018) Google Scholar 38Shao, H., et al.: Temporal interlacing network. AAAI Conference on Artif. Intell. 34, 11966– 11973, (2020) CrossrefGoogle Scholar 39Hara, K., et al.: Can spatiotemporal 3D cnns retrace the history of 2D cnns and imagenet? In: IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City (2018) Google Scholar 40Wang, X., et al.: Non-local neural networks. IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City (2018) Google Scholar 41Feichtenhofer, C., et al.: Slowfast networks for video recognition. In: IEEE/CVFInternational Conference on Computer Vision, Seoul (2019) Google Scholar 42Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. In: IEEE Conference On Computer Vision And Pattern Recognition, Boston (2015) Google Scholar 43Yue-Hei Ng, J., et al.: Beyond short snippets: Deep networks for video classification. In: IEEE Conference on Computer Vision and Pattern Recognition, Boston (2015) Google Scholar 44Sun, L., et al.: Lattice long short-term memory for human action recognition. In: International Conference on Computer Vision, Venice (2017) Google Scholar 45Li, Z., et al.: Videolstm convolves, attends and flows for action recognition. Comput. Vision Image Understanding. 166, 41– 50 (2018) CrossrefWeb of Science®Google Scholar 46Zhang, W., et al.: Progressive modality cooperation for multi-modality domain adaptation. IEEE Trans. Image Process. 30, 3293– 3306 (2021) CrossrefPubMedWeb of Science®Google Scholar 47Wang, Q., Gao, J., Li, X.: Weakly supervised adversarial domain adaptation for semantic segmentation in urban scenes. IEEE Trans. Image Process. 28(9), 4376– 4386 (2019) CrossrefPubMedWeb of Science®Google Scholar 48Li, Q., et al.: Exploring the relationship between 2D/3D convolution for hyperspectral image super-resolution. IEEE Trans. Geosci. Remote Sens. 1[SB]:Please provide the volume number in reference 48.—11 (2020) Web of Science®Google Scholar Volume15, Issue14Special Issue: Crowd understanding and analysisDecember 2021Pages 3425-3432 FiguresReferencesRelatedInformation