Recovering Complete Actions for Cross-dataset Skeleton Action Recognition

Authors: hanchao  liu
yujiang  li
tai-jiang  mu
shi-min  hu
  bnrist
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 08:40:53

Abstract: Despite huge progress in skeleton-based action recognition, its generalizability to different domains remains a challenging issue. 
In this paper, to solve the skeleton action generalization problem, we present a recover-and-resample augmentation framework based on a novel complete action prior. We observe that human daily actions are confronted with temporal mismatch across different datasets, as they are usually partial observations of their complete action sequences. By recovering complete actions and resampling from these full sequences, we can generate strong augmentations for unseen domains. At the same time, we discover the nature of general action completeness within large datasets, indicated by the per-frame diversity over time. This allows us to exploit two assets of transferable knowledge that can be shared across action samples and be helpful for action completion: boundary poses for determining the action start, and linear temporal transforms for capturing global action patterns. Therefore, we formulate the recovering stage as a two-step stochastic action completion with boundary pose-conditioned extrapolation followed by smooth linear transforms. Both the boundary poses and linear transforms can be efficiently learned from the whole dataset via clustering. We validate our approach on a cross-dataset setting with three skeleton action datasets, outperforming other domain generalization approaches by a considerable margin.

Main paper: 
 introduction: 
 Skeleton-based action recognition has recently achieved great success [57, 42, 6, 29] . The skeletonbased action representation has the advantage of removed background changes and camera positions, making it more robust compared to RGB representation. However, the generalizability to unseen domains under such a representation is still affected by the inherent spatiotemporal difference of 3D coordinates of a same action across domains, which yet largely remains an under-explored issue. In this paper, we study the single domain generalization problem [55] for skeleton-based action recognition, in which we do not have knowledge about the target domain.Essentially, both cross-subject and cross-view settings [41] fall under the category of cross-domain settings, and they can be well addressed by designing more powerful backbones and applying geometric transformations, achieving high accuracy in the test set [6, 29] . However, we find that in the cross-dataset setting where source and target data come from different datasets, the performance on accuracy degrades a lot (around or more than 20% in some cases [46] ) and cannot be well remedied by the above approaches. This indicates drastic domain gap in the inherent feature of human actions across datasets, posing great challenges for real-life applications [40, 18] and calling for research on domain generalization techniques for skeleton-based representation, which is the focus of this paper. Investigating action samples across multiple datasets, our observation is that a notable source of domain gap comes from the temporal mismatch of an action across different datasets (Fig. 1(a) ), which is usually caused by different definition or cropping criterion of human motions. Existing methods cannot perfectly handle this problem. Warping-based temporal alignment approaches, while being popular for few-shot recognition tasks [3, 53, 33, 45] , are inefficient and not optimal when given sufficient and diverse training data. Besides general domain generalization approaches [35, 51] , self-supervised skeleton representation learning methods [23, 47, 46, 59] , while showing good transferability of the learned feature, are still difficult to handle data with large gaps as the generalizability largely comes from seeing handcrafted augmentations in either contrastive learning [62] or auxiliary tasks [46] . Different from the above approaches, we aim to directly hallucinate strong augmentations for unseen domains by exploring some sort of human action priors.As in Fig. 1 (b), we train a skeleton auto-encoder and examine the per-frame latent feature diversity of the whole dataset (measured by standard deviation) for several skeleton action datasets, i.e., NTU [41] , PKU [25] , ETRI [15] (Details in Appendix A5). We observe that human action sequences start with relatively low feature diversity, which is actually a form that humans perform generally complete actions within large datasets, from rest poses that are less diverse (e.g. stand, sit) to rich-semantic poses that are more diverse. We summarize this pattern as a novel temporal prior named complete action prior. Although this prior can be detected by statistics in a general sense, in terms of individual samples, some exhibit strong action completeness while some are segments of their complete actions. This motivates us to learn an action completion function from the whole training data which can transform incomplete actions into complete ones. In this way, we can recover the complete sequence of an action from its partial observation, and resample from it to hallucinate data in unseen domains.Going deeper into such a statistical pattern, we can further mine some class-agnostic knowledge from the whole dataset that can be used for our action completion. First, the low diversity at beginning frames implies the existence of a set of representative boundary poses, which can be used for determining the start of an action. Second, the general completeness implies the existence of long and nearly complete actions within a large dataset. By studying the relationships between their raw and trimmed pairs, we can learn temporal patterns inherited in human actions.Based on the above observations, we propose a novel recover-and-resample augmentation framework for single domain generalization on skeleton action recognition. As in Fig. 1 (c), we recover complete actions from training samples and resample from them to further augment the training set. Especially, in the recovering stage, we adopt a two-step stochastic action completion, which first extrapolates the raw sequence conditioning on the boundary pose and then applies temporal transforms. A set of boundary poses are learned from the first frames of the training data. Altogether, a set of smooth linear transforms (linearity means re-organizing existing frames) are learned from reconstructing full sequence from its trimmed segment via a closed-form solution of context-aware frame similarity aggregation. We find such a simplified form of transform suitable and expressive enough for modeling common structural temporal patterns, e.g., shifting, scaling, symmetry, etc. The learning for both boundary poses and linear transforms can be achieved via clustering, for example, k-means, which makes our whole framework light and efficient. Also, we find a random cropping suffices for the resampling stage.We simplify the problem in cross-domain settings by studying well-defined daily actions in indoor settings. Specifically, we construct a new evaluation setting jointly with three large-scale datasets including PKU-MMD [25] , NTU-RGBD [41] and ETRI-Activity3D [15] . Our proposed augmentation method well solves the temporal misalignment issue. We improve the average accuracy on unseen datasets by 5%, outperforming other baseline methods by a large margin.In summary, the contributions of this paper are as follows:• We discover the complete action prior within large datasets by its statistical pattern, whose effect has not been well studied before. Building on such a prior, we present a novel recover-andresample augmentation framework for domain generalization on skeleton-based action recognition. • We propose an effective clustering-based approach to recover complete actions, which is achieved by boundary pose conditioned extrapolation and context-aware smooth linear transforms. • We demonstrate the superiority of our method over a wide range of methods by a large margin and we conduct extensive experiments to show the effectiveness of each proposed module.2 Related Work 
skeleton-based action recognition: 
 As for skeleton action recognition, the graph neural network [57, 42, 6, 29] has become a prevailing model due to its effectiveness to model spatiotemporal relations between joints. We use AGCN (Adaptive GCN) [42] as our backbone as it can model flexible graph layouts. Recently, there is a focus on studying skeleton action recognition with less labeled data. For self-supervised skeleton representation learning, a number of pretext tasks are proposed based on invariant augmentations [23, 47, 11, 59, 56] . These approaches improve the generalizability across datasets but are still bounded by the specific augmentation designs. For few-shot skeleton action recognition, metric learning approaches with different types of dynamic time warping [3, 53, 33, 45] are widely adopted. However, they are not optimal for domain generalization given large amount of training data. Our work especially explores the generalization problem for this field when data is not available in the target domain.General domain generalization approaches mainly rely on learning adversarial augmentation [51, 22, 55] and self-supervised auxiliary tasks [4] . They are not optimal for skeleton-based action recognition tasks since they do not make full use of the skeleton representation. The recent ST-Cubism [46] is most related to ours, which adapts learning jigsaw puzzles [46] to skeleton sequences to achieve domain generalization. While there are many augmentation methods intended for non-semantic related tasks like pose estimation [16, 2, 10, 9, 14] and human motion prediction [31] , skeleton augmentation for recognition tasks is essentially challenging as it is fragile to spoil the semantics of skeletons formed by low-dimensional joints. SFN [34] combines Mixup [61] with VAE for augmentation, but it still cannot deal with cross-domain settings. ModSelect [32] proposes a selection mechanism for multi-modal input for cross-domain action generalization. In this work we focus on the learnable temporal action augmentation for cross-domain settings.As human motions are constrained in the spatiotemporal space, many human motion priors are proposed, such as manifolds of valid human poses [19, 7, 48] and motion [63, 27, 17] , and other properties such as periodical human motion pattern [43, 12, 44] , body part interchangeability across samples [26, 24] and alignment with other modalities [65, 1] . ACTOR [38] learns a CVAE and samples data points conditioning on labels as an augmentation method for recognition tasks. However, it is only useful when data is scarce. Rate-invariant prior [50] is related to ours, but changing rate itself is not enough for cross-domain settings. In this paper, we explore the action completeness prior for better performance in cross-domain settings. Although the notion of action completeness [30, 64] appears in the temporal action localization (TAL) task [60] , we are the first to incorporate it for the skeleton-based domain generalization task. Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Appendix A6. Discussion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setting: 
 In the skeleton-based action recognition, suppose we have source data from one single domain S = {(x S , y S )}. Due to the unfixed length of input action sequences, the skeleton sequences x S are all uniformly resized to x S ∈ R T ×J×3 with sequence length T and joint number J [6, 15] . Our goal is to train a model G using S and test on other unseen target domains {(x T 1 , y T 1 ), (x T 2 , y T 2 ), • • • }. In particular y S and y T share the same action categories.Motivated by the observations of temporal mismatch (Fig. 1(a) ) and action completeness within a dataset (Fig. 1(b )), we first introduce a generalized form of temporal augmentation, which we name recover-and-resample augmentation. Given a training sample x, we first recover its complete action with a transform W recover and then sample a segment of it using W resample as an augmentation x ′ :EQUATIONIdeally W full can generate all kinds of temporal segments that may appear in the target domains. In contrastive self-supervised learning, [23] uses temporal shift as a recovering process, while [47] only uses resampling for constructing positive samples. We set the resample transform W resample as a common random sampling [47] , and focus more on the recovering stage W recover that can construct a complete action for the input motion x. Our contribution is that by exploiting the knowledge from complete action prior, we further decompose W recover into a linear transform and a nonlinear transform, i.e. W recover (x) = F L (F N (x)). The nonlinear transform F N extrapolates the motion to generate new poses which do not exist in the original motion. The linear transform F L reorganizes the motion sequence using existing frames. In this way we boost the expressive power of W recover for generating complete actions. Fig. 2 gives an illustration on how we generate augmentations for a partially observed action phone calling. Details for F L and F N are introduced in Sec. 3.3 and 3.4.Boundary pose clustering. When recovering a complete action, we first determine its boundary pose. Intuitively, boundary poses for common daily activities are more likely to be background poses (or rest poses). They do not convey strong action semantics and can be distinguished from meaningful foreground poses. On the other hand, the statistical finding that the initial poses of skeletal action sequences have low feature diversity (shown in Fig. 1 (b )) also validates our assumption: the boundary poses are to some extent constrained. Therefore, we propose to cluster the first frames x 0 of all the training samples in S to get a set of representative background (boundary) poses, which we denote as {p i }. During clustering, all the first frames {x 0 |x 0 ∈ R J×3 } are flattened to 1d vectors and L 2 norm is used as the distance measure.Conditional generation. We use nearest neighbour search to assign the boundary pose p ′ for x with first frame x 0 :EQUATIONNow we can extrapolate the original first frame x 0 to the new boundary pose p ′ . This is basically an infilling process conditioned on p ′ , and we control the length of extrapolation with parameter t p . Supposing the sequence length is T , we set the first frame as p ′ , and squeeze the original motion x to the segment from frame t p to the end frame T . We then infill the timestamps from the first frame to frame t p with new motion. Considering a stochastic process, we sample t p from a Beta distribution β(α, α)T /2 with high probability on the first frame and frame T /2. This means the conditional generation process is more likely to retain the original motion or infill a segment with length T /2.Motion infiller. The simplest form of the motion infiller F is linear interpolation since the skeleton already has good joint correspondence. We also propose an alternative to train an infiller with a neural network. We mask out some consecutive frames to get masked and full sequence pairs and train an action completion network. We then use this motion infiller to perform extrapolation. The comparison will be shown in the later experiments. Actually we find the parameter-free linear interpolation works well enough for generating reasonable motions for the recognition task.So formally our nonlinear transform can be represented as F N (x) = F(x, p ′ , t p ). We first find the boundary pose p ′ , sample t p to determine the sequence length to infill, and finally apply the motion infiller F.Note that the above nonlinear transform is still unable to capture global and structural patterns inherited in the human actions. Consequently, we further propose linear transforms F L to reorganize frames for more powerful transformations. We have seen great progress in self-supervised learning with very simple temporal linear transforms like crop & pad [23] . However they are manually designed and are not learned from data so they are not flexible. Here we propose to construct partial and full skeleton sequence pairs to learn a set of linear transforms in the form of F L (x) = W x, W ∈ R T ×T . Here x ∈ R T ×J×3 is a skeleton sequence with length T . Context-aware smooth linear transforms. Given partial sequence u and full sequence v (see Fig. 2 ) which are then both resized to length T , we hope to find a linear transform that minimizes the L 2 norm |W u -v|, where W has only one nonzero element for each row. The straightforward solution is to do it frame-wise, i.e. for each frame v i in v we find the index of its closest frame u j in u. However, such an operation cannot ensure that W is smooth and consistent with good transferability. Inspired by the context-aware alignment [20] , we propose context-aware smooth linear transform that aggregates information from semantically adjacent frames.Specifically, we first find the context-aware frame-wise similarity matrix of u and v. For example, the similarity score s ij between i-th frame of v and j-th frame of u can be obtained as following:EQUATIONwhere | • | denotes L 2 distance between two flattened skeletons (root translation already removed). Larger λ T means stronger context-aware smoothing. We then calculate the index by:EQUATIONThe transform matrix W is then obtained by setting W [i, round(k i )] = 1. This is to say, when deciding the most similar frame for v i , all frames in u are considered by their similarity weights. In this way the transform W becomes smooth and more resilient to noise frames.Clustering linear transforms. Again we utilize clustering algorithm to extract all the possible transform patterns that map partial sequences to full sequences. Since the original number of transforms for W obtained from randomly sampled trimmed and full pairs may be too large, with clustering we avoid the inefficient sampling of W . Meanwhile, during the clustering, some important transform patterns, e.g. mirroring, can stand out as they may originally only account for a small percentage in the whole pool of transforms.In details, we first randomly generate training pairs, i.e. partial motion sequences and full motion sequences {(u k , v k )} from S. We then use Eq. ( 3) to get similarity matrix M (k) for motion u k and v k . Instead of clustering directly on W , we perform clustering on all the similarity matrices {M (k) } to get representative cluster centers {M i }. We normalize each row for M i , and then calculate the index according to Eq. ( 4) and obtain the corresponding transform matrix W i . Again, during clustering, each M (k) ∈ R T ×T is flattened to 1d vector and we use L 2 norm as the distance measure.Once we obtain a set of {W i }, we can randomly sample W from {W i } and apply it to x. So the linear transform can be formally represented asF L (x) = W x, W ∼ {W i |W i ∈ R T ×T }.We follow the standard pipeline for training a recognition model. For input x with its action label y, we generate augmentations x ′ (refer to Algorithm 1 in Appendix A1), and train the model with the loss function:EQUATION)where L is the standard cross-entropy loss. In practice, in a batch, we augment the training samples with a ratio of m aug , and the rest of the samples remains unchanged.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: Code link is available on the first page.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
datasets and settings: 
 Datasets. We use four large-scale datasets, i.e, NTU60-RGBD [41] , PKU-MMD [25] , ETRI-Activity3D [15] and Kinetics [5] . The first three datasets are captured in indoor laboratory and household environments while the last one is collected from online videos.A new cross-dataset setting. We construct a new multi-domain cross-dataset setting, mainly comprising actions in indoor environment. We gather shared 18 actions (see Appendix A2) using the first three datasets described above and each dataset is treated as one domain. We train on one domain and test on the rest two domains. Specifically, we define four domain transfer sub-settings, i.e. N -→ E, N -→ P , E A -→ N , E A -→ P (each dataset is denoted with its first letter). Especially we use the adult split E A for training because in this way we keep the number of training samples for NTU and ETRI relatively the same. We do not include P for training since P majorly contains long and complete actions due to its annotation and it is relatively easy to perform sampling directly. We also reserve a subset P with mutually exclusive labels with P as a prior dataset to evaluate in an "Oracle" case when we have certain knowledge about how complete actions look like.Evaluation Metric. We use the average accuracy of the above four cross-dataset sub-settings to measure the performance of single domain generalization. Especially we use balanced accuracy [32, 39] to eliminate the influence of class bias. We train the model with ten different random seeds and report the mean accuracy.For fair comparison with ST-Cubism [46] , we also include P51 ← → N51 with 51 action classes paired between NTU and PKU dataset, and N12-→K12 with 12 action classes paired between NTU and Kinetics. We also follow their evaluation protocols when reporting results.Data preparation. We resize motion sequences from different domains to a fixed length T = 64. Following [42] , we remove camera rotation and trajectory movement in the pre-processing stage and we further apply random 3D rotation as spatial augmentation for all methods. [35] 67.3 ST-Cubism [46] 70.5 Skeleton-MAE [28] 70.3 HICLR [62] 66.5 CropPad [23] 69.0 CropResize [47] 67.0 Ours 72.2 Backbone models. For our backbone model, we adopt a slightly modified version [26] of Adaptive GCN (AGCN) [42] with fewer blocks and only use the joint stream for speed and simplicity. We reduce the number of blocks from 10 to 4. The output channels for each block are 64, 64, 128 and 256. The kernel strides for each block are 1, 1, 2 and 2. We find that such a design improves the base generalizability partially because it better fits the reduced length of motion sequences and avoids overfitting. Besides our AGCN backbone, we also test on ST-GCN [57] and CTR-GCN [6] , two representative GCN backbones. ST-GCN is a simple GCN that aggregates spatiotemporal information without elaborate design. CTR-GCN adopts a multi-level feature design and has much more parameters than our AGCN. HCN is a two-stream convolutional network used by [46] . We use it for fair comparison in P51 ← → N51 and N12-→K12 settings.Training details. We set training hyper-parameters the same as [42] . Furthermore, we set λ T = 0.1, the number of linear transforms N tr = 20, the number of background poses N bkg = 10, and m aug = 0.75. We set α = 0.1 for sampling t p . We fix the resampling method by randomly sampling a segment with length ratio r between 0.7 and 1.0. We use k-means for clustering boundary poses and linear transforms. Parameter settings for P51← →N51 and N12-→K12 are provided in Appendix A3.Baselines. Besides ST-Cubism [46] which is by far the only method that reports result for crossdataset settings, we compare with a wide range of baseline methods.(1) General domain generalization approaches: latent feature alignment CCSA [35] and adversarial augmentation ADA [51, 52] .(2) Self-supervised learning approaches: ST-Cubism [46] that solves jigsaw puzzles in spatial (Spa) and temporal (Tem) dimensions, Skeleton-MAE (partially borrowed from [28] ) that The result is shown in Table 1 . Our method, by explicitly exploring the action prior, improves ERM by 5.7% and outperforms the second best (ST-Cubism) by 3.0% in terms of average accuracy. All the self-supervised and augmentation-based methods improve the domain generalizability while the former perform better than the latter. However, neither of them can significantly improve the generalizability to deal with unseen data with large gaps. By learning transforms from training data, we are more flexible than handcrafted augmentations and perform especially better in recognizing long sequences (P domain). Test-time aggregation only improves on one specific setting and warpingbased matching with kNN is not as competitive as directly learning a deep classifier when given sufficient and diverse training data (See more discussion in Appendix A6). While sharing the same spirit that temporal alignment is important, we provide an alternative solution which combines raw training data and augmented "aligned" data from recovering and resampling complete actions.We further transfer boundary poses and linear transforms learned from the 18-class subset to N51← →P51 settings. In Table 3 , we compare to ST-Cubism [46] following their protocol using HCN [21] backbone. For N51-→P51, as in Table 2 and 3, our method outperforms ST-Cubism, showing the effectiveness when there are more action classes. For P51-→N51, ours is better than the temporal part of ST-Cubism but slightly worse than the whole ST-Cubism, which can be explained that P51 generally already consists of complete actions. For the challenging 2D setting N12-→K12, we use linear transform module only and also obtain slightly better performance than ST-Cubism.Effect of each component. We examine the effect of the non-linear transform F N , i.e. the boundary conditioned extrapolation, and the linear transform F L . As shown in Table 4 , both F N and F L improve the ERM baseline in terms of the average accuracy, showing they are crucial to construct complete actions and validating our two-step design. As a comparison, we see that recovering with handcrafted CropPad only leads to limited improvement (1.3%). Moreover, in cases when the action completeness is less significant in the target domain (e.g. N-→E), F L itself may be sufficient. Since F L and F N learned from different datasets are transferable, we can also use P as a high quality prior dataset to replace the original training data (denoted as Self in Table 4 ) to learn F L and F N .In this way we can investigate the performance upper bound of our method. By utilizing P we can further improve the average accuracy from 60.1 to 60.5. This shows that it would be more beneficial to directly learn from nearly complete actions since it is our aim to reconstruct complete actions from partial observations. On the other hand, the marginal improvement of 0.4% also indicates that even from the training data we already learn transforms and boundary poses in a quite decent manner.Parameter analysis. We examine three important parameters: number of clustered boundary poses N bkg , number of clustered linear transforms N tr , and context-aware coefficient λ T . The results are in Table 5 and 6 . As for N tr , a too small number of clusters will hurt the overall performance by around 1.2%, which shows that too few linear transforms cannot fully represent the various global motion patterns. As for N bkg , we empirically find that the number is not a very important issue probably because the variety of boundary poses does not significantly affect the semantics. However, directly extending with the raw first frame (shown as "None") will hurt the performance by 1.3%, indicating the importance of boundary pose clustering. We also investigate the parameter sensitivity of λ T . We find that λ T = 0.1 works well and generally it is not very sensitive (see result in Appendix A4).What is learned for augmentation? In order to better understand what is learned in F L and F N , we visualize the clustering results. Fig. 4 visualizes the clustered background poses. We find that the background poses mainly include common rest poses such as standing and sitting with different rotations. These poses do not always appear in all action sequences, so clustering from the whole dataset is reasonable. Fig. 3 visualizes the clustered linear transform matrices. We find that F L mainly learns temporal shifting and reflection operation from the training data, which indicates the presence of approximate symmetric pattern of many complete actions (see matrices in ">" and "<" shapes). We can also interestingly conclude that the CropPad transform [23] actually fits our complete action prior well as it appears in the set of F L transforms (such as the second matrix in Fig. 3 ). Ours is more flexible as we can learn a set of possible transforms. Fig. 5 provides examples of recovered complete actions from partial observations. In Fig. 6 , we show that with our augmentation we learn a more discriminative feature embedding compared to some other baselines.Comparison with other learning variants. Note that it is also possible to perform infilling and extrapolation for skeleton sequences by training an action completion network. In Table 7 we investigate the performance when learning by neural networks. We compare to the design of learning to extrapolate without conditioning on boundary poses, as well as the design of learning to infill instead of performing linear interpolation. For both variants we adopt the network design in [28] and more details are provided in Appendix A3. We find that in terms of average accuracy, both learning to extrapolate (F NN,extrap ) and learning to infill (F NN,infill ) are not as effective as our proposed learning-free interpolation, which shows that (1) assigning boundary pose via clustering is necessary as the network itself is difficult to learn automatically; (2) infilling by linear extrapolation is actually a simple but effective design as the infilling quality is good and robust for domain generalization.Per-class results. Table 8 shows the performance improvement for selected action categories. Our method can significantly improve the accuracy on several hand-related actions such as phone calling, hand waving and clapping. In the case of temporal mismatch, these actions are easily confused with other similar actions if only trained from partial observations. If an action can be well recognized by partial segments, the improvement may be small (e.g. taking off clothes).Generalizability for different backbones. We further apply our method on ST-GCN [57] and CTR-GCN [6] to see whether various backbones can benefit from the observed complete action prior. Note that CTR-GCN extracts temporal feature in a multi-scale fashion, which is more advanced than AGCN [42] . We observe consistent improvement over ERM, from 50.4 to 55.4 for ST-GCN and from 56.3 to 64.3 for CTR-GCN, as is shown in Table 9 . This shows that the domain gap caused by temporal mismatch is difficult to be mitigated by only designing network architecture itself. 
conclusion: 
 In this paper we present a recover-and-resample augmentation approach to deal with the single domain generalization problem for skeleton action recognition. By exploring the complete action prior, we recover complete actions with learned boundary poses and global linear transforms via clustering. Experiments on a cross-domain setting with three datasets validate our framework. In the future, we plan to investigate more effective resampling approaches, e.g. positional encoding, to further incorporate temporal resampling information into our whole framework.6 Appendix 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Section 4 and Appendix A.3 for experiment settings and implementation details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: See Appendix A4. Additional Experimental Results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Appendix A3. Additional Implementation Details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This research conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix A6. Discussion.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Not applicable.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See Appendix A2 and A3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The algorithmic description of our recover-and-resample augmentation framework is provided in Algorithm 1. Assign p ′ as the boundary pose for x from {pi} using Eq. ( 2). Sample tp ∼ β(α, α)T /2 and perform extrapolation x = F(x, p ′ , tp). Sample W from {Wi}, apply x = W x.x ′ = Wresample(x).Cross-dataset settings. Here we provide more details for our cross-dataset settings. The shared 18 action classes for NTU60-RGBD [41] , PKU-MMD [25] and ETRI-Activity3D [15] are listed in Table 10 . We evaluate on four cross-dataset sub-settings, i.e. N -→ E, N -→ P , E A -→ N and E A -→ P . We use cross-subject protocol for dividing training and test splits, and the number of training and test samples for each domain is shown in Table 11 . For each domain, we use training set for model training and test set for evaluation. For term of use, NTU60-RGBD [41] is free for research and non-commercial use. We submitted a license agreement to ETRI-Activity3D [15] website for downloading the dataset. License for PKU-MMD [25] is not stated on its official homepage. Evaluation details. For skeleton action recognition, most works [42, 6] report best results on the test set since there is no official validation set. We also follow this evaluation protocol for our domain generalization task. We train and evaluate for each sub-setting separately, and use average accuracy for evaluating the overall performance. Moreover, we run each experiment ten times with different random seeds and report mean accuracy to reduce performance oscillation. As the training sets for N and E A (the adult split) are approximately of the same sizes, we choose to use E A for training, which offers a fair basis and gives us a better understanding of the task difficulty for different sub-settings.Linear transform clustering. In order to cluster a set of linear transforms, we construct a set of training pairs, i.e. partial action sequences and full action sequences {(u k , v k )} from training data S. Although it is possible to randomly sample starting and end points for a sequence v to generate u, we propose to sample from some fixed interval timestamps for simplicity and regularity. Those timestamps are 0.0, 0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0. For each training sample v, we sample u according to the below pairs of starting and end points.[(0.0, 1.0), (0.0, 0.5), (0.5, 1.0), (0.25, 0.75), (0.125, 0.625), (0.375, 0.875), (0.0, 0.75), (0.25, 1.0)]Resampling stage. We mainly follow the CropResize [47] operation for our resampling stage, which first samples sequence length from uniform distribution U (r 1 , r 2 ) and then randomly samples a temporal segment with this length. Since effective resampling approach is not the primary goal of our paper, we fix it in our framework.Training details. For training AGCN, we follow [42] and use SGD optimizer with an initial learning rate of 0.1 and weight decay of 0.0001. We train the model for 50 epochs. The learning rate is reduced at 30th and 40th epoch with a decay of 0.1. For HCN we use the same training configuration as [46] . Parameter settings for other experiments are listed below. For N12-→K12 and N51← →P51, we set N tr = 10. For N51-→P51, N bkg = 10 and for P51-→N51, N bkg = 5. The boundary poses and linear transforms learned from the corresponding 18-class subset are transferred to N51← →P51 settings for efficiency. Also for N12-→K12, we set m aug to a smaller value of 0.5 and omit F N due to very high level of noise and missing joints in K12 dataset. We implement our method and other baselines using PyTorch [36] and conduct experiments on a single NVIDIA RTX 2080Ti. We will also release code in Jittor [13] implementation which supports faster training and inference.Design of learning variants for comparison. As in Table 7 , for learning to extrapolate (F NN,extrap ), we train a completion network using masked and full sequences, and during augmentation, we squeeze the original sequence to half of its original size and let the network extrapolate on both sides. We also compare with the design of learning to infill (F NN,infill ) instead of performing linear interpolation. Again we mask out a segment in the center of a motion sequence and use paired sequences to train an infilling network.Here we provide brief descriptions and more implementation details for other baseline methods.(1) General domain generalization approaches. CCSA [35] improves generalizability by aligning samples of the same label in the feature space via contrastive loss. For CCSA, the weight for contrastive loss is set to 0.1, and the margin for contrastive loss is set to 1.0. ADA [51] improves generalizability by augmenting with adversarial samples. ADA uses image-based perturbation in the original paper. We adapt it to the adversarial skeleton perturbation [52] as we find it yields better results. The training set is expanded with adversarial samples at 15th and 30th epoches. For adversarial augmentation we set the semantic regularizer to 1.0, learning rate to 1.0 and optimization steps to 5.(2) Self-supervised learning approaches. ST-Cubism [46] improves generalizability by solving auxiliary jigsaw puzzles. The spatial stream (Spa) and temporal stream (Tem) solves jigsaw puzzles in spatial and temporal dimensions respectively and the final prediction ensembles scores from the two streams. For ST-Cubism, the loss weight for jigsaw puzzle term is set to 0.1. Skeleton-MAE improves generalizability by learning to reconstruct the input skeletal motion sequence. For Skeleton-MAE, we borrow the masked auto-encoder module and reconstruction loss from [28] , and change the backbone to AGCN. During training, we randomly mask out a clip with a length ratio of 0.3 and let the model to reconstruct it. The weight for the reconstruction loss is set to 1.0. HICLR [62] is a skeleton representation learning method, which improves generalizability by pre-training a feature encoder via contrastive learning. For HICLR, we follow their supervised evaluation by first pre-training the encoder and then performing standard training using the training set. We adapt the backbone to AGCN as well.(3) Augmentation-based approaches. Mixup [61] generates augmentations by linearly interpolating between a pair of samples as well as their labels. For Mixup, training samples are augmented with Mixup with a probability of p = 0.5. CropPad [23] randomly crops a segment from the motion sequence and performs reflective padding on both sides. For CropPad, we use the crop ratio γ = 6.CropResize [47] randomly crops a segment from the motion sequence and resizes it to its original length. For CropResize, we randomly sample a segment with a length ratio between 0.7 and 1.0, which we find works the best. We also set the resampling process in our method the same as this baseline. For CropPad, CropResize, and spatial augmentations, we set m aug = 0.75 which is the same as ours. Uniform sampling [8] resizes a sequence by first uniformly dividing the input sequence into segments and then randomly sampling one frame within each segment. Such a sampling method can make full use of all frames in the raw motion sequence and therefore generate strong augmentations.(4) Alignment-based approaches. We use the feature encoder from the model trained with standard cross-entropy loss (ERM), and transform raw skeleton sequences into downsampled feature sequences (16 frames for each sample). We then use a kNN (k = 5) classifier [53] with warping-based distance OTAM [3] which is a representative warping-based distance measure used in few-shot video classification that deals with temporal mismatch. The final result comes from the ensemble of OTAM+kNN and the ERM model. We find that using distance in the raw joint space is inferior than in the feature space. Note that we implement temporal alignment method without further pre-training the feature encoder. Please refer to self-supervised learning for feature pre-training approaches.(5) Aggregation-based approaches. We build TSN [54] upon the AGCN backbone. The input sequence is uniformly divided into 3 clips before feeding to the backbone and we aggregate their features using average pooling. We implement multiple-crop testing as a post-processing method.Given a test sequence we sample 5 clips with length ratios of 0. Detailed results for ablation study. Table 12 presents results for the experiment of parameter analysis in the main paper. We find that λ T is generally not very sensitive, as choosing λ T over a large range (0.001 to 1) does not significantly affect the average accuracy. Table 13 provides detailed results for the per-class performance. Feature representations. We measure the similarity of two skeletons using L 2 norm in the joint coordinate space. However, it is also possible that we measure it in the latent space. Here we investigate the performance difference between these two feature representations. Specifically, we train a skeleton auto-encoder with MLP using training data and use it to obtain latent feature for the skeleton in each frame. For F N we perform boundary pose clustering in the latent feature space, and for F L we calculate similarity matrices using latent feature. As in Table 14 , we observe a lower but very close performance if using latent feature. This shows that our method does not depend on specific skeleton representations. Evaluation on different clustering algorithms. We investigate whether our method is compatible with different clustering algorithms. We conduct experiments using agglomerative clustering [37] , which is another type of clustering algorithm different from k-means. The result is in Table 15 .We see that the average accuracy is lower but still close and outperforms other baseline methods, which shows that our method is generally not sensitive to the choice of clustering algorithm. In the experiment, we keep hyper-parameters (e.g. number of clusters) the same as k-means. The result may be further improved by tuning more suitable parameters.Evaluation on resampling methods. For our experiments CropResize is used for the resampling stage. In Table 16 , we investigate the length of segments r as a key parameter. We find that sampling too short segments may introduce many noisy augmentations and hampers the overall performance.To resize the sampled segment to a fixed length sequence, linear and random frame sampling are also investigated. The results are shown to be generally comparable.Evaluation on standard in-domain action recognition. Although the proposed method mainly deals with cross-dataset settings, we find it can also work as an augmentation strategy for standard in-domain skeleton action recognition with training and testing within the same dataset. In Table 17 , we evaluate on the in-domain recognition on NTU-18 subset and ETRI Adult-18 subset from our constructed cross-dataset settings, as well as on the standard full NTU-60 dataset [41] . We adopt full AGCN backbone for N-18 and E A -18, and use full 2s-AGCN [42] for NTU-60. We set a smaller weight for augmentation m aug = 0.25. The performance improvement of our method is comparable to uniform sampling [8] , which is considered as an effective temporal augmentation. Note that for cross-dataset settings we surpass uniform sampling by 4.4% (see Table 1 ) since uniform sampling only samples the observed sequence. On the other hand, the fact that our improvement on N-18 and E A -18 is much smaller than in cross-dataset settings indicates that the training and test split of a dataset are generally well aligned. Our cross-dataset setting magnifies the issue of temporal mismatch and presents a more practical setting as in real-world applications. Spatial augmentation v.s. temporal augmentation. In Table 18 , we examine how spatial augmentation and temporal augmentation contribute to the improvement of generalizability, as they both prove to be effective in the self-supervised skeleton representation learning [23] . We choose typical spatial augmentations including shear transform [23] , bone length scaling [58] and Gaussian noise addition on joints [11] . To compare with spatial augmentations, we set shear ratio [23] β = 0.1, and set the translational noise with a normal distribution N (0, 0.01). For bone scaling, we randomly scale each bone with a length ratio between 0.9 and 1.1. We demonstrate that our method, in the form of temporal augmentation, is more effective than direct spatial augmentation. This shows that direct spatial transformation is more likely to change the semantics, especially for hand parts. Domain differences in human actions are easier to be handled in the temporal dimension than in the spatial dimension.More visualizations for F L . In Fig. 8 we visualize the clustered linear transforms when N tr = 20 and N tr = 5 using training set N . We find that a small number of N tr (e.g. N tr = 5) will hurt the diversity of linear transforms. In such a case, it is difficult to extract some important temporal action patterns (e.g. temporal shift and symmetry) via clustering.Error bars. The domain generalization task normally brings the issue of performance oscillation. In Table 19 we report 10-run standard deviation for the average accuracy in our cross-dataset settings. We show that our improvement is statistically significant.Qualitative results. In Fig. 9 we provide some qualitative examples showing our improvement over the baseline (ERM). With our proposed augmentation by action completion, we enable more accurate action recognition for long and full sequences when observing from partial action sequences.Experimental details for discovering complete action prior. Each input skeleton is flattened to a vector with a dimension of 75 (joint number J = 25). We design a simple skeleton auto-encoder consisting of six fully-connected layers and ReLU activation. The output dimensions for the fullyconnected layers are 128, 64, 32, 64, 128, 75, respectively. The L 2 norm loss is adopted to train the auto-encoder in order to reconstruct the original input skeleton. During training, we sample frames from processed skeletal data (uniformly resized to 64 frames and the first frame is aligned to coordinate) as training data.We train the auto-encoder separately for each dataset (PKU-MMD [25] , NTU60-RGBD [41] and ETRI-Activity3D [15] ) using data in our 18-class cross-dataset settings. We then use the trained model to transform all skeleton frames into latent features (extracted from the third layer with a dimension of 32). The feature diversity of dataset S at a certain timestamp t is the standard deviation of the obtained latent features z(•) averaged over all feature dimensions c ∈ C and all skeletonBeginning frames (NTU)Central frames (NTU) sequence samples x ∈ S.EQUATIONwhere z(x t ) is the mean of all latent features z(x t ) for x ∈ S, N S is the size of dataset S and N C is the number of latent feature dimensions.More analysis on the statistical pattern. We provide more analysis for curve patterns in Fig. 1 (b) .For PKU and NTU, as they are captured in labs and are generally clean with less occlusion and noise, they are of relatively low diversity and exhibit stronger level of complete action prior. For ETRI, as it is captured in home environment and is more challenging with larger noise, the frame-wise diversity would be higher. And as it is captured in real life with action boundaries that are hard to determine when collecting the dataset, the complete action prior is not as significant as NTU and PKU dataset. Fig. 7 shows examples of beginning frames and central frames for the NTU split in our cross-dataset settings, which is consistent with the statistical finding in Fig. 1 (b) .Also, we see that the global pattern differs across different datasets, for example, in the starting and end point diversity. For PKU and ETRI, they exhibit an overall temporal pattern of being symmetric.In contrast, the NTU dataset does not have such a symmetric pattern. There is also a shift for the peak diversity across datasets. The above differences are majorly caused by different definition of when an action starts and ends in different datasets, as can also be visualized in the Fig. 1 (a) example. These findings suggest that the domain gap between datasets, represented as temporal mismatch, may involve temporal shift, scaling and even symmetry/non-symmetry properties arising from the partial/full bias of human actions.Note that the two observations presented in the main paper that many human actions are partial observations and that human actions generally exhibit completeness within a large group of samples are not contradictory. This can be best explained by that (1) within a large dataset, some action categories or samples are nearly complete while some other action categories or samples are partial segments of complete actions, and (2) even if the actions are not exactly complete, e.g. only comprising central segments, many of them obey a similar statistical pattern which is only less significant. Here the "completeness" is the opposite of having uniform diversity over time. Also keep in mind that the curves in Fig. 1 (b) only reflect the statistical pattern for a group of actions in a dataset. For example, although the curve for NTU seems non-symmetric, we can still have approximate symmetry operations in the clustered linear transforms (see Fig. 8 ). This can be best explained by that there exist action samples/categories that are symmetric in the NTU split, but they only account for a small portion.Other implications on the proposed method. (1) The complete action prior generally relies on the assumption of clean skeleton data and daily routine pattern of human activities. Relying on this prior, our method is unable to work well when the source data is very noisy and contains very long motions with multiple stages, for example, taking K12 as source data. We find that K12 does not exhibit such strong prior as is observed in Fig. 1(b ). ( 2) When the target domain does not show strong action completeness or contains high level of noise, the extrapolation module F N may not have a very strong effect. This is partially shown in N-→E and N12-→K12 settings. This also explains our finding that using linear transform F L is generally sufficient for N12-→K12.Discussion for other baseline methods. We discuss some possible reasons that the alignment-based method is not as competitive as ours in the cross-dataset settings: (1) Given sufficient training data, the nearest neighbour classifier only memorizes training samples compared to network classifier and this in turn hampers its generalizability. Contrastive pre-training for the feature encoder with DTW-based loss may lead to better performance, but at a cost of much more computation. (2) Direct matching via kNN cannot fully take advantage of various types of augmentations, such as random rotation.(3) A better design of dynamic time warping is needed since there is domain shift across training and test datasets for either raw joint features or latent features. In widely-studied few-shot scenarios [3, 53] , the training and test data normally come from the same distribution. For aggregation-based method, clips sampled without action completion cannot well handle the partial/full bias. However, these methods and our proposed method are not contradictory and can be combined for further improvement.Stochastic action completion. In this paper we actually propose a stochastic action completion method that can be used to augment the training data. This stochastic action completion is more suitable for augmentation than a deterministic one. Note that F N has the probability to retain its original motion due to our sampling strategy for length of extrapolation t p . F L also learns identical mapping as is shown in the learned transform matrices (diagonal matrix). In this way, even for the input of a complete action, our approach is still able to output a reasonable complete action by sampling an identical mapping. A supporting evidence is that for P51-→N51 (P51 consists of generally complete actions), our method still improves the baseline ERM and even outperforms the temporal module of ST-Cubism [46] .Number of new parameters. During model training, the newly added parameters are two matrices, i.e., the boundary pose clusters of shape (N bkg , J, 3) and the linear transform clusters of shape (N tr , T, T ). Here N bkg = 10, J = 25, N tr = 20, T = 64. So the number of new parameters of these two matrices is ∼ 90000, which is less than two FC layers (suppose each layer is nn.Linear(256,256)).For other baseline methods, universal domain generalization methods CCSA [35] , ADA [51] and handcrafted augmentation methods such as uniform sampling [8] , Mixup [61] , CropPad [23] and CropResize [47] do not introduce new parameters. Self-supervised learning methods often have a new branch along with the original network for learning auxiliary tasks, therefore introducing new parameters (usually several FC layers). So generally, our method and those self-supervised methods have a similar magnitude of new parameters.Limitations. The limitations arising from the complete action prior are discussed in Appendix A5. Other implications on the proposed method. Moreover, we note that direct resampling from a complete action sequence might introduce ambiguity for some actions. For example, standing up and sitting down are two different actions, but both of them can be obtained by sampling from a complete action sequence in which a human stands up from sitting pose and then sits down. However, one should note that this problem also exists when constructing positive samples in contrastive self-supervised learning [11, 47] , as resampling or reversing an action may change the semantics for some action categories. To deal with this problem, we plan to add positional encoding and temporal masks in the resampling stage to further encode temporal information about how the sequence is sampled. We leave it for future work.Broader societal impact. Generally we don't find that our method has negative societal impacts. However, users should be careful about the result if action recognition along with our method is applied to critical human-related applications such as health monitoring and elderly fall detection. Current deep learning-based models cannot guarantee absolutely accurate action recognition. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: 3 of 4 reviewers were inclined to accept the paper with one reviewer flipping to positive. The negative review had very limited comments and did not respond to the rebuttal. The AC has read all reviews and agrees with the majority of reviewers that the paper should be accepted.

Reviews: review: paper_summary: paper_summary: In this paper, the authors address the issue of generalizing skeleton-based action recognition across different domains. They propose a novel recover-and-resample augmentation framework based on the concept of complete action prior. The approach is validated on different cross-dataset settings and demonstrates a significant improvement in cross-dataset accuracy compared to existing methods.
strength_weakness: strength_weakness: Although the method is intriguing and the performance is impressive, I am content with the current experimental setting. In real-world applications, dealing with cross-dataset issues makes it challenging to ensure that both datasets share the same action categories. Currently, the shared classes are manually selected. In my opinion, it would be better to adopt the new setting proposed by [1], where the source and target datasets have a category gap.

[1]   Collaborating Domain-shared and Target-specific Feature Clustering for Cross-domain 3D Action Recognition. ECCV 2022
questions: questions: N.A.
limitations: limitations: N.A.
paper_summary: paper_summary: This paper proposes a novel recover-and-resample augmentation framework to address the skeleton action generalization problem across different datasets. The framework utilizes a complete action prior to recover full action sequences from partial observations, employing boundary pose-conditioned extrapolation and smooth linear transforms. The proposed method demonstrates superior performance in cross-dataset settings compared to existing domain generalization approaches.
strength_weakness: strength_weakness: . Limited Scope: The evaluation is primarily focused on indoor datasets, which may limit the generalizability of the findings to other types of datasets or real-world applications.

. Complexity in Implementation: The two-step stochastic action completion and the need for clustering might pose implementation challenges for practitioners.

- Dependency on Clustering Algorithms: The approach heavily relies on clustering algorithms for learning boundary poses and linear transforms. The effectiveness of the method may be influenced by the choice of clustering algorithm and the parameters used, which might require extensive tuning and could impact the reproducibility and scalability of the approach. 

- Request more ablation experiments on diverse sampling and clustering methods.

- Limited Exploration of Resampling Techniques: While the paper proposes a robust framework for recovering and resampling action sequences, the resampling techniques themselves are relatively simple (random) and not extensively explored. More advanced or varied resampling strategies could potentially further enhance the augmentation process and improve generalizability.
questions: questions: No further questions.
limitations: limitations: Tackled in the draft.
paper_summary: paper_summary: The work proposes a novel recover-and-resample augmentation framework for domain generalization with application to skeleton-based action recognition. The authors aim to tackle a specific issue when moving from one dataset to another, i.e. the temporal misalignment of actions of the same class. In the experimental analysis, they provide comparisons with existing approaches and ablation studies.
strength_weakness: strength_weakness: - The introduction fails to guide the reader in understanding the motivations, methodologies and challenges. The motivations do not follow clear storytelling, and the description of the methodology is not fully comprehensible. Also, the language would need an improvement (the syntax of some sentences should be fixed)
- Also the SoA presentation fails to fully convince the reader of the value of the contributions of the proposed approach. The paper is related to different tasks and a more guided tour of the SoA would help to appreciate the contributions. Also, Domain generalization and Data Augmentation are two very important topics for the work, but the SoA discussion is very limited
- The methodology lacks technical details and appropriate, clear justifications for the different operations. 
- It is not clear if the results are fully reproducible. To my understanding, the experimental protocol is not  used in other papers and there are no details on the implementations that have been adopted, so it is very difficult to judge the results (although the authors say the code will be available upon publications, indications on the adopted implementation would be useful)
questions: questions: I would ask for clarifications on the following points

1.Introduction

- What do you mean by "It is actually a form that humans perform generally complete actions within large datasets"? If this is the case, what's the importance of alignment?
- "Although this prior can be detected by statistics in a general sense, in terms of individual samples, some exhibit strong action completeness while some are segments of their complete actions". 
- On Fig. 1(b): the behaviour of NTU and PKU seems different from the one of ETRI (more variability in the initial poses for the latter), while in your comments ETRI is treated on par with the other two. 
- "By studying the relationships between their raw and trimmed pairs, we can learn a temporal patterns inherited in human actions." 
- You named part of the method "Complete action prior" but the term seems never used in the next sections
- In addition to these questions, a more general comment is that the introduction fails to convey the message, which should be a clear and convincing description of the methodology.  The picture does not fully help the understanding

2.Related Work

- Some sentences are not well justified, for instance, "These approaches partially improve the generalizability across datasets but are still bounded by the specific augmentation design." or "...but they do not make full use of the skeleton representations". An intuition of the meaning of statements of this type and in what sense the proposed approach is better would be beneficial

3.Method

- "On the other hand, the statistical finding that the initial poses of skeletal action sequences have low feature diversity (shown in Fig. 1 (b)) also validates our assumption: the boundary poses are to some extent constrained and similar to each other." This does not seem true for ETRI
- If in the training dataset the actions are all starting from the rest position (so they are all aligned) how the proposed method helps dealing with misalignment when changing the dataset?
- With the extrapolation, is the length T remaining the same?
- "Note that the above nonlinear transform is still unable to capture global and structural patterns inherited in human actions". Why?
- Why is it useful to reorganize the frames?
- What are the differences between the proposed work and [17]?
- Eq. 4: in my understanding k_i should be an index between 1 and T, but from the formula it does not seem so
- How do you generate the similarity matrix?
- Why should you cluster the W? The need for clustering is not clear
- From the loss function, it seems that each training sample is paired with an "augmented" sample, which is in contrast with the use of a parameter m_aug

4. Experiments

- A quantification (or at least an idea) of the level of misalignment of the different datasets would be useful for the interpretation of the results
- Reporting the results with P as training would still be useful as a sanity check
- The transfer sub-settings used in the experiments are different from [41] (which, according to the authors, is the only one reporting results for cross-dataset settings. Why this choice?
- What implementations have you used for obtaining the results in the tables? 
- It is that Kinetic, mentioned in the adopted datasets, never appears in the experiments
- For the NTU datasets, have you considered the cross-view of the cross-subject problem?
- ERM corresponds to what version of the method?
- When reporting the per-class results, what about the actions that are not mentioned? It would be useful to have a more general idea 
- The experiment briefly described in "Generalizability for different backbones" is actually important and should go in the main paper (discussed more in-depth)

Minor

- I would not mention the use of NNs for the motion infiller in the methodology description
- The description of the experiment with P51, N51, N12, and K12 should be moved above, with the other details on the transfer settings

[After reading the other reviews and the rebuttal from the authors, I'm happy to increase my score as most of my main concerns have been clarified]
limitations: limitations: Limitations and potential societal impacts are discussed in the Appendix
paper_summary: paper_summary: This paper presents an innovative recover-and-resample augmentation framework to tackle the domain generalization challenge in skeleton-based action recognition. Utilizing the concept of a complete action prior, this method reconstructs entire action sequences from partial observations and resamples them to generate robust augmentations for unseen domains. The approach's effectiveness is confirmed through comprehensive experiments on three skeleton action datasets, showing considerable improvements over existing domain generalization methods.
strength_weakness: strength_weakness: 1. Domain generalization is mostly handled via RGB image, since due to the background changes and the camera positions there are obvious large distribution shifts. Regarding the skeleton data, since they are mostly recorded in 3D coordinates without background information, where the domain gap has already been reduced. The authors should justify the need of domain generalization in the skeleton-based human action recognition field with more details in the introduction section. Since in the ultilized datasets, e.g., NTU120, it involves cross view and cross subject evaluation, which can also be regarded as domain generalization.

2.  In Section 4.3, how did the authors implement these approaches on the skeleton-based human action recognition methods? More details are expected to be enriched.


3. What is the number of parameters for the proposed approach? The authors are encouraged to make comparison between the proposed approach and the baselines regarding the number of parameters.

4. The authors should provide more clarification regarding why they chose those GCN backbones, e.g., HCN and CTRGCN. Maybe can be described as a small subsection in Sec. 4.

5. TSNE visualization is interesting on the test domain to see the learnt embeddings from different approaches.
questions: questions: 1. Can the authors elaborate more on why domain generalization is critical in the context of skeleton-based human action recognition, despite the inherent reduction in domain gaps provided by 3D coordinates?

2. Why did the authors choose HCN and CTRGCN as the GCN backbones for their approach? Can this be detailed further, perhaps as a subsection in Section 4?

3. Could the authors provide a more detailed explanation of how the approaches in Section 4.3 were implemented for skeleton-based human action recognition methods, including any specific techniques or modifications used?

4. What is the total number of parameters in the proposed approach, and how does it compare to the baseline methods?

5. Can the authors provide t-SNE visualizations of the learned embeddings from different approaches on the test domain to illustrate the effectiveness of the proposed method compared to others?
limitations: limitations: yes it is in appendix

Comments: title: Rebuttal Response
Response to the author
comment: Based on my understanding, in real-world applications, there's no certainty that newly collected video or skeleton sequences will correspond to classes that the model was originally trained on. Therefore, I contest the claim that "As a result, such a setting is also limited in its practical use." This introduces a significant challenge in real-world contexts, necessitating further research and development to tackle it effectively.

Given that the main title of this paper is "Cross-Dataset Skeleton Action Recognition," as opposed to domain generalization in skeleton action recognition, my primary concern is with the accuracy of the cross-dataset setting definition. In my view, we should not align labels or categories before proceeding with cross-dataset experiments.

In conclusion, I will maintain my rating at '4: Borderline reject'.
Dear Reviewer Vnjk:

Thanks for the positive response and raising the score. We really appreciate your kind suggestions. In the revision for the introduction, we will add (1) clear insight/motivation for recovering complete actions as explained in the rebuttal, (2) a brief pipeline description for the full method, and (3) a clear justification for the need of clustering in each module as explained in the rebuttal. We will also address the issues for other parts of the paper as suggested.
I truly thank the authors for the care they put in their rebuttal. My main concerns have been solved (some of my observations were actually due to misunderstandings on my side), and  I'm happy to raise my score. In any case, I suggest the authors to revise the introduction to make the story more clear (see my original comments).
Thank you for the detailed response. I will keep my score as 6 according to the contribution of this work.