Recovering Complete Actions for Cross-dataset Skeleton Action Recognition
Authors: Hanchao Liu
Yujiang Li
Tai-Jiang Mu
Shi-min Hu
Venue: NeurIPS 2024 poster (2024)
Keywords: skeleton action recognition
domain generalization
data augmentation
Primary area: machine_vision
TL;DR: We present a novel recover-and-resample augmentation framework based on complete action prior for skeleton action generalization task.

Abstract: Despite huge progress in skeleton-based action recognition, its generalizability to different domains remains a challenging issue. 
In this paper, to solve the skeleton action generalization problem, we present a recover-and-resample augmentation framework based on a novel complete action prior. We observe that human daily actions are confronted with temporal mismatch across different datasets, as they are usually partial observations of their complete action sequences. By recovering complete actions and resampling from these full sequences, we can generate strong augmentations for unseen domains. At the same time, we discover the nature of general action completeness within large datasets, indicated by the per-frame diversity over time. This allows us to exploit two assets of transferable knowledge that can be shared across action samples and be helpful for action completion: boundary poses for determining the action start, and linear temporal transforms for capturing global action patterns. Therefore, we formulate the recovering stage as a two-step stochastic action completion with boundary pose-conditioned extrapolation followed by smooth linear transforms. Both the boundary poses and linear transforms can be efficiently learned from the whole dataset via clustering. We validate our approach on a cross-dataset setting with three skeleton action datasets, outperforming other domain generalization approaches by a considerable margin.

Recovering Complete Actions for Cross-dataset Skeleton Action Recognition
Hanchao Liu'
Yujiang Li
Tai-Jiang Mu
Shi-Min Hu'
BNRist Department of Computer Science and Technology. Tsinghua University
Abstract
Despite huge progress in skeleton-based action recognition, its generalizability different domains remains a challenging issue. In this paper, to solve the skeleton action generalization problem, we present recover-and-resample augmentation framework based on a novel complete action prior. We observe that human daily actions are confronted with temporal mismatch across different datasets, as they are usually partial observations of their complete action sequences. By recovering complete actions and resampling from these full sequences, we can generate strong augmentations for unseen domains. At the same time, we discover the nature of general action completeness within large datasets, indicated by the per-frame diversity over time. This allows to exploit two assets of transferable knowledge that can be shared across action samples and be helpful for action completion: boundary poses for determining the action start, and linear temporal transforms for capturing global action patterns. Therefore, we formulate the recovering stage two-step stochastic action completion with boundary cose-conditioned extrapolation followed by smooth linear transforms. Both the boundary poses and linear transforms can be efficiently learned from the whole dataset via clustering. We validate our approach on cross-dataset setting with three skeleton action datasets, outperforming other domain generalization approaches by considerable margin.
Introduction
The skeleton- based action representation has the advantage removed background changes and camera positions, making it more robust compared to RGB representation. However, the generalizability to unseen domains under such representation is still affected by the inherent spatiotemporal difference of 3D coordinates of a same action across domains, which yet largely remains an under-explored issue. In this paper, we study the single domain generalization problem (55) for skeleton-based action recognition, in which we do not have knowledge about the target domain.
akeleton-based action recognition has recently achieved great success 42 6 29
Essentially, both cross-subject and cross- -ross-view settings 4 settings, and they can be well addressed by designing more powerful backbones and applying geometric transformations, achieving high accuracy the test set 6 29 However, we find that in
fall under the category of cross-domain
the cross dross-dataset setting where source and target data come from different datasets, the performance on accuracy degrades lot (around or more than 20% in some cases
466 and cannot be well remedied
by the above approaches. This indicates drastic domain gap in the inherent feature of human actions across datasets, posing great challenges for real-life applications [40] [8] and calling for research on domain generalization techniques for skeleton-baue based representation, which is the focus of this paper.
"Tai-Jiang Mu is corresponding author. Email: taijiang tsinghua..duu.cn Code available at https // //githi giihhub.ce haw.aacchhoo.teeeeeeeeeeeeee Redemmmmple
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
Cross-dataset skeleton action recognition
(b)
Complete action prior
(c)
Recover and resample
Dataset
feature diversity
Complete Prior
3 Freoquer
ETRI
Knowledge
NTU
Fresample
Dataset
R
8
PKU
Aug. data
PKU
Training skeletons
Datasct
Raw data
classpfirr
ETRI
action start
action end
dasa
prior knowlolge
Figure 1: (a) Cross-dataset skeleton action recognition. Taking action phone calling as an example, temporal mismatch across datasets poses challenging issue. (b) Complete action prior. Human actions within large datasets exhibit statistical patterns from less feature diversity to more diversity, implying the nature of action completeness (shown for NTU, PKU and ETRI dataset). (c) Recover and Resample. After learning stochastic action completion function from the training data, we recover complete actions and resample from them to further augment the training set.
Investigating action samples across multiple datasets, our observation is that a notable source of domain gap comes from the temporal mismatch of an action across different datasets (Fig. Ta)), which is usually caused by different definition or cropping criterion of human motions. Existing methods cannot perfectly handle this problem. Warping- Warping-based temporal alignment approaches.
while being popular for few-shot recognition tasks [3 53 33 45). are inefficient and not optimal when given sufficient and diverse training data. Besides general domain generalization approaches 35 51]. self-supervised skeleton representation learning methods
[33 47 46 59), while showing good transferability of the learned feature, are still difficult to handle data with large gaps as the generalizability largely comes from seeing handcrafted augmentations in either contrastive learning [62] or auxiliary tasks [46]. Different from the above approaches. we aim to directly hallucinate strong augmentations for unseen domains by exploring some sort of human action priors.
As in Fig. (b), we train a skeleton auto-encoder and examine the per-frame latent feature diversity of the whole dataset (measured by standard deviation) for several skeleton action datasets, i.c., NTU 4T) PKU (25). ETRI [1] (Details in Appendix A5). We observe that human action sequences start with relatively low fcature diversity, which is actually a form that humans perform generally complete actions within large datasets, from rest poses that are less diverse (e.g. stand, sit) rich-semantic poses that are more diverse. We summarize this pattern action prior. Although this prior can be detected by statistics in a general sense, in terms of individual samples, some exhibit strong action completeness while some are segments of their complete actions. This motivates us to learn an action completion function from the whole training data which can transform incomplete actions into complete ones. In this way, we can recover the complete sequence of an action from its partial observation, and resample from to hallucinate data unseen domains.
novel temporal prior named complete
Going deeper into such statistical pattern, we can further mine some class-agnostic knowledge from the whole dataset that can be used for our action completion. First, the low diversity at beginning frames implies the existence of a set of representative boundary poses, which can be used for determining the start of an action. Second, the general completeness implies the existence of long and nearly complete actions within a large dataset. By studying the relationships between their raw and trimmed pairs, we can learn temporal patterns inherited in human actions.
Based on the above observations, we propose a novel recover-and-resamppe augmentation framework for single domain generalization skeleton action recognition. As in Fig. (c), we recover complete actions from training samples and resample from them to further augment the training set. Especially, in the recovering stage, we adopt a two-step stochastic action completion, which first extrapolates the raw sequence conditioning on the boundary pose and then applies temporal transforms. A set of boundary poses are learned from the first frames of the training data. Altogether, set of smooth linear transforms (linearity means re-organizing existing frames) are learned from reconstructing full sequence from its trimmed segment via closed-form solution of context-aware frame similarity aggregation. We find such a simplified form of transform suitable and expressive enough for modeling common structural temporal patterns, e.g., shifting, sealing, symmetry, etc. The learning for both boundary poses and linear transforms can be achieved via clustering, for example, k-means, which makes our whole framework light and efficient. Also, we find a random cropping suffices for the resampling stage.

---

We simplify the problem in cross-domain settings by studying well- daily actions in indoor settings. Specifically, we construct a new evaluation setting jointly with three large-scale datasets including PKU-MMI 225 NTU-RGBD 41 and ETRI-Activity3D [15]. Our proposed augmentation method well solves the temporal misalignment issue. We improve the average accuracy on unseen datasets by 5%, outperforming other baseline methods by a large margin.
In summary, the contributions of this paper are as follows:
We discover the complete action prior within large datasets by its statistical pattern, whose effect has not been well studied before. Building such a prior, we present novel recover-and- resample augmentation framework for domain generalization on skeleton- action recogni- tion.
We propose an effective clustering- approach to recover complete actions, which is achieved by boundary pose conditioned extrapolation and context-aware smooth linear transforms.
We demonstrate the superiority of our method over wide range of methods by large margin and we conduct extensive experiments to show the effectiveness of each proposed module.
2 Related Work
2.1 Skeleton-based Action Recognition
As for skeleton action recognition, the graph neural network [27] 6 29] has become a prevailing model due to its effectiveness to model spatiotemporal relations between joints. We use AGCN (Adaptive GCN) [42 as our backbone as can model flexible graph layouts. Recently, there is focus on studying skeleton action recognition with less labeled data. For self-supervised skeleton representation learning, a number of pretext tasks are proposed based on invariant augmentations
59 56). These approaches improve the generalizability across datasets but are still bounded by the specific augmentation designs. For few-shot skeleton action recognition, metric are widely adopted. However, they are not optimal for domain generalization given large amount of training data. Our work especially explores the generalization problem for this field when data is not available in the target domain.
23 47
learning approaches with different types of dynamic time warping 3 5
33 45]
2.2 Domain Generalization and Data Augmentation
General domain generalization approaches mainly rely on learning adversarial augmentation 51 22
55 and supe-sspe auxiliary tasks (4. They are not optimal for skeleton- based action recognition tasks since they do not make full use of the skeleton representation. The recent ST-Cubism [46 is most related to ours, which adapts learning jigsaw puzzles 46) skeleton sequences to achieve domain generalization. While there are many augmentation methods intended for non-semantic related tasks like pose estimation [6
2 TO 14] and human motion prediction [31,, skeleton augmentation for recognition tasks is essentially challenging as it is fragile to spoil the semantics of skeletons formed by now-dimensional joints. SFN (34] combines Mixup [61 with VAE for augmentation, but it still cannot deal with cross-domain settings. ModSelect (32) proposes selection mechanism for multi-modal input for cross-domain action generalization. In this work we focus on the learnable temporal action augmentation for cross-domain settings.
2.3 Human Motion Priors
As human motions are constrained in the spatiotemporal space, many human motion priors are proposed, such as manifolds of valid human poses [19] 7 48) and motion [63 27 17 and other body part interchangeability across 65 ACTOR (38) learns CVAE and samples data points conditioning on labels as an augmentation method for recognition tasks. However, it is only useful when data is scarce. Rate-invariant prior [50) is related to ours, but changing rate itself not enough for cross-domain settings. this paper, we explore the action completeness prior for better performance in cross doma-n settings. Although the notion of action completeness [30 64 appears in the temporal action localization (TAI.) task (0) we are the first to incorporate it for the skeleton-basee domain generalization task.
properties such as periodical human motion pattern 43
samples 26 24] and alignment with other modalities
3

---

Training set S
Boundary pose clustering
Linear transform clustering (W1
{pr}
Per-frame similarity
Cluster & Aggregate
W
Boundary conditioned extrapolation
Linear transformation
Augmented data
training sample from
5p- Retooaa a3r
W WWWW
Predicted
Predicted
(Label: phone calling)
Fx(x) = F(x,p',t))
F Fx(x) Wx
Resampling
Figure 2: Overview of Recovering and Resampling. Given training set S, learn boundary
poses (p:) and context-aware linear transforms (W;) via clustering. For a sample x from S, we first do extrapolation (Fxr conditioning the boundary pose p' with infilling length ly. and then perform linear transform (F2)) by sampling from {Wi}. The new data points x' are resampled from recovered complete actions as strong augmentations for unseen datasets. Skeletons in dark blue rectangles are new frames generated by FN and Fc. Both and æ' are used for training the classifier.
3 Method
3.1 Problem Setting
In the skeleton-based action recognition, suppose have source data from one single domain {(xs, ys)). Due to the unfixed length of input action sequences, the skeleton sequences xs are all uniformly resized to Is E Xx3 with sequence length 7 and joint number J 15). Our goal to train a model G using S and test on other unseen target domains ((111,911) (IT2, 972), ". In particular ys and yr share the same action categories.
3.2 A
Resample Framework
Motivated by the observations of temporal mismatch (Fig. [a]) and action completeness within dataset (Fig. [b)). we first introduce generalized form of temporal augmentation, which we name acover-and-resamppe augmentation. Given a training sample x we first recover complete action with transform W Wrecoeer and then sample segment of it using W Wessmmpl augmentation x':
s' Wnn(x)) = (Wresample W (x))
Ideally Winl can generate all kinds of temporal segments that may appear in the target domains. In contrastive self-supervised learning, (3 uses temporal shift as recovering process, while [47]
only uses resampling for constructing positive samples. We set the resample transform W ressaplle as common random sampling [47), and focus more on the recovering stage Wresoer that can construct complete action for the input motion Our contribution is that by exploiting the knowledge from complete action prior, we further decompose W Wreswer into a linear transform and nonlinear transform, i.e. W Wiecover(1) generate new poses which do not exist in the original motion. The linear transform FL reorganizes the motion sequence using existing frames. In this way boost the expressive power of W Wrecover for generating complete actions. Fig. [ givee an illustration on how we generate augmentations for partially observed action phone calling. Details for Fr and Fy are introduced in Sec. 3.3 and 3.4
(Fx (z)). The nonlinear transform FN extrapolates the motion to
3.3 Boundary-conditionel Extrapolation
Boundary pose clustering. When recovering a complete action, we first determine boundary pose. Intuitively, boundary poses for common daily activities are more likely to be buckground poses (or rest poses). They do not convey strong action semantics and can be distinguished from meaningful foreground poses. On the other hand, the statistical finding that the initial poses of skeletal

---

action sequences have low feature diversity (shown in Fig. (b)) also validates our assumption: the boundary poses are some extent constrained. Therefore, we propose cluster the first frames zo of all the training samples in S to get a set of representative background (boundary) poses, which we denote as {p:). During clustering, all the first frames ( Rxx33 are flattened to 1d vectors and L2 norm is used as the distance measure.
Conditional generation. We use nearest neighbour search assign the boundary pose p' for æ with first frame To:
p' arg min Zo Pil. {
(2)
Now we can extrapolate the original first frame 2g to the new boundary pose p'. This is basically an infilling process conditioned on p', and we control the length of extrapolation with parameter tp- Supposing the sequence length T. we set the first frame as p' and squeeze the original motion x: to the segment from frame (p to the end frame T. We then infill the timestamps from the first frame to frame ip with new motion. Considering a stochastic process, we sample ip from a Beta distribution 3(a, a 22 with high probability on the first frame and frame T/2. This means the conditional generation process is more likely to retain the original motion or infill a segment with length 7/2.
Motion infiller. The simplest form of the motion infiller F is linear interpolation since the skeleton already has good joint correspondence. We also propose an alternative to train an infiller with a neural network. We mask out some consecutive frames to get masked and full sequence pairs and train an action completion network. We then use this motion infiller to perform extrapolation. The comparison will be shown in the later experiments. Actually we find the parameter-free linear interpolation works well enough for generating reasonable motions for the recognition task.
So formally our nonlinear transform can be represented as Fx(x) = =(x p'pp). We first find the boundary pose p', sample ip to determine the sequence length infill, and finally apply the motion infiller F.
3.4 Learning Smooth Linear Transforms
Note that the above nonlinear transform is still unable to capture global and structural patterns inherited in the human actions. Consequently, we further propose linear transforms Fz to reorganize frames for more powerful transformations. We have seen great progress in self-supervised learning with very simple temporal linear transforms like crop & pad (23]. However they are manually designed and are not learned from data they are not flexible Here we propose to construct partial and full skeleton sequence pairs to learn a set of linear transforms in the form of Fr) = Wx, W E IR J is skeleton sequence with length T.
RT Here
Context-aware smooth linear transforms. Given partial sequence u and full sequence v (see Fig. 2) which are then both resized to length T, we hope to find linear transform that minimizes the Le norm [Wu v/. where W has only one nonzero element for each row. The straightforward solution is to do it frame-wise, i.e. for each frame i/ in However, such an operation cannot ensure that W is smooth and consistent with good transferability. Inspired by the context-awar alignment (20), we propose context-aware smooth linear transform that aggregates information from semantically adjacent frames.
find the index of its closest frame Uj in
Specifically, we first find the context-aware frame-wise similarity matrix of u and v. For example, the similarity score 8iy between i-th frame of v and j-th frame of u can obtained as following:
exp(-lu; U2 ----uu//00 l/T/
$1
exp( |14 umll/>T)
where | denotes Ly distance between two flattened skeletons (root translation already removed). Larger Xr means stronger context-aware smoothing. We then calculate the index by:
K
Σ j 8ij j=1
The transform matrix W is then obtained by setting Wii, rownd(k;)] = 1. This is to say, when deciding the most similar frame for U; all frames in u are considered by their similarity weights. In this way the transform W becomes smooth and more resilient to noise frames.
5

---

Clustering linear transforms. Again we utilize clustering algorithm to extract all the possible transform patterns that map partial sequences full sequences. Since the original number of transforms for W obtained from randomly sampled trimmed and full pairs may be too large, with clustering we avoid the inefficient sampling of W. Meanwhile, during the clustering, some important transform patterns, e.g. mirroring, can stand out as they may originally only account for small percentage in the whole pool of transforms.
In details, we first randomly generate training pairs, i.e. partial motion sequences and full motion sequences {(",, from S. We then use Eq. (3) to get similarity matrix M(&) for motion and Tg. Instead of clustering directly on W, we perform clustering on all the similarity matrices {M to get representative cluster centers }. We normalize each row for M, and then calculate the index according to Eq. (4) and obtain the corresponding transform matrix W,. Again, during clustering, each M (k) E T'xT is flattened to ld vector and we use L2 norm as the distance measure.
Once we obtain a set of {W.). we can randomly sample W from {Wa} and apply it to x. So the linear transform can be formally represented as Fz( Wz, W [W,[W, E R
3.5 Training
We follow the standard pipeline for training a recognition model. For input ø with its action label we generate augmentations x' (refer Algorithm in Appendix Al), and train the model with the loss function:
Liotll ((G(z). 3) + C(G(2'), v),,
(5)
where L is the standard cross-entropy loss. In practice, in batch, we augment the training samples with a ratio of Mag. and the rest of the samples remains unchanged.
4 Experiments
4.1 Datasets and Settings
Datasets. We use four large-scale datasets, i.e, NTU60-RGBD 41. PKU-MMD 255 ETRI- Activity3D [ISI and Kinetics [5]. The first three datasets are captured in indoor laboratory and household environments while the last one is collected from online videos.
new cross-dataset setting. We construct a new multi-domain cross- dooss-ataset setting, mainly comprising actions in indoor environment. We gather shared 18 actions (see Appendix A2) using the first three datasets described above and each dataset is treated as one domain. We train on one domain and test the rest
domains. Specifically, ie. N + E, N - P. EA -> N, EA - P (each dataset denoted with its first letter). Especially we use the adult split É' for training because in this way we keep the number of training samples for NTU and ETRI relatively the same. We do not include P for training since / majorly contains long and complete actions due its annotation and it relatively easy to perform sampling directly. We also reserve a subset P with mutually exclusive labels with P as a prior dataset to evaluate in an "Oracle" case when we have certain knowledge about how complete actions look like.
define four domain transfer sub-settings,
Evaluation Metric. We use the average accuracy of the above four cross-dataset sub-settings to measure the performance of single domain generalization. Especially we use balanced accuracy to eliminate the influence of class bias. We train the model with ten different random seeds and report the mean accuracy.
32 39
For fair comparison with ST-Cubism [46), we also include P51 ++ N51 with 51 action classes paired between NTU and PKU dataset, and N12- K12 with 12 action classes paired between NTU and Kinetics. We also follow their evaluation protocols when reporting results.
4.2 Implementation Details
Data preparation. We resize motion sequences from different domains to a fixed length T = 64. Following 42), we remove camera rotation and trajectory movement in the pre-processing stage and we further apply random 3D rotation as spatial augmentation for all methods.

---

Table Comparison with other methods in our cross cross-datased settings. The best result is in bold and the second best is with underlines.
Method
E
N -
EA
EA -
Avg.
ERM
54.9
70.5
42.4
49.7
54.4
CCSA
56.0
72.2
43.6
51.7
55.9
ADA
9
55.2
69.2
43.8
50.7
54.7
ST-Cubism 46
59.1
71.7
45.4
52.4
57.1
Skeleton MAAE 28
56.1
72.7
44.5
52.4
56.4
HICLR 62
54.0
46.7
53.8
56.3
Uniform sampling
56.7
45.3
51.3
55.7
Mixup 61
55.0
69.9
44.5
52.2
55.4
CropPad 23
56.8
70.1
44.0
50.9
55.4
CropResize 47
57.5
70.3
45.5
50.6
56.0
TSN
54.3
43.0
50.0
Multiple-crop testing
54.4
76.5
40.8
52.6
56.1
OTAM+KNN 3
54.2
72.7
42.9
50.8
55.2
Ours
58.4
75.8
48.4
57.8
60.1
Table 2: Comparison with other meth- ods in NTU PKU setting with 51 actions.
Table 3:
Comparison with ST-Cubism in multiple cross- dataset settings with HCN backbone.
Method
Backbone
N51-P551
P51-+N51
Method
N51 N51-+P51
ERM
HCN
57.6
50.5
ST-Cubism (Tem)
HCN
60.0
$2.7
ERM
66.3
ST-Cubism (Spa) 46
HCN
59.6
50.8
CCSA
67.3
ST-Cubism (6
HCN
61.3
53.8
ST-Cubism
46
70.5
Ours
HCN
62.8
53.3
Skeleton-Man
28
70.3
HICLR 62
66.5
Method
Backbone
N12->K12
CropPad
23
69.0
CropResize
67.0
ERM
HCN
14.4
ST-Cubism
46
HCN
15.6
Ours
72.2
Ours
HCN
15.9
Backbone models. For our backbone model, we adopt a slightly modified version (26 of Adaptive GCN (AGCN) 42 with fewer blocks and only use the joint stream for speed and simplicity. We reduce the number of blocks from 10 to The output channels for each block are 64, 64, 128 and 256. The kernel strides for each block are 1, and 2. We find that such design improves the base generalizability partially because better fits the reduced length motion sequences and avoids overfitting. Besides AGCN backbone, (6). two representative GCN backbones. ST-GCN is a simple GCN that aggregates spatiotemporal information without elaborate design. CTR-GCN adopts multi-level feature design and has much more parameters than our AGCN. HCN is two-stream convolutional network used by [46]. We use for fair comparison in P51 + N51 and N12-<K12 settings.
also test ST-GCN [57) and CTR-GCN
Training details. We set training pprammpareetters the same as [42]. Furthermore, we set X7 = 0. the number of linear transforms Nr 20, the number of background poses Noky 10. and Maug = 0.75. We set a = 0.1 for sampling fp. We fix the resampling method by randomly sampling segment with length ratio between 0.7 and 1.0. We use k-means for clustering boundary poses and linear transforms. Parameter settings for P51 <+N51 and N112>K12 are provided Appendix A3.
4.3 Results
Baselines. Besides ST-Cubism [46] which is by far the only method that reports result for cross-
dataset settings, we compare with wide range baseline methods.
(1) General domain gen-
eralization approaches: latent feature alignment CCSA (35) and adversarial augmentation ADA
52]. (2) Self-supervised learning approaches: ST-Cubism [46] that solves jigsaw puzzles in
spatial (Spa) and temporal (Tem) dimensions, Skeleton- MMRe (partially borrowed from (28) that

---

Table 4: Effect each component and the effect of using different prior datasets for our method. All ablation experiments in this table have the resampling step.
Method
N - E
-
EA -
Ea
P
Avg.
ERM
54.9
70.5
42.4
49.7
54.4
CropPad
23
67.9
45.2
51.7
55.7
Nonlinear FN
57.7
70.8
48.3
54.5
57.8
Linear Fz. Linear(SelD+NNonniieerSSSSl
58.6
72.8
45.3
52.2
57.2
58.4
75.8
48.4
57.8
60.1
Linear( P) +Nonlinnar((ee1f
58.2
76.2
48.5
57.7
60.2
Linear SimeattSeI++Noolineear
58.7
76.4
48.4
58.4
60.5
Linear( P)
+Nonlineard P)
57.9
76.3
49.2
58.6
60.5
Figure 3: Visualization selected linear transform matrices {Wi} via clustering using training sets N and Ea.
learns skeleton representation in the fashion of masked auto-encoder, and HICLR 62 which is representative pre-training method via contrastive learning. (3) Augmentation-baser approaches:
mixed sample data augmentation Mixup [6], Crop and reflective padding (CropPad) [23]. crop and
resizing (CropResize) [47] and uniform sampling [8] (4) Alignment- based approaches: OTAM
3 +KNN which uses nearest neighbour classifier [53] with variant OTAM (3) as the dynamic time warping distance. (5) Aggregation-base -based approaches: Temporal segment network design (54) and multiple-crop testing (5-crop) which samples clips from test sequence and average the output scores. Moreover, Empirical Risk Minimization (ERM) uses standard cross-entropy eessopy loss and serves as a performance lower bound. More implementation details provided in Appendix A3.
The result is shown in T Our method, by explicitly exploring the action prior, improves ERM by 5.7% and outperforms the second best (ST-Cubism) by 3.0% in terms of average accuracy. All the self-supervised and augmentation-baset the former perform better than the latter. However, neither of them can significantly improve the generalizability to deal with unseen data with large gaps. By learning transforms from training data, we are more flexible than handcrafted augmentations and perform especially better in recognizing long sequences (P domain). Test-tine time-ttimeeee aggregation only improves on one specific setting and warping- based matching with kNN is not competitive directly learning deep classifier when given sufficient and diverse training data (See more discussion in Appendix A6). While sharing the same spirit that temporal alignment is important, we provide alternative solution which combines raw training data and augmented "aligned" data from recovering and resampling complete actions.
methods improve the domain generalizability while
We further transfer boundary poses and linear transforms learned from the 18-class subset to N51++P51 settings. Table 3 we compare to ST-Cubism [46] following their protocol using HCN (21] backbone. For N51-P55 as in Table 2 and 3 our method outperforms ST-Cubism, showing the effectiveness when there are more action classes. For P51-N15, ours is better than the temporal part of ST-Cubism but slightly worse than the whole ST-Cubism, which can be explained that P51 generally already consists of complete actions. For the challenging 2D setting N12-XK1,, use linear transform module only and also obtain slightly better performance than ST-Cubism.
4.4 Analysis and Discussion
Effect of each component. We examine the effect of the non-linear transform Fy. i.e. the boundary conditioned extrapolation, and the linear transform Fi. As shown in Table 4 both Fx and FL improve the ERM baseline terms of the average accuracy, showing they are crucial to construct complete actions and validating our two-step design. As comparison, we see that recovering with handcrafted CropPad only leads to limited improvement (1.3%). Moreover, cases when the action completeness is less significant in the target domain (e.g. N-E),, Ft itself may be sufficient.

---

Table Effect of linear transforms.
Table 6: Effect of background poses.
Nu
N-E
N-PP
Ea
N
EA
P
Avg.
Noke
N-EE
N->P
Ea
N
Ex
+>
Avg.
ERM
54.9
70.5
42.4
49.7
54.4
ERM
54.9
70.5
42.4
49.7
54.4
3
57.8
69.8
49.2
58.7
58.9
None
58.6
73.7
47.6
55.3
58.8
5
58.2
72.4
48.9
57.2
59.2
5
58.1
76.3
48.1
57.2
59.9
58.2
56.9
59.3
58.4
75.8
57.8
60.1
20
58.4
75.8
48.4
57.8
60.1
20
58.4
75.7
48.5
57.4
60.0
Table 7: Comparison with other learning variants in our method.
Table 8: Per-class accuracy improvement of our proposed method compared to ERM.
Method
N-E N-P
Ea +N
E +
Avg.
Action
Avg. acc. across four settings
ERM
54.9
70.5
42.4
49.7
54.4
Phone calling
38.8(+25.3)
FNN.estip (-)
58.6
74.3
46.3
56.1
58,8
Hand waving
71.3(+22.1)
FINNinnn
58.6
73.8
46.7
55.6
58.7
Clapping
54.7(+15.0)
Pointing finger
92.5(+9.7)
Ours
58.4
75.8
48.4
57.8
60.1
Taking off clothes
88.0(+3.1)
(a)
Action: phone calling
ff
A Action: clapping
# # t & &
(b)
f
Action: hand waving
Figure Visualization for the boundary pose clustering result {p,} when Nang 5. (a) Pose clusters for training set N and (b) Pose clusters for training set EA.
Figure 5: Examples some recovered complete actions. The skeletons in blue are inputs and the skeletons in sky blue are new frames generated by our method.
Since Fr and Fx learned from different datasets are transferable, we can also use P as a high quality prior dataset to replace the original training data (denoted Self in Table 4 to learn Fz and FN- In this way we can investigate the performance upper bound of our method. By utilizing P we can further improve the average accuracy from 60.1 1 60.5. This shows that would more beneficial to directly learn from nearly complete actions since it is our aim to reconstruct complete actions from partial observations. the other hand, the marginal improvement of 0.4% also indicates that even from the training data we already learn transforms and boundary poses in quite decent manner.
Parameter analysis. We examine three important parameters: number of clustered boundary poses Nokg number of clustered linear transforms N and context-aware coefficient Xy. The results are in Table lndd
and 6 As for Nú. too small number clusters will hurt the overall performance by around 1.2%, which shows that too few linear transforms cannot fully represent the various global motion patterns. As for Nikg, we empirically find that the number not very important issue probably because the variety of boundary poses does not significantly affect the semantics. However, directly extending with the raw first frame (shown as "None") will hurt the performance by 1.3%, indicating the importance of boundary pose clustering. We also investigate the parameter sensitivity of Xy. We find that \y = 0.1 works well and generally it is not very sensitive (sce result in Appendix A4).
What is learned for augmentation? In order to better understand what is learned in Fz and Fx we visualize the clustering results. Fig. visualizes the clustered background poses. We find that the background poses mainly include common rest poses such as standing and sitting with different rotations. These poses do not always appear all action sequences, so clustering from the whole dataset is reasonable. Fig. 3 visualizes the clustered linear transform matrices. We find that FL mainly learns temporal shifting and reflection operation from the training data, which indicates presence of approximate symmetric pattern of many complete actions (see matrices in ">" and "<" shapes). We can also interestingly conclude that the CropPad transform 2 actually fits our complete action prior well it appears in the set of Fz transforms (such as the second matrix in Fig. 3. Ours

---

ERM
ST-Cubism (Spa)
ST-Cubism (Tem)
Ours
visualization of the feature embedding for the test set in N51 setting with HCN backbone. The color indicates the groundtruth label. Our method learns a more discriminative feature embedding.
Figure 6: The t-SNE 49
Table Performance our method on different backbones.
Backbone
N + E
N +
P
Ea
+ N
EΛ + P
Avg.
ST-GCN 57
51.7
67.7
37.8
44.5
50.4
ST-GCN+Ours
56.7
69.7
46.1
49.0
55.4
CTR-GCN O
54.7
72.9
46.6
50.9
56.3
CTR- Ourss
63.6
81.4
53.9
58.3
64.3
more flexible complete actions from partial observations. In Fig. [6] we show that with our augmentation we learn more discriminative feature embedding compared some other baselines.
learn set of possible transforms. Fig. 5 provides examples of recovered
Comparison with other learning variants. Note that also possible to perform infilling and extrapolation for skeleton sequences by training action completion network. In Table investigate the performance when learning by neural networks. We compare to the design of learning to extrapolate without conditioning boundary poses, as well the design of learning to infill instead of performing linear interpolation. For both variants we adopt the network design in [28 and more details provided in Appendix A3. We find that in terms of average accuracy, both learning to extrapolate (FNN.extrap) and learning to infill (FNNinnnn) are not as effective as our proposed learning- free interpolation. which shows that (1) assigning boundary pose via clustering is necessary as the network itself is difficult to learn automatically; (2) infilling by linear extrapolation is actually simple but effective design the infilling quality is good and robust for domain generalization.
Per-class results. [ the performance improvement for selected action categories. Our method can significantly improve the accuracy several hand-related actions such phone calling, hand waving and clapping. In the case of temporal mismatch, these actions are easily confused with other similar actions if only trained from partial observations. If action be well recognized by partial segments, the improvement may be small (e.g. taking off clothes).
Generalizability for different backbones. We further apply our method on ST-GCN [5 and CTR-GCN (6 to see whether various backbones can benefit from the observed complete action prior. Note that CTR-GCN extracts temporal feature in a multi-scale fashion, which is more advanced than AGCN 42.. We observe consistent improvement over ERM, from 50.4 to 55.4 for ST-GCN and from 56.3 to 64.3 for CTR-GCN is shown in Table This shows that the domain gap caused by temporal mismatch is difficult to be mitigated by only designing network architecture itself.
Conclusion
In this paper we present a weweer-nndrreaappli augmentation approach deal with the single domain generalization problem for skeleton action recognition. By exploring the complete action prior, we recover complete actions with learned boundary poses and global linear transforms via clustering. Experiments the future, we plan investigate more effective resampling approaches, e.g. positional encoding, to further incorporate temporal resampling information into our whole framework.
cross-domain setting with three datasets validate our framework. In
10

---

Acknowledgements
We thank all the reviewers for their useful suggestions. This work was supported by the National Natural Science Foundation of China (62220106003), the Research Grant Beijing Higher Institu- tion Engineering Research Center, the Esinghua-Tencenn Joint Laboratory for Internet Innovation Technology, and the Tsinghua University Initiative Scientific Research Program.
References
[1] R. Bellini, Y. Kleiman, and D. Cohen-Or. Dance to the beat: Synchronizing motion to audio. Computational Visual Media, 4:197-208, 2018.
[2] Y. Bin, X. Cao, X. Chen, Y. Ge, Y. Tai, C. Wang. J. Li, F. Huang, C. Gao, and N. Sang. Adversarial semantic data augmentation for human pose estimation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pages [3] K. Cao, J. Ji, Cao, C.-Y. Chang, and Niebles. Few-shot video classification via temporal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10618- 10627. 2020.
Springer, 2020.
[4] F. M. Carlucci, D'Innocente, Bucci, B. Caputo, and T. Tommasi. Domain generalization by solving jigsaw puzzles. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 2229-2238, 2019. 2019.
[5] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1299-6300
2017.
[6] Y. Chen, Z. Zhang, C. Yuan, B. Li, Y. Deng, and W. Hu. Channel-wise topology refinement graph convolu- tion for skeleton-based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1339911336 2021.
[7] A. Davydov, A. Remizova, V. Constantin, S. Honari, M. Salzmann, and P. Fua. Adversarial parametric
pose prior. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10997-1100s 2022.
[8] H. Duan, Y. Zhao, K. Chen, D. Lin, and B. Dai. Revisiting skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2969-29788 2022.
[9] Gholami, B. Wandt, H. Rhodin, R. Ward, and 7. J. Wang. Adaptpose: Cross-dataset adaptation for 3d human pose estimation by learnable motion generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13075-13085, 2022.
[10] K. Gong, Zhang. and J. Feng. Poseaug: A differentiable pose augmentation framework for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8575-8584, 2021.
[11] T. Guo, H. Liu, Chen, M. Liu, T. Wang, and R. Ding. Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36. pages 762-770, 2022.
[12] D. Holden, T. Komura, and J. Saito. funstionn neural networks for character control. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
[13] S.-M. Hu, D. Liang, G.-Y. Yang, G.-W. Yang, and W.-Y. Zhou. Jittor: novel deep learning framework
with meta-operators and unified graph execution. Science China Information Sciences, 63:1-21, 2020.
[14] L. Huang. Liang, and W. Deng. Dh-aug: Dh forward kinematics model driven augmentation for 3d human pose estimation.
Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23- 23-27, 2022, Proceedings, Part VI, pages 336-453. Springer, 2022.
[15] J. Jang, D. Kim, C. Park. M. Jang, J. Lee, and J. Kim. Etri-activity3d: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10990-109977. IEBE, 2020.
[16] W. Jiang, S. Jin, W. Liu, C. Qian, P. Luo, and S. Liu. Posetrans: A simple yet effective pose transformation augmentation for human pose estimation. Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. pages 643-659. Springer, 2022.
[17] K. Karunratanakul, K. Preechakul, E. Aksan, T. Beeler, S. Suwajanakorn, and S. Tang. Optimizing diffuusin noise
universal motion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1334- 1345, 2024.
[18] M. A. Khan, K. Javed, S. A. Khan, T. Saba, U. Habib, J. A. Khan, and A. A. Abbasi. Human action recognition using fusion of multiview and deep features: an application to video surveillance. Multimedia tools and applications, 18855514885-14911 2024.
[19] M. Kocabas, N. Athanasiou, and M. J. Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 5253 5263, 2020.

---

[20] T. Kwon, B. Tekin. S. Tang, and M. Pollefeys. Context-aware sequence alignment using 4d skeletal augmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8172-8182, 2022.
[21] C. Li, Q. Zhong, D. Xie, and S. Pu. Co-occurrence feature learning from skeleton data for action recognition and detection with hierarchical aggregation. Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 786-792. 2018.
2018.
[22] I.. Li, K. Gao, Cao, Z. Huang. Y. Weng. X. Mi, Z. Yu, X. Li, and B. Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 224-233, 2021.
[23] L. Li, M. Wang, B. Ni, H. Wang, J. Yang. and W. Zhang. 3d human action representation learning via cross- viess consistency pursuit. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4741-4750, 2021.
[24] L. Lin, J. Zhang, and Liu. Actionlet-dependent contrastive learning for unsupervised skeleton-based ac- tion recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 2363-2263333 2272,,23,22222 2023.
[25] C. Liu, Y. Hu, Y. Li, S. Song, and J. Liu. Pku-mmd: large scale benchmark for continuous multi-modal human action understanding. arXiv preprint arXiv:/703.07475. 2017.
[26] H. Liu, Y. Liu, T--J Mu, X. Huang, and S.-M. Hu. Skeleton-cutmix: Mixing up skeleton with probabilistic
bone exchange for supervised domain adaptation. IEEE Transactions on Image Processing, 2023.
[27] H. Liu, X. Zhan, S. Huang. T.-J. Mu, and Y. Shan. Programmable motion generation for open-set motion control tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1399-1408, 2024.
[28] Q. Liu and Z. Wang. Collaborating domain-shared and target-specific feature clustering for cross-domain 2022: 17th European Conference, Tel Aviv, Israel, October 23 27. 2022, Proceedings, Part IV. pages 137-155. Springer. 2022.
3d action recognition. In Computer
[29] Z. Liu, H. Zhang, Z. Chen, Z. Wang, and W. Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 143-152, 2020.
[30] C. Lu, R. Li, H. Fu, B. Fu, Y. Wang, W.-L. Lo, and Z. Chi. Precise temporal localization for complete actions with quantified temporal structure. In 2020 25th International Conference on Pattern Recognition (ICPR). pages 4781 1-788. IEEE, 2021.
[31] T. Maeda and N. Ukita. Motionaug: Augmentation with physical correction for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (427-6436, 2022.
[32] Z. Marinov, A. Roitberg, D. Schneider. and R. Stiefelhagen. Modselect: Automatic modality selection for synthetic-to-real domain generalization. In Computer Vision-ECCV 2022 Workshops: Tel Aviy, Israel, October 23-27, 2022, Proceedings. Part VIII. pages 326-346. Springer. 2023.
[33] R. Memmesheimer, S. Häring, N. Theisen, and D. Paulus. Skeleton-dml: Deep metric learning for skeleton- based one-shot action recognition. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 3702-3710. 2022.
[34] F. Meng. H. Liu, Y. Liang, J. Tu, and M. Liu. Sample fusion network: An end-ta-end data augmen- tation network for skeleton-based human action recognition. IEEF Transactions Image Processing. 2019.
28(11):5281-522
[35] S. Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision, pages 2017.
[36] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan, Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[37] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer.
R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[38] M. Petrowich, M. J. Black, and G. Varol. Action-conditioned 3d human motion synthesis with transformer vac. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10985-10995 2021.
[39] A. R. Punnakkal, A. Chandrasekaran, N. Athanasiou, A.
and M. J. Black. Babel: bodies. action and behavior with english labels. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 722-731, 2021.
[40] N. A. Rahmad, M. A. As' An. N. E. Ghazali, N. Shahar, and N. A. J. Sufri. A survey of video based action recognition in sports. Indonesian Journal of Electrical Engineering and Computer Science, 11(3):987-90 993, 2018
12

---

[41] A. Shahroudy, J. Liu. T.-T. Ng. and G. Wang. Nuu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010-1019, 2016.
[42] L. Shi, Y. Zhang, J. Cheng, and H. Lu. Two-stream adaptive graph convolutional networks for skeleton- based action recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12026-12035, 2019.
[43] S. Starke, I. Mason, and T. Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions Graphics (TOG), (14:) 2022.
[44] S. Starke, Y. Zhao, Komura, and K. Zaman. Local motion phases for learning multi-contact character movements. ACM Transactions on Graphics (TOG), 39(4):4--, 2020.
[45] B. Su and J. Wen. Temporal alignment prediction for supervised representation learning and fow-shot sequence classification. In International Conference on Learning Representations, 2021.
[46] Y. Tang, X. Liu, X. Yu, D. Zhang. J. Lu, and J. Zhou. Learning from temporal spatial cubism for cross- dataset skeleton-based action recognition. ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 18(2):1-24 24, 2022.
[47] F. M. Thoker, H. Doughty, and C. G. Snock.
3d action representation learning. In Proceedings of the 29th ACM international conference on multimedia, pages 1655-1663, 2021.
[48] G. Tiwari, D. Antić, E. Lenssen, N. Sarafianos, T. Tung, and G. Pons-Moll. Pose-ndf: Modeling human
pose manifolds with neural distance fields. Computer Vision-ECCV 2022: 17th European Conference,
Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V, pages 572-589. Springer, 2022.
[49] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research,
911). 2008.
[50] A. Veeraraghavan, A. Srivastava, A. K. Roy-Chowdhury, and R. Chellappa. Rate-invariant recognition of humans and their activities. IEEE Transactions on Image Processing, 1866):33661133, 2009.
[51] R. Volpi, H. Namkoong. O. Sener, J. C. Duchi, V. Murino, and S. Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018.
[52] H. Wang. F. He, Z. Peng. T. Shao, Y.-L. Yang. K. Zhou, and D. Hogg. Understanding the robustness of skeleton-baser based action recognition under adversarial attack. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14656-14665, 2021.
[53] L. Wang and P. Koniusz. Uncertainty-dwv for time series and sequences. In European Conference on Computer Vision, pages 176-195. Springer, 2022.
[54] L. Wang. Y. Xiong. Z. Wang. Y. Qiao, D. Lin, X. Tang, and L. Van Gool. Temporal segment networks:
Towards good practices for deep action recognition. In European conference on computer vision, pages
20- 30- Springer, 2016.
[55] 7. Wang. Y. Luo, R. Qiu, 7., Huang, and M. Baktashmotlagh. Learning to diversify for single domain generalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages
834-843, 2021.
[56] S. Xu. H. Rao, X. Hu. Cheng, and Hu. Prototypical contrast and reverse prediction: Unsupervised
skeleton based action recognition. IEEE Transactions on Multimedia, 2021.
[57] S. Yan. Y. Xiong, and D. Lin. Spatial temporal graph convolutional networks for skeleton-based action
recognition. In Thirty-second AAAI conference artificial intelligence, 2018.
[58] 7. Yang, W. Zhu, W. Wu, C. Qian, Q. Zhou, B. Zhou, and C. C. Loy. Transmomo: Invariance-driven unsupervised video motion retargeting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5306-5315, 2020.
[59] Q Zeng, C. Liu, M. Liu, and Q Chen. Contrastive 3d human skeleton action representation learning via crossmoco with spatiotemporal occlusion mask data augmentation. IEEE Transactions on Multimedia, 2023.
[60] C. Zhang, T. Yang, J. Weng, M. Cao, J. Wang, and Y. Zou. Unsupervised pre-training for temporal action localization tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14031-14044 2022.
[61] H. Zhang, M. Cisse, Y. N. Dauphin, and Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv
preprini wrXXi:1110.09444
2017.
[62] J. Zhang, L. Lin, and J. Liu. Hierarchical consistent contrastive learning for skeleton- based action recognition with growing augmentations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3427-3435, 2023.
[63] S. Zhang, Y. Zhang, F. Bogo. M. Pollefeys, and S. Tang. Learning motion priors for 4d human body capture 3d scenes. In Proceedings of the IEEE/CVE International Conference on Computer Vision,
pages 11343-115 1133,, 2021.
[64] Y. Zhao, Y. Xiong, L. Wang, Z. Wu, X. Tang, and D. Lin. Temporal action detection with structured segment networks. In Proceedings of the IEEE international conference on computer vision, pages 2914 2923,,,, 2017.
[65] Q. Zhou, M. Li, Q. Zeng, A. Aristidou, X. Zhang, L. Chen, and C. Tu. Let's all dance: Enhancing amateur dance motions. Computational Visual Media, 13):531-550, 2023.
13

---

Appendix
A1. Algorithm
The algorithmic description of our recover-and augmentation framework is provided in Algorithm
Algorithm 1 Recover and Resample
imput:
a sample E xx3x with length and joint number J, dataset S, resampling function Wiesample.
coefficient at.
output: an augmented sample z' E [xxxxx3
Cluster background poses {pr} from S. (Sec.
3.3
Cluster linear transforms {W;) from S. (Sec.
Define motion infiller F(.).
Assign p' as the boundary pose for æ from (pi} using Eq. 2
Sample 4p ((xx, cx)T and perform extrapolation æ Fa, .)).
Sample W from (W.) apply Wx.
Wiesample
A2. Description on Datasets and Evaluation Settings
Cross-dataset settings. Here we provide more details for our cross-dataset settings. The shared 18 action classes for STU60-RRGB [1]. PKU-MMD [25] and ETRI-Activity3D [I5] are listed in Table IO We evaluate on four cross- cros-dataset settinet i.c. + E, N + P, E + N and EA We use cross-subject protocol for dividing training and test splits, and the number of training and test samples for each domain is shown in Table TT. For each domain, we use training set for model training and test set for evaluation. For term of use, NTU60-RGBD [41 is free for research and non-commercial use. We submitted a license agreement to ETRI-Acto PFRI-AAtiviyyDD [15] website for downloading the dataset. License for PKU-MMD 225
stated on its official homepage.
Table 10: Action labels used in our cross cross-datased dataset settings.
Action label
eat
drink take off clothes
brush teeth
brush hair read
wear clothes
put on/take off glasses play with phone hug
write bow
phone call
clap
handshake
hand wave
point finger
fall down
Table 11: Number of training and test samples for each dataset split.
Domain
samples in training set
# samples in test set
N
12651
5212 7008
Ea
13568
P
6823
968
26930
13350
P
12017
1736
Evaluation details. For skeleton action recognition, most works 42 report best results on the test set since there is official validation set. We also follow this evaluation protocol for our domain generalization task. We train and evaluate for each sub- sub-setting separately, and use average accuracy for evaluating the overall performance. Moreover, we run each experiment ten times with different random seeds and report mean accuracy to reduce performance oscillation. As the training sets for N and FA (the adult split) are approximately of the same sizes, we choose to use FA for training, which offers fair basis and gives better understanding of the task difficulty for different sub- sub-settings.
14

---

A3. Additional Implementation Details
Linear transform clustering, In order to cluster a set of linear transforms, we construct a set of training pairs, L.c. partial action sequences and full action sequences {(ug, 2사)} from training data S. Although it is possible to randomly sample starting and end points for a sequence e to generate u, we propose to sample from some fixed interval timestamps for simplieity and regularity. Those timestamps are 0.0, 0.125, 0.25, 0.375, 0.5. 0.625, 0.75, 0.875, 1.0. For each training sample v, we sample according to the below pairs of starting and end points.
1.0), (0.0, 0.5), (0.5, 1.0), (0.25, 0.75), (0.125, 0.625), (0.375, 0.875), (0.0, 0.75), (0.25, 1.0)
Resampling stage. We mainly follow the CropResize 47 operation for our resampling stage, which first samples sequence length from uniform distribution U (11, r2) and then randomly samples a temporal segment with this length. Since effective resampling approach is not the primary goal of our paper, we fix in our framework.
Training details. For training AGCN, we follow 42 and use SGD optimizer with an initial learning rate of 0.1 and weight decay of 0.0001. We train the model for 50 epochs. The learning rate is reduced at 30th and 40th epoch with a decay of 0.1. For HCN we use the same training configuration as [46]. Parameter settings for other experiments are listed below. For N12- and N51 N51+ we set N₂ = 10. For N51-P51, Nikg = 10 and for P51-N51, Nikg 5. The boundary poses and linear transforms learned from the corresponding 18-class subset are transferred to N N51+>551 settings for efficiency. Also for N12-KI1,, 12, we set Mang to a smaller value of 0.5 and omit FN due to very high level of noise and missing joints in K12 dataset. We implement our method and other baselines using PyTorch [36] and conduct experiments on a single NVIDIA RTX 2080Ti. We will implementation which supports faster training and inference.
also release code in Jittor
Design of learning variants for comparison. As in 7 for learning to extrapolate (FNNesturp) we train a completion network using masked and full sequences. and during augmentation, we squeeze the original sequence to half of its original size and let the network extrapolate on both sides. We also compare with the design of learning to infill (FNN,innn) instead of performing línear interpolation. Again we mask out a segment in the center of a motion sequence and use paired sequences train an infilling network.
Other baseline methods. Here we provide brief descriptions and more implementation details for other baseline methods.
(1) General domain generalization approaches. CCSA 35 improves generalizability by aligning samples of the same label in the feature space via contrastive loss. For CCSA, the weight for contrastive loss is set to 0.1, and the margin for contrastive loss is set to 1.0. ADA (5I) improves generalizability by augmenting with adversarial samples. ADA uses image-based perturbation in the original paper. We adapt it to the adversarial skeleton perturbation 52 as we find it yields better results. The training set is expanded with adversarial samples at 15th and 30th epoches. For adversarial augmentation we set the semantic regularizer to 1.0, learning rate to 1.0 and optimization steps to 5.
improves generalizability by solving auxiliary jigsaw puzzles. The spatial stream (Spa) and temporal stream (Tem) solves jigsaw puzzles in spatial and temporal dimensions respectively and the final prediction ensembles scores from the two streams. For ST-Cubism, the loss weight for jigsaw puzzle term is set to 1. Skeleton- improves generalizability by learning to reconstruct the input skeletal motion sequence. For Skeleton-MAFE we borrow the masked auto-encoder module and reconstruction loss from (2). and change the backbone to AGCN, During training. we randomly mask out a clip with a length ratio of 0.3 and let the model to reconstruct it. The weight for the reconstruction loss is set to 1.0. HICLR [62] is skeleton representation learning method, which improves generalizability by pre-training a feature encoder via contrastive learning. For HICLR, we follow their supervised evaluation by first pre-training the encoder and then performing standard training using the training set. We adapt the backbone to AGCN well.
(2) Self-supervised learning approaches. ST-Cubism 46
(3) Augmentation- based approaches. Mixup 61 generates augmentations by linearly interpolating between a pair of samples as well as their labels. For Mixup, training samples are augmented with Mixup with a probability of p- = 0.5. CropPad [23] randomly crops a segment from the motion sequence and performs reflective padding on both sides. For CropPad, we use the crop ratio y 6.
15

---

CropResize 47 randomly crops segment from the motion sequence and resizes it to its original length. For CropResize, we randomly sample segment with length ratio between 0.7 and 1.0, which we find works the best. We also set the resampling process in our method the same as this baseline. For CropPad, CropResize, and spatial augmentations, we set May 0.75 which the same as ours. Uniform sampling [8] resizes sequence by first uniformly dividing the input sequence into segments and then randomly sampling one frame within each segment. Such sampling method can make full use of all frames in the raw motion sequence and therefore generate strong augmentations.
(4) Aligament-based approaches. We use the feature encoder from the model trained with standard cross coos-entropy loss (ERM), and transform raw skeleton sequences into downsampled feature sequences (16 frames for each sample). We then use kNN (k 5) classifier [53] with warping- based distance OTAM [3] which is representative warping- varping-based based distance measure used in few-shot video classification that deals with temporal mismatch. The final result comes from the ensemble of OTAM+NNN and the ERM model. We find that using distance in the raw joint space is inferior than the feature space. Note that we implement temporal alignment method without further pre-training the feature encoder. Please refer to self-supervised learning for feature pre-training approaches.
(5) Aggregation-based approaches. We build TSN SE upon the AGCN backbone. The input sequence is uniformly divided into 3 clips before feeding to the backbone and we aggregate their features using average pooling. We implement multiple-crop testing as a post-proccess poottprocsssing method. Given test sequence we sample 5 clips with length ratios of 0.5 and 0.75 and average the prediction for these clips altogether with the original sequence. The starting and end point pairs of the sampled clips are (0, 0.5), (0.25, 0.75], (0.5, 1.0), [0, 0.75], (0.25, 1.0] in our experiments.
A4. Additional Experimental Results
Detailed results for ablation study. Table 12 presents results for the experiment of parameter analysis in the main paper. We find that \y is generally not very sensitive, as choosing 'y over a large range (0.001 1) does not significantly affect the average accuracy. Table 13 provides detailed results for the per-class performance.
Table 12: Effect of coefficient by in context- aware linear transforms.
ÀT
N E
N P
EA +
N
EΛ + P
Avg.
ERM
54.9
70.5
42.4
49.7
54.4
58.0
73.5
48.0
58.3
59.4
0.1
58.4
75.8
48.4
57.8
60.1
0.01
58.0
73.3
48.5
58.4
59.5
0.001
58.1
73.7
47.9
57.4
59.3
Table 13: Per-class results of our proposed method compared to ERM.
Actions
N E
N P
Ea + N
Σ P
Avg.
Phone calling
20.8(+5.7)
49.6(+49.1)
32.7(+24.5)
51.8(+22.0)
38.8(+25.3)
Hand waving
80.8(+21.5)
96.8(+7.7)
41.8(+21.1)
65.9(+41.7)
71.3(+22.1)
Clapping
58.8(+23.1)
90.0(+5.4)
26.9(+14.4)
43.2(+17.2)
54.7(+15.0)
Pointing finger
94.0(+2.7)
100.0(40.0)
77.6(422.9)
98.2(+13.1)
92.5(+9.7)
Takinn off clothes
98.5(+1.3)
93.1(+2.9)
70.9(+5.3)
89.6(+2.7)
88.0(+3.1)
Feature representations. We measure the similarity of two skeletons using L2 norm in the joint coordinate space. However, it is also possible that we measure it in the latent space. Here we investigate the performance difference between these two feature representations. Specifically, we train a skeleton auto-encoder with MLP using training data and use it to obtain latent feature for the skeleton each frame. For FN we perform boundary pose clustering in the latent feature space, and for Fr. we calculate similarity matrices using latent feature. As in Table we observe lower but very close performance if using latent feature. This shows that our method does not depend on specific skeleton representations.
16

---

Table 14: Comparison between different skeleton representations for our method.
Representation
N -
E
N
P
N
Σ + P Avg.
ERM
54.9
70.5
42.4
49.7
54.4
Ours (joint space)
58,4
75.8
48.4
57.8
60.1
Ours (latent space)
58.4
74.9
48.3
57.6
59.8
Table 15: Comparison between different clustering algorithms.
Method
N->
N-PP
EA-NN
Ex-P
Avg.
Ours (k-means)
58.4
75.8
48.4
57.8
60.1
Ours (agglomerative)
58.5
74.7
48.0
56.7
59.5
Table 16: Comparison between different resizing strategies and segment lengths in the resampling stage. denotes the range of length for sampled segments.
Resizing
range r
N-
N-
Ex N
EX-P
Avg.
linear
[0.3,1.0]
57.4
71.5
47.9
56.2
58.2
linear
[0.5,1.0]
58.4
74.7
48.1
56.3
59.4
linear
[0.7,1.0]
58.4
75.8
48.4
57.8
60.1
random
[0.7,1.0]
57.9
75.6
48.3
57.1
59.7
Evaluation on different clustering algorithms. We investigate whether our method is compatible with different clustering algorithms. We conduct experiments using agglomerative clustering [37], which another type of clustering algorithm different from k-means. The result is in Table [5] We see that the average accuracy is lower but still close and outperforms other baseline methods, which shows that our method is generally not sensitive to choice of clustering algorithm. experiment, we keep may be further improved by tuning more suitable parameters.
(e.g. number of clusters) the same as k-means. The result
Evaluation on resampling methods. For our experiments CropResize is used for the resampling stage. In Table T6] we investigate the length of segments too short segments may introduce many noisy augmentations and hampers the overall performance. To resize the sampled segment to fixed length sequence, linear and random frame sampling are also investigated. The results are shown to be generally comparable.
key parameter. We find that sampling
Evaluation standard in-domain action recognition. Although the proposed method mainly deals with cross-dataset settings, we find it can also work as an augmentation strategy for standard in-domain skeleton action recognition with training and testing within the same dataset. In Table 17 we evaluate on the in-domain recognition on NTU-18 subset and ETRI Adult- 18 subset from our constructed cross-dataset settings, well on the standard full NTU-60 dataset [41). We adopt full AGCN backbone for N-18 and EA-18, and use full 2s-AGCN 42 for NTU-60, We set a smaller weight for augmentation Mang 0.25. The performance improvement of our method comparable to uniform sampling [8]. which is considered an effective temporal augmentation. Note that for cross-dataset settings we surpass uniform sampling by 4.4% (see Table [ since uniform sampling only samples the observed sequence. On the other hand, the fact that our improvement on N-18 and E1 is much smaller than cross-dataset settings indicates that the training and test split of dataset are generally well aligned. Our cross- setting magnifies the issue of temporal mismatch and presents a more practical setting as in real-world applications.
Table 17: Result of method for standard skeleton action recognition.
Method
N-18
Ea-18
NTU-60
ERM
86.0
89.2
88.7
Uniform Sampling S
86.5
90.3
89.4
Ours
86.8
90.1
89.5
17

---

Table 18: Comparison between spatial and temporal augmentations.
Method
N
E
N
P
ΣA
+
N
ΣA + P Avg.
ERM
54.9
70.5
42.4
49.7
54.4
Shear 23
54.9
69.7
42.6
49.2
54.1
Bone scaling
58
54.6
68.7
42.6
50.0
54.0
Translation
55.2
68.8
43.3
50.3
54.4
Ours (temporal)
58.4
75.8
48.4
57.8
60.1
Table 19: Standard deviation of cross-domain action recognition.
Method
Avg. accuracy
Std
ERM
54.4
0.8
Ours
60.1
0.6
Spatial augmentation v.s. temporal augmentation. In Table 18 we examine how spatial augmen- tation and temporal augmentation contribute to the improvement of generalizability, as they both prove to be effective in the self-supervised skeleton representation learning (23 We choose typical
spatial augmentations including shear transform [23, bone length scaling 158 and Gaussian noise
addition on joints [1]. To compare with spatial augmentations, we set shear ratio 23) 3 0.1, and set the translational noise with a normal distribution N (0, 0.01). . For bone scaling, we randomly scale each bone with length ratio between 0.9 and 1.1. We demonstrate that method, in the form of temporal augmentation, is more effective than direct spatial augmentation. This shows that direct spatial transformation is more likely to change the semantics, especially for hand parts. Domain differences in human actions are easier to be handled the temporal dimension than in the spatial dimension.
More visualizations for Fr., Fig. S we visualize the clustered linear transforms when N, = 20 and N,, 5 using training set N. We find that small number of Nr (e-g- N₂r = 5) will hurt the diversity of linear transforms. such a case, is difficult to extract some important temporal action patterns (e.g. temporal shift and symmetry) via clustering.
Error bars. The domain generalization task normally brings the issue of performance oscillation. In Table 19 we report 10-run standard deviation for the average accuracy in our cross-dataset settings. We show that our improvement statistically significant.
Qualitative results. In Fig. [we we provide some qualitative examples showing our improvement over the baseline (ERM). With our proposed augmentation by action completion, we enable more accurate action recognition for long and full sequences when observing from partial action sequences.
A5. Additional Analysis on Complete Action Prior
Experimental details for discovering complete action prior. Each input skeleton is flattened to vector with a dimension of 75 (joint number J = 25). We design a simple skeleton auto-encoder consisting of six fully-connected layers and ReLU activation. The output dimensions for the fully- connected layers 128, 64, 32, 64, 128, 75, respectively. The L2 norm loss is adopted to train the auto-encoder in order to reconstruct the original input skeleton. During training, we sample frames from processed skeletal data (uniformly resized to 64 frames and the first frame is aligned to coordinate) as training data.
We train the auto-encoder separately for each dataset (PKU-MMD 25). NTU60-RGBD 4] and -TTII-Acivity3DD (I5) using data our 18-class cross-dataset settings. We then use the trained model to transform all skeleton frames into latent features (extracted from the third layer with dimension of 32). The feature diversity of dataset S at a certain timestamp the standard deviation of the obtained latent features 2(-) averaged over all feature dimensions E C and all skeleton
18

---

Beginning frames (NTU)
Central frames (NTU)
Visualization of beginning and central frames that are sampled from NTU split in our 18-class cross-dataset setting. The beginning frames which are usually rest poses have less diversity, while the central frames which contain rich action semantics usually have more diversity.
Figure 7:
sequence samples a E S.
Diversity(S,1)
1
E((((r))- - #(wz)c)²
(6)
Ns #ES LEC
where (xxy)) is the mean of all latent features =(x,) for x E S, Ns is the size of dataset S and Ne is the number of latent feature dimensions.
More analysis on the statistical pattern. We provide more analysis for curve patterns in Fig. (b). For PKU and NTU, as they are captured in labs and are generally clean with less occlusion and noise, they are of relatively low diversity and exhibit stronger level of complete action prior, For ETRI, as is captured in home environment and is more challenging with larger noise, the frame-wis diversity would be higher. And as it captured in real life with action boundaries that are hard to determine when collecting the dataset, the complete action prior is not as significant as NTU and PKU dataset. Fig. 7 shows examples of beginning frames and central frames for the NTU split in our cross-dataset settings, which is consistent with the statistical finding in Fig. (b).
Also, we see that the global pattern differs across different datasets, for example, in the starting and end point diversity. For PKU and ETRI, they exhibit an overall temporal pattern of being symmetric. In contrast, the NTU dataset does not have such a symmetric pattern. There is also shift for the peak diversity across datasets. The above differences are majorly caused by different definition of when an action starts and ends in different datasets, as can also be visualized in the Fig. (a) example. These findings suggest that the domain gap between datasets, represented as temporal mismatch, may involve temporal shift, scaling and even symmeety/non-symmeet properties arising from the partial/full bias of human actions.
Note that the two observations presented in the main paper that many human actions are partial observations and that human actions generally exhibit completeness within a large group of samples are not contradictory. This can be best explained by that (1) within a large dataset, some action categories or samples are nearly complete while some other action categories or samples are partial segments of complete actions, and (2) even if the actions are not exactly complete, e.g. only comprising central segments, many of them obey similar statistical pattern which is only less significant. Here the "completeness" is the opposite of having uniform diversity over time. Also keep in mind that the curves in Fig. (bb only reflect the statistical pattern for group of actions in a dataset. For example, although the curve for NTU seems non-symmetric, we can still have approximate symmetry operations in the clustered linear transforms (see Fig. 88. This can be best explained by that there exist action samples/categories that are symmetric in the NTU split, but they only account for a small portion.
Other implications on the proposed method. (1) The complete action prior generally relies on the assumption of clean skeleton data and daily routine pattern of human activities. Relying on this prior, our method is unable work well when the source data is very noisy and contains very long motions with multiple stages, for example, taking K12 as source data. We find that K12 does not exhibit such strong prior as is observed Fig. 1(b). (2) When the target domain does not show strong action completeness or contains high level of noise, the extrapolation module Fx may not have very strong effect. This is partially shown in N and N12-K12 settings. This also explains our finding that using linear transform Fz is generally sufficient for N12-<K12.
19

---

A6. Discussion
Discussion for other baseline methods. We discuss some possible reasons that the alignment-basee method is not as competitive as ours in the cross dataset settings: (1) Given sufficient training data, the nearest neighbour classifier only memorizes training samples compared to network classifier and this in turn hampers its generalizability. Contrastive pre-training for the feature encoder with DTW-based loss may lead to better performance, but at a cost of much more computation. (2) Direct matching via kNN cannot fully take advantage of various types of augmentations, such as random rotation. (3) A better design of dynamic time warping is needed since there is domain shift across training and test datasets for either raw joint features or latent features. In widely-studied few-shot scenarios 3 53], the training and test data normally come from the same distribution. For aggregation- based method, clips sampled without action completion cannot well handle the partial/full bias. However, these methods and our proposed method are not contradictory and can be combined for further improvement.
Stochastic action completion. In this paper we actually propose a stochastic action completion method that can be used to augment the training data. This stochastic action completion is more suitable for augmentation than deterministic one. Note that FN has the probability to retain its original motion due to our sampling strategy for length of extrapolation lp. Fr. also learns identical mapping as is shown in the learned transform matrices (diagonal matrix). In this way, even for the imput of complete action, our approach is still able to output reasonable complete action by sampling an identical mapping. A supporting evidence is that for P51-N551 (P51 consists of generally complete actions), our method still improves the baseline ERM and even outperforms the temporal module of ST-Cubism [46]
Number of new parameters. During model training, the newly added parameters are two matrices, i... the boundary pose clusters of shape (Nakg. J) and the linear transform clusters of shape (Nur, T, T). Here Nokg 10. J 25, N₂r = 20, T = 64. So the number of new parameters of these two matrices ~ 90000, which less than two FC layers (suppose each layer is (2..inr((2666256) For other baseline methods, universal domain generalization methods CCSA (35. ADA [5T] and handcrafted augmentation methods such as uniform sampling [8) Mixup 611. CropPad 223 and CropResize (47) do not introduce new parameters. supersupe learning methods often have new branch along with the original network for learning auxiliary tasks, therefore introducing new parameters (usually several FC layers). So generally, our method and those self-supervised methods have a similar magnitude of new parameters.
Limitations. The limitations arising from the complete action prior are discussed in Appendix A5. Other implications on the proposed method. Moreover, we note that direct resampling from a complete action sequence might introduce ambiguity for some actions. For example, standing up and sitting down are two different actions, but both of them can be obtained by sampling from a complete action sequence in which a human stands up from sitting pose and then sits down. However, one should note that this problem also exists when constructing positive samples in contrastive self-supervised learning [T] 47), as resampling reversing some action categories. To deal with this problem, we plan to add positional encoding and temporal masks in the resampling stage to further encode temporal information about how the sequence is sampled. We leave it for future work.
action may change the semantics for
Broader societal impact. Generally we don't find that our method has negative societal impacts. However, users should be careful about the result if action recognition along with our method is applied to critical human-related applications such as health monitoring and elderly fall detection. Current deep learning-basse based models cannot guarantce absolutely accurate action recognition.
20

---

N 20
(b) Nor 5
Figure 8: Visualization for the clustered linear transform matrices {W) using training set N when
Nir 20 and N₂- 5.
Baseline: write
Ours: clap
Baseline: put on/take off glasses
Ours: clap
Baseline: put on/take off glasses
Ours: phone call
Baseline: brush teeth
Ours: point finger
R
Baseline: brush hair
Ours: hand wave
Figure 9: Qualitative examples showing improvement of method the baseline (ERM).
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A6. Discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Not applicable.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Section 3, Section and Appendix A.3 for method, experiment settings and implementation details.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
Code link is available on the first page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips.com wttp:ccpppbccccceec guwde//ddeeeeee
siinisiiccyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: See Section 4 and Appendix A.3 for experiment settings and implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See Appendix A4. Additional Experimental Results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix A3. Additional Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://uwrrp..ccpppp...cccaaaaaa
Answer: [Yes]
Justification:
This research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix A6. Discussion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix A2 and A3.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "8skLR0qFuK", "reviewer": "Reviewer_UxEw", "summary": "In this paper, the authors address the issue of generalizing skeleton-based action recognition across different domains. They propose a novel recover-and-resample augmentation framework based on the concept of complete action prior. The approach is validated on different cross-dataset settings and demonstrates a significant improvement in cross-dataset accuracy compared to existing methods.", "questions": "N.A.", "limitations": "N.A.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Introduces a new recover-and-resample framework that effectively addresses temporal mismatch in skeleton action recognition.\n\n2. Utilizes the concept of action completeness within large datasets, employing boundary poses and linear transforms to capture global action patterns.\n\n3. The experimental results on different cross-dataset settings outperform the previous methods.", "weaknesses": "Although the method is intriguing and the performance is impressive, I am content with the current experimental setting. In real-world applications, dealing with cross-dataset issues makes it challenging to ensure that both datasets share the same action categories. Currently, the shared classes are manually selected. In my opinion, it would be better to adopt the new setting proposed by [1], where the source and target datasets have a category gap.\n\n[1]   Collaborating Domain-shared and Target-specific Feature Clustering for Cross-domain 3D Action Recognition. ECCV 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UxEw:\n\nThanks for your effort for reviewing our paper and giving kind suggestions. \n\n[Q1] I am content with the current experimental setting\n\n[A1] Do you mean you are not content with the current experimental setting? Since the generalizability of skeleton-based action recognition is still less explored in the community, we focus on the task of domain generalization, which is the most formal and standard setting where source and target datasets share the same action categories. Our reason for adopting this setting is that we can put every effort on investigating the domain gaps in skeleton-based action recognition in a more fundamental way. Following such a goal, we explored the inherent nature of action completeness for improving generalizability.\n\nIn the new setting proposed by [1], where the source and target datasets have a category gap, useful information from large amount of unlabeled data can be mined. However, its final prediction is unable to give the actual action label in a straightforward way, especially for those unseen categories. Note that [1] needs to solve the label assignment problem to obtain the accuracy metric. It is more of a representation learning or a clustering problem. As a result, such a setting is also limited in its practical use.\n\nIf considering another new setting where the source and target datasets have a category gap and the model has to predict the exact action labels for unseen target actions, it would become a zero-shot setting for unseen categories and more information about the category itself needs to be considered.\n\nSo basically our standard setting and the new setting [1] address the generalization problem in different aspects. It is hard to say one setting is overwhelmingly better than the other. Hence, in our opinion, adopting a standard setting of domain generalization should not be blamed as core weakness of the paper. \n\n\n[1] Collaborating Domain-shared and Target-specific Feature Clustering for Cross-domain 3D Action Recognition. ECCV 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YlDjKhDMWw", "reviewer": "Reviewer_V13L", "summary": "This paper proposes a novel recover-and-resample augmentation framework to address the skeleton action generalization problem across different datasets. The framework utilizes a complete action prior to recover full action sequences from partial observations, employing boundary pose-conditioned extrapolation and smooth linear transforms. The proposed method demonstrates superior performance in cross-dataset settings compared to existing domain generalization approaches.", "questions": "No further questions.", "limitations": "Tackled in the draft.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": ". Innovative Framework: The recover-and-resample augmentation framework is a novel approach to tackling the skeleton action generalization problem by focusing on recovering complete actions.\n\n. Comprehensive Evaluation: The method is thoroughly validated across multiple datasets, showing significant improvement over baseline methods.\n\n. Efficient Learning: The use of clustering for learning boundary poses and linear transforms makes the framework efficient and scalable.\n\n. Powerful generalizability on cross datasets.\n\n. Utilize nature concept of skeleton-based dataset for generalizability.\n\n. Varied method suggestion: clustering concept and linear transformation algorithms", "weaknesses": ". Limited Scope: The evaluation is primarily focused on indoor datasets, which may limit the generalizability of the findings to other types of datasets or real-world applications.\n\n. Complexity in Implementation: The two-step stochastic action completion and the need for clustering might pose implementation challenges for practitioners.\n\n- Dependency on Clustering Algorithms: The approach heavily relies on clustering algorithms for learning boundary poses and linear transforms. The effectiveness of the method may be influenced by the choice of clustering algorithm and the parameters used, which might require extensive tuning and could impact the reproducibility and scalability of the approach. \n\n- Request more ablation experiments on diverse sampling and clustering methods.\n\n- Limited Exploration of Resampling Techniques: While the paper proposes a robust framework for recovering and resampling action sequences, the resampling techniques themselves are relatively simple (random) and not extensively explored. More advanced or varied resampling strategies could potentially further enhance the augmentation process and improve generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer V13L:\n\nThanks for your effort for reviewing our paper and giving kind suggestions. We really appreciate your positive comments on our paper.\n\n[Q1] Dependency on Clustering Algorithms: The approach heavily relies on clustering algorithms for learning boundary poses and linear transforms. The effectiveness of the method may be influenced by the choice of clustering algorithm and the parameters used, which might require extensive tuning and could impact the reproducibility and scalability of the approach.\n\n[A1] (1) We use k-means, which is one of the simplest clustering algorithms, and we conducted experiments for important parameters in the clustering process. (2) The insight behind is that we need to mine sufficient and important patterns for linear temporal transforms, e.g. scaling, shifting, reflection. So the number of linear transform clusters is important (Table 5), at least not to be set too small. (3) The number of clustered boundary poses is generally not very sensitive (Table 6). \n\n[Q2] Limited Exploration of Resampling Techniques: While the paper proposes a robust framework for recovering and resampling action sequences, the resampling techniques themselves are relatively simple (random) and not extensively explored. More advanced or varied resampling strategies could potentially further enhance the augmentation process and improve generalizability.\n\n[A2] (1) Yes, indeed we use very simple resampling method in our paper, namely randomly cropping a segment and then resampling it to remain the original sequence length. Despite its simplicity, it generally well addresses the case that many actions are only segments of their full sequences. (2) The whole recovering and resampling process already introduces some level of redundancy, since the recovering stage and resampling stage are both stochastic. (3) In limitation part we discussed the potential of more advanced resampling strategy in addressing ambiguity issue, but this seems to be a common challenge for all the augmentation-based methods. (4) Currently the resampling is done uniformly on randomly selected segments. We will try non-uniform sampling on random segments.  We really appreciate it if the reviewer is willing to offer some more insights on what kind of resampling method might improve the results. \n\n[Q3] Request more ablation experiments on diverse sampling and clustering methods.\n\n[A3] Thanks for the suggestion. In the revision we plan to add DBSCAN as another clustering algorithm to check the reproducibility, and add non-uniform resampling of random segments as another resampling method to see whether it can bring improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "da9xjMjzQl", "reviewer": "Reviewer_Vnjk", "summary": "The work proposes a novel recover-and-resample augmentation framework for domain generalization with application to skeleton-based action recognition. The authors aim to tackle a specific issue when moving from one dataset to another, i.e. the temporal misalignment of actions of the same class. In the experimental analysis, they provide comparisons with existing approaches and ablation studies.", "questions": "I would ask for clarifications on the following points\n\n1.Introduction\n\n- What do you mean by \"It is actually a form that humans perform generally complete actions within large datasets\"? If this is the case, what's the importance of alignment?\n- \"Although this prior can be detected by statistics in a general sense, in terms of individual samples, some exhibit strong action completeness while some are segments of their complete actions\". \n- On Fig. 1(b): the behaviour of NTU and PKU seems different from the one of ETRI (more variability in the initial poses for the latter), while in your comments ETRI is treated on par with the other two. \n- \"By studying the relationships between their raw and trimmed pairs, we can learn a temporal patterns inherited in human actions.\" \n- You named part of the method \"Complete action prior\" but the term seems never used in the next sections\n- In addition to these questions, a more general comment is that the introduction fails to convey the message, which should be a clear and convincing description of the methodology.  The picture does not fully help the understanding\n\n2.Related Work\n\n- Some sentences are not well justified, for instance, \"These approaches partially improve the generalizability across datasets but are still bounded by the specific augmentation design.\" or \"...but they do not make full use of the skeleton representations\". An intuition of the meaning of statements of this type and in what sense the proposed approach is better would be beneficial\n\n3.Method\n\n- \"On the other hand, the statistical finding that the initial poses of skeletal action sequences have low feature diversity (shown in Fig. 1 (b)) also validates our assumption: the boundary poses are to some extent constrained and similar to each other.\" This does not seem true for ETRI\n- If in the training dataset the actions are all starting from the rest position (so they are all aligned) how the proposed method helps dealing with misalignment when changing the dataset?\n- With the extrapolation, is the length T remaining the same?\n- \"Note that the above nonlinear transform is still unable to capture global and structural patterns inherited in human actions\". Why?\n- Why is it useful to reorganize the frames?\n- What are the differences between the proposed work and [17]?\n- Eq. 4: in my understanding k_i should be an index between 1 and T, but from the formula it does not seem so\n- How do you generate the similarity matrix?\n- Why should you cluster the W? The need for clustering is not clear\n- From the loss function, it seems that each training sample is paired with an \"augmented\" sample, which is in contrast with the use of a parameter m_aug\n\n4. Experiments\n\n- A quantification (or at least an idea) of the level of misalignment of the different datasets would be useful for the interpretation of the results\n- Reporting the results with P as training would still be useful as a sanity check\n- The transfer sub-settings used in the experiments are different from [41] (which, according to the authors, is the only one reporting results for cross-dataset settings. Why this choice?\n- What implementations have you used for obtaining the results in the tables? \n- It is that Kinetic, mentioned in the adopted datasets, never appears in the experiments\n- For the NTU datasets, have you considered the cross-view of the cross-subject problem?\n- ERM corresponds to what version of the method?\n- When reporting the per-class results, what about the actions that are not mentioned? It would be useful to have a more general idea \n- The experiment briefly described in \"Generalizability for different backbones\" is actually important and should go in the main paper (discussed more in-depth)\n\nMinor\n\n- I would not mention the use of NNs for the motion infiller in the methodology description\n- The description of the experiment with P51, N51, N12, and K12 should be moved above, with the other details on the transfer settings\n\n[After reading the other reviews and the rebuttal from the authors, I'm happy to increase my score as most of my main concerns have been clarified]", "limitations": "Limitations and potential societal impacts are discussed in the Appendix", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "- The paper considers an important task, related to enriching the size, quality and variability of a dataset to finally improve the generalization abilities of trained models\n- The methodology is modular, with intermediate outputs that can give valuable insights on the interpretation and the possibility of replacing the different tools with alternative choices\n- The experimental analysis is very extensive. The protocol is well described and largely allows for reproducibility", "weaknesses": "- The introduction fails to guide the reader in understanding the motivations, methodologies and challenges. The motivations do not follow clear storytelling, and the description of the methodology is not fully comprehensible. Also, the language would need an improvement (the syntax of some sentences should be fixed)\n- Also the SoA presentation fails to fully convince the reader of the value of the contributions of the proposed approach. The paper is related to different tasks and a more guided tour of the SoA would help to appreciate the contributions. Also, Domain generalization and Data Augmentation are two very important topics for the work, but the SoA discussion is very limited\n- The methodology lacks technical details and appropriate, clear justifications for the different operations. \n- It is not clear if the results are fully reproducible. To my understanding, the experimental protocol is not  used in other papers and there are no details on the implementations that have been adopted, so it is very difficult to judge the results (although the authors say the code will be available upon publications, indications on the adopted implementation would be useful)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Vnjk:\n\nThanks for reviewing our paper. Due to space limit, we answer main questions regarding the soundness of the paper. Feel free to raise further questions.\n\nIntroduction\n \n[Q1.1] What do you mean by \"human performs generally complete actions within large datasets\" and \"in terms of individual samples, some exhibit strong action completeness while some are segments of their complete actions\"?\n\n[A1.1] The two statements are not contradictory, which can be explained by within a large dataset, some action categories/samples are nearly complete and some action categories/samples are partial segments of complete actions. Meanwhile the overall statistics show the non-uniform diversity curve (meaning generally complete actions as opposed to uniform diversity). The insight is that the boundary poses and transforms mined from those complete categories/samples can be used to help perform action completion on those incomplete action ones, so we call them transferable knowledge (line47-48). \n\n\n[Q1.2] Clarify \"the above nonlinear transform is still unable to capture global and structural patterns inherited in human actions\" \n\n[A1.2] Sorry capture should be restore. Take phone calling as an example. Normally, a complete action phone calling consists of raising the arm, keeping hands close to head, and putting down the arm. During recovering, for a NTU sample (Fig 1(a)-NTU), we need to do mirroring so that the action becomes symmetric in time and thus becomes complete. For an ETRI sample (Fig 1(a)-ETRI), we have to first extrapolate the beginning pose to a rest pose, and then do mirroring so that the action ends with a rest pose as well. From above we see extrapolation only is not enough, and we still need global transforms to restore important properties of common complete actions, e.g., symmetry. Nonlinear transform refers to extrapolation here.\n\n[Q1.3] \"By studying the relationships between their raw and trimmed pairs, we can learn temporal patterns inherited in human actions.\" Why reorganizing the frames?\n\n[A1.3] We use a segment of an action (i.e., trimmed sample) to reconstruct the original action (i.e, raw sample) with a linear transform (See Fig.2 u and v). It is an approximation, but in this way we can extract global patterns of human actions. For example, by sampling the first half segment of a symmetric action (e.g. Fig.1(a)-PKU phone calling) and trying to reconstruct the full, we can obtain a transform that functions like a mirroring operation. Note that the mirroring operation is a linear transform which can be characterized by a matrix of shape (T, T), and applying this matrix essentially means re-organizing existing frames since it does not introduce new frames. The learned linear transforms are not restricted to mirroring operation shown in this example, and can be shifting, scaling and mirroring, etc (see Fig.3).\n\n[Q1.4] If in the training dataset the actions are all starting from the rest position (so they are all aligned), … ?\n\n[A1.4] We have both recovering and resampling. During resampling, we randomly sample a segment from the recovered complete sequence, so the starting point of samples in the augmented training data is random and not necessarily the rest pose.\n\n\n[Q1.5] On Fig. 1(b): the behaviour of ETRI.\n\n[A1.5] Please refer to appendix A.5 (line629-632). NTU and PKU are captured in lab, while ETRI is captured at home with more noise and pose diversity. Yet clustering rest poses from beginning frames is still the best solution because they have least diversity.\n\n\n\n3.Method\n\n[Q3.1] With extrapolation, is the length T remaining the same?\n\n[A3.1] Yes. See line 146-147. We squeeze the extrapolated sequence. \n\n[Q3.2] Differences between the proposed work and [17]?\n\n[A3.2] The goal of [17] is to learn alignment of two sequences. We adopt a similar way to find the alignment matrix between trimmed and full sequences. However, their task and our task are essentially different. We use those alignment matrices to augment the existing training data. \n\n[Q3.3] Eq. 4: k_i should be an index between 1 and T, but from the formula it does not seem so\n\n[A3.3] In Eq. 4, s_ij is the weight for each j and j is in the range [1,T], so the value of k_i is in  [1,T]. We have round(k_i) in line 179.\n\n[Q3.4] How to obtain the similarity matrix?\n\n[A3.4] See line 175-177 and Eq. (3).\n\n[Q3.5] Why clustering the W?\n\n[A3.5] (1) The original number of transforms for W is too large, note that n_W = n_training_samples x per_segment_in_one_sample. With clustering, we avoid inefficient sampling of W. (2) Important transform patterns (e.g. mirroring) can stand out during the clustering, as some of them only account for small percentage of the whole pool of W. We will add these explanation.\n\n[Q3.6] About the loss function.\n\n[A3.6] While it is certainly acceptable to have loss exactly as Eq 5, in practice, for efficiency, we randomly take batch_size x m_aug raw samples and batch_size x  (1-m_aug) augmented samples so that the total number of samples in a batch is still batch_size.\n\n4. Experiments\n\n[Q4.1] Reporting the results with P as training.\n\n[A4.1] The result with P as training is provided in Table 3 in P51->N51 setting. \n\n[Q4.2] The transfer sub-settings are different from [41]. Why this choice?\n\n[A4.2] In Table 3 we follow exactly the same setting as [41] for fair comparison. However, [41] only considers mutual shared actions between two datasets. Our new multiple-dataset cross-dataset setting allows us to study and evaluate generalizability in a more fundamental way, so we mainly report results on our new setting (Table 1 and 2). \n\n[Q4.3] Kinetic never appears in the experiments\n\n[A4.3] Kinetics (K12) is used in Table 3. \n\n[Q4.4] How about the cross-view for NTU?\n\n[A4.4] We will discuss it briefly (see Reviewer UadT [Q1,A1]).\n\n[Q4.5] ERM?\n\n[A4.5] ERM refers to training a classifier using backbone network with standard  cross-entropy loss. It serves as a basis for all the methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uk4kL2hGHR", "reviewer": "Reviewer_UadT", "summary": "This paper presents an innovative recover-and-resample augmentation framework to tackle the domain generalization challenge in skeleton-based action recognition. Utilizing the concept of a complete action prior, this method reconstructs entire action sequences from partial observations and resamples them to generate robust augmentations for unseen domains. The approach's effectiveness is confirmed through comprehensive experiments on three skeleton action datasets, showing considerable improvements over existing domain generalization methods.", "questions": "1. Can the authors elaborate more on why domain generalization is critical in the context of skeleton-based human action recognition, despite the inherent reduction in domain gaps provided by 3D coordinates?\n\n2. Why did the authors choose HCN and CTRGCN as the GCN backbones for their approach? Can this be detailed further, perhaps as a subsection in Section 4?\n\n3. Could the authors provide a more detailed explanation of how the approaches in Section 4.3 were implemented for skeleton-based human action recognition methods, including any specific techniques or modifications used?\n\n4. What is the total number of parameters in the proposed approach, and how does it compare to the baseline methods?\n\n5. Can the authors provide t-SNE visualizations of the learned embeddings from different approaches on the test domain to illustrate the effectiveness of the proposed method compared to others?", "limitations": "yes it is in appendix", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Domain generalization of the skeleton based human action recognition is an interesting and unexplored research direction. The authors propose a good test bed for this task which can contribute to the society.\n\n2. The authors propose a new augmentation method which is verified to be effective on various DG settings.\n\n3. Comprehensive ablation studies are delivered by the authors to show the efficacy of the proposed approach.", "weaknesses": "1. Domain generalization is mostly handled via RGB image, since due to the background changes and the camera positions there are obvious large distribution shifts. Regarding the skeleton data, since they are mostly recorded in 3D coordinates without background information, where the domain gap has already been reduced. The authors should justify the need of domain generalization in the skeleton-based human action recognition field with more details in the introduction section. Since in the ultilized datasets, e.g., NTU120, it involves cross view and cross subject evaluation, which can also be regarded as domain generalization.\n\n2.  In Section 4.3, how did the authors implement these approaches on the skeleton-based human action recognition methods? More details are expected to be enriched.\n\n\n3. What is the number of parameters for the proposed approach? The authors are encouraged to make comparison between the proposed approach and the baselines regarding the number of parameters.\n\n4. The authors should provide more clarification regarding why they chose those GCN backbones, e.g., HCN and CTRGCN. Maybe can be described as a small subsection in Sec. 4.\n\n5. TSNE visualization is interesting on the test domain to see the learnt embeddings from different approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UadT:\n\nThanks for your effort for reviewing our paper and giving kind suggestions. We really appreciate your positive comments on our paper.\n\n[Q1] Can the authors elaborate more on why domain generalization is critical in the context of skeleton-based human action recognition, despite the inherent reduction in domain gaps provided by 3D coordinates?\n\n[A1] Thanks for the suggestion. In the revision we will rewrite second paragraph in the introduction section elaborating on the importance of studying domain generalization in the context of skeleton-based human action recognition as following:\n\nSkeleton-based action representation has the advantage of removed background changes and camera positions, making it more robust compared to RGB representation. Despite that, generalizability under such a representation is still largely affected by the inherent spatiotemporal difference of 3D coordinates of a same action across domains. Essentially, cross-subject and cross-view settings are all cross-domain settings, yet they can be well addressed by designing more powerful backbones and applying geometric transformations, achieving high accuracy in the test set. However, we find that in the cross-dataset setting where source and target data come from different datasets, the performance degrades a lot (around or more than 20%) (comparing Table 1 and 15) and cannot be well remedied by the above approaches. This indicates drastic domain gaps in inherent feature of human actions across datasets, posing great challenges for real-life use and calling for research on domain generalization techniques for skeleton-based representation.\n\nInvestigating action samples across multiple datasets, our observation is that a notable source of domain gap comes from the temporal mismatch of an action across different datasets (Fig. 1(a)), which is usually caused by different definition or cropping criterion of human motions…\n\n\n[Q2] Why did the authors choose HCN and CTRGCN as the GCN backbones for their approach? Can this be detailed further, perhaps as a subsection in Section 4?\n\n[A2] Thanks for the suggestion. We will add a new subsection to Section 4 briefly describing backbones used in the experiments, AGCN, HCN, CTR-GCN and ST-GCN. (1) We use Adaptive-GCN with reduced number of blocks for our main experiments for its good balance between efficiency and performance. (2) HCN is a convolutional network used by [41]. We use it for fair comparison in their setting as shown in Table 3. (3) ST-GCN and CTR-GCN are representative GCN backbones. CTR-GCN has much more parameters and multi-level feature design. ST-GCN is a simple GCN without elaborate design. We use them to show that designing network itself is hard to improve generalizability. (4) We will move the results of generalizability for different backbones to the main paper.\n\n[Q3] Could the authors provide a more detailed explanation of how the approaches in Section 4.3 were implemented for skeleton-based human action recognition methods, including any specific techniques or modifications used?\n\n[A3] Thanks for the suggestion. We will add more details to Appendix A3: other baseline methods.\n\n[Q4] What is the total number of parameters in the proposed approach, and how does it compare to the baseline methods?\n\n[A4] (1) During model training, the newly added parameters are only two matrices, i.e., boundary poses of shape (N_bkg, J, 3) and linear transforms of shape (N_tr, T, T). Here N_bkg=10, J=25, N_tr=20, T=64. Considering the number of parameters of AGCN is large, our learned parameters for action completion add very little to total parameters, (around ~90000), which are less than two FC layers (suppose each layer is nn.Linear(256,256)). (2) Comparing to other methods, general domain generalization methods CCSA, ADA and handcraft augmentation methods such as uniform sampling, mixup, CropPad and CropResize do not introduce new parameters. Self-supervised learning methods often have a new branch along with the original network for learning auxiliary tasks, therefore introducing new parameters (usually several FC layers). So generally, ours and those self-supervised methods have similar magnitude of new parameters.\n\n[Q5] Can the authors provide t-SNE visualizations of the learned embeddings from different approaches on the test domain to illustrate the effectiveness of the proposed method compared to others?\n\n[A5] Thanks for the suggestion, we will add t-sne plot in the revision, which is a good visualization tool for checking learned features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: 3 of 4 reviewers were inclined to accept the paper with one reviewer flipping to positive. The negative review had very limited comments and did not respond to the rebuttal. The AC has read all reviews and agrees with the majority of reviewers that the paper should be accepted.
Decision: Accept (poster)