CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition
Authors: Yuhang Wen
Mengyuan Liu
Songtao Wu
Beichen Ding
Venue: NeurIPS 2024 poster (2024)
Keywords: Action Recognition
Skeletons
Multi-Entity Actions
Convex Hull
Primary area: machine_vision

Abstract: Skeleton-based multi-entity action recognition is a challenging task aiming to identify interactive actions or group activities involving multiple diverse entities. Existing models for individuals often fall short in this task due to the inherent distribution discrepancies among entity skeletons, leading to suboptimal backbone optimization. To this end, we introduce a Convex Hull Adaptive Shift based multi-Entity action recognition method (CHASE), which mitigates inter-entity distribution gaps and unbiases subsequent backbones. Specifically, CHASE comprises a learnable parameterized network and an auxiliary objective. The parameterized network achieves plausible, sample-adaptive repositioning of skeleton sequences through two key components. First, the Implicit Convex Hull Constrained Adaptive Shift ensures that the new origin of the coordinate system is within the skeleton convex hull. Second, the Coefficient Learning Block provides a lightweight parameterization of the mapping from skeleton sequences to their specific coefficients in convex combinations. Moreover, to guide the optimization of this network for discrepancy minimization, we propose the Mini-batch Pair-wise Maximum Mean Discrepancy as the additional objective. CHASE operates as a sample-adaptive normalization method to mitigate inter-entity distribution discrepancies, thereby reducing data bias and improving the subsequent classifier's multi-entity action recognition performance. Extensive experiments on six datasets, including NTU Mutual 11/26, H2O, Assembly101, Collective Activity and Volleyball, consistently verify our approach by seamlessly adapting to single-entity backbones and boosting their performance in multi-entity scenarios. Our code is publicly available at https://github.com/Necolizer/CHASE .

CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition
Yuhang Wen Sun Yat-sen University wenyh290mai12. sysu. sysu. edu. en
Mengyuan Liu"
State Key Laboratory of General Artificial Intelligence
Peking University, Shenzhen Graduate School mkliuyifang@gmail...
Songtao Wu Sony R&D Center China Songtao Wu@sony. com
Beichen Ding
Sun Yat-sen University sysu. edu, cn
dingbch@mail. .sy..
Abstract
Skeleton-based multi-entity action recognition is a challenging task aiming to identify interactive actions or group activities involving multiple diverse entities. Existing models for individuals often fall short in this task due to the inherent distribution discrepancies among entity skeletons, leading to suboptimal back- bone optimization. To this end, we introduce Convex Hull Adaptive Shift based multi-Entity action recognition method (CHASE), which mitigates inter-entity dis- tribution gaps and unbiases subsequent backbones. Specifically, CHASE comprises learnable parameterized network and an auxiliary objective. The parameterized network achieves plausible, sample-adas adapeeve repositioning of skeleton sequences through two key components. First, Implicit Convex Hull Constrained Adaptive Shift ensures that the new origin of the coordinate system is within the skeleton convex hull. Second, the Coefficient Learning Block provides lightweight param- eterization of the mapping from skeleton sequences to their specific coefficients in convex combinations. Morcover, to guide the optimization of this network for discrepancy minimization, Discrepancy as the additional objective. CHASE operates as a sample-adaptive normalization method to mitigate inter-entity distribution discrepancies, thereby reducing data bias and improving the subsequent classifier's multi-entity action recognition performance. Extensive experiments six datasets, including NTU Mutual 1/26, H2O, Assembly 101, Collective Activity and Volleyball, consistently verify approach by seamlessly adapting single-entity backbones and boost- ing their performance in multi-entity scenarios. Our code is publicly available at
propose the Mini-batch Pair-wise Maximum Mean
Introduction
Multi-entity action recognition, a challenging task derived from action recognition 1
5 6 7
99, aims to find the optimal estimator of the mapping from multi-entity motions semantic labels,
where entities involved can range from human bodies [10] [1I], hands [12 to various objects [
Recent approaches predominantly rely on skeletal data for addressing this challenge 10
[1 given
that skeletons serve concise representation of spatiotemporal features [15] 16 17
18 19
20
21
This task has broad applications in numan-robon interaction [22] 23) scene understanding
24
27 28),
human motion analysis 29
30
32
33
etc.
*Conresponding Authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

An Example
Point Cloud of 10 Samples
X-Y
X-2
Y-2
Top-1 Accuracy
Basaline cvertti)
Haa--rily Uairkainn thampanc
Wolk Tosowdk Walkkpprt
Beselinn 2 (SRCoM))
Durs
oHAsE (rumy
Wib
Maximum Neen
Discrepaccy
(A) Estimated Distributions of Point Clouds from Skaleton Sequences
(b) Experimental Results
Figure 1: Inter-entity distribution discrepancies in multi-entity action recognition task. (a) We delineate three distinct settings: Vanilla (a common practice), S2CoM (an intuitive baseline approach), and CHASE (our proposed method). Column 2 illustrates spatiotemporal point clouds defined by the skeletons over 10t sequences, Column 3-5 depict the projections of estimated distributions of these point clouds onto the x-y, Z-x, and y-z planes. These projections reveal significant inter-entity distribution discrepancies when using Vanilla. (b) The discrepancies observed in Vanilla introduce bias into backbone models, leading unsatisfactory performance. Although S2CoM can reduce these discrepancies. it makes the classifiers produce wrong predictions due to a complete loss of inter-entity information. With the lowest inter-entity discrepancy, our method unbiases the subsequent backbone to get the highest accuracy, underscoring its efficacy.
Experiments have revealed that network architectures tailored for single-entity actions get unsatis- factory performance when confronted with multi-entity actions [10 (35). This inadequacy can be
observed in treating interactions: each entity is encoded independently using the same single-entity backbone, and their features are averaged for recognition. This practice is based on an empirical assumption that each entity is independent and identically distributed (i.i.d).. But we demonstrate that different entities depicted by skeletons exhibit evident non-i.i.d. characteristics. Fig. (a) Row reveals significant inter-entity distribution discrepancies using estimated distributions of joints from distinct entities. Such discrepancies can introduce bias into the backbone models, leading to suboptimal optimization and performance.
attributed to common practice (36] 37
38
39,
40
explains why multi-entity action modeling usually diverges from the single-entity one.
Using local coordinates for each entity holds promise in rendering them i.i.d., achieved by shifting individual origins to the per-entity spatiotemporal centers of mass (S2CoM), as depicted in Fig. (a) Row S2CoM is a straightforward and intuitive bascline to address this problem. However, this approach exacts a significant toll as it entails complete loss of inter-entity information. Experimental results corroborate this notion, illustrated in Fig. (b), showcasing the detrimental impact of lacking inter-entity measurements on recognition performance. Nonetheless, this endeavor sparks an insightful realization: the potential for narrowing distribution gaps through origin shifts, thereby improving the performance of single-entity backbones in muuli--nnitty scenarios, A natural question arises: Can we reduce the bias by finding the optimal cample-adaptic shift in R33 that minimizes the distribution discrepancies among entities?
To address the inter-entity distribution discrepancy problem, we propose Convex Hull Adaptive Shift based multi-Entity action recognition method (CHASE). Serving as an additional normal- ization step, CHASE aims accompany other single-entity backbones for enhanced multi-entity action recognition. Our main insight lies in the adaptive repositioning of skeleton sequences to mitigate inter-entity distribution gaps, thereby unbiasing the subsequent backbone and boosting its performance. Specifically, CHASE consists of a learnable parameterized network and an auxiliary objective. The parameterized network can achieve plausible and cumple-adaptin repositioning of skeleton sequences through two crucial components. First, the Implicit Convex Hull Constrained Adaptive Shift (ICHAS) ensures that the new origin of the coordinate system is within the skeleton convex hull. Second, the Coefficient Learning Block (CLB) provides lightweight parameteriza- tion of the mapping from skeleton sequences to their specific coefficients in ICHAS. Moreover, to guide the optimization of this network for discrepancy minimization, propose the Mini-batch

---

Pair-wise Maximum Mean Discrepancy (MPMMD) as the additional objective. This loss function quantifies pair-wise entity discrepancies using maximum mean discrepancy and integrates bani-batch sampling strategies to estimate the expectation. In conclusion, CHASE works as a sample-adaple normalization method to mitigate inter-entity distribution discrepancies, which can reduce bias the subsequent classifier and enhance its multi-entity action recognition performance.
The contributions of this paper are three-fold:
To the best of our knowledge, we are the first to investigate the issue of inter-entity distribution discrepancies in multi-entity action recognition. Our proposed method, Convex Hull Adaptive Shift for Multi-Entity Actions, effectively addresses this challenge. Our main idea is adaptively repositioning skeleton sequences to mitigate inter-entity distribution gaps, thereby unbiasing the subsequent backbones and boosting their performance.
2. Serving as an additional normalization step for backbone models, CHASE consists of a Iearnable network and an auxiliary objective. Specifically, this network is formulated by the Implicit Convex Hull Constrained Adaptive Shift, together with the parameterization of a lightweight Coefficient Learning Block, which learns sample-adaptive origin shifts within skeleton convex hull. Additionally, the Mini-bate bini-batch Pair-wise Maximum Mean Discrepancy objective is proposed to guide the discrepancy minimization.
3. Experiments on NTU Mutual 11, NTU Mutual 26, H2O, Assembly101, Collective Activity Dataset and Volleyball Dataset consistently verify our proposed method by improving performance of single-entity backbones in multi-entity action recognition task.
2 Related Work
2.1 Skeleton-based Action Recognition
proffering annotated or estimated skeleton sequences support the development of skeleton- action recognition. Based on these benchmarks, significant body of works focus on the design of artificial neural network architecture for more effective skeleton-based action recognition. Early models rely on the basic architecture of Recurrent Neural Network to capture temporal motions [45] 46
Datasets & Models. Datasets [4
49 50). Graph Convolution Network (GCN) shows predominated popularity as various graph convolution operators being proposed B6 [6, 57 588. Recent progress of the model design is largely driven by adopting self attentotnn mechanism and transformer architecture
47
48
37 38 51 52, 53 54 [55
39 40
59
60
63
Optimization Objectives. Several works have explored additional optimization objectives beyond the commonly used cross-entropy (CE) loss ensure robust recognition (37 65 address challenging
open-set problems [16], or integrate supplementary natural language descriptions [66 66].
However, existing methods are usually developed under the empirical assumption that entities are i.i.d.
allowing the backbones to learn representations of actions concerning only one entity [36] 37 38 39
40 (. However, when confronted with multi-entity interactions, their common practice of feeding the backbone separately often proves inadequate. Our proposed approach can seamlessly adapt to these existing methods, boosting their performance by minimizing the distribution discrepancies.
2.2 Skeleton-based Multi-Entity Action Recognition
Interactive Actions. Addressing datasets featuring two-person actions [41 42 68
69
70
72 or
necessitates effective interaction modeling. This spurs the development of various interaction recognition models leveraging human body and hand graph priors [10 75). Notably, the introduction of the general interactive action recognition task (35]
egocentric hand-object interactions 122 B
73 74
unifies diverse interactions across various entity types, including person-to -or---toopprnnoo 10] [T] 14 35 76
77), hand-to-hand 212 35
78 79) and hand-to-object
[3] 35|
75
79
80
81 interactions.
Group Activities. Another interesting area of study is group activities more entities and may include irrelevant individual motions usually leverage compositional reasoning from group skeletons, either alone or in combination with
82
which involve
85
86
To this end, recent works
additional modalities, to achieve promising results [87] 88 89 90
9 92
93 94| 95 96).
3

---

Implicit Convex Hull Constrained Adaptive Shift (ection 3.1)
Feas P
Objective for Inter-entity Distribution Discrepancy Minimization (Section 3.31
Irfeasible 5r
Conwes Hill
pr
Mini-Batch Sampling
F "roofmm
8 X(I softmax(W)ha)
Discrepancy
Parameterized Mapping for Coefficients 3.2)
Minimization
S Skeleton Sequence
Coeflider Leening Biock
Eniites GAP4FC
Squezzo PeeazoOpcc Operntoo
Activetionn
Com 13121
Softree
Backtoone
Task
Pair-wise
Arpss
Comca
Comaat
Avg among
Metric
Eesgreadd
Mulisliction Additton
W (P(K)
Reproducing Kemel Hilbert Space (RKHS)
Ooly for Trerkig
Figure 2: The overall framework of the proposed CHASE for multi-entity action recognition. Given a skeleton sequence of multi-entity action as input. CHASE executes an implicit convex hull constrained adaptive shift with the Coefficient Learning Block, implemented as a lightweight backbone wrapper. CHASE also collects pair-wise shifted skeletons within mini-batcher effectively alleviating inter-entity distribution discrepancies by introducing an additional objective.
While these works demonstrate satisfactory performance through interaction modelling, encounter model scalability issues when confronted with the factorial growth of inter-entity interac- tions 10
may
3 35 75 Moreover, they usually lack sufficient justification for why multi multi--ntityy action modeling significantly diverges from the single-entity one [10 35 79 81] In this paper, we delve into the inter-entity distribution discrepancy problem and introduce CHASE solution minimize discrepancies. Through our proposed method, we aim to demonstrate that single-entity backbones work well in multi-entity settings.
3 CHASE
Fig. presents the framework of our proposed CHASE for skeleton-based multi-entity action recognition. We begin by presenting the formulation of the implicit convex hull constrained adaptive shift in Section 3.1 followed by the design of a lightweight Coefficient Learning Block in Section In Section 3.3 subsequently introduce an additional objective termed Mini-batch Pair-wise Maximum Mean Discrepancy to further mitigate inter-entity distribution discrepancies.
3.1 Implicit Convex Hull Constrained Adaptive Shift
The observed inter-entity distribution discrepancy in multi-entity skeleton sequences stems from the initial configuration of the world coordinate system. To mitigate this discrepancy, we propose an adaptive shift mechanism for each multi-entity skeleton sequence. It guides the origin to a sample- adaptive location, aiming render each entity approximately i.i.d.. Moreover, based the empirical assumption that the origin should not be far away from the skeletons. we implicitly constrain the new origin to remain within the skeleton convex hull by proving simple but crucial proposition.
Consider a scenario where E interactive entities (c.g. persons) engage in purposeful activities over duration of T, and the pose of each entity is indicated by J joints with C Cartesian coordinates. The skeleton sequence of a multi-entity action is defined as X RCCTxxxE For clarity we denote
U = T J x E. Given points P. Cxx1 in X E RCxU the subtraction 5. p p²11 < 5 U) defines a shift of origin for them, where h.. x Cx1, This can be expressed in matrix form
x = X - J1.,
(I)

---

The implicit skeleton convex hull constrained adaptive shift vector is formulated as where J1,u E IR IxU is a matrix of ones, and X is the shifted skeleton sequence. Now the problem is to make the shift vector p" adaptive to X. A naive implementation is the linear combination:
= - p. Ji. X(I - Wi,e),
where / RUXU and the weight matrix W E ERUx1
However, optimizing W can be challenging without constraints, as p² could potentially be any point in R It is therefore reasonable to constrain p² by incorporating the definition of the Convex Hull.
Definition 1 (Convex Hull 197). The convex hull S of given set X can be defined 1) The (unique) minimal convex set containing X. 2) The set of all convex combinations of points in X. These definitions are equivalent.
We jump to the formulation of the implicit skeleton convex hull constrained adaptive shift vector by proving the following proposition:
Proposition 1.
pd -xsoftmxx(W),
where X E RCXU W E RUX1, and pd E BCx1, p¹ in Eq. 3 is element in the set of all convex combinations of points in X. also a point that lies in the minimal convex set containing X.
Proof. The first half of this proposition is equivalent to show that the matrix product of X and
softmax(W) is a convex combination of X. X set of points ; Por with C' Cartesian coordinates. We denote softmax(W)) as W with component as, which formulated as
e
ā; En ea
(LSiSU),
(4)
where a; a component W. By applying function softmax RD - (0, 1)", each component āg of W will be in the interval (0, 1), and the components will add up to Thus we have p² E a.p%, where all ā; E IR satisfy ā; > 0 and Σ â; = 1. This is sufficient for the definition of convex combination, which only requires ā, 2 0. Then the second half of this proposition is evident with the equivalence of definitions in Def.
Proposition 1 also implies that all possible P" constitute a subset Š of the convex hull S defined by the skeleton joints for all entities during the action period:
5 = Σ a.p P. E X,
Σ a. 1,ā; e.==1, E cs.
(5)
which specifically is the interior of S (i.e., the open convex hull of X). We provide an example of the
feasible p² in the interior of S. marked by green circle in Fig. 2 The center of mass (CoM) p is
also in the set Š, proven by simply taking all ā; = 1/U(1 < U).
With Eq. and Eq. introduce Implicit Convex Hull Constrained Adaptive Shift as:
X softmax (W)J,,))
(6)
where W is coefficients needed to be optimized. In Eq. 2 the search space for p² encompasses the entire R² However, in Eq. 6 it's restricted to the open convex hull S. We optimize the weights for each point under the constraint of the skeleton convex hull, subsequently deriving the adaptive shift vector for each sample. Applying softmax function implicitly constrains p² to remain within the convex hull S, while preserving inter-entity measurements. Consequently, subtraction between the point set and the shift vector repositions the origin to specific point the open convex hull.
5

---

3.2 Parameterized Mapping for Coefficients
In this section, a lightweight Coefficient Learning Block is introduced to parameterize the mapping from the input skeleton sequence to the weight matrix. This parameterization allows CHASE to achieve sample-adaptive coefficients beyond sample-adaptive shifts formulated in Section 3.1
In Eq. we note that the first-order partial derivative of X with respect to X
0x I ax
Ju. Jo:
(7)
whose result is constant. This implies that the same learnt weight matrix W is applied all different Xs when getting adaptive p*s. To make the coefficients W dependent on the input X, mapping a/ R Rxx + R Ux is expected to map X to W.
As depicted in Fig. 2 parameterize the nonlinear mapping 1 sequence of learnable layers, termed the Coefficient Learning Block. This lightweight CLB can be formulated as follows:
W v X) WW + 6)).
(8)
where Wi E RC W2, E RC W3 E U2 are weight matrices, b is a bias matrix, Φ RCIxU RRx1 is squeeze operator 9X and is an activation function. Using dimensionality- layer around the non-linearity is common gating mechanism parameterization 98 99]. Hence, we ensure U C1 C2-
reduction layer and a aimmensionality-inrraattt
3.3 Objective for Inter-entity Distribution Discrepancy Minimization
To facilitate CHASE optimization, we introduce an additional objective aimed minimizing the inter-entity distribution discrepancies of the shifted skeleton sequences. This objective quantifies the pair-wise discrepancies and employs mini-batcls sampling strategies to estimate the expectation.
Maximum mean discrepancy is a metric used to measure the distance between distributions, defined as the distance between their embeddings in the reproducing kernel Hilbert space (RKHS) H:
MMD(P,Q)
sup 11II
(BIf(>)] (v)l))
where sup(-) denotes the supremum. It is equivalent finding the RKHS function that maximizes the difference in expectations between the two probability distributions P(x) and Q(v).
Suppose each entity distribution is denoted (1 < i < E") for E' entities, measure the distance of all pair-wise distributions using the empirical mean
E-1 Σ Σ MMD(P P()/CEE),)
Ex(2) MMD(2))
(10)
where == -Pt, S Sij j) with the probability density r(=), and C(E,2) denotes a combination F. things taken 2 at time without repetition. We adopt two approximations for computational efficiency. The first involves estimating [()(x)) in Eq. 9 uuiin mini-batch of The second approximation concerns the right-hand side of Eq. I0 which is impractical due to its complexity of (n!) in terms of the entity count. Instead, can be approximated by uniformly sampling a mini-batch of M entity pairs from all possible C(E, 2) combinations z:
1 M
En(a) MMD(=)) =
MMD(2m))
(11)
We denote Eq. II with the above two approximations be the Mini-batch Pair-wist Maximum Mean Discrepancy Loss Empmmmd thereby we have the total loss function for training:
f = LCLS Xmmpmmdd
(12)
where LOLS is the classification loss and À is the trade-off weight factor for Lmpmmad

---

NTU Mutual 11 26mimi intoraction calegories, e.g. giving object, shaking hands, high-five cheers drink, exchange rack paper scissors
interactive (bwo objecD engage
ASB101 Skeleten sequences should be classified into 1.380 fine-grained categories Iverb & noun) without ebject
CAD Recognize group activites from estimated 2D joints in the wild, with number of entiiie up to 13
Recognize vollebbll group activties from estimated joints of 12 players and the position the
Figure 3: Visualizations of multi-entity action samples and their skeleton convex hulls.
4 Experiments
4.1
Datasets & Settings
We conduct experiments on six multi-entity action recognition datasets. Fig. presents skeletal samples in these datasets and their skeleton convex hulls, showcasing their difficulties.
NTU Mutual 11 and NTU Mutual 26, respectively subsets of NTU RGB+D [41 and NTU RGB+D 120 422, consist of variety of inter person mutual actions. NTU Mutual 11 adopts the widely-used X-Sub and X- View criteria, while NTU Mutual 26 follows the X-Sub and X-Set criteria.
H2O 13 proffers 3D poses of human hands and bounding boxes of the manipulated objects, facilitating both hand-to-hand and hand-to-object interactions learning. We follow the training, validation, and test splits outlined in [3] in our experiments.
Assembly101 (ASB101) [12] is large and challenging 3D manual procedural activity dataset, with 1,380 categories of interactive actions. We follow the training. validation, and test splits described in [12 for evaluations. Fine-grained actions (verb & noun) are adopted labels in experiments.
Collective Activity Dataset (CAD) captures people and their behaviors in public using street cameras, categorizing pedestrian collective activities into 4 groups We adopt the same categories, individual labels, train-test split in (95] Only 2D joint coordinates are used our experiments.
consists of video clips from volleyball tournaments and includes group activity classes based on volleyball terminology. We follow the Original split described in for evaluation. Only estimated 2D joint coordinates are used as input features.
Volleyball Dataset (VD) 86
Settings. Experiments conducted on the GeForee RTX 3070 GPUs with PoTchh.. CTR-GCN (3)
InfoGCN (37], STe (40) and HD-GCN (38] are chosen as our baseline models. To ensure fair comparisons, adopt single intra-skeleton modality without multi-modality fusion following (35). For CTR-GCN in NTU Mutual 26, we adopt input shape E p3x64x26xx segment size 1,, and X = 0.1 in CHASE. SGD optimizer used with Nesterov momentum of 0.9, initial learning rate of 0.1 and a decay rate 0. at the 80th and 100th epoch. Batch size is set to 64, More detailed configurations for each model are provided in the Appendix.
4.2 Experimental Results
Table shows the experimental results on different benchmarks, reporting averaged top-1 accuracy and its standard deviation in runs with several seed initializations. We compare CHASE with vanilla counterparts (light red background) and the state-of-the-ar multi-entity action recognition methods (light yellow background). By adopting proposed CHASE, boost the vanilla counterparts'
7

---

Table 1: Comparisons with Skeleton- based Methods Multi Multi-Entity Action Datasets
NTU Mutual 26(%)
NTU Mutual 11(%)
Method
Venue
X-Sub
X-Set
X-Sub
View
GDCN
23
85.80
92.10
SkeleTR 76
ICCV'23
87.80
88.30
94.80
97.70
ISTA-Nel 35
IROS'23
90.56(=0.08)
(1.72
AHNet-lLarge 83
PR'24
86.43
86.64
90.85
93.38
mc-GCN 7/
arXiv'24
90.00
90.00
95.50
98.20
CTR-GCN 36
ICCV'21
(=000)
90.19 177)
95.94 36)
(8.32
CHASE (Ours)
91.30 (=0.22)
11.98
92.34
12.15
(=0.10)
96.45
(+0.05)
10.51
98.83
70.51
(+0.13)
InfoGCN (k=l)
CVPR 22
90.22(#0.102
(1.13 10.16)
95.51 (+:0.10)
97.76 (:0.22)
CHASE (Ours)
91.86 (=0.05)
11.64
92.41
11:28
96.35 (#0.18)
98.25 (#:025)
10.49
(=0.44)
STSA-Net [40
Neuro. '2
88.41
(=0001)
90.19
#0.11) (0.11
95.96 (40.000000
98.47
+CHASE (Ours)
89.77
(=0.18)
11.38
91.54
(=0.12)
11.35
10.08 96.63
(+0.10)
98.73 (+0008)
70.24
HD-GCN (CMM=l)
ICCV'23
(0.44))
(=0.44)
90.08
35.58(440.1)
97. (7.93(
CCASE (Ours)
90.81 (=0.13)
92.06
11.97
96.22 (+0005)
98.31 (+0.07)
(0022))
Method
Venue
H20(%)
ASB101(%)
CAD(%)
VD(%)
AT 26
CVPR 20
92.30
ISTA-Net 35
IROS' 23
89.09
28. 28.01
87 67.16
(1.40
H2OTR 80
CVPR'23
90.90
EftHandEgoNet 81
arXiv'24
91.32
AHNet-Large 83
PR'24
89.32
84.31
CTR-GCN 36
ICCV'21
81.68 (40.85)
27. 27.83
80.45
92.66
+ +CHASE (Ours)
91.05 (d.1.98)
19.37
28.03 10.21
89.61
19.16
92.89
70.24
(+0.30)
(=0.20)
(#0.15)
InfoGCN (k=1)
CVPR 22
76.24 (33.9))
27. 27.18
83.07 (0.46))
91.77 (=:011))
+ CCHASE (Ours)
83.47
17.23
27. 27.36
10.18
84.18
11.11
92.00
10.23
(9:
(+00112
(=291)
(=0.15)
STSA-Net
Neuro. 23
92.29 (20..52
27. (7.70(+0.19)
80.20 (1:3,00)
(2:0.52
CHASE (Ours)
94.77
27.81
10.11
85.93
15.73
92.78
10.26
L#:1)36)
(3:0.13)
(.2466)
(10.41)
HD-GCN (CMMel)
ICCV'23
(2.73 (+0.41)
72.73
27.31 27.31 (40.36)
76.93 +4.38)
(1-0.02)
CHASE (Ours)
81.61
18.88
27.50
82.39
92.00
(+1.03)
(4:024)
(#1.61)
(#0.07)
performance by a noticeable margin in most settings. yields varying degrees of accuracy improve- ment across different baseline models and benchmarks, owing to differences in model parameter count, training objective, data scale, etc. Compared models with complicated interaction designs, CHASE can help single action backbones achieve the seate-of-the-and performance in interaction recognition by outperforming ISTA-Net (35), AHNet-Large [33. etc. group activities recognition task, which is more challenging for single-entity backbones, CHASE can help achieve competitive performance. Fig. visualizes that CHASE can effectively alleviate the potential inter-entity distri- bution discrepancies across range of data scales, thereby ensuring robust backbone optimization and inference. UMAP [00) visualization in Fig. 5 demonstrates our proposed CHASE differentiate similar multi-entity actions better by assisting backbones to learn more distinctive representations.
Table 2: Comparison with Other Alternatives
4.3
Ablation Study
Method
Acc (%)
A (%)
Vanilla
89.32
In this section, conduct ablation studies the widely-adopted benchmarks NTU Mutual 26 and NTU Mutual 11 with only joint modality.
S2CoM
88.66(10.26)
0.67
BatchNorm
89.06(+0016)
00.7
ER 335
89.34/410155)
+0.02
Aug
89. 89.72(110.04
+0.40
Comparison with other alternatives. We com- pare our proposed CHASE with several alterna- tives as follows: 1) Vanilla: Use the raw world coordinates or pixel coordinates. 2) S2CoM: Shift
S2CoM+/STD
90. 90.29(+0.06)
+0.97
S2CoM
90.79(+0.100
+1.47
CHASE (Ours)
(0.22)
+1.88
8

---

Vanila
CHASE (Curd
Fewer S Samples
More
Samples
More (109)
Figure 4: Qualitative results of CHASE. Different entity distributions are denoted by blue and orange. CHASE effectively mitigates inter-entity distribution discrepancies, demonstrating its clear effectiveness across range of data scales, from small to large.
Table 3: Analysis of Inter-entity Distribution Discrepancies
Set
Method
Avg KLD +
JSD
BD
HD 1
MMD
Vanilla
(=0.25)
0.19
(0206)
0.46(44000)
(.94
CHASE (Ours)
(+3.00+009)
(.08(++.02)
0. 0.10 (+0002)
(.30(=0.03)
0.05(#0.02)
Vanilla
(2:0.35)
(.18
(
0.45(=0.05)
1.03(=00.0)
II
CHASE (Ours)
0.45(100.8)
0.10(=0002)
0. 0.11 (++0.2))
0.32 (=0.03)
(.07(4+0.0)
Vanilla
(=0.
0.14(#0.02)
(=0.00))
(10.
1.25
III
CHASE (Ours)
0.41 0.41(+0.00)
0.08
0. 0.10 (+:002)
0.30
0.05
Vanilla
(.7.(1)
(.14(10.03)
0.17 (L.003)
(.40((1..4))
1.15
IV
CHASE (Ours)
0.41 (10.07)
0.08 (.08(11001)
0.09 (110.02)
0.30 0.30 (10.03)
0.04 (10.03)
the individual origins to the spatiotemporal centers of mass for each entity. 3) BatchNorm: Apply an additional BatchNorm operation immediately when batches of samples are fed into the model. 4) ER (Entity Rearrangement [35]): technique aims to eliminate the orderliness of entities for interaction modelling. 5) Aug: Apply additional data augmentation by randomly shifting the skeleton quences. 6) S2CoM+: Shift the origin to the spatiotemporal center mass. 7) S2CoM+/STD: Scale according to the channel-wise standard deviations after applying S2CoM Results in Table 2 indicate that CHASE can outperform these alternatives by bringing the largest accuracy improvement to the vanilla CTR-GCN.
Analysis of inter-entity distribution diserepancies. Table presents metrics evaluating the inter- entity distribution discrepancies on test sets, including Averaged Kullback-Leilerr Divergence (Avg KLD), Jensen-Shannon Divergence (JSD), Bhattacharyya Distance (BD), Hellinger Distance (HD) and MMD. We measure the pair-wise distributions of sampled data points from different entities in test sets of NTU Mutual 11 X-Sub (I), X- View (II) and NTU Mutual 26 X-Sub (III). X-Set (IV). Table demonstrates that CHASE significantly minimizes discrepancies across all evaluation metrics, thereby benefiting backbone learning for each entity in multi-entity actions.
Analysis of Key Components. We validate the effectiveness of each key component in Table 4 When remov-
Table 4: Analysis of Key Components in CHASE
ICHAS
CLB
MPMMD
Acc (%)
(%)
AS
CHC
ing the skeleton convex hull constraint (CHC), there is significant drop in accuracy, exceeding 60%, for initial learning rates (Ir) of 0.1 and 0.01. This substantial decline highlights the importance of CHC constraint for learning the adaptive shift, Additionally, replacing Adap- tive Shift (AS) with X Ji.r results in dramatic decrease in accu- racy, indicating that simply adding an equivalent number trainable parameters without an adaptive shift formulation is ineffective. Table further shows CHASE also benefits from CLB and MPMMD.
d
0.1
91.30,+0.22
0.1
22. 12.35
68.65
0.01
(6.99
- 4.32
91.20(2a.13)
0.10
0.1
22. 22.75
68.56
critical
0.01
(30.3)
- -67.79
0.1
20.42
-70.88
0.1
91.17 (+0.18)
-0.13
0.1
(9.10)
-1.81

---

killing
p70 rour
widd o0%
Vanilla
CHASE (Ours)
Figure 5: UMAP [00] visualizations of multi-entity skeleton sequence representations on the test split of NTU Mutual 26 X-Sub. Compared with Vanilla, our proposed CHASE differentiate similar multi-entity actions better by assisting backbones to learn more distinctive representations.
Table 5: Mixed Recognition on NTU RGB+D 120
Method
X-Sub (%)
X-Set (%)
CTR-GCN 36
84.95 85.36 10.05)
86.90(
CHASE
(6.95
Evaluations Mixed Recognition of Single- & Multi-Entity Actions. Table 5 shows 0.41 % improvement in X-Sub accuracy on the entire NTU RGB+D 120. This implies that although CHASE proposed for multi-entity actions, it is also effective in mixed recognition settings.
illustrated in Table 6 the number of trainable parameters of CHASE in NTU Mutual 26 configurations is about 26.37 k, resulting in mere I 1%-2% parameter increase. For computational complexity, FLOPs CHASE approximately 2.50 M. These metrics demonstrate that CHASE is both efficient and lightweight.
Analysis of Efficiency.
Table 6: CHASE Trainable Parameters
Method
Param. (M)
CTR-GCT [36]
1.44
CHASE
1.46
STSA-Net 40
4.13
CCAAEE
4.1670.60%
5 Conclusion
This paper proposes the Convex Hull Adaptive Shift for Multi-Entity Action Recognition (CHASE) address the inter-entity distribution discrepancies. To the best of our knowledge, we the first to investigate this problem and leverage discrepancy minimization to unbias the classifiers. Our approach seamlessly adapt existing backbone architectures and demonstrate performance improvements across six multi-entity action recognition datasets.
Acknowledgments and Disclosure of Funding
This work was supported by Natural Science Foundation of Shenzhen (No. (CY120230807120800002 and National Natural Science Foundation of China (No. 62203476, No. 52105079).
10

---

References
[1] Pengfei Wei, Lingdong Kong. Xinghua Qu, Ren, Zhiqiang Xu, Jing Jiang, and Xiang Yin. Unsupervised video domain adaptation for action recognition: A disentanglement perspective. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
[2] Dongho Lee, Jongseo Lee, and Jinwoo Choi. CAST:
Cross-attention in space and time for video action recognition. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
[3] Filip Ilie, He Zhao, Thomas Pock, and Richard P. Wildes. Selective, interpretable and motion consistent privacy attribute obfuscation for action recognition. In Conference on Computer Vision and Pattern Recognition (CVPR). 2024.
[4] Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, and Lin Wang. Exact: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more. The IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[5] Duo Peng, Li Xu, Qiuhong Ke, Ping Hu, and Jun Liu. Joint attribute and model generalization learning for privcc-preseevvng action recognition. In Thirty-seventl Conference on Neural Information Processing Systems (NeurIPS), 2023.
[6] Bruce X.B. Yu, Yan Liu, Xiang Zhang, Sheng-hua Zhong, and Keith C.C. Chan. Mmnet:
A model-based multimodal network for human action recognition rgb-d videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3522-3588 2023.
[7] Junwei Liang, Enwei Zhang, Jun Zhang, and Chunhua Shen. Multi-datase training of transformers for robust action recognition. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[8] Jinfu Liu, Chen Chen, and Mengyuan Liu. Multi-modality co-Icarning for efficient skeleton- based action recognition. In Proceedings of the ACM Multimedia (ACM MM), 2024.
[9] Jinfu Liu, Runwei Ding, Yuhang Wen, Nan Dai, Fanyang Meng, Fang-Lue Zhang, Shen Zhao, and Mengyuan Liu. Explore human parsing modality for action recognition. CAAI Transactions on Intelligence Technology, 2024.
[10] Yunsheng Pang, Qiuhong Ke, Hossein Rahmani, James Bailey, and Jun Liu. Igformer: Interaction graph transformer for skeleton-based human interaction recognition. In Proceedings of the 17th European Conference on Computer Vision (ECCV), pages 605-622, 2022.
[11] Shuai Li, Xinxue He, Wenfeng Song, Aimin Hao, and Hong Qin. Graph diffusion convolutional network for skeleton based semantic recognition of two-person actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8477-8493, 2023.
[12] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly 101: cedural activities. In 2022 IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21064-21074 2022.
101: A large-scale multi-view video dataset for understanding pro-
[13] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEFSCVF International Conference on Computer Vision (ICCV), pages 10138-10148, October 2021.
[14] Mauricio Perez, Jun Liu, and Alex C. Kot. Interaction relational network for mutual action recognition. JEEE Transactions on Multimedia, 24:366 346,66-376,.2 2022.
[15] Xinshun Wang, Zhongbin Fang. Xia Li, Xiangtai Li, Chen Chen, and Mengyuan Liu. Skeleton- in-context: Unified skeleton sequence modeling with in-context learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[16] Kunyu Peng, Cheng Yin, Junwei Zheng, Ruiping Liu, David Schneider, Jiaming Zhang, Kailun Yang, M. Saquib Sarfraz, Rainer Stiefelhagen, and Alina Roitberg. Navigating open set scenarios for skeleton-basee action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 3885) Mar. 2024.

---

[17] Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang. Mo- tionbert: A unified perspective on learning human motion representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15088-15099, October 2023.
[18] Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-basee action recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2969- 2969-2978, 2978, June 2022.
[19] Xingzhe He, Bastian Wandt, and Helge Rhodin. Autolink: Self-supervised learning of human skeletons and object outlines by linking keypoints. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[20] Jun Liu, Amir Shahroudy, Gang Wang, Ling- Yu Duan, and Alex C. Kot. Skeleton-based online action prediction using scale selection network. IEEE Transactions on Pattern Analysis and Muchine Intelligence, 42(6): 4533-1477 2020.
[21] Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Christoph Feichtenhofer, and Jitendra Malik. On the benefits of 3d pose and tracking for human action recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Paltern Recognition (CVPR), pages 640-649, June 2023.
[22] Fan-Yun Sun, Isaac Kauvar, Ruohan Zhang, Jiachen Li, Mykel Kochenderfer, Jiajun Wu, and Nick Haber. Interaction modeling with multiplex attention. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[23] Simindokht Jahangard, Zhixi Cai, Shiki Wen, and Hamid Rezatofighi. Jrdb-social: A mul- tifaceted robotic dataset for understanding of context and dynamics of human interactions within social groups. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[24] Timur Bagautdinov, Alexandre Alahi, Francois Fleuret, Pascal Fua, and Silvio Savarese. Social scene understanding: End-to-end multi-person action localization and collective activity recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[25] Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. arXiv:2403.0m 08629, 2024.
[26] Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, and Cees G. M. Snoek. Actor-transformers for group activity recognition. In 2020 IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 836-845, 2020.
[27] Hongwei Ren, Yue Zhou, Haotian FU, Yulong Huang, Xiaopeng LIN, Jie Song, and Bojun Cheng. Spikepoint: An efficient point-based spiking neural network for event cameras action recognition. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[28] Evonne Ng, Donglai Xiang, Hanbyul Joo, and Kristen Grauman. You2me: Inferring body pose in egocentric video via first and second person interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[29] Fangzhou Hong, Liang Pan, Zhongang Cai, and Ziwei Liu. Versatile multi-modal pre-training for human-centric perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16156-16166 June 2022.
[30] Weihua Chen, Xianzhe Xu, Jian Jia, Hao Luo, Yaohua Wang, Fan Wang. Rong Jin, and Xiuyu Sun. Beyond appearance: A semantic controllable self- superviee learning framework for human-centrie visual tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15050-15061, June 2023.
12

---

[31] Shixiang Tang, Cheng Chen, Qingsong Xie, Meilin Chen, Yizhou Wang, Yuanzheng Ci, Lei Bai, Feng Zhu, Haiyang Yang, Li Yi, Rui Zhao, and Wanli Ouyang. Humanbench: Towards general human-centric perception with projector assisted pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21970-21982, June 2023.
[32] Junkun Yuan, Xinyu Zhang. Hao Zhou, Jian Wang, Zhongwei Qiu, Zhiyin Shao, Shaofeng Zhang, Sifan Long, Kun Kuang, Kun Yao, Junyu Han, Errui Ding, Lanfen Lin, Fei Wu, and Jingdong Wang. HAP: Structure-aware masked image modeling for human- perception. In seiirty-eventth Conference on Neural Information Processing Systems (NeurlPS), 2023.
[33] Yuanzheng Ci, Yizhou Wang, Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang. Unihep: A unified model for numan-centre perceptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 17840- 17852, June 2023.
[34] Yizhou Wang, Yixuan Wu, Shixiang Tang, Weizhen He, Xun Guo, Feng Zhu, Lei Bai, Rui Zhao, Jian Wu, Tong He, et al. Hulk: A universal knowledge translator for human- tasks. arXiv:2312.01697, 2023.
[35] Yuhang Wen, Zixuan Tang, Yunsheng Pang, Beichen Ding, and Mengyuan Liu. Interactive spatiotemporal token attention network for skeleton-based general interactive action recogni- tion. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7886-7892, 2023.
[36] Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel- wise topology refinement graph convolution for skeleton-bai -based action recognition. In IEEE International Conference on Computer Vision (ICCV), pages 13359-13366 13368, 2021.
[37] Hyung-Gun Loung-Gunno Chi, Myoung Hoon Ha, Seunggeun Chi, Sang Wan Lee, Qixing Huang, and Karthik Ramani. Infogen: Representation learning for human skeleton-based action recogni- tion. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20154-20164, 2022.
[38] Jungho Lee, Minhyeok Lee. Dogyoon Lee, and Sangyoun Lee. Hierarchically decomposed graph convolutional networks for skeleton- based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10444-10453, October 2023.
[39] Lei Shi. Yifan Zhang, Jian Cheng, and Hanqing Lu. Decoupled spatial-temporal attention network for skeleton-based action-gestus recognition. In 15th Asian Conference on Computer Vision (ACCV), page 38-53, 2020.
[40] Helei Qiu, Biao Hou, Bo Ren, and Xiaohua Zhang. Spatio-temporal segments attention for skeleton-based action recognition. Neurocomputing, 518:30-38, 2023. ISSN 0925-2312.
[41] Amir Shahroudy, Jun Liu, Tian-Tsong Ng. and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In 2016 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1010-1019, 2016.
[42] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C. Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10))2684-277) 2701, 2020.
[43] Jiang Wang, Xiaohan Nie, Yin Xia, Ying Wu, and Song-Chun Zhu. Cross-view action modeling, learning, and recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), page 2649-2656, 2014.
[44] Liu Chunhui, Hu Yueyu, Li Yanghao, Song Sijie, and Liu Jiaying. Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding. ACM Multimedia workshop, 2017.
13

---

[45] Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-tempora Istm with trust gates for 3d human action recognition. In Proceedings of the 17th European Conference on Computer Vision (ECCV), pages 816 136-333,2 2016.
[46] Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, and Xiaohui Xie. Co-occurrence feature learning for skeleton based action recognition using regularized deep Istm networks. In Proceedings of the Thirlieth AAAI Conference on Artificial Intelligence, 1AAI'16, page 3697-3703, AAAI Press, 2016.
[47] Jun Liu, Gang Wang, Ping Hu, Ling-Yo Yu Duan, and Alex C. Kot. Global context-aware attention Istm networks for 3d action recognition. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3671-3680, 2017.
[48] Pengfei Zhang, Cuiling Lan, Junliang Xing. Wenjun Zeng, Jianru Xue, and Nanning Zheng. View adaptive recurrent neural networks for high performance human action recognition from skeleton data. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2136-28 1136-2145, 2017.
[49] Jun Liu, Gang Wang. Ling- Yu Duan, Kamila Abdiyeva, and Alex C. Kot. Skeleton- hu- man action recognition with global context-aware attention lstm networks. IEEE Transactions Image Processing, 27(4):1586 2018.
[50] Pengfei Zhang, Cuiling Lan. Junliang Xing, Wenjun Zeng, Jianru Xue, and Namning Zheng. View adaptive neural networks for high performance skeleton-based human action recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 41(8):1963-199 2019.
[51] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI'1, 18, 2018.
[52] Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Actional-structur graph convolutional networks for skeleton-based action recognition. In 2019 IEEE/CVF Conference on Computer Vision and Paltern Recognition (CVPR), pages 3590-3598, 2019.
[53] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12018-12027, 2019.
[54] Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 140-149, 2020.
[55] Xiaoke Hao, Jie Li, Yingehun Guo, Tao Jiang, and Ming Yu. Hypergraph neural network for skeleton-base action recognition. IEEE Transactions on Image Processing, 30:2263-2275. 2021.
[56] Haodong Duan, Jiaqi Wang. Kai Chen, and Dahua Lin. Pyskl: Towards good practices for skeleton action recognition. In Proceedings of the 30th ACM International Conference on Multimedia (ACMMM), pages 7351-7354, 2022.
[57] Dongjingdin Liu, Pengpeng Chen, Miao Yao, Yijing Lu, Zijie Cai, and Yuxin Tian. Tsgenext: Dynamic-statio static multi graph convolution for efficient pooleton-based action recognition with long-term learning potential. urXiv::304. 116/,, 2023.
[58] Woomin Myung, Nan Su, Jing-Hao Xue, and Guijin Wang. Degen: Deformable graph convolutional networks for skeleton- based action recognition. IEEE Transactions on Image Processing, 33:2477-2490, 2024.
[59] Yuxuan Zhou, Zhi-Qi Cheng, Chao Li, Yifeng Geng, Xuansong Xie, and Margret Keuper. Hypergraph transformer for skeleton-based action recognition. (tt:::///10099// 2022.
[60] Qingtian Wang, Shuze Shi, Jiabin He, Jianlin Peng. Tingxi Liu, and Renliang Weng. lip- tranffommr: Intra-inter-part transformer for skeleton-basee action recognition. In IEEE International Conference on Big Data (BigData), pages 936-945, 2023.
14

---

[61] Nguyen Huu Bao Long. Step catformer: Spatial-temporal effective body-part cross attention transformer for dkeleton-based action recognition. arXiv:2312.03288, 2023.
[62] Yunyao Mao, Jiajun Deng, Wengang Zhou, Yao Fang, Wanli Ouyang, and Houqiang Li. Masked motion predictors are strong 3d action representation learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10181-10191, October 2023.
[63] Neel Trivedi and Ravi Kiran Sarvadevabhatla. Psumnet: Unified modality part streams are all you need for efficient pose-based action recognition, In Proceedings of the 17th European Conference on Computer Vision (ECCV), page 21 211-227, 2022.
[64] Jeonghyeok Do and Munchurl Kim. Skateformer: Skeletal-temporal transformer for human action recognition. WxXi:22403099508 2024.
[65] Xiaohu Huang, Hao Zhou, Bin Feng, Xinggang Wang, Wenyu Liu, Jian Wang. Haocheng Feng, Junyu Han, Errui Ding, and Jingdong Wang. Graph contrastive learning for skeleton-based action recognition. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[66] Haojun Xu, Yan Gao, Zheng Hui, Jie Li, and Xinbo Gao. Language knowledge-assisted representation learning for skeleton- action recognition. CoRR, abs/2305. 12398, 2023.
[67] Wangmeng Xiang, Chao Li, Yuxuan Zhou, Biao Wang, and Lei Zhang. Generative action description prompts for skeleton-based action recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages
October 2023.
[68] Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L. Berg, and Dimitris Samaras. Two-person interaction detection using body-pose features and multiple instance learning. In IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition Workshops, pages 28 35, 2012.
[69] Wen Guo, Xiaoyu Bie, Xavier Mlameda-Pinedaa and Francese Moreno-Noguer. Multi-person extreme motion prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1305--13064, June 2022.
[70] Yifei Yin, Chen Guo, Manuel Kaufmaann, Juan Jose Zarate, Jie Song, and Otmar Hilliges. Hi4d: 4d instance segmentation of close human interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17016-1700 1001611027,, June 2023.
[71] Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng. Yunhui Liu, Wenjun Zeng, and Xiaokang Yang. Inter- Towards versatile -uman-human interaction analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[72] Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-basse multi-human motion generation under complex interactions. International Journal of Computer Vision, Mar 2024.
[73] Guillermo Garcia-Hermane Shanxin Yuan, Seungryul Baek, and fae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[74] Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. Assemblyhands: Towards egocentric activity understanding via 3d hand pose estimation. In Proceedings of the IEE//CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 12999- June 2023.
[75] Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+o: Unified egocentric recognition of 3d hand-object poses and interactions. In 2019 IEEE/VE Conference on Computer Vision and Pattern Recognition inniiiionnCl (CVPR), pages 4506-4515, 2019.
15

---

[76] Haodong Duan, Mingze Xu, Bing Shuai, Davide Modolo, Zhuowen Tu, Joseph Tighe, and Alessandro Bergamo. Skeletr: Towards skeleton-based action recognition in the wild. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), pages 13634-13644, October 2023.
[77] Mengyuan Liu, Chen Chen, Songtao Wu, Fanyang Meng, and Hong Liu. Learning mutual excitation for hand-to-hand and human-to-human interaction recognition. arXiv:2402.0224 02240123,,,, 2024.
[78] Yilin Wen, Hao Pan, Lei Yang, Jia Pan, Taku Komura, and Wenping Wang. Hierarchical temporal transformer for 3d hand pose estimation and action recognition from egocentric rgb videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern June 2023.
Recognition (CVPR), pages 21243-21223
[79] Md Salman Shamil, Dibyadip Chatterjee, Fadime Sener, Shugao Ma. and Angela Yao. On the utility of 3d hand poses for action recognition. arXiv:2403.09803 2024.
[80] Hoscong Cho, Chanwoo Kim, Jihyeon Kim, Scongyeong Lee, Elkhan Ismayilzada, and Seungryul Back. Transformer-basee unified recognition of two hands manipulating objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4769-4778, June 2023.
[81] Wiktor Mucha and Martin Kampel. In my perspective, in my hands: Accurate egocentric 2d hand pose and action recognition. arXiv:2404.00300, 2024.
[82] Tian Lan, Yang Wang. Weilong Yang, Stephen N. Robinovitch, and Greg Mori. Discriminative latent models for recognizing contextual group activities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8):1549-1562 2012.
[83] Guoquan Wang. Mengyuan Liu, Hong Liu, Peini Guo, Ti Wang, Jingwen Guo, and Ruijia Fan. Augmented skeleton sequences with hypergraph network for self-supervised group activity recognition. Pattern Recognition, 152:1 10478, 2024.
[84] Che-Jui Chang, Danrui Li, Deep Patel, Parth Goel, Honglu Zhou, Seonghyeon Moon, Samuel S. Sohn, Sejong Yoon, Vladimir Pavlovic, and Mubbasir Kapadia. M3act: Learning from synthetic human group activities. In Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[85] Wongun Choi, Khuram Shahid, and Silvio Savarese. What are they doing? Collective activity classification using spatio-temport relationship among people. In IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pages
2009.
[86] Mostafa S. Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. A hierarchical deep temporal model for group activity recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). pages 1971-1980, 2016.
[87] Sina Mokhtarzadeh Azar, Mina Ghadimi Atigh, Ahmad Nickabadi, and Alexandre Alahi. Con- volutional relational machine for group activity recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[88] Jianchao Wu, Limin Wang, Wang, Jie Guo, and Gangshan Wu. Learning actor relation graphs for group activity recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). June 2019.
[89] Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, and Cees G. M. Snoek. Actor-transformers for group activity recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[90] Shuaicheng Li, Qianggang Cao, Lingbo Liu, Kunlin Yang, Shinan Liu, Jun Hou, and Shuai Yi. Groupformer: Group activity recognition with clustered spatial-temporal transformer. In Proceedings of the IEEF/CVE International Conference on Computer Vision (ICCV), pages 13668-13677, October 2021.
16

---

[91] Hangjie Yuan and Dong Ni. Learning visual context for group activity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4):3261-3265) May 2021.
[92] Masato Tamura, Rahul Vishwakarma, and Ravigopal Vennelakanti. Hunting group clues with transformers for social group activity recognition. In Proceedings of the 17th European Conference on Computer Vision (ECCV), page 19 35, 2022.
[93] Mingfei Han, David Junhao Zhang, Yali Wang, Rui Yan, Lina Yao, Xiaojun Chang, and Yu Qiao. Dual-ai: Dual-path actor interaction learning for group activity recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2990-2999, June 2022.
[94] Haritha Thilakarathne, Aiden Nibali, Zhen He, and Stuart Morgan. Pose is all you need: the pose only group activity recognition system (POGARS). Machine Vision and Applications, 33 (6):95, October 2022.
[95] Honglu Zhou, Asim Kadav, Aviv Shamsian, Shijie Geng, Farley Lai, Long Zhao, Ting Liu, Mubbasir Kapadia, and Hans Peter Graf. Composer: Compositional reasoning of group activity in videos with keypoint-only modality. In Proceedings of the 17th European Conference on Computer Vision (ECCV), 2022.
[96] Hangjie Yuan, Dong Ni, and Mang Wang. Spatio-temporal dynamic inference network for group activity recognition. In JEEE/CVF International Conference on Computer Vision (ICCV), pages 1456-7465, 2021.
[97] Ralph Tyrell Rockafellar. Convex analysis. Princeton Landmarks in Mathematics and Physics. Princeton University Press, Princeton, NJ, December 1996.
[98] Jie Hu, Li Shen, and Gang Sun. Squuezr-annd-excitatio networks. In Conference Computer Vision and Puttern Recognition (CVPR), pages 7132-7141, 2018.
[99] Rupesh Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In Twenty-eighth Conference on Neural Information Processing Systems (NeurIPS), volume 28, 2015.
[100] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
17

---

Appendix
The appendix is organized follows:
A. Supplementary Analysis of CHASE
B. Code for CHASE
C. Details of Multi-Entity Action Recognition Datasets
D. Evaluation Metries
E. Implementation & Configuration Details
F. Supplementary Experimental Results
G. Limitations & Broader Impacts
H. Licenses for Used Assets
A Supplementary Analysis of CHASE
A.1 Preliminaries
Here we clarify some crucial definitions in the skeleton-basee multi-entity action recognition task as follows.
Definition (Skeleton Sequence of A Multi-Entity Action). Suppose that E interactive entities (e.g. persons) engage in purposeful act during period of time T, and the pose of each entity is indicated by J/ joints with C Cartesian coordinates. We can define the skeleton sequence of multi-entity action as X E [CXXXXXE,
Definition 3 (Skeleton-based Multi-Entity Action Recognition). We define the task as finding the optimal estimator & of the mapping & X Y, where X is the skeleton sequence of a multi-entity action and Y is its corresponding label.
Definition (Joints & Bones). A joint within a skeleton is a point in the Cartesian coordinate system, which can also be viewed as vector (1 < J). A bone within skeleton is differential vector of two joints pú and Pj (1 < i,j S J), if and only if the two joints are connected or the same one according to a prior graph (e.g. the bone connection of the human body).
I.E.
Dimension
GCN Transformer Backbone for Single Entitt Action
J.
All Entities
4FP
Figure 6: The common practice in single-entity action recognition models to recognize multi-
entity actions. This late fusion strategy is used in many recent works [36 37 38 39 40).
We illustrate the common practice 36
when single-entity action recognition models meet multiple entities using Fig. 6 With vanilla world coordinates as input, they usually concatenate each entity along batch dimension, and extract dibeemsinnii features of each entity separately using GCN or transformer backbone. Subsequently, the individual features get averaged before undergoing global average pooling and full connection layers. Notably, this common practice based on an empirical assumption that each entity is independent and identically distributed.
39
A.2 Adaptive Shift Analysis Under Bone Representation
Most existing research uses Def. for bone modality definition [36. the learned adaptive shift is ineffective because shifting the origin does not affect the Fuclidean distance between any two points. However, the adaptive shift could still prove effective bones are defined differently.
54
56) In this context,
18

---

Table 7: Statistics of Multi-Entity Action Recognition Datasets
Datasets
Annolation
#Actions
#Joints
#Clips
#Valid Frames
#Entities
#Participants
Body
Hand
Object
NTC Mutual
10,347
2,00
NTC Mutual 26 42
26
25
24,732
59.36
2,00
106
H20 13
36
21
933
97.29
3.00
4
Assemblyll 101
1,380
21
85,252
105.91
2,00
53
CAD
17
2,511
5.22
VD 86
8
17
4,830
10:00
13.00
Definition (k-hop Bones (37 We denote X, as the joints at the moment in skeleton sequence. The k-hop Bone (()) at moment can formulated as
X (I P
(13)
where k 2 1, and P E RJxJ is binary adjacency matrix of directed graph without bi-directional edges.
In the context of human body skeletons, Def. J the joint j.. representing the center of the spine, as the root to construct the directed graph. If = max,d(v) + 1, where maz,d(() means the max
hop of all joints to the root, then the k-hop Bone X aligns with the joint definition in Def. 4 However, when = 1, this equivalence doesn't necessarily hold for the bone definition Def. The reason is that some joints (such the root) may have in-degree, causing them to retain their original joint coordinates in Def. Therefore, the k-hop bones may still contain joint information that can be modified through adaptive shift.
A.3 Analysis of Gradient
11 where go denotes the composition of MMD (Fq. and ICHAS (6) with CLB (Eq. 8) We assume that the gradient of ge with respect to the parameters exists. Since adopts uniform sampling, we note that the discrete probability density function r(z) of is independent of the parameters 0. Thus we have
Consider the expectation (E [go (z) in Eq.
(xEr(s)lo((e) = Vol Σ r=))
Σ Vogo
(14)
(o) Vogo
which indicates that the gradient of the expectation (e is equivalent the expectation of the gradient (E)())) ( [=)oo(=). The latter can be approximated using Monte Carlo methods.
B Code for CHASE
CHASE is implemented as a wrapper for various skeleton- based action backbones, as illustrated in Algorithm [ Line 18 indicates input of batch size N. channel C. number of frames T, number of joints V, number of entity M. Line 19 represents Eq. g which is to map X to W with function eb.
Line 21 represents the formulation of p² in Eq, 3 Line 25 indicate the subtraction in Eq. Line 26-27 are mini-batch sampling strategy for the Mini-batch Pair-wise Maximum Mean Diserepancy
Loss. Line 28 represents the hingle-entity backbone. Our code is publicly available at https htt.cuubccoo//eoliiee//////////e
with MIT license.
C Details of Multi-Entity Action Recognition Datasets
We conduct experiments on range of datasets, including NTU Mutual 11 (a subset of NTU
RGB+D 41). NTU Mutual 26 (a subset of NTU RGB+D 120 42).. H20 [13, Assembly101
(ASB101) 12]. Collective Activity Dataset (CAD) (85) and Volleyball Dataset (VD) (86). Table
19

---

Algorithm CHASE Wrapper: PyTorch-like Pseudo Code
1:
class CHASEW
2: def init (self,, backbone, in_channels, num frame, num_point, pooling_seg, num_entity,
c1, c2):
3:
Super(CHASEErappee, self). init
4:
out_channel = num num_frame num_point num_entity
5:
self.pooling_ seg = pooling_seg
6:
self.seg num
self.seg num list [(num frame//sell.poolng_eeese __e[[])), (num polio// seg(1]),
(num entity//slll.oooin__eeee
8:
self.seg num self.seg_num laargg_num__sssii [is[[]] self.seg_ [agg_num__ss]]]]] self.seg_num
9:
self.shift = Sequenttui
nn. Conv3d(in channels=in channels, out channels=cl, kernel kernel_size=l),
11:
self.pooling_seglle seg[]], self peolipolling_seeeeeeeeeee seg(2])),
md,CCon33d(c,,, c2, 1, bias=False).
13:
Nm.Rel.U(inplace=TTree
an.Conv3d(c2, out_channel, 1, bias=False),
15:
16:
self.backbone backbone
17:
def (self, x):
18:
N. C, T, V, x.size()
19:
self.sshitt(xx.viewc.h V
20:
rearrange(x, 'n c v v ne m)', ET, m=M, v=V).comtiguous)
21:
= (tx @ (ffofttma(ddim=)) (onostnnxdddmm...mmmmmeee///aaaa sf rearrange(sf, 'n (T V M) (t > (T (V v) (M m)',
self.seg, self.seg_num)
22:
23:
T=self.poolig__seg00 V= M=seef..ooling_seg)
24:
==eeff.ee lavee__nmmllssse v=self.seg_nnmm larre__nmm_lssse [is m=self.seg_ num list .ontiigguuss
25:
x xf
if self.training:
27:
pairs MiniBatchSampling(X)
out self. self.backbone(x)
self. self..trinneee training:
30:
return out, pairs
else:
32:
return out
provides details about each dataset, including their annotation types, numbers of action categories, numbers of joints, numbers of clips (samples), averaged counts of valid frames, counts of entities engaging in a multi-entity action, and numbers of participants in data collection. For CAD [85 and VD [86), which both capture videos in the wild with variety of individuals, difficult to determine the exact number of participants.
D Evaluation Metrics
In this section, we provide the detailed formulation for metrics ablation study. Given two discrete probability distributions and Q defined on the same sample space X,, we define the following metrics:
Averaged Kullback-Leible Divergence (Avg KLD):
Aug DKr. D (PI DKI. (QIP)
P(z)
Q(x) log (xx)
(15)
Σ p log
+ ΣQ(x)
]]
Q(x)
P(x))
20

---

Input
Model
Loss
30 J oints IX,
Backbones
Task Objective
CE
Action Label Untera
CTR GCN
-O
Entty
Action Label (indiiddal?
OR
CHASE Wrapper
InfoGCN
2D (x
CE
x12
GAP
Attion Label
Objective
InfoG CN
OR
- Prcca)
STSA
Mini-bathh Pair-wise Sampling
IRBF Kernts)
NTU
H20
ASB101
CAD
Figure 7: Implementation details for different models and benchmarks. CHASE can be adapted
various backbones including CTR-GCN (36), InfoGCN (37), STSA-Net (40) and HD-GCN (38
We implement linear classification heads for all the models and benchmarks, except for InfoGCN (37 Individual labels utilized action recognition performance in CAD 85 and ASB101 [12]
auxiliary classification objective further improve the group
Jensen-Shannon Divergence (JSD):
(QQI,M, JSD(P D(PIIM D(QIIM),
(16)
where M = J Q) is mixture distribution of P and Q
Bhattacharyya Distance (BD):
DB(P,Q)= --h Σ v(())(xx).
(17)
ZEX
Hellinger Distance (HD):
H(P,Q) = V2
Σ
(PP()) (Q(a))
(18)
ZEX
Maximum Mean Discrepancy (MMD):
MMD Q)
sup (E[[(X) E(f(Y))) 1
(19)
where [.] is the expectation of any function f the RKHS H.
When adopting these metrics in our experiments, the distributions are generated by the kernel density estimation (KDE). We sample the points 30 times with different seed initializations and report the averaged measurements.
E Implementation & Configuration Details
In this section, we provide more details of our experimental setup and model implementation for each benchmark. Experiments are conducted with 8 GeForce RTX 3070 GPUs (GPU Memory: 8GB), using torch version 1.9.+cull 11, torchvision version 0. 1.1000+cul1 and CUDA version 1.4. CTR-GCN (36), InfoGCN 37 STSA-Net 40 and HD-GCN 38 are chosen as our baseline models. To ensure fair comparisons, we adopt single intra-skeleton modality without multi-modality fusion,
following 35). Our implementation details are illustrated in Fig.
21

---

E.1 Dataset-related Configurations
X-Sub and X- K-Veew criteria [41 are adopted in NTU 42 are used in NTU Mutual 26. We evaluate models with only 3D joint inputs, applying data augmentations such as random rotation and spatial shift. During training and testing, we employ temporal cropping and resizing, adjusting based on the number of valid frames. Notably, we use distinct percentage intervals for training (0.5,1) and testing (0.95), For experiments conducted on the entire NTU-RGB+D 120 dataset, we maintain identical settings as those used for NTU Mutual 26.
NTU Mutual 11 [41 & NTU Mutual 26
42
Mutual 11, while X-Sub and X-Set criteria
H2O 13. We follow the training, validation, and test splits described in [3] To maintain consistency in GCNs, we use the hand graph structure, originally designed for human hands, for both hand entities and object entities. Models are evaluated with only 3D joint inputs. The same augmentations as NTU Mutual 26 are adopted in this benchmark.
ASB101 [1). We follow the training, validation, and test splits outlined in 12 for evaluations. 1,380 Fine-grained actions (verb & noun) are adopted as labels in experiments. We evaluate models with only 3D joint inputs. The same augmentations as NTU Mutual 26 are adopted in this dataset, except that the training percentage interval of the temporal cropping and resizing is set to (0.75,1).
We adopt the same data augmentations, group action categories, individual labels and train-test split in [95]. But different from (95), only 2D joint coordinates are used in our experiments. Individual labels are used as an auxiliary classification objective, as presented in Fig. 7
CAD 85
VD [86).
We follow the same data augmentations, group action categories, individual labels and Original train-test split in 195]. But different from (95), only 2D joint coordinates are used as input features in experiments. Besides, individual labels are leveraged as an auxiliary classification objective to further improve the group action recognition performance, volleyball position represented as X E RTxC To ensure inter-entity consistency, we apply padding to fit the shape X E IR RROxTxJx
shown in Fig. 7 The
To maintain consistency in GCNs, we employ human body graph structure for both human body entities and volleyball entities.
F,2 Model-related Configurations
CHASE, We set default segment size 11, G 64, C2 M and 0.1 in CHASE. In all experiments, we maintain consistent configurations for baseline models to ensure fair comparisons between models incorporating CHASE and their respective vanilla counterparts. To avoid unnecessary verbosity, we present implementation specifies solely for NTU Mutual 26. For training details pertaining to other benchmarks, please refer to the code repository at lhttps //github. tegithub.com//ii Ntweeeeeoooeeeaaeeee
CTR-GCN (36). Cross entropy is used the recognition loss function. SGD optimizer is used with Nesterov momentum of 0.9, initial learning rate of 0.1 and a decay rate 0.1 at the 80th and 100th epoch. Batch size is set 64. With the first 5 warm-up epochs, the training process terminated after 110 epochs.
InfoGCN (37]. By taking k 1, InfoGCN adopts their definition of 1-hop Bones 37 as input. Cross entropy is used as the loss function with label smoothing factor 0.1 and temperature factor 1.0. The information bottleneck objective [37] is also employed as the auxiliary loss. Following 377,
set A = 0.1, A2- = 0.0001, and the μ (=(y)) of each action class as random orthogonal vectors with a scale of 3. Diverging from conventional backbones, we substitute the standard classification head with the InfoGCN head, depicted in Fig. [ SGD optimizer is used with Nesterov momentum of 0.9, weight decay of 0.0005, initial learning rate of 0.1 and decay rate 0.1 the 90th and 100th epoch. Batch size is set to 120. With the first 5 warm-up epochs, the training process is terminated after 110 epochs.
STSA-Net [4]]. Cross entropy is used as the loss function with label smoothing factor 0.1 and temperature factor 1.0. SGD optimizer is used with Nesterov momentum of 0.9, initial learning rate of 0.1 and a decay rate 0.1 at the 60th and 90th epoch. Batch size is set to 64. With the first warm-up epochs, the training process is terminated after 110 epochs.
HD-GCN 38
We set CoM = in the hierarchy graph generation and evaluate on this setting. Cross entropy is used as the loss function. SGD optimizer is used with Nesterov momentum of 0.9,
22

---

NTU Mubual
Vanila
CHASE [Ours)
ASB101
CAD
Vanila
CHASE [Ours]
Samples/P
Samples/Pintts
More
Samples/Poins
Figure 8: Qualitative results on multi-entity action recognition datasets. For visual clarity. we display or 3 estimated entity distributions for each test set. Each subplot shows the projection of the multi-variant normal distributions generated by mean vectors and covariance matrices. Different entity distributions denoted by distinct colors. CHASE effectively mitigates inter-entity distribution discrepancies while preserving potential entity orderliness in these datasets.
initial learning rate of 0.1 and a decay rate 0. I at the 80th 100th epoch. Batch size is set to 64. With the first 5 warm-up epochs, the training process is terminated after 110 epochs.
F
Supplementary Experimental Results
More Analysis on Table In Table we observe that CHASE yields varying degrees of accuracy improvement across different baseline models and benchmarks. The performance gains are influenced by both the backbone models and the datasets, as CHASE functions as an additional normalization step that mitigates data bias introduced by inter-entity distribution discrepancies. For baseline backbones, this is owing differences in their backbone architecture design, parameter count and training objective. For example, STSA-Net [40] is relatively large backbone based on transformer architecture, which does not rely the prior definition of the skeleton graphs. Its adaptability to different graph structures makes outweigh GCN- based backbones in H2O benchmark. Another example is InfoGCN [7 which leverages auxiliary information bottleneck objective in its training. Though this method is proven more effective than the other backbones in some interaction settings, it doesn't 't ensure better performance in hand-to-object interaction and group activity benchmarks. For different benchmarks, it is owing to differences in data scale and label space (see Table For example, ASB101 is extremely challenging benchmark for its over 80,000 samples and 1,380 target categories. Therefore the accuracy improvement is modest compared with the other benchmarks.
More Qualitative Results. Fig. S visualizes how CHASE works with multi-entity skeletal sequences. By integrating CHASE, different entity distributions become more similar in both aspects of mean and covariance. It demonstrates that CHASE can lower the inter-entity distribution discrepancy, especially obviously in NTU Mutual 11, NTU Mutual 26, CAD, whose entities have orderliness. For H2O, ASB 101 and VD, whose entities are characterized by an intrinsic order (e.g. left hands, right hands, left-side volleyball players, right-side volleyball players, and objects), CHASE can also preserve their orderliness by letting the distributions be similar but different.
Evaluations on Mixed Recognition of Single-Enttity & Multi-Entity Actions. Table 5 concludes the action recognition results on the entire NTU RGB+D 120 [42]. By integrating CHASE, the baseline model gets accuracy improvement by 0.41% % and 0.05% X-Sub and X-Set, respectively.
23

---

(a) Confusion Matix (Vanila)
(b) Confusion Matrix (C (CASE))
(c) Comparison Category Accuracy
Figure 9: Confusion Matrices & Comparison of Category-level Accuracy NTU Mutual 26 X-Sub. (a) Confusion matrix of the vanilla CTR-GCN, (b) Confusion matrix of CTR-GCN with
CHASE. (c) We present detailed comparison of the category-level accuracy between the vanilla CTR-GCN (blue) and CTR-GCN with CHASE (orange). These results demonstrate the effectiveness of CHASE by improving recognition accuracy for most categories.
This suggests that CHASE is effective even in mixed recognition settings. However, the improvement is modest because single actions are the dominant category this dataset.
Analysis on Efficiency. We analyse CHASE's efficiency on NTU Mutual 26 configurations. Our proposed CHASE is implemented as a backbone wrapper, adaptable to variety of single-entity action models. As presented in Ta- ble X the number of trainable parameters is about 26.37 k, which only increases number of backbone's parameters by 1 1.-2%. We can approximate that the number of trainable parameters is increased by (U + 1+ C2) C + C2 U. For computational complexity, the FLOPs of CHASE is approximately 2.50 M. This analysis proves that CHASE is both efficient and lightweight for benefiting multi-entity action learning.
Table 8: CHASE Trainable Parameters Method
# Param. (M)
CTR-GCN 136
1.44
CHASE
1.46
InfoGCN [37]
1.54
+ CHASE
1.57f1.9%%
STSA-Net [40]
4.13
CHASE
4.1670.60%
HD-GCN 38
1.65
+ CHASE
1.68t1.60 6.0% 60%
Segment Size of Squeeze Operator. In CLB, which is
Table 9: Ablation on Segment Sizes
also the mapping &i to weight matrix, the squeeze oper-
(T", J', E')
Acc (%)
ator squeezes the tensor to specific shape, denoted as
(1, 1,
91.30
(T'J" "'). This segment size determines the point set in a multi-entity action sequence to which ICHAS ap- plies. For example, the segment size (1, 1, 1) indicates that ICHAS applies to all the points, and (1, 1, 2) means two different ICHAS apply to two entities separately. Table evaluates various segment sizes of the squeeze operator, demonstrating that the global ICHAS with
(1, 1,
91. 91.28 (=0.19)
(2, 1)
(0.00)=
(4, 1, 1)
(1.03 (20.09)
51, 1)
g
the segment size (1, 1, 1) achieves the best performance compared with the other settings. Therefore, we choose the default segment size as (1, 1, 1) in all the experiments. Besides, the reduction ratio between C1 and C2 is determined according to the experimental results in 98]
Weight À for MPMMD. Fig. 10 evaluates different values of the trade-off weight factor X in Eq. 12 varying from 10 to 10º. On NTU Mutual 26, it achieve the best performances when adopting À = 0.1 for MPMMD loss. It corresponds to our claim that MPMMD is an auxiliary objective to guide discrepancy minimization, additional to the recognition task objective. We also conduct experiments with MPMMD across variety of M values but find insignificant differences in performance. Hence, we set M = 1 for computational efficiency.
31.40
40.20
92.00
40.00
$0.00
30
10
Analysis on Confusion Matrix & Category-level Accuracy. We present the confusion matrices of the vanilla CTR-GCN and CTR- GCN with CHASE in Fig. 9 (a) (b). It indicates that our proposed CHASE is able to assist the backbone to differentiate similar multi- entity actions better. Fig. 9 (c) reports the category-level accuracy for 26 kinds of person-to-persee interactions. We observe that in very few categories, their performance slightly drops. One possible
Lambda scalo
Figure 10: Ablation study on different As for MPMMD.
24

---

Table 10: Analysis of Performances with Test-Time Skeleton Noises and Masking
+ Noise (%)
Mask (%)
Test-Time
0 10
0 0= 10
Pm 10
Pm 10
CTR-GCN [36]
88.55(10.03)
(10.3))
(10.13
56.37 (=0013)
+ CHASE
(1.24(40.01)
82.53
88.57 (#0.03)
60.65
reason that these actions, like point finger, rely heavily on cues from the individual movements of one of the entities, instead of the multi-entity interactions. This might not be addressed by mitigating inter-entity distribution discrepancies. But it could still conclude from Fig. (c) that adopting CHASE can achieve accuracy improvement for most of the categories, e.g. wield knife (+8.50%), shoot with gun (+8.00%), whisper (+5.39%), and punch/slap (+4.37%).
Analysis of Performances with Test-Time Skeleton Noises and Masking, To evaluate the robust- ness, we intentionally corrupt the multi-entity skeleton sequences with noise and masking during the inference phase. This aims at resembling possible skeleton occlusions or estimation errors during the test time. The noises X,, N(u, o²) used in this experiment are normally distributed with mean μ O and standard deviations = 10-3 10-2 Masking strategies are randomly masking the multi-entity skeleton sequences with probabilities Pmn 10-2, 10-1 Table 10 reports the averaged top-1 accuracy and its standard deviation in runs with several seed initializations for noises and masks. For recognizing multi-entity actions with corrupted test-time inputs, CTR-GCN integrating with CHASE consistently outperforms the vanilla counterpart, showcasing its robustness.
G Limitations & Broader Impacts
This work proposes the Convex Hull Adaptive Shift for Multi-Entity Action Recognition to resolve the inter-entity distribution discrepancies. Although CHASE offers a generic framework for various types of multi-entity actions, such as person-to-person interactions, hand-to condeto-object interactions, hand-to-hand interactions and group activities, application to single-entity actions warrants further investigation. One potential approach is to consider different parts as multiple entities, such as treating different limbs of human body as distinct entities. Moreover, it's promising to apply CHASE- like designs to the recently-develos enhance their performances on multi-entity skeletal data. These areas of exploration are left for future research.
human-centi foundation models [17 29 30 B1 32
33 34 to
This paper focuses on multi- mutti-entity action recognition, field with broad applications in physical human- interaction, social scene understanding, multi-agent systems, surveillance, healthcare monitoring, etc 224 22
27 28 23). Our work contributes to advancements in these domains by enhancing efficient and effective skeleton-based learning of multi-entity actions. Although the use of human-centered data can pose privacy concerns, our study utilizes only skeletal data estimated from sensors or RGB videos, thus mitigating potential privacy and ethical issues.
25
26
H Licenses for Used Assets
Datasets:
NTU Mutual NTU RGB+D 41 Custom (research-only, non-commercial, attribu- tion)
NTU Mutual 26 / NTU RGB+D 120
Custom (research-only) F
H2O [3]: Custom (research-only, non-commercial) |
Assembly101 12 cense
Creative Commons Attriuttion 4.0 International Li-
Collective Activity Dataset
85
Unknown
asp
httpe://booatasett.ttha htw.:assttteeth..c......
http://ccrettivecommnns. org http://ssoe..reooooooeeeoo//////
25

---

Volleybali Dataset 86
BSD 2-Clause license
Models:
Creative Commons www.ltatioooeoooooooooneneeeiiii 4.0 International License
CTR-GCN
36
InfoGCN
37
Unknown
STSANet
40
MIT License
HD-GCN
38
MIT License
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect this paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work, provided in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
We provide proof of our proposition 1 in Section 3.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe our proposed CHASE clearly and fully in Section 3. Section 3.2
and Section
3.3
We illustrate our experimental setting and details in Section and
Appendix
Our code is publicly available at //tttps://githi cwwc /te//Neolllii aeteoolizerr CHASE
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code publicly available at
We have provided sufficient instructions to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
biiaps..c/public//uue
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We illustrate our experimental setting and details Section 4.1 and Appendix We also provide all the configuration files in our code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: T the experimental results on different benchmarks, reporting the averaged t accuracy and its standard deviation in runs with several seed initializations. In Table 3, we sample the points 30 times with different seed initializations and report the averaged measurements with standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided sufficient information the computer resources in Appendix E
Experiments are conducted with 8 GeForce RTX 3070 GPUs (GPU Memory: 8GB), using torch version 1.9.0+cul torchvision version 0.10.0+cu11 11 11, and CUDA version 11.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: pettps://neuripeee httpp//cc//thhcccsseeeeeeell
Answer: [Yes]
Justification:
This research conforms in every respect with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative socictal impacts of our work, provided in Appendix
Guidelines:
30

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We carefully cited the original papers for existing assets and included their licenses in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
31

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code is publicty available at httttp:::/gtbu...oo//////// htgpthub.com//eeolliee/////////a with MIT license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "eGQRe24m50", "reviewer": "Reviewer_N2tt", "summary": "The paper proposes a normalization method for skeleton-based multi-entity recognition based on finding the center of mass within the convex hull of the spatio-temporal domain of the point cloud defined by the skeletons over a sequence. The main idea is to \"center\" the world of skeletons to unbias the subsequent detector and boost their performance. Building on this motivation, the authors find that a fixed, learnable parameterized network can be used for that purpose, facilitating the inference. The experiments demonstrate that their proposed algorithm boosts performance over the corresponding baselines", "questions": "I would insist on the authors to please elaborate a bit better on the motivation and an example where former normalization leads to the classifiers to produce wrong results, with their method mitigating such problem.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is technically sound and the authors follow a proper mathematical derivation that leads to the design of CHASE in a clever manner. \n\nThe paper includes an extensive supplementary material with code and further analysis that make the paper rather complete. \n\nThe method is elegant and simple, providing with a very efficient, lightweight network that shows provable performance on a broad variety of datasets. Ablation studies are conducted to validate the proposed parts, as well as to compare against other normalization alternatives. \n\nThe paper is well documented with an extensive coverage of related literature.", "weaknesses": "Overall I believe that the presentation should be improved, clearly stating the contribution and motivation for it. It takes a good read to understand that what the authors are proposing as a lightweight network is the result of mathematically deriving an iterative approach for normalization of skeletons within their convex hull. It should be clearly stated that the method aims to accompany other methods for multi-entity activity recognition by adding an extra normalization step, which consists of what is presented in Section 3. \n\nSimilarly, the sketch depicted in Figure 1 is a bit confusing and does not really illustrate what the authors aim to solve. I would suggest the authors to provide a clear motivation example that leads to their method. The caption in Fig. 1 is rather poorly written (I did not understand it at least).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of the methodology of CHASE, proper mathematical derivation, and extensive experiments. We understand that your concerns may stem from clarity and presentation of our contributions and motivation. Below, we address these issues one by one and have clarified them in the revised paper. We hope our response effectively addresses your concerns.\n\n---\n**Q1:** *The presentation should be improved, clearly stating the contribution and motivation for it. It should be clearly stated that the method aims to accompany other methods for multi-entity activity recognition by adding an extra normalization step, which consists of what is presented in Section 3.*\n\nWe deeply appreciate your time and thorough review of our paper. We apologize for any confusion in the presentation of our contribution and motivation. We have revised the final paragraph of the Introduction section to better articulate our contributions and motivation.\n\nThe contributions of this paper are three-fold:\n\n1. To the best of our knowledge, we are the first to investigate the issue of inter-entity distribution discrepancies in multi-entity action recognition. Our proposed method, Convex Hull Adaptive Shift for Multi-Entity Actions, effectively addresses this challenge. **Our main idea is to adaptively repositioning skeleton sequences to mitigate inter-entity distribution gaps, thereby unbiasing the subsequent backbones and boosting their performance.**\n2. **Serving as an additional normalization step for backbone models, CHASE consists of a learnable network and an auxiliary objective.** Specifically, the network is formulated by the Implicit Convex Hull Constrained Adaptive Shift, together with the parameterization of a lightweight Coefficient Learning Block, which learns sample-adaptive origin shifts within skeleton convex hull. Additionally, the Mini-batch Pair-wise Maximum Mean Discrepancy objective is proposed to guide the discrepancy minimization.\n3. Experiments on NTU Mutual 11, NTU Mutual 26, H2O, Assembly101, Collective Activity Dataset and Volleyball Dataset consistently verify our proposed method by enhancing performance of single-entity backbones in multi-entity action recognition task.\n\nOur motivation is:\n\nWhen using *Vanilla* (a common practice), the estimated distributions of joints from different entities show significant discrepancies, as shown in Figure 1(a). These discrepancies can introduce bias into backbone models, leading to suboptimal optimization and poor recognition performance, as depicted in the upper histogram of Figure 1(b). Although *S2CoM* (an intuitive baseline approach) can reduce these discrepancies, it results in wrong predictions by the classifiers due to a complete loss of inter-entity information. To address the inter-entity distribution discrepancy problem, we propose a Convex Hull Adaptive Shift based multi-Entity action recognition method (CHASE). Serving as an additional normalization step, CHASE aims to accompany other single-entity backbones for enhanced multi-entity action recognition. Our main insight lies in the adaptive repositioning of skeleton sequences to mitigate inter-entity distribution gaps, thereby unbiasing the subsequent backbone and boosting its performance.\n\n---\n**Q2:** *The sketch depicted in Figure 1 is a bit confusing and does not really illustrate what the authors aim to solve.  I would suggest the authors to provide a clear motivation example that leads to their method. The caption in Fig. 1 is rather poorly written.*\n\nWe apologize for any confusion caused by Figure 1. We clarify the motivation example in Figure 1 that leads to our method as follows:\n\n1. **What we aim to solve**: When using Vanilla (a common practice), the estimated distributions of joints from different entities show significant discrepancies, as shown in Figure 1(a). **These discrepancies can introduce bias into backbone models, leading to suboptimal optimization and poor recognition performance**, as depicted in the upper histogram of Figure 1(b). Though S2CoM (an intuitive baseline approach) can mitigate the discrepancies, **it makes the classifiers produce wrong predictions due to a complete loss of inter-entity information**. Therefore, this figure visualizes the problem we aim to solve.\n2. **The effectiveness of CHASE**: Figure 1(a) and the lower histogram of Figure 1(b) highlight that CHASE effectively mitigates these discrepancies. Our method helps reduce bias in the subsequent classifiers, thereby enhancing their performance in the recognition task.\n\nIn the revised version of the paper, we have updated Figure 1 and its caption to clearly illustrate the motivations and contributions mentioned above (see the pdf file in global response). The caption of Fig.1 is modified as follows:\n\nFigure 1: **Inter-entity distribution discrepancies in multi-entity action recognition task.** (a) We delineate three distinct settings: *Vanilla* (a common practice), *S2CoM* (an intuitive baseline approach), and *CHASE* (our proposed method). Column 2 illustrates spatiotemporal point clouds defined by the skeletons over $10^4$ sequences. Column 3-5 depict the projections of estimated distributions of these point clouds onto the x-y, z-x, and y-z planes. These projections reveal significant inter-entity distribution discrepancies when using *Vanilla*. (b) The discrepancies observed in *Vanilla* introduce bias into backbone models, leading to suboptimal optimization and poor performance. Although *S2CoM* can reduce these discrepancies, it makes the classifiers produce wrong predictions due to a complete loss of inter-entity information. With the lowest inter-entity discrepancy, our method unbiases the subsequent backbone to get the highest accuracy, underscoring its efficacy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjlh4abZb7", "reviewer": "Reviewer_hFtj", "summary": "This paper focuses on the interesting problem of the normalization strategy for multi-entity skeletons in skeleton-based action recognition. The proposed method is intuitive, and the authors provided detailed implementation details for reproduction. However, this work is unclear, and the experiments are unconvincing.", "questions": "What is the main difference between the proposed method and the simple strategy of shifting the origin of multi-entities to their common center? Please add a comparison experiment with this method.", "limitations": "(1)\tThe purpose of the multi-entity action recognition task is not clear. Is it to recognize each individual’s action or to classify group activities? If the purpose varies across different datasets, please clarify this in the experiment. Additionally, I am curious whether the optimal normalization strategy differs for these two purposes. For example, the method used in S2CoM seems more suitable for recognizing each individual’s action.\n(2)\tWhat is the main difference between the proposed method and the simple strategy of shifting the origin of multi-entities to their common center? Please add a comparison experiment with this method.\n(3)\tAlthough the normalization strategy is an important trick and can bring significant improvement in the action classification task, I still have a concern about whether it is worth conducting an additional network to achieve this simple trick by introducing extra learnable parameters. Some heuristic strategies may be more efficient and general. Accordingly, can the proposed module be transferred among different datasets without retraining the module?", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper focuses on the interesting problem of the normalization strategy for multi-entity skeletons in skeleton-based action recognition. The proposed method is intuitive, and the authors provided detailed implementation details for reproduction.", "weaknesses": "(1)\tThe purpose of the multi-entity action recognition task is not clear. Is it to recognize each individual’s action or to classify group activities? If the purpose varies across different datasets, please clarify this in the experiment. Additionally, I am curious whether the optimal normalization strategy differs for these two purposes. For example, the method used in S2CoM seems more suitable for recognizing each individual’s action.\n(2)\tWhat is the main difference between the proposed method and the simple strategy of shifting the origin of multi-entities to their common center? Please add a comparison experiment with this method.\n(3)\tAlthough the normalization strategy is an important trick and can bring significant improvement in the action classification task, I still have a concern about whether it is worth conducting an additional network to achieve this simple trick by introducing extra learnable parameters. Some heuristic strategies may be more efficient and general. Accordingly, can the proposed module be transferred among different datasets without retraining the module?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We appreciate your recognition of the motivation behind CHASE and the extensive implementation details for reproducibility. We understand that your concerns may stem from some misunderstandings and the presentation of ablation studies. Below, we address these issues one by one and have clarified them in the revised paper. We hope our response will address your concerns.\n\n---\n**Q1:** *The purpose of the multi-entity action recognition task is not clear. Is it to recognize each individual’s action or to classify group activities? If the purpose varies across different datasets, please clarify this in the experiment. Additionally, I am curious whether the optimal normalization strategy differs for these two purposes.*\n\nWe apologize for any confusion.\n\n1. The purpose of the multi-entity action recognition task is consistently to **recognize the comprehensive actions performed by multiple entities**, rather than individual actions. Multiple entities can include human bodies, hands, and objects. This is discussed in Section 1 (lines 21-23) and Section A.1 (lines 613-619) of our submission. This goal aligns with many related works focusing on interactive actions and group activities [11,25,85]. To avoid misunderstandings, we have clarified this better in Section.1 of the revised version.\n2. The aim **does not** vary across different datasets. All experiments are conducted with the purpose of recognizing multi-entity actions. Notably, in CAD and VD benchmarks, individual labels are leveraged as an auxiliary objective to further improve the group action recognition performance, as mention in line 708-709 and 712-713. This strategy is a common practice adopted by most models training on these two datasets [83,85].\n3. Exploring whether the optimal normalization strategy differs for these two purposes is indeed an interesting topic. However, it is not the focus of this paper. We will delve into this issue in future work.\n\n---\n**Q2:** *What is the main difference between the proposed method and the simple strategy of shifting the origin of multi-entities to their common center? Please add a comparison experiment with this method.*\n\nIn our initial submission, we have reported the experimental result of the comparison of CHASE and this simple strategy in Table 2, highlighting the superior performance of CHASE.\n\n1. **The main difference is that the common center of mass (CoM) of all entities is just a subset of the search space for** $\\vec{p^*}$ **in CHASE**. As indicated in line 149-150 in our manuscript, their common CoM $\\bar{\\vec{p}}$ is in the open convex hull of $X$, proven by simply taking all $\\tilde{\\alpha}_i=1/U(1\\leq i\\leq U)$. Therefore, **shifting the origin of multi-entities to their common center is just one possible result among all possible ones in CHASE**.\n2. While it is intuitive to shift the origin to their common center, **experimental results shows that this is not always the optimal choice**. In our main draft, we have reported the experimental result of the comparison you mentioned in Table 2 (the last two rows). Our proposed CHASE achieves 91.30\\% top-1 accuracy on NTU 26 X-Sub benchmark, while this simple (denoted as S2CoM†) obtains 90.79\\%. This comparison implies that adopting adaptive $\\vec{p^*}$ for each skeleton sequence sample is superior to this intuitive approach.\n\n---\n**Q3:** *I still have a concern about whether it is worth conducting an additional network to achieve this simple trick by introducing extra learnable parameters. Some heuristic strategies may be more efficient and general.*\n\n**Our proposed CHASE outperforms other alternative strategies**, as mentioned in Tables 2 of our submission. We acknowledge that there is an inevitable trade-off between efficiency and performance. However, we believe our proposed CHASE strikes a good balance in this trade-off. We have addressed your concerns with extensive ablation studies in Tables 2 and 9. In our initial submission, Table 2 presents a comparison with many alternative strategies, including BatchNorm, S2CoM, and data augmentations. Our CHASE achieves **the best recognition performance**, highlighting the advantage of adaptively shifting skeletons to unbias the subsequent classification backbone. Moreover, as shown in Table 9, the number of trainable parameters is approximately 26.37k, which **only increases the number of parameters by 1\\%-2\\%**. These findings show the advantages of adopting CHASE for this task.\n\n---\n**Q4:** *Accordingly, can the proposed module be transferred among different datasets without retraining the module?*\n\nIn most cases, **we can't directly transfer it without retraining, as the input shapes vary across different skeleton datasets**, indicated by Table 7. However, your insightful suggestion prompted us to investigate whether CHASE can be transferred if we align the input shapes of two datasets. We conducted an experiment on a modified version of the H2O dataset, aligning its input skeleton shape to the ASB101 dataset by discarding object poses and sampling 70 frames (as used in ASB101). The results in the following table demonstrate that both the frozen CHASE module (pretrained on ASB101) and the retrained CHASE module improve the performance of the CTR-GCN backbone. This implies the transferability of our proposed CHASE, provided the skeleton sequences are aligned.\n\nMethod | Acc (\\%) | $\\Delta$ (\\%)\n:-------- | :-----: | :-----:\nCTR-GCN | ${48.48}_{(\\pm2.91)}$ | -\n\\+ CHASE (retrained) | ${56.47}_{(\\pm1.59)}$ | $+7.99$\n\\+ CHASE (frozen, pretrained on ASB101) | ${56.61}_{(\\pm4.41)}$ | $+8.13$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c4YWAxdKIJ", "reviewer": "Reviewer_16iH", "summary": "This paper proposes CHASE, a multi-entity skeleton data augmentation/preprocessing technique, to mitigate inter-entity distribution gaps and improve the multi-entity action recognition. Specifically, the authors formulate a new constraint called ICHAS, design a lightweight block CLB to learn the nonlinear mapping from input to the weight matrix in ICHAS, and introduce an objective to guide the discrepancy minimization in CLB training. The authors conduct comprehensive experiments to show the effectiveness of the method.", "questions": "a)\tFigure 2 is very informative and good for understanding the whole method. However, there are still some notations needed to be explained, e.g. the green and red circles under the two skeleton coordinates. It would be helpful if the authors introduce the method details (in sections 3.1 to 3.3) with references to specific parts of Figure 2.\n\nb)\tIt’s good to have multiple runs and report the standard deviation. How many seed initializations exactly do the authors use?\n\nc)\tFrom Table 1, the performance margin varies a lot. On H2O and CAD, the top-1 accuracies have increased by over 9%. Could the authors explain or provide some discussion on this point?\n\nd)\tFrom Table 7 in the appendix, it seems like the authors use 25 ground-truth 3D joints as skeleton inputs for NTU-60 and NTU-120 datasets. Since 2D estimated joints tend to achieve better action recognition performance in dominant models, do the authors verify CHASE using 17 estimated 2D joints with COCO layout as input?\n\ne)\tTable 6 shows the mixed recognition results on the entire NTU-120 dataset with X-Sub setting. Table 8 in the appendix shows the mixed recognition results on NTU-120 X-Sub and X-Set settings. Why not replace Table 6 with Table 8? Similar to Table 5 and Table 9, why not replace Table 5 with Table 9 or integrate the number of parameters into Table 1 (may discard the column of ‘Venue’)?\n\nSmall details (no need to address them, just suggestions):\n\na)\tRegarding equations (9) and (10), it would be better to clarify the definitions of sup(*) and C(E,2).", "limitations": "Although the method has been verified on six benchmarks, these datasets are relatively small, with the number of categories ranging from 4 to 36, except Assembly 101 has 1380 action categories. However, the performance gain on Assembly 101 is very small (<=0.21%). The reviewer has some concerns about the generalization of this method.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "a) The paper is well-written, easy to understand, and well-organized.\n\nb) The method is well-motivated, well-ablated, and the experiments are presented clearly.", "weaknesses": "a)\tThere’re some typos need to be revised carefully, e.g. l131 “be be”, Table 4 “-68.56”.\n\nb)\tRegarding clarity, the captions of Figures and Tables can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and constructive comments. We appreciate your recognition of the writing, motivation and experiments. We understand that some of your concerns may stem from clarity and experimental details. Below, we address these issues one by one and have clarified them in the revised paper. We hope our response will address your concerns.\n\n---\n**Q1:** *There’re some typos need to be revised carefully, e.g. l131 “be be”, Table 4 “-68.56”. Regarding clarity, the captions of Figures and Tables can be improved.*\n\nWe are grateful for your indications of typos and suggestions for improving clarity. We have checked and revised the manuscript for these issues. For example, we have revised the caption of Figure 1 to provide a more concise illustration of our motivation.\n\n---\n**Q2:** *There are still some notations needed to be explained in Figure 2. It would be helpful if the authors introduce the method details (in sections 3.1 to 3.3) with references to specific parts of Figure 2.*\n\nWe have added notation explanations to Figure 2 in the revised version. For example, the green and red circles represent feasible and infeasible $\\vec{p^*}$ as formulated by Eq. 3, respectively. Additionally, we have provided details in sections 3.1 to 3.3 with references to Figure 2 to facilitate understanding.\n\n---\n**Q3:** *How many seed initializations exactly do the authors use?*\n\nWe adopt three seed initializations in most settings, which means three runs from training to evaluation.\n\n---\n**Q4:** *Why does the performance margin vary a lot?*\n\n**As CHASE functions as an additional normalization step, the performance gains are influenced by the bias both in datasets and introduced in backbone models, especially their different architecture design, data scale, and label space.** We have discussed it in detailed in Section F (line 746-757) of our submission:\n\n\"In Table 1, we observe that CHASE yields varying degrees of accuracy improvement across different baseline models and benchmarks. ... For different benchmarks, the variations in performance margins are due to differences in data scale and label space (see Table 7). For example, ... In contrast, H2O and CAD are relatively small in data scale. The significant accuracy increase implies that our CHASE can unbias the subsequent backbone more effectively with limited training data better.\"\n\n---\n**Q5:** *Do the authors verify CHASE using 17 estimated 2D joints with COCO layout as input?*\n\nWe have evaluated CHASE using 17 estimated 2D joints as input on the **CAD and VD benchmarks**, as visualized in Figure 3 of our initial submission. Additionally, we have conducted experiments on the **NTU Mutual 26 dataset** using 17 estimated 2D joints. The results are reported in the following table. It demonstrates that our proposed CHASE also enhances the performance of the vanilla counterparts when using 2D joint inputs.\n\nMethod | X-Sub (\\%) | X-Set (\\%)\n:----|:----:|:----:\nCTR-GCN | ${90.10}_{(\\pm0.05)}$ | ${91.35}_{(\\pm0.15)}$\n**+ CHASE (Ours)** | **${90.56}_{(\\pm0.09)}$** | **${92.38}_{(\\pm0.81)}$**\n\n---\n**Q6:** *Why not replace Table 6 with Table 8? Similar to Table 5 and Table 9, why not replace Table 5 with Table 9 or integrate the number of parameters into Table 1?*\n\nDue to the page limit of the initial submission, we included excerpted versions (Table 5 \\& 6) in the main draft and placed the full versions (Table 8 \\& 9) in the supplementary material. We will replace the excerpted versions with the full tables, if the paper is accepted, as additional content pages will be allowed. Integrating the number of parameters into Table 1 might cause confusion, as the number of model parameters varies across different benchmarks. Therefore, we opted to keep the parameter details separate to maintain clarity.\n\n---\n**Q7:** *It would be better to clarify the definitions of sup(\\*) and C(E,2).*\n\nWe have clarified the definitions in the revised version. The notation $\\sup(\\cdot)$ stands for the supremum. In Eq. (9), $\\sup(\\cdot)$ is used to denote the maximum value that the expression $\\mathbb{E}[f(x)]-\\mathbb {E} [f(y)]$ can attain, where the function $f$ is taken from a specific class of functions. $C(E,2)$ stands for the combination of $E$ things taken 2 at a time without repetition. In Eq. (10), $C(E,2)$ is used to denote the total count of possible entity pairs.\n\n---\n**Q8:** *These datasets are relatively small ... The reviewer has some concerns about the generalization of this method.*\n\nWe provide the following reasons to support the generalization capability of CHASE:\n\n1. We have compared our approach with recent methods in multi-entity action recognition, as shown in the table below. **These methods are typically evaluated on $\\leq 5$ datasets, with some using only a subset of the datasets we have utilized.** In contrast, we have verified CHASE across six diverse benchmarks, including person-to-person interactions, hand-to-object interactions, and group activities. Our approach outperforms these methods, as reported in Table 1 of our submission.\n\n2. Additionally, we have evaluated CHASE on ASB101, which is the largest and most challenging benchmark for this task. **Given the complexity of this dataset, achieving substantial accuracy improvements is inherently difficult.** Although the improvement on ASB101 is modest, it still demonstrates the effectiveness of CHASE in a challenging scenario.\n\nMethod | Venue | \\#Dataset | Max \\#Category | Datasets for Multi-entity Action Recognition\n:----|:----:|:----:|:----:|:----:\nIGFormer | ECCV'22 | 3 | 26 | NTU Mutual 11/26, SBU (200 samples)\nISTA-Net | IROS'23 | 4 | 1380 | NTU Mutual 26, SBU (200 samples), ASB101, H2O\nH2OTR | CVPR'23 | 2 | 45 | H2O, FPHA (1175 samples)\nme-GCN | arXiv'24 | 3 | 1380 | NTU Mutual 11/26, ASB101\nEffHandEgoNet | arXiv'24 | 2 | 45 | H2O, FPHA (1175 samples)\nAHNet-Large | PR'24 | 5 | 26 | NTU Mutual 11/26, CAD, VD, PKU-MMD mutual\n**Ours** | - | 6 | 1380 | NTU Mutual 11/26, H2O, ASB101, CAD, VD", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttEtB5GmjZ", "reviewer": "Reviewer_WotV", "summary": "This paper tackled the issue of inter-entity distribution discrepancies in multi-entity action recognition. The authors proposed convex hull adaptive shift method to minimize the cross entity discrepancies, where CLB and MPMMD are proposed to assist the learning procedure. The method is verified to be effective among various datasets and backbones.", "questions": "1. Improving the Introduction Section:\n\n    a. Why is it necessary to achieve discrepancy minimization (line 54)?\n\n    b.Why do we need to achieve sample adaptive coefficients (line 52)?\n\n    c. Can the authors highlight the motivation behind these needs and improve the links among the proposed items (lines 57-60) with more insights?\n\n\n2. Novelty of the CLB:\n\n    a. How does the concept bottleneck (CLB) differ from the commonly used format of attributes learning in concept learners, such as in Eq. 8?\n\n    b. What are the differences between the CLB and the concept bottleneck models discussed in Shin et al. (2023)?\n\n    c. If the concept learner from existing works (e.g., Wang et al. (2023)) were used, would it perform better than the CLB?\n\n3. Insights in Section 4.2:\n\n    a. Why does the proposed method provide benefits?\n\n    b. Can the authors provide a more detailed analysis to enrich Section 4.2?\n\n4. Computational Complexity:\n\n    a. What is the computational complexity introduced by the proposed method?", "limitations": "yes in supplementary", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper proposed an interesting idea by using implicit convex hull as constraints to achieve adaptive coordinate shift.\n\n2.The proposed approach is verified to be effective on various datasets and backbones.\n\n3.This method can serve as a good contribution to the skeleton-based human action recognition community.", "weaknesses": "1. The introduction section should  be improved. For example on line 54, why do we need to achieve the discrepancy minimization? on line 52, why do we need to achieve sample adaptive coefficients? The motivation should be highlighted. The links among these proposed items should be also improved on line 57-60 and need more insights.\n\n\n2. The novelty of the CLB is limited. The format of attributes learning shown in Eq.8 is commonly used to construct concept learners. What is the difference between the concept bottleneck [1] and the CLB? If you use the concept learner from some existing works, e.g., [2], will it be better than CLB?\n\n\n[1] Shin S, Jo Y, Ahn S, et al. A closer look at the intervention procedure of concept bottleneck models[C]//International Conference on Machine Learning. PMLR, 2023: 31504-31520.\n\n\n[2] Wang B, Li L, Nakashima Y, et al. Learning bottleneck concepts in image classification[C]//Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 2023: 10962-10971.\n\n3. More insights should be given in Section 4.2. Why does the proposed method help? The authors are encouraged to enrich the analysis.\nThe authors are encouraged to discuss the computational complexity brought by the proposed method.\n\n4. The authors are encouraged to discuss the computational complexity brought by the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging and constructive comments. We appreciate your recognition of the methodology, experimental results, and contributions. We understand that your concerns may arise from the need for greater clarity and detail in presenting our motivations and methodology. Below, we address these issues one by one and have clarified them in the revised paper. We hope our response will address your concerns effectively.\n\n----\n**Q1a.** *Why is it necessary to achieve discrepancy minimization?*\n\nThese discrepancies can introduce bias into the backbone models, leading to suboptimal optimization and performance, as shown in the upper histogram of Fig. 1(b). The lower histogram of Fig. 1(b) indicates that minimizing these discrepancies helps reduce bias in the classification backbone, thereby enhancing its recognition performance on this task.\n\n----\n**Q1b.** *Why do we need to achieve sample adaptive coefficients?*\n\nAchieving sample adaptive coefficients is a necessary step to adaptively reposition each skeleton sequence. As shown in Table 4 in our main draft, the recognition accuracy drops to 91.20\\% **without sample-adaptive coefficients**. It implies that sample-adaptive weight representations are important to achieve better recognition performance.\n\n----\n**Q1c.** *Can the authors highlight the motivation behind these needs and improve the links among the proposed items with more insights?*\n\nWe have revised the paragraph to clearly present our motivation and the connections among the proposed components:\n\n\"... **Specifically, CHASE consists of a learnable parameterized network and an auxiliary objective.** The parameterized network can achieve **plausible and sample-adaptive** repositioning of skeleton sequences through two crucial components. First, the Implicit Convex Hull Constrained Adaptive Shift (ICHAS) ensures that the new origin of the coordinate system is within the skeleton convex hull. Second, the Coefficient Learning Block (CLB) provides a lightweight parameterization of the mapping from skeleton sequences to their specific coefficients in ICHAS. Moreover, **to guide the optimization of this network for discrepancy minimization**, we propose the Mini-batch Pair-wise Maximum Mean Discrepancy (MPMMD) as the additional objective. This loss function quantifies ... **In conclusion, CHASE works as a sample-adaptive normalization method to mitigate inter-entity distribution discrepancies, which can reduce bias in the subsequent backbone and enhance its multi-entity action recognition performance.**\"\n\n----\n**Q2a \\& b.** *What is the difference between the concept bottleneck (Shin et al. (2023)) and the CLB?*\n\nAfter carefully reviewing the relevant literature [1,2], we have identified several key differences between our CLB and concept bottleneck models (CBMs):\n\n1. **Motivation**: CBMs are primarily designed to enhance interpretability [1]. Different from CBMs, our CLB is the parameterization of a mapping, which maps the input skeleton sequence to the coefficient matrix. Thus, **while CBMs focus on interpretability, CLB is geared towards adaptively adjusting the skeleton sequence representation to reduce inter-entity discrepancies**.\n2. **Inputs**: CBMs typically require input data $x\\in \\mathbb{R}^{d}$, binary concepts $c\\in \\{0,1\\}^{k}$, and target responses $y\\in Y$ [1,2]. However, **in multi-entity action recognition tasks, there are no binary concepts to work with**, making the direct application of CBMs unsuitable for our task.\n3. **Architecture Design**: The architecture of CBMs, as implemented in [1], is expressed as $\\hat{y} = f(\\delta(g(x))$, where $\\delta$ denotes ReLU, and $f,g$ denotes inceptionv3 \\& MLP. In [2], it's based on slot attention and is formulated as $a_k = \\phi(Q(c_k)^TK(F'))$, where $Q,K$ are nonlinear transformations. Our CLB, however, uses a different architecture: $W=\\psi(X)=W_3\\delta(W_2\\phi(W_1X+b))$. This distinction highlights that **our CLB does not rely on inceptionv3+ReLu+MLP or slot attention mechanisms**.\n\n----\n**Q2c.** *If the concept learner from existing works (e.g., Wang et al. (2023)) were used, would it perform better than the CLB?*\n\nCBMs require binary concepts [1,2], which is **not applicable to the multi-entity action recognition task**. Therefore, adapting a concept learner from existing works to our task is not feasible. Given this, it is not possible to directly compare their performance with our CLB.\n\n----\n**Q3a.** *Why does the proposed method provide benefits?*\n\nBy adaptively shifting skeleton sequences, CHASE effectively mitigates inter-entity distribution discrepancies in multi-entity skeletal data, which can unbias the subsequent classification backbone and boost their recognition performance. \n\n----\n**Q3b.** *Can the authors provide a more detailed analysis to enrich Section 4.2?*\n\nYes, we have provided a more detailed analysis of Section 4.2, as mentioned in Section F (line 746-757) of our initial submission:\n\n\"In Table 1, we observe that CHASE yields varying degrees of accuracy improvement across different baseline models and benchmarks. The performance gains are influenced by both the backbone models and the datasets, as CHASE functions as an additional normalization step that mitigates bias in the backbone introduced by inter-entity distribution discrepancies.\nFor baseline backbones, this is owing to differences in their backbone architecture design, parameter count and training objective. For example, ... \"\n\n----\n**Q4:** *What is the computational complexity introduced by the proposed method?*\n\nWe have discussed the computational complexity in Section 4.3 (line 270-275) \\& Section F (line 774-785) of our initial submission. As presented in Table 9, the number of trainable parameters is about **26.37 k**. We can approximate that the number of trainable parameters is increased by $(U+1+C_2)\\times C_1 + C_2 \\times U$. For computational complexity, the FLOPs of CHASE is approximately **2.50 M**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to all the reviewers for their time, insightful suggestions, and valuable comments. We deeply appreciate the positive recognition from the reviewers regarding our paper’s motivation (hFtj, 16iH, WotV), the elegance and simplicity of our methodology (N2tt, WotV), the rigorous mathematical derivation (N2tt), the extensive experiments demonstrating strong performance across a wide range of datasets and backbones (N2tt, 16iH, WotV), and the thorough implementation details provided for reproducibility (N2tt, hFtj). We address the common concerns raised by the reviewers below:

*1. Presentation of our motivations and contributions.*

We have revised the final paragraph of the Introduction section, Figure 1, and caption of Figure 1 to better articulate our motivations and contributions.

The contributions of this paper are three-fold:

- To the best of our knowledge, we are the first to investigate the issue of inter-entity distribution discrepancies in multi-entity action recognition. Our proposed method, Convex Hull Adaptive Shift for Multi-Entity Actions, effectively addresses this challenge. **Our main idea is to adaptively repositioning skeleton sequences to mitigate inter-entity distribution gaps, thereby unbiasing the subsequent backbones and boosting their performance.**
- **Serving as an additional normalization step for backbone models, CHASE consists of a learnable network and an auxiliary objective.** Specifically, the network is formulated by the Implicit Convex Hull Constrained Adaptive Shift, together with the parameterization of a lightweight Coefficient Learning Block, which learns sample-adaptive origin shifts within skeleton convex hull. Additionally, the Mini-batch Pair-wise Maximum Mean Discrepancy objective is proposed to guide the discrepancy minimization.
- Experiments on NTU Mutual 11, NTU Mutual 26, H2O, Assembly101, Collective Activity Dataset and Volleyball Dataset **consistently verify our proposed method by enhancing performance of single-entity backbones in multi-entity action recognition task**.

*2. Generalization of CHASE.*

- Compared with recent related works, **we have adopted more datasets with a wide range of types of entities and action categories**. As reported in Table 1 of our submission, we have verified CHASE across six diverse benchmarks, including person-to-person interactions, hand-to-object interactions, and group activities. Notably, we have evaluated CHASE on ASB101, which is the largest and most challenging benchmark for this task, featuring over 80,000 samples and 1,380 "verb+noun" categories, with absent object poses. (16iH Q8)
- We have evaluated CHASE using **both 3D joints and estimated 2D joints** as input. (16iH Q5)
- Experimental results demonstrate that **CHASE can be transferred without retraining the module if we align the input shapes of two datasets**. (hFtj Q4)

In addition to addressing the common concerns mentioned above, we have provided detailed responses to each specific question raised by the reviewers. We hope our responses will effectively address the reviewers’ concerns, and we look forward to engaging in comprehensive discussions in the coming days.

Meta-review: 3 of 4 of the reviewers found the paper's contributions significant with strong theoretical derivation for the technique and improved results across a number of benchmarks. The one negative reviewer found the motivation unclear and was not satisfied that model didn't support "transferr[ing] among different datasets without retraining the module". In the opinion of the AC these reasons don't qualify as reasons for rejection. Given this the AC recommends the paper be accepted to NeurIPs.
Decision: Accept (poster)