CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition

Authors: yuhang  wen
mengyuan  liu
songtao  wu
beichen  ding
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 09:42:28

Abstract: Skeleton-based multi-entity action recognition is a challenging task aiming to identify interactive actions or group activities involving multiple diverse entities. Existing models for individuals often fall short in this task due to the inherent distribution discrepancies among entity skeletons, leading to suboptimal backbone optimization. To this end, we introduce a Convex Hull Adaptive Shift based multi-Entity action recognition method (CHASE), which mitigates inter-entity distribution gaps and unbiases subsequent backbones. Specifically, CHASE comprises a learnable parameterized network and an auxiliary objective. The parameterized network achieves plausible, sample-adaptive repositioning of skeleton sequences through two key components. First, the Implicit Convex Hull Constrained Adaptive Shift ensures that the new origin of the coordinate system is within the skeleton convex hull. Second, the Coefficient Learning Block provides a lightweight parameterization of the mapping from skeleton sequences to their specific coefficients in convex combinations. Moreover, to guide the optimization of this network for discrepancy minimization, we propose the Mini-batch Pair-wise Maximum Mean Discrepancy as the additional objective. CHASE operates as a sample-adaptive normalization method to mitigate inter-entity distribution discrepancies, thereby reducing data bias and improving the subsequent classifier's multi-entity action recognition performance. Extensive experiments on six datasets, including NTU Mutual 11/26, H2O, Assembly101, Collective Activity and Volleyball, consistently verify our approach by seamlessly adapting to single-entity backbones and boosting their performance in multi-entity scenarios. Our code is publicly available at https://github.com/Necolizer/CHASE .

Main paper: 
 introduction: 
 Multi-entity action recognition, a challenging task derived from action recognition [1, 2, 3, 4, 5, 6, 7, 8, 9] , aims to find the optimal estimator of the mapping from multi-entity motions to semantic labels, where entities involved can range from human bodies [10, 11] , hands [12] to various objects [13] . Recent approaches predominantly rely on skeletal data for addressing this challenge [10, 11, 14] , given that skeletons serve as a concise representation of spatiotemporal features [15, 16, 17, 18, 19, 20, 21] . This task has broad applications in human-robot interaction [22, 23] , scene understanding [24, 25, 26, 27, 28] , human motion analysis [29, 30, 31, 32, 33, 34] , etc. The Higher The BetterThe Lower The BetterWalk Towards? Walk Apart?Inter-entity Distribution Discrepancies Figure 1 : Inter-entity distribution discrepancies in multi-entity action recognition task. (a) We delineate three distinct settings: Vanilla (a common practice), S2CoM (an intuitive baseline approach), and CHASE (our proposed method). Column 2 illustrates spatiotemporal point clouds defined by the skeletons over 10 4 sequences. Column 3-5 depict the projections of estimated distributions of these point clouds onto the x-y, z-x, and y-z planes. These projections reveal significant inter-entity distribution discrepancies when using Vanilla. (b) The discrepancies observed in Vanilla introduce bias into backbone models, leading to unsatisfactory performance. Although S2CoM can reduce these discrepancies, it makes the classifiers produce wrong predictions due to a complete loss of inter-entity information. With the lowest inter-entity discrepancy, our method unbiases the subsequent backbone to get the highest accuracy, underscoring its efficacy.Experiments have revealed that network architectures tailored for single-entity actions get unsatisfactory performance when confronted with multi-entity actions [10, 35] . This inadequacy can be attributed to a common practice [36, 37, 38, 39, 40] observed in treating interactions: each entity is encoded independently using the same single-entity backbone, and their features are averaged for recognition. This practice is based on an empirical assumption that each entity is independent and identically distributed (i.i.d.). But we demonstrate that different entities depicted by skeletons exhibit evident non-i.i.d. characteristics. Fig. 1 (a) Row 1 reveals significant inter-entity distribution discrepancies using estimated distributions of joints from distinct entities. Such discrepancies can introduce bias into the backbone models, leading to suboptimal optimization and performance. It explains why multi-entity action modeling usually diverges from the single-entity one.Using local coordinates for each entity holds promise in rendering them i.i.d., achieved by shifting individual origins to the per-entity spatiotemporal centers of mass (S2CoM), as depicted in Fig. 1 (a) Row 2. S2CoM is a straightforward and intuitive baseline to address this problem. However, this approach exacts a significant toll as it entails a complete loss of inter-entity information. Experimental results corroborate this notion, as illustrated in Fig. 1 (b), showcasing the detrimental impact of lacking inter-entity measurements on recognition performance. Nonetheless, this endeavor sparks an insightful realization: the potential for narrowing distribution gaps through origin shifts, thereby improving the performance of single-entity backbones in multi-entity scenarios. A natural question arises: Can we reduce the bias by finding the optimal sample-adaptive shift in R 3 that minimizes the distribution discrepancies among entities?To address the inter-entity distribution discrepancy problem, we propose a Convex Hull Adaptive Shift based multi-Entity action recognition method (CHASE). Serving as an additional normalization step, CHASE aims to accompany other single-entity backbones for enhanced multi-entity action recognition. Our main insight lies in the adaptive repositioning of skeleton sequences to mitigate inter-entity distribution gaps, thereby unbiasing the subsequent backbone and boosting its performance. Specifically, CHASE consists of a learnable parameterized network and an auxiliary objective. The parameterized network can achieve plausible and sample-adaptive repositioning of skeleton sequences through two crucial components. First, the Implicit Convex Hull Constrained Adaptive Shift (ICHAS) ensures that the new origin of the coordinate system is within the skeleton convex hull. Second, the Coefficient Learning Block (CLB) provides a lightweight parameterization of the mapping from skeleton sequences to their specific coefficients in ICHAS. Moreover, to guide the optimization of this network for discrepancy minimization, we propose the Mini-batch Pair-wise Maximum Mean Discrepancy (MPMMD) as the additional objective. This loss function quantifies pair-wise entity discrepancies using maximum mean discrepancy and integrates mini-batch sampling strategies to estimate the expectation. In conclusion, CHASE works as a sample-adaptive normalization method to mitigate inter-entity distribution discrepancies, which can reduce bias in the subsequent classifier and enhance its multi-entity action recognition performance.The contributions of this paper are three-fold:1. To the best of our knowledge, we are the first to investigate the issue of inter-entity distribution discrepancies in multi-entity action recognition. Our proposed method, Convex Hull Adaptive Shift for Multi-Entity Actions, effectively addresses this challenge. Our main idea is to adaptively repositioning skeleton sequences to mitigate inter-entity distribution gaps, thereby unbiasing the subsequent backbones and boosting their performance. 2 Related Work 
skeleton-based action recognition: 
 Datasets & Models. Datasets [41, 42, 43, 44] proffering annotated or estimated skeleton sequences support the development of skeleton-based action recognition. Based on these benchmarks, a significant body of works focus on the design of artificial neural network architecture for more effective skeleton-based action recognition. Early models rely on the basic architecture of Recurrent Neural Network to capture temporal motions [45, 46, 47, 48, 49, 50] . Graph Convolution Network (GCN) shows predominated popularity as various graph convolution operators being proposed [36, 37, 38, 51, 52, 53, 54, 55, 56, 57, 58] . Recent progress of the model design is largely driven by adopting self-attention mechanism and transformer architecture [39, 40, 59, 60, 61, 62, 63, 64] .Optimization Objectives. Several works have explored additional optimization objectives beyond the commonly used cross-entropy (CE) loss to ensure robust recognition [37, 65] , address challenging open-set problems [16] , or integrate supplementary natural language descriptions [66, 67] .However, existing methods are usually developed under the empirical assumption that entities are i.i.d. allowing the backbones to learn representations of actions concerning only one entity [36, 37, 38, 39, 40, 53] . However, when confronted with multi-entity interactions, their common practice of feeding the backbone separately often proves inadequate. Our proposed approach can seamlessly adapt to these existing methods, boosting their performance by minimizing the distribution discrepancies.Interactive Actions. Addressing datasets featuring two-person actions [41, 42, 68, 69, 70, 71, 72] or egocentric hand-object interactions [12, 13, 73, 74] necessitates effective interaction modeling. This spurs the development of various interaction recognition models leveraging human body and hand graph priors [10, 75] . Notably, the introduction of the general interactive action recognition task [35] unifies diverse interactions across various entity types, including person-to-person [10, 11, 14, 35, 76, 77] , hand-to-hand [12, 35, 78, 79] and hand-to-object [13, 35, 75, 79, 80, 81] interactions.Group Activities. Another interesting area of study is group activities [82, 83, 84] , which involve more entities and may include irrelevant individual motions [85, 86] . To this end, recent works usually leverage compositional reasoning from group skeletons, either alone or in combination with additional modalities, to achieve promising results [87, 88, 89, 90, 91, 92, 93, 94, 95, 96] . Given a skeleton sequence of multi-entity action as input, CHASE executes an implicit convex hull constrained adaptive shift with the Coefficient Learning Block, implemented as a lightweight backbone wrapper. CHASE also collects pair-wise shifted skeletons within mini-batches, effectively alleviating inter-entity distribution discrepancies by introducing an additional objective.While these works demonstrate satisfactory performance through interaction modelling, some may encounter model scalability issues when confronted with the factorial growth of inter-entity interactions [10, 13, 35, 75] . Moreover, they usually lack sufficient justification for why multi-entity action modeling significantly diverges from the single-entity one [10, 35, 79, 81] . In this paper, we delve into the inter-entity distribution discrepancy problem and introduce CHASE as a solution to minimize discrepancies. Through our proposed method, we aim to demonstrate that single-entity backbones can work well in multi-entity settings. 
chase: 
 Fig. 2 presents the framework of our proposed CHASE for skeleton-based multi-entity action recognition. We begin by presenting the formulation of the implicit convex hull constrained adaptive shift in Section 3.1, followed by the design of a lightweight Coefficient Learning Block in Section 3.2. In Section 3.3, we subsequently introduce an additional objective termed Mini-batch Pair-wise Maximum Mean Discrepancy to further mitigate inter-entity distribution discrepancies.The observed inter-entity distribution discrepancy in multi-entity skeleton sequences stems from the initial configuration of the world coordinate system. To mitigate this discrepancy, we propose an adaptive shift mechanism for each multi-entity skeleton sequence. It guides the origin to a sampleadaptive location, aiming to render each entity approximately i.i.d.. Moreover, based on the empirical assumption that the origin should not be far away from the skeletons, we implicitly constrain the new origin to remain within the skeleton convex hull by proving a simple but crucial proposition.Consider a scenario where E interactive entities (e.g. persons) engage in purposeful activities over a duration of T , and the pose of each entity is indicated by J joints with C Cartesian coordinates.The skeleton sequence of a multi-entity action is defined as X ∈ R C×T ×J×E . For clarity we denoteU = T × J × E. Given points ⃗ p i ∈ R C×1 in X ∈ R C×U , the subtraction ⃗ pi = ⃗ p i -⃗ p * (1 ≤ i ≤ U) defines a shift of origin for them, where ⃗ pi , ⃗ p * ∈ R C×1 . This can be expressed in matrix form as:EQUATIONwhere J 1,U ∈ R 1×U is a matrix of ones, and X is the shifted skeleton sequence. Now the problem is to make the shift vector ⃗ p * adaptive to X. A naive implementation is the linear combination:EQUATIONwhere I ∈ R U ×U and the weight matrix W ∈ R U ×1 .However, optimizing W can be challenging without constraints, as ⃗ p * could potentially be any point in R 3 . It is therefore reasonable to constrain ⃗ p * by incorporating the definition of the Convex Hull.Definition 1 (Convex Hull [97] ). The convex hull S of a given set X can be defined as: 1) The (unique) minimal convex set containing X.2) The set of all convex combinations of points in X. These definitions are equivalent.We jump to the formulation of the implicit skeleton convex hull constrained adaptive shift vector by proving the following proposition:Proposition 1. The implicit skeleton convex hull constrained adaptive shift vector is formulated asEQUATIONwhere X ∈ R C×U , W ∈ R U ×1 , and ⃗ p * ∈ R C×1 . ⃗ p * in Eq. 3 is an element in the set of all convex combinations of points in X. It is also a point that lies in the minimal convex set containing X.Proof. The first half of this proposition is equivalent to show that the matrix product of X and softmax(W ) is a convex combination of X. X is a set of points ⃗ p 1 , . . . , ⃗ p U with C Cartesian coordinates. We denote softmax(W ) as W with component αi , which is formulated asEQUATIONwhere α i is a component of W . By applying function softmax : R U → (0, 1) U , each component αi of W will be in the interval (0, 1), and the components will add up to 1. Thus we have ⃗ p * = U i=1 αi ⃗ p i , where all αi ∈ R satisfy αi > 0 and U i=1 αi = 1. This is sufficient for the definition of a convex combination, which only requires αi ≥ 0. Then the second half of this proposition is evident with the equivalence of definitions in Def. 1.Proposition 1 also implies that all possible ⃗ p * constitute a subset S of the convex hull S defined by the skeleton joints for all entities during the action period:EQUATIONwhich specifically is the interior of S (i.e., the open convex hull of X). We provide an example of the feasible ⃗ p * in the interior of S, marked by a green circle in Fig. 2 . The center of mass (CoM) ⃗ p is also in the set S, proven by simply taking all αi = 1/U (1 ≤ i ≤ U ).With Eq. 1 and Eq. 3, we introduce Implicit Convex Hull Constrained Adaptive Shift as:EQUATION)where W is coefficients needed to be optimized. In Eq. 2, the search space for ⃗ p * encompasses the entire R 3 . However, in Eq. 6, it's restricted to the open convex hull S. We optimize the weights for each point under the constraint of the skeleton convex hull, subsequently deriving the adaptive shift vector for each sample. Applying a softmax function implicitly constrains ⃗ p * to remain within the convex hull S, while preserving inter-entity measurements. Consequently, the subtraction between the point set and the shift vector repositions the origin to a specific point in the open convex hull.In this section, a lightweight Coefficient Learning Block is introduced to parameterize the mapping from the input skeleton sequence to the weight matrix. This parameterization allows CHASE to achieve sample-adaptive coefficients beyond sample-adaptive shifts formulated in Section 3.1.In Eq. 6, we note that the first-order partial derivative of X with respect to X isEQUATIONwhose result is constant. This implies that the same learnt weight matrix W is applied to all different Xs when getting adaptive ⃗ p * s. To make the coefficients W dependent on the input X, a mapping ψ : R C×U → R U ×1 is expected to map X to W .As depicted in Fig. 2 , we parameterize the nonlinear mapping ψ as a sequence of learnable layers, termed the Coefficient Learning Block. This lightweight CLB can be formulated as follows:EQUATIONwhereW 1 ∈ R C1×C , W 2 ∈ R C2×C1 , W 3 ∈ R U ×C2 are weight matrices, b is a bias matrix, ϕ : R C1×U → R C1×1is a squeeze operator [98] and δ is an activation function. Using a dimensionalityreduction layer and a dimensionality-increasing layer around the non-linearity is a common gating mechanism parameterization [98, 99] . Hence, we ensure U ≥ C 1 > C 2 .To facilitate CHASE optimization, we introduce an additional objective aimed at minimizing the inter-entity distribution discrepancies of the shifted skeleton sequences. This objective quantifies the pair-wise discrepancies and employs mini-batch sampling strategies to estimate the expectation.Maximum mean discrepancy is a metric used to measure the distance between distributions, defined as the distance between their embeddings in the reproducing kernel Hilbert space (RKHS) H:EQUATIONwhere sup(•) denotes the supremum. It is equivalent to finding the RKHS function f that maximizes the difference in expectations between the two probability distributions P (x) and Q(y).Suppose each entity distribution is denoted as P i (1 ≤ i ≤ E) for E entities, we measure the distance of all pair-wise distributions using the empirical meanEQUATIONwhere z = (P i , P j )(1 ≤ i, j ≤ E, i ̸ = j) with the probability density r(z), and C(E, 2) denotes a combination of E things taken 2 at a time without repetition. We adopt two approximations for computational efficiency. The first involves estimating E[f (x)] in Eq. 9 using a mini-batch of x.The second approximation concerns the right-hand side of Eq. 10, which is impractical due to its complexity of O(n!) in terms of the entity count. Instead, it can be approximated by uniformly sampling a mini-batch of M entity pairs from all possible C(E, 2) combinations z:EQUATIONWe denote Eq. 11 with the above two approximations to be the Mini-batch Pair-wise Maximum Mean Discrepancy Loss L mpmmd , thereby we have the total loss function for training:EQUATIONwhere L CLS is the classification loss and λ is the trade-off weight factor for L mpmmd . 4 Experiments 
datasets & settings: 
 We conduct experiments on six multi-entity action recognition datasets. Fig. 3 presents skeletal samples in these datasets and their skeleton convex hulls, showcasing their difficulties.NTU Mutual 11 and NTU Mutual 26, respectively subsets of NTU RGB+D [41] and NTU RGB+D 120 [42] , consist of a variety of inter-person mutual actions. NTU Mutual 11 adopts the widely-used X-Sub and X-View criteria, while NTU Mutual 26 follows the X-Sub and X-Set criteria.H2O [13] proffers 3D poses of human hands and bounding boxes of the manipulated objects, facilitating both hand-to-hand and hand-to-object interactions learning. We follow the training, validation, and test splits outlined in [13] in our experiments.Assembly101 (ASB101) [12] is a large and challenging 3D manual procedural activity dataset, with 1,380 categories of interactive actions. We follow the training, validation, and test splits described in [12] for evaluations. Fine-grained actions (verb & noun) are adopted as labels in experiments.Collective Activity Dataset (CAD) [85] captures people and their behaviors in public using street cameras, categorizing pedestrian collective activities into 4 groups. We adopt the same categories, individual labels, train-test split in [95] . Only 2D joint coordinates are used in our experiments.Volleyball Dataset (VD) [86] consists of video clips from volleyball tournaments and includes 8 group activity classes based on volleyball terminology. We follow the Original split described in [95] for evaluation. Only estimated 2D joint coordinates are used as input features.Settings. Experiments are conducted on the GeForce RTX 3070 GPUs with PyTorch. CTR-GCN [36] , InfoGCN [37] , STSA-Net [40] and HD-GCN [38] are chosen as our baseline models. To ensure fair comparisons, we adopt single intra-skeleton modality without multi-modality fusion following [35] .For CTR-GCN in NTU Mutual 26, we adopt input shape X ∈ R 3×64×25×2 , segment size (1, 1, 1) and λ = 0.1 in CHASE. SGD optimizer is used with Nesterov momentum of 0.9, a initial learning rate of 0.1 and a decay rate 0.1 at the 80th and 100th epoch. Batch size is set to 64. More detailed configurations for each model are provided in the Appendix.Table 1 shows the experimental results on different benchmarks, reporting the averaged top-1 accuracy and its standard deviation in runs with several seed initializations. We compare CHASE with vanilla counterparts (light red background) and the state-of-the-art multi-entity action recognition methods (light yellow background). By adopting our proposed CHASE, we can boost the vanilla counterparts' performance by a noticeable margin in most settings. It yields varying degrees of accuracy improvement across different baseline models and benchmarks, owing to differences in model parameter count, training objective, data scale, etc. Compared to models with complicated interaction designs, CHASE can help single action backbones achieve the state-of-the-art performance in interaction recognition by outperforming ISTA-Net [35] , AHNet-Large [83] , etc. In group activities recognition task, which is more challenging for single-entity backbones, CHASE can help achieve competitive performance. Fig. 4 visualizes that CHASE can effectively alleviate the potential inter-entity distribution discrepancies across a range of data scales, thereby ensuring robust backbone optimization and inference. UMAP [100] visualization in Fig. 5 demonstrates our proposed CHASE differentiate similar multi-entity actions better by assisting backbones to learn more distinctive representations. the individual origins to the spatiotemporal centers of mass for each entity. 3) BatchNorm: Apply an additional BatchNorm operation immediately when batches of samples are fed into the model. 4) ER (Entity Rearrangement [35] ): A technique aims to eliminate the orderliness of entities for interaction modelling. 5) Aug: Apply an additional data augmentation by randomly shifting the skeleton sequences. 6) S2CoM †: Shift the origin to the spatiotemporal center of mass. 7) S2CoM †/STD: Scale according to the channel-wise standard deviations after applying S2CoM †. Results in Table 2 indicate that CHASE can outperform these alternatives by bringing the largest accuracy improvement to the vanilla CTR-GCN.Analysis of inter-entity distribution discrepancies. Table 3 presents metrics evaluating the interentity distribution discrepancies on test sets, including Averaged Kullback-Leibler Divergence (Avg KLD), Jensen-Shannon Divergence (JSD), Bhattacharyya Distance (BD), Hellinger Distance (HD) and MMD. We measure the pair-wise distributions of sampled data points from different entities in test sets of NTU Mutual 11 X-Sub (I), X-View (II) and NTU Mutual 26 X-Sub (III), X-Set (IV). Table 3 demonstrates that CHASE significantly minimizes discrepancies across all evaluation metrics, thereby benefiting backbone learning for each entity in multi-entity actions. Evaluations on Mixed Recognition of Single-& Multi-Entity Actions. Table 5 shows a 0.41% improvement in X-Sub accuracy on the entire NTU RGB+D 120. This implies that although CHASE is proposed for multi-entity actions, it is also effective in mixed recognition settings.  
conclusion: 
 This paper proposes the Convex Hull Adaptive Shift for Multi-Entity Action Recognition (CHASE) to address the inter-entity distribution discrepancies. To the best of our knowledge, we are the first to investigate this problem and leverage discrepancy minimization to unbias the classifiers. Our approach can seamlessly adapt to existing backbone architectures and demonstrate performance improvements across six multi-entity action recognition datasets. Definition 5 (k-hop Bones [37] ). We denote X t as the joints at the moment t in a skeleton sequence. The k-hop Bone X(k) t at moment t can be formulated asEQUATIONwhere k ≥ 1, and P ∈ R J×J is a binary adjacency matrix of a directed graph without bi-directional edges.In the context of human body skeletons, Def. 5 uses the joint j r , representing the center of the spine, as the root to construct the directed graph. If k = max v d(v) + 1, where max v d(v) means the max hop of all joints to the root, then the k-hop Bone X(k)t aligns with the joint definition in Def. 4. However, when k = 1, this equivalence doesn't necessarily hold for the bone definition in Def. 4. The reason is that some joints (such as the root) may have no in-degree, causing them to retain their original joint coordinates in Def. 5. Therefore, the k-hop bones may still contain joint information that can be modified through adaptive shift. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Table 1 shows the experimental results on different benchmarks, reporting the averaged top-1 accuracy and its standard deviation in runs with several seed initializations.In Table 3 , we sample the points 30 times with different seed initializations and report the averaged measurements with standard deviations. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: This research conforms in every respect with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss both potential positive societal impacts and negative societal impacts of our work, provided in Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We carefully cited the original papers for existing assets and included their licenses in Appendix H.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Our code is publicly available at https://github.com/Necolizer/CHASE with MIT license.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Consider the expectation E r(z) [g θ (z)] in Eq. 11, where g θ denotes the composition of MMD (Eq. 9) and ICHAS (Eq. 6) with CLB (Eq. 8). We assume that the gradient of g θ with respect to the parameters θ exists. Since it adopts uniform sampling, we note that the discrete probability density function r(z) of z is independent of the parameters θ. Thus we haveEQUATIONwhich indicates that the gradient of the expectation ∇ θ E r(z) [g θ (z)] is equivalent to the expectation of the gradient E r(z) [∇ θ g θ (z)]. The latter can be approximated using Monte Carlo methods. We conduct experiments on a range of datasets, including NTU Mutual 11 (a subset of NTU RGB+D [41] ), NTU Mutual 26 (a subset of NTU RGB+D 120 [42] ), H2O [13] , Assembly101 (ASB101) [12] , Collective Activity Dataset (CAD) [85] , and Volleyball Dataset (VD) [86] . self.seg = self.pooling_seg[0]*self.pooling_seg [1] *self.pooling_seg [2] 7:self.seg_num_list = [(num_frame//self.pooling_seg[0]), (num_point//self.pooling_seg [1] ), (num_entity//self.pooling_seg [2] )] 8:self.seg_num = self.seg_num_list[0] * self.seg_num_list [1] * self.seg_num_list [2] 9:self.shift = nn.Sequential( 10:nn.Conv3d(in_channels=in_channels, out_channels=c1, kernel_size=1), 11: nn.AdaptiveAvgPool3d((self.pooling_seg[0], self.pooling_seg [1] , self.pooling_seg [2] )), 12: nn.Conv3d(c1, c2, 1, bias=False), 13: nn.ReLU(inplace=True), 14: nn.Conv3d(c2, out_channel, 1, bias=False), sf = (tx @ sf.softmax(dim=1)).unsqueeze(-1).expand(-1, -1, self.seg, self.seg_num)22: sf = rearrange(sf, 'n c (T V M) (t v m) -> n c (T t) (V v) (M m)', 23:T=self.pooling_seg[0], V=self.pooling_seg [1] , M=self.pooling_seg [2] , 24: t=self.seg_num_list[0], v=self.seg_num_list [1] , m=self.seg_num_list [2] ).contiguous() return out provides details about each dataset, including their annotation types, numbers of action categories, numbers of joints, numbers of clips (samples), averaged counts of valid frames, counts of entities engaging in a multi-entity action, and numbers of participants in data collection. For CAD [85] and VD [86] , which both capture videos in the wild with a variety of individuals, it is difficult to determine the exact number of participants.In this section, we provide the detailed formulation for metrics in ablation study. Given two discrete probability distributions P and Q defined on the same sample space X , we can define the following metrics:Averaged Kullback-Leibler Divergence (Avg KLD): Figure 7 : Implementation details for different models and benchmarks. CHASE can be adapted to various backbones including CTR-GCN [36] , InfoGCN [37] , STSA-Net [40] , and HD-GCN [38] . We implement linear classification heads for all the models and benchmarks, except for InfoGCN [37] . Individual labels are utilized as an auxiliary classification objective to further improve the group action recognition performance in CAD [85] and ASB101 [12] .EQUATIONJensen-Shannon Divergence (JSD):EQUATIONwhere M = 1 2 (P + Q) is a mixture distribution of P and Q. Bhattacharyya Distance (BD):EQUATION)Hellinger Distance (HD):EQUATION)Maximum Mean Discrepancy (MMD):EQUATIONwhere E[f (•)] is the expectation of any function f in the RKHS H.When adopting these metrics in our experiments, the distributions are generated by the kernel density estimation (KDE). We sample the points 30 times with different seed initializations and report the averaged measurements.In this section, we provide more details of our experimental setup and model implementation for each benchmark. Experiments are conducted with 8 GeForce RTX 3070 GPUs (GPU Memory: 8GB), using torch version 1.9.0+cu111, torchvision version 0.10.0+cu111, and CUDA version 11.4. CTR-GCN [36] , InfoGCN [37] , STSA-Net [40] and HD-GCN [38] are chosen as our baseline models.To ensure fair comparisons, we adopt single intra-skeleton modality without multi-modality fusion, following [35] . Our implementation details are illustrated in Fig. 7 .NTU Mutual 11 [41] & NTU Mutual 26 [42] . X-Sub and X-View criteria [41] are adopted in NTU Mutual 11, while X-Sub and X-Set criteria [42] are used in NTU Mutual 26. We evaluate models with only 3D joint inputs, applying data augmentations such as random rotation and spatial shift. During training and testing, we employ temporal cropping and resizing, adjusting based on the number of valid frames. Notably, we use distinct percentage intervals for training (0.5,1) and testing (0.95). For experiments conducted on the entire NTU-RGB+D 120 dataset, we maintain identical settings as those used for NTU Mutual 26.H2O [13] . We follow the training, validation, and test splits described in [13] . To maintain consistency in GCNs, we use the hand graph structure, originally designed for human hands, for both hand entities and object entities. Models are evaluated with only 3D joint inputs. The same augmentations as NTU Mutual 26 are adopted in this benchmark.ASB101 [12] . We follow the training, validation, and test splits outlined in [12] for evaluations. 1,380 Fine-grained actions (verb & noun) are adopted as labels in experiments. We evaluate models with only 3D joint inputs. The same augmentations as NTU Mutual 26 are adopted in this dataset, except that the training percentage interval of the temporal cropping and resizing is set to (0.75,1).CAD [85] . We adopt the same data augmentations, group action categories, individual labels and train-test split in [95] . But different from [95] , only 2D joint coordinates are used in our experiments. Individual labels are used as an auxiliary classification objective, as presented in Fig. 7 .VD [86] . We follow the same data augmentations, group action categories, individual labels and Original train-test split in [95] . But different from [95] , only 2D joint coordinates are used as input features in experiments. Besides, individual labels are leveraged as an auxiliary classification objective to further improve the group action recognition performance, as shown in Fig. 7 . The volleyball position is represented as X ∈ R T ×C . To ensure inter-entity consistency, we apply padding to fit the shape X ∈ R C×T ×J×1 . To maintain consistency in GCNs, we employ human body graph structure for both human body entities and volleyball entities. In all experiments, we maintain consistent configurations for baseline models to ensure fair comparisons between models incorporating CHASE and their respective vanilla counterparts.To avoid unnecessary verbosity, we present implementation specifics solely for NTU Mutual 26.For training details pertaining to other benchmarks, please refer to the code repository at https: //github.com/Necolizer/CHASE. [36] . Cross entropy is used as the recognition loss function. SGD optimizer is used with Nesterov momentum of 0.9, a initial learning rate of 0.1 and a decay rate 0.1 at the 80th and 100th epoch. Batch size is set to 64. With the first 5 warm-up epochs, the training process is terminated after 110 epochs.InfoGCN [37] . By taking k = 1, InfoGCN adopts their definition of 1-hop Bones [37] as input.Cross entropy is used as the loss function with label smoothing factor 0.1 and temperature factor 1.0. The information bottleneck objective [37] is also employed as the auxiliary loss. Following [37] , we set λ 1 = 0.1, λ 2 = 0.0001, and the µ r (z|y) of each action class as random orthogonal vectors with a scale of 3. Diverging from conventional backbones, we substitute the standard classification head with the InfoGCN head, depicted in Fig. 7 . SGD optimizer is used with Nesterov momentum of 0.9, a weight decay of 0.0005, a initial learning rate of 0.1 and a decay rate 0.1 at the 90th and 100th epoch. Batch size is set to 120. With the first 5 warm-up epochs, the training process is terminated after 110 epochs.STSA-Net [40] . Cross entropy is used as the loss function with label smoothing factor 0.1 and temperature factor 1.0. SGD optimizer is used with Nesterov momentum of 0.9, a initial learning rate of 0.1 and a decay rate 0.1 at the 60th and 90th epoch. Batch size is set to 64. With the first 5 warm-up epochs, the training process is terminated after 110 epochs.HD-GCN [38] . We set CoM = 1 in the hierarchy graph generation and evaluate on this setting. Cross entropy is used as the loss function. SGD optimizer is used with Nesterov momentum of 0.9, a More Analysis on Table 1 . In Table 1 , we observe that CHASE yields varying degrees of accuracy improvement across different baseline models and benchmarks. The performance gains are influenced by both the backbone models and the datasets, as CHASE functions as an additional normalization step that mitigates data bias introduced by inter-entity distribution discrepancies. For baseline backbones, this is owing to differences in their backbone architecture design, parameter count and training objective. For example, STSA-Net [40] is a relatively large backbone based on transformer architecture, which does not rely the prior definition of the skeleton graphs. Its adaptability to different graph structures makes it outweigh GCN-based backbones in H2O benchmark. Another example is InfoGCN [37] , which leverages an auxiliary information bottleneck objective in its training. Though this method is proven more effective than the other backbones in some person-to-person interaction settings, it doesn't ensure better performance in hand-to-object interaction and group activity benchmarks. For different benchmarks, it is owing to differences in data scale and label space (see Table 7 ). For example, ASB101 is an extremely challenging benchmark for its over 80,000 samples and 1,380 target categories. Therefore the accuracy improvement is modest compared with the other benchmarks.More Qualitative Results. Fig. 8 visualizes how CHASE works with multi-entity skeletal sequences. By integrating CHASE, different entity distributions become more similar in both aspects of mean and covariance. It demonstrates that CHASE can lower the inter-entity distribution discrepancy, especially obviously in NTU Mutual 11, NTU Mutual 26, CAD, whose entities have no orderliness. For H2O, ASB101 and VD, whose entities are characterized by an intrinsic order (e.g. left hands, right hands, left-side volleyball players, right-side volleyball players, and objects), CHASE can also preserve their orderliness by letting the distributions be similar but different.Evaluations on Mixed Recognition of Single-Entity & Multi-Entity Actions. Table 5 concludes the action recognition results on the entire NTU RGB+D 120 [42] . By integrating CHASE, the baseline model gets accuracy improvement by 0.41% and 0.05% on X-Sub and X-Set, respectively. This suggests that CHASE is effective even in mixed recognition settings. However, the improvement is modest because single actions are the dominant category in this dataset. (U + 1 + C 2 ) × C 1 + C 2 × U .For computational complexity, the FLOPs of CHASE is approximately 2.50 M. This analysis proves that CHASE is both efficient and lightweight for benefiting multi-entity action learning. Segment Size of Squeeze Operator. In CLB, which is also the mapping ψ to weight matrix, the squeeze operator squeezes the tensor to a specific shape, denoted as (T ′ , J ′ , E ′ ). This segment size determines the point set in a multi-entity action sequence to which ICHAS applies. For example, the segment size (1, 1, 1) indicates that ICHAS applies to all the points, and (1, 1, 2) means two different ICHAS apply to two entities separately. Table 9 evaluates various segment sizes of the squeeze operator, demonstrating that the global ICHAS with the segment size (1, 1, 1) achieves the best performance compared with the other settings. Therefore, we choose the default segment size as (1, 1, 1) in all the experiments. Besides, the reduction ratio between C 1 and C 2 is determined according to the experimental results in [98] . Weight λ for MPMMD. Fig. 10 evaluates different values of the trade-off weight factor λ in Eq. 12, varying from 10 -2 to 10 0 . On NTU Mutual 26, it achieve the best performances when adopting λ = 0.1 for MPMMD loss. It corresponds to our claim that MPMMD is an auxiliary objective to guide discrepancy minimization, additional to the recognition task objective. We also conduct experiments with MPMMD across a variety of M values but find insignificant differences in performance. Hence, we set M = 1 for computational efficiency.Matrix & Category-level Accuracy. We present the confusion matrices of the vanilla CTR-GCN and CTR-GCN with CHASE in Fig. 9 (a) & (b). It indicates that our proposed CHASE is able to assist the backbone to differentiate similar multientity actions better. Fig. 9 (c) reports the category-level accuracy for 26 kinds of person-to-person interactions. We observe that in very few categories, their performance slightly drops. One possible reason is that these actions, like point finger, rely heavily on cues from the individual movements of one of the entities, instead of the multi-entity interactions. This might not be addressed by mitigating inter-entity distribution discrepancies. But it could still conclude from Fig. 9 (c) that adopting CHASE can achieve accuracy improvement for most of the categories, e.g. wield knife (+8.50%), shoot with gun (+8.00%), whisper (+5.39%), and punch/slap (+4.37%).Analysis of Performances with Test-Time Skeleton Noises and Masking. To evaluate the robustness, we intentionally corrupt the multi-entity skeleton sequences with noise and masking during the inference phase. This aims at resembling possible skeleton occlusions or estimation errors during the test time. The noises X n ∼ N (µ, σ2 ) used in this experiment are normally distributed with mean µ = 0 and standard deviations σ = 10 -3 , 10 -2 . Masking strategies are randomly masking the multi-entity skeleton sequences with probabilities p m = 10 -2 , 10 -1 . Table 10 reports the averaged top-1 accuracy and its standard deviation in runs with several seed initializations for noises and masks.For recognizing multi-entity actions with corrupted test-time inputs, CTR-GCN integrating with CHASE consistently outperforms the vanilla counterpart, showcasing its robustness.This work proposes the Convex Hull Adaptive Shift for Multi-Entity Action Recognition to resolve the inter-entity distribution discrepancies. Although CHASE offers a generic framework for various types of multi-entity actions, such as person-to-person interactions, hand-to-object interactions, hand-to-hand interactions and group activities, its application to single-entity actions warrants further investigation. One potential approach is to consider different parts as multiple entities, such as treating different limbs of a human body as distinct entities. Moreover, it's promising to apply CHASE-like designs to the recently-developed human-centric foundation models [17, 29, 30, 31, 32, 33, 34] to enhance their performances on multi-entity skeletal data. These areas of exploration are left for future research.This paper focuses on multi-entity action recognition, a field with broad applications in physical human-robot interaction, social scene understanding, multi-agent systems, surveillance, healthcare monitoring, etc [24, 22, 25, 26, 27, 28, 23] . Our work contributes to advancements in these domains by enhancing efficient and effective skeleton-based learning of multi-entity actions. Although the use of human-centered data can pose privacy concerns, our study utilizes only skeletal data estimated from sensors or RGB videos, thus mitigating potential privacy and ethical issues.Datasets:• NTU Mutual 11 / NTU RGB+D [41] ): Custom (research-only, non-commercial, attribution) 2• NTU Mutual 26 / NTU RGB+D 120 [42] : Custom (research-only) 3• H2O [13] : Custom (research-only, non-commercial)4 • Assembly101 [12] : Creative Commons Attribution-NonCommercial 4.0 International License5 • Collective Activity Dataset [85] : Unknown• Volleyball Dataset [86] : BSD 2-Clause license 6Models:• CTR-GCN [36] : Creative Commons Attribution-NonCommercial 4.0 International License7 • InfoGCN [37] : Unknown • STSANet [40] : MIT License We also provide all the configuration files in our code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.http://rose1.ntu.edu.sg/Datasets/requesterAdd.asp?DS=3http://rose1.ntu.edu.sg/Datasets/actionRecognition.asphttps://h2odataset.ethz.ch/http://creativecommons.org/licenses/by-nc/4.0/https://github.com/mostafa-saad/deep-activity-rec/blob/master/LICENSEhttps://github.com/Uason-Chen/CTR-GCN/blob/main/LICENSEhttps://github.com/heleiqiu/STTFormer/blob/main/LICENSEhttps://github.com/Jho-Yonsei/HD-GCN/blob/main/LICENSE 


Decision: 3 of 4 of the reviewers found the paper's contributions significant with strong theoretical derivation for the technique and improved results across a number of benchmarks. The one negative reviewer found the motivation unclear and was not satisfied that model didn't support "transferr[ing] among different datasets without retraining the module". In the opinion of the AC these reasons don't qualify as reasons for rejection. Given this the AC recommends the paper be accepted to NeurIPs.

Reviews: review: paper_summary: paper_summary: The paper proposes a normalization method for skeleton-based multi-entity recognition based on finding the center of mass within the convex hull of the spatio-temporal domain of the point cloud defined by the skeletons over a sequence. The main idea is to "center" the world of skeletons to unbias the subsequent detector and boost their performance. Building on this motivation, the authors find that a fixed, learnable parameterized network can be used for that purpose, facilitating the inference. The experiments demonstrate that their proposed algorithm boosts performance over the corresponding baselines
strength_weakness: strength_weakness: Overall I believe that the presentation should be improved, clearly stating the contribution and motivation for it. It takes a good read to understand that what the authors are proposing as a lightweight network is the result of mathematically deriving an iterative approach for normalization of skeletons within their convex hull. It should be clearly stated that the method aims to accompany other methods for multi-entity activity recognition by adding an extra normalization step, which consists of what is presented in Section 3. 

Similarly, the sketch depicted in Figure 1 is a bit confusing and does not really illustrate what the authors aim to solve. I would suggest the authors to provide a clear motivation example that leads to their method. The caption in Fig. 1 is rather poorly written (I did not understand it at least).
questions: questions: I would insist on the authors to please elaborate a bit better on the motivation and an example where former normalization leads to the classifiers to produce wrong results, with their method mitigating such problem.
limitations: limitations: N/A
paper_summary: paper_summary: This paper focuses on the interesting problem of the normalization strategy for multi-entity skeletons in skeleton-based action recognition. The proposed method is intuitive, and the authors provided detailed implementation details for reproduction. However, this work is unclear, and the experiments are unconvincing.
strength_weakness: strength_weakness: (1)	The purpose of the multi-entity action recognition task is not clear. Is it to recognize each individual’s action or to classify group activities? If the purpose varies across different datasets, please clarify this in the experiment. Additionally, I am curious whether the optimal normalization strategy differs for these two purposes. For example, the method used in S2CoM seems more suitable for recognizing each individual’s action.
(2)	What is the main difference between the proposed method and the simple strategy of shifting the origin of multi-entities to their common center? Please add a comparison experiment with this method.
(3)	Although the normalization strategy is an important trick and can bring significant improvement in the action classification task, I still have a concern about whether it is worth conducting an additional network to achieve this simple trick by introducing extra learnable parameters. Some heuristic strategies may be more efficient and general. Accordingly, can the proposed module be transferred among different datasets without retraining the module?
questions: questions: What is the main difference between the proposed method and the simple strategy of shifting the origin of multi-entities to their common center? Please add a comparison experiment with this method.
limitations: limitations: (1)	The purpose of the multi-entity action recognition task is not clear. Is it to recognize each individual’s action or to classify group activities? If the purpose varies across different datasets, please clarify this in the experiment. Additionally, I am curious whether the optimal normalization strategy differs for these two purposes. For example, the method used in S2CoM seems more suitable for recognizing each individual’s action.
(2)	What is the main difference between the proposed method and the simple strategy of shifting the origin of multi-entities to their common center? Please add a comparison experiment with this method.
(3)	Although the normalization strategy is an important trick and can bring significant improvement in the action classification task, I still have a concern about whether it is worth conducting an additional network to achieve this simple trick by introducing extra learnable parameters. Some heuristic strategies may be more efficient and general. Accordingly, can the proposed module be transferred among different datasets without retraining the module?
paper_summary: paper_summary: This paper proposes CHASE, a multi-entity skeleton data augmentation/preprocessing technique, to mitigate inter-entity distribution gaps and improve the multi-entity action recognition. Specifically, the authors formulate a new constraint called ICHAS, design a lightweight block CLB to learn the nonlinear mapping from input to the weight matrix in ICHAS, and introduce an objective to guide the discrepancy minimization in CLB training. The authors conduct comprehensive experiments to show the effectiveness of the method.
strength_weakness: strength_weakness: a)	There’re some typos need to be revised carefully, e.g. l131 “be be”, Table 4 “-68.56”.

b)	Regarding clarity, the captions of Figures and Tables can be improved.
questions: questions: a)	Figure 2 is very informative and good for understanding the whole method. However, there are still some notations needed to be explained, e.g. the green and red circles under the two skeleton coordinates. It would be helpful if the authors introduce the method details (in sections 3.1 to 3.3) with references to specific parts of Figure 2.

b)	It’s good to have multiple runs and report the standard deviation. How many seed initializations exactly do the authors use?

c)	From Table 1, the performance margin varies a lot. On H2O and CAD, the top-1 accuracies have increased by over 9%. Could the authors explain or provide some discussion on this point?

d)	From Table 7 in the appendix, it seems like the authors use 25 ground-truth 3D joints as skeleton inputs for NTU-60 and NTU-120 datasets. Since 2D estimated joints tend to achieve better action recognition performance in dominant models, do the authors verify CHASE using 17 estimated 2D joints with COCO layout as input?

e)	Table 6 shows the mixed recognition results on the entire NTU-120 dataset with X-Sub setting. Table 8 in the appendix shows the mixed recognition results on NTU-120 X-Sub and X-Set settings. Why not replace Table 6 with Table 8? Similar to Table 5 and Table 9, why not replace Table 5 with Table 9 or integrate the number of parameters into Table 1 (may discard the column of ‘Venue’)?

Small details (no need to address them, just suggestions):

a)	Regarding equations (9) and (10), it would be better to clarify the definitions of sup(*) and C(E,2).
limitations: limitations: Although the method has been verified on six benchmarks, these datasets are relatively small, with the number of categories ranging from 4 to 36, except Assembly 101 has 1380 action categories. However, the performance gain on Assembly 101 is very small (<=0.21%). The reviewer has some concerns about the generalization of this method.
paper_summary: paper_summary: This paper tackled the issue of inter-entity distribution discrepancies in multi-entity action recognition. The authors proposed convex hull adaptive shift method to minimize the cross entity discrepancies, where CLB and MPMMD are proposed to assist the learning procedure. The method is verified to be effective among various datasets and backbones.
strength_weakness: strength_weakness: 1. The introduction section should  be improved. For example on line 54, why do we need to achieve the discrepancy minimization? on line 52, why do we need to achieve sample adaptive coefficients? The motivation should be highlighted. The links among these proposed items should be also improved on line 57-60 and need more insights.


2. The novelty of the CLB is limited. The format of attributes learning shown in Eq.8 is commonly used to construct concept learners. What is the difference between the concept bottleneck [1] and the CLB? If you use the concept learner from some existing works, e.g., [2], will it be better than CLB?


[1] Shin S, Jo Y, Ahn S, et al. A closer look at the intervention procedure of concept bottleneck models[C]//International Conference on Machine Learning. PMLR, 2023: 31504-31520.


[2] Wang B, Li L, Nakashima Y, et al. Learning bottleneck concepts in image classification[C]//Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 2023: 10962-10971.

3. More insights should be given in Section 4.2. Why does the proposed method help? The authors are encouraged to enrich the analysis.
The authors are encouraged to discuss the computational complexity brought by the proposed method.

4. The authors are encouraged to discuss the computational complexity brought by the proposed method.
questions: questions: 1. Improving the Introduction Section:

    a. Why is it necessary to achieve discrepancy minimization (line 54)?

    b.Why do we need to achieve sample adaptive coefficients (line 52)?

    c. Can the authors highlight the motivation behind these needs and improve the links among the proposed items (lines 57-60) with more insights?


2. Novelty of the CLB:

    a. How does the concept bottleneck (CLB) differ from the commonly used format of attributes learning in concept learners, such as in Eq. 8?

    b. What are the differences between the CLB and the concept bottleneck models discussed in Shin et al. (2023)?

    c. If the concept learner from existing works (e.g., Wang et al. (2023)) were used, would it perform better than the CLB?

3. Insights in Section 4.2:

    a. Why does the proposed method provide benefits?

    b. Can the authors provide a more detailed analysis to enrich Section 4.2?

4. Computational Complexity:

    a. What is the computational complexity introduced by the proposed method?
limitations: limitations: yes in supplementary

Comments: title: Appreciation for Your Support and Positive Review
Clarifications on Transferability and Multi-Entity Action Recognition
Rebuttal Comment
Answer
Thank You for Your Positive Feedback and Consideration
To the authors
Looking Forward to Further Discussions
Looking Forward to Further Discussions
Looking Forward to Further Discussions
Looking Forward to Further Discussions
comment: Dear Reviewer,

Thank you for your positive feedback and for your confidence in our work. We appreciate your thoughtful review and are pleased that our responses met your expectations.

Thank you again for your support and your constructive comments.

Best regards,
Dear Reviewer,

Thank you for your feedback. We apologize for not fully addressing your concerns in our initial response. 

1. We have followed your suggestions, conducting the experiments to evaluate the transferability of our CHASE module across different datasets without retraining. Specifically, we first trained the CHASE + CTR-GCN backbone on the challenging ASB101 dataset, achieving a top-1 accuracy of 28.03%. **We then transferred the CHASE module to the H2O (two-hand version) dataset without retraining, where it achieved 56.61% accuracy.** **This result outperforms both retraining the module on H2O (56.47%) and training only the CTR-GCN backbone on H2O (48.48%).** These findings demonstrate that our proposed module can indeed be transferred among different datasets without retraining.
2. Multi-entity action recognition **aims to classify interactions involving multiple entities, which could be people, objects, or other elements within a scene.** Examples of such actions include *cheers and drink*, *exchanging things*, *walking apart,* and *talking*. Group activities are a subset of multi-entity actions [73, 75, 76]. Unlike traditional action recognition, which typically focuses on a single subject performing a single action, multi-entity action recognition addresses the complexity of interpreting actions that involve multiple participants or objects interacting simultaneously. We hope this clarification addresses the confusion.

We appreciate your insights and hope this additional information helps to resolve your concerns.

Best regards,
I keep my initial score, where the authors did not struggle to handle my concerns, like "transferred among different datasets without retraining the module". Besides, the multi-entity action recognition task is still confusing.
I am happy with the provided response and I have no further questions in regards to it. I believe this paper lies above the acceptance threshold.
Dear Reviewer,

Thank you for your positive feedback and for taking the time to review our rebuttal. We're glad that our responses addressed your concerns, and we appreciate your willingness to improve the score. Thank you again for your thoughtful consideration.

Best regards,
Dear authors,

thank you very much for your rebuttal. I think most of my concerns are handled and I will improve my score to 6.

Best,
Dear Reviewer,

Thanks again for your insightful comments on our paper.

We have submitted the response to your comments and the global response with a PDF file. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider rasing the score.

Thank you
Dear Reviewer,

Thanks again for your insightful comments on our paper.

We have submitted the response to your comments and the global response with a PDF file. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider rasing the score.

Thank you
Dear Reviewer,

Thanks again for your insightful comments on our paper.

We have submitted the response to your comments and the global response with a PDF file. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider rasing the score.

Thank you
Dear Reviewer,

Thanks again for your insightful comments on our paper.

We have submitted the response to your comments and the global response with a PDF file. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider rasing the score.

Thank you