Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation
Authors: Jingmin Zhu
Anqi Zhu
Hossein Rahmani
Jun Liu
Mohammed Bennamoun
Qiuhong Ke
Venue: NeurIPS 2025 poster ()
Keywords: zero-shot learning
generalized zero-shot learning
skeleton-based action recognition
test-time adaptation
Primary area: applications
TL;DR: We propose a training-free test-time adaptation method that significantly improves zero-shot skeleton action recognition by using a training-free cache model during inference time.

Abstract: We introduce Skeleton-Cache, the first training-free test-time adaptation framework for skeleton-based zero-shot action recognition (SZAR), aimed at improving model generalization to unseen actions during inference. Skeleton-Cache reformulates inference as a lightweight retrieval process over a non-parametric cache that stores structured skeleton representations, combining both global and fine-grained local descriptors. To guide the fusion of descriptor-wise predictions, we leverage the semantic reasoning capabilities of large language models (LLMs) to assign class-specific importance weights. By integrating these structured descriptors with LLM-guided semantic priors, Skeleton-Cache dynamically adapts to unseen actions without any additional training or access to training data. Extensive experiments on NTU RGB+D 60/120 and PKU-MMD II demonstrate that Skeleton-Cache consistently boosts the performance of various SZAR backbones under both zero-shot and generalized zero-shot settings. The code is publicly available at https://github.com/Alchemist0754/Skeleton-Cache.

Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation
Jingmin Zhu
Anqi Zhu Monash University
Monash University
j inggin. in zhu1 @monash edu
maggie... zhu@monash.h.ooooommooo edu
Hossein Rahmani
Jun Liu
Lancaster University rwraaaaa@iiii @ww.iatteannnnnaeeaa ac uk
Lancaster University
j wellll1@lanneee @lancaster..r....e ac uk
Mohammed Bennamoun University of Western Australia mohammed. bennamoun@uwa. edu. au
Qiuhong Ke"
Monash University
Qiuhong. Ke@monash edu
Abstract
We introduce "Keleton-Cache, the first training-free test-time adaptation framework for kkeeed zero-shot action recognition (SZAR), aimed at improving model generalization unseen actions during inference. Skeleton-Cache reformulates inference as lightweight retrieval process over a non- parametremec cache that stores structured skeleton representations, combining both global and fine-gre mne-grained local descriptors. To guide the fusion of descriptor wise predictions, we leverage the semantic reasoning capabilities of large language models (L.LMs) to assign class- specific importance weights. By integrating these structured descriptors with L.MM- guided semantic priors, Skeleton-Cachh dynamically adapts to unseen actions without any additional training access to training data. Extensive experiments on NTU RGB+D 60/ 60/120 and PKU-MMD demonstrate that Ireleton-Cachi consistently boosts the performance of various SZAR backbones under both zero- shot and generalized zero-shot settings. The code publicly available https: //g //github. com/ Alchemist0754 wtt.ssstte///t/tooonennee
Swteeeeeeeennneeeeeeeeeeeeeeenae Cache.
Introduction
Action recognition is an important research area in computer vision with diverse applications in surveillance, monitoring, and human-computer interaction [12, 13]. While traditionally relying on RGB-based approaches, the field has evolved to include skeleton-based methods, enabled by advances in pose estimation techniques and depth sensor technologies [21, 27J. Skeleton-based recognition offers distinct advantages: robustness to lighting and background variations, enhanced privacy protection, and reduced computational demands [2, 14, 29, 23, 22]. These benefits have established it as a viable alternative to traditional video-based recognition, achieving competitive accuracy while addressing common challenges [25]. However, existing supervised skeleton-ber based methods rely extensive labeled training data, which is often expensive and impractical to obtain, particularly for rare actions [5].
To address this limitation, Skeleton-basee Zero-Shot Action Recognition (SZAR) [17, 1] has emerged, enabling recognition of unseen actions through supporting information such class names and
*Corresonning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

attributes. The core idea aligns skeleton features with textual descriptors in a shared semantic space, transferring knowledge from known to unknown classes [9, 28, 32, 33].
In SZAR, models often face significant distribution shift between training (seen actions) and testing (unseen actions) due to semantic gap between action categories, variations in human pose dynamics, differences in execution styles across individuals, and domain-spectt occlusions. Moreover, since unseen actions may involve novel motion patterns or compositional pose structures not observed during training, the model's learned representations can fail to generalize. Test-time adaptation (TTA) a compelling solution to this challenge it allows models to adjust to the test data distribution on the fly, improving generalization without requiring retraining from scratch. However, conventional TTA [26, 24] methods often rely on gradient-based updates or additional optimization, which can be computationally expensive and unstable. While recent training- free alternatives [10, 31] have emerged, they are primarily designed for image classification tasks and are not readily applicable to SZAR, given the unique spatio-temporal structure of skeleton sequences.
noise such as sensor errors or
To address this challenge, we propose Skeleton-Cacher training-fire test-time adaptation (TF-TTA) module that improves model generalizability to unseen actions during inference without requiring parameter updates or access the training data. Skeleton-Cache builds and dynamically updates a non-parametric cache during testing and reformulates SZAR as a lightweight retrieval-based classification task. Instead of relying on single holistic embedding-white cues essential for distinguishing similar actions- actions-Skeleton- Cache constructs a structured feature representation for both query and cache entries, composed of global descriptors and multiple fine- grained local descriptors that reflect key body regions and motion phases. This structured, multi-scale representation enables the model to capture subtle, discriminative patterns in skeletal sequences and allows the cache to store key patterns that remain consistent across different samples of the same class, leading to more precise feature matching and improved recognition of complex, previously unseen actions.
may overlook subtle
At test time, we compute the similarity between each descriptor of the query and the corresponding descriptors in the cache keys to obtain descriptor-wise class logits, which are then fused to obtain the final class prediction. A straightforward fusion approach would be to average these logits; however, such uniform treatment ignores the fact that different descriptors may vary in importance depending on the specific action. To address this, we leverage the strong semantic reasoning capabilities of large language models (LLMs) to provide linguistic priors in the form of importance weights for each descriptor. These weights encode high-level knowledge about the relevance of the global context, specific body regions, and motion phases for recognizing given action-for example, recognizing the action "kicking" relies more on leg movements and cues from the final temporal phase. By incorporating these LLM-derived weights, our model adaptively emphasizes the most informative cues for each class, all without introducing any additional training.
Skeleton- Cache is a plug-and and play. training-fiee free module that can be integrated into a wide range of SZAR becckbones-icluddng PÜRLS [33], SA-DVAE [15], SMIE [32] and SynSE [8]-offering dynamic and efficient test-time adaptation. Importantly, the cache evolves continuously during inference: features with high confidence are incorporated into the memory, while new or distinctive features create new entries. This parammemetricc update mechanism supports on-the- fly adaptation to novel inputs without needing gradients or data augmentation.
Our contributions are summarized as follows:
We introduce Skeleton-Cache, the first training-free test-time adaptation (TTA) framework for skeleton-based zero-shot action recognition (SZAR), enhancing the generalization of existing SZAR models to unseen actions without additional training.
We design a structured cache within Skeleton-Cache that captures both a global descriptor and fine-grained spatial and temporal descriptors of skeleton sequences, enabling the model retrieve subtle and discriminative cues critical for recognizing unseen actions more accurately.
We leverage the powerful semantic reasoning capabilities of LLMS to derive class-specific importance weights, which guide an adaptive fusion of descriptor-wise predictions in a lightweight manner without introducing extra training.

---

We conduct extensive experiments on NTU RGB+D 60/120 [20. 16] and PKU-MMD [[[4], showing that Skeleton Cache consistently boosts the performance of diverse SZAR backbones, with better results under both zero-shot and generalized zero-shot settings.
2 Related Work
2.1 Skeleton-based Zero-Shot Action Recognition
Skeleton ZSL has evolved rapidly from early metric- metric-learning approaches by Wray et al. [28], who projected skeleton features into DeViSE-style semantic spaces, to more sophisticated methods like SynSE-ZSL [8] with syntactically guided pseudo samples for verb-level transfer, and SMIE [32] which maximized mutual information between visual and textual distributions for temporal cues. Recent advances leverage language models, with PURLS [33] aligning part-aware skeleton patterns with word embeddings and STAR [3] refining cross-modal alignment through dual prompts and side-information constraints, while generative approaches like SA-DVAE [15] synthesize unseen features to close the semantic gap. However, these methods lack mechanisms to adapt at inference, making them vulnerable to distribution shifts between seen and unseen classes. These shifts can occur even within the same dataset due to novel compositions of motion patterns, temporal variations, underrepresented body part interactions. Without test-time adaptation, models unable to recalibrate predictions in response to such discrepancies, limiting their effectiveness real-world zero-shot scenarios. Our Skeleton-Cache addresses this by introducing the first training-free test-time adaptation module for SZAR-a plug-and-play solution that enhances diverse SZAR backbones without modifying learned parameters.
2.2 Test-Time Adaptation for Zero-Shot Learning
Test-Time Adaptation (TTA) research has progressed along two primary directions. Gradient-based methods update model parameters during inference, including Tent [26] which fine-tunes batch-norm statistics minimize prediction entropy, TPT [24] which optimizes textual prompts via augmented views, PromptAlign [18] which aligns prompt token statistics with source data distributions, and DifTTTT [6] which employs diffusion models for diverse synthetic views-all suffering from latency and overfitting risks on smaller SZAR datasets due to their reliance on back-propagation and multiple forward passes. In contrast, Training-fre free alternatives offer higher efficiency, such as T3A [10] which dynamically re-weights class prototypes based on prediction confidence, CALIP [7] introduced a parameter- free attention module for cross- rrosll features. AdaNPC [31] which mixes and source features to mitigate forgetting, and TDA [1]] which constructs positive and negative feature caches for adapting models like CLIP. However, these methods are primarily designed for image classification and treat features holistically, overlooking the structured and fine-grained spatial- temporal information essential for distinguishing similar skelctal actions (c.g., salute vs. wave). Our Skeleton-
Skeleton-Cache Cache addresses these limitations with a design tailored specifically for SZAR. It avoids gradient- baade updates entirely, mitigating overfitting and latency concerns. Besides, it models fine-grained information by decomposing skeleton inputs into structured global, spatial, and temporal descriptors. Furthermore, Skeleton- Cache uniquely leverages the semantic reasoning abilities of LLMs to guide descriptor fusion, enabling robust generalization to complex, unseen actions while remaining lightweight and plug-and aoug-and-llay.
3 Methodology
This section presents the full Skeleton-Cache pipeline, beginning with key preliminaries, followed by a detailed explanation of the proposed method
3.1 Preliminaries and Problem Definition
Zero-shot Learning (ZSL) [17] operates under specific conditions. Given labeled dataset D2 = {(x, " where x; E X' represents input features and yı E V, denotes seen class labels, the model is trained solely seen classes and must handle unseen class samples during testing. In
ZSL, the model is evaluated only on an unseen test set D., {xx, )y) where yj Va and
3

---

Statetio
Closifee
Stektton Soqueeee
(Cg c
Man)
Descripter-wise Prediction
LLM--Gaided Fusion
Y
ouai
Figure 1: (a) Overview of the proposed pipeline that integrates our Greleton-Cachie with a frozen skeleton-based zero-shot action recognition (SZAR) model. (b) Illustration of the cache update process. Kgg: entry count for class y. K: entry limit per class in cache. Amaz The maximum value of ef within 9 class block.
n., = 0. The prediction space is restricted to unseen labels: : X + Vu. In Generalized )1. n
7SL (GZSL), the model must recognize samples from both seen and unseen classes, expanding the prediction space V/, U...
Test-Time Adaptation (TTA). To combat distribution shift during deployment, TTA updates the inference pipeline on-the-fly. Gradient-bate bbased TTA adjusts model parameters 0 for every online mini-batch B. | through one or few optimisation steps:
0, 8 Vo
(I)
where v) the learning rate and L an adaptation objective. By contrast, training-free TTA (TF-TTA) freezes 0 and updates only a light-weight auxiliary state S-for example, a cache or running statistics [10, 11]. Given frozen features z, +(x;) and their zero-shot predictions y., TF-TTA performs
S. = ( 8 v
(2)
where U denotes the update function that modifies the auxiliary state hased on current batch informa- tion, achieving real-time throughput with zero gradient cost. Our Skeleton-Cache instantiates S as the cache detailed below.
3.2 Skeleton-Cache
Current methods for skeleton-basee based zero-shot action recognition (SZAR) are trained solely on seen classes and lack the ability adapt dynamically inference time, which limit their generalization to unseen actions. To address this, we present Skeleton-Cache, a training-free test-time adaptation module that can be integrated into SZAR pipelines for improved generalization to unseen actions without requiring access to training data or any model updates.
Specifically, Skeleton-Cacee Cache reframes action inference dynamic retrieval process. As illustrated in Fig. 1, it constructs and continuously updates a panapermtttiic cache during test time. For each incoming test sequence, the system computes deseriptor-wy predictions and then refines the final classification by leveraging linguistic priors derived from large language models (LLMs). Below we describe the details.
Cache Construction and Update: The cache structured as class-specific blocks, one for each action class, with each block holding up K entries. Each entry is tuple (eg eg. ej,) that captures essential information for retrieval and update: (i) eg is the feature representation (cache key) of a skeleton sequence, used to compute similarity with incoming queries. (ii) e, is the predicted label of the sequence (cache value), used in computing the final prediction for the query. (iii) e;, is the prediction confidence associated with the sample, used to assess the reliability of the sample and guide cache updates. The cache is initialized as empty. As test samples arrive, we selectively insert high-confidence entries into their corresponding class-specific blocks. This maintains compact yet representative memory for inference. Below, we describe how to extract each tuple and the strategy for updating the cache.
Cache Key eg. We denote an incoming test skeleton sequence as x E X where the channel dimension, T is the number of frames, V is the number of joints, and M is the number of

---

persons. Given a trained and frozen SZAR model with skeleton feature encoder (e.g., ST-GCN [30]). we extract features for each skeleton sequence and average over persons to obtain latent tensor F E TRNTTXV where N is the feature dimension after encoding. To obtain the cache key, one straightforward approach is to apply global pooling over F to produce single representation. However, relying solely on global context can be limiting for action recognition. While some actions are distinguishable at coarse level, others differ in subtle, local patterns that holistic features may overlook. For instance, "wave" and "salute" may appear similar globally but differ in fine-grained cues like hand posture or local motion dynamics. In addition, key local patterns often remain consistent across samples of the same class, making them more reliable for retrieval and generalization. To improve the model's ss ability to distinguish between similar actions and accurately recognize intra-class instances, we compute both global and local spatial temporal descriptors as the cache key:
T LLE t,
Sp
F SLE.
g
F
1Vp
VITa
VT
NEV,
IET.
where Sp E ERN, t2 E R and E RN denote the local spatial, local temporal and global descriptors, respectively. [Vp]p= denotes P predefined joint groups corresponding to body parts (e.g., head, torso, arms, legs). (T:) represents Z predefindd temporal segments (e.g., beginning, middle, end) that together span the entire sequence. The final cache key eg E [ is formed by first expanding each descriptor with an additional dimension to form row vectors in I and concatenating them along the first dimension to form a matrix:
e concal (g. S1. Sp, ,,,..., (z) E (P+
Cache Value eg and confidence ejc As shown in Fig. I(a), given testing skeleton sequence, pass it through a trained and fixed SZAR model to obtain prediction logits Φ. Applying the softmax function yields the class probability distribution ý softmax(() over the unseen classes Vu. The predicted label is 2 arg max, PJj and is encoded as a one-hot vector eg E RlV% The prediction confidence is quantified by the entropy of the distribution:
Du Σ j=1
e, =
Py log Pj.
(5)
where lower entropy indicates higher confidence.
Cache Update: For each incoming test sequence, we extract its tuple (cg. Cj,,,)) the cache block corresponding to the predicted class j. As shown in Fig. 1(b). if the number of entries for class ÿ (KCg) is below the maximum capacity K, the tuple is directly appended. Otherwise, we identify the entry within class ý that has the highest entropy h2 (i.c., the lowest confidence). If the current test sample's entropy is lower (indicating higher confidence), replaces the least confident entry with hg i otherwise, no update is made.
Cÿ)) We then examine
For each test sample, we compute its local spatial, local temporal, and global features as defined in Eq. 3, forming descriptor matrix eq (q0),, (2)), ,..,, ((..),
Cache Retrieval for Descriptor-wise Prediction:
(P+2) [R(P=2+)xxN where q()) is the global descriptor,
are the local spatial descriptors, and {q ( .... .q (P+2)) are the local temporal descriptors. Once the cache is populated, each query descriptor is compared with the corresponding descriptor of all cached entries to compute similarity scores. These similarity scores are used to weight the cached values (one-hot labels), producing descriptor-we wise prediction vector for each descriptor type. The set of descriptor-wise predictions is then fused to generate the final class prediction.
Specifically, for each descriptor type d E {0, 1,..., P + 2}. we compare the corresponding query vector q (d) with all cached keys of the same descriptor type. Let (id) represent the cached key of type d from the i-th entry belonging to class j. where j E {1, 2..... .. Dull and E (1,2...., K). The similarity, or affinity, between the query descriptor and cached key is calculated by:
/ = exp| 3(11 11 cos (1) kk99
(6)
5

---

where B > 0 is temperature parameter controlling the sharpness of the similarity distribution, and cos(..-) denotes the cosine similarity.
All affinity scores computed across the cached entries of the [)'ul classes are concatenated into a single vector. Assuming that each class has exactly K entries (resulting in total of Diu K entries), we define the affinity vector for descriptor type d a E (Dx) as:
(d) = [0 [01.
I, 02,11
(d) (d)
(d) 42
(d) DELT
(d) D...k
(7)
We also construct a label matrix Y E R from all entries, where each row a cache value, i.e., one-hot vector indicating the class label of an entry. The descriptor-wise prediction is then computed as:
(8) where (d) represents the retrieved class logits derived from the similarity scores associated with descriptor type d. Although we assume a fixed number of K entries per class for clarity, the above formulas naturally extend cases where the number of cached entries varies across classes.
(d) Y E I
We repeat the above process for all P + Z + 1 descriptor types, resulting in a collection of descriptor- wise prediction vectors {o d=- P+7 Each vector ((d) offers a complementary class prediction based spatial, or temporal. These predictions are subsequently
on different types of descriptors- fused using the LLM-guided weighting scheme described below.
It is worth mentioning that the above computation is highly efficient, as it primarily involves matrix multiplications with sparse one-hot label matrices and vectors of maximum length |Vl K. The computation can be naturally parallelized, further enhancing efficiency. Additionally, the number of descriptors is constant (independent of the skeleton sequence length), meaning that the computational complexity remains unaffected by the sequence duration. This enables Skeleton-Cache to operate in real-time, even for high-frame-rate streams.
LLM-Guided Prior for Fusing Descriptor-wise Predictions: To aggregate the descriptor-wise predictions {o P4Z into the final class logits for a query sequence, we account for the fact that different descriptor type-global, spatial, or temporal-may vary relevance depending on the specific action. Rather than using uniform averaging, we leverage a large language model (LLM) to derive a prior in the form of class-specific weights that capture the relative importance of each descriptor. LLMs are powerful general- purpose reasoning engines with rich semantic knowledge and contextual understanding, making them well-suited for estimating informative priors in training-free manner. These weights are then used to guide a weighted fusion of the descriptor-wiss predictions, enabling more informed and adaptive final classification. Below we describe the details.
Prompt Design and Weight Extraction. For each action category in V.. we issue a single prompt to the LLM to obtain: (i) importance scores for the P spatial body regions, normalized to sum 1, (ii) importance scores for the Z temporal phases, normalized to sum to 1, and (iii) a relative preference score 7 E [0, 1] balancing global versus local descriptors.
Our prompt captures three key aspects: which body parts are most distinctive (e.g., arms for "waving"), "), which temporal phases are critical (e.g.. the apex of "jumping"), and whether the action is better recognized holistically or through local components.
The LLM directly provides all necessary components in its response: spatial scores [wapa Jp=1
[p) L
quantifying body part relevance, temporal scores (x) ) reflecting the importance of temporal segments, and a global-local preference y E [0, 1] indicating whether the action is better represented holistic motion pattern or composition of spatial-temporal primitives. These raw outputs combined into a (P + 7 + + -)ddinensional importance vector as follows:
( = 17, (1 71 w(t). (1 (1-7)
(P). spa (1 (1-7) w( (1 11-7) w
spa
which is subsequently no-orrallzedd obtain the final weight vector w(e). All components (p) "imp- and 7-are derived from the LLM's feedback, effectively translating its semantic understanding of the action into computational attention over descriptors. Details on the prompt design, implemen- tation, and analysis of the LLM responses are provided in the Appendix.

---

Weighted fusion of Descriptor-wise Predictions: Given the set of descriptor-wist prediction vectors { P+Z where each o (d) E R R corresponds to the class logit vector of a descriptor, we stack {old)) d== them along the first dimension form matrix:
concat (o (0) o0) ..... o ofP2)) E E
(10)
where each row corresponds to the prediction logits from specific descriptor. Specifically, ( is the prediction from the global descriptor, {o( Ip=1 are the predictions from the local spatial descriptors,
and (or (P( 1==1 + are the predictions from the temporal descriptors.
We then apply the LLM-derived class-specific weight vector w(() to O to perform weighted fusion
of the descriptor-wiz -iie predictions:
s = w (c) OOORERN RI
(11)
This weighted fusion mechanism ensures that descriptors most relevant given action class- based on the LLM's ' semantic prior-receive greater emphasis in the final prediction. By leveraging linguistic priors to guide the combination of descriptor-wise logits, this approach enhances prediction accuracy without introducing any trainable parameters, thereby preserving the training-free free nature of Skeleton-Cache.
For each testing sample, we leverage the fused logits to enhance the original zero-shot logit & produced by the frozen SZAR model by:
+
(12)
where a, is a balancing coefficient. The posterior used for final prediction is p softmax(o)).
Experimental and Results
4.1
Datasets
NTU RGB+D 60 Dataset [20]. It contains 56,880 skeleton sequence samples of 60 actions, with 40 individual subjects captured from 80 distinet camera viewpoints. Each sample provides temporal sequence of the 3-D location coordinates for 25 human body joints per performer. The maximum performer number and the coordinate values are padded 0 when the corresponding performer is unavailable (e.g., single-person actions).
NTU RGB+D 120 Dataset [16]. It is the extension of the NTU RGB+D 60, contains 114,480 samples for 120 actions performed by 106 individual subjects captured from 155 distinct camera viewpoints.
PKU-MMD Dataset [4]. contains two phases for action recognition with increasing difficulty, which covers the same multi multi-modalities as the NTU dataset. The actions are collected into 51 action categories, and almost 20000 instances are included.
4.2 Descriptor Partitioning and L.MM-Guided Weight Generation
To construct the structured cache keys, we partition skeleton sequences along both spatial and temporal dimensions, enabling Skeleton-Cache to capture mne-grained motion patterns efficiently.
For spatial partitioning, we divide the skeleton into four semantically meaningful body regions based on the Kinect v2 25-joint model: head (joints 2, 3, 4, 8. 20), torso (joints O, 1, 8, 12, 16, 20), arms (joints 4- 21-24), and feet (joints O, 12-19). These groupings align with natural anatomical boundaries and capture localized motion patterns relevant different action types. For temporal partitioning, we uniformly segment each sequence of length T into three maases-beginning
E [1, [ /3])), middle (l E [[T/3] + 1, [2T/3]] and end (1 E [[2T/3] 1, 1,7)- without requiring manual annotation of action phases. The spatial features sp and temporal features 12 are then computed by averaging feature representations across the corresponding joints and frames as defined in Eq. 3. Complete joint assignments are detailed in the Appendix.
7

---

For the LLM-guided weighting mechanism, we leverage GPT-4o to generate flass-specific semantic priors in training-free manner. For each action class, we issue a single structured prompt requesting three components: normalized spatial importance scores across the four body regions, normalized temporal importance scores across the three phases, and a global-local preference parameter E [0. 1] indicating whether the action is better recognized holistically or through local components. The LLM directly returns these values, which we combine via Eq. 9 to form the final weight vector w( without any manual tuning or post-processing. This design preserves the training-free nature of our approach. The complete prompt template is provided in the Appendix.
LLM-Guided Weight Generation. The LLM serves as a source of structured semantic knowledge, generating class-specific priors in a completely training-free manner. To obtain weights for spatial importance, temporal importance, and global-vs-local preference, we query GPT-4o once per action class using a structured prompt without any manual adjustment or post-processing. The LLM directly pp=1. temporal weights {w 1z 2=11 and global-local preference parameter y [0, 1], which are combined according to Eq. 9. We deliberately avoid fine-tuning these weights to preserve the training-free nature of our approach. The quantitative results (Table 3) and qualitative visualizations (Appendix D.1) empirically validate that LLM- generated weights align well with human intuition and significantly outperform uniform averaging, demonstrating the effectiveness of incorporating semantic priors for test-time adaptation in zero-shot settings. The complete prompt template and example responses are provided in Appendix B.2.
returns normalized spatial weights {w (P)
4.3 Implementation Details
To evaluate the generalization ability of our method, we follow the training and testing protocols described in SynSE [8], SMIE [32], PURLS [33], and SA-DVAE [15]. For every baseline and for our Skeleton-Cache, the experimental settings strictly match those reported in the original papers. During inference all pre-trained weights are loaded once and kept frozen. The cache size is fixed to K = prototypes per unseen class; the body-part granularity is P = 4 (head, torso, arms, legs) and the temporal segmentation is Z 3 (begin, middle, end). We use test batch size of 1 to emulate streaming deployment. To construct the framework. All experiments run on a single NVIDIA RTX 4090 GPU. Additional implementation details are provided in the Appendix.
Prior Matrix, we query GPT-4o via the API
As described in the Methods section, we conducted our experiments under both zero-shot learning and generalized zero-shot learning settings. For zero-shot learning, we use Top-1 recognition accuracy on unseen test samples Dj as metric. For generalized zero-shot learning, we use the accuracy on both seen test samples Did and unseen test samples Di. as well as their harmonic mean as metrics.
Our method follows the settings of existing baseline methods, adopting the split settings as [8], with all model training conducted on the seen train dataset. Due to fixed splits potentially not reflecting method performance well, recent papers (SMIE [32], SA-DVAE [15], etc.) have established new setting where they implement three different random class splits of 55/5, 1 110/10, and 46/5 for NTU60, N2U- and PKU-MMD datasets respectively. Our paper has also completed this baseline, with results shown in Table 2.
4.4 Comparison with State-of-the-Art
Table contrasts Skeleton- with leading zero-shot skeleton action recognition approaches on NTU RGB+D 60 and 120, while Table 2 reports results under the random-split protocol and on PKU-MMD, underscoring generalization capabilities. Attaching the cache to heterogencous backbones yields uniform accuracy improvements of 4-8 percentage points across all datasets and splits. Notable results include SA-DVAE climbing from 82.37% 89.41% % (+7.00 700 pp) on NTU60 55/5, PURLS reaching 85.46 46.46% (+6.20 2. pp) on the same split, and similar gains on the more challenging NTU120 96/24 split. The SA-DVAE + Skeleton-Cachh combination achieves 89.866 89% ZSL accuracy on NTU60 and 71.05% on PKU-MMD, surpassing all published numbers. These improvements extend to the GZSL setting with harmonic mean increases of 4-7 pp, confirming benefits for both seen and unseen classes. Across varied evaluation settings, Skeleton-Cache consistently delivers otate-of-the-art performance while adding minimal memory overhead and latency, demonstrating practicality for real-time deployments.
8

---

Table 1: Comparison NTU RGB+D datasets. ZSL: zero-shot Top-1 accuracy. S/U/H: seen,
unseen, and harmonic mean in GZSL. SC: Skeleton-Cache our proposed method.
NTU RGB-D 60
NTU RGBID 120
SS15 Split
48/12 Spiit
110/10 Split
96/24 Spilt
Method
ZSL
U
11
ZSL
5
U
II
ZSL.
5
U
E
ZSL
S
11
ReViSE [P]
53.91
74.22
34.75
47:32
17:49
02.36.2.77
31.16
55,04
48.69
44
46.68
32.38
49.66
25.06
33.31
JPUSR [2g]
64.82
64.44
50.29
56.40
30.75
51.91
47.66
38
CAD--YAE (19)
76.84
69.38
61.79
65.37
28.90 96
51,52
35.41
59.53
47.16 49.7M
48.44
35.77
41.11
34.14
37.31
SynSE
75.81
61.27
56.93
59.02
33.30
52.21 27.85
36.33
62.69
52.51 57.60
54.94
36.70
56 39
32.25
41.04
SMIE
77.98
65.74
PURLS [SJ
79.22
40.99 99
71.95
52.01
SA DVAD [15)
$2.37
02.28
70,80
66.27
41.38
50,90
42.56
68.77
61.10
59.75
60.42
46.12
58.82
35,79
44.50
SSBS++SC
79.7319992
62:31
65.57
63.80144777
(8882
54,02
34.88
42.39
15.777
53,
64 64.80.55.55 58.57
44.1015
37.86
45.90"4..66
SMIF
62.631:4.65
44.1713.09
72.98 17.24
50.4415.14
PURLS+SC
85.4670.24
45.22
77.60 15.65
56,8314488
SA-DVAE+SC
89.41 64.60
79.
71.554
53.22 4844
15.29
62.
53.14
50.83/66633
Table 2: Random-split comparison on three datasets. ZSL: Top-1 zero-shot accuracy; GZSL: harmonic mean for GZSL; SC: Skeleton-Cache, our proposed method.
NTU
NTU 120
PKU-MMD
Method
ZSL.
GZSL
ZSL
GZSL
ZSL.
GZSL
ReViSE [9]
60.94
60.34
44,90
40.34
59.34
49,82
JPoSR [28]
59.44
60.05
46.69
43.69
57.17
51.64
CADA-VAB [19]
61.84
66.38
45.15
45.64
60.74
45.75
SyaSE [8]
64.19
67.47
47.28
43.47
53.85
49.47
SMIE [32]
65.08
46.40
60.83
SA-DVAE [15]
84,20
75,27
50.67
47.54
66.54
54.72
SynSE+SC
68.22::4.3
73.24
54.63
50.0816.61
$3.63
55.31 84
SMIEASC
71.2476.16
50.85
66.7615.93
SA-DVAE+SC
80.21
56.1875.51
51,9474.40
71.05
(3.77
4.5 Ablation Studies & Qualitative Analysis
Hyperparameter Analysis.
Figure 2 illustrates the impact of three key hy- perparameters. For cache size K (Fig. 2a), per- formance improves as we increase from 2 to 8 entries per class, with diminishing returns be- yond that point, indicating that a moderate cache effectively captures class-specific patterns. The balancing coefficient o. (Fig. 2b) shows op- timal performance around 5.0, balancing the influence of cache-retrieved logits against the original zero-shot predictions. The temperature parameter 3 (Fig, 2c) reaches peak performance at 3.0, providing the ideal sharpness for the similarity distribution used in descriptor comparison. Across all settings, performance differences are relatively small and trends are stable across different data splits, indicating that Skeleton-Cache is robust to hyperparameter choices.
Figure 2: Ablation studies on key hyperparameters of the "keleton-Cacht mechanism. (a) Cache size
K. (b) Balancing coefficient a... (c) Similarity temperature parameter 3.
Weighting strategy. To show the benefits of using LLM prior for fusing descriptor-wist predictions, we conduct two other baselines: Random weights: weights are randomly generated for each descriptor, Uniform weights: the same weight is assigned to all descriptors. The comparison results are shown in Table 3 (top). Random weights degrade accuracy because they over-emphasise noisy parts, whereas uniform weights yield modest gains by averaging all cues. LLM-guided weights provide the largest and most consistent improvements, confirming that class-specific priors help the cache focus on informative body regions and motion phases. The class-specific weight generated by the LLM for example actions are provided in the Appendix.
Descriptor granularity. We examine the contribution of different descriptor types in Table 3 (bottom). Using only the global descriptor (g) already surpasses the baseline, but adding spatial ([sp) } temporal ((t=)2=) descriptors is even more beneficial. Spatial information is slightly more valuable than temporal, suggesting that body part localization provides stronger cues for zero-shot generalization. Yet the full eight-part design that combines global, spatial, and temporal

---

Table Ablation study on Skeleton-Cacee Cache components. Weighting: Different fusion strategies for
8 8-part descriptors: Granularity: Comparison of descriptor types; Values show ZSL accuracy (%), i x indicates gain over baseline.
NTU Dataset Splits
Method
55/5 (NTU60)
48/12 (NTU60)
1101O0 (NTUI20)
96/24 (NTU120)
PURLS baseline
79.22
40.99
71.95
$2.01
Weighting strutegy (full descriptors)
Random weights
77.31 11,91
38,56
12.43
68.02 13.93
46.47 15.54
Uniform weights
83.67 14,45
44.88 13.89
75.54 13.59
54.37 12.36
weights
75.6015.65
$6.83
Descriptor gramularity (LLM weights fixed)
Global-only Spatial-only (4 parts)
82. 60.60 13.38
42.13 11.14
73.08 11.13
53.57 11.56
83.79
44.68
75.46
56.07 14.06
Temporal-ont only (3 parts)
82.14 12.92
43.32 72.33
74.52 12.57
54.41 12.40
Full (8 parts)
85.4616.24
45.22 14:23
77.6015.65
56,831
Table Comparison with other TTA methods on NTU-RGBD 60/120 under different SZAR splits. Throughput is measured in samples/ms.
NTU-RGBD
NTU-RGED 120
Throughpot
Method
55/5
48/12
110/10
96/24
PURLS baseline
79.22
40.99
10.09
TPT [24]
40.11
27.98
22.10
17.10
0.70
DifIFT [6]
42.27
28.34
23.43
20.01
0.59
CALIP [7]
80.02
45.10
72.27
51.84
2.48
Ours (Skeletion Cache)
85.46
45.22
77.60
56.83
3.07
descriptors delivers the best performance on every split. The accuracy margin over PURLS reaches 6.24 percentage points on NTU60 55/5.
Comparison with other TTA methods. As shown in the table on the right, we experimented with different TTA methods the PURLS baseline. TPT [24] and DiffTPT [6]perform poorly likely because they use learnable prompts that significantly alter CLIP's text features, disrupting the delicate aligmment between skeleton and text features a critical issue given the limited generalization capability of skeleton-basse based zero-shot models trained small datasets. CALIP [7] enhances modality alignment directly the feature level and achieves significant improvements. Our method further improves performance by introducing fine-grained descriptors and prior knowledge, which increases inter-class separability.
5
Conclusion
We present Saeletonn-Cahher a fully training-free test-time adaptation framework for zero-shot skeleton action recognition. By encoding each input sequence into structured, fine-grained descriptors, reframing recognition
retrieval task via lightweight, non-parametric cache, and integrating retrieved scores through --gidddd importance weights, our method enhances prediction accuracy on novel classes across diverse zero-shot action recognition models without any gradient updates.
Limitation. As a training-free method, our approach builds on the representations extracted by a pre-trained SZAR model. While this enables broad applicability and eliminates the need for additional training, the effectiveness of cache retrieval may vary depending on the quality of input skeletons. In challenging scenarios with severe occlusions pose noise, performance may be moderately affected.
Broader impacts. By combining test-time adaptation with zero-shot skeleton recognition, Skeleton- Cache can empower adaptive gesture understanding in assistive robotics and real-time analytics in sports, while also raising privacy considerations that warrant ethical deployment and clear consent protocols.
10

---

Acknowledgment
This research was supported by the Australian Government through the Australian Research Council's DECRA funding scheme (Grant No.: DE250100030) and Discovery Projects scheme (Grant No: DP210101682).
References
[1] W. Cao, Y. Wu, Y. Sun, H. Zhang. J. Ren, D. Gu, and X. Wang. A review multimodal zero-shot learning. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1488, 2023.
[2] Y. Cao, C. Liu, Z. Huang, Sheng, and Y. Ju. Skeleton-based action recognition with temporal action graph and temporal adaptive graph convolution structure. Multimedia Tools and Applications,
2910999999929999 2021.
[3] Y. Chen, Guo, T. He, and Wang. Fine-grained side information guided dual-prompts for zero-shot skeleton action recognition, 2024. URL https: arxiv. org/ abs//404 07487.
[4] L. Chunhui, H. Yueyu, L. Yanghao, S. Sijie, and L. Jiaying. Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding. arXiv preprint arXiv: TrXiv:1703.07475 2017.
[5] H. Duan, Y. Zhao, K. Chen, D. Lin, and B. Dai. Revisiting skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2978, 2022.
[6] C.-M. Feng, K. Yu, Y. Liu, S. Khan, and W. Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning, 2023. URL https: /atp:s////xxvv/v/ herrg/abs/2308... 2aw.b////00000088.... 06038.
[7] Z. Guo, R. Zhang, L. Qiu, X. Ma, X. Miao, X. He, and B. Cui. Calip: Zero-shot enhanceeent of clip with parameter-free free attention, 2022. URL /tttts://aaattiie /ttpiv////x/vvv//aa http:///awxxvv..raa//eas/ess httpr//aab///2000/// 14169.
[8] Gupta, D. Sharma, and R. Sarvadevabhatla. Syntacticallly guided generative embeddings for zero-shot skeleton action recognition, 2021. URL httpt: awwpvv.rivvvvvaa///i///aa//aaaaa org/ absp///1.... 11530.
[9] Y.HH. Hubert Tsai, L.-K. Huang, and R. Salakhutdinov. Learning robust visual-semantic embeddings. In Proceedings of the IEEE International conference on Computer Vision, pages 3571-3580, 2017.
[10] Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-244 2440, 2021.
[11] A. Karmanov, D. Guan, S. Lu, A. El Saddik, and E. Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14162-14171, 2024.
[12] I. Karunaratne, A. S. Atukorale, and H. Perera. Surveillance of fluman-compute interactions: way forward to detection users' psychological distress. In 2011 IEEE Colloquium on Humanities, Science and Engineering, pages 491-496. IEEE, 2011.
[13] A. I.. Kotian, R. Nandipi, U. M, U. R. S, VARSHAUK, and V. G. T. A systematic review human and computer interaction. In 2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIo')), pages 1214-1218, 2024. doi: 10.1109/IDCI1T597553 2024.
2044 10467622.
[14] M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, and Q. Tian. Actional-structural graph con- volutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3595-3603, 2019.
[15] S.-W. Li, Z.-X. Wei, W.-J Chen, Y-H. Yu, C.-Y. Yang. and J. Y. jen Hsu. Sa-dvae: Improving zero-shot skeleton-based action recognition by disentangled variational autoencoders, 2024. URL https:://// //tp///aavvvv/aa//////eee/eaaav/ org/ /2o. 1346.0 13460.

---

[16] J. Liu, A. Shahroudy, M. Perez, G. Wang, L.-Y. Duan, and A. C. Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence, 2(10):2684-2701 2019.
[17] F. Pourpanah, M. Abdar, Y. Luo, X. Zhou, R. Wang, C. Lim, X.-Z. Wang, and J. review of generalized zero-shot learning methods. IEEE Iransactions on pullern analysis and machine intelligence, 45(4):4051 000405114070, 2022.
J. Wu. A
[18] H.A. Samadh, H. Gani, N. H. Hussein, M. U. Khattak, M. Naseer, F. Khan, and S. Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[19] E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, and Z. Akata. Generalized zero-shot learning via aligned variational autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 54-57, 2019.
[20] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010-1019, 2016.
[21] M. B. Shaikh, D. Chai, S. M. S. Islam, and N. Akhtar. From cnns to transformers in multimodal human action recognition: A survey. ACM Transactions on Multimedia Computing, Commu- nications, and Applications, 10(8):1-24, July 2024. ISSN 1551-6866 URL http: / /txx doi. orgooo////// 10. 1145/3664815.
doi: 10.3645/36
[22] L. Shi, Y. Zhang, J. Cheng, and H. Lu. Skeleton-based action recognition with directed graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7912-7921, 2019.
[23] L. Shi, Y. Zhang, J. Cheng, and H. Lu. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12026- 2019.
[24] M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision- languae models. Advances in Neural Information Processing Systems, 35: 1211444444889, 2022.
[25] Z. Tang, Y. Zhao, Y. Wen, and M. Liu. A survey on backbones for deep video action recognition, 2024. URL https: //axxiv, org/ /a/b///////// abs/2405 .55840
[26] D. Wang, E. Shelhamer, Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint 2rXiv:2006.1077 2020.
[27] Z. Wang, Y. Yang, Z. Liu, and Zheng. Deep neural networks in video human action recognition: A review, 2023. URL https: //t:://///iiv/e// org/ aa/aaabb////...////
[28] M. Wray, D. Larlus, G. Csurka, and D. Damen. Fine- Fine-grained action retrieval through multiple pfrts-of-speech embeddings. In Proceedings of the IEEE/CVF international conference on computer vision, pages 450-459, 2019.
[29] D. Yang, Y. Wang, Dantcheva, Garattoni, G. Francesca, and Brémond. few-iivvriiant skeleton action representation learning via motion retargeting. International Journal of Com- puter Vision, pages 1-16, 2024.
[30] B. Yu, H. Yin, and Z. Zhu. Spatio-temport graph convolutional networks: deep learning framework for traffic forecasting. In Proceedings of the Swenty-Sevents International Joint Conference on Artificial Intelligence, IJCAI-2018, page 3634-3640. International Joint Confer- ences on Artificial Intelligence Organization, July 2018. doi: 10. 00.24963/jcaii20188555 URL http://dx... doi. org/10. 24963/ i ijcai. 2018/505.
[31] Y. Zhang, X. Wang, K. Jin, K. Yuan, Z. Zhang, L. Wang, R. Jin, and T. Tan. Adanpe: Exploring non con-maraettric classifier for test-time adaptation. In International conference on machine learning, pages 41647-41676. PMLR, 2023.
12

---

[32] Y. Zhou, W. Qiang, A. Rao, N. Lin, B. Su, and J. Wang. Zero-shot baseton-bbsed action recognition via mutual information estimation and maximization. ACM International Conference on Multimediaa, MM '23, page 5302-5310, ACM, Oct. 2023. doi: 10.1 00l11453358178..0
Proceedings of the 31st
1888. htthttpp:/dxx doi org/ 10. 1 1144/3181//111..1 36118861
[33] A. Zhu, Q. Ke, M. Gong, and J. Bailey. Part-aware unified representation of language and skele- ton for zero-shot action recognition, 2024. URL https: org orgooog///aoo...ooo...o /0oo//240.... 13327.
13

---

Appendix
This appendix provides detailed supplementary material to support the main paper, Boosting Skeleton- based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation. It is organized into four sections: (1) Algorithm gives the full Skeleton-Cache pseudocode and clarifies notation. (2) Implementation Details covers the efficiency analysis, LLM prompt design, and the spatial/temporal partitioning strategy required for replication. (3) More Quantitative Results extends the empirical study with extra ZSL methods, comparisons with generic TTA baselines, and cache update with adapted logits. Finally, (4) Analytical Visualizations analyzes adaptation effects through weight maps, confusion matrices, top-5 prediction changes, and per-class accuracy changes. These sections collectively enhance the understanding of Skeleton-Care functionality, reproducibility, and performance.
A Algorithm
This section provides a formal description of the Skeleton-Cache algorithm through pseudocode and clarifies the notation used throughout our paper. The algorithm is designed to enhance the general- ization Skeleton-based zero-shot action recognition (SZAR) models by dynamically maintaining non-parametrie cache of confident exemplars and integrating similarity-based predictions with LLM-guided weights, all without requiring model retraining.
Algorithm Skeleton-Cache: Training-Free Free Test-Time Adaptation for SZAR
Require: Test skeleton sequences, action class descriptions, number of unseen classes [Vul, cache size K, spatial groups P. temporal segments Z, hyperparameters a,, 3
Ensure: Adapted class predictions 9
1: Initialize caches Cj- t- for j L,.... PVul
2: for all test samples do
b Phase 1: Feature Extraction
3: Extract skeleton features FERNXTXV with M
4: 4 VT Σズ E 1= F.r..
5:
for p to do
6:
8p 5- V Evev, El= F.t..
7:
for 2 to Z do
8:
I F.I..v eg + concat(g, $1...... .........p.. sp,,.,, ti....,tt)
f. v.r.
EIeT.
9:
b Phase 2: Initial Prediction
10:
Obtain initial label ý and entropy from M
Phase 3: Cache Update
11:
if [Cyl < K then
Cg + Cg U h(ex,h)}
13:
else
Let maxax I/max) = arg (e.)e.)) h if it max then Replace (e mmax with (Ek, h)
15:
16:
Phase 4: Similarity Computation
17:
for j= 1 to Pal do for all (ky,s. E C, do cos(g (4), 1.f))) jj + expl exp(-3(1 (d)
18:
19:
20:
(d) y Y for each descriptor d
Phase 5: LLM-Guuded Adaptation
21:
Generate normalized weight w E IR P+++1 via LLM
< (ol))
- arg max(ssftmaxx( + ags))
We denote the original zero-shot logits produced by the frozen SZAR model Ô. The enhanced logits, augmented by Skeleton-Cache, are defined &o & + (see Eq. 12 in the main paper). The entropy of a probability distribution denoted by H(-). The cache for class is represented as C,. containing tuples of feature representations (cache keys ex). predicted labels
Notation.
14

---

(ey), and confidence scores (e₁). The descriptors include global (g), spatial (sp), and temporal (t.) components, concatenated to form the cache key e; The pseudocode outlines the Skeleton- Cache method, detailing the feature extraction, cache update, similarity computation, and L. L.LM-guided fusion processes.
B Implementation Details
This section provides comprehensive details on the implementation of Skeleton-Cache, including efficiency analysis, the LLM prompt template for weight extraction, and our spatial and temporal partitioning strategies. These details ensure reproducibility and clarify the computational overhead and practical considerations of our method.
B.1 Efficiency Analysis
As mentioned in Section 4.4, Table 5 presents the memory footprint and latency overhead of Skeleton- Cache across four zero-shot splits. The per-class memory consumption ranges from 120.0 KB to 128.0 KB, with the total memory requirement not exceeding 2.94 MB even for the largest split (NTU120 96/24). The computational latency varies from 0.40 seconds for the smallest split (NTU60 55/5) to 6.21 seconds for the largest (NTU120 96/24), reflecting the increased processing demands of cache operations as the number unseen classes grows. These metrics confirm that Skeleton Cachh introduces minimal overhead compared to the base SZAR model, making it lightweight solution suitable for real-time applications.
Table 5: Efficiency overhead of Skeleton-Cache on four zero-shot splits.
Unseen classes
Memory
Extra
Split
per class (KB)
total (MB)
latency (s)
NTU60 55/5
5
128.0
0.64
0.40
NTU60 48/12
12
120.0
1.44
1.98
NTU120 110/10
128.0
1.28
2.27
NTU120 96/24
24
122.7
2.94
6.21
B.2 Prompt Template for LLM-Guided Weights
Skeleton- Cache Prompt (per action class)
You are an expert in human action understanding. Given the action class <ACTION>, answer the following three questions without adding commentary.
1. Spatial importance. The human body divided into four regions: [Head, Torso, Arms, Legs]. Provide a list of four non non-negativv numbers that sum to 1, corresponding to the relative importance of each region for recognising <ACTION>.
Format: "spatial": [w_head, M_torso, arms, w_legs]
2. Temporal importance. The action sequence is divided into three phases: [Beginning, Middle, End]. Provide list of three non-negative numbers that sum to 1, indicating the relative importance of each phase.
Format: "temporal": [w_begin, w_mid, w_end]
3. Global vs local preference. Provide a single number 7 E [0. 1] indicating how much the action should be recognised holistically (r 1) versus by local parts/phases ().
Format: "gamma":
Return one compact JSON object with keys "spatial", "temporal", and "gamma". Do not include any other keys, text, explanations.
15

---

Example Response (action "Waving")
"epatial": [0. 05. 0.10, 0.70, 0.15], "temporal": [o.15, 15, 0.60, 0.25],
"gamma:: 0.30
LLM-guided weight extraction workflow. To obtain the dataset-specific fusion weights, we employ streamlined process leveraging LLM-based knowledge extraction. For each action class in the target dataset, we first construct a class-specific query by inserting the action name into our prompt template. These queries are then batch- setcc-processed through an LLM API (we use gpt-4-ture temperature 5 = 0), which returns structured JSON responses containing three key components: spatial importance weights Wapa R across body regions, temporal importance weights w w!n)
with
(c)
R across action phases, and global-local preference parameter 7 E [0, 1]. We then process these )
raw outputs to construct per-class weight vector (<) h (1 (1->(0) Wapa (c)T (1-7(0) (-)-()())(w)) Wimpl. followed by é -ooomallisaionn to obtain the final vector w(e), The complete LLM-prior matrix W E R. (P+(+++++ formed by stacking these normalized weight vectors for all classes and stored as NumPy array for efficient access during test-time inference. This entire process is performed once per dataset without requiring any model training or fine-tuning, making it computationally efficient and broadly applicable across different skeleton-basee action recognition tasks.
B.3 Spatial and Temporal Partitioning
This subsection describes the partitioning strategy for spatial and temporal descriptors used in Skeleton-Cacee for the NTU RGB+D and PKU-MMD datasets. The spatial partitioning divides the skeleton joints into semantically meaningful groups, while the temporal partitioning employs uniform segmentation approach to capture action dynamics.
Spatial Partitioning. For both NTU RGB+D and PKU-MMD datasets, we partition the skeleton joints into four spatial groups corresponding to distinct body regions: head, torso, arms and feet. These groups are defined based on the joint indices provided in the datasets, ensuring that each group captures anatomically relevant features for action recognition. The specific joint indices for each spatial group in the NTU RGB+D dataset are as follows:
Head: Joints (2, 3, 4, 8, 20), covering the head and upper neck regions, with additional
interpolation include seven joints for robust feature extraction.
Torso: Joints (0, 4, 8, 12, 16, 20), including the spine, shoulders, and central joints, with interpolation to include five additional joints for comprehensive torso representation.
Arms: Joints (4, 5, 6, 7, 8, 9, 10, 11, 21, 22, 23, 24), encompassing both hands and wrists to capture fine-grained hand movements critical for actions like "writing" or "waving."
Feet: Joints {0, 12, 13. 14. 15, 16, 17, 18, 19), covering the lower body. including hips, knees, and ankles, with interpolation to include three additional joints to enhance leg motion capture.
For the PKU-MMD dataset, the same semantic partitioning applied, mapping the corresponding joint indices to these four regions while accounting for any dataset-specific differences in joint definitions. The spatial features Sp for each group p E (1,2,33, 3,4) 4} are computed by averaging the feature representations F F.,.. over the joints E V, and time steps t [1, 7], as described in Algorithm 1, This partitioning ensures that Skeleton-Cache captures localized motion cues relevant to specific body parts, enhancing the discriminative power of the descriptors.
Temporal Partitioning. The temporal partitioning strategy divides each skeleton sequence into three equal segments to capture the dynamics of the action across its progression: heginning, middle, and end phases. For a sequence with T' frames, the temporal segments are defined as follows:
Beginning: Frames t [1, [T/3]].
16

---

Middle: Frames t E [[T/3] + 1, [3]
End: Frames E [[27/3] 1,7].
For each segment E f1,2.3). the temporal features t; are computed by averaging the feature representations F,,, over the frames t E T3 and joints v [1.V], as outlined in Algorithm 1. This uniform segmentation ensures that each phase of the action equally represented, allowing Skeleton-Cache to model temporal evolution without requiring manual annotation of action phases. The approach computationally efficient and generalizes well across datasets with varying sequence lengths, demonstrated in experiments NTU RGB+D and PKU-MMD.
C More Quantitative Results
To further assess the robustness and generalizability of Skeleton-Cache this section broadens the quantitative evaluation with additional zero-shot learning methods, comparisons against generic lest-time adaptation baselines, and analysis of cache update strategies with adapted logits.
C.1 ZSL Performance with Additional Methods
While the main paper evaluates Skeleton-Cache on several established ZSL methods. we extend the analysis here by experimenting with four additional ZSL methods on the NTU RGB+D 60 and NTU RGB+D 120 datasets. These experiments reinforce the trend observed in the main paper: more advanced ZSL methods exhibit greater performance improvements when augmented with Ekeleton-Cacher This is attributed to Skeleton- Cecleton-acches reliance on high-confieecc samples for populating the cache model, as advanced ZSL methods typically produce predictions with lower entropy compared earlier methods, which often suffer from high entropy and class imbalance issues, such as misclassifying all samples of one class into another.
Table 6: Comparison of ZSL accuracy (%) on NTU RGB+D datasets. Our proposed Eeeleton-Cachte (SC) method shows greater improvements on more advanced ZSL methods due to their ability to generate more confident predictions for caching.
NTU RGB+D 60
NTU RGB+D 120
Method
55/5 Split
48/12 Split
110/10 Split
96/24 Split
ReViSE [9]
53.91
17.49
55.04
32.38
JPOSE [28]
64.82
28.75
51.93
32.44
CADA-VAB [19]
76,84
28.96
59.53
35.77
SynSE [8]
75.81
33.30
62.69
38.70
SMIE [32]
77.98
40.18
65.74
45.30
PURLS [33]
79.22
40.99
71.95
52.01
SA-DVAE [1S]
82.37
41.38
68.77
46.12
STAR [3]
81.40
45.10
63.30
44,30
ReViSE
54.70
18.05
55.83
p3.01
JPOSE
66.31
29.84
53,44-1.51
33.56
CADA- VAEASC
78.94
30.49
61.53
37.30
SynSEasC
79.73
38.82
68.47 1577
44.10-5.0
SMIE+SC
82.63
44.17
72.98
50.44
PURLS
85.467424
45.22
77.60
56.83 r4.82
STAR
88.82 17.42
52.0316.2
69.54-6.24
50.02 -572
SA-DVAE+SC
89.41 17.04
47.83
74.29
53.14:7.0
Table compares the Top-1 accuracy of eight ZSL methods, including four additional methods (ReViSE, JPoSE, CADA-VAL VAE, and STAR) not evaluated in the main paper, on four zero-shot splits of NTU RGB+D 60 and NTU RGB+D 120. The results show that Steleton-Cacht (SC) consistently improves performance across all methods, with larger gains observed for more advanced methods PURLS. For instance, SA-DVAE+SC achieves 7.04% improvement on NTU60 55/5 split, compared to only 2.79% improvement for ReViSB+SC. This trend is consistent across splits, as advanced methods generate more confident predictions, enabling Skeleton-Cache to effectively cache representative exemplars, In contrast, carlier methods like ViSE and JPoSE
like SA-DVAE
17

---

often produce high-entropy predictions or collapse to predicting a single class, limiting the cache's
effectiveness.
C.2 Comparison with Generic TTA Baselines
Table 7 contrasts Skeleton-Care Cache with mainstream test-time adaptation (TTA) methhos--gradient based prompt-tuning (TPT [24], DiffTPT [6]), training- prototype or attention modulation (AdaNPC [31], CALIP [7]) and dual-cache retrieval (TDA [1i])using a shared frozen PURLS backbone. By leveraging liseleton-speciinn priors, our method arranges each sequence into eight struc- tured descriptors (global, four spatial parts, three temporal phases) and fuses their similarities under class-wise weights produced by a single LLM prompt. This dedicated design preserves topology, mitigates viewpoint drift, and consistently delivers the best accuracy on every NTU-RGBD split without back-propagation or synthetic augmentations.
In contrast, existing baselines exhibit limitations when applied to skeleton data. Gradient-bee based prompt tuning (TPT, DifITPT) requires iterative optimisation that easily over-fits the small and highly imbalanced target streams. Prototype/attention schemes ( (AdaNPC. CALIP) compress each clip into single holistic vector, discarding fine-gre tine-grained joint dynamics and thus confusing actions that differ only in local motion. TDA partially addresses this with a dual cache, yet still operates at a single global scale and ignores temporal phase diversity. None of these methods explicitly model skeleton topology or spatial-temporal structure, which explains their inferior performance compared with the skeleton-awar retrieval strategy proposed here.
Table 7: Comparison with other TTA methods on NTU-RGBD datasets.
NTU-RGBD
NTU-RGBD 120
Method
55/5
48/12
110/10
96/24
TPT [24]
40.11
27.98
22.10
17.10
DiffTPT [6]
42.27
28.34
23.43
20.01
AdaNPC[31]
81.77
42.52
72.50
53.16
CALIP [7]
80.02
45.10
72.27
51.84
TDA[U]]
82.84
43.95
74.92
54.60
Ours
85.46
45.22
77.60
56.83
C.3 Cache Update with Adapted Logits
To explore the impact of cache update strategies, conducted experiment where Skeleton-Cache is updated using the final adapted logits (post-fusion, as defined in Equation 12) instead of the zero-shot logits from the PURLS model. This approach leverages the enhanced predictions after combining cache-basee similarity logits with the original model logits, potentially incorporating more refined confidence estimates. Table compares the Top-1 accuracy of this adapted-logit strategy
Table 8: Top-1 accuracy (%) of Skeleton-Cache with cache updates using adapted logits versus zero-shot logits NTU RGB+D datasets.
Split
SC (Zero-Shot Logits)
SC (Adapted Logits)
Difference
NTU60 55/5
85.46
85.77
+0.31
NTU60 48/12
45.22
44.66
-0.56
NTU120 110/10
77.60
77.84
+0.24
NTU120 96/24
56.83
56.90
+0.07
against the standard Skeleton Caccht (SC) approach, which uses zero-shot logits for cache updates, across four zero-shot splits of NTU RGB+D 60 and 120. The results show minimal differences, with adapted-logits accuracies 85.77%, 44. 44.96%, 77. 77.84%, and 56.90% compared to 85 85.46%, 45.22%, 77.60% and 56.83% for the standard approach. Notably, a slight performance decline is observed on the NTU60 48/12 split (44. 4.%96 vs. 45.22%). This may stem from reduced overall entropy in the adapted logits, the fusion process (Section 3.2) enhances prediction confidence, potentially allowing
18

---

suboptimal samples to enter the cache despite the high-confidence selection criterion. These samples may not represent the class effectively, impacting retrieval accuracy. The limited performance variation across other splits suggests that the high-confidence filtering mechanism (Section 3.2) is robust, mitigating significant deviations when using adapted logits.
Analytical Visualizations
This section presents visualizations to illustrate the impact of Skeleton-Cache on prediction quality, focusing on confusion matrices, entropy shifts, and LLMM-derived weight distributions. These analyses provide insights into how the method improves class discrimination and confidence in predictions.
D.1 LLM Weight Heatmap Analysis
LLM-Prior Weights for NTU (5r)
Reading
0.30
Acton Chooes
Writing
0.55
0.20
0.15
Jamp
0.10
Pocket
Descriptors (G-Glsbal, AwAens. LeLego. B-Begin, M-Nisole, E-End)
Figure 3: Heat-map visualisation of 199--4-erived, weights w(e), Columns correspond to the eight descriptors; rows correspond to unseen classes of NTU 55/5 split.
LLM-Prior Weights NTU120 (10r)
drop
0.40
put on jacket
0.35
lute
Action Classes
punch or slap
Weight
juggle table tennis table
0.20
object into bag
0.05
wield knife
Descriptors (G-Global, H-Head, T- A-Arms, LoLegs, B-Begin, M M-diddle, E-End)
Figure 4: Heat-map visualisation of HPP--4-deivee weights (()) Columns correspond to the eight descriptors; rows correspond to unseen classes NTU 110/10 split.
Figures 3 and 4 display heatmaps of the GPT-4-derived weights w(e) for the NTU 55/5 and 110/10 splits, respectively. These weights align closely with human intuition about actions. For example, in the NTU 55/5 split, reading and writing emphasize the arms descriptor, while put on hat/cap assigns significant weight (approximately 40%) to head. Similarly, jump up focuses on legs and the middle temporal segment, reflecting the action's peak motion phase. This alignment with commonsense semantics enhances inter-class discriminability by assigning distinct weight patterns to different actions, enabling the model to differentiate between similar motion patterns effectively.
19

---

D.2 Confusion Matrix Analysis
Before Adapestior
Aferr Adaptotion
resding
reading
werting
vatting
But on hations
But habthag
Hump
273
Unsccl pocket
touch
masiing
writing
pit haticap
Jamp up
touch peoceet
mading
serting
pet Matticas
Jamu up
bauc pockest
Proficeed
Aodioced
Figure 5: Comparison of confusion matrices on NTU 55/5 split. Adaptation with Skeleton-Cache sharpens the diagonal and reduces off-diagonal confusion, indicating improved class discrimination.
Before Adaptation
After Adaptation
drop Jacket
drop Jacket
377
16
41
20
30
11
2P
salite
Salutm
headache
232
17
34
heartache
285
0
Tre
Iaggle
slap
THPP
34
320
72
15
244
657
cross smRjtt
CIOSS anRg
520
butti kicks
butt kicks
wleld kritte
121
weeld knite
67
13
545
drop
lacket
headache
slap
sggle
put
cross arms
butt kicks
wield knife
drop
Jacket
headache
mlap
Nggle
put
cross arrns
butt kicks
wied knife
Prodcted
Predicted
Figure 6: Comparison of confusion matrices on NTU 110/10 split. Adaptation with Skeleton-Cache
sharpens the diagonal and reduces off-diagonal confusion, indicating improved class discrimination.
Figures 5 and 6 compare confusion matrices before and after adaptation with Skeleton-Cache for the NTU 55/5 and 110/10 splits. Before adaptation, the model struggles distinguish between similar actions, such as reading and writing, due to their overlapping upper-body poses, resulting in significant off-diagonal confusion. Post-adaptation, the diagonal sharpens, and off-diagonal elements lighten, indicating that Skeleton-Cacee enables differentiation of previously indistinguishable classes. However. this improvement introduces a trade-off: the model occasionally exhibits overconfidence. leading to misclassification in some cases, as evidenced by residual off- off-diagonal elements.
D.3
Top-5 Prediction Changes
Figure visualizes the confidence score shifts in the top-5 predictions for 10 unseen classes in the NTU 110/10 split. Ceeleton-Cacht significantly enhances prediction confidence, particularly for challenging zero-shot tasks. For instance, actions like salute and cross arms show marked increases in confidence for the correct class, reducing uncertainty and refining the prediction distribution. This boost in confidence underscores Skeleton-Cache' 's ability to leverage structured descriptors and bashe- retrieval to improve recognition accuracy for unseen actions.
20

---

D.4 Per-Class Accuracy Changes Before and After Adaptation
To elucidate the impact of Skeleton-Cacht (SC) on per-class performance, we visualize the Top-1 accuracy changes for the NTU RGB+D 55/5 and 110/10 splits before and after test-time adaptation with PURLS+SC (Section 3.2). Figure 8 compares the base PURLS accuracies against the adapted accuracies, highlighting the contribution of the cache-basede based retrieval and L.M--uided fusion.
For the NTU 55/5 split (Figure 8a), Skeleton-Cache consistently improves aceuracy across all classes, with notable gains in writing (76. 10% to 95.96%) and reading (41.39% to 49.08%). These improvements stem from the structured descriptors capturing fine-grained motion patterns (e.g.. hand movements) and the LLM-guided weighting emphasizing relevant body parts (Section 3.2). For the NTU 110/10 split (Figure 8b), most classes show substantial gains, such cross arms (14.86% to 58.70%) and slap (17.45% to 31.27%). benefiting from the cache's 's ability to retrieve discriminative local patterns. However, juggle exhibits significant drop (13.37% to 0.87%), likely due to its complex, multi-limb coordination, which challenges the cache's ' retrieval when high- confidence exemplars are scarce. Overall, Skeleton- enhances generalization for most classes, with occasional limitations for actions requiring intricate motion patterns.
21

---

Iwadache
cross ams
Telbx
23p
Abo
0.0
0.2
0.4 Probabiity
0.8
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Probability
(a) drop
(b) jacket
salute
headache
headache
jettal
cross ams
$50
Aggle
Telox
Telox
0.0
0.2
0.4
0.5
0.8
0.0
0.2
D.4 Probability
0.5
0.8
Probsbility
(c) salute
(d) headache
slap
juggle
1-33
butt bice
step
jeceal
1.ggle
but: dice
cross ams
0.00
0.05
0.10
0.20
0.25
0.30
Probability
Probability
(c) slap
juggle
put
cross arms
emor AMMT
weld Infe
weld Infl
but dices
salute
tatore Alio
fafore
butt 1000
4/o
0.0
0.2
0.4
0.6
0.8
0.0
0.1
0.2
0.3
D.4
0.5
Probsbility
Probsbility
g put
(h) CrOSS arms
kiiks
welld 5m
s31u8t
dap
but ticce
Telow Alsx
Teloxe
d110
amc
Aloo
0.00
0.05
0.10 Probsbiltt
0.15
0.20
0.25
0.0
0.2
0.4 Probability
0.6
0.8
(i) butt kicks
wield knife
Figure 7: Visualization of Top-5 prediction changes before and after applying the Skeleton-Cache for 10 different unseen action classes. Each image displays the confidence score distribution shift when using our caching mechanism, demonstrating how the prediction accuracy improves for challenging shot action recognition tasks.
22

---

Accuracy Comparison 55/5 Split
Base (PURLS)
Accuracy (%)
80
Adapted (PURLS+SC)
60
20
reading
writing
put haticap
jump up
touch pocket
(a) NTU 55/5 Split
Accuracy Comparison for NTU 110/10 Split
100
Base (PURLS)
Accuracy (%)
80
Adapted (PURLS
60
20
drop
Jacket
saluta
headache
slap
juggle
put
cross arms
butt kicks
wield knife
(b) NTU 110/10 Split
Figure 8: Per-class accuracy comparison between base PURLS and adapted PURLS+SC for NTU 55/5 and 1 110/10 splits. Blue bars represent base accuracies, and orange bars represent adapted accuracies.
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: This paper's Abstract and Introduction (Sec. 1) state three key contribu- tions- (i) proposing Skeleton-Caee Cache for training-fire -fee test-time adaptation in SZAR, (ii) introducing structured key-value cache with body-partthemporal descriptors, and (iii) em- ploying a GPT-4-guided prior for descriptor fusion. Methodology (Sec. 3) and Experiments (Sec. validate each claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section contains a dedicated Linitations paragraph noting that performance degrades when skeleton quality is poor (e.g., heavy occlusion or pose noise) and that larger caches could further improve accuracy at modest memory cost.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper presents an empirical framework and does not introduce new theorems; therefore no formal proofs are required.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
25

---

Answer: [Yes]
Justification: Section 4.1 details dataset splits; Section 4.2 lists every hyper- apprr-prrmetter (K=8, P=4, 2=3, 3=3.0, a, =500, batch size 1) and hardware (single RTX 4090). The Appendix provides full LLM prompts and training scripts, enabling reproduction prior to code release.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: All datasets (NTU RGB+D 60/120, PKU-MMD II) public. The Appendix "The code will be released once the paper is accepted." Thus instructions are present, but the repository not public submission time.
states:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/ seeiic/gguieee//eee (CodeSSumsssiionP.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips. http://gggiiide//eeee////i/ie/ee
for more details.
26

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4.2 enumerates data splits, cache size, descriptor granularity, tem- perature, fusion coefficient, frozen backbones, and GPT-4 usage; optimizer details are unnecessary because the approach is training-free.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following previous works in the field, results are reported as single Top-1 accuracy and harmonic mean values without confidence intervals or variance measures. This is due to deterministic inference and computational constraints that limit repeated runs for statistical analysis.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
27

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Section 4.2 states that all experiments run on one NVIDIA RTX 4090 (24 GB). The Appendix reports an extra memory footprint of <5 MB and sub mi-mmiiilscoond latency per sequence.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http//euripp..// cc htc//pubic//Etiieee
lines?
Answer: [Yes]
Justification: This paper uses public datasets collected under institutional review, releases no sensitive data, and discusses privacy the Broader Impacts section (Sec. 5).
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 5 outlines positive applications (assistive robotics, sports analyties) and addresses privacy concerns, recommending explicit consent when Skeleton-Cache is deployed in surveillance contexts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
28

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No new high-risk models scraped datasets are released; Skeleton-Cache only refines predictions existing frozen backbones on public data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section 4.1 cites NTU RGB+D (CC BY-NC-SA 4.0) and PKU-MMD (aca- demic license) with URLs; the Appendix lists GitHub repositories and licenses (MIT/BSD) for all baseline codes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
29

---

Answer: [NA]
Justification:
This paper introduces a method but no new dataset or pretrained model.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: The study relies exclusively on publicly available datasets; no new human- subject data are collected.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No new human-subject research is conducted; IRB approval is therefore applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
30

---

Answer: [Yes]
Justification: Section 3.2 explains that GPT-4 is prompted once per class to derive descriptor- importance weights; this LLM prior is integral to Skeleton-Cache and fully documented.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https: : /ttp///erpppp//eeee Cwterrisscc//oooei wwtcccc///ooooeeeeeeeee /otooeoeeenneee22//////// /LLM) for what should or should not be described.
31


Reviews:
[{"review_id": "QYXitG28Q7", "reviewer": "Reviewer_1vAy", "summary": "The paper addresses zero-shot learning of actions described with skeletons by shaping the task as a retrieval-based classification. The authors propose to use a cache of descriptors of unknown classes to be used for the computation of a cache-based classification score that is weighted with LLM-generated weights and combined with the score provided by a frozen zero-shot model. The experiments are conducted on public datasets and show improved results.", "questions": "In addition to what was already reported in the previous sections, I mention the following points:\n- It is not immediately clear on what population the cache entries are computed; some more details can help clarify\n- Row 161: the size of the skeleton sequence is CxTxVxM. While T and V are clear, I have doubts about the meaning of C and M. Is C referring to the coordinates in the original skeletons? Why do we need M if we only have a person in the sequence? Do we refer to the index of the person?\n- It would be useful, for reproducibility, to have more details about the body and temporal groups\n- At the end of Sec. 4.2 you mention that in addition to the data splits in [8] you also adopted a random one, recently proposed in other approaches, and that the latter is in Table 2. As a consequence, I assume that Table 1 refers to the first split. However, I find in Table 1 different splits, and this is confusing \n- What are the motivations for the choice of the backbones used in your approach?\n- Fig. 2: Is there coherence among backbones?\n- The fact that K does not seem to have a big impact on the results is counterintuitive. Any intuition?\n- Any specific reasons for using PURLS for the last analysis?", "limitations": "Limitations are briefly discussed in the Conclusion", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is well written in almost all its parts, with convincing motivations and a good discussion on existing approaches. The proposed approach revisits a known task, but the application to skeleton-based zero-shot action classification looks like a novel contribution. The experiments appear convincing and rather extensive. \n\nOn the weaknesses, I mention the following points (more detailed questions, if needed, will be in the Question section):\n- Sec. 1 has a good flow, but maybe a more gentle introduction to zero-shot learning would better prepare the unfamiliar reader for the rest of the paper. There is a general definition, but some more practical details on training and test in zero-shot could be beneficial (e.g. how to move from the classification of known classes to the classification of unknown classes in the architecture?)\n- In the same section, but also in the rest of the manuscript, sometimes the motivations seem more related to action classification in general rather than specific to zero-shot settings. For instance, the fact that different action classes might be better recognised with different local/global space/time characteristics is intrinsic in the complexity of the task, I believe. This means, in my opinion, that the classification of known classes should always be improved. These aspects should be discussed in the Introduction and then in the experiments\n- In the experiments, I find the description of the different splits a bit confusing (more details later), so this part should be revised", "was_revised": true, "final_justification": "Considering the reviews, the rebuttal and the discussion with the authors, I confirm my positive opinion on this paper. In the rebuttal, the authors successfully addressed my doubts.", "rebuttal": "We sincerely thank you for your detailed evaluation and constructive feedback. We are encouraged that you found our work well-written and the experiments comprehensive. We address your insightful comments below, which will significantly improve our paper.\n\n### W1 & W2: Introduction Needs Gentler ZSL Overview and ZSL-Specific Motivations\n\nThank you for your insightful comments. Our existing introduction indeed might pose a barrier for readers less experienced in this area, and we will revise it based on your suggestions. We will clarify that Zero-Shot Learning (ZSL) aims to recognize classes absent during training. During training, models learn from labeled samples of seen classes, leveraging auxiliary information to establish connections between visual patterns and semantic knowledge. At test time, models must classify samples from entirely new classes by leveraging learned relationships and auxiliary information, without having seen training examples from these classes. This fundamental requirement creates significant distribution shift, directly motivating our use of Test-Time Adaptation (TTA) to improve model performance on unseen actions.\n\nWe appreciate you raising point (W2). We agree that structured descriptors are a general challenge in action recognition. However, this becomes significantly more difficult in ZSL. For seen classes, models learn to distinguish fine-grained actions through abundant examples. For unseen classes, models rely on high-level semantic guidance, which often fails to capture subtle distinctions. Our TTA method addresses this by dynamically refining structured understanding of unseen actions at inference. As highlighted in Table 1, we observe particularly significant gains for unseen classes. For example, SA-DVAE with TTA achieves a 8.36% increase on unseen accuracy, reflecting the method's effectiveness in this zero-shot context. To better reflect this, we will revise the Introduction and Experiment sections to more clearly emphasize the ZSL-specific motivation.\n\n### Q1: Cache Population Details Not Clear\n\nThank you for the question. The cache is populated exclusively with test-set samples during the inference phase. The process is as follows: The cache is initialized as empty. For each incoming test sample, the frozen SZAR model first generates a prediction. If this prediction is highly confident (i.e., has low prediction entropy, calculated as $\\hat{h} = -\\sum_j p_j \\log p_j$ per Eq. (5)), the sample's feature representation is added to the cache for its predicted class. This selective mechanism ensures that the cache stores only reliable and representative exemplars of the unseen classes encountered at test time.\n\n### Q2: Meaning of C and M in CxTxVxM Tensor Dimensions\n\nWe appreciate the request for clarification. For the input tensor $x \\in \\mathbb{R}^{C \\times T \\times V \\times M}$:\n\n- $C$ is the input channel dimension. C typically equals to 3 representing the (x, y, z) coordinates of each joint.\n- $M$ represents the number of people in the skeleton sequence. Although many sequences contain only one person, the data format sets M = 2 to accommodate up to two individuals—common in datasets like NTU RGB+D, which include two-person interactions (e.g., hugging, shaking hands). When only one person is present, the second person's data is typically zero-padded. This structure ensures consistency across samples and supports both single-person and multi-person actions. This format adheres to the standard data representation widely adopted in the skeleton-based action recognition field, following previous works.\n\n### Q3: More Details Needed on Body and Temporal Groups for Reproducibility\n\nThank you for this suggestion. To improve reproducibility, here are concrete examples of our partitioning strategy:\n\n**(1) Spatial Groups (P=4):** Joints are grouped semantically based on body parts, where joint indices correspond to specific anatomical landmarks in the skeleton structure according to the Kinect v2 25-joint model (e.g., spine base, neck, shoulder center). Each group contains joints from a specific body part across the entire sequence. For our datasets (NTU RGB+D & PKU-MMD), the groups are defined by specific joint indices:\n\n- **Head:** Joints {2, 3, 4, 8, 20},\n- **Torso:** Joints {0, 1, 4, 8, 12, 16, 20},\n- **Arms:** Joints {4, 5, 6, 7, 8, 9, 10, 11, 21, 22, 23, 24},\n- **Feet:** Joints {0, 12, 13, 14, 15, 16, 17, 18, 19}.\n\n**(2) Temporal Segments (Z=3):** Each segment contains all joints within a specific temporal phase. For a sequence of length $T$:\n\n- **Beginning:** Frames $t \\in [1, \\lfloor T/3 \\rfloor]$\n- **Middle:** Frames $t \\in [\\lfloor T/3 \\rfloor+1, \\lfloor 2T/3 \\rfloor]$\n- **End:** Frames $t \\in [\\lfloor 2T/3 \\rfloor+1, T]$\n\nA more comprehensive description of all spatial joint group assignments is provided in the Appendix to ensure full reproducibility.\n\n### W3 & Q4: Confusing Split Descriptions in Experiments and Table 1 vs Table 2\n\nThank you for highlighting this point of confusion. We sincerely apologize for the ambiguity in our use of the term \"split.\" We will revise Section 4.2 to explicitly distinguish between protocols and splits, and to clearly label the evaluation settings in both tables.\n\nTo clarify:\n\n- **Protocol** refers to the evaluation methodology, which defines how classes are divided into seen and unseen sets (e.g., fixed vs. random divisions).\n- **Split** refers to a specific ratio or division of seen and unseen classes within a given protocol.\n\nTable 1 reports results under the Fixed Protocol, as originally introduced in SynSE [8]. This protocol defines four benchmark splits with fixed seen/unseen class divisions:\n\n- NTU60 (55/5 split): 55 specific seen classes, 5 specific unseen classes\n- NTU60 (48/12 split): 48 specific seen classes, 12 specific unseen classes\n- NTU120 (110/10 split) : 110 specific seen classes, 10 unseen classes\n- NTU120 (96/24 split): 96 specific seen classes, 24 specific unseen classes\n\nEach of these splits has a predefined class partition that remains consistent across papers using this protocol.\n\nTable 2, on the other hand, adopts the Random Protocol used in more recent works, which defines three different random seen/unseen class splits and the results need to be averaged over the three splits: \n\n- NTU60: 3 random divisions of 55/5 split and report the averaged performance\n- NTU120: 3 random divisions of 110/10 split and report the averaged performance\n- PKU-MMD: 3 random divisions of 46/5 split and report the averaged performance\n\nWe use the exact same random divisions as prior work to ensure fair comparison. We will revise the manuscript to consistently use this terminology and explicitly label the tables to avoid any ambiguity.\n\n### Q5: Motivations for Choice of Backbones\n\nThank you for this question. We chose these backbones because they are popular and frequently-used baselines in the SZAR literature, enabling direct comparison with prior work. Moreover, they represent diverse ZSL strategies: SynSE uses syntactic guidance, SMIE employs mutual information maximization, PURLS leverages CLIP-style alignment, and SA-DVAE adopts a generative approach. This diversity of methods—from discriminative to generative, from simple alignment to complex synthesis—demonstrates the broad adaptability of Skeleton-Cache across different architectural paradigms. The consistent performance gains across all of them (Tables 1, 2) validate that our method is truly model-agnostic and can enhance various existing approaches.\n\n### Q6: Coherence Among Backbones in Fig. 2\n\nThank you for this insightful question. The trends shown in Fig. 2 are indeed consistent across different backbones. We hypothesize this is because our method operates directly on the final logits from the SZAR backbone through a simple, stable addition via the update rule $\\hat{\\phi} = \\phi + \\alpha_s s$ (from Eq. (12)). While different backbones may produce varying numerical logit values for the same samples due to their distinct architectures, the relative prediction patterns and confidence distributions tend to remain remarkably similar. Since our operation is additive and model-agnostic, it preserves these underlying prediction tendencies while effectively incorporating the support information, providing a stable foundation that leads to the consistent gains shown in Tables 1 and 2 under the same hyperparameter settings regardless of the specific backbone employed.\n\n### Q7: Impact of Cache Size K Counterintuitively Small\n\nWe appreciate this question. The result is indeed less sensitive to $K$ than one might expect. The reason lies in our selective, confidence-based cache update mechanism (detailed in Section 3.2). The cache is not a passive buffer; it actively filters for and stores only the most representative exemplars (those with low prediction entropy). Because of this high-quality selection, a small number of \"clean\" samples (performance saturates around $K=8$) is sufficient to define a robust class prototype, making a larger cache largely redundant.\n\n### Q8: Specific Reasons for Using PURLS for Analysis\n\nThank you for asking. We used PURLS [33] for our main ablation studies (Table 3) and TTA comparisons (Table 4) because it is a strong, recent, and non-generative method. This provides a clean testbed that allows for a more direct and interpretable assessment of our method's contributions without confounding factors. Additionally, many existing TTA methods such as TPT and CALIP, which were originally designed for CLIP-based models, are only convenient to experiment with on PURLS, given its CLIP-like characteristics that make it naturally compatible with these approaches. Therefore, to ensure fair and consistent evaluation, we primarily adopted PURLS as our baseline for comprehensive comparative analysis.\n\nThank you again for your valuable feedback, which will significantly improve our paper."}, {"review_id": "1yEhNLO5zd", "reviewer": "Reviewer_qjj5", "summary": "The paper introduces Skeleton-Cache, a novel framework designed to enhance skeleton-based zero-shot action recognition (SZAR). The key innovation is a training-free test-time adaptation (TF-TTA) module that dynamically adapts to unseen actions during inference without requiring additional training or access to training data. Skeleton-Cache uses a non-parametric cache to store structured skeleton representations, combining global and fine-grained local descriptors. The framework leverages large language models (LLMs) to assign class-specific importance weights, guiding the fusion of descriptor-wise predictions.", "questions": "1. Please clarify the retrieval process details. How is each component used for retrieval?\n2. How is the weighting implemented? Is it class-specific or not? \n3. What are the exact weight values? Did you tweak them a little bit after getting them from LLMs?\n4. Please add a comparison of inference latency with other TF-TTA methods.", "limitations": "The authors could discuss more societal impact. As human data is sensitive, the authors should add more discussions about ethics, etc.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Skeleton-Cache (SC) introduces a novel training-free test-time adaptation framework, which is the first TF-TTA method used in the field of SZAR. In addition, the authors considers local spatial, local temporal features, and a global feature for cache keys.\n2. The method is computationally efficient, avoiding gradient-based updates and relying on a lightweight retrieval process.\n3. Integration with spatial and temporal-specific weights from LLMs: The use of LLMs to derive spatial/temporal-specific importance weights is a unique approach that improves the accuracy of action recognition.\n4. The paper is well written.\n\nWeakness:\n1. Some details of cache retrieval are not clear. From Fig. 1 and Eq. (4), I thought the vector $e_\\hat{k}$ is used for retrieval. However, Eq.(6) suddenly introduced $k_{j,i}^d$. It took me several readings to understand $k_{i,j}^d$ might be one of $g$, $s$, $t$ used in Eq.(4). \n2. The details of the weighting are not clear. In line 234, it seems the weighting is for each class. However, Eq.(9) seems the weighting is the same for all the classes.\n3. The use of LLM may unnecessarily be overclaimed. These weights are essentially a set(s) of predefined weights. LLMs may hallucinate and may lead to worse performance.\n3. The authors should report a comparison of inference latency.", "was_revised": true, "final_justification": "I thank the authors for providing detailed responses. My concerns are solved. I would like to keep my positive rating.", "rebuttal": "We sincerely thank you for your thoughtful and constructive review. Your detailed feedback is immensely helpful, and we appreciate the opportunity to clarify these points and improve our manuscript.\n\n### W1 & Q1: Cache Retrieval Process Details Not Clear ($e_k$ vector vs $q^{(d)}$ notation)\n\nWe sincerely apologize for the lack of clarity in our notation regarding the cache retrieval process. To clarify, the final cache key for a single sequence is a structured matrix $\\mathbf{e}_{\\hat{k}} \\in \\mathbb{R}^{(P+Z+1) \\times N}$, which is formed by concatenating all individual descriptor vectors as you correctly noted: \n\n$\\mathbf{e}_{\\hat{k}} = \\mathrm{concat}(g, s_1, ..., s_P, t_1, ..., t_Z)$\n\nTo simplify notation, we use $q^{(d)}$ and $k_{j,i}^{(d)}$ to denote the $d$-th descriptor (row) of the query matrix and cache key matrix, respectively, where $j \\in \\{1, 2, ..., |Y_u|\\}$ denotes the class index and $i \\in \\{1, 2, ..., K\\}$ denotes the entry index within class $j$'s cache block.\n\nThe retrieval process is performed in a strictly descriptor-wise manner. Given a test sample, we first extract its descriptors to form the query matrix. Then, for each descriptor type $d$, we compute the affinity between the query descriptor $q^{(d)}$ and all corresponding cached descriptors $k_{j,i}^{(d)}$ **across all entries in the cache (i.e., for all $j$ classes and all $i$ entries within each class, totaling up to $|Y_u| \\times K$ comparisons)**. These affinity scores are used to weight the cached class labels, producing descriptor-wise predictions that are subsequently fused. In the revised manuscript, we will restructure **Section 3.2** to first define the full cache key as a matrix of descriptors, and then explicitly explain that the query and cache descriptors are simply the corresponding rows of their respective matrices, ensuring a logical flow from **Eq. (4)** to **Eq. (6)**.\n\n### W2 & Q2: Weighting Implementation Details - Class-Specific or Same for All Classes?\n\nThe weights derived from the LLM are indeed **class-specific**. As stated in **line 234**, \"For each action category $c$ in $\\mathcal{Y}_u$, we issue a single prompt to the LLM...\". This procedure yields a unique set of raw importance scores for each individual class, which are then formulated into a class-specific weight vector $w^{(c)}$. In Eq.9, we did not add class notion for simplicity. We will revise **Eq. (9)** to eliminate ambiguity:\n\n$$w^{(c)} = \\left[ \\gamma^{(c)}, (1-\\gamma^{(c)}) \\cdot w_{\\text{spa}}^{(c,1)}, \\dots, (1-\\gamma^{(c)}) \\cdot w_{\\text{spa}}^{(c,P)}, (1-\\gamma^{(c)}) \\cdot w_{\\text{tmp}}^{(c,1)}, \\dots, (1-\\gamma^{(c)}) \\cdot w_{\\text{tmp}}^{(c,Z)} \\right]$$\n\nThe superscript $(c)$ on all components explicitly denotes their dependence on the action class $c$. This ensures, for example, that the action \"writing\" will have a weight vector $w^{(\\text{writing})}$ that prioritizes the `arms` descriptor, while the action \"jump up\" will have a different vector $w^{(\\text{jump up})}$ that emphasizes the `legs` and `middle` temporal phase descriptors. To further clarify, we will add a declarative sentence immediately following the revised **Eq. (9)** to re-emphasize that the weight vector $w^{(c)}$ is uniquely generated for each unseen class.\n\n### W3 & Q3: LLM Role Overclaimed and Weight Values/Tweaking Details\n\nWe believe it is critically important to avoid overclaiming the role of the LLM, and we thank you for raising this valid point. In our work, the LLM serves as a source of structured knowledge, generating class-specific semantic priors in a completely training-free manner within our test-time adaptation framework. The LLM's key contribution is providing nuanced, class-specific guidance that is otherwise absent, especially valuable in zero-shot settings. To maintain our training-free approach and ensure transparency, we obtain weight values for spatial importance, temporal importance, and global-vs-local preference directly from the LLM using the structured prompt below, without any manual adjustment or tuning. Notably, even without tuning these weights, our method already achieves significant improvements over various baselines, demonstrating the effectiveness of LLM-generated priors. We deliberately chose not to fine-tune these weights to preserve the training-free nature of our approach, though future work could explore weight optimization for potentially better performance.\n\nWe acknowledge that LLMs can exhibit biases and hallucinations; however, the quantitative and qualitative results in our appendix show that these weights align well with human intuition about action recognition, empirically validated by the superior performance of LLM weights over uniform averaging. This LLM-guided weighting makes our adaptation more semantically aware, opening broader implications for domains like medical imaging or scientific discovery, where feature-based and attribute-based learning is limited by expensive expert annotation. Our method offers a pathway to inject domain knowledge without manual labeling. We will revise our discussion to frame the LLM's contribution more precisely in this context.\n\n**Prompt (per action class)**\n\n```\nYou are an expert in human-action understanding. Given the action class <ACTION>, answer the following three questions without adding commentary.\n\n1. Spatial importance. The human body is divided into four regions: [Head, Torso, Arms, Legs]. Provide a list of four non-negative numbers that sum to 1, corresponding to the relative importance of each region for recognising <ACTION>.\nFormat: \"spatial\": [w_head, w_torso, w_arms, w_legs]\n\n2. Temporal importance. The action sequence is divided into three phases: [Beginning, Middle, End]. Provide a list of three non-negative numbers that sum to 1, indicating the relative importance of each phase.\nFormat: \"temporal\": [w_begin, w_mid, w_end]\n\n3. Global vs local preference. Provide a single number γ∈[0,1] indicating how much the action should be recognised holistically (γ≈1) versus by local parts/phases (γ≈0).\nFormat: \"gamma\": γ\n\nReturn one compact JSON object with keys \"spatial\", \"temporal\", and \"gamma\". Do not include any other keys, text, or explanations.\n```\n\n**Example Response (action = \"Waving\")**\n\n```json\n{\n\"spatial\": [0.05, 0.10, 0.70, 0.15],\n\"temporal\": [0.15, 0.60, 0.25],\n\"gamma\": 0.30\n}\n```\n\n\n### W4 & Q4: Inference Latency Comparison with Other TF-TTA Methods Missing\n\nThis is an excellent and very practical suggestion. We thank you for pushing us to conduct this important experiment, as a direct latency comparison was indeed missing from our initial submission. In response to your feedback, we have now benchmarked the performance and inference latency of our method against several prominent TTA baselines, using the PURLS model as a shared backbone for a fair comparison.\n\nTo facilitate direct latency comparison across methods, we employ throughput (samples/ms) as a standardized metric, which measures the number of samples processed per millisecond on an RTX 4090 GPU. **Higher throughput values indicate better computational efficiency**. We measured the throughput on each of the four benchmark splits and report the average in the table below:\n\n|Method|NTU-RGBD 60 55/5|NTU-RGBD 60 48/12|NTU-RGBD 120 110/10|NTU-RGBD 120 96/24|Throughput (samples/ms)|\n|---|---|---|---|---|---|\n|baseline|79.22|40.99|71.95|52.01|10.09|\n|TPT|40.11|27.98|22.10|17.10|0.70|\n|AdaNPC|81.77|42.52|72.50|53.16|2.15|\n|CALIP|80.02|45.10|72.27|51.84|2.48|\n|TDA|82.84|43.95|74.92|54.60|3.22|\n|Ours (Skeleton-Cache)|85.46|45.22|77.60|56.83|3.07|\n\nThe results clearly illustrate the efficiency of our approach. Our method achieves a throughput of 3.07 samples/ms, demonstrating the second-best efficiency among all TTA methods. Notably, our approach achieves superior accuracy performance while maintaining competitive efficiency, significantly outperforming other methods that exhibit slower throughput rates. While our framework shows moderately lower throughput compared to the non-adapting baseline, this computational investment directly translates to achieving new state-of-the-art accuracy across all benchmark splits. This demonstrates a highly favorable trade-off, where our framework's significant performance improvements are attained while maintaining practical efficiency suitable for real-time applications.\n\n### Societal Impact and Formatting Issues (Line 333-334 Break, Fig. 1 Clarity)\n\nThank you for your keen eye in identifying these issues. You are absolutely correct, and we appreciate you bringing these formatting and broader impact points to our attention, as they are essential for a high-quality publication. We will certainly expand the \"Societal Impacts\" subsection to include a more thorough discussion of the dual-use nature of action recognition technologies, the critical privacy implications of processing human skeleton data, and the potential for biases inherited from large language models to affect fairness in downstream applications. We will also add a note on the importance of ethical deployment and clear consent protocols. Furthermore, we will correct the line break issue between lines 333-334 and will redesign Figure 1 for improved clarity, better separating the feature extraction, cache retrieval, and LLM-guided fusion stages to make the overall pipeline more intuitive for the reader."}, {"review_id": "bYgRszAwfQ", "reviewer": "Reviewer_27pj", "summary": "This paper proposes Skeleton-Cache, a novel training-free test-time adaptation framework designed for skeleton-based zero-shot action recognition. Instead of relying on fixed representations, Skeleton-Cache reformulates inference as a retrieval task over a non-parametric feature cache. Each incoming skeleton sequence is encoded into a structured set of descriptors which are stored in class-specific cache entries. During testing, the model retrieves similar descriptors from the cache to refine its predictions on-the-fly. Importantly, the framework leverages large language models to assign semantic importance weights to each descriptor type. This LLM-guided weighting adaptively fuses the descriptor-level predictions, highlighting the most informative body regions or motion phases for each action. Skeleton-Cache integrates seamlessly with existing zero-shot skeleton action recognition models as a plug-and-play module and requires no additional training or gradient updates during deployment. Experiments on standard benchmarks show that adding Skeleton-Cache consistently boosts accuracy on unseen action classes under both conventional zero-shot and generalized zero-shot settings. The proposed method thus addresses the distribution shift between seen and unseen actions by dynamically adapting to test data without retraining, and achieves new state-of-the-art results in SZAR.", "questions": "None", "limitations": "None", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1. The paper is well-written and easy to follow.\n2. The idea of applying test-time adaptation in zero-shot skeleton action recognition is novel. The paper identifies a clear gap, as existing SZAR methods cannot adapt at inference. The authors introduce Skeleton-Cache as the first training-free TTA module for the task, reformulating skeleton action recognition as a cache-based retrieval problem, combined with a structured, multi-scale representation of skeleton data.\n3. The experimental section is thorough and convincing. Results on three benchmarks (NTU-60, NTU-120, PKU-MMD) and multiple data splits show consistent improvements across the board. For instance, aattaching Skeleton-Cache to the SA-DVAE model boosts its zero-shot accuracy on NTU-60/120 by ~6% in top-1 accuracy.\n4. Skeleton-Cache is a training-free, plug-and-play module that can work with different backbone models, which is a practical strength. It does not alter the learned model parameters and introduces only a tiny overhead. \n\nweakness\n1. The paper could offer a deeper analysis of when and why the method might fail or underperform. For instance, one potential concern is how Skeleton-Cache handles misclassifications during the adaptation process. Since the cache is updated using the model’s predicted labels (which could occasionally be wrong for unseen classes), there is a risk that an early misprediction could insert an incorrect prototype into the cache, possibly confusing subsequent retrievals. \n2. The absence of code implementation may undermine the reproducibility of the proposed method.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I will keep my recommendation.", "rebuttal": "We sincerely thank you for your positive and detailed review. We are greatly encouraged by your recognition of our work's novelty, clarity, and experimental thoroughness. We appreciate the insightful points you raised and offer the following clarifications.\n\n### W1: Deeper Analysis of Failure Cases and Misclassification Handling\n\nWe sincerely thank the reviewer for this insightful and important observation. As noted by the reviewer, our Skeleton-Cache mechanism relies on predicted labels to update the prototype cache. If misclassifications occur early, there is a risk that incorrect prototypes are inserted into the cache, potentially misleading future predictions. We have observed this behavior manifest in two primary failure modes:\n\n**Failure Mode 1: Confusion Between Semantically Similar Actions**  \nIn some scenarios, actions like reading and writing share overlapping joint dynamics and can be easily confused by the backbone model, as shown in the following confusion matrix. The early mispredictions can propagate through the cache. \n\nTo address this, in our skeleton-cache (SC), we first employ entropy-based confidence filtering to ensure that only high-confidence predictions are considered for caching, excluding uncertain predictions from the cache. Second, we implement a dynamic replacement policy that continuously monitors cache quality and replaces the least confident entries when the cache reaches capacity. These mechanisms work in tandem to maintain high-quality prototypes: the entropy filtering prevents low-confidence mispredictions from entering, while the replacement policy ensures that any early confident but incorrect samples are gradually displaced by higher-quality examples over time, directly mitigating the risk of performance degradation from initial mispredictions.\n\nWhile we observe some improvements in the diagonal elements (reading: 113→134, writing: 207→261), the cross-confusion remains substantial. Specifically, 61 reading samples (22.3%) are still misclassified as writing after adaptation, demonstrating the limited effectiveness when incorrect prototypes contaminate the cache early in the process.\n\n**Before Adaptation**\n\n|True\\Predicted|reading|writing|put_on_hat/cap|jump_up|touch_pocket|\n|---|---|---|---|---|---|\n|**reading**|113|62|57|25|16|\n|**writing**|20|207|27|4|14|\n|**put_on_hat/cap**|6|23|228|8|7|\n|**jump_up**|18|0|0|256|2|\n|**touch_pocket**|0|5|6|1|263|\n\n**After Adaptation**\n\n|True\\Predicted|reading|writing|put_on_hat/cap|jump_up|touch_pocket|\n|---|---|---|---|---|---|\n|**reading**|134|61|49|20|9|\n|**writing**|2|261|6|1|2|\n|**put_on_hat/cap**|2|11|247|9|3|\n|**touch_pocket**|0|1|0|1|273|\n\n**Failure Mode 2: Amplified Errors from Occlusion and Noise**  \nActions involving complex hand interactions, self-occlusions, rapid movements, and subtle gestures produce inherently noisy features that can be misclassified with high confidence due to model overconfidence. When these incorrect but confident predictions are cached, they actively degrade performance by providing misleading retrieval candidates. In the NTU120 96/24 split, such challenging actions show limited improvements:\n\nWhile our method provides improvements, these actions remain significantly below the overall average accuracy of 56.83%, indicating the substantial gap that persists. Particularly for actions with very poor initial recognition (e.g., thumb up), the improvements are minimal because our mechanism fundamentally depends on the backbone model's representational quality.\n\n|Action|Before Adaptation|After Adaptation|\n|---|---|---|\n|pick up|26.60%|40.86%|\n|take off hat or cap|15.38%|35.54%|\n|rock paper scissors|15.45%|22.46%|\n|thumb up|2.04%|2.77%|\n\n### W2: Code Availability\n\nWe completely agree that reproducibility is important. As stated at the end of our abstract (line 13), we have always planned to release our code. We reaffirm our commitment to make our work fully reproducible. We apologize that we cannot provide the link during the rebuttal stage due to the latest NeurIPS 2025 policy restrictions. We will release the complete source code on a public GitHub repository. This will include:\n\n(1) The full implementation of the Skeleton-Cache module.  \n(2) Pre-generated LLM-guided prior files for all datasets.  \n(3) Scripts to reproduce the main results reported in our paper.  \n\nWe will add a clear link to this repository in the final version of the paper.\n\nThank you once again for your constructive feedback and your confidence in our work. We hope these clarifications address your concerns and demonstrate our commitment to thorough evaluation and reproducibility."}, {"review_id": "VZ0BRPzAQd", "reviewer": "Reviewer_LMYG", "summary": "This paper introduces Skeleton-Cache, a training-free test-time adaptation framework for skeleton-based zero-shot action recognition. By dynamically constructing and updating a non-parametric cache during testing, the model adapts to unseen actions without additional training. Experimental results demonstrate that Skeleton-Cache significantly improves recognition accuracy across multiple benchmark datasets, showcasing its effectiveness and efficiency in zero-shot learning.", "questions": "Would the sequence of the samples influence the test results.", "limitations": "No", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. Innovative Training-Free Test-Time Adaptation. The Skeleton-Cache module operates without additional training, dynamically updating a non-parametric cache at inference time to improve generalization to unseen actions. This design is both practical and efficient, enabling real-time deployment and serving as a plug-and-play enhancement for various skeleton-based action recognition models.\n2. LLM-Guided Semantic Fusion. By leveraging a large language model to assign class-specific importance weights to different body regions or temporal segments, the method enhances focus on the most relevant features for unseen actions. This novel use of semantic reasoning to guide descriptor fusion helps the model better emphasize critical skeleton regions and motion phases in zero-shot scenarios, leading to improved recognition performance.\n\nWeaknesses\n1. Dependence on Pre-trained Feature Extractors. The approach relies heavily on the quality of pre-trained skeleton encoders. If the underlying backbone struggles with noisy, occluded, or low-quality pose estimates, Skeleton-Cache’s adaptation will be limited by those upstream errors.\n2. Cache Memory Overhead. Although non-parametric, the dynamic cache can grow large during long inference sessions or on large-scale datasets. Without careful management, this may introduce nontrivial memory and runtime overhead in resource-constrained deployments.\n3. Reproducibility Concerns. The paper states that code release will follow acceptance, but no pre-release implementation is available. This delays independent verification and hinders rapid adoption by the community.", "was_revised": true, "final_justification": "The authors addressed most of my concerns, and I will keep my recommendation.", "rebuttal": "We sincerely thank you for your valuable time and constructive feedback. We are encouraged that you recognized our framework's innovation and applicability. We address your concerns below.\n\n### W1: Dependence on Pre-trained Feature Extractors Quality Limitations\n\nWe appreciate this thoughtful observation regarding the fundamental dependence on backbone quality—a challenge that extends beyond our method to the broader skeleton-based action recognition field. Indeed, when skeleton pose estimation is severely compromised, all downstream methods face inherent limitations. However, we would like to highlight several design aspects of Skeleton-Cache that provide inherent robustness to these challenges: our structured representation naturally decomposes skeleton sequences into multi-scale descriptors (global, spatial, temporal), providing redundancy when certain body parts are occluded or noisy; the LLM-guided fusion adaptively weights descriptor contributions based on semantic priors, naturally downweighting unreliable features from compromised body regions; and our entropy-based cache selection acts as a confidence filter, preferentially storing high-quality samples while discarding uncertain predictions. These mechanisms collectively help maximize feature utility even when the backbone occasionally produces suboptimal representations.\n\nOur experimental results demonstrate that Skeleton-Cache provides consistent improvements across diverse backbone architectures with varying baseline performance levels. As shown in Tables 1, 2, and 6, we achieve substantial gains on both established methods like SynSE (+3.92% on NTU60 55/5) and state-of-the-art approaches like SA-DVAE (+7.04% on the same split). These improvements hold across different data splits and datasets, suggesting that our framework effectively enhances feature utilization within each backbone's representational capacity. We acknowledge that addressing severe pose estimation failures remains an open challenge for the entire skeleton-based recognition community, but our contribution lies in demonstrating how structured representation and adaptive fusion can maximize the utility of available features, regardless of the underlying backbone's limitations.\n\n### W2: Cache Memory Overhead in Resource-Constrained Deployments\nThis is a very practical concern. We have carefully analyzed the memory footprint and the results confirm that the overhead is minimal. For clarity, we present the full data below:\n\n|Split|Unseen Classes|Memory per Class (KB)|Total Memory (MB)|Sample Memory(MB)|Sample Number|\n|---|---|---|---|---|---|\n|NTU60 55/5|5|128.0|0.64|105.1|1368|\n|NTU60 48/12|12|120.0|1.44|252.7|3291|\n|NTU120 110/10|10|128.0|1.28|326.3|4249|\n|NTU120 96/24|24|122.7|2.94|759.7|9892|\n\n\nAs shown, the total memory required peaks at only 2.94 MB for the largest split with 9892 samples (759.7 MB). Compared to the sample memory, the cache memory overhead is minimal. This analysis demonstrates that Skeleton-Cache is a lightweight solution suitable for real-world deployment where resources may be constrained.\n\n### W3: Reproducibility Concerns - No Pre-release Implementation Available\n\nWe fully agree that reproducibility is paramount. We apologize that we cannot provide the link during the rebuttal stage due to the latest NeurIPS 2025 policy restrictions. However, we are fully committed to releasing our complete source code, pre-trained features, and weight files upon acceptance. To facilitate replication in the meantime, we have provided extensive implementation details in the appendix. This includes:\n\n(1) Detailed pseudocode of the entire pipeline in Algorithm 1.  \n(2) The exact LLM prompt template used for weight generation in Appendix B.2.  \n(3) Precise definitions of our spatial and temporal partitions for feature extraction in Appendix B.3.\n\nWe believe these resources provide a clear blueprint for reproducing our results, and we will release our code as soon as possible.\n\n### Q1: Impact of Sample Sequence on Test Results\n\nThank you for this insightful question. Our system is designed for high robustness against sample order. As detailed in Section 3.2, \"Cache Construction and Update\", the cache is populated based on prediction confidence, where high-confidence samples are added and low-confidence ones are replaced. This ensures the cache quickly stabilizes with representative class exemplars, making its final state largely insensitive to the initial sample sequence. To empirically validate this, we evaluated the PURLS backbone on four splits, each with five different random shuffles of the test set. The results show a negligible standard deviation, confirming that performance is highly robust and not dependent on sample order.\n\n|Run (Random Seed)|NTU60 55/5|NTU60 48/12|NTU120 110/10|NTU120 96/24|\n|---|---|---|---|---|\n|1|85.67|45.32|77.01|56.64|\n|2|85.18|45.62|77.33|56.86|\n|3|84.96|45.12|77.33|56.73|\n|4|85.32|44.94|77.73|57.02|\n|5|85.69|45.46|77.41|56.43|\n|**Mean ± Std. Dev.**|**85.36 ± 0.32**|**45.29 ± 0.27**|**77.36 ± 0.26**|**56.74 ± 0.22**|\n\nThank you again for your valuable feedback, which we believe has helped strengthen our paper."}]

Author rebuttal:

Meta-review: The paper introduces a cache-based approach to improve skeleton-based zero-shot action recognition at inference time without updating model parameters. Reviewers highlighted that the paper is well written, introduces an efficient algorithm, and presents extensive experiments. On the other hand, the contribution is limited to a specific application setup, which appears insufficient to justify a higher rating. Code release is expected upon acceptance notification for reproducibility and community benefits.
Decision: Accept (poster)