Title: Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: zero-shot learning

generalized zero-shot learning

skeleton-based action recognition

test-time adaptation
Abstract: We introduce Skeleton-Cache, the first training-free test-time adaptation framework for skeleton-based zero-shot action recognition (SZAR), aimed at improving model generalization to unseen actions during inference. Skeleton-Cache reformulates inference as a lightweight retrieval process over a non-parametric cache that stores structured skeleton representations, combining both global and fine-grained local descriptors. To guide the fusion of descriptor-wise predictions, we leverage the semantic reasoning capabilities of large language models (LLMs) to assign class-specific importance weights. By integrating these structured descriptors with LLM-guided semantic priors, Skeleton-Cache dynamically adapts to unseen actions without any additional training or access to training data. Extensive experiments on NTU RGB+D 60/120 and PKU-MMD II demonstrate that Skeleton-Cache consistently boosts the performance of various SZAR backbones under both zero-shot and generalized zero-shot settings. The code is publicly available at https://github.com/Alchemist0754/Skeleton-Cache.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Skeleton-Cache, a training-free test-time adaptation framework for skeleton-based zero-shot action recognition. By dynamically constructing and updating a non-parametric cache during testing, the model adapts to unseen actions without additional training. Experimental results demonstrate that Skeleton-Cache significantly improves recognition accuracy across multiple benchmark datasets, showcasing its effectiveness and efficiency in zero-shot learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. Innovative Training-Free Test-Time Adaptation. The Skeleton-Cache module operates without additional training, dynamically updating a non-parametric cache at inference time to improve generalization to unseen actions. This design is both practical and efficient, enabling real-time deployment and serving as a plug-and-play enhancement for various skeleton-based action recognition models.\n2. LLM-Guided Semantic Fusion. By leveraging a large language model to assign class-specific importance weights to different body regions or temporal segments, the method enhances focus on the most relevant features for unseen actions. This novel use of semantic reasoning to guide descriptor fusion helps the model better emphasize critical skeleton regions and motion phases in zero-shot scenarios, leading to improved recognition performance.\n\nWeaknesses\n1. Dependence on Pre-trained Feature Extractors. The approach relies heavily on the quality of pre-trained skeleton encoders. If the underlying backbone struggles with noisy, occluded, or low-quality pose estimates, Skeleton-Cache’s adaptation will be limited by those upstream errors.\n2. Cache Memory Overhead. Although non-parametric, the dynamic cache can grow large during long inference sessions or on large-scale datasets. Without careful management, this may introduce nontrivial memory and runtime overhead in resource-constrained deployments.\n3. Reproducibility Concerns. The paper states that code release will follow acceptance, but no pre-release implementation is available. This delays independent verification and hinders rapid adoption by the community.\n\nQUESTIONS:\nWould the sequence of the samples influence the test results.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Skeleton-Cache, a novel training-free test-time adaptation framework designed for skeleton-based zero-shot action recognition. Instead of relying on fixed representations, Skeleton-Cache reformulates inference as a retrieval task over a non-parametric feature cache. Each incoming skeleton sequence is encoded into a structured set of descriptors which are stored in class-specific cache entries. During testing, the model retrieves similar descriptors from the cache to refine its predictions on-the-fly. Importantly, the framework leverages large language models to assign semantic importance weights to each descriptor type. This LLM-guided weighting adaptively fuses the descriptor-level predictions, highlighting the most informative body regions or motion phases for each action. Skeleton-Cache integrates seamlessly with existing zero-shot skeleton action recognition models as a plug-and-play module and requires no additional training or gradient updates during deployment. Experiments on standard benchmarks show that adding Skeleton-Cache consistently boosts accuracy on unseen action classes under both conventional zero-shot and generalized zero-shot settings. The proposed method thus addresses the distribution shift between seen and unseen actions by dynamically adapting to test data without retraining, and achieves new state-of-the-art results in SZAR.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper is well-written and easy to follow.\n2. The idea of applying test-time adaptation in zero-shot skeleton action recognition is novel. The paper identifies a clear gap, as existing SZAR methods cannot adapt at inference. The authors introduce Skeleton-Cache as the first training-free TTA module for the task, reformulating skeleton action recognition as a cache-based retrieval problem, combined with a structured, multi-scale representation of skeleton data.\n3. The experimental section is thorough and convincing. Results on three benchmarks (NTU-60, NTU-120, PKU-MMD) and multiple data splits show consistent improvements across the board. For instance, aattaching Skeleton-Cache to the SA-DVAE model boosts its zero-shot accuracy on NTU-60/120 by ~6% in top-1 accuracy.\n4. Skeleton-Cache is a training-free, plug-and-play module that can work with different backbone models, which is a practical strength. It does not alter the learned model parameters and introduces only a tiny overhead. \n\nweakness\n1. The paper could offer a deeper analysis of when and why the method might fail or underperform. For instance, one potential concern is how Skeleton-Cache handles misclassifications during the adaptation process. Since the cache is updated using the model’s predicted labels (which could occasionally be wrong for unseen classes), there is a risk that an early misprediction could insert an incorrect prototype into the cache, possibly confusing subsequent retrievals. \n2. The absence of code implementation may undermine the reproducibility of the proposed method.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Skeleton-Cache, a novel framework designed to enhance skeleton-based zero-shot action recognition (SZAR). The key innovation is a training-free test-time adaptation (TF-TTA) module that dynamically adapts to unseen actions during inference without requiring additional training or access to training data. Skeleton-Cache uses a non-parametric cache to store structured skeleton representations, combining global and fine-grained local descriptors. The framework leverages large language models (LLMs) to assign class-specific importance weights, guiding the fusion of descriptor-wise predictions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Skeleton-Cache (SC) introduces a novel training-free test-time adaptation framework, which is the first TF-TTA method used in the field of SZAR. In addition, the authors considers local spatial, local temporal features, and a global feature for cache keys.\n2. The method is computationally efficient, avoiding gradient-based updates and relying on a lightweight retrieval process.\n3. Integration with spatial and temporal-specific weights from LLMs: The use of LLMs to derive spatial/temporal-specific importance weights is a unique approach that improves the accuracy of action recognition.\n4. The paper is well written.\n\nWeakness:\n1. Some details of cache retrieval are not clear. From Fig. 1 and Eq. (4), I thought the vector $e_\\hat{k}$ is used for retrieval. However, Eq.(6) suddenly introduced $k_{j,i}^d$. It took me several readings to understand $k_{i,j}^d$ might be one of $g$, $s$, $t$ used in Eq.(4). \n2. The details of the weighting are not clear. In line 234, it seems the weighting is for each class. However, Eq.(9) seems the weighting is the same for all the classes.\n3. The use of LLM may unnecessarily be overclaimed. These weights are essentially a set(s) of predefined weights. LLMs may hallucinate and may lead to worse performance.\n3. The authors should report a comparison of inference latency.\n\nQUESTIONS:\n1. Please clarify the retrieval process details. How is each component used for retrieval?\n2. How is the weighting implemented? Is it class-specific or not? \n3. What are the exact weight values? Did you tweak them a little bit after getting them from LLMs?\n4. Please add a comparison of inference latency with other TF-TTA methods.\n\nLIMITATIONS:\nThe authors could discuss more societal impact. As human data is sensitive, the authors should add more discussions about ethics, etc.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses zero-shot learning of actions described with skeletons by shaping the task as a retrieval-based classification. The authors propose to use a cache of descriptors of unknown classes to be used for the computation of a cache-based classification score that is weighted with LLM-generated weights and combined with the score provided by a frozen zero-shot model. The experiments are conducted on public datasets and show improved results.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written in almost all its parts, with convincing motivations and a good discussion on existing approaches. The proposed approach revisits a known task, but the application to skeleton-based zero-shot action classification looks like a novel contribution. The experiments appear convincing and rather extensive. \n\nOn the weaknesses, I mention the following points (more detailed questions, if needed, will be in the Question section):\n- Sec. 1 has a good flow, but maybe a more gentle introduction to zero-shot learning would better prepare the unfamiliar reader for the rest of the paper. There is a general definition, but some more practical details on training and test in zero-shot could be beneficial (e.g. how to move from the classification of known classes to the classification of unknown classes in the architecture?)\n- In the same section, but also in the rest of the manuscript, sometimes the motivations seem more related to action classification in general rather than specific to zero-shot settings. For instance, the fact that different action classes might be better recognised with different local/global space/time characteristics is intrinsic in the complexity of the task, I believe. This means, in my opinion, that the classification of known classes should always be improved. These aspects should be discussed in the Introduction and then in the experiments\n- In the experiments, I find the description of the different splits a bit confusing (more details later), so this part should be revised\n\nQUESTIONS:\nIn addition to what was already reported in the previous sections, I mention the following points:\n- It is not immediately clear on what population the cache entries are computed; some more details can help clarify\n- Row 161: the size of the skeleton sequence is CxTxVxM. While T and V are clear, I have doubts about the meaning of C and M. Is C referring to the coordinates in the original skeletons? Why do we need M if we only have a person in the sequence? Do we refer to the index of the person?\n- It would be useful, for reproducibility, to have more details about the body and temporal groups\n- At the end of Sec. 4.2 you mention that in addition to the data splits in [8] you also adopted a random one, recently proposed in other approaches, and that the latter is in Table 2. As a consequence, I assume that Table 1 refers to the first split. However, I find in Table 1 different splits, and this is confusing \n- What are the motivations for the choice of the backbones used in your approach?\n- Fig. 2: Is there coherence among backbones?\n- The fact that K does not seem to have a big impact on the results is counterintuitive. Any intuition?\n- Any specific reasons for using PURLS for the last analysis?\n\nLIMITATIONS:\nLimitations are briefly discussed in the Conclusion", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}