TASAR: Transfer-based Attack on Skeletal Action Recognition
Authors: Yunfeng Diao
Baiqi Wu
Ruixuan Zhang
Ajian Liu
Xiaoshuai Hao
Xingxing Wei
Meng Wang
He Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Human Activity Recognition
Transfer-based adversarial attack
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: This work proposes a new transfer-based attack on skeletal action recognition and builds a large-scale robust evaluation benchmark for this task.

Abstract: Skeletal sequence data, as a widely employed representation of human actions, are crucial in Human Activity Recognition (HAR). Recently, adversarial attacks have been proposed in this area, which exposes potential security concerns, and more importantly provides a good tool for model robustness test. Within this research, transfer-based attack is an important tool as it mimics the real-world scenario where an attacker has no knowledge of the target model, but is under-explored in Skeleton-based HAR (S-HAR). Consequently, existing S-HAR attacks exhibit weak adversarial transferability and the reason remains largely unknown. In this paper, we investigate this phenomenon via the characterization of the loss function. We find that one prominent indicator of poor transferability is the low smoothness of the loss function. Led by this observation, we improve the transferability by properly smoothening the loss when computing the adversarial examples. This leads to the first Transfer-based Attack on Skeletal Action Recognition, TASAR. TASAR explores the smoothened model posterior of pre-trained surrogates, which is achieved by a new post-train Dual Bayesian optimization strategy. Furthermore, unlike existing transfer-based methods which overlook the temporal coherence within sequences, TASAR incorporates motion dynamics into the Bayesian attack, effectively disrupting the spatial-temporal coherence of S-HARs. For exhaustive evaluation, we build the first large-scale robust S-HAR benchmark, comprising 7 S-HAR models, 10 attack methods, 3 S-HAR datasets and 2 defense models. Extensive results demonstrate the superiority of TASAR.  Our benchmark enables easy comparisons for future studies, with the code available in the https://github.com/yunfengdiao/Skeleton-Robustness-Benchmark.

Published as a conference paper at ICLR 2025
TASAR: TRANSFER-BASED ATTACK ON SKELETAL
ACTION RECOGNITION
Yunfeng Diao'
Baiqi Wu l-f Ruixuan Zhang' Xingxing Wei Meng Wang'
Ajian Liu? Xiaoshuai Hao
He WangSt S.t
Hefei University of Technology 2 Institute of Automation Chinese Academy of Sciences
3 Beijing Academy of Artificial Intelligence Beihang University
UCL Centre for Artificial Intelligence, Department of Computer Science, University College London
d Sengyunnennghehee eddovon, (2021214516, 202021292 292: ".mail.hfutt vedovon, jien iienniigee 1u92 9gmm con
xshaoRbari. cn.co, wxweiibuaa.eet adu.co, aai..marawaneRamai..... com, www.whawww...ccc.coo ac.
ABSTRACT
Skeletal sequence data, as widely employed representation of human actions, are crucial in Human Activity Recognition (HAR). Recently, adversarial attacks have been proposed in this area, which exposes potential security concern,, and more importantly provides a good tool for model robustness test. Within this research, transfer-based attack is an important tool as mimics the real-world scenario where an attacker has no knowledge of the target model, but is under- explored in Skeleton- exhibit weak adversarial transferability and the reason remains largely unknown. In this paper, we investigate this phenomenon via the characterization of the loss function. We find that prominent indicator of poor transferability is the low smoothness of the loss function. Led by this observation, we improve the transfer- ability by properly smoothening the loss when computing the adversarial exam- ples. This leads to the first Transfer-based Attack on Skeletal Action Recognition, TASAR. TASAR explores the smoothened model posterior of pre-trained surro- gates, which is achieved by new post-train Dual Bayesian optimization strategy. Furthermore, unlike existing transfer-based methods which overlook the tempo- ral coherence within sequences, TASAR incorporates motion dynamics into the Bayesian attack, effectively disrupting the
HAR (S-HAR). Consequently, existing S-HAR attacks
coherence of S-HARs. For exhaustive evaluation, we build the first large-scale robust S-HAR benchmark, comprising 7 S-HAR models, 10 attack methods, S-HAR datasets and defense models. Extensive results demonstrate the superiority of TASAR. Our bench- mark enables easy comparisons for future studies, with the code available in the https//ggiubbb.oom/yyyucgggaa//e http://iioooSkeeeeteoooossssssss Bowiooo-oooosssssennnnnhhee
INTRODUCTION
S-HAR has been important research topic in computer vision. Recently, S-HAR classifiers have been found to be susceptible to adversarial attack (Wang al., 2021; Diao al., 2021), suggesting adversarial attack potentially provides useful tool for robustness tests for S-HAR classifiers. But not all attacks are equally practical. Existing S-HAR attacks are mainly proposed under white-box settings (Liu et al., 2020a; Tanaka et al., 2022), where the attacker has full access to the victim model's architecture, weights, and training details, or under query-based black-box settings (Diao et al., 2021; Kang et al., 2023b), where the attacker can make numerous queries (Diao et al., 2024a). However, neither approaches are impractical in real-world scenarios (e.g. autonomous driving (Guo al., 2024), intelligent surveillance (Garcia-Cobo & SanMiguel, 2023) and human-computer in- teractions (Wang et al., 2020)), where either accessing the victim model or numerous queries is not attainable. Therefore, transfer-based attack, i.e. generating adversarial examples by attacking a surrogate model and then transfer them to target black-box models, is proposed as a promising alternative (Dong et al., 2018; Wang et al., 2021).
Corresponding author

---

Published as a conference paper at ICLR 2025
Deterministic ersarial Allack
Bayesiue
Pre-tmined Surrogate i8
Beyesumn Starogate @
_LLoss Surface
Las Surthie
Spe Altack
Spanial
BOpo
8.0)
-eg pte-pee 18,0)
High Transter Success Rate x
High Thausfer Success Smoothened Loss Surthee
Smoothened Surface
Cleas Moticn
Tauporal Colereius
Clean Motion
Tengoral Caherence X
No Rerrining v/
No Retminirg X
Lightweight Modell Sizee
Lightweiht Moddl Size X
Post train Dual Bayesian Motion Attack
Sturogate Model
Loss Surface
Pre-tmined Suanogate
Appended
Sperial-ttuporal. Atrack
pio'-0o' 18,6, 8,0")
Ursus Success Rate Suvvthenne Loss Surfece of Temporal Coberence of
Clean Motism
A8"- ("))
/ Frozen
Trainable
6' 20
Reiminn
Dual Bayesian Optimization
Lightwightt Model
Figure I: A high-level illustration of our proposed method. Results marked with 'check mark' (v) indicate superior performance compared those marked with 'cross' (x). Spatial attack: treats each frame independently. Spatial-tempon Attack: integrates temporal motion gradients to disrupt the -emporll coherence of S-HAR models.
However, current transfer-based attack on S-HAR is far from ideal due to their generally poor and unreliable performance. Recently, few studies have attempted to apply white-box S-HAR attacks against black-box models via surrogate models (Wang al., 2021; Liu et al., 2020a). However, results show that their transfer success rate is highly determined by the specific choice of the S-HAR surrogate, so that its general adversarial transferability low (Wang et al., 2023; Lu et al., 2023), also referred to as low/weak transferability. Although similar research in other fields (Dong et al., 2018; Huang et al., 2023; Diao et al., 2024b) has achieved success, a direct application of them on S-HAR still shows low transferability. raising doubt on the usefulness of adversarial transferability in this domain (Lu al., 2023). More importantly, the reason for this failure remains unclear.
We begin by investigating the underlying causes of the low transferability in S-HAR attacks. By first systematically investigating the sensitivity of attack transferability on the choice of surrogates, we compare the loss surface smoothness of the surrogates, inspired by (Wu & Zhu, 2020; Qin et al., 2022). A visual comparison is shown in Figure 2, which gives clear indication of high correlations between loss smoothness and transferability. Consequently, based S-HAR attack should smoothen the surrogate's 's loss during training. Various strategies aim to achieve smoother loss landscapes, via e.g. regularization (Zhao et al., 2022; Foret et al., 2021) or Bayesian learning (Izmailov et al., 201 8; Nguyen et al., 2024; Maddox et al., 2019). We explore the latter and Bayesian Neural Networks (BNNs). This is because BNNs tend to have smooth loss landscapes (Blundell et al., 2015; Izmailov et al., 2018; Nguyen et al., 2024). More importantly, it enables to attack the whole distributions of models, i.e. Bayesian attacks, which has been proven to enhance the transfcrability in other fields (Li et al., 2023; Gubri et al., 2022).
argue that the transfer-
However, it not straightforward to design such a transferable Bayesian attack for S-HAR. First, attacking a distribution of models requires sampling from the posterior distribution. But S-HAR classifiers contain least several millions of parameters (Liu al., 2020b), which makes sampling computationally expensive. Second, most prior transferable attacks are specifically designed for statie data, e-g. images. However, most S-HAR models learn the spatial-temporal features because skeletal data contains rich motion dynamics. A naive adaptation of them ignores the coherence during attack, leading to either lower transfcrability or excessive attack which raises sus- picion. How incorporate the motion dynamics in Bayesian attacks has not been explored.
To tackle these challenges, we propose the first Transfer- Attack specifically designed for Skeletal Action Recognition, TASAR, with key novelties shown in Figure 1. First, our post-train Bayesian strategy keeps pre-trained surrogate intact by appending lightweight Bayesian compo- nents behind it, without the need for re-training of the pre-trained surrogate. Second, propose
2

---

Published as a conference paper at ICLR 2025
novel dual Bayesian optimization for smoothed posterior sampling, which effectively smoothens the rugged loss surface. Finally, unlike previous transfer-based attacks that treat each frame inde- pendently, overlooking the temporal dependencies between sequences, integrate the temporal motion gradient in a Bayesian manner to disrupt the spatial-temporal coherence of S-HAR models. For exhaustive evaluation, we build the first comprehensive robust S-HAR evaluation benchmark RobustBenchHiAR. RobustBenchHAR consists of S-HAR models with diverse GCN structures and latest Transformer structures, 10 attack methods, 3 datasets and 2 defense methods. Extensive ex- periments demonstrate the superiority and generalizability of TASAR.
2 RELATED WORK
Skeleton-Based Human Action Recognition. Early S-HAR research employed convolutional neu- ral networks (CNNs) (Ali et al., 2023) and recurrent neural networks (RNNs) (Du et al., 2015) to extract motion features in the spatial and temporal domains. However, skeleton data topological graph challenges feature representation with traditional methods. Recent advances with graph con- volutional networks (GCNs) (Kipf& & Welling, 2016) have improved performance by modeling skele- tons as topological graphs, with nodes corresponding to joints and edges to bones (Yan et al., 2018). Subsequent improvements in graph designs and network architectures include two-stream adaptive GCN (2s-AGCN) (Shi et al., 2019a). directed acyelic GCN (DGNN) (Shi et al., 2019b), multi-scale GCN (MS-G3D) (Liu et al., 2020b), channel-wise topology refinement (CTR-GCN) (Chen et al., 2021) and auxiliary feature refinement (FR-HEAD) (Zhou et al., 2023). Alongside advancements in GCN-based models, recent studies have explored temporal Transformer structures for S-HARs (Do & Kim, 2024; Qiu et al., 2022; Guo et al., 2024), but their vulnerability remains unexplored. Re- cently, robust S-HAR against adversarial noise has gained attention, with works such as Diao et al. (2024a) exploring adversarial sample distributions and Tanaka et al. (2024) applying Fourier anal- ysis. BEAT (Wang et al., 2023) employs a post-train Bayesian strategy to achieve full Bayesian treatment on clean data, adversarial distribution and classifier. Although post-train Bayesian strat- egy is suggested to be more robust (Wang et al., 2023), its application in S-HAR attacks has not been explored. To address this, we introduce new post-train Dual Bayesian strategy improve adversarial transferability.
Adversarial Attacks on S-HAR. Adversarial attacks (Szegedy et al., 2013) have been applied to various data types, with increasing focus on S-HAR. CIASA (Liu et al., 2020a) proposes a con- strained iterative attack via GAN (Goodfellow et al., 201 4a) to regularize the adversarial skeletons, while SMART (Wang et al., 2021) uses perception loss gradient. Tanaka et al. (2022) suggests only perturbing skeletal lengths. These methods are all white-box attacks, requiring full knowledge the victim model. Different from existing white-box attacks leverage dynamics physical constraints to preserve visual naturalness within white-box settings, we focus on disrupting apatial-temporia coherence to improve adversarial transferability. In contrast, BASAR (Diao al., 2021; 2024a) proposes motion manifold searching to achieve the query-based black-box attack. FGDA-GS (Kang al., 2023a) estimates gradient signs further reduce query numbers. Compared to white-box and query-based attacks, transfer- based attacks (Liu et al., 2016) pose a more practical threat as real- world HAR scenarios typically cannot access white-box information or extensive querying. While existing white- box S-HAR attacks (Wang et al., 2021; Liu et al., 2020a) can be adapted for transfer- based scenarios, they suffer from low transferability and sensitivity to surrogate choices. Lu et al. (2023) proposes a no-box attack for S-HAR, but it also fails in basse attacks. Various type of transfer-based attacks, including gradient- based (Dong et al., 2018; Ma al., 2023; Ge et al., 2023), input transformation (Xic et al., 2019; Zhu et al., 2024; Wang et al., 2024), and ensemble- based methods (Xiong et al., 2022; Li et al., 2023; Tang et al., 2024), exhibit high transfcrability across various tasks but struggle in skeletal data (Lu et al., 2023). Therefore, there is an urgent need to develop a transferable attack for skeleton based action recognition.
3 METHODOLOGY
3.1 PRELIMINARIES
We denote a clean motion x E X and its corresponding label y E y. Given a surrogate action recognizer fo parametrized by 0, is trained map a motion to predictive distribution p(y
3

---

Published as a conference paper at ICLR 2025
0). The white-box attack aims to find adversarial examples & within the neighborhood (x)x) {x: : ||5 - x S e} that misleads the target model fa:
arg min p(y 8x 0) |8 18
where c is the perturbation budget. || ||, is the lp norm distance. The procedure of transfer-based attack is firstly crafting the adversarial example x by attacking the surrogate model, then transferring attack the unseen target model. In Equation (1), since the transferable adversarial examples are optimized against one surrogate model, the ad adversarial transferability heavily relies on the surrogate model learning a classification boundary similar to that of the unknown target model. While possible for image classification, it proves unrealistic for S-HAR (Wang et al., 2023; Lu et al., 2023).
3.2 MOTIVATION
Existing S-HAR attacks have shown outstanding white-box attack performance but exhibit low transferabilityy (Wang et al., 2023). Similarly, previous transfer- based attacks (Dong et al., 2018; Xiong et al., 2022), successful on image data, also show poor transferability when applied to skele- tal motion (Lu al., 2023). Naturally, two questions occur to us: (!) Why do existing adversarial attacks fail to exhibit transferability in skeletal data? (2) Do transferable adversarial examples truly exist in S-HAR?
To answer these questions, we start by generating adversarial examples using various surrogate skeletal recognizers and then evaluate their adversarial transferability. Obviously, in Table 1, the transferability is highly sensitive to the chosen surrogates, e.g. CTR TTR-GCN (Chen et al., 2021) as the surrogate exhibits higher transferability than ST-GCN (Yan et al., 2018). This observation mo- tivates us to further investigate the differences between surrogate models. Previous research (Wu & Zhu, 2020; Qin et al., 2022) has proven that adversarial examples generated by surrogate models with less smooth loss landscape are unlikely to transfer across models. Therefore, we investigate the smoothness of the loss landscape across different surrogate models. In Figure 2, we visualize the loss landscape of ST-GCN and CTR-GCN trained on the skeletal dataset NTU-60 (Shahroudy et al., 2016), and compare their smoothness to the ResNet- (He et al., 2016) trained on CIFAR- 10 (Krizhevsky al., 2009). More landscape visualizations can be found in Appendix C. By ana- lyzing the loss surface smoothness, we have two findings: (1) The loss surface of models trained on skeletal data is much sharper than those trained image data, leading ability. This suggests that adversarial examples within sharp local region are less likely to transfer across models in S-HAR, potentially explaining our first question. (2) CTR-GCN has flatter loss landscape compared to ST-GCN, making it a more effective surrogate for higher transferability. Consequently, we argue that using a surrogate with a smoothed loss landscape will significantly enhance adversarial transferability in S-HAR.
relatively low transfer-
In this work, motivated by evidence that Bayesian neural networks (BNNs) exhibit low sharpness and good generalization (Blundell et al., 2015; Maddox et al., 2019), we aim to construct a Bayesian surrogate by sampling from the model posterior space to smoothen the rugged loss landscape. From a Bayesian perspective, Equation (1) can be reformulated by approximately minimizing the Bayesian posterior predictive distribution:
arg min p(y xx D) 1
arg min Ea- (y x)
where p(0 D) & p(D ())(O), in which D is the dataset and p(0) is the prior of model weights.
3.3 A POST-TRAIN BAYESIAN PERSPECTIVE ON ATTACK
Unfortunately, directly sampling from the posterior distribution of skeletal classifiers is not straightforward task due to several factors. First, directly sampling the posterior is intractable for large-scale skeletal classifiers. Although approximate methods such as MCMC sampling (Welling & Teh, 2011) or variational inference (Blei et al., 2017) are possible, sampling is prohibitively slow and resource-intensive due to the high dimensionality of the sampling space, which typically involves at least several million parameters in skeletal classifiers. In addition. skeletal classifiers normally contain large number of parameters and pre-trained on large-scale datasets (Liu et al., 2019).
4

---

Published as a conference paper at ICLR 2025
SEGGN
STAGCM/BA
STGCNMPB
M8 ST-GCN((-DBR
He
Renott-i((magp
CIR-GUNOLA)
CTR-GEN(PE)
CTR-GNN(-DD
Figure 2: Comparison of loss landscapes of trained models. The I and axis represent two random direction vectors sampled from Gaussian distribution, which are added the model's parameter space along these directions. These random direction vectors are used to assess the sensitivity of the model's s loss function. The z axis represents the loss value. More details can be found in Li et al. (2018) 8). BA means the Bayesian Attack proposed by Li et al. (2023). PB means the post-train Bayesian optimization, and P-DB means the improved post-train Dual Bayesian optimization. The loss landscape optimized by post-train Dual Bayesian is significantly smoother than those of vanilla post-train Bayesian and baseline methods. More visualizations can be found in Appendix C.
Consequently, is not practical for end-users re-train the surrogate Bayesian manner, as the training process is time-consuming
To solve the above issues, we propose new post-train Bayesian attack. We maintain the integrity of the pre-trained surrogate while appending tiny MLP layer gor behind it, connected via skip connection. Specifically, the final output logits can be computed as: logits = go (fo(x)) ) + fx).. In practice, we adopt Monte Carlo sampling to optimize the appended Bayesian model:
max #r
0,0 so max
E
Ep x x, 0,0,6) 64 ~ p(0' (D.)),
(3)
K
where K the number of appended models. Directly training such Bayesian component is in- tractable, so the posterior distribution p(0' D.0) needs to be approximated through sampling, where p(0' 0) x p(D 0, 0'))))) ((()) and p(f") is the prior of appended model weights. Cor- respondingly. Equation (2) can be approximately solved by performing attacks on the ensemble of tiny appended models:
K
1
arg min
E (,0,0))) p(0' (D,0).
(4)
Se K
Our post-train Bayesian attack offers two advantages. First, the appended models composed of tiny MLP layers, getting a similar memory cost to a single surrogate. Second, by freezing fe- post-train Bayesian strategy keeps the pre-trained surrogate intact, avoiding re-training the pre- trained surrogate. More importantly, training on go is much faster than on fo due to the smaller model size of ger
3.4 POST-TRAIN DUAL BAYESIAN MOTION ATTACK
In our preliminary experiments, we found that naive application of post-train Bayesian attack (Equation (4)) already surpassed the adversarial transfer performance of existing S-HAR attacks, which demonstrates the effectiveness of smoothening the loss surface of surrogates. However, its performance remains slightly inferior to the Bayesian attack via re-training a Bayesian surrogate (Li al., 2023)(Equation (2)). This performance gap is understandable, as avoid the prohibitively slow process of sampling the original posterior distribution ~ p(0) D) by using a tiny Bayesian component for post-training instead. To further eliminate the trade-off between attack strength and

---

Published as a conference paper at ICLR 2025
efficiency, propose novel post-train dual Bayesian optimization for smoothed posterior sam- pling, to sample the appended models with high smoothness for better transferability (Figure 2). Moreover, unlike previous transfer-based attacks that assume each frame is independent and ignore the temporal dependency hetween sequences, we integrate motion dynamics information into the Bayesian attack gradient to disrupt the spatial-temporal coherence of S-HAR models. We name method Post-train Dual Bayesian Motion Attack.
3.4.1 POST-TRAIN DUAL BAYESIAN OPTIMIZATION
This motivation is based on the view that models sampled from a smooth posterior, along with the optimal approximate posterior estimating this smooth posterior, have better smoothness (Nguyen et al., 2024). To this end, we aim for proposing a smooth posterior for learning post-train BNNs, hence possibly possessing higher adversarial transferability. Specifically, inspired by the observation that randomized weights often achieve smoothed weights update (Izmailov et al., 2018; Dziugaite & Roy, 2017; Jin al., 2023), add Gaussian noise to smooth the appended network weights. This is achieved by new post-train dual Bayesian optimization:
max Egr Op(o" D.O) Ey Sor oorraaa (u x. 0.0,0' A0)
(5)
For any appended model sampled from the posterior, Equation (5) ensures that the neighborhood around the model parameters has uniformly low loss. We further use dual Monte Carlo sampling to approximate Equation (5):
min
Σ k=1
(x.v.0,00 + Alkm.) A00m ~ N (0,0'1)
(6)
"p MK
where L is the classification loss. Considering dual MCMC samplings computationally intensive, we instead consider the worst-case parameters from the posterior, followed by Li al. (2023). Hence Equation (6) can be equivalent to min-max optimization problem, written as:
min 6 " D.O) (0.02
max
Σ
L (x.vv0000 + A0) .(A0') 2
The confidence region of the Gaussian posterior is regulated by Ç. We discuss the sensitivity to & in the Appendix C. The entanglement between /' and A0' complicates gradient updating. To simplify this issue, we utilize Taylor expansion at 0' to decompose the two components:
8, ( 12.9)
min
(1)
max
K k=1
(x., + Va, Vo (x.,00000)) (0),M(A)) 2E.
(8)
Since A sampled from zero-mert isotropic Gaussian distribution, the inner maximization can be solved analytically. We introduce the inference details, mathematical deduction and algorithm in Appendix B. As shown in Figure 2, the loss landscape optimized by post-train Dual Bayesian is significantly smoother than vanilla post-train Bayesian.
3.4.2 TEMPORAL MOTION GRADIENT IN BAYESIAN ATTACK
Post-train Dual Bayesian Motion Attack can be performed with gradient-based methods such as FGSM (Goodfellow et al., 2014b):
* = x a sign
VL VL Al%m))
(9)
where a is the attack step size. Mcanwhile, for notational simplicity, we notate the classification loss I. (x, y, 0, 0% Allmm) L (x). Assume motion with / frames {x), 02, ... ,.a,, this attack

---

Published as a conference paper at ICLR 2025
gradient consists of a set of partial derivatives over all frames VL(x)
at(x)
8L(x)
87(x)
Barg
axz
Jez assumes each frame is independent, ignoring the dependeney between frames over time. This assumption is reasonable for attacks static data such as PGD (Madry et al., 2017) while infeasible for skeletal motion attacks. In skeletal motion, most S-HAR models learn the apatial-temporal features (Yan et al., 2018), hence considering motion dynamics in the computing of attack gradient can disrupt the spatial-temporal coherence of these features, leading to more general transferability. To fully represent the motion dynamics, first-order (velocity) gradient (VL (x))at and recond-order (acceleration) gradient information (VL (x))dz should also be considered. To this end, we augment the original position gradient with the motion gradient, then Equation (4) becomes:
The partial derivative @L(x)
K
M
signu A=1 m==n=n
Σ W. (VL (V)( dn),
Σ ==0
Wn
(10)
where (VL (x)) = VL(x). Motion gradient can be computed by explicit modeling (Xia et al., 2015) or implicit learning (Tang et al., 2022). Given that implicit learning requires training an additional data-driven model to learn the motion manifold, which increases computational overhead, opt for explicit modeling. Inspired by Lu et al. (2023), we employ tame-narying autoregressive models (TV-AR) skeleton sequences by modeling the temporary non-stationary signals (Xía et al., 2015). We first use first-order T'V-AR(Ja)) and second-order TV-AR(((2)) to model human motions respectively:
et al.. 2017) because TV-AR can effectively estimate the dynamics of
fai a A.- + Be + Yt:
faz 8e = Ct 1 + Dr l/-2 + E. +
(12)
where the model parameters /3, [A,, B,] and /8 IC, Dr, E.j are all time-varying parameters and determined by data-fitting. is a time-dependent white noise representing the dynamics of stochasticity. Using Equation (11), the first-order motion gradient can derived as:
OL(x')
OL(x))
OL(X') A.
(13)
0
0i._1
dz;
Similarly, second-order dynamics can be expressed as below by using Equation (12):
0L(x2)
0L(X))
0L(X') dē;_1
OL(X*) 0z
Co-1 +
(D+ C. C.. CC-1),
(14)
02
Di,
where C.
and Dr
After computing 8 Cc-1 10-1-11-1 2i + De-1 2i +
02:
as5
E-11 + 7t-1, we compute C
Overall, the high-order dynamics gradients
02
over all sequences can be expressed as (VI (x))d1
[(
[L(x))
(BL(x)
[L(x)) chre
and
d1 8r2
LVL(x))as
[(
aL(x)
[L(x)
OL(X)
Bieg
82
d2
4
EXPERIMENTS
4.1
RobustBencHHAR SETTINGS
To our best knowledge, there is no large-scale benchmark for evaluating transfer-based S HAR attacks. To fill this gap. we build the first large-scale benchmark for robust S-HAR evaluation, named RobustBenchHAAR. We briefly introduce the benchmark settings here, with additional details available in Appendix D.
(A) Datasets. RobustBenchHAR incorporates three popular S-HAR datasets: NTU 60 (Shahroudy al., 2016) NTU 120 (Liu et al., 2019) and HDMOSSMüller et al., 2007). Since classifiers not have the same data pre-processing setting. we unify the data format following (Wang et al., 2023). For NTU 60 and NTU 120, we subsampled frames to 60. For HDM05, segmented the data into 60-frame samples.
(B) Evaluated Models. We evaluate TASAR in three categories of surrogate/victim models. (1) Normally trained models: We adapt commonly used GCN-based models, i.c., ST-GCN (Yan al., 2018), MS-G3D (Liu

---

Published as a conference paper at ICLR 2025
Table 1: The attack success rate(%) of untargeted transfer-based attacks on NTU60 and NTU120.
Ave' was calculated as the average transfer success rate over all target models except for the surro- gate. 'Former' represents SkateFormer and MI stands for MI-FGSM.
Daiase:: NTL.00
Dataset: NTTTE2
Surrogute Method
Target Models
Ave
Target Models
Ave
STGON
2AGGCN
MSGM>
CTRGEN FRHEAD SFommer
STGON
25AGCN
MSG31 CTRGON FRHEAD SFomme
IRGSM
99.26
11.76
8.35
14.22
16.22
15:44
13:25
96.81
5.82
7.10
13.97
16.42
24.75
14.27
MI
100.00
17.76
27.20
14.95
26.59
11:76
99.63
18.75
28.18
15.07
23.03
SMART
93.28
5.62
2.19
6.88
7.19
10.08
6.39
94.06
8.28
7.66
11.09
10.16
16.12
10.66
STOUN
CIASA
100.00
3.43
3.43
7,60
9.80
8.33
6.52
100,00
4.16
4.41
9.07
8,08
14.95
8.13
MIG
99.50
39,60
18.14
27.91
98.01
17.45
23.01
21.53
DIM
77.97
20.54
34.03
12.13
28. 83
13.11
21.75
75.61
10.76
12.25
12.75
16.21
23.01
15.00
TASAR
99.29
42.55
64,60
20.33
49.41
17.22
38.82
99.26
19.60
19.37
15.28
22.79
25.24
20.46
FGGM
25.49
22.79
100.00
20.10
24.75
16.66
21.36
26.56
16.42
100.00
15.20
18.38
27.20
20.35
MI
22.42
13:72
100.00
14,83
20.22
12.25
16,09
25.49
12.25
100.00
14.46
16.78
22.
18.26
SMART
100,00
12.09
31.25
13.96
100,00
17.92
23.38
MSG3D
CIASA
17.40
5.88
100.00
11:27
11.51
11.76
11.56
22.79
5.88
100.00
11.03
12.50
19.11
14.26
MIG
31.92
39.65
100.00
24.44
36.15
23.06
11.04
32.17
27.22
100.00
23.27
31.18
33.54
29.48
DIM
28.58
47.27
100.00
17.82
35.27
17. 69
29.35
30.94
38.24
100.00
19.43
30.19
29.
29.72
TASAR
48.87
51.18
99.61
41.49
40.14
23.90
41.11
41.16
47,22
100.00
28.83
40.60
40.37
39.65
IFGSM
27.45
13.72
33.33
14.95
14.33
97.30
31.49
25.02
MI
25.36
23.52
36.51
99. 02
51.34
85
$1.32
%0.14
19.73
29. 16
$9.
29. 16
28, 80
27.30
SMART
15.00
5.00
4.69
99.69
15.31
9.27
9.85
19.75
5,84
4.63
99.60
9:27
17.13
11.32
CTRGON CIASA
14.70
4.65
5.88
99.75
15.93
9.31
10.09
19.60
5.88
4.65
99.75
10.53
16.91
11.51
MIG
28,86
35,34
48.19
93.55
53.46
21.04
37.38
30.94
24.75
32.67
94,18
34.03
29.45
30.37
DIM
53.16
17:51
29.51
19.49
24.87
62.31
24.57
TASAR
33.76
52.31
66.74
97.06
$8.32
21.07
46.44
33.59
26.22
33.82
92.89
35.78
32.84
32.45
IFGSM
23,03
15.19
11.27
14,95
16,42
13:48
15.72
26.26
13.97
12.99
13:44
20.83
24.30
19,00
MI
18.13
12.29
19.36
12.25
19.36
10.78
15.36
26.22
21.07
32.35
15,20
22.54
23:77
23.55
SMART
21.77
6.04
6.04
11.29
10.08
10.88
11.02
23.79
9.27
4,43
9.27
12.90
21.37
13.51
ST
CIASA
10.53
MIG
22.31
21.44
18.89
16.77
23.44
16.77
19.94
30.54
20.32
21.88
16.46
21.25
24.87
22.55
DIM
23.39
53.04
32,67
15:47
28.71
14.72
24.67
29.82
15.84
14.72
13.99
19:05
24.50 40
19.65
TASAR
26.44
54.32
42.78
16.35
37.98
18.38
34.61
34.61
46.63
26.92
32.45
al., 2020b), CTR-GCN (Chen et al., 2021), 2s-AGCN (Shi et al., 2019a), FR-HEAD (Zhou et al., 2023), and two latest Transformer-hase models SkateFormer((o & Kim, 2024) and STTForeer(Quu et al., 2022). To our best knowledge, this is the first work to investigate the robustness of Transformer-basee S-HARs. (2) Ensemble models: an ensemble of ST-CGN. MS-G3D and DGNN (Shi et al., 2019b). (3) Defense models: We employ BEAT (Wang et al., 2023) and TRADES (Zhang et al., 2019a), which demonstrate their robustness for skeletal classifiers.
(C) Baselines. We compare with state-of-the-art (SOTA) S-HAR attacks, i.c. SMART (Wang et al., 2021) and CIASA (Liu et al. 2020a). We also adopt the SOTA transfer-based attacks baselines. including gradient- based, i.e., I-FGSM (Kurakin et al., 2018), MI-FGSM (Dong al., 2018) and the latest MIG (Ma et al., 2023), input transformation method DIM (Xie et al., 2019), and baeembllebbsee/Bayeeee et al., 2018), SVRE (Xiong et al., 2022) and BA (Li et al., 2023). For a fair comparison, we ran 200 itcrations for all attacks under loo norm-bounded perturbation of size 0.01. For TASAR, we use the iterative gradient attack instead of FGSM in Equation (10).
attacks, i.e., ENS (Dong
(D) Implementation Details. Our appended model is simple two-layer fully-comnected layer network. Unless specified otherwise, we use K 3 and M 20 in Equation (10) for default and explain the reason in the ablation study later. More implementation details can be found in Appendix D.
4.2 EVALUATION ON NORMALLY TRAINED MODELS
Evaluation of Untargeted Attack. As shown in in Table 1, TASAR sigmificantly surpasses both S-HAR attacks and transfer-based attacks under the black-box settings, while maintaining comparable white-box attack performance. Specifically, TASAR achieves the highest average transfer success rate of 35.5% % across different models and datasets, surpassing SMART (Wang et al., 2021) (the SOTA S-HAR attack) and MIG (Ma et al., 2023) (the SOTA transfer-based attack) by a large margin of 23.4% and 8.1 % respectively. Moreover, TASAR shows consistent transfcrability across all surrogate models, target models and datasets. These improvements break the common belief that transfer-basee attacks in S- S-HAR suffer from low transfcrability and highly rely on the chosen surrogate (Lu et al., 2023).
Evaluation of Targeted Attack. In this section, we focus on targeted attacks under the black-box setting. Improving targeted attack transferability on S-HAR generally more challenging than untargeted attacks. This primarily due to the significant semantic differences between the randomly selected class and the original one. Attacking a 'running' motion to 'walking' is generally easier than to 'drinking'. This is why targeted attacks have lower success rate than untargeted attacks. However. Table 2 shows TASAR still outperforms the baseline under most scenarios. Moreover, TASAR can successfully attack the original class a target with an obvious semantic gap without heing detected by humans. The visual examples can be found in Figure 4.
8

---

Published as a conference paper at ICLR 2025
Table 2: The targeted attack success rate (%)
Table 3: The untarget attack success rate (%)
of targeted transfer- based attack on NTU60.
against defense models on HDM05 (top) and
NTU 60 (bottom).
Surrogate
STOCN
SMART
TASAR
Method
MI
MI
STGUN
27.45
26.79
28.02
2,08
25AGCN
6.06
1.20
3.05
MSG3D
52.72
6,06
2.32
1.81
CTRGCN
8,33
1.41
1.71
1.33
FREIEAD
6.82
1.81
1.71
6.82
2.20
1.56
Surogate
STGOK
KILFISM
SMART
Methodi
TASAR
3
2,8
3.35
TRADEES MIGENMSSICCCRRR
3.13
3.75
2.94
3.54
1.58
STOCCN MSSTICCRRE
80.13
96,45
60.16
22.29
3.34
BEAT
15.45
2,50
MSG3D
SMART
0.80
0.60
44,95
1.01
1.01
0.86
MSG3D
SMART
2,50
3.13
6.09
82.36
TASAR
9.09
9.09
57.58
9.85
9.33
9.34
TASAR
12:26
10.29
12:25
59.38
100.00
58,59
MI
3.06
3.30
2.81
29.53
4.53
3.43
SI FGSM
16,05
5.31
3,40
53.33
20.19
15.05
CTRGCN
SMART
1.11
1.61
43.95
STGIN
SMART
12.50
5.78
9.06
73.95
4.68
8.28
TASAR
8.33
9.09
8.33
22.73
9.09
8.71
12.50
12.50
MI
1.47
98.61
1.83
1.83
1.47
1,65
MLPCISM
23.4
13:11
2aAGON
SMART
2,21
1.20
2.62
MSG3D
SMART
19.45
26.71
79.68
TASAR
10.61
4.56
10.61
TASAR
19.79
14.58
17.71
40.63
100.00
32.29
4.3 EVALUATION ON ENSEMBLE AND DEFENSE MODELS
Evaluation Ensemble Models. TASAR benefits from the additional model parameters added by the ap- pended Bayesian components. For a fair comparison, we compare with SOTA ensemble-base based methods, iee. ENS Dong et al. (2018) and SVRE Xiong et al. (2022), and the Bayesian Attack (BA) Li et al. (2023), because they also benefit from the model size. ENS and SVRE take three models ST-GCN, MS-G3D and DGNN as ensemble of surrogate models, while BA and TASAR only take MS-G3D as the single substitute architec- ture. Unlike BA re-training the surrogate into BNN, TASAR instead appends small Bayesian component for post-training. We choose ST-GCN, 2s-AGCN, MS-G3D, CTR-GCN, FR-HEAD as target models, and evaluate the average white-box attack success rate (WASR), average black-box attack success( (BASR) and the number of parameters in Figure 3. We can clearly see that TASAR (blue line) achieves the best attack perfor- mance under both white-box and black-box settings, with an order of magnitude smaller model size. When using MSG3D (12.78M) as the surrogate model, the Bayesian components appended by TASAR only increase 0.012M parameters of the surrogate size, resulting in a memory cost comparable to that of a single surrogate. In contrast, the Bayesian surrogate model used by BA has 15 times more parameters (255.57M) than the single surrogate.
Since both BA and TASAR are Bayesian- attacks, we compare the smoothness of their loss landscape in Figure 2. It can seen that both BA and TASAR exhibit the ability to smoothen the loss landscape, providing empirical evidence for the Bayesian surrogate's effectiveness smoothening the loss surface, Further, TASAR and BA achieve the top-2 performance in transfer- based attacks, highlighting the high correlations between loss smoothness and transferability. Compared to BA, TASAR exhibits significantly flatter loss landscape, aligning with the higher transfer success rate than BA. The key difference between BA and TASAR is that TASAR samples from a smoothed posterior, which shows the benefit of smoothed posterior sampling for improving adversarial transferability.
Evaluation on Defense Models. As BEAT shows high robustness against S- HAR white-box attack (Wang al., 2023), it is also interesting evaluate its defense performance against black-box box attack. We also em-
LNG
-TUS WASR
SYTE
8A
HDMOS-BASR
TASAR
NTU60-BASR
21%
/imh
HDMOS-WASR
original motion (Throw)
10%
NTU120-WASR
M00
2000
Model Size
NTU120-BABR
adversarial example (Lie down)
Comparisons with ensemble and Baycsian attacks. We calculate the model size and evaluate the average white-box (WASR) and black-box attack success rate (BASR) on the HDM05, NTU60, and NTU120 datasets, respec- tively.
Figure 3:
Figure 4: The ground truth label 'Throw" can be misclassified as 'Lie down' on tar- geted attack by TASAR. The semantic differ- ences between ground truth labels and target labels are large.

---

Published as a conference paper at ICLR 2025
ploy the adversarial training method TRADES (Zhang et al., 2019a) baseline due its robustness in S-HAR (Wang al,, 2023). Obviously, in Table 3, TASAR still achicves the highest adversarial transfcrability among the compared methods against defense models, further validating its effectiveness.
4.4 ABLATION STUDY
Dual MCMC Sampling. TASAR proposes a new dual MCMC sampling in the post-train Bayesian formulation (Equation (10)). To see its contribution, conduct ablation study the number of appended models (K and in Equation (10)). To isolate impact of the number of appended models, employ TASAR without motion gradient. The contribution
motion gradient will be discussed in the subsequent ablation experiment. As shown in Table 4, compared with vanilla Post-train Bayesian strategy (M=0), the dual sampling significantly improves the attack performance. Furthermore, although TASAR theoretically requires intensive sampling for inference, in practice, we find small number of sampling is sufficient (K and M 20). More sampling will cause extra computation overhead. we use K 3 M 20 by default.
Surrogate Motion Gradient STOCH NO MB
Table 4: Ablation Study on NTU 60 with ST-GCN as the surrogate. M and K are the number dual MCMC sampling.
STOCH MG
MSG2D NO MG
MSGID MG
22.17
K
Target
20.45 19.65
ST-GEN
3=AGCN
MS-G3D
CTR.GCN
97.46
19.00
58.39
19.53
43.75
98.24
60.35
20
98.05
41.21
59.57
18.36
45.72
97.46
39.25
56.45
19.34
43.15
98.07
42.01
60.57
19:73
20
99.29
42.55
64.60
20.33
49.41
Dateset
97.92
56.77
10
96.88
41.15
63.80
16.93
45.05
Figure 5: The ablation experiments of
motion gradient. MM MG' means whether using motion gradient in TASAR. Temporal Motion Gradient. TASAR benefits from the interplay between temporal Motion Gradient (MG) Bayesian manner. We hence conduct ablation studies(MG/No MG) show the effects of motion gradient and report the results in Figure 5. Compared with TASAR without using motion gradient, TASAR with motion gradient consistently improves the attack success rate in both white box and transfer-based attacks, which shows the benefit of integrating the motion gradient into the Bayesian formulation.
20
97.14
39.84
60.94
20.57
45.21
4.5 SURROGATE TRANSFERABILITY
is widely believed that transfer-based attacks S-HAR are highly sensitive the surrogate choice (I.u et al., 2023; Wang et al., 2023; 2021). In this subsection, we provide a detailed analysis of the factors contributing to this phenomenon. When looking at the results in Table 1 and the visualization of loss landscape in Figure 2 and Appendix C, note that loss surface smoothness correlates with the adversarial transferability. For example, CTR-GCN, manifesting smoother regions within loss landscape. demonstrates higher transferability than ST-GCN and STTFormer. STTFormer trained on NTU 120 has a smoother loss surface than ST-GCN (sec Appendix C), resulting in higher transfcrability than ST-GCN, For NTU 60, STTFormer shows similar loss surface to that of ST-GCN and exhibits comparable transferability. Therefore, suspect that the loss surface smoothness plays pivotal role in boosting adversarial transferability for S-HAR, polentially outweighing the significance of gradient-based optimization techniques. Next, two-stream MS-G3D shows the highest transfer- ability. Unlike other surrogates, which solely extract joint information, MS-G3D uses a two-stream ensemble incorporating both joint and bone features, thereby effectively capturing relative joint movements. In conclu- sion, suggest that skeletal transfer-basee attacks employ smoother two-stream surrogates incorporating both joint bone information.
5 CONCLUSION
In this paper, we systematically investigate the adversarial transferability for S-HARs from the view of loss landscape. and propose the first transfer-based based attack on skeletal action recognition, TASAR. build Ro- bustBenchHHAR the first comprehensive benchmark for robustness evaluation in S-HAR. We hope that Robust- BenchHAR could contribute to the adversarial learning and S- S-HAR community by facilitating researchers to casily compare new methods with existing ones and inspiring new research from the thorough analysis of the comprehensive evaluations.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENT
This project has received funding from National Natural Science Foundation of China (No. 62302139, No. 62406320), FRECU-HFUT (IZ2023HGAA0202, J22023 HGQA0101).
REFERENCES
Ayman Ali, Ekkasit Pinyoanuntapong, Pu Wang, and Mohsen Dorodchi. Skeleton- based human action recog- nition via convolutional neural networks (cnn). arXiv preprint arXiv:2301.13360 2023. 3
David Blei, Alp Kucukelbir, and Jon McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 1 1188518) 2017. 4
Charles Blundell, Julien Cornehise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015. 2,
Laura F Bringmann, Ellen Hamaker, Daniel Vigo, André Aubert, Denny Borsboom, Francis Tuer- linckx. Changing dynamics: time-varying autoregressive models using generalized additive modeling. Psy- chological methods, 22(3):409, 2017."
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel-wise topology refinement graph convolution for skeleton-basee action recognition. In Proceedings of the IEEE/CVF inter- national conference computer vision, pp. 13359-13368, 2021. 3, 4, 8, 20
Yunfeng Diao, Tianjia Shao, Yong-I iiang Yang. Kun Zhou, and Hc Wang. Basar:black-bo attack on skeletal action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR). pp. 7597-7607, June 2021. 3,
Yunfeng Diao, He Wang, Tianjia Shao, Yongliang Yang. Kun Zhou, David Hogg, and Meng Wang. Understand- ing the vulnerability of Akeleton-base human activity recognition via black-box attack. Pattern Recognition, 153:110564 2024a. 1,3
Yunfeng Diao, Naixin Zhai, Changtao Miao, Xun Yang, and Meng Wang. Vulnerabilities in ai- generated image detection: The challenge of adversarial attacks. arXiv preprint arXivv2407.20886 2024b. 2
Jeonghyeok Do and Munchurl Kim. Skateformer: Skeletal-temporal transformer for human action recognition.
In European Conference Computer Vision (ECCV), 2024. 8,
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su. Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting ad- versarial attacks with momentum, In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). June 2018. 1. 3, 4, 8, 2.21
Yong Du, Wei Wang. and Liang Wang. Hierarchical recurrent neural network for skeleton based action recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1110 1118, 2015. 3
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. Proceedings of the Thirty- Third Conference on Uncertainty in Artificial Intelligence, UAI, 2017.
Pierre Foret, Ariel Kleiner. Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for effi- ciently improving generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: : wwpp//peeeeveeeeeeeeeeeeeeee / forum?id d- 6 61m1mpssl1..
Guillermo Garcia-Coho and Juan C SanMiguel. Human skeletons and change detection for efficient violence detection in surveillance videos. Computer Vision and Image Understonding, 233: 233:103339, 2023.
Zhijin Ge, Hongying Liu, Wang Xiaosen, Fanhua Shang, and Yuanyuan Liu. Boosting adversarial transferabil- ity by achieving flat local maxima. Advances in Neural Information Processing Systems, 36:70141-70161, 2023. 3
Ian Goodfellow. Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair. Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 4a. 3,20
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamessing adversarial examples. arXiv preprint arXiv:1412.6677 2014b. 6
11

---

Published as a conference paper at ICLR 2025
Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Efficient and transferable adversarial examples from bayesian neural networks. Uncertainty in Artificial Intelligence, pp. 738-748. PMLR, 2022.
Xiaofeng Guo, Qing Zhu, Yaonan Wang, and Yang Mo. Mg-get: A motion- guidedd graph convolutional trans- former for traffic gesture recognition. IEEE Transactions on Intelligent Transportation Systems, 2024. 1,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hao Huang, Ziyan Chen, Huanran Chen, Yongtao Wang, and Kevin Zhang. T-sea: Transfer-based self- ensemble attack on object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 00514-20524 June 2023. 2
Pavel Izmailov, Dmitri Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Averaging weights leads wider optima and better generalization. In 34th Conference Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 876-885, 2018. 15
Gaojie Jin, Xinping Yi, Dengyu Wu, Ronghui Mu, and Xiaowei Huang. Randomized adversarial training via taylor expansion. In Proceedings of the IEEFJCVF Conference on Computer Vision and Pattern Recognition, pp. 16447-16457, 2023. 6
Zi Kang, Hui Xia, Rui Zhang, Shuliang Jiang, Xiaolong Shi, and Zuming Zhang. Fgda-gs: Fast guided decision attack based on gradient signs for skeletal action recognition. Computers & Security, 135:103522, 2023a.
Zi Kang, Hui Xia, Rui Zhang, Shuliang Jiang, Xiaolong Shi, Zuming Zhang, Fgda-gs: Fast guided decision attack based on gradient signs for skeletal action recognition. Computers & Security, 135:103522 2023b.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. CoRR, abs/1609.02907, 2016. URL htttp://a /ttx/aaivvv./////////////aa org/ abs 109. 02907. 3
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 4
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pp. 99-112. Chapman and Hall/CRC, 2018. 8, 20
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Making substitute models more bayesian can enhance transferability of adversarial examples. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali. Rwanda, May 1-5, 2023, 2023, URL https:tttt//ttttcttttttttttttttt /tttps://ppeneeveeww wttp://wwwweeeeettttteeee.ae 15,
bjpPypblire. 2,
Jian Liu, Naveed Akhtar, and Ajmal Mian. Adversarial attack on skeleton-based human action recognition. IEEE Transactions on Neural Networks and Learning Systems, 33(4):1609-160 1622, 2020a. 1, 2, 3, 8, 20
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. Nuu rgh+ 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence, (2) 2701, 2019. 4, 19
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, 2016.
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 143-152, 2020b. 2, 2,
Zhengzhi Lu, He Wang. Ziyi Chang, Guoan Yang, and Hubert P. H. Shum. Hard no-box adversarial attack on skeleton-basse basod human action recognition with skeleton doleeetonmotioniinnee motion-informed gradient. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4597-4606, October 2023. 2, 3, 4, 7, 8, 10, 15
Wenshuo Ma. Yidong Li, Xiaofeng Jia, and Wei Xu. Transferable adversarial attack for both vision trans- formers and convolutional networks via momentum integrated gradients. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 4630-4639, 2023. 3, 21
12

---

Published as a conference paper at ICLR 2025
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson A simple bascline for bayesian uncertainty in deep learning. Advances neural information processing systems, 32, 2019. 2, 4, 15, 21
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706. 06083, 2017. 7
Mcinard Müller, Tido Röder, Michael Clausen, Bernhard Eberhardt, Bjöm Krüger, and Andreas Weber. Mocap database hdm05. Institut für Informatik H. Universität Bonn, 2(7), 2007. 7, 19
Van-Anh Nguyen, Tung-Longg Vuong, Hoang Phan, Thanh-Toan Do, Dinh Phung, and Trung Le. Flat seeking bayesian neural networks. Advances in Neural Information Processing Systems, 36, 2024. 2,
Zeyu Qin, Yanbo Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu. Boosting the transferability of adversarial attacks with reverse adversarial perturbation. Advances in Neural Information Processing Systems, 35:29845-29858, 2022. 4
Helei Qiu, Biao Hou, Bo Ren, and Xiaohua Zhang. Spatio-temporal tuples transformer for skeleton- based action recognition. CoRR, abs/2201.0222 02844, 2022. URL wwt.pppp//aaaae.. wwt.://gg////aaeeeo abaa///s/2221 02849. 3, 8, 20
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Nuu rgh+ d: large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1010-1019, 2016. 4, 7, 19
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton-basee based action recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12026 12035, 2019a. 3, 2.
Lei Shi, Yifan Zhang, Jian Cheng, and Hanging Lu. Skeleton-based action recognition with directed graph neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7912-7921, 2019b. 3, 8, 20
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. Advances in neural information processing systems, 29, 2016. 15, 21
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint urXiv:1312.6199 2013. 3
Nariki Tanaka, Hiroshi Kera, and Kazuhiko Kawamoto. Adversarial bone length attack on action recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 2335 2343, 2022. 1.3
Nariki Tanaka, Hiroshi Kera, and Kazuhiko Kawamoto. Fourier analysis on robustness of graph convolutional neural networks for skeleton-based action recognition. Computer Vision and Image Understanding, 240: 103936, 2024. 3
Bowen Tang, Zheng Wang, Yi Bin, Qi Dou, Yang Yang, and Heng Tao Shen. Ensemble diversity facilitates adversarial transfcrability. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 24377-24386, 2024. 3
Xiangjun Tang, He Wang, Bo Hu, Xu Gong, Ruifan Yi, Qilong Kou, and Xiaogang Jin. Real-time controllable motion transition for characters. ACM Transactions on Graphics (TOG), 41(4):1-10, 2002.002.77 7
He Wang and Yunfeng Diao. Post-train black-box defense via hayesian boundary correction. arXiv preprint arXiv:2306. 16979, 2023. 15
He Wang. Feixiang He, Zhexi Peng, Tianjia Shao, Yong-Liang Yang, Kun Zhou, and David Hogg- Under- standing the robustness of skeleton- based action recognition under adversarial attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14656 14665, 2021. 1,2, 3, 8, 10, 25.20
He Wang. Yunfeng Diao, Zichang Tan, and Guodong Guo. Defending black-box skeleton-based human activity classifiers. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty-Seventh AAAI Conference on Artificial Intelligence, pp. 2546-2554, 2023. 2, 3. 4, 7, 8, 9, 10, 15, 20
Kunyu Wang, Xuanran He, Wenxuan Wang, and Xiaosen Wang. Boosting adversarial transferabiliiy by block shuffle and rotation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 44336-24346 2024. 3
13

---

Published as a conference paper at ICLR 2025
Tiancai Wang. Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1121-122,, 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of 28th international conference on machine learning (ICML-I1), pp. 681-688. Citeseer, 2011. 4
Lei Wu and Zhanxing Zhu. Towards understanding and improving the transferability of adversarial examples in deep neural networks. In Asian Conference on Machine Learning, pp. 837-850. PMLR, 2020. 2, 4
Shihong Xia, Congyi Wang. Jinxiang Chai, and Jessica Hodgins. Realtime style transfer for unlabeled hetero- geneous human motion. ACM Transactions Graphies (TOG), 34(4): 10, 2015.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan I. Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2730 2739, 2019. 3, 81
Yifeng Xiong, Jiadong Lin, Min Zhang, John E Hopcroft, and Kun Hc. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14992, 2022. 4, 8, 21
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-bart basdd action recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 8. 3, 4, 20
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled tradc-off between robustness and accuracy. In International conference machine learning, pp. 7472-7482 PMLR, 2019a. 10, 20
Pengfei Zhang, Cuiling Lan, Wenjun Zeng, Jianru Xue, and Nanning Zheng. Semantics-guided neural networks for efficient skeleton-basee human action recognition. CoRR, abs/1904.01 189, 2019b. URL http://
arxiv orgg abs/ / 1904. 01189.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference Machine Learning, pp. 26982-26992. PMIR, 2022. 2
Huanyu Zhou, Qingjie Liu, and Yunhong Wang. Learning discriminative representations for skeleton based ac- tion recognition. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1668-10617,, June 2023. 3, 8,8,,2
Rongyi Zhu, Zeliang Zhang, Susan Liang, Zhuo Liu, and Chenliang Xu. Learning to transform dynamically for better adversarial transferability. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24223-2222222 24282,2222222 2024.
A ETHICS AND REPRODUCIBILITY STATEMENT
Ethics Statement. TASAR is capable of generating natural-looking adversarial examples in S-HAR that transfer across different skeletal classifiers. We acknowledge the possibility that TASAR might pose signif- icant practical threat to the current S-HAR models. However, we believe that order to build a reliable and robust action recognizer, is of great necessity to investigate their vulnerability. Therefore, this paper can raise awareness of vulnerability in existing S-HAR models, which greatly outweighs its risk. TASAR can be employed evaluate the robustness of skeletal classifiers in real-world applications improve their robustness through adversarial training.
Reproducibility Statement. To ensure the reproducibility of our work, we have included a comprehensive Reproducibility Statement. For the datasets used in our experiments, all the datasets used in this paper are open dataset and are available to the public. We have provided thorough description of the data processing sleps in the supplementary materials. For the novel model and algorithms presented in this work, we have included a link to the downloadable source code and model checkpoint approach. The source code and model checkpoint can also be found in the supplementary materials. Addition- ally. all inference details and mathematical deduction can be found in the Appendix B. This Reproducibility Statement is intended to guide readers to the relevant resources that will aid in replicating our work, ensuring transparency and clarity throughout.
use our proposed benchmark and build our
14

---

Published as a conference paper at ICLR 2025
B INFERENCE DETAILS
The detailed inference process for Post-train Dual Bayesian Motion Attack is outlined in Algorithm
Post-train Dual Bayesian Optimization. The confidence region of the Gaussian posterior in Equa- tion regulated by &. As A0' is sampled from zero-mean isotropic Gaussian distribution, the inner maxi- mization can solved analytically:
AO de,o V (x, v. v..,0) /Vv (x.
(15)
Then the gradient of Vog L (x, y, 8, 0%) A Equation (8) becomes Vo, L (x, v, 0, + HAØ, which HAC, can be approximately estimated finite difference method:
HAO ~
1
V
1 K
+ +20)) Vo, 1 K (x.
(x.xx00006)
(16)
where y a small positive constant. Therefore, our final optimization objective is:
k Vo, (x, + (1/7) (ve, L (x. + -Ve, ov (x.,00000000)))
(17)
Followed by Wang & Diao (2023), use Stochastic Gradient Adaptive Hamiltonian Monte Carlo (SGAHMC) (Springenherg
2016) for the post-train dual Bayesian optimization. Our dual Bayesian optimization assume Ø' is sampled from Gaussian posterior, which has a presumed isotropic covariance matrix. In practice, we follow the suggestions from Li et al. (2023) to calculate the mean and the covariance from data by using SWAG (Maddox et al., 2019). as SWAG can offer improved approximation to the posterior over parameters. While the posterior still relies Gaussian approximation, specifically incorporates the SWA(lzmailov al., 2018) 8) solution its mean, and decomposes the covariance into low-rank matrix and diagonal matrix:
0% ~ (Bk,SWA, ESWAG)
ESWAG = = (Edug + Eaow-rank) )
(18)
where 1 (2 0) can be set to other coefficients, which represent the scaling factor of SWAG for disassociating the learning rate of the covariance. Note that the posterior discussed in the preceding section is formulated based on the worst cases, thus facilitating its effortless integration with SWAG expand diversity and flexibility. Since Og.SWA is unknown before training terminates, the dispersion of 0% the final Bayesian model originates from the combination of two distinct independent Gaussian distrihutions, with their covariance matrices aggregated. Consequently Equation (18) becomes:
0% .V (OL,SWA, Eot) Egg (Eeug Etow-rank ) 31,
(19)
where | controls the covariance matrix of the isotropic Gaussian distribution mentioned before.
C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
C.I ADDITIONAL LANDSCAPE VISUALIZATIONS
To explore the sensitivity of transfer-based attacks surrogate models (Lu al., 2023; Wang al., 2023; 2021), we present in Figure and Figure the loss landscapes different surrogate models trained on NTU60 and NTU120. It evident that both the post-train Bayesian optimization(PB) and the improved post-train Dual Bayesian pptimization(P-DD can smooth the loss landscape: Notably, surrogates refined by P-DB display smoother loss landscapes, leading superior transferability over normally trained models and those optimized by PB.
15

---

Published as a conference paper at ICLR 2025
Algorithm 1: Inference for Post-train Dual Bayesian Motion Attack
Input: x: training data; -rost-trainn the number of post-train iterations; duut the number of Post-train Dual Bayesian optimization iterations; My: sampling iterations for 8%;3 c: moment update frequency; Kithe number appended models;/ :attack iterations; 0: pre-trained surrogate weights;
Output: {8] + Aff 8x + x0 appended surrogate weights; x: adversarial
examples;
// Post-train Bayesian Init: randomly initialize (0'.-...,00 k)
for == to N ppost-train do
for k= K do
Randomly sample a mini-batch data (x,y):
6% - 6 VV, (x, y, 0 1%));
for 1= MM do
Update 0g via SGAHMC;
end
11
end
12 end
// Post-train Dual Bayesian Optimization
each B% < 0 86 6 13
14
to Nduat do
15
for k= to K do
16
Randomly sample a mini-batch data (x, y) 8% - _% -/V L (x, y, 0, (,))) Compute A0', via Equation (15); Solving outer min optimization in Equation (8) via Equation (17); if MOD(i, c) = then
18
20
21
n i/c:
no 02
nO
2 +0
23
8
n+1
x+
23
end
24
end
end
26
&kSWA = 03 // Attack
Ediag 8% 8x
27 Initialization:xx... = x;
obtain the time-varying parameters with TV-AR;
29 for i = to 1-1 do
models=[] for == to K do
31
for m = to M do
Draw 0% A0' kmm in Equation (19);
models.append 6 + Alk.):
33
34
end
35
end
36
Obtain the ensemble gradient;
OL(x') Ba__,
and OL(x') 0z
Calculate the motion gradient
with Equation (13) and
Equation (14); Update x via Equation (10);
end
40
return (0) A0...... .......,.1. 8xx A0
16

---

Published as a conference paper at ICLR 2025
(TR-GCN(PB)
CTR-GCN(P
ST-GCN
ST-GCN(PB)
ST-GIN((-DDB)
STTFomer
(I)))))))(P
STTF DDB)
Figure 6: Loss landscapes of trained models with different methods on NTU60. The loss landscape in each plot generated from the original data randomly selected from the test dataset of NTU60.PB means the post-train Bayesian optimization, P-DB means the improved post-train Dual Bayesian optimization. The first row, second row and third represent the loss surface of CTR- GCN, ST-GCN and STTFormer trained normally, as well as PB and P-DB, respectively, with the axis ranging from 0 to 16.
CTR-GCN
CTR-GCN(PB)
CTR-GCN(P-DB)
ST-GCN
ST:GCN(PB)
ST-CU((-DB)
STTFomee
STTForeer
STIFomee
Figure 7: Loss landscapes of trained models with different methods on NTU120. loss landscape in each plot are generated from the same original data randomly selected from the test dataset of NTU120. PB means the post-train Bayesian optimization, P-DB means the improved post-train Dual Bayesian optimization. The first row, second row, and third row correspond to the loss surfaces of CTR-GCN, ST-GCN, and SoTFrrmer under normal training. PB and P-DB, respectively. For the normally trained plots, the axis ranges from 30, while for PB and P-DB, the range is from to 20.
17

---

Published as a conference paper at ICLR 2025
C.2 ADDITIONAL RESULTS
The Sensitivity to Ç. our Post-train Dual Bayesian optimization, consider the worst-case parameters from posterior. The confidence region of the Gaussian posterior is regulated by p(A0') 2 &. influencing extent of exploration within the posterior distribution. Therefore, we investigate the relationship between the sensitivity of & and the performance of TASAR. Based our assumption of an isotropic Gaussian distribution, got the analytical solution of the worst
below:
Adl Ae.o o I (x. 000 / (x.
(20)
original motion (Walk)
adversarial example (Cartwheell)
The ground truth label 'Walk' can be misclassified 'Cartwheel' on targeted attack by TASAR. The semantic differences between ground truth labels and target labels are large.
Figure 8:
Equation (20), & can be reparameterized as a hyper-paramest λç.o. Consoquently, we conduct ablation studics to investigate the relationship between the performance of our method and the sensitivity of the hyper- parameter λç.r. We varied with λεo E {0.001, 0.01, 0.05, 0.1, 1, 1.5, 2} on NTU120 dataset and show the success rates of attacking victim models. We choose ST-GCN shown in Table 5. We found that setting small value of λç,r achieves the best adversarial transferability while maintaining a high benign accuracy. Hence we sample a collection of new surrogates near to the original surrogates and set A 0.001 as default.
pre-trained model and the results are
Table 5: Comparsions attack success rate(%) with different Aç. The surrogate model uniformly selected as ST-GCN on NTU120.
Target
Ag.o
Accuracy
ST-GCN
2s-AGCN
M3D
CTR-GCN
FR-HEAD
0.001
99.26
19.60
19.37
15.28
22.79
63.60
0.01
95.88
17.62
13.10
14.29
20.23
64.30
0.05
96.43
17.26
13.10
14.88
21.43
63.60
0.1
96.43
17.62
13.10
13.69
21.43
60.56
1
96.43
18.45
11.90
16.07
23.21
60.39
1.5
96.43
18.45
13.10
15.48
23.81
56.58
2
97.02
17.26
13.10
16.67
22.02
54.61
Perturbation Budget In this section, we analyze the impact of attack strength on adversarial transferabil- ity. We increase the perturbation budget from 0.01 to 0.05, the results are shown in Figure 9. The general pattern of the lines aligns with the threshold setting. indicating that larger perturbation budget consistently enhances adversarial transferability across various surrogate models.
The Visual Quality of Targeted Adversarial Examples TASAR can successfully attack the original class to target with an obvious semantic gap without being detected by humans. show an additional visual example in Figure 8.
Comparsion with Different Training Strategies. We conducted additional experiments to compare the performance of modeling (0' D.O), p(0 D) and p (0,0) D). The default setting of TASAR corresponds (0' D. 0). where 0' is trainable while 0 remains fixed. p(0 D) represents standard Bayesian neural network without adding appended models. For this case, we train the Bayesian surrogates with cyclical variant of Stochastic Gradient Markov Chain Monte Carlo[1] to sample 3 models from the posterior distribution of neural network weights. For p00,6 o² D). appended models are added behind BNNs and both 0 and 0' are trainable. Due to the high optimization complexity(updating both @ and 8'), use vanilla post-train Bayeian optimization instead of improved post-train dual Bayesian optimization optimization for updating 6'.
18

---

Published as a conference paper at ICLR 2025
PDBA
100
90
80
70
60
50
40
25-AGCN
MSG-3D
30
HRHEAD
0.010
0.015
0.020
0.025
0.030
0.035
0.040
0.045
0.050
Figure 9: The success rate with different perturbation size. The surrogate model is uniformly chosen as MSG-3D and the dataset is NTU60.
Table 6: The attack success rate(%) of untargeted attacks on NTU60.
NTU60
Surrogate
Method
ST-GCN
2s-AGCN
MS-G3D
CTR-GCN
PR-HRAD
SFormer
((011,,) (TASAR)
99.29
42.55
64.60
20.33
49.41
17.22
ST-GCN
p(0 D)
90.63
37.11
69.53
17.58
43.36
25.39
p00 (D)
93.75
41.80
73.04
18.75
46.87
22.65
The results are presented in Table Compared to modeling p(0 D) and p(0, (') TASAR achieves supe- rior performance in transfer attacks on three out of five black-box models and demonstrates the best white-box attack performance ST-GCN. This advantage arises from the of improved post-training Dual Bayesian optimization, which enables smoothed posterior sampling and improves adversarial transfcrability. Moreover, unlike modeling p(0 D) and p(0, D') where averaging gradients
multiple surrogales diminish white- box attack effectiveness, our post-training strategy preserves the pre-trained model intact, not reducing the original white-box attack strength. Further, TASAR significantly reducing computational overhead and accel- crating training process, shown in Table 7.
the changes in loss perturbed along random direction with varying magnitudes. Specifically, we first sample d from Gaussian distribution and normalize it onto the la unit norm ball as d <- ||]]e"" Then, we calculate the loss change (smoothness) (a) for different magnitudes a:
Quantification of Model Smoothness. We quantitatively
f(a) |T(x -. v. T(xx, 0))
(21)
We calculate f(a) times with different randomly sampled d. and take the average results. For fair com- parison, we use the same cach sampling in both 'NT' and 'TASAR'. The experimental results reveal that TASAR achieves a significantly smoother loss landscape compared to Normally Train (NT) across all magni- tudes of perturbation. For large perturbations ((a) 1.0), TASAR reduces the loss change by approximately threefold compared to NT. Additionally, TASAR maintains
uniform smoothness across different mag- nitudes, while exhibits sharper variations, with larger loss changes. This indicates that TASAR effectively smoothens the loss landscape, contributing to improved transferability.
D DETAILED RobustBenchHAR SETTINGS
Here we report the detailed experimental settings in our experiments. All experiments are conducted on one
NVIDIA GeForce RTX 3090 GPU.
(A) Datasets. We choose three widely adopted datasets: NTU60 (Shahroudy al., 2016). NTU120 (Liu al., 2019) and HDMOS(Müller al., 2007). The HDM05 dataset comprises 130 action classes and includes
19

---

Published as a conference paper at ICLR 2025
Table Model Size and Training Time measurement on NTU60, 'MS' means The Model Size(M) and "TT" means The Training Time(hours).
NTU60
Surrogate
Method
MS
TT
p(6'D,0 8)
3.54
0.65
ST-GCN
D)
9.30
5.7
p(8,6'
9.36
6.4
Table 8: Loss changes (f()) measurement for normally trained Surrogate and TASAR on HDM05.
"NT" means "Normally Training".
Magnitude
Surrogate
Method
-0.8
-0.6
-0.4
0.4
0.6
0.8
1.0
NT
7,46
6.05
4.34
2.08
2.06
4.33
5.91
7.24
ST-GCN
TASAR
2.66
1.70
1.03
1.77
2.27
1.40
1.65
2.31
2337 sequences performed by 5 con-professionl actors. NTU60 offers 60 action classes, it comprises 56,880 vidcos captured from 40 subjects across 155 camera viewpoints. NTU120 extends NTU60 with 120 action classes, it contains 11 114,480 vidcos from 106 subjects across 155 camera viewpoints. Due to variations in data pre-processing settings among S-HAR classifiers (such as data requiring Daaaammling(Zangg et al., 2019b)), we unify the data format following Wang et al. (2023). For NTU60 and NTU120, we subsample frames to 60. For HDM05, we segment the data into 60-frame samples(Diao al., 2021).
(B) Evaluated Models. We evaluate TASAR in three categories of surrogate/ victim models.
(1) Normally trained models: We adapt commonly used GCN-based models, i... ST-GCN(Yaa et al., 2018), 2S- ASCN(Sii et al., 2019a), CTR-CTRR--CC GCN(CCen et al., 2021), MS- G3D(Liu ct al., 2020b), FR- HEAD(Zhou et al., 2023) and Transformer-based models, i.c., STTFormer Qiu et al., 2022) and SkateFormer(Do & Kim, 2024). Below introduce the seven skeletal classifiers details. ST-GCN (Yan al., 2018) 8) is the first time to apply graph convolution S-HAR, constructing nodes and edges in the topology using skeletal information. CTR- GCN (Chen et al., 2021) improves the design GCNs ST-GCN and proposes to dynamically learn different topologies by learning& a shared topology as a common prior for all channels and refining it for each channel. 2s-AGCN (Shi et al., 2019a) enables the model to learn graph topologies end-te no-eto-en through self-attention. It also incorporates dual-stream framework model first-order and second-order information simultaneously. MS-G3D (Liu et al., 2020b) proposes a disentangled multi-scale aggregation scheme to eliminate redundant dependeneies between node features from different neighborhoods, therehy capturing g global joint relationships on human skeletons. FR-HEAD (Zhou et al., 2023) applics contrastive feature refinement at various stages of GCNs to build multi-level feature extraction. This allows ambiguous samples to be dynamically discovered and calibrated the feature space. STTFormer (Qiu al., 2022) divides the skeleton sequence into temporal tuples to capture the relationships between different joints in consecutive and non-adjacent frames. SkateFormer (Do & Kim, 2024) classifies essential skeletal-temporal relationships for S-HAR into four distinct categories and utilizes olf-attention in each partition to focus on key joints and frames crucial for recognition. To the best of knowledge, this is the first work to investigate the robustness of Transformer based S S-HARs.
(2) Ensemble model: An ensemble of ST-CGN, MS-G3D and DGNN (Shi al., 2019b).
(3) Defense models: We employ BEAT (Wang al., 2023) and TRADES (Zhang et al., 2019a). which all demonstrate their robustness for skeletal classifiers. BEAT (Wang ct al., 2023) proposes a black-bos defense framework, which transforms any pre-trained classifier into a more robust one by maximizing the joint proba- bility of clean data, adversarial examples and the classifier through joint Bayesian treatments. TRADES (Zhang al., 2019a) is white-box defense method that introduces -L-divergence loss function for adversarial train- ing. TRADES not only accounts for natural error but also incorporates adversarial error, balancing robustness and accuracy.
treatments.RRDDSS
(C) Baselines. Unlike images or videos, the space available for attacking skeletons is much smaller, mak- ing adversarial perturbations on skeletons more easily detectable, here choose two state-of-the-art attacks against S-HAR: (1) SMART (Wang et al., 2021) ensures the imperceptibility of the attack by introducing ad- versarial attack perceptual loss function for S-HAR. (2) CIASA (Liu et al., 2020a)maintain))) the spatio-temporal constraints of joint connections and skeletal bone lengths through spatial skeleton realignment and further en- sures the anthropomorphic plausibility by utilizing GAN(Goodfellow et al., 2014a) for regularization.
Besides the attacks specifically designed for S-HAR, we also select general transfer-based attacks as haselines, these attacks include (1) Gradient-based attacks, such I-FGSM (Kurakin al., 2018), an iterative fast gra-
20

---

Published as a conference paper at ICLR 2025
dient method; MIFGSM (Dong al., 2018), which integrates momentum into I-FGSM to stabilize the update direction and prevent getting stuck poor local maxima; and the latest MIG (Ma al., 2023), which uses integrated gradients to steer the generation of adversarial perturbations and adjusts them according to the mo- mentum updating strategy. (2) Input transformation attacks, such as DIM (Xic et al., 2019), which improves the transferability of adversarial examples by creating diverse input patterns. (3) including ENS (Dong al., 2018). which attacks multiple models with fused logit activations; SVRE (Xiong al., 2022), which escapes poor local optima by computing an unbiased gradient estimate through variance reduction for each iteration; and BA (Li al., 2023), which fine-tunes the weights of the surrogate model in a Bayesian manner, thereby creating an ensemble of infinitely many DNNs as surrogates.
attacks,
For fair comparison, we run 200 iterations for all attacks under loo norm-bounded perturhation of size 0.01. For TASAR, use the iterative gradient attack instead of FGSM.
(D) Implementation Details or TASAR. Our appended model a simple two-layer fully fully-connected layer network. Unless specified otherwise, we use K 3 and M 20 in TASAR for default and explain the reason in the ablation study later.
During the post-train, we set a learning rate of 0.03 with five epochs. We use SGA/IMC optimizers (Springen- berg al., 2016), within itr automatically chosen, the friction coefficient = 10 and Mg 30 steps for sampling.
During dual Bayesian optimization, we set = 0.1/ || llz and perform training for epochs with learning rate of 0.03. Additionally, we always set λe, = 0.001.
During Inference, SWAG adjusts the covariance matrix using a constant multiplier to decouple the learning rate from covariance(Maddox et al., 2019). Here we always use 1.5 the rescaling factor. When performing attacks, we set 0.009 for models with SWAG. The w1.w2 wa are as 0.8. 0.1 and 0.1.
(E) Computing Resource. The experimental platform used in this study equipped with AMD EPYC 7542 32-Core CPU operating at a clock speed of 2039.81 8139 GHz, four NVIDIA GeForce RTX 3090 GPUs, and 24 GB of memory GPU. The proposed method was implemented using the open-source machine learning framework PyTorch.
21


Reviews:
[{"review_id": "1wr6gFOJQ8", "reviewer": "Reviewer_NGYv", "summary": "This paper studies transfer-based attacks for skeleton-based action recognition. A last-layer Bayesian approches is applied to smooth the loss surface to improve adversarial transferability. The authors setup a benchmark for transfer-based skeleton-based action recognition attacks. The proposed method is evaluted using different backbone approaches on NTU-60, NTU-120, and HDM05. The results demonstrate the effectiveness of the proposed approaches.", "questions": "1) In Table 4, why K=3 yields better results than K=5? I thought more samples lead to better approximation", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1) The paper is well-written with clear motivations of each step\n2) The proposed approch is easy to apply on existing methods since only a few layers are needed at the end of the networ.\n3) Thorough experiments and ablation studies", "weaknesses": "1) As the pretrained models are frozen, modeling p(\\theta’|D, \\theta) is not as good as modeling p(\\theta, \\theta’|D). If trainable, it would be better to justify the performance difference on a small subset of data.\n2) Besides the visualization of loss surfaces, quantitative evaluation of the surface smoothess can be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ntqV95WNuz", "reviewer": "Reviewer_BzGe", "summary": "For the transfer-based attack on skeletal action recognition tasks, this paper builds the first comprehensive benchmark, comprising 7 skeletal models, 10 attack models, 3 skeletal datasets, and 2 defense methods. Additionally, a novel post-train Dual Bayesian Motion attack is presented, aiming to explore a smoother model posterior by incorporating a small MLP layer. Unlike image data, the spatial-temporal characteristics, represented by first-order velocity and second-order acceleration, are considered.", "questions": "Please see in Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments.\n2. The post-train Dual Bayesian Motion attack method is reasonable in general.", "weaknesses": "1. Although it is reasonable to assume that a smoother loss landscape benefits transfer attacks, exploring the reasons for poor transferability in S-HAR tasks and comparing the differences among surrogate models may reveal various factors. Therefore, directly concluding that smoothing the loss landscape is the sole solution may not be entirely logical.\n\n2. In this method, an MLP is primarily added after the original model. However, the consideration of first-order velocity and second-order acceleration to capture spatial-temporal information, as mentioned in this paper, has already been addressed by SMART (Wang et al., 2021), which limits the novelty of this approach.\n\n3. The Bayesian optimization approach used in this method is similar to BEAT (Wang et al., 2023), and the differences between the two approaches should be discussed further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xULJOfuySu", "reviewer": "Reviewer_LgjH", "summary": "This paper explores an interesting research problem in skeleton-based action recognition.\n\nThis paper proposes the transfer-based attack on skeletal action recognition.\n\nThe proposed model incorporates motion dynamics into the Bayesian attack gradient, effectively disrupting the spatial-temporal coherence of the skeletal action recognition.\n\nThe authors explore and analyse several model architectures, and conduct experimental analysis and evaluations on several benchmarks and show that the proposed model achieves good performance compared to existing baselines.", "questions": "Please refer to my main comments above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ Overall this paper explores an interesting research problem.\n\n+ The authors conduct large-scale evaluations to show that the proposed model achieves good performance on several benchmarks.\n\n+ Some of the figures presented in the paper are quite nice and provide some insights to readers.", "weaknesses": "Major: \n\n- One of the major issues of this paper is that some of the concepts presented in the paper are not properly and clearly explained. For example, in abstract section, what is \"white-box scenarios\". Also what does it mean by saying \"its sharpness contributes to the weak transferability in S-HAR\"?\n\n- Fig. 1 is not being introduced and explained clearly. For example, what is spatial attack, spatial-temporal attack, why frozen, trainable, being lightweight and demanding? What are appended models and dual bayesian? These concepts should be clearly explained to readers, either in texts or figure caption. Moreover, how to tell which one is better? \n\n- The introduction section is not very well-motivated. For example, in Line 83, referring to Table 1; however, the concepts to understand the table are not being explained. The authors are encouraged to reorganise the section, explain the concepts properly before going deep into the technical jargon. What is \"white-box\"? In contribution section, what is \"low adversarial transferability\"?\n\n- In related work section, the authors outline some closely related works; however, they do not highlight what are the major differences between existing works and the proposed method, what are the insights, how the proposed method has the potential to address the challenges etc.\n\n- Fig. 2, the font sizes are invisible to reviewer, what are the x and y axis represent for? What do \"random direction vectors\" mean? These concepts should be properly explained. \n\n- In Eq. (2), what are boldface $x$ and $\\tilde{x}$ mean?  In Line 158 and 183, it says \"a clean motion\" and \"the adversarial sample\", what do they mean and how they align with each other, what would be the relationship between them? These concepts and maths symbols are not properly explained. It is suggested to have a notation section detailing the maths symbols and operations used in the paper. Eq. (3) right hand side is a bit confusing, the reviewer cannot understand that, could you explain step by step?\n\n\nMinor:\n\n-  Remove the  full stop \".\" inside Sec. 3.4.1 and 3.4.2 headings.\n\n- Table of font sizes should be consistent in the whole paper. In the main paper, table 1, 2, 3 and 5 font sizes are too small, and hard to read; however, table 6 in appendix section, the font sizes are too big. The authors should make those representations consistent and nicer.\n\n- While the maths presented in the paper seem to be accurate, its presentations, especially in the method section, needs extensive improvement. The reviewer gets lost many times, and some of the concepts are not very well introduced and explained. A good paper should explain these concepts clearly and in a more intuitive way. \n\n- An excellent paper should use efficient figures and plots to show the experimental results. In this paper, most experimental results are presented in the form of tables, it would be better if the authors could provide some plots and visualisations to make the evaluations, comparisons and analysis more interesting to wide communities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7ErsQugECQ", "reviewer": "Reviewer_WYTN", "summary": "This paper investigates the issue of weak adversarial transferability in skeleton-based Human Activity Recognition (S-HAR) attacks. It proposes a new transfer-based attack called TASAR, which aims to improve transferability by smoothing the loss surface. The authors also establish a large-scale robust S-HAR benchmark for comprehensive evaluation.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper identifies a crucial limitation of existing S-HAR attacks and proposes a practical solution. The proposed TASAR attack is innovative and demonstrates potential improvements in adversarial transferability. The evaluation on the large-scale benchmark provides valuable insights into the effectiveness of existing and proposed methods.\n\nThe author has submitted the code, making the work reproducible.", "weaknesses": "I suggest that the authors cite more papers related to recent research from top conferences over the past two years.\n\nMore related literature can be included, for instance, by adding a topic on noise-robust human action recognition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)