Subspace Kernel Learning on Tensor Sequences
Authors: Lei Wang
Xi Ding
Yongsheng Gao
Piotr Koniusz
Venue: ICLR 2026 Poster (2026)
Keywords: Kernel
tensor
subspace learning
action recognition
Primary area: learning on time series and dynamical systems
TL;DR: We introduce Uncertainty-driven Kernel Tensor Learning (UKTL), a novel kernel framework for M-mode tensors that compares mode-wise subspaces derived from tensor unfoldings.

Abstract: Learning from structured multi-way data, represented as higher-order tensors, requires capturing complex interactions across tensor modes while remaining computationally efficient. We introduce Uncertainty-driven Kernel Tensor Learning (UKTL), a novel kernel framework for $M$-mode tensors that compares mode-wise subspaces derived from tensor unfoldings, enabling expressive and robust similarity measures. To handle large-scale tensor data, we propose a scalable Nystr\"{o}m kernel linearization with dynamically learned pivot tensors obtained via soft $k$-means clustering. A key innovation of UKTL is its uncertainty-aware subspace weighting, which adaptively down-weights unreliable mode components based on estimated confidence, improving robustness and interpretability in comparisons between input and pivot tensors. Our framework is fully end-to-end trainable and naturally incorporates both multi-way and multi-mode interactions through structured kernel compositions. Extensive evaluations on action recognition benchmarks (NTU-60, NTU-120, Kinetics-Skeleton) show that UKTL achieves state-of-the-art performance, superior generalization, and meaningful mode-wise insights. This work establishes a principled, scalable, and interpretable kernel learning paradigm for structured multi-way and multi-modal tensor sequences.

Published as a conference paper at ICLR 2026
SUBSPACE KERNEL LEARNING ON TENSOR SEQUENCES
Piotr Koniusz 3,2,1 1 Griffith University, Data6l University of New South Wales {1 . wang4, x. Riding yongsheng Baciangg.a@@@pppom gaofegriffith. edu. au, piotr. kcnnuszzuusst edu.au au
Lei Wang ..2 Xi Ding" Yongsheng Gao?!
ABSTRACT
Learning from structured multi-way data, represented as higher-order tensors, re- quires capturing complex interactions across tensor modes while remaining putationally efficient. We introduce Uncertainty-drive Kernel Tensor Learning (UKTL), a novel kernel framework for M-mode tensors that compares mode-wise subspaces derived from tensor unfoldings. enabling expressive and robust similar- ity measure. To handle large-scale tensor data, we propose a scalable Nyström kernel linearization with dynamically learned pivot tensors obtained via soft k- means clustering. key innovation of UKTL is its uncertainty-awar subspace weighting, which adaptively down-weights unreliable mode components based on estimated confidence, improving robustness and interpretability in comparisons between input and pivot tensors. Our framework is fully end-to-end trainable and naturally incorporates both multi-way and multi-mod interactions through struc- tured kernel compositions. Extensive evaluations action recognition bench- marks (NTU-60, NTU-120, Kinetics-Skeleton) show that UKTL achieves state- of-the-art performance, superior generalization, and meaningful mode-wise wise in- sights. This work establishes principled, scalable, and interpretable kernel learn- ing paradigm for structured multi-way and multi-modal tensor sequences.
INTRODUCTION
The rapid growth of high-dimensional, multi-modal data, from video streams to biomedical signals, demands learn- ing frameworks capable of capturing rich multi-way structures. Tensors, or multi-way arrays, naturally represent such data by preserving relationships across multiple modes such as space, time, and features (Kolda & Bader, 2009; Koniusz et al., 2013; Koniusz & Cherian, 2016; Koniusz et al., 2021; Wang et al., 2024b). Despite their ex- pressiveness, tensors introduce unique challenges for machine learning. Tradi- tional approaches often flatten tensors into vectors or matrices before applying standard techniques such as Principal Component Analysis (PCA). This sim- plification destroys the inherent struc- ture of tensors, leading to models that are inefficient or less expressive, especially when non-linear dependencies or sparse observations present.
0.84
-0.24
-0.38
0.67
20
20
3.0
0.500
Taster
0.00 0.0
coardratis
Figure 1: Mode-wise factor matrices from the Tucker de- composition for the action called "draw x". Each row shows one latent factor (from 4 leading factors), and each column corresponds to one tensor mode: temporal block, body joints, 3D coordinates, and time. Structured patterns reveal interpretable, mode-specific information which motivates our approach.
Kernel methods offer a powerful alternative by embedding data into high-dimensional Reproducing Kernel Hilbert Spaces (RKHS), enabling non-linear modeling through pairwise similarities without
Equul contribution.
Corresponding authors.

---

Published as a conference paper at ICLR 2026
explicit feature mappings. However, most kemeel methods rely vectorized imputs, limiting their ability to directly use tensors. Furthermore, their computational cost grows rapidly with dataset size. impeding large-scale or real-time applications. On the other hand, tensor decomposition meth- ods such as CP, Tucker, and Tensor-SVD provide effective low-rank representations that respect multi-way relations but fundamentally linear and do not easily integrate with non-linear learn- ing frameworks. Recent tensor subspace learning approaches seek to combine these advantages but often assume equal importance across tensor modes or resort to heuristic regularization.
In this work, we introduce a novel perspective: learning kernels over tensor subspaces with explicit modeling of uncertainty in each mode. Our key insight is that mode-wise unfolding of tensors re- veals interpretable subspaces that can be robustly compared in RKHS using kernels (see Fig. 1). Building on this, we propose (i) a product subspace kernel that captures interactions across modes more effectively than direct tensor comparisons, (ii) Nyström- kernel linearization with dy- namic pivot tensor selection via soft clustering for scalable training, and (iii) an uncertainty-ave regularization framework, grounded in likelihood maximization, that adaptively weights subspaces by their informativeness.
Our end-to-end framework learns non-linear, structured embeddings directly from tensor sequences, while maintaining computational efficiency. We validate our approach on challenging action recog- nition benchmarks, where it outperforms state-of-thee ofrtt---ttttttttte graph convolutional, hypergraph, and trans- former models, all with a simpler and more interpretable design. Our main contributions are:
i. We propose a principled kernel learning framework for tensor data that captures non-linear re- lationships through mode-wise subspace comparisons, avoiding destructive flattening or costly direct tensor matching.
ii We introduce a Nyström kernel lincarization with soft clustering to dynamically construct rep- resentative pivot tensors, enabling scalable training on large datasets.
iii. We develop an uncertainty-aware regularization that adaptively weights tensor subspaces based their reliability using maximum likelihood estimation.
2 RELATED WORK
Tensor decomposition and subspace learning. Tensor decomposition methods have been widely adopted for learning lowdiimesioonal representations of high-order data. Canonical models such as CANDECOMPPPARAFAA
(CP) (Kruskal, 1977), Tucker decomposition (Lathauwer et al., 2000), and tensor singular value decomposition (t-SVD) (Kilmer al., 2013) provide multilinear analogs to classical matrix factorizations such as PCA or SVD. These techniques exploit the inherent multi- mode structure of tensor data and are effective for compression, denoising, and subspace extraction. Building upon these foundations, approaches such as Multilinear PCA (MPCA) (Lu et al., 2008), Tensor Train PCA (TT-PCA) (Bengua et al., 2017), and higher-order embedding methods (Liu et al., 2010: Wang et al., 2018) adapt tensor decomposition (semi- semmi-suuprrvised learning settings.
Despite their broad applicability, these methods are inherently linear and typically assume rigid structural constraints, such orthogonality fixed rank across modes (Lu al., 2011). More critically, they often treat all tensor modes as equally informative, applying the same modeling as- sumptions regardless of the variation or relevance of each dimension (Wang & Koniusz, 2023; Zhang et al., 2025b; Dong et al., 2025). This uniform treatment limits results in real-world scenarios, where different modes (e.g., spatial, temporal, semantic) may contribute unequally the task hand. Our approach diverges from traditional methods by introducing a non-linear, kernel-based formulation on soeeesspccfiic subspaces from tensor unfolding. Unlike standard tensor subspace techniques, we explicitly model mode-wise uncertainty, enabling adaptive, data-driven regularization based on subspace relevance.
Kernel methods on tensor data. Kernel methods offer powerful tools for capturing non-linear relationships in data through implicit mappings into high-dimensional RKHS, Early extensions of kernel PCA (Schölkopf et al., 1998) and support vector machines to tensor data often relied on flattening tensors into vectors or defining simple tensor- productt kernels (Chen et al., 2022). More sophisticated formulations use structured kernels based on mode- decompositions or Kronecker- product designs (He et al., 2017), seeking to preserve multi-way dependencies within the kernel framework. While these methods successfully bring non-linearity into tensor-based modeling, they
2

---

Published as a conference paper at ICLR 2026
B
B
X
Unfold
MSN
1,..r, M
MLP * HoT
SVD
Soft k-means
KTL
FC
sample with index i
( 1y3
y
(U
Figure 2: Overview of the proposed Uncertainty-driven Kernel Tensor Learning (UKTL) pipeline for action recognition. For brevity, we use skeletons as an example. Each skeleton sequence is divided into temporal blocks ......, B,, embedded via an MLP, and processed by Higher-order Trans- former (Ho'T) to obtain feature tensor X. These tensors undergo mode-m matricization (l....: .......M)) M) and SVD to extract M subspaces per sample. Soft k-means clustering yields C' Nyström pivots, each represented by also M subspaces. A Multi-mode SigmaNet (MSN) estimates uncertainty vectors over all subspaces, which are used to regularize kernel computations. The Nyström- approxiappro KTL maps inputs to compact, uncertainty-awer aware representations, g; for final classification. The en- tire model trained end-to-end.
face significant challenges in practice. The most notable limitations include high computational cost due to dense kernel matrices, poor scalability to large datasets, and reliance on handcrafted or static kernel definitions. Moreover, they typically lack mechanisms for adapting kernel behavior based on the heterogeneous informativeness of different modes, a gap that is particularly critical for tasks involving dynamic or structured input data like tensor sequences.
Our method addresses these limitations through a product subspace kernel that compares tensors via their mode- wise unfolded bases. This structured formulation enables efficient, interpretable comparisons while preserving the multi-way organization of the input. Furthermore, our kernel is not predefined, but learned end-to-end using Nyström approximation scheme with dynamically selected pivot tensors, allowing both scalability and adaptability to the underlying data distribution.
Scalable kernel approximation. The expressive power of kernel methods often comes the cost of computational inefficiency, as computing and storing kernel matrices scales quadratically with the number of samples. To overcome this limitation, several approximation strategies have been proposed. The Nyström method (Williams & Seeger, 2000) approximates the full kernel matrix using subset of representative samples, while random Fourier features (Rahimi & Recht, 2007) and convolution-based features (Koniusz et al., 2016) approximate shift-invariant kernels via explicit feature maps. Structured low-rank approximations (Gittens & Mahoney, 2016) further aim to reduce complexity by exploiting algebraic structure in the data.
Although effective in general settings, most existing kernel approximation techniques are designed for vector or matrix data, and often rely on static, randomly selected dictionary elements. These approximations are typically fixed prior to training and disconnected from the learning process, Myström-based
limiting their ability to adapt to task-specific data distributions. We introduce approximation strategy tailored for tensor sequences, where pivots are constructed dynamically via soft k- f-mmans clustering. This approach allows the pivot set evolve during training, ensuring that the kernel approximation remains relevant and responsive to the structure of both the sample tensors and the learned subspaces. Importantly, such an approximation mechanism is fully differentiable, enabling seamless integration into our end-to-end learning pipeline.
Uncertainty modeling in representation learning. Modeling uncertainty in neural representations enjoys growing attention, particularly in the context of Bayesian deep learning. Techniques such as variational inference (Blundell al., 2015) and Monte Carlo dropout (Gal & Ghahramani. 2016) aim to quantify epistemic or alcatoric uncertainty in network predictions or parameter estimates. However, most existing methods focus on scalar outputs, latent features or rescaling distance (Wang & Koniusz, 2022) and do not extend uncertainty modeling to mode-related subspaces underlying tensor decompositions. In the domain of tensor learning, uncertainty is rarely modeled explicitly. Standard approaches apply uniform regularization across all tensor modes, implicitly assuming that each contributes equally to model performance. This assumption is problematic in applications where the signal strength, noise level, semantic relevance differs substantially across modes. To address this gap, we propose a maximum likelihood- framework for learning the relative un- certainty of mode-specific subspaces. This yields interpretable, data-driven regularization terms that
3

---

Published as a conference paper at ICLR 2026
reflect the variability and informativeness of each subspace. By incorporating uncertainty directly into the learning objective, our method enhances both robustness and generalization, particularly in cases where certain modes prone to noise or contain less discriminative information.
3 PRELIMINARIES
Notations. We follow standard notational conventions in multilinear algebra (Vasilescu & Terzopou- los, 2003). Scalars are denoted by lowercase letters (I)) vectors by bold lowercase (x), matrices by bold uppercase (X), and higher-order tensors by calligraphic symbols (X)). The set Zy denotes the integer set {1, ... J.. An M-th order tensor A R x/1xxxxxxxxx has elements indexed by d11--iar- The mode-m unfolding (or matricization) of A denoted A(m) E I I 4,, where each column corresponds to mode-m fiber (obtained by fixing all indices except im.. The Frobe-
nius norm of a tensor is given by ||Al|== EEi,...ee -i----se- For more details, see (Huckle, 2019).
Higher-order Transformer (HoT) layers. Let the HoT layer (Kim et al., 2021) be fm- RJ J'"d + J with two sub-layers: (i) higher-order self-attention am-n R + R xx and (ii) a feed-forward MLP R J + Rd Moreover, let indexing vectors i E Iÿ = I x...... II (m modes) and j IJ II x... xI, (n modes). For the input tensor X R xd with hyper-edges of order m, HoT layer evaluates:
am in (X)
Σ
N. X, X,W. W
h.p h.u
MLP MLPn-n((mm-a (X) = L (LLLUL) n-m (a (am-m (x))))
(2)
fm (X)= = (X) + (a (am-n (X)),
(3)
where " E En is the so-called attention coefficient tensor with H heads, and dij
A, ER' is
vector, WV RRbda and Wo hi Rddaxd are learnable parameters. Moreover, fa indexes over the so-called equivalence classes of order-(m + n)) in the same partition of nodes, L R J and 1 R. J"ddp + J are equivariant linear layers and dr is the hidden dimension.
hi
The HoT layers in our model operate directly on this hyper-edge tensor using the higher-order tention and feed-forward formulations provided in Eqs. - Specifically, the HoT attention term (Eq. 1) computes higher-order interactions among hyper-edges through the multi-head attention coefficients i.j The equivariant feed-forward defined by - and L (Eq. 2); and the overall block applies a residual update followed by LayerNorm (Eq. 3). This two-layer HoT stack aggregates information across hyper-edges (and over temporal blocks, when applicable), producing the higher-order representation used by the subsequent tensor kernel module.
METHOD
In this paper, mode refers only tensor dimension (e.g., spatial, temporal, hyper-edge) and not sensor modality or multi-modal inputs. We propose an Uncertainty-driven Kernel Tensor Learning (UKTL) framework for action recognition (see Fig. 2). For simplicity, we illustrate using skeletons. Skeleton sequences are segmented into temporal blocks, embedded by an MLP, and encoded by a НoT produce feature tensors. These tensors are unfolded along each mode and decomposed via SVD to extract subspace projections. Soft k-means clustering identifies representa- tive Nyström pivot tensors, also projected into subspaces. Multi-mode SigmaNet (MSN) estimates mode-wise uncertainty vectors, which regularize kernel computations, enhancing robustness. The Nyström approximation enables scalable, compact, and uncertainty-awee kernel embeddings, fol- lowed by classification. The entire pipeline is trained end-to-end for effective learning.
method
4.1 TENSOR REPRESENTATION OF SEQUENCES
Definition of hyper-edges. A hyper-edge in our model is defined as an unordered triplet of joints, e.g., (2 js), representing three-way way structural relation in the skeleton. We enumerate all valid joint triplets, the total number of hyper-edges is No (2)
4

---

Published as a conference paper at ICLR 2026
For skeletal action recognition, consider two common forms: (i) raw tensors X' E R where d the spatial dimensionality (e.g.. 2D/3D joint coordinates), J is the number of body joints, and T is the temporal length; and (ii) higher-order feature tensors X' E RxxNxx, where N is the number of (hyper-Jedges and T is the number of temporal blocks extracted from the sequence.
To keep the backbone lightweight while capturing high-order structure, we composed of (a) a 3-layer MLP
compact encoder and (b) a HoT (Kim et al,, 2021). Each temporal block (with T' consecutive frames) encoded into a d J feature map by the MLP (the encoder produces per-joint embeddings æt.j E Rd for each temporal block (). Let X, E RRdxJ be the ML.P output at temporal block f. E -T,...,T). A hyper-edge feature is then obtained by aggregating the three joint embeddings through the shared MLP. The HoT module then aggregates these (stacking hyper-edges across all triplets and temporal blocks forms the input tensor to HoT) to produce structured more tensors encoding hyper-edge interactions.
To form the final input tensor for UKTL, we (i) permute the feature and hyper-edge modes; (ii) extract upper- trianulti third-order hyper-edges (size Ng = -2)): and (iii) stack the tensors along the temporal axis. This results compact representation E IRd NgxT, capturing rich multi-way relationships.
Mode-m matricization & subspace representation. Let X R denote an Mth-order tensor. The mode-m matricization (also called unfolding) transforms X' into matrix X (m) E
RR.mxIIe XIle Is by rearranging its entries such that the mtth mode becomes the row dimension and all other modes are flattened into the columns. The full set of matricizations is denoted as X (X()).... (M))). For our case, each skeleton sequence is encoded into a third-order tensor 2g E Rdd XNxx, where d' is the feature dimension, Ng = =(2) is the number of third-order hyper-edges, and 7 is the number of temporal blocks. We apply mode-m matricization to each tensor X, resulting X E ((N))) (1))) E R N& (ddT), and ((3)) T
Each matrix X (()) captures the structure of X; along mode m by aligning its corresponding slices as rows. These matrices serve as inputs to subspace- based kernel functions. To obtain a compact representation, we perform Tucker decomposition (Kolda & Bader, 2009) on each tensor:
X C; X1 US(1) X2 U1()) X3 U2(3).
(4)
where C₁ is the core tensor, and U(()) contains the leading left singular vectors of Xim) (i.e., the mode-m subspace basis). These orthonormal matrices summarize the dominant variation in each mode and serve as the input to our product subspace kernel. Each satisfies:
(m)
((m) (m) Am)) s(m))
(5)
where (m)) is a diagonal matrix of singular values. This decomposition ensures that our kernel compares tensor sequences in terms of their most informative multi-mode subspaces, leading to better generalization and robustness.
4.2 SUM-PRODUCT GRASSMANN KERNEL
While many kernel methods have been proposed for tensor data, few explicitly account for the inherent multi-linear structure of the tensor space. Signoretto et al. (2011) introduced a framework for RKHS of multilinear functions, allowing kernels to operate directly on tensors rather than on vectorized representations. This is achieved achiveed via bounded multilinear mappings 1b H1 x.. X - R. where each H.n is an RKHS. These mappings form a Hilbert space of multilinear functions, effectively extending the kernel framework to infinite-dimensional tensor-valued functions. Given an RKHS embedding ò Rh x + H. the associated tensor kernel is defined as:
kx,, X2) (o()), ((2)))
(6)
To exploit mode-wise structural information, we adopt a product kemel formulation. Let Xs(m) and (m)) be the mode-m matricizations of tensors X, and Xz. The product kernel is expressed
M
k(X, X,) = II
k k(Ka(m)) Kymm),
(7)
m=
5

---

Published as a conference paper at ICLR 2026
where each factor kernel (m)) (m)) measures similarity between subspaces spanned by mode- m unfoldings. Rather than comparing the raw unfoldings directly, which are high- high-dimensional and sensitive noise, project them onto oow-dimensional subspaces via SVD. Specifically, for mode- matricization (m)) TR x (with I. = -II m Ik), we compute:
[m)) = ( (m) (mmy"
where Ux TR xp contains the top-p left singular vectors. This matrix defines p-dimensional subspace, interpreted as a point on the Grassmann manifold G(p, Im), the of all p-dimensional linear subspaces in I We embed this point into the space of orthogonal projection matrices via:
Φ G(p, 1m) - R Im
span (()))) + U U X(m3"
(9)
This embedding is isometric with respect to the projection metric, enabling the use of Euclidean distances the space projection matrices. The projection kernel, which defines similarity on the Grassmann manifold, is given by:
((Ux,.)))) Ux ) = I
Xeton) Xx
(10)
and is known to be positive definite. Inspired by the Gaussian RBF kernel, we extend Eq. 10 to define Grassmann-basee factor kernel for each mode:
[[Uxi(m) UI Unim) Uxs(m UX ((m)
ke (()i())) {z(m))= exxp
(11)
202
Combining across all M tensor modes, the final product kernel becomes:
[|Uxum) Uxam -Ux(m)) UX
11
4 A(,, X2)
II exp
(12)
20²
This kernel is positive definite due to the closure properties kernels (the product of valid kernels is valid). For flexibility, also introduce sum kernel:
M
[[Uxam)
UT Xecms
Uxsmm)
U U 11
ka X2)
exp
(13)
202
Kernelized Tensor Learning (KTL). Finally, to balance global and local structure, we propose sum-product kernel:
M 25) Σ (( + (1-1) -4) II & ((Ka Kz(m)); m=1
(14)
m==
where μ E [0, 1] controls the relative contribution of the sum and product components.
Representing tensors via mode-wise wise subspaces on Grassmann manifolds offers two key benefits: (i) computational efficiency, subspaces are significantly lower-dimensional than unfoldings. and (ii) robustness to noise and occlusion, since the subspace can effectively approximate missing information. These properties make our Grassmann based factor kernel particularly suitable for high high-ddmensionnll, structured, and often noisy data, e.g., skeleton sequences.
4.3 UNCERTAINTY-DRIVE SUBSPACE LEARNING
Uncertainty-driven KTL (UKTL). Different tensor modes may exhibit varying levels of uncer- tainty due to noise, missing data, or modality imbalance. To capture this, model uncertainty individually across each mode rather than assuming a global, constant uncertainty as in prior works (Matthies, 2007; Kiureghian & Ditlevsen, 2009; Indrayan, c2008.; Hüllermeier & Waege- man, 2021; Kendall & Gal, 2017). Inspired by (Wang Koniusz, 2022; 2025), we introduce a mode-wise uncertainty-aware mechanism called Multi-mode SigmaNet (MSN).

---

Published as a conference paper at ICLR 2026
The MSN consists of M branches, per tensor mode. Each branch takes the projection matrix U Xxtt)) Uxit))
Xiim) (i.e., the mode-m subspace X;) and outputs corresponding uncertainty vector (am) ER². Each branch is composed of an FC layer followed by scaled sigmoid activation to
ensure positivity and boundedness: (())()))) ScaledSigmoid(FC(UX U Uxum)) )) To incorporate
uncertainty into the kernel, we define:
U Ux
U Xitnn / Xi(m)
(15)
where the division is element-wise across the rows of U Uayyy effectively down-weighting directions with higher uncertainty. We update our projection- based kernels (Eqs. 12 and 13) by replacing (i with Ux Ux,,,.) The resulting kernels are more robust to noisy or uncertain features in each tensor mode, and similarly for the sum-produee kernel (Eq. 14).
4.4 NYSTRÖM KERNEL LINEARIZATION
Pivot selection. To reduce the computational complexity of kernel methods on tensorial data, we use a Nystrôm-based low-rank approximation of the kernel matrix. This allows to obtain an explicit, finiteddimensional feature representation from the proposed tensor kernel. Given N tensors {X, J we first identify C' pivots { through soft k-means clustering (Koniusz & Mikolajezyk, 2011), which differentiable and suitable for end-to-end learning. Each pivot 2, IRd' x approxi- mates local cluster center (a.ka. local prototype) in the tensor space. We solve:
min [Z Zc]
X
3, [a
(16)
j=1
where o; ERC denotes X''s soft assignment to the C pivots.
Nyström approximation. Let KNc NxC be the kernel matrix between the data and pivots, and Koc E CCC the kernel matrix among pivots:
KNcly Z,), Kxclyy=((2, 2,).
(17)
We stabilize inversion via eigendecomposition of Kee:
Koc UAUT,
(18)
and compute the inverse square root via:
p-1 -1-2/
(19)
The Nyström feature embedding is then given by:
G = KxCPP G= G G
(20)
where is the mean across columns of G to ensure centering. The resulting G E NxC serves as the low-rank linearized kernel features.
End-to-end model integration. Our complete model stacks four modules: tensor encoder (MLP HoT), MSN for subspace uncertainty modeling, sum-product Grassmann kernel with Nyström kernel linearization, and a final FC classifier. Formally, the model function is:
|X (T2: PMPPM PHoT::
(21)
where P= PPMMP PHOT] are encoder parameters, PMSN for uncertainty, and PeC for classification.
Our framework is modality- and supports any input representable as a structured tensor, including skeletons, RGB frames, depth maps, and other sensor streams, without architectural changes. Each modality is encoded using an uncertainty-aware tensor kernel that captures intra- modal correlations across spatial, temporal, and channel dimensions. Modality-specific tensor representations are projected into shared kernel space and fused via learnable weighted sum- mation, preserving modality structure while modeling inter-modal complementarities. The fused embedding is then processed by MLP+HoT for end-to-end multi-modll learning.

---

Published as a conference paper at ICLR 2026
Table 1: Results on NTU-60, NTU-120 and Kinetics-Skeleton, UKTL outperforms graph, hyper- graph, and transformer models by using uncertainty-an tensor kernels. All tensor methods use the same MLP + HoT backbone for fair comparison.
Method
NTU-60
NTU-120
Kinetics- Skeleton
X-Sub(S)
X-Viow(S))
X-SDb(Re)
X-Setupe (6)
Top-I(5)
Top-5(%)
TCN (Kim & Reiter, 2017)
20.3
40.0
ST-GON 2018)
70.7
AS GCN (Li et 2019)
86.8
94.2
78.3
79.8
34.8
56.5
(Shi al., 2019)
NAS GCN (Peng el 2020)
89.4
95.7
37.1
60.1
Graph-based
Shin-GCN (Cheng
MS-G3D (Lio at.,, 2020)
91.3
96.2
86.9
88.4
38.0
60.9
Sym-GNN (1 et 2022)
90.1
96.4
37.2
58.1
SSL (Yan et 2023)
92.8
96.5
84.8
85.7
CTR-GCN (Zhang
2025g)
92.6
96.7
89.6
91.0
FD-GON (Huo et 2025)
92.8
96.7
89,4
90.7
DSDC-GCN (Zhunng et al, 2025)
93.0
97.1
89.9
90.6
38.6
63.4
Hyper-GNNN (Hao al., 2021)
89.5
95.7
37.1
60.0
DHGCN (Wei al., 2021)
90.7
96.0
86.0
87.9
37.7
60.6
Hypergraph-
SD.HGGN al.,
based
Selective HCN (Zhu et al., 2022)
90.8
96.6
38.0
61.1
Hype--GCN
87.0
STTR (Plizzari el all,, 2021)
90.3
96.3
85.1
87.1
38.0
60.5
(2hang et al. 2021)
Transformer-
MTT (Kong et al., 2022)
90.8
96.7
86.1
87.6
37.9
61.3
4s GSTN (Jiang et 2022)
91.3
96.6
86.4
88.7
MAMP (Mao et al., 2023)
84.9
89.1
78.6
79.1
STJD-MP (Gunasekara al., 2025)
85.9
90.0
77.1
79.3
Backbone
90.8
95.8
85.2
87.4
36.7
59.5
KPCA (baseline)
92.0
96.8
88.6
90.1
37.1
59.8
Temsorbused
TPCA (baselinee)
96.8
88.2
90.0
KTT.
92.5
97.1
88.8
90.3
38.9
61.9
Training & inference. The training loss combines classification and uncertainty regularization:
P
Σ, No +1 ).
Kk.Neton) +1
e (x (U.(
log
(22)
where ((-) is cross-entropy, 3 weights the uncertainty regularization across modes. Moreover, (0., xx-
At inference, the model (including learned pivots) remains fixed. Given N' test sequences, the same pipeline is applied generate predictions for sequence-level action classification. We now present our experiments, followed by detailed analysis and discussion.
EXPERIMENT
5.1 DATASETS AND SETUPS
Datasets & protocols. We evaluate method three large-scale benchmarks: (i) NTU RGB+D
(NTU-60) (Shahroudy et al., 2016) contains 56,880 sequences across 60 action classes, featuring variable sequence lengths, high intra-class variability, and up two subjects per clip, each with 25 joints. We follow two standard protocols: cross-subject (X-Sub) and cross-view (X-View) (ii) NTU RGB+D 120 (NTU-120) (Liu al., 2019) extends NTU-60 120 classes and 114,480 samples, captured from 106 subjects and 155 viewpoints. We adopt the cross- (X-Sub) and cross-setup (X-Setup) evaluation protocols. We also evaluate other modalities (e.g.. RGB and depth) and their fusion on NTU-60 and NTU-120 using standard protocols. (iii) Kineticx-Skeleton is derived from the Kineties dataset (Kay et al., 2017), with around 300,000 videos spanning 400 action categories. Skeletons extracted using OpenPose (Cao et al., 2017) (18 joints per frame) in ST-GCN (Yan et al., 2018). We use their released skeleton data and report Top-1 and Top-5 accuracy.
Baselines. We compare against several tensor-based baselines using shared encoder (an MLP followed by a HoT block). Kernel PCA (KPCA) applies kernel functions to vectorized features, dis- carding tensor structure. Tensor PCA (TPCA) preserves multi-way structure via Tucker HOSVD decomposition but remains inherently linear. Our proposed KTL introduces mode-win wise kernel func- tions over tensor subspaces, enabling non-linear, structure-aware similarity. Uncertainty-driven KTI.
8

---

Published as a conference paper at ICLR 2026
(UKTL) further enhances this by modeling mode-specific uncertainty adaptively weight subspace contributions. All models are trained under identical settings for fair comparison.
Setups. Experiments are implemented in PyTorel and trained with SGD (momentum 0.9, weight decay 0.0001, batch size 32) and an initial learning rate of 0.1. For NTU-60/ 120, the LR decays x 10 epochs 40 and 50, ending at 60; for Sinetics-Skeleton Models use two HoT layers for NTU-60/120 and four for Sinetics-Skeletar with hidden dimension 16 and 4/8 attention heads, respectively. Videos split into overlapping 30-frame blocks (stride 10). Hyperparameters (e.g., J4, 3. Nyström pivots) are tuned via HyperOpt (Bergstra et al., 2015).
decay occurs 50 and 60, ending 80.
5.2 COMPARISONS WITH THE STATE OF THE ART
Single-modality evaluation. Table shows that KTL and UKTL achieve strong performance across all three benchmarks. Graph-based methods like CTR-GCN and FD-GCN perform well, leveraging spatial-tempora skeleton modeling via graph convolutions. Hypergraph-baad based approaches further benefit from capturing higher-order joint relationships, while transformer based models improve performance by modeling long-range dependencies with global attention.
Our tensor-based approaches, built on consistent encoder backbone, systematically outperform these competing paradigms. KPCA improves over the backbone by introducing non-linear feature mappings but loses the inherent multi-way structure of the skeleton data. TPCA respects tensor modes and their multilinear structure but remains limited by its linear nature, showing improve- ments mainly on certain metrics. KTL advances these baselines by learning structured, non-linear kernel functions tensor subspaces, effectively combining the advantages of non-linearitt with tensor-mode awareness. This results in clear gains across all evaluation metrics, evidencing bet- ter feature discrimination and robustness. Crucially, UKTL,, which incorporates uncertainty- regularization to adaptively weight different tensor modes based on their relevance and reliability, delivers the best overall performance. The consistent improvements of UKTL across all datasets and metrics, such as a 0.6% boost over KTL on NTU-60 X-Sub and 1.2% on NTU-120 X- X-Sub, highlight the effectiveness of modeling uncertainty complex skeletal data representations.
These results collectively underscore two key insights: (i) maintaining the tensor structure vital for capturing the rich cilti-ddmennional correlations inherent in the data, e.g., for skeletal action recognition; (ii) adapting model behavior through uncertainty estimation leads to more robust and discriminative representations, enhancing generalization in challenging scenarios.
Multi-modal evaluation. Table presents the results. RGB and depth inputs exhibit spatial-temporal struc- tures fundamentally different from skeletons. Nevertheless, our method achieves competitive RGB-only (83- 86%) and depth-only (85-87%) per- formance. This confirms that the framework is not specialized to skele- geometry and can effectively en- code heterogeneous modalities once represented tensors. Fusing Skele- ton+RGB or Skeleton+Depth con- sistently yields significant improve- ments, reaching up to 95% on NTU- 60 X-Sub. These gains indicate that kernel-basee fusion successfully ploits complementary cues, e.g., motion structure from skeletons, appearance from RGB, and geom- etry from depth, without modality-specine architectures. Even RGB+Depth fusion, without skeleton input, enjoys improvements over single- modality baselines due to effective inter-modal modeling.
Table Evaluation of single- and multi-modal perfor- mance on NTU-60 and NTU-120. Skeleton alone provides strong, competitive baseline, while RGB and Depth demon- strate the framework's 's flexibility across modalities. Fusing multiple modalities consistently improves accuracy, show- ing that UKTL effectively captures complementary infor- mation and inter-modal correlations.
Modality
NTU-60
N7U--120
X-Sub (%)
X-View (S)
(%)
X-Setup
Skeleton
93.
97.3
90.0
91.4
RGB
83.2
86.1
79.0
81.0
Depth
85.0
87.5
80.5
82.1
Skolcton RGB
94.5
97.9
91.3
92.5
Skeleton Depth
94.8
98.0
91.5
92.7
RGB Depth
87.2
89.6
82.5
84.3
Skeleton RGB + Depth
95.5
98.5
92.8
94.0
Combining Skeleton, RGB, and Depth achieves the best overall results (95.5/98.5 NTU-60 and 92.8/94 /4.00 on NTU-120), outperforming both uni-modal baselines and multi-stream models. All fusion is performed within unified tensor-kernel formulation, without ad-hoc modality-specific

---

Published as a conference paper at ICLR 2026
100
100
Accurcy (85)
80
00
1p=21
(pm5)
(ourg)
prod
60
100
150
180
202
80
5um
Fr03
Sum gr
(a) Subspace order p.
(b) Kernel choice.
(c) Number of pivots C. (d) Kernel composition.
Figure 3: Ablation study evaluating the effects of subspace order (on NTU-60/120), and of kernel choice, Nyström pivots, and kernel composition (on NTU-60) within the UKTL framework.
components. Although evaluated on human action datasets, the consistent gains across modalities and their combinations demonstrate that UKTL generalizes beyond skeleton-based inputs and natu- rally extends to other structured tensor modalities.
These results suggest that UKTL's gains arise from its ability to operate in shared tensor-kernel space where modality structure is preserved while aligmment occurs implicitly through kernel interactions, rather than through explicit architectural coupling. This implicit alignment makes fusion resilient to modality imbalance and noise, allowing informative modalities to domi- nate when present without suppressing weaker ones. Moreover, the absence modality-specific design choices indicates that performance improvements stem from the kernel formulation itself, highlighting UKTL as a scalable and principled approach for mudti-modal learning in settings where modality availability, quality, or structure may vary.
5.3 ABLATION STUDY
Below, we conduct an ablation study on skeleton sequences from NTU-60 and NTU-120 to evaluate key hyperparameters and configurations. See Appendix for additional details and evaluations.
Subspace order p. Fig. 3a shows that increasing the subspace order consistently improves ac- curacy NTU-60 and NTU-120 (X-Sub), with performance saturating beyond a dataset-specific threshold. This suggests that low-dimensional subspaces effectively capture the essential discrimi- native structure. Optimal values are p = 8 for NTU-60 and p 10 for NTU-120.
Kernel choice. To evaluate the effectiveness of our kernel (Eq. 14), we compare it with standard linear and polynomial kernels. As shown Fig. 3b, the linear kernel performs worst (77.5% on X- Sub), while polynomial variants offer moderate gains. In contrast, our sum- poorruuuct kernel achieves 93.1% (X-Sub) and 97.3% (X-View), highlighting its superior ability to capture complex subspace interactions by jointly modeling additive and multiplicative relationships.
Number Nyström pivots. We examine how UKTL's performance varies with the number of Nyström pivots used for kernel approximation. As shown in Fig. 3c, accuracy improves significantly up to 150 pivots, then stabilizes beyond 180, reflecting a balance between approximation quality and computational cost. To ensure optimal performance across datasets, et al., 5) to automatically tune the pivot number during training.
HyperOpt (Bergstra
Kernel composition. Below compare sum-only, product-only, and combined (sum- (oun-product) ker- nels. As shown in Fig. 3d, the sum-kernel yields 81.6% (X-Sub), while the product-kemms achieves 91.8%. Their combination reaches the highest accuracy, 93.1% (X-Sub) and 97.3% (X-View), demonstrating that additive and multiplicative structures capture complementary information.
6 CONCLUSION
We introduced Uncertainty-driven Kernel Tensor Learning (UKTL), scalable and principled frame- work for structured multi-way and multi-modal data. UKTL combines mode-wise subspace kernels, dynamic Nyström linearization, and uncertainty-aware regularization to enable expressive, adap- tive, and efficient tensor comparison without flattening or losing structure. Experiments on ac- tion recognition datasets show UKTL consistently surpasses of graph. hypergraph, and transformer models, while offering interpretable mode-wise insights. This establishes a powerful kernel-based approach for learning from structured tensor data.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Xi Ding, a visiting scholar at the ARC Research Hub for Driving Farming Productivity and Dis- ease Prevention, Griffith University, conducted this work under the supervision of Lei Wang. Lei Wang proposed the algorithm and developed the theoretical framework, while Xi Ding implemented the code and performed the experiments. We thank the anonymous reviewers for their invaluable insights and constructive feedback, which have contributed improving our work.
This work was supported by the Australian Research Council (ARC) under Industrial Transforma- tion Research Hub Grant 1H180100002. This work was also supported by the CSIRO Allocation Scheme (Lead CI: Piotr Koniusz), the National Computational Merit Allocation Scheme 2025 (NC- MAS 2025; Lead CI: Lei Wang) and the ANU Merit Allocation Scheme (ANUMAS 2025; Lead CI: Lei Wang), with computational resources provided by NCI Australia, an NCRIS-enabled capability supported by the Australian Government.
REFERENCES
Johann A Bengua, Phien N Ho. Hoang Duong Tuan, and Minh N Do. Matrix product state for higher-order tensor compression and classification. IEEE Transactions on Signal Processing, 65 (15):4019-4030, 2017. 2
James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D Cox. Hyperopt: a python library for model selection and hyperparameter optimization. CSD, 8(1):014008, 2015. 9, 10
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017. 8
Cong Chen, Kim Batselier, Wenjian Yu, and Ngai Wong. Kernelized support tensor train machines. Pattern Recognition, 122:108337 2022. 2
Panqi Chen, Lei Cheng, Jianlong Li, Weichang Li, Weiqing Liu, Jiang Bian, and Shikai Fang. Func- tional complexity-adaptio temporal tensor decomposition. In The Thirty-ninth Annual Confer- ence on Neural Information Processing Systems, 2025a. URL https: / http//ppneeeeeeeeeeeeeeeeeeeeeep RhRLbCt RRRnnn TMS. 21
forum?id
Qixiang Chen, Lei Wang, Piotr Koniusz, and Tom Gedeon. Motion meets attention: Video motion prompts. Asian Conference on Machine Learning, pp. 591-606. PMIR, 2025b. 25
Ke Cheng, Yifan Zhang, Xiangyu He, Weihan Chen, Jian Cheng, and Hanqing Lu. Ekeleton-basee action recognition with shift graph convolutional network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 8
Dexuan Ding, Lei Wang, Liyun Zhu, Tom Gedeon, and Piotr Koniusz. Learnable expansion of graph operators for multi-modal feature fusion. In The Thirteenth International Conference on Learning Representations, 2025a. URL https: wwwpreeieeieeeeeeeeeeee net forum Pimumiii-SSaa 2id=SM2qIOsa1N. 25
Xi Ding and Lei Wang. The journey of action recognition. In Companion Proceedings of the ACM on Web Conference 2025, pp. 1869-1884, 2025a. 25
Xi Ding and Lei Wang. Do language models understand time? In Companion Proceedings of the ACM on Web Conference 2025, pp. 1855-1868. 2025b. 25
Xi Ding, Lei Wang, Piotr Koniusz, and Yongsheng Gao. Graph your own prompt. In The Thirty- ninth Annual Conference Neural Information Processing Systems, 2025b. URL https: // openreview net/ forum
25
Xi Ding, Lei Wang, Piotr Koniusz, and Yongsheng Gao. Learning time in static classifiers. In The Fortieth AAAI Conference Artificial Intelligence, 2025c. 25

---

Published as a conference paper at ICLR 2026
Junhao Dong, Piotr Koniusz, Liaoyuan Feng, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Robustifying zero-shot vision language models by subspaces alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 21037- 21047, October 2025. 2
Shikai Fang, Akil Narayan, Robert Kirby, and Shandian Zhe. Bayesian continuous- tucker decomposition. In International conference on machine learning, pp. 6235-6245. PMLR, 2022.
21
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016. 3
Alex Gittens and Michael W Mahoney. Revisiting the nyström method for improved large-scale machine learning. The Journal of Machine Learning Research, 17(1):397744011, 2016.
Shanaka Ramesh Gunasekara, Wanqing Li, Philip Ogunbona, and Jack Yang. Spatio-temporal joint density driven learning for skeleton-based action recognition. IEEE Transactions on Biometrics, Behavior, and Identity Science, pp. 1-1, 2025. doi: 10.1 10.1000/000M.2225.555666
8
Xiaoke Hao, Jie Li, Yingchun Guo. Tao Jiang, and Ming Yu. Hypergraph neural network for skeleton-bard based action recognition. IEEE Transactions on Image Processing, 30:2263-2275, 2021. doi: 10.1109/TIP.20111 3.551......5995....
Mehrtash Harandi, Mathieu Salzmann, and Richard Hartley. Dimensionality reduction on spd man- ifolds: The emergence of geometry-aware methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4041):48-4) 62, 2018. doi: 10. 20.1009/PAA...017266555 2.655008.. 20
Changxiang He, Chen Xiao. Shuting Liu, Xiaofei Qin, Ying Zhao, and Xuedian Zhang. Single- skeleton and dual-skeleton hypergraph convolution neural networks for skeleton-based action recognition. In Teddy Mantoro, Minho Lee, Media Anugerah Ayu, Kok Wai Wong, and Achmad Nizar Hidayanto (eds.), Neural Information Processing. pp- 15-27, Cham, 2021. Springer International Publishing. ISBN 978-3-030-92270-2.
Lifang He, Xiangnan Kong, Philip S Yu, Xiaowei Yang, Ann Ragin, and Zhifeng Hao. Dusk: A dual siuuctre-ppeservvie kernel for supervised tensor learning with applications to neuroimages. In Proceedings of the 2014 SIAM International Conference on Data Mining, pp. 127-135. SIAM, 2014. 24
Lifang He, Chun-Ta Lu, Guixiang Ma, Shen Wang, Linlin Shen, Philip S. Yu, and Ann B. Ra- gin. Kernelized support tensor machines. In Doina Precup and Yee Whye Teh (eds.), Pro- ceedings of the 34th International Conference on Machine Learning, volume 70 of Proceed- ings of Machine Learning Research, pp. 1442-1451. PMLR, 06-11 Aug 2017. URL https: //proceedings mllr. 2.ress/v0//het htt.hell........... 2
Thomas Huckle. www5. in. tum. de wwwppeesons//ccckieeeeeeeeeeeeee wwwecckeeeeensooeeeiee
pdf, 2019. 4
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach. Learn., 110(3)457--00 2021. doi: 10.1007/
110994-01-00546655 6
Jinze Huo, Haibin Cai, and Qinggang Meng. Fast distance-enhanced graph convolutional network for skeleton- based action recognition. Pattern Recognition, pp. 111827, 2025. 8
Abhaya Indrayan. Medical biostatistics. Chapman & Hal//CRC, Boca Raton :, 2nd ed. edition, c2008. URL http: //t:://wwwww www. loc. /tt.://////addda///////o////////
html. 6
Yujian Jiang, Zhaoneng Sun, Saisai Yu, Shuang Wang, and Yang Song. A graph skeleton trans- former network for action recognition. Symmetry, 14(8), 2022. ISSN 2073-8994, doi: 10.3390/sym1408155
URL https: www. wddpi..p..m. com wtt.c/mmm/2///// 2003-0332000 9994 22/44///115//
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola. Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017. 8
12

---

Published as a conference paper at ICLR 2026
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for com- puter vision? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. 6
Misha E. Kilmer, Karen Braman, Ning Hao, and Randy C. Hoover. Third-order tensors operators on matrices: A theoretical and computational framework with applications in imaging. SIAM J. Matrix Anal. Appl., 144 172, jan 2013. ISSN 0895-4798. doi: 10. 10.1133 137/1 10837711. URL
ht.ps: htt.///oiio/ orgg C . 137 137/110837711.
Jinwoo Kim, Sacyoon Oh, and Seunghoon Hong. Transformers generalize deepsets and can be extended to graphs & hypergraphs. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:// 4,5
openreview net forum? ddsscnnnnnnnnnn
Tae Soo Kim and Austin Reiter. Interpretable 3d human action analysis with temporal convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1623 1631, 2017. doi: (0.100/CVPPR.2207..300 8
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does matter? Structural Safety, 31(2):105-112, 2009. ISSN 0167-4730. doi:
Risk
Acceptance and Risk Communication. 6
Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Review, 51(3):45--500 2009, doi: 10.1137/0700011XX URL https: /tte:://dco.ooo /ttpd/ii..oo.o1111..111.11111111 07070111x. 5,5
Jun Kong, Yuhang Bian, and Min Jiang. Mtt: Multi-scale temporal transformer for skeleton-based
action recognition. IEEE Signal Processing Letters, 29:528-532, 2022. doi: 10. 1091.00/2... 3142675.
P. Koniusz and A. Cherian. Sparse coding for third-order super-symmetric tensor descriptors with application to texture recognition. CVPR, 2016.
P. Koniusz, F. Yan, P. Gosselin, and K. Mikolajezyk. Higher-order occurrence pooling on mid- and low-level features: Visual concept detection. Technical Report, 2013.
Piotr Koniusz and Krystian Mikolajezyk. Soft assignment of visual words as linear coordinate coding and optimisation of its reconstruction error. In 2011 18th IEEE International Conference on Image Processing, pp. 2413-2416, 2011. doi: 10. (0.110///CP.222111111111
Piotr Koniusz, Anoop Cherian, and Fatih Porikli. Tensor representations via kernel linearization for action recognition from 3d skeletons. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings. Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 37-53. Springer, 2016. doi: 10.1007/978-3-1 0111/99988333199 46493-00 0 URL https: / /rrg 10.1007/000 0000//978333111 319-31193-----11 64493-0_3333
Piotr Koniusz, Lei Wang, and Anoop Cherian. Tensor representations for action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(2):648-663) 2021.
Joseph B. Kruskal. Three-way way arrays: rank and uniqueness of trilinear decompositions, with appli-
cation to arithmetic complexity and statistics. Linear Algebru and its Applicationss, 18(2):95-138, 1977. ISSN 0024-3795. doi: http:://o...og///o.00066024777/7
URL https:
/ //www.ccienceiire.....ooo//// wwwpeeetcccccciiiieioooo http://www.....eeeeeaaaaaaaa///a
2
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decompo-
sition. SIAM J. Matrix Anal. Appl., 21: 2000. 2
Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Actional-structural graph convolutional networks for sicleton-baseee action recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 8
13

---

Published as a conference paper at ICLR 2026
Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang. and Qi Tian. Symbiotic graph neural networks for 3d skeleton-base based human action recognition and motion prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6)331163333, 2022. doi: 10.
1109/TPAMI.22213333335 8
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yo Yu Duan, and Alex C. Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions Pattern Analysis and Machine Intelligence, 2019. doi: 10. 1109/PPAMM.20112216687777
Yang Liu, Yan Liu, and Keith C. C. Chan. Tensor distance based multilinear locality- maximum information embedding. IEEE Transactions Neural Networks, 21(11): (481))1844-1854, 2010. doi: 10.1 100.1//TN.2110.266555.1
2
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2020. 8
Haiping Lu, Konstantinos N. Plataniotis, and Anastasios N. Venetsanopoulos. Mpea: Multilinear principal component analysis of tensor objects. IEEE Transactions on Neural Networks, 19(1): 18-39, 2008. doi: 10. 2007 901297.
Haiping Lu, Konstantinos N Plataniotis, and Anastasios Venetsanopoulos. A survey of multilinear subspace learning for tensor data. Pattern Recognition, 44(7):1540-155) 2011. 2
Yunyao Mao, Jiajun Deng, Wengang Zhou, Yao Fang, Wanli Ouyang, and Houqiang Li. Masked motion predictors are strong 3d action representation learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10181-10191, 2023. 8
Hermann G. Matthies. Quantifying uncertainty: Modern computational representation of probability and applications. In Adnan Ibrahimbegovic and Ivica Kozar (eds.), Extreme Man-Made and Natural Hazards in Dynamics of Structures, pp. 105-135, Dordrecht, 2007. Springer Netherlands. ISBN 978-1-4020-5656-1
Wei Peng, Xiaopeng Hong, Haoyu Chen, and Guoying Zhao. Learning graph convolutional network for skeleton-based human action recognition by neural searching. Proceedings of the AAAI Con- ference Artificial Intelligence, 34(03)::669-2678 Apr. 2020. doi: 10.1609/aaiiv 3a3o9//aaaiv3433.55555 v34003.5652. URL https:tp///.mcc
aaai. org/ indei. htt.ee/..../.....tatiiieeeeeeeaa
8
Chiara Plizzari, Marco Cannici, and Matteo Matteucci. Skeleton-based action recognition via spa- tial and temporal transformer networks. Computer Vision and Image Understanding, 208-209; 103219, 2021. ISSN 1077-3142. doi: http::/00..oo///00000fffrrii.212 URL https: / /www. sciencedirectt com http:///.s.eaeetttaa//////..0... 22
8,
Ali Rahimi and Benjamin Recht. Random features for large-scale kernll machines. In J. Platt, D. Koller, Y. Singer. and S. Roweis (eds.), Advances in Neural In- formation Processing Systems, volume 20. Curran Associates, Inc.. 2007. URL https: http://ppcceeennn...ep www.eeiiiiiiinnnninieeeeeeeeeeee neurins cw/paper.pppp.eeeeepeeeceeee...e files/ ftt.ss/paper//00///50
Pwp.ppaa..aaaaa.appp. pdd.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlincar component analysis as kernel eigenvalue problem. Neural Computation, 1099):129,,,3,9, 1998. doi: 10.1162/ 01107666680001144676
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In IEEE Conference Computer Vision and Pattern Recognition, June 2016. 8
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In CVPR, 2019.
14

---

Published as a conference paper at ICLR 2026
Marco Signoretto, Lieven De Lathauwer, and Johan A.K. Suykens. A to tensorial data analysis. Neural Networks, 24(8):861-874, 2011. ISSN 0893-6080. doi: https://doi..rr/IIII 1tt.11.0.11111.... www.://..eemeet.211........ science/a art i ic http://11////0/0011111111111
framework
URL https: /www. scie sciencedi com/
Artificial Neural Networks: Selected Pa-
pers from ICANN 2010. 5
Zerui Tao, Toshihisa Tanaka, and Qibin Zhao. Undirected probabilistic model for tensor decompo- sition. Advances in Neural Information Processing Systems, 36:25837-25853, 2023. 21
M.A.O. Vasilescu and D. Terzopoulos. Multilinear subspace analysis of image ensembles. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceed-
ings. volume 2, pp. II-93, 2003. doi: 10.10 121 4
Lei Wang and Piotr Koniusz. Uncertainty-DITW for time series and sequences. In European Con- ference on Computer Vision, pp. 176-195, Springer, 2022. 3,6
Lei Wang and Piotr Koniusz. 3mformer: Multi-order multi-mode transformer for skeletal action recognition. In Proceedings of the JEEF/CVF Conference on Computer Vision and Puttern Recog- nition, pp. 5620-5631, 2023. 2
Lei Wang and Piotr Koniusz. Feature hallucination for self-supervised action recognition. Interna- tional Journal of Computer Vision, 133(11):7612-766) 2025. 6
Lei Wang, Jun Liu, Liang Zheng, Tom Gedeon, and Piotr Koniusz. Meet jeanie: a similarity measure for 3d skeleton sequences via remporal-viewpoinn alignment. International Journal of Computer Vision, 132(9):4091 1004449991122,, 44122, 2024a. 25
Lei Wang, Ke Sun, and Piotr Koniusz. High-order tensor pooling with attention for action recog- nition. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3885-3889, 2024b, doi: 10.1 10.11099/CASS444455.20000 000 10446900.
Wenqi Wang, Vaneet Aggarwal, and Shuchin Aeron. Tensor train neighborhood preserving em- bedding. IEEE Transactions on Signal Processing, 66(10):2724-2732, 2018, doi: 10.1 109/TSP. 2018.28 2
Zheng Wang, Shikai Fang, Shibo Li, and Shandian Zhe. Dynamic tensor decomposition via neural diffusion-reaction processes. Advances in Neural Information Processing Systems, 36:23453-
23467, 2023. 21
Jinfeng Wei, Yunxin Wang, Mengli Guo, Pei Lv, Xiaoshan Yang, and Mingliang Xu. Dynamic hy- pergraph convolutional networks for skeleton- based action recognition. CoRR, 12.5211 2021. URL htttp:::/arrii..orggbbbb////////
10570. 8
Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In T. Leen, Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Sys- tems, volume 13. MIT Press, 2000. URL httpppss//ppooceeeddng...ss/ pallss/papr/200000 wiloe/iill/////// wtt.:feeeeeeeeeeeeeeeeeeeeeeeeee http://eeee..etaaaaaaaeeeeeaeeaa
cc/paper_
Paper. .pf.
Hong Yan, Yang Liu, Yushen Wei, Zhen Li. Guanbin Li, and Liang Lin. Skeletonmae: graph- based masked autoencoder for skeleton sequence pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5606-5618, 2023.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. In AAAI, 2018. 8
Shaojie Zhang, Jianqin Yin, and Yonghao Dang. A generically contrastive spatiotemporal represen- tation enhancement for 3d skeleton action recognition. Pattern Recognition, 164:111521, 2025a. 8, 22
Yifei Zhang, Hao Zhu, Junhao Dong, Haoran Shi, Ziqiao Meng, Piotr Koniusz, and Han Yu.
Crossspectra: Exploiting cross-layer smoothness for parameter-efficient fine-tuning. In The Thitt Annual Conference on Neural Information Processing Systems, 2025b. URL https://openrevieet net
15

---

Published as a conference paper at ICLR 2026
Yuhan Zhang, Bo Wu, Wen Li, Lixin Duan, and Chuang Gan. Stst: Spatial-tempont specialized transformer for skeleton- based action recognition. Proceedings of the 29th ACM International Conference Multimedia, MM '21, pp. 3229-3237, New York. NY, USA, 2021. Association for doi: 10.1 1457744740083..445473. URL https:
Computing Machinery. ISBN 978 9781 450386517.
Youwei Zhou, Tianyang Xu, Cong Wu, Xiaojun Wu, and Josef Kittler. Adaptive hyper-graph con- volution network for deeleton-basee human action recognition with virtual connections. arXiv preprint arXiv22411.14790 2024. 8
Yiran Zhu, Guangji Huang, Xing Xu, Yanli Ji, and Fumin Shen. Selective hypergraph convolu- tional networks for Axcleton-based action recognition. In Proceedings of the 2022 International Conference on Multimedia Retrieval. ICMR '22, pp. 518-526, New York, NY, USA, 2022. Asso- ciation for Computing Machinery. ISBN 9781450392389, doi: 10.1 10.1145/512577:3331166 URL /tt.//5111227.33367...
https: //coii www.:oo..rro....0
Tianming Zhuang, Zhen Qin, Yi Ding, Zhiguang Qin, Ji Geng, Yi Liu, and Kim-Kwang Kwang Raymond Choo. Dsde-gen: Decoupled static-dynamic co-occurrence graph convolutional networks for skeleton-based action recognition. IEEE Transactions on Circuits and Systems for Video Technol- ogy, 311) 2025. doi: 10. 10.100//CCSVT 133.
16

---

Published as a conference paper at ICLR 2026
A DERIVATIVE OF UNCERTAINTY-DRIVER KERNELIZED TENSOR LEARNING
For two M-order tensors X; and 2j, we design kernel function that captures their multi-mode structure via uncertainty-awart subspace similarities. Specifically, we formulate two variants of the kernel, sum and product, based on mode-wiss subspace comparisons. These are defined as follows.
Urrertainty-driven sum kernel:
M ke ((Ki(m)) R((m))
Σ
=
m=1
|Üxicmm U
Ux)(m) U Xitma
Σ exp
Xi(m)
(23)
202
Uncertainty-drivee product kernel:
M ,() = II
ks (()a())) Kz(m))
m=11
[Uxi(m) ÚT
Uxsom ÜT Ky
II exp
K((m)
(24)
20²
Here, k(,,, Xxj denotes the overall tensor kernel, while k( (m))))) Rz(m)) is the m-th mode kernel based on mode-m matricizations Xi(m) and (mm)- The matrices Ux and Ux represent the seetraintt-weigghee subspaces derived via singular value decomposition (SVD) and reweighted by uncertainty estimates.
Uncertainty-drinn sum-product kernel: We combine the sum and product kernels to form more expressive hybrid:
h X, X3) = Σ ke (()(m))) Ky(m)) 11)
II & (m)) {(mm)
M
Ü ( X(m)
ÜT Üx X Kycm)
ÚT
exp
20
m=
M
Ü Uxim)) Ú
Ü (m))
U
Xim;;
+ (1 (4)
exp
20
m=1
Ux Uxs(m
Ux UJ xxy(m)
xss(m))
exp
202
m=1
U Ux UT Uxitm
UT U Ux pmm)
(1) II mmm
exp
(25)
202
In this formulation: Ux Ux and U Uxy(m are the unitary matrices obtained from SVD of mode-m matricizations. X.mmm and Xytmn are uncertainty scores obtained via our proposed Multi-mode SigmaNet (MSN), which takes as input the projections Ux..m) U Xxxm and Ux UT Xx respec- tively. l E [0, 1] controls the contribution of the versus product kernel: setting μ 1 results in
17

---

Published as a conference paper at ICLR 2026
pure sum kemnel, while μr 0 yields product kernel. or is the bandwidth of the RBF kernel, kept constant across all experiments.
Remarks.
The sum kernel captures independent mode-wise similarities, while the product ker- nel emphasizes joint agreement across all modes, leading to stricter similarity enforcement. By interpolating between them with μt, the proposed sum sum-produs cum-product kernel offers a flexible mechanism to balance robustness and expressivity. Moreover, the uncertainty weighting via MSN enables the model to emphasize more reliable subspace components, leading to more stable and discriminative tensor comparisons.
B MAXIMUM LIKELIHOOD INTERPRETATION OF MODE-WISE UNCERTAINTY
We provide probabilistic justification for the uncertainty weighting mechanism in UKTL. In partic- ular, we interpret the learned uncertainty vectors Xx.m) as estimates of mode-specific heteroscedas- tic noise variances, modeled via maximum likelihood.
Probabilistic model. Let Uxam) E RPxr denote the orthonormal basis the mode-m subspace for sample X; obtained via truncated SVD. We assume each row Uxam), is noisy observation of a latent clean subspace vector S (((m).A E R' corrupted by Gaussian noise with component-specific variance:
Ux ~ N (Si(m)))) ØkI), ==
(26)
Here, Xstm (o.....,00 opl upl captures the noise variances across the p components of the mode-m subspace.
Maximum likelihood estimation. Given the dataset, the total negative log-likelihood under this model is:
-(a)=- Σ
log p (Uxi(m)) Si(m)= Xecms)
(27)
2,12
U Uniinn Sa() w12 +l2++lo ox
const,
(28)
1,m
which simplifies to weighted Frobenius loss plus a log-barrier:
U Uxi(m) Si(
min
A
log dk-
(29)
X
The division is applied rosewise, and the log o% term acts regularizer to prevent variance plosion.
Practical realization. In practice, we do not have access ground truth clean subspaces S(())) Instead, we approximate them using the observed Ux itself as input to a small network, MSN, which predicts the corresponding uncertainty vector X'ma))" To integrate this into the kernel com- putation, we define scaled subspace basis:
Üx = U Xxim))
(30)
where division is performed row-wise. This transformation suppresses unreliable subspace direc- tions by shrinking their contribution in the kernel, while preserving well-estimated ones.
Uncertainty-aware kernel. The transformed basis Ü Uxtm is then used in the kernel computation (see Eq. 14), yielding a kernel function that is robust to subspace noise.
Remarks. This probabilistic formulation shows that our uncertainty-ant aware kernel arises naturally from maximum likelihood perspective under heteroscedastic Gaussian noise. It enables UKTL to down-wert down-weight unstable or noisy subspace components and focus on discriminative structure, especially when mode-wise quality varies.
18

---

Published as a conference paper at ICLR 2026
M
-0.4
26
-0.00
MW
60-
03
25-
MW
(a) Draw
x
(b) Draw tick
MM
6.75 nome
m
25
MMM
co
WW
MN
W
bedy
(c) Draw x (performed by different subject)
(d) Draw tick (performed by different subject)
Figure 4: Visualization of Tucker decomposition in each mode tensor representations for action draw and action draw tick.
C SKELETON DATA PREPROCESSING
All skeleton sequences are processed to have uniform temporal length of T 200 across datasets. Specifically, (i) if sequence has fewer than T frames, we repeat eyclically until it reaches length T: (ii) if has T' more frames, uniformly sample T frames. We then divide each sequence into temporal blocks length 7" using stride S, ensuring a consistent number temporal blocks across all sequences for fair comparison.
Before feeding the skeleton sequences into the MLP and Higher-order Transformer (HoT) modules, perform joint-wise normalization. First, each joint Vy in frame is centered relative to reference joint (e.g., the torso joint vy.a):
f'y, Uz. ff..
(31)
where indexes the joint, and e denotes the reference joint. Next, we normalize the coordinates of each joint into the range [-1, independently along each axis E {2,
(32)
max (E (et..e.......((((........... T.EE(L.....)
where T is the number of frames and J is the number of joints per frame.
For skeleton sequences involving multiple subjects, we handle each skeleton independently. Specifi- cally, (i) each skeleton is normalized separately and processed individually through the MLP to learn its temporal dynamics; (ii) the resulting per- per-steleton features are passed separately through the HoT block. Finally, aggregate the HoT outputs from all subjects in sequence using average pooling.
19

---

Published as a conference paper at ICLR 2026
D MOTIVATION FOR KERNEL TENSOR LEARNING
To motivate our KTL framework, we analyze the basis components obtained from Tucker decom- position on raw tensor representations two visually similar actions: draw x and draw tick.
Each skeleton sequence represented as a tensor X' E IFxxJx3xT", where r denotes the number of temporal blocks, J is the number of joints, and 7 is the length of each temporal block. We apply Tucker decomposition to extract the basis matrices in each mode of the tensor.
As illustrated in Fig. 4, the decomposed bases for sequences of the same action label exhibit strong similarity across all modes. For example, the basis components for two different samples draw x (Fig. 4a and Fig. 4c) are highly aligned, even though draw x and draw tick are semantically and visually similar actions in terms of 3D joint trajectories, their respective basis components reveal clear differences.
those for draw tick (Fig. 4b and Fig. 4d). In contrast,
This observation suggests that the mode-wise bases obtained through tensor decomposition encode discriminative and compact representations of action dynamics. Rather than comparing full high- dimensional tensors, we use these subspace representations for more efficient and meaningful comparisons. This motivates our approach in KTL, where we model and compare tensor-mode subspaces using kernel methods to capture non-linear relationships among sequences.
E MANIFOLD GEOMETRY AND SUBSPACE LEARNING
Symmetric Positive Definite (SPD) Manifolds. The set of d symmetric positive definite (SPD) matrices, denoted Sym forms a Riemannian manifold, ie., a curved non-Fuclidean space embed- ded Rdxd Riemannian geometry provides a more appropriate notion of dissimilarity for SPD matrices than the Euclidean metric.
commonly used distance on Sym is the affine-invariant Riemannian metric:
(s = log (55"//
(33)
where S1, S2 E Sym log(.) is the matrix logarithm, and || || denotes the Frobenius norm. This metric invariant to affine transformations and aligns with the manifold's curvature.
In learning tasks, SPD matrices can be projected to lower-dimensional SPD spaces via bilinear maps of the form W XW, where W E I2xm SPD (Harandi et al., 2018).
n, ensuring that the projected matrix remains
Grassmann Manifolds. The Grassmannian G(r, n) is the space of all r-dimensional linear sub- spaces of R". point on this manifold is represented by an orthonormal matrix Y E Raxr whose columns span the subspace. The geodesic distance between two subspaces span(Y) and span( (Y') is defined via their principal angles 0g as:
1/2
da Y. Y')
Σ
02
(34)
An alternative is to embed Grassmann points into the SPD cone using projection matrices Y enabling simpler kernel operations via the Frobenius norm:
dist(Y Y')
IY Y'Y 1|.
(35)
This embedding supports RBF kernels over subspaces:
YY
k Y') exp
(36)
20²
which we exploit in constructing our mode-wise subspace kernels and their incertainty-awar ex- tensions.
20

---

Published as a conference paper at ICLR 2026
F ADDITIONAL DETAILS AND CLARIFICATIONS
F.1 CLARIFICATION TECHNICAL CONTRIBUTIONS
While UKTL builds on established primitives such as Grassmann kernels, tensor subspace represen- tations, and Nyström approximations, the core contributions are far from simple combination. In- stead, we introduce new formulations that make tensor subspace kernels differentiable, uncertainty- aware, and scalable, capabiliiies that do not exist in prior work.
Our multi-mode sum-product tensor kernel fundamentally novel. Existing Grassmann or tensor not end-to-end learnable, and cannot capture
kernels operate single-mode matricizations, cross-mode interactions. In contrast, our sum prod-rrduct kernel jointly models additive and multiplica- tive interactions across all tensor modes, is fully differentiable with respect to the subspace bases, and allows the upstream encoder to directly optimize representations through the kernel itself. This design enables the model to exploit nulti-dimensional structure in way that no prior kernel ap- proach supports.
Our uncertainty-awer subspace kernel introduces a probabilistic derivation rather than a heuristic. By modeling heteroscedastic noise over subspace bases, we implement row-wise variance scaling and a bog-berrier regularizer, producing novel uncertainty-weighee kernel. Experiments show that including this component consistently improves accuracy by 0.8-1.2% (Table 1), confirming its material contribution robustness.
We propose differentiable Nyström linearization for tensor kemels.. Standard Nyström methods rely on fixed pivots and are non-differentiable, limiting end-to-end learning. Our approach incor- porates soft k-means pivot learning and gradient-safe eigendecomposition, allowing scalable kernel computation on high-order tensors while preserving positive semi-definiteness. Ablations indicate that learning the pivots provides 0.5-0.9% gain over fixed selections, demonstrating that this design is essential rather than incidental.
We emphasize that the perceived complexity is not overengineering. Each module addresses a dis- tinct challenge: structure preservation through multi-mode subspaces, robustness via uncertainty weighting, scalability with differentiable Nyström, and expressive representation through the HoT encoder. Removing any of these components consistently reduces performance, validating their ne- cessity. These three contributions: (i) uncertainty-awar Grassmann kernels, (ii) multi-mode sum- product kernel family, and (iii) differentiable Nyström linearization, integrate in a fully differentiable pipeline for tensor representation learning.
To our knowledge, no prior work combines manifold kernels. uncertainty modeling, and transformer-basee encoders in a unified end-to-end framework, and our experiments demonstrate that these innovations materially improve performance, particularly in multi-modal fusion settings (Table 2). This confirms that UKTL's contributions are substantive, novel, and directly advance the state of the art in structured tensor learning.
F.2 RELATION TO BROADER TENSOR LEARNING LITERATURE
The broader literature on temporal tensor learning and tensor time-series models (Fang et al., 2022; Tao et al., 2023; Wang et al., 2023; Chen et al., 2025a), including recent advances in continuous-time Tucker decomposition, probabilistic tensor dynamics, and structured temporal f'ctoriiation, focus primarily on sparse reconstruction. forecasting, and generative modeling of temporal tensors through low-rank factorization or probabilistic temporal priors. While UKTL addresses different problem, discriminative recognition (e.g., complex human motions / actions) rather than generative temporal modeling, acknowledging these studies helps situate our contribution within the larger landscape of temporal tensor research.
Unlike decomposition-basee temporal tensor models, UKTL adopts a kernelized, discriminative perspective. By operating on mode-wise tensor subspaces and introducing uncertainty-awart projection kernel with differentiable Nyström linearization, our framework enables end-to-end su- pervised learning directly structured tensor sequences. This design allows UKTL to compare and discriminate high-order temporal representations through subspace geometry, rather than modeling or forecasting the tensor dynamics themselves. Consequently, UKTT complements prior temporal
21

---

Published as a conference paper at ICLR 2026
Table 3: End-to-end efficiency comparison on NTU-60.
Model
#Params
FPS
Memory (train)
CTR-GCN (Zhang et al., 2025a)
1.46M
520
2.6 GB
ST-TR (Plizzari et al., 2021)
10 12M
410
3.2 GB
UKTL (Ours)
1.2 1.5M
450 - 480
2.9 GB
lensor models: it uses tensor structure for discriminative tasks, whereas existing approaches focus on generative or predictive objectives.
F.3 COMPUTATIONAL COMPLEXITY ANALYSIS
Below, detail the per-iteration computational cost of UKTL and isolate how the number of Nyström pivots C' and the subspace dimension p influence both training and inference.
Tensor encoder. For sequence with T frames and J joints, feature extraction is dominated by the MLP and HoT layers. (i) MLP complexity: JdT) which is constant w.r.t p and C. (ii) Higher- order Transformer (HoT): HoT attention operates hyper-edges of order With hidden size d, H attention heads, and Ng- (2) hyper-edges, the complexity is O(H T .g..d).. Since this encoder is shared across all tensor-based baselines, it does not constitute the differentiating computational cost of UKTL.
Mode-wise SVD for subspaces. For each tensor X; E Rdd XNxxr, mode-m unfolding yields the following matrix sizes: Mode-1: d' x (NęT), Mode-2: Ng (d'r), Mode-3: 7 (d' (d' Ng). We compute only the top-p left singular vectors via truncated SVD, The complexity per mode is Opp d' NeT). Since there are three modes, the total SVD cost is O(3p d' Ne))), indicating that the subspace dimension enters linearly.
Uncertainty estimation. For each mode, uncertainty estimation operates on the low-rank form of the projection matrix UUT, using p x p statistics. The complexity per mode is Opp²), and over all three modes: (33²)). ). This cost is negligible compared SVD and kernel computations.
Kernel compulation against Nyström pivots. For each training sample, kernel values are computed against C Nyström pivots. single Grassmann kernel evaluation consists of (i) projection matrix difference: (ppmm) and (ii) Frobenius norm between two rank-p matrices: O(p²) ). The dominant term is (pp)). Thus, the kernel computation cost per iteration is O - CMM..MMM. -p", where M
is the number of tensor modes. This simplifies to (N CCpp²) p²)). Hence, the pivot count C affects complexity linearly, while the subspace dimension p affects it quadratically.
Nyström eigen-update The pivot-pivot kernel matrix Kee RCxC requires eigendecomposi- tion every few epochs, with complexity O(C*). This cost amortized over training iterations and remains moderate since C < N.
Final complexity summary. The overall per-iteration complexity of UKTL is (p d' NgT + Nuatch C. p²). The subspace dimension contributes linearly through SVD and quadratically through kernel evaluation, while the number of pivots C' contributes linearly during training and cubically during infrequent pivot updates.
During inference, only kernel evaluations against the pivots are required, yielding a complexity of O(Cp²) ). This aligns with the observed empirical behavior: performance saturates beyond 8-10 and C 150-180 (Fig. 3), offering the best accuracy-efficiency trade-off.
End-to-end efficiency comparison. We evaluate the end-to-end efficiency of UKTL against repre- sentative strong baselines, including the graph-based CTR-GCN and Transformer--asee ST-TR, on the NTU-60 dataset. We report approximate throughput (frames per second, FPS), parameter count, and peak training memory usage, measured under the same hardware and batch-size settings.
Several observations can be drawn from Table 3. First, UKTL achieves throughput within approx- imately 10% of strong deep baselines, despite incorporating tensor subspace modeling and kernel- based operations. Profiling indicates that runtime is dominated by the shared MLP HoT back- bone, while the muccertainty-war tensor kernel contributes only a minor overhead. Second, UKTL maintains parameter count comparable to
and is order of magnitude smaller than
22

---

Published as a conference paper at ICLR 2026
Table Sensitivity analysis of the mixture weight ju between sum and product interactions.
1
0.25
0.3
0.4
0.5
0.6
0.75
1
Acc (%)
91.8
92.6
92.7
93.1
93.1
92.6
92.4
81.6
Transformer-basee models, reflecting the parameter efficiency of kernelized representations. Third, while UKTL incurs slightly higher memory usage than CTR-GCN due subspace and kernel com- putations, it remains more memory-efficient than Transformer variants.
These results demonstrate that UKTL scalable and computationally competitive, achieving strong performance without introducing prohibitive runtime or memory costs relative to widely adopted state-of-the-are baselines.
F.4 SENSITIVITY ANALYSIS THE MIXTURE WEIGHT
The mixture weight j in Eq. 14 is treated as a learnable scalar parameter, initialized to 0.5, and opti- mized jointly with the remaining model parameters. We intentionally make y learnable because the optimal balance between additive and multiplicative interactions depends on the underlying dataset structure.
Table 4 reports the classification accuracy under different fixed values of μ.. Performance remains stable when μ lies in the range [0.25, 0.6]. Using product-only interactions (μ 0) achieves strong performance but exhibits reduced robustness, while sum-only interactions (μx 1) suffer from the loss of multiplicative structure. Importantly, the learned μ consistently converges this optimal range, eliminating the need for manual tuning. On NTU-60, the learned value converges to 0.41; on NTU-120, converges 0.47; and on Kinetics-Skeleton Skeleton, converges 0.58. This indicates that product interactions dominate in NTU-60 and NTU-120, whereas additive contributions become more important for Senetics-Skeletee due to noisier skeleton observations.
F.5 CLARIFICATION THE INITIAL MIP
We clarify that the initial MIP is not a simple flattening operation; rather, it a learnable joint en- coder specifically designed produce meaningful embeddings for the HoT module and subsequent tensor-mode processing.
Function of the MLP. Given per-frame joint inputs X; E Jx3 the MLP maps each joint inde- pendently into a latent feature space: Jta MLP( E Rd. This preserves the joint-level structure, maintaining spatial locality across joints while producing embeddings suitable for higher- order attention. Crucially, it does not flatten the skeleton, so each joint remains distinct token for hyper-edge construction.
Necessity of the MLP. The HoT module operates on three-joint hyper-edges to capture higher-order spatial-temporaa correlations. Constructing meaningful hyper-edge embeddings requires each joint to reside in consistent latent space. A naïve flattening of the skeleton would destroy the per- joint structure, eliminate learnable joint-specific embeddings, and reduce generalization to unseen subjects or motion scales. In contrast, the MLP functions localized joint encoder, analogous to the per-token embedding layer in transformers, providing a trainable representation for each joint that HoT can combine into hyper-edge features.
Architectural details. The MLP used in our model is a three-layer network following the architecture described in the main paper. This produces feature maps X, R for each temporal block t, which then serve as inputs to the HoT module. These per-joint embeddings are subsequently organized into hyper-edge features for downstream tensor-mode processing.
Therefore, the MLP is a learnable joint encoder, not a flattening layer, producing per-joint latent features required for higher-order attention and tensorial subspace learning.
23

---

Published as a conference paper at ICLR 2026
F.6 CLARIFICATION ON UNCERTAINTY VECTORS
In our method, uncertainty vectors refer to the per-mode trust (or confidence) scores produced by the uncertainty module after the mode-wise SVD. They quantify how reliable each mode's 's subspace representation is.
Where they come from. After computing the mode-wise projection matrices ÓT = UT E R/. XIm the uncertainty module processes each UT through a single fully connected layer: Um = )FC(("")) E Rº (we write um here for simplicity), where m E 21, indexes the tensor modes, p is the subspace dimension, and a is a scaled sigmoid activation. Thus, each uncertainty vector um is p-dimensional vector of confidences for the latent subspace directions of mode m.
What they represent. Each uncertainty vector Umm. measures the reliability of each basis direction of the subspace, down-weije weights noisy unstable components, and enhances well-conditioned di- rections that correspond to discriminative motion patterns. This provides a fine-grained, direction- aware trust estimate rather than single scalar confidence. Intuitively, high value indicates stable and meaningful geometric direction, while a low value indicates a noisy or unreliable axis of variation.
How they are used. The uncertainty vectors weight the projection matrices when constructing kernel features: U. = Um @ Um (Eq. 25; here tmm fam see also Eq. 15), where @ denotes element- wise scaling across basis directions. This allows the kernel to emphasize trustworthy structure and suppress unreliable subspace components. This is why the kernel is termed UKTL: the uncertainty vectors act as trust gates each tensor mode before embedding into the Grassmannian kernel space.
F.7 COMPATIBILITY WITH DUSK
DuSK (He et al., 2014) uses a CP decomposition and constructs a dual structure-pressrrving ker- (m) (mm) where (m!) and !!! are mode-
nel: İDUSK( Y) ΣR D s=1 Arps I... km
(m)
specific CP factors.
Because UKTL already produces mode-wise low-rank factors (U1, U2 U3), these can directly serve as inputs to the mode kernels (...) inside DuSK. In other words, UKTL provides the factorized, mode-specific representations required by DuSK, enabling replacement the Grassmann factor kernels with minimal modification to the rest of the pipeline.
What would change if DuSK used? Only the factor-wise kernel needs to be replaced. Our projection- based kernel kproj (m!m) (:(m)) can be substituted by the DuSK mode kernels km((.) applied to the corresponding mode factors. All upstream components (MLP, HoT, SVD, uncertainty modeling) and the Nyström linearization remain unchanged, as they only require the kernel to be positive definite over the mode-wise representations.
Why we choose the Grassmann kernel. We adopt the Grassmann projection kernel because it aligns naturally with subspace geometry, integrates uncertainty in principled manner, and enables stable Nyström approximation, which is essential for scalability on large datasets such as action and motion tensors. DuSK is powerful but computationally heavier due to its CP-factor summation structure and was originally designed for static neuroimaging tensors rather than dynamic sequences such as human skeleton data.
UKTL does not rely on a specific kernel family. DuSK can be incorporated as an alternative factor- ized kernel with minimal changes.
G LIMITATIONS AND FUTURE WORK
While proposed UKTL framework demonstrates strong performance and sets new benchmarks on multiple action recognition datasets, several limitations remain, which also open avenues for future research.
First, our method relies on predefined hyperparameters such as the number of pivots in Nyström approximation, the subspace order p, and kernel fusion weights. Although HyperOpt to
24

---

Published as a conference paper at ICLR 2026
automate their tuning, this process be computationally expensive and dataset-dependent. Future work could explore adaptive or learnable strategies, e.g., (Ding et al., 2025b;a), to determine these parameters in data-driven and efficient manner.
Second, while our use of gensor-mode subspaces provides compact and discriminative representa- tions, the current approach assumes fixed temporal block structure across all sequences (Ding et al., 2025c). This may limit the model's 's ability to capture fine-grained temporal variations or han- dle actions with varying durations effectively, e.g., (Wang et al., 2024a; Chen al.. 2025b; Ding & Wang, 2025b;a). Future extensions could incorporate dynamic temporal block segmentation or attention-based mechanisms to allow flexible and context-aware temporal modeling.
Third, the kernel functions used in our model are predefined (e.g.. sum- num-product kernels). Although they are expressive and show superior performance, learning the kernel function directly from data, such as through neural kernel learning or meta-kernel search, could further enhance model adapt- ability and generalization.
Fourth, our current framework is designed and evaluated primarily on structured datasets with clean annotations (e.g., skeletons). real-world applications, for example, skeleton data may contain significant noise, occlusions, missing, joints. Incorporating robust strategies for uncertainty es- timation, noise modeling, or missing data imputation would make the method more applicable to real-world deployment, such in surveillance healthcare scenarios.
Finally, while the proposed KTL model is lightweight compared to some deep architectures, the overall pipeline involves multiple components (e.g., decomposition, kernel approximation, and un- certainty modeling), which could limit its scalability to extremely large datasets or real-time appli- cations. Future research could investigate end-to-ent differentiable approximations of each module to simplify the pipeline and improve inference speed.
H LLM USAGE DECLARATION
We disclose the use of Large Language Models (LLMs) as general-purpose assistive tools during the preparation of this manuscript. LLMs were used only for minor tasks such grammar and style improvement, code verification, and formatting suggestions. No scientific ideas, analyses, experimental designs, or conclusions were generated by LLMs. All core research, methodology, experiments, and results were performed and fully verified by the authors.
The authors take full responsibility for all content presented in this paper, including text code suggestions that were refined with the assistance of LLMs. No content generated by LLMs was treated original scientific work, and all references and claims have been independently verified. LLMSs did not contribute in a manner that would qualify them for authorship.
25


Reviews:
[{"review_id": "atNu1EjVdt", "reviewer": "Reviewer_uWeY", "summary": "This paper introduces Uncertainty-driven Kernel Tensor Learning (UKTL), a scalable kernel framework for learning from structured tensor sequences, particularly applied to skeleton-based action recognition.\nThe central idea is to represent each tensor by its mode-wise subspaces obtained via unfolding and SVD, and then define product and sum Grassmann kernels to measure similarity across these subspaces.\nTo handle scalability, the authors employ a Nyström kernel linearization with dynamically learned pivot tensors via soft k-means clustering, and further propose uncertainty-aware subspace weighting (Multi-mode SigmaNet) that adaptively down-weights unreliable tensor modes.\n\nThe model is trained end-to-end, integrating a lightweight encoder (MLP + Higher-order Transformer), the proposed kernel layer, and a classifier.\nExperiments on three large-scale benchmarks (NTU-60, NTU-120, and Kinetics-Skeleton) show that UKTL achieves state-of-the-art performance, outperforming strong graph, hypergraph, and transformer-based baselines. The paper also provides ablation studies on kernel composition, pivot selection, and subspace order, as well as visualizations of Tucker components for interpretability.", "questions": "- Could the authors provide a formal analysis (in big-O notation) of the computational complexity per iteration, especially how Nyström pivot count C and subspace dimension p affect training and inference?\n\n- How sensitive is performance to the mixture weight between sum and product kernels? Is μ learned or manually tuned?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The principled formulation bridging tensor subspaces and kernels is well-formulated. The idea of defining kernels over mode-wise Grassmann subspaces is conceptually elegant and mathematically well-grounded, combining multilinear structure preservation with nonlinear kernel flexibility.\n\n- The introduction of the Multi-mode SigmaNet to learn mode-wise uncertainty is a thoughtful innovation that improves robustness and interpretability, addressing a long-standing issue in tensor learning where all modes are treated equally.\n\n- The experiments are extensive and convincing. UKTL consistently surpasses graph and transformer baselines across three challenging datasets. The ablation results are well-structured and demonstrate the individual benefit of each component (subspace order, kernel choice, uncertainty weighting, pivot count).", "weaknesses": "- The paper combines multiple ideas—Grassmann kernels, Nyström approximation, HoT encoders, and uncertainty modeling—making it dense and difficult to parse. The derivations are mathematically sound but the overall design is somehow overcomplex. What's more, as each component (tensor subspace kernel, Nyström approximation, uncertainty weighting) has precedent, the paper’s contribution lies in their combination and engineering coherence rather than a fundamental breakthrough. \n\n- The related work section omits a substantial body of research on temporal tensor learning and tensor time-series models， which  have also been extensively studied in ML community, focusing mainly on sparse reconstruction and forecasting tasks through low-rank tensor factorization or probabilistic temporal models like[1][2][3][4]. While UKTL focuses on action recognition, more discussion on these studies would help situate the contribution more clearly within the broader literature on tensor-based temporal representation learning, and highlight the discriminative, kernelized nature of the proposed approach.\n\n- Despite its potential generality, all experiments are in one domain of skeleton action recognition. Without tests on other structured tensor data (e.g., video or neuroimaging), the claimed “multi-modal generality” remains unproven.\n\n\n[1]: Fang, et al. \"Bayesian continuous-time Tucker decomposition.\" ICML 2022.\n[2]: Tao, et al.. \"Undirected probabilistic model for tensor decomposition.\" NeurIPS 2023\n[3]: Wang et al. \"Dynamic tensor decomposition via neural diffusion-reaction processes.\" NeurIPS 2024\n[3]: Chen, et al. \"Functional Complexity-adaptive Temporal Tensor Decomposition.\" NeurIPS 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MwPzvGX8au", "reviewer": "Reviewer_o5ts", "summary": "The paper proposes Uncertainty-driven Kernel Tensor Learning (UKTL), a framework for learning from high-order tensor data such as skeletal action sequences. Unlike traditional tensor or kernel methods that flatten data or ignore mode-specific variability, UKTL compares tensors through mode-wise subspaces obtained from tensor unfoldings, preserving spatial, temporal, and semantic structure.\n\nThe method builds a kernel that captures both independent and joint correlations across tensor modes. To scale to large datasets, the authors use a Nyström approximation with dynamically learns pivot tensors—cluster centers that represent “local modes” of the data distribution. Anothr module estimates uncertainty for each mode, weighting subspaces based on their reliability. \n\nExperiments on NTU-60, NTU-120, and Kinetics-Skeleton show that UKTL surpasses graph, hypergraph, and transformer baselines while maintaining efficiency.", "questions": "Please refer to the weakness above, which highlight what I believe are the main limitations to be addressed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a novel approach for learning from structured, high-order data with original compnents such as uncertainty-driven subspace weighting and sum–product Grassmann kernels.\n\n- The theoretical formulation is sound and is a valuable contribution bridging kernel theory,and uncertainty modeling.\n\n- The paper is clearly written and  well-organized", "weaknesses": "1. Evaluation is limited to skeleton data. This is a key limitation which hinders the generality of the method. Although these datasets are standard and challenging, they represent a very narrow class of structured motion data. Demonstrating the approach on a different tensor domains (uch as video features, fMRI signals, or audio visual data) would better support claims of generality and broad applicability. Even a small-scale study in a non-skeletal domain would reinforce the method’s versatility.\n\n2. The use of the term 'mode' should be better clarified from the very beginning, since multi-modality is usually more often used to imply multiple sensor modalities  (also 'mode' is used in yet another fashion in the paper when the authors introduce 'local modes' which as I understand are 'modes within modes' ).  Maybe, clarifying the distinction between tensor-mode structure and cross-modal fusion would avoid overstated claims. This brings to the next point:\n\n3. As already mentioned, while the paper repeatedly refers to “multi-modal” learning, the experiments involve only skeleton data, where “modes” correspond to tensor dimensions (spatial, temporal, coordinate). I believe the framework is also well-suited to genuine multimodal sensor fusion (e.g., RGB + depth + skeleton, avaialble in the NTU datasets by the way). However, such potential is not empirically demonstrated. This point relates also to point n. 1.\n\n4. The full pipeline (MLP + Higher-order Transformer + SVD decomposition + uncertainty module + Nyström kernel) is intricate. While each component is (mostly) motivated, a complexity analysis comparing runtime and memory with strong deep baselines (e.g., CTR-GCN or transformer variants) is missing and would help quantify scalability.\n\n5. The initial MLP: I would appreciate some more details on this module, which at first glance looks like a standard \"flattening\" operation for the input tensors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8OZ6STUYpi", "reviewer": "Reviewer_5vU1", "summary": "The authors present a tensor-based learning approach with uncertainty modeling.", "questions": "- I would like a more detailed description of the HoT layers. The hyperedges are undefined on the same page.\n- What is the index H in equation (1)?\n- What are uncertainty vectors? \n- Is the method also applicable for different kernels, such as the Dusk kernel? (DuSK: A Dual Structure-preserving Kernel for Supervised Tensor Learning with Applications to Neuroimages Lifang He, Xiangnan Kong, Philip S. Yu, Ann B. Ragin, Zhifeng Hao, Xiaowei Yang)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The problem of tensor learning is very interesting.", "weaknesses": "The boilerplate goes until page 4 using space that seems to be missing for details later. I find the description of the uncertainty vectors and the other material in Sections 3 and 4 not easy to follow. Many details are missing and the description is too brief to be able to follow the discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)