Functional Virtual Adversarial Training for Semi-Supervised Time Series Classification
Authors: Qingyi Pan
Yicheng Li
Venue: NeurIPS 2025 poster ()
Keywords: Semi-Supervised classification; Virtual Adversarial Training; Function Space; Sobolev Space
Primary area: general_machine_learning
TL;DR: We propose functional Virtual Adversarial Training (f-VAT) that incorporates the functional structure of data into perturbations.

Abstract: Real-world time series analysis, such as healthcare, autonomous driving, and solar energy, faces unique challenges arising from the scarcity of labeled data, highlighting the need for effective semi-supervised learning methods. While the Virtual Adversarial Training (VAT) method has shown promising performance in leveraging unlabeled data for smoother predictive distributions, straightforward extensions of VAT often fall short on time series tasks as they neglect the temporal structure of the data in the adversarial perturbation. In this paper, we propose the framework of functional Virtual Adversarial Training (f-VAT) that can incorporate the functional structure of the data into perturbations. By theoretically establishing a duality between the perturbation norm and the functional model sensitivity, we propose to use an appropriate Sobolev ($H^{-s}$) norm to generate structured functional adversarial perturbations for semi-supervised time series classification. Our proposed f-VAT method outperforms recent methods and achieves superior performance in extensive semi-supervised time series classification tasks (e.g., up to $ \approx 9$ % performance improvement). We also provide additional visualization studies to offer further insights into the superiority of f-VAT.

Functional Virtual Adversarial Training for Semi-Supervised Time Series Classification
Qingyi Pan Department of Statistics and Data Science
Yicheng Li Department of Statistics and Data Science Tsinghua University Beijing, China
Tsinghua University
Beijing, China
pqy228mails tsinghua.
liyc22@mails 1rse22mmmils..ssinnnaa.aaa twwignua.. edu. en
Abstract
Real-world time series analysis, such as healthcare, autonomous driving, and solar energy, faces unique challenges arising from the scarcity of labeled data, highlighting the need for effective semi-supervised learning methods. While the Virtual Adversarial Training (VAT) method has shown promising performance in leveraging unlabeles data for smoother predictive distributions, straightforward extensions of VAT often fall short on time series tasks they neglect the temporal structure of the data in the adversarial perturbation. In this paper, we propose the framework of functional Virtual Adversarial Training (f-VAT) that can incorporate the functional structure of the data into perturbations. By theoretically establishing duality between the perturbation norm and the functional model sensitivity, we propose to use an appropriate Sobolev (H +) norm to generate structured functional adversarial perturbations for semi-supervised time series classification. Our proposed f-VAT method outperforms recent methods and achieves superior performance extensive seni-supervised time series classification tasks (e.g.. up to ss 9% performance improvement). We also provide additional visualization studies to offer further insights into the superiority of f-VAT.
Introduction
Time series analysis has attracted considerable attention from both academia and industry, due to its relevance critical domains such electrocardiogram (ECG) interpretation in medical diagnosis [14] and photovoltaic module power calibration in solar energy systems [35]. However, capturing intrinsic temporal structural properties of time series data, such noisy fluctuations, long-term trends, and periodic patterns, is still challenging (22), particularly with scarce labeled data [41] Because manual annotation is labor-intensive and costly with massive unlabeled data, semi-supervised time series classification methods have become a promising research direction.
In this regard, existing methods attempt leverage massive unlabeled data and limited labeled data to alleviate overfitting. Some studies generate high-confidence pseudo-labels for unlabeled samples and then train deep model the expanded dataset 6 36). Although such methods can iteratively improve predictive performance, they require careful selection of confidence thresholds and remain susceptible "confirmation bias": earlier errors in pseudo-labels are likely be reinforced during training, making difficult for deep models to correct early biases or capture global trend information [10 2 Some recent studies enhance pseudo-labeling with strong data augmentation and self-supervised representation learning to better utilize unlabeled data. For instance, TS-TCC (18) generates the pscudo-label on a weakly augmented view, while encouraging consistent representations
*Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

under strong augmentation. However, naive data augmentation methods (e.g.. local cropping, random shifting) [4] can disrupt key temporal information, thus affecting generalization.
Recently, Virtual Adversarial Training (VAT) based consistency regularization 31 constructs ad- versarial perturbations that maximally change the predictions of the model and then penalize the difference between the original and perturbed outputs, thus enforeing local smoothness and effectively tightening the decision boundary in the data manifold. While VAT and its variants have shown promis- ing performance in the fields of computer vision KT and natural language processing [28] most of these methods are based on the classical Euclidean norm to bound perturbations. Unfortunately, straightforward extension of VAT to semi-supervised time series classification often proves less effective because bounding virtual perturbations via the Euclidean norm ignores the time series nature of the input data, producing jagged and spiky anomalous patterns that disrupt low-frequency trend information. Consequently, adversarial perturbations generated by standard VAT are not truly worst- case for time series data, making VAT inefficient in improving smoothness of predictive distributions and predictive performance.
To address these challenges, we propose functional Virtual Adversarial Training (f-VA7) for semi- supervised time series classification. F-VAT constructs adversarial perturbations in various (possibly) infinite-dimensional spaces, such as the Sobolev spaces M. which can better capture the underlying structure of the data. For time series, this approach enables us generate adversarial perturbations that can simultancously preserve low-frequency trend information and flexibly explore input space, facilitating smoother predictive distribution and alleviating severe overfitting to limited labeled data.
In this paper, we theoretically establish the duality between the perturbation and the smoothness of (non-)linear functional models. Based theoretical analysis, for time series data, we use an appropriate Sobolev norm to generate functional adversarial perturbations that preserve low-frequency trend information while avoiding "jagged" anomalous patterns. We conduct extensive experiments on real-world datasets to verify that F-T significantly outperforms other competitive baselines (e.g.. up to 9.42% on CricketX and 8.30% on SelfReg). Further visualization indicates that, compared to the original VAT, our proposed functional adversarial perturbations lead to more stable convergence and better final performance.
In summary, our main contributions are as follows.
We propose the framework of functional Virtual Adversarial Training (f- VA-V that allows us to construct perturbations in various function spaces. For linear and non-linear models, we theoretically establish the duality between the perturbation norm and gradient sensi- tivity (Theorem 3.3 and Theorem 3.4), showing how to generate structured adversarial perturbations.
For the semi-supervised time series classification problem, we propose to use an appropriate Sobolev norm (H-* norm in Section 3.3 to generate structured adversarial perturbations. The Sobolev norm allows us to properly control the model's sensitivity to high-frequency noises while capturing low-frequency trend information.
Extensive experimental results (Section 4.2p on semi-supervised time series classification are provided to demonstrate the superiority of f-VAT over existing methods. Additional visualization results (Section 4.3) indicate that functional adversarial perturbations can significantly smooth the loss landscape to achieve stable convergence and better performance.
2 Preliminaries on Virtual Adversarial Training
Let us briefly introduce the procedure of Virtual Adversarial Training (VAT) [31]. We consider the semi-supe -umi-supereieed classification problem with a small labeled dataset and large unlabeled dataset. Let (X1, .,........................ Xx) E X denote the entire input data, where each X, can be associated with a label y E = 21, ... K if it labeled, or z/ is unknown if it is unlabeled. We denote by D the labeled dataset, n the unlabeled dataset, and D D' U DM the entire dataset. Our goal is to leam the conditional probability (vector) p(- 0) = (X) E K
The idea of VAT is to generate adversarial perturbations for each sample X; and then penalize the difference between the original and perturbed outputs, facilitating the local smoothness of predictive distribution. Formally, the Local Distributional Smoothness LDS(X,, ru: fe) is used to measure such

---

difference for the model outputs. In this paper, we mainly focus on the squared loss
LDS fo) (X, + r.) - (x;))
(1)
while another common choice is the Dullback-Leible (KL) divergence [I]. We adopt the squared
error, because its symmetry yields better numerical stability in the early stages of training [13]
The adversarial perturbation r; aims to maximize the LDS loss, characterizing the worst-case local perturbation for the model fo
arg max LDS(X,, se
(2)
where 0 is small hyperparameter controlling the perturbation norm in Euclidean space. Then, the VAT loss over X; is defined as
C (X, fo) = r = max relledl
LD:((,,,;:.o))
In practice, since the analytic expression of r; is intractable, an interactive gradient ascent 31 is used to approximate the optimal perturbation rī. The overall objective function combines the supervised loss Lo on the labeled set DD' and the VAT loss CVA on the entire dataset is
L(D: fo) = Lo(D); ; fo) + L (D;
where (VA(D: = +. E. CVA (X,; fe) is the VAT loss over the entire dataset D.
3 Functional Virtual Adversarial Training
One key aspect of VAT is the choice of the nomn used to bound the local perturbation ||||| S in Eq. ( In the literature [1. seems to be straightforward to use the Fuclidean norm ||-1|2 as the input is typically represented vector in R.d However, when the input space X' consists of functions, which is the case for time series data, the choice of the norm can become more intricate. As we will see the following theoretical analysis, the choice of the norm can have a significant impact on VAT loss and its interpretation. In this section, for simplicity, we consider one-dimensional output f: + IR and focus on some fixed sample zig, while our results can be easily extended to the case where f is vector-valued function.
3.1 The Duality Perspective of VAT
To start with, let first consider the setting where the input æ is vector in Rd, but consider a more general setting where the perturbation is bounded by a general norm. Let Σ be positive definite matrix, we denote the E-norm by ||ul| = Ve Ev. Replacing the Euclidean norm in Eq. 2 with the Σ-norm, we introduce the following VAT loss:
C f)
sup LDS(2o. ff)- re
(5)
To illustrate the impact of the -nnorm on the VAT loss, let us first consider the case where f(x) is linear function. We have the following result.
Proposition 3.1. Let f(z) = x) be a linear model. Then, the VAT loss Eq. 5 is equivalent to
c = =
(6)
Proof. Since f(x) is linear, we have
2 NA (No, f)
1(3, mp) - -(3, x0 + +)) sup 1(,,))1
sup r
rc
This supremum is achieved when is in the direction of Σ 1,, i.e.,
= - - Bb15 -I / |/11
and thus
(2o, ()) = sup
((,r,) -1, = =1113112
rc
3

---

Moreover, the result can be extended to the case where f()) is continuously differentiable function as follows. The proof is deferred Appendix [A.3] Proposition 3.2. Let (x) be continuously differentiable. Then, we have
limn E 3-2 c-0+
The results from Proposition 3.1 and Proposition 3.2 reveal simple but profound duality structure underlying the VAT loss. First, as the gradient (a(aa) reflects the sensitivity of the function, the VAT loss can be interpreted as a measure of the model's 's sensitivity to perturbations in the input space. Moreover, the use of the E-norm the perturbation constraint ||/||s S e naturally introduces dual norm - penalizing the sensitivity of f along the different directions. This is reminiscent of duality optimization, where constraints the primal space translate to penalties in the dual space.
Therefore, one of the crucial questions is how to choose the matrix Σ when designing the VAT loss. If Σ is isotropic (e.g., Σ T), the VAT loss reduces to the Euclidean sensitivity [V((z)) l If E is anisotropic, the VAT loss prioritizes robustness along directions where Σ assigns lower curvature. This coincides with our intuition: allowing larger adversarial perturbations results in more constraint on the model's sensitivity in those directions. While seems to be trivial when Σ a diagonal matrix that simply scales the input, the choice of Σ can be more complex and encompasses structural information about the input space, which would lead to substantial differences the VAT loss. This is what will see under the time series data.
3.2 VAT under Functional Inputs
In the setting of the time series data (24], the input æ is viewed as function some interval T. which would require more sophisticated treatment the theoretical analysis. Let L - (T) be the Hilbert space square integrable functions on and denote the inner product by (..)) We suppose that the input E (T)). For the perturbation, let us take another Banach space E and denote by ||- lg the norm E. Typically, we can choose E to be a Sobolev space, which will be discussed in the next subsection. The dual space E* of E is defined the space of continuous linear functionals on E, which equipped with the dual norm ||-1|g- defined by
II
sup =EB:
[g(x))-
To avoid technicalities, will only provide informal results here. For formal treatment and rigorous proofs, we refer the readers to Appendix i appendix.
The functional virtual adversarial training (f-VAT) introduces more general form of the VAT loss, which is defined
C (xo, fi
sup PEL
LDS(
(8)
Our first result parallels Proposition 3.1 in the functional setting.
Theorem 3.3 (Informal). Let f(x) (3, ())x))1(7 la(T)- 3 (T())) be a functional linear model. Then, the loss Eq. 8 is equivalent to
c f =1
(9)
where | ø- can be infinite if 3 E
Then, the next result the functional version of Proposition 3.2 Theorem 3.4 (Informal). Let f: LL((T) R. be differentiable xo E LT)... Then,
lim e 2 2 f) = [Vf(=o)) c0
(10)
Theorem 3.3 and Theorem 3.4 establish the duality of VAT under general functional spaces, showing that a constraint over the E-norm of the perturbation results penalizing the dual space E* -norm of the model's sensitivity. These results also recover Proposition 3.1 and Proposition cases, since I| and I|-1ly are dual under the canonical Fuclidean norm. While finite dimensional spaces under different norms are equivalent, the infinite dimensional spaces be substantially
special

---

different. The functional setting generalizes the vector-based analysis in the preceding subsection, but its ciimite-dimensional nature allows for richer specification of perturbation constraints that align with the geometry of functional data.
These findings offer several key insights into the behavior and potential of VAT when applied to functional data, such as time series, A critical implication of these results lies in the role of the space E (or equivalently, the norm |- which defines the constraint of the perturbation r. It known that the stronger norm of E' (thus a "smaller" space) leads to weaker norm of E (thus a "bigger" space), which can be seen from the duality between 11-1ls and I|-ls--.. We will use the Sobolev spaces to show a concrete example in the next section. Consequently, the result highlights that, if we want to impose a stronger norm on the sensitivity of the method, we should, in contrast, relax the norm of the perturbation.
Moreover, this perspective also has practical implications for designing VAT-based training procedures. The choice of E and thus E" determines the directions and structure of perturbations against which the model is made robust. For time series data, where the inputs are functions over time interval T, this flexibility is particularly valuable. Time series often exhibit structural properties such as smoothness, periodicity, or specific temporal dependencies. By selecting space that captures these characteristics-forr instance, a Sobolev space that penalizes high-frequency oscillations through derivative- based norms- the VAT loss can be tailored to prioritize robustness under a particular sense. This could be especially beneficial in applications where high- high-frequency components are considered noise, which encourages the model focus on the underlying smooth signal.
3.3 Structured Adversarial Perturbations via Sobolev Spaces
Sobolev spaces 1 are powerful tool in functional analysis, particularly useful for analyzing the smoothness and regularity of functions. Using Sobolev spaces, we can define a structured norm for the F-VAT constraint that captures the smoothness properties of the perturbations. In this subsection, we will briefly introduce the Sobolev spaces, their duality, and practical ways to compute their norms with discrete points. We refer the readers Appendix B in the appendix for more details.
For a positive integer s, the Sobolev space H* comprises functions in (T) whose weak derivatives up order s are also in (T). The weak derivative generalizes differentiation to functions lacking classical smoothness, enabling control over their regularity. A function with classical continuous derivatives up to order is in H+(T), but the converse is not necessarily true. The norm in a Sobolev space is simply the sum of the I,²(T) norms of all derivatives up to order s and the inner product can be similarly defined, which make H+(T) Hilbert space.
More broadly, Ha (T) can be defined for any real a E R. Non-integer a captures fractional smoothness, while negative encompasses distributions rougher than (TT functions. Moreover, it is known that for s > 0, H² is densely embedded into LT)T"), while (T) is dense in H "TT).
The duality of Sobolev spaces can be easily identified with the index s. The dual of HT) is H HT) under the L (T) inner product. This duality enables us to frame the VAT loss using the dual norm
-I, assessing model sensitivity to perturbations constrained by || [| for > 0.
Computing the Norm with Discrete Observations. In practical settings, time series are discrete vectors in R For simplicity, we assume that the time series sampled N equidistant points, but the results can be extended to non-equidistant points with minor adjustments. Then, we can approximate the Sobolev norms using finite difference operators (.. Define the first-order difference matrix D1 E IR N with (Dır), = T'+1 r, and higher-order differences D2 = Df. For integer k, the squared H norm of l with discrete observations IN E N can be approximated as:
|r R AR,NN, where A = IN + D. D1 + D.J D2 +
D2 DR:
where IN is the N x N identity matrix. This sums the E2-norm of r and its discrete derivatives up to order k.
However, we actually will use the H norm for the perturbation, the above formulation is not directly applicable. For general a e R. we will use the spectral method [9] instead. is known that the Sobolev space Hd (T) can be represented via the fractional power of the Laplacian operator.
We have Ha(Rd) = {s = (T - -)))-/aa A)-a/2u u E (22(RR)) and ||/ = [|( - A)//2 | (
5

---

Therefore, let us take the discrete negative Laplacian matrix
-1
2
0
Ln
0
-1
-1 2
0
0
-1
can compute the 4a norm by
Ir RJ T + LN)"TN, a R
(11)
where (JN + In defined via the spectral theorem: IN + Ln = QAQ with diagonal A, then (IN + LN) = QA=QT, where A takes the exponent of each diagonal entry. This formulation is also valid for non-integer or negative s, where we note that (IN + LN) is positive definite and thus the diagonal entries of A are positive, This gives us practical ways compute the Sobolev norms with discrete observations.
3.4 The Algorithm
Let us now summarize the f-VAT procedure. Overall, we apply mini-batch gradient descent to minimize the total loss. In practice, since the analytic expression of optimal perturbation r" is intractable in Eq. 2). we approximate it using gradient ascent. We illustrate each step of updating in Algorithm where recall the labeled loss Lo ff); and LDS in Eq. (1). According to our theory in Theorem 3.3 and the duality of Sobolev spaces, we use H for the norm of adversarial
perturbation to impose smoothness constraint on the functional model. In this paper, we mainly use = 2 to penalize (weak) derivative up to the second order of the model, empirically verified by ablation studies in Fig,
Algorithm Functional Virtual Adversarial Training Step
1: Input: Data batch D, D', model fe. order the Sobolev norm 8 2 0, radius e, adversarial iterations L, learning rate y).
2: for each sample X; E D do
b Approximate r
3: Randomly initialize perturbation vector over [|rll Se
4: for 6 + L do
ri 5: Gradient ascent T4 77 (D)
6: Normalize r; < [|r.
7: end for
8: end for
9: 8 - - VoL((). where L(0) Lo(DD; + [D] Exed LDS(X,,
4 Experiments
4.1 Experimental Settings
We use dozens of publicly available datasets from the UCR and UEA repositories [T]. including the representative univariate dataset (i.c., CricketX, UWave, and InsectWing) and the multivariate dataset (i.e., SelfReg, NATOPS, and Heartbeat) difficult to easy. and widely-used in semi-supervised time series classification 19 21). Additionally,
[19]. These representative datasets are from
we construct more empirical results on several large-scale China Securities Index (CSI) datasets (i.e.. CSI 50 and 500 futures) spanning from 2020 to 2023 for predicting directions (upward or downward) of futures prices [44 33 The dataset collects records spanning from 2020 to 2022. Each time step contains bid/ask prices and corresponding volumes. Following [19, each dataset is split into train (60%), valid (20%), and test set (20%). We rescale each dataset into the range (0, for numerical stability. We refer readers to Appendix for more details.

---

4.2 Performance Evaluation
Semi-Supervised Performance Following 19 we compare F-V with six deep learning-baser methods, including recent SemiTime [9 and TapNet [3]. The baseline SupL. represents the deep model trained only on the labeled dataset. We adopt the baseline results to Table the original work [9 23 43 Following [40], we average the performance of each method over five runs with different random seeds and train/valid/test splits. T the mean and standard error from multiple runs.
Table I:
The semi-supervised classification accuracy (%) with standard deviation across six real-world datasets. Best performance in boldface.
Dutasct
Ratio
SupL.
Pl
MTL
SemiTime
TapNet
VAT
F-VAT
10%
44.88 #0.61
38.87 +2.28
40.94
#1.07
44,88
#3.18
39.42
+0.52
42.85
+3.57
49.18 =1.90
CricketX
20%
51.61
44.44
13.91
50.
1.:22
51.61
10.00
51.41
49.14
57.91
40%
58.71
53.39 #2.18
55.10
+1.12
58.71
+2.78
58.97 +0.72
58.63
68.39
10%
81.46 10.18
81.53 20.54
76.35
20:55
81.46
20.00
82.34 10.58
94.41 10.00
94,82 20:33
U'aave
20%
84.57
+0.87
81.66
+0.74
81.77
#0.94
84.57
+0.49
86.35
+-.48
95.53 +0.31
96.45 =0.27
40%
86,91
86.45 +1:20
86.91 +00.8
86,91
+0.47
89.24 +0.69
94.76
97.23
10%
54.96
#1.25
43.16 +3.20
50.45 1.00
54.96
#1.01
55.53 +1.18
55,49
58.01
InsectWing
20%
59.01
#1.13
48.35 #0.81
56.43 20.85
59.01 #1.00
60.36
1:0 .8
61.27 =0.1
61.28 =1.86
40%
62,38
#1.30
55.32 #2.04
60.90 =00.7
62.38 #0.76
63.87 #1.41
63.48 =0.30
64.81 =1:15
10%
46,49
#2.01
50.44 20.70
50,88
=:.01
49.68
#2.88
50.87 43.01
53,12 #4.51
59.31
=3.06
SelfReg
20%
52.44
53.94
52.19
1:2.00
52.63
11.31
$4.39
55.76
61.60
40%
51.31
13.48
55.69 12.74
56.14
1:2.01
49.56 11.72
54.38 10076
53.47 11.04
64.44
23.13
10%
68.98
12.89
75.83 1:4.9
73.91
1:3.3
68.52
10.81
70.37 17.12
82.38
10.95
86.04
11.41
NATOPS
20%
81.02 #1.00
82.51
+1.26
82,41
+2.89
80.09
#2.12
77.77 #1.89
82.81
+0.52
86.25
=1.88
40%
88.89
88.27 +1.19
90.27
#1.30
87.49
#2.41
82.87
42.12
90.15 #1.60
93,13
=0.15
10%
67.08
72.13 +1.00
71.61
+2.47
71.61
#1.71
72.84
73.86
76.25
Heartheat
20%
73.25 20.71
72.01
1.0.7e
73.66
74.49
11.43
73.24 11.88
71.59
1.0
76.46
11.06
40%
67.08
11.80
73.28
1:1.50
73.61
2:3007
72.43
73.66
10.71
75.00
20.11
77.28
20040
Table I shows that f-VAT consistently outperforms other competitive baselines across all real-world datasets reported in various label ratios. For instance, f-VAT achieves up to 9.42% performance improvements on CricketX and 8.30% on SelfReg with label ratio 0.4, while in the label-scarce scenario a 0.1, V-V still achieves up 6. 6.19% performance improvements on SelfReg.
To further verify the superiority of f-VAT, we randomly sample 30 datasets from UCR/UEA datasets [6]. Then, we add classical baseline meanTeacher (38 and a recent state-of-the-art Class-Aware Temporal and Contextual Contrasting (CA-TCC) 18 for fair comparison. Table 2 shows that f- f-VAT still consistently outperforms other competitive baselines all settings. Due the page limit, the details of more empirical results can be found in Appendix
Table 2: The average accuracy (%) and average rank under different label ratios.
Method
10%
20%
40%
AvgAcc
AvgRank
AvgAce
AvgRank
AvgAcc
AvgRank
Supl.
36.92
Pr
53.09
3.93
55.16
4.40
63.60
4.47
MTL
45.19
5.70
45,72
5.87
46.11
6.80
mean Teacher
50.94
4.13
SemiTime
56.53
3.57
58.93
3.77
69.02
3.13
TapNet
58.67
3.70
60.41
3.40
70.28
3.17
CA-TCC
E-VAT
65.85
1,50
68.87
1.50
76.24
1.53
Additionally, we construct more empirical results on several large-scale China Securities Index (CSI) datasets (i.e., CSI 50 and 500 futures). Table presents the performance of various semi- supervised methods across different label ratios, and shows that f-VAT significantly outperforms other competitive baselines on futures datasets, especially on more volatile CSI 500 futures. This is because f-VAT's adversarial perturbations incorporating key temporal structure facilitate deep models to effectively use unlabeled samples to yield smoother predictive distribution with better generalization.
7

---

Table 3: The performance comparison on domestic futures datasets like CSI 50 and 500 futures. Best performance in boldface.
Futures
Ratio
Supl.
Pl
MTI.
SemiTime
TapNet
CA-TCC
[-VAT
10%
21.3
42.87
41
53.94 20.
55.19 20.77
54.62 10.54
55.72
11:16
58.64
20.03
20%
45.08
11.45
47.26
10.73
54.97
10.1
56.69
10:50
$6.93
11.28
58.21
_0.41
62.09
10.26
40%
50.69
13.53
52.42
11:31
$6.97
10.57
57.34
10.20
59.75
11.08
_1.32
10:45
10%
34.23
20.24
44.00
+0.00
39.53 +1.35
38.86
+2.04
40.53 +0.83
39.79
=1.54
43.77
#0.75
500
20%
35.38
21.00
46.57 10.48
45.26
20.29
46.04
10.12
44.58
1:1.0
45.05
11.74
52.14
20.25
40%
43.85
=1.40
49.25
#0.86
47.66
+1.30
50.65
#1.06
51.30
#1.96
54.29
=0.87
58.66
#0.44
Fully- Supeevised Performance The proposed f-VAT can be easily extended to fully supervised settings. We compare f-VAT with several competitive supervised learning methods. ED [ the clas- sical one-nearest-neighhor classifier based on Euclidean distance. TapNet 43 and ShapeNet (26) are deep bearning-based methods that leverage manual shapelet-hased haaeelet-based features for better representations. Additionally, we include two non-neural methods called ROCKET 17 and HiveCOTE [4] The empirical results in Table 4 are taken from the original work [3] 43 26
We average the performance of each method over five runs with different random seeds, and report mean values with standard deviations.
Table 4: The performance comparison between VAT and other baselines in fully-supervise settings. We report the mean and standard deviation over five runs. Best performance in boldface.
Dataset
Hive-COTE
ROCKET
ED
TapNet
ShapeNet
VAT
F-VAT
CricketX
74.10
76.10 20.01
62.90
66.20
68.30
68.54 11.40
77.25 10.44
UWave
92.10
93.70 #0.04
88.10
89.40
90.60
92.43 #0.47
97.75 +0.18
Inne
62,20 #0.01
64.70 =0.01
60.20
67.30
66.30
70.01 +1.16
71.70
SeliReg
51.60
51.40
48.30
55.10
57.80 17,80-u +0.0s
58.75 +1.25
60.21
NATOPS
82.80
88.50 #0.44
85.10
88.30 +0.0s
87,58
+1.00
97.50
+0.01
Heartbeat
72.20 4. no
71.70
20.00
61.90
ac.ce
72.10
11.43
75.60 o2
76.08
20.92
78.75 40.41
Table shows that f-VAT consistently outperforms other baselines with an average improvement of 2.4% in all settings, especially for datasets containing long-term trend structures (i.e., up to 4.05% improvement in UWave). Also, f-VAT exhibits lower variance than VAT, as the functional adversarial perturbation imposes stronger regularization on the model's gradient sensitivity and improves the model's 's stability in fullv-supervised settings.
Altering Order of Sobolev Norm We further ex- plore how the order 8 of Sobolev norm influences the performance of f- VAV. Fig. Iillustrates that the performance of f-VAT in NATOPS with label ratio 0.4 under different s, where the error bar reports the standard deviation over runs. We observe that the model's accuracy peaks at s 2. This is because under under-regularization (s < 1) or over-regularization (s 2 3) both would degenerate predictive perfor- mance. Based on this, we choose
95
Accuracy (%)
90-
80
2 in the rest of experiments. Additionally, we evaluate the per- formance on more datasets in Table 5 where = 0 reduces to the original VAT. Table 5 shows that set- ting 2 achieves the best performance in almost all settings. For high volatile time series data, setting high order s = 3 generates adversarial perturbations that preserve low- cow-frequency temporal information, while for relatively stable time series data, the small order 1 allows the perturbation to flexibly explore input space. In practice, it suffices to evaluate E 3,42 based on the validation set and report the average performance
3
5
Order
Figure 1: Various order of Sobolev norm.
several runs.
Moreover, to verify the model-agnostic property f- F-V, we compare the performance different architectures trained by f-VAT in Appendix [) The empirical results suggest that Temporal Con- volutional Network (TCN) S outperforms other architectures most settings, so we adopt the eight-layer TCN main architecture.
8

---

Table 5: The performance of f-T with different Sobolev norm order s.
CricketX
UWave
InsectWing
NATOPS
SelfReg
58.63
94.76
63.48
90.15
53.47
59.91
96.54 +o.07
66.70
89.58 #0.12
56.16 +1.73
2 61 66 saas
97.16 20:25
67.08
93.13 2s
58.86 20.ss
60.44 202s
96, to.1s
64. 10 =0.00
90.10 mo.es
51.39 +1.21
58.22 =s.1p
96.71 +o.01
66.11 =0.52
87.51
#o.a2
50.93 eo.12
4.3 Further Analysis
In this subsection, conduct additional qualitative experiments further analyze the behaviors of the deep model trained by f-VAT in the semi-supervised time series classification.
Perturbation Visualization Fig. 2 visualizes vari- ous virtual perturbations on NATOPS with label ratio 0.4. The color line represents the original time series data, the color line adds ad adversarial perturba- tion generated by the original VAT, and the color line adds our proposed functional adversarial perturbation by VAV.T. As shown in Fig. 2 the functional adver- sarial perturbation closely aligns with the original sample across all key time steps and only adds subtle fluctuations to non-critical regions, while the origi- nal VAT's perturbation introduces large spikes that disrupt underlying trend structure. These observa- tions show that functional adversarial perturbations preserve trend information without introducing anomalous patterns. facilitating deep models to achieve smoother predictive distributions and more stable convergence.
Sample PVAT
1.0
0.0
500
800
700
800
900
Time
Figure 2: Visualization of perturbations.
100
Accuracy (%)
40
0.2
0.2
20
VAT
MVTT
0.2
0.1
0.0
0.2
0.1
50
100
150
200
Y
0.0
-0.1
0.1
X
-0.1
Epochs
-0.2
0022
Accuracy Curve
(b) VAT
(c) f-VAT
Figure 3: Visualization of performance curve (left) and the loss landscape of deep models trained by VAT (mid) and f-VAT (right) the UWave dataset with label ratio 0.4. The landscape of VAT contains many local minima, trapping the model in suboptimal solutions. The landscape of f- F-V is significantly smoother due to the adversarial perturbation in function space. Thus, f-VAT achieves faster and more stable convergence and better final results (left).
Loss Landscapes To validate the smoother predictive distribution offered by the functional adver- sarial perturbation in Sobolev space and demonstrate that f-VAT leads to easier optimization, we visualize the loss landscape of deep model trained by f-VAT and VAT using "filter normalization" [27] on NATOPS with label ratio = 0.4. From Fig. miid) we observe that the loss landscape of VAT highly chaotic, which causes the model be easily trapped in suboptimal minima, leading to an
unstable training curve. In contrast, deep models trained by f-VAT (as shown in Fig. 3 (right)) enjoy a significantly smoother loss landscape, where the functional adversarial perturbation the Sobolev space preserves low- oow-frequency trend information to provide more consistent gradient directions.

---

Consequently, f- VAT facilitates deep models for faster and more stable convergence with better final performance, demonstrated in Fig. (left).
Feature Importance Analysis To further analyze the behaviors of deep models trained by f-VAT semi-supervised settings, Fig. 4 visualizes the gradient-based feature importance map [34] on NATOPS with different label ratios. The sample corresponds to "Fold wings" action, whose feature importance significantly changes within the time interval [25,3] 35]. These observations show that as supervision signals increase, deep model trained by f-VAT captures essential "shapelets" (i.e., "elbow's movement to the right") [23] and reduces its reliance on irrelevant features (i.e., feature "shoulder rotation"), which aligns with existing knowledge that "Fold wings" action primarily relies on "elbow movement" with less reliance on "shoulder notation" [37]. This qualitative analysis validates the importance of capturing functional trend information within time series data and demonstrates that f- VAT is highly effective and efficient method for this purpose.
1.0
1.0
0.8
5
0.8
Festuress
Features
10
0.6
10
0.4
0.4
15
15
0.2
20
20
10
40
30
40
50
Time
Time
-0.68
-0.2
-0.70
-0.4
-0.72
-0.6
-0.8
10
20
10
20
30
40
50
Time
Time
(a) Label ratio 0.1 on NATOPS
(b) Label ratio 0.4 on NATOPS
Figure 4: As the label ratio increases, the deep model trained by f-VAT ignores less important features (e.g., feature 2) and captures more critical regions (e.g., feature 10).
Limitations and Future Directions
In this paper, propose functional Virtual Adversarial Training (f-VAT), which incorporates the functional structure of data into perturbations. Considering semi-supervised time series classification, provide both theoretical insights and extensive empirical results showing the superiority of the proposed f-VAT method. We believe that our f-VAT method can serve as a general framework for semi-supervised learning methods with functional data.
Nevertheless, there are still some limitations in this work. We follow the implementations of repre- sentative semi-supervised time series classification settings, which primarily train backbone models from scratch, rather than fine- tuning them based on pretrained weights. consider examining the effectiveness of pre-trained backbones
potential future work.
Another future direction is explore and design different adversarial perturbation norm for other functional models. For example, for data with known seasonality or periodic patterns, we can design norm that additionally penalizes the non- periooddiyy of the perturbation to align with the periodic patterns, which could be done using the Fourier domain decomposition. We believe that investigating additional structure of the time series would further enhance the performance of the f-VAT method. Also, if the functional data lie a manifold, one can introduce norm induced by the manifold structure to better align the perturbations. These further exploration would shed the light on the development of semi-superes cemi-supervieed learning methods.
10

---

Acknowledgments and Disclosure of Funding
We would like thank the anonymous reviewers and area chairs for their valuable comments and suggestions.
References
[1] Robert A Adams and John JF Fournier. Sobolev spaces, volume 140. Elsevier, 2003.
[2] Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. In 2020 International joint conference neural networks (IJCNN), pages 1-8. IEEE, 2020.
[3] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv: 1100000 2018.
[4] Anthony Bagnall, Michael Flynn, James Large, Jason Lines, and Matthew Middlehurst. A tale of two toolkits, report the third: on the usage and performance of hive-cote v1. O. Advanced Analytics and Learning on Temporal Data, 2020.
[5] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint 10X3v:8803.0122 2018.
[6] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, 32, 2019.
[7] Niels Birbaumer, Nimr Ghanayim, Thilo Hinterberger, Iver Iversen, Boris Kotchoubey, Andrea Kübler, Juri Perelmouter, Edward Taub, and Herta Flor. A spelling device for the paralysed. Nature, 388(6)25)2277-298 1999.
[8] George EP Box and Gwilym M Jenkins. Time series analysis, forecasting and control. Holden- Day, 1976.
[9] Claudio Canuto, M. Youssuff Hussaini, Alfio Quarteroni, and Thomas A. Zang. Spectral Methods: Fundamentals in Single Domains. Scientific Computation. Springer Berlin Heidelberg. 2006.
[10] Mingeai Chen, Hao Cheng, Yuntao Du, Ming Xu, Wenyu Jiang. and Chongjun Wang. Two wrongs don't make a right: Combating confirmation bias in learning with label noise. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 14765-14773, 2023.
[11] Yanping Chen, Eamonn Keogh, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, and Gustavo Batista. The ucr time series classification archive, July 2015. www. ucr edu/
"eamonn/time_sett"" seteeeeerrie__dddeeeeeeeeeeeeeee
[12] Yanping Chen, Adena Why, Gustavo Batista, Agenor Mafra-Neto, and Eamonn Keogh. Flying insect classification with inexpensive sensors. Journal of Insect Behavior, 27(5)x657-677, 2014.
[13] Constantin Christof. On the stability properties and the optimization landscape of training problems with squared loss for neural networks and general nonlinear conic approximation schemes. Journal of Machine Learning Research, 2772 2021.
[14] Mooi Choo Chuah and Fen Fu. Ecg anomaly detection via time series analysis. In Frontiers of High Performance Computing and Networking ISPA 2007 Workshops: ISPA 2007 International Workshops SSDSN, UPWN, WISH, SGC, ParDMCom, HiPCoMB, and IST-AWSN Niagara Falls, Canada, August 28-September 1, 2007 Proceedings 5, pages 123-135. Springer, 2007.
[15] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555 2014.
11

---

[16] Hoang Anh Dau. Anthony Bagnall, Kaveh Kamgar, Chin-Chir Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The uer time series archive. IEEF/CAA Journal of Automatica Sinica, 6(6): (00):1293-1033 2019.
[17] Angus Dempster, François Petitjean, and Geoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and 2020.
Knowledge Discovery,
[18] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-KKong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(12):15604115111 2023.
[19] Haoyi Fan, Fengbin Zhang, Ruidong Wang, Xunhua Huang, and Zuoyong Li. Semi-supervised time series classification by temporal relation prediction. In International Conference on Acoustics, Speech, and Signal Processing, pages 3545-3549. IEEE, 2021.
[20] Nehla Ghouaiel, Fierre-François Marteau, and Marc Dupont. Continuous pattern detection and recognition in stream-a benchmark for online gesture recognition. International Journal of Applied Pattern Recognition, 4(2):146-1664 2017.
[21] Jann Goschenhofer, Rasmus Hvingelby, David Rügamer, Janek Thomas, Moritz Wagner, and Bernd Bischl. Deep semi-supervised learning for time series classification. arXiv preprint arXiv:2102.03622 2021.
[22] James D Hamilton. Time series analysis. Princeton university press, 2020.
[23] Jon Hills, Jason Lines, Edgaras Baranauskas, James Mapp, and Anthony Bagnall. Classification of time series by shapelet transformation. Data mining and knowledge discovery, 28:851-881, 2014.
[24] Tailen Hsing and Randall Eubank. Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators. Wiley Series in Probability and Statisties. Wiley, edition, 2015.
[25] Shayan Jawed, Josif Grabocka, and Lars Schmidt-Thiemm Self-supervised learning for semi- supervised time series classification. Advances in Knowledge Discovery and Data Mining, 12084:499, 2020.
[26] Guozhong Li, Byron Choi, Jianliang Xu, Sourav S Bhowmick, Kwok-Pan Chun, and Grace Lai-Hung Wong. Shapenet: A shapelet-neural network approach for multivariate time series classification. In Proceedings of the AAAI Conference Artificial Intelligence, pages 8375- 8383, 2021.
[27] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances neural information processing systems, 31, 2018.
[28] Linyang Li and Xipeng Qiu. Token-aware virtual adversarial training in natural language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8410-8418, 2021.
[29] Chengyu Liu, David Springer, Qiao Li, Benjamin Moody, Ricardo Abad Juan, Francisco J Chorro, Francisco Castells, José Millet Roig. Ikaro Silva, Alistair EW Johnson, et al. An open access database for the evaluation of heart sound algorithms. Physiological Measurement, 37(12):2181, 2016.
[30] Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. uwave: Accelerometer- based personalized gesture recognition and its applications. Pervasive and Mobile Computing, 566):657-675, 2009.
[31] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEFE Transactions on Pattern Analysis and Machine Intelligence, 11(88::979-19993 2018.
12

---

[32] Abdullah Mueen, Eamonn Keogh, and Neal Young. Lhaieel-ssapeless: an expressive primitive for time series classification. In International Conference on Knowledge Discovery and Data Mining, pages 1154-1 162,, 2011.
[33] Qingyi Pan, Suyu Sun, Pei Yang, and Jingyi Zhang. Futuresnet: Capturing patterns of price fluctuations domestic futures trading. Electronics, 13(22):4482, 2024.
[34] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[35] Hussein Sharadga, Shima Hajimirza, and Robert S Balog. Time series forecasting of solar power generation for large-scale photovoltaic plants. Renewable Energy, 150:797-807, 2020.
[36] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
[37] Yale Song, David Demirdjian, and Randall Davis. Tracking body and hands for gesture recognition: Natops aircraft handling signals database. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), pages 500-506. IEEE, 2011.
[38] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve remi-supervised deep learning results. Advances neural information processing systems, 30, 2017.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, pages 5998-6008 2017.
[40] Haishuai Wang. Qin Zhang, Jia Wu, Shirui Pan, and Yixin Chen. Time series feature learning with labeled and unlabeled data. Pattern Recognition, 89:55 66, 2019.
[41] Li Wei and Eamonn Keogh. semi-supervised time series classification. International Conference on Knowledge Discovery and Data Mining, pages 748-753, 2006.
[42] Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao. Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey, arXiv preprint 1004:2002..2477. 2020.
[43] Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Tien Lu. Tapnet: Multivariate time series classification with attentional prototypical network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6845-6852, 2020.
[44] Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deeplob: Deep convolutional neural networks for limit order books. IEEE Transactions on Signal Processing, 67(11):3001-30) 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our paper discusses the limitations of the work in Section Limitations and Futures Directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: Our paper provides full set of assumptions and a complete (and correct) proof in Appendix and Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our paper fully discloses the information needed to reproduce the main experimental results in Section 4. and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification: We include our codes and instructions to reproduce our main experimental results in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppbllccguuudooeo Cotee//ddeeeSbbmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our paper specifies the training and test details in Section 4.] and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the mean and standard error over five runs in Table I and Fig. D within Section 4.2
Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources in Ap- pendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://euut http///wrwssscpcuuprreeeeeeeeeee ceciae//eeehesnieeeeeeeeeeeeeeee
Answer: [Yes]
Justification:
The research conducts with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is essentially fundamental research in machine learning, without negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed. There is no societal impact of the work performed
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
20

---

A Duality Characterization of VAT Loss in General Spaces
Let us first introduce some preliminaries on the duality with respect to L2, Let I'' be an interval in IR and we denote by (T') the space of square-integrable functions on I" with the inner product
(u, 12(T)) = u(z)v(r)dz.
Let E be Banach space with norm 11 Let E= be the dual space of E, which consists of all continuous linear functionals on E. For f E E* and u E E, we denote by (S, () g- = f(u) the duality pairing between f and Suppose that E is either densely embedded in (T) (T) is densely embedded E. We will discuss the relationship between the dual space E and the space 2 (T). We denote by ||-|| the operator norm.
The case E L?. Leti E L be the embedding. Suppose that || S K K|/uull For v E L and u E E, we define the functional
Φn(u) 12))
Then, we find that
(w)) [0.(u)] = 1(ig- (0) S -
1 = sup (d)( & -| 11
This shows that Φe E E.. Moreover, suppose d E" identically zero. Then, for all u E E, we have o. (u) (1), e) 0. But since the range of i is dense in L we must have v = 0. Therefore, v + o. defines an embedding 1.2 + E", which we denote by i In addition, also have
(v, (2)) (i (v), u) F- U)
namely, "-L12 = i12
we view the embeddings as inclusions, we have E IC C
The case 2 E. Let L E be the embedding. For an element & E E", goir-- also defines continuous functional on L Consequently, from the Riesz representation theorem. there exists unique element Ui E T.2 such that (d o i12=)))) (u, vi) 12- Moreover,
||o|ll2 = sup 1 [() = sup || 5I
| < |o - ||
Therefore, the mapping di + ug a bounded linear operator from E L², which we denote by gg 1.2.. Furthermore, if Uy is identically
then é must also be zero by the density of the range of i,--EE Consequently, & +1 is an embedding. Similarly, we have
(0, (0,ir=-(u)) F (1g.-1>(0), u) 12 namely, i2-4 ig. ig.
Viewing the embeddings as inclusions, we have F" C C.2
A.1 The Duality Characterization of VAT Loss
With the above notations. let give more rigorous statement of the theorem. When E L2, we define the VAT loss as
VA (2O, =
sup (f(zo) - f(xo + r))
(12)
r=i 1 (ro)e Prolll
while when 1 E, we define the VAT loss as
sup (2o 6f)= reL2:|l:
If(xo) f(vo + r)
(13)
Let us give more rigorous statement of the theorem.
21

---

Theorem A.1. Les f(x) = (3,z) L2(T) be functional linear model with 3 E L
For the case E + L?, the loss Eq. 13 is equivalent to
C (3o, E; f) [=)
(14)
For the case I + E, the loss Eq. 12
equivalent to
c Boll ir- (3n) for some Bo E",
C (Io,
(15)
otherwise.
If we view the embeddings inclusions, unify the two cases _ (xo. e; f) 21
(16)
Proof. Let Eg be the set which the supremum is taken in Eq. 113 and Eq. (12)
Then, the VAT
loss writes
CA (xo. f) sup (8, zo) (8, zo + t)]l sup
(B,r)
(2(2)
re Eo
Be
Let us consider the two cases of separately.
The T.?. Using the relation between the dual spaces, have
C (ro, (f
(3, (
sup reig (2))) St
sup Br
(8, (1
(:12- (8),
sup Ir
= = (B)1
The case 1 + F., In this case, have
sup (I0, PEL2.11222 St
(B,r)
3 = ig) for some 30 E E", we have
L (Ma,
(1 1 (30),
sup rELP:lli2a (1) St
sup
(Bo, (1),r)
sup
MEBinins
On the other hand, if (No, 6 f) is finite, then we can define a function on Ran 2 that <(r1) L)(r))
=i>=E(r),
which is well-defined since F is injective. It is easy to verify that is linear. Moreover,
21100 sup
(d(r)))==
(B,r) = (zo < oc.
sup
rEL2:|||ua
Hence, we have E.. Moreover, the definition of Φ and the duality relation gives
(3,7) = @(11) = (o, (o. (0, ().-))=))))))))))
(ir (1x-)
which implies that 3 ig- Consequently, (M < xo if and only if 3 Ran i 1 1.2 and we conclude the theorem.
22

---

A.2 Proof of Theorem 3.4
Let us first recall that Fréchet differentiability of f at 20 means that there exists a continuous linear functional V (x(o)) (T) such that
f(20 + +r) = f(xo) + (VI(2)), r) (3(T)) + (((rl as |rlle + 0.
Moreover, if I + E, say that the Fréchet differentiability holds under the E-norm if (xro) - (8o) for some Po E Ex and for any E L?, we have
f(zo + r) f(x) (Vf(zo).
+(llir===(r)le))) ||r|l - 0.
First, give rigorous version of Theorem 3.4 in the following.
Theorem A.2 (Theorem restated,, Let f (T) + R be Fréchet differentiable at zo (T)
with gradient f(xo) E (T).
Ef + (2))), then
lim C ff 1111=- (V[())) c-
1f (T) Fr, assume further that f = +1- (B0) for some Bo E F and the Fréchet differentiability holds under under E-norm, we have
lim 2 11 c1+01
Proof. We consider the two cases separately.
The case E L2(T") In this setting, the dual embedding i,.- (T) + E= satisfies (12---(1)) (V), u) (v. ()ar) for all v E LT),,, E E.
Combining it with the Fréchet differentiabilliy of at zu. Ifr - (r)) we have
f(o +r) = ((0o) + ()) +
= f(zo) + (i1.- (VF(oo))) r
E. o(IrollE))
Therefore, recalling that the VAT loss is given by Eq.
we have
C (NO, (0) sup nolBi]rollesse
((r2 +
sup meBBllrollsse
Consequently,
BeBBllrole (V/( +
sup
(VL== +o(1)
sup M
= (V +0(1)
sup
= (V((o)) +
23

---

The case (T) + E satisfies
In this case, let us recall that the dual embedding i E - (T)
()), = u) for all E E", u E (T).
Thus, since Vz(00) = -L (8o) for some Bo E E", using the Fréchet differentiaaility of f at xo
under the E-norm, we have
f(ao + +r) = f(zo) + (Vf(2)), (arry +(lia>>=((()))))
=f(z)) (i (6o) r) LoCT, + +0l1a-
= =((o)) + ( (1 ((l
Recall the VAT loss is given by Eq. 13 we have
C (xo, fif)=
sup
(5. 2 (2)5)())) e =((lir==
= Sr
+ + a((r)lll)
sup
where we use the density of 722-EE Therefore,
6 2 (w0: = f;
( (E + o0))
sup
M.B.: Ilese
sup
s 1.80l g- o(1).
A.3 Proof of Proposition 3.2
3.2
Since f() is continuously differentiable, we have
f(xo) - f(xo + r) = (Vf(z)), + o(11r11)).
Noticing that |rll S implies ||r|l2 - 6 -
we have
LVA (Vo fi = sup
113, 2o) - (3, r)
re
((V r) + o(1rrl12))
sup
re
((V[(20),))++a +
sup
rlllllg
Therefore, when 0+, we obtain that
2 (zo, (ff
(V((z)))) - +o(1)
sup
- (V)(z))
sup
ri
= =
24

---

B Sobolev Spaces
Sobolev spaces are cornerstone functional analysis, providing framework to quantify the smoothness of functions and distributions. This section outlines their construction, duality properties, and practical computation discrete settings, which are particularly relevant for introducing dual loss in the VAT method. We refer to books for details.
B.1 Construction of Sobolev Spaces via Fourier Transforms
Sobolev spaces (R*) for E IR are defined using Fourier transforms, offering unified approach across all real smoothness indices. Let ((R)) denote the Schwartz space of smooth, rapidly decaying functions I&d, and S' (((RR)) its dual, the space of tempered distributions. Moreover, the Fourier transformation ú(E) Fu(E) SRa u(x)e= + dz of u E S(RR) and extend the Fourier transformation S' (Rd).
define
For E IR, the Sobolev space H* (RR.) is defined as the space of tempered distributions u such that (1 + (2 E L22 (RR")
(17)
with the norm and inner product
||u|| (E (1+11199)))))))) dg)
1/2 (u, U) / ( (1)))))(()))))) (18) JRd
Moreover, (TRI)) is also equivalent to the completion of S(R4) with respect to the He norm. The weight (1 + (g/ modulates the contribution different frequencies: for 8 >> 0, penalizes high-frreueecc components, enforcing smoothness; for < 0, emphasizes them, allowing rougher distributions; and for s = O, it reduces to the (IRd)) norm, since H ( = L²(())).
well
known that the Sobolev space H*R*) is Hilbert space under the inner- mmer-product (:,')
For positive integer 8, H (RR) coincides with the space of T,2 functions whose weak derivatives up order also in L (IR)). For negative (RR)) includes distributions that lack the integrability of 2 functions but are constrained by the decay of their Fourier coefficients.
In applications like time series, we often consider functions on a bounded domain, such as T = [0. 1]. For domains C RRd, ( can be defined as the restriction of functions from H*((R)) to Ω with the norm being the infimum of the H H((d)) norm over all extensions, or as the completion C (S2) with respect to the appropriate norm.
B.2 Duality and Embeddings of Sobolev Spaces
Sobolev spaces exhibit rich duality structure, which is essential for understanding their properties and applications. For I, the dual space of H=R(R)) with respect the (IR)) inner product H HR*( That is, any continuous linear functional (H"R)))" be represented as:
((u) = (v, 4) J v(z)u(z) dr,
(19)
some E H HRRd), with the norm equivalence:
[[l](и))> ||
(20)
This duality arises because the pairing (u, u) is well-defined when E H' and u E H-*, as the Fourier transform ensures that (1 + (51?) -(0())) and (1 + (g(2)s/2((E) yield product in (IRd).
Sobolev spaces also satisfy embedding theorems, which relate them other function spaces based on smoothness. A key result is the Sobolev embedding theorem: for s > d/2, H (RR)) embeds continuously into (IRd), the space of bounded continuous functions, with:
| |ulla, Cl
(21)
for some constant C. This embedding implies that functions in H with sufficiently large s are not only continuous but also bounded, property useful for ensuring regularity in optimization problems.
25

---

More generally, for $1 > (R*) + H ( reflecting that higher s corresponds to greater smoothness.
These properties- duality and embeddings-arr critical applications like Virtual Adversarial Training (VAT), where the dual norm [=)(= F-- quantifies model sensitivity to perturbations constrained by ||r]|=> S e, and embeddings ensure perturbations maintain desirable regularity.
B.3 Fractional Powers of the Laplacian and Sobolev Norms
The fractional power of the Laplacian provides an operator-theoretic perspective on Sobolev spaces, unifying the definition of (²*) across all R. The Laplacian - is positive, self-adjoint operator (RR)), and its fractional power (- A ) for
is defined via the Fourier transform:
F((-A) T ul(E) = K 2a (C)(
where ú(E) = Fu(E) is the Fourier transform of u. However, the Sobolev norm incorporates a shifted operator. 1 A, to ensure positivity and handle low frequencies effectively, Specifically, the fractional power (I A) satisfies:
Fi(I - A) ul(E) = (1 +1+ 1613)0/2
so that:
|| = |1(I A////2ulzz
This equivalence follows from:
II(T - -) //0/////lll / + (1+1ER)/2(()))
(1 (1+16))))00))) de =|
For > 0, (1 - A/0/2 acts as a differential operator of order penalizing high-frequency oscillations Au. and the norm ||u Aull
and enforcing smoothness. For example, when (I A)u measures both the function and its second derivatives. For $ < 0, (I A)///2 smoothing operator, and H includes distributions whose images under (1 A)- -//2 are in L
The dual norm in H (IR") relates to the inverse fractional power. For E H we have:
|| = [1(I A) -/2
since:
| H-* sup (w..4)
sup II ((
Ku, (I 2 w) 1.2,,
||ui #111
and setting w (I- - A) s/2 yields the result via the self-adjointnnes of I A. This operator formulation is particularly useful in discrete settings, as it translates directly to matrix powers, as discussed previously.
The fractional Laplacian (-A)* itself (without the identity shift) is also of interest, with norm:
[I(-A))/2ul3 = / (EE))
While this norm emphasizes derivative behavior alone, 7 A ensures baseline T,2 contribution, making H norms more robust for small or low frequencies. Choosing between (7 - -/)//2 (-A)s/2 or (al - 2 for perturbation constraints can tailor the robustness profile: the former balances function magnitude and smoothness, while the latter focuses purely on derivative control.
B.4 Discrete Computation via Spectral Methods
practical applications such as time series analysis, data are represented as discrete vectors in R-W To compute Sobolev norms in this discrete setting, we employ spectral methods that approximate the continuous operators while preserving their spectral properties, ensuring consistency with the operator perspective introduced in the discussion fractional powers and Sobolev norms.
26

---

We recall that in the continuous case, the Sobolev norm of function H*()) is defined using the fractional power the operator A, where is the Laplacian:
|| hr- = |1(I A) A) A
This can be expressed in the Fourier domain as:
( (1+1
| =
where à is the Fourier transform of u, and + |g|2 arises from the eigenvalues of - A. In the discrete setting, we aim to approximate this norm for a vector E IR representing time series sampled at N equally spaced points. For simplicity, we assume periodic boundary conditions, which are common in time series analysis and allow for efficient computation via the Fast Fourier Transform (FFT).
B.4.1 Discrete Laplacian and Sobolev Norm
The discrete Laplacian LN E R is constructed using second-order finite differences. For uniform grid with n points, Lx is tridiagonal matrix defined as:
-1
-
2
-1
... 0
0
LN =
0
-1
-1
0
0 -1 2
This matrix Ln approximates the negative Laplacian -A via twice difference, which is positive semi-definite matrix. Thus, IN + Ln. where IN is the N x N identity matrix, approximates the operator I - A. Consequently, the squared H norm the discrete vector TN E IR is then approximated as:
|r|pir & NlNN LN)
B.4.2 Speeding up the Computation under Periodic Boundary Conditions
For many applications of time series, we assume periodic boundary conditions, which allow us to use the discrete Fourier transform (DFT) to compute the Sobolev norm efficiently. Under periodic boundary conditions, we take the discrete Laplacian as
LN
-
1 2
-1
0
0
0
-1
1
0
0
-1 2
where we note the off-diagonal entries -1 the corners (e.g., I1,N -1, LN.1 = -1) enforce periodicity. Then, this matrix L, is circulant matrix, which can be diagonalized using the discrete Fourier transform (DFT). The matrix can be diagonalized by the discrete Fourier transform matrix F. defined by:
F= = (P5N)
FG
25jA/N
N-1,
VN
which unitary (i..,, F+F In. with 5" being the conjugate transpose). Then, the spectral decomposition of LN is given by:
I FAF"
where A diag(Ao, A AN-1) is diagonal matrix of eigenvalues. For the discrete Laplacian defined above, the eigenvalues are:
T
A
4sin
O.I....,,,
n
These eigenvalues are non-negative (Au 2 0), with An 0 corresponding to the constant mode. Consequently, IN + Lx is decomposed as:
IN + +N F(IN + A)F
27

---

and the fractional power of the operator is then:
(IN + LN) = F(IN + A)*F"
Thus, the norm can computed as:
1|N ll (NN + IN) FFF((N + +A)
Σ k=
where f, = PN is the discrete Fourier transform of rN. Since the fast Fourier transform (FFT) can compute fn. in O(N log N) time and other operations are O(N), the overall complexity of computing the Sobolev norm is O(N log N). improving efficiency compared to direct matrix-vector multiplication, which would be (N
B.4.3 Discrete Laplacian for Mon-Unifore Points
In many practical applications, such as time series analysis with irregularly sampled data, the points t l2 < .... < f. are uniformly spaced. In this case, we have adjust the discrete Laplacian to account for the non-uniform spacing between points.
For non-uniform grid, let define the forward and backward step sizes at each point:
h f++1
i= ................................ N 11
b5 4-11 i 2, N,
and define hi = hij and hy = hy- We introduce the coefficients
2
b The entries of the discrete Laplacian LN E NNN are given by
a; h (h+ k,)
C; 1 (1t + +,,)"
h
i= 2,..,,N 1.
b
C1
0
NN
02
b2
02
GN-1
bN-1
CN
aN
bN
With the adjusted discrete Laplacian, compute the Sobolev norm in similar manner before.
28

---

C Experiment Details
C.1
Dataset Details
We choose several representative datasets benchmarks, ranging from easy difficult. These datasets include different properties like sampling rate, allowing for a more comprchensive evaluation of various semi-supervised time series classification methods. Table summaries the statistics of various datasets. The details of other 30 UCR/UEA datasets can be found in [16].
CricketX [32. The dataset contains gesture data with position of the X axis collected from accelerometers in 3D space. CricketX includes 12 classes: "Cancel Call", "Dead Ball", "Four", "Last Hour Leg Bye", "No Ball", "One Short", "Out", "Penalty Runs", "Six", "TV Replay". Both the training and test sets contain 390 samples.
UWaveGestureLibbaayy All (UWave) [0 The dataset from the gesture recognition system used by Nokia's search engine collects user-phone interaction motions. It contains 4,478 samples. The training and test sets contain 896 and 3,582 samples, respectively.
Insect Wingeetingga (InsectWing) [1]] The dataset is released by the Computational Entomology group at the University of California Riverside for insect classification.
includes wingheat audio signals from male and female mosquitoes, different species of flies, and other insects. The training set contains 220 samples and the test set contains 1,980 samples.
SelfRegulationSCP2 (SelfReg) 77 The University of Tuebingen releases SelfRegulation- SCP2 [7], which contains EEG data with seven columns and 1,152 rows. These sensors record signals of slow cortical potentials from auditory and visual feedback. The training set contains 200 samples, and the test set contains 180 samples.
The AALTD competition releases the NATOPS dataset (20]. These sensors collect data from hands, elbows, wrists, and thumbs. The dataset consists of position coordinates. The six categories represent different actions: "I have to command", "all clear", "not clear", "spread wings", "fold wings", and "lock wings". Both training and test sets contain 180 samples.
NATOPS [20].
Heartbeat [29]. The Hcartbeat dataset relcased by the PhysioNet Challenge 2016 primarily includes heart sound signals from volunteers in clinical or non-clinical environments. The signals are categorized into two classes: "normal" (113 samples) and "abnormal" (296 samples). The sensors sequentially positioned the aortic, pulmonic, tricuspid and mitral auscultation sites patients spanning a broad age range.
Table 6: The statistics of univariate and multivariate datasets in UEA UCR archive, including three univariate datasets and three multivariate datasets for evaluation.
Dataset
Samples
Length
Dim
Class
CricketX
780
300
12
UWave
4478
948
8
InsectWing
2200
11
SelfReg
380
1152
7
2
NATOPS
360
51
24
6
Hcartbeat
409
405
61
5
C.2 Hyperparameter Setting
We use stochastic gradient descent with learning rate of 10 The batch size is set to 64 with maximum of 300 epochs. Due to the model-agnostic properties of f-VAT, we use an eight-layers Temporal Convolutional Network (TCN) 5 as the backbone architecture to compare with other competitive baselines. We run our experiments on eight NVIDIA A10 GPUs (each with 24 GB memory).
29

---

D Ablation Study
D.1 Various Architectures
To verify the model-agnostic property of our proposed f-VTT, select several representative deep architectures with comparable parameter size, including Gated Recurrent Units (GRU), Self-Attention encoder and Temporal Convolutional Network (TCN).
Gated Recurrent Unit (GRU) integrates the output and memory gates to address short-term memory challenges [15]. GRUs are combined with attention mechanisms better capture trend information within time series data.
Self-Attenten encoder (SA) [39. SA encoder composed of four self ceff-attention layers and positional encoding layer, which can effectively extract trend information from modeling time series data.
Temporal Convolutional Network (TCN) is a widely used for sequence modeling. Its causal convolutional operations effectively capture both short-term fluctuations and long- term temporal dependencies [5]. TCN achieves superior performance in various sequence modeling tasks.
We report the mean performance each architecture over five runs with different random seeds. As shown in Fig. [ TCN significantly outperforms other deep architectures on NATOPS with various label ratios. These observations show that causal convolution of TCN can effectively capture trend information, thus effectively utilizing unlabeled data further improve generalization. In this paper, we adopt the 8-layer TCN as our main architecture.
GRU SA
96
Accuracy (%)
90
TCN
80
75
65
60
Label ratio
Figure 5: The performance comparison of different architectures on NATOPS with different label
ratios. TCN significantly outperforms other architectures in all settings.
Table 7: Comparison of VAT, VAT-step, and VAT on five datasets under different label ratios.
Medrals
29%
10t6
4tr
199
200
47%
109
10%
40%
199
206
10%
209
VAr
25.85
56.3
6141
65.51
55.90
41.31
63.48
53.1
66.0
53-1
85
VAT-dup
-047 2as.
54300
95.00
17.45 jav
61.3
63.0
(86) com
4367
52.55 poer
81
$100
SNAY
1918
3341 200
6834
9133
WuB com
61.3 2ew
64 1on 46.3
60/0
To verify the straightforward improvement to VAT may be unsuitable for time series, we design variant called VAT-step, which constrains the magnitude of perturbations at each time step, and conduct extensive experiments across multiple datasets with different label ratios. T7ble shows that VAT-step only provides marginal performance gains compared the original VAT most settings, and even degrades performance on more challenging multivariate datasets (e.g.. SeltReg and NATOPS) containing complex temporal structure. This because clipping the perturbation each time step struggles to adaptively scale the magnitude on critical regions for prediction (like "shapelets" in Fig. 4.. unless the per-step magnitude hyperparameters are carefully tuned. Additionally. VAT-step can easily converge to the ppermutationniivariintt [8] local optimum, where arbitrarily reordering time steps can still generate identical perturbations satisfying each step-wise constraint. By contrast, adversarial perturbations generated by J-VAT, which incorporate the temporal structure of time series, more efficiently utilize sequential information improve the smoothness of predictive distributions and the final predictive performance.
30

---

E More Empirical Results
We randomly sample 30 datasets from UCR/UEA datasets and add a state-of-the-art Class- Aware Temporal and Contextual Contrasting (CA-TCC) [8] within available resources. Tables 8 to T0 show that tVAT consistently outperforms other baselines all settings.
Table 8: The accuracy and rank of each method across 30 datasets with label ratio or 0.1.
233e
11:0
CEF
11.8
035
6
553
316
4.0
31-8
1500
1181
6
15.4
33.8
325
-0
343
2731
12:4
1487
13.88
035
110
3.5
B0200
5631
Table 9: The accuracy and rank of each method across 30 datasets with label ratio a = 0.2.
Scatiiee
Tiple:
CA-TC
5NST
AlComarc mack Sookl
1330
56.09
41.0
2000-
11.6
101
6
811
(13)
3.00%
1140.-4
1741
1335
Dolgs Lacihe
11.8
1855--.
1300
200
23
0113
160
113
133
Table 10: The accuracy and rank of each method across 30 datasets with label ratio a = 0.4.
Egt.
Tamhs
2537
Typle
140
10
15.0
45.0
000
aaC aboo
010
0230
3000
502
300
500
1133
250
016
120
180
310-1
127h
0330
500
11
023.5
11:3--m
133
300
133
30
422
31h
115
15
100
1533
512
203
63
3.0--
128
0.4
A11
A32
9.34
400
X12
F Runtime comparison
Compared with VAT (O(N)), the computational complexity of f-VAT O(N log N) with a small constant factor, due to the FFT-based ||||=, normalization. The empirical results in Table show
31

---

that the extra computational cost remains in the same order as VAT. The proposed VAT achieves competitive performance without incurring significant computational costs, making particularly suitable for semi-supervised time series classification with limited computational resources.
Table 11: Runtime comparison of f-VAT and VAT.
Method
CricketX
U'avee
InsectWing
NATOPS
SelfReg
VAT
15.67
51.66
35.58
28.04
30.68
F-VAT
20.45
62.05
45.92
38.98
44.95
(%)
30.50
20.11
29.06
39.02
46.51
32


Reviews:
[{"review_id": "k6H7QQizzh", "reviewer": "Reviewer_Jy1d", "summary": "This paper proposes functional Virtual Adversarial Training (f-VAT), a novel method for semi-supervised time series classification. The key insight is that standard Virtual Adversarial Training (VAT), when applied to time series, uses Euclidean norm-bounded perturbations, which ignores the temporal structure of the data and leads to unrealistic \"jagged\" adversarial examples. To address this, f-VAT incorporates the structure of time series by constraining the adversarial perturbations using a Sobolev norm. Empirically, the authors demonstrate that f-VAT significantly outperforms standard VAT and other state-of-the-art methods on several semi-supervised and fully-supervised time series classification benchmarks.", "questions": "- Regarding the choice of the Sobolev order `s`, do the authors have any practical guidance on selecting `s` for a new task without performing an exhaustive search?\n- Could you elaborate on how one might design other norms for different types of time series? For instance, for data with known seasonality or periodic patterns, could a norm be designed in the Fourier domain to specifically encourage or penalize perturbations in certain frequency bands?", "limitations": "The authors have adequately addressed the limitations of their work in Section 5", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "##### Strengths\n- The paper introduces a novel and well-motivated approach, where the authors use functional analysis tools (Sobolev norms) to generate structured adversarial perturbations for time series data\n- The work is technically strong, where it provides a theoretical foundation for the proposed framework, establishing a duality between the perturbation norm and the model's sensitivity.\n- The empirical evaluation is comprehensive, where the proposed method is tested on six datasets, consistently showing significant performance improvements over multiple baselines, including standard VAT. The results hold across different ratios of labeled data and in a fully-supervised setting.\n- The paper is well-written. The qualitative analyses, including visualizations of perturbations and loss landscapes, provide good intuition and support the paper's claims.\n##### Weaknesses\n- The paper's empirical validation of the Sobolev order `s` is based on a single dataset (NATOPS), where `s=2` was found to be optimal. It is not entirely clear if this choice is universally optimal or if `s` is a sensitive hyperparameter that would require tuning for each new dataset. A brief discussion on the generalization of this choice would strengthen the paper.\n- While the paper mentions the efficacy of the norm computation (O(N log N)), I could not find an explicit comparison of the total training time between f-VAT and standard VAT. A discussion on the practical computational overhead would be a valuable addition.", "was_revised": true, "final_justification": "The author response provides the justification for s and the empirical runtime data, which address my initial concern. As a result, I'm increasing my score.", "rebuttal": "### W1 & Q1: Selection of $s$\nThanks for your suggestions. We evaluate the performance on more datasets in Table, where $s=0$ reduces to the original VAT[Takeru et al., 2018]. Table shows that setting $s=2$ achieves the best performance in almost all settings. For high volatile time series data, setting high order $s=3$ generates adversarial perturbations that preserve low-frequency temporal information, while for relatively stable time series data, the small order $s=1$ allows the perturbation to flexibly explore the input space. In practice, it suffices to evaluate $s\\in[1,2,3,4]$ based on the validation set and report the average performance over several runs.\n\nThe performance of f-VAT with different Sobolev norm order $s$:\n| $s$ | CricketX               | UWave          | InsectWing            | NATOPS               | SelfReg        |\n|:---:|-----------------------:|-----------------------:|----------------------:|---------------------:|----------------------:|\n| 0 | 58.63 $\\pm$ 0.50 | 94.76 $\\pm$ 0.54 | 63.48 $\\pm$ 0.30 | 90.15 $\\pm$ 1.60 | 53.47 $\\pm$ 1.04 |\n| 1 | 59.91 $\\pm$ 2.32 | 96.54 $\\pm$ 0.67 | 66.70 $\\pm$ 0.50 | 89.58 $\\pm$ 0.12 | 56.16 $\\pm$ 1.73 |\n| 2 | 61.66 $\\pm$ 2.33 | 97.16 $\\pm$ 0.28 | 67.08 $\\pm$ 0.86 | 93.13 $\\pm$ 0.15 | 58.86 $\\pm$ 0.35 |\n| 3 | 60.44 $\\pm$ 0.23 | 96.82 $\\pm$ 0.16 | 64.10 $\\pm$ 0.86 | 90.10 $\\pm$ 0.65 | 51.39 $\\pm$ 1.21 |\n| 4 | 58.22 $\\pm$ 3.39 | 96.71 $\\pm$ 0.61 | 66.11 $\\pm$ 0.82 | 87.51 $\\pm$ 0.52 | 50.93 $\\pm$ 0.12 |\n\nReference \n\n- [Takeru et al., 2018] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.\n\n### W2: Runtime comparison \nThanks for your suggestions. Compared with VAT ($\\mathcal{O}(N)$), the computational complexity of f-VAT is $\\mathcal{O}(N\\log(N))$ with a small constant factor, involving the FFT-based $\\|\\cdot\\|_{H^{-s}}$ normalization. The empirical results show that the extra computational cost remains in the same order as VAT.\n\nRuntime comparison f-VAT and VAT:\n| Method  | CricketX | UWave | InsectWing | NATOPS | SelfReg |\n|-------|---------|-------------|-----------|-------|---------------|\n| VAT     | 15.67 | 51.66 | 35.58 | 28.04 | 30.68 |\n| f-VAT   | 20.45 | 62.05 | 45.92 | 38.98 | 44.95 |\n| Δ (%)   | 30.50 | 20.11 | 29.06 | 39.02 | 46.51 |\n\n### Q2: Norms for different types of time series\nThanks for your insightful comments. Indeed, we should design other norms to align with different types of time series. For example, for data with known seasonality or periodic patterns, we can design a norm that additionally penalizes the non-periodicity of the perturbation to align with the periodic patterns, which could be done using the Fourier domain decomposition. We believe that investigating additional structure of the time series would further enhance the performance of the f-VAT method, and it can be one of our future directions."}, {"review_id": "lY0lEqQ7yb", "reviewer": "Reviewer_jNPY", "summary": "This paper proposes a framework of functional Virtual Adversarial Training (f-VAT) for semi-supervised time series classification. The theoretical analysis and foundation are established for the proposed  algorithm. The experimental results demonatre its effectiveness.", "questions": "Accoding to Eq.(11), the H^s norm can be only approximated, how about the approximation error here? Is there the other method to compute  the norm? How to select the proper s for a practical problem or task?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n    This paper tries to utlize the Virtual Adversarial Training (VAT) method on leveraging the unlabeled data of semi-supervised time series classification for smoothing the predictive distributions and proposes the framework of functional Virtual Adversarial Training  (f-VAT) to incorporate the functional structure of the data into perturbations. It theoretically establishes a duality between the perturbation norm and the functional model sensitivity, and then use an appropriate Sobolev norm to generate structured functional adversarial perturbations for semi-supervised time series classification. The proposed algorithm is reasonable  and novel for  semi-supervised time series classification. The experiments  strongly support the theoretical  analysis.\n\nWeeknesses: The description of theoretical results is not so logic or explainable. It is a big jump to make the derivation from the linear model to the continously differentiiable case. The authors should give a good explain for this generalization in the paper.", "was_revised": true, "final_justification": "The authors addressed my concerns and questions and I still kept my previous scores.", "rebuttal": "### W1: Description of theoretical results\nWe apologize for not making the generalization clear in the original version. The case of **linear model (as also functional linear model in the infinite-dimensional case) provides main insights into the duality between the adversarial perturbation norm and the model sensitivity norm**, while the generalization to the continuously differentiable case serves mainly as additional theoretical supports for more complex models, though, due to the highly complex nature of neural networks, further theoretical analysis would be beyond the scope of this paper.\nNevertheless, our existing theory has already provided much insight for the design of f-VAT with the duality theory--we should choose the proper Sobolev $H^{-s}$ norm (but not $H^s$) for adversarial perturbations, which is key for the success of our method.\n\nWe hope that this explanation clarifies your doubts, and we will make it more clear in the revised version.\n\n### Q1: Approximation error\nThank you for your insightful question. The discrete approximation of the $H^s$ norm is known as the spectral method [Canuto et al., 2006], which is the most common way to approximate the Sobolev norm with discrete observations. The approximation error is determined by multiple factors, including the order $s$, the distribution of the samples, and the smoothness of the function. Roughly speaking, assuming that $u \\in H^r$ for some $r > s \\in \\mathbb{R}$, the approximation error of the $H^s$ norm with $N$ equidistant samples would be $\\mathcal{O}(N^{r-s})$.\n\nThere are indeed other methods to compute the Sobolev norm, such as finite element methods and wavelet methods. However, they are more complicated and less efficient. Since the approximation error is not our main focus, we think the spectral method is sufficient for our purpose.\n\nReferences:\n- [Canuto et al., 2006] Claudio Canuto, M. Youssuff Hussaini, Alfio Quarteroni, and Thomas A. Zang. Spectral Methods: Fundamentals in Single Domains. Scientific Computation. Springer Berlin Heidelberg,2006.\n\n### Q2: Selection of $s$\nThanks for your suggestions. We evaluate the performance on more datasets in Table, where $s=0$ reduces to the original VAT[Takeru et al., 2018]. Table shows that setting $s=2$ achieves the best performance in almost all settings. For high volatile time series data, setting high order $s=3$ generates adversarial perturbations that capture low-frequency temporal structure, while for relatively stable time series data, the small order $s=1$ allows the perturbation to flexibly explore the input space. In practice, it suffices to evaluate $s\\in{1,2,3,4}$ based on the validation set and report the average performance over several runs.\n\nThe performance of f-VAT with different Sobolev norm order $s$:\n| $s$ | CricketX               | UWave           | InsectWing            | NATOPS               | SelfReg        |\n|:---:|-----------------------:|-----------------------:|----------------------:|---------------------:|----------------------:|\n| 0 | 58.63 $\\pm$ 0.50 | 94.76 $\\pm$ 0.54 | 63.48 $\\pm$ 0.30 | 90.15 $\\pm$ 1.60 | 53.47 $\\pm$ 1.04 |\n| 1 | 59.91 $\\pm$ 2.32 | 96.54 $\\pm$ 0.67 | 66.70 $\\pm$ 0.50 | 89.58 $\\pm$ 0.12 | 56.16 $\\pm$ 1.73 |\n| 2 | 61.66 $\\pm$ 2.33 | 97.16 $\\pm$ 0.28 | 67.08 $\\pm$ 0.86 | 93.13 $\\pm$ 0.15 | 58.86 $\\pm$ 0.35 |\n| 3 | 60.44 $\\pm$ 0.23 | 96.82 $\\pm$ 0.16 | 64.10 $\\pm$ 0.86 | 90.10 $\\pm$ 0.65 | 51.39 $\\pm$ 1.21 |\n| 4 | 58.22 $\\pm$ 3.39 | 96.71 $\\pm$ 0.61 | 66.11 $\\pm$ 0.82 | 87.51 $\\pm$ 0.52 | 50.93 $\\pm$ 0.12 |\n\nReference \n- [Takeru et al., 2018] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018."}, {"review_id": "7gtURY1Jd2", "reviewer": "Reviewer_QUWH", "summary": "This paper addresses the flaw of a famous semi-supervised learning method, VAT, when adopted in time series classification tasks, which is suggested to be caused by neglecting the temporal structure of the data in the adversarial perturbation. In response, the paper proposes functional-VAT (f-VAT), using an appropriate Sobolev norm to generate structured functional adversarial perturbations to incorporate the functional structure of the data into perturbations.", "questions": "Please check the weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n\n- Semi-supervised learning for time series classification is an important but highly under-explored topic. I believe it is meaningful to specifically consider the characteristics of time series data, just as this paper did.\n\n- Utilizing the duality between perturbation norm and gradient sensitivity and introducing the Sobolev norm to generate structured adversarial perturbations is reasonable to me. The theoretical derivation looks elegant and solid.\n\n- The experimental results look promising, showing clear improvement of the proposed method.\n\n\n[Weaknesses]\n\n- My biggest concern is that the motivation is not convincing enough. Specifically, I could understand that adopting VAT in time series classification tasks would face the suggested problem, inspiring the proposed improvement. However, in the first place, why is it necessary to use VAT? VAT is a quite classical method proposed several years ago, with various more advanced semi-supervised learning methods appearing subsequently. Does VAT have any particular advantages over them in time series tasks? From my perspective, building upon VAT is afraid to limit the contribution of the proposed method to the current real-world practices.\n\n- As UCR and UEA repositories include 100+ time series classification datasets, just using six datasets is not sufficient and representative for me. I would suggest either including the majority of the datasets or pre-setting a specific rule to select the experimental datasets to avoid cherry-picking. For the compared baseline methods, there is the same case. While it is argued that the settings of both datasets and baselines are the same as [17], please note that it is published on ICASSP. The experiment range that is solid enough for ICASSP may not be similarly solid enough for a top conference like NeurIPS.\n\n- There is a lack of empirical evidence for some minor points. For instance, while the paper provides a theoretical complexity for implementing the Sobolev norm, it would be beneficial to include an empirical comparison of the actual training overhead of f-VAT versus standard VAT. \n\n- The code is not available for reproduction.\n\n\n[Recommendation]\n\nOverall, the novelty, insights, and contributions of the paper are borderline for NeurIPS. The suggested method is quite interesting, and the current results seem promising. However, I have certain concerns about the motivation, and the experiments are not solid enough. Therefore, I recommend a borderline rejection at this time, and would like to change my score based on the rebuttal and the discussion with other reviewers later.", "was_revised": true, "final_justification": "I appreciate the authors' efforts in the rebuttal, and apologize for leaving out the code in the supplementary materials. \n\nAfter carefully reading the rebuttal and other reviewers' comments, I maintain my position that the novelty of this work is borderline for NeurIPS. And to be honest, regarding the motivation, I'm still not quite convinced by the choice of VAT. Specifically, although the suggested f-VAT achieves a non-trivial improvement in performance compared with the existing method, integrating the same \"functional\" adversarial data augmentation into other more advanced SSL frameworks may still help achieve better improvement. That is, instead, what I'm mainly concerned about.\n\nHowever, the authors provide a large number of new empirical results as suggested, which look promising and indeed address some points of my concern. Therefore, I increased my rating to borderline accept.", "rebuttal": "### W1: Motivations for selecting VAT\nWe appreciate the reviewer’s concern about the choice of Virtual Adversarial Training (VAT) [Takeru et al., 2018]. Some recent self-supervised methods like TS-TCC [Emadeldeen et al., 2023] require careful selection of confidence thresholds and remain susceptible to \"confirmation bias\", while VAT-based methods do not require selecting thresholds and can adaptively construct adversarial perturbations to effectively utilize unlabeled data for better generalization. Additionally, our proposed f-VAT within the VAT-based framework greatly improves performance by effectively leveraging underlying data structure of time series data. Please read our response to the next question for additional empirical results on the performance comparison of different methods across multiple datasets. Additionally, we provide a theoretical understanding of f-VAT, which gives us insights into the selection of the perturbation norm.\n\n### W2: More empirical results\nThanks for your suggestions. In the original paper, we follow the settings [Haoyi et al., 2021] to compare f-VAT with other competitive baselines and report results on six UCR/UEA datasets. To further verify the superiority of f-VAT, due to the time limit, **we randomly sample 30 datasets from UCR/UEA datasets and add a recent state-of-the-art Class-Aware Temporal and Contextual Contrasting (TS-TCC) [Emadeldeen et al., 2023]** within available computing resources. Table shows that tVAT still consistently outperforms other competitive baselines in all settings. Due to the page limit, please refer to details of more empirical results in the comments.\n\nAverage accuracy (%) and average rank under different label ratios:\n| Method   | 10% |      | 20% |      | 40% |      |\n|:--------:|:----:|-----:|:----:|-----:|:----:|-----:|\n|          | AvgAcc | AvgRank | AvgAcc | AvgRank | AvgAcc | AvgRank |\n| SupL     | 35.31 | 6.67 | 36.92 | 7.00 | 37.15 | 7.33 |\n| PI       | 53.09 | 3.93 | 55.16 | 4.40 | 63.60 | 4.47 |\n| MTL      | 45.19 | 5.70 | 45.72 | 5.87 | 46.11 | 6.80 |\n| SemiTime | 56.53 | 3.57 | 58.93 | 3.77 | 69.02 | 3.17 |\n| TapNet   | 58.67 | 3.70 | 60.41 | 3.40 | 70.28 | 3.20 |\n| TS‑TCC   | 58.07 | 3.37 | 59.84 | 3.67 | 63.27 | 4.27 |\n| f-VAT     | 65.85 | 1.50 | 68.87 | 1.50 | 76.24 | 1.53 |\n\nAdditionally, we construct more empirical results on several large-scale China Securities Index (CSI) datasets (i.e., CSI 50 and 500 futures) spanning from 2020 to 2023 for predicting directions (upward or downward) of futures prices [Zihao et al., 2019] [Qingyi et al., 2024]. The dataset collects over $42\\times 10^{3}$ records spanning from 2020 to 2022, and each time step contains bid/ask prices and corresponding volumes. Table presents the performance of various semi-supervised methods across different label ratios, and shows that f-VAT significantly outperforms other competitive baselines on futures datasets, especially on more volatile CSI 500 futures. This is because f-VAT's adversarial perturbations incorporating key temporal structure facilitate deep models to effectively use unlabeled samples to yield smoother predictive distribution with better generalization.\n\nThe performance comparison on domestic futures datasets:\n| Futures | Ratio | supL | PI | MTL | SemiTime | TapNet | TS-TCC | f-VAT |\n|:------:|:----:|---------------:|-----------------------:|--------------:|-------------------:|-----------------:|-----------------:|--------------------:|\n|        | 10% | 40.05 $\\pm$ 1.38 | 42.87 $\\pm$ 1.45 | 53.94 $\\pm$ 0.13 | 55.19 $\\pm$ 0.77 | 54.62 $\\pm$ 0.54 | 55.72 $\\pm$ 1.16 | **58.64 $\\pm$ 0.53** |\n| 50     | 20% | 45.08 $\\pm$ 1.45 | 47.26 $\\pm$ 0.73 | 54.97 $\\pm$ 0.61 | 56.69 $\\pm$ 0.50 | 56.93 $\\pm$ 1.28 | 58.21 $\\pm$ 0.44 | **62.09 $\\pm$ 0.26** |\n|        | 40% | 50.69 $\\pm$ 3.53 | 52.42 $\\pm$ 1.21 | 56.97 $\\pm$ 0.57 | 57.34 $\\pm$ 0.20 | 59.75 $\\pm$ 1.09 | 59.80 $\\pm$ 1.32 | **64.78 $\\pm$ 0.45** |\n|        | 10% | 34.23 $\\pm$ 0.24 | 44.00 $\\pm$ 0.06 | 39.53 $\\pm$ 1.35 | 38.86 $\\pm$ 2.04 | 40.53 $\\pm$ 0.83 | 39.79 $\\pm$ 1.54 | **43.77 $\\pm$ 0.73** |\n| 500    | 20% | 35.38 $\\pm$ 1.06 | 46.57 $\\pm$ 0.48 | 45.26 $\\pm$ 0.29 | 46.04 $\\pm$ 0.12 | 44.58 $\\pm$ 1.70 | 45.05 $\\pm$ 1.74 | **52.14 $\\pm$ 0.25** |\n|        | 40% | 43.85 $\\pm$ 1.49 | 49.25 $\\pm$ 0.55 | 47.66 $\\pm$ 1.50 | 50.65 $\\pm$ 1.05 | 51.50 $\\pm$ 1.36 | 54.29 $\\pm$ 0.87 | **58.66 $\\pm$ 0.46** |\n\n### W3: Runtime comparison \nThanks for your suggestions. Compared with VAT ($\\mathcal{O}(N)$), the computational complexity of f-VAT is $\\mathcal{O}(N\\log(N))$ with a small constant factor, involving the FFT-based $\\|\\cdot\\|_{H^{-s}}$ normalization. The empirical results show that the extra computational cost remains in the same order as VAT.\n\nRuntime comparison f-VAT and VAT:\n| Method  | CricketX | UWave | InsectWing | NATOPS | SelfReg |\n|-------|---------|-------------|-----------|-------|---------------|\n| VAT     | 15.67 | 51.66 | 35.58 | 28.04 | 30.68 |\n| f-VAT   | 20.45 | 62.05 | 45.92 | 38.98 | 44.95 |\n| Δ (%)   | 30.50 | 20.11 | 29.06 | 39.02 | 46.51 |\n\n### W4: The available code for reproduction\nWe provide the training logs and the source code in the supplementary materials for reference. Could you please provide more details while trying to reproduce empirical results, such as error messages or missing packages?\n\n- [Takeru et al., 2018] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.\n- [Emadeldeen et al., 2023] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, , 2023.\n- [Haoyi et al., 2021] Haoyi Fan, Fengbin Zhang, Ruidong Wang, Xunhua Huang, and Zuoyong Li. Semi-supervised time series classification by temporal relation prediction. In ICASSP, IEEE, 2021.\n- [Zihao et al., 2019] Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deeplob: Deep convolutional neural networks for limit order books. IEEE Transactions on Signal Processing, 2019.\n- [Qingyi et al., 2024] Qingyi Pan, Suyu Sun, Pei Yang, and Jingyi Zhang. Futuresnet: Capturing patterns of price fluctuations in domestic futures trading. Electronics, 13(22):4482, 2024"}, {"review_id": "ihf9TY2mcn", "reviewer": "Reviewer_RR58", "summary": "The paper points out that the Virtual Adversarial Training falls short on time series as the added perturbations bounded by  Euclidean norm disrupt time series patterns. The paper proposes the Sobolev norm to replace the Euclidean norm to generate structured functional adversarial perturbations.", "questions": "Please refer to Weaknesses.", "limitations": "The evaluation is limited and some statements are confusing.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\nThis paper points out that the Virtual Adversarial Training falls short on time series: the perturbation vector constrained by Euclidean norm disrupts temporal patterns. This insight is very enlightening for designing effective adversarial methods tailored for time series.\n\n\n\n\n\n### **Weakness 1**\n\nWhy is time series considered functional data, while images are not? Time series is a function of some interval $T$, and images can also be considered functions in a spatial context. \n\nDespite \"Functional\" being a key term in the title, I still do not understand its specific meaning in this paper after reading the introduction. Additionally, it's unclear what unique designs are applied to time series (at least not sufficiently articulated in the introduction). From this perspective, I believe the writing of the paper needs improvement.\n\n### **Weakness 2**\n\nThe clarity and motivations in Section 3 are confusing.\n\n- **W 2.1**: How do Theorem 3.3 and Theorem 3.4 assist in the method design? The condition of Theorem 3.3  is that the model is linear, and the condition of  Theorem 3.4 is $\\epsilon\\rightarrow0^+$. **These two conditions are not satisfied in practical training**.\n- **W 2.2**: Based on Line 163~170, the conclusions of Theorem 3.3 and Theorem 3.4 state that \"if we want to impose a stronger norm on the sensitivity of the method, we should relax the norm of the perturbation.\" How is this related to the introduction of the Sobolev norm?\n- **W 2.3**: Why does the Sobolev space penalize high-frequency oscillations? Is the benefit of the Sobolev space solely that it penalizes high-frequency noise? This can be achieved through common high-frequency filtering, which has already been widely applied in existing time series analysis methods.\n- **W 2.4**: Compared with VAT: \n  - The VAT only needs a hyperparameter $\\epsilon$, but f-VAT needs **four hyperparameters** $\\epsilon$,  $s$, $L$, and $\\eta$. VAT only needs to compute once to obtain the optimal perturbation $r$, whereas f-VAT requires $L$ iterations and the hyperparameter $\\eta$. \n  - Do the perturbation vector $r$ and the model parameters $\\theta$ share the same learning rate $\\eta$ as shown in Algorithm 1?  \n  - In addition to the Sobolev norm order, the sensitive analysis of other hyperparameters should be included, along with a comparison of the training time between VAT and f-VAT.\n- **W 2.5**: In Line 6 of Algorithm 1, why adopt gradient ascent and normalize  $r_i$? In the original paper of VAT, the optimal $r$ is obtained by calculating the first dominant eigenvector of the Hessian matrix $\\nabla\\nabla_r D(r, x, \\hat{\\theta})|_{r=0}$. Is the approach not suitable for f-VAT? For instance, calculate the optimal $r$ whose Sobolev norm is $\\epsilon$ in the direction of the first dominant eigenvector? \n- **W 2.6**: Could the authors provide the formula of $||\\cdot|| _ {H ^ {-s}}$ as well, rather than just $||\\cdot|| _ {H ^ {s}}$. It is confusing that Eq 11 only introduces $||\\cdot|| _ {H ^ {s}}$, while Algorithm 1 actually constrains $r$ using $||\\cdot|| _ {H ^ {-s}}$. Additionally, in Eq 11 and its context, $L$ and $L _ N$ seem to be confused. \n\n### **Weakness 3**\n\nThis paper lacks a discussion of related works, particularly in the area of semi-supervised time series. Additionally, the experiments do not compare with the latest semi-supervised methods [1, 2]. More importantly, the paper only includes three datasets from UCR and three from UEA. However, TS-TFC [1] includes **106 datasets** from UCR. As far as I know, these time series datasets are relatively small-scale and highly heterogeneous, making the evaluation in the paper seem inadequate.\n\n[1] Temporal-Frequency Co-training for Time Series Semi-supervised Learning. *AAAI*, 2023.\n\n[2] Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification. TPAMI, 2022.\n\n\n\n### **Weakness 4**\n\nIn Section 4.3, the authors claim that functional adversarial perturbations add subtle disturbances only in non-critical areas, while keeping them small in important regions. However, I did not observe this phenomenon. **Where are the critical regions versus the non-critical regions?** It appears that f-VAT applies small perturbations at each time point, while VAT applies larger perturbations at certain time points.\n\nTherefore, a straightforward improvement to VAT could involve not only constraining the norm of the overall perturbation vector but also limiting the magnitude of perturbations at each time point, or constraining the variance of perturbations at each time point. Would this be viable？", "was_revised": true, "final_justification": "I appreciate the thorough experimental results provided by the authors, and I would increase my rating to 4.", "rebuttal": "We would like to point out that the contribution of this paper is to propose the **theory-inspired f-VAT algorithm that performs well in practice**, which is supported by our experiments (and additional ones, see answers below) that f-VAT consistently outperforms other competitive baselines in all settings. \nWhile the theory considers simplified settings, it provides us with key insights into designing the f-VAT algorithm, particularly in constructing perturbation that aligns with the time series structure of the data.\nTherefore, we believe that our paper is novel and significant both empirically and theoretically.\n\n### W1.1\nWe agree that both time series and images can be viewed as functional data, but their intrinsic structures can be different. In this paper, we mainly focus on time series data, but our general f-VAT framework can be applied to broader settings. Exploring f-VAT for functional data with other structures, such as image data, can be one of our future directions.\n### W1.2\nWe use the term \"Functional\" to emphasize the function structure of the data, particularly for time series data. Focusing on the additional function structure of the data, our novel f-VAT method generates adversarial perturbations that align with the key temporal structure, i.e., the dynamic correlations within variables changing over time steps.  This is the key improvement of f-VAT over VAT.\n\n### W2.1\nOur theory aims to provide insights into designing adversarial perturbations when the data have additional structure, particularly for time series data. We believe that our existing theory has already provided much insight for the design of f-VAT with the duality theory, and due to the highly complex nature of neural networks, further theoretical analysis would be beyond the scope of this paper. Moreover, strong empirical results on various time series datasets demonstrate the effectiveness of our method. \n\nPlease see our response to the next point for more details on how the theory is used in the method design.\n\n### W2.2\nIn our theory, we first establish the general theory on duality between the perturbation norm and the model sensitivity norm (Theorem 3.3 and Theorem 3.4). However, we still need to choose a norm (or the space for the norm) in the concrete implementation. For time series data, it is natural to consider Sobolev spaces $H^s$ (of positive order) to capture the smoothness/regularity for model sensitivity. Then, with the duality theory, we transfer $H^s$ to $H^{-s}$ for the perturbation norm, where we use the properties of Sobolev spaces that the dual space of $H^s$ is $(H^s)^* = H^{-s}$. In fact, it is one of the key points in our paper to use the Sobolev norm $H^{-s}$ of negative order for the perturbation, which is critical for the success of the proposed f-VAT.\n  \nWe would like to clarify that our point is \"The Sobolev norm allows us to properly control the model's sensitivity to high-frequency noises''. This can be seen from duality theory: the Sobolev norm $H^{-s}$ imposed on the perturbation effectively serves as a regularization term that penalizes the model sensitivity in terms of the Sobolev space $H^s$.\n\n### W2.3\nThe infinite-dimensional nature of the Sobolev space allows for a richer specification of perturbation constrains that align with the geometry of functional data like periodicity, rather than penalizing high-frequency filtering. Thus, the functional adversarial perturbation generated by f-VAT in Sobolev space can capture the temporal structure of time series data.\n\n### W2.4\n- W2.4.1:\nWe would like to clarify that f-VAT and VAT require the same set of hyperparameters: $\\epsilon$ controls the perturbation radius; $L$ is the number of power-iteration steps (typically set to $L=1$); and $\\eta$ is the step size for gradient ascent during the power-iteration. The only additional hyperparameter of f-VAT is the order $s$ of norm $\\|\\cdot\\|_{H^s}$. \n\n- W2.4.2:\nYes, both the perturbation vector $r$ and the model parameters share the same learning $\\eta$, as clarifies in [Takeru et al., 2018].\n\n- W2.4.3:\nCompared to original VAT, f-VAT only introduces an additional hyperparameter $s$. We evaluate the performance on more datasets, which shows that setting $s=2$ achieves the best performance. Table shows that the extra computational cost remains in the same order as VAT. Due to the page limit, please refer to our replies to Reviewer #Jy1d Q1 & W2.\n \n###  W2.5\nThe original VAT also adopts gradient ascent and normalizes $r_i$ to approximate the first dominant eigenvector of the Hessian matrix $\\nabla\\nabla D(r,x,\\hat{\\theta})|_{r=0}$, while the proposed f-VAT adopts the same power-iteration procedure with $L=1$. The key difference is that the projection into the $\\epsilon$ ball in the $L_2$ space is replaced by the $H^{-s}$ space.\n\n###  W2.6\nWe apologize for the typo in Eq.(11) and the confusion caused by the notation. In fact, as emphasized in the paragraph before Eq.(11), the formula $\\|r\\|_{H^\\alpha}^2 \\approx r_N^\\top (I_N + L_N)^\\alpha r_N$ applies to all $\\alpha \\in \\mathbb{R}$, including negative orders, so we do not distinguish $s$ or $-s$. Here, the operator $L$ should always be replaced by $L_N$ which represents the discrete Laplacian operator. We have corrected the notation in the revised version of the paper, and we hope this resolves the confusion.\n\n### W3\nWe have add more detailed discussion of literature review in the revised version. \nWe follow the settings [Haoyi et al., 2021] to compare f-VAT with other competitive baselines and report results on six UCR/UEA datasets. We **randomly sample 30 datasets from UCR/UEA datasets and add a state-of-the-art Class-Aware Temporal and Contextual Contrasting (TS-TCC) [Emadeldeen et al., 2023] within available resources**. Table shows that tVAT consistently outperforms other baselines in all settings. Please see our additional comments for more experiment details.\n\nAverage accuracy (%) and average rank across the dataseets under different label ratios:\n| Method   | 10% |      | 20% |      | 40% |      |\n|:--------:|:----:|-----:|:----:|-----:|:----:|-----:|\n|          | AvgAcc | AvgRank | AvgAcc | AvgRank | AvgAcc | AvgRank |\n| SupL     | 35.31 | 6.67 | 36.92 | 7.00 | 37.15 | 7.33 |\n| PI       | 53.09 | 3.93 | 55.16 | 4.40 | 63.60 | 4.47 |\n| MTL      | 45.19 | 5.70 | 45.72 | 5.87 | 46.11 | 6.80 |\n| SemiTime | 56.53 | 3.57 | 58.93 | 3.77 | 69.02 | 3.17 |\n| TapNet   | 58.67 | 3.70 | 60.41 | 3.40 | 70.28 | 3.20 |\n| TS‑TCC   | 58.07 | 3.37 | 59.84 | 3.67 | 63.27 | 4.27 |\n| f-VAT     | 65.85 | 1.50 | 68.87 | 1.50 | 76.24 | 1.53 |\n\nAdditionally, we construct empirical on several large-scale China Securities Index (CSI) datasets (i.e., CSI 50 and 500 futures) spanning from 2020 to 2023 for predicting directions (upward or downward) of futures prices [Zihao et al., 2019]. The dataset collects over $42\\times 10^{3}$ records spanning from 2020 to 2022, and each time step contains bid/ask prices and corresponding volumes. Table presents the performance of various semi-supervised methods across different label ratios, and shows that f-VAT significantly outperforms other competitive baselines on futures datasets, especially on more volatile CSI 500 futures. \n\nThe performance comparison on domestic futures datasets:\n| Futures | Ratio | supL | PI | MTL | SemiTime | TapNet | TS-TCC | f-VAT |\n|:------:|:----:|---------------:|-----------------------:|--------------:|-------------------:|-----------------:|-----------------:|--------------------:|\n|        | 10% | 40.05 $\\pm$ 1.38 | 42.87 $\\pm$ 1.45 | 53.94 $\\pm$ 0.13 | 55.19 $\\pm$ 0.77 | 54.62 $\\pm$ 0.54 | 55.72 $\\pm$ 1.16 | **58.64 $\\pm$ 0.53** |\n| 50     | 20% | 45.08 $\\pm$ 1.45 | 47.26 $\\pm$ 0.73 | 54.97 $\\pm$ 0.61 | 56.69 $\\pm$ 0.50 | 56.93 $\\pm$ 1.28 | 58.21 $\\pm$ 0.44 | **62.09 $\\pm$ 0.26** |\n|        | 40% | 50.69 $\\pm$ 3.53 | 52.42 $\\pm$ 1.21 | 56.97 $\\pm$ 0.57 | 57.34 $\\pm$ 0.20 | 59.75 $\\pm$ 1.09 | 59.80 $\\pm$ 1.32 | **64.78 $\\pm$ 0.45** |\n|        | 10% | 34.23 $\\pm$ 0.24 | 44.00 $\\pm$ 0.06 | 39.53 $\\pm$ 1.35 | 38.86 $\\pm$ 2.04 | 40.53 $\\pm$ 0.83 | 39.79 $\\pm$ 1.54 | **43.77 $\\pm$ 0.73** |\n| 500    | 20% | 35.38 $\\pm$ 1.06 | 46.57 $\\pm$ 0.48 | 45.26 $\\pm$ 0.29 | 46.04 $\\pm$ 0.12 | 44.58 $\\pm$ 1.70 | 45.05 $\\pm$ 1.74 | **52.14 $\\pm$ 0.25** |\n|        | 40% | 43.85 $\\pm$ 1.49 | 49.25 $\\pm$ 0.55 | 47.66 $\\pm$ 1.50 | 50.65 $\\pm$ 1.05 | 51.50 $\\pm$ 1.36 | 54.29 $\\pm$ 0.87 | **58.66 $\\pm$ 0.46** |\n\n### W4.1\nThe critical areas refer to the input feature essential for prediction. Figure 4 visualizes the essential \"shapelets'' in yellow and the irrelevant regions in blue. Our proposed f-VAT can adaptively concentrate its perturbations on these non-critical regions, aligning with the original samples across all key time steps, while the original VAT may introduce large spikes in some critical regions and disrupt the underlying trend structure.\n\n### W4.2\nThe straightforward improvement to VAT may be unsuitable for time series. If we limit the magnitude of perturbations at each time step, the order of time steps can be arbitrarily permutable to destroy the temporal structure. On the other hand, our proposed f-VAT can capture the key trend structure, i.e., the implicit correlations within variables changing over time, by generating theory-inspired functional adversarial perturbations.\n\n- [Takeru et al., 2018] Takeru Miyato, etc., Virtual adversarial training: a regularization method for supervised and semi-supervised learning. TPAMI, 2018.\n\n- [Haoyi et al., 2021] Haoyi Fan, etc, Semi-supervised time series classification by temporal relation prediction. ICASSP, 2021.\n\n- [Emadeldeen et al., 2023] Emadeldeen Eldele, et. al., Self-supervised contrastive representation learning for semi-supervised time-series classification. TPAMI, 2023.\n\n- [Zihao et al., 2019] Zihao Zhang etc, Deeplob: Deep convolutional neural networks for limit order books, 2019."}]

Author rebuttal:

Meta-review: This paper proposes functional Virtual Adversarial Training (f-VAT), a new framework for semi-supervised time series classification. Unlike standard VAT, which relies on Euclidean norm perturbations that fail to capture temporal structures, f-VAT introduces structural functional perturbations. By establishing a duality between perturbation norms and model sensitivity, the authors design structured adversarial perturbations that preserve low-frequency information while suppressing high-frequency noise. Extensive experiments on multiple datasets demonstrate consistent and significant performance improvements over baselines.

The strengths of this paper include:

1.	The work introduces a novel and well-motivated approach by applying Sobolev norms to construct structured adversarial perturbations that are more appropriate for time series data.

2.	The paper is theoretically solid, providing a clear foundation and duality between perturbation norm and model sensitivity.

3.	The empirical evaluation is comprehensive, demonstrating improvements across multiple semi-supervised datasets and settings.

4.	The paper is also clearly written, with qualitative visualizations that support the claims.

The authors further clarified their theory contributions during the rebuttal period. They have also provided additional experiments on more datasets, runtime comparisons, and clarifications on the selection of the Sobolev order, which resolved most of the reviewers’ concerns. Reviewers acknowledged that their initial issues were addressed.

I recommend acceptance. This is a technically strong paper with clear theoretical contributions and thorough empirical validation. That said, the revision should
Decision: Accept (poster)