Cross-Window Self-Training via Context Variations from Sparsely-Labeled Time Series
Authors: Yooju Shin
Susik Yoon
Hwanjun Song
Jae-Gil Lee
Byung Suk Lee
Venue: Submitted to ICLR 2023 (2023)
Keywords: semi-supervised learning
time series
pseudo labeling

Abstract: A real-world time series is often sparsely labeled due to the expensive annotation cost. Recently, self-training methods have been applied to a dataset with few labels to infer the labels of unlabeled augmented instances. Accelerating this trend for time-series data, fully taking advantage of its sequential nature, we propose a novel data augmentation approach called context-additive augmentation, which allows a target instance to be augmented easily by adding preceding and succeeding instances to form an augmented instance. Unlike the existing augmentation techniques which may alter the target instance by directly perturbing its features, it preserves a target instance as is but still gives various augmented instances with varying contexts. Additionally, we propose a cross-window self-training framework based on the context-additive augmentation. The framework first augments target instances by applying context-varying windows over a given time series. Then, the framework derives reliability-based cross-window labels and uses them to maintain consistency among augmented instances across the windows. Extensive experiments using real datasets show that the framework outperforms the existing state-of-the-art self-training methods.

Under review as a conference paper at ICLR 2023
CROSS-WINDON SELF-TRAINING VIA CONTEXT VARIATIONS FROM SPARSELY-LABELED TIME SERIES
Anonymous authors Paper under double-blind review
ABSTRACT
A real-world time series is often sparsely labeled due to the expensive annotation cost. Recently, self-training methods have been applied dataset with few la- bels to infer the labels of unlabeled augmented instances. Accelerating this trend for time-series data, fully taking advantage of its sequential nature, we propose novel data augmentation approach called context-additive augmentation, which allows a target instance to be augmented easily by adding preceding and succeed- ing instances form augmented instance. Unlike the existing augmentation techniques which may alter the target instance by directly perturbing its features, it preserves target instance is but still gives various augmented instances with varying contexts. Additionally, we propose a cross-window self-training frame- work based the context-additive augmentation. The framework first augments target instances by applying context-varying windows over given time series. Then, the framework derives reliability- to maintain consistency among augmented instances across the windows. Ex- tensive experiments using real datasets show that the framework outperforms the existing state-of-the-art self-training methods.
cross-window labels and uses them
INTRODUCTION
A time series is a collection of consecutive data points, often annotated with temporally coherent timestamp labels R4: and this work deals with a model aiming to classify every timestamp in time series correctly. However, due to the length of and complexity in a time series, labeling every timestamp in the time series requires prohibitively high cost, and therefore, in reality a
time
series are only sparsely labeled (Moltisanti et al. 2019 Ma et al.) [2200] Deldari et al.] 2021: Shin ct al. 2022). In this regard, self-training is used as a promising way to train a model from sparse labels, by leveraging the model's output to infer new labels for unlabeled data points (Laine & Aila Rizve et al. (021). Recent state-of-the-art self-training methods, mostly developed for image
2017
data, necessitate domain-specific data augmentation (Sohn et al. 2020 Zhang et al.) 2021 Kim &
Lee
2022).
Such conventional data augmentation generates multiple different instances from target instance R2.R4: (i.e., an instance for pseudo-labeling) by way of data perturbation. If data instances are independent of one another as in image data, there no other way than to perturb the target instance itself. In contrast, using the sequential nature of time series, where data instances (segments or data points) are temporally correlated, it is feasible to generate multiple different instances from target instance without perturbing it but by adding its surrounding sequence (i.e., context). R4: As shown in Figure a), given a target instance sampled from a time series, contexts of varying lengths are added to the preceding and succeeding positions of the target instance to generate different pairs of "augmented"" instances. We call this type of data augmentation the context-additive augmentation.
The key property of context-additive augmentation is to achieve the effect of data augmentation without perturbing a target instance. Being free of data perturbation brings several benefits. First, consistency between augmented instances can be enforced more reliably because a target instance itself is exactly the same among its augmented instances. Second, a sufficient number of augmented instances can be easily obtained by context variations. Third, it is computationally inexpensive, only requiring the retrieval of a sub-sequence from a time series. Moreover, context-additive augmenta- tion can be used together with conventional data augmentation such as jittering and scaling. Thus,

---

Under review as a conference paper at ICLR 2023
sparsely labeled time series
A pair of augmented instances left
Sampling window A tr label
x
wy w W
Context-additive A ugmentation
Model
Shared
Model
Model upeblee
Contest
Target
Tampe
Centans
Context Target instance Context-
Softmax probabilttes
fa (xlen)
WW RoDA to (xtight)
Add
Replicate
Add
Timestamps
Timestamps
Reliability
Seiaablity-Weiighhee Weighted Mixing
Varying contexts
M000
M 3 (xlet) (yelele).P)
Cross-window labels
Pairs of augmented instances
True labels
((6 (xlem) (xried)).))
(a) Context-additive augmentation.
(b) Model update with cross-windows soft labels.
Figure 1: R1,R2,R3, R4. Illustrati of CrossMatch.
the novel concept of context-additive augmentation opens new direction of data augmentation for sequential data, i.e., time series.
Despite its seme-series--avvy concept and big benefits, applying context-additive augmentation for self-training is challenging. First, requires determining the proper number and range of context variations based on the trade-off between the expected performance improvement and training cost. Note that varying the context length augmented instances incurs different complexity for a down- stream task; intuitively, compared with the conventional data augmentation, a short context gives weak augmentation and a long context gives strong augmentation R4: considering the intensity of perturbation. Second, a consistency regularization method is needed fully take advantage of the benefit of context-additive augmentation, which does not perturb a target instance unlike the conventional data augmentation.
In order to address the aforementioned challenges, we propose a novel self-training approach, called CrossMatch, for time series. In existing self-training methods such MixMatch (Berthelot al. 2019) and FixMatch (Sohn et al., 2020), an artificial label is created as a form of a hard label; because the model's outputs for augmented instances could be biased by the perturbation of the target instance, the most confident label is chosen by averaging and sharpening in MixMatch and weak augmentation and thresholding in FixMatch. In CrossMatch, on the other hand, due to its target-preserving property, the model's outputs with the contexts of the same length are considered equally meaningful, and therefore, the model's s output from an augmented instance (i.e., window) crossed to other augmented instances in the form of a seft label. H2: As shown in Figure bb, pair of augmented instances generated from a target instance fed to a model to get the two softmax outputs of the target instance. Then, single set of cross-windon soft labels is derived and enforced to each output for consistency regularization. The same procedure repeats using diverse pairs of augmented instances.
In summary, for time-series self-training, CrossMatch conducts context-additive augmentation with varying contexts and consistency regularization among augmented instances using cross-window soft labels. Especially for the first aforementioned challenge, we empirically analyze the impact of context variations classification accuracy in Section 4.2 Through extensive evaluation using three sparsely-labeled time-series datasets, despite its simplicity CrossMatch is shown to achieve higher classification accuracy than the existing state-of-the-ee methods, significantly outperforming the FixMatch style methods with jittering and scaling up to 23p%.
RELATED WORK
2.1
DATA AUGMENTATION
Data augmentation perturbs given data instances to generate diverse and sufficient data instances prevent overfitting (Shorten & Khoshgoftaar 2019). The techniques used in data augmentation
2

---

Under review as a conference paper at ICLR 2023
usually manipulate features and can be classified into two categories: (1) inner-augmentation that changes the features within a data instance and (2) anter-augmentation that exploits features across multiple data instances. Rotation, flipping, scaling, cutout, and random erasing are the examples of inner-augmentation (DeVries & Taylor) 2017, Cubuk et al. 2019). Mix-up, cut-mix, and copy-paste the representatives of inter-augmentation that mixes two full or partial images (Zhang et al. 2018 Yun et al. 20119] Ghiasi et al. 2021). However, these studies have not considered context-additive augmentation because they deal with independent data instances such as images.
Time-series augmentation techniques have been devised in recent literature (Wen et al. 2021). They include jittering, scaling, window warping, window cropping, and the Fourier transform special- ized in the time and frequency domains, and belong to the iuner-uggmentttion category (Um et al.) 2017 Eldele et af.] [021 Yue al. 2022; Chen et al.) 2022). The inter-augmentation category is considered to be ineffective because temporal patterns could be lost after mixing two time-series segments (Iwana & Uchida, 2021). These studies also consider set of already segmented time se- ries as candidate augmentation targets and assume that the instances in each segment have the same labels. Thus, they are not directly applicable to a single continuous time series with sparse labels, which is more practical and challenging setting. Moreover, all of them perturb the target instances, following the common trend of existing data augmentation.
2.2 SEMI-SUPERVISED LEARNING BASED ON AUGMENTATIONS
Semi-supervised learning (SSL) trains a model with both labeled and unlabeled data instances
Van Engelen & Hoos 2020). Unlabeled data instances are harnessed by two popular approaches:
(1) self-supervised learning which is mostly based on contrastive learning or pretext tasks (Chen)
al. 2020: Grill et al. 2020 Singh et al. 202I) and (2) self-training which produces artificial
[Lce|2013] Xie et al. 2020 Pham et al. 2021). We focus on self-training owing to its simplicity and effective-
pseudo-labels for unlabeled data instances from model predictions (Lee) 2013) Chen et al.) [2018b] a
ness demonstrated in recent studies Yang et al. (021).
State-of-the -frte---t self-training methods force consistency in model predictions from multiple augmen- tations of a data instance. Mix.March (Berthelot et al. (019) averages out the predictions from mul- tiple augmentations and sharpens the averaged prediction to reduce the entropy the pseudo-label ReMixMatch (Berthelot et al., (220) generates sharpened pseudo-label from a weak augmentation and matches against the predictions from multiple strong augmentations. FixMatch (Sohn et al.) 2020) generates one-hot pseudo-label by choosing single class above a fixed confidence thresh- old. FlexMatch (Zhang et al. 2021) is a variation of FixMatch, which uses a dynamic confidence threshold to adapt to different learning speeds among different classes. Propagation regularizer (Kim) & Lee 2022) also reduces confidence in incorrect predictions to make FixMatch robust to a more sparse label setting. However, all these methods heavily rely on domain-specific augmentation and lack for consideration of time series.
There are several time-series semi-supervised learning methods in the literature, but most of them are based on self-supervised learning. In their settings, a model is first pre-trained with time-series self- supervision and then fine-tuned with initial labels. The examples of time-series self-supervisions are (1) pretext tasks such as forecasting and temporal relation prediction (Jawed et al. 2020 Fan et al., 2021), (2) contrastive learning with the aforementioned time-series augmentation techniques (Singh ett al. 2021 Xiao et al. (022).. and (3) clustering results (Singhania et al. 2022). These methods target an already -eegeedeed time series, which cannot deal with continuous time series with sparse
labels (Ma et al. 2021 Goschenhofer et al. 2021 Xu et al. 2022).
CROSSMATCH: BROSS-WINDOL TIME-SERIES SELF-TRAINING
3.1 PRELIMINARIES AND PROBLEM SETTING
Table T
summarizes basic notations used in this paper.
Let D = X' x = ((x,, v1) t E T be a time series, where T is an index set of timestamps, Z₁ Rd is dddiienssionl data point at timestamp and yı is corresponding class label if x, is labeled or null otherwise. Let Tr. be the index set of labeled timestamps and To be the index set of unlabeled timestamps, where Tr, UTo T and [Tr| < <TTl Here, Tr. is sparse, that
Dataset and Model:
3

---

Under review as a conference paper at ICLR 2023
is, its members are few and scattered. In this work, multiple consecutive timestamps, referred to as a segment instance, is usually processed in a batch. An instance X = { EE [m- w + is a set of consecutive 2w data points (timestamps) with d features sliced from D, where [m-w: - represents an integer interval from m-w through m+ww-1. Likewise, Y = {ye LE (m-w= m+w) is set of the corresponding class labels sliced from D, where yt E (T..... K} and K is the number of classes. A model fe predicts the sequential softmax probabilities of X, i.e., fx( E [0, 1]2xxX The classification loss for training the model given X and Y is formulated as
{(X,
Σ Lu (X)((X)) 2
where H.,,)) is sparse categorical cross-entropy and ',,: means indexing at timestamp i.
Pseudo-labeling: For each instance X, using the maximum softmax probabilities conditioned on a confidence threshold T, pseudo-label Dr at each timestamp E [m-w : m)+) is derived by
Table 1: A summary of notations.
E R data point at timestamp
X
an instance, set of data points
D1 arg (I..., ()x if (X TT Inull
X
a target instance augmented instance
otherwise.
timostamp
A set Y (i tE [m- m++w) of the pseudo-labels for
half of the length of X'
X is constructed by Equation 2 Then, the classification loss for an instance X and its set Y of the pseudo-labels obtained formulated as E( X, Y) with Equation (1. R2,R4: Note that we do not store those pseudo-labels but discard them after the model update from ((X, Y) whenever a target instance is sampled for pseudo-labelingg
the middlee timestamp length half of the length of context,
Consistency regularization: Recent self-training methods force consistency between the model outputs of augmentations as the class information preserved across augmentations. For example, FixMatch matches the pseudo-label from a weak augmentation against the prediction from a strong augmentation SShn et al. 2020). That is, {(X, Ý) zw E: 1 max, (o(a()))), ((et)))))>>+(((AA( ((x((X))) it), where Út arg max, [a (((((()) )),,,k and a and AA represent weak and strong augmentations. R2.R4: The goal of consistency regularization is to offer informative supervision to update the model by diverse augmentations and reliable pseudo-labels. In the example of FixMatch, pseudo-labels from weak augmentations are reliable due to weak perturbation, and strong augmentations become diverse due to strong perturbation.
3.2 CONTEXT-ADDITIVE AUGMENTATION
Given a target instance X {x, t [m [m-o: m)))}, we add a context of total length c to its surroundings generate
Target instance
Contexts
Time series
augmented instance X' = {2t/) E [m [m-o-et Cj m +o++)))). There numerous design choices for generating X' where the context lengths a and or control the degree of perturbation and the allocation of con- text around X determines the multiplicity of aug- mentation. Increasing the multiplicity between augmented instances helps derive more informa- tive pseudo-labels, but this high multiplicity also increases the redundancy between similar augmented instances and causes inefficient computations.
X www
+ +
Figure 2: Illustration of a target instance and its context.
In this work, we follow a simple heuristic for the allocation of the context R1,22,R2 for effective consistency reguluarization. A context of a given length c is added on the left and right sides of the target instance, and thus two augmented instances, X = {xr|t E [m
m + o)}
and right = {rz[t E [m-o 0 m +)++c} are generated as shown in Figure 2
R11 R3, R44.
The underlying reason is that forcing consistency between the model outputs from the augmented
instances with more difference gives more strong supervision to the model (Wang et al.) 2022).
This heuristic is practical and reasonable, as it not only maximizes the difference between the two
4

---

Under review as a conference paper at ICLR 2023
augmented instances but also increases efficiency in further model inference due small number of augmented instances. Each augmented instance is fed to model, and pseudo-labels are generated only from the target instance (i.e., the overlap between the two augmented instances) by Equation (2). As a result, pseudo-label sets Y-left and Yright for the left and and right augmented instances, respectively, are obtained.
The value of o, which is half of the length of a target instance, needs be carefully chosen for each data set. Too large o may include semantically irrelevant data points, while too small o may not give enough temporal information for prediction. Either way leads to incorrect pseudo-labels and ultimately degrades the performance. Thus, we set 2o, the length of a target instance, as value lower than the mean length of a label-coherent segment, which could be known in advance or estimated by given set Tz of labeled timestamps.
More important factor is the length of context c, which directly affect the information quality of consistency regularization. If с is too short, then two augmented instances become too similar so that the model barely changes after matching the outputs of augmented instances (Wang et al. 2022). On the other hand, if is too long, the outputs diverge so that the matching would weld the representation of instances from different class, We empirically study this trade-off Section To prevent the potential divergence in self-training, we devise Ediability-weighee mixing of pseudo-labels generated from two augmented instances.
3.3 RELIABILITY- WEIGHTTED MIXING
Though all pseudo-labels generated for target instance satisfy the confidence threshold in Equa- tion 2 we treat them differently based on the reliability of a pseudo-label. Our rationale is that the pseudo-label becomes more reliable if (1) the model ʃo receives a larger number of data points on the left and right sides of the pseudo-label and (2) the number data points is balanced between the two sides so that the prediction is not biased to the preceding or succeeding interval. We design a reliability function that follows our rationale and compute reliability of each pseudo-label. Using the reliability as a weight, we mix the pseudo-labels into single cross-window label, which will be matched against two softmax probabilities from both of augmentations.
left.
left--
m
n4
Xlin
WWW
n
right.
right,-
n
172
m
m
C
171
+
m
+
Figure 3: Visualization of Equation (3)
Let's consider two pseudo-labels glsat and right generated from fo fx and fo(Xrigh)), shown in Figure 3 For each timestamp [m-o: m) of target instance, the length n, nooc+ of the left side and the length n !oe.- of the right side along each augmented instance are easily calculated by
loc,t
-t- ft-m+o+e
if loc left
Ioc, and n
fm+oott
if loc left
(3)
1-
if loc right
Im+o+o-t if loc right,
where loc indicates the location of context addition either left or right.
Per our design rationale, the reliability score becomes higher as n, loc,t and n, loc.- are larger and +oc,+ and noc.-- are more similar with each other. This requirement can be achieved by exploiting a bell-shaped function,
T())
2
r(p) 2p D2
(4)
where 0 S 1. Here, p is a normalization of loc,t with respect to the full length of an augmented instance, i.c., (t) = n loc,t /20 c). That is, using Equation (4), we obtain two reliability scores (()) and (t)) for each pseudo-label from the two augmented instances Xte/t and Xyight- Due to the bell shape in Figure 4 timestamp with sufficienlly long side length on both sides has an adequately high reliability in augmented instance. If side length on either side is too long (i.e.,
1
p
Figure 4: Equation
33
5

---

Under review as a conference paper at ICLR 2023
biased to one side), then the side length on the other side is too short and consequently the reliability score is too low. The reliability score is maximized at the center of a given augmented instance.
Last, in order to treat each pscudo-label differently based on the reliability scores on the two sides, (t) and r(p*)"(t)) the weight of each pseudo-label is assigned as wf = r(ploc(t) (t))/(r(p)))))))) window soft label for the target instances in the left and right instances is
+ r(p"sh(()) (t)) by normalizing the two reliability scores. The final cross-
(5) where the function A normalizes the input to make the sum of the softmax probabilities as and the onehot function converts a scalar to one-hot encoded vector. In other words, reliability-weigheee mixing generates soft label by adding and normalizing the two pseudo-labels. Using the cross- window label set and two target instance outputs, we compute classification loss as follows:
right Th (u onehot(v) + wi
lu (Xlen Xright Y)
1. 40
1 Σ Aenall (h(So )t,s, Ut) ((oo (xx Be);
(6)
where h is soft cross-entropy and Y foe (m- m+))) By matching single cross- soft label to both augmented instances, we can reduce the variance of the label to be predicted. In addition, self-traning with soft labeling is more effective than with hard labeling since soft labeling can be robust to erroneous pseudo-labels when at least one pseudo-label is correct (Müller et al. 2019).
3.4 OVERALL SELF-TRAINING PROCEDURE USING CROSSMATCH
R1,22,33.R: Figure summarizes the overall procedure of CrossMatch using context-additive aug- mentation and seaabllit-weiighhe mixing. Given a time series, CrossMatch first takes a target in- stance sampled from the time series context-additive augmentation (see Figure [a)). Then, a model fe infers the pair of augmented instances X and Xright to produce two independent softmax prohabilities. The pseudo-labels spectively generated from the two softmax probabilities are averaged (+) to give cross-window soft labels Y with siiability-weighted mixing. Finally, the cross-entropy losses are computed from the cross-window soft labels Y as well as the true labels Y and used to update the model fo (see Figure [b)). These steps are repeated with randomly sampled target instance for iterations. CrossMatch in Appendix A details each step.
input and generates pair of augmented instances with
4 EVALUATION
4.1 EXPERIMENT SETTING
Datasets: We use three widely-used bench- mark datasets summarized in Table mHealth is an action recognition dataset recorded with wearable sensors, such as 3D accelerom- eters, 3D gyroscopes, 3D) magnetometers, and electrodes, whose sampling frequency is 50Hz (Banos et al. (2014). HAPT is also a sensor time-series dataset tracking human movements in a laboratory sampled with the frequency of 50Hz (Anguita et al. 2013). Opportunity is a collection of sensor recordings at 100Hz capturing daily natural human activities with wearable, object, and ambient sensors (Roggen al. 2010). R2,R4: For each originally fully-labeled dataset, we ran- domly sample the same number of timestamps for each class and drop the labels from the rest of the timestamps to generate a sparsely-
Table Datasets and configurations.
[7]
Length
Acless
HAPT
408K
967
6
512
0.1%
mHlalthh
343K
2933
12
23
384
256
1.0%
50K
Opportunity
190K
109
17
113
512
64 1.0%
30K
time series, leaving multiple labeled time spans located randomly. The sampled timestamps become labeled timestamp set Tr for the given ratio of labeled timestamps = = [Tz |/ Table 2 summarizes the statistics of datasets and the data-specific param- eters, specifying the number of timestamps, the average length of segment with single class, the number of classes, the dimension of a data point, the half length of target instance o, the context length c, the default ratio of labeled timestamps 2, and the maximum number of iterations I.
Implementation Details: We use a popular multi stage temporal convolutional network named MS-TCN (Farha & Gall 2019). which is applicable to our problem setting as it gives softmax prob-

---

Under review as a conference paper at ICLR 2023
Table 3: Timestamp accuracy and Fl @25 score averaged over the last 20 iterations with adjusting the label ratio 2 (1x, 5x, and 10x of a default value). The best values are marked in bold.
FixMarch
FlexMatch
PropRey
CrossMatch
Method
Ratio
TS Accuracy
25
TS Accuracy
FL@25
TS Accuracy
F5
TS Accuracy
F1
1x
0.78 = 0.01
0.55 0.04
0.78 :: 0.01
0.54 = 0.04
0.79 0.01
0.52 4 0.04
0.84
0.01
0.75
0.02
НAPT
5x
0.85
4 0.01
0.51 2 0.04
0.84 1: 0.00
0.50 1 0.04
0.82 1: 0.01
0.49 4: 0.04
0.88
1: 0.01
0,70
0.02
10x
0.96
0.00
0.77+0000
0.96 + 0.00
0 0.01
0.96 0.00
0.89++0.01
0.95
0.01
0.84
0.02
1x
0:77 + 0.01
0.23 0.01
0.77 + 0.01
0.13 + 0.01
0.81 0.01
0.3 +:03
0.85
0.01
0.45 0.02
mHealth
5x
0.91
= 0.01
0.65 0.02
0,91 + 0.01
0.62 = 0.03
0.91 + 0.01
0.67 0:03
0.94
= 0.01
0,74 0.03
0.91
0.02
0.00
0.66
0.01
0.70
0,95
0.03
1x
0.61 = 0.04
0.65 1: 0.05
0.59 4: 0.03
0.63 # 0.05
0.63 1 0.03
0.65
0.05
0.67 = 0.02
0,73 0.04
Opportunity
0.73 0.03
0.73 0.04
0.72 4: 0,03
0.75 + 0.05
0.73 0.03
0.74 0.05
0.78 0.02
0.82 0.03
10x
0.75 + 0.03
0.75 0.04
0.73 + 0.03
0.77 + 0.04
0.75 0.03
0.76 0.04
0.83 0.02
0.86 0.02
abilities for each timestamp in an input instance X. We follow the same hyperparameter and config- uration in the original MS-TCN, except the learning rate and an optimizer adjusted for a self-training environment with sparse labels. Please refer to Appendix for more details.
For CrossMatch, we set the confidence threshold T to 0.95 and the weight of the unlabeled loss À to 1. The model is first trained without self-training, i.e., only using the labeled batches (Algorithm [ Line 7 We start to update a model with pseudo-labels after the number of pseudo-labels in each class for a batch is balanced. Formally, this condition is satisfied when the entropy of the numbers of pseudo-labels per class is above 0.99 for the last 100 iterations; it is enforced to prevent early confirmation bias in self-training (Kim & Lee 2022). If a data point xe in an instance X has a true label (i.e.,& t E T.). CrossMatch uses the true label instead of the generated pseudo-label.
Evaluation Metrics: We measure timestamp accuracy and segmental Fl score with five-fold cross validation and report the average value with standard deviation of five runs. For sequential classifiers such as MS- TCN, timestamp accuracy (denoted as TS accuracy) and segmental F1 score (denoted as F1 @ 25) measure the performance of classification at each timestamp and segment re- spectively (Li et al. 2021; Kumar et al. 2022). To evaluate pseudo-labeling performance, we report pseudo-label Fl score (denoted as PLF) as well. We define each metric in detail in Appendix C
Compared Self-Training Methods: We compare CrossMatch with three state-of-the-art self- training methods: FixMatch (Sohn et al. 2020), FlexMatch (Zhang al., 2021), and PropReg (Kim) & Lee 2022). As discussed in Section 2 these methods require inner-instee instance augmentation for con- sistency regularization. We use two popular time-series augmentations: jittering and scaling, where weak augmentation (xX)= = jittering( (X) and strong augmentation A(X) = seteerin(scaaingg (X)) (Um et al. 2017). Throughout the experiments, we use the same hyperparameters for all methods except the instance length due to the context-additiv augmentation. As an additional evaluation for fair comparison, we also compare CrossMatch with the variants of the compared methods which modified in support of our context-additive augmentation.
4.2 COMPARISON WITH STATE-OF-THE-AA SELF-TRAINING METHODS
Overall Comparison: Table shows timestamp accuracy and F1 @25 on three datasets with vary- ing label ratios; each value is obtained by averaging over the last 20 iterations for reliable results. Compared with other methods, CrossMatch achieves the best classification performance R1: with a statistical significance of 0.05 using independent (unpaired) t-test for all datasets except HAPT 10x. This is mainly because consistency regularization using context-additive augmentation is more in- formative than inser-instance augmentation used in other self elff-trannin methods. In particular, Cross- Match exhibits much better performance than others especially when the initial label ratio is low. For instance, with only 0.1 0.1% of the labeled timestamps in HAPT, CrossMatch outperforms FixMatch by 20p% Flex by 21p%, and PropReg by 23p% in F1 @25 (see the first row in HAPT data of Table 3. Therefore, the performance dominance of CrossMatch indicates the context-additive augmentation with Milabbiitty-eighhee mixing indeed helps the model select more reliable pseudo- labels even when softmax probabilities fluctuate due to label scarcity.
Training Curve Analysis: Figure 5 shows the training curves of classification and PLF over the entire training iteration. Please refer to Appendix D for the same results with standard deviation and R2, R3, R4. other metrics related to pseudo-labeling. CrossMatch shows R3: much higher per- formance than the other methods with respect to timestamp accuracy and Fl @25 even in the early
7

---

Under review as a conference paper at ICLR 2023
CrusMatch
FixMatch
FlexMaich
PropReg
TS Accuracy
0.9
0.9
0.8
0.7
0.6
0.7
0.5
0.4
0
12500
25000
25000
50000
0
15000 Iteration
30000
Iteration
Iteration
F1@25
0.8
0.6
App
0.8
0.5
0.3
0.6
0.2
0
12500
25000
0.0
25000 Iteration
50000
0
15000 Iteration
30000
Iteration
0.8
0.6
0.6
0.4
0.3
0.3
0.0
0.0
0.0
0
12500
25000
25000
50000
0
15000 Itcration
30000
Iteration
Iteration
(a) HAPT
(b) mHHalth
Opportunity
Figure 5: Training curve of the classification performance (the first two rows) and R1,R2: pseudo- labeling performance (the last row) over self-training iterations in Algorithm
stage of training, reaching the highest performance in most cases (see the first two rows in Figure 5 This is attributed to its robustness in reliability and its credibility in enforcing consistency between two augmented instances from context-additive augmentation. For example, low-quality pseudo-labels at the iteration of around 17, 000 when the warm-up period ends, thereby showing a temporary drop in timestamp accuracy. Although the accuracy recovers gradually, the final accuracy is far behind the accuracy CrossMatch achieves.
mixing in the early stage of self-training
shown in mHealth of Figure [5])), other methods suffer from the
The effectiveness of CrossMatch also supported by R2.R3.R4itt PLF computed from precision and recall (see the last row in Figure Figure[]). Precision is the ratio of the number of correctly predicted timestamps the number of pseudo-labeled timestamps, while recall is the ratio of the number of correctly predicted timestamps to the length of a target instance (refer to Appendix C Co for more details). We averaged PLF over the target instances in batch. CrossMatch reaches the highest PLF continuously in most cases due to the reliability of our cross-window labels.
4.3 EXTENSIONS WITH INNER- AND INTER-INSTANCE AUGMENTATION
We investigate another possible exten- sions: (I) CrossMatch to combine the two inner-instance augmentation (+IA) of jit- tering and scaling and (2) the variant existing self-training methods combine our proposed context-additive augmenta- tion (-CA). For the former, we perform jittering and scaling before applying the context-additive augmentation. For the latter, we modify the existing methods in support of the -ontett-addiivv augmenta- tion. The two augmented instances (the left and right instances Figure are re- spectively treated weakly and strongly augmented instances for existing methods; the instance with a higher reliability score than the other becomes the weakly augmented instance.
Table 4: Classification accuracy of the compared
methods with context-additive augmentation (CA) and CrossMatch with inner-instanee augmentation (IA).
HAPT
mHsalth
Dataset
Accuracy
F1625
Accuracy
F16:25
CrosMaach
0.84 +: 0.01
0.75 4 0.02
0.85 i: 0.01
0.45 4: 0.02
CrosMatch+AA
0.77
0.01
0.67
FixMarch
0.78 1 0.01
0.55 1 0.04
0.77 1 0.01
0.23 1: 0.01
FixMatch<CA
0.62 : 0.02
0.39 = 0.03
0.76 0.04
0.35 2 0.04
FlexMarch
0.54
0.01
DeeaachbCA
0.64 1: 0.02
0.49 a 0.03
0.82 à: 0.01
0.31 4: 0.03
Propleg
+: 0.01
0.52
0.81 0.01
0.43 0.03
PropReg-CA
0.68 4 0.01
0.45 = 0.02
0.77 1 0.03
0.30 1 0.03
Table 4 summarizes the performance of possible extensions for HAPT and mHealth datasets with their default label ratios of 0.1% and 1.0%. First, CrossMatch with jittering and scaling shows additional performance gain which is about 4p% in timestamp accuracy and 2-22p% in Fl @2@5 for the two datasets. These improvements demonstrate that jittering and scaling can be applied together with context-additive augmentation without performance drop under the proposed self-
8

---

Under review as a conference paper at ICLR 2023
Table 5:
Timestamp accuracy and FI @25 averaged over the last 20 iterations with varying context lengths using CrossMatch. The best values are marked in bold.
НAPT
mHealth
Opportunity
TS Accuracy
@1@25
TS Accuracy
F1
TS Accuracy
FI 6255
0.25
0.78 + 0.01
0.67 + 0.01
0.69 + 0.01
0.29 + 0.02
0.65 + 0.03
0.72 + 0.04
0.50
0.79 0.02
0.65 0.03
0.79 0.02
0.37 0.02
0.69 0.03
0.73 0.04
1.00x
0.81 0.01
0.70
0.02
0.85 0.01
0.45 0.02
0.67 0.02
0.73 0.04
x.00
0.84 0.01
0.78 0.01
0.82 1 0.01
0.38 1 0.04
0.66 1 0.02
0.71 0.03
x.00x
0.84 1 0.01
0.76 1.062
0.77 1 0.02
0.28 1 0.03
0.55 1 0.02
0.56 1 0.04
Table 6: R1. R1, Timestamp accuracy and F1 @25 averaged over the last 20 iterations with the variations of context-additive augmentation (CA). The context length for fixed-context CA set as the best value observed in Table 5 The best results are marked in bold.
Lefl-only CA
Right-only
Fixed-comtett
CrossMatch
Variations
TS Accuracy
61
TS Accuracy
F1625
TS Accuracy
F1
TS Accuracy
Fl (125
HAPT
0.01
+ 0.03
0.73
0.02
0.67 0.02
0.65
0.01
+
mHlealth
0.77 0.02
0.41 1: 0.04
0.76
0.02
0.43 0.05
0.73 1: 0.02
0.22
0.02
0.85
0.01
0.45
0.02
training framework. However, in general, the simple extensions of existing methods for context- additive augmentation rather suffer from a significant performance drop. Therefore, our method is robust the addition of existing inner-instance augmentation.
4.4 ANALYSIS OF VARYING CONTEXT LENGTHS
Table 5 summarizes the timestamp accuracy and FI @25 score of CrossMatch by adjusting the de- fault context length e (in Table 2) from 0.25 to 4.0x. If the context length too large, augmented instances bear too much perturbation hand, if the context length too small, informative consistency regularization between two aug- mented instances becomes trivial since they show high similarity. We found that the optimal context length is highly correlated with the average segment length of each dataset. For instance, the best timestamp accuracy of Opportunity data achieved with relatively smaller context length (i.e.. 64 x 0.5 = 32) than that of mHealth data (i.e., 256 x 1.0 256) because Opportunity data has much shorter mean segment length: as can be seen Table g Opportunity and mHealth ex- hibit the shortest and the longest mean segment length among the three datasets. Therefore, the best timestamp accuracy and F1 @25 score of each dataset achieved with different context lengths.
generate high-quality cross-window labels. On the other
R1,R2,R3: Table shows the classification performance of CrossMatch with the variations of context-additive augmentation (CA). The context is added to only either side a target instance (i.e., left-only CA or right-only CA), and the length of the context is fixed rather than varying (i.e.. fixed-context CA). We set the fixed length to the best value found in Table These variations weaken the diversity of augmented instances, which degrade the effect of consistency regularization, all of them result in lower classification performance than Crosh.atch.
CONCLUSION
In this paper, we propose a novel time-series self-training method CrossMatch equipped with context-additive augmentation. It adds a context instance to target instance on its left and right sides within window to generate two augmented instances with different contexts. To reduce vari- ance in pseudo-labeling. CrossMatch mixes the two sets of pseudo-labels obtained from the two augmented instances with inferred reliability scores. Our extensive experiments demonstrate that CrossMatch achieves considerably higher classification accuracy than other state-of-the-an self- training methods-by up to 23p% even when only 0.1% 1% of timestamps are labeled. CrossMatch introduces a new direction of data augmentation for sequential data and has the potential to be ap- plied to variety of time-series applications. For further work, plan theoretical analysis of the optimal context length and performance maximization of self-training using context- addtext-additive aug- mentation.

---

Under review as a conference paper at ICLR 2023
CODE OF ETHICS
In this paper, we propose a time-series self-training framework with context-additive augmentation. Our algorithm improves model performance despite sparsity labels time-series data, and this improvement can reduce the cost of label annotations. In experiments we use popular time-series datasets open to the public and available from the websites the links provided in the references. Each human subject in the datasets is encoded as a random number for anonymization.
REPRODUCIBILITY
We elaborate on the details of our algorithm and dataset setting in the main text for reproduction. Section 4.T and Appendix [ describe the dataset setting for sparsely-labeled time series and ex- plain hyperparameters used in the training and configuration. Algorithm summarizes our frame- work as a pseudo-code for better understanding. Finally, our work can be reproduced using the guidelines and the source codes in https: www dropbox.....oooooooo.oooooo..o.. com/ sh/ 12n2hr wtt.beeeess....
AADCRye a 40 www.cowww2gaadddeeoom
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al. A public domain dataset for human activity recognition using smartphones. In ESANN, pp. 2013.
Oresti Banos, Rafael Garcia, Juan A Holgado-Terrizz Miguel Damas, Hector Pomares, Ignacio Rojas, Alejandro Saez, and Claudia Villalonga. mHHalthDroid: A novel framework for agile development of mobile health applications. In Ambient Assisted Living Workshop, pp. 91-98, 2014.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. In ICLR, 2020.
Dong-Dong Chen, Wei Wang, Wei Gao, and Zhi- Hua Zhou. Tri-net for semi-supervised deep learn- ing. In IJCAI, pp. 2014-2020 2018a.
Minghao Chen, Fangyun Wei, Chong Li, and Deng Cai. Frame-wise action representations for long videos via sequence contrastive learning. In CVPR, pp. 13801-13810, 2022.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. NeurlPS, pp. 22243-22255, 2020.
Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Semi-supervised deep learning with memory. In ECCV, pp. 268-283, 2018b.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In CVPR, 2019.
Shohreh Deldari, Daniel V Smith, Hao Xue, and Flora D Salim. Time series change point detection with self-supervised contrastive predictive coding. In WebDef, pp. 3124-3135, 2021.
Terrance Dree and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint /708v 2017.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In IJCAI, pp. 2352-2359, 2021.
Haoyi Fan, Fengbin Zhang, Ruidong Wang, Xunhua Huang, and Zuoyong Li. Semi-supervised time series classification by temporal relation prediction. In ICASSP, pp. 3545-3544 2021.
10

---

Under review as a conference paper at ICLR 2023
Yazan Abu Farha and Jurgen Gall. MS-TCN: Multi-stage temporal convolutional network for action segmentation. In CVPR, pp.
2019.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D. Cubuk, Quoe V. Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In CVPR, pp. 2918-2928 2021.
Jann Goschenhofer, Rasmus Hvingelby, David Rügamer, Janek Thomas, Moritz Wagner, and Bernd Bischl. Deep semi-supervised learning for time series classification. In ICMLA, pp. 422 428, 2021.
Jean- Cann-Battinn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurlPS, pp. 21271- 21284, 2020.
Brian Kenji Iwana and Seiichi Uchida. An empirical survey of data augmentation for time series classification with neural networks. Plos One, 16(7)=e0254411, 2021.
Shayan Jawed, Josif Grabocka, and Lars Schmidt- Thiemme Self-supervised learning for semi- supervised time series classification. In PAKDD, pp. 499-511, 2020.
Noo-ri Kim and Jee-Hyong Lee. Propagation regularizer for semi-supervised learning with tremely scarce labeled samples. In CVPR, pp. 14401-14410, 2022.
Sateesh Kumar, Sanjay Haresh, Awais Ahmed, Andrey Konin, Zeeshan Zia, and Quoc-Huy Tran. Unsupervised action segmentation by joint representation learning and online clustering. In CVPR, pp. 201 20174-20185 2022.
Samuli Laine and Timo Aila. Temporal ensembling for semi- semi-supervised learning. In ICLR, 2017.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, pp. 896, 2013.
Zhe Li, Yazan Abu Farha, and Jurgen Gall. Temporal action segmentation from timestamp supervi- sion. In CVPR, pp. 8365-8374, 2021.
Fan Ma, Linchao Zhu, Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and Zheng Shou. SF-Net: frame supervision for temporal action localization. In ECCV, pp. 420 420-437, 2020.
Qianli Ma. Zhenjing Zheng, Jiawei Zheng, Sen Li, Wanqing Zhuang, and Garrison W Cottrell. Joint-label learning by dual augmentation for time series classification. AAAI, pp. 8847-8855, 2021.
Davide Moltisanti, Sanja Fidler, and Dima Damen. Action recognition from single timestamp pervision untrimmed vidcos. CVPR, pp. 9915-9924, 2019.
Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? Ad- vances in neural information processing systems, 32, 2019.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In CVPR, pp. 11557- 11568, 2021.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An imcertainty-awant pseudo-label selection framework for semi-supervised learning. In ICLR, 2021.
Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, al. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh international conference networked sensing systems (INSS), pp. 233-2400 IEEE, 2010.
Yooju Shin, Susik Yoon, Sundong Kim, Hwanjun Song, Jae-Gil Lee, and Byung Suk Lee. Coherence-ban label propagation over time series for accelerated active learning. In ICLR, 2022.
11

---

Under review as a conference paper at ICLR 2023
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Ankit Singh, Omprakash Chakraborty, Ashutosh Varshney, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Semi-supervised action recognition with temporal contrastive learning. In CVPR, pp. 10389-10399, 2021.
Dipika Singhania, Rahul Rahaman, and Angela Yao. Iterative contrast-classify for semi-supervised temporal action segmentation. In AAAI, pp. 2262-2270, 2022.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NewrIPS, 33:596 608, 2020.
Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. Data augmentation of wearable sensor data for parkinson's 's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international conference on multimodal interaction, pp. 216 216-220, 2017.
Jesper Van Engelen and Holger H Hoos. survey on sepi-supervised learning. Machine Learn- ing, 109(():373-44)) 2020.
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In CVPR, pp. 16570-16579, 2022.
Qingsong Wen, Liang Sun, Fan Yang. Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: survey. In IJCAI, 2021.
Junfei Xiao, Longlong Jing, Lin Zhang, Ju He, Qi She, Zongwei Zhou, Alan Yuille, and Yingwei Li. Learning from temporal gradient for semi-supervised action recognition. In CVPR, pp. 3252- 3262, 2022.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In NeurIPS, 2020.
Yinghao Xu, Fangyun Wei, Xiao Sun, Ceyuan Yang, Yujun Shen, Bo Dai, Bolei Zhou, and Stephen Lin. Cross-moded pseudo-labeling for semi-supervised action recognition. In CVPR, pp. 2959- 2968, 2022.
Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-suprrrie learning. CoRR, abs/2103.00555 2021. https://: http:p::////xv.....// org wbw.////l.... 00550
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAI, pp. 8980-8987, 2022.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp- 6023-6032, 2019.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting sumi-super learning with curriculum pseudo la- beling. NeurlPS, pp. 18408-18419, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In /CLR, 2018.
12

---

Under review as a conference paper at ICLR 2023
A OVERALL TRAINING ALGORITHM
Algorithm Time-series self-training with CrossMatch
Input: A time series with initial labels D, labeled timestamp set Tz. unlabeled timestamp set Tü, labeled batch size Bt, unlabeled batch size Bu, a model fe, confidence threshold т, loss weight X, learning rate 77, half the length of target instance o, context length c, max number of iterations I.
Output: Final model fe.
I: for each iteration up to I do
2: T f Sample Bt timestamps from Tr: To +- Sample B,, timestamps from Tr:
3:
c' <- Sample context length from Uniform(2,c):
4:
w < +d;;
5:
f** Loss computation for labeled batch **/
6:
X1 + { (H(m-w:mtu)) m. E Ti): 3/- < {) (m-ummtu)) t Ti):
7:
L B, ExEX, {(X,))
// See Equation
8:
*** Loss computation for unlabeled batch #*/
9:
N ((((m-wmmto) Xmm-ammtu)) ), m Tu):
// See Section 3.2
10:
y. & PSEUDOLABELINGI fe, T);
// See Equation 2
11:
Y + RILIABILTTVEIIITINNIOO..
// See Section 3.3
12:
Lu = By Eix Ex. lu (x X Y);
// See Equation
13:
C+ ALui 1 + VoC:
14: return fo:
Algorithm describes how CrossMatch works in time-series self-training. For each iteration, there two steps for batch initialization: center timestamp sampling from Tr and To (Line 2 and context length sampling from a uniform distribution (Line 3). Note that we assign w as o 0 after
the sampling (Line 4p. From labeled middle timestamps, instances are sliced from X' with length 2w to construct a labeled batch, Xj and }%, where X (m-wmm-e) = [ 1 E [m-w : m m++)) w)) and
y (im-ummtt) = {ve, E [m-w w m }w]} (Line 6 The classification loss for a labeled batch C
the batch (Line From unlabeled middle timestamps, instances are sliced with context- additiee augmentation that generates a pair of instances whose length is o+ (Line 9). The target instance of each instance in an augmented unlabeled batch X, is then pseudo-labeled using a confidence threshold - (Line (O). CrossMatch softens the pseudo-labels with reliability weighting across two instances with different contexts, transforming two pseudo- labels into a single cross-window label shared across the instances (Line II]. The classification loss with cross-window labels is computed for each pair of augmented instances and is averaged over the batch (Line [2]. Finally, the losses for labeled and unlabeled batches are then integrated into single loss L with a hyperparameter X, and the model fa is updated using its gradient (Line) 13).
computed and averaged
B DETAILS OF TRAINING THE CLASSIFIER
R2: As described in implementation details of 4.1 we use MS-TCN as backbone sequential clas- sifier (Farha & Gall 2019). It can classify each data point in segment instance X, generating sequential softmax probabilities at each timestamp. MS-TCN has four stages, and each stage is composed of eleven dilated convolution layers and softmax output layer. The first stage takes a subsequence of the whole time series and outputs softmax probability distribution at each times- tamp. After the first stage, every stage fed with softmax probabilities and then outputs another softmax probabilities. For all datasets, we use the same training hyperparameters and classifier as listed in Table 7 We set the labeled batch size as 4 and the unlabeled batch size as 8, use SGD optimizer with momentum and Nesterov method. The initial learning rate is 0.005 and is scheduled with a cosine decay function. The letter in Scheduling denotes the current iteration number during training, and the letter 7 denotes the total number of iterations.
13

---

Under review as a conference paper at ICLR 2023
Table 7: Training hyperparameters.
Stage
Layer
Bg
Bu
Optimizer
Momentum
Nesterov
Scheduling
11
SGD
True
0.005
cos (Ti)
C DETAILS IN EVALUATION METRICS
Timestamp accuracy is the ratio of the number of timestamps with correctly predicted labels the total number of timestamps a times series, computed as follows.
TS accuracy
1y
|Tisst
Segmental F1 score is performance measure for judging whether a classifier outputs correct and coherent labels for consecutive timestamps. Segmental precision and recall are first computed by counting the number of correct matches between predicted segments set Y and segments Y with Jaccard similarity threshold (here, we set 0.25) as follows.
Precision
Σ
Σ
Recall
Σ
Y
Jaceard(y .Y)50.25
Y
Juccard(( (2)20.25
YE& YEY
FERYEY
Finally, Segmental Fl score is computed as F1@25
2-precisiomereaall
precision++
each iteration, we measure pseudo-label precision and pseudo-label recall from each times- tamp of target instances in a batch, as follows:
PL Precision the number of correct PLs
PL Recall the number of correct PLs
the number of PLs
the number of timestamps
For every timestamp, we check the existence of pseudo-label and the class of the true label the class of a pseudo-label is correct to count the numerators and denominators the above formula. When there no pseudo-label any timestamp, the denominator of precision becomes 0 and the precision can diverge. So, we define the precision with no pseudo-label as O. Finally, pseudo-label F1 score is PLF 2aPL precision= recall After a few iterations, the entire data is expected to be used PL precision++PL.
by pseudo-labeling the sampled target instances.
D DETAILED ILLUSTRATIONS
Figure 6 is detailed version of Figure with standard deviation error bars added. R2,R3.R Figure 7 shows the pscudo-labeling metrics such as precision, recall, and the total number of pseudo-labels. Figure g shows the accuracy convergence trend when context length is varied factor multiple of the context length c, illustrating accuracy curve during the training iterations.
14

---

Under review as a conference paper at ICLR 2023
CrusMatth
FixMatch
FlexMaich
PropReg
TS Accuracy
0.9
0.9
0.8
0.8
0.7
0.6
0.7
0.5
0.4
12500
25000
25000
50000
15000 Iteration
30000
Iteration
Iteration
0.8
0.6
0.8
F1,022
0.5
0.3
0.6
0.2
0.0
0.4
12500
25000
25000
50000
15000 Iteration
30000
Iteration
Iteration
PLF
0.4
0.3
0.3
0.0
0.0
0.0
12500 Iteration
25000
25000 Iteration
50000
15000
30000
Iteration
(a) HAPT
(b) mlHealhh
(c) Opportunity
Figure 6: Training curve of the classification performance (the first two rows) and pseudo-labeling performance (the last row) over self-training iterations in Algorithm Error bars represent the standard deviation of five runs.
CrosaMatch
FixMatch
FlexMatch
PropReg
Precision
1.0
1.0
1.0
0.8
0.7
0.8
0.6
0.4
0.6
12500
25000
25000
50000
15000
30000
Iteration
Iteration
Iteration
Recall
0.3
0.2
0.2
0.0
0
12500
25000
0.0
25000
50000
0.0
0
15000
30000
Iteration
Iteration
Iteration
Number of Pt.
Number PL
Number of PL
2e4
2e4
1e4
le4
164
0
12500 iteration
25000
25000 iteration
50000
15000 iteration
30000
(a) HAPT
(b) mHHeath
(c) Opportunity
Figure 7: Precision, recall, and the number of pseudo-labels of Cross compared with those of FixMatch, FlexMatch, and PropReg.
15

---

Under review as a conference paper at ICLR 2023
0.25x
0.5x
1x
2X
4x
TS Accuracy
0.9
0.8
0.7
0.7
0.5
0.4
12500
25000
25000
50000
0
15000 Iteration
30000
Iteration
Iteration
0.9
0.6
0.8
11222
0
12500 Iteration
25000
25000 Iteration
50000
0
15000 Itcration
30000
(a) HAPT
(b) mlalth
(c) Opportunity
Figure 8: Classification accuracy of CrossMatch for varying context length
16


Reviews:
[{"review_id": "pBnoJWOEG0", "reviewer": "Reviewer_4aZf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, a method for time series self training is being proposed. The model is named CrossMatch and it utilizes context-additive augmentation, in other words, it adds a context instance to the left and the right part of the time-series to generate two augmented views with different contexts. Then, the authors design a reliability function for more reliable pseudo-labels, and they mix the pseudo-labels into a single cross-window label, which will be matched against two softmax probabilities from both of augmentations. The CrossMatch model is test against three other state of the art models (FixMatch, FlexMatcha and PropReg) in three publicly available datasets (HAPT, mHealth and Opportunity). The results show that in most cases the proposed model outperforms the rest in segmented Accuracy and segmented F1 Score. In the experimental setup, the authors also show results for the inter- and inner-instance augmentations analysis and an analysis of varying context length for the augmented instances.", "summary_of_the_review": "Overall, I enjoyed reading this paper. It is easy to follow, well-written and well-structured. The authors do a good job providing the motivation, the challenges and the related works. A few suggestions for improvements can be found in this review. The results show that the proposed model outperforms the rest of the comparison methods in three real world datasets..", "strength_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow. The authors have done a good job in the writing and the structure of the paper.\n- The problem is interesting to the research community.\n- The authors have cover the related works and show what are the current challenges and how the proposed method will help on those and increase performance.\n- The authors have conducted an extensive experimental setup and the results are very promising as they show that in most cases the proposed model outperforms the comparison methods.\n- The authors have included analyses about the augmentations types that have been selected and the varying context length for the augmented instances.\n- The authors will release the code.\n\nWeaknesses:\n- The novelty of the paper is incremental as the authors rely on existing methodology, however they do add extra analyses and they do have their own contributions.\n- It would be interesting to discuss examples of the additive augmentations that have been successful from the datasets in the experimental setup.\n- The whole idea of either left or a right augmentation needs more motivation. Why do the authors propose only one of the two? Because in this concept the time is changing, how do only left (previous TS segments) perform? How does the performance get affected by only right additive augmentations? What does that mean for the TS segments on the right (future)? Have the authors experiment with combinations of the augmentations?\n- The idea of the notation and equations etc to be calculated based on the half and the middle timestamp of the time-series segment is a bit confusing, wouldn’t it be easier to follow if the paper was written with 1/2*{length of X’} for example?\n- In the definitions of the evaluation metrics, what is the relation of the segmental accuracy and F1-Score to Jaccard similarity? This particular paragraph needs a better writing, as it is not clear how the segmental scores are being calculated, which is important for the understanding of the results.\n- In Table 3, TS Accuracy in FixMatch, FlexMatch and PropReg should all be in bold, as they are all the same.\n- The results in the tables of the experimental setup, have small differences between the proposed and the comparison methods. Have the authors checked for statistical significance in those results?\n- It is not clear what the iteration axis refers to in Figure 5.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mPgkH_JrJi", "reviewer": "Reviewer_gKcX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Edit: I would like to thank the authors for explaining their method and updating the scripts. Although they resolved several of my concerns,  I think the proposed method of using context is not very new, but using left and right context has some novelty, but not very high. \n\nAs the authors said, the labels are spreading evenly with one class per segment. In addition, the label length is relatively long without switching often. F1 score might not be that sensitive. The problem setting is relatively ideal. I am not sure whether we need this method as maybe some simpler solution would work in this case. \n\nThus, my score will remain the same. \n\n\n---------------------------------------------------------------------------------------------------------------------------------------------------------\nOriginal review: \n\nThis paper proposed a new framework for self-training for generating pseudo labels for a long time series. The proposed framework propose some training strategy on the few label cases and then use post-processing by utilizing the left and the right context to reweight the labels. \n\n", "summary_of_the_review": "I will consider raising my evaluation if my concerns are addressed by the authors. I think the problem setting, the overall model, and the experiment setting needs more explanation. \n\n", "strength_and_weaknesses": "Edit: \nUsing a segment for context in time series is not really new but contrasting \n\n\nStrength: \n1. The problem is very important since time series are often lacking of labels. \n2. The performance is better than the state-of-the-art. \n\nWeakness: \n1. unclear problem setting\n2. unclear experiment setting\n3. the proposed method is very heuristic and not clean. \nsee below in detailed comments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KAHis73eHOP", "reviewer": "Reviewer_Cx6c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel data augmentation method called context-additive augmentation. This method is easily-implemented, and preserves a target instance without perturbing it. Based on this method, the authors additionally propose a cross-window self-training framework. This framework employs reliability-based cross-window labels to improve consistency among augmented instances. The experiment results show the good performance of the proposed method. ", "summary_of_the_review": "This paper proposes a novel data augmentation method for self-supervised learning. The experiment results show the good performance of the proposed method. ", "strength_and_weaknesses": "Strength: \n1. The idea of reliability-weighted mixing is interesting. This can help readers design other soft-mixing methods.\n2. The results of the experiments show that the proposed method achieves SOTA performance.\n\nWeakness:\n1. One might don't agree with the rationale in Section 3.3 and the curve r(p) should be. In my opinion, the reliability value of the middle point of the augmented instance is maximum is not proper. In my opinion, the overlap ( m-o to m+o) should have the largest reliability. If the rationale you proposed holds, assuming that c < 2o, the m - c/2 -th position on X_left will have much larger reliability than the same position but on X_right. Please explain the intuition of your rationale.\n2. It is confused with the augmentation method you proposed. Assuming that the target instance is X[m-o, m+o], the generated instances are X[m-o-c, m+o] and X[m-o, m+o+c]. In my option, this is equal to the case that the target is X[m-o-c, m+o+c] and cut the target into X[m-o-c, m+o] and X[m-o, m+o+c]. So I think the novelty is limited.\n3. The statements in Section 3 sometimes are not clear enough. The details can be referred to as the minor weaknesses below. Please modify the statements in order to make the paper easily understood by the readers.\n \nMinors:\n1. In Section 3.2, the authors say that the heuristic \"not only maximizes the difference between the two augmented instances but also minimizes the overlap between them\". However, I am not sure what is the difference between \"maximizing the difference\" and \"minimizing the overlap\". I think the authors just expressed the same meaning in two ways. If there is some difference, please explain it in detail.\n2. The purpose of Figure 5 is unclear. Since the accuracy has been shown in the tables, and it is difficult to see that CrossMatch converges most quickly. Even if it converges quickly, the speed of convergence is likely related to the learning rate.\n3. Section 4.4 only shows the performance under different \"c\"s. More discussions are recommended.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4tXPceJvd4Y", "reviewer": "Reviewer_Rri3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the concept of context additive augmentation for self-supervised training for sparsely labelled time series. The idea is to propagate the labels on surrounding timesteps, and then used the pseudo labels for self-training. This work explores multiple real-world datasets to compare performance against other state-of-the-art approaches.", "summary_of_the_review": "The paper proposes an interesting heuristic that could be strengthened by clarifications.", "strength_and_weaknesses": "The paper explores an important problem as time series often present sparse labelling.\n\nHowever, the clarity of the paper weakens its impact. The introduction contains all necessary elements but would benefit from clarifications and examples to give the intuition behind the proposed heuristic. \"Target instance\", \"by adding surrounding sequences\", and \"weak and strong augementaions\" are terms that need to be more clearly introduced. The specific problem should also be introduced earlier, the paper seems to focus on temporal prediction for which each time step needs to be classified. At the end of the introduction, it is still not clear what challenges the proposed method addresses and what the motivation for this approach is.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject