Self-Supervised Time Series Representation Learning by Inter-Intra Relational Reasoning
Authors: Haoyi Fan
Fengbin Zhang
Yue Gao
Venue:  (2021)
Keywords: self-supervised learning
time series
deep learning
relational reasoning

Abstract: Self-supervised learning achieves superior performance in many domains by extracting useful representations from the unlabeled data. However, most of traditional self-supervised methods mainly focus on exploring the inter-sample structure while less efforts have been concentrated on the underlying intra-temporal structure, which is important for time series data. In this paper, we present SelfTime: a general self-supervised time series representation learning framework, by exploring the inter-sample relation and intra-temporal relation of time series to learn the underlying structure feature on the unlabeled time series. Specifically, we first generate the inter-sample relation by sampling positive and negative samples of a given anchor sample, and intra-temporal relation by sampling time pieces from this anchor. Then, based on the sampled relation, a shared feature extraction backbone combined with two separate relation reasoning heads are employed to quantify the relationships of the sample pairs for inter-sample relation reasoning, and the relationships of the time piece pairs for intra-temporal relation reasoning, respectively. Finally, the useful representations of time series are extracted from the backbone under the supervision of relation reasoning heads. Experimental results on multiple real-world time series datasets for time series classification task demonstrate the effectiveness of the proposed method. Code and data are publicly available.

Under review as a conference paper at ICLR 2021
SELF-SUPERVISED TIME SERIES REPRESENTATION LEARNING BY INTER-INTRA RELATIONAL REASON- ING
Anonymous authors Paper under double-bling brind review
ABSTRACT
Self-supervised learning achieves superior performance many domains by ex- tracting useful representations from the unlabeled data. However, most of tradi- tional sulf--sue methods mainly focus on exploring the sater- struc- ture while less efforts have been concentrated structure, which is important for time series data. In this paper, we present Self- Time: a general Self-supervised Time series representation learning framework, by exploring the inter-sample relation and intra-temporal relation of time series to learn the underlying structure feature on the unlabeled time series. Specifically, we first generate the inter-sample relation by sampling positive and negative sam- ples of a given anchor sample, and intra- untra-temporal relation by sampling time pieces from this anchor. Then, based on the sampled relation, shared feature extraction backbone combined with two separate relation reasoning heads are employed quantify the relationships of the sample pairs for inter-sample relation reasoning, and the relationships of the time piece pairs for intra-temporal relation reasoning, respectively. Finally, the useful representations of time series are extracted from the backbone under the supervision of relation reasoning heads. Experimental re- sults multiple real-world time series datasets for time series classification task demonstrate the effectiveness of the proposed method. Code and data are publicly available
the underlying intra-temporal
INTRODUCTION
Time series data is ubiquitous and there has been significant progress for time series analysis (Das) 1994] in machine learning, signal processing, and other related areas, with many real-world applica- tions such as healthcare (Stevner et al., 2019), industrial diagnosis (Kang et al. 2015). and financial forecasting (Sen al. 2019).
Deep learning models have emerged as successful models for time series analysis (Hochreiter & Schmidhuber 1997] Graves al.) 2013 Shukla & Marlin 2019 Fortuin al. 2019 Oreshkin et al.. (020)) Despite their fair share of success, the existing deep supervised models are not suitable for high-dimensional time series data with a limited amount of training samples as those data-driven approaches rely on finding ground truth for supervision, where data labeling is labor-intensive and time-consuming process, and sometimes impossible for time series data. One solution to learn useful representations from unlabeled data, which substantially reduce dependence on costly manual annotation.
Self-supervised learning aims to capture the most informative properties from the underlying struc- ture of unlabeled data through the self-generated supervisory signal to learn generalized representa- tions. Recently, sulf-supe learning has attracted more and more attention in computer vision by designing different pretext tasks on image data such as solving jigsaw puzzles (Noroozi & Favaro, 2016), inpainting (Pathak al. 2016). rotation prediction (iidars
al.) 2018), and contrastive learning of visual representations(Chees et al.] (020)) and on video data such as object tracking
Wang & Gupta 2015), and pace prediction (Wang et 2020). Although some video-base ap-
Anonymous repository link.

---

Under review as a conference paper at ICLR 2021
Positive Sample
Inter-sample Relation
Piece
Piece Sampling
Anchor Sample
Negative Sample
Sample Relation:
Positive
Negative
intra-temporal Relation
Temporal Relation:
Short-term
Middle-term
Long-term
Figure Exploring inter- relation, and multi-scale intra-temporal relation of time series. Here, an example of 3-scale temporal relations including short-term, middle-term and long-term temporal relation is given for illustration.
proaches attempt to capture temporal information in the designed pretext task, time series far dif- ferent structural data compared with video. More recently, in the time series analysis domain, some metric learning based sep-ssuee methods such as triplet loss (Franceschi et al. 2019) and con- trastive loss (Schneider et al. 2019 (Saeed et al. 2020). multi-task learning based self-supervised methods that predict different handcrafted features Pascual et all||2019 Ravanelli et al. 2020) and
different signal transformations (Saced et al. 2019 Sarkar & Etemad] 2020p have emerged. How- ever, few of those works consider the intra-temporal structure of time serie. Therefore, how to design an efficient pretext task in self-supprvised manner for time series representation learning is still an open problem.
In this work, present SelfTime: general superssu time series representation learning framework. Inspired by relational discovery during self-supervised human learning, which attempts discover new knowledge by reasoning the relation among entities (Goldwater al., 2018 Patac- chiola & Storkey| 2020). we explore the inter-sample relation reasoning and intra-temporal relation reasoning of time series to capture the underlying structure pattern of the unlabeled time series data. Specifically, as shown in Figure for inter-sample relation reasoning, given an anchor sample, we generate from its transformation counterpart and another individual sample as the positive and nega- tive samples respectively. For intra-temporal relation reasoning, we firstly generate an anchor piece, then, several reference pieces are sampled to construct different scales of temporal relation between the anchor piece and the reference piece, where relation scales are determined based on the temporal distance. Note that in Figure [ we only show an example of 3-scale temporal relations including short-term, middle-term. and long-term relation for illustration, whereas in different scenarios, there could be different temporal relation scale candidates. Based on the sampled relation, shared feature extraction backbone combined with two separate relation reasoning heads employed to quantify the relationships between the sample pairs or the time piece pairs for inter-sample relation reasoning intra-temporal relation reasoning, respectively. Finally, the useful representations of time series are extracted from the backbone under the supervision of relation reasoning heads on the unlabeled data. Overall, SelfTime is simple yet effective by conducting the designed pretext tasks directly on the original input signals.
Our main contributions are three-fold: (1) we present a general self-supervised time series represen- tation learning framework by investigating different levels of relations of time series data including inter-sample relation and intra-temporal relation. (2) We design simple and effective antra-temporal relation sampling strategy to capture the underlying temporal patterns of time series.
(3) We con- duct extensive experiments on different categories of real-world time series data, and systematically study the impact of different data augmentation strategies and temporal relation sampling strategies on self-supervised learning of time series. By comparing with multiple state-of-the-art baselines, experimental results show that SelfTime builds new state-of-the the-ttr-tt on self- self-supervised time series representation learning.
RELATED WORK
Time Series Modeling. In the last decades, time series modeling has been paid close attention with numerous efficient methods, including bissedce-based methods, feature-based methods, ensemble- based methods, and deep learning based methods. Distance-based methods (Berndt & Clifford
2

---

Under review as a conference paper at ICLR 2021
Górecki & Luczak 2014) try to measure the similarity between time series using Euclidean distance or Dynamic Time Warping distance, and then conduct classification based on I-NN classi- fiers. Feature-based methods aim to extract useful feature for time series representation. Two typical types including bag-of-feature based methods (Baydogan et al. 2013: Schäter] 2015) and shapelet based methods (Ye & Keogh| 2009; Hills al.) 2014). Ensemble-
1994
methods (Lines & Bagnall 2015 BBanall et al. 2015) aims at combining multiple classifiers for higher classification perfor- mance. More recently, deep learning based methods (Karim et al., 2017 Ma et al.) 2019 Cheng et al. 2020) conduct classification by cascading the feature extractor and classifier based on MLP,
RNN, and CNN in an end-to-end manner. Our approach focuses instead on self-supervised represen- tation learning of time series on unlabeled data, exploiting inter-sample relation and intra-temporal relation of time series to guide the generation of useful feature.
Relational Reasoning, Reasoning the relations between entities and their properties makes signifi- cant sense to generally intelligent behavior (Kemp & Tenenbaum 2008). In the past decades, there has been an extensive researches about relational reasoning and its applications including knowl- edge base (Socher et al.] 2013), question answering (Johnson al.] 2017 [Santoro et al.; (017). video action recognition Zhou et al.) 2018), reinforcement learning (Zambaldi et al., 20119) and graph representation (Battaglia et al.) 2018), which perform relational reasoning directly on the con- structed sets or graphs that explicitly represent the target entities and their relations. Different from those previous works that attempt to learn a relation reasoning head for a special task, inter-sampl relation reasoning based on unlabeled image data is employed in (Patacchiola & Storkey 2020) to learn useful visual representation focus on time series data by exploring both inter-sample and intra-temporal relation for time series representation in self-supervised scenario.
the underlying backbone. Inspired by this, in our work, we
Self-supervised Learning. Self-supervised learning has attracted lots of attention recently in dif- ferent domains including computer vision, audio/speech processing, and time series analysis. For image data, the pretext tasks including solving jigsaw puzzles (Noroozi & Favaro 2016), rotation prediction (Gidaris et al. 2018), and visual contrastive learning (Chen al.] 2020) are designed for self-supervised visual representation. For video data, the pretext tasks such as frame order validation
(Misra et al. 20166] Wei et al. 2018). and video pace prediction (Wang et al. 2020) are designed which considering additional temporal signal of video. Different from video signal that includes plenty of raw feature in both spatial and temporal dimension, time series is far different structural data with less raw features at each time point. For time series data such as audio and ECG, the metric learning based methods such as triplet loss (Franceschi et al. 2019) and contrastive loss (Schneider et al. 2019 (Seed et al.) 2020), or multi-task learning based methods that predict different hand- crafted features such as MFCCs, prosody, and waveform (Pascual et al. [209a]] Ravanelli et al. 2020), and different transformations of raw signal (Sarkar & Etemad 2020 Saced et al. 2019) have emerged recently. However, few of those works consider the intra-tempora structure of time series. Therefore, how to design an efficient supe-supe pretext task capture the underlying structure of time series is still an open problem.
3 METHOD
Given an unlabeled time series set T = (b where each time series bn = (I ...b, contains T ordered real values. We aim to learn useful representation in so st() from the backbone encoder f-(-) where 0 is the learnable weights of the neural networks. The architecture of the proposed SelfTime is shown in Figure [ which consists of an inter-sample relational reasoning branch and an intra-temporal relational reasoning branch. Firstly, taking the original time series signals and their sampled time pieces as the inputs, a shared backbone encoder fo(.) extracts time series feature and time piece feature to aggregate the inter-sample relation feature and intra-temporal relation feature respectively, and then feeds them to two separate relation reasoning heads r, (-) and (.)))) to reason the final relation score of inter-sample relation and intra-temporal relation.
3.1 INTER-SAMPLE RELATION REASONING
Formally, given any two different time series samples bm and tn from T, we randomly generate two (5) K
sets of K augmentations A(tm. (1(0) 1== and A(t..) (+ ) J== where (0) and 1/0 the i-th
3

---

Under review as a conference paper at ICLR 2021
Input
Backbone Encoder
Sample Feature
Relation Feature
Relation Reasoning Head
Output
Inter Inter-Sampp Sample Relational Reasoning
fe(.)
Positive Relation ?
Negative Relation ?
Temporal Relation Sampling
Intra Temporal Relational Reasoning
Relation Class 1
fe(.)
Relation Class
Tip
Relation Class C?
Figure 2: Architecture of SelfTime.
augmentations of im and i, respectively. Then, construct types of relation pairs: positive relation pairs and negative relation pairs. positive relation pair is (tlf) +(()) sampled from the
same augmentation A(tm), while negative relation pair is (tm (G) sampled from different augmentation sets A(tm) and A(t,). Based on the sampled relation pairs, we use the backbone encoder fe learn the relation representation follows: Firstly, we extract sample representations 2(1) = fo(mm))
(1) ). 2m = fe((t))) and z!! = fo(t?"). Then, we construct the positive relation
-5)
representation [=m _2m"), and the negative relation representation [=1 _201),, where [...] denotes the vector concatenation operation. Next, the inter-sample relation reasoning head r.)()) takes the
generated relation representation input reason the final relation score h (4,5) r ([= =
(i) ==² for negative relation, respectively. Finally, the for positive relation and (a,j) [
inter-sample relation reasoning task is formulated as a binary classification task and the model is trained with binary cross-entropy loss Linter follows:
2N
K
K
Linter
(65)
(bg( + (1 (1-y!- ((:d))) logg1 A(()))))
(1)
n=1
where (1.)) = 1 for the positive relation and yn (t5) = 0 for the negative relation.
3.2 INTRA-TEMPORAL RELATION REASONING
To capture the underlying temporal structure along the time dimension, we try to explore intra- temporal relation among time pieces and ask the model to predict the different types of temporal we define an L-length time piece Pa,u of t, starting at time step u as a contiguous subsequence Pn,u = (tn,us fn.411..... Ln Firstly, sample different types of temporal relation among time pieces follows: Randomly sample two L-length pieces Pr,u and Pr,v of t, starting at time step u and time step e respectively. Then, the temporal relation between Pr.u and Pr.. is assigned based du.,, e.g., for similarity, we define the temporal distance du,, difference between two starting step and Next, define C types of temporal relations for each pair of pieces based on their temporal distance, e.g., for similarity, we firstly set distance threshold as [T/C]. and then, if the distance du,, of piece pair less than D, we assign the relation label as O, if du is greater than D and less than 2D, we assign the relation label as 1, and so on until we sample types of temporal relations. The details of the intra-temporal relation sampling algorithm are shown in Algorithm [
relation. Formally, given time series sample t, = (tr.1:
their temporal distance v) as the absolute value of the
Based on the sampled time pieces and their temporal relations, we use the shared backbone encoder fa to extract the representations of time pieces firstly, where Zn, fo(pn.))) and 2n,r = fo(P...). Then, we construct the temporal relation representation as [= =, Next, the intra-temporal relation reasoning head (-)-) takes the relation representation as input to reason the final relation
score h (0,0) = r ro([,,, >). Finally, the intra intra-temporal relation reasoning task is formulated multi-class classification problem and the model is trained with cross-entropy loss Lintra as
4

---

Under review as a conference paper at ICLR 2021
follows:
(a. log exp(hn...) (1.2)))
Cintro
2o exp(In (1(1)))))
By jointly optimizing the inter-sample relation reasoning objective (Eq. [ and intra-temporal relation reasoning objective (Eq. D the final
Algorithm 1: Temporal Relation Sampling.
Require:
training loss is defined as follows:
tn: A T-length time series.
ua pn.vi L-Iength pieces of bn-
L Linter + Lintra
(3)
C: Number of relation classes.
Ensure:
An overview for training SelfTime is given in Algorithm 2 in Appendix A SelfTime
C}: The label of the tem- poral relation between Pr.. and Pn.e-
(u,1)
E
is an efficient algorithm compared with the traditional contrastive learning models such
1: da |u 1/. D [T/C
2: if du, S D then
as SimCLR. The complexity of SimCLR is O(N* k2), while the complexity of SelfTime )(NK3) + O(NK), where O(NK²) is the complexity inter-sample relation reasoning module, and ONN)) the complexity of intra- temporal relation reasoning module. It can be seen that SimCLR scales quadratically in both training size N and augmentation number K. However, in SelfTime, inter-sample relation reasoning module scales quadratically with the number of augmentations K, and linearly with the training size N, and intra-temporal relation reasoning module scales linearly with both augmentations and training size.
(u,r) 0
3: yn
4: else if du S 2 D then
5: Yn (u.v)
6:
7: else if du, S (C- 1) * then
8: yh (u,r) C -
9: else
10: Un
(u,e)
C --1
11: end if
(a,=)
12: return yn
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. To evaluate the effectiveness of the proposed method, in the experiment, we use three categories time series including four public datasets CricketX, UWaveGes- tureLibraryAll (UGLA), DodgerLoopDay (DLD), and Insec:WingbeatSound (IWS)
Category
Dataset
Sample
Length
Class
CricketX
300
Motion
UWaveGesture
Librrars
4478
945
8
Dodgerl Dop
158
288
Sensor
InsettWingbeatSounn
2200
256
Device
MrPT
2574
1024
15
XITU
1920
1024
15
from the UCR Time Series Archive DDau
Table I: Statistics of Datasets.
jet al. 2018). along with two real Seal-world
bearing datasets XJTU and MFP1 (Zhao et al. 2020). All six datasets consist of various num- bers of instances, signal lengths, and number of classes. The statistics of the datasets are shown in Table 0
Time Series Augmentation The data augmentations for time series are generally based on random transformation in two domains (Iwana & Uchida (020): magnitude domain and time domain. In the magnitude domain, transformations are performed on the values of time series where the values at each time step are modified but the time steps are constant. The common magnitude domain based augmentations include jittering, scaling, magnitude warping (Um et al.) 2017), and cutout (DeVries & Taylor 2017). In the time domain, transformations that the elements of the time series are displaced to different time steps than the original sequence. The common time domain based augmentations include time warping (Um al.) (017). window slicing, and window warping (Le Guennec et al., 2016). More visualization details of different
performed along the time axis
augmentations are shown in Figure 3
series data 2018/ https: www. htttss//www.mfpt..or be-arr--yybbbbrrnnn hnna-ddttssss//
sets/
5

---

Under review as a conference paper at ICLR 2021
Original
Jittering
Scaling
Cutout
0.00
-0.25
-0.25
-0.5
-0.50
-0.50
-0.55
0.75
-1.0
-0.75
100
200
300
100
200
300
100
200
300
100
200
300
Magnitude Warping
Time Warping
Window Slicing
Window Warping
100
200
300
100
200
300
100
200
300
100
200
300
Figure 3: Data augmentations examples from CricketX dataset. The blue solid line the original signal and the red dotted lines are the transformations.
Baselines. We compare SelfTime against several state-of-the-art methods of self suff-super repre- sentation learning:
Supervised consists of backbone encoder as the same with SelfTime and a linear classifier, which conducts fully supervised training over the whole networks.
Random Weights the same as Supervised in the architecture, but freezing the backbone's weights during the training and optimizing only the linear classifier.
Triplet Loss (Franceschi et al. 2019) is unsupervised time series representation learning model that uses triplet loss to push a subsequence of time series close to its context and distant from randomly chosen time series.
Deep InfoMax (Hjelm al., 2019) framework of unsupervised representation learning by maximizing mutual information between the input and output of a feature encoder from the local and global perspectives.
Forecast (Jawed et al., 2020) is semi-supervised time series classification model that leverages features learned from the self-supervised forecasting task on unlabeled data. In the experiment, we throw away the supervised classification branch and use only the fore- casting branch to learn the representations of time series.
Transformation (Sarkar & Etemad] 2020) is self-supervised model by designing transfor- mation recognition of different time series transformations pretext task.
SimCLR (Chen et 2020 is a simple but effective framework for self-supervised rep- resentation learning by maximizing agreement between different views of augmentation from the same sample via contrastive loss in the latent space.
Relation (Patacchiola & Storkey 2020 is relational reasoning based self-supervised repre- sentation learning model by reasoning the relations between views of the sample objects as positive, and reasoning the relations between different objects as negative.
Evaluation. As common evaluation protocol, linear evaluation is used in the experiment by train- ing a linear classifier on top of the representations learned from different self-supervised models to evaluate the quality of the learned embeddings. For data splitting, set the crainin/vaaliatioon split as 50%/25%/25% During the pretraining stage, we randomly split the data times with differ- seeds, and train the backbone on them. During the linear evaluation, we train the linear classifier 10 times on each split data, and the best model on the validation dataset was used for testing. Finally, we report the classification accuracy as mean with the standard deviation across all trials.
Implementation. All experiments were performed using PyTorch (v1.4.0) (Paszee et al. 2019 A simple 4-layer 1D convolutional neural network with ReLU activation and batch normalization (Toffe & Szegedy 2015) were used as the backbone encoder fe for SelfTime and all other base- lines, and use two separated 2-layer fully-connected networks with 256 hidden-dimensions as the inter-sample relation reasoning head rn and intra-temporal relation reasoning head T. respectively (see Table 4 in Appendix B for details). Adam optimizer (Kingma & Ba 2015) was used with learning rate of 0.01 for pretraining and 0.5 for linear evaluation. The batch size set as 128 for all models. For fair comparison, we generate K 16 augmentations for each sample although

---

Under review as a conference paper at ICLR 2021
2nd
Pansfommtioo
2nd
Frandoomation
2nd
Iransfomtion
Cet
Siak
VEN
T:W
WA..
NGW
Rvp
35
Cat
Sox
SKW
TUW
V..55
Arg
Sia
MGW
TSW
W1S5
Apg:
24
65.9
46.2
07.3
50.5
61.7
50.5
$4.4
00.1
61.4
54.5
65.0
64.9
04.0
64.6
64.2
62.2
61.7
63.1
81.8
66.0
64.7
025
62.9
68.0
57.9
36.1
32.9
64.9
64.4
61
9
63.9
66.5
65.6
64.7
64.6
65.8
66
65.1
62.6
62.0
68.0
68.8
68.1
68.1
67
63.5
54
R0.0
33.5
93.7
10 Temndor
65.8
13k
570
82.5
62
63
18
65
84
45
05
155
61.9
58.1
68.6
53.2
63.8
00.4
61
62.0
61.8
64.8
64.8
06.2
64.9
65.0
64
64.7
66.5
63
68.0
68
06.2
64
65.0
05.0
1838
58.0
61.5
86.0
65.2
Inter-sample Relation
Intra-temporal Relation
SelfTime
Figure 5: Linear evaluation CricketX under individual or composition of data augmentations. For all columns but the last, diagonal entries correspond to single transformation, and off-diagonals correspond to composition of two transformations (applied sequentially). The last column reflects the average over the row.
more augmentation results in better performance (Chen et al. 2020 Patacchiola & Storkey 2020).
More implement details of baselines are shown in Appendix D More experimental results about the
impact of augmentation number K are shown in Appendix
4.2 ABLATION STUDIES
In this section, we firstly investigate the impact of different temporal relation sampling settings on cetra-temporl relation reasoning. Then, we explore the effectiveness of iater-sample relation reasoning. intra-temporal relation reasoning, and their combination (SelfTime), under different time series augmen- tation strategies. Experimental results show that both inter- sample relation reasoning and imtra-lemporal relation reason- ing achieve remarkable performance, which helps the network to learn more discriminating features of time series.
Aless
65
Linear
63
61
2
3
4
5
6
Class Number (C)
67
Alsss
65
63
0.1 0.15 0.2 0.25 0.3 0.35
Piece Size ( L/T)
Temporal Relation Sampling. To investigate the different settings of temporal relation sampling strategy on the impact of linear evaluation performance, in the experiment, we set different numbers of temporal relation class C and time piece length L. Specifically, to investigate the impact of class number, we firstly set the piece length 0.2 * T as 20% of the original time series length, then, we vary C' from to 8 during the temporal relation sampling. As shown in Figure 4 we show the results of parameter sensitivity experiments on CricketX, where blue bar indicates class reasoning accuracy on training data (Class ACC) and brown line indicates the linear evaluation accuracy on test data (Linear ACC). With the increase of class number, the Linear ACC keeps increasing until C' = 5, and we find that a small value C = 2 and big value C' 8 result in worse performance. One possible reason behind this is that the increase of class number drops the Class ACC and makes the relation reasoning task too difficult for the network learn useful representation. Similarly, when set the class number C and vary the piece length L from 0.1 * T to 0.4 * T, we find that the Linear ACC grows up with the increase of piece size until 0.3 T, and also, either small value or big value of L will drop the evaluation performance, which makes the relation reasoning task too simple (with high Class ACC) or too difficult (with low Class ACC) and prevents the network from learning useful semantic representation. Therefore, as consistent with the observations of self-supervised studies in other domains (Pascual et al. 2019b] Wang et al. 2020), an appropriate pretext task designing is crucial for the self-supervised time series representation learning. In the experiment, to select moderately difficult pretext task for different datasets, we set (class number (C), piece size (L/T)) as {3, 0.2) for CricketX, {4. 0.2} } for UWaveGestureLLibarrAll. {5, 0.35) for DodgerLoopDay, {6. 0.4) for Insect WingbeatSondd {4. 0.2) for MFPT, and {4, 0.2) for XJTU. More experimental results on other five datasets for parameter sensitivity analysis are shown in Appendix
Figure 4: Impact of different tem- poral relation class numbers and piece sizes on CricketX dataset.
7

---

Under review as a conference paper at ICLR 2021
Method
Dataset
ChicetX
UGLA
DLD
TWS
MFPT
XTTU
Supervised
62.44=1.53
87.83=0.32
37.05=1.61
66,23-0.45
80.29:10.8
95,940042
Random Weights
36.910092
70.01
32.95-2.57
12.85=1.36
46.68=22.35
42.5814.67
Triplet Loss (Fanceschhi et 2010
40.01=2.044
7141=111
41.372247
53.61
47.87=2.97
53.31=3343
Deep InfoMax [Fjelm al 2019
49.16-+.00
2.37
38.95++.447
55,99-1.31
58.9942.72
1.827+1.83
Forecast (Jawed ét al 2020
44.59=1.09
75.
38,74=3.05
54,89=1.99
12.6++1.65
62.28+2.55
Transfommatio Sarkar & Exenaad
2020
52.12=2.02
75.41=0.27
1.56
1..68
60.33=3.29
85.08:2.01
SimCLR Chen
2020
59.0+#3.19
74.9-0922
37. 3..78433
19.99
11.21
88.84=0663
Relation Patacciuooa & Strkey
2020
65.3:10.43
0.787
42.84
62.0:11449
73.53=0065
05.1222
SelfTTme (ours)
68.6++0.66
84.97-0.83
49.14-2.33
66.87--0.71
78.48=00.4
96.73-00.6
Table 2: Linear evaluation of representations learned by different models on different datasets.
Impact of Different Relation Modules and Data Augmentations. To explore the effectiveness of different relation reasoning modules including inter-sample relation reasoning, intra-temporal rela- tion reasoning, and their combination (SelfTime), in the experiment, we systematically investigate the different data augmentations on the impact of linear evaluation for different modules. Here, we consider several common augmentations including magnitude domain based transformations such as jittering (Jit.), cutout (Cut.), scaling (Sca.), magnitude warping (M.,.,, and time domain based transformations such time warping (T.W.), window slicing (W.S.). window warping (W.W.). Fig- ure shows linear evaluation results on CricketX dataset under individual and composition of trans- formations for inter-sample relation reasoning, intra-temporal relation reasoning, and their combina- tion (SelfTime). Firstly, we observe that the composition of different data augmentations is crucial for learning useful representations. For example, inter-sample relation reasoning is more sensitive to the augmentations, and performs worse under Cut., Sca., and M. W. augmentations, while intra- temporal relation reasoning is less sensitive the manner of augmentations, although it performs better under the time domain based transformation. Secondly, by combining both the inter-sample and intra-temporal relation reasoning, the proposed SeifTime achieves better performance, which demonstrates the effectiveness of considering different levels of relation for time series representa- tion learning. Thirdly, we find that the composition from a magnitude-based transformation (c.g. scaling, magnitude warping) and time-based transformation (e.g. time warping, window slicing) facilitates the model to learn more useful representations. Therefore, in this paper, we select the composition of magnitude warping and time warping augmentations for all experiments. Similar experimental conclusions also hold on for other datasets. More experimental results on the other five datasets for evaluation of the impact of different relation modules and data augmentations are shown in Appendix
4.3 TIME SERIES CLASSIFICATION
In this section, we evaluate the proposed method by comparing with other state-of-the-arts on time series classification task. Firstly, we conduct linear evaluation to assess the quality of the learned representations. Then, evaluate the performance of all methods in transfer learning by training on the unlabeled source dataset and conduct linear evaluation on the labeled target dataset. Finally, qualitatively evaluate and verify the semantic consistency of the learned representations.
Linear Evaluation. Following the previous studies (Chen et al., 2020 Patacchiola & Storkey, 2020). we train the backbone encoder for 400 epochs on the unlabeled training set, and then train a linear classifier for 400 epochs on top of the backbone features (the backbone weights are frozen without back-propagation). As shown in Table 2, our proposed SelfTime consistently outperforms all baselines across all datasets. SelfTime improves the accuracy over the best baseline (Relation) by 5.05% (CricketX), 5.06% (UGLA), 14.61% (DLD), 7.85% (IWS), 6.73% (MFPT), and 1.67% (XITU) respectively. Among those basclines, either global features (Deep InfoMax. Transforma- tion, SimCLR, Relation) or local features (Triplet Loss, Deep InfoMax, Forecast) are considered during representation learning, they neglect the essential temporal information of time series ex- cept Triplet Loss and Forecast. However, by simply forecasting future time pieces, Forecast cannot capture useful temporal structure effectively, which results in low-quality representations. Also, in Triplet Loss, a time-based negative sampling is used to capture the inter-sample temporal relation among time pieces sampled from the different time series, which is cannot directly and efficiently capture the intra-sample temporal pattern of time series. Different from all those baselines, Self- Time not only extracts global and local features by taking the whole time series and its time pieces
8

---

Under review as a conference paper at ICLR 2021
Method
UGLA-
Chdaa--ULLA
IWS-DLD
Source
Target
DLD-IWS
XITC-METT
MFPT-XXTU
Supervisel
31.31:2776
7135212
22.9-255
44.31:3:25
63.15:2208
82,38
Random Weights
36.910092
70.01=1668
32.95-2..57
12.85
46.68.22.35
52.58.44667
Triplet Loss
Pranceschi
2019
4..08:4.666
55,32=251
34.67 4.17
45 224
53.75-2.966
59.24
Deep InfoMax 775glm
4..75
Forecast Jadraa of
32.67+0866
1.172-1.17
25.47+2.93
55 19+1
53.08-2.75
3..74-3.92
Trunsformation barear
2120
19.244+2.25
70.4= 198
1.06
57.71+0.83
54.71=1.66 1.68
69.81=8.03
Simd'l
36.2444003
Patnck/iiols Stirkey
52.55+2677
75.67-0.54
56.29+11.82
70.27-1.14
92.27-1.115
Seli lime Louis)
55.04:2.558
77.77-0.35
45.0 1.48
57.8:1335
75.06:11.84
93.79:22466
Domain transfer evaluation by training with self-supervision on the unlabeled source data and linear evaluation the labeled target data (e.g. source-ttareet: UGLA->Crikett,
Table 3:
Triplet Loss
Deep Infomax
Forecast
Transformation
SimCLR
Relation
SelfTime
Figure 6: t-SNE visualization of the learned feature on UGLA dataset. Different colors indicate different labels.
as inputs during feature extraction, but also captures the implicit temporal structure by reasoning intra-temporal relation among time pieces.
Domain Transfer. To evaluate the transferability of the learned representations, we conduct exper- iments in transfer learning by training on the unlabeled source dataset and conduct linear evaluation on the labeled target dataset. In the experiment, we select two datasets from the same category as the source and target respectively. outperforms all the other baselines under different conditions. For example, SelfTime achieves an transfer, and over Deep InfoMax
shown in Table experimental results show that our SelfTime
improvement over the Relation by 4.73% on 20.2% IWS-DLD transfer, and over Relation 6.81% XJTU- transfer, respectively, which demonstrates the good transferability of the proposed method.
Visualization. To qualitatively evaluate the learned representations, extract the features and visualize them in 2D space using t-SNE (Maaten & Hinton (008) to verify the semantic consistency of the learned representations. Figure 6 shows the visualization results of features from the baselines and the proposed SelfTime on UGLA dataset. is obvious that by capturing global sample structure and local temporal structure, SeliTime learns more semantic representations and results in better clustering ability for time series data, where more semantic consistency is preserved in the learned representations by our proposed method.
the trained backbone to
5 CONCLUSION
We presented self-supervised approach for time series representation learning, which aims ex- tract useful feature from the unlabeled time series. By exploring the inter-sample relation and intra- temporal relation, SelfTime is able capture the underlying useful structure of time series. Our main finding is that designing appropriate pretext tasks from both the global-sample structure and local-temporal structure perspectives crucial for time series representation learning, and this find- ing motivates further thinking of how to better leverage the underlying structure of time series. Our experiments on multiple real-world datasets show that our proposed method consistently out- performs the state-of-the-art self-supervised representation learning models, and establishes new state-of-the-aa in self superviee time series classification. Future directions of research include ex- ploring more effective intra-temporal structure (i.e. reasoning temporal relation under the time point level), and extending the SelfTime to multivariate time series by considering the causal relationship among variables.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anthony Bagnall, Jason Lines, Jon Hills, and Aaron Bostrom. Time-series classification with cote: the collective of transformation-aee based ensembles. IEEE Transactions on Knowledge and Data Engineering, 27(9):2522-2335, 2015.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1r0.v::806.0/26. 2018.
Mustafa Gokce Baydogan, George Runger, and Eugene Tuv. bag-of-feature framework to clas- sify time series. IEEE transactions on pattern analysis and machine intelligence, 35(11):2796- 2802, 2013.
Donald J Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In KDD workshop, volume 10, pp. 359-370. Seattle, WA, USA: 1994.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th international conference on machine learning (ICML), 2020.
Ziqiang Cheng, Yang Yang, Wei Wang, Wenjie Hu, Yueting Zhuang, and Guojie Song. Time2graph: Revisiting time series modeling with dynamic shapelets. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, pp. 3617-3624, 2020.
Samarjit Das. Time series analysis, volume 10. Princeton university press, Princeton, NJ, 1994.
Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bag- nall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML. The ucr time series classification archive, October 2018. https::/ http://wwwwcom cs. ucr edu/ eamonn/tiiee series_data 2018/
Terrance DeVrics and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 2017.
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som- vae: Interpretable discrete representation learning time series. In 7th International Conference on Learning Representations, ICLR, 2019.
Jean-Y Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series, In Advances in Neural Information Processing Systems, pp. 4650 4661, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR, 2018.
Micah Goldwater, Hilary Don, Moritz F Krusche, and Evan J Livesey. Relational discovery in category learning- Journal of Experimental Psychology: General, 147(1):1-35, 2018.
Tomasz Górecki and Maciej Euezak. Non-isometric transforms in time series classification using dtw. Knowledge-Basse Based Systems, 61:98-108, 2014.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference acoustics, speech and signal processing. ICASSP, pp. 6645-6649, 2013.
Jon Hills, Jason Lines, Edgaras Baranauskas, James Mapp. and Anthony Bagnall. Classification of time series by shapelet transformation. Data Mining and Knowledge Discovery, 28(4):851-881, 2014.
10

---

Under review as a conference paper at ICLR 2021
R Devon Hjelm, Alex Fedorov, Samuel Prroie-Marchhldon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations. ICLR, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03160 2015.
Brian Kenji Iwana and Selichi Uchida. Time series data augmentation for neural networks by time warping with a discriminative teacher. arXiv preprint arXiv:2004.08780, 2020.
Shayan Jawed, Josif Grabocka, and Lars Schmidt-Thiee supervised time series classification. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD, pp. 499-511. Springer, 2020.
Self-supervised learning for semi-
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2901-2910, 2017.
Myeongsu Kang, Jaeyoung Kim, Linda M Wills, and Jong-Myon Kim. Time-varying and multires- olution envelope analysis and discriminative feature analysis for bearing fault diagnosis. IEEE Transactions on Industrial Electronics, 62(122:7749-776) 2015.
Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Shun Chen. Lstm fully convolutional networks for time series classification. IEEÉ access, 6:1662-1665 1669, 2017.
Charles Kemp and Joshua B Tenenbaum. The discovery of structural form. Proceedings of the National Academy of Sciences, 105(11::1088-10692, 2008.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 3th Interna- tional Conference on Learning Representations, ICLR, 2015.
Arthur Le Guennec, Simon Malinowski, and Romain Tavenard. Data Augmentation for Time Series Classification using Convolutional Neural Networks. In ECML/PKDD Workshop on Advanced Analytics and Learning Temporal Data, 2016.
Jason Lines and Anthony Bagnall. Time series classification with ensembles of elastic distance measures. Data Mining and Knowledge Discovery, 2963)556--592, 2015.
Qianli Ma, Wanqing Zhuang, and Garrison Cottrell. Triple-shapelet networks for time series classi- fication. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 1246-1251. IEEE, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2579 2008
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference Computer Vision, ECCV, pp. 527- 544. Springer, 2016.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European Conference on Computer Vision, ECCV, pp. 69-84. Springer, 2016.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In 8th International Conference on Learning Representations, ICLR. 2020.
Santiago Pascual. Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio. Learning -annssi-gnnostic speech representations from multiple sulf-supe tasks. In Proc. the Conf. of the Int. Speech Communication Association (INTERSPEEEC)) pp. 161-165, 2019a.

---

Under review as a conference paper at ICLR 2021
Santiago Pascual, Mirco Ravanelli, Joan Serrä, Antonio Bonafonte, and Yoshua Bengio. Learning Problem-Agnosee Speech Representations from Multiple Self- Supervisd Tasks. In Proc. of the Conf. of the Int. Speech Communication Association (INTERSPEECH), pp. 161-165, 2019b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, NeurlPS, pp. 8026-8037, 2019.
Massimiliano Patacchiola and Amos Storkey. Self-supervised relational reasoning for representation learning. arXiv preprini arXiv:2006.0588 058000 2020.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, CVPR, pp. 2536-2544, 2016.
Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In /EEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6989-6993, 2020.
Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien. Multi-task self-supervised learning for human activity detection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(2):1-30, 2019.
Aaqib Saeed, Flora D Salim, Tanir Ozcelebi, and Johan Lukkien. Federated self-supervised learning of multi-sensor representations for embedded intelligence. IEEE Internet Things Journal, 2020.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, NIPS, pp. 4967-4976, 2017.
Pritam Sarkar and Ali Etemad. Self-supervised learning for ecg-based emotion recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3217-3221. IEEE, 2020.
Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 2906):105--1500 2015.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. In Proc. of the Conf. of the Int. Speech Communication Association (INTERSPEECH)) pp. 3465-3469, 2019.
Rajat Sen, Hsiang-Fu Yu, and Inderjit Dhillon. Think globally, act locally: A deep neural net- work approach to high-dimensional time series forecasting. In Advances in Neural Information Processing Systems, NeurIPS, pp. 4837-4846, 2019.
Satya Narayan Shukla and Benjamin M Marlin. Interpolation networks for irregularly sampled time series. In 7th International Conference on Learning Representations, ICLR, 2019.
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems, NIPS, pp. 026-934, 2013.
ABA Stevner, Diego Vidaurre. Joana Cabral, K Rapuano, Søren Føns Vind Nielsen, Enzo Tagli- azucchi, Helmut Laufs, Peter Vuust, Gustavo Deco, Mark W Woolrich, et al. Discovery of key whole-brain transitions and dynamics during human wakefulness and non-rem sleep. Nature communications, 10(1):1-14, 2019.
Terry Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulié. Data augmentation of wearable sensor data for parkinson's 's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 216-220, 2017.
12

---

Under review as a conference paper at ICLR 2021
Layer Deseription
Output Tensor Dim. 7 (or x I.)
#0
Input time series (or time piece)
Backhone Encoder
#1
Conv ID 8, 4. D+-BatchNorm+Rele
8 T/2 (or 8 x L/2)
#2
Conv ID(8, 16, 4, 1)+ D>-BatchNorm-Rell.
16x 7/4 (or 16 L,4)
#3
ConvID(16, 32, 4, Dy4BatchNorm-ReeU Co3y 64, 4, I/+BatchNorm-ReLU
32
32 L/8)
#4
64
+AvgPoolD--Platee--Nomaaa
Inter-Sample Relation Reasoning Head
#1
Sanncar-BatchNNom-...aaa ccakyRel.U
256
#2
Ligear+
Intra- Temporal Relation Reasoning Head
Linear--aathNorrm- Linear-BatthhWrmm-Lcayyyllll.. eakyRellUU Lokkellll
256
Linear+Solimax
Table 4: Implementation detail of SelfTime. Here, we denote ID convolutional layer as Conv chhincchanelss out.channels, kernel_size, stride, padding).
Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self-supervised video representation learning by pace prediction. In European conference on computer vision, ECCV, 2020.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, ICCV, pp. 2794-2802, 2015.
Donglai Wei, Joseph Lim, Andrew Zisserman, and William T Freeman. Learning and using the ar- row of time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 8052-8060, 2018.
Lexiang Ye and Eamonn Keogh. Time series shapelets: new primitive for data mining. In Pro- ceedings of the 15th ACM SIGKDD international conference Knowledge discovery and data 2009.
mining, KDD. pp.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap. Edward Lockhart, al. Deep reinforcement learning with relational inductive biases. In 7th International Conference on Learning Representations, ICLR, 2019.
Zhibin Zhao, Tianfu Li, Jingyao Wu, Chuang Sun, Shibin Wang, Ruqiang Yan, and Xuefeng Chen. Deep learning algorithms for rotating machinery intelligent diagnosis: An open source benchmark study, arXiv preprint arXiv:2003.03315 2020.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 803-818, 2018.
A PSEUDO-CODE OF SELFTIME
The overview of training process for SelfTime is summarized Algorithm
B ARCHITECTURE DIAGRAM
SelfTime consists of backbone encoder, inter-sample relation reasoning head, and intra- temporal relation reasoning head. The detail architectural diagrams of SelfTime are shown in Table
C DATA AUGMENTATION
In this section, we list the configuration details of augmentation used in the experiment:
13

---

Under review as a conference paper at ICLR 2021
Algorithm 2: SelfTime
Require:
Time series set T = {tn } I==
fo: Encoder backbone.
Ty: Inter-sample relation reasoning head.
rg: Intra-temporal relation reasoning head.
Ensure:
fo: An updated encoder backbone.
1: for tmrtm; tn E T do
2: Generate two augmentation sets A(tm)) and A(tn)
3: Sample positive relation pair (tm = and negative
relation pair (tm), tl?)) from A(tm) and A(t,)
4:
2m (tm)
(1)
>> Sample representation
5:
2m so (t()
P Sample representation
6:
( fo (th)
>> Sample representation
hmm - = rpp(Em 20
(b.5
7:
Reasoning score positive relation
12m (1d) (e((z 2
8:
Reasoning score of negative relation
9:
Sample time piece relation pair (Pn,es Pn.v) by Algorithm I
Zn,u = fo (P(,,
Time piece representation
11:
Zn.r = (P
Time piece representation
12:
(u,v) r ([z,,, zn.ul)
> Reasoning score of intra-temporal relation
13:
end for
14: Linter
2N ΣK EØ= (1.))
(1.))) log(h (1)))
+(1 - yn (1.5)) log(1 1 (((25)))
D Inter-sample relation reasoning loss Intra-temporal relation reasoning loss
15: Lintra
= ΣN Un (u,v) log DE oxp((hln.)))
exp(A(uie))
16: Update Se. Tu. and rip by minimizing
C Linter Lintra
17: return encoder backbone fe. throw away Tu. and Tip
14

---

Under review as a conference paper at ICLR 2021
Jittering: We add the gaussian noise the original time series, where noise is sampled from Gaussian distribution N 0.2).
Scaling:
We multiply the original time series with random scalar sampled from Gaussian distri- bution (O, 0.4).
Cutout: We replace random 10% part of the original time series with zeros and remain the other parts unchanged.
Magnitude Warping: We multiply warping amount determined by cubic spline line with 4 knots on the original time series at random locations and magnitudes. The peaks or valleys of the knots set as μ and 0.3 (Um et al. 2017).
Time Warping: We set the warping path according to a smooth cubic spline-basse curve with knots, where the random magnitudes is μ 1 and 0.2 for each knot (Um et al. (017).
Window Slicing: We randomly crop 80% of the original time series and interpolate the cropped time series back to the original length (Le Guennec al. 2016).
Window Warping: We randomly select a time window that is 30% of the original time series length, and then warp the time dimension by 0.5 times or 2 times (Le Guennec et al. 2016).
D BASELINES
Triplet Loss Franceschi et al. 2019) We download the authors' official source code and the same backbone as SelfTime, and set the number of negative samples as 10. We use Adam optimizer with learning rate 0.001 according grid search and batch size 128 as same with SelfTime.
Deep Info To Hjjelm et al. 2019) We download the authors' official source code and use the same backbone as Seme and the parameter a 0.5, 1.0, 0.1 through grid search. We use Adam optimizer with learning rate 0.0001 according grid search and batch size 128 as same with
SelfTime.
Forecas (Jawed et al. 2020) Different from the original multi-task model proposed by authors, we throw away the supervised classification branch and use only the proposed forecasting branch to learn the representation in a fully sulf-supe manner. We use Adam optimizer with learning rate 0.01 according to grid search and batch size 128 as same as SelfTime.
Transformation [ (Sarkrr & Etemad, 2020) We refer to the authors' official source code and reim- plement it in Py Torch by using the same backbone and two-layer projection head as same with SelfTime. We use Adam optimizer with learning rate 0.001 according to grid search and batch size 128 as same with SelfTime.
P CCen et al., (020) We download the authors' official source code by using the same backbone and two-layer projection head as same with SelfTime. We use Adam optimizer with learning rate 0.5 according grid search and batch size 128 same SelfTime.
Relation 10 Patacchiola & Storkey, 2020) We download the authors' official source code by using the same backbone and relation module as same with SelfTime. For augmentation, we set K 16, and use Adam optimizer with learning rate 0.5 according to grid search and batch size 128 as same with SelfTime.
E PARAMETER SENSITIVITY
Figure shows ssoww the impact of different augmentation number K on all datasets. It's obvious that more augmentations result in better performance, which demonstrates that introducing more reference
15

---

Under review as a conference paper at ICLR 2021
CricketX
UWaveGestureeliraayyAll
73
87
70
85
67
64
83
61
81
4
16
32
64
16
32
64
Day
InsectWingbeatSound
52
69
ACC
50
68
67
48
66
46
65
16
64
32
64
MFPT
XJTU
81
99
79
97
77
95
75
93
73
91
16
32
64
4
16
32
48
64
Number of augmentations (K)
Number of augmentations (K)
Figure 7: Impact of augmentation number K.
samples (including positive samples and negative samples) for the anchor sample raises the power of relational reasoning.
Figure shows the impact different temporal relation class numbers and piece sizes other five datasets: UUWaveGestueLiibaayy.t where the blue bar indicates class reasoning accuracy on training data (Class ACC) and the brown line indicates the linear evaluation accuracy test data (Linear ACC). We find an interesting phe- nomenon that both small values of class number C' piece size L/T, and big values C or L,T,, result in worse performance. One possible reason behind this is that the increase of class number drops the Class ACC and makes the relation reasoning task too simple (with high Class ACC) or too difficult (with low Class ACC) and prevents the network from learning useful semantic representa- tion. Therefore, an appropriate pretext task designing is crucial for the self-supervised time series representation learning. In the experiment, to select moderately difficult pretext task for different datasets, we set {class number (C), piece size (L/T)) as (4, 0.2) for SaaaveGettreeLbbrary.. 0.35) for Dodgerl Loodery {6, 0.4) ) for InsectWingeeaSSonn, (4, 0.2) for MFPT, and [4, 0.2) for XITU.
DodgerLooppay., InseetWingbettSondd, MFPT, and XJTU,
(5,
F ABLATION STUTY
In this section, we additionally explore the effectiveness of different relation reasoning modules including inter-sample relation reasoning, intra-temporal relation reasoning, and their combina- tion (SeliTime) other five datasets including UWaveGestuueeLibaary- sectWingbeatSound, MFPT, and XJTU. Specifically, in the experiment, we systematically inves- tigate the different data augmentations Here, we consider several common augmentations including magnitude domain based transforma- tions such jittering (Jit.), cutout (Cut.), scaling (Sca.), magnitude warping (M.W.), and time do- main based transformations such as time warping (T.W.), window slicing (W.S.), window warping (W.W.). Figure and Figure I0 show linear evaluation results on five datasets under individual and composition of transformations for inter-sample relation reasoning, intra-tem sera-temporal relation reason- ing, and their combination (SelfTime). As similar the observations from CricketX, firstly, we observe that the composition of different data augmentations is crucial for learning useful represen- tations. For example, inter-sample relation reasoning more sensitive to the augmentations, and performs worse under Cut., Sca., and M.W. augmentations, while intra-temporal relation reason- ing less sensitive to the manner of augmentations on all datasets. Secondly, by combining both the satele and intra-tempont relation reasoning, the proposed SelfTime achieves better per- formance, which demonstrates the effectiveness of considering different levels of relation for time series representation learning. Thirdly, overall, find that the composition from magnitude-
DodgerLoopDay, In-
the impact of linear evaluation for different modules.
16

---

Under review as a conference paper at ICLR 2021
Class ACC (Train)
91
84
Linear ACC
Class ACC
82
85
Linear ACC
79
(Test)
(Train)
72
83
(Test)
82
67
62
81
55
80
52
79
2
3
8
15
2
.5
.35
.4
Class Number C)
Piece Size L/T)
(a) UWaveGestureLlbbaryyAll
Class ACC
98
Class ACC
91
55
(Train)
91
53
(Test)
(Train
81
53
(Test)
52
Lineat
71
51
Linear
61
49
70
50
51
47
15
.35
.4
Class Number ( ()
Piece Size L/T)
(b) Dodger
69
Linear ACC
Class ACC
94
70
Linear ACC
(Train) rain)
78
67
(Test)
(Train)
66
(Test)
Class
70
65
72
62
62
61
58
54
61
50
54
2
3
8
.1
.15
2
.25
3
.35
.4
Class Number ( ())
Piece
Size
L/T)
(c) InsectWingbeatSound
Class ACC
76
Linear 0
Class ACC
73
78
Linear ACC
(Train)
70
74
(Test)
(Train)
62
74
(Test)
60
70
50
70
51
66
40
68
40
62
2
3
6
7
8
1
15
2
25
3
.35
.4
Class Number ()
Piece Size
L/T)
(d) MFPT
Class ACC
74
92
Class ACC
71
90
Linear ACC
(raii)
63
87
(Test)
(Train)
63
88
(Test)
52
86
41
77
55
84
30
72
47
82
2
3
6
7
8
1
15
2
.25
.3
.35
.4
Class Number C))
Piece Size
L/T)
(e) XJTU
Figure 8: Impact different temporal relation class numbers piece sizes on other five datasets.
17

---

Under review as a conference paper at ICLR 2021
2nd
ransfommtion
2nd Franfoomttion
Iransfmmtionn
35
Cdt
Scs
VGN
TJW
W.S.
NGW
Avg.
35
Cat
Sox
SUW
TOW
V.S5
WINN.
Arg
Sca
MGW
TIM
W.S.
Axg.
74.5
05.0
70.7
99.4
73.0
12.2
73.1
11.1
78.
12.5
03.0
03.5
022
02.4
02.2
+8.0
60.9
05.0
53.1
01.3
00.7
01
01.2
1st Urendfr taaon
74,2
69.2
71.3
95.5
80.0
79.5
76.2
72.6
75.4
53.4
83.2
63.8
88
63.2
68
82.0
78.6
39.6
03.4
68.3
82.9
68.6
83
02.5
58.2
65.3
62.7
85.0
136-
80.8
78
74.0
13.8
836
13
82
82
63.3
T
84
73
82
85
75.7
73.2
793
79.0
73.6
72.3
71.2
74.9
78.6
52.8
83.2
63.8
82.0
63.0
82
02.4
77.5
78.6
82.
84.0
78.7
60.9
20.6
00.7
18.0
76.0
75.9
52.5
80.9
50.6
20
Iater- Relation
Intra-temporal Relation
SelfTime
(a) UWaveGestureLLibaayyAll
2nd transformation
2nd traneforatioon
Iranstoration
52.5
92
45.5
41.4
52.7
30%
52.7
49.5
51.9
54.3
52
51.5
50.4
51.00
50.0
$1.6
51.8
$5.5
40.6
52.4
52.77
52.0
62.3
282
27.1
53.1
471
52.3
41.2
C
93.9
51.8
53
82.1
52.7
51.3
52.3
92.5
52.0
$1.0
6
48.0
65.5
56.0
51.5
Tel transfmmman
28.
27
42.7
96.8
50.7
$4.0
51.0
40.7
43.6
40.6
40
56.7
48.7
Fw
54.9
51.8
52
51.6
55.9
53
52.4
92.6
Pak
52.9
$5.1
48
50
56.4
52
54.5
52.6
61.8
47.5
428
41.8
49.5
47
440
66.5
61.6
58
52
48.9
50.0
527
51.3
61.5
51.9
52.0
47
47.
62.1
40
50.6
52.5
30
Inter-sample Relation
intra-temporal Relation (b) Dodgery
SelfTime
2nd transfomation
2nd transfomatioon
2nd transformation
58.6
58.9
58.0
57.6
03.1
53..5
63.33
00.00
07.0
67
6B44
06.7
67.9
17.65
6T.6
67.2
00:2
C5
68:2
G6.00
00
05
58.7
68.0
07.2
06.5
06.9
66.6
60.0
06
banaffarmlionn Max.
57.9
60.8
Tsl Ialrraformetoo
66.9
66.8
G7.0
Tal wanaaaarmmen molion
67.4
56.7
55.1
52.2
51.5
63.2
60.5
63.0
57.7
66.0
67.0
67.4
66.8
66.9
67.6
96
67.1
65,1
66
65.2
66.0
96
67.4
05.8
Edtt
83.4
50.8
85.1
52.9
81.3
90.9
61.7
62.3
87.9
68.0
67.4
67.7
87
67.1
66
07.5
06.0
67.3
67.3
67.3
58.7
86.7
08.5
WBB
66
60.5
59
592
50
60.8
81
8733
88:3
67
R6
67.
Inter-sample Relation
Intra-tempont Relation (c) InsectWingbeatSound
SelfTime
Figure 9: Linear evaluation on UWaavGesstureLibraryAll DodgerLoopDay, and InsectWingbeat Sound datasets under individual or composition of data augmentations. For all columns but the last, diagonal entries correspond to single transformation, and off-diagonals correspond to composition of two transformations (applied sequentially). The last column reflects the average over the row.
based transformation (e.g. scaling, magnitude warping) and a time-based transformation (e.g. time warping, window slicing) facilitates the model to learn more useful representations. Therefore, in this paper, we select the composition of magnitude warping and time warping augmentations for all datasets, although other compositions might result in better performance.
18

---

Under review as a conference paper at ICLR 2021
2nd ransfommtion
2nd Iransformaioon
2nd transforatiion
Stk.
VUN
Avp
Arg:
TiM
W1S.
77.0
68.6
65.1
67.1
67.8
65.6
57.7
55.5
54.4
74.8
72.8
71.8
64.6
75.0
78.5
78.3
77.6
76.8
76.7
76.5
78.4
78.5
70.0
78.1
78.9
76.1
78.2
78
71.6
84.1
60.8
56.5
50.8
72-0
72.7
14.4
87.2
712.5
78.6
22.7
11.2
P0.2
77.8
76.7
77.8
77-5
78.5
70.7
77.
77.5
136
02.6
63.9
72.9
73.3
68.2
56.1
59.8
64.8
8
56.5
78.5
07.8
66.6
64.5
86
67.4
64.5
66.2
78.2
77.8
64.6
62.7
63.
68.2
96.4
61.4
72:7
72.8
85.0
58.4
52.9
62.6
64.8
77.4
09.5
64.8
61.9
65.9
67.2
81.8
65
77.5
78.2
85.0
69.0
81.
67.6
Intere Relation
Intra- terrprll Relation
SelfTime
(a) MFPT
2nd transfomation
transfomationn
Sick
MIN
T/W
W.S5
M/W
AvD
31
Gat
Sia
M.W
TWW
W.S.
W/W
AvD
Gut
Rick
M/W
T/W
W1S
M//M
AvD
03.2
05
81.3
$7.0
79.9
$9.5
91.0
90.9
85.8
82.9
92.8
87.8
84.2
88.9
87
96.9
88
85.0
CP
90.0
88.9
91.5
90.6
93.6
86.8
01
90.8
23
86.1
$1.3
76.8
72-4
94.3
94.0
94-8
85.
77.0
58.7
85.5
88
85.7
83.4
84.2
108 Tiansformaion
01.7
98
15.50
06
MdK-
85
91.2
91.1
94.6
54.7
51.0
21.2
81.3
82
78.3
58.5
84.7
88.0
86.
85
86
85.0
00.2
52
85.6
96
92.4
92.4
93
93.3
85.2
31.3
93.4
38.2
91.8
50.5
88.2
90.7
78.7
87.8
81.7
88.0
87.9
84.9
862
85.0
87.2
89
96.2
26.0
93.2
52.3
21
92.5
W0R
Inter-sample Relation
intra-temporal Relation (b) XJTU
SelfTime
Figure 10: Linear evaluation on MFPT and XJTU datasets under individual composition of data augmentations. For all columns but the last, diagonal entries correspond to single transformation, and off-diagonals correspond to composition of two transformations (applied sequentially). The last column reflects the average over the row.
19


Reviews:
[{"review_id": "fyqTqWYZan7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach for self-supervised time series representation learning by using inter-sample and intra-temporal relational reasoning. The paper builds upon the existing ideas on relational reasoning [3] and self-supervised learning to train models from unlabeled data.\nSelf-supervised learning for time series is still under-explored and this paper attempts to bridge this gap in time series literature.\n\nThe key novelty of the paper as claimed by the authors is in the design of inter-sample and intra-temporal tasks and loss functions to learn a feature extractor from unlabeled data. \nThis idea of designing pretext tasks using global-sample structure and local-temporal structure and adapting it for time series is interesting. However, the inter-sample loss function seems to be a straightforward application of ideas in self-supervised learning literature that rely on various augmentations to create positive and negative pairs of samples. The intra-temporal task is defined where the distance between subsequences (referred to as time pieces in the paper) of a time series is used to create a classification task. This is a potentially novel (albeit incremental) aspect of the approach but the empirical evaluations (as detailed below) fail to highlight the impact of the same on performance clearly.\nFurthermore, the idea of using subsequences to define self-supervised (intra-temporal) tasks for time series has been explored earlier, e.g. in [1,2]. The authors seem to be unaware of papers like [1,2], and Introduction and Related Work sections suggest that self-supervised learning for time series has not been attempted earlier.\n\nApart from lack of clarity on novelty and contribution of the work, I have following concerns regarding empirical evaluation: \n1. The ablation studies show the effect of number of classes and the length of subsequences (pieces) on linear evaluation accuracy (Linear ACC) for the downstream classification tasks. However, the sensitivity of results to the hyperparameters of intra-temporal task is reported only on CricketX. Do the same results hold on other datasets? Also, the results on downstream task seem to be very sensitive to the choice of parameters C and L/T. The authors state that \"In the experiment, to select a moderately difficult pretext task for different datasets, we set {class number (C), piece size (L/T )} as {3, 0.2} for CricketX, {4, 0.2} for UWaveGestureLibraryAll, {5, 0.35} for DodgerLoopDay, {6, 0.4} for InsectWingbeatSound, {4, 0.2} for MFPT, and {4, 0.2} for XJTU\" - It is not clear how these choices for hyperparameters are arrived at, or what \"we set\"  and \"moderately difficult pretext task\" mean. Also, the ablation study is more of a sensitivity analysis of the choice of values for C and L/T. It is not clear what happens if intra-temporal relation reasoning task or the inter-sample relation reasoning task is removed from SelfTime. Does that make SelfTime same as one of the other baselines, e.g. \"Relation\"? Since most of the baselines used are adaptations from image domain, it is difficult to gauge where the proposed approach stands w.r.t. time-series specific methods like [1,2].\n2. For CricketX, DLD, and XJTU datasets, SelfTime performs significantly better than the supervised learning model (Table 2). How does one explain this observation as supervised methods would typically perform better or at least as good as the self-supervised methods?\n3. The authors observe that \"we find that the composition from a magnitude-based transformation (e.g. scaling, magnitude warping) and a time-based transformation (e.g. time warping, window slicing) facilitates the model to learn more useful representations.\" As per the description, it seems that this observation is based on analysis of just one dataset (CricketX) and on one algorithm (SelfTime). I think a more thorough description and evaluation across datasets and baselines could be useful.\n4. In the ablation results, it would help to see results across all datasets when using only inter-sample or intra-temporal losses. Also, the results in Fig. 5 and Fig. 6 are for only one dataset, and that too different ones: CricketX for Fig. 5 and UGLA for Fig. 6. Do the observations from Fig. 5 and 6 hold across datasets? Similarly, what motivates the particular pairing of source-->target for the Domain Transfer Evaluation? Given six datasets, several other combinations are possible - does this observation hold across all such combinations?\n5. The authors loosely mention that \"more augmentation results in better performance\" on the basis of references from existing literature. Though this might hold in other domains, an evaluation of the same for this setting would be useful to claim the same in the given context. \n\nGiven the above, a more thorough and consistent description of the contribution in light of recent related work and empirical evaluation is needed. Given works like [1,2,3], the contribution of the paper is limited to defining the intra-temporal task, but the same has not been evaluated carefully enough.\n\nThe write-up can be improved at several places, e.g.:\nEqns. 1 and 2 need a minus sign to be called as losses? \nIs i=j valid for the positive inter-sample relation pair? If not, Eqn. 1 might need an update.\nfire --> fair?\n$z_n$ is bold at a few places and not at other\nlimited amount training samples \nOr for video data\nthere is not enough feature\nwe generate from its transformation counterpart and another individual sample as the positive and negative samples respectively\nIn the experiment, we achieve new state-of-the-art results, and outperforms existing methods by a significant margin on multiple real-world time series datasets for classification task\na intra-temporal\nFirstly, takes the original time series signals and their sampled time pieces as the inputs\nshow that both two kinds of relation reasoning\nwill drop the evaluation performance\nt-SNE visualization of the learnt feature.\n\nReferences:\n[1] Unsupervised scalable representation learning for multivariate time series. Franceschi et. al. NeurIPS'19\n[2] Multi-task Self-Supervised Learning for Human Activity Detection. Saeed et. al. PACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2019.\n[3] Self-supervised relational reasoning for representation learning. Patacchiola and Storkey. NeurIPS'20\n\n** updating the score to 5 **", "extra_scores": {}, "extra_text": {}}, {"review_id": "U4IqNAxpJPQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper presents a general Self-supervised Time Series representation learning framework. It explores the inter-sample relation reasoning and intra-temporal relation reasoning of time series to capture the underlying structure pattern of the unlabeled time series data.  The proposed method achieves new state-of-the-art results and outperforms existing methods by a significant margin on multiple real-world time-series datasets for the classification tasks.\n\n**Contributions**\n1. The paper is well written and easy to follow. The organization is good. \n2. The architecture is well motivated. It is reasonable to use unsupervised temporal relations to learn video features. \n3. The qualitative results are numerous, insightful, and convincing on multiple datasets. The authors conduct extensive experiments to demonstrate the effectiveness of the proposed method, including inter-sample relation reasoning and intra-temporal relation reasoning.\n\n**Details**\n1. The novelty of the proposed method.\nThe Inter-sample relation reasoning is very similar to SimCLR, which also maximizes agreement between different views of augmentation from the same sample via a contrastive loss.  Considering this, the novelty is relatively incremental.\n\n2. Additional video recognition experiments.\nThe used dataset is small-scale that makes the task simple. I would have wanted to see results on large-scale classification tasks, such as video action classification.  The performance of action classification is closely related to the temporal feature modeling. So the effects on this task can make the proposed method more convincing. \n\n**Conclusion**\noverall, this paper proposes an interesting architecture for self-supervised temporal modeling. But the novelty is relatively limited compared to the recent SimCLR work. And it requires additional experiments on harder video classification task and datasets to show the effects and robustness of the proposed method.\n\n**After rebuttal**\n\nThanks for the detailed response.\nThis paper can be seen as an interesting attempt to use self-supervised on time series data. \nAlthough the basic idea is similar to SimCLR, It is still interesting work considering the  computation complexity and new loss function.\nSo I update my score to 6.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T1_UQN6H_c", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "===========Update after rebuttal==================\n\nI will remain my score of weak rejection.\n\n\n=============================================================\n\nThis paper presents to model both inter-sample and intra-temporal relations. The idea is naive but is easy to follow and reasonable. Experimental results support the effectiveness of the method.\n\nStrengths: The idea is reasonable and the application is important. \n\nWeakness:\n1. The notations are messy. The author should consider clean up some unused notations for better readability. Especially for section 3.1, I spend quite a few time thoroughly digest the notations. The explanation in section 3.2 is even scarier. I can easily understand the concept from the intro and Figure 1, yet I spend much more time reading sections 3.1 and 3.2.\n2. The presentation of the experimental section is also hard to follow. Specifically, what is the message conveyed in Figure 5? I would suggest the author highlight the region that we can pay attention to. Nonetheless, why only considering the composition of two augmentations? Instead of providing a matrix that considers all the combinations between two different augmentations, why not providing a table summarizing the sets of different combinations of augmentations? The sets of combinations can be the sets that the author likes us to focus on. \n3. An important paper [1] is missing. The author should discuss this missing reference.\n\nOverall, I am leaning positive toward this paper, yet I feel the presentation can be greatly improved. I am giving the score 5 for now, and I may update the score after the rebuttal.\n\n[1] Temporal Relational Reasoning in Videos, Zhou et al., ECCV 2018.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a general self-supervised time series representation learning framework. The organization is good, and the architecture is well motivated. However, the paper has limited novelty, and is a straightforward application of ideas in self-supervised learning literature.

Experimental results are not entirely convincing. The used dataset is small-scale that makes the task simple. A more thorough comparison with recent related work is needed. The presentation is also sometimes hard to follow.
Decision: Reject