Learning Through Limited Self-Supervision: Improving Time-Series Classification Without Additional Data via Auxiliary Tasks
Authors: Ian Fox
Harry Rubin-Falcone
Jenna Wiens
Venue:  (2020)
Keywords: Sequential Representation Learning
Self-Supervision
Function Approximation

Abstract: Self-supervision, in which a target task is improved without external supervision, has primarily been explored in settings that assume the availability of additional data. However, in many cases, particularly in healthcare, one may not have access to additional data (labeled or otherwise). In such settings, we hypothesize that self-supervision based solely on the structure of the data at-hand can help. We explore a novel self-supervision framework for time-series data, in which multiple auxiliary tasks (e.g., forecasting) are included to improve overall performance on a sequence-level target task without additional training data. We call this approach limited self-supervision, as we limit ourselves to only the data at-hand. We demonstrate the utility of limited self-supervision on three sequence-level classification tasks, two pertaining to real clinical data and one using synthetic data. Within this framework, we introduce novel forms of self-supervision and demonstrate their utility in improving performance on the target task. Our results indicate that limited self-supervision leads to a consistent improvement over a supervised baseline, across a range of domains. In particular, for the task of identifying atrial fibrillation from small amounts of electrocardiogram data, we observe a nearly 13% improvement in the area under the receiver operating characteristics curve (AUC-ROC) relative to the baseline (AUC-ROC=0.55 vs. AUC-ROC=0.62). Limited self-supervision applied to sequential data can aid in learning intermediate representations, making it particularly applicable in settings where data collection is difficult.

Under review as a conference paper at ICLR 2020
LEARNING THROUGH LIMITED SELF-SUPERVISION" IMPROVING TIME-SERIEE CLASSIFICATION WITHOUT ADDITIONAL DATA VIA AUXILIARY TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervision, in which target task improved without external supervision, has primarily been explored in settings that assume the availability of additional data. However, many cases, particularly healthcare, one may not have access to additional data (labeled or otherwise). In such settings, self-supervision based solely on the structure of the data at-hand can help. We ex- plore novel sulf-supervi framework for time-series data, in which multiple auxiliary tasks (e.g., forecasting) are included to improve overall performance on sequence-level target task without additional training data. We call this approach limited self-supervision, as we limit ourselves to only the data at-hand. We demon- strate the utility of limited self-supervision on three sequence-level classification tasks. two pertaining to real clinical data and one using synthetic data. Within this framework, we introduce novel forms of self-supervision and demonstrate their utility in improving performance on the target task. Our results indicate that lim- ited self-supprvision leads to a consistent improvement over a supervised baseline, across range of domains. In particular, for the task of identifying atrial fibril- lation from small amounts of electrocardiogram data, we observe a nearly 13% improvement in the under the receiver operating characteristics curve (AUC- ROC) relative to the baseline (AUC-ROC=0.55 vs. AUC-ROC=0.62), Limited self- self-supervision applied sequential data can aid in learning intermediate rep- resentations, making particularly applicable in settings where data collection is difficult.
hypothesize that
INTRODUCTION
Many problems involving sequential data, such as machine translation, sentiment analysis, and mor- tality prediction, naturally framed sequence-level tasks (Harutyunyan et al.] [017]] Hassan jet al. 2018: Radford et al. (017). Sequence-level tasks map a sequence of observations Xo.T to single label y. Learning this mapping is often made challenging due to high-D (dimension) low-N (number of samples) setting (Nasrabadi) 2007). Such problems are particularly prevalent in healthcare tasks, which often involve limited quantities of labeled data captured at high temporal resolution (e.g., electrocardiogram waveforms).
In high- D low-N settings, researchers have had success with transfer learning techniques, by lever- aging additional data to learn intermediate representations that are then used in the target task. When additional data are unavailable, it may be possible to improve the intermediate learned representa- tion of the data with respect to the target task by considering additional tasks intrinsic to the data. In particular, we hypothesize that the structure of sequential data provides rich source of innate supervision. For example, signal reconstruction forecasting could improve the intermediate rep- resentation by capturing the underlying data-generating process. Such approaches are examples of self-supervision, where labels derived from the input (as opposed to external sources).
In this paper, we show that leveraging the sequential structure of the data at-hand can lead to im- proved performance ering supervuper auxiliary tasks (e.g., signal reconstruction), in addition to the sequence-level task, one can learn useful intermediate representations of the data. Past work investigating self-
sequence-leve tasks (i.e., the target task). More specifically. by consid-

---

Under review as a conference paper at ICLR 2020
supervision for sequential data has focused full-signal reconstruction (Dai & Le, 2015), and to a lesser extent forecasting (Ramachandran et al.; 2016). Building on past work, we examine the sequential data when additional data unavailable, and propose new types of self-supervision tasks. We refer to this approach as 'limited self-supervision. We limit the self-supervision to the data at-hand, and focus on self-supervised auxiliary tasks relevant to sequential data ordered by time (i.e., time-series data).
utility of self-supervisiin
Our main contributions are follows:
We demonstrate the efficacy of the proposed limited self-supervision framework for im- proving performance across datasets/tasks with no additional data.
We compare the utility of several different existing forms of self- self-supervision in our limited- data setting, identify consistent trends across supervision types, and demonstrate the utility of combining multiple different forms of self-supervision.
We propose a new form of self-supervision, piecewise- autoencoding, that trades off fine-grained signal modeling and long-term dependency propagation. We demonstrate that this is the best form of limited self-supervision across all tasks.
Our work suggests that there a wide range of time-series and sequence classification tasks where limited self- self-supervision could improve performance. It also shows the value of including multiple, simultaneous streams of auxiliary self-supervision. Our findings present a methodological contribu- tion, in the form of useful type of self supervsin piecewise-linear autoencoding. Further, our empirical findings on when and how auxiliary tasks help can inform future work in developing self- supervvperv techniques.
RELATED WORK
Previous work has found value in self-supervised pretraining with large amounts of unlabeled data (Devlin al., 2018 Ramachandran al., (016). In our work, focus on time-series data instead of language data. Several previous works have examined self-supervision for pretraining or feature extraction the context of time series (Hyvarinen Morioka, 2016 Oord et al., 2018). We sub- stitute the pretraining framework with multitask learning framework. removing the requirement to train multiple individual models. However, in contrast to standard pretraining or multitask learning setups, we do not assume the availability of additional data for training. We posit that even in the absence of additional data, self-supervision can lead to improved performance on the larget task.
Multitask learning deals with training single model to perform well on multiple tasks. By simulta- neously training on multiple related tasks and sharing representations, multitask models can improve generalization (Caruana 1998). Multitask learning has been used successfully across number of different clinical tasks Wiens et al., 2016 Ahmed et al. 2016 Razavian et al., 2016 Harutyunyan et al. 2017). In particular, the success of multitask learning in deep learning demonstrates its value for representation learning (Ruder) (017). Within the context of supervised learning, Schwab el al. considered multitask framework for learning from sequential health data (Schwab et al.) 2018). Though they used self-supervision, they only considered setting where large amounts of unlabeled data were available.
Our work was inspired by the findings of Dai and Le (Dai & Le, 2015). Dai and Le compared sequence-autoencoding and language modeling as auxiliary tasks for leveraging large pools of unla- beled data for natural language tasks (e.g., sentiment analysis). Their approach led to offte--o--he-an led to larger improvements
performance on a range problems. They found than language modeling. Interestingly, they found jointly training on the main and auxiliary task de- creased performance relative the baseline. contrast, we foeus on self-supervision for time series and do not assume access to additional data.
Choi et al. 2018)) examines self-supervision without additional data applied to hierarchical EHR data. They also demonstrate the benefit of adding auxiliary supervision. Our work differs from this by i) using sequential as opposed to hierarchical structure, ii) examining multiple streams of simultaneous supervision and ii) comparing broad range of auxiliary tasks general time series data. [Co et al. (2018) find that training a sequence classification model to simultancously forecast the data imputation method improves overall performance, though they suggest this is due to
2

---

Under review as a conference paper at ICLR 2020
better handling missing data. We examine the impact of auxiliary sulf--upervision more generally as a way to improve supervised representation learning.
3 LEARNING WITH SELF-SUPERVISED AUXILIARY TASKS
In this section, we present our proposed limited self-supervision framework. After describing our notation, we present our baseline encoder-decoder architecture and describe four self-supervised auxiliary tasks.
3.1 PROBLEM DEFINITION AND NOTATION
We define a sequence as a set of observations {xt 1x=) Xe E Rd ordered by the index t. We denote such a sequence as Xo:T. Each observation X₃ is a d-dimensional vector. We focus on univariate, evenly sampled time series.
We categorize time-series tasks across three dimensions: i) target vs. auxiliary tasks, external supervision vs. superviirn and iii) sequence-level vs. subsequence-level tasks. A target task the task of interest, whereas auxiliary tasks are only useful insofar as they improve performance on the target task. External supervision occurs when the task labels are provided by external source (e.g., object recognition). Self-supervision occurs when no additional supervision is required to generate the ground truth label (e.g., in autoencoding, the input itself serves the supervision). A sequence-level task is one where the supervision pertains to the entire sequence (i.e., sequence classification). A subsequence-lewe nal. In our work, all of the target tasks are sequence- tasks requiring external supervision. We denote the label of the target task y. Our auxiliary tasks are all self-supervised and may be either sequence- or subsequence-level.
task provides multiple instances of supervision across the sig-
3.2 BASELINE ARCHITECTURE
In this work, we compare four different self-supervised auxiliary tasks. Throughout, we consider a fixed encoder architecture, focusing the improvements offered by the auxiliary tasks. Specifically, we focus on recurrent neural networks with LSTM cells. We use a 1-layer LSTM with a number of hidden units determined on a task-by-task basis. These architectures have proven useful for sequential tasks in many domains (Oord et al. [2166] Ramachandran et al.) 2016). including health data (Harutyunyan et al. 2017). We note, however, that nothing in our framework presupposes particular architecture, these techniques could work with any representation-leamiin gradient- based approach.
Figure la depicts our baseline architecture and its relation to auxiliary tasks. The encoder is im- plemented as single-layer I.TT. The target decoder is single fully connected layer mapping from hidden state 27 to the output. This simple output layer purposely places a heavy burden on the encoder, since the representation learned by the encoder is shared by (and thus may improve from) all tasks. We train the model by minimizing the multi-class cross-entropy between our predictions y (put through a softmax activation) and the one-hot distribution representing the correct label class (Eqn. 1).
min Ltarget 8p
Σ (0) (p)
(1)
2T
Exxor: 0g)
Where 0g and @D are parameters for the encoder and decoder respectively and indexes class labels. We compare this baseline architecture trained with no auxiliary tasks to models trained with up to four auxiliary tasks.
3.3 SELF-SUPERVISED AUXILIARY TASKS
We consider four self-supervised auxiliary tasks: i) autoencoding (i.e. reconstruction), ii) forecast- ing, iii) partial-signa autoencoding, and iv) piecewise-lines autoencoding (shown in Figure 1b). The auxiliary tasks take input the output of the encoding network. Task X is implemented
3

---

Under review as a conference paper at ICLR 2020
Xy
X
%
Shared Encoder
Auxiliary Tasks
OAF
O
OAE
8t 11-
-2 r+
Or
Xt
A
Zxp
20
Xo
-Z0
DF
8t+1
2++2 Op
Xt+2
Dps
RAE
RAE
Xt
2
Dr
xx+1
DF
X1
E
-Z1
XT
Zt
Dps
Z-T
DAE
DF
Lincar interpalation
DA
XT
E
-TT
Drl
8 1--++1100
Ros
2i-h
PAE
Ops
&t-h
(Vn-li Pa-1)
Ra
PLA
O,,
Oa
Vn.
Pn.
D
1
Ros
7
PAB
Ops
xt-2
(vo.Po
0)
Opee
Our
11
Target
Xt
Rys zi
PAE...
R.
L.d
-V0
Decoder
Dps
Zt
DPL
2T
(a)
(b)
Figure I: a) Our full task architecture. The green encoder layer is shared across all tasks. The target decoder fully connected layer mapping the correct label size. The auxiliary tasks connect to hidden states. Note the sequence-level tasks only connect to the final hidden state. b) Each auxiliary task decoder Dx is composed of a recurrent layer Rx and an output layer Ox. DAE) Autoencoder. DF) Forecaster. Dps) Partial Partial-Signal Autoencoder. DPL) Piecewise- Linear Autoencoder.
decoder Dx composed recurrent layer Rx and a fully connected output layer Ox. While our sequence-level target task requires only prediction the end of the sequence, some of our aux- iliary tasks are subsequence-level, in which case the task takes as input the intermediate encoding representation Zμ.
i) Autoencoding (AE). This is a standard method for unsupervised training in which we seek to, given the final hidden state produced by the encoder, output a reconstruction of the complete signal that minimizes error (Eqn. 2). This task requires that the hidden state encode compressed version of the input, compression encourages learning latent structure and discourages learning noise. The decoder, parameterized by single-layer autoregressive LSTM, outputs sequence: DAE(ZT) = So. Note zt E' 0g), and thus depends 0g.
min LAE 0g.AAE
| DAE(ZT;0AE))
ii) Forecasting. In this task, the decoder takes the hidden state produced any time step by the de- coder and attempts to predict the next h (for example, 6) elements in the sequence. This requires the hidden state to encode the dynamics of the data-generating process. The decoder outputs Xt+1:4+h given past values Xo.t, minimizing (Eqn. 3). We use single-layer LSTM similar to the AE decoder, though it is used at each time-step to decode the next h observation values.
T-h
min LF 0g.8F
Σ
| Dr(z; 0F)
E(xn=;0x:)
iii) Partial-Signal Autoencoding (PS-AE). This auxiliary task is a variant of AE that differs in three ways: 1) instead of decoding the full signal Xo.7 it decodes only the previous h (for example, 6) steps of the signal Xt-h:1-1- 2) instead of one prediction being made at the end of the encoder, a prediction is made every encoding step from X₁, onwards, and 3) the input to the decoding layer includes the current value. x, (Eqn. 4). implemented identically to the Forecast Decoder, the only difference is that predicts the previous h observation values. This task allows examine
4

---

Under review as a conference paper at ICLR 2020
the impact of signal reconstruction without requiring learning tong-term dependencies, allowing for a more meaningful comparison with Forecasting.
min Lps 0g.Pps
X - 1aat-11Dps(a..) Drs(2;;0s)))
(4)
iv) Piecewise-Linear Autoencoding (PI.-AE). piecewise-linear approximation, or combina- tion of line segments, is capable of efficiently representing wide class of signals (particularly non-periodic signals). It is promising choice for an auxiliary task as it encourages compact representation capturing the most important details of the signal. A piecewise-linear representation consists of two length n n++ vectors (where ni is the number linear segments the signal), a value vector and position vector p. The positions length, between 0 and The decoder produces a series of points which define a piecewise- linear reconstruction of the complete input signal. The reconstruction is defined by linear interpolation between the series of points (po. on) (Pa, "a), where vú is the value of the signal at the point i, and p; is the relative position (or time) where the point occurs. Our decoder produces n + such points using a singlc-layer autoregressive LSTM where the hidden state is fed to two output layers which map z!" 2L Pt). For each point i [0, n 1] we feed the previous point value v and the sum of previous positions Σ5=0 to the decoder. After we have generated the target number of points, we normalize the position values to enforce Σj=0 Pj T and perform the interpola- tion. As this entire process is differentiable, we optimize the decoder directly on the interpolated reconstruction loss (Eqn. 5).
defined proportions of the original signal
min LPL = ||xO.T DpL(ZT;0PL))1 0g.0pl
(5)
Additional details on these tasks can be found in Appendix A.1
3.4 TRAINING
We optimize our model to minimize the loss:
L = Ltarget + Luw
(6)
Where Learger is cross-entropy loss and
LA =aAELar aFLF apsLps apıLpI.
weighted summation of auxiliary MSE losses. The weighting terms ax are defined as the auxiliary task is not being used for training. If the task is being used, then ox = Lx
Lirrgt
where the losses are calculated at the beginning of training using the newly-initialized network on the training data. This ensures that all tasks have losses of similar magnitude.
4 EXPERIMENTAL SETUP
To test our hypothesis, that auxiliary self-superviiion improves performance on sequence-level tasks, we consider at variety of sequence-level tasks across different types of synthetic and real data.
4.1 TARGET TASKS DATASETS
We consider the following three tasks (two of which are based on publicly available real datasets):
Piecewise-Linear Segment Prediction (PLA). We begin with simulated data, as allows us to estimate the ability of self-supervised auxiliary tasks to identify long-term& dependencies in the data. signals, each length 100. Point values are drawn from
The dataset is composed piecewise- uniform distribution and lie between -1 and 1. The number of line segments also varies uniformly between and 6. The target task for this dataset to estimate the number of distinct segments that occurred in the signal. 1,000 training, validation, and test sequences were generated independently.
Patient Classification using Glucose Data (TID). This task uses publicly available continuous glucose monitor (CGM) data collected from people with type diabetes (TID) (Fox et al.) 2018).
5

---

Under review as a conference paper at ICLR 2020
Each signal Xu:7 consists of 288 glucose measurements sampled every five minutes over the course of a day. This dataset contains 1,863 days of data from 40 patients. In this task, we aim to classify patients based on their data. Here, y . 40} represents the patient. Classifying patients is proxy for the important problem of identifying signal dynamics. We preprocess the data by removing physiologically implausible glucose measurements, and linearly interpolating missing chunks of data. We exclude signals in which more than 20% of the measurements are missing and those that are missing a contiguous block longer than two hours. Data were collected by series of multi-day sessions separated by three-month intervals. As our test set, we consider the final recording session from each patient. We select our validation set randomly from the remaining data.
Atrial Fibrillation Detection (AF). Our final task uses electrocardiogram (ECG) data from the publicly available 2017 PhysioNet Challenge (Clifford et al.) 2017), in which the goal was to auto- matically diagnose atrial fibrillation (AF). This dataset contains four unevenly distributed classes: normal sinus rhythm, AF, other arrhythmia, and noise. We use the training data provided for the competition (the test data are not publicly available), resulting in 8,528 samples. 771 of those sig- nals are labeled AF. We exclude signals with less than 30 seconds of data (967 signals total, 127 with AF) and truncate all signals to exactly 30 seconds. We also downsample the data, reducing signal size from 9,000 to 125. This speeds up training time and eases memory requirements. We use the validation set provided for the challenge as the test set (removing those examples from the training set), and randomly sample 10% of the training data for use as a validation.
4.2 IMPLEMENTATION DETAILS
We implement all models in PyTorch (Paszke al., 2017), and optimize model parameters using Adam (Kingma & Ba 2014) with an initial learning rate of le 3 (the default PyTorch value). In practice we found altering the decoding horizon had little effect on performance, so we used h = 6 for all experiments. All encoding/decoding layers are composed of a single recurrent layer an identical number of hidden units, set on a per-task basis using performance on the validation set to balance training time, memory constraints, and target-task performance (evaluated using the early- stopping validation set). We used 128 hidden units for the PLA target task, 512 hidden units for the TID target task, and 256 hidden units for the AF target task. The decoding layers also have (or two for PL-AE) fully connected output layers. For the PL-AE, we set the number of line segments = 6 as reasonable size to approximate many signal types. We mitigate the risk of overfitting by using early stopping on a withheld validation set, training until we fail to improve performance for over 50 epochs and reporting test performance for the best performing model on the validation data. Some auxiliary tasks make predictions at multiple points in the signal. This helps prevent the vanishing gradient problem, which can impede learning with large sequences. To avoid conflating these sorts of improvements with those caused by learning better representations, we use label propagation with our target task (sequence classification), linearly annealing contributions to the loss function over the length of the signal (Dai & Le. 2015). The propagated losses are combined using a weighted average, with the weights linearly annealed from 0 to 1 over the length of the signal. Our code and synthetic data will be made publicly available to allow for replication and extension. For the purposes of double-blind peer review, we have released the code and PLA data on an anonymous Google Drive accoun
RESULTS
We begin by establishing that the proposed self-supervised auxiliary task framework improves target task performance. We then look more carefully at the effects of different types of auxiliary tasks. We conclude by looking at the relationship between auxiliary task and target task performance, which sheds light on the mechanism by which auxiliary tasks improve performance.
To evaluate the results of these experiments, we measure the macro- averaged AUC-ROC on the target task, and the mean absolute percent error (MAPE) for the auxiliary tasks. We repeat all experiments using three random initializations and average the results.
The Benefit of L Limited Seff-Supervision. We begin by examining our main hypothesis, that limited self-supervision improves sequence-level task performance without additional data. In Figure 2
https://bit..t 1y

---

Under review as a conference paper at ICLR 2020
Auxiliary Improvement: T1D
Auxiliary Improvement; AF
Auxiliary improvement: PLA
0.650
0.9
0.625
0.65
0.600
AUC-ROC
0.8
0.575
0.60
0.7
0.550
0.6
0.55
0.525
0.500
0.50
Auxiliary Tasks
# Auxiliary Tasks
Number of Auxiliary Tasks
(a)
(b)
(c)
Figure 2: Performance across three target tasks by number of auxiliary tasks used (averaged over all possible orderings). In general, we observe that the greater the number of auxiliary tasks, the greater the performance for all three tasks. The marginal improvement from including additional auxiliary tasks appears to taper off as the number of tasks increases.
Averaged Improvement: PLA
Averaged Improvement:
Average Improvement: T1D
Improvment to AUC-ROC
0.150
0.008
0.015
0.125
0.100
0.006
0.075
0.004
0.005
0.002
0.025
0.000
0.000 AE ForecastPS-AE PL-AE
0.000
AE ForecastPS-AE PL-AE
ForecasIPS-AE PL-AE
Auxillary Task
Auxiliary Task
Auxillary Task
(a)
(b)
(c)
Figure 3: The average marginal contribution of each auxiliary task across target tasks. The PL-AE task consistently offers the greatest marginal improvement, and the forecasting task consistently provides the least.
we plot target-task performance across our three tasks with a varying number self-supervised auxiliary tasks. We estimate performance for given number of auxiliary tasks by averaging the performance of all possible combinations. For three sequence-level tasks, the inclusion of four auxiliary tasks improves performance relative performance tends to increase with the number of auxiliary tasks. The one exception is the T1D task, where improvement peaks at three auxiliary tasks.
auxiliary tasks. Moreover, observe that
Relative Contribution of Different Auxiliary Tasks. We now examine how different auxiliary tasks contribute differently to performance. To measure the impact of individual auxiliary tasks, we average performance across all models that include the auxiliary task versus all models that do not. This allows us observe an auxiliary task's individual contribution, and its ability to usefully combine with other streams of self-supervision. The averaged change in AUC-ROC indicates the marginal improvement offered by the task (Figure(3. PL-AE outperforms all other auxiliary tasks, including AE, on all three datasets. Since PL-AE limits the temporal granularity of the output, this suggests that modeling fine temporal granularity does not help, and may even hurt performance. The forecasting task underperforms all other auxiliary tasks. This finding is in line with the findings of (Dai & Le 2015). However, the explanation Dai and Le provide (that the AE encourages long-ten term dependency modeling) is inconsistent with the performance of PS-AE, which also does not model long-term dependencies. The fact that the PS-AE outperforms the forecasting task, but generally un- derperforms AE, suggests that while modeling long-term dependencies may improve performance,

---

Under review as a conference paper at ICLR 2020
Table 1: The performance of particular combinations of auxiliary tasks. All methods include the dataset-specific target task. AE and Forecast refers to Autoencoding and Forecasting respectively, the auxiliary tasks explored in (Dai & Le 2015). PLAE and PSAE refer to Piecewise-Linear Au- toencoder and Partial-Signa Autoencoder, our novel forms of self-supervision. We see that our newly proposed forms of self-supervision outperform the other approaches on all datasets.
AUC-ROC 100
Auxiliary Tasks
PLA
TID
AF
None
50.0 0.4
67.1
55.2 4.
AE
88.6 0.3
66.7 1 0.4
59.8
122
Forecast
53.5 5.2
1 0.5
57.0
2..0220
Ao+Foreusst (Dai Le 2015
89.5
6..4+10
6..0
PLAE+PSAE (Ours)
89.6 0.4
60.0 0.5
All (Ours)
90.7 0.3
67.8
6.14
Effect of Training Data: PLA
AE MAPE AUROC
Improvement in AUC-ROC
0.4
0.70
0.3
AUC-ROC
0.65
0.2
Auxiliary
Data
500
0.1
0.60
1000
1500
0.0
100
500
1000
0.2
0.4
0.6
0.8
Training Data Size
AE MAPE
(a)
(b)
Figure 4: A) The effect training data size has on auxiliary tasks. As the amount of data increases, tend see increase the improvement afforded by the auxiliary tasks. B) The relationship between auxiliary task performance (in MAPE, lower is better) and target task performance (in AUC-ROC, higher is better) the TID data. Calculated over all training data sizes, there is somewhat weak relationship (Pearson R = -0.53, line not shown). However, when we condition on the amount of training data, we find that different relationships emerge (Pearson 0.34 when training size = 1,000 vs. -0.72 when training size 1,500).
there is likely some other reason that AE works well. We also examine the performance of particular combinations of self-supervised auxiliary tasks in Table We that our proposed auxiliary tasks outperform and are complementary to standard auxiliary task combinations.
Why do Self-Supervised Auxiliary Tasks Help? To investigate the underlying mechanism by which auxiliary tasks improve performance, we examine model performance as we vary the number of auxiliary tasks and the amount of training data (results shown in Figure 4a). As the amount of training data increases, the added value from the auxiliary tasks increases on average. This suggests that auxiliary tasks are not simply acting as a form of regularization, since otherwise we would expect to see larger improvements on smaller training sets.
If the auxiliary tasks result in better representations (not just regularized representations), their im- pact on performance should correlate with the auxiliary task performance. A decreuse in auxiliary task error should lead to a better intermediate representation and an increase in target task perfor- mance (i.e, AE MAPE and AUC-ROC should be negatively correlated). Mcanwhile, if the auxiliary tasks work as regularizers, exhausting representation capacity, we would expect to see little effect or positive correlation (if the regularization effect is too strong, and auxiliary performance comes at the cost of target performance).
This relationship is highly dependent on the amount of data. When training data are limited, there is a weak positive correlation between auxiliary task error and target performance, suggesting regularizing effect. However, with
We explore this, specifically for AE with the TID data in Figure 4b
8

---

Under review as a conference paper at ICLR 2020
the full amount of training data, see strong negative correlation. Auxiliary tasks give sizable improvements at 1,000 and 1,500 training examples (averaged improvement of 0.016 and 0.010 respectively). This suggests that 1) auxiliary tasks act both to regularize and to enhance intermediate representations, depending on the amount of data, and 2) there an amount of training data where they are effective in either role. These findings suggest that auxiliary self-supervised tasks may be useful across wide range of training set sizes.
CONCLUSIONS
In this paper, we introduced a limited self-supervised framework, in which we sought improve sequence-level task performance without additional data. By jointly training our target task with auxiliary seff-supervised tasks, we demonstrated small but consistent improvements across three different sequence classification tasks. Our novel piecewise-lineal autoencoding task emerged the most useful auxiliary task across all datasets. In contrast, forecasting, which presents an intuitively appealing form of self-supervision, led to the smallest improvements.
Across a range of training set sizes, we showed that the value of auxiliary tasks lies improving intermediate representations learned by the network. When limited training data are available, these tasks serve as a form of regularization. With more training data, performance on the auxiliary tasks improves and does performance on the target task, suggesting more useful intermediate representations. In the context of time-series analysis, limited self-superviiion is an effective form of supervision and comes at little cost. Going forward, researchers seeking improve performance on sequence-level target tasks should consider incorporating self-supervised auxiliary tasks.
REFERENCES
Bilal Ahmed, Thomas Thesen, Karen Blackmon, Ruben Kuzniecky, Orrin Devinsky, Jennifer Dy, and Carla Brodley. Multi-task learning with weak class labels: Leveraging iEEG to detect cortical lesions in eryptogenic epilepsy. Machine Learning for Healthcare Conference, pp. 115-133, 2016.
Wei Cao. Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, pp. 6775- 6785, 2018.
Rich Caruana. Multitask learning. In Learning to learn, pp. 95-133. Springer, 1998.
Edward Choi, Cao Xiao, Walter Stewart, and Jimeng Sun. Mime: Multilevel medical embedding of electronic health records for predictive healthcare. In Advances in Neural Information Processing Systems, pp. 4547-4557, 2018
G. Clifford, Chengyu Liu, Benjamin Moody, L. Lehman, Ikaro Silva, Qiao Li, A. Johnson, and R. Mark. AF classification from short single lead ECG recording: The Physionet Computing in Cardiology Challenge 2017. Computing Cardiology, 44. 2017.
Andrew M. Dai and Quoc V. Le. Semi-supervised Sequence Learning. arXiv:1511.01432 [cs), November 2015. URL htttt::/aaxiv.orrttt 1511
arXiv: 1511.01432.
Jacob Devlin, Ming- Mei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint /8 1..000080000000. 2018.
Ian Fox, Lynn Ang, Mamta Jaiswal, Rodica Pop-Busui, and Jenna Wiens. Deep Hulti-Output Fore- casting: Learning Accurately Predict Blood Glucose Trajectories. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18, pp. -195, New York, NY, USA, 2018, ACM. ISBN 978- 1-450--5552222 doi: 10.1 3219819.3220102. URL htR:ttt::// //tp:: acm. wtt/o0.../.... 1141 011444//2219999138 3220102
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.

---

Under review as a conference paper at ICLR 2020
Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, and Aram Galstyan. Multitask Learning and Benchmarking with Clinical Time Series Data. arXiv:1703.077 [cs. stat], March 2017. URL http: org/
arXiv: 1703.07771.
Hany Hassan, Anthony Aue. Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dommun William Lewis, and Mu Li. Achieving Human Parity on Automatic Chinese English News Translation. arXiv preprint arXiv:1803.05567 2018.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, pp. 3765-3773, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.66944, 2014.
Nasser M. Nasrabadi. Pattern recognition and machine learning. Journal of electronic imaging, 16 (4):049901, 2007.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel Recurrent Neural Networks. arXiv: 1601006700 [cx]. January 2016. URL http://arxiv. org htt///b//1// wwwpaabsla....e..ae......bb....b 06759 arXiv: 1601.06752
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-WS W, 2017.
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to Generate Reviews and Discovering Sentiment, S.3iv:1704.0144: [cs], April 2017. URL http:///arxi..rr//bbb/// 01444 arXiv: 1704.01444.
Prajit Ramachandran, Peter J. Liu, and Quae V. Le. Unsupervised pretraining for sequence to se- quence learning. arXiv preprint 16 1oXiv:6611.02600 1,02683,, 2016.
Narges Razavian, Jake Mareus, and David Sontag. Multi-task prediction of disease onsets from longitudinal lab tests. arXiv preprint 1786471608.00647, 2016.
Sebastian Ruder. An Overview of Multi-Task Learnig Deep Neural Networks. arXiv: 706.05099 [cs, stat), June 2017. URL http::ttt////rii.oor///aaa/////
05098 arXiv: 1706.05098.
Patrick Schwab, Emanuela Keller, Carl Muroi, David Mack, Christian Strssle, and Walter Karlen. Not to Cry Wolf: Distantly Supervised Multitask Leaming in Critical Care. arXiv:1802.05000 [cs, stat], February 2018. Uttt:/tt://arxx
abs 1802. 05027 arvi: 1802.05027.
Souhaib Ben Taieb, Antti Sorjamaa, and Gianluca Bontempi. Multiple-output modeling for multi- 2010.
step-ahead time series forecasting. Neurocomputing,
Jenna Wiens, John Guttag, and Eric Horvitz. Patient risk stratification with time- varying parameters: multitask learning approach. The Journal of Machine Learning Research, 17(1):2977-2819 2016.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 AUXILIARY TASKS: ADDITIONAL MOTIVATION AND DETAILS
A.1.1 AUTOENCODING
Motivation. Autoencoding encourages the hidden state retain all relevant information from the original input. By reducing the size of the hidden state, our model must learn a compressed representation of the signal. Compression encourages learning latent structure and discourages learning noise.
Details. We use single-layer autoregressive LSTM for the recurrent layer RA which initialized using the hidden and cell states from the encoder LSTM and sequentially decodes hidden states z AE These hidden states are fed into a fully-connected output layer, OAB. that maps 2A Xy. The output & is then fed into RAE, generating 2 continuing until Ro:r is fully generated. To provide a shorter path for gradient flow, we decode in reverse order, generating Ro instead of 8o.7 (Goodfellow et al. 2016).
A.1.2 FORECASTING
Motivation. Forecasting encourages to encode the dynamics of the data-generating process. Without information about the underlying dynamics, future value prediction is either challenging or trivial (if the signal does not change). Such dynamics may carry valuable information for range of sequence-level tasks, though the aspects of the dynamics most relevant to the target task may differ from those that predict future values. Details. We focus simultancously (Taieb et al. 2010). This is done using a recurrent decoder, similar to the autoen- coder, but applied at each encoding step and expanded only h steps, where is hyperparameter.
multi-output forecasting architecture, predicting several future values
A.1.3 PARTIAL-SIGNA AUTOENCODING
Motivation. Previous work has found advantages reconstruction over prediction (Dai & Le 2015). Dai and Le hypothesize the observed superiority of sequence autoencoding over forecast- ing may result from the short-term nature of the language modeling task (only predicting the next word). To investigate the effect of short-term dependency auxiliary tasks, casting system, where we can examine the effect of varying the prediction horizon. Analogously, we use PS-AE examine the effect of producing short-term reconstructions.
a multi-step fore-
Details. PS-AE implemented using setup identical to our forecasting approach, except we estimate the previous h values instead of the subsequent values (Figure Dps).
A.1.4 PIECEWISE-LINEAE AUTOENCODING
Motivation. AE encourages fine gre-grained modeling of the signal over long periods, whereas PS- AE encourages fine ain-ggrinedd modeling over a short period. While PS-AE can control the range of temporal dependencies modeled using the decoding horizon h, it does not vary the granularity at which the output is modeled. What level of signal granularity is required for reasonable target-task performance? To explore this question, we introduce Piecewise-Linen (PI.) Autoencoding.
Details. To generate PL representation of a signal with n distinct pieces, take the encoded representation of the signal, žr, and feed into a recurrent layer RpL. Figure 1 a, Der. il- lustrates our
system. The sequential output of the decoder, 2,"', generates the point value and position. Specifically, we use two fully-connected output layers, Opat and O pos. that map z,"L + Uj.P; respectively. Since we know = O, at the first step we generate only an initial value. We continue the decoding process, feeding V,, Ek=o Pk to the decoder to generate z PL until we have generated n + 1 points. We then use linear interpolation to map (Vo. Po) ( P.) + &n.T. We normalize the position vector and use the cumulative summa- tion to determine segment positions.
11

---

Under review as a conference paper at ICLR 2020
P.A
PLA
FomeLt
NOTAE
Not
N07 Nece
ALL
Figure 5: Average AUC-ROC for every combination of Auxiliary task for all three analyses. Here, columns marked 'NOT included all but the indicated auxiliary task.
Table 2: Accuracy on 7 data sets from the UCR Archive.
Data Set
Baseline-ACe
Best-ACC
Best-Tasks Forecaster Inline PLA
SOTA- ACC
ElevtrieDevices
59.6
FordB
60.3
75.4
FordA
61.4
86.6
Not Forecaster
96.5
Wafer
89.1
95.2
AE
100.0
HandOutlines
54.7
54.7
None
92.4
TwoPatterns
64.4
Forecaster+Inline
StarLighttCurves
Not PLA
98.0
A.2 ALL COMBINATIONS OF AUXILIARY TASKS
In Figure 5 we present plots showing average AUC-ROC for all possible combinations of auxiliary tasks, for each data set. Here the contribution of each task can be examined.
A.3 APPLICATION TO UCR TIMES SERIES DATA-SETS
In order to evaluate the use of self-supervist auxiliary tasks in an casily replicable setting, and in order to facilitate comparisons with state-of-the are methods, we performed classification on data sets from the UCR Time Series Archive http::ttps//// /wwp///www.// cs wcr. www.:eddueeammonn seriees_aaae _atts____aaa__ 20187] We included tasks that had fixed length sequences and sample to label ratio greater than 100. We only included the subset of these tasks for which our baseline architecture performed better than random chance. These selections were to ensure that the tasks were reasonably well suited to baseline architecture. For each dataset, the last 20% of the training sample was used as a validation set, with the testing sample used as a test set.
iime
Table 2 shows bascline results for each task (accuracy when model was run with auxiliary tasks), the highest accuracy achieved with any combination of auxiliary tasks, and state-of-the-art performance according tecture is routinely and drastically outperformed by state-of-the-att methods, the addition of auxil- iary tasks improved performance be significantly closer to the state-of-the-ae level in most cases, and our model achieved at- state-of-the-ar accuracy for the Two Patterns task. Although our model was unable achieve state-of-the-art performance on most tasks, the improvement over baseline indicates that the addition of self-supervised auxiliary tasks could enhance better suited baseline architectures on these and other tasks.
fi cation. com Although the baseline archi-
12


Reviews:
[{"review_id": "H1llIYPjFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a so called self-supervised method for learning from time series data in healthcare setting. Specifically, here self-supervision is achieved via designing auxiliary tasks based on data's internal structure to create more labeled auxiliary training tasks.\n\nFrom both perspectives of methods and applications, the proposed model has very limited novelty. It is just one application of multitask learning. Also very similar idea has been implemented by [1]. In [1], the authors learn multi-level embedding to make disease/risk prediction, where the embedding was jointly trained by performing auxiliary prediction tasks that rely on this inherent EHR structure. The authors need to state what is the novelty of the proposed method compared with [1].\n\nIn addition, the performance evaluation missed many baselines. Table 1 seems more like a ablation study rather than a performance comparison. You need to compare with all state-of-the-art models in computational phenotyping in order to show the performance advantage brought by the proposed mode design.\n\n[1] Edward Choi, Cao Xiao, Walter Stewart, Jimeng Sun, MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare,  NeuRIPS, 2018\n\n ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BklmVCchYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose an approach for self-supervised learning on time series.\nThree datasets are considered (simulation and 2 healthcare datasets).\nThe gist of the contribution is to both optimize prediction loss\nof the true task and at the same time do a good job for a family\nof auxiliary tasks. 4 auxiliary tasks are considered. While\nthe first 3 auxiliary tasks are quite common, the 4th tasks\ncalled piecewise-linear autoencoding appears novel. The idea\nis that the hidden representation of the LSTM should be a good predictor\nof the past using a piecewise-linear approximation.\nThe author coin the term \"limited self-supervision\" for their approach\nalthough it's not clear why it is fundamentally not just self-supervised\nlearning as it has been proposed in the past.\n\nThe paper is overall well written and addresses the relevant issue\nof learning from limited annotated data.\n\nMajor concerns\n\n- It is yet another way to do self-supervised learning on time series\nand no clear benchmark with alternatives is provided (time contrastive\nlearning (TCL) or Contrastive Predictive Coding (CPC) https://arxiv.org/pdf/1807.03748.pdf\netc.)\n\n- On any of the applied problem it is not clear if the proposed\napproach brings an improvement on the state-of-the-art or if it's\njust an illustration of the method disconnected from the literature\nof the application.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BklOzy2atr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the use of many auxiliary tasks to boost the performance on a target task by means of `'self-supervision'. Specifically, they considered auto-encoding, forecasting, partial-segment auto-encoding, and piecewise-linear auto-encoding. \n\nThere are major concerns that should be clarified or described in detail.\n1) The overall architecture is not complete. the architectures used in the experiments are not described concretely.\n2) To this reviewer, the idea of self-supervision is similar to the unsupervised learning for representation learning. \n3) The methods of BERT (Bidirectional Encoder Representations from Transformers) [Devlin et al., 2018] or BRITS (Bidirectional Recurrent Imputation for Time Series) [Cao et al., 2018], although different for their target tasks in their original work, could be also regarded as self-supervision technique and could be interesting to compare with them.\n4) The experimental settings are not described well, thus lack of reproducibility\n5) It is unclear which aux-tasks were applied in Fig. 2. Further to better understand and analyze the results, it is required to conduct more rigorous ablation studies.\n6) There is no comparison with recent work on the same datasets.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses  an important problem of self-supervised learning in the context of time-series classification. However, all reviewers raised major concerns regarding the novelty of the approach and the quality of empirical evaluation, including insufficient comparison with the state-of-art and reproducibility issues. The reviewers agree that the paper, in its current state, does not path the ICLR acceptance threshold, and encourage the authors to improve the paper based on the provided suggestions.
Decision: Reject