Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes the use of many auxiliary tasks to boost the performance on a target task by means of `'self-supervision'. Specifically, they considered auto-encoding, forecasting, partial-segment auto-encoding, and piecewise-linear auto-encoding. 

There are major concerns that should be clarified or described in detail.
1) The overall architecture is not complete. the architectures used in the experiments are not described concretely.
2) To this reviewer, the idea of self-supervision is similar to the unsupervised learning for representation learning. 
3) The methods of BERT (Bidirectional Encoder Representations from Transformers) [Devlin et al., 2018] or BRITS (Bidirectional Recurrent Imputation for Time Series) [Cao et al., 2018], although different for their target tasks in their original work, could be also regarded as self-supervision technique and could be interesting to compare with them.
4) The experimental settings are not described well, thus lack of reproducibility
5) It is unclear which aux-tasks were applied in Fig. 2. Further to better understand and analyze the results, it is required to conduct more rigorous ablation studies.
6) There is no comparison with recent work on the same datasets.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper propose an approach for self-supervised learning on time series.
Three datasets are considered (simulation and 2 healthcare datasets).
The gist of the contribution is to both optimize prediction loss
of the true task and at the same time do a good job for a family
of auxiliary tasks. 4 auxiliary tasks are considered. While
the first 3 auxiliary tasks are quite common, the 4th tasks
called piecewise-linear autoencoding appears novel. The idea
is that the hidden representation of the LSTM should be a good predictor
of the past using a piecewise-linear approximation.
The author coin the term "limited self-supervision" for their approach
although it's not clear why it is fundamentally not just self-supervised
learning as it has been proposed in the past.

The paper is overall well written and addresses the relevant issue
of learning from limited annotated data.

Major concerns

- It is yet another way to do self-supervised learning on time series
and no clear benchmark with alternatives is provided (time contrastive
learning (TCL) or Contrastive Predictive Coding (CPC) https://arxiv.org/pdf/1807.03748.pdf
etc.)

- On any of the applied problem it is not clear if the proposed
approach brings an improvement on the state-of-the-art or if it's
just an illustration of the method disconnected from the literature
of the application.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper proposes a so called self-supervised method for learning from time series data in healthcare setting. Specifically, here self-supervision is achieved via designing auxiliary tasks based on data's internal structure to create more labeled auxiliary training tasks.

From both perspectives of methods and applications, the proposed model has very limited novelty. It is just one application of multitask learning. Also very similar idea has been implemented by [1]. In [1], the authors learn multi-level embedding to make disease/risk prediction, where the embedding was jointly trained by performing auxiliary prediction tasks that rely on this inherent EHR structure. The authors need to state what is the novelty of the proposed method compared with [1].

In addition, the performance evaluation missed many baselines. Table 1 seems more like a ablation study rather than a performance comparison. You need to compare with all state-of-the-art models in computational phenotyping in order to show the performance advantage brought by the proposed mode design.

[1] Edward Choi, Cao Xiao, Walter Stewart, Jimeng Sun, MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare,  NeuRIPS, 2018

 
review_title: Official Blind Review #2

Meta-review: The paper addresses  an important problem of self-supervised learning in the context of time-series classification. However, all reviewers raised major concerns regarding the novelty of the approach and the quality of empirical evaluation, including insufficient comparison with the state-of-art and reproducibility issues. The reviewers agree that the paper, in its current state, does not path the ICLR acceptance threshold, and encourage the authors to improve the paper based on the provided suggestions.
Decision: Reject