Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**
This paper presents a general Self-supervised Time Series representation learning framework. It explores the inter-sample relation reasoning and intra-temporal relation reasoning of time series to capture the underlying structure pattern of the unlabeled time series data.  The proposed method achieves new state-of-the-art results and outperforms existing methods by a significant margin on multiple real-world time-series datasets for the classification tasks.

**Contributions**
1. The paper is well written and easy to follow. The organization is good. 
2. The architecture is well motivated. It is reasonable to use unsupervised temporal relations to learn video features. 
3. The qualitative results are numerous, insightful, and convincing on multiple datasets. The authors conduct extensive experiments to demonstrate the effectiveness of the proposed method, including inter-sample relation reasoning and intra-temporal relation reasoning.

**Details**
1. The novelty of the proposed method.
The Inter-sample relation reasoning is very similar to SimCLR, which also maximizes agreement between different views of augmentation from the same sample via a contrastive loss.  Considering this, the novelty is relatively incremental.

2. Additional video recognition experiments.
The used dataset is small-scale that makes the task simple. I would have wanted to see results on large-scale classification tasks, such as video action classification.  The performance of action classification is closely related to the temporal feature modeling. So the effects on this task can make the proposed method more convincing. 

**Conclusion**
overall, this paper proposes an interesting architecture for self-supervised temporal modeling. But the novelty is relatively limited compared to the recent SimCLR work. And it requires additional experiments on harder video classification task and datasets to show the effects and robustness of the proposed method.

**After rebuttal**

Thanks for the detailed response.
This paper can be seen as an interesting attempt to use self-supervised on time series data. 
Although the basic idea is similar to SimCLR, It is still interesting work considering the  computation complexity and new loss function.
So I update my score to 6.


review_title: Interesting architecture for self-supervised temporal representation learning, but the novelty is limited  compared to contrastive learning.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes an approach for self-supervised time series representation learning by using inter-sample and intra-temporal relational reasoning. The paper builds upon the existing ideas on relational reasoning [3] and self-supervised learning to train models from unlabeled data.
Self-supervised learning for time series is still under-explored and this paper attempts to bridge this gap in time series literature.

The key novelty of the paper as claimed by the authors is in the design of inter-sample and intra-temporal tasks and loss functions to learn a feature extractor from unlabeled data. 
This idea of designing pretext tasks using global-sample structure and local-temporal structure and adapting it for time series is interesting. However, the inter-sample loss function seems to be a straightforward application of ideas in self-supervised learning literature that rely on various augmentations to create positive and negative pairs of samples. The intra-temporal task is defined where the distance between subsequences (referred to as time pieces in the paper) of a time series is used to create a classification task. This is a potentially novel (albeit incremental) aspect of the approach but the empirical evaluations (as detailed below) fail to highlight the impact of the same on performance clearly.
Furthermore, the idea of using subsequences to define self-supervised (intra-temporal) tasks for time series has been explored earlier, e.g. in [1,2]. The authors seem to be unaware of papers like [1,2], and Introduction and Related Work sections suggest that self-supervised learning for time series has not been attempted earlier.

Apart from lack of clarity on novelty and contribution of the work, I have following concerns regarding empirical evaluation: 
1. The ablation studies show the effect of number of classes and the length of subsequences (pieces) on linear evaluation accuracy (Linear ACC) for the downstream classification tasks. However, the sensitivity of results to the hyperparameters of intra-temporal task is reported only on CricketX. Do the same results hold on other datasets? Also, the results on downstream task seem to be very sensitive to the choice of parameters C and L/T. The authors state that "In the experiment, to select a moderately difficult pretext task for different datasets, we set {class number (C), piece size (L/T )} as {3, 0.2} for CricketX, {4, 0.2} for UWaveGestureLibraryAll, {5, 0.35} for DodgerLoopDay, {6, 0.4} for InsectWingbeatSound, {4, 0.2} for MFPT, and {4, 0.2} for XJTU" - It is not clear how these choices for hyperparameters are arrived at, or what "we set"  and "moderately difficult pretext task" mean. Also, the ablation study is more of a sensitivity analysis of the choice of values for C and L/T. It is not clear what happens if intra-temporal relation reasoning task or the inter-sample relation reasoning task is removed from SelfTime. Does that make SelfTime same as one of the other baselines, e.g. "Relation"? Since most of the baselines used are adaptations from image domain, it is difficult to gauge where the proposed approach stands w.r.t. time-series specific methods like [1,2].
2. For CricketX, DLD, and XJTU datasets, SelfTime performs significantly better than the supervised learning model (Table 2). How does one explain this observation as supervised methods would typically perform better or at least as good as the self-supervised methods?
3. The authors observe that "we find that the composition from a magnitude-based transformation (e.g. scaling, magnitude warping) and a time-based transformation (e.g. time warping, window slicing) facilitates the model to learn more useful representations." As per the description, it seems that this observation is based on analysis of just one dataset (CricketX) and on one algorithm (SelfTime). I think a more thorough description and evaluation across datasets and baselines could be useful.
4. In the ablation results, it would help to see results across all datasets when using only inter-sample or intra-temporal losses. Also, the results in Fig. 5 and Fig. 6 are for only one dataset, and that too different ones: CricketX for Fig. 5 and UGLA for Fig. 6. Do the observations from Fig. 5 and 6 hold across datasets? Similarly, what motivates the particular pairing of source-->target for the Domain Transfer Evaluation? Given six datasets, several other combinations are possible - does this observation hold across all such combinations?
5. The authors loosely mention that "more augmentation results in better performance" on the basis of references from existing literature. Though this might hold in other domains, an evaluation of the same for this setting would be useful to claim the same in the given context. 

Given the above, a more thorough and consistent description of the contribution in light of recent related work and empirical evaluation is needed. Given works like [1,2,3], the contribution of the paper is limited to defining the intra-temporal task, but the same has not been evaluated carefully enough.

The write-up can be improved at several places, e.g.:
Eqns. 1 and 2 need a minus sign to be called as losses? 
Is i=j valid for the positive inter-sample relation pair? If not, Eqn. 1 might need an update.
fire --> fair?
$z_n$ is bold at a few places and not at other
limited amount training samples 
Or for video data
there is not enough feature
we generate from its transformation counterpart and another individual sample as the positive and negative samples respectively
In the experiment, we achieve new state-of-the-art results, and outperforms existing methods by a significant margin on multiple real-world time series datasets for classification task
a intra-temporal
Firstly, takes the original time series signals and their sampled time pieces as the inputs
show that both two kinds of relation reasoning
will drop the evaluation performance
t-SNE visualization of the learnt feature.

References:
[1] Unsupervised scalable representation learning for multivariate time series. Franceschi et. al. NeurIPS'19
[2] Multi-task Self-Supervised Learning for Human Activity Detection. Saeed et. al. PACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2019.
[3] Self-supervised relational reasoning for representation learning. Patacchiola and Storkey. NeurIPS'20

** updating the score to 5 **
review_title: Official Blind Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: ===========Update after rebuttal==================

I will remain my score of weak rejection.


=============================================================

This paper presents to model both inter-sample and intra-temporal relations. The idea is naive but is easy to follow and reasonable. Experimental results support the effectiveness of the method.

Strengths: The idea is reasonable and the application is important. 

Weakness:
1. The notations are messy. The author should consider clean up some unused notations for better readability. Especially for section 3.1, I spend quite a few time thoroughly digest the notations. The explanation in section 3.2 is even scarier. I can easily understand the concept from the intro and Figure 1, yet I spend much more time reading sections 3.1 and 3.2.
2. The presentation of the experimental section is also hard to follow. Specifically, what is the message conveyed in Figure 5? I would suggest the author highlight the region that we can pay attention to. Nonetheless, why only considering the composition of two augmentations? Instead of providing a matrix that considers all the combinations between two different augmentations, why not providing a table summarizing the sets of different combinations of augmentations? The sets of combinations can be the sets that the author likes us to focus on. 
3. An important paper [1] is missing. The author should discuss this missing reference.

Overall, I am leaning positive toward this paper, yet I feel the presentation can be greatly improved. I am giving the score 5 for now, and I may update the score after the rebuttal.

[1] Temporal Relational Reasoning in Videos, Zhou et al., ECCV 2018.
review_title: Review

Meta-review: This paper presents a general self-supervised time series representation learning framework. The organization is good, and the architecture is well motivated. However, the paper has limited novelty, and is a straightforward application of ideas in self-supervised learning literature.

Experimental results are not entirely convincing. The used dataset is small-scale that makes the task simple. A more thorough comparison with recent related work is needed. The presentation is also sometimes hard to follow.
Decision: Reject