Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, a method for time series self training is being proposed. The model is named CrossMatch and it utilizes context-additive augmentation, in other words, it adds a context instance to the left and the right part of the time-series to generate two augmented views with different contexts. Then, the authors design a reliability function for more reliable pseudo-labels, and they mix the pseudo-labels into a single cross-window label, which will be matched against two softmax probabilities from both of augmentations. The CrossMatch model is test against three other state of the art models (FixMatch, FlexMatcha and PropReg) in three publicly available datasets (HAPT, mHealth and Opportunity). The results show that in most cases the proposed model outperforms the rest in segmented Accuracy and segmented F1 Score. In the experimental setup, the authors also show results for the inter- and inner-instance augmentations analysis and an analysis of varying context length for the augmented instances.
strength_and_weaknesses: Strengths:
- The paper is well-written and easy to follow. The authors have done a good job in the writing and the structure of the paper.
- The problem is interesting to the research community.
- The authors have cover the related works and show what are the current challenges and how the proposed method will help on those and increase performance.
- The authors have conducted an extensive experimental setup and the results are very promising as they show that in most cases the proposed model outperforms the comparison methods.
- The authors have included analyses about the augmentations types that have been selected and the varying context length for the augmented instances.
- The authors will release the code.

Weaknesses:
- The novelty of the paper is incremental as the authors rely on existing methodology, however they do add extra analyses and they do have their own contributions.
- It would be interesting to discuss examples of the additive augmentations that have been successful from the datasets in the experimental setup.
- The whole idea of either left or a right augmentation needs more motivation. Why do the authors propose only one of the two? Because in this concept the time is changing, how do only left (previous TS segments) perform? How does the performance get affected by only right additive augmentations? What does that mean for the TS segments on the right (future)? Have the authors experiment with combinations of the augmentations?
- The idea of the notation and equations etc to be calculated based on the half and the middle timestamp of the time-series segment is a bit confusing, wouldn’t it be easier to follow if the paper was written with 1/2*{length of X’} for example?
- In the definitions of the evaluation metrics, what is the relation of the segmental accuracy and F1-Score to Jaccard similarity? This particular paragraph needs a better writing, as it is not clear how the segmental scores are being calculated, which is important for the understanding of the results.
- In Table 3, TS Accuracy in FixMatch, FlexMatch and PropReg should all be in bold, as they are all the same.
- The results in the tables of the experimental setup, have small differences between the proposed and the comparison methods. Have the authors checked for statistical significance in those results?
- It is not clear what the iteration axis refers to in Figure 5.
clarity,_quality,_novelty_and_reproducibility: Ideas and suggestions on how to improve the clarity quality and the presentation of this work are mentioned above in the weaknesses. The code and the data will be available online for reproducibility. The novelty of the paper is incremental as discusses above.
summary_of_the_review: Overall, I enjoyed reading this paper. It is easy to follow, well-written and well-structured. The authors do a good job providing the motivation, the challenges and the related works. A few suggestions for improvements can be found in this review. The results show that the proposed model outperforms the rest of the comparison methods in three real world datasets..
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Edit: I would like to thank the authors for explaining their method and updating the scripts. Although they resolved several of my concerns,  I think the proposed method of using context is not very new, but using left and right context has some novelty, but not very high. 

As the authors said, the labels are spreading evenly with one class per segment. In addition, the label length is relatively long without switching often. F1 score might not be that sensitive. The problem setting is relatively ideal. I am not sure whether we need this method as maybe some simpler solution would work in this case. 

Thus, my score will remain the same. 


---------------------------------------------------------------------------------------------------------------------------------------------------------
Original review: 

This paper proposed a new framework for self-training for generating pseudo labels for a long time series. The proposed framework propose some training strategy on the few label cases and then use post-processing by utilizing the left and the right context to reweight the labels. 


strength_and_weaknesses: Edit: 
Using a segment for context in time series is not really new but contrasting 


Strength: 
1. The problem is very important since time series are often lacking of labels. 
2. The performance is better than the state-of-the-art. 

Weakness: 
1. unclear problem setting
2. unclear experiment setting
3. the proposed method is very heuristic and not clean. 
see below in detailed comments. 
clarity,_quality,_novelty_and_reproducibility: The paper works on an important problem. The technique is okay, but on the heuristic side. But in general it has several unclear parts need to be explained: 

1.  What is the target instance for? Is it the instance you would like label, or these are already labeled? 
2.  What is the input and output for $f_{theta}$? Please include the used network structure and how many layers you used? 
3.  Can you please give an example of your input and output for the overall framework? For example, what is figure 1, what is the time series after your mask? What is the measure for "copy"?  How many labels per instances? Can you give the location and distribution (maybe some heatmap to visualize the original label and the label generated after equation 2, as well as the label after "reliability weighting")? It is really confusing about the problem setting. 

Algorithm 1: 
For algorithm 1: should line 10 be $Y_u$? 


Experiment: 
1.  Why table 3 needs to be F1@25 score "over the last 20 iterations"? Why don't we compare with the last iteration directly since there is a proper stop condition? 
2. Table 5, it seems like precision was sacrificed while recall is increasing, any comments for it?  Also, the precision and recall do not overlap with the peak with F1@25?  
3. How does a fixed c vs a random c value affect the performance? 
4. CrossMatch you set tau=0.95, but your pseudo-label per class is trained to 0.99, any reason for that? Is it contributing the performance? 
5. Table 4, did you use jitters and scaled jitters when you test the performance with IA or CA? 
6.  Also, for the experiment setting, are you concatinating all the instance into a single time series, then perform the pseodo labeling for every dataset? Or you were just doing rolling based on a single instance? I am just confused about the setting. 

summary_of_the_review: I will consider raising my evaluation if my concerns are addressed by the authors. I think the problem setting, the overall model, and the experiment setting needs more explanation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel data augmentation method called context-additive augmentation. This method is easily-implemented, and preserves a target instance without perturbing it. Based on this method, the authors additionally propose a cross-window self-training framework. This framework employs reliability-based cross-window labels to improve consistency among augmented instances. The experiment results show the good performance of the proposed method. 
strength_and_weaknesses: Strength: 
1. The idea of reliability-weighted mixing is interesting. This can help readers design other soft-mixing methods.
2. The results of the experiments show that the proposed method achieves SOTA performance.

Weakness:
1. One might don't agree with the rationale in Section 3.3 and the curve r(p) should be. In my opinion, the reliability value of the middle point of the augmented instance is maximum is not proper. In my opinion, the overlap ( m-o to m+o) should have the largest reliability. If the rationale you proposed holds, assuming that c < 2o, the m - c/2 -th position on X_left will have much larger reliability than the same position but on X_right. Please explain the intuition of your rationale.
2. It is confused with the augmentation method you proposed. Assuming that the target instance is X[m-o, m+o], the generated instances are X[m-o-c, m+o] and X[m-o, m+o+c]. In my option, this is equal to the case that the target is X[m-o-c, m+o+c] and cut the target into X[m-o-c, m+o] and X[m-o, m+o+c]. So I think the novelty is limited.
3. The statements in Section 3 sometimes are not clear enough. The details can be referred to as the minor weaknesses below. Please modify the statements in order to make the paper easily understood by the readers.
 
Minors:
1. In Section 3.2, the authors say that the heuristic "not only maximizes the difference between the two augmented instances but also minimizes the overlap between them". However, I am not sure what is the difference between "maximizing the difference" and "minimizing the overlap". I think the authors just expressed the same meaning in two ways. If there is some difference, please explain it in detail.
2. The purpose of Figure 5 is unclear. Since the accuracy has been shown in the tables, and it is difficult to see that CrossMatch converges most quickly. Even if it converges quickly, the speed of convergence is likely related to the learning rate.
3. Section 4.4 only shows the performance under different "c"s. More discussions are recommended.
clarity,_quality,_novelty_and_reproducibility: Good
summary_of_the_review: This paper proposes a novel data augmentation method for self-supervised learning. The experiment results show the good performance of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the concept of context additive augmentation for self-supervised training for sparsely labelled time series. The idea is to propagate the labels on surrounding timesteps, and then used the pseudo labels for self-training. This work explores multiple real-world datasets to compare performance against other state-of-the-art approaches.
strength_and_weaknesses: The paper explores an important problem as time series often present sparse labelling.

However, the clarity of the paper weakens its impact. The introduction contains all necessary elements but would benefit from clarifications and examples to give the intuition behind the proposed heuristic. "Target instance", "by adding surrounding sequences", and "weak and strong augementaions" are terms that need to be more clearly introduced. The specific problem should also be introduced earlier, the paper seems to focus on temporal prediction for which each time step needs to be classified. At the end of the introduction, it is still not clear what challenges the proposed method addresses and what the motivation for this approach is.
clarity,_quality,_novelty_and_reproducibility: The paper would benefit from clarification throughout to make the model more intuitive.

The work proposes an interesting idea that would benefit from further comparisons with label propagation methods and clarification of the experimental settings. For instance, the studied datasets present multiple individuals, but it is said that "all included time series are concatenated in chronological order to form a single continuous time series", what does it mean in this context?
It would also be beneficial to display the number of pseudo labels used for each method (if possible having a fixed number across methodologies to compare the quality of the data added, which may be confounded by the number of points added). Formalisations of the assumptions made on the time series and the label missingness would be key to understanding the limitations of the proposed methodology.

The code is provided for reproducibility.


summary_of_the_review: The paper proposes an interesting heuristic that could be strengthened by clarifications.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This interesting paper has been assessed by four knowledgeable reviewers two of whom rated it as marginal reject and two as marginal accept. The key complaints include incremental novelty some issues with clarity and fairness of comparison versus relevant alternatives. The authors have actively engaged in extensive discussions with some reviewers during the rebuttal period, but these discussions and provided clarifications did not lead to improved ratings. This work in its current form needs improvements, it is not too far from being publishable, so the authors should be encouraged to push forward, but at its current state it does not meet the acceptance criteria for ICLR.
justification_for_why_not_higher_score: Incremental contribution, limited comparison to the relevant alternatives.
justification_for_why_not_lower_score: n/a
Decision: Reject