Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a framework to learn a domain-invariant representation of time series. Using the idea of domain generalization, this paper removes the influence of varying domains/distributions across patients/datasets through pseudo domain classification. Although this kind of model has been rather studied in images, but not a lot in time series. 
main_review: Strengths:
- Good representation.

- The motivation and model design makes sense to me. 

- Extensive experiments. This work used 6 datasets from 3 different scenarios, worked on 4 different challenging tasks. I also appreciate the visualizations. 


Weaknesses:

No obvious weakness to me. Here are some suggestions for further revision:

- This is an important point. Section 3.3, regarding cross-person generalization setting: I read the descriptions in the first set and noticed the footnote 2, but how to split 14  (USC-HAD) and 9 (PAMAP2) subjects into 4 groups? So I guess you are not splitting the subjects but splitting the segments, right? In detail, you shuffled all segments from all subjects, and split them into 4 groups (each group contains segments from all subjects). 

I also understand that the term of 'cross-person' is used in some papers (especially in EEG related publications) referring to shuffling all segments from all subjects. However, in their case, 'cross-person' is in contrast to 'person dependent' (use the segments from the same subject for both training and test) and 'person independent' (use several subjects for training and others for test). 

In your case, it's not good to call it 'cross-person generalization'. Because, in the context of 'cross-position' and 'cross-dataset' where the training set and test set are from different positions/datasets, so readers suppose 'cross-person' means the training set and test set are from different subjects. Thus, my suggestion is to change the setting from splitting segments to splitting subjects. For example, in PAMAP2, you can use 7 subjects for training and 2 residuals for test (it's kind of like setting 4 but not the same). At least, please provide the results of splitting subjects over one dataset. 

- BTW, I didn't find the results for setting one-person-to-another (not in Table 2 nor Table 3), am I missed something? 

- Please enlarge panel c of Fig 1, the point shapes are hardly readable.

- This paper takes univariate time series as an example, please discuss how to deal with multivariate time series? Specifically, are the signals from different sensors (in the same segment) belong to the same domain label? If not, how to solve the problem?

- Please add more descriptions and details of domain-invariant representation learning (bottom in Page 4 or add to appendix). It's not clear to me that why the GRL learned representation is invariant to domains?

- Please make Eq 1 an in-line equation. It's not important for the whole paper.

- Please clarify, are the h_f, h_b, h_c are all functions? Implemented by neural networks? Please let me know if I missed some related information in your experimental setup.
summary_of_the_review: This paper focuses on one meaningful topic, targets the challenge, and proposed an interesting solution to solve it. The presentation is easy to follow, the model is good (maybe the pseudo self-classification is not very novel, but still ok to me), and the experiments are sound to me. So I suggest accepting it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a time series classification method with loss function that simultaneously promotes diverse distribution among different sub-domains within each domain and domain invariant feature representation within the same class. Here the domain is considered in a rather granular manner such as different person may be considered as different domains in gesture recognition application. An iterative process is proposed to learn a classification model. Both theoratical analysis and empirical evaluation on three different applications are provided. The proposed method shows better accuracy especially on generalizing across different domains than other competing methods.
main_review: ### Strength
1. This paper extended the domain generalization on time series classification problem by modeling the distribution of sub-domains within each domain and demonstrate its benefit in learning better representation.
2. The proposed approach shows superior performance on three different benchmarks compared to state-of-the-art domain generalization methods. Multiple generalization settings are provided in human activity recognition task and the improvement is consistent across different settings.


### Weakness
1. The overall technical novelty is marginal. First, the pseudo-labeling strategy is adopted directly from (Liang et al. 2020). Secondly, the adversarial training strategy is adopted from (Ganin et al., 2016). Finally, the determination of number of sub-domains is based on simple heuristics i.e. cross-validation on different K values, which is not really a new concept compared to decoding time series using a state-space model such as hidden Markov model. The authors need to emphasize their contributions on existing development of domain generalization.
2. The theoretical insight is based on existing conclusion from (Sicilia et al., 2021). It does not add to the novel contribution of the paper other than providing an interpretation of loss function.
3. The discussion on classification model is not sufficient. Is the same model architecture as described in Appendix B used for all datasets and experiments? What is the impact of using more complex or simpler architectures? The ablation study can be strengthened by comparing different choices of model artitechtures. 


### Additional Comments
1. It would be good to elaborate more implementation details such as how data is pre-processed, how convolution is performed for multi-variate time series, what is the kernel size, and what is the criteria for convergence in repeating step 2-4. The discussion on model architecture in Appendix B should be placed in main paper instead.
2. Figure 6(d) shows even when K=2, the accuracy is much higher than ERM. I'm wondering what is the performance when K=1, which can be considered as degenerated baseline of the proposed approach. This can provide additional evidence on how effective is characterizing sub-domains and promote diversity.
summary_of_the_review: Overall, the proposed approach showed quite promising results on several different time series classification applications. The problem formulation and techniques are mostly adopted from existing work, which reduce the technical novelty. The authors need better justification on the performance improvement achieved by proposed approach, and highlight their contributions in domain generalization methodology. Please refer to the main review for more suggestions on improvement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on time-series classification problems, such as gesture or speech command recognition, where the data for any given sequence contains information from one user, but for which a priori you don't know anything about that user. Their goal is to be able to train on data with one set of users ("domains") and evaluate on a different set of users ("domains"). The users (at train and test) are unknown and are modeled by discrete latent variables. 

Contributions include an algorithm ("DIVERSIFY") for learning distributions of domains/users. Their approach ultimately results in a model for time-series classification and for latent segmentation of data into discrete domains (aka user profiles). The model is framed using a min-max adversarial formulation. 

The paper documents results on three datasets (EMG signals, Speech Commands, and Human Activities) show improvements over other distributional algorithms.  
main_review: This is an interesting paper and potentially the start to something really interesting. There are a number of questions/concerns I have as described below. Sorry if this comes off as overly negative. I enjoyed reading the paper and thought this is interesting work. 

Contributions/Arguments/Writing:
* There is a lot of discussion on stationarity early in the paper, with claims seeming to indicate that all time-series signals are non-stationary and other modalities, like images, are stationary. Strictly speaking, non-stationarity is a problem with practically all real-world classification problems, including images and other areas that "distributional" and "domain" work has been done. Perhaps the argument here should be around user-specific effects (e.g., the way one person wears the Myo EMG band may make gestures look different than another person? Or the same person wearing the device on two different days may require different calibration?).
* One thing I didn't understand was why "viewing time-series classification from a distributional perspective," at least using the methods in this paper, is important. Perhaps the authors can address this? Based on my cursory understanding of related work like GroupDRO, their goal is to minimize worst-case errors on specific distributions of data. You could imagine uses in ML fairness where you want the worst case performance for minority classes to be sufficiently good. However, the distributions used in this paper are latent and seemingly don't have semantic meaning. 
* Given the distributional slant, are the metrics described the this paper the right ones for the task? (caveat: as noted below, I don't believe the metrics are named). Results are provided holistically for all distributions whereas perhaps we should be more curious to see results on specific distributions. Based on the theory it seems like worst case performance should be improved. 
* The claims around what is or isn't a domain were not consistent between modalities. In the image modality (Fig 1a) a sketch is one domain whereas a cartoon is another. However, in Fig 1b it seems that two subsequences within one series are considered separate domains? 

Models / algorithms / experiments
* The models used (as described in the appendix) are very small, especially for the Speech Commands dataset. It would be interesting to see state of the art (or at least recent) architectures for each problem (e.g., MatchBox or similar on Google Speech Commands) and to apply the paper's adversarial algorithm to those models. Based on a quick search it seems that this [1] is state of the art on this dataset achieves 97.41% compared to around 95% in this paper. 
* Metrics: It would be useful to describe the metrics being used for each experiment. I can try to infer (I assume it's mostly per-clip accuracy?) but I don't know for sure. More broadly it's unclear how metrics are compute computed, whether or not they're the "right" metrics for each dataset, or if they're the right metrics for evaluating distributional approaches like the one described in the paper. 
* It's hard to understand how substantial performance improvements are, especially on the EMG and Human Activity datasets. I could only find one other recent paper using the EMG dataset and performance was in the 90%s [2]. Assumably these are not apples-to-apples comparisons with the results in this paper... but having some external validation would be appreciated (especially on Speech Commands).
* Re: Ablation study. I don't know what dataset these experiments are being performed on. Is there a reason the performance decreases with more domains? I was surprised by the visualizations in Fig 8 and Fig 9. These seem to be contrary to what is described in the rest of the paper: it seems that the domains actually don't do a good job of capturing latent structure? Perhaps I'm reading these incorrectly. 
* The EMG and HAR datasets are very small. I suppose this is philosophical, but would it be more impactful to develop algorithms that better fit this data or to collect 10x more data from each set of sensors? It's hard to know how well performance generalizes on datasets like this. 

[1] Mordido et al. "Compressing 1D Time-Channel Separable Convolutions using Sparse Random Ternary Matrices." May 2021.
[2] Rubio et al. "Identification of hand movements from electromyographic signals using Machine Learning" 2020.


summary_of_the_review: I could be swayed to change my ratings fro "correctness" and "empirical novelty" but in general there are too many unanswered questions for me to vote to accept this paper.
review_title: null

Meta-review: This paper has been reviewed by three reviewers, two scoring it borderline leaning towards an accept, and one scoring it as accept. The key criticism from reviewers is the lack of novelty (*the technical novelty is limited due to the adoption of existing methods without substantial changes (pseudolabeling for latent distribution, adversarial learning for domain generalization)*) and limited technical analysis (*Theoretical analysis is weak due to the use of existing conclusion from Sicilia et al., 2021*). On the other hand, authors argue that the problem they address is new (*learn the generalized representation for time series data, which is a new and challenging problem*).

As it stands, AC sees this paper as borderline leaning towards reject for the above reasons even though the evaluations are interesting.
Decision: Reject