Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces GroupGAN a method for generating multivariate time series data that share a single source or cause. GroupGAN uses a separate generator and discriminator for generating each data channel and a central discriminator for forcing the model to keep the correlation between different channels the same as the real data. Synthetic data as well as EEG eye state dataset is used to evaluate GroupGAN compared to other models in this area.
strengths_and_weaknesses: Strengths:

I believe the generation of multi-variate time-series data is a significant and very challenging problem. As mentioned by the authors it is urgently needed in many fields. The idea of using a central discriminator and separate generator discriminator pairs for each channel is very interesting and the authors show through experiments that this idea improves the data generation and downstream tasks.

Weaknesses:
- In the related works section it would be good to mention the weaknesses of the existing methods that the current model is trying to fix
- I think some additional comparisons would've strengthened the claim of the paper that GroupGAN works better than the baseline model (a single multi-channel GAN model for generating the data) for data generation (on lines 116 - 118), for example in tables 1 and 2 I expected to see a comparison with the baseline model.
- On line 155 the authors used LLD for the first time and then define them later it is better to define a term before using it.
- The paper can be improved by changes in language and writing style. There are typos and grammar problems that if fixed can improve the clarity of the paper. I bring some examples below:
- - on line 13 it should be "performs very well in downstream tasks"
- - on line 95 "channels" should be "channel"
- - on line 223 "EGG" should be "EEG"
-- on line 292 "compare" should be "comparison"
questions: - On lines 151-154 you mention that for central discriminator MLP networks work better than LSTM networks because "if the central discriminator is too powerful, the results will be of lower quality, as it will strive to make the signals more correlated at the expense of realistic individual TS." can't this be controlled with the gamma weight?
- Is the "All-synthetic experiment" described on lines 243-255 the same as the "Train-on-fake, Test-on-real" experiment mentioned in table 3?
- In Figure 4, I wonder why GroupGAN does so much better than baseline on the All-Synthetic experiment while on the Augmentation experiment it is not doing as well? Is it because of a lower diversity of samples generated compared to the baseline?
- In Figure 4c, does augmenting more synthetic data reduce the performance of GroupGAN with respect to the baseline model, or does it improve it? It is not obvious from this figure.
- What is your hypothesis on why GroupGAN does not do as well on 2 channel datasets?
limitations: The limitations are discussed adequately.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose Group GANs to generate Multivariate Time Series (MTS) data. MTS can be used for augmentation and privacy protection. To generate realistic MTS, the model should (1) generate realistic single-channel data (2) preserve the correlation between channels. So this paper uses a paired generator and discriminator to generate single-channel data and uses a Central Discriminator to force correlation preservation. They verify the results on synthetic and EEG datasets.
strengths_and_weaknesses: ## Strengths:
* The paper is well structured and nicely written. The authors provide a clear motivation for why generating MTS is useful and explain the key challenges.
* The authors tear down the problem of generating MTS into (1) generating realistic single-channel TS (2) preserving correlation. The authors propose Group GANs (many channel GANS + Central Discriminator) to address these problems. The method is straightforward and intuitive. I believe the proposed method will provide insight and inspire people working on related problems.
It’s good to see the authors submitted detailed supplementary materials.
* The authors use many metrics and methods (eg, Feature-based Analysis) to show how Diversity and Correlation are preserved. They also discussed the implementation details and trade-offs of their model, which can be valuable for the community.

## Weaknesses:
* I think the algorithm and training details are very important for understanding the paper. Maybe you can consider moving them (mainly Algorithm 1) from the Supplementary Materials to the body part.
* Some questions regarding the objectives remain. See Questions.
* The datasets are small, and they don’t look very practical. (As the authors state “our method could be extended to more practical use cases” in line 312).
* Whether the proposed model can scale very well remains a question. Performance limitations can be an issue. Another issue is that the authors only test MLP and LSTM, which is not a full guarantee that they can generalize to complex models like Transformers.
questions: Q1:

Eq (4) is the objective for “a given channel” i. What is the objective for all channels (for i generators + i discriminators + one central discriminator)?

Q2: 

In eq (4), when updating channel i, does the Central Discriminator treat the generated samples in  other channels (j != i) as real ones? It seems the case because in line 100 you say “the goal is to estimate the conditional distribution”.

Q3:

What is the output of the central discriminator? I still don’t get it after reading Algorithm 1. The central discriminator takes in data of all the channels, but does it output a single score for all the channels combined or does it output a score for every channel (i scores in total)? And how do you update the central discriminator exactly (Describe the loss)?
limitations: More useful datasets and advanced model architectures can be used.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors present GroupGAN for synthesis of multivariate time-series data.  In contrast to previous approaches where features are jointly learned, GroupGAN trains a separate GAN while also training a central discriminator which classifies feature vectors as either real or fake.  The intuition behind the latter is to preserve correlations among the individually trained generators.  Synthetically trained data are evaluated for low-dimensional synthetic time-series data and supervised EEG data.
strengths_and_weaknesses: Strengths: The proposed architecture is novel and interesting.  The central disciminator, which enforces preservation of correlations among independent GANs, is intuitive and could be helpful regularizing different GAN instances to promote consistency/correlations.  While the authors claim: "The main reason for having channel GANs as opposed to a single multichannel generator discriminator pair is to make each generator powerful in its own channel distribution, and by including the central discriminator, we enforce realistic correlations between the
channels as much as possible."
It seems like another major advantage of this approach is tractability; jointly modeling all TS at once may lead to a computational explosion, but the current setup allows the computationally efficient benefits of training individual generators while also ensuring correlations are learned among the individual TS through the central discriminator.

Weaknesses: The writing and description of the model could be much improved to align with time-series literature.  In particular, the description and focus on channels both confuses and limits the paper to vision data.  Channels typically refers to vision data, but moving the model description and text to describing time signals as comprised of features broadens the work and its applicability (i.e., a large number of time series work are not vision data).  Furthermore, the description in terms of channels breaks down for different applications (i.e., MFCCs in speech recognition).  The clearest/most general description in time-series analysis is a (possibly multirate as well as multivariate) process is composed of various features collected at different times.  Furthermore, there seems to be a general misconeption on what a time series is in the paper; the EGG eye state data is a single time series, each time-instance of which consists of 14 features (or measurements) and a binary label.

Evaluation of the proposed method requires significant improvement.  The synthetic data is extremely low-dimensional (only 2 dimensions) and resulting hyperparameter tuning, performance, an(d (particularly) catch22 results on this data are thus unconvincing.  Furthermore, the real data is preprocessed in an extremely ad-hoc way (lines 229-234) without any explanation.  E.g., the EEG data is supervised, why are labels re-estimated during preprocessing?  How many outliers were present in the data, and what observed effects on GroupGAN and competitors warranted removal?

Necessary details are also lacking from the paper.  In particular: "The baseline method is an LSTM-based GAN that generated all of the
channels simultaneously" <- this competitor requires much more explanation.  What were the exact details of the architecture and learning schedule?  Why were they chosen?  Why were existing methods (i.e., TimeGAN)  not compared to?
questions: The following is a major contribution of the paper: "this is the first study to analyse how to generate multivariate time series
with individual channel generation originating from a common noise while inter-channel
correlation preservation is forced with a central discriminator." <- How realistic is this scenario?  E.g., for the listed examples "the biometric values from a medical patient, the stock prices from economic events or geographically separated seismic measurements from a single earthquake," do these assumptions hold?  E.g., while biometric values are collected from a single medical patient, different signals may have different noise properties (e.g., ECG, blood pressure, etc.).

We hypothesize that this will penalize unrealistic (un)correlation patterns between channels."  <- Can you supply evidence to support
 this, or point out where in the text evidence will be demonstrated?

Suggestions:
" it is well known that sharing data associated with 26 single individuals, even anonymized, can lead to unexpected privacy breaches" <- Well known ML examples:
          -Narayanan, Arvind, and Vitaly Shmatikov. "Robust
          de-anonymization of large sparse datasets." 2008 IEEE
          Symposium on Security and Privacy (sp 2008). IEEE, 2008.

          -Gong, Maoguo, et al. "A survey on differentially private
          machine learning." IEEE computational intelligence magazine
          15.2 (2020): 49-64.
limitations: A discussion of the tradeoff between learning a single multivariate GAN vs multiple GANs would help strengthen the paper.  While the authors have mentioned learning a large number of individual GANs can be prohibitive for high-dimensional data, in situations where jointly learning the distribution of all features is intractable, the presented architecture is highly parallelizable while maintaining a mechanism to encourage the separately trained GANs generate correlated data.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a generative method for multivariate time series. The method consists of individual GANs for each channel of time series with a shared initial noise, and a global discriminator for preserving the correlations between different channels. The design is to model the common factors of different channels.
strengths_and_weaknesses: Strengths

1. The problem of synthetic time series generation is important in many tasks, e.g., data sharing, data augmentation.
2. The proposed method uses a reasonable approach to enforce common factors of multivariate time series and preserve correlations.
3. The experiments on synthetic and real datasets indicate the effectiveness of the proposed method to some extent.

Weakness:

1. The method is sort of incremental on GAN for time series generation. Preserving correlations using a central discriminator is similar to enforce a regularization of different channels (Fig 1), which is a common way of coordinating multivariate time series.
2. In Eq 2, the condition conditional distributions are assumed independent. That is, the equivalence between the joint condition and the product of individual condition. This is not clearly justified.
3. Enforcing a shared initial noise in Eq 3 and modeling the joint condition in Eq 2 seem have similar effects of regularizing different channels of time series. The description is not clear on how could these two parts both be important.
4. Enforcing a single initial noise may be a strong constraint. Adding K initial noises, where K is smaller than the number of channels, could make the model more flexible and fit training data better.
5. In the experiments, mode time generative model could be compared, on both synthetic and real datasets for comprehensiveness and consistency.

questions: 1. In Eq 2, why to assume the conditional distributions are independent in the conditions (channels)?
2. Will the shared initial noise in Eq 3 and the modeling of the joint condition in Eq 2 have similar effects? How could they both be indispensable in the method?
3. Is it reasonable to add several initial noises, instead of one, for improving the flexibility?

limitations: The authors addressed the potential negative societal impact well.
review_title: null

Meta-review: metareview: The authors propose GroupGAN which uses a separate generator and discriminator for generating each data channel and a central discriminator for accurately capturing the correlation structure between different channels. This is a borderline paper and there were extensive discussions among the reviewers about this paper. The reviewers all agreed that having two sets of discriminators is a good idea. However, the evaluation for this paper were only done with low-dimensional time series and the reviewers voiced concerns about the generalization of this method to higher dimensions. Some parts still need further clarification: for example, as Reviewer zeED pointed out, dimensionality reduction in this paper remains an extremely odd preprocessing step. 

Overall, accepting this paper may start new discussions in the community. I hope the authors address the concerns raised by the reviewers in the camera-ready version of the paper.
confidence: Less certain
Decision: Accept