Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary

The authors are concerned with the classification of EEG signals, in order to predict age, gender, depression and Axis 1 disorder diagonosis from EEG signals. After standard preprocessing and optionnal averaging to obtain evoked responses, the authors feed the samples into a $\beta$-VAE , and then either use a standard classification algorithm or the SCAN method to predict the labels.
The authors report better results than the usual methods based on the late positive potential. They also show that their method can be trained with non-averaged EEG data and still yield good results when tested on ERP , and conversely. Finally, the authors inspect the learned representations.

Major comments

- The paper is very well written, easy and pleasant to read, and well structured.
- The automation of EEG pipelines, like this paper does, is extremely important.
- The SCAN + $\beta$-VAE or SCAN+ VAE method does not seem to perform much better than LR +LPP. Even though SCAN allows for interpretable components, it is arguably much less interpretable than LPP.
- The article validates carefully  a machine learning pipeline on a specific task and dataset, but there is little contribution in terms of machine learning, so I'm wondering whether ICLR is a good fit for this paper, rather than a more neuroscience-oriented conference.

Minor comments
- The authors propose an original pipeline, yet the dataset and the code to reproduce the results are not provided, which hinders reproducibility and the potential impact of this work. 
- In my understanding, the LPP seems to only use one feature for classification: the average amplitude difference between waveforms. Could the authors also consider methods using more hand-crafted features? 
- The EEG signals are only acquired with 3 sensors, it would be interesting to add a word about how the method scales to datasets with more sensors. 
-  It would be interesting to add some ROC curves for the logistic regressions, which would complement nicely the summary statistic used by the authors.

Misc. 

- The software used to perform the study should be acknowledged.
- The ERP are normalized to [0, 1] before going in the VAE. The authors could be more accurate: is each channel normalized individually?
- The references that state that EEG contains important biomarkers of clinical disorders, and that averaging trials yields ERP could also point to more historic papers.

review_title: An extensive and rigorous validation of the power of autoencoders for EEG based classification, but with little novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The authors propose a beta-VAE network to learn EEG representation as biomarkers for diagnosing depression from EEG data. They show improved performance compared to an off-the shelf linear classifier. The paper is well-written but lacks a description of related work in the field and also a detailed analysis of the results to support the claims. 

Novelty:  The use of VAE and beta-VAE for EEG data is not novel and this line of literature should be better discussed in the paper. 

A few more comments/questions for the authors:

1. The details of the AE and beta-VAE architecture should be described in the main paper not the supplementary. Also, a detailed description of the number of trainable parameters and the amount of available data should be added to the main manuscript. 

2. On page 2, the authors mention that LDA and SVM are not commonly used in EEG literature for ERP classification, but these two are actually very common. For instance see: Blankertz, B., Lemm, S., Treder, M., Haufe, S., & Müller, K. R. (2011). Single-trial analysis and classification of ERP components—a tutorial. NeuroImage, 56(2), 814-825.

3. The authors mention that the EEG “ground truth” markers are not available for a condition like depression, and yet they mention that their method provides interpretable biomarkers. There seems to be some discrepancy here that should be explained further. 

4. The authors mention that their paper is the first to use DL representation learning for clinical EEG biomarkers. This claim is not true, please see the following for a list of relevant papers: Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., & Faubert, J. (2019). Deep learning-based electroencephalography analysis: a systematic review. Journal of neural engineering, 16(5), 051001.

5. The axis 1 label is rather confusing, Could the authors please explain it further and say how it is relevant for the depression diagnosis study? 

6. Did the authors do pre-processing (including ICA and removing eye-blinks) at test time? ICA needs a lot of EEG data to be able to reliably remove the eye blinks. That means a pre-trained beta-VAE would not be able to reduce training time. 

7. The authors do not mention which electrodes are selected for their analysis and why. 

8. It is not clear if LR is even significantly better than the other baseline classifier in table A2. Also, the authors should apply the rest of the baseline classifiers to the beta-VAE and AE representations. 

9. In table 1 and A2, the upper limit of the chance level should be calculated as a function of the available data as described in: Müller-Putz, G., Scherer, R., Brunner, C., Leeb, R., & Pfurtscheller, G. (2008). Better than random: a closer look on BCI results. International Journal of Bioelectromagnetism, 10(ARTICLE), 52-55.
In the same table, the overall chance is reported as 0.45. Why is that? 

10. How did the authors evaluate the “significantly better” performance of their proposed classifier? 

11. On page 7, how is the 37x less time need to be spent in the lab by vulnerable populations measured? Again, did the authors used ICA-cleaned data for this evaluation? 

12. Is the separation of the ERPs in Figure 3C significant? Please provide the original ERP for the depression and healthy subjects for comparison. 
Also, the two colors are very hard to tell apart, please use another color combination such as blue and red/orange. 


review_title: Review #1

final_score: rating: 7: Good paper, accept
review_content: 
POST REVISION

Following our discussions and taking the changes made to the manuscript into account, I have decided to increase my score and recommend acceptance. My concerns have been adequately addressed. I believe the paper has been clarified, results are more carefully evaluated and claims are sound. I believe that this work consitutes a well-selected application that addresses a relevant research question with important clinical implications. In my opinion, this deserves to be aknowledged and may be of interest to others in the ICLR audience. My hope is that this application stimulates more work in EEG-based machine learning and also encourages others not to shy away from difficult application domains such as psychiatry, where we really are in need of new solutions to old and - unfortunatley - quite persistent problems.

Thank you very much for your hard work!







---------------------------------------------------------------------------
The authors propose a disentanglement approach (bVAE with/without SCAN) to obtain sparse and interpretable clinical features from EEG time series to aid clinical decisions and automatize EEG preprocessing. They obtain convincing disentangled representations that show some promise for future applications. 

 In the following, I point out strengths and weaknesses, my recommendation and its justification as well as additional detailed comments for the authors. I hope you will find my comments helpful and constructive.

---------------------------------------------------------------------------
Strengths and weaknesses

Strengths
-	Providing ML solutions for psychiatry is a very challenging problem, as psychiatric diseases are of a very complex nature and many crucial processes remain opaque at best. I believe that it is very important that the ML community starts to tackle these issues. I thank the authors for trying to solve this difficult real-world problem and not showcasing their approach on toy examples. Even though the overall accuracies are not high, I believe it is very important to show such an unbiased estimate of where we are at with ML applications in psychiatry. This merits recognition.
-	Choosing EEG data for applications in psychiatry is a very good choice given its feasibility, cost-efficiency and availability in a clinical setting. Often authors do not think carefully enough, whether the data they train their classifier on can actually be acquired in a real-world clinical setting.
-	I applaud the authors’ choice to emphasize interpretability in this healthcare setting. This is a commendable choice that is often not made in other healthcare applications. This is a crucial prerequisite to improve the chance of such an approach ending up in clinical practice as interpretability will be the precursor for acceptance by patients and medical staff.
-	Including study site prediction was a very commendable effort to assess geographic confounders, which is a major concern in recent psychiatric studies with the advent of more and more large scale multi-center datasets.

Weaknesses:
-	Depression outcome label appears to have a proof-of-concept character, rather than addressing a more relevant clinical question (differential diagnosis for bipolar vs unipolar depression for example).
-	Misleading representation of results speaking to superiority of their approach (SCAN approach is much worse than LR combined with bVAE and VAE; therefore, I would advise a more cautious interpretation with respect to the SCAN results and being more precise in the abstract).
-	Confidence intervals for classification results are missing.
-	The claim that their approach reduces ‘hand-engineering’ seems to be unsubstantiated given the description of their methodology.

---------------------------------------------------------------------------
Recommendation
-	Overall, I would like to see this paper accepted, should the authors agree to address the concerns raised above. Unfortunately, I cannot support acceptance as it stands, but I would increase my score, if my concerns have been addressed.

---------------------------------------------------------------------------
Justification of recommendation
-	I believe the research question addresses a very important problem in trying to identify sparse and interpretable clinical markers from EEG data. The method is well-selected to address this question and takes real-world clinical constraints into account (multi-center data, feasibility of data acquisition => EEG, interpretability of identified features => bVAE, uncertainty regarding clinical labels and about which features are required => transfer learning to new labels and unsupervised feature identification). Therefore, I believe this paper deserves to be accepted and represents a good example that hopefully will inspire others to tackle these challenging questions.
-	However, I cannot support acceptance as it stands, because of the weaknesses highlighted above. Most of the issues pertain to clarity of writing, over-interpretation (or at least misleading representation of the results) some missing information on methodology, and choice of the clinical outcome label. For more detailed elaboration on these points see below. 

---------------------------------------------------------------------------
Detailed feedback

Abstract
-	You describe your method (bVAE and SCAN) followed by the sentence: “We demonstrate that our method is able to outperform the canonical hand-engineered baseline classification method on a number of factors, including participant age and depression diagnosis”. This gives the impression that the bVAE+SCAN approach is substantially superior to the hand-engineered baseline. However, in the results table this specific combination is only marginally better than baseline, while bVAE+LR or indeed AE+LR substantially outperform the baseline. This is misleading in the abstract and should also be discussed later on.


Introduction
-	You make point to criticize the ‘hand-engineering’ of conventional EEG analysis (section 1, paragraph 2, p. 1-2). You fail to mention that automatic options for eye blink correction exist, for example PCA based methods like Berg & Scherg (1994): https://doi.org/10.1016/0013-4694(94)90094-9 Indeed fully automated pipelines do exist.
Furthermore, from your methods description (2.1, EEG preprocessing, p. 4;  2.1 Autoencoder, last paragraph p. 4-5), it appears that you follow the conventional preprocessing and only then feed the EEG into the autoencoder, which is of course, in principle, fine, but I fail to see why you bring up the whole critique of the hand-engineered pipeline. Your approach does not seem to address this issue, or did I miss something? 

Methods
-	Some aspects of the methodological description of the analysis are missing. Please, state which software was used to perform the preprocessing. Were ICA-based eye blink correction results visually inspected? Furthermore, for the AE you state that preprocessed trialwise EEG data goes into it as input, for the bVAE this information is missing. Did you follow this approach here as well?
-	The choice of your clinical classification label appears arbitrary and in my opinion does not address a clinically important question. Could you elaborate why you lumped these specific diagnoses together and not others? The axis I label is so broad that it is basically meaningless. What was the rationale for this analysis? I also assume that you classify depression and axis I vs. controls, is this correct? If so, this is not very relevant clinically. The challenge does not lie in determining, whether a person is healthy or has a mental disorder (clinicians usually can tell within a few minutes of conversation), but rather differential diagnosis or prognosis. A more clinically relevant comparison would be, for example, classifying MDD vs bipolar depression. This is a challenging clinical question, because both patient groups can present with depressive symptoms initially and only time tells which diagnose and also (importantly) medication is appropriate. Why did you not choose such a classification problem?
-	In your description of the bVAE you state that each disentangled factor corresponds to an ‘interpretable’ transformation of the data. This is misleading. Interpretability is of course the goal of a reduced latent space, but by no means the default result. Whether a compression is indeed interpretable needs to be carefully assessed.
-	Was test data selected across subjects (other individuals to test) or within subjects (other parts of the data from the same individual, but all individuals included in the training data)? This is important information as the goal would be to generalize to unseen individuals rather than unseen data from the same individual.

Results
-	Please, add confidence intervals for balanced accuracy (for example by computing the posterior balanced accuracy: Broderson et al. 2010, https://ieeexplore.ieee.org/document/5597285/) otherwise it is impossible to assess the relative quality of the different classifiers. Statements such as: “the classification accuracy is still significantly higher for deep representations compared to the LPP” should be backed up by a statistical test of some sort. Along the same lines how was “significantly higher than chance“ assessed? By permutation tests, based on confidence intervals/Bayesian confidence intervals?
-	You state: “This suggests that replacing the more manual canonical LPP pipeline with deep representation learning can allow for both better training data efficiency and a reduction in time that the (potentially vulnerable) participants have to spend in the lab by up to 37x,…” For this interpretation to hold, you would need to show, that you can train your classifier on a single trial, otherwise, you would still need all the data for pretraining to then be able to make predictions based on single trials. I would advice clarifying this.
-	Lastly, I would like to state, that I was very impressed that you were able to re-discover the anhedonia neurocorrelates with your approach. I think the major challenge is indeed to make sure, that your bVAE identifies meaningful latent variables which I believe you successfully showed.

review_title: Solid application of well-selected method to difficult application domain with interesting results - Recommend acceptance 

Meta-review: The approach is novel and according to the reviewers' comments addresses a relevant and important problem on EEG data analysis. Differences to related work are discussed. Methods and Experimental results are sound. The authors have provided a comprehensive response to the reviews.

Decision: Accept (Poster)