Representation learning for improved interpretability and classification accuracy of clinical factors from EEG
Authors: Garrett Honke
Irina Higgins
Nina Thigpen
Vladimir Miskovic
Katie Link
Sunny Duan
Pramod Gupta
Julia Klawohn
Greg Hajcak
Venue: ICLR 2021 Poster (2021)
Keywords: EEG
ERP
electroencephalography
depression
representation learning
disentanglement
beta-VAE

Abstract: Despite extensive standardization, diagnostic interviews for mental health disorders encompass substantial subjective judgment. Previous studies have demonstrated that EEG-based neural measures can function as reliable objective correlates of depression, or even predictors of depression and its course. However, their clinical utility has not been fully realized because of 1) the lack of automated ways to deal with the inherent noise associated with EEG data at scale, and 2) the lack of knowledge of which aspects of the EEG signal may be markers of a clinical disorder. Here we adapt an unsupervised pipeline from the recent deep representation learning literature to address these problems by 1) learning a disentangled representation using $\beta$-VAE to denoise the signal, and 2) extracting interpretable features associated with a sparse set of clinical labels using a Symbol-Concept Association Network (SCAN). We demonstrate that our method is able to outperform the canonical hand-engineered baseline classification method on a number of factors, including participant age and depression diagnosis. Furthermore, our method recovers a representation that can be used to automatically extract denoised Event Related Potentials (ERPs) from novel, single EEG trajectories, and supports fast supervised re-mapping to various clinical labels, allowing clinicians to re-use a single EEG representation regardless of updates to the standardized diagnostic system. Finally, single factors of the learned disentangled representations often correspond to meaningful markers of clinical factors, as automatically detected by SCAN, allowing for human interpretability and post-hoc expert analysis of the recommendations made by the model.

Published as a conference paper at ICLR 2021
REPRESENTATION LEARNING FOR IMPROVED INTER- PRETABILITY AND CLASSIFICATION ACCURACY OF CLINICAL FACTORS FROM EEG
Garrett Honke*t
Irina Higgins i
Nina Thigpen'
X, the Moonshot Factory Mountain View, CA
Deep London, UK
X. the Moonshot Factory Mountain View, CA, USA
Vladimir Miskovic
Katie Link
Sunny Duan
X. the Moonshot Factory Mountain View, CA, USA
X. the Moonshot Factory Mountain View, CA, USA
DeepMind
Mountain View, CA, USA
Pramod Gupta
Julia Klawohn Florida State University Tallahassee, FI., USA
Greg Hajcak Florida State University Tallahassee, FL., USA
X, the Moonshot Factory Mountain View, CA, USA
ABSTRACT
Despite extensive standardization, diagnostic interviews for mental health disorders encompass substantial subjective judgment. Previous studies have demonstrated that EEG-based neural measures can function as reliable objective correlates of depression, or even predictors of depression and its course. However, their clinical utility has not been fully realized because of 1) the lack of automated ways to deal with the inherent noise associated with EEG data at scale, and 2) the lack of knowledge of which aspects of the EEG signal may be markers of a clinical disorder. Here we adapt an unsupervised pipeline from the recent deep representation learning literature to address these problems by 1) learning disentangled representation using 3- VAE to denoise the signal, and 2) extracting interpretable features associated with sparse set of clinical labels using Symbol-Concepp Association Network (SCAN). We demonstrate that our method is able to outperform the canonical baseline classification method number of factors, including participant age and depression diagnosis. Furthermore, our method recovers a representation that can be used to automatically extract denoised Event Related Potentials (ERPs) from novel, single EEG trajectories, and supports fast supervised re-mapping to various clinical labels, allowing clinicians re-use single EEG representation regardless updates to the standardized diagnostic system. Finally, single factors of the learned disentangled representations often correspond to meaningful markers of clinical factors, as automatically detected by SCAN, allowing for human interpretability and post-hoc expert analysis of the recommendations made by the model.
INTRODUCTION
Mental health disorders make up one of the main causes of the overall disease burden worldwide (Vos et al., 2013), with depression (e.g., Major Depressive Disorder, MDD) believed to be the second leading cause of disability (Lozano al., 2013; Whiteford et al., 2013), and around 17% of the population experiencing its symptoms at some point throughout their lifetime (McManus et al., 2016; 2009; Kessler et al., 1993; Lim et al., 2018). At the same time diagnosing mental health disorders has many well-identified limitations (Insel al., 2010). Despite the existence of diagnostic manuals
'Equal contribution
ghonk, irinah, hiegn,,,, vmiskovic, katielink, sunnyd, pramodg egoogle.......oooooooooog com JJK is now at Hunboldt--niversiitl zu Berlin, Berlin, Germany; julia. klawchn@hn caannehuubeee werrbberrndd. i reg. ha hajcak@medd fsu. edu

---

Published as a conference paper at ICLR 2021
Neutral IAPS image axamples:
SGAN INSormaTaa
SCAN desoder P,9
moon
Gender
B-VAEIRE emooder 9,O-,H
KLITJEP
SCAN encodar CUAc,00
Depresdion
IAPS image
examples:
B-WAEE
MAEIINE P.PATL))
SCAN rpptt trl
Trating
B-NAAA
ClessFcctioo from ERPVERS
ERF
ETG
Vnjeclotics from single
pertciparti
Figure I: Pipeline schematic. Participants are presented with images from the International Affective Picture System (IAPS). EEG trajectories recorded from the same participant over multiple trials are averaged to create ERPs. Each EEG sample consists of 256 time samples (-248-772 ms, where 0 is stimulus onset time) of stimulus-locked activity recorded from three sites as participants view either neutral (red) or positive (blue) TAPS images. ERP responses to positive and negative images are concatenated and normalised between [0. 1] across all channels simultaneously. The resulting "images" are used to train 3-VAE or AE. well disentangled pre-trained 3- VAE model used to train SCAN. Each SCAN training example consists of a 5-hot binary classification label presented to the SCAN encoder, and its corresponding EEG "image" x presented to the 3- V-E encoder. 3- VAE weights are fixed during SCAN training. To obtain classification, encoder, then the inferred z, means are fed through the SCAN decoder, where a per-class softmax is applied to obtain the predicted label (red pathway). To analyse SCAN classification decisions, 1-hot binary vector is fed into SCAN encoder. Samples from the inferred distribution Zy are then fed through the 3- VAE decoder to visualise the corresponding ERP reconstructions (purple pathway).
EEG "image" is presented to the 3- 3-VAE
like Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders (SCID) (DSM-V, 2013), diagnostic consistency between expert psychiatrists and psychologists with decades of professional training can low, resulting different diagnoses in upwards of 30% of the cases (Cohen's Kappa = 0.66) (Lobbestael et al., 2011) 1). Even if higher inter-rater reliability was achieved, many psychological disorders do not have fixed symptom profile, with depression alone having many hundreds of possible symptom combinations (Fried & Nesse, 2015). This means that any two people with the same SCID diagnosis can exhibit entirely different symptom expressions. This is core challenge for developing objective, symptom driven diagnostic tool in this domain.
Electroenncephalograpyy (EEG) is measurement of post-synaptic electrical potentials that can be taken non-invasively the scalp. EEG signals can function important biomarkers of clinical disorders (Hajcak et al., 2019) but they are difficult to clean and interpret at scale. For example, components of the EEG signal can often significantly overlap interfere with each other. Furthermore, nearby electronics, line noise, hardware quality, signal drift and other variations in the electrode-scalp connection can all distort the recorded EEG signal. Hence, the extraction of BBG data of sufficient quality is usually a laborious, semi-automate process executed by lab technicians with extensive training. A typical EEG analysis pipeline consists of collecting EEG recordings evoked from large number of stimulus presentations (trials) in order to have sufficient data to average out the noise. Independent Components Analysis (ICA) is often used to visually identify and remove the component that corresponds to eye blinks (Delorme & Makeig, 2004; Makeig et al., 2004; Jung et al., 2000) (although see Weber et al. (2020); Nolan et al. (2010) as examples of fully automated artifact removal pipelines). . This can be followed by trial rejecton stage where anomalous trials are identified and removed from the EEG data scroll, sometimes also through visual examination. The cleaned up EEG recordings from a large number of trials are then averaged produce Event Related Potential (ERP) (Luck, 2012). This allows a clinician to extract specific ERP components relevant to the clinical factor of interest, average the event-locked activity within them, and then either perform statistical group comparison, or-in
2

---

Published as a conference paper at ICLR 2021
the case of the diagnosis classification (LR) to obtain the final diagnostic results. Some more advanced classification approaches might include Support Vector Machines (SVM), Linear Discriminant Analysis (LDA), or Random Forest (RF) (Parvar et al,, Güler& & Übeyli, 2007; Subasi & Gursoy, 2010; Tomioka et al., 2007; Bashivan et al., 2016).
off off-the-shelff classifier, like Logistic Regression
To summarise, EEG recordings are noisy measures of electric activity from across the brain. There is evidence that these signals are useful as markers of depression, but we lack understanding of what aspects of depression they index. Furthermore, the field of clinical psychopathology still lacks consensus on the etiopathogenesis of mental health disorders, which means that there is no such thing as the "ground truth" diagnostic labels. Hence, while EEG is routinely used to diagnose conditions like epilepsy (Smith, 2005), memory (Stam et al., 1994) or sleep disorders (Karakis et al., 2012), its promise for being a reliable diagnostic tool for clinical conditions like depression has not been fully realised so far. In order to make EEG a viable diagnostic tool for a broader set of clinical conditions it is important to have an automated pipeline for extracting the relevant interpretable biomarker correlates from the (preferably individual trial) EEG data in robust manner. Furthermore, this process should not depend fully on diagnostic labels which are often subjective and at best represent highly heterogeneous classes.
Recent advances in deep learning have prompted research into end-to-end classification of EEG signal using convolutional and/or recurrent neural networks (Bashivan et al., 2016; Mirowski et al., 2009; Cecotti & Graser, 2011; Güler et al., 2005; Wang et al., 2018; Farahat et al., 2019; Solon et al., 2019; Cecotti et al., 2014), 4), holding the promise of automated extraction of relevant biomarker correlates. However, deep classifiers operate best in the big data regime with clean, well-balanced ground truth classification targets. In contrast, even the largest of EEG datasets typically contain only few hundred datapoints, and the classification labels are subjective, noisy and unbalanced, with the majority of the data coming from healthy control participants. Hence, in order to utilise the benefits of deep learning but avoid the pitfalls of over-reliance on the classification labels, we propose two-step pipeline consisting of unsupervised representation learning, followed by supervised mapping of the pre-trained representation to the latest version of the available diagnostic labels. The hope is that the unsupervised learning step would denoise the input signal and extract the broad statistical regularities hidden in thus serving as an alternative for the existing automatic EEG pre- pre-processing pipelines (Weber et al., 2020; Nolan et al., 2010) while minimising the need for priori knowledge. resulting in a representation that can continue to be useful even if the label taxonomy evolves.
Recently great progress has been made in the field of deep unsupervised representation learning (Roy et al., 2019; Devlin et al., 2018; Brown et al., 2020; Chen et al., 2020b; Grill et al., 2020; Chen et al., 2020a; Higgins et al., 2017; Burgess al., 2019). Disentangled representation learning a branch of deep unsupervised learning that produces interpretable factorised low-dimensional representations of the training data (Bengio al., 2013; Higgins al., 2017). Given the requirement for model interpretability in our use-case, we use Beta Variational Autoencoders (3-VAE) (Higgins et al., 2017)-one of the state of the art unsupervised disentangled representation learning methods-to discover din-eimnsionnl disentangled representations of the EEG data. We then train the recently proposed Symbol Concept Association Network (SCAN) (Higgins et al., 2018) to map the available classification labels to the representations learnt by 3-VAE (see Fig. 1). We demonstrate that our proposed pipeline results better classification accuracy than the typical approach for extracting a known ERP pattern for use as a biomarker-a process that is often heavily influenced by priori knowledge. This holds true when predicting a number of factors, including age, gender, and depression diagnosis. Furthermore, SCAN is able to produce arguably interpretable classification recommendations, whereby its decisions on different clinical factors are grounded in a small number (often single) latent dimensions of the V-VAE, allowing the clinicians opportunity to interpret the recommendations produced by SCAN, and visualise what aspects of the EEG signal are associated with the classification decision host-hoc. This opens up the opportunity to use our proposed pipeline as a tool for discovering new EEG biomarkers. We validate this by -r-discovering" known biomarker for depression. Finally, we demonstrate that once a 3-VAE pre-trained on ERP signals, it can often produce ERP-like reconstructions even when presented with single noisy EEG trajectories. Furthermore, the representations inferred from single EEG trials produce good classification results, still outperforming the canonical baseline method. This suggests that once good disentangled representation is learnt, the model can be used online as new EEG data is being recorded, thus lowering the burden of keeping potentially vulnerable participants in the lab for extended recording sessions.
3

---

Published as a conference paper at ICLR 2021
2 METHODS
2.1 DATA
Anhedonia. This work targets one of the two cardinal symptoms of depression annseoonn- Anhedonia is the lack of pleasure and/or interest in previously pleasurable stimuli and activities (DSM-V, 2013). One established approach for objectively quantifying this symptom is the use of EEG to measure neural responses elicited by emotionally salient visual stimuli. Research in this domain has uncovered a stereotyped neural activation pattern in healthy control participants, where amotionally-salices stimuli evoke a Late Positive Potential (LPP) in ERPs the averaged timeseries of stimulus time-locked EEG recordings. This pattern has been identified as a potential biomarker for depression because (on average) this positive deflection in amplitude is attenuated absent in individuals who exhibit symptoms of anhedonia (Brush et al., 2018; Foti et al., 2010; Klawohn et al., 2020; MacNamara et al., 2016; Weinberg et al., 2016; Weinberg & Shankman, 2017).
Participants. The data were collected part of mental health study across multiple laboratory sites. The multi-site aspect of the study meant that more data could be pooled together. however, also meant that the data was noisier. Participants (N = 758, agex 16.7, age = [111.0, 59.8). 398 398 female) were samples of healthy controls (Nue = 485) and people diagnosed with depression (among other mental illnesses) (see Sec. A.1.1 and Tbl. A1 for further breakdown).
Stimuli and Experimental Design. Participants were shown series of 80 images from the International Affective Picture System (IAPS) (Lang et al., 2008) presented in random order up to 40 times each. The images varied in valence: either positive (affiliative scenes or cute animals designed to elicit the LPP ERP component), or neutral (objects or scenes with people). Each image trial consisted of white fixation cross presented for random duration between 1000-2000 ms (square window) followed by a black and white image presented for 1000 ms.
EEG Preprocessing. While participants completed the picture viewing task, EEG was continuously recorded. Each picture trial was then segmented to contain a 200 ms pre-stimulus baseline and 1200 ms post-stimulus interval. The raw EEG signal was digitized, bandpass filtered and cleared of the eye movement artifacts and anomalous trials as described in Sec. A.1.2.
Classification labels. The following classification labels were used in this study: age (adult or child), gender (male or female), study site, and the presence or absence of two clinical conditions: depression diagnosis and broader Axis disorder diagnosis. All classification labels were binary, apart from study site, which contained four possible values corresponding to four different sites where the data were collected. Participants 18 years of age and older were classified as adults. Gender was classified based on self-reported values. Positive depression labels 1n=110) include all participants that were diagnosed with Major Depressive Disorder (MDD), Persistent Depressive Disorder (PDD), and depressive disorder NOS tnot-othhrriise-spccffeel] by expert clinicians through clinical interview Je.g..SCII for adults, KSADS for children). Axis 1 is broad category consisting of the most prevalent psychological disorders in the population (now discontinued in DSM-V V) that excludes intellectual disabilities and personality disorder (DSM-V, 2013). Positive Axis labels (n (n=22 273) encompassed all participants with positive depression labels plus individuals diagnosed with Cyclothemia, Dysthemia, anxiety disorders, mood disorders (e.g.. Bipolar Disorder), panic disorders, and substance and eating disorders (all of which are sparse). The Axis class is provided to compare model behavior on a transdiagnostic measurement of psychopathology! While recruitment for the study was primarily focused on depression, the SCID produces a large set of diagnostic decisions, and we collapsed this sparser set of positive diagnoses into the existing Axis DSM-IV superordinate category. We include this label in modeling and analysis to maximize the number of positive labels for training and evaluation and to give the reader a sense of what the algorithm may have learned that is generalizable across disorders-akin to the P factor (Caspi et al., 2014).
1 'Dioorders in this class that are present in the data include Major Depressive Disorder, Persistent Depressive Disorder, Depression NOS, Cyclothemia. Dysthemia. Bipolar I, Bipolar 11, Bipolar NOS, Mania, Hypomania, Agoraphobia, Social Phobia, Separation Anxiety, Generalized Anxiety Disorder, Panic Disorder. Panic Disorder with Agoraphohia, Anorexia, Bulimia, Fating disorder NOS, Alcohol Abuse, Alcohol Dependence, and Substance Abuse and Substance Dependence disorders.
4

---

Published as a conference paper at ICLR 2021
2.2 REPRESENTATION LEARNING
Canonical LPP analysis baseline The canonical approach for extracting the Late Positive Potential (LPP) effect serves as the baseline for this work. The LPP effect is calculated as the average amplitude difference between ERP waveforms (i.e., averaged stimulus time-locked EEG segments) evoked from emotionally-salien and neutral stimuli. Before the delta was calculated, each HRP was normalised with respect to the baseline average activity within the 100 ms window preceding stimulus onset. Finally, the normalised delta signal within the 300-700 ms window after stimulus onset was averaged to provide the canonical baseline LPP representation.
Autoencoder An AutoEncoder (AE) is a deep neural network approach for non-linear dimensionality reduction (Hinton & Salakhutdinov, 2006; Baldi, 2012). typical architecture consists of an encoder network, which projects the input high- ligh-dimensional data x into cow-dimensional representation z, and a decoder network that is the inverse of the encoder. projecting the representation z back into the reconstruction of the original high-dimensional data [x; 6,0), where e and 0 are the parameters of the encoder and decoder respectively (see Fig. 1 for model schematic). AE is trained through backpropagation (Rumelhart et al., 1986) using the reconstruction objective:
(x; 6.0) -
The input to the AE in our case 256x6 "image" of the EEG signal (see Fig. 1), where 256 corresponds to the 1024 ms of the recorded EEG trajectory sampled at 250 Hz and pre-processed as described Sec. A. 1.2 and 6 corresponds to three electrodes per each the two image valence conditions (i.e., stimulus classes): neutral and positive. These input images were further normalised to the [0, 1] range across all channels before being presented to the AE.
The AE was parametrised to have two convolutional encoding layers with 32 filters each of size 6, with strides of step size 2 along the time axis, followed by a single fully connected layer of size 128, projecting into (O-dimensional representation The decoder the model consisted of around 106,752 parameters. The model had ReLU activations throughout, and was optimised using Adam optimizer with learning rate of le-4 over I mln iterations of batch size 16.
the inverse of the encoder. Overall
3- o-Variational Autoencoder A 3- Variational Autoencoder (3-VAB) (Higgins et al., 2017) is a gen- erative model that aims to learn a disentangled latent representation z of input data x by augmenting the Variational Autoencoder (VAE) framework (Rezende et al., 2014; Kingma & Welling, 2014) (see Aec.A.2.1 for more details) with an additional /3 hyperparameter. Intuitively, a disentangled representa- tion is factorised latent distribution where each factor corresponds of the training data (c.g. in a disentangled representation of a visual scene, individual factors may represent change in lighting or object position). neural network implementation of 3 3-VAE consists of an inference network (equivalent to the AE encoder), that takes inputs x and parameterises the (disentangled) posterior distribution q(z|x), and a generative network (equivalent to the AE decoder) that takes sample from the inferred posterior distribution 2 ~ N' [u(z[x),or (z(x)) (z]x)) and attempts to reconstruct the original image (see Fig. 1). The model trained through a two-part loss objective:
interpretable transformation
GS-VAE=Eppsa (
(z)x) (x)x)) pe (x L p(z))
where p(x) is the probability of the input data, g(z(x) is the learnt posterior over the latent units given the data, p(z) is the unit Gaussian prior with a diagonal covariance matrix (O,II),, ) é and are the parameters of the inference (encoder) and generative (decoder) networks respectively, and 3 is hyperparameter that controls the degree of disentangling achieved by the model during training. Intuitively, the objective consists of the reconstruction term (which aims to increase the likeliiel of the observations) and a compression term (which aims to reduce the Kullback-Leibler (KL) divergence between the inferred posterior and the prior). Typically a 3 1 is necessary to achieve good disentangling, however the exact value differs for different datasets. In order to find good value of 3 to disentangle our EEG dataset, we perform hyperparameter search, training ten models with different random initialisations for each of the ten values of / [0.075, 2.0) sampled uniformly. Well disentangled 3- VAE models were selected using the Unsupervised Disentanglement Ranking (UDR) score (Duan et al., 2019) described in Sec. A.4 (see Fig. A1 1 for visualisation of the resulting UDR
5

---

Published as a conference paper at ICLR 2021
Age
Gender
CORVERP
DRPIUUPL
SNFLIRRP
SUPLIGUPL
ERPGUAL
SMPLERR
GUPUSINPL
ERPEER
ERRSANPL
SMPUERR
SNPUSHP.
Depression
Axis
Figure 2: Posterior distribution of balanced classification accuracy (mean, error bars 95% confidence interval) (Carrillo et al., 2014; Brodersen et al., 2010). ERP averaged trajectories, SMPL- single sampled EEG trials. LPP- canonical late positive potential baseline representation. LR- L2 regularised logistic regression (see Tbl. A2 for other baseline classifiers). Models trained on single EEG trajectories (SMPL) are tested on different single EEG trajectories the SMPL/SMPL train/test case. Black horizontal line- chance accuracy, horizontal dashed lines 955 confidence interval calculated according to Müller-Putz et al. (2008).
scores). All B- 3-VAE models had the same architecture as the AE?, and were trained in the same manner and on the same data that was pre- pre-processed in the same way to lie in the [0, 1] range.
2.3 CLASSIFICATION
Baseline classifiers To evaluate the quality of representation in terms of how useful for classifying different clinical factors, we applied a range of baseline classification algorithms: Support Vector Machine (SVM), Random Forest (RF), Logistic Regression (LR) and Linear Discriminant Analysis (LDA) (see Sec. A.3 for details). For all classification results we report the posterior distribution of balanced accuracy (Carrillo et al., 2014; Brodersen et al., 2010). Balanced accuracy was chosen, because it correctly matches chance accuracy even for unbalanced datasets. Chance accuracy and its confidence bounds were calculated according to Müller- et al. (2008).
Symbol-Concepp Association Network The baseline classifiers described above produced uninter- pretable decisions. This is undesirable if we were to have a chance at discovering new clinical biomark- ers in the BEG data. To address this interpretability challenge, we leverage recent model proposed for visual concept learning in the machine learning literature- the Symbol-Concepp Association Network (SCAN) (Higgins et al., 2018). While SCAN was not originally developed with the classification goal in mind, it has desirable properties to utilise for the current application. In particular, is able to automat- ically discover sparse associative relationships between discrete symbols (5-hot classification labels in our case, see Fig. 1) and continuous probability distributions of the disentangled posterior representation discovered by a trained 3- model. Furthermore, the associative nature of the grounding used to train SCAN allows it to deal with noisy data gracefully, and to learn successfully from small number of pos- itive examples and from highly unbalanced datasets. SCAN is in effect another VAE model. In our case it takes 5-hot classification labels y as input, and aims to reconstruct them from the inferred posterior q(zly) (see Fig. for more details). To train SCAN the original VAE objective is augmented with an ad- ditional KL term that aims to ground the SCAN posterior in the posterior of a pre-trained ,3- VAE model:
LSCAN = Eply) (x (x) [log (ylz,)) 1 K (( (2,ly) ll LL (z (x) go (zyly))))
NNote that the AE can be seen as special case of the 3 VAR with 0=

---

Published as a conference paper at ICLR 2021
ERP inputs
Reconstructions
SMPL inputs
(ERP)
Figure 3: Reconstructions (middle six columns) by two disentangled 3- VAEs pre-trained on either ERPs or single EEG sample trajectories (SMPL), as well as an AE pre-traines on ERPs. Each model reconstructs either single EEG trajectories (SMPL. inputs) or the corresponding ERPs (ERP inputs). Models pre-trained on ERPs are able to reconstruct ERP-like trajectories even from single EEG samples, as demonstrated by the closer similarity of the 3- V-Es (ERP) and AE (ERP) reconstructions ERP inputs compared to SMPL inputs, regardless whether ERPs or single EEG samples are reconstructed.
where v and the parameters of the SCAN encoder and decoder respectively, go (z(2,, ly) is the posterior SCAN inferred from a 5-hot label y, and g (z [x] ) is the posterior of the pre-trained 3-VAE model inferred from EEG "image" corresponding to the label presented to SCAN. Note that the 3- V-V weights are not updated during SCAN training. The extra grounding term K (Z. [x] || go (xyz)))) allows SCAN to discover associative relationship between subset of disentangled factors and each 1-hot dimension of the given label.
We paramterised SCAN encoder and decoder MLPs with hidden layers of size 128, ReLU non-linearities and a 10-dimensional posterior to match the dimensionality of the 3- V-E representation. The model resulted in around 22,016 parameters. Like the other models, SCAN was trained with batch size 16, over I min iterations and with Adam optimizer with learning rate of le-4.
RESULTS
Deep representation learning improves classification accuracy. We evaluated the classification accuracy for predicting participant age, gender, study site, and the presence absence of two clinical labels: depression and Axis 1. We first obtained the canonical LPP representations, and applied Support Vector Machine (SVM), Random Forest (RF), Logistic Regression (LR) and Linear Discriminant Analysis (LDA) to obtain balanced classification accuracy. Table A2 and Figure A2 show that LR produced some of the best results overall for the baseline, hence we report only LR results for the VAE E and AE representations in the main text (see Tables A3-A4 and Figures A3-A4 for the other classifiers applied to the 3-VAE and AE representations). Figures and 5 (ERP/ERP train/test condition) demonstrate that representations extracted through / VAE and AE pretraining resulted in higher overall classification accuracy than those obtained through the baseline canonical LPP pipeline (see also Table AS ERP/ERP cells). This effect holds across all classification tasks, including depression and Axis diagnoses. Furthermore, the pattern of classification results in line with what might have been expected by the expert clinicians, whereby age and depression classification accuracy is significantly higher than chance, while gender is harder to decode from the EEG signal.
similar pattern appears to hold in most cases when all the models trained on single EEG trials instead of ERPs (SMPL/ERP train/test in Figures 2 and 5, see also Table A5). While the maximum classification accuracy drops for all models when trained from the noisier single EEG trials rather than ERPs, the classification accuracy obtained from 3-VAE and AE representations is still often higher than that obtained from the LPP baseline.
Classification based on deep representations generalises better to single EEG trials. One possible clinical application of the proposed classification pipeline is to enable online diagnosis recommendations from single EEG trials. Hence, we tested how well the classification accuracy of the different representations generalises to novel single EEG trial trajectories (*/SMPL in Figures and 5, see also SMPL columns Tbl. A5). As expected, the performance often drops when classifying the noisy single EEG trajectories, regardless whether the representations were pre-trained using
7

---

Published as a conference paper at ICLR 2021
A
-2
Latent traversal
2
Gender
Axis 1
No depression
SCAN scaN samples
22
Deta
25
z6
Depression
Study site
Depression
27
SCAN Aammrn Commen
z8
z9
Data
Figure 4: A: Latent traversals of a pre-trained disentangled |3- BAE. Each row reconstructs an ERP trajectory the value of each latent dimension is traversed between [-2, 2] while keeping the values of all other latents fixed. B: Inferred distributions for each of the 10 latent dimensions a SCAN trained with the B- VAE shown in A. Each subplot shows the result of SCAN inference a set of 1-hot symbols corresponding to all the different conditions of a single factor (e.g. M/F for Gender). Different factors have automatically become associated with different (often single) latent dimensions. For example, the presence (red) or absence (blue) of Axis diagnosis is represented by the narrow inferred Gaussian distributions centered around different opposite values of latent 2g, while other latent dimensions defaulted back to the unit Gaussian prior. C: five samples from each of the two conditions associated with the depression factor from the SCAN shown B, well as corresponding ERP samples from the training dataset. Red - neutral responses, blue pleasant responses from each of the three EEG sensors. SCAN has automatically discovered that the presence of depression is associated with the lack of clear separation between responses in the two valence conditions anhedonia. Its samples are qualitatively similar to the samples from the dataset.
ERPs (ERP/SMPL) or single EEG samples (SMPL/SMPL, note that we tested the models with different single EEG samples to those used for training). However, the classification accuracy is still often significantly higher for deep representations compared to the LPP baseline. This suggests that replacing the more manual canonical IPP pipeline with deep representation learning can allow for both better training data efficiency and a reduction in time that the (potentially vulnerable) participants have spend in the lab by up to 37x, which is the average number of trials per condition that made it through EEG pre- rre-processing in our dataset and were used for generating ERPs.
Deep representation learning reconstructs ERPs from single EEG trials. Since 3- VAE and AE had good classification accuracy when presented with single EEG samples, tested whether they could also reconstruct ERPs from single EEG trials. Figure shows that this is indeed the case- reconstructions produced by the pre-trained models from single noisy EEG samples look remarkably similar to those produced from the corresponding ERPs (also see Figs, A5-A7 for more examples), Note that this only holds true for models trained using ERP data and not those trained on single EEG samples.
Disentangled representations allow for interpretable classification. To obtain SCAN classifica- tion results, we inferred the posterior qe (z.]() |x) of pre-trained (3- VAE in response to an EEG "image" x. We then used the pre- pre-trained SCAN decoder obtain the reconstructed label logits p, (y (z[z) using the 3-VAE posterior mean as input (Fig. 1, red path). Finally, we applied softmax over the produced logits obtain the predicted 5-hot label for the EEG "image". When SCAN was trained on top of well disentangled 8 VAE, it was often able to outperform the canonical LPP baseline in terms of classifica- tion accuracy (see Figure and Table A5, SCAN+ VAE).. This is despite the fact that SCAN was not developed with the classification goal in mind. Note, however, that SCAN can only work when trained with disentangled representations. Indeed, SCAN classification performance was in most conditions not distinguishable from chance when trained on top of entangled AE representations (see Figure 2 and Table AS, SCAN+AE). To further confirm the role of disentanglemenn, we calculated Spearman correlation between the quality of disentanglemest as measured by the UDR score and the final SCAN classification accuracy. Table shows significant correlation for age, depression and Axis 1 diagnoses, suggesting that average these factors were classified better if representations The same, however, is not the case for gender or study site. This implies that some of the better disentan- gled models did not contain information necessary for classifying these factors. Such information loss is
more disentangled.
8

---

Published as a conference paper at ICLR 2021
Age
Gender
Site
Depression
Axis
r
p
p
p
0.28
.01
-0.1
.31
0.19
.06
0.33
<.001
0.31
<,001
Table 1: Spearman correlation between disentanglement quality (as approximated by UDR score) of trained 3- VAE models (n=100) and balanced classification accuracy from their corresponding SCAN models.
in fact known trade-off of 3- V-VAE training, with more disentangled 3- 3-VAE models often compromising on the informativeness of the learnt representation due to the increased compression pressure induced by the higher 3 values necessary achieve disentanglement (Higgins al., 2017: Duan et al., 2019).
When trained on top of well disentangled B- V-V models, SCAN decisions were not only accurate, but they were also based which obtained average sparsity of just 3.25% compared to the 87.5% for SCAN, even when LI regularisation was used. Furthermore, the disentangled dimensions used by SCAN were often arguably interpretable, hence making SCAN classification decisions amenable to post-hoc analysis. Figure 4B visualises the inferred SCAN posterior when presented with 1-hot labels corresponding to the male or female gender, one of the four study sites, and the presence or absence of depression and Axis 1 diagnoses. most cases, SCAN was able to associate the label with a single latent dimension the pre-trained 3-VAE (e.g., gender is represented by latent dimension 25), and the different values of the same class label corresponded to disjoint narrow Gaussians on those latents (c.g.. male gender is represented with Gaussian N (0.89, 0.49), while female gender is represented by Gaussian (-0.82,, 0.52), both on 25). We can also visualise what the different VAE latents have learnt to represent by plotting their traversals as shown in Figure 4A (see more in Figs. A8-A10).
small number of disentangled dimensions. This is unlike the LR classifier,
Finally, we can also sample from the SCAN posterior 2, ~N((uzz (o) (z, y)) and reconstruct the re- sulting ERPs using the VAE decoder Ps (x(2,) (Fig. 1, purple path). Such samples for the depression label are visualised in Figure 4C. It is clear that the reconstructed ERP samples corresponding to partic- ipants with positive depression diagnosis contain difference between the positive and neutral trials (overlapping blue and red lines), while those corresponding to participants without positive depression diagnosis do have an obvious gap between the responses to positive and neutral trials. Hence, we were able to "re-discover an objective biomarker for the symptom of anhedonia (and depression generally), thus opening up the potential for discovering new biomarkers hidden in EEG data the future.
4 CONCLUSION
This work provides the first evidence that disentanglement-focusee representation learn- ing and the resulting models are powerful mea- surement tools with immediately applicability to applied and basic research for clinical psychol- ogy and electrophysiology. One caveat, however, should be considered: more work is needed to further explore how this approach could be used to produce truly meaningful clinical inferences. Toward that goal, this study limited because of (I) the choice to focus analysis on the superordi- nate depression class opposed its constituent disorders, (2) the use of Axis 1 as point of com- parison as opposed a similarly represented co- morbid disorder, and (3) the choice to train mod- els without netween-particinge
FRPERP
FRPOMPI
EMPLERP
Figure 5: Precision-recall curves for balanced clas- sification accuracy of depression diagnosis calcu- lated as the average of precision and recall for L2 regularised logistic regression. ERP averaged trajectories, SMPL single sampled EEG trajecto- ries. LPP canonical late positive potential baseline representation. Models trained single EEG tra- jectories (SMPL) are tested on different single EEG trajectories in the SMPL/SMPL train/test case.
voss-vvaidation. Clearly, there is need for follow-up work to examine factors that have more clinical impact such as differences in types of depression (persis- tent vs. major depressive disorder) and treatment response.
We have demonstrated that disentangled repre- sentation learning can be successfully applied to

---

Published as a conference paper at ICLR 2021
EEG data, resulting in representations that can be successfully re-used to predict multiple clinical factors through fast supervised re- mapping, out- performing baseline typical for the field. Our method recovers representation that can be used to automatically extract denoised Event Related Potentials (ERPs) from novel, single EEG trajectories. Finally, single factors of the learned disentangled representations often correspond meaningful mark- ers of clinical factors (as automatically detected by SCAN), allowing for human interpretability and potentially providing novel insight into new biomarkers and the neurofunctional alterations underlying mental disorders. While SCAN does not always produce statistically-relabb accuracy advantages over more traditional methods (all using B- VAE encoding), the ability show the user exactly how patterns in the raw data manifest across clinical groups of interest is a considerable advantage.
ACKNOWLEDGMENTS
We thank Sarah Laszlo, Gabriella Levine, Phil Watson, Obi Felten, Mustafa Ispir, Edward De Brouwer, the Amber team, Nader Amir and the Center for Understanding and Treating Anxiety, Dr. Kristen Schmidt, Alec Bruchnak, and Nicholas Santopetro.
REFERENCES
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on Unsupervised and transfer learning. pp. 37-49, 2012.
Pouya Bashivan, Irina Rish, Mohammed Yeasin, and Noel Codella. Learning representations from EEG with deep recurrent- ceuureet--onvvluutiona neural networks. ICLR, 2016.
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representa- tions. In International Conference on Machine Learning, pp. 552-560, 2013.
K. Brodersen, C. Ong, K. Stephan, and J. M. Buhmann. The balanced accuracy and its posterior distribution. In 2010 20th International Conference on Pattern Recognition, pp. 3121-3114 2010. doi: 10.1100//CPR.2000.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Christopher Brush, Peter Ehmann, Greg Hajcak, Edward A Selby, and Brandon L Alderman. Using multilevel modeling to examine blunted neural responses reward in major depression. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 3(12): 1 032-1039, 2018.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MoNET: Unsupervised scene decomposition and representation. arXiv preprint arXiv: /901.1111111 1390, 2019.
Henry Carrillo, Kay H Brodersen, and José A Castellanos. Probabilistic performance evaluation for multiclass classification using the posterior balanced accuracy. In ROBOT2013: First Iberian Robotics Conference, pp. 347-361. Springer, 2014.
Avshalom Caspi, Renate M Houts, Daniel W Belsky, Sidra Goldman-Melloo HonaLee Harrington, Salomon Israel, Madeline H Meier, Sandhya Ramrakha, Idan Shalev, Richie Poulton, et al. The p factor: one general psychopathology factor in the structure of psychiatric disorders? Clinical Psychological Science, 19-1377, 2014.
Hubert Cecotti and Axel Graser. Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3): 433-445. 2011.
Hubert Cecotti, Miguel P Eckstein, and Barry Giesbrecht. Single-trial classification of event-related potentials in rapid serial visual presentation tasks supervised spatial filtering, IEEE transactions Neural Networks and Learning Systems, 25(1) 25(11):2038) ):2030--0445 2014.
10

---

Published as a conference paper at ICLR 2021
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference Machine Learning, 2020a.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020b.
Arnaud Delorme and Scott Makeig. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component componnnnt analysis. Journal of Neuroscience Methods, 134(1): 9-21, 2004.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
DSM-V. Diagnostic and statistical manual of mental disorders. American Psychiatric Association, Washington, DC, 5 edition, 2013.
Sunny Duan, Loic Matthey. Andre Saraiva, Nicholas Watters, Christopher P Burgess, Alexander Lerch- ner, and Irina Higgins. Unsupervised model selection for variational disentangled representation learning. arXiv preprint /905: /2614, 2019.
Amr Farahat, Christoph Reichert, Catherine M Sweeney-Reet and Hermann Hinrichs. Convolu- tional neural networks for decoding of covert attention focus and saliency maps for EEG feature visualization. Journal of Neural Engineering, 16(6):066010, 2019.
Dan Foti, Doreen M Olvet, Daniel N Klein, and Greg Hajcak. Reduced electrocortical response to threatening faces in major depressive disorder. Depression and Anxiety, 27(9))813-820, 2010.
Eiko I Fried and Randolph M Nesse. Depression is not a consistent syndrome: an investigation of unique symptom patterns in the STAR* D study. Journal of Affective Disorders, 172:96-102, 2015.
Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A Engemann, Daniel Strohmeier, Christian Brodbeck, Lauri Parkkonen, and Matti S Hämäläinen. Mne software for processing meg and eeg data. Neuroimage, 66:446-460, 2014.
Gabriele Gratton, Michael GH Coles, and Emanuel Donchin. A new method for off-line removal of ocular artifact. Electreenceehaalggrapyy and Clinical Neurophysiology.555 55(4))468-448 1983.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et Bootstrap your own latent: A new approach self-supervised learning. arXiv preprint Stti:22006.0733: 2020.
Inan Güler and Elif Derya Übeyli. Multiclass support vector machines for EEG-signals classification. IEEE transactions on Information Technology in Biomedicine, 11(2):11--166, 2007.
Nihal Fatma Güler, Elif Derya Übeyli, and Inan Güler. Recurrent neural networks emploronsepplliinnnnnooooooon Lyapunov exponents for EEG signals classification. Expert Systems with Applications, 29(3):506-514, 2005.
Greg Hajcak, Julia Klawohn, and Alexandria Meyer. The utility event-eelated potentials in clinical psychology. Anmual Review of Clinical Psychology, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAEE Learning basic visual concepts with a constrained variational framework. ICLR, 265);6, 2017.
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerehner. SCAN: Learning hierarchical compositional visual concepts. ICIR, 2018.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 31 355568)):0044507 2006.
11

---

Published as a conference paper at ICLR 2021
Thomas Insel, Bruce Cuthbert, Marjorie Garvey, Robert Heinssen, Daniel S Pine, Kevin Quinn, Charles Sanislow, and Philip Wang. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders, 2010.
Tzyy-Ping Jung, Scott Makeig, Colin Humphries, Te- Won Lee, Martin Mckeown, Vicente Iragui, and Terrence Sejnowski. Removing electroencephalloraahii artifacts by blind source separation. Psychophysiology, (7(2))
2000.
Ioannis Karakis, Keith H Chiappa, Marta San Luciano, Kenneth C. Sassower, John W Stakes, and Andrew Cole. The utility of routine EEG in the diagnosis of sleep disordered breathing. J Clin Neurophysiol,, 29(4))333-33) 338, 2012.
Ronald C Kessler, Katherine A McGinngle, Marvin Swartz, Dan G Blazer, and Christopher Nelson. Sex and depression in the national comorbidity survey i: Lifetime prevalence, chronicity and recurrence. Journal of affective disorders, 1993.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. Proceedings of the 2nd International Conference on Learning Representations (ICLR),
Julia Klawohn, Kreshnik Burani, Alec Bruchnak, Nicholas Santopetro, and Greg Hajcak. Reduced neural response to reward and pleasant pictures independently relate to depression. Psychological Medicine, pp. 1-9, 2020.
Peter J Lang, Margaret M Bradley, and Bruce N Cuthbert, International affective picture system (IAPS): Technical manual and affective ratings. The Center for Research in Psychophysiology, University of Florida., 1:39-58, 2008.
Grace Y Lim, Wilson W Tam, Yanxia Lu, Cyrus S Ho, Melvyn W Zhang, and Roger Co. Prevalence of depression in the community from 30 countries between 1994 and 2014. Scientific Reports, 8(1): 1-10, 2018.
Jill Lobbestael, Maartje Leurgans, and Arnoud Arnzz. Inter-rater relibbiity of the Structured Clinical Interview for DSM-IV Axis disorders (SCID I) and Axis disorders (SCID II). Clinical psychology & psychotherapy, 18(1):75-79, 2011.
Rafael Lozano, Mohsen Naghavi, Kyle Foreman, Mohammad AlMazroa, and Ziad Memish. Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: systematic analysis for the Global Burden Disease Study 2010 (vol 380, pg 2095, 2012). The Lancet, 381:628-666 628, 2013.
Steven Luck. Event-related potentials. American Psychological Association, 2012.
Annmarie MacNamara, Roman Kotov, and Greg Hajcak. Diagnostic and symptom-based predictors of emotional processing in generalized anxiety disorder and major depressive disorder: An event-related potential study. Cognitive Therapy and Research, 40(3):275-285 2299, 2016.
Scott Makeig, Stefan Debener, Julie Onton, and Arnaud Delorme, Mining event-related brain dynamics. Trends in Cognitive Sciences, 8(5):204-210 2004.
Sally McManus, Howard Meltzer, Traolah Brugha, Paul E Bebbington, and Rachel Jenkins. Adult Psychiatric Morbidity in England 2007: results of household survey. NHS Information Centre for Health and Social Care. NHS Information Centre for Health and Social 2009. 2009.
Sally McManus, Paul E Bebbington, Rachel Jenkins, and Traolah Brugha. Mental health and wellbeing in England: Adult Psychiatric Morbidity Survey 2014. Leeds: NHS Digital, 2016.
Piotr Mirowski, Deepak Madhavan, Yann LeCun, and Ruben Kuzniecky. Classification of patterns of
EEG synchronization for seizure prediction. Clinical Neurophysiology, 120(11):1927-1940. 2009.
Germot Küiller-Putz. Reinhold Scherer, Clemens Brunner, Robert Leeb, and Gert Pfurtscheller. Better than random: a closer look on bei results. International Journal of Bioelectromagnetismm, 10 (ARTICLEE::2-55, 2008.
12

---

Published as a conference paper at ICLR 2021
Hugh Nolan, Robert Whelan, and Richard B Reilly. Faster: fully automated statistical thresholding for eeg artifact rejection. Journal of neuroscience methods, 192(1):152 162, 2010.
Hossein Parvar, Lauren Seulthorpe-Petley, Jason Satel, Rober Boshra, Ryan CN D' A'Ary, and Thomas P Trappenberg. Detection of event-related potentials in individual subjects using support vector machines. Brain Informatics, 2(1): 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint 0rXiv:/401.4085, 2014.
Yannick Roy, Hubert Banville, Isabela Albuquerque, Alexandre Gramfort, Tiago H Falk, and Jocelyn
Faubert. Deep learning- electroencephalograahy analysis: systematic review. Journal of neural engineering, (((5):051001,200
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. Nature, 523(6088) 1986.
SJM Smith. EEG in the diagnosis, classification, and management of patients with epilepsy. Journal of Neurology, Neurosurgery & Psychiatry, 76(suppl 2):i12-i17, 2005.
Amelia J Solon, Vernon J Lawher,, Jonathan O Touryan, Jonathan R McDaniel, Anthony J Ries, and Stephen M Gordon. Decoding P300 variability using convolutional neural networks. Frontiers in Human Neuroscience, 13:201, 2019.
Kees J Stam, Dénes LJ Tavy, Brechtje Jelles, Herbert AM Achtereekte, Joris Slaets, and Ruud WM Keunen. Non-linear dynamical analysis of multichannel EEG: clinical applications in dementia and Parkinson's 's disease. Brain Topography. 7(2):141-155) 1994.
Abdulhamit Subasi and M Ismail Gursoy. EEG signal classification using PCA, ICA, LDA and support vector machines. Expert Systems with Applications, (7(12))8559-8666 2010.
Ryota Tomioka, Kazuyuki Aihara, and Klaus-Robert Müller. Logistic regression for single trial EEG classification. In NeurlPS, pp. 1377-1384, 2007.
Theo Vos, Ryan M Barber, Jashua A Salomon, and Christopher J L Murray. Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990-2013; systematic analysis for the Global Burden of Disease Study, The Lancet, (86((995))7773--- 800, 2013.
Haofei Wang, Bertram Shi, and Yiwen Wang. Convolutional neural network for target face detection using single-trial EEG signal. 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 20008-2011. 2018.
Lilian A Weber, Andreea O Diaconescu, Christoph Mathys, André Schmidt, Michael Kometer, Franz Vollenweider, and Klaas E. Stephan. Ketamine affects prediction errors about statistical regularities: computational single-trial -erial analysis of the mismatch negativity. Journal of Neuroscience, 40(29): 5658-5668, 2020.
Anna Weinberg and Stewart
Shankman. Blunted reward processing in remitted melancholic depression. Clinical Psychological Science, 5(1)::4-25 2017.
Anna Weinberg, Greg Perlman, Roman Kotov, and Greg Hajcak. Depression and reduced neural response to emotional images: Distinction from anxiety, and importance of symptom dimensions and age of onset. Journal of abnormal psychology, 125(1):26, 2016.
Harvey A Whiteford, Louisa Degenhardt, Christopher J L Murray, and Theo Vos. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. The Lancet, 382(9904)::155-- 1586, 2013.
13


Reviews:
[{"review_id": "psZpJ79yp1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nPOST REVISION\n\nFollowing our discussions and taking the changes made to the manuscript into account, I have decided to increase my score and recommend acceptance. My concerns have been adequately addressed. I believe the paper has been clarified, results are more carefully evaluated and claims are sound. I believe that this work consitutes a well-selected application that addresses a relevant research question with important clinical implications. In my opinion, this deserves to be aknowledged and may be of interest to others in the ICLR audience. My hope is that this application stimulates more work in EEG-based machine learning and also encourages others not to shy away from difficult application domains such as psychiatry, where we really are in need of new solutions to old and - unfortunatley - quite persistent problems.\n\nThank you very much for your hard work!\n\n\n\n\n\n\n\n---------------------------------------------------------------------------\nThe authors propose a disentanglement approach (bVAE with/without SCAN) to obtain sparse and interpretable clinical features from EEG time series to aid clinical decisions and automatize EEG preprocessing. They obtain convincing disentangled representations that show some promise for future applications. \n\n In the following, I point out strengths and weaknesses, my recommendation and its justification as well as additional detailed comments for the authors. I hope you will find my comments helpful and constructive.\n\n---------------------------------------------------------------------------\nStrengths and weaknesses\n\nStrengths\n-\tProviding ML solutions for psychiatry is a very challenging problem, as psychiatric diseases are of a very complex nature and many crucial processes remain opaque at best. I believe that it is very important that the ML community starts to tackle these issues. I thank the authors for trying to solve this difficult real-world problem and not showcasing their approach on toy examples. Even though the overall accuracies are not high, I believe it is very important to show such an unbiased estimate of where we are at with ML applications in psychiatry. This merits recognition.\n-\tChoosing EEG data for applications in psychiatry is a very good choice given its feasibility, cost-efficiency and availability in a clinical setting. Often authors do not think carefully enough, whether the data they train their classifier on can actually be acquired in a real-world clinical setting.\n-\tI applaud the authors’ choice to emphasize interpretability in this healthcare setting. This is a commendable choice that is often not made in other healthcare applications. This is a crucial prerequisite to improve the chance of such an approach ending up in clinical practice as interpretability will be the precursor for acceptance by patients and medical staff.\n-\tIncluding study site prediction was a very commendable effort to assess geographic confounders, which is a major concern in recent psychiatric studies with the advent of more and more large scale multi-center datasets.\n\nWeaknesses:\n-\tDepression outcome label appears to have a proof-of-concept character, rather than addressing a more relevant clinical question (differential diagnosis for bipolar vs unipolar depression for example).\n-\tMisleading representation of results speaking to superiority of their approach (SCAN approach is much worse than LR combined with bVAE and VAE; therefore, I would advise a more cautious interpretation with respect to the SCAN results and being more precise in the abstract).\n-\tConfidence intervals for classification results are missing.\n-\tThe claim that their approach reduces ‘hand-engineering’ seems to be unsubstantiated given the description of their methodology.\n\n---------------------------------------------------------------------------\nRecommendation\n-\tOverall, I would like to see this paper accepted, should the authors agree to address the concerns raised above. Unfortunately, I cannot support acceptance as it stands, but I would increase my score, if my concerns have been addressed.\n\n---------------------------------------------------------------------------\nJustification of recommendation\n-\tI believe the research question addresses a very important problem in trying to identify sparse and interpretable clinical markers from EEG data. The method is well-selected to address this question and takes real-world clinical constraints into account (multi-center data, feasibility of data acquisition => EEG, interpretability of identified features => bVAE, uncertainty regarding clinical labels and about which features are required => transfer learning to new labels and unsupervised feature identification). Therefore, I believe this paper deserves to be accepted and represents a good example that hopefully will inspire others to tackle these challenging questions.\n-\tHowever, I cannot support acceptance as it stands, because of the weaknesses highlighted above. Most of the issues pertain to clarity of writing, over-interpretation (or at least misleading representation of the results) some missing information on methodology, and choice of the clinical outcome label. For more detailed elaboration on these points see below. \n\n---------------------------------------------------------------------------\nDetailed feedback\n\nAbstract\n-\tYou describe your method (bVAE and SCAN) followed by the sentence: “We demonstrate that our method is able to outperform the canonical hand-engineered baseline classification method on a number of factors, including participant age and depression diagnosis”. This gives the impression that the bVAE+SCAN approach is substantially superior to the hand-engineered baseline. However, in the results table this specific combination is only marginally better than baseline, while bVAE+LR or indeed AE+LR substantially outperform the baseline. This is misleading in the abstract and should also be discussed later on.\n\n\nIntroduction\n-\tYou make point to criticize the ‘hand-engineering’ of conventional EEG analysis (section 1, paragraph 2, p. 1-2). You fail to mention that automatic options for eye blink correction exist, for example PCA based methods like Berg & Scherg (1994): https://doi.org/10.1016/0013-4694(94)90094-9 Indeed fully automated pipelines do exist.\nFurthermore, from your methods description (2.1, EEG preprocessing, p. 4;  2.1 Autoencoder, last paragraph p. 4-5), it appears that you follow the conventional preprocessing and only then feed the EEG into the autoencoder, which is of course, in principle, fine, but I fail to see why you bring up the whole critique of the hand-engineered pipeline. Your approach does not seem to address this issue, or did I miss something? \n\nMethods\n-\tSome aspects of the methodological description of the analysis are missing. Please, state which software was used to perform the preprocessing. Were ICA-based eye blink correction results visually inspected? Furthermore, for the AE you state that preprocessed trialwise EEG data goes into it as input, for the bVAE this information is missing. Did you follow this approach here as well?\n-\tThe choice of your clinical classification label appears arbitrary and in my opinion does not address a clinically important question. Could you elaborate why you lumped these specific diagnoses together and not others? The axis I label is so broad that it is basically meaningless. What was the rationale for this analysis? I also assume that you classify depression and axis I vs. controls, is this correct? If so, this is not very relevant clinically. The challenge does not lie in determining, whether a person is healthy or has a mental disorder (clinicians usually can tell within a few minutes of conversation), but rather differential diagnosis or prognosis. A more clinically relevant comparison would be, for example, classifying MDD vs bipolar depression. This is a challenging clinical question, because both patient groups can present with depressive symptoms initially and only time tells which diagnose and also (importantly) medication is appropriate. Why did you not choose such a classification problem?\n-\tIn your description of the bVAE you state that each disentangled factor corresponds to an ‘interpretable’ transformation of the data. This is misleading. Interpretability is of course the goal of a reduced latent space, but by no means the default result. Whether a compression is indeed interpretable needs to be carefully assessed.\n-\tWas test data selected across subjects (other individuals to test) or within subjects (other parts of the data from the same individual, but all individuals included in the training data)? This is important information as the goal would be to generalize to unseen individuals rather than unseen data from the same individual.\n\nResults\n-\tPlease, add confidence intervals for balanced accuracy (for example by computing the posterior balanced accuracy: Broderson et al. 2010, https://ieeexplore.ieee.org/document/5597285/) otherwise it is impossible to assess the relative quality of the different classifiers. Statements such as: “the classification accuracy is still significantly higher for deep representations compared to the LPP” should be backed up by a statistical test of some sort. Along the same lines how was “significantly higher than chance“ assessed? By permutation tests, based on confidence intervals/Bayesian confidence intervals?\n-\tYou state: “This suggests that replacing the more manual canonical LPP pipeline with deep representation learning can allow for both better training data efficiency and a reduction in time that the (potentially vulnerable) participants have to spend in the lab by up to 37x,…” For this interpretation to hold, you would need to show, that you can train your classifier on a single trial, otherwise, you would still need all the data for pretraining to then be able to make predictions based on single trials. I would advice clarifying this.\n-\tLastly, I would like to state, that I was very impressed that you were able to re-discover the anhedonia neurocorrelates with your approach. I think the major challenge is indeed to make sure, that your bVAE identifies meaningful latent variables which I believe you successfully showed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VAg5rkJ55Fj", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe authors are concerned with the classification of EEG signals, in order to predict age, gender, depression and Axis 1 disorder diagonosis from EEG signals. After standard preprocessing and optionnal averaging to obtain evoked responses, the authors feed the samples into a $\\beta$-VAE , and then either use a standard classification algorithm or the SCAN method to predict the labels.\nThe authors report better results than the usual methods based on the late positive potential. They also show that their method can be trained with non-averaged EEG data and still yield good results when tested on ERP , and conversely. Finally, the authors inspect the learned representations.\n\nMajor comments\n\n- The paper is very well written, easy and pleasant to read, and well structured.\n- The automation of EEG pipelines, like this paper does, is extremely important.\n- The SCAN + $\\beta$-VAE or SCAN+ VAE method does not seem to perform much better than LR +LPP. Even though SCAN allows for interpretable components, it is arguably much less interpretable than LPP.\n- The article validates carefully  a machine learning pipeline on a specific task and dataset, but there is little contribution in terms of machine learning, so I'm wondering whether ICLR is a good fit for this paper, rather than a more neuroscience-oriented conference.\n\nMinor comments\n- The authors propose an original pipeline, yet the dataset and the code to reproduce the results are not provided, which hinders reproducibility and the potential impact of this work. \n- In my understanding, the LPP seems to only use one feature for classification: the average amplitude difference between waveforms. Could the authors also consider methods using more hand-crafted features? \n- The EEG signals are only acquired with 3 sensors, it would be interesting to add a word about how the method scales to datasets with more sensors. \n-  It would be interesting to add some ROC curves for the logistic regressions, which would complement nicely the summary statistic used by the authors.\n\nMisc. \n\n- The software used to perform the study should be acknowledged.\n- The ERP are normalized to [0, 1] before going in the VAE. The authors could be more accurate: is each channel normalized individually?\n- The references that state that EEG contains important biomarkers of clinical disorders, and that averaging trials yields ERP could also point to more historic papers.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "12VH4YIICeO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose a beta-VAE network to learn EEG representation as biomarkers for diagnosing depression from EEG data. They show improved performance compared to an off-the shelf linear classifier. The paper is well-written but lacks a description of related work in the field and also a detailed analysis of the results to support the claims. \n\nNovelty:  The use of VAE and beta-VAE for EEG data is not novel and this line of literature should be better discussed in the paper. \n\nA few more comments/questions for the authors:\n\n1. The details of the AE and beta-VAE architecture should be described in the main paper not the supplementary. Also, a detailed description of the number of trainable parameters and the amount of available data should be added to the main manuscript. \n\n2. On page 2, the authors mention that LDA and SVM are not commonly used in EEG literature for ERP classification, but these two are actually very common. For instance see: Blankertz, B., Lemm, S., Treder, M., Haufe, S., & Müller, K. R. (2011). Single-trial analysis and classification of ERP components—a tutorial. NeuroImage, 56(2), 814-825.\n\n3. The authors mention that the EEG “ground truth” markers are not available for a condition like depression, and yet they mention that their method provides interpretable biomarkers. There seems to be some discrepancy here that should be explained further. \n\n4. The authors mention that their paper is the first to use DL representation learning for clinical EEG biomarkers. This claim is not true, please see the following for a list of relevant papers: Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., & Faubert, J. (2019). Deep learning-based electroencephalography analysis: a systematic review. Journal of neural engineering, 16(5), 051001.\n\n5. The axis 1 label is rather confusing, Could the authors please explain it further and say how it is relevant for the depression diagnosis study? \n\n6. Did the authors do pre-processing (including ICA and removing eye-blinks) at test time? ICA needs a lot of EEG data to be able to reliably remove the eye blinks. That means a pre-trained beta-VAE would not be able to reduce training time. \n\n7. The authors do not mention which electrodes are selected for their analysis and why. \n\n8. It is not clear if LR is even significantly better than the other baseline classifier in table A2. Also, the authors should apply the rest of the baseline classifiers to the beta-VAE and AE representations. \n\n9. In table 1 and A2, the upper limit of the chance level should be calculated as a function of the available data as described in: Müller-Putz, G., Scherer, R., Brunner, C., Leeb, R., & Pfurtscheller, G. (2008). Better than random: a closer look on BCI results. International Journal of Bioelectromagnetism, 10(ARTICLE), 52-55.\nIn the same table, the overall chance is reported as 0.45. Why is that? \n\n10. How did the authors evaluate the “significantly better” performance of their proposed classifier? \n\n11. On page 7, how is the 37x less time need to be spent in the lab by vulnerable populations measured? Again, did the authors used ICA-cleaned data for this evaluation? \n\n12. Is the separation of the ERPs in Figure 3C significant? Please provide the original ERP for the depression and healthy subjects for comparison. \nAlso, the two colors are very hard to tell apart, please use another color combination such as blue and red/orange. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The approach is novel and according to the reviewers' comments addresses a relevant and important problem on EEG data analysis. Differences to related work are discussed. Methods and Experimental results are sound. The authors have provided a comprehensive response to the reviews.

Decision: Accept (Poster)