MBrain: A Multi-channel Self-Supervised Learning Framework for Brain Signals
Authors: Donghong Cai
Junru Chen
Yang Yang
Teng Liu
Yafeng Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: brain signals
self-supervised learning
multi-channel time series
seizure detection

Abstract: Brain signals are important quantitative data for understanding physiological activities and diseases of human brain. Meanwhile, rapidly developing deep learning methods offer a wide range of opportunities for better modeling brain signals, which has attracted considerable research efforts recently. Most existing studies pay attention to supervised learning methods, which, however, require high-cost clinical labels. In addition, the huge difference in the clinical patterns of brain signals measured by invasive (e.g., SEEG) and non-invasive (e.g., EEG) methods leads to the lack of a unified method. To handle the above issues, in this paper, we propose to study the self-supervised learning (SSL) framework for brain signals that can be applied to pre-train either SEEG or EEG data. Intuitively, brain signals, generated by the firing of neurons, are transmitted among different connecting structures in human brain. Inspired by this, we propose to learn implicit spatial and temporal correlations between different channels (i.e., contacts of the electrode, corresponding to different brain areas) as the cornerstone for uniformly modeling different types of brain signals. Specifically, we capture the temporal correlation by designing the delayed-time-shift prediction task; we represent the spatial correlation by a graph structure, which is built with the goal to maximize the mutual information of each channel and its correlated ones. We further theoretically prove that our design can lead to a better predictive representation. Extensive experiments of seizure detection on both EEG and SEEG large-scale real- world datasets demonstrate our model outperforms several state-of-the-art time series SSL and unsupervised models.

Under review as a conference paper at ICLR 2023
MBRAIN: A MULTI-CHANNEL SELF-SUPERVISED LEARNING FRAMEWORK FOR BRAIN SIGNALS
Anonymous authors Paper under double-blind review
ABSTRACT
Brain signals are important quantitative data for understanding physiological ac- tivities and diseases of human brain. Meanwhile, rapidly developing deep learn- ing methods offer wide range of opportunities for better modeling brain signals, which has attracted considerable research efforts recently. Most existing studies pay attention to supervised learning methods, which, however, require high-cost clinical labels. In addition, the huge difference in the clinical patterns of brain signals measured by invasive (e.g., SEEG) and non-invasive (e.g., EEG) methods leads to the lack of a unified method. To handle the above issues, in this paper, we propose to study the self-supervised learning (SSL) framework for brain sig- nals that can be applied to pre-train either SEEG or EEG data. Intuitively, brain signals, generated by the firing of neurons, are transmitted among different con- necting structures in human brain. Inspired by this, we propose to learn implicit spatial and temporal correlations between different channels (i.e., contacts of the electrode, corresponding to different brain areas) as the cornerstone for uniformly modeling different types of brain signals. Specifically, we capture the temporal correlation by designing the Selaaydd-ime--siee prediction task; we represent the spatial correlation by graph structure, which is built with proposed multi-channel CPC whose goal is to maximize the mutual information of each channel and its correlated ones. We further theoretically prove that our design can lead to a better predictive representation and propose the instantaneou-tie -iteeooo---i---iff prediction task based on it. Finally, aeplace-discrmiinae learning task is designed to preserve the characteristics of each channel. Extensive experiments of seizure detection both EEG and SEEG large-scale real-world datasets demonstrate model outperforms several state-of-the-an time series SSL and unsupervised models.
INTRODUCTION
Brain signals are foundational quantitative data for the study of human brain the field of neuro- science. The patterns of brain signals greatly help to understand the normal physiological function the brain and the mechanism of related diseases. There are many applications brain signals, such cognitive research (Ismail & Karwowski] 2020 Kuanar al., 2018)) emotion recog- nition (Song et al. 2020 Chen et al., 2019), neurological disorders (Alturki et al.] (2220) Yuan et al.) 2019) and on. Brain signals can be measured by noninvasive invasive methods (Paluszek et al. 2015). The noninvasive methods, like electroencephalograpah (EEG), cannot simultancously consider temporal and spatial resolution along with the deep brain information, but they are eas- ier to implement without any surgery. As for invasive methods like stereoclectnoencehhllgee (SEEG), they require extra surgeries to insert the recording devices, but have access to more precise and higher signal-to-noise data. For both EEG and SEEG data, there are multiple electrodes with contacts (also called channels) that are sampled at fixed frequency to record brain signals.
Recently, discoveries in the field of neuroscience have inspired advances of deep learning tech- niques, which in turn promotes neuroscience research. According to the literature, most deep learning-based studies of brain signals focus supervised learning (Shocibi et al.) 202 I Rasheed et al. 2020 Zhang et al.) 2021 Craik et al. 2019). which relies on a large number of clinical labels. However, obtaining accurate and reliable clinical labels requires high cost. In the meantime, the emergence of self-supervised learning (SSL) and its great success (Chen & He 2021; Grill et al.
2020 He et al.] 2020: Brown al. 2020 Devlin et al. 2018 Raffel et al. 2020 Van den Oord

---

Under review as a conference paper at ICLR 2023
et al. 2018) makes it a predominant learning paradigm in the absence of labels. Therefore, some recent studies have introduced the means of SSL to extract the representations of brain signal data. For example, [Bavville et al.] 202I) directly applies general SSL tasks to pre-train EEG data, in- cluding relative position prediction (Doersch et al., 2013), temporal shuffling (Misra et al. 2016) and contrastive predictive coding (Van den Oord al.] 2018). [Mhsennand al. (2020) designs data augmentation methods, and extends the self-supervised model SimCLR (Chen et al. 2020) in computer vision to EEG data. In contrast to the numerous works investigating EEG, few studies focus on SEEG data. Martini et al. (2021) proposes self-supervised learning model for real-time epilepsy monitoring multimodal scenarios with SEEG data and video recordings.
Despite the advances on representation learning of brain signals, two main issues remain to be overcome. Firstly, almost all existing methods are designed for a particular type of brain signal data, and there is lack of a unified method for handling both EEG and SEEG data. The challenge mainly lies in the different clinical patterns of brain signals that need to be measured in different ways. On the one hand, EEG collects noisy and rough brain signals on the scalp; differently, SEEG collects deeper signals with more stereo spatial information, which indicates more significant differences of different brain areas (Perucca et al. 2014). On the other hand, in contrast to EEG with a gold- standard collection location, the monitoring areas of SEEG vary greatly between patients, leading to different number and position of channels. Therefore, how to find the commonalities of EEG and SEEG data to design a unified framework is challenging.
Another issue is about the gap between existing methods and the real-world applications. In clinical scenarios, doctors typically locate brain lesions by analyzing signal patterns of each channel and their holistic correlations. straight-forward way for this goal is model each of the channels sep- arately by single-channel time series models, which, however, cannot exploit correlations between brain areas (Davis et al., 2020; Lynn Bassett 2019). As for the existing multivariable time series models, most of them can only capture implicit correlation patterns (Zerveas et al. 2021] Chen & Shi 202T), whereas explicit correlations are required by doctors for identifying lesions. Moreover, although some graph-based methods have been proposed to explicitly learn correlations, they focus on giving an overall prediction for all channels at a time but overlook the prediction on one specific channel (Zhang al. [0022] Shang et al. 2021). Therefore, how to explicitly capture the spatial and temporal correlations while giving thannel-wise prediction is another issue to be overcome.
In this paper, our main contribution is to propose multi channel self sulf-supervised learning framework MBrain, which can be generally applicd for learning representations of both EEG and SEEG data. In addition, we pay special attention to its application seizure detection. Based on domain knowledge and data observations, we propose to learn the correlation graph between channels as the common cornerstone for both two types of brain signals. In particular, we employ Contrastive Predictive Coding (CPC) (Van den Oord et al. 2018)) as the backbone model of our framework by extending it for handling multi-channel data with theoretically guaranteed efféctiivness. Based on the multi- channel CPC, we propose the instantaneous time shift task to learn the spatial correlations between channels, and the delayed time shift task and the replace discriminative task are designed to capture the temporal correlation patterns and to preserve the characteristics of each channel respectively. Extensive experiments show that MBrain outperforms several state-of-the-ar baselines on large- scale real-world EEG and SEEG datasets for the seizure detection task.
PRELIMINARY: THEORETICAL ANALYSIS OF MULTI-CHANNEL CPC
We employ Contrastive Predictive Coding (CPC) (Van den Oord et al. 2018) as the basis of our framework. 'Te pretext task of CPC is to predict low-level local representations by high-level global contextual representations C, at the f-th time step. Theoretically, the optimal InfoNCE loss proposed by CPC with N negative samples C is a lower bound of the mutual in- formation between contextual semantic distribution p(cx) and raw data distribution p(*++*), i.e., CTT > -[ cx) + log N, where k is the prediction step size. CPC is originally designed for single-channel sequence data only, and there are two natural ways to extend single channel CPC to multi-channel version. The first one is to use CNNs with multiple kernels to encode all channels simultaneously, which cannot offer explicit correlation patterns for doctors to identify lesions. The second one is to train shared CPC regarding all channels as one, which has ability to capture the correlation patterns. Taking a comprehensive consideration, we propose multi- channel CPC in this paper. Our motivation is to explicitly aggregate the semantic information of multiple channels
2

---

Under review as a conference paper at ICLR 2023
to predict the local representations of one channel. Formally, we propose the following proposition as our basic starting point.
Proposition 1. Introducing the contextual information of the correlated channels increases the amount of mutual information with the raw data of the target channel.
(1) where and j are indexes of the channels. ((-) represents some kinds of aggregate function, which has no additional formal constraints other than the need to retain information the target channel.
1 ( ((ce)) - I [ ce (14)))))) 2 21 (z)
Proof. We the linear operation of mutual information to obtain: 1 ct (((c)it))
[at ei) + 25 (((c) ) (())) According to the non-negativity of the conditional mutual information, we complete the proof.
It seems natural that the predictive ability of multiple channels is stronger than that of a single channel, which is also consistent with the assumption of Granger causality (Granger 1969) some extent. Therefore, we choose to approximate the more informative I [z]+++) ((())) obtain more expressive representations. Specifically, followed by InfoNCE, we define our loss function LN as
fu(Ie4k. ((c))) Enyex fe(1,, ((c)))
LN =
Σ Ex log
where X denotes the data sample set consisting of one positive sample and N 1 negative samples of the i-th channel. We then establish the relationship between LNN and [ (cc)).
Theorem 1. Given a sample set for each channel X {-f..... æ'y J. i 1..... . consisting of one positive sample from p(æt+ (cx)) and N 1 negative samples from Σ, p(x((())/m where n is the number of channels. The optimal _ is the lower bound of E, (())))
((c))):
CN Σ - 1 ( + No.
(3)
Proof. The optimal fa(x++k, (ct)) is proportional to p( (c))) p(z)) which is the same as single-channel CPC, And can directly replace the data distributions in the proof of single-channel CPC (see details in Appendix B) to obtain the inequality below:
Σ p(T(+))
[ Σ, p( P(
N
Σ
E x log
+ log N ExI,x x log
ln N.
p(r ((c.)
11,p(1 + (cx)))
(4)
According to the Jensen Inequality, we obtain that (E log log.oggpp// p(=x(++)))10) 1)/n s log p( p(2444)/m) By exponentiating the two equations, we have
1,p(s/+)) n EE(
(5)
With the help of equation
5 we can further obtain the lower bound of equation 4 H)p(a(+4)
CN Ex x log 11p(s/44 (((e))
+ +n
Σ [-I ((c))) + log No. (6)
Then we complete the proof.
We next analyze the advantages of multi-channel CPC over single-channel CPC. Our loss function CN leads to a better predictive representation because we approximate a more informative objective T( (bcx)), if the optimal loss function for each channel has log N gap with [ ((c))), which is the same in single-channel CPC. Moreover, with the same GPU memory, the more channels, the smaller the batch size that can be accommodated. But we can randomly sample negative samples across all channels, which increases the diversity of negative samples. However, in order to narrow the approximation gap, equation 5 should be considered. The equality sign in this inequality holds if and only if samples from each channel follows the same distribution. In fact, for many large- scale mulli-channe time series data (e.g., brain signal data used in this paper), by normalizing each channel, they all exhibit close normal distributions leading to small gaps in equation 5
3

---

Under review as a conference paper at ICLR 2023
3 PROPOSED METHOD
In this section, we introduce the details of our proposed self-supervised learning framework MBrain. For the commonality between EEG and SEEG, we are inspired by the synergistic effect of brain function and nerve cells, that is, different connectivity patterns correspond to different brain states (Lynn & Bassett 2019). In particular, for brain signals, nerve cells will spontaneously gener- ate traveling waves and spread them out (Davis et al.. 2020), maintaining some characteristics such as shape during the process. Therefore, the degree of channel similarity implies different propaga- tion patterns of traveling waves, reflecting the differences in connectivity patterns to some extent. Both EEG and SEEG brain signals follow the inherent physiological mechanism. Therefore, we pro- pose to extract the correlation graph structure between channels (brain areas) as the cornerstone of unifying EEG and SEEG data (Section B.T.. Next, we introduce three self-supervised learning tasks to model brain signals in Section 3.2 We propose instantaneous time shift task based on multi- channel CPC and delayed time shift task to capture the spatial and temporal correlation patterns. Then Replace discriminative task is further designed to preserve characteristies of each channel.
Notations. For both EEG and SEEG data, there are multiple electrodes with C channels. We use X == (an RCD ]= to represent raw time series data with L time points. and j denote the index of channels. Yı,- E {0, } is the label for the 1-th time point and i-th channel. We use a W-length window with no overlap to obtain the time segments S = { ||
(see details in Appendix A The label corresponding to the 1-th time segment and the i-th channel is denoted as Y2,
3.1 LEARNING CORRELATIONS BETWEEN CHANNELS
As mentioned above, the correlation patterns between different brain areas can help us to distin- guish brain activities in downstream tasks to large extent. Taking the seizure detection task as an example, when seizures occur, more rapid and significant propagation of spike-and -wave discharges will appear (Proix al. 2018), which greatly enhances the correlation between channels. This phe- nomenon is also verified by data observations in Appendix C which supports us to treat correlation graph structure learning as the common cornerstone of our framework. However, correlations be- tween brain regions are difficult to be observed and recorded directly. Therefore, for each time step t, our goal is learn the structure of the correlation graph, whose adjacency matrix is Ar, where nodes in the graph indicate channels and weighted edges denote the correlations between channels.
Considering that the brain is in normal and stable state most of the time, we first define the coarse- grained correlation graph as the prior graph for a particular individual as
A cnans (i,j) Es, [Cosine(sr..: st,;)]
where the expectation operation averages over all the correlation matrices computed in only one time segment 8t, and Cosine(.,.) denotes the cosine similarity function.
Next, based on A coarse, for each pair of channels, further model their fint- fite-grained short-term cor- relation within each time segment. We assume that the fine-grained correlations follow a Gaussian distribution element-wise whose location parameters are elements of A coarse and scale parameters will be learned from the data. By means of the reparameterization trick, the short-term correlation matrix of the t-th time segment sampled from the learned Gaussian distribution:
(i = SoftPlus self self no (i ~ (0.
(8)
(9)
A fine (i.j) A coarse (i,j) or. x ne(i,)).
(10)
SoftPlus() )) is commonly used activation function to ensure the learned standard deviation is posi- .self"
tive. c, is the contextual representation of raw time segments extracted by encoders (see details in
Section 3.2 To remove the spurious correlations caused by low frequency signals and enhance the
sparsity, which is a common assumption in neuroscience (Yu et al., 2017). we filter the edges by a threshold-basee function obtain the final correlation graph structure Ag;
A (i,
A fic (i, 2 01,
(i( =
(11)
0,
A fine (i,j) < 0
4

---

Under review as a conference paper at ICLR 2023
Jene
Task 3
Bur
Discriminator
Learning
Discriminative
Replace
replace
replaced?
Lg loss
sequancee
predict
L loss
Instantaneous Time Shift
Encoder
GNN
Learning
Correlations
faz
Task 1
+
sell
pooling
Delayed Time Shift
highly correlated?
Chamns
Task 2
Lg loss
Cosine
Overview of MBrain The leftmost is the raw multi-channel brain signals. We use an encoder to map the raw data into a low-dimensional representation space. To capture the spatial and temporal correlation patterns, propose three SSL tasks to guide the encoder to learn informative and distinguishable representations.
Figure 1:
3.2 SELF-SUPERVISED LEARNING TASKS FOR BRAIN SIGNALS
To capture the correlation patterns in space and time, propose two self self-supe tasks: in- stantaneous time shift that based on multi-channel CPC and captures the short-term correlations focusing on spatial patterns; and delayed time shift for temporal patterns broader time scales.
Instantaneous Time Shift. For spatial patterns, we aim to leverage the contextual information of correlated channels to better predict future data of the larget channel. Therefore, we apply multi- channel CPC and utilize the fine-grained graph structure A, obtained in Section 3.1 as the correla- tions between channels.
We first use a non-linear encoder Jene (ID-CNN with d kernels) mapping the observed time seg- ments to the local latent d-dimensional representations 2, (enc( E xxxxx for each channel separately. T is the sequential length after down sampling by Jenc. Then autoregressive model Jar is utilized to summarize the historical r-length local information of each channel itself to obtain the respective contextual representations:
c,
Far (2 2.
(12)
In this step, only extract the contextual information of all channels independently. Based on the graph structure A,, we instantiate the aggregate function Φ(-) in equation 4 as GNNs due to their natural paassig ability on a graph. Here we use one-layer directed GCN (Yun al. 2019) to show the process:
Ene Arli,) EiHA((i.))
self CT.J
other ReLU
0)
(13)
where is the learnable matrix. Considering that we only aggregate other channels' information, the self-loop in GCN is removed here. Finally, by combining both c:e and oother to obtain the global representations Ct,r. the model can predict the local representations -step away 21,++k, based on the multi-channel CPC loss:
Ct Concat self onher), (I,7) C
(14)
T Wey
C1 Cn -,, log E,, Ex,ex G.,, Wk,
(15)
where Xi denotes the random noise set including one positive sample 2t,+ and N negative samples. We, is the learnable bilinear score matrix of the k -th step prediction.
5

---

Under review as a conference paper at ICLR 2023
Delayed Time Shift. For brain areas far apart, there exists delayed brain signal propagation, which confirmed by the data observations showed in Appendix C We should consider these significant temporal correlations across several time steps in model.
Our motivation is that a simple classifier can easily predict whether two time segments are highly correlated, the segment representations will be significantly different from those with weaker cor- relations. We thus define the delayed time shift task to encourage more distinguishable segment representations. Similar with instantaneous time shift, we first compute the cosine similarity matrix based on raw data between time segments across several time steps. For the i-th channel in the 1-th time segment, the long-term correlation matrix B; is computed as
B2 j) = Cosine(st, St+kg.));
(16)
where j traverses all channels including the target channel and it2 traverses at most K₂ prediction steps. Then we construct pseudo labels Y, according to B; to encourage the segment representations with higher correlations to be closer. A predefined threshold 0g is set to assign pseudo labels:
(k2,,, 2 O2:
(k2, 0.
(17)
(k2,, j) < 0z.
With the pseudo labels, we define the cross entropy loss of the delayed time shift prediction task:
h Pooling(G .colf) ,G,f.
(18)
p- Softmax(MLP(CCnnat(Hi. hes he))))))))
(19)
C2 B (k2. log (1 (12. log(1 $)]
(20)
where p is the predicted probability that the two segments are highly correlated. In practical appli- cation, we randomly choose 50% labels from each Y for efficient training.
Replace Discriminative Learning. Consistently exploiting correlation for all channels will weaken the specificity between channels. However, there are sigmificant differences the physiological signal patterns of different brain areas recorded by channels. Therefore, retaining the characteristics of each channel cannot be ignored for the modeling of brain signals. For this purpose, we further design the replace discriminative learning task.
Following BERT (Devlin et al. 2018), we randomly replace r% local representations throughout 2, by 2. which is sampled from any 7 sequences and any C channels in We use the notation I(&) to represent the new local representations after replacement and the corresponding channel indexes of & in the original sequence. We generate pscudo labels Y, of the task as below:
(T,,,) = (0, f1,
I(2 +
(21)
I(2,r,))= =
and i traverse T sequences and C channels of 2, After obtaining â, we put it into the autoregres- sive model to get the new contextual representations é = gar(2), Finally, a simple discriminator implemented by MLP is utilized to classify whether & are replaced by other channels not:
La E,, [Yr(T, i) log q (1- (1-Y(T,, log(1 4)
(22)
where q is the predicted probability that é,.,, replaced. When the accuracy of discrimination increases, different channel representations output by the autoregressive model are easier to distin- guish. Therefore, the task preserves the unique characteristics of each channel.
Combining the multi-task loss functions equation 15 equation 20 and equation 22, we jointly train MBrain with C = (1 (1-A1- A₁ - -1 +C1+: A1 + A2 After the SSL stage, the segment representations ht obtained from equation [8 are used for downstream tasks.
4 EXPERIMENTS
4.1 DATASETS AND BASELINES
SEEG dataset. The SEEG dataset used in our experiment anonymized and provided by a first-
class hospital cooperate with. For patient suffering from epilepsy, 4 to 10 invasive electrodes

---

Under review as a conference paper at ICLR 2023
with 52 to 124 channels used for recording signals. It is worth noting that since SEEG data collected in a high frequency (1,000Hz or 2,000Hz) through multiple chamnels, our data is massive. In total, we have collected 470 hours of SEEG signals with total capacity of 550GB. Professional neurosurgeons help us label the epileptic segments for each channel. For the self-supervised learn- ing stage, randomly sample 5,000 10-second SEEG clips for training and validation. As for the downstream seizure detection task, we first obtain 5,000 sampled 10-second SEEG clips (80% for training and 20% for validation). For the testing, we sample another 2,550 10-second SEEG clips with positive and negative sample ratio of 1:50. We use a 1-second window to segment each clip without overlap and our target is to make predictions for all channels in each 1-second segment.
EEG dataset. We use the Temple University Hospital EEG Seizure Corpus (TUSZ) v1.5.2 (Shah) et al. 2018) as our EEG dataset. It is the largest public EEG seizure database, containing 5,612 EEG recordings, 3,050 annotated seizures from clinical recordings, and eight seizure types. We include 19 EEG channels in the standard 10-20 system. For experimental efficiency, we generate a smaller dataset from TUSZ. We randomly sample 3,000 12-second EEG clips for self-supervised learning- As for the downstream seizure detection task, we first obtain 3,000 sampled 12-second EEG clips (80% for training and 20% for validation). For the testing, we sample another 3,900 12-second EEG clips with positive and negative sample ratio of 1:10. It is worth noting that the labels of EEG data are coarse graine,, which means we only have the label of whether epilepsy occurs in an EEG clip.
Baselines. We compare MBrain with state-of-the-ant models including one supervised time ries classification model MiniRocket (Dempster et al. [02T] and several self-supervised and un- supervised models: CPC (Van den Oord et al. 2018)). SimCLR (Chen et al. (020), Triplet- Loss (T-Loss) (Franceschi et al. 2001)). Time Series Transformer (TST) (Zerveas et al. (021),
GTS (Shang et al. 20221) TS-TCC et al. 2021 and TS2Vee (Yue et al. 202 of the baselines Appendix
See details
4.2 EXPERIMENTAL SETUP
To demonstrate the effectiveness of MBrain, we first define the seizure detection task and perform the experiments on EEG and SEEG datasets respectively. To examine the generalization of our model, we further conduct experiments corresponding to two clinically feasible schemes on SEEG dataset. The ablation study, hyperparameter analysis, variance and more experimental results are showed in Appendix G H and [ We also show the case study of the correlation graph learned by Section 3.1 in Appendix to confirm the ability of identifying different brain states.
Task 1 (Seizure Detection). Given time-ordered set including Is consecutive time segments with the index of the first segment being lo: S = {Sto:...... Is models predict the labels Ye, for all time segments in S (i.c., t = to...., to + Is) and all channels in each segment (i.e., i = 1....... C).
Seizure detection experiment. In this experiment, we first perform the self-supervised learning of the model on unlabeled data. Then the segment representations learned by the model are used for downstream seizure detection task (see details in Appendix D). During the training phase of downstream task, the encoder of SSL model will be fine-tuned with a very low learning rate. For EEG data, there is overlap between the patients of training and testing sets. In addition, since EEG labels are coarse- graiied,,, the segment representations in each 12-second EEG clip are pooled to one representation and then be used for seizure detection. For SEEG data, considering the difference in the number and location of implanted electrodes for patients, we conduct experiments for each patient independently and report the average performance.
Transfer learning experiments. To meet practical clinical needs, we design two clinically feasible schemes for SEEG data. The first the domain generalization experiment, that is, training the model on data of existing patients and directly predicting data of unknown patients. More specif- ically, we follow the "3-1-1" setting, where 3 patients are used for training, patient used for validation and patient is used for testing. We conduct experiments for all combinations, pick up the best result for each patient, and report the average results over all patients. The second is the domain adaptation experiment (Motiaan et al. 2017). In this experiment, we first perform SSL on one patient (i.e., source domain) and then fine- tine-tuning is performed using partial labeled data from another patient (i.e,, target domain) in the testing set. Finally, we perform seizure detection the
7

---

Under review as a conference paper at ICLR 2023
remaining data of the target domain. We report the results 12 cross-domain scenarios covering all combinations of four patients with typical seizure patterns in the SEEG dataset.
4.3 RESULTS OF SEIZURE DETECTION EXPERIMENT
The average performance of seizure detection on the SEEG dataset are presented Table ! Since the positive-negative sample ratio of SEEG dataset is imbalanced, F-score is metric to evaluate the performance of models than only considering precision or recall. Especially in clinical applications, doctors pay more attention to finding as much seizures as possible, we thus choose Fsore and Fccoe in the experiment. Overall, MBrain improves the Fi Ficrc by 28.92% and the Fcos by 26.85% on SEEG dataset compared the best result of baseline methods, demonstrating that MBrain can learn informative representations from SEEG data.
appropriate
Table I: The average performance of the seizure detection experiment SEEG and EEG datasets.
SEEG
EEG
Models
Pre.
Rec.
Fi
F2
Pre.
Rec.
F1
F2
AUROC
MiniRocket
22.98
66.24
31.79
43.58
22.86
63.08
33.56
46.66
75.30
CPC
27.65
55.07
34.20
42.73
22.81
58.31
32.50
44.02
74.53
SimCLR
11.06
51.54
16.60
25.41
12.63
74.88
21.33
36.78
55.86
T-Loss
29.29
51.55
36.00
43.13
20.72
69.25
31.82
47.00
75.88
TST
13.60
44.65
19.80
28.41
15.65
28.59
19.65
23.87
58.20
GTS
24.29
40.39
29.16
34.17
18.86
62.51
28.88
42.54
71.69
TS-TCC
22.10
49.94
25.32
32.74
15.55
39.76
21.89
29.60
58.63
TS2Vec
30.56
52.83
36.03
43.35
21.40
58.31
31.24
43.24
73.35
MBrain 37.97
65.07
46.45
55.28
22.13
76.99
34.32
51.34
77.96
Table |ass shows the results of seizure detection experiment on EEG dataset. Following the com- evaluation scheme EEG dataset (Tang et al.) 2022), also add Area Under the Receiver Operating Characteristic (AUROC) metric in our experiment. Our model is designed to learn the representation for each channel, while there only one label for an EEG clip. Therefore, requires the pooling operation to aggregate representations output by our model over channels and time seg- ments for seizure detection. This setting makes the performance improvement of our model not as significant as that in the SEEG experiment. Nevertheless, MBrain still outperforms all baselines on Fi -corr,, F2-score and AUROC with an increase of 2.26%, 9.23% and 2.74%, respectively.
Table 2: The average performance the domain generalization experiment on SEEG dataset.
Models
Pre.
Rec.
Fi
Fg
CPC
22.88=5.06
23.9243.90
20.11
21.23:+2.49
T-Loss
21.38-14.25
28.5044.007
23.48+3.30
25.9043.006
TS2Vec
27.93+5.23
29.4943.97
26.7813.22
27.8843.52
MBrain
30.69-5.92
38.94114.4
32.6113.60
35.6443.04
4.4 RESULTS OF DOMAIN GENERALIZZATION EXPERIMENT
This because, to some extent, the results in Table represent upper bound on the performance of these models. We point out that although GTS and MBrain are both graph- models, GTS cannot be trained on multiple patients, since the DCRNN (Li et al. 2018) encoder used in GTS can only process graphs with fixed nodes. contrast, our model is designed to learn the correlations between each pair of nodes and utilizes inductive GNN, it easily handle graphs with different numbers of input nodes. In general, the performance of models under the domain generalization setting decreases significantly (40.37% average in terms of F2-score) compared with that in epilepsy detection
In this experiment, we compare the baseline models that perform well in Table T
8

---

Under review as a conference paper at ICLR 2023
experiment. The drop for recall metric is more pronounced, confirming that the distribution shift of patients in SEEG data is more significant than that in EEG. This results from the fact that differ- ent brain regions and different types of epileptic waves have different physiological properties and patterns. MBrain in this experiment still improves F Fi-score and F2-score by 21.77% and 27.83% respectively, compared the best baseline. The results prove that MBrain has superior general- ization ability benefiting from rational inductive assumption of model design.
The performance of the domain adaptation experiment on SEEG dataset in terms of F2- score. DA row denotes the performance of MBrain in the domain adaptation experiment setting, Max-base and Non-DA rows represent the best performance of baselines and Mrainn in the seizure detection experiment. Bold numbers and indicate the best and the second best performance.
Table 3:
Group A
Group B
Group C
Group D
Setting
B-> C-*A D-+A A-BB C-B D-B A-C B-C D-C A-D B B-DD C-++
DA
68.55 69.14* 68.78
41.08
46.06
46.12* 40.04
39.34
48.64
80.82*
79.90 80.72 75.35
Max-base
62.49
39.78
33.59
Non-DA 70.63
46.62
46.09*
83.27
4.5 RESULTS OF DOMAIN ADAPTATION EXPERIMENT
According to the results of domain generalization experiment, it is difficult for MBrain to achieve competitive performance with Table T Due to the long record, clinical SEEG data contains tens even hundreds of seizures, allowing predict the remaining data. Therefore, as a clinical alternative to domain generalization, we con- duct compromise domain adaptation experiment. Table 3 shows the performance of the domain adaptation (DA) experiment for four patients with typical seizure patterns provided by doctors from SEEG dataset. More specifically, we train MBrain on one patient and fine-tune on all other three patients. B-A denotes that the SSL model is trained on Patient-B and then performs seizure de- tection experiment with the SSL, model being fine-tuned on Patient-A. The results of "Max-base" and "No rows correspond to the performance of the best baseline and MBrain respectively in scenarios A-A, BB C-CC and D-+D
model to
subset of data to fine-tune and then to
Compared with the results in the condition that the self-supervised model and downstream model are both trained on the same patient, the F2-scores all 12 cross-domain scenarios reduce by less than 15% Additionally, it can be observed that in all cross-domain scenarios, MBrain beats the best baseline in the corresponding scenarios without domain adaptation. It is worth noting that D-C' scenario outperforms corresponding "Non-DA" result. The possible reason is that the signal patterns on Patient-D are more significant and recognizable than those on Patient-C. Therefore, the SSL model trained on higher quality source domain better distinguish signal states when performing downstream tasks on target domain. Overall, the domain adaptation experiment makes MBrain achieve competitive performance with Table fine-tuning domain. The results suggest that MBrain captures the inherent features and outputs generalized representations between patients, because we fine-tune the SSL model with a very low learning rate (1c-6). From the perspective of pre-training, the SSL model trained on the source patient gives good initial parameters for the fine-tuning stage on the target patient.
only a subset of the target
5 CONCLUSION AND DISCUSSION
In this paper, we propose a general multi-channel SSL framework MBrain, which can be applied for learning representations of both EEG and SEEG brain signals. Based on domain knowledge and data observations, we succeed to use the correlation graph between channels as the cornerstone of our model. The proposed instantaneous and delayed time shift tasks help us capture the correlation patterns of brain signals spatially and temporally. Extensive experiments of seizure detection on large-scale real-world datasets demonstrate the superior performance and generalization ability of MBrain. However, there are still some limitations of our work. For example, negative sampling of multi-channel CPC consumes certain memory and time. Besides, determine the time range of long-tec temporal patterns. As for the future work, we plan to collect more types of brain signals and extend MBrain to more downstream tasks.
lack more automatic way to

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
We provide the source code of our model MBrain in the Supplementary Material. Some implemen- tation details of MBrain can be found Appendix [Dand default hyperparameters can be found in the code. Users can run MBrain on their own datasets, following the same generation method of database mentioned in Section
7 ETHICS STATEMENT
This paper proposes a novel self-supervised learning framework MBrain for brain signals, and con- duct experiments on real-world large-scale EEG and SEEG datasets. The EEG dataset is public and SEEG dataset is non-public but anonymous. Overall, this work inherits some of the risks of the ex- isting works implementing the EEG dataset and does not introduce any new ethical or future social concerns for SEEG dataset.
REFERENCES
Fahd A. Alturki, Khalil AlSharabi, Akram M. Abdurraqeeb, and Majid Aljalal. Eeg signal analysis for diagnosing neurological disorders using discrete wavelet transform and intelligent techniques. Sensors, 20(9), 2020.
Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time ries classification bake off: A review and experimental evaluation of recent algorithmic advances. Data mining and knowledge discovery, (1(3):606-66) 660, 2017.
Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexande Engemann, and Alexandre Gramfort, Uncovering the structure of clinical EEG signals with self-supervised learning. Journal of Neural Engineering, 18(4):046020, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish. Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. NIPS, pp. 1877-1901, 2020.
J. X. Chen, P. W. Zhang, Z. J. Mao, Y. F. Huang. D. M. Jiang, and Y. N. Zhang. Accurate ceg- based emotion recognition on combined features using deep convolutional neural networks. IEEE Access, 7:44317-4433 443288. 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, pp. 1597-1607, 2020.
Wei Chen and Ke Shi. Multi-scale attention convolutional neural network for time series classifica- tion. Neural Networks, 136:126-144 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pp. 15745-15753, 2021.
Alexander Craik, Yongtian He, and Jose L Centreeas-Vidal Deep learning for electroenceehallgramm (eeg) classification tasks: a review. Journal of neural engineering, 16(3):031001, 2019.
Zachary W Davis, Lyle Muller, Julio Martinez- Trujillo, Terrence Sejnowski, and John Reynolds. Spontaneous travelling cortical waves gate perception in behaving primates. Nature, 587(7834): 432-436, 2020.
Angus Dempster, François Petitjean, and Geoffrey Webb. Rocket: exceptionally fast and accu- rate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 3455): 2020.
10

---

Under review as a conference paper at ICLR 2023
Angus Dempster, Daniel F Schmidt, and Geoffrey I Webb. Minirocket: A very fast (almost) deter- ministic transform for time series classification. KDD, pp. 248-234822 257, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini arXiv: /0X000880000405, 2018.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In ICCV, pp. 1422-1430, 2015.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, In IJCAL, pp. 2352-2359, 2021.
Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In NIPS, 2019.
Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: journal of the Econometric Society, pp. 424-438, 1969.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your latent new approach to self-supervised learning. In NIPS, pp. 21271-21284, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9726-9735, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735- 80, 1997.
Lina Elsherif Ismail and Waldemar Karwowski. Applications of eeg indices for the quantification of human cognitive performance: A systematic review and bibliometric analysis. PloS one, 15(12): c0242857, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Shiba Kuanar, Vassilis Athitsos, Nityananda Pradhan, Arabinda Mishra, and K.R. Rao. Cognitive analysis of working memory load from eeg, by deep recurrent neural network. In ICASSP, pp. 2576-2580, 2018
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In ICLR, 2018.
Christopher W Lynn and Danielle S Bassett. The physies of brain network structure, function and control. Nature Reviews Physics, 1(5)3318-32,, 2019.
Michael L Martini, Aly A Valliani, Claire Sun, Anthony B Costa, Shan Zhao, Fedor Panov, Saadi Ghatan, Kanaka Rajan, and Eric Karl Oermann. Deep anomaly detection of seizures with paired stereclectteenchhallgrahyy and video recordings. Scientific Reports, 11(1):1-1,, 2021.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, pp. 527-544. Springer, 2016.
Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. Contrastive representation
learning for electreencephalogram classification. In PMLR, pp. 238-253, 2020.
Saeid Motiaan, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. In ICCV, pp. 5715-5725, 2017.
M. Paluszek, D. Avirovik, Y. Zhou, S. Kundu. A. Chopra, R. Montague, and S. Priya. 11 mag- netoelectric composites for medical application. In Composite Magnetoelectrrics pp. 297-327. Woodhead Publishing, 2015.
11

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Piero Perucca, François Dubeau, and Jean Gotman. Intracranial electroenceephalograpiic seizure- onset patterns: effect of underlying pathology. Brain, 137(1):183196, 2014.
Timothée Proix, Viktor K Jirsa, Fabrice Bartolomei, Maxime Guye, and Wilson Truccolo. Predict- ing the spatiotemporal diversity of seizure propagation and termination in human focal epilepsy. Nature communications, 91):11 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140):1-67, 2020 2020.
Khansa Rasheed, Adnan Qayyum, Junaid Qadir, Shobi Sivathamboo, Patrick Kwan, Levin Kuhlmann, Terence O'Brien, and Adeel Razi. Machine learning for predicting epileptic seizures using ceg signals: review. IEEE Reviews in Biomedical Engineering, 14:139-155, 2020.
Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 29(6):1505-111 1530, 2015.
Vinit Shah, Eva Von Weltin, Silvia Lopez, James Riley McHugh, Lillian Veloso, Meysam Gol- mohammadi, Iyad Obeid, and Joseph Picone. The temple university hospital seizure detection corpus. Frontiers in Neuroinformatics, 12:83, 2018.
Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. In ICLR, pp. 1-14, 2021.
Afshin Shoeibi, Marjane Khodatars, Navid Ghassemi, Mahboobeh Jafari, Parisa Moridian, Roohal- lah Alizadehsani, Maryam Panahiazar, Fahime Khozeimeh, Assef Zare, Hossein Hosseini-Nejad, et al. Epileptic seizures detection using deep learning techniques: a review. International Journal of Environmental Research and Public Health, 18(1 18111),,780, 2021.
Tengfei Song, Wenming Zheng, Peng Song, and Zhen Cui. Eeg emotion recognition using dy- namical graph convolutional neural networks. IEEE Transactions Affective Computing, 11(3): 532-541, 2020.
Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I Webb. Monash university, uea, ucr time series regression archive. arXiv preprint arXiv:2006. 200.::00006/09966, 2020.
Siyi Tang, Jared Dunnmon, Khaled Kamal Saab, Xuan Zhang, Qianying Huang, Florian Dubost, Daniel Rubin, and Christopher Lee-Messer. Self-supervised graph neural networks for improved electroencephaalggapiic seizure analysis. In ICLR, 2022.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv e-prints, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Renping Yu, Han Zhang, Le An, Xiaobo Chen, Zhihui Wei, and Dinggang Shen. Connectivity werength-weiihhted sparse group representation--base brain network construction for ci classifi- cation. Human brain mapping, 38(5):2370-2383, 2017.
Ye Yuan, Guangxu Xun, Kebin Jia, and Aidong Zhang. A multi-view deep learning framework for eeg seizure detection. IEEE Journal of Biomedical and Health Informatics, 23(1) (3(1):83994) 2019.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. arXiv preprint arXiv:2106. 10466, 2021.
Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph trans- former networks. In NIPS, 2019.
12

---

Under review as a conference paper at ICLR 2023
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer- based framework for multivariate time series representation learning. In KDD, pp. 21 2114-2124, 2021. 2021.
Xiang Zhang, Lina Yao, Xianzhi Wang, Jessica Monaghan, David Mcalpine, and Yu Zhang. A survey on deep leaming-based non-invasive brain signals: recent advances and new frontiers. Journal of Neural Engineering, 18(3):031002 2021.
Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In ICLR, 2022. URLhttts:///ppereeee
met/forund cutt/forum?ice Simmmmidddwm11@oomm Xwm&I7d0-13
Zhe Zhu. Change detection using landsat time series: review of frequencies, preprocessing, algorithms, and applications. ISPRS Journal of Photogrammetry and Remote Sensing, 130:370- 384, 2017.
13

---

Under review as a conference paper at ICLR 2023
A PRELIMINARIES
Brain signal data. For both EEG and SEEG data, there are multiple electrodes with contacts that are sampled at a fixed frequency to record the brain signals. We also call these contacts channels. For every sampling point, each channel records the potential value of the brain region in which they are located, constituting abstract multi-channel time series data. A complete record file contains a total of time points, for which we use the notation X {2) E RR) to represent. In reminder of this paper, we use and j to denote the index of channels, such as i = { For every 2i, we assign binary label Yi, E { to it according to the start and end time of epileptic brain signals marked by doctors. The time points are in the epileptic state with positive labels (Y 1), while zero labels (Yi,, 0) represent the normal data.
Preprocessing. Following the existing time series works (Zuu 2017 Bagnall et al., 2017 Schäter 2015) with the common preprocessing of segmentation, we use a W-length window divide the original brain signal data X into time segments S = {s, E R gWxc number of segments |S| = L/WJ. The segment label is obtained from the time points of the whole segment, i..., Y max 1ixw+1, -1,i.... ((t+))x..).
1t=1 without overlapping. The
B SINGLE-CHANNER CPC
Contrastive Predictive Coding (CPC), a pioneering model for self-supervised contrastive learning, sets the pretext task to predict low-level local representations by high-level global contextual infor- mation Gz. In this way, the model can avoid learning too many details of the raw data and pay more attention to the contextual semantic information of the sequence data. The InfoNCE loss proposed in CPC has become the basic design of the contrastive learning loss function. Specifically, given raw data sample set X {Ii...., IN consisting of one positive sample from p(#(+a|c)) (c)) and N 1 negative samples from the noisy distribution p(=+=*), InfoNCE will optimize:
fa fx( Σx,ex fa (z)
LN - Ex log
(23)
In order to obtain the best classification probability of the positive sample with the cross entropy loss function, the optimal fu ( c²) is proportional to p(x++x/c)) (1tt+=//l//p((ttt/ttt p(x+)))).. Furthermore, the optimal loss function is also closely related to mutual information, as below:
p(Fc=xle)/p(xxt)
CN
Ex
log
((0ctulce//((04t)) + Ea,exo, p(zylc)/p(zz)
p(a++*) N plac+* ct)
2 Ex
log
(24)
--((2++i;)) log
(25)
Therefore, we can conclude that while minimizing the loss function Lx. we are also constantly approximating mutual information of raw data distribution p(#++x) and contextual semantic distribution p(cz). It turns out that InfoNCE is indeed well-established loss function designed for self-supervised contrastive learning.
C DATA OBSERVATIONS
As Figure shows, for both EEG and SEEG data, we observe that the correlation matrices almost the same on two normal segments without overlapping in the same patient. To the opposite, the correlation matrix in the epileptic states differs from the normal ones greatly. These data ob- servations verify the conclusion that correlation patterns can help us to distinguish different brain states, and support to the correlation matrix the cornertoone of EEG and SEEG data.
14

---

Under review as a conference paper at ICLR 2023
Correlations of normal data
Normal correlation matrix after substraction
Seizure correlation matrix after substraction
4.3
SEEG
0.0
-42
EEG
1101 1533333
The normal seizure correlation matrices EEG and SEEG brain signals. The top row is for SEEG and the bottom row is for EEG. For clear presentation, SEEG data. The leftmost column including two figures are the base correlation matrices on normal data. The two figures in the middle column represent the matrices after subtracting another normal correlation matrices from the base matrices, and the rightmost column includes matrices after subtracting seizure correlation matrices from the base matrices.
Figure 2:
sample some channels
Correlation matrices delayed time shift (SEEG)
0330
Temporal steps
-0015
1.33
+
-025
-033
Channels
Figure 3: The correlation matrices of delayed time shift SEEG data. The top figure shows the average correlation matrix over all 10-second SEEG clips of particular patient. And the bottom figure represents the correlation matrix of one particular sampled 10-second clip. We compute the cosine similarity matrix between the first time segment in the clip of the first channel and the time segments of other channels in the next consecutive 7 time steps. For clear presentation, we sample 26 channels and set correlations below 0.5 to 0 for the bottom figure.
The data observations showed in Figure and Figure 4 confirm that there still exist significant relations between time segments across several time steps. Unlike instantaneous time shift, delayed correlations stable. This be concluded from the numerical difference between the aver-
15

---

Under review as a conference paper at ICLR 2023
Correlation matrices of delayed time shift (EEG)
6/42
6.55
0.36
6.22
Tempora steps
C28
64
-02
-04
-00
Channels
Figure 4: The correlation matrices of delayed time shift of EEG data. The top figure shows the averaged correlation matrix over all 12-second EEG clips, And the bottom figure represents the correlation matrix of one particular 12-second clip which randomly sampled. The computation and operation are the same as Figure 3
aged correlation matrix and the one-clip correlation matrix in both figures. Therefore, we design self-supervised task different from that in the instantancous time shift to learn the delayed correla- tions.
D IMPLEMENTATION DETAILS OF MBrain
The non-linear encoder Jene used in MBrain is composed of three 1-D convolution layers and one-layer LSTM model (Hochreiter & Schmidhuber 1997 is used as the autogressive model Jar- The model optimized using Adam optimizer (Kingma Ba 2013) with a learning rate of 2c-4 and weight decay of le-6 for the self-supervised learning stage. And for the downstream seizure detection stage, the downstream model is optimized with learning rate of 5e-4 and weight decay of le-6 while the SSL model is fine-tuned with a low learning rate of 1e-6. For the hyperparameters of MBrain, we set 0, 0.5 and 0g 0.5. We set the maximum value k in instantaneous time set K₂ 7
shift task as 8. As Figure 5 shows, most significant correlation in delayed time shift task. Lastly, we build our model using Py PoTocc 1.8 (Paszke et al.] 2019) and trained it workstation with four NVIDIA TESLA T4 GPUs.
to take into account the step with the
For the downstream task, we first utilize an LSTM model (Hochreiter & Schmidhuber 1997) to encode the segment representations of each channel in chronological order (10-second in SEEG clips and 12-second in EEG clips) independently. One-layer self- attention (Vaswani et al.) 2017) is then adopted to all channels within the same time step. Finally, two-layer MLP classifier used to predict whether epilepsy occurring in the time segments. All baselines share the same downstream model our experiments.
E IMPLEMENTATION DETAILS OF BASELINES
MiniRocket (Dempster al. 2021): ): Rocket (Dempster al. 2020 is state of the art supervised time series classification method based on evaluations on public benchmarks (Bagnall et al. 2017] Tan et al. 2020). involves training a linear classifier top of features extracted by flat collection numerous and various random convolutional kernels. MiniRocket is a variant Rocket which improves processing time, while offering essentially the same accuracy. We uese the open source
16

---

Under review as a conference paper at ICLR 2023
Analysis of hyperparameter K₂
0.010
Cosine correlations
0.005
0.000
-0.010
2
3
4
9
Temporal steps
Figure 5: The data observation of how to choose hyperparameter K2. We first average the correla- tions of all channels of each channel in each time step. Then we average those of all channels in the same time step. As the figure shows, we empirically choose K2 7
code from htom/ httpg/ghhw....o//nnng/////nnnnno For each subject, we use the features obtained through MiniRocket to train independent logistic regression classifier for each channel and test it on the test set that channel.
CPC (Van den Oord et al. 2018): This a sulf-supervie learning method based on a contrastive loss InfoNCE. The pretext task of CPC is set to predict future local low-level representations obtained from multi-layer CNNs by contextual high-level representations obtained from an au- toregressive model. This is the backbone model in this paper. We use the open source code of the corrected version from
SimCLR (Chen et al. 2020): This is simple yet effective framework for contrastive learning of visual representations and we use time-series specific augmentations to adapt our application. We implemented SimCLR on time series data by ourselves. We use the same encoder architecture and parameter configuration as TS-TCC. In the meantime, we also follow TS-TCC and use scaling (sigma=1.1) as the data augmentation way.
Triplet-Loss (T-Loss) (Franceschi et al. 2019): The approach employs time- based negative sampling and a triplet loss to learn representations for time series segments. We use the de- fault model architecture from the source code provided by the author hhttps://itthub.com/ chite- wwwaaaannnnnnnnirreeeeeere TimeSeries). For the sampling method of neg- the data of the previous batch as the candidate set of negative samples of the current batch data (the negative sample candidate set for the first batch is itself). Since the dataloader is shuffled the end of each epoch, there is no need worry about the case where the set of sampled negative samples does not change.
ative samples,
Time Series Transformer (TST) (eeveas et al. 2021): This a unsupervised repre- sentation learning framework for multivariate time series by training a transformer model to extract dense vector representations of time series through an input denoising objec- tive. We use the default model architecture from the source code provided by the author http:///itbubb.oomgggeeeaaaaeeee
transformer).
GTS (Shang et al. 2021): This is a time series forecasting model that learns a graph structure among multiple time series and forecasts them simultaneously with GNN. In view of this, this model can learn useful representations from unlabeled time series data. We use the default model architecture from the source code provided by the author In the pre-training stage, divide each time series segment into 10 parts average, and learn time series forecasting model that predicts the next steps based on the previous steps. In downstream task stage, series segment for the seizure detection task.
the representation after step 10 the representation of the time
TS-TCC (Eldele et al. 202T): This is an unsupervised time-series representation learning frame- work, applying a temporal contrasting module and contextual contrasting module to learn robust
17

---

Under review as a conference paper at ICLR 2023
and discriminative representations. We the default model architecture from the open source code provided by the author
TS2Vec Yue et al. 2021): This a universal representation learning framework for time series, that applies hierarchical contrasting learn scale-invariant representations within augmented con- text views. We use the default model architecture from the source code provided by the author
F
CASE STUDY
(a) Normal correlation graph.
(b) Scizure correlation graph.
Figure 6: Case study on correlation graphs learned by MBrain.
In this section, study the correlation graphs between the channels learned by MBrain. We ran- domly sample normal and seizure SEEG clips of one particular patient, and visualize their cor- relation graphs. The correlation graphs showed in Figure 6 we use the threshold 8₂ defined in Section 3.1 for the preservation of the edges. In addition, the thickness of an edge represents the edge weight, and the size of a node represents the sum the edge weights of all the edges linked to that node. It can be observed that during the normal phase, the correlation is sparser and the weights between edges are smaller, indicating a weaker correlation between channels. contrast, during the seizure phase, the pattern between channels varies, with the correlation becomes denser and the edge weights become larger. Furthermore, in the correlation graph of the seizure phase, edges with larger weights usually connected to 2 seizure channels, like Channel-2, Channel-35 and Channel-38 in Figure 6(b) which can help surgeons to better localize the seizure lesions.
G ABLATION STUDY
We study the effectiveness of each component in MBrain. Specifically, we compare MBrain with different model variants removing following different components. We firstly remove the corre- lation graph structure learning module from the instantancous time shift task and degenerate the task to single-channel CPC while still uniformly sampling negative samples in all channels. This variant is denoted as "MBrain Graph".. Next, we respectively remove the whole instantaneous time shift task, the delayed time shift task and replace discriminative task. These model variants de- noted as "MBrain- Instantt" "MBrain- -Dela" and "MBrain-Ree Replace"" Finally, we consider the con- dition of preserving only self-supervised task. "MBrain-onlyInstaan" "Mirain--nly and
18

---

We can observe that the performance "CPC-Conv" decreases dramatically. We speculate that this is because the channels are relatively independent, and the correlation between most channels weak or even non-existent. Direct adoption of multi-channel convolution may introduce spurious and noisy correlations. However, the graph structure learning proposed by has sparsity assump- tion, and the representation extraction of each channel relatively independent, so it can effectively MLP aggregate Under review as a conference paper at ICLR 2023
"MBrain-onie -onBrrainoollReplacee indicate that MBrain only performs instantaneous time shift task, delayed time shift task and replace discriminative task respectively.
We repeat the experiments five times in the fine-tuning stage of the downstream seizure detection shows the results of ablation study on SEEG dataset. It can be observed that the com- F₃ and F2 metrics, demonstrating the effectiveness
task.
plete MBrain achieves the best performance of each component our model design. For "MBrain-Instamt", the significant decrease in perfor- mance illustrates that capturing the spatial and short-term patterns is quite important and is the key to learning the essential representations in multi-channel brain signals. For "MBrain-Grapee Graph" the decrease in performance demonstrates that multi-channel CPC can greatly help learn more infor- mative representations. Additionally, the performance in EiBrain--eelay and Ce-Brain-Repacee also decreases significantly, illustrating that modeling long-term temporal patterns and preserving the characteristics of channels can help learn more distinguishable representations. In the condition where only one self-supervised task is preserved, it can be observed that the instantaneous time shift task is the most important, which is expected, and the delayed time shift task and the replace discriminative task contribute similarly to the performance of the complete model.
Table 4: The results of ablation study.
Models
Pre.
Rec.
F1
F2
CPC
27.65=1.44
55.0713.52
34.20:13.40
42.73-2.57
CPC-Conv
0.39
33.21#4.00
10.53+1.07
17.46=1.42
CPC-MLP
2..84+3.07
52.70+33.65
32.18+22.46
20.34
MBrain-Graph
36.72:4.559
60.48:14.47
43.61++3.08
51.47=2.68
MBrain-Instant
34.49-44.37
55.4143..90
31.57++3.48
48.38
MBrain-Delay
35.00=14.49
65.6112.94
42.9743.61
12.51
Mepain-Replace
36.08=5.35
63.67:4.24
43.6613.66
52.49-2.32
MBlaianonlylnstant
36.4324.44
63.6612.12
43.35.13.83
51.82.12.77
MBlain-onlyDelay
31.59=4.22
55.0345.226
38.56+22.84
46.05=22266
MByeineonlyReplace
34.13+6..8
56.06++3.68
40.02++4.47
47.44+2240
MBrain
37.97:2.75
65.07
46.45
55.28
In addition to removing some components and tasks from MBrain, also design some ablation experiments to verify the effectiveness of our proposed graph structure learning. We have proposed two ideas on how directly implement the multi-channel CPC in Section [2] For the second idea, we have reported the results of a shared CPC regarding all channels as one on the CPC row of Table T For the first idea, we design two strategies to combine multi-channel CNN or MLP into CPC respectively to learn representations for each channel.
Directly use I-Dimension CNN to encode the whole time series data and the number of channels during the process is - 256 256 + C x 256, and split the output into C representations, each of which is -i6-dimensional representation. Then an LSTM is implemented to it. Then we execute the self-supervised task and the downstream task of CPC based on the representations for each channel as MBrain does, this variant is denoted as "CPC-Conv".
We use the contextual representations of all n channels input to MLP in fixed order, but we set the representation the target channel to 0 tensor when we aggregate them. By using the output of MLP the aggregated representation of other channels, perform subsequent experiments following exactly the same steps as MBrain. We name this variant as "CPC-MLP".
learn and aggregate significant information. For "CPC-MLP
19

---

Under review as a conference paper at ICLR 2023
the representations of other channels, and then concatenate it with the representation of the target channel to predict future data. Unlike "CPC-Con" which directly adopts multi-channel convolution for the raw data to obtain the "mixed" low-level representations, "CPC-MLP", like MBrain, lears the correlation of channels based on the "separate" high-level representations. Therefore, the per- formance of "CPC-MLP"" does not drop dramatically that of "CPC-Conv". It can be observed that "CPC-MLP" outperforms "CPC" on the EEG dataset. This may result from the fact that the number of channels in EEG dataset only 19, while that in SEEG dataset is 52 124. Conse- quently, the ablation results show that the graph structure learning we design has reasonable and parameter-effie efficiett inductive assumption.
H HYPERPARAMETER ANALYSIS EXPERIMENTS
A2=0.3
=1=0.5
0.750
MBrain
0.750
MBrain
0.725
0.725
Fy-core
Fy-score
0.675
0.650
0.625
0.625
0.600
0.600
0.575
0.575
0.550
0.530
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
(a) Weights search for Lz.
(b) Weights search for Lz.
Figure 7: Sensitivity analysis on loss weights.
Sensitivity analysis on loss weights. Our loss function is defined as: C (1- C2+ X2h3, where L1, L2 and L3 are the loss of instantaneous time shift prediction task, delayed time shift prediction task and replace discriminative task respectively. and a1 and A hyperparameters to balance the three pre- pre-training tasks. We search both of the weights of t₁ and X2 in the set {0.1, 0.2, 0.3, 0.4, 0.5) and report the tuning results with F2-score for seizure detection task patient-A from SEEG dataset. In 7(a) and 7 we can see that A1 = 0.5 and X2 0.3 lead to the optimal performance, In addition, MBrain consistently performs better than the best baseline.
Sensitivity analysis of replace ratio
Mohav
0.75
0.70
0.55
0.50
Replace ratio (16)
Figure 8: Sensitivity analysis on replace ratio r%.
Sensitivity analysis on replace ratio. We perform sensitivity analysis on replace ratio r% from replace discriminative task. We search the replace ratio from 5% to 95% and report the tuning results with Fo-score for seizure detection task on patient-A from SEEG dataset. As Figure shows, when the replace ratio is 45%, MBrain has the best performance of 71.06 While MBrain gets
20

---

Under review as a conference paper at ICLR 2023
the smallest standard deviation and the second best performance of 70.63=11.4 when the replace ratio is set as 15%.
I FULL RESULTS
I.1 FULL RESULTS OF SEIZURE DETECTION EXPERIMENT
Table 5: The average performance of the seizure detection experiment on the SEEG dataset.
SEEG
Models
Pre.
Rec.
Fi
F2
MiniRocket
22.98:+0.15
66.24+1.226
3..19:10.19
43.28:10.22
CPC
27.65=+4.49
55.07=3.52
34.20+3.40
42.73:2.57
SimCLR
11.06++3.95
51.54
16.60+44.68
25.41+44.95
T-Loss
29.29:12.65
51.55++2.33
36.00
43.13:11.57
TST
13.60=13.48
44.65#4.21
19.80#3.73
28.41+3.29
GTS
24.29:t4.26
40.39:5.80
29.16#2.97
34.17:22.66
TS-TCC
22.10
49.94415.41
25.3248.02
32.74+4..95
TS2Vec
30.56:12.17
52.83+2.89
36.03#1.72
13.35
MBrain
37.97
65.07
46.45
55,28
Table The average performance the seizure detection experiment on the SEEG dataset with the encoder of the SSL model frozen.
SEEG
Models
Pre.
Rec.
F
F2
MiniRocket
22.98:10.15
6..24:0.26
31.79:1.119
43.58:10.22
CPC
26.9914.335
54.16:14.49
32.9813.25
41.17::2.45
SimCLR
10.36::2.76
50.89:16.68
15.6943.21
24.1943.34
T-Loss
29.20+:3.21
50.24++3.52
35.03+11.73
41.85++1.99
TST
13.92-13.96
45.59:1..37
19.4043.91
28.24:12.86
GTS
23.13+3.51
42.34:1.955
28.9242.70
34.7842.008
TS-TCC
23.07++..45
50.3247.11
26.2916.08
34.39
TS2Vec
30.79.13.08
51.43:43.36
35.9942.78
4..4
MBrain
37.72+15.46
63.34
44.47+4.57
53.00
Seizure detection experiment. First, we keep the initial parameters of the trained self-supervised model unchanged, repeat the experiments five times the fine-tuning stage of the downstream seizure detection task, and report the mean and standard deviation results in Table 5 and 7 Next, we freeze the entire encoder SSL model and only train the downstream model and the classifier in the seizure detection task. The same reproducible experimental results are shown in Table 6 and Table g We can see that the performance of most models drops slightly compared with the results model still has competitive performance in
without freezing the SSL model. Nevertheless, Fz-score For the supervised model MiniRocket without a learnable representation extractor, the results remain unchanged in the experimental settings.
21

---

Under review as a conference paper at ICLR 2023
Table 7: The average performance of the seizure detection experiment on the EEG dataset.
EEG
Models
Pre.
Rec.
Fi
F2
AUROC
MiniRocket
22.86-+00.4
63.08 + 1.47
33.56++1.11
46.66+1..33
75.30++0.77
CPC
22.81+:2.04
58.31=7.55
32.50=11244
44.02-+2.43
74.53+1.00
SimCLR
12.63:11.62
14.88+
1.93
36.78
53.86-1-536
T-Loss
1..7
69.2513.99
31.82++1088
47.00=0.500
75.88
TST
15.65=1.54
28.59+12.93
19.65=
23.87=8.09
58.20=4.227
GTS
18.86+1.099
62.51+5.04
2..88
42.54+1.48
71.69+
TS-TCC
15.55:10.88
39.76+11.00
1..89+1.200
29.60=1..64
58.63
TS2Vec
21.40:10.63
58.31=6.14
31.244
43.24=2.78
73.35+
MBrain
2..13:1..03
76.99-1..49
34.92.L0.90
51.34
77.96:10.97
Table 8: The average performance of the seizure detection experiment on the EEG dataset with the encoder of the SSL model frozen.
EEG
Models
Pre.
Rec.
Fi
F2
AUROC
MiniRocket
22.86=:0.84
63.08
33.56=11.11
46.66=1.33
75.30+1.77
CPC
23.00
53.6917.47
31.90+1.40
41.95=3.19
13.4611.35
SimCLR
11.69:10.49
71.73+11.90
2..05+1.220
35.22+3.25
11.35:
T-Loss
21.69942..1
6.254
31.63++2.24
4..59
75.25+
TST
13.42:12.22
13.66
17.62.55.05
23.11=8.09
53.82:15.07
GTS
18.07=0.78
65.01+2.92
28.27
42.76=11553
71.42+22113
TS-TCC
14.03++2..0
34.34+10.20
19.35++3.06
25.89
54.80+4..93
TS2Vec
20.23
64.82+11.92
30.43
44.21
73.43+2260
MBrain
11.99
66.02+5.05
+4.82
47.88 1.52 76.27+1.91
Significant Analysis. Due to the large variance of the experimental results, further conduct the significance test of the mean values to show that the performance of MBrain is indeed significantly primarily
better than that of other baseline models. We would like to emphasize that cerned with the Fcosc of the models, because the clinical practice, doctors focus on finding as many seizures as possible. Thus, we focus on the significant analysis of Fa-score for the results of Table
We perform significance analysis according the following procedures: (1) For SEEG data, first combine the repeat results of each model for each patient into one vector. For EEG data, we directly use the vector of repeat results. Significance analysis is performed on F2 score vectors of all models pair by pair. (2) We use Levene's 's tes to test whether two populations have homogeneity of variance, which a critical property in sigmificant analysis. (3) On the condition of homogeneity of variance, we conduct independent sample T-tes for the two populations. We show the p-values of Table Table 7 in Figure in in the form of thermal maps respectively. Considering the symmetry of the significance test, we only show the half of the p-value matrix.
From these two figures, we can see that for EEG data, MBrain significantly outperforms other base- lines. For SEEG data, although the performance of T-Loss and TS2Vec is not significantly different
https: http:s//eaawiipppaaaaoo////////a s_ttsseeene_________________ test
teat #Unpa red__nd_ _ed___nd__p_ red
Ewo sawpaamtttlettttt
22

---

Under review as a conference paper at ICLR 2023
Mmnwn
Mitren
Miiktcktt
1.06
MiiRocktt
-.
10
5mtLR
Simtin
T Lo55
1,000
Loss
8647
131
15.44
1.666
0..04
151
01
4544
-1.14
G75
8.24
6..20
0.69
6.117
GTS
02.0
+5.44
0.827
-0.44
-0.21
TS-CC
1:03
6:447
3845
0300
3342
TS-TCC
+0.03
+3.3
5.000
+0.00
0,44
+6-3
B:544
6022
2258
6:20
42.L
8662
0.683
8.144
Misttodet
DPC
Sioncue
Tdduu
137
15/T02
153rec
Mitrnon
Mosttorttt
DPC
SinctL
Tiled
TSTCE
15300
(a)
SEEG dataset.
(b) EEG dataset.
Figure 9: Significant analysis.
from that MBrain at the tolerance of 0.05, the p-value was only slightly above 0.05. Therefore, in general, MBrain is superior other baselines models in terms of F2-score. For baselines, can see that MiniRocket, CPC, T-Loss, TS2 and GTS have similar performance on the two datasets. And SimCLR, TST and TS-TCC perform relatively poorly.
1.2 FULL RESULTS OF DOMAIN ADAPTATION EXPERIMENT
Table 9:
The performance of the domain adaptation experiment on SEEG dataset in terms of Fg- score. DA row denotes the performance of MBrain in the domain adaptation experiment setting. Max-base and Non-DA rows represent the best performance of baselines and MBrain in the seizure detection experiment. Bold numbers and indicate the best and the second best performance.
Group A
Group B
Setting
B-+A
C-+A
D-
A-BB
C-BB
D-+B
DA
68.55+4.27
69.14
68.78+4.12
41.08:2.59
46.06=3.05
46.12.42.04
Max-base
62.49:12.30
39.78-1..04
Non-DA
70.63
46.62.12.42
Group C'
Group D
Setting
A+C
B-C
D-C
A+D
B++
C+D
DA
40.04:1..98
39.3442.11
48.64:15.48
80.82.10.5**
19.10
80.72-E1.31
Max-base
33.59++2.23
75.35
Non-DA
46.09+2.35*
83.27-10.95
In this experiment, we still keep the self-supervised model trained on the source domain unchanged and repeat the experiments on the target domain five times in the fine-tuning stage of the down- stream seizure detection task. The mean and standard deviation results can be observed that the variances of the results under DA setting are generally higher than those under Non-DA setting, indicating a large variation in the patterns of patients in SEEG dataset. Nev- ertheless, high-quality source patient data can still be used to pre-train an SSL encoder with good average performance on the target patient. practice, we can make predictions in an ensemble way by training multiple classifiers.
presented in Table
23


Reviews:
[{"review_id": "MSt70z_54z", "reviewer": "Reviewer_9fFf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a semi-supervised learning (SSL) method that pre-trains on both SEEG and EEG data. It learns the correlation graph between channels from three SSL tasks: instantaneous time shift task, delayed time shift task, and replace discriminative task. Experiments on seizure detection is described with promising results compared to other semi-supervised learning methods.", "summary_of_the_review": "The idea of using SSL on both EEG and SEEG signals is interesting, and learning correlation graphs improves the interpretability of the model, which can be useful for clinicians.", "strength_and_weaknesses": "Strength: SSL on EEG has been studied before, but the authors generalize the idea to SEEG signals as well, and tries to learn correlation graphs which helps with interpretability of the model. Experimental results on seizure detection are encouraging, and are compared to other existing models. \n\nWeakness: Figure 1 is a little confusing. Also, can the authors comment on the large standard deviation values in Tables 5-7? Is the improvement by the proposed method significant given that observation? How long does the fine-tuning take, and what do the prediction results look like without fine-tuning? What happens when the SEEG channels and EEG channels don't overlap?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IDwhJJK412N", "reviewer": "Reviewer_DmoA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes a novel multi-channel self-supervised framework for SEEG and EEG data. The framework comprises delayed-time-shift prediction, instantaneous time shift, replacement discriminative tasks, and an additional graph module. The results show improvements over previous related work and supervised models for multiple performance metrics for seizure detection.", "summary_of_the_review": "The authors are solving significant problems for healthcare, and there are many exciting ideas: theoretical justification for multi-channel vs. single channel, variants of multi-channel CPC, other self-supervised tasks or modules for EEG and SEEG data, and interpretability (appendix). However, each part of the contribution lacks theoretical justification, technical sophistication, or empirical evidence in a detailed examination; thus, it does not look clear and complete. Overall, It is hard to read the manuscript and understand its significance. Authors should consider exploring one idea at a time and provide concise empirical evidence or theoretical justification for each part. ", "strength_and_weaknesses": "#### Strength\n- The experiments have been performed with SEEG and EEG datasets.\n- The authors have considered many baselines.\n- The authors have performed an ablation study to show the benefit of each proposed component.\n\n#### Weaknesses\n- In section 2 after Theorem 1, it is unclear how the proposed objective is more informative and better. When discussing mutual information for CPC, we talk about lower bounds and tightness. For example, the authors (Tschannen et al., 2019) show that tighter bounds on MI can result in worse representation. While the authors note the gap $\\log N$, which is constant and only shifts the lower bound.\n- Further, I did not find any experiments with empirical evidence on how multi-channel improves over a single channel without additional modification. The authors provide multiple ideas on how to implement the multi-channel CPC. However, they do not evaluate them. For example, in multimodal learning (Liang et. al, 2022), fusing different modalities with only architectures is expected to improve the performance over unimodal baselines assuming channels are different modalities. So, it could be a simple baseline where the models take multi-channel input, but the loss function is vanilla CPC. I agree with the authors that the correlation is not explicit in this case, but it is still a viable option to improve performance. The comparison could also show the trade-off between performance and interpretability.\n- Main experiments have been performed on the dataset splits where the same subject exists in each subset. The authors must evaluate models on the dataset splits that do not overlap on the same subjects. Models can exploit slow features in time series (persistent temporal feature learning) (Feichtenhofer et al., 2021).\n- Subject-to-subject domain translation experiments are not stable and depend on the random sample. Consider studying data efficiency over a number of subjects instead of the proposed experiment. Clearly, the model does not generalize in a such low data regime.\n- There is no pairwise statistical comparison of the model's performance for significance. Table 1, 2, and 3 does not have a standard deviation, while they are reported in the appendix. Some results, specifically, SimCLR on Recall, have high variance and mean within the variance; thus, selecting the proposed model as bold is misleading. In table 5, CPC is equivalent to the MiniRocket or MBrain.\n\nFeichtenhofer, Christoph, et al. \"A large-scale study on unsupervised spatiotemporal representation learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nTschannen, Michael, et al. \"On mutual information maximization for representation learning.\" arXiv preprint arXiv:1907.13625 (2019).\n\nLiang, Paul Pu, Amir Zadeh, and Louis-Philippe Morency. \"Foundations and Recent Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions.\" arXiv preprint arXiv:2209.03430 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DenMN9mXnr2", "reviewer": "Reviewer_Pj9B", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach for self-supervised representation learning of brain signal, as recorded by EEG and SEEG. The representations are trained to optimize a loss with three components. The first component is a multi-channel version of the InfoNCE loss, used in contrastive predictive coding. The channels are aggregated according to their instantaneous correlations with each other. The second component gives weight to correlations that may occur at larger time delays. The third component encourages individual channels to be distinguishable. The authors find that their model results in embeddings that can be used downstream for better seizure detection, as compared to other self-supervised representation learning approaches. They further find that these representations can be learned for one subject and then used for prediction on a different subject.", "summary_of_the_review": "The paper delivers on what it promises: a self supervised representation learning approach for brain signal. Improvements are shown over other reasonable SSL approaches. And transfer learning is shown to be possible. The main downside of this approach is its complexity. The paper could be greatly strengthened if the complexity of each component was justified, especially by way of comparisons to simple linear baselines.", "strength_and_weaknesses": "# Strengths\n- Proposes an architecture that can accommodate two very different modalities: EEG and SEEG\n- Thorough comparison with other SSL representations\n- Domain transfer analysis indicates that this approach can be trained on subjects and then used for held-out subjects\n\n# Weaknesses\n- The theoretical discussion in section 2 is a little disconnected from the proposed method, since proposition 1 seems to require that the aggregation function $\\Phi$ be linear and non-trivial, but this is not necessarily fulfilled by the learned autoregressive model (eq 12). I believe that in almost all cases, the autoregressive model will not be trivial, but could the authors comment on why they believe the conclusions of proposition 1 should usually hold in practice.\n- Section 3.2: There are many ways that multiple channels could be aggregated to form a single context vector. In this work, the context vector is created using a graph, where the edge weights are taken from a correlation matrix. This adds a non-trivial amount of complexity to the system, so it would be appropriate to justify this choice with a few ablation tests. It would be good to compare with aggregation strategies that make fewer assumptions about the structure of the data. For example, a fully connected feed-forward that reduces the dimensionality from $n$ channels to 1 channel could be used instead. \n- Section 4.3: I think there is a very simple and informative baseline that should be included: a linear classifier or autoregressive model trained on the same data and targets. \n- Section 4.5: An ablation that could be informative: how does the model perform when only the \"Replace Discriminative Learning\" objective is used? The hyper-parameter analysis shows that this component of the loss has the greatest weight. And similar losses have been sufficient for good performance in the audio domain (see wav2vec2 -- Baevski et al 2020, Mockingjay -- Liu et al 2019).\n- Section 4.2. I don't completely understand how domain adaptation is possible between subjects. Doesn't each subject have different electrode placements/a different correlation matrix? How can the pre-trained weights be re-used?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8S1Qnk0Pwqg", "reviewer": "Reviewer_dttW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-channel self supervised learning method to overcome two issues in analyzing brain signals: (1) existing methods are often limited to a particular type of brain signal data, either the invasive data (e.g., SEEG) or non-invasive data (e.g., EEG). (2) correlations amongst different brain areas need to be explored to achieve a better understanding of brain activity. To address these issues, the authors propose a CPC-based method on capture the spatial correlations amongst different channels, which also take into account the delayed time shift. Theoretical analysis is also provided. The method is applicable to learning representations of both EEG and SEEG data. The effectiveness has been shown in the application of seizure detection. ", "summary_of_the_review": "Overall, the paper is well written with an interesting idea. Novel self-supervised algorithms have been proposed to explore correlations amongst multiple channels in brain signal data. Effectiveness of the proposed method has been validated using real data from a hospital. It would be more convincing to compare to some non-contrastive-learning-based self-supervised learning methods. ", "strength_and_weaknesses": "Strengths: \n1.\tExploring spatial correlations amongst channels is critical to knowledge discovery from multi-channel brain signal data. \n2.\tThe method is in a self-supervised fashion, which can help alleviate the need for large annotations.\nWeaknesses:\n1.\tCan this method be used on both SEEG and EEG simultaneously?\n2.\tIt would be better to compare with other self-supervised learning methods that are not based on contrastive learning. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-wqNqmXMyXP", "reviewer": "Reviewer_tXY7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an SSL framework for EEG and evaluate for seizure detection.", "summary_of_the_review": "The paper is nicely written with good results on two different datasets. The demonstrated application is important and the strong performance indicates the significance of the work. However, including the same subjects in training and testing invalidates the results. It is not clear if that was done for other approaches as well. Even if it was, both should be done by selecting different subjects for training and testing.", "strength_and_weaknesses": "Strengths:\n- Nicely written\n- Baselines are chosen well\n- Outperforms all baselines for both F1 and F2 (weaknesses highlight why this is not really valid)\n- Evaluated on both SEEG and EEG datasets\n- Includes clinical collaboration further signifying the significance of the work\n- The domain adaptation experiment in Table 2 is also a great way to show the significance of the work\n\n\nWeaknesses:\n- Split table 3 into two tables?\n- Domain adaptation not shown on TUSZ dataset\n- Including the same subject in both testing and training invalidates the results", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject