Group-level Brain Decoding with Deep Learning
Authors: Richard Csaky
Mats W.J. Van Es
Oiwi Parker Jones
Mark Woolrich
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
transfer learning
decoding
neuroimaging
MEG
permutation feature importance

Abstract: Decoding experimental variables from brain imaging data is gaining popularity, with applications in brain-computer interfaces and the study of neural representations. Decoding is typically subject-specific and does not generalise well over subjects. Here, we propose a method that uses subject embedding, analogous to word embedding in Natural Language Processing, to learn and exploit the structure in between subject variability as part of a decoding model, our adaptation of the WaveNet architecture for classification. We apply this to magnetoencephalography data, where 15 subjects viewed 118 different images, with 30 examples per image; to classify images using the entire 1s window following image presentation. We show that the combination of deep learning and subject embedding is crucial to closing the performance gap between subject- and group-level decoding models. Importantly, group models outperform subject models on low-accuracy subjects (but impair high-accuracy subjects) and can be helpful for initialising subject models. The potential of such group modelling is even higher with bigger datasets. To better enable physiological interpretation at the group level we demonstrate the use of permutation feature importance developing insights into the spatio-temporal and spectral information encoded in the models. All code is available on GitHub.

Under review as a conference paper at ICLR 2023
GROUP-LEVEL BRAIN DECODING WITH DEEP LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Decoding experimental variables from brain imaging data is gaining popularity, with applications in brain-computer interfaces and the study of neural represen- tations. Decoding is typically subject-specific and does not generalise well over subjects. Here, propose method that uses subject embedding, analogous word embedding in Natural Language Processing, to learn and exploit the structure in between subject variability as part of decoding model, our adaptation of the WaveNet architecture for classification. We apply this to magnetoenceepalloraphh data, where 15 subjects viewed 118 different images, with 30 examples per image; to classify images using the entire Is window following image presentation. We show that the combination of deep learning and subject embedding crucial to closing the performance gap between subject- and group-level decoding models. Importantly, group models outperform subject models on low low-accuracy subjects (but impair high-accuracy subjects) and can be helpful for initialising subject mod- els. The potential of such group modelling is even higher with bigger datasets. To better enable physiological interpretation at the group level we demonstrate the of permutation feature importance developing insights into the spatio-temporal and spectral information encoded in the models. All code is available on GitHub'.
1 INTRODUCTION
In recent years, decoding has gained in popularity in neuroscience (Kay et al., 2008), specifically decoding external variables (e.g. stimulus category) from internal states (i.e. brain activity). Such analyses can be useful for brain-computer interface (BCI) applications (Willett et al., 2021) or to gain neuroscientific insights (Guggenmos et al., 2018; Kay et al., 2008). Analysing deep learning methods on such data is also beneficial for the machine learning community. Namely, the small, noisy, high-dimensional datasets test the limits of popular architectures on real data and demand research into new methods (Zubarev et al., 2019; Kostas et al., 2021). Applications of decoding to brain recordings typically fit separate (often linear) models per dataset, per subject (Guggenmos et al., 2018; Dash et al., 2020b). This has the benefit that the decoding is tuned to the dataset/subject,, but has the drawback that is unable to leverage knowledge that could be transferred across datasets/subjects. This is especially desirable for the field of neuroimaging, because gathering more data is expensive and often impossible (e.g. in clinical populations). More practical drawbacks of subject-specific (subject-level) models include increased computational load, higher chance of overfitting, and the inability to adapt to new subjects. We aim to leverage data from multiple subjects and train shared model that can generalise across subjects (group-level). A conceptual visualisation of subject-level (SL) and group-level (GL) models is given in Figure 1.
Magnetoeneephallgrapyy (MEG) measures magnetic fields induced by electrical activity in the brain, and it is one of the main noninvasive brain recording methodologies, next to electroencephallggaphh (EEG) and functional Magnetic Resonance Imaging (fMRI. Due high temporal resolution and relatively good spatial resolution, MEG is an excellent method for studying the fast dynamics of brain activity. MEG highly suitable for decoding analyses (Du et al., 2019), which is mostly done using SL models. This is because between-subject variability of neuroimaging data limits the application of a single shared model between subjects without capturing the structure of between-subjeer variability (Olivetti al., 2014; Li et al., 2021). Such approach, which we call naive group modelling,
Anonymized.

---

Under review as a conference paper at ICLR 2023
model 1
model 2
model
model
subject 1 triale
subject 2 triala
subject 3 triala
wabject 3 Iriale
subjact Iriala
T
subjjctt i itals
(a) Subject-level (SI.) models.
(b) Naive group-levell (GL. model.
Figure 1: Comparison of subject-level (SL) and naive group-leved (GL) modelling. (a) A separate model is trained on the trials (examples) of each subject. (b) A single, shared model is trained on the trials of all subjects without capturing between subject variability. Each trial is Cx (channels timesteps) dimensional. Each of the $ subjects has trials.
effectively pretends that all data comes from the same subject (see Figure lb). Between-subject variability has multiple sources, such as different anatomical structures, different positions in the scanner, signal-to -oise ratio, etc. (Saha & Baumert, 2020). To overcome this, we propose general architecture capable of jointly decoding multiple subjects with the help of subject embeddings (Figure 2). The scope of this paper is full-epoch decoding, and comparisons with sliding- -windoww decoding approaches often used in neuroscience left for future work.
To qualify how we aim to improve on SL models, we will next describe the two main approaches to evaluating decoding models, with different underlying assumptions and goals. One approach is to construct separate train and test splits for each subject that are made up of different, non-overlapping trials. This be called within-subject splitting evaluation. SL models are evaluated by definition in this way, and it is very common setup in the neuroscience literature (Guggenmos et al., 2018; Cooney al., 2019b; Cichy Pantazis, 2017; Dash al., 2020b;a; Nath al., 2020). In this work. our main aim is to improve over SL. models in the context of within-subject splitting evaluation and improve the prediction of left-out trials, by using a single group decoding model that generalises across subjects. We call this GL method across-subject decoding. We are motivated by the fact that GL models that perform well in this manner can useful for gaining neuroscientific insights that are relevant at the group level, as we will show in Sections 4.4 and 4.5. The other prominent approach to evaluating group models, leave-one-subee sube-subject-ott (LOSO) analysis, also presented in Section 4.3. In this scenario, GL models are trained on data from multiple subjects and tested on subject (Zubarev et al., 2019), which can be especially useful in zero-shot BCI applications. Although in this case, find no improvement using our embedding-aiee change with larger datasets with many more subjects. Our aim is to improve across-subject decoding of MEG data by using group model that generalizes across subjects. To be clear this objective and the datasets we use are not related to any kind of direct BCI application.
unseen
group model, we think this may
We make the following contributions using MEG dataset with visual task (Cichy et al., 2016): 1. A GL model with subject embeddings is introduced, substantially improving over naive group modelling. 2. Insight provided into how non-linearity and subject embedding helps group modelling. 3. Neuroscientific insights are gained from the deep learning- baaed decoding model. 4. Analysis of model weights reveals how meaningful spatio-temporal and spectral information is encoded.
RELATED WORK
Decoding can be applied to most tasks/modalitics, such as images (Cichy et al., 2016), phonemes (Mugler al., 2014), words (Cooney et al., 2019b; Hultén et al., 2021), sentences (Dash et al., 2020b), and motor movements such as imagined handwriting (Willett et al., 2021), 1) jaw movements (Dash used image categorisation because
al., 2020a), or finger movements (Elango et al., 2017). Here, it is widely studied decoding task and we had access to dataset which is relatively large for the field of neuroimaging. Our results should readily generalise to other decoding modalities. Chaibub Neto
2

---

Under review as a conference paper at ICLR 2023
et al. (2019) strongly argue against using SL modelling, especially for clinical applications. Recently, some transfer learning approaches have been proposed to deal with the problem of variability between subjects. Kostas & Rudzicz (2020) have proposed two distinct methods. The euclidean alignment is very similar to a spatial whitening of the data. We have tried this in conjunction with our group model, however, we found this to lower performance, and thus opted for simpler channel-wise standardization. The other method, mixup regularization is entirely complementary to our approach and can be used in conjunction. This is general regularization/data augmentation technique and does not specifically deal with inter-subject variability. Most transfer learning frameworks consist of applying a model trained on one subject to different (target) subject (Elango et al., 2017: Dash et al., 2019; Cooney et al., 2019a; Olivetti et al., 2014; Halme & Parkkonen, 2018; Li et al., 2021). Some approaches use learnable affine transformations between subjects (Elango et al., 2017), while others finetune the whole model on target subjects (Cooney et al., 2019a; Dash et al., 2019). However, these approaches offer only a marginal improvement over naive group modelling and do not use a shared model across subjects. We aim significantly improve this by using a general framework and model capable of decoding multiple subjects at once.
We must note that while there is a clear distinction between within-subjes splitting and LOSO suuue-one-subject-ee
evaluation, most of the transfer learning works that aim to improve in a LOSO scenario do use some form of training on the left-out subject (Zubarev et al., 2019). Thus, the distinction between the two evaluation frameworks gets less transparent and we discuss in Section 4.3 how our methods could be useful in this less stringent LOSO definition. We believe that ultimately the way a method is intended to be used in practise should determine how that method is benchmarked.
Transfer learning is popular in the wider machine learning field. Parallels can be drawn with domain adaptation (Long et al., 2015), or transferring knowledge from large to small datasets within the same domain (Wang et al., 2019; Zhuang et al., 2020). Natural language processing (NLP) datasets often contain data from widely different sources (Radford et al., 2022), but due to the sheer size of the dataset and model complexity, training on joint data achieves good results (Brown al., 2020; Devlin et al., 2019). As discussed before, this naive concatenation of subjects does not work well on small neuroimaging datasets. Perhaps the most relevant parallels can be drawn with dialogue and speech modelling work, where inter-speaker differences are modelled using speaker embeddings (Li et al., 2016; Zhang et al., 2018; Saito et al., 2019; Mridha et al., 2021). Chehab et al. (2021) have similarly found that subject embeddings provide small but significant improvement in encoding MEG data from a language task. However, limited information is provided on how subject embedding helps, and their results cannot be directly generalised MEG decoding. We build this work, expanding it to the task of decoding MEG data, and provide additional insight into how deep learning and subject embeddings help GL decoding models.
3 METHODS
3.1 DATA
In this work, a task-MEG dataset is used where 15 subjects view 118 different images, with each image viewed 30 times (Cichy al., 2016). The data is publicly available", however, we obtained the continuous raw MEG data directly from the authors to be able to run our preprocessing pipeline using MNE-Python (Gramfort et al., 2013). Raw data is bandpass filtered between 0.1 and 125 Hz and line noise is removed with notch filters. After downsampling to 250 Hz. 1.024-second epochs are extracted, starting 100 ms before stimulus presentation. This resulted in 306 356-dimensional trials (channels x timesteps) from the 306 MEG sensors. Whitening is used to remove covariance between channels for SL models, whereas for GL models, a standardisation is performed per channel. We do multiclass decoding, predicting separate probability for each of the 118 classes (images).
3.2 MODELS
Our deep learning model, the WaveNet Classifier, is inspired by previous approaches to applying WaveNet (van den Oord et al., 2016) for classification (Zhang et al., 2020). We do not aim to design a new kind of architecture for decoding MEG data, but rather build our model based on previously
htttp:/usserpage.
Sureeeee..fubreeee fu-ber1 1n. rmar chy/ Fusi
page/mai html
3

---

Under review as a conference paper at ICLR 2023
pubject trial subject & trial
306
subject trial
Coscarunate
50Rud 12x1
dilated conT
Heshape
2448 21000
bally wally-ccoomnn connocted
1000x0000
nulty- come cood
400x118
clasaiboction
256
sublact Bubl loct
28
subject
16x10
Figure 2: Group-level WaveNet Classifier with subject embeddings. Dashed boxes represent parts of the model which differ between SL and GL versions of our architecture. Red boxes represent learnable parameters. For convolutional layers the numbers represent inpul channels x output channels x kernel size. For fully-connected layers the numbers represent inpul neurons x oulput neurons. The embedding layer dimensionality is given as s x E (15 x 10), where s is number of subjects, and E is the embedding size. Embeddings are concatenated with input trials to provide information about which trial is coming from which subject. The classification loss is cross-entropy.
proven CNN-based architectures for EEG and MEG data (Lawhern et al., 2018). While Wavenet has proved to be successful in the audio domain, the best of our knowledge, there is no prior work applying Wavenet to neural decoding. Our model consists of 2 parts: the (temporal) convolutional block, intended to act as a feature extractor; and the fully-connected block, which is designed for classification (Figure 2). The convolutional block uses a stack of 1D dilated convolutional layers, which include dropout and the inverse hyperbolic sine activation function. The dilated convolutions in WaveNet are effective for modelling time series data as successive layers extracts complementary frequency content of the input (Borovykh et al., 2018). Since the dilation factor is doubled in successive layers, the receptive field of the convolutional block is 2 lagers Given there is pooling and a convolution stride of 1, the output of each layer preserves the temporal dimensionality? At the end of the convolutional block, we downsample temporally by the size of the receptive field. In the model with convolutional layers, this means that the initial input of size 256 is downsampled by factor of 64, resulting in 4 values per channel. Next, this output is flattened and fed into fully-connected block. The final output is a logit vector corresponding to the 118 classes. The model trained with the cross-entropy loss for classification.
For SL modelling, the Wavenet Classifier contains 3 convolutional layers, whereas for group mod- elling has 6, further motivated in Section 4. Our improvement of the naive group model includes subject embeddings, which are introduced as way of dealing with between-subject variability, similarly to Chehab et al. (2021). Like word embeddings in NL., each subject has a corresponding dense vector (Mikolov et al., 2013). This vector is concatenated with the channel dimension of the input trial across all timesteps. Thus, in the embedding- aided GL model, input trials are (C+E) x T dimensional, where E is the embedding size. Subject embeddings are learnt together with other model weights using backpropagation. We reasoned that an embedding-aides model can learn general features across subjects, with the capability of adapting its internal representations for each subject.
3.3 MODEL ANALYSIS
In this section, we describe several approaches to uncover the information encoded in the WaveNet Classifier. In Kernel FIR Analysis, we investigate the frequency characteristics of the convolutional kernels. Random noise fed into a trained model, and the power spectral density of the output of specific kernels is computed to assess their finite impulse response (FIR) properties. Permutation Feature Importance (PFI) is a powerful method to assess which features contribute the most to model performance (Altmann et al., 2010; Chehab et al., 2021). Chehab et al. (2021) specifically have showed the power of PFI for analysing how certain language features like word frequency affect the performance of forecasting MBG data at different temporal and spatial locations. Conversely, in MEG decoding we can directly assess both spatial and temporal information by permuting across timesteps (for each channel) and across channels (for each timestep), respectively. We call these
Exceptt the amount that gets chopped off because of the kernel size itself, since we do not use padding.
4

---

Under review as a conference paper at ICLR 2023
1.4
1.2
Validation accuracy
1.0
subject models
group models with embedding
finetuned group models
0.8
group models
0.6
0.4
chance
0.0
linear subject
nanlinear
linear group
nanlinear group
linear
nanlinear group emb
nonlinear finetuned
nonlinear group emb finetuned
subject
group emb
Figure 3: SL and GL models evaluated on the validation set of each subject. Paired samples T-tests
are shown for comparisons of interest (+ = p < 5e -, ** le-2, *** = le 3, ***
p < 1e - 4). The non-li von-linear near group-emb finetuned model is finetuned separately on each subject, initialized with the non-linear group-emb model. Chance level is 1/118.
temporal and spatial PFI, respectively. A decrease from the original accuracy (with unpermuted inputs) indicates that (visual) stimulus-related information is present in the MEG data in certain time periods or sensors. Alternatively, when looking at individual kernels, our feature importance measure the absolute difference between the kernel output using the original and permuted inputs. We reason that a more important feature will cause a higher output deviation. To assess the frequency sensitivity of individual kernels. we introduce the spectral PFI. First, the data in each channel of each trial is Fourier transformed, and the Fourier coefficients are shuffled across channels for each frequency (or frequency band). Then, the inverse Fourier transform is computed, obtaining trial with disrupted information in specific frequency bands.
4 RESULTS
4.1 SUBJECT EMBEDDING CLOSES THE GAP BETWEEN SL AND GL MODELS
Our main evaluation metric classification accuracy across the 118 classes. Train and validation splits with a 4:1 ratio were constructed for each subject and class. This means that classes are balanced (i.e., contain the same amount of examples) across subjects and splits. The first 20% continuous MEG data is used to extract validation examples, with the remaining used for training trials. SL and GL models are trained and evaluated on the same splits. For each model, an extra training is conducted wherein the (linear) identity function is used the influence of non-linearity, which is the bedrock of deep learning. Linear and non-linear models trained for 500 and 2000 epochs (full passes of the training data), respectively, with the Adam optimiser (Kingma & Ba, 2015). More training details are provided in the Appendix. We compute paired samples T-tests for comparisons of interest over methods, where the pairing within subject and samples are the subject-level mean accuracies over validation trials. We used Py PyTorch for training (Paszke et al.. 2019) and several other packages for analysis and visualisation (Pedregosa et al., 2011; Virtanen et al., 2020; Harris et al., 2020; Wes McKinney, 2010; Waskom, 2021; Hunter, 2007).
activation function assess
Validation accuracies for all models are shown in Figure 3. Interestingly, the subject level, linear models performed slightly better than non-linear (4% increase, 5.7e 4). We think that both the limit data size and noise levels in the data contribute to the subpar performance of non-linear models. Although some studies have found deep learning to improve over simpler linear models, this improvement is often marginal (Cooney et al., 2019b; Schirmeister et al., 2017). Such results are difficult to generalise across different MEG datasets, due variability in tasks, the number of subjects, and the amount and quality of data (Schirrmeister et al., 2017). The large between- subject variability common MEG datasets is apparent, with individual subjects' accuracy ranging from 5% 88%. As
5

---

Under review as a conference paper at ICLR 2023
expected, the naive group models, i.e. naive application of either the linear non-linear WaveNet Classifier to the group modelling problem, result in much worse performance than SL models (30% decrease). Inferring such high variability between so few subjects implicitly is not trivial. Adding subject embeddings to the non-linear model improves performance by 24% (p = 1.9e 6), with no increase for the linear model. This shows that leveraging subject embeddings in conjunction with non-linear activations can narrow the gap with SL models (6% difference, p = 1.3e 2). Limiting the non-linearity to the first layer resulted in a subpar performance, similar that of linear model. This indicates that non-linearity is needed within multiple layers to benefit from subject embeddings.
We also finetuned the embedding- GL model on the training data of each subject separately (non-linear group-emb finetunes for 500 epochs. We effectively use the GL model as an initialisation for SI. models, improving over SI. models trained from scratch, achieving 50% accuracy (5% increase, p le 3). This shows that representations learned at the group level are useful for SL modelling. In contrast, finetuning a naive group model (non-lineax group finetuned) only achieved 42% accuracy (8% lower), showing that finetuning perfor- mance depends on good GL performance. Thus, in addition to closing the gap between SL and GL modelling, finetuning our ambedding-aided model provides the best overall accuracy for SL modelling. The variance of non lineerr group-emb (0.19) and non-linear group-emb finetuned (0.24) is lower than the SL models (0.26). Generally, the group model is reducing between-subject variability. In the case of non-1 low accuracies (15-30%) had higher accuracies than SL models (even though the mean across sub- jects is lower). This shows that training a group model is a good approach to improve low-accuracy subjects. Indeed, strong negative correlations of -0.88 and -0.54 are obtained between the SL accu- racy and the change in accuracy achieved by the ncn-linear group-emb and non-linear group-emb finet uned models, respectively. Comparing finetuning to from-seratch SL models, only 2 high-accuracy subjects are slightly worse, and generally low/nid-accuracy subjects show more improvement than high-accuracy subjects (see Appendix for plots). In addition to the main dataset used in this section also analysed our main findings on another publicly available visual MEG dataset (see Appendix for results).
1 i near group-emb, 4 subjects with generally
In summary, these results suggest the following recommendations for decoding MEG task data. 1. Subject embeddings and non-linearity should be used for achieving good group models. 2. GL models can be used improve over SL models on www-perfommace subjects. 3. For the best SL performance, the finetuning approach should be used, benefitting low- www-performance subjects the most.
4.2 INSIGHTS INTO SUBJECT EMBEDDINGS AND OTHER MODELLING CHOICES
For the embedding-aided GL setup, 4 further models were trained for 5-fold cross-validation Average accuracy was 37.4% (as opposed the 38% reported in Figure 3), with a 95% confidence interval of 0.8%. Thus, the proposed GL model is robust to different random seeds and dataset partitions. more extensive robustness analysis is omitted due to computational constraints. In non-linear SL models, performance improves as use fewer convolutional layers. This effect plateaus at layers, whereas for non-linear GL models using more layers improved accuracy (see Table in the Appendix for exact accuracies). Thus, SL models seem to rely more the fully-connected block as they are unable to extract good features, and GL. models rely more on the convolutional block to learn shared features across subjects. To be clear, because of how we perform the downsampling after the convolutional layers (described in Section 4.1), using fewer convolutional layers increases the overall parameter count because the fully-connected block has to be enlarged. Thus, the group model (with 6 conv layers), is about 2.5x smaller than the SL models (with 3 conv layers). However, nen-linear group-emb finetuned models achieve higher accuracy than from-scratch SL models. This shows that when initialised well (with group model trained on multiple subjects) even SI. models can benefit from non-linearity and more convolutional layers.
We tried different approaches to understand how subject embeddings help the group model. A clustering or 2D projection of the embedding space such as PCA or t-SNE (Van der Maaten & Hinton, 2008) did not show any clusters (see Figure 8 in the Appendix). This is likely to be consequence of only having 15 subjects, since cases where such visualisations work well (Liu et al., 2017) typically have thousands of dimensions (e.g. words in word-embeddings). To assess whether the embeddings simply encode which subjects are good, we transformed the embeddings with PCA and correlated all components with the accuracies across subjects. We found no significant correlations; therefore,

---

Under review as a conference paper at ICLR 2023
group>subject (p<0.05)
Subject
Accuracy loss
0.5-
group-emb
group
8.1
Volidation accuracy
chance
0.25
0.4
Accurscy loce
0.20
5-0
0.15
0.2
0.1
0.05
0.00
0.0
0.2
0.4
0.6
0.8
1.0
200
600
800
Training set ratio
(ms)
(a) Generalisation and finetuning on left-out subjects. The horizontal axis shows the amount of training data used from the left-out subject; training ratio of O corre- sponds to zero-shot approach. sub ject is trained from scratch, while group-cmb and group are initialised with the non-linear GL model with and without embed- dings, respectively. The 95% confidence interval of the accuracy across left-out subjects is shown with shading.
(b) Temporal (line) and spatial (sensor space map) PFI for the non con-linear group-ente model. For temporal PFI accuracy loss (vertical axis) plotted with respect to time since visual image presentation (horizontal axis). Shading shows the 95% confidence interval which is not visible due to low variability. For spatial PFI, darker red shading is equivalent higher accuracy loss.
Figure 4: Generalisation performance (a); Temporal and Spatial PFI (b).
embeddings do not appear encode information about SI. accuracy. To assess how much embeddings contribute to a trained model, we tried both setting the embeddings to zero and shuffling them. This reduced the accuracy to 10% (from 38%). Thus, embeddings encode crucial information to aid decoding, but the model is still better than chance without them. For further insights see Appendix.
4.3 LEANE-ONE--SUJJECT-OO EVALUATION
In this section we report leave-one-suhject-onr (LOSO) cross-validation results, a common evaluation setup with several use cases described in Section 1. We also analyse how performance improves when we allow models to use increasing amounts of data (finetuning) from the left-out subject. We compared the naive group and group-embedding models (non-linear) with a from-scratch SL model (linear) trained only on the left-out subject. To be clear, as in the finetuning approach in Section 4.1 the group models are only used as initialisation for an SL model trained on the left-out subject. The only difference is that the subject which we are finetuning is not in the training data of the group models (as opposed to the finetuning setup in Section 4. 4)... For the embedding-aidee group model, the left-out subject's embedding group models achieve 5% accuracy (Figure 4a). Up to the case when 70% of the training set is used finetuned GL. model is much better than from-scratch SL model (p < 0.05, corrected for multiple comparisons). The benefit of GL. models in LOSO analysis has been previously established (Elango al., 2017). Unsurprisingly, the embedding-aided group model does not improve over the naive group model, but is. importantly, not worse. As opposed to the finetuning setup in Figure 3, when adapting to new subjects, better group performance does not translate to better finetuning performance, as the model is unable to leverage the new subject's embedding.
initialised randomly. In the LOSO (zero-shot) evaluation,
Exploiting subject embeddings pure LOSO framework is not trivial, as some additional approach is needed to initialise/learn the embedding of the left-out subject in an unbiased manner. In larger datasets with more subjects, netween-subjes similarities in the embeddings could be exploited and different heuristics explored, e.g. initialising the embedding with the average of all learned subject embeddings. However, research aimed at improving performance in new subjects often leverages transfer learning in some way, where a limited amount of data from the new subject can be used (Zubarev et al., 2019). In this scenario, we think our across- subject group model could be helpful, by, for example, using the limited data from the new subject or by learning a useful embedding for the new subject an unsupervised manner. As we have shown in Section 4.1 this could be especially useful for subjects with low performance. As opposed a naive continuation of the trends
7

---

Under review as a conference paper at ICLR 2023
in Figure 4a, expect that with more trials, the gap between group initialisation and training from scratch would continue, up to some limit. We believe that the reason why the gap closes at 100% training data is due to the ratio of training and validation sets and the low number of examples. The small validation set (6 examples per class) is probably not representative of the full data distribution.
4.4 NRUROSCIENTIFE INSIGHTS ARE GAINED FROM A DEEP LEARNING BASED GL MODEL
An established critique of deep learning models applied to neuroimaging data is the lack of inter- pretable insight (Murdoch et al., 2019) they provide about the underlying neural processes that drive the decoding. To gain such neuroscientific insights, it is useful to assess the time- and space-resolved information/disccimnnabiiiii
within trials. As mentioned in Section 3.3, permutation feature impor- tance (PFT) is a suitable, model-agnostic measure do this. Figure 4b shows the temporal and spatial PFI of the non-linear group-embedding model. To make the results robust and smooth, the shuffling for temporal PFI was upplied to 100 ms windows, and magnetometers and gradiometers in the same location were shuffled together for spatial PFI. Time windows or channels with higher accuracy loss than others are interpreted as containing more information about the neural discriminability of the visual images. This indicates when and where information processing related to the presented images happening in the brain. Temporal PFI shows a large peak around 150 ms which is often observed in neuroscientific decoding studies employing sliding window analysis to assess temporal information content (Higgins et al., 2022). After this, the information content rapidly decreases, with a second, smaller peak around 650 ms, which could correspond presentation at 500 ms. Spatial PFI shows that the most important channels are in the back of the head in the sensors in visual areas, which is expected for visual task. We found good agreement between this PFI analysis and a standard gradient- baaee analysis, further detailed in the Appendix.
brain response following the end of image
4.5 WEIGHTS ENCODE MEANINGFUL SPATIO-TEMPORA AND SPECTRAL INFORMATION
To provide further insight into non -inear group model, we next show that neuro- scientifically interpretable spatial, temporal, and spectral information can be obtained by analysing the learnt weights. All visualisations are from 3 convolutional layers, with all 6 layers shown in the Appendix. Kernels within a layer seem to have similar temporal sensitivity, even though only 5 are shown from over 1e5 total kernels (Figure 5c). Output deviations are standardised to compare temporal PFI across kernels with different output magnitudes. In the early layers, sensitivity peaks around 100 ms (as in Figure 4b), then rapidly decreases, eventually climbing again slowly. Kernels in early layers have somewhat random spatial sensitivity (Figure 5a), but this gets narrowed down to channels over the visual cortex in deeper layers, with some differences between individual kernels. This sensitivity is similar to the spatial features that were shown to be most informative for classifica- tion performance (see Figure 4b). Figure 5b shows the temporal profile of the spatial PFI. This is achieved by focusing the shuffling to 100 ms time windows and 4-channel neighbourhoods (3 closest channels for each channel), repeated across all timesteps and channels. Spatial sensitivity does not seem to change with time; i.e. the most important channels are always the same.
Spectral PFI measures the change in kernel output to perturbations in specific frequency bands (see Figure 10a in Appendix). Across all layers and kernels, the profile has a 1/f (frequency) shape with clear peak at 10 Hz. These are common features of the MEG signal (Demanuele et al., 2007: Drewes al., 2022), indicating that the spectral sensitivity of the kernels coincides with the power spectra of the data. We also looked at the spectral PFI of 4-channel neighbourhoods (Figure 10b in Appendix) and found that kernels are sensitive the same channels (in the visual area) across all frequencies. with these channels having larger 10 Hz peaks. Kernel FIR analysis shows the power spectra of kernels' outputs when input examples are Gaussian noise (see Figure 10c in Appnedix). The subject embedding was set to a subject with average accuracy. The power spectra were normalised to make visual comparisons between kernels easier. Since the WaveNet architecture uses dilated filters with only 2 values per filter, early layers show broad filtering characteristics, but already in layer 2 more emphasis is put on lower frequencies. In deeper layers, filters (kernels) become more tuned to specific frequencies, generally below 20Hz. This is in line with the spectral properties of MEG data as discussed above. Both the spectral PFI and kernel FIR analysis shows that there is significant variability between the spectral information encoded by various kernels. From the analysis presented in this section, we can conclude that kernels are sensitive to interpretable temporal, spatial, and spectral features of the MEG data.
8

---

Under review as a conference paper at ICLR 2023
Kernel 0
Kernel 1
Kernel
Cutbur Cutbut-deeall destatton Dusse decatton Ourpet desstton
Sarnel Layer =
Oaggut deviutio
Layer
Layero
Gand =1 Layys -
Layst =
Layor
2..5
Garnd Layer
C dedtatto
Layst =
30
Layer
250
500
750
250
520 Time (ms)
730
Time Ims)
(a) Spatial
(b) Channel-wise temporal
(c) Temporal
Figure 5:
insights can be obtained using PFI. Spatial (a), channel-wist temporal (b),
and temporal (c) PFI across non-lineat group-eno emb kernels within 3 layers (rows). For spatial PFI, kernels are plotted separately; whereas for temporal PFI, 5 kernels (lines) are plotted together. Channel-wise temporal PFI shows the temporal PFI of each channel for Kernel 2. Channel colouring matched to the corresponding spatial PFI map, and darker reds mean higher output deviation. For temporal PFI, output deviation is normalised. The horizontal axis shows the time clapsed since image presentation, for both temporal PFI types. 95% confidence intervals are shown with shading.
DISCUSSION
We have proposed a deep learning-based group-level (GL) model that improves significantly over naive GL modelling. in which nothing is learnt about the nature of between-subject variability, achieving similar performance to subject-level (SL) models. We have shown how subject embeddings and non-linearity crucial for this. These are important insights towards the goal of using group models in decoding neuroimaging data, which would allow for better use of this inherently limited resource. Our proposed GL model is especially beneficial in the case of low-accuracy subjects, and can be useful as an initialisation for SL models in the case of across addition, we have demonstrated that we can use PFI to obtain insight into which time points and channels contributed to the decoding and to obtain meaningful information encoded in convolutional kernels. Using this and other methods, such as representational similarity analysis, neuroscientific investigations can be performed at the group level.
decoding. In
We found that in the context of leave-one seave-ooe-uubect-onn evaluation, using subject embeddings did not improve performance. The outcome may be different when our method is applied to larger neuroimaging datasets with more subjects or when subject embeddings are combined with other transfer learning methods, as discussed in Section 4.3. Further research is needed into deep learning models capable of implicitly learning inter-subject variability. An important question is whether scaling up models large datasets would achieve this goal. We expect the subject embedding and group modelling to generalise to different task and recording modalities (EEG, fMRI, etc.) because they face similar decoding challenges. The specific Wavenet-based model is readily generalisable to other clectrophysiologica data such as EEG and Electrocortiocograpty because of the same temporal dynamics they capture. The scope of this work is full-epoch decoding, and it remains to be seen how our methods perform with sliding-window window decoding.
Other than being useful for fine-tuning, our embedding- aided group model can be useful in the case of much larger datasets where we cannot afford to have a separate model for each subject. In neural decoding, group models are widely understood to perform worse than individual models ((Guggenmos et al., 2018; Dash et al., 2020b)). But why is this? By plotting per-subject performance in both kinds of models (see Figure in the Appendix), we see something revealing. While group results are worse average, some subjects are decoded better in the group model than in their individual models. These generally appear to be the subjects who perform worst on the individual models. This suggests that group models could be successfully used for some subjects if those subjects could be identified. Our results suggest follow-up studies to understand why some subjects performed better worse.

---

Under review as a conference paper at ICLR 2023
REFERENCES
André Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer. Permutation importance: a corrected feature importance measure. Bioinformatics, (56100:1340-1333 2010.
Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. Dilated convolutional neural networks for time series forecasting. Journal of Computational Finance, Forthcoming. 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: http:/poooeddnn/// neure neurips. cc/ pcc/paper/2020// (t0p222/f1le/// 45w.14467700000000 967 4 Bbfb8ac1 42 22
Elias Chaibub Neto, Abhishek Pratap, Thanneer M Perumal, Meghasyam Tummalacherla, Phil Snyder, Brian M Bot, Andrew D Trister, Stephen H Friend, Lara Mangravite, and Larsson Omberg. Detecting the impact of subject characteristics on machine learning-based diagnostic applications. NPJ digital medicine, 2(1):1-6, 2019.
Omar Chehab, Alexandre Defossez, Jean- -hrritotte Loiseau, Alexandre Gramfort, and Rean-Remi King. Deep Recurrent Encoder: A scalable end-to-end network to model brain signals. arXiv preprint wwXii:2203.022::0 2021.
Radoslaw Martin Cichy and Dimitrios Pantazis. Multivariate pattern analysis of meg and eeg: A comparison of representational structure in time and space. Neurolmage, 158:441-454 2017.
Radoslaw Martin Cichy, Dimitrios Pantazis, and Aude Oliva. Resolving human object recognition in space and time. Nature neuroscience, 17(3):555-46) 2014.
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific Reports, 6(1):1-13, 2016.
Ciaran Cooney, Raffaella Folli, and Damien Coyle. Optimizing layers improves CNN generalization and transfer learning for imagined speech decoding from EEG. In 2019 IEEE international conference on systems, man and cybernetics (SMC), pp. 1311-1316. IEEE, 2019a.
Ciaran Cooney, Attila Korik, Folli Raffaella, and Damien Coyle. Classification of imagined spoken word-pairs using convolutional neural networks. In The 8th Graz BCI Conference, 2019, pp. 338-343, Verlag der Technischen Universitat Graz, 2019b.
Debadatta Dash, Alan Wisler, Paul Ferrari, and Jun Wang. Towards a Speaker Independent Speech- BCI Using Speaker Adaptation. In INTERSPEECH, pp. 864-868, 2019.
Debadatta Dash, Paul Ferrari, and Jun Wang. Decoding Speech Evoked Jaw Motion from Non- invasive Neuromagnetic Oscillations. In 2020 International Joint Conference Neural Networks (IJCNN) pp. 1-8. IEEE, 2020a.
Debadatta Dash, Paul Ferrari, and Jun Wang. Decoding imagined and spoken phrases from non- invasive neural (MEG) signals. Frontiers in Neuroscience, 14:290, 2020b.
Charmaine Demanuele, Christopher James, and Edmund JS Konuga-Barke Distinguishing low frequency oscillations within the 1/f spectral behaviour of electromagnetic brain signals. Behavioral and Brain Functions, 3(1):1-14, 2007.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota. June 2019. Association for Computational Linguistics. doi: 18.186 10865//11/N19.1..1 1423. URL https::////......... / / maclannhhll ogg. 9- 1423.
10

---

Under review as a conference paper at ICLR 2023
Jan Drewes, Evelyn Muschter, Weina Zhu, and David Melcher. Individual resting-state alpha peak frequency and within-trial changes in alpha peak frequency both predict visual dual- -ull-p segregation performance. Cerebral Cortex, 2022.
Changde Du, Jinpeng Li, Lijie Huang, and Huiguang He. Brain encoding and decoding in fMRI with bidirectional deep generative models. Engineering, 5(5):948-553, 2019.
Venkatesh Elango, Aashish N Patel, Kai J Miller, and Vikash Gilja. Sequence transfer learning for neural decoding. bioRxiv, pp. 210732, 2017.
Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, et al. MEG and EEG data analysis with MNE-Python. Frontiers neuroscience, pp. 267, 2013.
Matthias Guggenmos, Philipp Sterzer, and Radoslaw Martin Cichy. Multivariate pattern analysis for MEG: A comparison of dissimilarity measures. Neuroimage, 173:434 2018
Hanna-Leena Halme and Lauri Parkkonen. Across-subject offline decoding of motor imagery from MEG and EEG. Scientific reports, 8(1):1-12, 2018.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchamn Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy, Nature, 585(7825) September 2020. doi: 10. 1088//41586-000226000 -20-0000022200000
URL htwps::pp/..... /tw.///oii/ii/////// htt.//ooooooooo/o//oo//oo/ooo / 1038011300000000
Cameron J Higgins, Mats WJ van Es, Andrew J Quinn, Diego Vidaurre, and Mark W Woolrich. The relationship between frequency content and representational dynamics in the decoding of neurophysiological data. bioRxiv, 2022.
Annika Hultén, Marijn van Vliet, Sasa Kivisaari, Lotta Lammi, Tiina Knuutila Ali Faisal, and Ritta Salmelin. The neural representation of abstract words may arise through grounding word meaning in language itself. Human brain mapping, 42(15):493344984 2021.
J. D. Hunter. Matplotlib: A 2D graphics environment. Computing In Science & Engineering, 9()): 90-95, 2007.
Kendrick N Kay, Thomas Naselaris, Ryan Prenger, and Jack Gallant. Identifying natural images from human brain activity. Nature, 452
2008.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLhttp: http//aaxvvv.../.aaea / 1422. 6980.
Demetres Kostas and Frank Rudzicz. Thinker invariance: enabling deep neural networks for bei across more people. Journal of Neural Engineering, 17(5):056008, 2020.
Demetres Kostas, Stephane Aroca-Ouellette, and Frank Rudzicz. Bendr: using transformers and a contrastive self-supervised learning task learn from massive amounts of eeg data. Frontiers in Human Neuroscience, pp. 253, 2021.
Vernon J Lawhern, Amelia Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent Lance. Eegnet: a compact convolutional neural network for eeg-based brain compucr interfaces. Journal of neural engineering, 15(5)0056013, 2018.
Jingcong Li, Jiahui Pan, Fei Wang, and Zhuliang Yu. Inter-Subject MEG Decoding for Visual Information with Hybrid Gated Recurrent Network. Applied Sciences, 11(3):1215, 2021.
Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. A Persona- Based Neural Conversation Model. In Proceedings the 54th Anmual Meeting of the Association for Computational Linguistics, pp. 994-1003. Association for Computational Linguisties, August 2016. URL http: / www. web. org/ /aaatthllogy/peeeeee 1094.
11

---

Under review as a conference paper at ICLR 2023
Shusen Liu, Peer-Timo Bremer, Jayaraman Thiagarajan, Vivek Srikumar, Bei Wang, Yarden Livnat, and Valerio Pascucci. Visual exploration of semantic relationships neural word embeddings. IEEE transactions visualization and computer graphics, 24(1):553-56,, 2017.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks, In International conference on machine learning, pp. 97-105. PMLR, 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems
26, pp. 3111-3119. Curran Associates, Inc., 2013. URL https: /tttt wwtpp.i......i.t.......
Muhammad Firoz Mridha, Abu Quwsar Ohi, Muhammad Mostafa Monowar, Md Hamid, Md Islam, Yutaka Watanobe, et al. U-vectors: Generating clusterable speaker embedding from unlabeled data. Applied Sciences, 1(21):10079, 2021.
Emily M Mugler, James L Patton, Robert D Flint, Zachary A Wright, Stephan U Schuele, Joshua Rosenow, Jerry J Shih, Dean Krusienski, and Mare W Slutzky. Direct classification of all American English phonemes using signals from functional speech motor cortex. Journal of Neural Engineering, 1 (3):03505,,, 2014.
W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings the National Academy of Sciences, (16(44)22071-22268 2019.
Debarshi Nath, Mrigank Singh, Divyashikha Sethia, Diksha Kalra, and Indu. A comparative study of subject-dependent and subject-independent strategies for eeg-based emotion recognition using 1stm network. In Proceedings of the 2020 the 4th International Conference on Compute and Data Analysis, pp. 142-147, 2020.
Emanuele Olivetti, Seved Mostafa Kia, and Paolo Avesani. MEG decoding across subjects. In 2014 International Workshop Pattern Recognition in Neuroimaging, pp. 1-4. IEEE, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn; Machine learning in Python. Journal of Machine Learning Research, 22:2825-28300 2011.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Ro- bust speech recognition via large-scale weak supervision. 2022. URL hattps://ceet openai. com/papes rs/ wh whiisper... .df.
Simanto Saha and Mathias Baumert. Intra-and inter-subject variability in EEG-based sensorimotor brain computer interface: a review. Frontiers in computational neuroscience, pp. 87, 2020.
Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari. DNN-based speaker embedding using subjective inter-speaker similarity for multi-speaker modeling in speech synthesis. arXiv preprint arXiv:1907.08294, 2019,
Robin Tibor Schirmeesterr, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping, 38(11):5391-544 2017.
12

---

Under review as a conference paper at ICLR 2023
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. In Proc. 9th ISCA Workshop on Speech Synthesis Workshop (SSW 9), pp. 125, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 1), 2008.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Fric W. Moore, Jake VanderPlas, Denis Laxalde. Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-277 2020. doi:
Kafeng Wang, Xitong Gao, Yiren Zhao, Xingjian Li, Dejing Dou, and Cheng-Zhong Xu. Pay attention to features, transfer learn faster CNNs. In International conference learning representations, 2019.
Michael L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60): 3021, 2021. doi: 10.21 10.21 105/josss.000000 03021. URL https:://dot wtw.:.....r........ 21105/ 03021.
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 56 61, 2010.
doi: 10.25080/Majora- (attlorr-92bf19999 92bf1922-066
Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. High-performance brain-to-text communication via handwriting. Nature, 593(7858): 249-254, 2021.
Saizheng Zhang, Emily Dinan, Jack Urbanck, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing Dialogue Agents: I have dog, do you have pets too? In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Long Papers), pp. 2204 2213, Association for Computational Linguistics, July 2018. URL http: hawpwwb.wwbb/g/// anthology/Pi /P18 1205.
Xulong Zhang, Jianzong Wang. Ning Cheng, and Jing Xiao. MDCNN-SID: Multi-scale Dilated Convolution Network for Singer Identification. arXiv preprint 2xXi:2004.003371, 2020.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76, 2020.
Ivan Zubarev, Rasmus Zetter, Hanna-Leena Halme, and Lauri Parkkonen. Adaptive neural network classifier for decoding meg signals. Neuroimage, 197:425 434,,43422200 2019.
A APPENDIX
Dropout was set to 0.4 and 0.7, and a batch size of 590 and 59 was used for GL and SL models, respectively. Learning rate was set to 0,0001 for GL. and 0.00005 for SL models. Training of single SL and GL model took 5-15 minutes and 4 hours on an NVIDIA A100 GPU, respectively. For linear models, validation losses (cross-entropy) and accuracies were negatively correlated, i.e. loss decreases while accuracy increases, and eventually both suggested overfitting. Since non-linear models are more expressive, they overfitted sooner according to the loss, but accuracy kept improving until it reached a plateau, never overfitting. Analysing the loss distribution across validation examples (for non- -inear models) shows that even during overfitting most examples' loss keeps decreasing with few high-loss outliers disproportionately influencing the mean. Since accuracy is binary, outliers are diminished, explaining the apparent difference in learning behaviour. For linear models, this unintuitive behaviour was not observed probably due to inherent model simplicity.
13

---

Under review as a conference paper at ICLR 2023
0.0005
0.0004
Gradient magnitude
0.0003
0.4
Gradien
0.4
0.3
0.0002
0.2
0.2
0.1
0.0001
0.1
0.0
200
400
600
800
Time
(a) Temporal gradient analysis
(b) Spatial gradient analysis
Figure 6: Using gradient analysis by backpropagating the loss to randomly initialized inputs. In (a) we can see the temporal profile of the gradients averaged over channels. In (b) we can see the spatial profile of the gradients averaged over time.
We analysed our main findings on another publicty available visual MEG dataset with 92 different images (15 subjects, and 30 trials per image) (Cichy et al., 2014). Linear SL models achieved 35% accuracy, whereas a linear group model without embeddings had 12%, and nonlinear group model with embeddings had 28%. Thus we can see that our approach behaves similarly on this dataset, improving lot over the naive group baseline, but not quite achieving the performance of the linear SL models. Finetuning the group model separately on individual subjects achieved 38% accuracy surpassing from-scratch SL models.
Training with an embedding dimensionality of and 14, resulted in 20% and 38% accuracy, re- spectively. We tried these two settings to see how embedding size in the lower and upper limits influences performance, As an embedding dimensionality of 14 performs the same as 10, we could draw the conclusion that 10 is not limiting factor. From the much worse result with embedding dimensionality of 3 we could draw the conclusion that compressing the embedding representations too much is not possible. As with the clustering analysis, this is likely to be due having few subjects. To gain further insight into the learned subject embeddings we computed accuracy on each subject's s validation data using other subjects' embeddings. In the resulting subject-by-subie confusion matrix the value in the i-th row jnd j-th column shows how well the embedding of subject i can be replaced with the embedding of subject j (Figure 7). After division with the original accuracies the metric shows how much accuracy can be retained when swapping subject embeddings. Some subjects' embedding cannot be replaced by others (e.g. subject 3), and some subjects' embedding can be more easily replaced (e.g. subject 12). Conversely, some subjects' embeddings are more general as they can replace many others (e.g. subject 14), and some are less general (e.g. subject 2). We tried clustering this matrix, and looked at correlation with both embedding distance and subject accuracy, however no meaningful results were found.
We compared the accuracy beased PFI analysis with standard gradient- based analysis where saliency map is obtained by backpropagating to randomly initialized inputs (Figure 6. We smoothed the temporal profile with the same window size for the PFI analysis. Temporally that the agreement between the two methods is high, with peaks aligning very well (less than 10ms difference). Spatially the two methods do show some differences, but overall gradient analysis still points to the most important information being in the visual cortex.
can see
14

---

Under review as a conference paper at ICLR 2023
linear subject
nonlinear subject
nonlinear group-emb
3 conv layers conv layers
0.45
0.22 0.39
0.41
0.25
0.38
Table 1: Effect of number of convolutional layers on the validation accuracy of two subject-level and one group-level model.
E0
E1
E2
E3
E4
E5
E6
E7
E8
89
E10
E11
E12
E13
E14
Vo
0.16
0.09
0.08
0.26
0.21
0.13
0.12
0.13
0.18
0.12
0.10
0.20
0.11
0.19
V1
0.13
0.04
0.11
0.11
0.09
0.10
0.13
0.11
0.18
0.12
0.10
0.13
0.03
0.15
V2
0.09
0.05
0.14
0.06
0.10
0.08
0.10
0.10
0.11
0.13
0.10
0.10
0.08
0.14
V3
0.04
0.04
0.07
0.03
0.06
0.03
0.05
0.05
0.06
0.06
0.06
0.08 0.03
0.07
V4
0.36
0.16
0.12
0.07
0.17
0.33
0.24
0.13
0.25
0.16
0.16
0.23
0.15
0.35
V5
0.15
0.08
0.08
0.11
0.11
0.09
0.16
0.18
0.14
0.08
0.08
0.27
0.11 0.11
V6
0.14
0.11
0.05
0.05
0.23
0.08
0.09
0.09
0.15 0.08
0.12
0.09
0.10
0.28
V7
0.23
0.16
0.17
0.13
0.26
0.29
0.21
0.24
0.21
0.26
0.33
0.25
0.23 0.29
V8
0.15
0.10
0.05
0.07
0.11
0.11
0.07
0.17
0.13 0.14
0.13
0.18
0.07
0.10
V9
0.25
0.24
0.12
0.12
0.19
0.15
0.19 0.20
0.16
0.27
0.16
0.20
0.03
0.32
V10
0.29
0.13
0.11
0.10
0.16
0.15 0.15
0.24 0.24 0.25
0.15 0.25
0.06
0.23
V11
0.11
0.15
0.05
0.10
0.18
0.12
0.10
0.21 0.13 0.19 0.10
0.15
0.18
0.18
V12
0.60
0.30
0.23
0.19
0.40
0.49
0.30
0.37 0.47 0.42 0.49
0.42
0.28 0.28
V13
0.10
0.06
0.05
0.05
0.14
0.07
0.10
0.10
0.07
0.04
0.06
0.14
0.13
0.09
V14
0.19
0.10
0.05
0.06
0.30
0.10
0.29
0.15
0.10 0.20
0.12
0.17
0.18
0.17
Figure 7: Subject embedding confusion matrix. Columns (E0-E14) refer to subject embedding indices and rows (V0-V (14) refer subject validation sets. Greener shading (higher values) shows subjects with higher retained accuracy when their embeddings are swapped.
200
100
0
-100
-200
-300
-200
-100
100
200
Figure 2D Tsne projection of the subject embeddings in the nonlinear group-embedding model.
15

---

Under review as a conference paper at ICLR 2023
0.9
0.9
0.9
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.8
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0
0
nonlinear subject
nonlinear
nonlinear group-emb
nonlinear
nonlinear
nonlinear
group-emb
group-emb finetuned
subject
group-emb
finetuned
(a) Subject to group
(b) Subject to finetuned
(c) Group to finetuned
Figure 9: Accuracy changes across all 15 subjects (individual colours), when comparing SI., GL., and finetuned group models. Group models clearly reduce the variability of accuracies across subjects, and are especially helpful for low-accuracy subjects. When finetuning our group model on individual subjects (c), we see that accuracy increases for all subjects, and especially for high accuracy subjects. This is unsurprising because these subjects have good enough data on their own for SL models to be able to learn well. As seen in (a) and (b) these high-accuracy subjects are usually impaired by GL models, exactly for the aforementioned reason.
DMECAE dnlitnn outtr aenatin Oxipr devatin
devation duppe deciattonn
Layer
Layer 01K
Layer OKK1
Layer
Power
0.0
Layer 2
outpue desatton outpur deratto
Layer 21K
Layer - 1K=1
Layer - 2
poverr
2.5
25-
2.5
0.0
Layer
Layer
Layer 5 51K=
Layer +5 5|K=
Layer - K5
Power
100 Marhhadee
200
50 Prequency dicl
50
100
50
100
Frequenry (12)
Frequency (10)
Trequency (19)
Trequency 0to)
(a) Spectral PFI
(b) Chn-wise spectral PFI
(c) Kernel FIR analysis
Figure 10: Frequency sensitivity of kernels via spectral PFI (a), channel-wise spectral PFI (b), and frequency characteristics via kernel FIR analysis (c), from 3 layers (rows). Kernels are plotted together (lines) for spectral PFI, and in separate columns for kernel FIR analysis (normalised). Each channel-wise spectral PFI plot is for kernel, where lines show the spectral PFI of corresponding channels in the sensor space map. 95% confidence intervals are shown with shading for spectral PFI. Due to small variability across permutations, this is barely visible. For spectral PRI the hand-width was set to 5 Hz. to obtain smooth frequency profile.
16

---

Under review as a conference paper at ICLR 2023
Kernel 0
Kerne 1
Kernel 2
Kernel 3
Kernel 4
Layer O
Layer
Layer 2
Layer 3
Layer
Layer
Figure 11: Spatial PFI across 6 layers (rows), with kernels per row. Darker reds mean higher output deviation.
17

---

Under review as a conference paper at ICLR 2023
Output deviation Output deviation
Kernel 4 Layer =
Output deviation
Layer 0
e
10-
0
4 Layer
Output deviation
2.5
Layer 1
30
20
10
-2.5
Output deviation
Kernel Layer
Output deviation
2.5
Layer
40
0.0
20
-2.
Output deviation
Kernel Layer
Output deviation
Layer
30
10
Output deviation Output deviation
Kernel 4 Layer
Output deviation
Layer
0.0
20
-2.5
Kernel 4 Layer =
Layer
Output deviation
30
10
-2
250
500
750
250
500
750
(ms)
Chn Temporal PFI
(b) Temporal PFI
Figure 12: Channelwise temporal PFI (a), and temporal PFI (b) across kernels of the non-lineaz group-emb model 6 layers (rows). For temporal PFI 5 kernels (lines) are plotted together. Channelwise temporal PFI shows the temporal PFI of each channel for Kernel 5. Channel coloring is matched to the corresponding spatial PFI map, and darker reds mean higher output deviation. For temporal PFI output deviation is normalized. The horizontal axis shows time elapsed since image presentation for both temporal PFI types. 95% confidence interval shown with shading.
18

---

Under review as a conference paper at ICLR 2023
Layer
Output deviation
200
Output deviation
7.5
100
2.5
Layer
Layer
Output deviation
Output deviation
40
200
20
100
Layer
Output deviation
Output deviation
200
40
100
20
Layer
Layer
Output deviation
Output deviation
20
0
Layer
Layer
Output deviation
Output deviation
75
50
25
Layer 5
Layer 5
Output deviation
Output deviation
100
40
50
20
50
100
50
100
Frequency (Hz)
Frequency (H2)
(a) Spectral PFT
(b) Chn Spectral PFI
Figure 13: Frequency sensitivity of kernels via spectral PFI (a), channelwise spectral PFI (b) from layers (rows). Kernels plotted together (lines) for spectral PFI. Each channelwise spectral PFI plot for kernel, where lines show the spectral PFI of corresponding channels the topomap. 95% confidence interval is shown with shading for spectral PFI. Due small variability across permutations this is barely visible.
19

---

Under review as a conference paper at ICLR 2023
Layer K
Layer k|
Layer O =
Layer K
Layer 0 K
Power
2
25
0.0
0.0
1
K
1|K-3
Layer 11K-4
5.0
Power
5,0
2.5
25
2.5
Layer 2| K =
Layer 2
Layer 2 2
Layer 2 2|K
Layer 2|K-4
5.0
2.5
25
25
2.5
0.0
0.0
0.0
Layer K
Layer 3|K-1
Layar K
Layer - 3|K
Layer K|K=
5.0-
Power
2.5-
2.5
2.5
0.0
0.0
Layer K
Layer 4
=|k=3
Layer K
Power
-mmm
2.5
2.5
2.5
Layer - K
Layar - 51K-1
Layor - 51K-2
Layer - 5
Layer 5 1K
Power
Frequency (H2)
Irequency (Hz)
trequency 0tzt
Prequency (Mt)
Irequency 012)
Figure 14: Frequency characteristics of 5 kernels across 6 layers (rows) via kernel FIR analysis. The power spectra is normalized.
20


Reviews:
[{"review_id": "rXdaH_WRnl", "reviewer": "Reviewer_MQXD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript analyzes the use of subject embeddings to enhance deep learning models trained to decode MEG recordings of several subjects. Concretely, the work analyzes decoding MEG data to classify which of 118 different images 15 different subjects looked at. They use a convolutional neural network using dilated convolutions inspired by WaveNet. and evaluate using a vanilla model without subject embeddings as well as a larger model that also uses subject embeddings. They report that using the subject embeddings improves accuracies and when combined with finetuning on an individual subject can even slightly outperform subject-level models. Furthermore, analysis of the trained models partly shows plausible results.", "summary_of_the_review": "Overall, this seems a nice manuscript with some additional analyses needed in my view.", "strength_and_weaknesses": "**Update**\nDue to the including of the reference, the clarifications regarding number of layers and the additional gradient-based analyses, have increased my score.\n\n**Pre-rebuttal**\nThe manuscript is quite detailed and evaluates many aspects of a clearly described idea.\n\nIt is valuable that the work looks at various settings like linear/nonlinear with/without embedding and also including subject-level finetuning. Also the additional analysis of the trained models is interesting.\n\nSome open questions for me:\n“For SL modelling, the Wavenet Classifier contains 3 convolutional layers, whereas for group modelling it has 6, further motivated in Section 4. “\n\nI could not quite find the part where this is written maybe I missed it? In any case, would be nice to also evaluate the 6-layer classifier on subject-level decoding, as it may also extract different frequencies with its deeper structure. So these results would be an important addition, also to disentangle effects of larger model and effect of subject embedding etc.\n\nIn general, it would be good to evaluate further published EEG deep learning models/pipelines on this task to know how the reported accuracies compare to the literature. Also, I assume this dataset has been decoded before, how where accuracies in the literature?\n\nI was also missing at least one reference on transfer learning for neurophysiological recordings, \nhttps://iopscience.iop.org/article/10.1088/1741-2552/abb7a7 \nAlso this or other works would provide interesting baselines for the method presented in this manuscript.\n\n\nRegarding the PFI analysis, it would be nice to additionally perform a computationally simpler gradient-based analysis, note that both fourier transform and inverse fourier transform are differentiable and hence one can also compute gradients on fourier coefficients. It would be good to see if these analysis agree or in how far they disagree.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gJgCoxL0u06", "reviewer": "Reviewer_bWS9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is concerned with improving classification of brain activity (MEG). Typically, this would be done by training a model for each subject. But this work presents a strategy for pooling data across multiple subjects (n=15) to produce a \"group-level\" model. In the naive approach, this can be done by combining data from multiple subjects into one training set. But in this work, they augment each training example with a subject embedding, which is updated during training. They find that a group-level model produces predictions which have a 6% disadvantage to the subject-level models, but a 5% advantage after fine-tuning. The paper also includes an analysis of the model's trained weights and what they reveal about visual processing in the brain. ", "summary_of_the_review": "In the work's current state, I vote to reject. The approach as it stands cannot be used for transfer-learning, which could be one of the main benefits of a group-level model, if accomplished. In its current form, the proposed method doubles the training time and number of weights in order to achieve a 5% increase over the baseline subject-level model.", "strength_and_weaknesses": "# Strengths\n- The text is well written\n- The analysis of MEG signal using trained model weights (section 4.5) shows promise for future neuroscientific study.\n\n# Weaknesses\n- The model is not suitable for use in transfer learning. The leave-one-subject-out analysis shows that the naive baseline performs just as well as the proposed model, and that better group performance does not translate to better fine-tuning performance on the held-out subject. The authors rightly note that this is a non-trivial problem, but similar transfer-learning solutions have shown good progress for NLP and vision, and it would greatly strengthen the significance of these results if something analogous could be accomplished for the MEG domain.\n- It should also be noted that the group-level architecture is ~2x larger than the subject-level architecture. The authors say that subject-level performance plateaus beyond a certain number of parameters (3 layers), but it would be nice to see this in a table.  Alternatively, the authors could show results for a group-level architecture that is the same size (3 layers) as the subject-level model. As it stands, there remains room to believe that the presented gains over the baseline could be entirely explained by simply using a larger model.\n- See below for comments on significance and novelty ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L-OlJDZYuN", "reviewer": "Reviewer_kPfZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a deep learning-based group-level (GL) neural decoding model that can be adapted to different subjects. The key module of the GL model is the learnable subject embeddings. Experiments were conducted on an MEG neural dataset. Experimental results indicated that the proposed GL model can achieve significantly higher classification accuracy than the naive GL modelling. The paper also gained insight into how subject embeddings help the group model and analyzed the spatial and temporal feature importance.\nThe main contributions of this paper are as follows.\n1.The paper builds a GL neural decoding model that can be adapted to different subjects.\n2.The paper gains insights into the learned subject embedding.\n3.The paper gives analysis of model weights to reveal how meaningful spatio-temporal and spectral information is encoded.", "summary_of_the_review": "More than an innovative work on representation learning, the work applies an existing method to a new task. I think the quality of the paper is not up to the average quality of the papers accepted by ICLR. Also, the paper is not quite in line with the main scope of ICLR. The article also does a rich analysis from the neuroscience perspective. I suggest the authors to submit their paper to conferences or journals in the field of computational neuroscience after major revisions.", "strength_and_weaknesses": "Strength:\n1.This paper builds a GL neural decoding model that can be adapted to different subjects. The key module is the learnable subject embedding, which is simple to apply. In this way, researchers can aggregate multi-subject neural signals to train a GL model and finetune it on the specific target subject. Considering the sample size of neural datasets is always limited, the GL model is meaningful to improve the model performance. Although the model is only validated on an MEG dataset in this paper, I think it can also be applicable to other modalities, such as EEG and fMRI.\n\nWeaknesses:\n1.The methodology novelty is very limited. The feature extraction network is similar to the WaveNet. The subject embedding trick and the analyses on the relationship between neural responses and features with PFI have been applied to MEG neural encoding in (Chehab et al. 2021). \n\n2.The paper is a work in progress. The model is only validated on an MEG dataset. It is recommended that the authors perform validation on at least two neural dataset. The results and analyses are not impressive.\n\n3.The LOSO experimental results show that the advantage of the subject embeddings cannot transfer to new subjects. How to leverage transfer learning with the proposed model and learn a useful embedding for the new subject in an unsupervised manner as described in Page 7? The authors only show a concept and don’t conduct any experiment.\n\n4.Many descriptions on the experimental results rather than concrete data in the form of figures and tables. For example, the results of finetuning a naive group model indicated in Page 6 have not been shown in Figure 3. In Section 4.2, the impact of the network layer on accuracy is not shown in corresponding charts. The authors said that the visualization of subject embeddings did not show any clusters and no visualization is shown in the paper or the appendix.\n\n5.Many details are missing. For example, why did the authors change the embedding dimensionality from 10 only to 3 and 14? How about other settings?\n\n6.The improvement of GL model compared with the naïve GL model is significant. However, the naïve group models are too weak to be supportive baseline models. A network module is also proposed in the literature [1] to address the between-subject variability. I suggest that the authors should take a boarder review and compare their model with stronger baselines.\n\n[1] Défossez, A., Caucheteux, C., Rapin, J., Kabeli, O., & King, J. R. (2022). Decoding speech from non-invasive brain recordings. arXiv preprint arXiv:2208.12266.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9DePmgbSiE", "reviewer": "Reviewer_NEjD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an interesting discussion and MEG classification results in SL vs. GL settings. The submission is not a good match for ICLR, and it would interest the applied neuroscience community. The technical contribution of a relatively standard/off-the-shelf WaveNet to MEG is the major weakness of the paper. Technical details on the reproduction of the model (see details in a section below) are missing, and a promise in the abstract, \"All code is available on GitHub,\" lands an anonymized link (a GitHub search for the code would result probably in the authors' identification). ", "summary_of_the_review": "The manuscript is not a good match for the ICLR. Little ML novelty and extended results discussion from a neuroscientific application point of view would contribute to the SfN poster but not the ICLR. ", "strength_and_weaknesses": "Strength: A relatively standard WaveNet application to MEG is unfortunately out of scope for ICLR.\n\nWeaknesses: Missing technical details. The authors include very vague technical descriptions of MEG preprocessing as follows:\n\"Raw data is bandpass filtered between 0.1 and 125 Hz, and line noise is removed with notch filters. After downsampling to 250 Hz, 1.024-second epochs are extracted, starting 100 ms before stimulus presentation. This resulted in 306 x 256-dimensional trials (channels x timesteps) from the 306 MEG sensors. Whitening is used to remove covariance between channels for SL models, whereas for GL models, standardization is performed per channel. We do multiclass decoding, predicting a separate probability for each of the 118 classes (images).\"\nWhat kind of filters (FIR, IIR, etc.) were used? What downsampling and whitening procedures were applied (the reviewer had no access to the code)? Why 118 classes (that would be a rather revolutionary BCI)? \n\nSimilar problems continue with the WaveNet model's vague description. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject