Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This manuscript analyzes the use of subject embeddings to enhance deep learning models trained to decode MEG recordings of several subjects. Concretely, the work analyzes decoding MEG data to classify which of 118 different images 15 different subjects looked at. They use a convolutional neural network using dilated convolutions inspired by WaveNet. and evaluate using a vanilla model without subject embeddings as well as a larger model that also uses subject embeddings. They report that using the subject embeddings improves accuracies and when combined with finetuning on an individual subject can even slightly outperform subject-level models. Furthermore, analysis of the trained models partly shows plausible results.
strength_and_weaknesses: **Update**
Due to the including of the reference, the clarifications regarding number of layers and the additional gradient-based analyses, have increased my score.

**Pre-rebuttal**
The manuscript is quite detailed and evaluates many aspects of a clearly described idea.

It is valuable that the work looks at various settings like linear/nonlinear with/without embedding and also including subject-level finetuning. Also the additional analysis of the trained models is interesting.

Some open questions for me:
“For SL modelling, the Wavenet Classifier contains 3 convolutional layers, whereas for group modelling it has 6, further motivated in Section 4. “

I could not quite find the part where this is written maybe I missed it? In any case, would be nice to also evaluate the 6-layer classifier on subject-level decoding, as it may also extract different frequencies with its deeper structure. So these results would be an important addition, also to disentangle effects of larger model and effect of subject embedding etc.

In general, it would be good to evaluate further published EEG deep learning models/pipelines on this task to know how the reported accuracies compare to the literature. Also, I assume this dataset has been decoded before, how where accuracies in the literature?

I was also missing at least one reference on transfer learning for neurophysiological recordings, 
https://iopscience.iop.org/article/10.1088/1741-2552/abb7a7 
Also this or other works would provide interesting baselines for the method presented in this manuscript.


Regarding the PFI analysis, it would be nice to additionally perform a computationally simpler gradient-based analysis, note that both fourier transform and inverse fourier transform are differentiable and hence one can also compute gradients on fourier coefficients. It would be good to see if these analysis agree or in how far they disagree.

clarity,_quality,_novelty_and_reproducibility: The manuscript is fairly clearly written, seems novel to the best of my knowledge and code is available.
summary_of_the_review: Overall, this seems a nice manuscript with some additional analyses needed in my view.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present an interesting discussion and MEG classification results in SL vs. GL settings. The submission is not a good match for ICLR, and it would interest the applied neuroscience community. The technical contribution of a relatively standard/off-the-shelf WaveNet to MEG is the major weakness of the paper. Technical details on the reproduction of the model (see details in a section below) are missing, and a promise in the abstract, "All code is available on GitHub," lands an anonymized link (a GitHub search for the code would result probably in the authors' identification). 
strength_and_weaknesses: Strength: A relatively standard WaveNet application to MEG is unfortunately out of scope for ICLR.

Weaknesses: Missing technical details. The authors include very vague technical descriptions of MEG preprocessing as follows:
"Raw data is bandpass filtered between 0.1 and 125 Hz, and line noise is removed with notch filters. After downsampling to 250 Hz, 1.024-second epochs are extracted, starting 100 ms before stimulus presentation. This resulted in 306 x 256-dimensional trials (channels x timesteps) from the 306 MEG sensors. Whitening is used to remove covariance between channels for SL models, whereas for GL models, standardization is performed per channel. We do multiclass decoding, predicting a separate probability for each of the 118 classes (images)."
What kind of filters (FIR, IIR, etc.) were used? What downsampling and whitening procedures were applied (the reviewer had no access to the code)? Why 118 classes (that would be a rather revolutionary BCI)? 

Similar problems continue with the WaveNet model's vague description. 

clarity,_quality,_novelty_and_reproducibility: The dataset is freely available, but the GitHub code would probably be available after manuscript acceptance, which is a problem for the reviewer.
The current ML model and MEG preprocessing descriptions are too vague for reproducibility. The vague description makes the WaveNet application appear standard and without significant novelty. 
summary_of_the_review: The manuscript is not a good match for the ICLR. Little ML novelty and extended results discussion from a neuroscientific application point of view would contribute to the SfN poster but not the ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work is concerned with improving classification of brain activity (MEG). Typically, this would be done by training a model for each subject. But this work presents a strategy for pooling data across multiple subjects (n=15) to produce a "group-level" model. In the naive approach, this can be done by combining data from multiple subjects into one training set. But in this work, they augment each training example with a subject embedding, which is updated during training. They find that a group-level model produces predictions which have a 6% disadvantage to the subject-level models, but a 5% advantage after fine-tuning. The paper also includes an analysis of the model's trained weights and what they reveal about visual processing in the brain. 
strength_and_weaknesses: # Strengths
- The text is well written
- The analysis of MEG signal using trained model weights (section 4.5) shows promise for future neuroscientific study.

# Weaknesses
- The model is not suitable for use in transfer learning. The leave-one-subject-out analysis shows that the naive baseline performs just as well as the proposed model, and that better group performance does not translate to better fine-tuning performance on the held-out subject. The authors rightly note that this is a non-trivial problem, but similar transfer-learning solutions have shown good progress for NLP and vision, and it would greatly strengthen the significance of these results if something analogous could be accomplished for the MEG domain.
- It should also be noted that the group-level architecture is ~2x larger than the subject-level architecture. The authors say that subject-level performance plateaus beyond a certain number of parameters (3 layers), but it would be nice to see this in a table.  Alternatively, the authors could show results for a group-level architecture that is the same size (3 layers) as the subject-level model. As it stands, there remains room to believe that the presented gains over the baseline could be entirely explained by simply using a larger model.
- See below for comments on significance and novelty 
clarity,_quality,_novelty_and_reproducibility: - I'm worried about the significance of this work. The presented group-level model performs 6% worse than the subject-level baseline. Fine-tuning on the subject data results in a 5% performance increase, but this greatly narrows the real world applicability of this approach. In the very best case, to see any benefit from this method, one has to pool all their data, train a group level model, and then fine-tune these weights for each subject, essentially doubling the training time and size of the architecture. And even then, the benefit is only a 5% increase.
- And this benefit is not uniform across subjects, indeed for some subjects, the performance slightly decreases (first paragraph of pg. 6).
- The novelty of the approach also concerns me. Compared to the naive group-level approach, the presented approach is essentially identical, but with an extra piece of input data, namely the identity of the subject. Simplicity in itself is not necessarily a bad thing. But I would argue that this paper does not present any new approach, but rather a simple comparison between models: one which has access to a particular bit of information, and one that does not. 
summary_of_the_review: In the work's current state, I vote to reject. The approach as it stands cannot be used for transfer-learning, which could be one of the main benefits of a group-level model, if accomplished. In its current form, the proposed method doubles the training time and number of weights in order to achieve a 5% increase over the baseline subject-level model.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a deep learning-based group-level (GL) neural decoding model that can be adapted to different subjects. The key module of the GL model is the learnable subject embeddings. Experiments were conducted on an MEG neural dataset. Experimental results indicated that the proposed GL model can achieve significantly higher classification accuracy than the naive GL modelling. The paper also gained insight into how subject embeddings help the group model and analyzed the spatial and temporal feature importance.
The main contributions of this paper are as follows.
1.The paper builds a GL neural decoding model that can be adapted to different subjects.
2.The paper gains insights into the learned subject embedding.
3.The paper gives analysis of model weights to reveal how meaningful spatio-temporal and spectral information is encoded.
strength_and_weaknesses: Strength:
1.This paper builds a GL neural decoding model that can be adapted to different subjects. The key module is the learnable subject embedding, which is simple to apply. In this way, researchers can aggregate multi-subject neural signals to train a GL model and finetune it on the specific target subject. Considering the sample size of neural datasets is always limited, the GL model is meaningful to improve the model performance. Although the model is only validated on an MEG dataset in this paper, I think it can also be applicable to other modalities, such as EEG and fMRI.

Weaknesses:
1.The methodology novelty is very limited. The feature extraction network is similar to the WaveNet. The subject embedding trick and the analyses on the relationship between neural responses and features with PFI have been applied to MEG neural encoding in (Chehab et al. 2021). 

2.The paper is a work in progress. The model is only validated on an MEG dataset. It is recommended that the authors perform validation on at least two neural dataset. The results and analyses are not impressive.

3.The LOSO experimental results show that the advantage of the subject embeddings cannot transfer to new subjects. How to leverage transfer learning with the proposed model and learn a useful embedding for the new subject in an unsupervised manner as described in Page 7? The authors only show a concept and don’t conduct any experiment.

4.Many descriptions on the experimental results rather than concrete data in the form of figures and tables. For example, the results of finetuning a naive group model indicated in Page 6 have not been shown in Figure 3. In Section 4.2, the impact of the network layer on accuracy is not shown in corresponding charts. The authors said that the visualization of subject embeddings did not show any clusters and no visualization is shown in the paper or the appendix.

5.Many details are missing. For example, why did the authors change the embedding dimensionality from 10 only to 3 and 14? How about other settings?

6.The improvement of GL model compared with the naïve GL model is significant. However, the naïve group models are too weak to be supportive baseline models. A network module is also proposed in the literature [1] to address the between-subject variability. I suggest that the authors should take a boarder review and compare their model with stronger baselines.

[1] Défossez, A., Caucheteux, C., Rapin, J., Kabeli, O., & King, J. R. (2022). Decoding speech from non-invasive brain recordings. arXiv preprint arXiv:2208.12266.
clarity,_quality,_novelty_and_reproducibility: Quality: Not very good. The novelty is very limited. Although the experimental results show that the proposed method can significantly improve the performance, the authors only validate their method on one MEG dataset and compare it with weak baselines.
Clarity: The method description is clear. But some detailed descriptions on the experimental results are not very clear.
Originality of the work: not very original.
Reproducibility: The code is publicly available, and the datased is also publicly available. The reproducibility is high.
summary_of_the_review: More than an innovative work on representation learning, the work applies an existing method to a new task. I think the quality of the paper is not up to the average quality of the papers accepted by ICLR. Also, the paper is not quite in line with the main scope of ICLR. The article also does a rich analysis from the neuroscience perspective. I suggest the authors to submit their paper to conferences or journals in the field of computational neuroscience after major revisions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: Improving decoding of brain activity from MEG by aggregating data across subjects while tagging the training data with a subject-specific embedding.

Strengths: The method may be useful for increasing decoding performance.

Weaknesses: The method is of limited interest and this is reflected in the scores. It's unclear what the rest of the community has to learn from the method or add to it. The general idea of subject/task embeddings is not new and the novelty of the application is limited. A different venue may well receive the work with more enthusiasm.
justification_for_why_not_higher_score: Even the most enthusiastic reviewers were not particularly convinced. The method is of limited interest for ICLR; it's unclear how this community can engage with the work. The core idea behind the method is well-known and the prospects for building on the work are limited.
Decision: Reject