Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an SSL framework for EEG and evaluate for seizure detection.
strength_and_weaknesses: Strengths:
- Nicely written
- Baselines are chosen well
- Outperforms all baselines for both F1 and F2 (weaknesses highlight why this is not really valid)
- Evaluated on both SEEG and EEG datasets
- Includes clinical collaboration further signifying the significance of the work
- The domain adaptation experiment in Table 2 is also a great way to show the significance of the work


Weaknesses:
- Split table 3 into two tables?
- Domain adaptation not shown on TUSZ dataset
- Including the same subject in both testing and training invalidates the results
clarity,_quality,_novelty_and_reproducibility: The writeup is clear and the work has sufficient novelty. Code is included for reproducibility.
summary_of_the_review: The paper is nicely written with good results on two different datasets. The demonstrated application is important and the strong performance indicates the significance of the work. However, including the same subjects in training and testing invalidates the results. It is not clear if that was done for other approaches as well. Even if it was, both should be done by selecting different subjects for training and testing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a semi-supervised learning (SSL) method that pre-trains on both SEEG and EEG data. It learns the correlation graph between channels from three SSL tasks: instantaneous time shift task, delayed time shift task, and replace discriminative task. Experiments on seizure detection is described with promising results compared to other semi-supervised learning methods.
strength_and_weaknesses: Strength: SSL on EEG has been studied before, but the authors generalize the idea to SEEG signals as well, and tries to learn correlation graphs which helps with interpretability of the model. Experimental results on seizure detection are encouraging, and are compared to other existing models. 

Weakness: Figure 1 is a little confusing. Also, can the authors comment on the large standard deviation values in Tables 5-7? Is the improvement by the proposed method significant given that observation? How long does the fine-tuning take, and what do the prediction results look like without fine-tuning? What happens when the SEEG channels and EEG channels don't overlap?
clarity,_quality,_novelty_and_reproducibility: The paper is explained well, especially section 3 that describes the methods. 
summary_of_the_review: The idea of using SSL on both EEG and SEEG signals is interesting, and learning correlation graphs improves the interpretability of the model, which can be useful for clinicians.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose an approach for self-supervised representation learning of brain signal, as recorded by EEG and SEEG. The representations are trained to optimize a loss with three components. The first component is a multi-channel version of the InfoNCE loss, used in contrastive predictive coding. The channels are aggregated according to their instantaneous correlations with each other. The second component gives weight to correlations that may occur at larger time delays. The third component encourages individual channels to be distinguishable. The authors find that their model results in embeddings that can be used downstream for better seizure detection, as compared to other self-supervised representation learning approaches. They further find that these representations can be learned for one subject and then used for prediction on a different subject.
strength_and_weaknesses: # Strengths
- Proposes an architecture that can accommodate two very different modalities: EEG and SEEG
- Thorough comparison with other SSL representations
- Domain transfer analysis indicates that this approach can be trained on subjects and then used for held-out subjects

# Weaknesses
- The theoretical discussion in section 2 is a little disconnected from the proposed method, since proposition 1 seems to require that the aggregation function $\Phi$ be linear and non-trivial, but this is not necessarily fulfilled by the learned autoregressive model (eq 12). I believe that in almost all cases, the autoregressive model will not be trivial, but could the authors comment on why they believe the conclusions of proposition 1 should usually hold in practice.
- Section 3.2: There are many ways that multiple channels could be aggregated to form a single context vector. In this work, the context vector is created using a graph, where the edge weights are taken from a correlation matrix. This adds a non-trivial amount of complexity to the system, so it would be appropriate to justify this choice with a few ablation tests. It would be good to compare with aggregation strategies that make fewer assumptions about the structure of the data. For example, a fully connected feed-forward that reduces the dimensionality from $n$ channels to 1 channel could be used instead. 
- Section 4.3: I think there is a very simple and informative baseline that should be included: a linear classifier or autoregressive model trained on the same data and targets. 
- Section 4.5: An ablation that could be informative: how does the model perform when only the "Replace Discriminative Learning" objective is used? The hyper-parameter analysis shows that this component of the loss has the greatest weight. And similar losses have been sufficient for good performance in the audio domain (see wav2vec2 -- Baevski et al 2020, Mockingjay -- Liu et al 2019).
- Section 4.2. I don't completely understand how domain adaptation is possible between subjects. Doesn't each subject have different electrode placements/a different correlation matrix? How can the pre-trained weights be re-used?
clarity,_quality,_novelty_and_reproducibility: - Quality: The connection between the theoretical discussion and the proposed method should be clarified (see above section). But otherwise, seems fine
- Originality: The system is very complex, and for a few design choices, there is not too much justification for the approach taken.The paper could be strengthened by showing that the components of their loss are all necessary, and could not be replaced by simpler components (see above).
- Clarity: Fine. The precise indexing of variables was often helpful.

## small comments/questions
- Proposition 1 proof. What Venn diagram is this referring to? I assume something like this https://en.wikipedia.org/wiki/Information_diagram? Would be good to include it.
- Proposition 1 proof. $\Phi$ needs to be restricted to the set of linear functions, no?
- Equation 2: suggestion -- remove the "*" from the notation.
- Ablation study: "degenerate the task to single-channel CPC" -- does this mean that positive examples are only drawn from a single channel? If so, how is that channel selected?
- Table 1: I know that the results are averaged over subjects, but are they also averaged over channels?
- Table 2: I think I'm missing something. Shouldn't the Non-DA row also have 3 numbers? One for each subject?
- How many patients are used in the SEEG seizure-detection experiment?
summary_of_the_review: The paper delivers on what it promises: a self supervised representation learning approach for brain signal. Improvements are shown over other reasonable SSL approaches. And transfer learning is shown to be possible. The main downside of this approach is its complexity. The paper could be greatly strengthened if the complexity of each component was justified, especially by way of comparisons to simple linear baselines.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a multi-channel self supervised learning method to overcome two issues in analyzing brain signals: (1) existing methods are often limited to a particular type of brain signal data, either the invasive data (e.g., SEEG) or non-invasive data (e.g., EEG). (2) correlations amongst different brain areas need to be explored to achieve a better understanding of brain activity. To address these issues, the authors propose a CPC-based method on capture the spatial correlations amongst different channels, which also take into account the delayed time shift. Theoretical analysis is also provided. The method is applicable to learning representations of both EEG and SEEG data. The effectiveness has been shown in the application of seizure detection. 
strength_and_weaknesses: Strengths: 
1.	Exploring spatial correlations amongst channels is critical to knowledge discovery from multi-channel brain signal data. 
2.	The method is in a self-supervised fashion, which can help alleviate the need for large annotations.
Weaknesses:
1.	Can this method be used on both SEEG and EEG simultaneously?
2.	It would be better to compare with other self-supervised learning methods that are not based on contrastive learning. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The idea is novel. 
summary_of_the_review: Overall, the paper is well written with an interesting idea. Novel self-supervised algorithms have been proposed to explore correlations amongst multiple channels in brain signal data. Effectiveness of the proposed method has been validated using real data from a hospital. It would be more convincing to compare to some non-contrastive-learning-based self-supervised learning methods.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes a novel multi-channel self-supervised framework for SEEG and EEG data. The framework comprises delayed-time-shift prediction, instantaneous time shift, replacement discriminative tasks, and an additional graph module. The results show improvements over previous related work and supervised models for multiple performance metrics for seizure detection.
strength_and_weaknesses: #### Strength
- The experiments have been performed with SEEG and EEG datasets.
- The authors have considered many baselines.
- The authors have performed an ablation study to show the benefit of each proposed component.

#### Weaknesses
- In section 2 after Theorem 1, it is unclear how the proposed objective is more informative and better. When discussing mutual information for CPC, we talk about lower bounds and tightness. For example, the authors (Tschannen et al., 2019) show that tighter bounds on MI can result in worse representation. While the authors note the gap $\log N$, which is constant and only shifts the lower bound.
- Further, I did not find any experiments with empirical evidence on how multi-channel improves over a single channel without additional modification. The authors provide multiple ideas on how to implement the multi-channel CPC. However, they do not evaluate them. For example, in multimodal learning (Liang et. al, 2022), fusing different modalities with only architectures is expected to improve the performance over unimodal baselines assuming channels are different modalities. So, it could be a simple baseline where the models take multi-channel input, but the loss function is vanilla CPC. I agree with the authors that the correlation is not explicit in this case, but it is still a viable option to improve performance. The comparison could also show the trade-off between performance and interpretability.
- Main experiments have been performed on the dataset splits where the same subject exists in each subset. The authors must evaluate models on the dataset splits that do not overlap on the same subjects. Models can exploit slow features in time series (persistent temporal feature learning) (Feichtenhofer et al., 2021).
- Subject-to-subject domain translation experiments are not stable and depend on the random sample. Consider studying data efficiency over a number of subjects instead of the proposed experiment. Clearly, the model does not generalize in a such low data regime.
- There is no pairwise statistical comparison of the model's performance for significance. Table 1, 2, and 3 does not have a standard deviation, while they are reported in the appendix. Some results, specifically, SimCLR on Recall, have high variance and mean within the variance; thus, selecting the proposed model as bold is misleading. In table 5, CPC is equivalent to the MiniRocket or MBrain.

Feichtenhofer, Christoph, et al. "A large-scale study on unsupervised spatiotemporal representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Tschannen, Michael, et al. "On mutual information maximization for representation learning." arXiv preprint arXiv:1907.13625 (2019).

Liang, Paul Pu, Amir Zadeh, and Louis-Philippe Morency. "Foundations and Recent Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions." arXiv preprint arXiv:2209.03430 (2022).
clarity,_quality,_novelty_and_reproducibility: #### Clarity
- The paper proposes four components, but the abstract only talks about the delayed time shift and graph module, but in the end, all four components are beneficial. In the introduction, the main contribution is the multi-channel CPC. I found it quite confusing.

#### Quality
- Overall, there is a lack of technical sophistication in the evaluation (statistical significance, proper dataset splits) and empirical evidence in experiments (single vs. multi-channel CPC design). Please refer to Weaknesses for details.

#### Novelty
- The objectives look novel. However, it is hard to evaluate the result's significance without addressing the weaknesses.

summary_of_the_review: The authors are solving significant problems for healthcare, and there are many exciting ideas: theoretical justification for multi-channel vs. single channel, variants of multi-channel CPC, other self-supervised tasks or modules for EEG and SEEG data, and interpretability (appendix). However, each part of the contribution lacks theoretical justification, technical sophistication, or empirical evidence in a detailed examination; thus, it does not look clear and complete. Overall, It is hard to read the manuscript and understand its significance. Authors should consider exploring one idea at a time and provide concise empirical evidence or theoretical justification for each part.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: Create embeddings of either EEG or SEEG data then use it to improve seizure detection.

Strengths: Investigating representation learning in other domains like neuroscience is of wide interest. The system appears to work better than baseline systems for SEEG and is marginally better at seizure detection with EEG data.

Weaknesses:
The method is tuned very specifically to neural data and is quite complex, making applications to other domains unlikely. There is no central key idea; the method essentially consists of three pieces that are combined together. This is fine, if the result is very flexible and generic; but the method is only demonstrated on seizure detection. Even there, improvements are only major on SEEG data, EEG shows a very incremental improvement.

Critically, the manuscript is unclear on specific details about experiments, for example, when it comes to adapting to multiple subjects. Authors promise to later improve the manuscript to clarify these points ("According to your suggestion, we will further describe the specific steps of our experiments and the logic of the three experiments in detail." on Dec 4th), but fundamental questions about what was done so late without an updated manuscript show that it is not yet ready for publication. At best the work is not reproducible in its current form, at worst, it was not fully described by the authors and thus parsed by the reviewers.

What a reader of the manuscript at ICLR could take away from this work is very unclear. As described in the manuscript itself, it is unclear if this method would improve performance on any other task. Had an array of decoding tasks been shown, readers at ICLR could imagine using this model.

The authors in their responses to reviewers to a range of concerns come back to one central point: the utility of this model to a clinical setting. To that end, a venue related to seizure detection would be a much better fit.
justification_for_why_not_higher_score: The method is tightly coupled to neural data and rather complex. There is no core idea for ICLR readers to import into their own work. It is not clear if the method would have any utility for any other decoding tasks. This would be a great fit for a conference where seizure detection performance was the determining factor for publication.

Setting all of the above aside, questions about the methodology of the manuscript which are not fully resolved with reviewers make it impossible to recommend acceptance (specifically with reviewer DmoA, ending on Dec 4th with a pledge to clear up specific steps in the experiments performed). A pledge to provide further clarity on the experiments performed shows that the manuscript is not yet mature enough.
Decision: Reject