Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper casts the removal of inter-experimental variability from functional data in systems neuroscience in the theoretical framework of domain adaptation. The paper adapts various approaches and subsequently demonstrate improved performance of cell type assignment. The paper also shows that the proposed theoretical framework can produce predictions that are best aligned to the anatomical data, as well as effectively reduces inter-experimental variability to reveal unobscured biological effects.
main_review: The paper clearly delineates related work in the area and provides various citations for the reader to refer to. The discussion is acceptable in terms of detail, but not excellent. To be an excellent , it would be useful to cite some more related works and (optionally) make a small comparison table to delineate similarities/differences of each and how the current proposed method addresses these limitations. The methodology is presented clearly with easy-to-understand equations and diagrams. In addition, the method is compared to three other baselines (two of which were mentioned in the recent works). The performance between the different methods make it clear that the current methodology is superior in accuracy, IPL depth, and identifying JS divergence, compared to the other methods, showing success in reducing inter-experiment variability. The paper also explicitly shows a nice example that demonstrates that their method can be used to reveal biological effects previously unnoticed when the dataset is contaminated with inter-experiment variability. Limitations and a clear discussion of the results/ethical implications are provided. Overall, the paper is solid work, and proves to be a good candidate for acceptance to Neurips.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors develop a method for removing inter-experimental variability in systems neuroscience experiments. The general idea is to learn a new representation that maximizes the information retained about the data (and in the supervised case, also the signal of interest), while minimizing the information about the individual experiments. To do this, they use an adversarial training approach, where one network tries to remove information about the individual experiments while retaining total information in the new representation, and another network tries to predict the individual experiment from this new representation. They demonstrate their approach on experimental 2-photon recordings of retinal bipolar cells, recorded in 2 different experiments.
main_review: Quality:

Overall, I found the quality of the paper high. It is technically sound, and compared to a few alternative methods on an experimental neuroscience dataset. 

It would have been nice (perhaps in the appendix), to show that the method can successfully recover the ground truth when it is known in Simulations.


Originality:
As the authors state, they are using an existing general approach within machine learning of adversarial training to deal with domain shifts. That being said, I agree it is novel in its application to systems neuroscience - at least I am not aware of anyone who has used this approach.


Clarity:

I found the clarity to be mixed in this paper. 

In particular, I found it challenging to follow the description of the model. For instance:
-Why does Eq 3 include an S - isn’t this unsupervised? Should the S say X?  And how do you get from Eq 1 to Eq 3 - is the order of the terms flipped?
-I would be more explicit that minimizing d is maximizing information (if I am properly understanding). This relates to my above point that it would be helpful to more clearly walk through going from Eqs 1 and 2 to 3 and 4.

I also found myself frequently looking back and forth between different parts of the paper, which made it more difficult to read/understand.
-It would be helpful to have a greater description of the figures in the Figure legends, or have the figures closer to where they’re being described in the text
-When reading the Methods, I didn’t know how it was going to be used or why it mattered, and then in the Results, there was no quick description of any of the methods being used
(so I needed to go back and forth). Some synthesis would be helpful.


Significance:

I think the authors are addressing an important problem of controlling for inter-experiment variability, which is rarely addressed. I think this general approach could be very useful in systems neuroscience.

While I feel this could be useful in theory, I wasn’t very convinced of the usefulness towards answering scientific questions based on the demonstrations in the paper. For instance, what would be the reason for a researcher to do the analysis in Fig. 6 based on a dataset with unknown cell types (besides demonstrating that your method works) - wouldn’t you just try to address this question in datasets where you actually have cell labels?


Other:

Line 141 - what is phi? (and why is it necessary to introduce new  nomenclature here?)

Fig. 4 - in the bottom panel, how are you coloring dots for dataset B, since you don’t know those cell types?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In the paper "Removing Inter-Experimental Variability from Functional Data in Systems Neuroscience", the authors propose a novel framework for alleviating inter-experimental variability. They present a supervised as well as an unsupervised variant of their approach and evaluated the framework on cell type datasets. They further compared their results to EM data, giving further credibility to their findings.
main_review: Contrary to what the authors said in their checklist, they did not include code or data in their submission! I believe not all of the datasets are publicly available. In the manuscript, the authors promise to make the code available upon acceptance.

Originality:
The idea to combine datasets by dealing with inter-experimental variability is not new. The authors do a reasonable job of summarizing related work. For systems neuroscience, they refer to Zhao et al. as well as Shah et al. However, their discussion of related work is not complete. In Sorochynskyi, Deny, Marre and Ferrari, PLoS Comput Biol 2021, the authors proposed a method involving statistical models of noise correlations to deal with inter-experimental variability. I think this is relevant and should have been discussed.
The authors have applied methods from machine learning for domain adaptation. For all I know such methods have not been applied to systems neuroscience before.

Quality:
The authors used state-of-the-art methods from domain adaptation. They introduce their framework based on concepts from information theory. This is done in a rigorous way. The demonstration of their framework is done on real data where they could validate their results by means of EM data. This is convincing and impressive.

Clarity:
The clarity of the manuscript is outstanding. The text is very well written, motivates the problem well and introduces the framework in a formal yet easily accessible way. The figures are also very good and easy to follow.

Significance:
From the systems neuroscience perspective, the proposed methods are of great importance. Inter-experimental variability is a profound problem due to natural data limitations occurring in neuroscience. A systematic and flexible way to remove this variability opens up new possibilities for data analysis.
I am not an expert in domain adaptation, so I cannot assess whether this is a contribution that is valuable to the machine learning for domain adaptation community as well.
limitations_and_societal_impact: The authors consider limitations of their work in a dedicated section where they consider the problem of different stimuli. They consider benefits with respect to the 3Rs which I find convincing.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces a new framework to remove non-biological variability to facilitate data aggregation across studies. The problem is viewed as a transformation of the input data, such that the transformed output is not informative of the “domain” (e.g., experimental condition or study) while simultaneously capturing relevant information from the data. This high-level procedure is implemented via an adversarial training strategy, where the mutual information terms of the objective function are approximated via a standard cross-entropy or MMSE loss. The authors validate their method using two murine datasets, in which the neural recording procedures are slightly different; the proposed framework outperforms the standard Harmony and scGen in the supervised and unsupervised settings.
main_review: Overall, this is a solid paper that presents an interesting new twist on data fusion. The authors complement their mathematical framework with a straightforward implementation using GANs. The evaluation is also comprehensive and clearly demonstrates the promise of the method. With that said, there are a few questionable assumptions. Specific comments on the paper are as follows:

Strengths

+ The paper tackles the ubiquitous yet challenging problem of removing non-biological data variability.
+ The authors propose a simple and elegant framework for nuisance removal that relies on distributional properties of the data. 
+ The authors develop both a supervised and unsupervised version of the model, thus allowing it to be used in a variety of application domains.
+ The experimental results are comprehensive. The authors compare their method (RAVE and RAVE+) with state-of-the-art approaches, and in each case, numerous performance metrics are extracted.

Weaknesses

- In the case where S (signal of interest) is continuous, the authors approximate the mutual information between Z (transformation) and S via a MSE objective. Depending on the dimensionality and characteristics of S, the MSE optimum can be quite far from the MI optimum.
- The training setup, model architecture, data processing, and hyperparameter tuning should be described in the main text and not buried in the Appendix. The authors have almost 1/2 page of free space.
- The closing sentence of the paper (condemning the use of this method in warlike applications) is puzzling. How would RAVE or RAVE+ be used for such purposes? More details are needed.
limitations_and_societal_impact: The authors discuss two key limitations of their approach: (1) the experimental paradigm must be the same across datasets, and (2) the method does not work well if the data is noisy (e.g., whole-brain fMRI). These limitations do hinder the societal impact of this particular work, particularly in human neuroimaging studies. With that said, the need for data aggregation methods is key in many application domains, so the authors are working on a critical problem with broad impact.
review_title: null

Meta-review: This paper provides a method for removing inter-experimental variability from functional datasets arising in systems neuroscience. All reviewers agreed that the results are significant and the paper was well written and executed.
Decision: Accept (Spotlight)