Removing Inter-Experimental Variability from Functional Data in Systems Neuroscience
Authors: Dominic Gonschorek
Larissa Höfling
Klaudia P. Szatko
Katrin Franke
Timm Schubert
Benjamin Adric Dunn
Philipp Berens
David A. Klindt
Thomas Euler
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: systems neuroscience
inter-experimental variability
cell type classification
retina
domain adaptation
autoencoder
adversarial optimization

Abstract: Integrating data from multiple experiments is common practice in systems neuroscience but it requires inter-experimental variability to be negligible compared to the biological signal of interest. This requirement is rarely fulfilled; systematic changes between experiments can drastically affect the outcome of complex analysis pipelines. Modern machine learning approaches designed to adapt models across multiple data domains offer flexible ways of removing inter-experimental variability where classical statistical methods often fail. While applications of these methods have been mostly limited to single-cell genomics, in this work, we develop a theoretical framework for domain adaptation in systems neuroscience. We implement this in an adversarial optimization scheme that removes inter-experimental variability while preserving the biological signal. We compare our method to previous approaches on a large-scale dataset of two-photon imaging recordings of retinal bipolar cell responses to visual stimuli. This dataset provides a unique benchmark as it contains biological signal from well-defined cell types that is obscured by large inter-experimental variability. In a supervised setting, we compare the generalization performance of cell type classifiers across experiments, which we validate with anatomical cell type distributions from electron microscopy data. In an unsupervised setting, we remove inter-experimental variability from the data which can then be fed into arbitrary downstream analyses. In both settings, we find that our method achieves the best trade-off between removing inter-experimental variability and preserving biological signal. Thus, we offer a flexible approach to remove inter-experimental variability and integrate datasets across experiments in systems neuroscience. Code available at https://github.com/eulerlab/rave.

Removing Inter-Experimental Variability from Functional Data in Systems Neuroscience
Dominic Gonschorek
Doniincc.gonchoreekeeeeo uni uni-tuebingen.
Larissa Höfling
Klaudia P. Szatko
harissa. tuubingeeneeeeennnneeme
klaudia. klaudia.szatkkoo www.koottueiinnnnn.noa... mpg de
Katrin Franke'
Timm Schubert'
katrin. franke@cin, uni umi-tuebingen.de de
timm schubert@en uni umi-tuebingen.co de
Benjamin A. Dunn
Philipp Berens
benjamin. dunn@ntnu.nn
philipp bwwensuuni.nnn.ee..e.ee.. wwwbbingttebbnnnn.mmm
David A. Klindt klindt. david@gmaill...... com
Thomas Euleri:
thomas. ewlarseinlrrriin uni tel--ttteinnendde
Abstract
Integrating data from multiple experiments common practice in systems neuro- science but it requires inter-experimental variability be negligible compared the biological signal of interest. This requirement is rarely fulfilled; systematic changes between experiments can drastically affect the outcome of complex anal- ysis pipelines. Modern machine learning approaches designed to adapt models across multiple data domains offer flexible ways of removing enter-experimental variability where classical statistical methods often fail. While applications of these methods have been mostly limited to single-cell genomics, in this work, develop a theoretical framework for domain adaptation in systems neuroscience. We imple- ment this in an adversarial optimization scheme that removes inter-experimental variability while preserving the biological signal. We compare method to previ- ous approaches on large-scale dataset of two-photon imaging recordings of retinal bipolar cell responses to visual stimuli. This dataset provides unique benchmark as contains biological signal from well-defined cell types that is obscured by large enter-experimental variability. In supervised setting, compare the gener- alization performance of cell type classifiers across experiments, which we validate with anatomical cell type distributions from electron microscopy data. In un- supervised setting, we remove inter-experimental variability from data which can then be fed into arbitrary downstream analyses. In both settings, find that method achieves the best trade-off between removing inter-experimental variability and preserving biological signal. Thus, offer flexible approach to remove inter-experimental variability and integrate datasets across experiments in systems neuroscience. Code available httttt://ggthhiii com/ eul mulerrab/rave
I Equal contributions, University of Tübingen, Norwegian University of Science and Technology.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Introduction
Systems neuroscientists often concerned with identifying and characterizing how properties of neurons vary along certain dimensions of interest. Differences in these properties between neurons form the basis for sorting them into discrete categories. Both the advance of large-scale data acquisition techniques in experimental neuroscience as well as the development of more efficient and powerful data analysis methods allow collecting and analyzing datasets of increasing size; and hence the discovery of more subtle variations in neural function between cell types [e.g. 4 However, as data acquisition is often an incremental process, has become common practice to pool data from multiple experiments. This practice ignores variability in the data stemming from external factors, which include non-biological ones (e.g. sample handling resulting in small differences tissue quality, or temperature fluctuations affecting the rates of biochemical processes) but potentially also unforeseen biological ones (e.g. subtle genetic variations) B 5 6 Such variability due to external factors, here referred to as inter-experimental variability, can confound and obscure the biological signals of interest. In some cases, the source of inter-experimental variability is known and can be modeled [5], but if this is not the case, method for removing it from the data is required.
The issue of experimxpei variability in systems neuroscience is analogous to the problem of domain shift in machine learning, where the data distribution changes between training ( ("source') and test (" ('target') data, causing an algorithm to fail when deployed on data from an unseen target domain I010 Methods that address this issue have to perform some form of domain adaptation, i.e. adapting the algorithm to work both on the training as well as some (usually unseen) test domain [1I. In single-cell genomics, a number of different studies have proposed methods for removing inter-experimental variability (see Section 2). but related works in systems neuroscience are lacking, despite the recognized need for such approaches 3 5 Here, we contribute to closing this gap as follows:
We cast the removal of inter-experimental variability from functional data in systems neuroscience in the theoretical framework of domain adaptation (Figure Tand Section 3
We adapt and evaluate different approaches and demonstrate improved performance of cell type assignment, while preserving the biological signals of interest (Table [ and Figure
We demonstrate that our method produces cell type predictions new dataset that best aligned with anatomical data (Table 2 and Fig
Finally, we showcase in a downstream analysis sis that the corrected data (Figure 3 clearly exhibits biological effects that were obscured by inter-experimental variability (Figure
2 Related Work
As mentioned before, few studies have proposed specialized solutions to the issue of inter- experimental variability in systems neuroscience. Two studies have approached the problem of temporal alignment of neural responses across experiments. Zhao et al. [5] proposed a solution to deal with the specific effects of temperature fluctuations on the response kinetics of retinal neurons by modeling them explicitly. Williams et al. [12] proposed a more general method for the temporal alignment of data across trials or recording sessions. Other studies have suggested models of neural function that integrate data across experiments. Shah al. B build encoding models to predict the responses of retinal ganglion cells across different experiments [see also [3] and compare it to covariates such as the gender of an animal. Sorochynsky et al. 14 propose a way to measure noise correlations in each recording and integrate those into models of neural populations of specific cell type. This latter approach is complementary to our method because it allows the study of the structure and function [see also [5] of noise correlations, which we discard nuisance variability. Somewhat related to the example application in our paper, Jouty et al. [16] suggested a method to perform non-parametric physiological classification of retinal ganglion cells in the mouse retina while trying to find matching clusters of cell types across experiments. Crucially, all of these approaches offer specialized solutions that do not represent general purpose correction methods.
In single-cell genomics, number of approaches for removing inter-experimental variability from data have been developed 217 Two such methods are Harmony 25 and scGen (26). Hurmony performs iterative clustering using a variant of soft k-means until convergence to align cells from different datasets in a joint embedding. scGen, on the other hand, combines variational autoencoder

---

adapted for scRNA-seq data with latent space arithmetics to predict gene expression, while removing inter-experimental variability between datasets. this paper. we compare our approach against these methods as they have been found to perform particularly well in two benchmarking studies 22
28
3 Theoretical Framework
The generative process of data denoted by a random variable X with image X' is depicted in Figure [ The biological signal shared across experiments (e.g. vari- ation due to cell types) is represented by random variable S ('signal") with image S. We define D ('domain") " as random variable with image DD that represents experimeeti variability. Now, our objective is to learn function that transforms the data into a new random variable Z := f(X) with image Z. Importantly, we distinguish two settings: (i) unsupervised - where is unknown and we sim- ply try to retain in Z much information about the data as possible while removing enerrexperimettll variability; (ii) supervised
X
Z
Figure 1: Problem Setting. Arrows repre- sent given (solid lines) and modeled (dashed lines) relations. Capital letters denote random variables, small letters transformations (see Section 33). The setting with known S (white circle) is supervised, while unknown S (gray circle) is an unsupervised setting.
where S known and we additionally try to retain in Z as much informa- tion about S possible. These objectives can be formulated in terms of mutual information, giving
the unsupervised loss function
L = I(z D) ((Z
(1)
and, provided knowledge about S, we obtain the supervised loss function
C L T)2
Now, I(Z; D) attains its minimum for T(f(X); D) O because of the non-negativity of mutual information. And 7( S) attains its maximum for I(f(X):S) = T(X;5) because of the data processing inequality. If were bijection, it would follow that T(f(X);S) = [(X; I(
I); but also But by assumption, 1(X; D) > 0 (otherwise there is no inter-experimental done) and so we would have I(S(X); D) > 0. Thus, the minimum of I(2; D). cannot be bijection. Generally, if there is an interaction between recording, signal and domain i.e. [(X; S; D) * O, then there will be trade-off between maximizing (Xx():: 8) and minimizing [(f(X); D). This trade-off becomes even more apparent in the unsupervised setting where [(f(X);XX and (Xx();; D) are clearly competing.
variability and
Mutual information quantifies the dependence between two variables but it is difficult to estimate 22 Instead, we measure dependency through nonlinear regression with appropriate distance metric Usually D is discrete random variable indicating the experiment of recording, and so, to estimate ((Z D), we can perform classification with classifier function à Z DD minimizing the standard cross-entropy dc Z), D) (Figure [ Lemma 10 in [4 shows that this gives variational lower bound I(2; D) [see also 35). In some cases S may also be discrete (e.g. cell types) and we can do the same, in other cases it might be (high-dimensional) continuous random variable and so, to approximate I 1[2; S), we can perform regression, minimizing the mean squared error (yse( (2)), S). Similarly, in the unsupervised setting, to approximate I X)), we minimize dMssEg(()) X)). To keep notation simple, in the unsupervised setting we define the mapping g
X, and in the supervised setting g Z X' x S. Putting this together, in the unsupervised setting our objective is
min C min max Xd(b(f(X)), -(
h a.f
where we have introduced hyperparameter À that mitigates the trade-off discussed above. In the supervised setting objective becomes
min L+ - min max X xd(h(((X)) D) -)d
(4)
9.
'If the joint probability density of two random variables is a bivariate normal distribution, then the mutual information is proportional to their linear correlation (33).
3

---

In both equations we find a min max optimization, where h is trying to predict D from Z, tightening the lower bound on [(;; D) [see 34]. while f is trying to prevent that by removing information about D) from Z, effectively lowering T(:; D). Practically, this optimization scheme has become standard adversarial setting in machine learning, for instance, in the training of generative adversarial networks [36] or for discriminative adversarial domain adaptation [37
4 Methods
4.1
Datasets
To test our model approach, we use two datasets of two-photon imaging recordings 38 40 from the
14 mouse retinal bipolar cell (BC) 44 types' responses to two visual stimuli, a local and full-field chirp stimulus (Figure (3. The axon terminals of BCs stratify at distinct, cell type-specific depths within the second synaptic layer of the retina, the inner plexiform layer (IPL). The functional BC data were obtained by imaging the glutamate output at their axon terminals using the genetically encoded glutamate-sensing fluorescent reporter iGluSnIR 42 In our study, we refer to these two datasets as A 2 and B [5] (for further preprocessing see Appendix).
functional clustering approach to group the BCs into the 14 functional types was applied to dataset A, thus providing functional reference cell type labels, which do not exist for dataset B. However, even both datasets recorded the same cell types, they suffer from inter-experimental variability making it difficult to match and, for example, to use dataset A to predict the cell type labels for dataset We discuss potential sources of inter-experimental variability in the Appendix. For preprocessing, both BC datasets are high-pass filtered above 0.1 Hz (to remove the trends decreasing fluorescence signal over time) and resampled to 30 Hz. Each cell's response is normalized to zero mean and standard deviation one. In addition, to ensure high quality responses, only cells with a sufficient response quality are used (for details about quality criterion see (2).
In (2] an anatomy-
4.2 Models
All methods transform the data, either into a fow-dimensional embedding E Z or directly into reconstruction á X' from which ixter-experimental variability has been removed a varying degree. Usually we have dim( Z) < dim(X') and so, for different downstream evaluations, we map between these representations: (f) with least squares reconstructions (Z - X)), or (ii) principle component projections (X - 2) (see Appendix).
4.2.1 Unsupervised Model
We parameterize the functions f.g and h (Figure I with neural networks. In the unsupervised model, the function g 2 X' provides a reconstruction v := g(2), â E X' of the data. With the concurrent task (eq. (3) of minimizing the predictability of the domain D, this reconstruction should only contain parts of the original data that are indiscernible across experiments. Since the purpose of our method to Remove, Adversarially, Variability from datasets collected in different Experiments, we term our model RAVE.
4.2.2 Supervised Model
In the supervised setting, we have partial knowledge about the biological signal The function g 2 - X' x S now returns a reconstruction well as a prediction of that signal (œ, 8) = g(2), & S. When optimizing equation (4). this additional task is equivalent to discriminative adversarial domain adaptation
37 In the particular data that work with, we have two datasets D (A, B). but the biological signal consists of cell type labels which are only available in the first dataset A. Thus, more accurately, this presents semi-supervised scenario where one wishes to classify newly recorded dataset according to some existing classification scheme. We term this extended version of model RAVE+.
4.2.3 Training and Optimization Details
All of our models are implemented and optimized in PyTorch 43 For both RAVE and RAVE+, we use the same model architecture, they only differ in the objective function. We randomly split the data

---

into training, validation and test set and train all models with empirical risk minimization. Model weights are trained with stochastic gradient descent using one instance of the Adam optimizer 44 for the outer minimization of f and y in equations (3) and ( and then second instance of Adam for the inner minimization of h in those equations. We optimize hyperparameters through random search [45] on the validation set and report performances on the test set which is only used for final evaluation. the random search, we test different learning rates for both optimizers, and also different training schedules. We additionally search over depth, width and drop-out rate for each of the neural networks (f, 9, ht., as well as the trade-off parameter À introduced equation (3). Finally, we explore training the inner optimization (h, estimating [(Z;D) more often than the outer optimization, which proved more stable and effective in early experiments.
4.2.4 Comparison Models
We test three different methods for comparison with our model. Our simplest comparison model (Linear) is linear model that projects out the contrast between the dataset indicator variables (see Appendix). This has an analytic solution and no hyperparameters, serving as a baseline to get estimate of the correction quality achieved by standard method in classical statistics. The other two methods (scGen and Harmony) are run in an unsupervised learning mode without cell type information. Even though scGen could be utilized run in a supervised mode with cell type information, this is not specified in a semi-supervised setting with only partial cell type labels available.
4.3 Performance Evaluation
For evaluating the correction performed by the various methods, analyze their output with respect to dataset-mixing (achieved by removing inter-experimente variability) and preservation of signal information.
4.3.1 Dataset-Mixing
The Rand index [46) measures similarity between clusterings; the adjusted Rand index (ARI) is the Rand index adjusted for chance level (see Appendix) which was recently used by Tran et al. (28 assess the quality of dataset-mixing in genomics. It takes as input the true and the predicted labels for a set of samples. We define ARldom (2) := ARI (d, d=) with d the original domain labels (of the test set) and dæ the domain labels predicted by a classifier trained on z. On the raw data (z = 2), we expect ARL to be high due to inter-experimental variability. After successful correction (with the output of model), we expect ARldom to be low indicating good dataset-mixing.
In addition, we compute the accuracy (Accaom) of a domain classifier with the objective to predict the domain labels based on the input data. For low dataset-mixing, we expect a high Acc as should be trivial for the classifier to differentiate the datasets. However, after removing inter-experimental variability, Accdom is supposed to be close to chance level (~ 64%, cf. Table D. which would indicate successful dataset-mixing. For the domain classifier, we use a random forest classifier with cross-validated hyperaarameters for each model (see Appendix). This crucial, because a powerful encoder f might hide (through multiple nonlinear transformations) domain information from simple classifier, but still recover that information an equally powerful decoder g. Conversely, observe that overly expressive random forest classifiers, tend to overfit the training set, thus underestimating the preserved domain or type information on the test set.
4.3.2 Preservation of Signal Information
In the unsupervised setting, to assess the amount of information preserved about the original data during the process of removing inter-experimental variability, we evaluate the rank correlation Conr(z, .)) between input x and reconstruction é. In the (semi-) supervised setting, we have reference cell type labels & for dataset A. To estimate how much of this information is preserved, we predict cell type labels & from n with random forest classifiers like above (see Figure) Appendix for further details).
If a method succeeds at preserving signal information in z after removing inter-experimental vari- ability, then expect the classifier to have high accuracy (AcCtype). Deteriorating classification
5

---

performance between predicting $ from raw data A versus predicting it from the model output 2 would indicate signal loss.
Additionally, we would like to evaluate how well cell types can be distinguished and how biologically plausible they are for the unlabeled dataset B. To this end, we apply the classifiers to predict cell type labels šB from zB- One direct comparison is between the distributions over cell types in $3 and as expected from
(x)5)A
(2,5)
A
train apply predict
Clessifier
(EM) data (0] (Figure 5A). However, we can also evaluate the accuracy of these predictions by making use of BC axonal stratification profiles obtained from the same
XB
B
EM data. From those data, we know where in the IPI. a BC type stratifies its axon terminals. Thus, we can compare the distribution over IPI. depth for the predicted cell types (&g) with the distributions expected from EM data. We quantify the difference between the expected and predicted distributions by calculating the Jensen-Shannon distance. We define the depth score (DS) as the mean Shannon distance between those two distributions (Table 2 Additionally. we evaluate the robustness of cell type labels & by fitting the classifier ten times with different seeds and calculating the average ARI between different runs, giving the ARltype score (Table T).
5
8zB
Figure 2: Workflow. Evaluating the preservation of signal information: A classifier gets trained on the labeled dataset A (either (x. $)A or (2, 8)4) and applied to dataset B to predict labels (ci- ther Sx, or 8zg). ). The predicted labels are then used for further evaluation.
5 Results
5.1
Simulation Experiments
First of all, we validated that our model performs as expected on simulated ground truth data. To do this, we generated bipolar cell responses for all 14 cell types based on the published bipolar cell model in Schröder et al. [51] To simulate different individual neurons, we added small perturbations to the model weights for each cell type until we matched the intra-cell-type variability observed in the real data. Thus, we generated N 1000 distinct neurons for each of the 14 BC types. Approximating the differences of the two datasets in the paper, we presented the model with the slightly altered versions of the stimulus from the actual experiments (see Appendix Bb. We additionally added white noise to match realistic signal to noise ratios, as estimated from repeated stimulations of the real neurons. This resulted in two datasets ('A' and 'B') with similar intra- and exper-experimental variability as observed in the real data, but with known ground truth cell type labels.
The results are discussed here and, additionally, they are presented in Appendix Fig. 9 confirmed that the classifiers are indeed perfectly able to separate these two artificial datasets based on their systematic differences (domain accuracy raw simulated data: 1.0). However, cell type classifiers trained on dataset A fail completely on dataset B indicating severe inter-experimental variability and a failure to transfer models across datasets (type accuracy on dataset A: 0.98, type accuracy on dataset B: 16). In contrast, after correction with RAVE+, the performance of a classifier trained to distinguish the two datasets drops from 1.0 to 0.66 (chance level 0.5), indicating strong removal of experimapel variability. Importantly, we find that a classifier trained on the output of RAVE+ on dataset A does now generalize to dataset B and recovers the ground truth cell labels nearly perfectly (type accuracy 0.99). This constitutes an important validation of our model.
We first
5.2 Unsupervised Removal of Inter-Experimental Variability
All methods tested (Linear, Harmony, scGen, RAVE and RAVE+) succeed at retaining a significant amount of information about in á. reflected by high correlations between x and & (Table [] Com(z,, x, 2) reaches similar levels for data from both datasets, suggesting that both datasets are modified to find a midway representation. This impression confirmed when visualizing A, EB and A and in next to each other (3. Moreover, we note that side-effect of the alignment more general denoising that RAVE tends to perform along with removing inter-experimental variability. We recognize that this desirable feature that we will study further in future research.

---

Model
CorrA 1
CorrB
Acc.om
Acciype 1
ARIdom 1
ARI
Raw
100
100
99.8 (0.1)
77.4 (0.9)
99.3 (0.4)
37.3 (3.3)
Linear
99.0 (0.5)
97.0 (1.7)
99.5 (0.2)
83.4 (0.8)
98.1 (0.7)
(1.7)
Harmony
72.0 (10.7)
72.0 (13.8)
94.2 (0.4)
82.5 (0.5)
78.0 (1.6)
31.4 (2.3)
scGen
78.0 (9.8)
80.0 (10.5)
99.6 (0.1)
84.7 (0.8)
98.7 (0.3)
14.3 (2.6)
RAVE
60.0 (12.8)
58.0 (17.3)
77.5 (0.5)
69.5 (0.4)
28.9 (1.2)
81.2 (2.7)
RAVE+
59.0 (14.8)
58.0 (19.1)
65.9 (0.9)
78.6 (0.8)
10.0 (1.2)
83.7 (2.3)
Table Model Comparison. All entries percentage. Mean and standard deviation metric scores across 10 random seeds. Bold font in each row indicates best score. Corra (CortB) is the correlation of corrected data from dataset (B) with raw data. AcC.om (AcCrype) is the accuracy of the domain (cell type) classifier. For ARIdom and ART see Section 4.3
We show mean traces for exemplary cell types from dataset A, and mean traces of cells from dataset B whose cell type labels we predict twice, first based on æ (left pathway in Figure [ and then again based on å (right pathway in Figure 2 but on εaLve instead of 2). As expected, inter-experimental variability obscuring the common signal s behind FA and z causes the cell type assignment to fail; the similarity between responses of cells assigned to the same cell type, but coming from the different datasets is low (Figure 3 BC type 5t). Repeating the classification pipeline based on æ with the same classifier architecture and different seeds yields highly variable cell type predictions for dataset (Table I ARIp despite high prediction accuracy on dataset (Table [ AcCzype). This demonstrates a failure in transferring dataset B, and not the classification itself. These results on raw data æ affected by inter-experimental variability were expected; however, the same pattern type (dataset B) and high Acctype (dataset A) is observed for Harmony, scGen and the Linear model. This suggests that these methods fail at removing experimmeillt variability. The high domain accuracy achieved by a classifier trained the outputs of these models confirms this conclusion. RAVE, on the other hand, succeeds at significantly lowering domain accuracy AcCdoms while the same time maintaining high scores for ARlepe and AcCrype-
low type
5.3 Supervised Removal of Inter-Experinn Exeerimxpeiltttll Variability
RAVE+ extends RAVE to the (semi-) supervised setting where (partial) signal information is present. RAVE+ excels at removing inter-experimental variability (Table Accaom and ARldom) and the same time retaining signal information (Table T Acctype and ARl A low dimensional t-SNE
Dataset A
MALL
ا
XRow
Dataset B
ReavE
XRaw
Rowe
Cell Type
Figure 3: Exemplary Cell Type Responses from both Datasets to the Chirp Stimuli. Four bipolar cell type responses of the types 2, 3a, 5t and 5i to the local (top panel) and full-field (bottom panel) chirp of raw data New (two top upper panels) and reconstructed data êRAVE (two bottom lower panels) by RAVE for both datasets A and B. Each column shows the mean responses of one cell type (standard deviation shaded).
7

---

52 embedding (Figure 4 shows that cells from datasets A and B are mapped onto the same cell
type "islands". The distribution of types across IPL depth predicted by classifier trained on RRAVE+
matches the expected anatomical distributions better than for all other methods (Figure and and Table
2). This provides valuable validation of the estimate 8B learned by RAVE+ the absence of ground
truth knowledge of 8B-
Raw
RAVE+
Dataset
Cell Type
3a
3b
8
5t 9
50
Figure 4: Dataset Embeddings. t-SNE embeddings of the test set raw (left column) and corrected output data by RAVE+ (right column). Embedded cells color-coded by dataset (top row) and cell type (bottom row). Cell type labels for the raw data of dataset B (bottom left) were predicted using a cell type classifier trained on the raw data of dataset A (Figure 2 left pathway).
plec type)
0.2
0.1
0.0
IPL depth
Js distance
0.5
XRAVE=
0.3
33
3b
4
St
50
51 types
X
8
R
BC
Figure 5: Distribution Across BC Types and IPL Depths. We compare the expected and predicted distribution of BCs from dataset B across the 14 types and across IPL depth. (A) Probability that BC belongs certain type as estimated from EM data; as estimated from BC type labels predicted on GRAVE and as estimated from BC type labels predicted on .r Error bars indicate SD across 10 seeds the classifier. (B) Distributions per cell type over IPL depth for EM data (distribution shown
to the left), RAVE+ output (solid line to the right) and raw data (dashed line to the right). Shaded area around the distributions shown the right indicate SD across 10 seeds of the classifier. (C) JS distances corresponding to the distributions in B).
5.4 Downstream Analyses Reconstructed Traces
As in our unsupervised setting, it common that no particular signal information is available and that one wants to remove inter-experimental variability from the data perform further downstream

---

BC Type
1
2
3a
3b
4
5o
X
6
7
8
9
R
all
Raw
34
52
38
42
56
63
62
54
60
52
51
38
58
37
50
31
Linear
58
40
53
51
57
62
57
56
62
55
57
58
58
53
56
34
Harmony
32
42
54
59
58
56
37
59
26
58
38
48
23
scGen
51
38
48
48
56
59
56
59
63
55
57
56
56
42
53
31
RAVE
32
38
40
57
58
62
55
66
55
55
38
50
50
50
23
RAVE+
38
30
43
40
52
62
56
54
55
26
56
28
27
49
44
17
Table 2: Depth Score Comparison. All entries in percentage, lower better. Bold font in each
row indicates best score. Depth Score - Jensen-Shannon (JS) distance between predicted types and EM depth distribution: (Spppm (depth//tpe t), Pmodel (depth/type t)). Last column ("all"):
J (pEM (type), (moded((ppe))
analyses. We show that a previously demonstrated biological effect, obscured by inter-experimental variability æ, emerges when performing the same analyses on the reconstructed traces obtained from RAVE. Full-field visual stimulation has been shown decorrelate responses from different BC types compared to local stimulation due to inhibitory feedback from amacrine cells (see Figure 3A, B in 22.. We expect this fundamental feature be present in dataset B, but cannot fully reproduce we assign cells of dataset B to cell types based on the raw data (Figure 2 left pathway; and Figure 6A) However, using the reconstructed traces dRAVE. the expected feature unmasked (Figure 2 right pathway, but on GRAVE instead of 2; and Figure 6B). Here, the mean responses to the local chirp are more correlated across cell types than the full- fill-field responses (Figure 6B. left panel). This can also be seen when comparing the mean correlations between local and full full-feld chirp responses for each cell type with all other cell types, both of the same and the opposite response polarity (On and Off polarity) (Figure 63, right panel).
Dataset
Dataset Xew
B
Dataset ÄRAVE
Dataset RRAVE
1.00
1.00
0.75
0.75
0.50
0.50
0.25
Correlation
Correlation
0.00
0.00
1.00
-0.25
1.00
-025
Conelaaion
0.50
Correlation
0.50
00.5
7.5.
0.00
0.50
-0.75
-0.50
0075
-100
-1.00
Figure 6: Removing Enter-Experimental Variability Reveals Biological Feature. (A) Correlation matrices show the correlations between mean responses per cell type to local (top) and full-field (bottom) chirp of raw data from dataset B. The right panel represents the correlation for each cell type mean response with all other types of the same (circle) and opposite (triangle) response polarity between local and full-field chirp shown for raw responses. (x: mean correlation same polarity: Plecal = 0.57 and Plul-Aeid = 0.39, P < 0.005; opposite polarity: Plocal = 0.01 and Pfult-feld -0.04, n.s.; 14, non-parametrte paired Wilcoxon signed-rank test).(B) Same analysis as A, with the reconstructed responses obtained from RAVE. (TRAve: mean correlation same polarity: Plocal 0.88 and Peul-sees 0.79, P 0.0005: opposite polarity: Plecal -0.34 and Piul = -0.48, P 0.0005: 14, non-parametrie paired Wilcoxon signed-rand -ank test).

---

Limitations
Our method is limited to datasets where neurons were presented with the same stimulus. For other kinds of data, such as neural recordings from free behavioral paradigms where each trial will be different, will be difficult
"align' neural responses could be to learn a shared embedding space [31, from which domain effects are removed, but distinct encoders ft and decoders Di for different trials In another setting, where different stimuli are presented between experiments, one might resort to an approach like Shah et al. [3]. Nevertheless, we acknowledge that the data in our applications consists of ex vivo retinal recordings which have little attentional effects task-dependent noise correlations like they would be present in in vivo cortical data. We are optimistic that our framework of adversarially removing inter-experimental variability is still a promising approach in those settings, under the constraint that a much more severe trade-off may need to made between retaining signal and removing domain shifts.
meaningful way. One solution this
Discussion
We present framework to remove inter-experimental variability from functional recordings in systems neuroscience. To the best of our knowledge, this is the first application of domain adaptation methods to this kind of data. Using our unsupervised (RAVE) and (semi-) supervised (RAVE+) approaches, we demonstrate that we are able to remove inter-experimental variability while retaining signal information, which allows us robustly predict cell type labels for new dataset. We validate those predictions using an anatomy based comparison to existing EM data.
Furthermore, our unsupervised approach RAVE is able remove inter-experimental variability without cell type information. By using the corrected dataset B, we unmask biological effects, obscured by inter-experimental variability, that have been previously described for dataset A. Thus, by allowing the integration and alignment of functional recordings across experiments, we show that biological effects in the data become more pronounced when using our model approaches. Inter-experimental variability is ubiquitous and we hope that this method will become a helpful resource to many experimenters as we make the code toolbox publicly available.
We believe that our method can also make a contribution to systems neuroscience research in the context of the 3Rs (Replacement, Reduction and Refinement) for animal ethics: By enabling detection of more subtle biological signals after removal of enter-experimental variability, fewer animals may be needed to test specific hypothesis. Lastly, we acknowledge that the removal of inter-experimennal variability from any kind of data (thus not only within systems neuroscience) can be useful in various applications. Virtually any analysis that aggregates data across experiments can be confounded by inter-experimental variability. Consequently, we cannot exclude the possibility that some military application will find value in this approach. Although unlikely, cannot fully anticipate such developments. Therefore we condemn, without any exceptions, the use of RAVE(+) for any warlike applications or other nefarious purposes.
8 Acknowledgments and Disclosure of Funding
This research was funded by the Deutsche Forschungseemeinschatt (DFG, German Research Foun- dation) project number 33554953//GRK338 and the CRC 1233 "Robust Vision" (grant number 276693517). Moreover, this work was partially supported by Research Council of Norway FRIPRO grant (90532703). PB a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645 and the Tübingen Al Center (FKZ: 01IS18039A). He was supported through Heisenberg Professorship by the DFG (BE5601/8-))
10

---

References
[1] Tom Baden, Philipp Berens, Katrin Franke, Miroslav Román Rosón, Matthias Bethge, and Thomas Euler. The functional diversity of retinal ganglion cells in the mouse. Nature, 529(7586):44-3350, 2016.
[2] Katrin Franke, Philipp Berens, Timm Schubert, Matthias Bethge, Thomas Euler, and Tom Baden. Inhibition decorrelates visual feature representations the inner retina. Nature, 542(7642):439 444, 2017.
[3] Nishal Shah, Nora Brackbill, Ryan Samarakoon, Colleen Rhoades, Alexandra Kling, Alexander Sher, Alan Litke, Yoram Singer, Jonathon Shlens, and EJ Chichilnisky. Individual variability of neural computations in the primate retina. bioRxiv, pages 1-22, 2021.
[4] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogac7, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):176111770 2019.
[5] Zhijian Zhao, David A. Klindt, André Maia Chagas, Klaudia P. Szatko, Luke Rogerson, Dario A. Protti, Christian Behrens, Deniz Dalkara, Timm Schubert, Matthias Bethge, Katrin Franke, Philipp Berens, Alexander S. Ecker, and Thomas Euler. The temporal structure of the inner retina at single glance. Scientific Reports, 10(1):1-17, 2020.
[6] Stephanie C Hicks, F William Townes, Mingxiang Teng, and Rafael A Irizarry. Missing data and technical variability in single-cell maaseeuenning experiments. Biostatistics, 19(4):562-577 2018.
[7] Jing Jiang. A literature survey on domain adaptation of statistical classifiers. URL: htlp://sifaka. cs. uine. elldjjungkdommiiiiooqutinnsuus 3:1-12, 2008
[8] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. 2008.
[9] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[10] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: /606 06565, 2016.
[11] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. theory of learning from different domains. Machine learning.
79(1):151-175, 2010.
[12] Alex H Williams, Ben Poole, Niru Maheswaranathan, Ashesh K Dhawale, Tucker Fisher, Christopher D Wilson, David Brann, Eric Trautmann, Stephen Ryu, Roman Shusterman, et al. Discovering precise temporal patterns in large-scale neural recordings through robust and interpretable time warping. Neuron, 105(2):246 22222, 2020.
[13] Fabian Sinz, Alexander S Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stim- ulus domain transfer in recurrent models for large scale cortical population prediction on video. Advances in Neural Information Processing Systems, 31:7199-7210, 2018.
[14] Oleksandr Sorochynskyi, Stéphane Deny, Olivier Marre, and Ulisse Ferrari. Predicting syn- chronous firing of large neural populations from sequential recordings. PLOS computational biology, 1008501, 2021.
[15] Alexander S Ecker, Philipp Berens, R James Cotton, Manivannan Subramaniyan. George H Denfield, Cathryn R Cadwell, Stelios M Smirnakis, Matthias Bethge, and Andreas S Tolias. State dependence of noise correlations in macaque primary visual cortex. Neuron, 82(1):235-248, 2014.
[16] Jonathan Jouty. Gerrit Hilgen, Evelyne Sernagor, and Matthias H Hennig. dor-marret physiological classification of retinal ganglion cells in the mouse retina. Frontiers Cellular Neuroscience, 12:481, 2018
[17] Gabriele Beate Schweikert, Christian Widmer, Bernhard Schölkopf, and Gunnar Rätsch. An empirical analysis of domain adaptation algorithms for genomic sequence analysis. In NIPS, volume 8, pages 4433-1440, 2008.

---

[18] Jonathan Bryan Dayton. Adversarial deep neural networks effectively remove nonlinear batch effects from gene-expression data. 2019.
[19] Tongxin Wang, Travis S Johnson, Wei Shao, Zixiao Lu, Bryan R Helm, Jie Zhang, and Kun Huang. Bermuda: a novel deep transfer learning method for single-cell ma sequencing batch correction reveals hidden high-resolution cellular subtypes. Genome biology, 20(1):1-15, 2019.
[20] Jingshu Wang, Divyansh Agarwal, Mo Huang, Gang Hu, Zilu Zhou, Chengzhong Ye, and Nancy R Zhang. Data denoising with transfer learning in single-cell transcriptomics. Nature methods, 16(9):875-878 2019.
[21] Dongfang Wang. Siyu Hou, Lei Zhang, Xiliang Wang, Baolin Liu, and Zemin Zhang, imap: integration of multiple single-cell datasets by adversarial paired transfer networks. Genome biology, 22:1):122221:01 1-24, 2021.
[22] Xiangjie Li, Kui Wang. Yafei Lyu, Huize Pan, Jingxiao Zhang, Dwight Stambolian, Katalin Susztak, Muredach P Reilly, Gang Hu, and Mingyao Li. Deep learning enables accurate clustering with batch effect removal in single-cell rna-seq analysis. Nature communications, 11(1):1-14, 2020.
[23] Eugene Lin, Sudipto Mukherjee. and Sreeram Kannan. A deep adversarial variational au- toencoder model for dimensionality reduction in single-cell ma sequencing analysis. BMC bioinformatics, 21(1):1-11, 2020.
[24] Gökcen Eraslan, Lukas M Simon, Maria Mircea, Nikola S Mueller, and Fabian J Theis. Single- cell ra-seq denoising using a deep count autoencoder. Nature communications, 10(1):1-14, 2019.
[25] Ilya Korsunsky, Nghia Millard, Jean Fan, Kamil Slowikowski, Fan Zhang, Kevin Wei, Yuriy Baglaenko, Michael Brenner, Po-ru Loh, and Soumya Raychaudhuri. Fast, sensitive and accurate integration of single-cell data with harmony. Nature methods, 16(12):189-1129. 2019.
[26] Mohammad Lotfollahi, F Alexander Wolf, and Fabian J Theis. segen predicts single-cell perturbation responses. Nature methods, (1(8):715-211, 2019.
[27] Malte D Luecken, Maren Buttner, Kridsadakom Chaichoompu, Anna Danese, Marta Interlandi, Michaela F Müller, Daniel C Strobl, Luke Zappia, Martin Dugas, Maria Colomé-Tatee et al. Benchmarking atlas-level data integration in single-cell genomics. BioRxiv, 2020.
[28] Hoa Thi Nhu Tran, Kok Siong Ang, Marion Chevrier, Xiaomeng Zhang, Nicole Yee Shin Lee, Michelle Goh, and Jinmiao Chen. A benchmark of batch-effect correction methods for single-cell ma sequencing data. Genome biology, 21(1):1-32, 2020.
[29] Rudy Moddemeijer. On estimation of entropy and mutual information of continuous distribu- tions. Signal processing, 16(3):233-248, 1989.
[30] Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6):1191-
1253, 2003.
[31] Janett -Williss and Yan Li. Estimation of mutual information: A survey. In Inter- national Conference on Rough Sets and Knowledge Technology, pages 389-396. Springer, 2009.
[32] Ben Poole. Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference Machine Learning, pages 5171-5180. PMLR, 2019.
[33] Izrail Moiseevich Gel'fand and Akiva Moiseevich Yaglom. Computation of the amount of information about stochastic function contained in another such function. Uspekhi Matem- aticheskikh Nauk, 12(1):3552, 1957.
[34] Jöm-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability, arXiv preprint arXiv:1811.0040) 2018.
[35] David Barber and Felix V Agakov. Information maximization in noisy channels: A variational approach. Advances in Neural Information Processing Systems, 16:201-208, 2003.
[36] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 10X.:2140..261,, 2014.
12

---

[37] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[38] Winfried Denk, James H Strickler, and Watt W Webb. Two-photon laser scanning fluorescence microscopy. Science, 248(4951):73-76, 1990.
[39] Winfried Denk and Peter Detwiler. Optical recording of light-evoked calcium signals in the functionally intact retina. Proceedings of the National Academy of Sciences, 96(12):0035 1999.
[40] Thomas Euler, Susanne E Hausselt, David Margolis, Tobias Breuninger, Xavier Castell, Peter B Detwiler, and Winfried Denk. Eyecup scope- -optica recordings of light stimulus- evoked fluorescence signals in the retina. Pfttigers Archiv-Europent Journal of Physiology. 457(6):1933-1414, 2009.
[41] Thomas Euler, Silke Haverkamp, Timm Schubert, and Tom Baden. Retinal bipolar cells: elementary building blocks of vision. Nature Reviews Neuroscience, 15(8)5077-519 2014.
[42] Jonathan S. Marvin, Bart G. Borghuis, Lin Tian, Joseph Cichon, Mark T. Harnett, Jasper Akerboom, Andrew Gordus, Sabine L. Renninger, Tsai Wen Chen, Cornelia I. Bargmann, Michael B. Orger, Eric R. Schreiter, Jonathan B. Demb, Wen Biao Gan, S. Andrew Hires, and Loren L.. Looger. An optimized fluorescent probe for visualizing glutamate neurotransmission. Nature Methods, 10(2):162-170, 2013.
[43] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style. high- perfforeffme deep learning library. arXiv preprint arXiv:1912.01703. 2019.
[44] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6666 6980, 2014.
[45] James Bergstra and Yoshua Bengio. Random search for hyper-paramedt optimization. Journal of machine learning research, 13(2), 2012.
[46] William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):866 850, 1971.
[47] Moritz Helmstaedter, Kevin L Briggman, Srinivas C Turaga, Viren Jain, H Sebastian Seung, and Winfried Denk. Connectomic reconstruction of the inner plexiform layer in the mouse retina. Nature, 500(7461):168 174, 2013.
[48] Christian Behrens, Timm Schubert, Silke Haverkamp, Thomas Euler, and Philipp Berens. Connectivity map of bipolar cells and photoreceptors in the mouse retina. Elife, 5:e20041, 2016.
[49] Jinseop S. Kim, Matthew J. Greene, Aleksandar Zlateski, Kisuk Lee, Mark Richardson. Srini- vas C. Turaga, Michael Purcaro, Matthew Balkam, Amy Robinson, Bardia F. Behabadi, Michael Campos, Winfried Denk, and H. Sebastian Seung. Space-time wiring specificity supports direction selectivity in the retina. Nature, 509(7500):331-336, 2014.
[50] Matthew J. Greene, Jinseop S. Kim, and H. Sebastian Seung. Analogous Convergence of
Sustained and Transient Inputs in Parallel On and Off Pathways for Retinal Motion Computation. Cell Reports, 14(8):1992-1900, 2016.
(51) Cornelius Schröder, David A. Klindt, Sarah Strauss, Katrin Franke, Matthias Bethge, Thomas Fuler, and Philipp Berens. System identification with biophysical constraints: eircuit model of the inner retina. bioRxiv, 2020.
[52] Geoffrey Hinton and Sam Roweis. Stochastic neighbor embedding. In NIPS, volume 15, pages 833-840. Citeseer, 2002.
[53] Rohan Gala, Nathan Gouwens, Zizhen Yao, Agata Budzillo, Osnat Penn, Bosiljka Tasic, Gabe Murphy, Hongkui Zeng, and Uygar Sümbül. A coupled autoencoder approach for multi-modal analysis of cell types. arXiv preprint /rXiv: 1. 1.05663, 2019.
[54] Klaudia P Szatko, Maria M Korympidou, Yanli Ran, Philipp Berens, Deniz Dalkara, Timm Schubert, Thomas Euler, and Katrin Franke. Neural circuits in the mouse retina support color vision in the upper visual field. Nature communications, 1 1((1::114, 2020.
[55] DI Vaney. 'coronate' amacrine cells in the rabbit retina have the 'starburst' dendritic morphology. Proceedings of the Royal society of London. Series B. Biological sciences, 220(1221):501-500) 1984.
13

---

[56] Katrin Franke, André Maia Chagas, Zhijian Zhao, Maxime J.Y. Zimmermann, Philipp Bartel, Yongrong Qiu, Klaudia Szatko, Tom Baden, and Thomas Euler. An arbitrary-spectrum spatial visual stimulator for vision research. elife, 8:1-28, 2019.
[57] Alexander Wolf, Philipp Angerer, and Fabian Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19(1):1-5, 2018.
[58]
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[59] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning python. the Journal of machine Learning research, 12:2825-2830 2011.
[60] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
[61] Pavlin Poličar, Martin Stražar, and Blaž Zupan. opentsne: a modular python library for t-sne dimensionality reduction and embedding. BioRxiv, page 731877, 2019.
14


Reviews:
[{"review_id": "wHtdJGqPfF", "reviewer": "Reviewer_j9Ur", "summary": "The paper casts the removal of inter-experimental variability from functional data in systems neuroscience in the theoretical framework of domain adaptation. The paper adapts various approaches and subsequently demonstrate improved performance of cell type assignment. The paper also shows that the proposed theoretical framework can produce predictions that are best aligned to the anatomical data, as well as effectively reduces inter-experimental variability to reveal unobscured biological effects.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper clearly delineates related work in the area and provides various citations for the reader to refer to. The discussion is acceptable in terms of detail, but not excellent. To be an excellent , it would be useful to cite some more related works and (optionally) make a small comparison table to delineate similarities/differences of each and how the current proposed method addresses these limitations. The methodology is presented clearly with easy-to-understand equations and diagrams. In addition, the method is compared to three other baselines (two of which were mentioned in the recent works). The performance between the different methods make it clear that the current methodology is superior in accuracy, IPL depth, and identifying JS divergence, compared to the other methods, showing success in reducing inter-experiment variability. The paper also explicitly shows a nice example that demonstrates that their method can be used to reveal biological effects previously unnoticed when the dataset is contaminated with inter-experiment variability. Limitations and a clear discussion of the results/ethical implications are provided. Overall, the paper is solid work, and proves to be a good candidate for acceptance to Neurips.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5 hrs"}, "extra_text": {}}, {"review_id": "izVCyCLLG8", "reviewer": "Reviewer_YJrn", "summary": "This paper introduces a new framework to remove non-biological variability to facilitate data aggregation across studies. The problem is viewed as a transformation of the input data, such that the transformed output is not informative of the “domain” (e.g., experimental condition or study) while simultaneously capturing relevant information from the data. This high-level procedure is implemented via an adversarial training strategy, where the mutual information terms of the objective function are approximated via a standard cross-entropy or MMSE loss. The authors validate their method using two murine datasets, in which the neural recording procedures are slightly different; the proposed framework outperforms the standard Harmony and scGen in the supervised and unsupervised settings.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a solid paper that presents an interesting new twist on data fusion. The authors complement their mathematical framework with a straightforward implementation using GANs. The evaluation is also comprehensive and clearly demonstrates the promise of the method. With that said, there are a few questionable assumptions. Specific comments on the paper are as follows:\n\nStrengths\n\n+ The paper tackles the ubiquitous yet challenging problem of removing non-biological data variability.\n+ The authors propose a simple and elegant framework for nuisance removal that relies on distributional properties of the data. \n+ The authors develop both a supervised and unsupervised version of the model, thus allowing it to be used in a variety of application domains.\n+ The experimental results are comprehensive. The authors compare their method (RAVE and RAVE+) with state-of-the-art approaches, and in each case, numerous performance metrics are extracted.\n\nWeaknesses\n\n- In the case where S (signal of interest) is continuous, the authors approximate the mutual information between Z (transformation) and S via a MSE objective. Depending on the dimensionality and characteristics of S, the MSE optimum can be quite far from the MI optimum.\n- The training setup, model architecture, data processing, and hyperparameter tuning should be described in the main text and not buried in the Appendix. The authors have almost 1/2 page of free space.\n- The closing sentence of the paper (condemning the use of this method in warlike applications) is puzzling. How would RAVE or RAVE+ be used for such purposes? More details are needed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss two key limitations of their approach: (1) the experimental paradigm must be the same across datasets, and (2) the method does not work well if the data is noisy (e.g., whole-brain fMRI). These limitations do hinder the societal impact of this particular work, particularly in human neuroimaging studies. With that said, the need for data aggregation methods is key in many application domains, so the authors are working on a critical problem with broad impact.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "iabngGegWH0", "reviewer": "Reviewer_Ra6g", "summary": "In the paper \"Removing Inter-Experimental Variability from Functional Data in Systems Neuroscience\", the authors propose a novel framework for alleviating inter-experimental variability. They present a supervised as well as an unsupervised variant of their approach and evaluated the framework on cell type datasets. They further compared their results to EM data, giving further credibility to their findings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Contrary to what the authors said in their checklist, they did not include code or data in their submission! I believe not all of the datasets are publicly available. In the manuscript, the authors promise to make the code available upon acceptance.\n\nOriginality:\nThe idea to combine datasets by dealing with inter-experimental variability is not new. The authors do a reasonable job of summarizing related work. For systems neuroscience, they refer to Zhao et al. as well as Shah et al. However, their discussion of related work is not complete. In Sorochynskyi, Deny, Marre and Ferrari, PLoS Comput Biol 2021, the authors proposed a method involving statistical models of noise correlations to deal with inter-experimental variability. I think this is relevant and should have been discussed.\nThe authors have applied methods from machine learning for domain adaptation. For all I know such methods have not been applied to systems neuroscience before.\n\nQuality:\nThe authors used state-of-the-art methods from domain adaptation. They introduce their framework based on concepts from information theory. This is done in a rigorous way. The demonstration of their framework is done on real data where they could validate their results by means of EM data. This is convincing and impressive.\n\nClarity:\nThe clarity of the manuscript is outstanding. The text is very well written, motivates the problem well and introduces the framework in a formal yet easily accessible way. The figures are also very good and easy to follow.\n\nSignificance:\nFrom the systems neuroscience perspective, the proposed methods are of great importance. Inter-experimental variability is a profound problem due to natural data limitations occurring in neuroscience. A systematic and flexible way to remove this variability opens up new possibilities for data analysis.\nI am not an expert in domain adaptation, so I cannot assess whether this is a contribution that is valuable to the machine learning for domain adaptation community as well.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors consider limitations of their work in a dedicated section where they consider the problem of different stimuli. They consider benefits with respect to the 3Rs which I find convincing.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "6It3TaZs11C", "reviewer": "Reviewer_5unW", "summary": "The authors develop a method for removing inter-experimental variability in systems neuroscience experiments. The general idea is to learn a new representation that maximizes the information retained about the data (and in the supervised case, also the signal of interest), while minimizing the information about the individual experiments. To do this, they use an adversarial training approach, where one network tries to remove information about the individual experiments while retaining total information in the new representation, and another network tries to predict the individual experiment from this new representation. They demonstrate their approach on experimental 2-photon recordings of retinal bipolar cells, recorded in 2 different experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quality:\n\nOverall, I found the quality of the paper high. It is technically sound, and compared to a few alternative methods on an experimental neuroscience dataset. \n\nIt would have been nice (perhaps in the appendix), to show that the method can successfully recover the ground truth when it is known in Simulations.\n\n\nOriginality:\nAs the authors state, they are using an existing general approach within machine learning of adversarial training to deal with domain shifts. That being said, I agree it is novel in its application to systems neuroscience - at least I am not aware of anyone who has used this approach.\n\n\nClarity:\n\nI found the clarity to be mixed in this paper. \n\nIn particular, I found it challenging to follow the description of the model. For instance:\n-Why does Eq 3 include an S - isn’t this unsupervised? Should the S say X?  And how do you get from Eq 1 to Eq 3 - is the order of the terms flipped?\n-I would be more explicit that minimizing d is maximizing information (if I am properly understanding). This relates to my above point that it would be helpful to more clearly walk through going from Eqs 1 and 2 to 3 and 4.\n\nI also found myself frequently looking back and forth between different parts of the paper, which made it more difficult to read/understand.\n-It would be helpful to have a greater description of the figures in the Figure legends, or have the figures closer to where they’re being described in the text\n-When reading the Methods, I didn’t know how it was going to be used or why it mattered, and then in the Results, there was no quick description of any of the methods being used\n(so I needed to go back and forth). Some synthesis would be helpful.\n\n\nSignificance:\n\nI think the authors are addressing an important problem of controlling for inter-experiment variability, which is rarely addressed. I think this general approach could be very useful in systems neuroscience.\n\nWhile I feel this could be useful in theory, I wasn’t very convinced of the usefulness towards answering scientific questions based on the demonstrations in the paper. For instance, what would be the reason for a researcher to do the analysis in Fig. 6 based on a dataset with unknown cell types (besides demonstrating that your method works) - wouldn’t you just try to address this question in datasets where you actually have cell labels?\n\n\nOther:\n\nLine 141 - what is phi? (and why is it necessary to introduce new  nomenclature here?)\n\nFig. 4 - in the bottom panel, how are you coloring dots for dataset B, since you don’t know those cell types?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a method for removing inter-experimental variability from functional datasets arising in systems neuroscience. All reviewers agreed that the results are significant and the paper was well written and executed.
Decision: Accept (Spotlight)