Beyond GANs: Transforming without a Target Distribution
Authors: Matthew Amodio
David van Dijk
Ruth Montgomery
Guy Wolf
Smita Krishnaswamy
Venue:  (2020)
Keywords: GAN
domain transfer
computational biology
latent space manipulations

Abstract: While generative neural networks can learn to transform a specific input dataset into a specific target dataset, they require having just such a paired set of input/output datasets. For instance, to fool the discriminator, a generative adversarial network (GAN) exclusively trained to transform images of black-haired *men* to blond-haired *men* would need to change gender-related characteristics as well as hair color when given images of black-haired *women* as input. This is problematic, as often it is possible to obtain *a* pair of (source, target) distributions but then have a second source distribution where the target distribution is unknown. The computational challenge is that generative models are good at generation within the manifold of the data that they are trained on. However, generating new samples outside of the manifold or extrapolating "out-of-sample" is a much harder problem that has been less well studied. To address this, we introduce a technique called *neuron editing* that learns how neurons encode an edit for a particular transformation in a latent space. We use an autoencoder to decompose the variation within the dataset into activations of different neurons and generate transformed data by defining an editing transformation on those neurons. By performing the transformation in a latent trained space, we encode fairly complex and non-linear transformations to the data with much simpler distribution shifts to the neuron's activations. Our technique is general and works on a wide variety of data domains and applications. We first demonstrate it on image transformations and then move to our two main biological applications: removal of batch artifacts representing unwanted noise and modeling the effect of drug treatments to predict synergy between drugs.

Under review as a conference paper at ICLR 2020
BEYOND GANS: TRANSFORMING WITHOUT A TAR- GET DISTRIBUTION
Anonymous authors Paper under double-blind review
ABSTRACT
While generative neural networks can learn transform specific input dataset into a specific target dataset, they require having just such a paired set of in- put/output datasets. For instance, to fool the discriminator, a generative adversar- ial network (GAN) exclusively trained to transform images of black-haired men alond-haired men would need change gender-related characteristics as well as hair color when given images of black-haired women as input. This is problem- atic, as often it is possible to obtain a pair of (source, target) distributions but then have second source distribution where the target distribution is unknown. The computational challenge is that generative models are good at generation within the manifold of the data that they trained on. However, generating new sam- ples outside of the manifold or extrapolating "out-of- sittoff--ample a much harder problem that has been less well studied. To address this, introduce technique called neuron editing that learns how neurons encode an edit for a particular trans- formation in a latent space. We an autoencoder decompose the variation within the dataset into activations of different neurons and generate transformed data by defining an editing transformation on those neurons. By performing the transformation in a latent trained space, we encode fairly complex and non-linear transformations the data with much simpler distribution shifts the neuron's 's activations. Our technique has the advantage of being generally applicable to a wide variety of data domains, modalities, and applications. We first demonstrate it image transformations and then move our two main applications in bi- ology: removal of batch artifacts representing unwanted noise and modeling the effect of drug treatments to predict synergy between drugs.
INTRODUCTION
common situation arises when we have two datasets and seek to learn a transformation that is a mapping from one (the source) to the other (the target). While much existing work has been done on this, less studied is the case where we want to learn a transformation from this source/target pair of datasets and apply it a second source dataset for which there is no known target.
If the second source distribution differs from the first source distribution, any transformation naively learned with a neural network on only the first source/target pair will suffer from problems including domain shift (the second source distribution systematically differs from the first) and overfitting (which aspects of the target only exist because it started as the first source distribution, and shouldn't be part of the learned transformation?).
This problem is important address. as learning transformations is common task in many different contexts, and often it is infeasible or impossible to obtain known source/target pairing information for every distribution needing to transformed. For example, many experiments biology are con- ducted to study the effect of treatment However, due to their expense and difficulty, clinical trials are often performed on only small subset of the samples. The challenge is isolating reatment-induced. variation from the confounding sample-specifi variation.
set of samples, such as tissues from different patients.
We propose neural network-basee method for learning a transformation that is general enough to be used across a wide range of data modalities and applications, from image-to saage-to-image translation to treatment in the biological setting. Popular neural network architectures like GANs pose the problem

---

Under review as a conference paper at ICLR 2020
of learning to output data in the region of the space occupied by the target distribution, no matter where the input data is coming from. To fool the discriminator, the generator's i output must end up in the same part of the space the target distribution. The discriminator does not take into account the input points into the generator in any way.
Instead, reframe the problem as learning transformation towards the target distribution that is more sensitive to where the input data starts. Thus, we could learn an edit between one source and target pair, and apply it a second source without needing to assume it has no systematic differences from the first source.
We propose learn such an edit, which we term neuron editing, in the latent space of autoencoder neural network with non-linear activations, First we train an autoencoder on the entire population of data which we are interested in transforming, This includes both the paired source/target data and the second source data. Neuron editing then involves extracting observed differences between the source/target activation distributions for neurons in this layer and then applying them to the second source data to generate a synthetic second target dataset. Performing the edit node-by-node in this space actually encodes complex multivariate edits in the ambient space, performed on denoised and meaningful features, owing to the fact that these features themselves are complex non-linear combinations of the input features.
Neuron editing is a general technique that could be applied to the latent space of any neural network, even GANs themselves. We focus exclusively on the autoencoder in this work, however, to leverage its denoising ability, robustness to mode dropping, and superior training stability compared to GANS. We demonstrate that neuron editing can work on a variety of architectures, while offering the advantages of introducing no new hyperparameters to tune and being stable across multiple runs.
While latent space manipulation has been explored in previous work, ours differs in several ways. For example, Raadordd al.] (2015) represents transformation between two distributions constant shift in latent space. In addition to assuming the latent transformation is the same for all points in the distribution, [upchurch et al. (2017) also uses an off-the-shelf pre-trained Imagenet classifier network. Our work, on the other hand, does not require a richly supervised pre-trained model; also, we model the shift between two distributions as a complex, non-constant function that learns different shifts for different parts of the space. We compare to this "constant-shi shifftttttssf"" approach and demonstrate empirically why it is necessary to model the transformation more complexly.
single
Some neurons are not heavily edited but still influence the output jointly with those neurons that are edited due to their integration in the decoding layers, propagating their effect into the output space. Thus, even relatively simple transformation in the internal layer of an autoencoder allows for modeling complex transformations in the ambient data space.
This aspect of neuron editing draws close connections with the field of domain adaptation, where the goal is to learn features on one labeled dataset that meaningfully separate points in another, unlabeled dataset (Tzeng et al. 2017). Similarly to that task, want to learn a transformation from the known source to the known target samples that will also apply to the second source dataset where the target is unknown. Thus, neuron editing represents an extension of domain adaptation, where instead of learning a classifier that can be used on the unlabeled data, we are learning a distribution transformation that can be used on the unlabeled data. Further differences include that while domain adaptation attempts to make features for the unlabeled dataset overlap with those of the labeled dataset, neuron editing transforms the second source dataset without first trying to align it to the first source dataset (Sankaranarayanaa et al. 2018). Also, different from many domain adaptation techniques, we do not need any sort of pre-trained classifier to yield an informative feature map for the data, as we learn our autoencoder de novo (Long et al.) (017). Given the near exclusive focus of the domain adaptation community on learning classifiers on labeled data and applying it to unlabeled data, we are excited to expand the field to also learning transformations on data with known targets and applying it to data with unknown targets.
We demonstrate that neuron editing extrapolates better than generative models on two important criteria. First, as to the original goal, the predicted change on the second source dataset more closely resembles the predicted change on the original source dataset. Second, the editing process produces more complex variation, since it simply preserves the existing variation in the data rather than need- ing a generator learn to create it. We compare to standard GAN approaches, dedicated parametric statistical methods used by computational biologists, and alternative autoencoder frameworks. In
2

---

Under review as a conference paper at ICLR 2020
(8)
Training
Inference
Encoder
Decoder
Encoder
Neuron Edit
Decoder
Input
Neuron Edit
Nouron
Train Auttenooder
Entoot Hidden
M
Sourne
Gee
Extepoation
Torget
Output
M
Pro-
Pork
Complete Forward
Reprace Hadden Loyar
Figure (a) Neuron editing interrupts the standard feedforward process, editing the neurons of trained encoder/decoder to include the source-tor source-to-targe to-target variation, and letting the trained decoder cas- cade the resulting transformation back into the original data space. (b) The neuron editing process. The transformation is learned on the distribution of neuron activations for the source and applied to the distribution of neuron activations for the extrapolation data.
each case, we see that they stumble on one or more of several hurdles: out-of-sample input, desired output that differs from the target of the training data, and data with complex variation.
In the following section, we detail the neuron editing method. Then, we motivate the extrapolation problem by trying perform natural image domain transfer the canonical CelebA dataset (Liu et al. 2018). We then move to two biological applications where extrapolation is essential: correct- ing the artificial variability introduced by measuring instruments (batch effects), and predicting the combined effects of multiple drug treatments (combinatorial drug effects) (Anchang el al. 2018).
MODEL
Let S E Rnsxd T Xx represent d-dimensional source, target, and second source distributions with nss n7, and nx observations, respectively. We seek a transformation such that: when applied to S it produces distribution equivalent to T 2. when applied 7' it is the identity function and 3. when applied to X it does not necessarily produce GANs learn transformation with the first two properties, they fail the third property due the fact that T is the only target data we have for training, and thus the generator only learns to output data like Therefore, instead learning such a transformation parameterized by a neural network, we learn simpler transformation on a space learned by neural network (summarized in Figure
S is different from X. While
We first train an encoder/decoder pair E/D map the data into an abstract neuron space decom- posed into high-level features such that can also decode from that space, i.e., the standard autoen- coder objective
L(S, X) MSE D(E(S,T,
where MSE is the mean squarrd error. The autoencoder is trained on all three data distributions S, T. and X and thus leams to model their joint manifold. Then, without further training, separately extract the activations of an n-dimensional internal layer of the network for inputs from S and from T, denoted by as S R R²,ar or Ra, We define piecewise linear transformation, called
3

---

Under review as a conference paper at ICLR 2020
NeuronEdit, which apply to these distributions of activations:
NeuronEdil(a)) - PJ+1- P5
(05 10 -17) + +D
where a E R" consists of n activations for single network imput, p5. R" consist of the jth percentiles of activations (i.e., for each of the n neurons) over the distributions of ag, or cor- respondingly, and all operations are taken pointwise, i.e., independently on each of the n neurons in the layer. Then, we define N Newron (as) : S - IR" given by x + Neuron (dil(as(x)) and equivalently for ar and any other distribution (or collection) of activations over a set of network inputs. Therefore, the NeuronEdis function operates on distributions, represented via activations over network input samples, and transforms the input activation distribution based on the difference between the source and target distributions (considered via their percentile disctretization).
We note that the NeuronEdit function has the three properties we stated above: (in terms of the represented n--immnnsioal distributions) Edil(ax)))( = Newron Edil(u))) = ax = uss This last property is crucial since learning to generate distributions like T, with GAN for example, would produce a discriminator who encourages the output to be funneled as close to T as posssible no matter where in the support start from.
1. Neuron Edit(a)) z Newron (dil(ar) = ar
To apply the learned transformation to X, we first extract the activations of the internal layer com- puted by the encoder, ax Then, we edit the activations with the neuron editing function åx. Finally, we cascade the transformations applied to the neuron activations through the decoder without any further training. Thus, the transformed output X is obtained by:
= Edi
We emphasize that at this point, since we do no further training of the encoder and decoder, and since the neuron editing transformation has no weights to learn, there no further objective term to minimize at this point and the transformation is fully defined.
Crucially, the nomenclature of an autoencoder no longer strictly applies. If we allowed the encoder or decoder to train with the transformed neuron activations, the network could learn to undo these transformations and still produce the identity function. However, since we freeze training and apply these transformations exclusively on inference, we turn autoencoder into a generative model that need not be close to the identity.
Training a GAN in this setting could exclusively utilize the data in S and T, since we have no real examples of the output for X to feed to the discriminator. Neuron editing, on the other hand, is able to model the variation intrinsic to X in unsupervised manner despite not having real post- transformation data for X. Since we know a priori that X will differ substantially from S, this provides significantly more information.
Furthermore, GANs are notoriously tricky to train (Salimans et al. 2016 Gulrajani et al. 2017 Wei let al. 2018). Adversarial discriminators suffer from oscillating optimization dynamics (Li al., 2017). uninterpretable losses (Barratt & Sharma. 2018 Arjovsky et al. 2017). and most debilitat- ingly, mode collapse (Srivastava al.) [2017] Kim al. 2017 Nagarajan
Kolter 2017). Under
mode collapse, significant diversity that should exist in the output of the generator is lost, instead producing synthetic data that severely degenerated version of the true target distribution.
Neuron editing avoids all of these traps by learning an unsupervised model of the data space with the easier-to- to-tait autoencoder. The essential step that facilitates generation the isolation of the variation in the neuron activations that characterizes the difference between source and target distri- butions.
There is relationship between neuron editing and the well-known word2vec embeddings in natural language processing (Goldberg& & Levy 2014). There, words are embedded in a latent space where meaningful transformation such changing the gender of a word is constant vector in this space. This vector can be learned on one example, like transforming man to woman, and then extrapolated to another example, like king, to predict the location in the space of queen. Neuron editing is an extension in complexity of word2vec's vector arithmetic, because instead transforming a single point into another single point, it transforms entire distribution into another distribution.
4

---

Under review as a conference paper at ICLR 2020
In-sample
Out-of-sample Penskes sablack har
Malss witacck
WMire wo
Orginal
Editing
Regularized
Autoenoder
GAN
CyeRGAN
ResnetGAN
Constani Shift
"PCA Ediino" Insooed Neuron Editing
Figure 2: Data from CelebA where the source data consists of males with black hair and the target data consists of males with blond hair. The extrapolation is then applied females with black hair. (a) A comparison of neuron editing against other models. Only neuron editing successfully applies the blond hair transformation. (b) An illustration that neuron editing must be applied the neurons of deep network, opposed to principle components.
ColebA
Neuron Editing
GAN
CycleGAN 153.03 6.55
ResnetGAN 184.31 +/ 9.71
RegAE
Constant Shift
FID
121.63 +/- 2,12
282.26
13.32
272.12 1.10
320.97 +/. 1.04
Table I: FID scores on the CelebA extrapolation task.
EXPERIMENTS
We compare the predictions from neuron editing to those of several generation-basee approaches: traditional GAN, a GAN implemented with residual blocks (ResnetGAN) to show generating resid- uals is not the same editing (Szegedy et al. 2017). and a CycleGAN (Zhu al., 2017). While in other applications, like natural images, GANs have shown an impressive ability to generate plau- sible individual points, we illustrate that they struggle with these two criteria. We also motivate why neuron editing is performed on inference by comparing against a regularized autoencoder that performs the internal layer transformations during training, but the decoder learns to undo the trans- formation and reconstruct the input unchanged (Amodio et al.] 2018). Lastly, we motivate why the more complex neuron editing transformation is necessary by comparing against naive "latent vec- tor arithmetic" approach. We find the constant vector between the mean of the source and the mean of the target in the internal layer of our pre-trained autoencoder, and apply this single shift to all neurons in the target (Constant Shift).
For the regularized autoencoder, the regularization penalized differrnnces in the distributions of the source and target in a latent layer using maximal mean discrepancy (Amodio et al. 2018 Dziugaite et al. 2013). The image experiment used convolutional layers with stride-two filters of size four, with 64- 221-22256-122 128-64 filters in the layers. All other models used fully connected layers of size 500-250-50-250-500 Leaky ReLU activation was used with 0.2 leak. Training was done with minibatches of size 100, with the Adam optimizer (Kingma & Ba 2014), and learning rate 0.001.
5

---

Under review as a conference paper at ICLR 2020
3.1 CELEBA HAIR COLOR TRANSFORMATION
We first consider a motivational experiment on the canonical image dataset of CelebA (Liu et al. 2018). If we want to learn transformation that turns given image of person with black hair to that same person except with blond hair, a natural approach would be to collect two sets of images, one with all black haired people and another with all blond haired people, and teach a generative model to map between them. The problem with this approach is that the learned model may perform worse on input images that differ from those it trained on. This has troubling consequences for the growing concern of socially unbiased neural networks, as we would want model performance to go unchanged for these different populations [Tatman 2017
This is illustrated in Fag where we collect images that have the attribute male and the attribute black hair and try to map to the set of images with the attribute male and the attribute blond hair. Then, after training on this data, extrapolate and apply the transformation to females with black hair, which had not been seen during training. The GAN models are less successful at modeling this transformation (everything but the hair color), they do not always generate a recreation of the input. In the hair color, only sometimes is the color changed. The regular GAN model especially has copious artifacts that are a result of the difficulty in training these models. This provides further evidence of the benefits of avoiding these complications when possible, for example by using the stable training of an autoencoder and editing it as we do in neuron editing.
out-of-sample data. In the parts of the image that should stay the same
We quantify the success of neuron editing by using the common metric of Frechet Inception Dis- tance (FID) that measures how well the generated distribution matches the distribution targeted for extrapolation. These scores are reported [able where we see neuron editing achieve the best re- sult on an average of three runs. Notably, due to the autoencoder's 's more stable training, the standard deviation across multiple runs is also lower than the GAN-based methods.
In Figure 2b, we motivate why we need to per- form the NewronFFde transformation on the internal layer of neural network, as opposed to applying on some other latent space like PCA. Only in the neuron space has this com- plex and abstract transformation of changing the hair color (and only the hair color) been decomposed into a relatively simple and piece- wise linear shift.
Original Edited
Original
Edited
Constant Shift
No Glasses <->> Glasses
Original
Edited
Original
Edited
Constent Shift
Mustache <->>>
Mustache
Beyond hair color transformation, neuron edit- ing is able learn general transformations on CelebA males and apply them to females. In Figure 3 we learn to transform between having/not
Figure 3: Additional CelebA transformations.
having the mustache attribute and having/not having the glasses attribute. The latter transformation on glasses demonstrates the importance of learning non-constant transformation. The glasses at- tribute bimodal, with both examples of sunglasses and reading glasses in the dataset. With neuron editing, we are able to learn to map to each of these different parts of the latent space, as opposed to the constant shift which adds dark sunglasses to the entire distribution.
3.2 BATCH CORRECTION BY OUT-OF-SAMPLE EXTENSION FROM SPIKE-IN SAMPLES
We next demonstrate another application of neuron editing's 's ability learn transform a distri- bution based on a separate source/target pair: biological batch correction. Many biological experi- ments involve using an instrument to measure different populations of cells and then characterizing the features that distinguish between them. However, these complex instruments can be difficult to calibrate and use consistently, and thus can introduce technical artifacts into the data they are used to measure. In fact, we can even measure the same population of cells twice and get two very differ- ent datasets back. When we measure different populations, these technical artifacts (batch effects) get confounded with the true differences between the populations. Batch effects are a ubiquitous problem in biological experimental data can lead to incorrect conclusions in downstream analysis. Addressing batch effects is goal of many new models (Finck et al.) 2013 Tung al. 2017 Butler & Satija [2077] Haghverdi et al. 2018), including some deep learning methods (Shaham et al. 2017 Amodio et al. 2018).

---

Under review as a conference paper at ICLR 2020
correction
Combinatorial prediction
(0)
(b)
16)
(d)
Fre-franttomm elonn
Pro Pre-tranafoon coneeoaattion
Poat-transhomttrr
Figure 4: Neuron editing corrects the variation in IFNg while preserving the variation in CCR6 and correctly predicting the effect of combining two drugs.
Neuron Editing 0.96275
GAN
CycleGAN 0.4310
ResnetGAN 0.6268
RegAE .005088
Constant Shift
CCA 0.6034
MNN
ComBat 0.5569
Limma 0.5431
0.5108
0.88205
0.5339
Table 2: Correlation between observed change in spike-ins and applied change to samples. Neu- ron editing most accurately applies just the transformation observed as batch effect and not true biological variation.
One method for grappling with this issue is to repeatedly measure an unvarying control (called a spike-in) set of cells with each population of interest (called sample) (Bacher & Kendziorski 2016). Because we know any observed differences in the spike-in are technical artifacts, we can model and then remove this artifact in the population of interest. In our previous terminology, the two spike-in distributions are our known source/target pair while the actual population of interest is second source that lacks known target.
Existing methods of batch correction based on spike-ins work directly in the data space, operate independently each dimension, and only do crude matching of distribution statistics. The most common approach is to simply subtract the difference in means between the spike-ins from the sample. We believe this natural opportunity for deep learning, where the same concept can be extended to an abstract feature space, composed of combinations of features, and a more powerful transformation. Moreover, we expect neuron editing to shine as the spike-ins ins likely differ drastically from the sample.
The dataset we investigate in this section comes from a mass cytometry (Bandura et al.) 2009) experiment which measures the amount of particular proteins in each cell in two different individuals infected with dengue virus (Amodio et al. 2018). We note that these data are a drastically different format from the images of the previous experiment, as they are in tabular form with cell being row i and the amount of protein j in column j. We believe a key strength to neuron editing is its general applicability to a wide range of data types and modalities. In this particular experiment, there are four datasets, each consisting of measurements of 35 proteins: the two spike-ins we refer to as Controll and Control2 are shape 18919 x 35 and 22802 x 35, respectively, while the two populations actually want to study, called Samplel and Sample2, are shape 94556 x 35 and 55594 x 35. To better grasp the problem of batch effects, we visualize biaxial plot with two of the proteins where there is a batch effect in one dimension and true underlying biological difference in the other dimension (Figure 4p. By using the controls, we seek to correct the artificially low readings of the protein IFNg in Sample1 I (along the x-axis) without removing the biologically accurate readings of higher amounts of protein CCR6 (along the y-axis).
We would like our model to identify this source of variation and compensate for the lower values of IFNg without losing other true biological variation in Sample1. For example. Samplel also has higher values of the protein CCR6, and as the controls show, this is true biological difference, not a batch effect (the y-axis in Figure 4a).
We quantify the performance of the models at this goal by measuring the correlation between the change in median marker values observed in the spike-in with the change applied to the sample. If this correlation is high, we know the transformation applied to the samples only removes the variation where we have evidence, coming from the spike-ins, that it is a technical artifact. This data is presented in Table where we compare to not only the deep generative models have already introduced, but also dedicated batch correction methods commonly used by practitioners (Johnson) et al. 2007 Haghverdi et al. 2018 Butler & Satija 2017). We that neuron editing outperforms
7

---

Under review as a conference paper at ICLR 2020
Figure 5: (a) The global shift in the two controls (light blue to red) is isolated and this variation is edited into the sample (dark blue to red), with all other variation preserved. (b) The median change in the sample in each dimension corresponds accurately with the evidence in each dimension in the controls.
TCAI
Chrigh Merre//errt
14]
(8)
Neuron Editing
GAN 0.87014/0.55 581511
CycleGAN
ResnerGAN
RegAE
Constant Shift 0.9668000.96922
0.99661/0.99 /0,99222
1.9268700883388
0.9455990.99300 93032
0.91 16590096053
Table 3: Correlation between real and predicted means/variances on the combinatorial drug predic- tion data, The GANs generate data that is less accurate (means are off) and less diverse (variances are smaller) than the real data, while neuron editing best models the true distribution.
all of the alternatives at extrapolating from the spike-ins to the samples. This is unsurprising, as the GAN methods are only trained to produce data like Control2, and thus will not preserve much of the variation in the sample. The traditional batch correction methods make specific parametric distributional assumptions on the data that are not held in practice, and thus also perform poorly. The regularized autoencoder, since the transformation is performed during training rather than after training like neuron editing, just reproduces its input unchanged.
In Figure 5h PCA embedding of the data space is visualized for Controll (light blue), Control2 (light red), Samplel (dark blue), and post-transformation Samplel (dark red). The transformation from I to Control2 mirrors the transformation applied to Sample Notably, the other varia- tion (intra-sample variation) is preserved. In Figureßp. we see that for every dimension, the variation between the controls corresponds accurately the variation introduced by neuron editing into the sample. These global assessments across the full data space offer additional corroboration that the transformations produced by neuron editing reasonably reflect the transformation evidenced by the controls.
3.3 COMBINATORIAL DRUG TREATMENT PREDICTION ON SINGLE-CELL DATA
Finally, we consider biological data from combinatorial drug experiment on cells from patients with acute lymphoblastic leukemia (Anchang et al. 2018). The dataset we analyze consists of cells under four treatments: no treatment (basal), 235 (Bez), Dasatinib (Das), and both Bez and Das (Bez+Das). These measurements also come from mass cytometry, this time on 41 dimensions, with the four datasets consisting of 19925, 20078, 19843, and 19764 observations, respectively. In this setting, define the source to be the basal cells, the target be the Das cells, and then extrapolate to the Bez cells. We hold out the true Bez+ Das data and attempt to predict the effects of applying Das cells that have already been treated with Bez.
Predicting the effects of drug combinations is an application which is typically approached through regression, fitting coefficients to an interaction term in a multiple linear regression model. This limitation of only fitting linear relationships and treating cach protein independently, greatly restricts the model in biological contexts where know nonlinearity and protein regulatory networks exist and play a large role in cellular function. Using neuron editing in this context facilitates learning a much richer transformation than previous, non-deep learning methods.
We quantitatively evaluate whether neuron editing produces meaningful transformation in Table where we calculate the correlation between the real and generated means and variances of each dimension. Neuron editing more accurately predicts the principle direction and magnitude of trans- formation across all dimensions than any other model. Furthermore, neuron editing better preserves the variation in the real data. The GANs have trouble modeling the diversity in the data, mani- fested by their generated data having significantly less variance than really exists.
We see example of the learned transformation by looking at characteristic effect of applying Das: a decrease in p4EBP1 (seen on the x-axis of Figure 44). No change in another dimension, PSTATS, is associated with the treatment (the y-ax of Figure b. Neuron editing accurately models this change in P4EBP1, without introducing any change in PSTATS or losing variation within the extrapolation dataset (Figure 41).
8

---

Under review as a conference paper at ICLR 2020
We note that since much of the variation in the target distribution already exists in the source distri- bution and the shift is relatively small one, we might expect the ResnetGAN to be able to easily mimic the target. However, despite the residual connections, it still suffers from the same problems as the other models using the generating approach: namely, the GAN objective encourages all output to be like the target it trained on. This leaves it unable produce the correct distribution if it differs from the target of the learned transformation, as we see in this case.
4 DISCUSSION
In this work, we have only consider learning from single pair of distributions and applying it to another single distribution. We consider it an interesting direction for future work to extend this to multiple distributions, either for learning from and application to. Additional future work along these lines could include training parallel encoders with the same decoder, or training to generate conditionally.
REFERENCES
Matthew Amodio, David van Dijk, Krishnan Srinivasan, William S Chen, Hussein Mohsen, Kevin R Moon, Allison Campbell, Yujiao Zhao, Xiaomei Wang. Manjunatha Venkataswamy, et al. Explor- ing single-cell data with deep multitasking neural networks. bioRxiv, pp. 237065, 2018.
Benedict Anchang, Kara L Davis, Harris G Fienberg, Brian D Williamson, Sean C Bendall, Loukia G Karacosta, Robert Tibshirani, Garry Nolan, and Sylvia K Plevritis. Drug-nem: Op- timizing drug combinations using single-cell perturbation response to account for intratumoral heterogeneity. Proceedings of the National Academy of Sciences, 11((1))E4294-448) 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein
gan. arXiv preprint
arXiv: 1701.1877. 2017.
Rhonda Bacher and Christina Kendziorski. Design and computational analysis of single-cell ma- sequencing experiments. Genome biology, 17(1):63, 2016.
Dmitry R Bandura, Vladimir Baranov, Olga I Ornatsky, Alexei Antonov, Robert Kinach, Xudong Lou, Serguei Pavlov, Sergey Vorobiev, John E Dick, and Scott D Tanner. Mass cytometry: tech- nique for real time single cell multitarget immunoassay based inductively coupled plasma time-of-flight mass spectrometry. Analytical chemistry, 81 81(16)668136682, 2009.
Shane Barratt and Rishi Sharma. A note the inception score. arXiv preprint arXiv: arXiv:1801.0111 1801.01933, 2018.
Andrew Butler and Rahul Satija. Integrated analysis of single cell transcriptomic data across condi- tions, technologies, and species. bioRxiv, pp. 164889, 2017.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint
039900 2015.
Rachel Finck, Erin F Simonds, Astraea Jager, Smita Krishnaswamy, Karen Sachs, Wendy Fantl, Dana Pe' er, Garry P Nolan, and Sean C Bendall, Normalization of mass cytometry data with bead standards. Cytometry Part 83(5):883-494, 2013.
Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al.'s regative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp- 5767-5777, 2017.
Laleh Haghverdi, Aaron TL Lun, Michael D Morgan, and John C Marioni. Batch effects in single- cell massequenin data are corrected by matching mutual nearest neighbors. Nature biotechnol- ogy, 36(5):221,, 2018.

---

Under review as a conference paper at ICLR 2020
W Evan Johnson, Cheng Li, and Ariel Rabinovic. Adjusting batch effects in microarray expression data using empirical bayes methods. Biostatistics, 8(1):118-18) 127, 2007.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. arXiv preprint /tt3::/700.5199 1922 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412. 69800, 2014.
Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. Towards understanding the dy-
namics of generative adversarial networks. arXiv preprint arXiv:/706.0955 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15:2018, 2018
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2208 2217. JMLR. org, 2017.
Vaishnavh Nagarajan and Zico Kolter. Gradient descent gan optimization is locally stable. In Advances in Neural Information Processing Systems, pp. 5585-5595, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:151.0664: 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234-2242, 2016.
Swami Sankaranarayanan, Yogesh Balaji, Carlos Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8503-8512, 2018.
Uri Shaham, Kelly P Stanton, Jun Zhao, Huamin Li, Khadir Raddassi, Ruth Montgomery, and Yuval Kluger. Removal of batch effects using distribution-maccinng residual networks. Bioinformatics, 33(16):2539-2546 2017.
Akash Srivastava, Lazar Valkoz, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Infor- mation Processing Systems, pp. 3308-3318, 2017
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, volume 4, pp- 12, 2017.
Rachael Tatman. Gender and dialect bias in youtubes automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pp. 53-59, 2017.
Po- Yuan Tung, John D Blischak, Chiaowen Joyce Hsiao, David A Knowles, Jonathan E Burnett, Jonathan K Pritchard, and Yoav Gilad. Batch effects and the effective design of single-cell gene expression studies. Scientific reports, 7:39921, 2017.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Paul Upchurch, Jacob Gardner. Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7064-7073, 2017.
Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of wasserstein gans: A consistency term and its dual effect. arXiv preprint M.XXv:1803.01555 2018.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint, 2017.
10


Reviews:
[{"review_id": "SylMq1hk9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a method for learning a transformation between two distributions and applying it to an out-of-sample extrapolating distribution. The method relies on an autoencoder, instead of a GAN, to which neuron-editing is applied. A transformation of one of the inner layer is learned using the source and target distributions is estimated and then applied to the extrapolating distribution. This leads to better reconstructions than GAN approaches as judged by some visualizations and results. The method is also applied to biological datasets and some improvement is shown (accuracy increases on specific prediction tasks).\n\nThe idea of applying neuron-editing to an autoencoder is pretty interesting. It's a simple manipulation that makes a lot of sense and judging by the image transformation examples works well. This method also numerically greatly outperforms others on the CelebA extrapolation task so the extrapolation is believable, even though more examples in the appendix would be good. The motivation of applying the method to medical data to correct for instrument variability is also very interesting.\n\nHowever I felt that I could not fully see the benefit of the application for medical data because the area, task, datasets etc are not well introduced. I think the datasets should be explained better, and examples of the images should be given. I understand the gist of Figure 4 but it's not well explained and I do not see why these dimensions were picked. I think there is more work to do there. I think a more careful introduction to the field, with explanations of the data, why deep methods are applicable there, what people have tried etc are necessary.\n\nAlso all of the tables in the paper with classification tasks should have sections in the appendix to explain everything about those tasks. In general, it seems the main weakness of the paper is in exposing the information/writing. \n\nSmaller points:\nIn section 2, source, target and extrapolation distributions are not introduced properly. In equation 1 and the text around it, it's hard to tell that each dimension is edited independently without a couple of reads. \n\" biologicl batch correction\" page 6\nI think the second paragraph in page 4 (\"To apply the learned...\") is missing a sentence in the middle about the actual editing step.\n\nPerhaps more can be said about multiple extrapolation datasets (measurements from different dates instead of only two), if possible/available.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgLTQdTYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of mismatched training/test data by directly modifying the latent representation learned with an auto-encoder. The proposed method employs a piece-wise linear function to transform the representation of samples from a source distribution to that of samples from the corresponding target distribution. The paper argues that the same transformation can be applied to samples from a distribution that are different from the source/target distributions. The method is empirically justified by three different tasks in computer vision and biology.\n\nIn my opinion, the motivation of the paper is clear and the writing is easy to follow, but one potential limitation is the lack of comparison with recent related work, e.g.,\n\nGenerate To Adapt: Aligning Domains using Generative Adversarial Networks\nSwami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, Rama Chellappa\nhttps://arxiv.org/abs/1704.01705\n\nDeep Transfer Learning with Joint Adaptation Networks\nMingsheng Long, Han Zhu, Jianmin Wang, Michael I. Jordan\nhttps://arxiv.org/abs/1605.06636", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkxm9SqlqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: This paper presents a new generative modeling approach to transform between data distributions via a technique the authors dub “neuron editing”. Their approach for “neuron editing” can be used to learn how neurons in a DNN encode particular transformations in the latent space, with the hope of using it to be able to generate data that are out-of-sample and/or do not lie on the same manifold as the data used to train the generative model. Experiments were performed on the CelebA dataset. The authors further demonstrate the usefulness of their proposed idea by using it to remove noise/batch effects from data and also predicting synergy between drugs by using their idea to model the effect of drug treatments.\n\nWhile the idea behind this paper is interesting and their method does seem to provide improvements over other generative models, the paper is a bit difficult to follow and the applications to biology are not explained well enough to understand the implications of their results. The fact that the authors did not compare to StyleGANs also seems a bit suspicious because these models are sota for the types of problems that this paper addresses. Additionally, the authors should consider rewriting the motivation for this problem to be more general. This is a convincing new method to transform distributions in any setting, but the introduction would lead one to believe its applicability is mostly to computational biology. \n\nDetailed comments:\n- The writing and motivation needs to be reworked in order to ensure that the introduction, results, and abstract match in their tone and content. The abstract uses a face recognition/generation application to motivate the work, the introduction focuses solely on computational biology, the results switch between biology and face recognition.\n\n- Section 2: Three desirable properties for a transformation are listed, but what makes these properties desirable?\nEquation (1): Not obvious why the piecewise function definition is necessary, since the second equation seems to hold for j = 0 and j = 99 as well. This should either be corrected or simplified.\n\n- “Mode collapse refers to the discriminator being unable to detect differences in variability between real and fake examples.” Not true. Mode collapse is when a large region of the model’s input space maps onto the small region around a single (often bad) sample.\n\n- Section 3: It would be helpful to see gradual interpolation from the original latent representation of a sample along the direction the neuron edit will be performed, and continued beyond the proposed final latent activations, to demonstrate how this process actually affects the model’s output.\n\n- Doesn’t really seem like the authors tried very hard with the other methods (especially the vanilla GAN) and the omission of StyleGAN, the current state-of-the-art in this kind of transformation learning, is conspicuously omitted. It is highly likely that a well-trained StyleGAN would do better than the other GAN/AE techniques compared against.\n\n- A PCA-based transformation is applied directly to the image data, but it would be interesting (and perhaps more informative) to perform an alignment of the PCs of activations in the latent space.\n\n- Section 4: Again, the motivation distinctly focuses on computational biology when one could easily imagine this approach being applicable to a variety of problems.\n\n- Comparisons with StyleGANs would’ve been appreciated, especially given the fact that they’re now considered state-of-the-art when it comes to modifying the latent space and creating out of sample images as proposed in this work. \n\n- The idea is interesting, though one wonders if there isn’t any other work where the neurons have been “edited” to accommodate different transformations, given that the idea is itself rather intuitive. A more thorough literature review in that regard would be helpful.\n\n- The CelebA experiments help lay establish an intuitive understanding of the proposed technique and were helpful. However, the ideas are a little disconnected with the biological applications of the technique. Better motivation/bridging of the two sets of experiments would be nice.\n\n- A more detailed explanation and analysis of the combinatorial drug application would’ve been helpful to understand the results.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new generative modeling approach to transform between data domains via a neuron editing technique. The authors address the scenario of source to target domain translation that can be applied to a new source domain. While the reviewers acknowledged that the idea of neuron editing is interesting, they have raised several concerns that were viewed by AC as critical issues: (1) given the progress that have been made in the field, an empirical comparison with SOTA GANs models is required to assess the benefits/competitiveness of the proposed approach -- see R1’s comments, also [StarGAN by Choi et al, CVPR 2018], (2) the literature review is incomplete and requires a major revision -- see R1’s and R3’s suggestions, also [CYCADA by Hoffman et al, ICML 2018], (3) presentation clarity -- see R1’s and R2’s comments. AC suggests, in its current state the manuscript is not ready for a publication. We hope the detailed reviews are useful for improving and revising the paper. 
Decision: Reject