user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Beyond GANs: Transforming without a Target Distribution

Paper content:
While generative neural networks can learn to transform a specific input dataset into a specific target dataset, they require having just such a paired set of input/output datasets.
For instance, to fool the discriminator, a generative adversarial network (GAN) exclusively trained to transform images of black-haired *men* to blond-haired *men* would need to change gender-related characteristics as well as hair color when given images of black-haired *women* as input.
This is problematic, as often it is possible to obtain *a* pair of (source, target) distributions but then have a second source distribution where the target distribution is unknown.
The computational challenge is that generative models are good at generation within the manifold of the data that they are trained on.
However, generating new samples outside of the manifold or extrapolating "out-of-sample" is a much harder problem that has been less well studied.
To address this, we introduce a technique called *neuron editing* that learns how neurons encode an edit for a particular transformation in a latent space.
We use an autoencoder to decompose the variation within the dataset into activations of different neurons and generate transformed data by defining an editing transformation on those neurons.
By performing the transformation in a latent trained space, we encode fairly complex and non-linear transformations to the data with much simpler distribution shifts to the neuron's activations.
Our technique is general and works on a wide variety of data domains and applications.
We first demonstrate it on image transformations and then move to our two main biological applications: removal of batch artifacts representing unwanted noise and modeling the effect of drug treatments to predict synergy between drugs.
A common situation arises when we have two datasets and seek to learn a transformation that is a mapping from one (the source) to the other (the target).
While much existing work has been done on this, less studied is the case where we want to learn a transformation from this source/target pair of datasets and apply it to a second source dataset for which there is no known target.
If the second source distribution differs from the first source distribution, any transformation naively learned with a neural network on only the first source/target pair will suffer from problems including domain shift (the second source distribution systematically differs from the first) and overfitting (which aspects of the target only exist because it started as the first source distribution, and shouldn't be part of the learned transformation?).
This problem is important to address, as learning transformations is a common task in many different contexts, and often it is infeasible or impossible to obtain known source/target pairing information for every distribution needing to be transformed.
For example, many experiments in biology are conducted to study the effect of a treatment on a set of samples, such as tissues from different patients.
However, due to their expense and difficulty, clinical trials are often performed on only a small subset of the samples.
The challenge is isolating treatment-induced variation from the confounding sample-specific variation.
We propose a neural network-based method for learning a transformation that is general enough to be used across a wide range of data modalities and applications, from image-to-image translation to treatment in the biological setting.
Popular neural network architectures like GANs pose the problem as one of learning to output data in the region of the space occupied by the target distribution, no matter where the input data is coming from.
To fool the discriminator, the generator's output must end up in the same part of the space as the target distribution.
The discriminator does not take into account the input points into the generator in any way.
Instead, we reframe the problem as learning a transformation towards the target distribution that is more sensitive to where the input data starts.
Thus, we could learn an edit between one source and target pair, and apply it to a second source without needing to assume it has no systematic differences from the first source.
We propose to learn such an edit, which we term neuron editing, in the latent space of an autoencoder neural network with non-linear activations.
First we train an autoencoder on the entire population of data which we are interested in transforming.
This includes both the paired source/target data and the second source data.
Neuron editing then involves extracting observed differences between the source/target activation distributions for neurons in this layer and then applying them to the second source data to generate a synthetic second target dataset.
Performing the edit node-by-node in this space actually encodes complex multivariate edits in the ambient space, performed on denoised and meaningful features, owing to the fact that these features themselves are complex non-linear combinations of the input features.
Neuron editing is a general technique that could be applied to the latent space of any neural network, even GANs themselves.
We focus exclusively on the autoencoder in this work, however, to leverage its denoising ability, robustness to mode dropping, and superior training stability as compared to GANs.
We demonstrate that neuron editing can work on a variety of architectures, while offering the advantages of introducing no new hyperparameters to tune and being stable across multiple runs.
While latent space manipulation has been explored in previous work, ours differs in several ways.
For example, Radford et al. (2015) represents a transformation between two distributions as a single constant shift in latent space.
In addition to assuming the latent transformation is the same for all points in the distribution, Upchurch et al. (2017) also uses an off-the-shelf pre-trained Imagenet classifier network.
Our work, on the other hand, does not require a richly supervised pre-trained model; also, we model the shift between two distributions as a complex, non-constant function that learns different shifts for different parts of the space.
We compare to this "constant-shift" approach and demonstrate empirically why it is necessary to model the transformation more complexly.
Some neurons are not heavily edited but still influence the output jointly with those neurons that are edited due to their integration in the decoding layers, propagating their effect into the output space.
Thus, even a relatively simple transformation in the internal layer of an autoencoder allows for modeling complex transformations in the ambient data space.
This aspect of neuron editing draws close connections with the field of domain adaptation, where the goal is to learn features on one labeled dataset that meaningfully separate points in another, unlabeled dataset (Tzeng et al., 2017) .
Similarly to that task, we want to learn a transformation from the known source to the known target samples that will also apply to the second source dataset where the target is unknown.
Thus, neuron editing represents an extension of domain adaptation, where instead of learning a classifier that can be used on the unlabeled data, we are learning a distribution transformation that can be used on the unlabeled data.
Further differences include that while domain adaptation attempts to make features for the unlabeled dataset overlap with those of the labeled dataset, neuron editing transforms the second source dataset without first trying to align it to the first source dataset (Sankaranarayanan et al., 2018) .
Also, different from many domain adaptation techniques, we do not need any sort of pre-trained classifier to yield an informative feature map for the data, as we learn our autoencoder de novo (Long et al., 2017) .
Given the near exclusive focus of the domain adaptation community on learning classifiers on labeled data and applying it to unlabeled data, we are excited to expand the field to also learning transformations on data with known targets and applying it to data with unknown targets.
We demonstrate that neuron editing extrapolates better than generative models on two important criteria.
First, as to the original goal, the predicted change on the second source dataset more closely resembles the predicted change on the original source dataset.
Second, the editing process produces more complex variation, since it simply preserves the existing variation in the data rather than needing a generator to learn to create it.
We compare to standard GAN approaches, dedicated parametric statistical methods used by computational biologists, and alternative autoencoder frameworks.
In Figure 1 :
(a) Neuron editing interrupts the standard feedforward process, editing the neurons of a trained encoder/decoder to include the source-to-target variation, and letting the trained decoder cascade the resulting transformation back into the original data space.
(b) The neuron editing process.
The transformation is learned on the distribution of neuron activations for the source and applied to the distribution of neuron activations for the extrapolation data.
each case, we see that they stumble on one or more of several hurdles: out-of-sample input, desired output that differs from the target of the training data, and data with complex variation.
In the following section, we detail the neuron editing method.
Then, we motivate the extrapolation problem by trying to perform natural image domain transfer on the canonical CelebA dataset .
We then move to two biological applications where extrapolation is essential: correcting the artificial variability introduced by measuring instruments (batch effects), and predicting the combined effects of multiple drug treatments (combinatorial drug effects) (Anchang et al., 2018) .
, and second source distributions with n S , n T , and n X observations, respectively.
We seek a transformation such that:
1. when applied to S it produces a distribution equivalent to
T 2. when applied to T it is the identity function and
3. when applied to X it does not necessarily produce T if S is different from X. While GANs learn a transformation with the first two properties, they fail at the third property due to the fact that T is the only target data we have for training, and thus the generator only learns to output data like T .
Therefore, instead of learning such a transformation parameterized by a neural network, we learn a simpler transformation on a space learned by a neural network (summarized in Figure 1 ).
We first train an encoder/decoder pair E/D to map the data into an abstract neuron space decomposed into high-level features such that it can also decode from that space, i.e., the standard autoencoder objective L:
where MSE is the mean-squared error.
The autoencoder is trained on all three data distributions S, T , and X and thus learns to model their joint manifold.
Then, without further training, we separately extract the activations of an n-dimensional internal layer of the network for inputs from S and from T , denoted by a S : S → R n , a T : T → R n .
We define a piecewise linear transformation, called N euronEdit, which we apply to these distributions of activations:
where a ∈ R n consists of n activations for a single network input, p S j , p T j ∈ R n consist of the j th percentiles of activations (i.e., for each of the n neurons) over the distributions of a S , a T correspondingly, and all operations are taken pointwise, i.e., independently on each of the n neurons in the layer.
Then, we define N euronEdit(a S ) : S → R n given by x → N euronEdit(a S (x)), and equivalently for a T and any other distribution (or collection) of activations over a set of network inputs.
Therefore, the N euronEdit function operates on distributions, represented via activations over network input samples, and transforms the input activation distribution based on the difference between the source and target distributions (considered via their percentile disctretization).
We note that the N euronEdit function has the three properties we stated above:
This last property is crucial since learning to generate distributions like T , with a GAN for example, would produce a discriminator who encourages the output to be funneled as close to T as posssible no matter where in the support we start from.
To apply the learned transformation to X, we first extract the activations of the internal layer computed by the encoder, a X .
Then, we edit the activations with the neuron editing functionâ X .
Finally, we cascade the transformations applied to the neuron activations through the decoder without any further training.
Thus, the transformed outputX is obtained by:
We emphasize that at this point, since we do no further training of the encoder and decoder, and since the neuron editing transformation has no weights to learn, there is no further objective term to minimize at this point and the transformation is fully defined.
Crucially, the nomenclature of an autoencoder no longer strictly applies.
If we allowed the encoder or decoder to train with the transformed neuron activations, the network could learn to undo these transformations and still produce the identity function.
However, since we freeze training and apply these transformations exclusively on inference, we turn an autoencoder into a generative model that need not be close to the identity.
Training a GAN in this setting could exclusively utilize the data in S and T , since we have no real examples of the output for X to feed to the discriminator.
Neuron editing, on the other hand, is able to model the variation intrinsic to X in an unsupervised manner despite not having real posttransformation data for X. Since we know a priori that X will differ substantially from S, this provides significantly more information.
Furthermore, GANs are notoriously tricky to train (Salimans et al., 2016; Gulrajani et al., 2017; Wei et al., 2018) .
Adversarial discriminators suffer from oscillating optimization dynamics , uninterpretable losses (Barratt & Sharma, 2018; , and most debilitatingly, mode collapse (Srivastava et al., 2017; Kim et al., 2017; Nagarajan & Kolter, 2017) .
Under mode collapse, significant diversity that should exist in the output of the generator is lost, instead producing synthetic data that is a severely degenerated version of the true target distribution.
Neuron editing avoids all of these traps by learning an unsupervised model of the data space with the easier-to-train autoencoder.
The essential step that facilitates generation is the isolation of the variation in the neuron activations that characterizes the difference between source and target distributions.
There is a relationship between neuron editing and the well-known word2vec embeddings in natural language processing (Goldberg & Levy, 2014) .
There, words are embedded in a latent space where a meaningful transformation such as changing the gender of a word is a constant vector in this space.
This vector can be learned on one example, like transforming man to woman, and then extrapolated to another example, like king, to predict the location in the space of queen.
Neuron editing is an extension in complexity of word2vec's vector arithmetic, because instead of transforming a single point into another single point, it transforms an entire distribution into another distribution.
In this work, we have only consider learning from a single pair of distributions and applying it to another single distribution.
We consider it an interesting direction for future work to extend this to multiple distributions, either for learning from and application to.
Additional future work along these lines could include training parallel encoders with the same decoder, or training to generate conditionally.

assistant: text: A method for learning a transformation between one pair of source/target datasets and applying it a separate source dataset for which there is no target dataset