user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Out-of-Sample Extrapolation with Neuron Editing

Paper content:
While neural networks can be trained to map from one specific dataset to another, they usually do not learn a generalized transformation that can extrapolate accurately outside the space of training.
For instance, a generative adversarial network (GAN) exclusively trained to transform images of cars from light to dark might not have the same effect on images of horses.
This is because neural networks are good at generation within the manifold of the data that they are trained on.
However, generating new samples outside of the manifold or extrapolating "out-of-sample" is a much harder problem that has been less well studied.
To address this, we introduce a technique called neuron editing that learns how neurons encode an edit for a particular transformation in a latent space.
We use an autoencoder to decompose the variation within the dataset into activations of different neurons and generate transformed data by defining an editing transformation on those neurons.
By performing the transformation in a latent trained space, we encode fairly complex and non-linear transformations to the data with much simpler distribution shifts to the neuron's activations.
We showcase our technique on image domain/style transfer and two biological applications: removal of batch artifacts representing unwanted noise and modeling the effect of drug treatments to predict synergy between drugs.
Many experiments in biology are conducted to study the effect of a treatment or a condition on a set of samples.
For example, the samples can be groups of cells and the treatment can be the administration of a drug.
However, experiments and clinical trials are often performed on only a small subset of samples from the entire population.
Usually, it is assumed that the effects generalize in a context-independent manner without mathematically attempting to model the effect and potential interactions with the context.
However, mathematically modeling the effect and potential interactions with background information would give us a powerful tool that would allow us to assess how the treatment would generalize beyond the samples measured.We propose a neural network-based method for learning a general edit function corresponding to treatment in the biological setting.
While neural networks offer the power and flexibility to learn complicated ways of transforming data from one distribution to another, they are often overfit to the training dataset in the sense that they only learn how to map one specific data manifold to another, and not a general edit function.
Indeed, popular neural network architectures like GANs pose the problem as one of learning to generate the post-treatment data distributions from pre-treatment data distributions.
Instead, we reframe the problem as that of learning an edit function between the preand post-treatment versions of the data, that could be applied to other datasets.We propose to learn such an edit, which we term neuron editing, in the latent space of an autoencoder neural network with non-linear activations.
First we train an autoencoder on the entire population of data which we are interested in transforming.
This includes all of the pre-treatment samples and the post-treatment samples from the subset of the data on which we have post-treatment measurements.The internal layers of this autoencoder represent the data with all existing variation decomposed into abstract features (neurons) that allow the network to reconstruct the data accurately BID28 BID4 BID17 BID24 .
Neuron editing involves extracting differences between the observed pre-and post-treatment activation distributions for neurons in this layer and then applying them to pre-treatment data from the rest of the population to synthetically generate post-treatment data.
Thus performing the edit node-by-node in this space actually encodes complex multivariate edits in the ambient space, performed on denoised and meaningful features, owing to the fact that these features themselves are complex non-linear combinations of the input features.While neuron editing is a general technique that could be applied to the latent space of any neural network, even GANs themselves, we instead focus exclusively on the autoencoder in this work to leverage three of its key advantages.
First, we seek to model complex distribution-to-distribution transformations between large samples in high-dimensional space.
While this can be generally intractable due to difficulty in estimating joint probability distributions, research has provided evidence that working in a lower-dimensional manifold facilitates learning transformations that would otherwise be infeasible in the original ambient space BID32 BID21 BID29 .
The non-linear dimensionality reduction performed by autoencoders finds intrinsic data dimensions that esentially straighten the curvature of data in the ambient space.
Thus complex effects can become simpler shifts in distribution that can be computationally efficient to apply.Second, by performing the edit to the neural network internal layer, we allow for the modeling of some context dependence.
Some neurons of the internal layer have a drastic change between preand post-treatment versions of the experimental subpopulation, while other neurons such as those that encode background context information not directly associated with treatment have less change in the embedding layer.
The latter neurons are less heavily edited but still influence the output jointly with edited neurons due to their integration in the decoding layers.
These edited neurons interact with the data-context-encoding neurons in complex ways that may be more predictive of treatment than the experimental norm of simply assuming widespread generalization of results context-free.Third, editing in a low-dimensional internal layer allows us to edit on a denoised version of the data.
Because of the reconstruction penalty, more significant dimensions are retained through the bottleneck dimensions of an autoencoder while noise dimensions are discarded.
Thus, by editing in the hidden layer, we avoid editing noise and instead edit significant dimensions of the data.We note that neuron editing makes the assumption that the internal neurons have semantic consistency across the data, i.e., the same neurons encode the same types of features for every data manifold.
We demonstrate that this holds in our setting because the autoencoder learns a joint manifold of all of the given data including pre-and post-treatment samples of the experimental subpopulation and pre-treatment samples from the rest of the population.
Recent results show that neural networks prefer to learn patterns over memorizing inputs even when they have the capacity to do so BID31 .We
demonstrate that neuron editing extrapolates better than generative models on two important criteria. First
, as to the original goal, the predicted change on extrapolated data more closely resembles the predicted change on interpolated data. Second
, the editing process produces more complex variation, since it simply preserves the existing variation in the data rather than needing a generator to learn to create it. We compare
the predictions from neuron editing to those of several generationbased approaches: a traditional GAN, a GAN implemented with residual blocks (ResnetGAN) to show generating residuals is not the same as editing BID26 , and a CycleGAN BID33 . While in other
applications, like natural images, GANs have shown an impressive ability to generate plausible individual points, we illustrate that they struggle with these two criteria. We also motivate
why neuron editing is performed on inference by comparing against a regularized autoencoder that performs the internal layer transformations during training, but the decoder learns to undo the transformation and reconstruct the input unchanged BID0 .In the following
section, we detail the neuron editing method. Then, we motivate
the extrapolation problem by trying to perform natural image domain transfer on the canonical CelebA dataset . We then move to two
biological applications where extrapolation is essential: correcting the artificial variability introduced by measuring instruments (batch effects), and predicting the combined effects of multiple drug treatments (combinatorial drug effects) BID1 .
In this paper, we tackled a data-transformation problem inspired by biological experimental settings: that of generating transformed versions of data based on observed pre-and post-transformation versions of a small subset of the available data.
This problem arises during clinical trials or in settings where effects of drug treatment (or other experimental conditions) are only measured in a subset of the population, but expected to generalize beyond that subset.
Here we introduce a novel approach that we call neuron editing, for applying the treatment effect to the remainder of the dataset.
Neuron editing makes use of the encoding learned by the latent layers of an autoencoder and extracts the changes in activation distribution between the observed pre-and post treatment measurements.
Then, it applies these same edits to the internal layer encodings of other data to mimic the transformation.
We show that performing the edit on neurons of an internal layer results in more realistic transformations of image data, and successfully predicts synergistic effects of drug treatments in biological data.
Moreover, we note that it is feasible to learn complex data transformations in the non-linear dimensionality reduced space of a hidden layer rather than in ambient space where joint probability distributions are difficult to extract.
Finally, learning edits in a hidden layer allows for interactions between the edit and other context information from the dataset during decoding.
Future work along these lines could include training parallel encoders with the same decoder, or training to generate conditionally.

assistant: text: We reframe the generation problem as one of editing existing points, and as a result extrapolate better than traditional GANs.