Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Overview: This paper presents a new generative modeling approach to transform between data distributions via a technique the authors dub “neuron editing”. Their approach for “neuron editing” can be used to learn how neurons in a DNN encode particular transformations in the latent space, with the hope of using it to be able to generate data that are out-of-sample and/or do not lie on the same manifold as the data used to train the generative model. Experiments were performed on the CelebA dataset. The authors further demonstrate the usefulness of their proposed idea by using it to remove noise/batch effects from data and also predicting synergy between drugs by using their idea to model the effect of drug treatments.

While the idea behind this paper is interesting and their method does seem to provide improvements over other generative models, the paper is a bit difficult to follow and the applications to biology are not explained well enough to understand the implications of their results. The fact that the authors did not compare to StyleGANs also seems a bit suspicious because these models are sota for the types of problems that this paper addresses. Additionally, the authors should consider rewriting the motivation for this problem to be more general. This is a convincing new method to transform distributions in any setting, but the introduction would lead one to believe its applicability is mostly to computational biology. 

Detailed comments:
- The writing and motivation needs to be reworked in order to ensure that the introduction, results, and abstract match in their tone and content. The abstract uses a face recognition/generation application to motivate the work, the introduction focuses solely on computational biology, the results switch between biology and face recognition.

- Section 2: Three desirable properties for a transformation are listed, but what makes these properties desirable?
Equation (1): Not obvious why the piecewise function definition is necessary, since the second equation seems to hold for j = 0 and j = 99 as well. This should either be corrected or simplified.

- “Mode collapse refers to the discriminator being unable to detect differences in variability between real and fake examples.” Not true. Mode collapse is when a large region of the model’s input space maps onto the small region around a single (often bad) sample.

- Section 3: It would be helpful to see gradual interpolation from the original latent representation of a sample along the direction the neuron edit will be performed, and continued beyond the proposed final latent activations, to demonstrate how this process actually affects the model’s output.

- Doesn’t really seem like the authors tried very hard with the other methods (especially the vanilla GAN) and the omission of StyleGAN, the current state-of-the-art in this kind of transformation learning, is conspicuously omitted. It is highly likely that a well-trained StyleGAN would do better than the other GAN/AE techniques compared against.

- A PCA-based transformation is applied directly to the image data, but it would be interesting (and perhaps more informative) to perform an alignment of the PCs of activations in the latent space.

- Section 4: Again, the motivation distinctly focuses on computational biology when one could easily imagine this approach being applicable to a variety of problems.

- Comparisons with StyleGANs would’ve been appreciated, especially given the fact that they’re now considered state-of-the-art when it comes to modifying the latent space and creating out of sample images as proposed in this work. 

- The idea is interesting, though one wonders if there isn’t any other work where the neurons have been “edited” to accommodate different transformations, given that the idea is itself rather intuitive. A more thorough literature review in that regard would be helpful.

- The CelebA experiments help lay establish an intuitive understanding of the proposed technique and were helpful. However, the ideas are a little disconnected with the biological applications of the technique. Better motivation/bridging of the two sets of experiments would be nice.

- A more detailed explanation and analysis of the combinatorial drug application would’ve been helpful to understand the results.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors present a method for learning a transformation between two distributions and applying it to an out-of-sample extrapolating distribution. The method relies on an autoencoder, instead of a GAN, to which neuron-editing is applied. A transformation of one of the inner layer is learned using the source and target distributions is estimated and then applied to the extrapolating distribution. This leads to better reconstructions than GAN approaches as judged by some visualizations and results. The method is also applied to biological datasets and some improvement is shown (accuracy increases on specific prediction tasks).

The idea of applying neuron-editing to an autoencoder is pretty interesting. It's a simple manipulation that makes a lot of sense and judging by the image transformation examples works well. This method also numerically greatly outperforms others on the CelebA extrapolation task so the extrapolation is believable, even though more examples in the appendix would be good. The motivation of applying the method to medical data to correct for instrument variability is also very interesting.

However I felt that I could not fully see the benefit of the application for medical data because the area, task, datasets etc are not well introduced. I think the datasets should be explained better, and examples of the images should be given. I understand the gist of Figure 4 but it's not well explained and I do not see why these dimensions were picked. I think there is more work to do there. I think a more careful introduction to the field, with explanations of the data, why deep methods are applicable there, what people have tried etc are necessary.

Also all of the tables in the paper with classification tasks should have sections in the appendix to explain everything about those tasks. In general, it seems the main weakness of the paper is in exposing the information/writing. 

Smaller points:
In section 2, source, target and extrapolation distributions are not introduced properly. In equation 1 and the text around it, it's hard to tell that each dimension is edited independently without a couple of reads. 
" biologicl batch correction" page 6
I think the second paragraph in page 4 ("To apply the learned...") is missing a sentence in the middle about the actual editing step.

Perhaps more can be said about multiple extrapolation datasets (measurements from different dates instead of only two), if possible/available.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper tackles the problem of mismatched training/test data by directly modifying the latent representation learned with an auto-encoder. The proposed method employs a piece-wise linear function to transform the representation of samples from a source distribution to that of samples from the corresponding target distribution. The paper argues that the same transformation can be applied to samples from a distribution that are different from the source/target distributions. The method is empirically justified by three different tasks in computer vision and biology.

In my opinion, the motivation of the paper is clear and the writing is easy to follow, but one potential limitation is the lack of comparison with recent related work, e.g.,

Generate To Adapt: Aligning Domains using Generative Adversarial Networks
Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, Rama Chellappa
https://arxiv.org/abs/1704.01705

Deep Transfer Learning with Joint Adaptation Networks
Mingsheng Long, Han Zhu, Jianmin Wang, Michael I. Jordan
https://arxiv.org/abs/1605.06636
review_title: Official Blind Review #3

Meta-review: This paper presents a new generative modeling approach to transform between data domains via a neuron editing technique. The authors address the scenario of source to target domain translation that can be applied to a new source domain. While the reviewers acknowledged that the idea of neuron editing is interesting, they have raised several concerns that were viewed by AC as critical issues: (1) given the progress that have been made in the field, an empirical comparison with SOTA GANs models is required to assess the benefits/competitiveness of the proposed approach -- see R1’s comments, also [StarGAN by Choi et al, CVPR 2018], (2) the literature review is incomplete and requires a major revision -- see R1’s and R3’s suggestions, also [CYCADA by Hoffman et al, ICML 2018], (3) presentation clarity -- see R1’s and R2’s comments. AC suggests, in its current state the manuscript is not ready for a publication. We hope the detailed reviews are useful for improving and revising the paper. 
Decision: Reject