Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Surreal-GAN aims to create fake pathological data with a latent variable and a healthy input and includes an inverse function that predicts the latent variable from a fake/real pathological data. However, unlike its predecessor model Smile-GAN, this adds 5 regularizations (L1 loss between generated output and input, L2 loss between the latent variable and the reverse generated latent variable, decomposing the latent variable, making the latent variables orthogonal to each other, and adding positive correlation to disease pattern and the latent variable) to make the latent variable more interpretable and continuous. This would increase the interpretability since instead of clustering the real data it could be used as understanding how severe the patient’s disease progress is. The results show that the model has a higher c-index than NMF, LDA and FA model predictions. It also shows that the best number of patterns for the model to predict is two, and it indeed has positive correlation to the severity of the disease.
main_review: The model proposed provides an interesting idea of using GAN to create an inverse function to detect the pattern and severity of disease from real data. The regularizations suggest many ways it could be used for other types of data that has a continuous output space. The reasoning behind the regularizations is overall reasonable. There could be more explanation, however, on why double decomposition is a good idea for the model.
Regarding the experiment with simulated data, it is hard to understand why the number of patterns were chosen to be 3, unlike 2 which was suggested to be optimal from the real data. Also, since the regularizations are the major extensions of the model from Smile-GAN, it would be more comprehensible for the reader if the paper had cases where some parameters were set to 0 (where some regularizations were not used). If the regularizations indeed created the effect the authors propose them to, removing them one by one would show the significance of each loss.
The paper also compares between other methods, without a reasonable explanation on why they were selected for comparison. Other references on cases where these approaches were used for the same goal would be helpful.
Some things that could be considered would be using W-GAN’s loss. It has been proposed to solve some issues of GAN by making the model architecture more stable and easier to test (Arjovsky, Chintala, Bottou, 2017), which is useful in generalization of the model or training using different datasets. Additionally, there are words that gradient clipping is a better way than weight clipping to ensure Lipschitz continuity as it still can suffer problems from having a wrong weight clipping window.
One trivial thing is Figure 2.b axis description is cut.
summary_of_the_review: Overall, the paper is convincing and has reasonable suggestions in creating a correlated continuous latent space for pathological data. However, there could be more comparisons important in proving their case. Also, there could be more considerations on new ways the model could be developed, given how GAN has been evolving over the years.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Surreal-GAN method for learning representations of underlying disease-related imaging patterns. This model has overcome limitations in previously published semi-supervised clustering methods and shown great performance on semi-synthetic data sets.
main_review: Strengths:

(1) The proposed method models disease as a continuous process and learning infinite transformation directions from CN to PT.

(2) Several key components(function continuity, transformation sparsity, and inverse consistency) can guide the model to capture meaningful imaging patterns).

(3) The experimental results have shown the effectiveness of the proposed model.



Weaknesses:

(1) The overall objective function includes several hyper-parameters, and the authors set fixed values for some parameters. Thus, how to set them? The effects of different settings should be discussed. Besides, when applying to other datasets, whether we can use the same parameters?

(2) As a significant extension of Smile-GAN, the authors should compare the proposed model with Smile-GAN.

(3) In Table 1, the compared methods are out of date, thus more state-of-the-art methods should be included. 

(4) It is interesting to validate AD diagnosis performance when using the proposed representation learning method.

(5) The authors state that there are four novelties in this study, however, some of them only introduce the existing technologies into the proposed framework.

summary_of_the_review: This paper proposes a Surreal-GAN method for learning representations of underlying disease-related imaging patterns. However, the comparison experiments are not sufficient, and the novelty section needs to be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors pointed out the lack of a good method to predict disease progression in a continuum based on representations explicitly modeling specific disease effects. To tackle this problem, they proposed a novel method called Surreal-GAN, a tailored version of generative adversarial networks, to learn separate representations of neurological and neuropsychiatric diseases in an unsupervised manner. The method frames the diseased brains as multiple disease-related features at various severity imposed on normal brains, and by learning these features and severities, it can generate a representation with improved reliability and explanability.
main_review: Strong Points

1.	Perhaps the biggest shining point is the astonishing number of carefully crafted regularizations – a total of six regularization terms in a single loss function. They are delineated in significant detail from section 2.2.1 to 2.2.5. I appreciate the amount of thinking and hard work that probably goes behind the scene, and I believe the author did a good job persuading me why each of these regularizations are helpful and what they respectively achieve. I particularly liked how the authors pointed out some potential trivial solutions the model can cheat with if proper regularization were not in place.
2.	As for the results, I would consider the voxel-wise statistical comparisons on the real data a rather definitive result demonstrating that Surreal GAN was able to isolate two major sources/locations of atrophy in Alzheimer’s disease. The two distinctive patterns, one showing up across multiple cortical regions and the other localized to the medial temporal lobe would probably be appreciated by many neurobiologists. Meanwhile, the statistical power is quite strong, given that it has been corrected for multiple comparison using the FDR method.


Weak Points

1.	As many as six regularization terms, while I stated as a “strong point”, can also been viewed as a sign of over-engineering. It will be better rationalized if the authors can conduct an ablation study on these five regularizations. With that being said, I acknowledge it might be an unrealistic commitment depending on how long each experiment would take, so this is a soft suggestion rather than an explicit request.
2.	It appears that the authors are utilizing the gradient clipping method to ensure Lipschitz continuity (section 3.2). However, that method is relatively deprecated, as emphasized in the original Wasserstein GAN paper, “weight clipping is a clearly terrible way to enforce a Lipschitz constraint”. While the authors are aware of the case and are conscious about their different use case, nevertheless, alternative ways such as gradient penalty (Improved Training of Wasserstein GANs by Gulrajani et al 2017, aka., WGAN-GP) may be worth considering.
3.	The baseline that the authors compared against are not very competitive. It might be beneficial if a supervised method (is it possible?) can be included, perhaps to set a “practical upper-bound” so that we can see how far the gap there is between that and the proposed method.


Questions to authors

1.	Section 3.3. When the authors mentioned that “the model was trained for at least 100000 epochs and saved until the reconstruction loss is smaller than 0.003 and the monotonicity loss is smaller than 6E-4”, I was surprised by the huge number of epochs. My primary concerns are: 1) does that mean the initial learning rate is a bit too low and/or a bad learning rate scheduling is used? 2) how do you avoid overfitting, besides using the suite of regularizations in the loss function?
2.	Section 2.3. The full loss formula is indexed twice (14 and 15). Is it a mistake?
summary_of_the_review: Overall, I would recommend this paper to be accepted. Personally I find the problem the authors are tackling legitimate and usually neglected, and I am persuaded that the proposed solution has some generalizability.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present a method for learning a representation of disease-related image patterns from regional volume information generated from structural MRI images. The method is based on a GAN approach, and different regularization methods on the latent representation are presented to improve the learning of disease-related representations. There are four contributions presented in the paper:
1. Modelling a disease as a continuous process
2. Controlling the monotonicity of the disease pattern
3. Separation of captured disease pattern
4. Inverse Mapping to ensure that disease patterns synthesized are captured

The evaluation of the method is performed on semi-synthetic data sets and an actual Alzheimer’s disease data set. The results show that the method can identify two clinically informative patterns.

main_review: The paper is well clearly organized, and the general idea of the method is very interesting. However, there are a few things I would like to address.
1. It is not easy to understand the actual model input and how it is generated from the structural MR images. An example image of the preformed atlas segmentation would be beneficial for understanding.
2. Figure 1 is difficult to understand. Is Figure 1b the result of the experiments or an illustration of what to expect from the presented method?
3. In the introduction and later in the paper, the authors refer to the low dimensional representation as R indices r_i and refer to Figure 1b. However, Figure 1b only shows z_1 and z_2. It would be great if the authors could comment on the relation between R indices and the latent representation z.
4. For the model architecture, the latent variable and the input are mapped into a 34-dimensional space. According to the paper, the dot product is applied to both vectors. However, the input to the decoder is 34x1. Is an element-wise multiplication used instead of the dot product?
5. It is not clear how the evaluation of the model f for each z_i is done separately for the pattern decomposition. It would be great if the authors could comment on this.
6. It is not clear what the number of features S relates to in the paper. 
7. For the evaluation, all necessary information is provided. However, the selection method of the hyperparameter for the loss function is not given. Furthermore, it is not evaluated how the selection of the hyperparameter influences the outcome of the model. This is only done for lambda and M.
8. The y axis label of Figure 2b is cut off.
9. The authors mentioned that this method is an extension of the Smile-GAN. Is there a specific reason why the Smile-GAN is not used for comparison?

summary_of_the_review: The presented methods present some interesting ideas for modelling disease-related representation. However, the description of the presented method needs some improvements.
review_title: null

Meta-review: The authors present a GAN for learning a continuous representation of disease-related image patterns from regional volume information generated from structural MRI images.
The reviewers find the problem relevant and appreciate the proposed solution. They find the paper well-written and find the empirical results on Alzheimer brain MRIs relevant for the neuroscience community.
 
The overall objective function includes several hyper-parameters. As pointed out as the main weak point by multiple reviewers this may hint at overengineering/overfitting to a data set. However, the reviewers also mention that the regularizers are all sufficiently well-motivated in the paper and the author response.

Reviewers highlight comparisons on the real data as a strong result demonstrating that Surreal GAN was able to isolate two major sources/locations of atrophy in Alzheimer’s disease. Overall, the reviews are positive in majority.
Decision: Accept (Poster)