Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a method for reducing the representation error generative convolutional neural networks by combining them with untrained deep decoder. The method is evaluated on compressive sensing and super-resolution, where a better performance than the isolated use of Deep Decoders and GAN priors. The main contribution of the paper is not the performance, but the simplicity of this approach.



For the title, I would suggest to replace the word Removing with Reducing.
Furthermore, the clarification of "GAN prior" is very nice in the introduction, maybe you could already clarify it in the abstract.

You should perform a critical grammar check. There are too many commas, for example:
"At sufficiently difficult superresolution problems, the Hybrid model outperforms, the Deep
Decoder, Bicubic upsampling, the BEGAN prior, and the BEGAN as DIP prior." -> there should be no comma after "outperforms"
The sentence from Page 3 to 4 reads strangely, probably a word is missing after "For our GAN prior, we use the BEGAN architecture, and we demonstrate similar results"
"Philosophically, they hybrid" -> "Philosophically, the hybrid"

Fig 6 caption - shouldn't it be 49152 instead of 49512?

You perform various very good analysis experiments, which is well appreciated. Still, it would be good to think about some more experiments (and include at least one of them in the paper):
1. You compare to IGAN and show that you achieve similar performance. You describe that a state-of-the-art approach are invertible generative models and that they are very time consuming (e.g., 15 minutes for a 64x64 image). How good would the invertible models be in terms of performance? Could you perform tests as well?
2. It would be great if you report the runtime of all experiments as well - maybe also the memory usage.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary: This paper proposes to use a combination of a pretrained GAN and an untrained deep decoder as the image prior for image restoration problem. The combined model jointly infers the latent code for the trained GAN and the parameters in the untrained deep decoder. It also jointly infers the mixing coefficient alpha and beta during test time for each image, thus learning how much we should rely on GAN. The proposed hybrid model is helpful on compressed sensing experiments on the CelebA dataset; however, it is only marginally better than deep decoder on image super resolution and out-of-distribution compressed sensing.

Detailed comments:
-	The writing is clear and I was able to understand the model part of the paper. The algorithm box is helpful. However, I would still appreciate if the authors can provide an overall model figure in the model section to help understanding. 
-	Jointly learning the mixing coefficient is an interesting part of the model.
-	The motivation in the abstract and intro could be strengthened. A smaller version of Figure 1 can be probably moved to the beginning of the paper to illustrate the problem of GAN. But even with the help of Figure 1, it is still unclear what is the fundamental problem for GAN. Simply combining a GAN with an untrained decoder model doesn’t help elucidate the source of the problem. 
-	The proposed Hybrid model seems to help on compressed sensing experiments on CelebA. However, it doesn’t help much on out-of-distribution experiments. Moreover, in the super-resolution task, as shown in Figure 5, the improvement over deep decoder is also not significant.
-	The out-of-distribution experiments seems lack of thorough study. In particular, the paper only studies the transfer between CelebA -> Caltech-UCSD Bird dataset. It would be better if the paper can study a variety of other image datasets as well. Also some visualization on the Bird dataset should also be included.
-	Effect of n_pre needs to be further investigated. Why not directly train both models together? It would be good if the authors could comment on how sensitive the n_pre is and what is the intuition.
-	For figures, I would recommend rename “Hybrid” to “Hybrid (Ours)” to highlight the paper’s contribution, and use a brighter color.
-	Figure 5 should be renamed as a Table. 
-	Hyperparameter details should be moved to the Experiment section.

Conclusion:
The paper proposes a simple combination of a trained GAN and an untrained decoder model for the task of image restoration. Although the method is clear and straightforward, in the experiments, the influence of the new model component seems marginal. Moreover, the motivation is not strong enough. Therefore, I recommend weak reject.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: [Update after rebuttal period]
I am sorry that the response cannot address my confusion. I still doubt the motivation of this paper and the actual experimental performance compared with state-of-the-art methods are still ignored. Thus I decrease my score.

[Original reviews]
This paper proposed to modeling image as the combination of a GAN with a Deep Decoder, to remove the representation error of a GAN when used as a prior in inverse problems. The proposed methods are evaluated on two image restoration tasks, including compressive sensing and image super-resolution. The effectiveness of the combination is also presented.

Authors devote themselves to remove the representation error of the GAN image prior. Intuitively, the manner of the proposed linear combination model is rough and less reasonable. In Alg.1, the detailed algorithmic process is presented, it is clear that authors need to pre-train the used GAN and Deep Decoder, then combine them to train one network. If the motivation of this paper is to remove the representation error of GAN, GAN should be viewed as the main body. However, the authors view GAN and Deep Decoder as the same position against the original intention.

Additionally, in the experimental part, the ablation studies indeed reflect the effectiveness of the proposed algorithm, but it looks like the Deep Decoder plays a key role in all cases. In other words, the GAN image prior just plays a supporting role. This is also far away from motivation.

More importantly, I cannot see the surprising results because of this work only compare themselves with some basic version or naïve methods. All state-of-the-art approaches to different tasks are ignored, which is the other big disadvantage. 

In Table 2, what is the meaning of ‘CSGM’？ The authors should describe it.

In a word, from the algorithmic and experimental perspective, this paper cannot achieve satisfying performance.

review_title: Official Blind Review #1

Meta-review: The paper introduces a method for removing what they call representation error and apply the method to super resolution and compressive sensing. 

The reviewers have provided constructive feedback. The reviewers like aspects of the paper but are also concerned with various shortcomings. The consensus is that the paper is not ready for publication as it stands.

Rejection is therefore recommended with strong encouragement to keep working on the method and submit elsewhere.
Decision: Reject