Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces a transformation from the deep image prior (DIP) to an embedding with an autoencoder (MMES). The authors aim to use this transformation to explain ("in words") why the DIP works so well and explain why convolutions are needed in the DIP. The contributions are summarised as a) providing an interpretable analogue to the convnet, b) demonstration of the proposed method's effectiveness, and c) characterisation of the DIP as a "low-dimensional patch-manifold prior".

I think the MMES approach is interesting and potentially a good analogue to the DIP, and explicitly draws out the locality prior the authors claim is integral to DIP. The good results and comparison to DIP demonstrates that this locality prior may be important to the task.

I disagree that this method is "interpretable"/"explainable", at least without any evidence toward this presented in the paper. There is still fundamentally a deep network as in DIP. The discussion on interpretability is limited and mostly provided through comparison with other methods.

All up I think this is a useful paper, even though the paper overstates its contributions. I would like to see the clarity improved: it took me a long while to make the connection between the method presented in the paper and the implications for DIP. This connection should have been more explicit in the paper. I would also like to see the "interpretability" statement either clearly explained or removed (I am not convinced that this method is interpretable).
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors present a natural image model based on the manifold of image patches.  It is similar to the Deep Image Prior in that it is untrained and has a convolutional-like structure.  It leads to an optimization problem with a reconstruction loss term and an auto encoding term.  The authors show empirical results in time series recovery, non-semantic inpainting, and super resolution.  In the image processing tasks, the performance of the proposed algorithm is on par (sometimes slightly worse, sometimes slightly better) than that of DIP.

I think the perspective of image patch analysis is a useful addition to the knowledge base for unlearned image priors.  That said, the paper says it tackles the questions for why the DIP "works so well" and why convolution operations are "essential for image reconstruction or enhancement".  After reading the paper, it is unclear to me how this work addresses these questions.  In particular, demonstrating a similar convolutional system does not rule out the possibility that there are non-convolutional systems that also explain the effect.  For example, the Deep Geometric Prior paper is nonconvolutional (it is entirely a MLP), and it also has the effect of fitting a smooth signal without training (subject to early stopping).  The DGP could be applied to images as well, resulting in a nonconvolutional deep image prior.  I think the authors should address more clearly and thoroughly the logical connection between their results and the explanation of the DIP, especially in light of the DGP.

The paper claims that the proposed method is more interpretable, and it would be nice if they could demonstrate this interpretability and the benefits it brings in solving image reconstruction problems.  

As a result, I am inclined to recommend a weak reject, but if the concerns above are addressed, I envision my rating could improve upon rebuttal.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
This paper proposed a low-dimensional patch-manifold prior perspective for reinterpreting the deep image prior. I think this is very interesting work that could have a lot of impact in vision and beyond, since effective construct the problem in reconstruction tasks are highly relevant to a number of tasks. I was initially quite excited about this paper, but as I drilled into the details of this work. 

For the manifold modeling, though the authors defined each part of the formulation in equations (1)-(5), but it is not clear how to design corresponding efficient structures for different low-level problems. In other words, the application ability is not clear. For example, if the proposed MMES scheme is used in image deconvolution tasks, how to design the corresponding structure?

For the parameters, there is no description of the parameters set in subsection 4.2 and 4.3. And what are the criteria for selecting these parameters?

For the experimental part, the comparison experiments in subsection 4.1(Toy examples) and subsection 4.3 (Color image superresolution) lack comparisons with the latest methods. I think, more comparison experiments should be provided. 


[Update after rebuttal period]
The revision and responses have addressed most of my concerns, so I keep my original score.



review_title: Official Blind Review #2

Meta-review: The paper proposes a combination of a delay embedding as well as an autoencoder to perform representation learning. The proposed algorithm shows competitive performance with deep image prior, which is a convnet structure. The paper claims that the new approach is interpretable and provides explainable insight into image priors.

The discussion period was used constructively, with the authors addressing reviewer comments, and the reviewers acknowledging this an updating their scores.

Overall, the proposed architecture is good, but the structure and presentation of the paper is still not up to the standards of ICLR. The current presentation seems to over-claim interpretability, without sufficient theoretical or empirical evidence.

Decision: Reject