Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Could you talk about the relationship between RIMs and the following papers:

1. Chen et al., "On learning optimized reaction diffusion processes for effective image restoration," CVPR 2015.
2. Klatzer et al., "Learning joint demosaicing and denoising based on sequential energy minimization," ICCP 2015.

Both seem to also be about discriminatively learning the free parameters of an iterative inverse-estimation / restoration process, rather than with an explicit generative model. 

Another example that's a little more hand-engineered for blind deconvolution, but still instantiates multiple "iteration layers":

3. Schuler et al., "Learning to Deblur," PAMI 2015.
review_title: Related work

final_score: rating: null
review_content: Interesting work. The idea of unrolling inference into a deep trainable net has been re-discovered and proved efficient and effective.

Could you please also discuss the relationship with two very related papers? It seems to me that MAP inference as a recurrent net and learning the energy models have been covered by the two previous work. Also the applications are quite similar. 

I would like to know what is the difference, especially what is novel here in this paper. 

[1] Belanger, David, and Andrew McCallum. "Structured Prediction Energy Networks." ICML 2016.
[2] Shenlong Wang, Sanja Fidler and Raquel Urtasun, "Proximal Deep Structured Models." NIPS 2016. 
review_title: Discussion with two recent papers

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes the RIMs that unrolls variational inference procedure. 

The author claims that the novelty lies in the separation of the model and inference procedure, making the MAP inference as an end-to-end approach. The effectiveness is shown in image restoration experiments.

While unrolling the inference is not new, the author does raise an interesting perspective towards the `model-free' configuration, where model and inference are not separable and can be learnt jointly. 

However I do not quite agree the authors' argument regarding [1] and [2]. Although both [1] and [2] have pre-defined MAP inference problem. It is not necessarily that a separate step is required. In fact, both do not have either a pre-defined prior model or an explicit prior evaluation step as shown in Fig. 1(a). I believe that the implementation of both follows the same procedure as the proposed, that could be explained through Fig. 1(c). That is to say, the whole inference procedure eventually becomes a learnable neural network and the energy is implicitly defined through learning the parameters.  

Moreover, the RNN block architecture (GRU) and non-linearity (tanh) restrict the flexibility and implicitly form the inherent family of variational energy and inference algorithm. This is also similar with [1] and [2].

Based on that fact, I have the similar feeling with R1 that the novelty is somewhat limited. Also some discussions should be added in terms of the architecture and nonlinearity that you have chosen. 
review_title: This paper is interesting but I remain some concerns regarding the author's response. 

final_score: rating: 7: Good paper, accept
review_content: This paper presents a method to learn both a model and inference procedure at the same time with recurrent neural networks in the context of inverse problems.
The proposed method is interesting and results are quite good. The paper is also nicely presented. 

I would be happy to see some discussion about what the network learns in practice about natural images in the case of denoising. What are the filters like? Is it particularly sensitive to different structures in images? edges? Also, what is the state in the recurrent unit used for? when are the gates open etc.

Nevertheless, I think this is nice work which should be accepted.
review_title: Interesting work

final_score: rating: 4: Ok but not good enough - rejection
review_content: Unfortunately, even after reading the authors' response to my pre-review question, I feel this paper in its current form lacks sufficient novelty to be accepted to ICLR.

Fundamentally, the paper suggests that traditional iterative algorithms for specific class of problems (ill-posed image inverse problems) can be replaced by discriminatively trained recurrent networks. As R3 also notes, un-rolled networks for iterative inference aren't new: they've been used to replace CRF-type inference, and _also_ to solve image inverse problems (my refs [1-3]). Therefore, I'd argue that the fundamental idea proposed by the paper isn't new---it is just that the paper seeks to 'formalize' it as an approach for inverse problems (although, there is nothing specific about the analysis that ties it to inverse problems: the paper only shows that the RIM can express gradient descent over prior + likelihood objective).

I also did not find the claims about benefits over prior approaches very compelling. The comment about parameter sharing works both ways---it is possible that untying the parameters leads to better performance over a fewer number of 'iterations', and given that the 'training set' is synthetically generated, learning a larger number of parameters doesn't seem to be an issue. Also, I'd argue that sharing the parameters is the 'obvious' approach, and the prior methods choose to not tie the parameters to get better accuracy.

The same holds for being able to handle different noise levels / scale sizes. A single model can always be trained to handle multiple forms of degradation---its just that its likely to do better when it's trained for specific degradation model/level. But more importantly, there is no evidence in the current set of experiments that shows that this is a property of the RIM architecture. (Moreover, this claim goes against one of the motivations of the paper of not training a single prior for different observation models ... but to train the entire inference architecture end-to-end).

It is possible that the proposed method does offer practical benefits beyond prior work---but these benefits don't come from the idea of simply unrolling iterations, which is not novel. I would strongly recommend that the authors consider a significant re-write of the paper---with a detailed discussion of prior work mentioned in the comments that highlights, with experiments, the specific aspects of their recurrent architecture that enables better recovery for inverse problems. I would also suggest that to claim the mantle of 'solving inverse problems', the paper consider a broader set of inverse tasks---in-painting, deconvolution, different noise models, and possibly working with multiple observations (like for HDR).
review_title: Official Review

Meta-review: This paper presents an approach for learning both a model and inference procedure at the same time using RNNs. The reviewers agree that the idea is interesting, but discussion and considering the responses to the reviews, still felt that more is needed to motivate and contextualise the work, and to make the methods presented more convincing. For this reason, the paper is unfortunately not yet ready for inclusion in this year's proceedings.
Decision: Reject