Title: Recurrent Inference Machines for Solving Inverse Problems
Authors: Patrick Putzky
Max Welling
Venue: ICLR 2017
Keywords: Optimization
Deep learning
Computer vision

Abstract: Inverse problems are typically solved by first defining a model and then choosing an inference procedure. With this separation of modeling from inference, inverse problems can be framed in a modular way. For example, variational inference can be applied to a broad class of models. The modularity, however, typically goes away after model parameters have been trained under a chosen inference procedure. During training, model and inference often interact in a way that the model parameters will ultimately be adapted to the chosen inference procedure, posing the two components inseparable after training. But if model and inference become inseperable after training, why separate them in the first place?

We propose a novel learning framework which abandons the dichotomy between model and inference. Instead, we introduce Recurrent Inference Machines (RIM), a class of recurrent neural networks (RNN) that directly learn to solve inverse problems.

We demonstrate the effectiveness of RIMs in experiments on various image reconstruction tasks. We show empirically that RIMs exhibit the desirable convergence behavior of classical inference procedures, and that they can outperform state-of- the-art methods when trained on specialized inference tasks.

Our approach bridges the gap between inverse problems and deep learning, providing a framework for fast progression in the field of inverse problems.


Review discussion:
Interesting work. The idea of unrolling inference into a deep trainable net has been re-discovered and proved efficient and effective.
Could you please also discuss the relationship with two very related papers? It seems to me that MAP inference as a recurrent net and learning the energy models have been covered by the two previous work.
Also the applications are quite similar.
I would like to know what is the difference, especially what is novel here in this paper.
[1] Belanger, David, and Andrew McCallum. "Structured Prediction Energy Networks." ICML 2016.
[2] Shenlong Wang, Sanja Fidler and Raquel Urtasun, "Proximal Deep Structured Models." NIPS 2016.
Our approach can be understood as a generalisation as well as a simplification of the approaches in [1] and [2]. It is a generalization because our model formulation allows training of models which have the same properties as in [1] or [2].
Further, it is a simplification for practitioners because it will allow them to iterate through different model architectures (i.e. different neural networks) quickly and to choose the best performing one for their problem.
The simplest way to see the novelty of our viewpoint can be seen in figure 1.
Whereas [1] and [2] operate in the regime of figure 1A, i.e. both approaches seperate prior and inference procedure, we acknowledge (see figure 1B) that both prior and inference are part of an internal model which can be learned jointly.
[2] also learns prior and inference jointly.
However, the approach is restricted to a family of hand-crafted inference procedures, much in the same way as data features used to be hand-crafted. With our viewpoint there is no need for a practitioner to explicitly choose an inference procedure or to explicitly choose a prior.
Instead, it all boils down to training an RNN. The RNN in turn could possibly learn an inference procedure which is much more efficient than anything that could be done through hand-crafted inference procedures.
We believe that many deep learning driven advances in computer vision over the past few years were possible because practitioners could iterate through and try new model architectures quickly. We hope that through our viewpoint the same can happen in inverse problems.
We thank the reviewer for pointing us to these papers. We will make sure to better point out the differences of our viewpoint to previous methods in the introduction and the related works section.
Could you talk about the relationship between RIMs and the following papers:
1. Chen et al., "On learning optimized reaction diffusion processes for effective image restoration," CVPR 2015.
2. Klatzer et al., "Learning joint demosaicing and denoising based on sequential energy minimization," ICCP 2015.
Both seem to also be about discriminatively learning the free parameters of an iterative inverse-estimation / restoration process, rather than with an explicit generative model.
Another example that's a little more hand-engineered for blind deconvolution, but still instantiates multiple "iteration layers":
3. Schuler et al., "Learning to Deblur," PAMI 2015.
Much of the same arguments as in our answer to AnonReviewer3 also hold for these papers. The approaches presented in 1. and 2. represent a particular choice of architecture of a RIM without a state variable. Please note a couple of further advantages in the formulation of RIM over the above approaches:
- Constraining the measure of interest (x in our paper) to be in a certain range of values becomes trivial in RIMs (see 2.3).
- Adding other covariates as additional input to the inference procedure also becomes trivial. The additional covariates can simply be an additional (fixed) input to the RIM. This could allow us to use information which was unusable with previous approaches.
- In our paper we show that RIMs can be trained to perform multiple tasks (different corruption processes, different noise levels, different scales of super-resolution). This is often not the case in other discriminatively trained models. Typically, a new model has to be trained for every noise level for example.
- The lack of parameter sharing in 1. and 2., as well as in [1] and [2] which were mentioned by AnonReviewer3, can be a limiting factor in practice. The assumption in these cases is that the number of necessary iterations is ultimately independent of the observation or the measure of interest.
In practice, however, we will often observe that this is not the case.
For example, uniform images (such as images of the sky) can often be denoised much faster than non-uniform images (such as images of grass or rocks). In our paper we present a RIM with shared parameters which can be a building block for adaptive computation time approaches (see Graves (2016)).
As mentioned in our response to AnonReviewer3 we will make sure to clarify novelty of our viewpoint in contrast to the mentioned approaches.
1. Graves A. Adaptive Computation Time for Recurrent Neural Networks. 2016. http://arxiv.org/abs/1603.08983.
This paper presents a method to learn both a model and inference procedure at the same time with recurrent neural networks in the context of inverse problems.
The proposed method is interesting and results are quite good. The paper is also nicely presented.
I would be happy to see some discussion about what the network learns in practice about natural images in the case of denoising. What are the filters like? Is it particularly sensitive to different structures in images? edges?
Also, what is the state in the recurrent unit used for?
when are the gates open etc.
Nevertheless, I think this is nice work which should be accepted.
We thank the reviewer for the positive review. The reviewer raises an interesting perspective on further analysis of the chosen model architecture. We have not done this type of analysis so far. We believe, however, that this approach would allow us to make better informed architectural choices in the future. Here, we wanted to put less focus on the chosen architecture from an implementation level but more of a focus on the top-down viewpoint. In applications, we see this type of analysis as part of an iterative workflow for finding a good model architecture.
Unfortunately, even after reading the authors' response to my pre-review question, I feel this paper in its current form lacks sufficient novelty to be accepted to ICLR.
Fundamentally, the paper suggests that traditional iterative algorithms for specific class of problems (ill-posed image inverse problems) can be replaced by discriminatively trained recurrent networks. As R3 also notes, un-rolled networks for iterative inference aren't new: they've been used to replace CRF-type inference, and _also_ to solve image inverse problems (my refs [1-3]).
Therefore, I'd argue that the fundamental idea proposed by the paper isn't new---it is just that the paper seeks to 'formalize' it as an approach for inverse problems (although, there is nothing specific about the analysis that ties it to inverse problems: the paper only shows that the RIM can express gradient descent over prior + likelihood objective).
I also did not find the claims about benefits over prior approaches very compelling. The comment about parameter sharing works both ways---it is possible that untying the parameters leads to better performance over a fewer number of 'iterations', and given that the 'training set' is synthetically generated, learning a larger number of parameters doesn't seem to be an issue.
Also, I'd argue that sharing the parameters is the 'obvious' approach, and the prior methods choose to not tie the parameters to get better accuracy.
The same holds for being able to handle different noise levels / scale sizes. A single model can always be trained to handle multiple forms of degradation---its just that its likely to do better when it's trained for specific degradation model/level.
But more importantly, there is no evidence in the current set of experiments that shows that this is a property of the RIM architecture. (Moreover, this claim goes against one of the motivations of the paper of not training a single prior for different observation models ...
but to train the entire inference architecture end-to-end).
It is possible that the proposed method does offer practical benefits beyond prior work---but these benefits don't come from the idea of simply unrolling iterations, which is not novel. I would strongly recommend that the authors consider a significant re-write of the paper---with a detailed discussion of prior work mentioned in the comments that highlights, with experiments, the specific aspects of their recurrent architecture that enables better recovery for inverse problems. I would also suggest that to claim the mantle of 'solving inverse problems', the paper consider a broader set of inverse tasks---in-painting, deconvolution, different noise models, and possibly working with multiple observations (like for HDR).
The reviewer makes some claims about our paper which to us appear not to be a reflection of the actual paper but a result of the communication between Reviewer 1 and Reviewer 3. We therefore chose to answer each paragraph separately:
R1: “Fundamentally, the paper suggests that traditional iterative algorithms for specific class of problems (ill-posed image inverse problems) can be replaced by discriminatively trained recurrent networks. As R3 also notes, un-rolled networks for iterative inference aren't new: they've been used to replace CRF-type inference, and _also_ to solve image inverse problems (my refs [1-3]).
Therefore, I'd argue that the fundamental idea proposed by the paper isn't new---it is just that the paper seeks to 'formalize' it as an approach for inverse problems (although, there is nothing specific about the analysis that ties it to inverse problems: the paper only shows that the RIM can express gradient descent over prior + likelihood objective).”
A: In no place in the paper do we claim that “unrolling inference” is the novelty of our approach. We do claim however, that we identify 3 minimal components that we deem necessary for well formed iterative inference methods: 1) the current estimate (external state), 2) an error gradient from the likelihood under the current estimate, and 3) an internal state. We define RIMs in a top-down fashion with these 3 components. The implementation is then up to the practitioner (see our argument about generalisation).
In contrast [1-3], define their models in a bottom up fashion: First start with an architecture (graphical model, prior model, likelihood model), then choose an inference method and finally construct the algorithm. We claim that with the 3 components, we could choose any architecture (on an implementation level) that we find to work best for a given problem. This perspective can significantly speed up the workflow in finding new models.
R1: “I also did not find the claims about benefits over prior approaches very compelling. The comment about parameter sharing works both ways---it is possible that untying the parameters leads to better performance over a fewer number of 'iterations', and given that the 'training set' is synthetically generated, learning a larger number of parameters doesn't seem to be an issue.
Also, I'd argue that sharing the parameters is the 'obvious' approach, and the prior methods choose to not tie the parameters to get better accuracy.”
A: It is surprising the reviewer raises this point. In the paper we clearly show that although we use parameter sharing, our approach (marginally) outperforms Regression Tree Fields (RTF-5) which is a discriminatively trained reconstruction method with non-shared parameters per time step (trained for each noise level separately). Given the reviewers claims, we put the presented RIM at a disadvantage in comparison to RTF-5, while still outperforming said method.
Overall, we find the discussion about parameter sharing tedious because it is not part of the central claims of our paper. Yes, we could have chosen to not share parameters over time, but this would yet have been another architectural choice. In our last response to the reviewer we tried to explain the benefits that parameter sharing can bring (e.g. figure 2 can only be done with a model that has shared parameters), unfortunately the reviewer chose to ignore our argument in their review.
R1: “The same holds for being able to handle different noise levels / scale sizes. A single model can always be trained to handle multiple forms of degradation---its just that its likely to do better when it's trained for specific degradation model/level.
But more importantly, there is no evidence in the current set of experiments that shows that this is a property of the RIM architecture. (Moreover, this claim goes against one of the motivations of the paper of not training a single prior for different observation models ...
but to train the entire inference architecture end-to-end).”
A: The reviewer makes it sound very trivial that a model can handle different noise levels/scales (or other corruption processes,see section 4.4 Multi-task learning).
In fact, if this was the case there would be few arguments for using RIMs or other inference approaches that motivate through Bayes’ theorem, whatsoever.
However, the reviewer does not give any evidence for their claims. It is, in fact, the most important result of our paper, that we have a discriminatively trained model that can handle different corruption processes at the same time. None of the papers that are presented by Reviewer 2 and Reviewer 3 have this feature. Either the presented methods trained a new model for every noise level, or results were only presented for a single noise level. In the super-resolution task we compare the RIM with SRCNN which are discriminatively trained CNNs for super-resolution.
SRCNNs do not explicitly take into account the corruption model, but instead a new model is trained for each of the 3 scales. In our paper we show that one RIM which is trained for all 3 scales can outperform the SRCNN on all scales although the RIM has less parameters and it uses parameter sharing over time. To us this clearly shows that model definition of RIMs can benefit performance.
Further the argument of the reviewer that training a new model for every degradation model/level would be always beneficial does not take into account any practical applications.
In practice, it is simply impossible to have different models for every possible noise level (noise levels are discrete in experiments but not in practical applications). And even if there were only 10 possible noise levels, it would imply that it was still necessary to have 10 different models on each device just for the task of denoising. In our paper we show that RIMs can handle different corruption processes under the hood of one model. It’s quite surprising that the reviewer tries to turn this positive feature of the model into a flaw.
I'm inclined to my score after reading the rebuttal. The authors may have missed the point of my criticisms, so I'll respond to specific points in the rebuttal below:
1. The entire motivation for RIMs, **as presented in the paper**, is that traditionally inverse-problems are solved using iterative methods with hand-chosen generative priors, and that one can do better with RIMs that are discriminatively trained with individual time-steps of the recurrent network carrying out the processing in traditional iterative algorithms.
The paper doesn't discuss other methods that use 'recurrent'-type discriminatively trained networks for solving specific inverse tasks, and doesn't compare to them (by omitting such discussion, it makes an implicit claim that all aspects of their approach are new). The specific formulation of RIMs is indeed new (with an inter, but without As I say in my original review, it is possible that adopting the RIM framework (with a clear use of the likelihood gradient, and internal state provided as input) may offer an advantage.
But we can't know without comparisons (to say simply a network that stacks multiple networks that go from image-to-image---i.e., only have the external state). Such a comparison would use the same 'architecture' for each step, to show the benefit of explicitly maintaining an internal state and providing the likelihood gradient.
2. I only brought up parameter sharing since authors cited it as an important difference to the prior discriminatively-trained iterative methods. I merely stated that I don't find the advantage of allowing an adaptive number of steps per example to be particularly compelling (note that in Fig 2, the performance seems to flatten out after a fairly small number of iterations).
I agree---the discussion about shared weights is tedious. I'd recommend it be left out as a major claimed advantage over prior work (when they are discussed).
Otherwise, this claim needs to be developed more with experiments / comparisons (variance in number of steps to reach acceptable quality across images, some indication that an automatic stopping criterion can be formulated, etc.).
3. Authors say, "It is, in fact, the most important result of our paper, that we have a discriminatively trained model that can handle different corruption processes at the same time." But here different corruption processes only mean different 'levels' of the same process---the paper doesn't show a single model that can be used for say both denoising and super-resolution.
"The reviewer makes it sound very trivial that a model can handle different noise levels/scales ...
In fact, if this was the case there would be few arguments for using RIMs or other inference approaches that motivate through Bayes’ theorem, whatsoever"---please read my comment again. Knowing the precise parameters of the corruption process obviously helps, but there's nothing preventing one from discriminatively training a recurrent, or regular, neural network for denoising, where the training samples include inputs with a range of noise levels. Obviously, performance will degrade as the range of noise levels grows. This is why an experiment (with the same architecture, but without the specific RIM formulation---and perhaps something trivial like just the noise level as an input, instead of gradients of the likelihood) is necessary.
For the specific example of denoising, there are infact not that many levels of noise to consider in practice. One could train a separate model for each ISO level (which is typically a small discrete set). As I suggest in my review, if the paper wants to claim the advantage of generality over different versions of the same corruption process, then this claim would be made compelling by considering other inverse problems like deconvolution or in-painting---where the space of blur kernels / inpainting masks is in-fact too large to learn separate models.
Overall, the main concerns in my review remain. The paper should add the missing discussion to related work pointed out by Reviewer 3 and I, and it should include experiments to back up the benefits over these methods that authors have claimed in their responses.
This paper proposes the RIMs that unrolls variational inference procedure.
The author claims that the novelty lies in the separation of the model and inference procedure, making the MAP inference as an end-to-end approach. The effectiveness is shown in image restoration experiments.
While unrolling the inference is not new, the author does raise an interesting perspective towards the `model-free' configuration, where model and inference are not separable and can be learnt jointly.
However I do not quite agree the authors' argument regarding [1] and [2].
Although both [1] and [2] have pre-defined MAP inference problem. It is not necessarily that a separate step is required.
In fact, both do not have either a pre-defined prior model or an explicit prior evaluation step as shown in Fig. 1(a). I believe that the implementation of both follows the same procedure as the proposed, that could be explained through Fig. 1(c).
That is to say, the whole inference procedure eventually becomes a learnable neural network and the energy is implicitly defined through learning the parameters.
Moreover, the RNN block architecture (GRU) and non-linearity (tanh) restrict the flexibility and implicitly form the inherent family of variational energy and inference algorithm. This is also similar with [1] and [2].
Based on that fact, I have the similar feeling with R1 that the novelty is somewhat limited.
Also some discussions should be added in terms of the architecture and nonlinearity that you have chosen.
R3: “This paper proposes the RIMs that unrolls variational inference procedure.”
A: To be clear, we do not claim to unroll a variational inference procedure. In no place in the paper do we define a lower bound on an objective function, neither do we write down an approximation of a probability distribution. We motivate our approach from MAP inference, and we emphasise that the resulting model needs three input components: 1) the current estimate (external state), 2) an error gradient from the likelihood under the current estimate, and 3) an internal state. This insight is summarised in figure 1.
Although we motivate the approach through MAP inference, after training the resulting model can step away from this scheme. There is no explicit objective function to be optimised during inference, in fact the objective function becomes implicit in the iterative process. This acknowledges the fact that objective functions in the context of inverse problems are often only used as a utility to define an inference procedure rather than as a measure of goodness-of-fit. Under these conditions, it is simply unnecessary to explicitly define an objective function for inference.
R3: “However I do not quite agree the authors' argument regarding [1] and [2].
Although both [1] and [2] have pre-defined MAP inference problem. It is not necessarily that a separate step is required.
In fact, both do not have either a pre-defined prior model or an explicit prior evaluation step as shown in Fig. 1(a). I believe that the implementation of both follows the same procedure as the proposed, that could be explained through Fig. 1(c).
That is to say, the whole inference procedure eventually becomes a learnable neural network and the energy is implicitly defined through learning the parameters.“
A: We do not disagree with the reviewer on this point. Yes, both approaches ([1] and [2]) can be described through Fig. 1(c). This goes in hand with our argument that RIMs generalise over these classes of models. The difference lies in the way model architectures are chosen. In [1] and [2], both the prior and inference method are chosen by hand to form the model. The update equations for the respective inference procedures can then be interpreted as a single step in an RNN.  This follows a bottom-up approach: First define the implementation level to then form an algorithm.
However, due to the choice of prior and inference methods, the interaction between both components remains fixed through a predefined set of update equations. In RIMs we motivate iterative inference in a top-down fashion: define a computational model (the RIM), then choose an implementation (a specific model architecture). The use of neural networks in this context generally allows RIMs to implement a broad class of prior and inference methods, more so it relaxes any possible interactions between prior components and inference components, all of which is done through learning.
R3: “Moreover, the RNN block architecture (GRU) and non-linearity (tanh) restrict the flexibility and implicitly form the inherent family of variational energy and inference algorithm. This is also similar with [1] and [2].”
A: The reviewer appears to have an insight into our chosen model architecture which is not apparent to us. It is not clear to us why our chosen architecture would restrict the implicit objective function. We would very much appreciate further insights from the reviewer on this comment.
R3: “Based on that fact, I have the similar feeling with R1 that the novelty is somewhat limited.
Also some discussions should be added in terms of the architecture and nonlinearity that you have chosen.”
A: The choice of architecture is uncommon in the context of inverse models, but more common in the context deep learning. GRUs often show similar performance to LSTMs while having less parameters. We chose this architecture to emphasise that with the RIM perspective these inverse problems can be addressed solely from a deep learning perspective without any particular inspiration from model architectures in the inverse modeling context.

Meta-review:
role: PC
data: comment: This paper presents an approach for learning both a model and inference procedure at the same time using RNNs. The reviewers agree that the idea is interesting, but discussion and considering the responses to the reviews, still felt that more is needed to motivate and contextualise the work, and to make the methods presented more convincing. For this reason, the paper is unfortunately not yet ready for inclusion in this year's proceedings.

Decision: Reject