Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The proposed work, LUSER, aims to improve the computational efficiency of deep unfolding network at training for solving  general image inverse problems.  To do so,  the proposed LUSER incorporated the fixed-point training of the neural network itself within each unrolling iterations by using the deep equilibrium models (DEQ). DEQ is a recently proposed framework for memory efficient learning of an infinite-depth unrolled network as implicitly defined by a fixed point of an operator. The authors have also conducted experiments to show that their approach is able to get similar quality reconstructions with reduced memory, compared to normal deep unfolding.
strength_and_weaknesses: Strengths:

The studied problem is interesting and worth devoting to. The main advantages of the presented technique are the reduced execution time of the data-consistency layer compared to the one based on deep equilibrium models (DEQ4IP) and improved network capacity compared to traditional deep unfolding (LU-DnCNN). 

Weakness:

However, there are some issues that I would like to highlight:

1. One major concern is that the proposed method seems only improve the neural networks learning capacity by running it multiple times within each unrolling iteration. When using the same number of parameters, the memory costs suppose to be the same with normal deep unfolding networks. Unlike DEQ, it not fundamentally solves the memory burden issue when unfolding more iterations (layers) or using more sophisticated networks such as transformers. 

2. The conducted experiments cannot sufficiently support the authors’ claim and the numerical results are somehow vague to read. For example, the various properties of different approach in Table 4 are not well supported by the explicit numerical validation. One better way is to present the training/validation loss against time and epoch, compared to other baseline methods. Besides, it is unclear why DEQ4IP has the same PSNR/SSIM results to LU-DnCNN for MRI reconstruction. Since DEQ4IP can run more iterations during training, it should result in better performance, as reported in [Gilton et al. (2021)].

3. While the proposed method aims to improve computational efficiency of the forward model, no large-scale imaging tasks are conducted in this paper, which also diminishes the contribution.

clarity,_quality,_novelty_and_reproducibility: 1.Clarity/ Quality:  The paper is overall well structured and written. However, some technical and experimental setting is not clearly stated. For example, the stopping criteria of the inner fixed-point iteration of the neural network and how is the anderson acceleration applied are not clearly stated. The image dimension is unknown in this paper.

2.Novelty/Reproducibility: While applying DEQ framework to reduce the memory cost of inner iterations of the deep unfolding network, it not fundamentally reduces the memory cost compared to the original DEQ module, which make the novelty very marginal. Since this work is the integrations of existing works, the reproducibility seems not to be issue.

summary_of_the_review: Consequently, given the pros and cons on balance, I feel this is a very borderline paper, and I vote for borderline reject tentatively
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper combines algorithm unrolling, i.e. loop unrolling, with the deep equilibrium models for reduced memory and improved results. 
strength_and_weaknesses: Strengths:

1. The paper reduces the memory cost for loop unrolling by combining it with the deep equilibrium models. 

Weaknesses:
1) The novelty of this paper is quite limited. Authors combine LU and DEQ, but results seem to be worse than DEQ. Author claims their approach to be better than LU and DEQ in several aspects such as stability. However, extensive experiments have not been conducted to backup these claims.

2) Authors claim for even medium scale problems there is a noticeable tradeoff for accuracy for LU models. Authors have focused on 2D data to show that their approach requires less memory and can deal with large scale complex inverse problems. However, for many applications involving complex operations such as MRI,  LU have achieved a state-of-the-art results (see fastMRI challenge, Muckley et. al, TMI 2021) for 2D datasets.  Thus, memory cost has not been a main challenge for processing 2D data. This issue arises when processing 3D datasets. However, authors have not included any experiments focusing on processing 3D datasets.  

3) Authors mention that their approach can tackle stability issues seen in DEQ4IP.  However, there is no experiment backing up this claim.

4) Experimental results from Table 2 shows that, DEQ4IP outperforms LUSER. Luser can *quantitatively* achieve slightly better performance when parameters are not shared.  However, quantitative metrics may not be reflective of the performance.  In figure 2, MRI images for LUSER approach seem to be more blurry than DEQ4IP. Similarly, DEQ4IP achieves sharper results for CT. Thus, in practice, LUSER provides worse results than the counterpart approach. Also, why LUSER-SW is not included in figure 2?
 

clarity,_quality,_novelty_and_reproducibility: The paper is not well-written and contains lots of grammar errors. The novelty and experiments are limited. 
summary_of_the_review: The paper combines loop unrolling with deep equilibrium models. The experimental results show that the proposed approach does not achieve better results than DEQ. Its benefit over LU is also questionable as the paper does not show results for a large-scale problem such as processing 3D datasets.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper aims to propose a memory-efficient deep equilibrium model (DEQ) for solving inverse problems. However, after reading the paper, I am not sure if I have a clear understanding of what are the key contributions of the proposed LUSER model. It seems to me that the key difference between LUSER and prior DEQ models are:

1. LUSER uses pre-learned DEQ models in its architecture (*"...LUSER achieves this by adopting DEQ models as the learned
regularizer update in a standard LU architecture...."*)
2. Somehow a shallow network can be used so that LUSER only needs to update a small number of parameters so that the memory requirement can be reduced.
strength_and_weaknesses: *Weakness

1. The paper is not well written. The methodology section lacks enough technical details as well as explanations for readers to understand the proposed model. For example, Fig. 1 is not illustrative, and the caption does not provide any explanation of the figure. Due to the above weakness, I am not equipped with enough knowledge of the model to evaluate the contribution of the model and interpret the numerical results.

2. Please explain what is key difference between LUSER and prior models, such as DEQ4IP. I think DEQ4IP also only learns the regularizer network, isn't it? (Data-consistency layer is formed using the fixed forward model, while the regularizer layer is a neural network).

3. The resolution of Fig. 2. is too low to see the visual difference.
clarity,_quality,_novelty_and_reproducibility: The manuscript is not clear. I am not sure I understand the whole work.
summary_of_the_review: Not a good paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new unrolled network architecture for inverse problems. It's main contribution is to replace the large feed-forward architecture (e.g., DnCNN) used in typical unrolled architectures with a small recurrent architecture, known as a Deep Equilibrium (DEQ) model. The DEQ model iterates until it converges to a fixed point and only the gradients associated with this fixed point need to be stored, thus making the proposed architecture more efficient to train.

The proposed architecture is applied to simulated image deblurring, CT, and MRI where it performs incrementally better than a standard unrolled projected gradient descent baseline. The proposed architecture is far more memory efficient to train than the baseline, but actual training times are not provided.

strength_and_weaknesses: # Strengths
Solving large scale imaging inverse problems with unrolled network architectures is an important task and memory efficient training is one of the largest open problems in this field. This work represents a meaningful step in solving this problem. 

The paper is generally well-written and easy to follow. 

The proposed network architecture is well-motivated and simple -- this is a good thing.

# Weaknesses
While the paper is not claiming to be state-of-the-art, the comparisons in the paper are somewhat lacking and the unrolled DnCNN baseline is far from state-of-the-art. Even removing biases from the network may have significantly increased performance [A] so it's unclear if there are really any performance (rather than memory) benefits associated with the proposed method.

[A] Mohan, Sreyas, et al. "Robust And Interpretable Blind Image Denoising Via Bias-Free Convolutional Neural Networks." International Conference on Learning Representations. 2019.

The training procedure description states only that "The weights \theta of the model can be trained via implicit differentiation, removing the need to store all the intermediate activations". This could use a citation or additional explanation. 

A number of statements lack evidence or citations. E.g., "The implicit DEQ models are smaller in size but just as expressive as typical convolutional models".

While Table 4 is a useful summary of the relative difference in training times, evaluation times, etc., the actual training times, evaluation times, etc. need to be in the paper. "Fast" vs "slow" is almost meaningless without context and this imprecision makes it nearly impossible to fully evaluation the submission -- the proposed method could have 1 day inference times.

In table 5, two comparisons isn't enough to show any trends. Would 7 layers further improve performance? When does performance plateau?

I would avoid stating LUSER provides a path forward for non-linear inverse problems. While this may be true, there's zero evidence for this in the paper.

clarity,_quality,_novelty_and_reproducibility: The paper is well written, there seems to be enough info to reproduce the results, their are substantial memory savings associated with the method (though other info is missing), and the proposed technique, while perhaps a natural evolution of existing ideas, is to my knowledge novel. 
summary_of_the_review: The paper proposes a simple and novel technique for significantly reducing the memory costs associated with solving large scale imaging inverse problems with unrolled neural networks. While the current submission is missing important info (e.g., inference times), I remain mostly positive about this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes combining a loop unrolled network with deep equilibrium model to solve inverse problems such as CT, MRI, and image deblurring. The paper was found to lack clarity, and the performance of the proposed method did not appear to improve upon deep equilibrium models.
justification_for_why_not_higher_score: Lack of clarity and poor empirical performance. Reviewers all recommend rejection.
Decision: Reject