Solving Inverse Problems with Model Mismatch using Untrained Neural Networks within Model-based Architectures
Authors: Peimeng Guan
Naveed Iqbal
Mark A. Davenport
Mudassir Masood
Venue: Accepted by TMLR ()

Abstract: Model-based deep learning methods such as loop unrolling (LU) and deep equilibrium model (DEQ) extensions offer outstanding performance in solving inverse problems (IP). These methods unroll the optimization iterations into a sequence of neural networks that in effect learn a regularization function from data. While these architectures are currently state-of-the-art in numerous applications, their success heavily relies on the accuracy of the forward model. This assumption can be limiting in many physical applications due to model simplifications or uncertainties in the apparatus. To address forward model mismatch, we introduce an untrained forward model residual block within the model-based architecture to match the data consistency in the measurement domain for each instance. We propose two variants in well-known model-based architectures (LU and DEQ) and prove convergence under mild conditions. Our approach offers a unified solution that is less parameter-sensitive, requires no additional data, and enables simultaneous fitting of the forward model and reconstruction in a single pass, benefiting both linear and nonlinear inverse problems. The experiments show significant quality improvement in removing artifacts and preserving details across three distinct applications, encompassing both linear and nonlinear inverse problems. Moreover, we highlight reconstruction effectiveness in intermediate steps and showcase robustness to random initialization of the residual block and a higher number of iterations during evaluation.

Published in Transactions on Machine Learning Research (04/2024)
Solving Inverse Problems with Model Mismatch using Untrained Neural Networks within Model-based Architec- tures
Peimeng Guan
pguan6 edu
Department of Electrical and Computer Engineering, Georgia Institute of Technology
Naveed Iqbal
naveediqhal@kuupm edu..
Department of Electrical Engineering and Intendisciplinary Reseorch Center for Communication Systems and Sensing (IRC-CSS), King Fahd University of Petroleum and Minerals
Mark A. Davenport
mdov@gatech. edu
Department Electrical and Computer Engineering, Georgia Institute of Technology
Mudassir Masood
mudossir@kfuum. edu.sa Department of Electrical Engineering and Intendisciplinary Reseorch Center for Communication Systems and Sensing (IRC-CSS), King Fahd University of Petroleum and Minerals
Reviewed on OpenReview: https: // openreview. net/ form?? id= XHEhjDDPDD
Abstract
Model-based deep learning methods such as loop unrolling (LU) and deep equilibrium model (DEQ) extensions offer outstanding performance in solving inverse problems (IP). These methods unroll the optimization iterations into a sequence of neural networks that in effect learn a regularization function from data. While these architectures are currently state-of- the-art in numerous applications, their success heavily relies on the accuracy of the forward model. This assumption be limiting in many physical applications due to model simplifi- cations uncertainties in the apparatus. To address forward model mismatch, introduce an untrained forward model residual block within the model-based architecture to match the data consistency in the measurement domain for each instance. We propose two variants in well-known model-based architectures (LU and DEQ) and prove convergence under mild conditions. Our approach offers a unified solution that is less parameter-sensitive requires no additional data, and enables simult nneous fitting of the forward model and reconstruction in single pass, benefiting both linear and nonlinear inverse problems. The experiments show significant quality improvement in removing artifacts and preserving details across three dis- linct applications, encompassing both linear and noulinear inverse problems. Moreover, we highlight reconstruction effectiveness in intermediate steps and showcase robustness to ran- dom initialization of the residual block and a higher number of iterations during evaluation. Code is available at https: / //gtttub.. com/InvProboa A- adwptive--eeeeee modee-odeellle -ased-aaaaaaeee moteeddmeehdd..
1 Introduction
Consider an inverse problem of the following form: = A(x) +e.
(1)
The goal is reconstruct the latent signal from the measurements y in the presence of noise e, where typically the forward model A is assumed to known. Inverse problems are generally challenging because they often ill-posed, i.e., the solution is not unique, or the reconstruction is highly sensitive to noise and/or model mismatch. The traditional approach to recovering x from the measurements y is by solving a regularized optimization problem of the form:
1 min
2 1 A(=)( +
(2)

---

Published in Transactions on Machine Learning Research (04/2024)
Xg
X1
X2
X3
X X5 X Xy X
With accurate A
With small mismatch in
Figure 1: A proximal LU network is trained for deblurring task using single forward model. The top row shows the intermediate reconstructions over 8 iterations using the true model, while the bottom row shows the evaluation results when a small perturbation is added to the forward model (the Peak Signal-to-Noise Ratio of the true kernel to the noisy kernel is 40.9 dB). This quality degradation is due to the accumulation of errors in the forward model.
where 7 2 0 is an appropriateel--cosenn parameter. The regularizer is usually predetermined based on some known or desired structure, o.g.; lı-, f2-, or total variation (TV) norm to promote sparsity, smoothness, or edges in image reconstructions, respectively. Solving (2) requires careful consideration of the underlying physics or the forward model A to obtain stable and accurate reconstruction. However, knowing A can be challenging in practice. The reasons for this include inaccurate measurements and challenging calibration, highly nonlinear and/or computationally expensive models replaced by simplified versions, as well as access to only approximations of certain features. Usually, some knowledge of the true model, designated Ao in this work, is available. This occurs in many applications, including blind rocovering scismic layer models using the simplified acoustic wave cquation as the forward model (Mousa) & Al-Shuhaill )011), determining fault locations in media with unknown structures (Mahmoud & Khalid 2013), computational tomography (CT) when the dynamic behavior of human object is observed during the measurement period Blanke et al. 22020). sparse signal recovery when the sensing matrix is not known perfectly, etc.
problems, coarvolltioondebburiitt
When the forward model is preciscly known, the inverse problem in 2 can be solved via classical optimization methods, where r is predefined. Machine-learning approaches, as summarized in Arridge et al. (2019 Iqbal et al. (2023); Ongie et al.] (2020). have demonstrated superior reconstruction performance by effectively
learning regularizer from data. For example, the Plug-and-Play method (Venkatakrishnan et al. 2013)
trains general denoiser independently of the forward model and iteratively minimizes (2) with the learned denoiser as the regularization updates. Another class of approaches, the loop unrolling (LU) method (Gregor & LeCun 2010 Hershey et al. 2014 Adler & Öktem 2018) and its extension using deep equilibrium models (Gilton et al.) 2021a), build on the observation that many itcrative algorithms for solving (2) can be re- expressed as a sequence of neural networks (Gregor & LeCun 2010), which can then be trained to remove artifacts and noise patterns associated with a known A, resulting in higher quality reconstructions.
While these model-based machine learning solvers have demonstrated impressive performance, they encounter challenges while dealing with inaccurate forward models. Plug-and-Plat LU, and its deep equilibrium extensions all entail gradient update of the data-fidelity term, which can be in error when the forward model is inaccurate. This causes the learned denoiser or regularizer to be ineffective. In particular, while LU often exhibits state-of-the-art performance, has faster runtime, and has improved stability in training Guua et al. 2022) compared to other approaches, it requires precise knowledge of the forward model. Fig. demonstrates the scnsitivity of LU to mismatch in A.
Some works passively address errors in the y-space by enhancing the robustness of the neural network. For example, Krainovic et al. (2023) suggests that introducing jittering in the y-space can achieve robustness in worst-case & perturbations. In Hu et al. (2023). deep equilibrium solver is trained using various incorrect A's, demonstraling greater robustness to variations in A compared to the Plug-and-Plec method. However, it is noteworthy that such robust networks often experience a tradeoff between accuracy and stability, as discussed in Krainovic et al. (2023): Gottschling et al. (2023). In contrast, our proposed approach involves an active strategy for mitigating model mismatch. The experimental results in Section t illustrate the effectiveness of our method compared to the same LU network that merely learns robust. mapping passively.
2

---

Published in Transactions on Machine Learning Research (04/2024)
Various active forward model matching algorithms are tailored to different tasks. For linear IPs, model mismatches can be more easily resolved by alternatively updating the model parameters and the underlying signal. For instance, Fergus et al. (2006))(Cai et al. (000));[eviin et al. (2009); ) Cho & Lee (2009) reconstructed the latent images with updates in the forward model solved directly from least squares. However, these methods are oflen constrained to linear scenarios and rely on lask-specifie recovery techniques, which are typically highly sensitive to hyperparameters Cho & Lee (2009). Later works Nan & Jil (2020); (Gilton et al. (2021b) learn a regularizer from data, resulting in improved performance on linear inverse problems. The structure of the initial forward model in a nonlinear inverse problem often differs from the true model. For instance, linear models may be used to estimate nonlinear functions. This motivates the use of neural networks to learn the true forward model (Candiani et al.) 2021 Koponen et al., 2021 Lunz et al., 2021 However, traditional neural network training approaches require separate stages for learning the forward model and signal reconstruction. This requires a substantial amount of training data to ensure accurate mapping (Arjas et al.) (023) from any æ (which represents ground truth and intermediate reconstructions) to its corresponding y in iteralive reconstructions. With limited training data, this method falls out of the discussion.
In contrast, we introduce an untrained residual network capable of handling forward model mismatch for general IPs. The proposed method is a unified framework that applies to both linear and nonlinear IPs, offering enhanced stability during training, requiring no additional data, and allowing fitting the forward model and reconstruction simultaneously in a single pass. We further list our contributions as follows:
General model-based architecture in handling model-mismatch: We present a general model-based architecture for solving TPs when only an approximation of the forward model is avail- able. Unlike classical model-based architectures which require precise knowledge of A, we propose Iwo variants of the model-based algorithms that can iteratively update the forward model along with the reconstruction.
Introducing untrained neural network for model-mismatch: We introduce a novel ap- proach by integrating an untrained neural network into a model-based architecture to handle model- mismatch in both linear and nonlinear inverse problems. By updating its parameters alongside the reconstruction process, we circumvent the need for additional data and pre-training the forward model in solving nonlinear inverse problems. Indeed, we show that random initialization of this network during evaluation yields the same level of reconstruction.
Proof of convergence and empirical validation: We establish the convergence of the proposed algorithm under mild conditions and empirically verify its convergence using a Deep Equilibrium Model (DEQ) structure.
Improved performance in lincar and nonlinear IP tasks: In contrast to the passive ro- bust training approaches within general IP solvers using model-based architectures, our methods demonstrate substantial improvement in image blind deblurring (linear), seismic blind deconvolu- tion (linear), and landscape defogging (nonlinear) tasks. It's worth noting that while our application showeases improvements in these specific arcas, its scope extends beyond to fields like medical imag- ing.
Effectiveness in iterative reconstructions and robustness: We show that proposed algorithms lead to more effective intermediate reconstructions. They also exhibit robustness different random initializations of the residual block and are more stable in scenarios involving higher number of iterations during evaluation.
2
Related Works
2.1 Review of Model-based Architectures
Model-based architectures in solving inverse problems are motivated by classical optimization. When the forward operator A is precisely known, one can unroll the objective function in T into numbers of iterative steps, and learn the regularization update from data. One subclass of model-based IP solvers is called loop unrolling (LU) or deep unfolding, which has a fixed number of itcrations. LU first introduced to solve sparse coding Gregor & LeCun (2010): Chen et al. 2018). and later on extended many image restoration
problems Monga et al. 22000); Gilton et al. (022aa) Ongie et. al. (2020); Zhao et al. 2023 2024
Variations
3

---

Published in Transactions on Machine Learning Research (04/2024)
of LU emerge according to different optimization algorithms, such as projected gradient descent Lohit et al. (2019), proximal gradient descent Hosseini et. al. 2020), alternating direction method of multipliers (ADMM) Yang et al. (2020), and momentum gradient descent Chun et al. (2023). More works can be found in Monga
ct al.
2020
The concept of unrolling optimization iterations is not restricted to fixed number but is often limited by memory constraints during backpropagation. To address this, Giltonn al. (2021a) proposes to train the iterative updates of the reconstructions until convergence using fixed point solver. Similarly, [Cai et al.] (222)) extends loop unrolling to an infinite number of iterations for robust principal component analysis, achieved by layer-wise training the initial iterations and updating the additional penalty paramcters for subsequent iterations. In our work, we demonstrate the proposed forward model matching technique using DEQ, known for its infinite-layer extension with consistent architecture across all iterations.
2.2
Methods for Addressing Model Mismatch
A linear inverse problem with forward model mismatch can be interpreted as parameter tuning within a predetermined structure. In other words, given matrix A of dimension
n, the entries contain errors.
This happens in many imaging tasks, such as blind deblurring (Li et al. 2020 Dong et al. 2020 Zhang et al. 2022 and blind deconvolution (Krishnan et. al. 2011]. These methods are tailored to specilic lasks. For example, Li et al. (2020) is recast from IVrreuularization due to the sparse gradients in natural images, Dong ctt al. (020) incorporates operations in Fourier space, which is hard to generalize to other inverse problem lasks. Their Lask-specific designs often involve numerous hyperparameters, leading to high sensitivity in reconstruction (Cho & Lee, 2009). For example, we observe the instability in training the blind deblurring algorithm in Li et al. (2020) to hyperparameters such as threshold value, parameter initialization, scale of proximity, etc. In contrast, we propose simple but stable solution applicable to general IPs with only est forward models.
On the other hand, when initial and true forward models differ structurally, updating parameters in Ao often yields suboptimal solutions, common in many nonlinear inverse problems. Oftentimes, Ao is the best approximation within this structure. This motivates the use of a more complex structure, like a neural network, to approximate the true model. In nonlinear applications, reconstructions with erroncous model estimates are lypically tackled in two steps. First, training a neural nelwork to learn the true forward model [Candiani et al., 2021 Hauptmann et al., 2018 Koponen et al., 2021 Liu et al., 2013 Lunz et al. 2021 Then, solving for the reconstruction using a prodetermined regularizer like TV-norm (Candiani et al.
2021 | and pseudo-Huber function Lunz et al. 2021), or learning a regularizer using the pre-trained forward model [Koponen et al.) 2021). These approaches assume either a "perfect" learned forward model Koponen iet al. 2021), or errors can be corrected via the learned regularizer (Hauptmann et al. (2018) which is less interpretable. In this work, we propose to use an untrained neural network to match the data consistency, which allows updating the forward model and the reconstruction in one step adaptively for each instance.
3
Background
3.1
Loop Unrolling Methods
The traditional approach to solving inverse problems by formulating them as an optimization problem of the form in ( A natural approach to solving 2 via a prozimal gradient descent algorithm, which can be applied even when the regularizer r is not differentiable, as is often the case (Parikh & Boyd 2014). The resulting update involves first taking gradient step (with a fixed step size 17) that aims to minimize the data-fidelity term in (2) This is then followed by proximal step, resulting in the iteration (Parikh & Boyd 2014
xs+1 prox,., (x + nA (y A(xx))) where A denotes the adjoint operator of A, 7 > 0 is again the parameter that controls the weight of the regularizer, and the proximal operator of a function is defined as:
(3)
1
prox, (z) arg min
= 2 || z =1 + 9(z).
(4)
As we can see, the choice of regularizer manifests itself entirely through the proximal operator. The LU algorithm essentially keeps the update in (3) but replaces the proximal operator with neural network, and
4

---

Published in Transactions on Machine Learning Research (04/2024)
limits the algorithm to finite number of iterations K. The final output æg is compared with the ground truth a and the network parameters are updated accordingly through end-to-end training.
By structuring the network in way that mirrors proximal gradient descent, and taking advantage of an accurate descent direction derived from the knowledge of A, the learned portion of the network can be interpreted as the proximal operator learned regularizer that enforces desired signal structures. However, when the forward model inexact or only approximately known, the gradient update in (3) can introduce errors. Since LU is trained end-to-end, the error will manifest itself in the learned proximal operators. As result, the neural network used in LU will no longer act pure proximal operator since it must both enforce signal structure as well as compensate for the errors in A, potentially becoming less effective and interpretable.
3.2 Deep Equilibrium Models
Deep equilibrium models (DEQ) form class of implicit neural networks, where the output the fixed point solution of neural network block Bai et al. (2019). For initial input x!0), and neural network Φo, DEQ repeatedly applies da.
a (k+1) de (x (x) (0) (k)),
until convergence to an equilibrium solution. The forward pass can be performed using any efficient root- finding algorithm. The backward pass, instead of saving all computational graphs do backpropagation in one pass (also known as end-to-end training), can be performed by either direct computation of the vector-Jacobian product (Bai et al. 2019) using the "Jacobian-free" strategy proposed in Fung al.
Later work in Gilton et al. (2021a) extends LU using a DEQ architecture, so instead of having pre- determined fixed number of iterations, this work allows potentially high number of iterations until a fixed-point solution is reached. Using DEQ shows a higher reconstruction quality, lower memory usage, and observe consistent improvement in reconstruction.
3.3 Half-Quadratic Splitting (HQS)
Another key tool that will rely in approach is variable splitting. Variable splitting is iterative optimization method that solves problems where the objective function is sum of multiple components
(Geman & Yang 1995 Nikolova & Ng 2005 Bergmann et al. 2015 Hurault et al. 2022 Yang & Wang 2017). It works by introducing an auxiliary variable and iteratively optimizing the objective function with respect to each variable while fixing the others. For example, by introducing an auxiliary variable x, we can
re-express the optimization problem in (2) as
] ly A(w)11 + tr(2), s.t. x
min 2.z
The solution can be approximated using the following unconstrained problem,
min 1 y 1(x) ! 11 x-= =11
2
where μ 2 0 is a tuning parameter. This optimization can be solved by iteratively updating an and 2 until convergence:
0h+1 arg min 7 ll A(x)
1 - - foa--zll: 2x112.
#t+1 arg min (r(2) + f1= 21+1
4 Proposed Method A-adaptive Loop Unrolling Architecture
The discussion above has assumed exact knowledge of A. Here suppose that have initial guess of the forward model Ao and consider a neural network fo that learns the measurement residual due to model misfit given the signal æ and Au, i.c.,
y = A(x) + Ao(x)) fo)

---

Published in Transactions on Machine Learning Research (04/2024)
We assume that Ao is useful estimate of the true forward model, in other words, A(x) and Ao(z) close.
We can then express the optimization ion problem in 2 as
min 2,,
ly Ao(x)
+ (r(x) + fil (2. Ao)l
(5)
Introducing an auxiliary variable 2, the solution of 5 is equivalent to solving
1 2 リ Ao(z) - Ao) l2 + (r(x)) + TI]Jo(2, + A|[a - =112 2.2.0
min
Ao(z)
(6)
Similar to the HQS updates, we first initialize æo 20, and Og- We then update each variable in the objective function by keeping other variables fixed. For h = 1.2,..., we have
2k+1 = arg min
2 lly - Ao(z) - S0 Ao) 12 + fo (z, Ao)l + A| - =112-
0k+1 arg min
2 51ly Ab(zx+1) fa(z Ao)l (1
(7)
#+++ prox A. (mk 1/= 2k+1))-
It should be noted that the update on z no longer has closed-form solution due to the nonlinearity of fe. However, this can be efficiently computed using Autograd in Pytorch (Paszke et al. 2017] or other differentiation computing algorithms. Meanwhile, the update on 0 follows the regular backpropagation for network parameters. We can then replace the proximal operator with a neural network, and the update on a connects all the components to form an LU network as illustrated in Fig. 2 The parameters in the proximal network, are learned through end-lo-end training, where v) is a trainable step size.
While we refer to se as network in the context of learning forward model mismatch, it differs from classical training approaches where weights remain fixed during evaluation. In our framework, the paramcters 0 in f are updated both during training and evaluation for each instance. The objective is to iteratively align the data fidelity term for a specific measurement through a nonlinear estimation f with the minimal fo-norm. Our proposed technique involves using untrained (random) weights 9g at the initial iteration and adjusting 6g based on the loss outlined in the second line of (7). The concept of employing an untrained neural network is reminiscent of Deep Image Prior (Ulyanov et al. 2018), which initializes a reconstruction network randomly, treating the weights as an implicit prior for the reconstruction. In our approach, however, we suggest using untrained neural network as integral part of the reconstruction process to address model mismatch while still being capable of learning regularization updates from all training data. Subsequent experiments demonstrate that during evaluation, the residual network f can be initialized with various untrained weights, resulting in different performance levels.
Note that the proposed iterative updates look reminiscent the alternating direction method of multipliers (ADMM), like Yang et al.] (2016). When the forward model is preciscly known, ADMM aims to minimize the Lagrangian of the objective function with an additional auxiliary variable. The proposed algorithm uses HQS as simpler method to split variables in solving optimization problem, but the algorithm could casily be generalized to other solvers such as ADMM.
5 Proposed Method II: A-adaptive Deep Equilibrium Architecture
Loop unrolling algorithms typically restrict the number of iterations to small value during end-to-end training, primarily to alleviate the high memory costs associated with processing large number of steps. In such cases, the focus often leans more toward managing computational resources rather than achieving convergence. However, when it comes to another well-known model-based architecture the deep equilibrium model (DEQ) extension, the situation differs. The DEQ exlension seeks a fixed-pole solution, which requires careful consideration of convergence to ensure the model's effectiveness in finding the desired equilibrium. In this section, we first prove the convergence of the proposed iterative updates (7) under mild conditions and thus propose an extension of A-adaptive LU through the integration of deep equilibrium model, namely A-adaptive DEQ.
6

---

Published in Transactions on Machine Learning Research (04/2024)
Auxiliary updato
W/
Wk+1
WK
80 0x
Correction
Bu+1
0x
update
Xg Xg
NN
Xk+1
xx
Figure Illustration of the hth iteration of an A-adaptive LU network. æo fed into the network, the auxiliary update and the correction update corresponding updates in 0 and respectively, and the proximal network in green is updated using end-to-end training. The final output contains the parameters for estimating the function mismatch and the reconstruction estimate.
Proposition 5.1. Assume r is convex, and use the Sact that || =11 is L-soooth with respect to æ, in another word let f(x) = || z 12, [[](( - V f((22)) I L| 22112 for some L > 0. The algorithm in 7 converges when the step size U I/L for all k 2 1.
Aalx 21 For a fixed æs and Ox. J(z6+1, 0,,411,,,000 xx) S J(z. vx) because we are minimizing over z. Similarly, given fixed 21+1 and æk: J[zx+1, 0k+1, σx) & J 8k, xx) when minimizing over 6. When fixing 0x41 and #k+1, from the convergence result. of the proximal gradient method (Parikh Boyd (014)) J(z++1, ((*++1,0++10 0k+1, S J (z [x) 27 (x - xx+1)1l2 S [Z 0x+1, Therefore, the value of the objective function decreases for updates of zx. O2, Tb at all k. In addition, because this value is also bounded below, the algorithm converges.
Proof. Let x(),0,x)
1 b1 Ao(z) Ao) + (r()) + (z)
In contrast to employing fixed number of ilerations K in A-adaptive LU, as outlined in equation A- adaptive DEQ allows F& to converge to fixed point solution. This convergence is achieved over a potentially large number of iterations, motivated by the convergence property established in Proposition 5.1
We denote Fp., as the function representing one iterative update of variables 2k. 0, and 2y in 7 where p parameterizes the proximal neural network and v denotes the trainable step size. According to Proposition
5.1 for sufficiently large number of iterations ko, a fixed point solution reached for all ko.
2k. 0z. 2& (x. 6k. xx)-
A-adaptive DEQ shares the same architecture as A-adaptive LU, where f is still an untrained neural network and the proximal operator is replaced using a neural network, but it is trained differently. Following the training scheme introduced in Gilton et al. (2021a), the forward pass of A-adaptive DEQ solves for fixed point solution, which can be efficiently achieved using Anderson acceleration (Walker & Ni, 2011). Whereas in the backward pass, we adopted the "Jacobian-Tree" backpropagation strategy stated in Fung et al.] We show in the experiments that the A-adaptive DEQ performs even better than A-adaptive LU due higher number of effective iterations. In addition, we verify its convergence during evaluation.
6 Experiments and Discussion
In this section, we demonstrate the efficacy of the proposed A-adaptive LU algorithm and show an im- provement in reconstruction by learning a more accurate forward model compared to the following baseline mothods: 1) a neural network that maps from the initial reconstruction æo the ground-truth æ, referred to a direct inverse mapping, which is independent of the forward operator, and 2) robust LU that is trained with inexact Ao's s from the initial æp- Notice that 2o is initialized with Aiy when and y from different spaces or 2p y if they are from the same space. We evaluate the algorithms on linear IP tasks (image blind deblurring and seismic blind deconvolution), and a nonlinear IP task (landscape defogging). For each task, the direct inverse work and the proximal network in LU and the proposed methods share the same architecture.
Notice that in linear IPs, the forward model is defined by matrix in specific dimension, so the true forward model can be determined by either solving for the least squares solution or by using gradient
7

---

Published in Transactions on Machine Learning Research (04/2024)
X
Robust LU
A-adaptive LU A-adaptive DEQ
Figure 3: Comparing the deblurring results using robust LU, A-adaptive LU and A-adaptive DEQ to the ground truth, where 2g the initial blurry images and æ is the ground truth. The bottom row shows the zoomed regions in red boxes. The proposed methods generate sharper edges.
X
Robust LU
A adaptive LU A-uduptive DEQ
X
Figure 4: Comparing the deconvolution results using robust LU, A-adaptive LU and A-adaptive DEQ to the ground truth. The bottom row shows the zoomed regions in red boxes. A seismic layer in the red box is missing in reconstruction using the bascline robust LU method.
descent update inside the loop unrolling method. Our experiments first show the proposed methods work in linear cases, then more interestingly, when no precise formulation of the true forward model is known in nonlinear IPs (infeasible to solve directly), our proposed method shows its effectiveness in reconstruction. Image Blind Deblurring In this problem, we aim remove the blur from images y when small amount of noise is present. The forward model is defined by a Gaussian blur kernel where have inaccurate knowledge of the variance and size of the kernel. To validate this approach, experiments are conducted on the CelebA dataset (Liu el al. 2015), which was resized to 3 x 120 x 100. The blurry images are generated with different Gaussian kernels for each pair of data (æ;, yi, Ao.s. Seismic Blind Deconvolution Here acoustic wave generated by vibroseis truck on the surface of the earth propagates through the earth's layers. The reflected signals are collected by the geophones and stored as raw measurements. The measurements are obtained in the form of [ However, the frequencies of the acoustic wave are often inaccurately recorded due to the limited resolution and noise in the measurement process (Zabihi Naeini & Sams 2017), which leads to an inaccurate estimate of the wavelet in the forward model. The forward process can be viewod as a convolution between the acoustic wave and the layer reflectivity æ. Therefore, the goal of seismic deconvolution is reconstruct the layer reflectivities æ, with the inaccurate estimate of the wavelet in the forward model taken into account. The true model for each data pair (æ,, Yr, Ao,)) can be expressed follows,
y: Ag, + fo(a;, w:) + Ei;
where Ao, is a Toeplitz matrix derived from the inexact source wavelet w, The measurement is simulated by applying inaccurate wavelet with small additive Gaussian noise to the forward model. Notice that noise added to the true model may result in artifacts due to extra magnification factor applied by ø; which is distinct from the measurement noise e; The data is generated following the procedure in Iqbal et al. 2019).
Landscape Image Defogging We also demonstrate the proposed algorithms with a nonlinear IP. Con- sider the problem of removing the effects of fog, haze or mist from image y to restore its true appearance
8

---

Published in Transactions on Machine Learning Research (04/2024)
Robust A-adaptive
A-adaptive DEQ
Figure 5: Comparing the defogging results using robust LU, A-adaptive LU and A-adaptive DEQ to the ground truth. The bottom row shows the zoomed regions in red boxcs. The proposed methods reproduce cleaner images.
æ. Examples of and æ are illustrated in Fig. 5 The degradation process can be modeled as follows,
y
L(x) @ (I - T((x)) tE,
where T E [0, 1]3xxxx is an unknown transmission map as function of the fog-free image m and n demole the width and height of the image. T(x) is then applied pointwise (0) to æ. L(x) is the unknown atmospheric light profile (Tffall et al. 2019). The goal is to estimate the model mismatch and recover the clean image. When the transition map matrix with all ones, the initial forward model is simplified to identity mapping pixel-wise. Since T and L are unknown functions with respect to H, we use two separate networks to learn transmission mapping and the light profile respectively. This example shows the flexibility of our proposed methods. Instead of having one single network fo that learns the residual, if we know the explicit formulation of the true model, can customize the residual network in more structured way. To evaluate the proposed approaches, the CityScapes Depth and Segmentation dataset given in Cordts et al. (2016) is used. This dataset provides depth maps, as well as foggy and clean views of urban street scenes.
6.1 Reconstruction Results
Table compares the average and the standard deviation of the testing peak signal-to-noise ratio (PSNR) in dB and Structural Similarity Index Measure (SSTM) of the reconstructions. The neural network learns direct inverse regardless of Au,i, which is the worst in roconstruction. It is notable that, even with some degree of inaccuracy in A, the robust LU still outperforms the direct inverse mapping. This is because end- to-end training enables LU to fix the gradient-step errors resulting from the model mismatch to some extent, thereby maintaining reasonably higher-quality results. Our results align with the finding in Krainovic ot al. (2023) that the robust LU trains an inverse mapping that less sensitive to perturbations in the y-space, by seeing inaccuracies in the forward model at train time. However, in the next section, we will discuss the intermediate reconstruction ineffectiveness of robust LU compared to the proposed methods.
The proposed methods make an improvement by actively adjusting for the model mismatch for each data pair. Instead of having the neural network learn both the proximal step and the model correction as in a black box, the proposed methods separate the tasks and each part of the network is now learning a clearer and easier task, resulting in a substantial performance gain over both the robust LU and the direct inverse mapping in all tasks. Furthermore, A-adaptive DEQ allows for a higher number of iterations than A-adaptive LU, resulting in superior performance both PSNR and SSIM. Notice that the A-adaptive LU tends have a smaller standard deviation than A-adaptive DEQ because the latter varies in number of iterations.
Apart from the quantitative evaluation, offer visual comparison of the reconstruction results using the robust LU and the proposed methods for the presented tasks in Figs. 3 and 5 In blind image deblurring, Fig. 3 shows that the proposed methods preserve a sharper jawline. In seismic blind deconvolution, the baseline method (Robust LU) fails to reconstruct a seismic layer as highlighted in red boxes in Fig. 4 whercas both proposed mothods successfully capture the layer. Furthermore, in the landscape defogging example, Fig. 5 demonstrates the proposed methods remove more artifacts while preserving more detailed information.
9

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Average and standard deviation testing PSNR and SSIM for direct inverse mapping, robust LU, A-adaptive LU, and A-adaptive DEQ. The best performances for each task are bold and the second best performances are underlined.
Deblur.
Deconv.
Defog-
PSNR.
24.002 0.101
22.816 + 0.112
21.802 = 1.706
Direct Inverse
SSIM
0.7689 + 0.0007
0.7421 + 0.0041
0.8738 = 0.0347
PSNR
34.380 + 0.437
24.785 + 0.836
29.645 = 1.939
Robust LU
SSIM
0.9429 + 0.0014
0.8407 = 0.0214
0.9563 = 0.0164
A-adaptive LU
PSNR
36.621 +: 0.330
27.271 +: 0.787
31.112 1 1.854
(proposed meth. I)
SSIM
0.9731 + 0.0010
0.8960 + 0.0157
0.9661 = 0.0133
A-adaplive DEQ
PSNR
37.809
0.532
28.236
0.321
32.149 2.069
(proposed meth. IT)
SSIM
0.9774 + 0.0041
0.9082 + 0.0201
0.9723 + 0.0124
6.2 Effectiveness of Reconstruction
We also explore the effectiveness of reconstruction by comparing the intermediate reconstruction æy in the robust LU and A-adaptive LU in Fig. 6 'he average testing Mean-Squared Error (MSE) between ay 's and the ground truth are recorded for all & 1,...,,, While LU is trained to be robust to mismatch in A's, the intermediate MSE of the robust LU remains high until the final iteration. In fact, at iteration 3 in the image deblurring task and iteration 2 and 4 in the landscape defogging task, we observe noticeable MSE increments in the robust LU (solid blue curve). It indicates that the robust LU lacks the capacity to learn how to effectively correct the errors a single iteration. Because the proximal network in the robust LU aims to perform both the proximal step and the model mismatch, it learns a less interpretable mapping from the erroneous gradient step to the true In contrast, A-adaptive LU learns model mismatch and the proximal step separately, It thus allows effective error correction and quality improvement in every iteration, resulting in consistent MSE decrement.
Image Blind Deblurring
Seismic Blind Daconvalution
Landscape Datogging
10-1
0922
robust LU
Akadaptive LU
20"4
10-3
bdoptrrel LU
a.aapppcc LJ
benatons
herethons
hersbonns
Figure 6: Average testing MSE of the intermediate reconstructions for the 5-iteration robust LU and the proposed A-adaptive LU.
6.3
Intermediate Reconstructions of A-adaptive DEQ
We show the intermediale reconstructions of the DEQ variant of the proposed method in Fig. 7 The proposed network trained with a maximum of 30 iterations or when a fixed point solution is reached, whereas when evaluating A-adaptive DEQ with more itcrations (i.c., 100), the reconstruction error remains low. This result aligns with the finding in Gilton et al. (2021a) where the reconstruction quality remains high after reaching the fixed point, particularly when the forward model is precisely known.
6.4 Robustness of A-adaptive LU with More Iterations
In this section, we empirically assess the stability of our proposed A-adaptive LU variant by evaluating it with a greater number of unrolling iterations than those employed during training. The result can be found in Across all three tasks, a noticeable Mean Squared Error (MSE) gap emerges between the two curves. The solid blue curve, representing robust LU, exhibits a drop in MSE itcration 5, which corresponds to the target output during training. However, as the number of iterations exceods the max iterations of training (in this set of experiments k 5), the MSE rises significantly. Notice that although the MSF
10

---

Published in Transactions on Machine Learning Research (04/2024)
Image Blind Dealumnng
Selsmic Blind Deconvolution
Lardscape Datogging
adaptiie
wulupp ve
edaptine DeD
10-2
1073
30"2
20
40
60
30
100
C0
33
100
20
40
60
80
100
Benshons
# benations
Beratiom
Figure 7: Average testing MSE of the intermediate reconstructions for the proposed A-adaptive DEQ. A max of 30 iterations is used in training, this figure depicts the extended evaluation result when allowing a max of 100 iterations.
at iteration 10 is relatively low for the landscape defogging task, the reconstruction is not stable and less prodictable. In contrast, the dashod red curve (A-adaptive LU) oscillates around the lowest MSE. This observation highlights the enhanced robustness of the proposed A-adaptive LU when subjected to a higher number of iterations during evaluation.
Selsmic
Landscape Datogging
16"1
Aradaptine
Jeadaptive
Jbadactive
30
Herabons
benetions
Beratons
Figure 8: Average testing MSE of intermediate reconstructions using the robust LU and A-adaptive LU with 10 unrolling iterations, while trained on 5 unrolling iterations.
6.5 Random Initialization of the Mismatch Network
The residual network fo is updated along the reconstruction in both training and evaluation for each data instance. Because the update of in 7 aims to minimize the data-fidelity term with consideration of the forward model mismatch in the smallest f2-norm, the initialization of O is less important. In this section, show that obtain the same level of performance in both proposed architectures. We evaluate the PSNR and SSIM of reconstruction results from both proposed methods using saved model fø, and two commonly used neural network initialization methods: Kaiming uniform (He et al. 2015) and Xavier (Glorot & Bengio 2010 initialization, as shown Table
be initialized by untrained neural network at even evaluation time, and still
Table 2: Average testing PSNR and SSTM for A-adaptive LU and A-adaptive DEQ using so with the saved model, uniform random initialization, and Xavier initialization.
PSNR/SSIM
A-adaptive LU
A-adaptive DEQ
saved model uniform
06.621/0.9771
37.809/0.9774
Deblurring
36.604/0.9733
37.808/0.99 /0.9774
Xavier
36.606/0.9733
37.809/0.9777
saved model
27.271/0.896
28.236 28.2366/0.9088 /0.9082
Deconvolution
uniform
27.268/0.899)
28.239
28.239/009088 /0.9084
Xavier
27.272/0.8960
28.238 /0.9044
saved model
31.112/0.9661
32.149 1149/099722
Defogging
uniform
31.277 31.277700.6661
32. /0.9722
32.147/0.9922
Xavier
31.276/0.9661
32.117/00977
11

---

Published in Transactions on Machine Learning Research (04/2024)
6.6 Discussion of Evaluation Runtime
While the suggested methods demonstrate superior numerical and visual reconstruction qualities, they do incur runtime tradeoff to be more accurate. Table 3 compares the average and standard deviation of runtime per batch during evaluation. The same batch sizes maintained across all three methods for each lask to ensure comparable compulational overhead. The loop unrolling variant of the proposed method is not significantly slower than the robust LU, despite the A-adaptive LU involving extra optimization steps within the reconstruction process compared to the one-step gradient update in the robust LU. Howover, the DEQ variant is noticeably slower due its high number of iterations in solving for a fixed point solution. The A-adaptive DEQ method also has higher variance because the number of iterations not fixed due the design of DEQs.
Average and standard deviation of runtime ms during evaluation for robust LU, A-adaptive LU and A-adaptive DEQ. The same batch size enforced for all methods across each task.
Table 3:
Tasks
Methods
Evaluation time (ms)/batch
robust LU
80.26 0.52
Deblurring (batch size 32)
A-adaptive LU
253.06 = 0.82
A-adaptive DEQ
2219.53 + 15.89
robust LU
81.14 = 0.67
Deconvolution (batch size = 32)
A-adaptive LU
256.94 + 0.85
A-adaptive DEQ
2249.60 1 28.37
robust LU
112.91 = 1.03
Delogging (batch size = 8)
A-adaptive LU
370.93 1 1.14
A-aduptive DEQ
2433.66 1 15.79
7 Conclusion
While the model-based machine learning IP solvers are powerful, they demand precise knowledge of the forward models, which can be impractical in many applications. One way to handle the model mismatch is to train a model with inaccurate forward models, which will implicitly correct the errors due to end-to- end training and demonstrate reasonable performance. However, this falls short in terms interpretability and efficacy in reconstruction. To address this problem, we introduce a novel procedure to adapt for the forward model mismatches using untrained neural networks actively based on two well-known model-based machine learning architectures, denoted A-adaptive LU and A-adaptive DEQ, Experimental results across both linear and nonlinear inverse problems showcase the effectiveness of the proposed methods in learning forward model updates. Consequently, they surpass the baseline methods significantly when trained as a robust reconstruction mapping to accommodate variations in A. We further demonstrate the robustness of the proposed methods to random initialization of the residual block and a higher number of iterations during evaluation. Finally, we show the eecuracy-runtim tradeoff in handling model mismatch using the proposed variants of the model-based architectures.
Broader Impact Statement
This work adaptively addresses the errors in the forward model by optimizing the auxiliary variable and the untrained mismatch neural network within model-based architecture. We show the flexibility of the pro- posed forward model matching module using two well-known architectures. However, the accuracy-runtime tradeoff still exists. Although the proposed A-adaptive DEQ variant achieves the best performance for all tasks, when solving large-scale inverse problems, careful consideration of runtime is necessary. Algorithms such as momentum methods accelerate the convergence speed of an optimization problem. Investigating more sophisticated design with accelerated algorithms offers an intriguing prospect for future work expedite A-adaptive DEQ.
Acknowledgments
This work was supported by the Center for Energy and Geo Processing (CcGP) at Georgia Toch, the Deanship of Research Oversight and Coordination (DROC), King Fahd University of Petroleum and Minerals (KFUPM), under Project GTEC2013.
12

---

Published in Transactions on Machine Learning Research (04/2024)
References
Jonas Adler and Ozan Öktem. Learned primal-dual reconstruction. IEEE Transactions Medical Imaging, 37(6):132211332, 2018.
Arttu Arjas, Mikko J. Sillanpää, and Andreas Hauptmann. Sequential model correction for nonlinear inverse problems, 2023.
Simon Arridge, Peter Maass, Ozan Öktem, and Carola-Bibiana Schönlicb. Solving inverse problems using data-driven models. Acla Numerica, 28:1-174, 2019.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
Ronny Bergmann, Raymond Chan, Ralf Hielscher, Johannes Persch, and Gabriele Steidl. Restoration of manifold-valued images by half-quadratic minimization. arXiv preprint Arxi::1505.0 07029, 2015.
Stephanic E Blanke, Bernadette N Hahn, and Anne Wald. Inverse problems with inexact forward operator: ilerative regularization and application in dynamic imaging. Ineerse Problems, 36(12):124001, oct 2020. doi: 10.1088/ (e111-662//bbbbab
HanQin Cai, Jialin Liu, and Wotao Yin. Learned robust pea: A scalable deep unfolding approach for high-dimensional outlier detection, 2021.
Jian-Feng Cai, Hui Ji, Chaoqiang Liu, and Zuowei Shen. Blind motion deblurring from a single image using sparse approximation. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 104 111. IEEE, 2009.
Valentina Candiani, Nuutti Hyvönen, Jari P Kaipio, and Ville Kolehmainen. Approximation error method for imaging the human head by electrical impedance tomography. Inverse Problems, 37(12):125008, 2021.
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ista and its practical weights and thresholds. Advances in Neural Information Processing Systems, 31, 2018.
Sunghyun Cho and Seungyong Lee. Fast motion deblurring. In ACM SIGGRAPH Asia 2009 pepers, pp. 1 8. 2009.
II Yong Chun, Zhengyu Huang, Hongki Lim, and Jeffrey A. Fessler. Momentum-net: Fast and convergent iterative neural network for inverse problems. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):4915-4931, April 2023. ISSN 1939-3539. doi:
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schielo. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213 3223, 2016.
Jiangxin Dong, Stefan Roth, and Bernt Schiele. Deep wiener deconvolution: Wiener meets deep learning for image deblurring. Advances in Neural Information Processing Systems, 33:1048 1111048-159, 2020.
Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman. Removing camera shake from single photograph. In Acm Siggroph 2006 Papers, pp. 787-794. 2006.
SW Fung, H Heaton, Li, McKenzie, Osher, and W Yin. Fixed point networks: Implicit depth models with jacobian-froe backprop. arXiv e-print. Xrxiv:1003. 12803, 2021.
D. Geman and Chengda Yang. Nonlincar image recovery with half-quadratic regularization. IEEE Trans- actions on Image Processing, 4(7):932 946, 1995. doi:
Davis Gilton, Gregory Ongie, and Rebecca Willett. Deep equilibrium architectures for inverse problems in 2021a.
imaging. IEFE Transactions Compulational Imaging,
13

---

Published in Transactions on Machine Learning Research (04/2024)
Davis Gilton, Gregory Ongie, and Rebecca Willett. Model adaptation for inverse problems in imaging. IEEE Transactions on Compulational Imaging, 7:661-695 674, 2021b.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Slatislics, volume Proceedings of Machine Leurving Research, pp. 249 256, Chia Laguna Resort, Sardinia, Italy, 13 15 May 2010. PMLR.
Nina M. Gottschling, Vegard Antun, Anders C. Hansen, and Ben Adcock. The troublesome kernel hallucinations, no free lunches and the accuracy-stabile tradc-off in inverse problems, 2023.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on Machine Learning, pp. 399-406, 2010.
Peimeng Guan, Jihui Jin, Justin Romberg, and Mark A Davenport Loop unrolled shallow equilibrium regularizer (Iuser) memory-efficien inverse problem solver. arXiv preprint vrXiu:2210.04980 2022.
Andreas Hauptmann, Ben Cox, Felix Lucka, Nam Huynh, Marta Betcke, Paul Beard, and Simon Arridge, Approximate k-space models and deep learning for fast photoacoustic reconstruction. In Machine Learning for Medical Image Reconstruction: First International Workshop, MLMIR 2018, Held in Conjunction with MICCAT 2018, Granada, Spain, September 16, 2018, Proceedings 1, pp- 103- 111. Springer, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification, 2015.
John R Hershey, Jonathan Le Roux, and Felix Weninger. Deep unfolding: Model-based inspiration of novel deep architectures. arXiv preprinl or 149::1009.255 2014.
Seyed Amir Hossein Hosseini, Burhaneddin Yaman, Steen Moeller, Mingyi Hong, and Mehmet Akcakaya. Dense recurrent neural networks for accelerated mri: History-cognizant unrolling optimization algo- rithms. IEEE Journal of Selected Topics in Signal Processing, 14(6):1280 1291, October 2020. ISSN 1941-0484. doi: 0ttp0//jsssp222.300033333
Junhao Hu, Shirin Shoushtari, Zihao Zou, Jiaming Liu, Zhixin Sun, and Ulugbek S Kamilov. Robustness of deep equilibrium architectures to changes in the measurement model. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Proximal denoiser for convergent plug-and-plas optimization with nonconvex regularization. In International Conference on Machine Learning, pp. 9483 9505. PMLR, 2022.
Naveed Iqbal, Entao Liu, James H. Mcclellan, and Abdullatäf A. Al-Shuhail. Sparse multichannel blind deconvolution of seismic data via spectral projected-gradient. IEEE Access, 7:23740 23751, 2019. doi: 10.1109/ACCESS Bttp://CCEESSS22.28999..0
Naveed Iqbal, Mudassir Masood, Motaz Alfarraj, and Umair Bin Waheed. Deep seismic cs: A deep learning assisted compressive sensing for seismic data. IEEE Transactions on Geoscience and Remote Sensing, 61: 1-9, 2023. doi: 2ttp:/GGRS.200.3222222
Janne Koponen, Timo Lähivaara, Jari Kaipio, and Marko Vauhkonen. Model reduction in acoustic inversion by artificial neural network. The Journal of the Acoustical Society of America, 150(5):3435 11133333444.1 2021.
Anselm Krainovic, Mahdi Soltanolkotabi, and Reinhard Heckel. Learning provably robust estimators for inverse problems via. jittering. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Dilip Krishnan, Terence Tay, and Rob Fergus. Blind deconvolution using a normalized sparsity measure. CVPR 2011, pp. 233 233-240, 2011. doi: (0.1109/CVPR.200 .5995521.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Understanding and evaluating blind deconvolution algorithms. In 2009 IEFE Conference on Computer Vision and Pollern Recognilion, pp.
1964 1971. IEEE, 2009.
Yuelong Li, Mohammad Tofighi, Junyi Geng, Vishal Monga, and Yonina C. Eldar. Efficientt and interpretable deep blind image deblurring via algorithm unrolling. IEEE Transactions Computational Imaging, 6: 666 681, 2020. doi: 21109/TCC12020)
Bo Liu, Qingfu Zhang, and Georges GE Gielen. A gaussian process surrogate model assisted evolution- ary algorithm for medium scale expensive optimization problems. IEEE Transaclions on Evolulionary Computation, 18(2):180-122, 2013.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Suhas Lohit, Dehong Liu, Hassan Mansour, and Petros T. Boufounos. Unrolled projected gradient descent for multi-spectral image fusion. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7722, 2019. doi: /0.1109/ICASSPP20011
Sebastian Lunz, Andreas Hauptmann, Tanja Tarvainen, Carola-Bibiane Schonlieb, and Simon Arridge. On learned operator correction in inverse problems. STAM Journal on Imaging Sciences, 14(1):92-127, 2021.
Magdi S Mahmoud and Haris M Khalid. Expectation maximization approach to data-based fault diagnostics. Information Sciences, 235:80-96 9000099,-40 2013.
Vishal Monga, Yuelong Li, and Yonina C. Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing, 2020.
Wail A Mousa and Abdullatif A Al-Shuhail. Processing of scismic reflection data using ma Synthesis lectures signal processing, 5(1):1-97, 2011.
Yuesong Nan and Hui Ji. Deep learning for handling kernel/modet uncertainty in image deconvolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2388-2397, 2020.
Mila Nikolova and Michael K Ng. Analysis of half-quadratic minimization methods for signal and image recovery. SIAM Journal on Scientific computing, 27(3):937-966) 2005.
Gregory Ongie, Ajil Jalal, Christopher A Metzler, Richard G Baraniuk, Alexandros G Dimakis, and Rebecca Willett. Deep learning tochniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39 56, 2020.
Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in Optimization, 1(3):127-239, 2014. ISSN 2167-3888. doi: 10.1561/24000000
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Zahid Tufail, Khawar Khurshid, Ahmad Salman, and Khurram Khurshid. Optimisation of transmission map for improved image defogging. IET Image Processing, 13()):1161-1169, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018
Singanallur V Venkatakrishhan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference Signal and Information Processing, pp. 945 948. IEEE, 2013.
Homer F Walker and Peng Ni. Anderson acceleaatinn for fixed-point iterations. SIAM Journal on Numerical Analysis, 49(4):1715-15 17755 2011.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Xiaojuan Yang and Li Wang. Fast half-quadratic algorithm for image restoration and reconstruction. Applied Mathemalical Modelling, 50:92-104, 2017.
Yan Yang, Jian Sun, Huibin Li, and Zongben Xu. Deep admm-net for compressive sensing mri. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Yan Yang, Jian Sun, Huibin Li, and Zongben Xu. Admm-csnet: deep learning approach for image compressive sensing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(3)):21-533, 2020. doi: TT000//TAAM..1.....
Ehsan Zabihi Naeini and Mark Sams. Accuracy of wavelets, seismic inversion, and thin-bed resolution. Interpretation, 5(4):T523 1530, 2017.
Meina Zhang, Yingying Fang, Guoxi Ni, and Tieyong Zeng. Pixel screening based intermediate correction for blind deblurring. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 01 09, 2022. doi:
Yanan Zhao, Yuclong Li, Haichuan Zhang, Vishal Monga, and Yonina C. Eldar. A convergent neural network for non-blind image deblurring. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 1505-1509, 2023. doi: 10.1109/ICIP49333 493.//91199999.220000 21t.:.222.....2222.
Yanan Zhao, Yuelong Li, Haichuan Zhang, Vishal Monga, and Yonina C. Eldar. Deep, convergent, unrolled half-quadratic splitting for image deconvolution, 2024.
16

---

Published in Transactions on Machine Learning Research (04/2024)
A Appendix: Training Details
The proximal networks for LU and A-adaptive LU are kept the same for each task, with 5-layer DnCNN for image deblurring and 7-layer DrCNN for landscape defogging. In the forward mismatch network, An( combines with Ao features, processed through 3-layer CNN. Ao features comprise incorrect Gaussian kernel for deblurring and landscape depth profiles. Because the initialization of fo is less important, the parameters are not reinitialized for each training instance for better time efficiency. Both LU and A- adaptive LU employ a maximum of = 5 iterations. The parameeassmeters A-adaptive LU (6) are chosen via sensitivity analysis, where defogging. Learning rates are set at 0.0001 for updating and network paramcters. All networks are trained NVIDIA RTX 3080 with 10GB of RAM.
0.01, 7 = 0.1 for deblurring, and = 0.0001, T = 0.1 for
B Appendix: Effect of Ao in Reconstruction
In addition, we show that our proposed methods are less sensitive the quality of the initial forward model. We demonstrate the idea using the image deblurring task, where one has flexiibility in choosing the initial forward model. Assuming the true forward model for an image deblurring problem has Gaussian kernel size of 5 with a variance of 7, we consider initial forward models Ao constructed with Gaussian kernels of sizes 5, but variances of 1, 3, 5, and 7. Figure
g illustrates the reconstruction results with different A 's for each method. When Au deviates from A, robust LU exhibits noticeable artifacts (with o = and 3), while our proposed methods show minimal degradation ion quality even with significant. differences ween Au and A. This is due to the adaptive nature of our methods in iteratively addressing model mismatches. Notice that a and 3 are treated as extreme cases, which were never encountered in A and Ao during training. The proposed methods generalize better to unseen estimated forward models.
0 1
σ 3
o 5=5
σ = 7
blurry input
Robust
27.54
36:68
35:06
0.961
0.963
10.909
A-adaptive LU
36-33
36.30
38:37
36/37
D.972
0.972
0.973
0.973
GT
A Adadptiiee DEQ,
40:28
40:31
40/32
40/92
0.987
0.987
Figure 9: Visualizing the deblurred images using different Ao, where denotes the variance of the Gaussian kernel of the estimated forward model. The numbers in each image denote the PSNR and SSTM values.
17


Reviews:
[{"review_id": "W4NN0YMv3G", "reviewer": "Reviewer_tfs5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper tackles important scenario of inverse problem beyond linear convolution model, which is an important open question\n- The paper is well-written (with some aspects of clarity to be addressed)\n\nWeaknesses:\n\n(1) Motivation: the central question of the paper needs to be clarified. The paper’s title and abstract are centred around inverse problems in general; however, in the narrative one can see that it only focuses on image data (see, e.g., Figure 1), furthermore, it is not clear whether the authors want to centre the narrative around the problem of nonlinear inverse problems or robustness, therefore the authors should clearly state the benefits of the proposed scheme. \n\n(2) References: Some of the recent approaches such as Zhang et al (2022) need to be cited; furthermore, it is important to show how this work fits into the wider range of inverse problems solving tasks (for images and beyond) such as Chung et al (2022).\n\n(3) Experiments: While the approach permits non-linear models, it would be important to see what is the benefit of the proposed model against the linear counterpart, perhaps by direct comparison on the fog dataset.\n\n(4) Clarity: it would be great to separate the related works section from the introduction. It would be also important to highlight which works have targeted the problem of nonlinear inverse problems. \n\nChung et al (2022) Improving Diffusion Models for Inverse Problems using Manifold Constraints, NeurIPS 2022\n\nZhang et al (2022) Pixel screening based intermediate correction for blind deblurring, CVPR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes two new methods for inverse problem solving based on loop unrolling (LU) and deep equilibrium model (DEQ).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- the motivation statement improvement (see Weakness 1)\n- addressing Weaknesses 2-4 on the references, experiments and clarity\n\nThe authors also need to address the following questions:\n- The Section 5.1 mentions: “The hyper-parameters in A-adaptive LU (6) are chosen via sensitivity analysis, where λ = 0.01, τ = 0.1 for deblurring, and λ = 0.0001, τ = 0.1 for defogging.” Would the authors clarify whether such sensitivity analysis needs to be conducted for every new task? \n- Is there any suggested procedure for designing the initial approximation operator A0, or is it limited to the assumption that it is some prior knowledge beyond the scope of the paper? \n- The comparison is centred around the baseline of the Robust LU which may not be enough to conclude about superior performance; however, it would be interesting to see how the proposed model compares to the state-of-the-art deblurring such as  Zhang et al (2022), or the cited works such as Krainovic (2023)?\n- It would be interesting to see how the choice of A0 impacts the performance, perhaps by varying the operator between the perfect match (A) and random parametrisation.\n- In Table 1, is it possible to provide confidence intervals for the experimental results?\n- Section 5.4 “This result aligns with the finding in Gilton et al. (2021a).” Is it possible to expand upon what was the finding in Gilton et al (2021a)?", "broader_impact_concerns": "The broader impact concerns should address the intersection of the proposed methods and the problems of security such as, e.g.adversarial attacks (see Boloor et al (2019)).\n \nBoloor et al (2019) Simple Physical Adversarial Examples against End-to-End Autonomous Driving Models, Arxiv", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9nV5oT5m3W", "reviewer": "Reviewer_Wz4e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This paper proposes a simple but efficient architecture dubbed \\mathcal{A}-adaptive LU and \\mathcal{A}-adaptive DEQ taking advantage of residual network for solving the forward model mismatch problem. This paper is easy-understanding and straightforward, and it also provides a comprehensive analysis of the proposed model, including stability, random initialization as well as evaluation runtime. \n\nWeaknesses: Some details on the model’s architecture should be provided in case of any confusion, and more experimental comparison with some state-of-art methods also need to be provided. The detailed concerns are listed in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In order to solve the forward model mismatch problem, this paper introduces a novel procedure to adapt for the forward model mismatches using untrained neural networks actively based on two well-known model-based machine learning architectures, denoted as A-adaptive loop unrolling (LU) and A-adaptive deep equilibrium model (DEQ). Theoretically, convergence of A-adaptive DEQ has been proved under mild conditions. Practically, experiments have been conducted to demonstrate a significant quality improvement in removing artifacts and preserving details within three distinct applications, covering both linear and nonlinear inverse problems. In addition, comprehensive analysis has been provided from the perspective of robustness, random initialization as well as evaluation runtime.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are certain details I am concerned about. First of all, I am little bit confused about the f_{\\theta} as a network in the context of learning forward model mismatch in both \\mathcal{A}-adaptive loop unrolling architecture and \\mathcal{A}-adaptive deep equilibrium architecture. It is said that the network f is untrainable however the parameters \\theta in f are updated both during training and evaluation, which causes me a certain confusion on whether the network f is trainable or not. Probably I have a misunderstanding about that, could you please give me more details or specific example on it. Besides, I am also lost F_{\\rho,\\eta} in the section 4 Proposed Method II: \\mathcal{A}-adaptive deep equilibrium architecture, and it is illustrated that \\rho parameterizes the proximal neural network and \\eta denotes the trainable step size, so do you mean the proximal neural network is trainable and how \\rho parameterizes the neural network. Please provide more details about it.\n\nIn terms of the experiments for linear inverse problems, like the image blind deblurring, the forward model is defined by a Gaussian blur kernel with inaccurate knowledge of the variance and size of kernel. Is it possible to use the blur kernel in our real world, for example, like the non-linear kernel generated by recording the camera motion trajectories mentioned in [1]? It seems more meaningful to use the real blur kernel compared with the Gaussian blur one in order to valid the effectiveness of the proposed \\mathcal{A}-adaptive LU and \\mathcal{A}-adaptive DEQ. \n\n[1] Y. Li, M. Tofighi, J. Geng, V. Monga and Y. C. Eldar, ``Efficient and interpretable deep blind image deblurring via algorithm unrolling’’, IEEE Transactions on Computational Imaging, vol. 6, 2020.\n\nMoreover, it seems that no matter \\mathcal{A}-adaptive LU and \\mathcal{A}-adaptive DEQ, it is necessary to know the initial guess of the forward model \\mathcal{A}_{0}. So how do you determine or what do you adopt the \\mathcal{A}_{0}, especially in the defogging case?\n\nLast but not least, I recommend the author to compare his proposed methods (\\mathcal{A}-adaptive LU and \\mathcal{A}-adaptive DEQ) with other latest methods for solving the inverse problems (IP), for example, (1) the Deep Unrolling for Blind Deblurring (DUBLID)[1], the one who unrolled the HQS algorithm with learnable parameters, (2) the Deep Convergent Unrolling for Non-blind deblurring (DECUN)[2] which also provides convergence similar to that of \\mathcal{A}-adaptive DEQ, and (3) Momentum-Net [3] .etc\n\n[1] Y. Li, M. Tofighi, J. Geng, V. Monga and Y. C. Eldar, ``Efficient and interpretable deep blind image deblurring via algorithm unrolling'', IEEE Transactions on Computational Imaging, vol. 6, 2020.\n[2] Y. Zhao, Y. Li, H. Zhang, V. Monga and Y. C. Eldar, ``Deep, convergent, unrolled half-quadratic splitting for image deconvolution'', arXiv preprint arXiv:2402.12872.\n[3] I. Y. Chun, Z. Huang, H. Lim, and J. A. Fessler, “Momentum-Net: Fast and convergent iterative neural network for inverse problems,” IEEE Trans. Pattern Anal. Mach. Intell., 45(5):4915–4931, Apr. 2023. doi: 10.1109/TPAMI.2020.3012955.", "broader_impact_concerns": "The presented work innovatively tackles inaccuracies within the forward model by refining the auxiliary variable alongside an untrained neural network, specifically designed to handle discrepancies, all within a structured model-based framework. This approach demonstrates its versatility by integrating seamlessly with two established architecture types. Nevertheless, the balance between accuracy and computational time remains a challenge. The newly introduced \\mathcal{A}-adaptive DEQ model variant outperforms existing solutions across various tasks. It is crucial to weigh the computational demands, especially for extensive inverse problem-solving scenarios, indicating that achieving higher performance levels may necessitate increased processing time. This careful balancing act underscores the need for a judicious evaluation of runtime implications in large-scale applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pPU3SfUADt", "reviewer": "Reviewer_43k2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nAs far as I read, the proposed technique has no major flaws. The paper is well-written. There is a reasonable amount of numerical evidence to support the effectiveness of the proposed methods.\n\nWeaknesses:\n\nThe image applications tackled by this paper may not attract much interest in the ML community as deep learning methods have extremely strong performance in those CV tasks. However, the base optimization problem may find other interesting applications.It would be interesting if the authors could list some more applications where a mismatched $\\mathcal{A}$ is common.\n\nThe convergence theorem based on monotony objective value decreasing is a weak theoretical result. It would be interesting if the authors could show more useful theoretical insights, such as the convergence rate.\n\nSome numerical experiments feature the $\\mathcal{A}$-adaptive LU, some feature the $\\mathcal{A}$-adaptive DEQ, and the rest feature both. How were those decisions made? In the case both $\\mathcal{A}$-adaptive LU and $\\mathcal{A}$-adaptive DEQ are compared, it seems $\\mathcal{A}$-adaptive DEQ is also significantly better. Does it suggest that proposing only $\\mathcal{A}$-adaptive DEQ is enough for this paper? Is $\\mathcal{A}$-adaptive LU just a step stone here?\n\nSome other deep unrolling methods are not limited to a finite number of iterations, but were missed in the discussion. For example, [R1] proposed an FNN-RNN-mixed unrolling which extended the LU model to infinitely many iterations. This mixed structure has the advantages of both LU and DEQ. The author may find it interesting.\n\n[R1]  Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection. NeruIPS 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the regularized inverse problem with loop unrolling (LU) or deep equilibrium (DEQ) frames. It focuses on the case when there is small mismatching in the sensing matrix $\\mathcal{A}$. Two methods (one for LU and one for DEQ) were proposed to adapt the mismatched $\\mathcal{A}$ and find better solutions than the existing robust loop unrolling method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness.", "broader_impact_concerns": "No Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers model-based deep learning methods (such as loop unrolling (LU) and deep equilibrium model (DEQ)) for solving inverse problems. The authors solve a challenging problem where the forward model mismatch problem and introduce an untrained forward model residual block into the model-based methods, resulting in A-adaptive loop unrolling (LU) and A-adaptive deep equilibrium model (DEQ).  Convergence of A-adaptive DEQ has been proved under mild conditions. Experiments have been conducted to demonstrate a significant quality improvement in removing artifacts and preserving details for both linear and nonlinear inverse problems. 

All the reviewers found that the claims in both theoretical analysis and practical experiments are well supported, illustrating the effectiveness and efficiency of the proposed architecture. They all recommend acceptance or leaning acceptance. I concur with the reviewers' recommendation to accept the paper.
Decision: Accept (poster)