Improved Imaging by Invex Regularizers with Global Optima Guarantees
Authors: Samuel Pinilla
Tingting Mu
Neil Bourne
Jeyan Thiyagalingam
Venue: NeurIPS 2022 Accept (2022)
Keywords: Invexity
Regularizer
Imaging
Global optima
Non-convex optimization
Convexity

Abstract: Image reconstruction enhanced by regularizers, e.g., to enforce sparsity, low rank or smoothness priors on images, has many successful applications in vision tasks such as computer photography, biomedical and spectral imaging. It has been well accepted that non-convex regularizers normally perform better than convex ones in terms of the reconstruction quality. But their convergence analysis is only established to a critical point, rather than the global optima. To mitigate the loss of guarantees for global optima, we propose to apply the concept of invexity and provide the first list of proved invex regularizers for improving image reconstruction. Moreover, we establish convergence guarantees to global optima for various advanced image reconstruction techniques after being improved by such invex regularization. To the best of our knowledge, this is the first practical work applying invex regularization to improve imaging with global optima guarantees. To demonstrate the effectiveness of invex regularization, numerical experiments are conducted for various imaging tasks using benchmark datasets. 

Improved Imaging by Invex Regularizers with Global Optima Guarantees
Samuel Pinilla 1.2, Tingting Mu Neil Bourne", Jeyan Thiyagalingam'
Scientific Computing Department, Science and Technology Facilities Council, Harwell, UK University of Manchester at Harwell, UK
Computer Science, University of Manchester, UK caamuel.pinille t. jeyann@sstfte ac. uk
(tingting. mu, neil. bourne] @manchester ac. uk
Abstract
Image reconstruction enhanced by regularizers, e.g., enforce sparsity, low rank or smoothness priors on images, has many successful applications in vision tasks such computer photography, biomedical and spectral imaging. It has been well accepted that non non-convex regularizers normally perform better than convex ones terms of the reconstruction quality. But their convergence analysis only established to a critical point, rather than the global optima. To mitigate the loss of guarantees for global optima, we propose to apply the concept of invexity and provide the first list of proved invex regularizers for improving image reconstruction, Moreover, we establish convergence guarantees global optima for various advanced image reconstruction techniques after being improved by such invex regularization. To the best of our knowledge, this is the first practical work applying invex regularization to improve imaging with global optima guarantees. To demonstrate the effectiveness of invex regularization, numerical experiments are conducted for various imaging tasks using benchmark datasets.
1 Introduction
Image reconstruction (restoration) enhanced by regularizers has a wide application in vision tasks such computed tomography 21 optical imaging [3] 4 magnetic resonance imaging (5,6), computer photography 7 [) biomedical and spectral imaging 9 I0 In general, an image reconstruction task
can be formulated as the solution of the following optimization problem:
minimize F(x) f(x) + g(x).
Here f(x) models a data fidelity term, which usually corresponds to an error loss for image recon- struction, and assumed to be differentiable. The other function (xx) acts as regularizer which can be non-smooth. It imposes image priors such as sparsity, low rank or smoothness [II. The use of an appropriate regularizer plays an important role in obtaining robust reconstruction results.
because can result in guaranteed global optima. The most well-known examples include the -norr and nuclear norm, which are the continuous and convex surrogates of the é fo-pseudo norm and rank, respectively [16] Although convex regularizers have demonstrated their success signal/image processing, biomedical informatics and computer vision applications [13 17 [8] 19], they are suboptimal in many cases, as they promote sparsity and low rank only under very limited conditions (more measurements from the scene are needed 2 21)). To address such limitations, non-convex regularizers have been proposed.
Convex regularization has been popular in the last decade [1 12
13
14 15
RRtheeforrd Appleton Laboratory.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table Comparison between the assumptions made in this work for f(x)), and g(x) to optimized Eq.
and the most common/suucessful assumptions in the state-of-the-art.
Method name
Assumption
Global optimizer
IRLS [4
special f and 9
No
General descent 33|
No
GIST 37
nonconvex f.g =
92. 91.92 convex
No
iPiano
38
nonconvex J, convex
No
Proposed
convex J. invex
Yes
For instance, several interpolations between the Eo fo-psuudoooom and the -norr have been explored
including the ep-quasinorms (where 0 < p < 1) (22). Capped-é penalty [23]. Log-Sum Penalty [20]. Minimax Concave Penalty [24], Geman Penalty [25]. However, these non non-convex regularizers unfortunately come with the price of losing global optima guarantees.
Image reconstruction methods based on Eq. (T) include model- approaches that directly solve Eq. using well-established optimization techniques, e.g., proximal operators and gradient descent
rules 26 27 28 learning-ban approaches that train an inference neural network [29 30 as well
as hybrid approaches that draw links between iterative signal processing algorithms and the layer-wise neural network architectures [31 32]. Many of these exploit non-convex assumptions over f(x) and/or g(x), for which we present summary of some commonly used or successful ones in Table
The table includes algorithms like the iterative reweighted least squares (IRLS) [33] 34], where the regularizer is composition between the one-dimensional Ly fp-quasinorm and the trace of a matrix.
In 35 36.. the objective function F(x) is assumed to form aali-algebaaic or tame optimization problem solved by gradient descent algorithms. In (37), the regularizer g(x) is assumed to be the subtraction of two convex functions, and the general iterative shrinkage and thresholding (GIST) algorithm is proposed to optimize F Fxx). Lastly, (38) assumes non-convex f(x but convex g(x) and proposes the inertial proximal (iPiano) algorithm for optimization.
For algorithms with the convexity assumptions removed, e.g., those in Table their convergence analysis unfortunately can only be established for a critical point. Ideally, we always prefer algorithms that can find the optimal solution for the target problem. One way to mitigate the loss of guarantees for global optima by revisiting the concept of invexity which was first introduced by Hanson [39], Craven and Glover [40] in the 1980s. What makes this class of functions special is that, for any point where the derivative of a function vanishes (stationary point), is a global minimizer of the function. Convexity is special case of invexity. Since 1990s, a lot of mathematical implications for invex functions have been developed, but with the lack of practical applications [41]. Examples of the few successful works implementing the invexity theory include 442 43 44
To the best of our
knowledge, there is no existing work on the application of invex regularization for imaging.
In this paper, we focus image reconstruction problems formulated in the form of Eq. ( where the data fidelity term f()) is based on the f--norm and an invex regularizer g(x) is used. Most invex theory research lacks clarity how to benefit practical applications, and this does not encourage the
practitioners to exploit the invex property 4T. We aim filling this gap by providing for the first time concrete and useful invex optimization formulations for imaging applications.
Specifically, we make the following contribution:
Provide the first list of regularizers with proved invexity that fits optimization problems for imaging applications.
Establish convergence guarantees global optima for three types of advanced image reconstruction techniques enhanced by invex reguarlizers.
Empirically demonstrate the effectiveness of invex regularization for various imaging tasks.
2 Preliminaries
Throughout this paper, we use boldface lowercase and uppercase letters for vectors and matrices, respectively. The i-th entry of a vector w, is w (i]. For vectors, ||w/|| is the f,-norm. An open ball is defined as B(æ;r) = (y E Ra |/ - æ1122 r). The operation conv(AA) represents the convex hull of the set A. and the operation sign(w) returns the sign of w. We use az (W) to denote the i-th singular value of W assumed in descending order.

---

We present several concepts needed for the development of this paper starting with the definition of locally Lipschitz continuous function.
Definition I (Locally Lipschitz Continuity). A function f I + R is locally Lipschitz continuous point IR if there exist scalars K and > 0 such that
If(y) f(z) - K 2||2,
for y, 2 c).,
Since the ordinary directional derivative being the most important tool in optimization does not necessarily exist for locally Lipschitz continuous functions, it is required to introduce the concept of subdifferential [45] which is calculated in practice as follows.
Theorem 1 (Subdifferentia), [45] Theorem 3.9] Let f R + R be a locally Lipschitz continuous function æ IR", and define 52 {ær E Re| i' not differentiable at the point æ). Then the subdifferential of is given by
(.))) = conv ((S E ""1exiits (æ,) E R" Gy such that æ; + a and Vf(x,) - c})
(3)
The notion of subdifferential is given for locally Lipschitz continuous functions because it is always nonempty 45 Theorem 3.3]. Based on these, the concept of invex function presented as follows.
Definition 2 (Invexity). Let f IR + IR be locally Lipschitz; then f is invex if there exists function 17 : R' E. 1= such that
f(z) - f(y) <" (),
Yz, E R", JA E 0f(y).
It is well known that convex function simply satisfies this definition for nf y)
The following classical theorem 46 Theorem 4.33) makes connection between an invex function and its well-known optimum property that supports the motivation of designing invex regularizers. Theorem 2 (Invex Optimality). [46] Theorem 4.33]) Let f I n R. be locally Lipschitz. Then the following statements are equivalent.
1. f is invex.
2. Every point y E R" that satisfies O E Of( is a global minimizer of f.
3. Definition satisfied for 7) IR" RR. + RM given by
f(x) 2 fly). otherwise,
nT y) f(=)- f(v)
(5)
where с is an element in 0f(y) of minimum norm.
Invex Functions
We start this section by firstly presenting five examples invex functions that are useful for imaging applications. Four of these have been labelled as non-convex in existing works [47 48 This is the
first time that they are formally proved be invex functions. We prove their invexity by showing they satisfy Statement of Theorem (see proof in Appendix Af supplementary material).
Lemma (Invex Functions). All of the following functions are invex:
g(x)=
[[x[i] + e)),, for E (0, 1) and 2 (p(1 - -p)) -p))=->
(6)
g(x)
log(1 [x={)]
|0z [i]
((x)
x
((z)
(9)
+
x
3

---

[2:
( log(1 + [=]]]]
(10)
2
We provide further insights of these functions in Section A.1 of Supplemental material. Table 2
summarizes their applications. Specifically, Eq. (6) known as quasinorm, and has attracted a lot of attention because it has resulted in theoretical improvements for matrix completion and compressive
sensing [22 49.
The analysis on the quasinorms is valid with and without the constant We prefer
to add c in order to formally satisfy the Lipschitz continuity in Definition Eqs. 7 and (8) enhance
the convex é noorr regularizer, and they have significantly improved image denoising
Eq.
has been used as the loss function to improve support vector classification (50)
Table 2: List of invex functions studied in this work.
Reference
Invex function
Application
22
33
51
Eq. (6)
Matrix completion
20
37
53
48
Eq.
Enhancing compressive sensing
47
54.
55
14. (8)
Image denoising
50
Eq. (9)
Support vector classification
Proposed
Eq. (10]
Compressive sensing
We propose the last function in Eq. (10) by the subtraction between Eq. and Eq. (8). This design is motivated by the optimization framework in [37] where the regularization term is assumed to be the subtraction of two convex functions (see GIST in Table D This has been found to be highly successful in imaging applications (see the survey [48). But until now there is no evidence that this subtraction produces another convex function (if exists) potentially useful in imaging applications. Therefore, we propose this example to show that at least this is possible in the invex case.
Additionally, we present another way of constructing an invex function in the following lemma. establishes that an invex function f R -> composed with an affine mapping H b for H E m Ø E Ih" and b E IR. is also invex if H is full row-rank. This condition on H is a mild assumption, because we show in Section $ iimaging application examples that satisfy this criterium.
Lemma 2 (Affine Invex Construction). Let f R. -> R be continuously differentiable invex function, H E Rmxn have full row rank, and E IR be a vector. Then the function h(x)) = f(Hæ b) is invex.
Similar to Lemma [1 it is proved by showing that the composed function satisfies Statement 2 of Theorem 2 (see Appendix of supplementary material). Eq. (9) an example of such an invex construction that satisfies the continuously differentiable assumption in Lemma Z
easily verified its proof in Appendix B practical implication of Lemma for imaging applications appears when we want to solve linear system of equations (e.g. [50). We demonstrate an application of this kind of invex construction in Section 4.2.2 to improve widely used image restoration framework.
4 Invex Imaging Examples, Algorithms and Convergence Analysis
In this section, we demonstrate the use of invex regularizers improve some advanced imaging methodologies. To benefit both practitioners and theory development, we present practical invex imaging algorithms and prove their convergence guarantees to global optima which was only possible for convex functions.
4.1 Image Denoising
Image denoising plays critical role in modern signal processing systems since images are inevitably contaminated by noise during acquisition, compression, and transmission, leading to distortion and loss of image information [56]. Plenty of denoising methods exist, originating from wide range of disciplines such as probability theory, statistics, partial differential equations, linear and nonlinear filtering, spectral and multiresolution analysis, also classical machine learning and deep learning [57 58 56). All these methods rely on some explicit or implicit assumptions about the true (noise-free) signal in order to separate it properly from the random noise.

---

One of the most successful assumptions is that signal can be well approximated by a linear combination of few basis elements in transform domain (59 60]. Under this assumption, denoising method can be implemented as two-step procedure: i) to obtain high-magnitude transform coefficients that convey mostly the true-signal energy, ii) to discard the transform coefficients which are mainly due to noise. Typical choices for the first step are the wavelet, cosine transforms, and principal component analysis (PCA) [59] 60 611. The second step is seen as a filtering procedure that is formally modelled as proximal optimization problem [62]
(g(x)) +
1 ||00
Prox (u) = arg min =ER
(11)
where g(x) acts regularization term, and u represents the noisy transform coefficients. In fact, the usefulness of Eq. (II) is not just limited to denoising, but other imaging problems like computer
tomography [63], optical imaging [64]. biomedical and spectral imaging [65]. In general, global
optima guarantees in Eq. is restricted to convex g(x),, e.g., -norm.
We improve this important proximal operator by incorporating invex regularizers. Specifically, using those invex functions g(x) as listed Table 2 global minimization is achieved in Eq. [T.. The
result is presented in the following theorem:
Theorem (Invex Proximal). Consider the optimization problem in Eq. for all functions in Table 2
Then the following holds:
1. The function h(x) = g(x) + 2 - u1l is convex (therefore invex).
2. The resolvent operator of the proximal is (I 0g) and it is treated as singleton because
it always maps to a global optimizer.
It is classically known that the sum of two invex functions is not necessarily invex in general [46]. Therefore, presenting examples like above, where the sum of fax and g(x) invex, is important to both invexity and imaging communities, We present the proof of Theorem 3 and provide the solution to Eq. II for each function in Table Appendix of supplementary material.
4.2 Image Compressive Sensing
Image compressive sensing has been extensively exploited in areas such as microscopy, holography, optical imaging and spectroscopy [66] 67 68
an inverse problem that aims recovering an image f E R" from its measurement data vector b = & f. where & E R=*x is the image acquisition matrix (m n)). Since m < n, compressive sensing assumes f has a k-sparse representation æ IR" (k < non-zero elements) in basis Vi E R=** that is f = æx in order to ensure uniqueness under some conditions. Examples of this sparse basis V in imaging are the Wavelet (also Haar Wavelet) transform, cosine and Fourier representations (69. Hence, one can work with the abstract model b = + a Ha. where encapsulates the product between $ and V with normmaaled columns [66 70 Under this setup, compressive sensing enables to recover using much lesser samples than what are predicted by the Nyquist criterion [7]. The task formulation is
minimize #ER"
f(x) + >g(x)= = 2
Hæ - b112 >xgxx)
(12)
where À E (0, 1] a typical choice in practice. When the regularizer g(x) takes the convex form of (i-norm, and when the sampling matrix H satisfies the restricted isometry property (RIP) for any
k-sparse vector æ E R", i.e., (1 1= <
| - (1 82x ||>|| for 828
69 Theorem
6.9], it has been proved that z can be exactly recovered by solving Eq.
[71].
We are interested invex regularizers. It has been proved that, when g(x)) takes the particular invex form in Eq. (6), x can be exactly recovered by solving Eq. (12) [49. Below we further generalize this result to all the invex functions as listed in Table The generalized result is presented in Theorem 4
Theorem 4 (Invex Image Compressive Sensing). Assume H = b. where
R" is k-sparse, the matrix H E x (m < n)) with fz-normalized columns that satisfies the RIP condition for any k-sparse vector, and b E I is noiscless measurement vector. If g( (z) in Eq. (2) takes the form of
the functions in Table 2 then the following holds:
1. The objective function $
H b12 + >g(æ)) is invex.
5

---

2. a can be exactly recovered by solving Eq. (12 i.e. only global optimizers exist. When g takes the form of Eq. (9) extra mild conditions on x are needed.
We clarify that if H satisfies the mentioned RIP. then each sub-matrix with k-columns of H, selected according to indices of the nonzero elements of the k-sparse signal is full row-rank matrix. This result important invex community, because supports the validity of Lemma (2 build invex functions with affine mappings. Additionally, we present another proved form of function sum that can result in an invex function, i.e., the sum of g(x) and the 22-norm composed with the affine mapping H - b. The complete proof is provided in Appendix E of supplementary material.
Next, we present different algorithms solve Eq. 122 using invex g(xr) in Table 2 We select few of the most important and successful image reconstruction techniques to start from, and develop their invex extensions. Taking advantage of the invex property, we prove convergence to global minimizers for each extended algorithm, which is unexplored up to date.
Algorithm Accelerated Proximal Gradient
1: input: Tolerance constant E (0, 1), initial point xz and number of iterations T.
2: initialize: xz! (0 z (0), 1, 0, an 02 < 1. and X E (0, 1]
3: for = too do
4: y()) = (t) 1_ (2 ()) + "2--((200) - xxll111)
5:
2(++1) = -prox 02 An
(1) 02 v2fl()))
6:
((+1) -prox
(t) a1 (f((())))
7: T1+1 +1(
2 2(++1),
if f(((t))) + )g(2(++))
( f(o(tt!)) *g(
8: (1+1)
(((-1),
otherwise
9: end for return: (T)
4.2.1 Accelerated Proximal Gradient Algorithm
The accelerated proximal gradient (APG) method [72 has been shown to be effective solving Eq.
(1). achieving better imaging quality in less iterations than its predecessors 13
36
37
38
73 and
been frequently used by recent imaging works (55 74 75 76.. Its convergence to global optima
only guaranteed for convex loss [72]. For non-convex cases, convergence to a critical point has been
stated [72]. Its pseudo-code for solving Eq. [2 is provided in Algorithm [
Taking advantage that the loss function f(x) + >g(x) in Eq. (12) is invex, and the uniqueness result in Theorem 3 we formally extend APG in the following lemma stating that the sequence { (x+(++))
generated by Algorithm converges a global minimizer of Eq. (12).
Lemma 3 (Invex APG). Under the setup of Theorem 4 and using L = 01 (H"H) (maximum
singular value), the sequence (x)
T-1 It=
generated by Algorithm converges to a global minimizer.
To prove Lemma we apply the Statement 2 of Theorem 2 to Eq. 12 and the unicity of the proximal
operators for functions in Table
Z The proof is provided in Appendix of supplementary material.
4.2.2 Plug-and-play with Deep Denoiser Prior
Plug-and-play (PnP) is a powerful framework for regularizing imaging inverse problems (65 and has gained popularity in a range of applications in the context of imaging inverse problems 22 65 77
79] It replaces the proximal operator in an iterative algorithm with an image denoiser, which
does not necessarily have a corresponding regularization objective. This implics that the effectiveness
of PnP goes beyond standard proximal algorithms such as primal-dual splitting [80] 81 82] It has
guarantees to a fixed point only when convex objective functions are employed (81).
To apply the PnP framework, we modify Algorithm T by replacing the proximal operator (Line 6 in its pseudo-code) with neural network based denoiser Noise2 Void (58), resulting in
D (++1) Void
(0) an a. (z)
(13)

---

The complete pseudo-code is presented in Algorithm B of Appendix G in supplemental material. We remark that in Algorithm 3] Line 5 of the Algorithm [ is retained to allows the comparison between regularizers (invex and convex). More specifically, Line 5 computes the proximal step, while Line 6 relies on neural network for the same purpose (3). This offers an avenue for simultaneously exploiting both the model-based and data-driven approaches. The output of Algorithm 3 is a close estimation to the solution of Eq. (12) [1]. The benefit of using this denoiser is that it does not require clean target images in order to be trained. We present the following convergence result for this modified algorithm under the assumption of [(x) in Eq. (12 being invex which is generalization of [8I] (restricted to convex functions only).
Lemma 4 (Invex Plug-and-play). Assume f(x:) in Eq. 12) is invex with Lipschitz continuous gradient, and denoiser d _ - R. Under the setup of Theorem 4 and some mild conditions on d, the sequence {x(() O generated by Algorithm satisfics
T
(1) (x01) Vs (2 (x(())
2 7
|00 1=(00)x''1 -x01
(14)
T
for any x=" d(x= an V)()))) (fixed point) and for some a E (0, 1).
Eq. guarantees that (z((1) is arbitrarily close to the set of fixed points of d(-), which is considered close estimation to the solution of Eq. (12) [T] Its proof is provided in Appendix of
supplementary material. Eq. (9) is an example satisfying assumption required in Lemma
4.2.3 Unrolling
The unrolling unfolding framework is another imaging strategy for solving Eq. It offers
systematic connection between iterative algorithms used in signal processing and the neural networks
31
32 33
Unrolled neural networks become popular due to their potential in developing efficient
and high-performing network architectures from reasonably sized training sets [4] 85 A folded
version of the proximal gradient algorithm is presented in Algorithm 2 Particularly, existing works
86 87 have shown that the efficiency of Algorithm can be improved by simulating a recurrent neural network that its layers mimic the iterations in Line 4 of Algorithm 2 Specifically, each x (f=1) constitutes
linear operation which models layer of the network, followed by proximal operation that models the activation function. Thus, one forms a deep network by mapping each iteration to a network layer and stacking the layers together to learn H, at and (!) for which is equivalent to executing iteration of Algorithm 2 multiple times. Their study was conducted only for g(x) in the form -norm.
Algorithm 2 Folded Proximal Gradient Algorithm
1: input: initial point x (0), number of iterations T 2 2: initialize: a₂ < T.42. and X E (0,
3: for t -0 T do
4: (+++1) prox n. Ag (x - ai H (Hx( - b))
5: end for
6: return: (7)
Convergence guarantees to global optima for Algorithm 2 has been established in 13 but it is restricted to convex objective functions. Therefore, due to the success and importance of unrolling aim to extend the global optima guarantees of Algorithm 2 to invex objectives, and present the results in the following lemma:
Lemma (Invex Unrolling). Under the setup of Theorem and using L = (H"H) (maximum
singular value) and ai < 2 the sequence (() T-1 generated by Algorithm converges to global minimizer.
T-1
==
The key to proving Lemma 5 relies on the uniqueness result the proximal operator for functions in Table as stated in Theorem 3 The proof is presented in Appendix of supplementary material. Such results confirm that the invex unrolled network of Algorithm 2 which uses the proximal operators of invex mappings as the activation functions, can reach the optimal solution during training.
7

---

5 Experiments and Results
A number of datasets have been merged to formulate one unique dataset for our training and evaluation purposes. These are DIV2K super-resolution [88) the McMaster [89]. Kodak [90]. Berkeley Segmentation (BSDS 500) 1911, Tampere Images (TID2013) [92] and the Color BSD68 [93] datasets. We conduct various experiments to study the performance of those invex regularizers as listed in Yable2jin in non-ideal conditions. We compare them against the state-of-the- the-art methods originally developed for convex regularizers (f -norm) ensuring global optima. When neural network training is involved, we take a total of 900 images which are randomly divided into a training set of 800 images, validation set of 55 images, and test set of 45 images, For all the experiments, the images are scaled into the range [0, 1]. For the invex regularizer in Eq. (6) we vary the value of p.
5.1 Image Compressive Sensing Experiments
We assess signal reconstruction, in these experiments, by averaging the saai---iigna-t--onii-----e- (PSNR) in dB over the testing image set. We consider additive white Gaussian noise in the mea- surements data vector with three different levels of SNR (Signal-to-Noise Ratio) = 20, 30, and co (noiseless case). For Algorithm and its plug-and-play variant, the parameters X, 01, and 02 were chosen to be the best for each analyzed function determined by cross validation, and the initial point (0)) was the blurred image b. The results are summarized in Table where the best and least efficient among invex functions is highlighted in boldface and underscore, respectively. Additional results are reported in Appendix of supplemental material for each experiment, using the structural similarity index measure to assess imaging quality.
Experiment 1 studies the effect of different invex regularizers, the Smoothly Clipped Absolute Deviation (SCAD) 194], and the Minimax Concave Penalty (MCP) (24), under Algorithm I A deconvolution problem is studied to formulate Eq. (12) which is an important problem in signal processing due to imperfect artefacts in physical setups such as mismatch, calibration errors, and loss of contrast (95] To compare, the used state-of-the-art methods that employ convex regularization are the Total Variation Minimization by Augmented Lagrangian (TVAL3) [96), and the fast iterative trrinkage-thresholdee
algorithm (FISTA) 133 which ensures global optima. Further, to comparing with convolutional neural networks methodologies, the non-iterative reconstruction methodology ReconNet [97] is used. To model this problem, all pixels of the testing set are fixed to 256 x 256 pixels. The images went through Gaussian blur of size 9 x 9 and standard deviation 4, followed by an additive zero-mean white Gaussian noise. The sensing matrix H is built as H = & v (for all methods except ReconNet), where & represents the blur operator over the images and V is the inverse of three stage Haar wavelet transform. This experiment is extremely ill-conditioned, where the condition number of H" T H is significantly higher than 1. This means that in practice the RIP condition is not guaranteed. To achieve fair comparison, the number of iterations was fixed for all functions as T = 800. The deconvolution problem follows a compressive sensing setup because the Gaussian filter remove high frequency information of the input image.
In the case of ReconNet, we follow existing setting in 97.. For the learning of ReconNet, we extract patches of size 33 x 33 from the noisy blurred training image set, and we train it using the Adam optimization algorithm and a learning rate x 10-4 for 512 epochs with a batch size of 128.
Experiment 2 studies the invex regularizers under the allg-and-plan modification of Algorithm I as described in Section 4.2.2 (58). The same deconvolution problem as in Experiment is used. The interesting aspect of this scenario is that Algorithm| Thas proximal step in Line 5 that allows to compare between regularizers (invex and convex) while using neural networks in Line (see Algorithm 3 in Appendix of Supplemental material). Noise2 Void is trained by randomly extracting patches of size 64 64 pixels from the training images where zero-mean white Gaussian noise was added for SN R = 20, 30dB. Data augmentation on the training dataset is used, by rotating each image three times by 90 and also added all mirrored versions. The learning rate is fixed as 0.0004.
Experiment 3 compares the invex regularizers but under the unrolling framework as described in Section 4.2.3 The gold standard convex regularizations to compare with are the learned iterative shrinkage and thresholding algorithm (LISTA) (87). and the Interpretable
cemmiizationnnsppiee deep network (ISTA-Net) 98. Also, to comparing with convolutional neural networks methodologies.
We used Noid implementation http:////gghhhuubco//oggeee
8

---

Table Performance comparison, in terms of PSNR (dB), where the best and least efficient among invex functions is highlighted in boldface and underscore, respectively.
(Experiment 1) Alguritam
P
0.5 for
Eq.
FISTA 13
ReconNet [
TVAL3 [6]
SCAD [04]
MCP 24
SNR
Eq.
Eq: F2
Eq. 8
Eq.
9
Eq. [10]
f Fy-oorm
33.40
31.25
31.93
30.00
32.65
29.97
27.01
28.77
30.55
31.30
20dB
24.60
22.83
23.39
22.00
23.98
21.80
19.99
20.49
22.60
23.01
300B
27.61
26.56
26.90
26.00
27.25
24.91
22.01
23,99
26.10
26.77
(Experiment
2) Algorithm
p= 0.8 for Eq.
[
(Dencising experiment) Algorithm
BM3D [59]
Noid [S8]
SNR
Eq:
[1]
Eq. [
Eq: 8
Eq.
[
Eq.
[[0]
F fy-norm
0.5 for Eq. 5
34.51
32.37
33.06
31.20
33.76
31.10
Metric
Fqg. 6
Fqg
[10
fy-norm
20dB
25.55
23.92
24.44
23.00
24.98
22.95
SNR (dB)
49.40
43.85
46.46
41.52
39.43
30dB
28.30
26.87
27.33
26.05
27,90
26.00
SSIM
0.886
0,872
0.876
0.869
0,853
(hepriisen 1) Algovihen
InfstroeLISTI p 0.55 for
(Esperiment 33
Algonihmm unfalded ISTA--Net p 0.85 for Fu
SNR
SNK m//a
10
150
[98]
30.55
76.59
32.50
30.15
30.89
25.04
31.07
11.6
36.10
35.50
56.54
32.15
35.30
11.01
31.86
20
0.4
3M.33
35.72
36.55
$4.92
57.41
34,17
43.61
40.07
41.15
30.02
42.35
38.02
11.3
26,00
25:45
26.94
25.M5
25.01
23.52
23:00
0.2
28.29
26:22
2687
25,02
27.55
2501
204B
32.29
26:24
203B
0.4
33.96
32.01
066
34.38
35.00
$8:28
32.94
33.54
32.60
30.30
0.6
36.77
34.68
35.03
34.33
35,39
$3.99
37:65
26.20
26.06
26.16
27.15
25.32
23.54
29.34
28.30
28.95
27.65
114
34.35
$1.59
32.66
11.12
33.47
30.46
29.58
20dB
0.4
35.41
33.33
$3.99
32.09
34.68
$2.08
35.54
36.25
38.00
34.65
the non-iterative reconstruction methodology ReconNet 997 is used. We follow the existing setting for LISTA in 86 and for ISTA-Net in 887. For the training stage we extract 10000 patches
b
E
R. 16 16
at random positions of each image, with all means removed. We then learn a dictionary
D
DED I 25612x12 from the extracted patches, using the same strategy as in [86] Gaussian i.i.d sensing
NO 1/m) and then normalize its columns to have the unit fn-nomm, where m is selected such that 256 = 0.2, 0.4, 0.6. The matrix H is built as H = + q with T = 16 (number of layers). We follow the same two-step strategy in [86] to train a recurrent neural network. First, perform a layer-wise pre-training solving Eq. vr. Second, append a learnable fully-connected layer at the end of the network structure, initialized by V.. Then, perform
matrices + R=m2250 are created from the standard Gaussian distribution, $ $
(12) for each extracted patch b by fixing
end-to-end training solving Eq. (2) where H in this case is learnt by updating the initial matrix V. For each testing image, we divide it into non non-overlapping 16 x 16 patches. When g(x) is the the fnormm we recover [86].
In the case of ISTA-Net, and ReconNet, for their learning stage we extract patches from the training image set of size 33 x 33. Gaussian i.i.d sensing matrices +b E m Posh
are created with ég- normalized columns as for LISTA, where m is selected such that 1089 0.2, 0.4, 0.6. The optimizer 10-1 for 200 and 512 epochs for ISTA-Net
employed was Adam algorithm and learning rate and ReconNet respectively, with batch size of 64 for both networks. For ISTA-Net T = 16 (number of unrolled iterations). We recall that when g(x) is the the fn-mr in ISTA-Net, we recover (98].
5.2 Image Denoising Experiment
Two image datasets, which we merge (80 images in total), are used for this experiment comes from a neutron image formation phenomenor[] [l These type of images contain the neutron attenuation properties of the object which helps analyze material structure. Performance is assessed by averaging
along all the images the experimental SNR in dB given by SN 20 log Izlla where and &
[
stand for the noisy and the denoised image, respectively, and the structural similarity index measure (SSIM) computed between z and ě. Taking advantage of results observed from previous experiments, we compare the top three regularizers in Eqs. (6). (8), and (TO) with two state-of-the-art denoising techniques including the block block-matching and 3-D filtering (BM3D) (59] using nn-ormm regularizer and the deep learning technique Void (trained as in Experiment 2) (58). We follow the two-step denoising procedure described in Section 4.1 In the first step, the transform domain is built using
We used the implementation from 86 at httph: //github//ttbbbte/ httpi/ttub.coom////e / WwT.TTA-G.//LLSTAAA---O CPSS
'Acquired with the ISIS Neutron and Moun Source system Harwell Science and Innovation Campus.

---

PCA as in [6]]. To build this transform we extract patches of 16 x 16 from the noisy image that are then used to adaptively construct a tight frame (nearly orthogonal matrix) tailored to the given noisy data 5 Results are summarized in 3 We report examples of denoised images obtained by Eqs. (6). S). (TOp, BM3D, and Noise2 are illustrated in Appendix of supplementary material, along with the algorithm used for the invex regularizers to denoise these images.
6 Discussion, Limitations and Conclusion
Application advancement of invex theory has paused for decades due to the lack of practical examples, which has caused significantly reduced interest in invexity research. To address this issue, we present for the first time a list of invex regularizers for image reconstruction applications, and formulate corresponding optimization problems. Particularly, for image compressive sensing, we improve three advanced imaging techniques using the listed functions in Table 2|as as invex regularizers. We present their solution algorithms and develop theoretical guarantees on their convergence to global minimum. We also conducted various image compressive sensing and denoising experiments to demonstrate the effectiveness of invex regularizers under practical scenarios that are non-ideal with noisy data observed and RIP condition not guaranteed. Significant benefit of using invex regularizers have been proved from both theoretical and empirical aspects. In fact, Table and theoretical results in Section 4 revive the potential of exploring invex theory in practical applications.
The numerical results presented in Table 3 confirm performance improvement by using invex regular- izers over the & fy-norm-base based methods (e.g FISTA, TVAL3) in unexplored scenarios. These tables and theoretical results in Section D revive the potential of exploring invex theory in practical applications. The best result is obtained with Eq. (6. and Eq. ( is the least efficient. The intuition behind the superiority of Eq. (6) comes from the possibility of adjusting the value of p in data-dependent manner [49]. This means that when the images are strictly sparse, and the noise is relatively low, a small value of p should be used. Conversely, when images are non-strictly sparse and/or the noise is relatively high, a larger value of p tend to yield better performance (which seems to be the case for the selected image datasets). We believe that the remaining invex, SCAD, and MCP regularizers have lesser performance than Eq. (6) as they do not have the flexibility of adjustment to the sparsity of the data, In fact, Eq. ( shows the poorest performance because in the proof of Theorem 4 we theoretically guarantee that Eq. (9) cannot sparsify all images. Therefore, this analysis leads to the conclusion
that the invex function Eq. (6) offers the best performance for the metrics concerns and the imaging problems studied here.
Although, we have presented theoretical results with global optima using invexity for some of most important and successful image reconstruction techniques, we highlight several limitations of our analysis. Specifically, we focused on reconstructed the image of interest an ideal scenario, that is, without the present of noise (Theorem 4). Additionally, we have limited our numerical results to tasks like denoising, and deconvolution. And, the convergence guarantees for the plug-and-play result only ensures close estimate of the solution (Lemma D. Therefore, we see there are a number of future directions this research can be taken further improving the results even further. One aspect is to explore avenues for improving convergence guarantees to global optima the plug-and-play framework. Another direction the study of inclusion of noise in the analysis of imaging applications, which may be an enabler to improve downstream tasks like invex robust image reconstruction. Finally, we feel that the application domains for invex functions can go well beyond denoising, and deconvolution imaging problems, especially around deep learning research, which can improved number of downstream applications.
Broader Impact
We believe that the presented mathematical and empirical analysis over the studied regularizers has the potential to unlock the benefits of invexity for further applications in signal and image processing. This may be an enabler to improve downstream tasks like deep learning for imaging, and to provide more robust image reconstruction algorithms.
Acknowledgments
This work was partially supported by the Facilities Funding from Science and Technology Facilities Council (STFC) of UKRI, and Wave the UKRI Strategic Priorities Fund under the EPSRC grant EP/T001569/1, particularly the "AI for Science" theme within that grant, by the Alan Turing Institute.
'ee used implementation at https:tps
hkust. edu. hh
10

---

References
[1] Emil Y Sidky, Jakob H Jørgensen, and Xiaochuan Pan. Convex optimization problem prototyping for image reconstruction in computed tomography with the chambolle- pecboolll--ekkkooooo algorithm. Physics in Medicine Biology, 57610)33065, 2012.
[2] Haibo Zhang, Linqi Hai, Jiaojiao Kou, Yuqing Hou, Xiaowei He, Mingquan Zhou, and Guohua Geng. OPK SNCA: Optimized prior knowledge via sparse non-convex approach for cone-beam x-ray lumines- cence computed tomography imaging. Computer Methods and Programs in Biomedicine, page 106645, 2022.
[3] Tim Meinhardt, Michael Moller, Caner Hazirbas, and Daniel Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In Proceedings of the IEEE International Conference on Computer Vision, pages 1781-1790) 2017.
[4] Manya V Afonso, José M Bioucas- Diass, and Mário AT Figuciredo. Fast image recovery using variable splitting and constrained optimization. JEEE transactions on image processing, 19(9):2345 (99):2345-2336 2010.
[5] Matthias J Ehrhardt and Marta M Betcke. Multicontrast mri reconstruction with structure-guide total variation. SIAM Journal on Imaging Sciences, 93)::084-1106,, 2016.
[6] Jeffiey A Fessler. Optimization methods for magnetic resonance image reconstruction: Key models and optimization algorithms. IEEE signal processing magazine, 37(1):33 40, 2020.
[7] Seyyed Reza Miri Rostami, Samuel Pinilla, Igor Shevkunov, Vladimir Katkovnik, and Karen Egiazar- ian. Power-balanced hybrid optics boosted design for achromatic extended depth-of-field imaging via optimized mixed OTF. Applied Optics, 60(30))9355-9378 2021.
[8] Felix Heide, Mushfiqur Rouf, Matthias B Hullin, Bjom Labitzke, Wolfgang Heidrich, and Andreas Kolb, High-quality computational imaging through simple lenses. ACM Transactions on Graphics (TOG), 32(5):1-14, 2013.
[9] Lizhi Wang. Chen Sun, Ying Fu, Min H Kim, and Hua Huang. Hyperspectral image reconstruction using a deep spatial-spectral prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2019.
[10] Cishen Zhang and Ifat-Al Baqee Parallel magnetic resonance imaging reconstruction by convex opti- mization. In Third International Conference on Innovative Computing Technology (INTECH 2013), pages 473-478. IEFF, 2013.
[11] Vishal Monga. Handbook of Convex Optimization Methods in Imaging Science, volume 1. Springer, 2017.
[12] Yuli Sun, Hao Chen, Jinxu Tao, and Lin Lei. Computed tomography image reconstruction from few views via log-norm total variation minimization. Digital Signal Processing, 88:172-181, 2019.
[13] Amir Beck and Mare Teboulle. A iterative shrinkage-thresholdiee thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183-200) 2009.
[14] Yunsong Liu, Zhifang Zhan, Jian- Fen- Cai, Di Guo, Zhong Chen, and Xiaobo Qu. Projected iterative soft-thresholding al algorithm for tight frames in compressed sensing magnetic resonance imaging. IEEE transactions on medical imaging, 35(9):21 110-2140,, 2016.
[15] Fernando Soldevila, Clemente, Enrique Tajahuerce, Patte-Patarroyo, Pedro Andrés, and Jesús Lancis. Computational imaging with a balanced detector. Scientific Reports, 6(1):1-10, 2016.
[16] Yun Fu, Low-rank and sparse modeling for visual analysis. Springer, 2014.
[17] Shirish Krishnaj Shevade and S Sathiya Keerthi. A simple and efficient algorithm for gene selection using sparse logistic regression. Bioinformatics, 19(1)):22662253, 2003.
[18] John Wright, Allen Y Yang, Arvind Ganesh, S Shankar Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE transactions on pattern analysis and machine intelligence, 31(2):210-20 2008.
[19] Jieping Ye and Jun Liu. Sparse methods for biomedical data. ACM Sigkdd Explorations Newsletier. 14(1):4-15 15, 2012.
[20] Emmanuel J Candes, Michael B Wakin, and Stephen P Boyd. Enhancing sparsity by reweighted 4 minimization, Journal of Fourier analysis and applications, 14(5):877-94) 9057, 2008.
11

---

[21] Tong Zhang. Analysis of multi-stage convex relaxation for sparse regularization. Journal of Machine Learning Research, 11(3), 2010.
[22] Goran Marjanovic and Victor Solo. On &g optimization and matrix completion. IEEE Transactions on signal processing, 60(1 1):5714-5722 2012.
[233] Zhihua Zhang, James T Kwok, and Dit- Yan Yeung. Surrogate maximizaaioo//iiniiiaaatooo algorithms and extensions. Machine Learning, ):-33,,,,, 2007.
[24] Cun-Hui Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of statistics, 38(2):894-944 942, 2010.
[25] Donald Geman and Chengda Yang. Nonlinear image recovery with half-quadratic regularization. IEFE transactions on Image Processing, 4(7):932-946, 1995.
[26] Amir Beck. First-orler methods in optimization. SIAM, 2017.
[27] Jikai Jin, Bohang Zhang, Haiyang Wang, and Liwei Wang. Non-convex distributionally robust optimiza- tion: Non-asymptotic analysis. Advances in Neural Information Processing Systems, 34, 2021.
[28] Stefano Sarao Mannelli and Pierfrancesco Urbani. Analytical study of momentum-basee acceleration methods paradigmatic high-dimensional non-convex problems. Advances in Neural Information Processing Systems, 34, 2021.
[29] Kai Zhang. Yawei Li. Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and aong-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.
[30] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[31] Samuel Pinilla, Kumar Vijay Mishra, Igor Shevkunov. Mojtaba Soltanalian, Vladimir Katkovnik, and Karen Egiazarian. Unfolding-aide bootstrapped phase retrieval in optical imaging. arXiv preprint arXiv:2203.016695, 2022.
[32] Vishal Monga, Yuelong Li, and Yonina Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(())18-44, 2021.
[33] Karthik Mohan and Maryam Fazel. Iterative reweighted algorithms for matrix rank minimization. The Journal of Machine Learning Research, 13(1):3441 3473,, 2012.
[34] Peter Ochs, Alexey Dosovitskiy, Thomas Brox, and Thomas Pock. On itcratively reweighted algorithms for nonsmooth nonconvex optimization in computer vision. SIAM Journal on Imaging Sciences, 8(1 3(31: 372, 2015.
[35] Hedy Attouch, Jérôme Bolte, and Benar Fux Svaiter. Convergence of descent methods for semi alge-alge and tame problems: proximal algorithms. forward backward splitting, and regularized seisel methods. Mathematical Programming. 137(1):91-122) 2013.
[36] Pierre Frankel, Guillaume Garrigos, and Juan Peypouquet. Splitting methods with variable metric for kurjseaatojasiewwiez functions and general convergence rates. Journal of Optimization Theory and Applications, 165(3):874-900, 2015.
[37] Pinghua Gong, Changshui Zhang, Zhaosong Lu, Jianhua Huang, and Jieping Ye. A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems. international conference on machine learning, pages 37 45. PMLR, 2013.
[38] Peter Ochs, Yunjin Chen, Thomas Brox, and Thomas Pock. ipiano: Inertial proximal algorithm for nonconvex optimization. SIAM Journal on Imaging Sciences, 7(2):1388-1419 2014.
[39] Morgan A Hanson. On sufficiency of the kuhn-tucker conditions. Journal Mathematical analysis and applications, 80(2):545.. 550, 1981.
[40] Bruce D Craven and Barney M Glover. Invex functions and duality, Journal of the Australian Mathematical Society, 39(1):1-20 1985.
[41] Constantin Zălinescu. A critical view invexity. Journal of Optimization Theory and Applications. 162(3))695-704, 2014.
[42] Adarsh Barik and Jean Honorio. Fair sparse regression with clustering: An invex relaxation for combinatorial problem. Advances in Neural Information Processing Systems, 34, 2021.
12

---

[43] Mujahid Syed, Panos Pardalos, and Jose Principe. Invexity of the minimum enror entropy criterion. IEEE 2013.
Signal Processing Letters,
[44] Badong Chen, Lei Xing. Haiquan Zhao, Nanning Zheng, José Pri, et al. Generalized correntropy for robust adaptive filtering. IEEE Transactions on Signal Processing, 64(13):3376 3387, 2016.
[45] Adil Bagirov, Napsu Karmitsa, and Marko M Mäkelä. Introduction to Nonsmooth Optimization: theory, practice and software. Springer. 2014.
[46] Shashi Mishra and Giorgio Giorgi. Invexity and optimization, volume 88, Springer Science Business Media, 2008.
[47] Yongjun Wu, Guangjun Gao, and Can Cui. Improved wavelet denoising by non-convex sparse regulariza- tion under double wavelet domains. IEEE Access,
2019.
[48] Fei Wen. Lei Chu. Peilin Liu, and Robert C Qiu. A survey on nonconvex regularization bassd sparse and low-rank recovery in signal processing, statistics, and machine learning. IEEE Access, 2018.
[49] Rui Wu and Di-Rong Chen. The improved bounds of restricted isometry constant for recovery via & minimization. IEEE transactions information theory, (199))612-61477 2013.
[50] Zhenxun Zhuang, Ashok Cutkosky, and Francesco Orabona. Surrogate losses for online learning of stepsizes in stochastic non-convex optimization. In International Conference on Machine Learning, pages 7664 7672. PMLR, 2019.
[51] Haibo Zhang. Guohua Geng, Shunli Zhang, Kang Cheng Liu, Yuqing Hou, and Xiaowei He. Sparse non-convex tp regularization for cone-heam x-ray luminescence computed tomography. Journal of Modern Optics, 65(20))2278-2288, 2018.
[52] Fan Lin, Yingpin Chen, Lingzhi Wang, Yuqun Chen, Wei Zhu, and Fei Yu. An efficient image recon- struction framework using total variation regularization with Ep qu-quusmo and group gradient sparsity. Information, 10(3):1 15, 2019.
[53] Quamming Yao and James Kwok. Efficient learning with a family of nonconvex regularizers by redis- tributing nonconvexity. In International Conference on Machine Learning, pages 2645-2654. PMLR, 2016.
[54] Zhanxuan Hu, Feiping Nie. Rong Wang. and Xuelong Li. Low rank regularization: A review. Neural Networks, 136:218-232. 2021.
[55] Weina Wang and Yunmei Chen. An accelerated smoothing gradient method for nonconvex nonsmooth minimization in image processing. Journal of Scientific Computing, 90(1):1-28, 2022.
[56] Linwei Fan, Fan Zhang, Hui Fan, and Caiming Zhang. Brief review image denoising techniques. Visual Computing for Industry, Biomedicine, and Art, 2(1):1 12, 2019.
[57] Mona Mahmoudi and Guillermo Sapiro. Fast image and video denoising via nonlocal means of similar neighborhoods. IEEE signal processing letters. 12(12):839 842, 2005.
[58] Alexander Krull, Tim-Oliver Buchholz, Florian Jug. Noise2voiid-earming denoising from single noisy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[59] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d connsform-dmmatie collaborative filtering. IEEE Transactions on image processing, 16(8):2080-20) 2007.
[60] Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions Image processing, 15(12)3373-377 3745, 2006.
[61] Jian-Feng Cai, Hui Ji, Zuowei Shen, and Gui-Bo Ye. Data-driven tight frame construction and image denoising. Applied and Computational Harmonic Analysis, 37(1):89 105, 2014.
[62] Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in optinization, 1(3):177- 239, 2014.
13

---

[63] Jakob S Jørgensen, Evelina Ametova, Genoveva Burca, Gemma Fardell, Evangelos Papoutsellis, Edoardo Pasca, Kris Thielemans, Martin Tumer, Ryan Warr, William RB Lionheart, et al. Core imaging library-pan part i: versatile python framework for tomographic imaging. Philosophical Trunsactions of the Royal Society
A, 379(2204)20220112,, 2021.
[64] Yu Sun, Shiqi Xu, Yunzhe Li, Lei Tian, Brendt Wohlberg, and Ulugbek Kamilov. Regularized fourier ptychography using an online plug-and-play algorithm. In ICASSP 20/9-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7665-7669. IEFF, 2019.
[65] Yu Sun, Brendt Wohlberg, and Ulugbek S Kamilov. An online plug-and-play algorithm for regularized image reconstruction. IEEE Transactions Computational Imaging. 5(3):395 408, 2019.
[66] Gonzalo R Arce, David J Brady, Lawrence Carin, Henry Arguello, and David S Kittle. Compressive coded aperture spectral imaging: An introduction. IEEE Signal Processing Magazine, 31(1):105-115, 2013.
[67] Andrés Jerez, Samuel Pinilla, and Henry Arguello. Fast target detection via template matching in compressive phase retrieval. IEEE Transactions on Computational Imaging, 6:934 944, 2020.
[68] Andrés Guerrero, Samuel Pinilla, and Henry Arguello. Phase recovery guarantecs from designed coded diffraction patterns in optical imaging. IEEE Transactions on Image Processing, 29:5687-5697, 2020.
[69] Simon Foucart and Holger Rauhut. A Mathematical Introduction to Compressive Sensing. 2013.
[70]
Emmanuel Candès and Michael B Wakin. An introduction to compressive sampling. IEEE signal processing mugazine, 25(2):21330. 2008.
[71] Emmanuel J Candès, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions information theory, 52(2):489-509, 2006.
[72] Huan Li and Zhouchen Lin. Accelerated proximal gradient methods for nonconvex programming. Advances in neural information processing systems, 28, 2015.
[73] Radu Ioan Bot, Emoö Robert Csetnek, and Szilárd Csaba László. An inertial forward backward algo- rithm for the minimization of the sum of two nonconvex functions. EURO Journal on Computational Optimization, 4(1):3-25, 2016.
[74] Trang C Mai, Hien Quoc Ngo, and Le-Nam Tran. Energy-efficient power allocation in cell-free massive mimo with zero-forcing: First order methods. Physical Communication, 51:101540, 2022.
[75] Jingxin Zhang, Donghua Zhou, Maoyin Chen, and Xia Hong, Continual learning for multimode dynamic process monitoring with applications an attrasuuperritticl thermal power plant. IEEE transactions on Automation Science and Engineering, 2022.
[76] Zhili Ge, Xin Zhang, and Zhongming Wu. fast proximal itcratively reweighted nuclear norm algorithm for nonconvex low-rank matrix minimization problems. Applied Numerical Mathematics, 2022.
[77] Kaixuan Wei, Angelica Aviles-Rivero, Jingwei Liang, Ying Fu, Hua Huang, and Bibbaa- Schönlieb. TFPNP: Tuning- free plug-and-pla proximal algorithms with applications to inverse imaging problems. Journal of Machine Learning Research, 23(16):1 2022.
[78] Ulugbek S Kamilov, Charles A Bouman, Gregery T Buzzard, and Brendt Wohlberg. Plug-and- play methods for integrating physical and learned models in computational imaging. arXiv preprint arXiv:2203.1 10003.17000 17061, 2022.
[79] Yuyang Hu, Jiaming Liu, Xiaojian Xu, and Ulugbek S Kamilov. Monotonically convergent convrgent regularization by denoising, arXiv preprint arXiv:2202. 04961, 2022.
[80] Shunsuke Ono. Primal- dual plug- and-play -lay image restoration. IEEE Signal Processing Letters, 24(8):1 108 1112, 2017.
[81] Ulugbek Kamilov, Hassan Mansour, and Brendt Wohlberg. A plug-and-play priors approach for solving nonlinear imaging inverse problems. JEEE Signal Processing Letters, 24(12):1822-188 2017.
[82] Zhiyuan Zha, Bihan Wen, Xin Yuan, Jiantao Zhou, and Ce Zhu. Simultaneous noulocal low-rank and deep priors for poisson denoising. In ICASSP 2022-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 2320-230 3320-2324. IEEE, 2022.
14

---

[83] Qiyu Hu, Yunlong Cai. Qingjiang Shi, Kaidi Xu, Guanding Yu, and Zhi Ding. Iterative algorithu induced deep-unfolding neural networks: Precoding design for multiuser mimo systems. IEEE Transactions on Wireless Communications, 20(2):194--1400, 2020.
[84] Arindam Chowdhury, Gunjan Verma, Chirag Rao, Ananthram Swami, and Santiago Segarra. Unfolding wmmms using graph neural networks for efficient power allocation. IEEE Transactions on Wireless Communications, 2009)6004-6017, 2021.
[85] Naveed Naimipour, Shahin Khobahi, and Mojtaba Soltanalian. UPR: A model-driven architecture for deep phase retrieval. In Asilomar Conference on Signals, Systems, and Computers, pages 205-209, 2020.
[86] Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ISTA and practical weights and thresholds. Advances in Neural Information Processing Systems, 31, 2018.
[87] Jialin Liu and Xiaohan Chen. ALISTA: Analytic weights are as good as learned weights in LISTA. In International Conference on Learning Representations (ICLR), 2019.
[88] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 126-135, 2017.
[89] Lci Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging, 20(2):023016, 201
[90] Kodak image dataset. http://zOk... us/ wa/graaphiss/koaae Accessed: 2022-05-05.
[91] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2. pages 416 4234423,22 July 2001.
[92] Nikolay Ponomarenko, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Lina Jin, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al. Color image database TID2013: Peculiarities and preliminary results. In European workshop on visual information processing (EUVIP), pages 106-111. IEEE, 2013.
[93] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedingy Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416-423. IEEE, 2001.
[94] Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association, 96(456):1348 1360, 2001.
[95] Li-Hao Yeh, Jonathan Dong, Jingshan Zhong, Lei Tian, Michael Chen, Gongguo Tang. Mahdi Soltanolkotabi, and Laura Waller. Experimental robustness fourier ptychography phase retrieval algorithms. Optics Express, 23(26):3311-333244 2015.
[96] Chengbo Li, Wotao Yin, Hong Jiang, and Yin Zhang. An efficient augmented lagrangian method with applications to total variation minimization. Computational Optimization and Applications, 56(3):507- 530, 2013.
[97] Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, and Amit Ashok. ReconNet: Non- iterative reconstruction of images from compressively sensed measurements. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 449-458, 2016.
[98] Jian Zhang and Bernard Ghanem. ISTA-Net: Interpretable optimization appimization-inspired deep network for image compressive sensing. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1828-1837, 2018.
[99] Rémi Gribonval and Morten Nielsen. Highly sparse representations from dictionaries are unique and independent of the sparseness measure. Applied and Computational Harmonic Analysis, 22(3):335-355, 2007.
[100] Joseph Woodworth and Rick Chartrand. Compressed sensing recovery via nonconvex shrinkage penalties. Inverse Problems, 32(7):075004, 2016.
[101] Heinz H Bauschke, Patrick L Combettes, et al. Convex analysis and monotone operator theory in Hilbert spaces, volume 408. Springer, 2011.
15


Reviews:
[{"review_id": "p4PdNCvwGiw", "reviewer": "Reviewer_2EmJ", "summary": "The authors consider the task of image reconstruction with regularization. Beginning at least a few decades past, regularizers have been adapted by various communities first from L2 quadratic penalties to L1-like edge-preserving penalties. More recently, motivated by compressed sensing researchers have used L1 or even non-convex penalties. Non-convex penalties are more difficult to analyze from a theoretical perspective, but empirically have been observed to produce the best results for image reconstruction tasks.\n\nIn this manuscript the authors propose to focus on classes of regularizers that are invex functions. Invex functions are functions where \"at any point where the derivative vanishes (stationary point), it is a global minimizer of the function.\" The manuscript's contributions are three-fold: 1) the manuscript provides the first list of regularizers with proved invexity for image reconstruction, 2) the manuscript establishes convergence guarantees for three types of image reconstruction problems with invex regularizers and 3) the manuscript provides empirical data on the performance of the regularizers.", "questions": "1. How can the H matrix in Theorem 4 be unitary if m < n? If m < n, H cannot have an orthonormal basis for vectors of size n.\n2. Could the annotation of Table 3 be improved? This is one of the most important results of the paper. I would bold the best-performing PSNR value in each row as is standard practice at machine learning conferences rather than the regularizer class. Also, the performance of Eq. (6) should be discussed.\n3. How were the datasets used? Did the authors merge all datasets under consideration? These details should be further specified for the experiments.", "limitations": "The authors discussed limitations under the relevant theorems and lemmas, but for practitioners I think it would be better to have a dedicated Limitations section. A few items that might be considered for such a section:\n\n- The plug-and-play result only shows that the optimization sequences converges to a close estimate of the solution.\n- The theory does not seem to consider noisy measurements.\n- The reconstruction tasks under consideration seem to be limited to deconvolution.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The authors present convergence theory for three problems of substantial interest to the reconstruction community: denoising, compressed sensing, and plug-and-play reconstruction. Each of these has been applied in practical settings. The paper includes global convergence guarantees for each setting.\n- The invex approach seems to be a new and interesting perspective into inverse problems. In particular, the quasi-norm of Equation (6) has long been of great interest to the image reconstruction community due to its relations to compressed sensing, so it is encouraging to see that the current manuscript's theory can be applied to this case.\n- In addition to theoretical results, the paper also includes empirical results for each of the regularizers under consideration.\n\n**Weaknesses**\n- Some properties of H in Theorem 4 seem inconsistent with how it is used in the subsequent algorithms. For example, Theorem 4 states that H is unitary, but H is elsewhere specified to have full row rank. The specific properties of H necessary for each theorem setting should be further clarified.\n- The presentation of Table 3 is somewhat confusing.\n- The variations in invex regularizer performance is not discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LcilE_ky0vB", "reviewer": "Reviewer_8pbb", "summary": "The paper provides convergence results attached with invex regularizers. Few non-convex regularizers with no global convergence guarantees profited from this analysis through invexity. Authors establish convergence guarantees to global optima for various advanced image reconstruction techniques with invex regularizers.  Authors prove few functions as invex functions and provide the proximal operators corresponding to them. These invex regularizers were utilized for various imaging tasks.\n", "questions": "Suggestions: (I am not asking authors to work on this right away. These are merely suggestions for a camera-ready version.)\n1. The paper presentation could be improved. A comparative plot between all invex functions would've clarified things.\n2. A visual aid with optimization landscape of Eq (11) i.e., $\\| x - u \\|^2$ + Invex function, will give more insights into the convexity of the problem, and comparison with $\\| x - u \\|^2 + \\ell_1$ will help us understand why invex regularizers are performing better than convex regularizers. \n3. Convergence analysis has been provided in theory. It would be better if the comparison of the algorithms has been provided in practice (experimental validation). plots with reconstruction MSE vs iterations, MSE vs time would be suggestible.\n4. Proximal operators of non-convex penalties are known to have longer computation time. So, a comparison of the proposed methods running times will establish solid ground for the paper.", "limitations": "The authors discussed the limitations of their work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Invex regularizers give a decent theoretical analysis of a few non-convex regularizers.\n2. Experiments included various image restoration tasks such as compressed image recovery and image denoising.\n\nAcknowledgments:\n1. Authors included more comparisons with TVAL3, ReconNet, and ISTA-Net as part of the revision.\n2. More Image reconstruction examples are provided in the supplementary document as part of the revision.\n\nIssues:\n1. The revised paper is more than ten pages long. I believe it has to be limited to 9 pages until acceptance. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E1I3QWsO1dv", "reviewer": "Reviewer_xZpv", "summary": "This manuscript revisits the concept of invexity and provides the first list of proved invex regularizers for improving image reconstruction. Convergence conditions are analyzed and experiments are conducted on practical applications.", "questions": "1.\tCan you compare the performance with some CNN-based baselines?\n2.\tFor different regularizers, were the parameters (e.g. \\lambda in eq.12) chosen as the same value?\n", "limitations": "No potential negative societal impact is mentioned. Both theoretical analysis and practical applications of invex functions are provided. I believe the proposed finite list will inspire exploration in related fields. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1.\tThe paper is well-written with compact structure and clear interpretations.\n2.\tSome useful regularizers in imaging applications are formally studied in a new perspective of invex function. Besides, a simple and valid way of constructing an invex function is provided, which may potentially inspire later researchers.\n3.\tExperimental comparisons indicate that the proposed invex regularizers have potential to improve performance in practical applications. \nWeakness:\nThe proposed new invex function doesn’t achieve the best efficiency compared with the existing non-convex functions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept