Global Optimality for Non-linear Constrained Restoration Problems via Invexity
Authors: Samuel Pinilla
Jeyan Thiyagalingam
Venue: ICLR 2024 poster (2024)
Keywords: Constrained optimization
Invexity
Quasi-invexity
Global optima
Primary area: optimization

Abstract: Signal restoration is an important constrained optimization problem with significant applications in various domains. Although non-convex constrained optimization problems have been shown to perform better than convex counterparts in terms of reconstruction quality, convex constrained optimization problems have been preferably for its global optima guarantees. Despite the success of non-convex methods in a large number of applications, it is not an overstatement to say that there is little or no hope for non-convex problems to ensure global optima. In this paper, for the first time, we develop invex constrained optimization theory to mitigate the loss of guarantees for global optima in non-convex constrained inverse problems, where the invex function is a mapping where any critical point is a global minimizer. We also develop relevant theories to extend the global optima guarantee to a set of quasi-invex functions - the largest optimizable mappings. More specifically, we propose a family of invex/quasi-invex of functions for handling constrained inverse problems using the non-convex setting along with guarantees for their global optima. Our experimental evaluation shows that the proposed approach is very promising and can aid in extending existing convex optimization algorithms, such as the alternating direction method of multipliers, and accelerated proximal gradient methods.

Published as a conference paper at ICLR 2024
GLOBAL OPTIMALITY FOR NON-LINEAR CON- STRAINED RESTORATION PROBLEMS VIA INVEXITY
Samuel Pinilla, & Jeyan Thiyagalingam
'Scientific Computing Department, Science and Technology Facilities Council samuel nini mi pllpllilnstttttiiilllllllllnmlne flaastff.ac. ack.uk
ABSTRACT
Signal restoration is an important constrained optimization problem with signif- icant applications various domains. Although non-convex constrained opti- mization problems have been shown to perform better than convex counterparts in terms of reconstruction quality, convex constrained optimization problems have been preferred for their global optima guarantees. Despite the success of non- convex methods in many applications, it is not an overstatement to say that there is little hope for non-convex problems to ensure global optima. In this paper, for the first time, we propose a family of invex functions for handling constrained inverse problems using the non-convex setting along with guarantees for their global optima- invex function is a mapping where any critical point is a global minimizer. We also develop relevant theories to extend the global optima guaran- tee to family of quasi-invex functions - the largest set of optimizable mappings. Our theoretical results show that the proposed family of invex and quasi-invex functions can aid in extending existing convex optimization algorithms, such as the alternating direction of multipliers and accelerated proximal gradient methods. Moreover, our numerical tests show that the proposed family of invex/quasi-inve functions overcome the performance of convex mappings terms of reconstruc- tion quality across several relevant metrics and imaging tasks.
INTRODUCTION
Signal restoration is an optimization problem that seeks to estimate unknown original signal
x E In from set noisy observations y E R". This optimization problem falls under the purview of inverse problems, which common across all disciplines. from physics (Mountrakis
et al. 2011 Vouras et al., 2022), to medical imaging (Ginat & Guptal [2014] [Piilla et al.) (022b)
to signal processing (Pinilla et al. 2023), and machine learning (Harris al.] 2020 Wei al,
2022 In general, an inverse problem can be formulated as follows:
minimize g(x) subject to f(x)= S, #ER"
where e > 0, f()) is a mapping constructed as a reconstruction error (fidelity term), and g(z) is a regularizer model the priors of the signal such as sparsity, low-rankness, and smoothness (Beck)
2017 Chapter 10). Here, f(x) is often considered to be differentiable (i.e., smooth), while g(x) is not. For some imaging tasks, the fidelity term has the form of s(Ax: y). where A Rm When m < n for A. the inverse problem reduces to compressive sensing. This technique is extensively exploited many areas such microscopy, optical imaging, and spectroscopy (Arce et al. 2013
Jerez et al. 2020 Guerrero et al. 2020).
Solving a guarantee for global optimality is of paramount importance to ensure that the best- possible solution is found subject to the relevant constraints (Parikh & Boyd, 2014 Pinilla et al.) 2022ah. Convex constructs of f())) gzx), ), as demonstrated the literature (Beck & Teboulle 2009 Beck, 2017), are helpful for seeking guaranteed global optima, when either the measurements can be considered as ideal and noiseless (i.c., E = 0) (Parikh & Boyd 2014) or when the regularizer g(æ)) exactly models the priors of the data. However, real-world problems do not fit this assumption, where noise is rather prevalent in all measurements (i.e.. e > 0), and convex regularizers can easily inaccurately model the sparsity, low-rankness, smoothness, and abnormal patterns in the data (Can-) des et al.] 2008; Laghrib et al.) 2023). Although approaches with assumptions tied to noise models (such as Gaussianity) perform better than plain convex approaches, they too fail the real-world noise models often deviate from these assumptions (Sun & Pong 2022 Tankovich et al. 2021).
"Rtterffrrd Appleton Laboratory, Harwell, UK.

---

Published as a conference paper at ICLR 2024
Table 1: State-of tete--f--hhe-rn art proposition for fax), and g(æ) optimization problem
Type of function
Ref
Global optima
(
g(x)
Sun & Pong 2022
SCAD
No
Carrillo et al. 2010
Huber loss
Log-sum penalty
No
Carrillo et al.
2016
Cauchy loss function
4
No
PPnilla et al.,
2022a
f2-norm
invex
Yes
Proposed
invex/quasi-invex
invex/quasi-invex
Yes
An alternative approach is consider non-convex mappings for f(x)) and g(æ).. From (Kaul & Kaur 1985), it can be seen that there exists a large collection of mappings that can be well-optimized com- pared to pure convex settings. We summarize important non-convex mappings from the literature in Table These are a set of well-accepted mappings that perform better than convex ones in terms of restoration quality (Pinilla et al. 2022a; [Sun & Pong, 2022). Various regularizers have been consid- ered in the literature to overcome the limitations of pure convex approaches, such as the E f-normm total-variation (Chambolle| 2004 Selesnick al. 2020) and nuclear norm (Candès & Recht 2006), which are the continuous and convex surrogates of the fo-pseudo norm and rank, respectively (Ful 2014). More specifically, several interpolations between the Lo psepddnoomm and the & -n-mm have been explored as regularizers including the eg fg-quasinorms (where 0 < p < 1) (Marjanovic & Solo, 2012]. Capped-E, penalty (Zhang et al., 2007). Log-Sum Penalty (Candes et al., 2008), Convex-
concave penalty (Kovalev & Gasnikov 2022). Minimax Concave Penalty (Zhang] (2010), Geman Penalty (Geman & Yang]1995), and the Smoothly Clipped Absolute Deviation (SCAD) (Fan & Li, 2001). Furthermore, 2--norm incorporated part of the constraint in (I) is often replaced by other loss functions to reflect different noise models or robustness requirements in the noisy data y, such as those outlined in (Carrillo et al. 2016 Zoubir et al.
2012). Concrete alternative reconstruction
error functions include the Cauchy loss (Carrillo et al. 2016). the Huber function (Hedman et al.)
2021], and the Tukey biweight loss (Mosteller & Tukey 1977p. However, while offering outstanding restoration quality, none of these can offer a guarantee for global optima, which would have been ideal for providing the soundness of these approaches.
The lack of approaches for guaranteeing the global minima in non-convex settings leaves any optimization-basee based algorithmic solution incomplete and non-unique. It hence cannot be categori- cally accepted as the best possible solution despite their improved performance over convex func- tions. Motivated by this problem of seeking guarantees for global optimality when solving non- convex problems, we investigate broader classes of functions that exploit the concept of invexity and quasi-invexity (Hanson| 1981 Reilandd 1990), which have been initially defined early 1980s (Han- son 1981). The specialty of invex functions is that any critical point in invex function is global minimizer itself (Pinilla et al., (022a). Quasi-inver invex mappings generalize invex functions and hence the largest set of optimizable functions as outlined in (Kaul & Kaur 1985). This places the quasi- invex functions in a special and unique position for handling inverse problems. Although mathemat- ical implications of quasi-inver (and hence invex) functions are well developed (Syed et al.) 2013)
Barik & Honorio 2021] Pinilla et al., (022a)). their practical utility is less understood (Zälinescu 2014). To the best of our knowledge, there no existing body of work in the invex literature on applying and developing quasi-invex (and invex) based methods for seeking global optimality in constrained optimization problems, such as signal restoration.
In this paper, we focus on demonstrating the practical utility of assuming f(x). g(z) to be invex/quasi- to handle inverse problems as in (T). with guarantees for global optima. Hence, in contrast with results in (Pinilla et al. 2022a), we make the following key contributions in:
We develop a set of mathematical tools construct a family of f(x=) and g(x) in (1 with proved suaeit///uaiiiivexiity for model-based inverse imaging problems. Additionally, we prove that our proposed family of invex/quasi-invex functions closed under summation result that has not been available before in the invex literature of optimization Zäälinescu] 2014).
We prove global optima under the invex/quasi-invex settings. These results extend the capability of alternating direction method of multipliers and accelerated proximal gradient methods to handle non-convex cases otherwise limited to convex settings. We also report their convergence rates, which show both methods using vrex/quasi-inves mappings are as efficient as solving the well- established convex optimization problems;
We conduct an effective evaluation of the proposed approach in handling a number of signal restoration problems against state-of-the thetthhh-att algorithms and baselines.
2

---

Published as a conference paper at ICLR 2024
2 PRELIMINARIES
The i-th entry of vector w. is w(1] For vectors, ||w||p is the e--norm. An open ball is defined as (( r) = y E R"; |1y - x 2||2 < r}. The operation conv(A) represents the convex hull of the set A. We use (W) to denote the i-th singular value of a matrix W in descending order.
We present several concepts needed for the development of this paper starting with the definition of locally Lipschitz continuous function.
Definition 1. A function f R + is locally Lipschitz continuous at a point x E R if there exist scalars K 0 and > 0 such that for all y. B(x, c) we have (f( - f(x) - K 21|2-
Since the ordinary directional derivative being the most important tool in optimization does not necessarily exist for locally Lipschitz continuous functions, it is required introduce the concept of subdifferential (Bagirov et al. 2014) which is calculated in practice as follows.
Theorem 1. Bagirov et al. 2014 Theorem 3.9) Let f R= R be locally Lipschitz continuous function at x E R", and define S2 {x E | is not differentiable at the point æ}. Then the
subdifferential of f is given by
0f(z) cony ({C R. exists (a::) E R" \ By such that x; + and fx())) c))
(2)
The notion of subdifferential is given for locally Lipschitz continuous functions because it is always nonempty (Bagirov et al., 2014 Theorem 3.3). Based on this, we present the concept of invexity. Definition Let f
I + RR be locally Lipschitz; then ʃ is invex if there exists a function 7: R" x RD R" such that vz, y R", VÇ ay we have f(x) f(y) 2 <" (=x y).
well known that convex function simply satisfies this definition for () y) = x - y. The following theorem (Mishra & Giorgi 2008 Theorem 4.33) makes connection between invexity and its well-known optimum property which motivates the design of invex optimization problems.
Theorem 2. (Mishra& & Giorgi 2008 Theorem 4.33) Let R" + R be locally Lipschitz. Then the following statements are equivalent: 1) f is invex. 2) Every point y E IR" that satisfies 0 E Of(y)
is global minimizer of f.
We finalize this section by introducing a generalization of invexity; indeed (Reiland 1990) intro- duced also the following classes of generalized convex functions.
Definition 3. Let f R" + IR be locally Lipschitz; then f is quasi-invex if there exists v
R" R" + I= such that Vx, E R", VÇ Of(y) we have f(x) f(y) S <" (T)((xy)= S 0.
3 INVEX AND QUASI-INVEX FUNCTION SETS
We address the necessity of handling the issues around real-world data by constructing invex/quasi- invex functions for data fidelity loss (æx) and regularizer g(æ) functions. To this end, we first define the family of functions this paper investigates and their properties.
Definition 4. (Admissible Function) Let h : IR" - R such that h(x) Σi= s( {((x),]), where 8 [0, x<) + [0, x)) and 8' (w) > O for w E (0, oo). If with s(0) = such that s(w)/ut /w2 is non-increasing on oo), then hxx) said be an admissible function.
Definition 4 places requirement on fax) and g(xr) in ① to operate element-wise positive values. This condition is mild since it holds for the vast majority of inverse problem formulations, including loss functions in deep learning, whether they are convex or non-convex (Carrillo al., [0016] Wen et al. 2018 Sun & Pong 2022). Using clement-wise mapping offers notable advantages, especially in developing high-performance algorithms. One key advantage lies in the ability to parallelize, effectively reducing runtimes, making them suitable for addressing large-scale problems (Parikh &
Boyd 2014 Wang & Chen [2022] Zoubir et al.) 2012).. Other conditions in Definition 4 are crucial as they establish vital connection to invexity, which we discuss next in Theorem (see Appendix for the proof).
Theorem 3. Let f.g: R. + R be two admissible functions in Definition 4. such that f(æ)
E' sy([x[i]] and g(x) = -E
Then the following holds: i) f (x), and g(z) are
invex: ii) (x) = af(xx) + (g(x) an admissible function (therefore invex) for every a, 3 2 0;
iii (x))) = (
is an admissible function. In addition to these, if we assume that
& (1), 8 (w) for oo), then f(x), g(x), and (2(2,,, he(a:) are all quasi-invex.
3

---

Published as a conference paper at ICLR 2024
The essential breakthrough of Theorem compared with results in the invex literature such as (Pinilla et al. 2022a) lies in the fact that Definition 4 introduces the first family of invex/quasi- invex functions closed under summation. This implies that the sum of invex/quasi-invex functions is also invex/quasi-invex. The significance of this result manifests on two fronts. Firstly, Theo- ] paves the way for estahlishing theoretical guarantees for global optima of (T) beyond the realm of convexity. Secondly, it bestows practical benefits to practitioners, as both Definition 4 and the third statement of Theorem 3 offer systematic methodology for constructing invex/quasi- invex functions. These results also give away another side consequence: for any admissible function (xx) = -i= ([x]((]]) such that s (w) 2 0. then f is not invex. The proof stems from the fact that (wx) can be zero in non-minimizer values, leading to its quasi-invexity. In the following sections, we leverage the results in Theorem 3 to present illustrative examples of invex/quasi-ine invex functions
rem
that apply in diverse scientific domains.
3.1 INVEX REGULARIZERS
In addition the list of original invex regularizer functions presented in (Pinilla et al. 2022a Lemma 1), consider the following Theorem[ Appendix B for proof).
Theorem 4. For p (0, 1], O, the following mapping
(Lll-regur)) g(x) Σ log(1+ + [[]lli)++))))
is an admissible function. Additionally, let D E 1==1))xn be the first-order difference matrix (further details in Appendix If g is an admissible function, then grv (ax) = g( Dx)) is invex.
The key to prove Theorem is centred around the fact that any admissible function is invex, and it a point-wise non-increasing when divided by quadratic mapping. It is worth mentioning that the invex function presented in equation (3) is one of the pivotal functions in the domain of compressive
sensing (Keshavarzian et al.) 2019). Although the analysis of equation (3) is valid with or without c,
we prefer to retain to satisfy Definition I Although the literature identified this function as non- convex (Wang& & Chen 2022 Liu et al. 2018), there is no notion of invexity or proof of invexity.
The second part of Theorem 4 presents invex generalizations of the classical convex total-variation
(TV) regularizer (Chambolle 2004; Selesmick et al., (0200) with the & fnnormm replaced by invex func- tions missing result in the invex literature such (Pinilla et al. (022a). TV-based regularization is effective in various applications such as computer tomography (Sun et al. 2019). hyperspectral imaging (lordache et al. [2012] Wang et al. (023), ) and computer vision (Estrela et al.] (016)). where the goal is purely signal restoration. Classical TV-based algorithms suffer from an issue where the fi-norm tends to underestimate the amplitudes of signal discontinuities owing to the limitation of the fi-norm regularization itself (Selesnick et al. 2002)). Several studies in the literature (Liu et al., 2018 Selesnick et al., 2020) show that l qu-quasioomm instead of fn-norm can significantly address this issue, albeut the literature not demonstrating the benefits ascribed to the invexity. Again, similar
to equation (3), the literature identified this function as non-convex (Wang & Chen 2022 2018) without any notion of these functions being invex or proof of invexity.
Liu al.,
Driven by the success of equation 3 in compressive sensing (Keshavarzian et al. 2019), we propose its TV-like regularizer version. Further insights of Theorem including the extension of TV-like regularizers to two- and three-dimensional signals, are presented Section B.
3.2 INVEX FIDELITY LOSSES
Now, to complement the list of invex regularizers stated in Section 3.1 fidelity functions fxx in the following theorem.
we present five invex data
Theorem 5. All the following functions for c, d > 0, and a E IR are admissible
x [i]
(Cauchy)
f(w) Σ log 1 1
(4)
82
2x:2
(Geman-Mcclure) f(x)
(5)
x
+ 482
1=1
[2
(Welsh)
f(x) =
Σ
1
cxp
(6)
282
i=1
4

---

Published as a conference paper at ICLR 2024
(Adaptive robust) [(x)
a -2
((
(ali)/ee²
[x - 2|
+
a/2
-1
a
m
x
[1]
log (1 + x (1))) [(x) Σ
(8)
2x:2
+
Fidelity losses given in equations (4)-7) have existed in the literature without ever being identified as invex functions or the benefits being ascribed their invexity. Equation (6) was identified as an
7 are found in the literature replacing the traditional & 2n-norm. This was account for different noise models and
invex function, and its invexity was proved in (Syed et al., 2013; 2014). Equations
improve the solution's 's robustness towards noisy measurements in (I) Eguations (4)-(7) have been
used improve the performance the original Variational Autoencoder (Kingma Welling 2014)
by incorporating equations
4 as robust loss functions in (Barron (019). Finally, equation (8)
a novel invex function we propose to provide more flexibility around losses (harder to achieve in
convex setting). The proof of Theorem 5 is deferred to Appendix
3.3 QUASI-INVEX REGULARIZERS
We list three quasi-inver invex function constructs for g(x) in Theorem below. Theorem 6. For constants X, p E (0, 1]. and >> 0, the following mappings are quasi-invex
(x)) = A re(a=()) 0 2 1, (a])) { A|ズ|i|| x² (²)0)((0) 0x2/2
if x
< 0A
(9)
if
a
N 0A
1=1
(x( = Σ min (log(1 + (|æ|)) + )))) 0); 0.
(10)
Further, let D E (n-1)xxn be the first-order difference matrix. If is admissible such that g(x) Em 8, (]][[i]]]) with sf (w) 2 0 for w E (0, oo), then grv (2) = (xx) is quasi-invex.
The critical aspect of proving Theorem 6 centred around the quasi-invex construct in Theorem and that it is point-wise non-increasing when divided by quadratic mapping. Equation 9 re- ferred to as the Minimax Concave Penalty (MCP) (Zhang 2010) the non-convex literature, with
range of upplications in various domains (Kovalev & Gasnikov 2022 Zhang 2010). Although this mapping function is standard in imaging applications (Ding et al. 2018: Song et al., 2022). the knowledge of its quasi-invexity has
been identified nor proven. For the first time, we provide formal proof of MCP being quasi-invex in Appendix D Additional functions, such as the Clipped Absolute Deviation (SCAD) (Fan & Li 2001), are proven to be quasi-invex in Section D Further- more, following an analysis analogous to Theorem 4 we can extend to two- and three-dimensional TV-like quasi-invex regularizers. In addition these three quasi-invex constructs, propose an- other way to produce quasi-invex loss functions. We present this in Lemma below.
3.4 QUASI-INVEX FIDELITY LOSSES
We finalize the exposition on quasi-invex functions by presenting the following Lemma D for con- structing a quasi-invex function for f(x). This result states that the maximum of two invex functions (not necessarily with respect to the same 17) is quasi--nvex. This lemma has applications in robust optimization or robust imaging problems (Vargas al., 2018b). where controlling the accuracy of the final solution is crucial. The route to robustness is achieved by minimizing the maximum be- tween two invex functions, which will bound the highest error to desirable value (Boyd al. 2004 Section 1.2.2). The proof of this lemma is deferred to Appendix [
Lemma 1. Let f. h R" + R be two invex functions as in Definition (not necessarily with respect the same 17). Then, q(x)= = (x) h(x)) is quasi-invex.
4 APPLICATIONS
In this section, demonstrate the of invex/quasi-invex functions several applications and guarantees for global optima, which has only been possible for convex functions. To this end, we commence by (1) ensures global optima when / (ax), g(x) are admissible functions missing result
5

---

Published as a conference paper at ICLR 2024
in the invex literature such as (Pinilla et al., 2022a). The importance of this result, which prove in Appendix [P] lies in the fact that it is the first time that global optima is achieved for a family of 0). The key to the proof is that
invex constrained optimization problems (for the noiseless
both f.g functions are admissible.
To complement this result, we provide extensive list of applications that would directly benefit from the global optima guarantees of (T. and the family of invex/quasi-inver constructs in Theorem 3 Summmrrzeed in Table and for the reasons of brevity, we selectively discuss three of the important applications. These include two imaging tasks, namely, compressive image restoration and total variation filtering, and two algorithmic applications, namely, extended Alternating Direction Method of Multipliers (ADMM) in Section and extended accelerated proximal gradient method (APGM) which we present due to space limitations in Appendix [ Additionally, the extended guarantees of ADMM allows to state the global optimality of solving program (1) when (z)) g (x)) are the family of quasi-invex functions introduced in Theorem B
Table 2: List of Invex and Quasi-invex functions studied in this work.
Reference
Invex
Application
Reference
Quasi-inves
Application
Wang et 2022
equation
Compressive sensing
Song et al 2022
equation
Model selection, ADMM
Tomography
Invex (3)
Toeal variation filtering
Tankowich
Robust Learning
This work
equation
Dictionayy Learning
Park es al.
oquation
Neural Radiance Fickd
Adiptiie Fillering
Quasi inves (rv((a)
Inpuinting
Thi work
equation
Supervised L Leantnng
4.1
COMPRESSIVE IMAGE RESTORATION
The reconstruction of signal/image from compressed set of measurements is essential in range of scientific domains that rely on the analysis and interpretation of image content (De los Reyes) al. 2017). In this context, program (T) becomes an inverse problem that aims recovering an image f E Rhth assuming that f has a k-spaase representation 0 E IR" (k < n. non-zero elements) in a basis T RRxn, that is f ax, in order to ensure uniqueness under some conditions. Examples of this sparse basis V in imaging are the Wavelet (also Haar Wavelet) transform, cosine, and Fourier representations (Foucart& y H Ax + 77, where 77 the noise, A encapsulates the product between acquisition matrix H, and V with & fo-normalieed columns (Arce et al., 2013] Candès & Wakin 2008). Under this setup, compressive sensing enables the recovery of using much fewer samples than predicted by the Nyquist criterion (Candès & Wakin 2008). When the regularizer g gxx) takes the convex form of fi-nrmm, ((Ax y) is the f2-norm, and when the sampling matrix A satisfies the restricted isometry property (RIP) for any k-sparse vector æ E IR", i.c., (1 - 62x) | <
Rauhut 2013). Hence, one can work with the abstract noisy model
|| | < (1+ d2x) || li for d2s < 3 (Foucart& & Rauhut 2013 Theorem 6.9), has been proved that æ can be exactly recovered by solving ① with c 0.
It is classically known that II has unique solution under convexity assumptions on g(æ), and fAAx y) for e O. We are interested in applying invex and quasi-invex regularizers and fi- delity functions. It has been proven that, when g(x)) takes the particular invex form (Pinilla et al. 2022a| Theorem 4), and f(Ax y) the f2-norm, æ can be exactly recovered by program (T) for c (Pinilla et al.] (002a)). Below Theorem 7 we further generalize this result to all the invex/quasi-invex constructs as in Theorem missing result in the invex literature such as (Pinilla ct al. 2022a).
Theorem 7. Assume Ax + 1 = y. where æ E R is k-sparse, the matrix A E RRxx (m < n) with nn-ormalized columns that satisfies the RIP condition with dzk <
4 V the noise vector satisfies f(Ax= - 3) < c for e > O and y E I is a noisy measurement vector. gaz), and f(z) are admissible functions where ((x) satisfies the triangle inequality (further details in Appendix G. then the solution x* of (I) holds ||æ x I|1 < C. (x) + Deee),,(..)) where the constants C, D > 0 depend only on 02k. and (x), (z). We show that a similar result holds for quasi- invex constructs as in Theorem In addition, if 0 then w can be exactly recovered by solving ( using both invex/quasi-invee constructs.
The key to prove Theorem 7 centred around the properties of invex/quasi-invex constructs in
Theorem 3 We present the proof of Theorem 7 in Appendix G It also includes insights on using
Lemma [ to have more quasi-invex functions, for which Theorem] is also valid. We highlight that
Theorem 7 is in the same form of convex recovery result (Foucart & Raubut 2013] Theorem 6.1)), suggesting the effectiveness of our proposed family of invex/quasi- functions.

---

Published as a conference paper at ICLR 2024
4.2 TOTAL VARIATION FILTERING
Total Variation (TV) regularization is deterministic technique that safeguards discontinuities in images (Ehrhardt & Betcke| 2016; Selesnick et al., 2020). It is based on the principle that images with sharp edges have high variation while smooth transitions will show low variations (Lin al.) 2019 Liu et al. (018)) These variations are modeled using the first-order difference matrix in Theo- rem4/more details in Appendix(B) and have been instrumental on wide range of image restoration problems, including denoising (Hu & Selesnick (2200) deblurring (Selesnick et al., 20020), inpaint- ing (Schönlieb et al., 2009), intrared super-resolution (Liu et al., 2018). and magnetic resonance
imaging (Ehrhardt & Betcke 2016), among others (Chambolle (004)).
Mathematically, classical TV regularization of an image u E R" the solution of
1
minimize 9rv (x) + 2X x u u122,
(11)
#ER"
for x E (0, 1] (typical choice in practice), where grv (x)) the sum of the absolute value differences between adjacent pixel values, i.e., fn-mr of differences between adjacent pixel By solving for u, TV regularization encourages the output image to have smooth regions while maintaining sharp edges. The main advantage of (II) is that it has no extraneous local minima, and the minimizer is unique (Selesnick et al. 2020). However, owing to the limitations of & f--nrrm (Selesnick et al.) 2020 Liu et al. 2018), optimization problem in (TT) can have undesirable effects, for example, in denoising, by underestimating signal discontinuities' amplitudes.
We improve program [1I by incorporating the invex/quasi invex regularizers grv (œ) presented in Theorems 4 andd 6 as follows (extensions for two-dimensional in Section [B.I)
Theorem 8. Consider the optimization problem in (TT) for invex and invex TV regularizers as in Theorems and 6. respectively. Then: 1) The function h(æ) grv (z) + 2X
A || u11 is
convex for a proper range of X. 2) The resolvent operator (I + A0grv) is singleton.
is classically known that the sum of two invex functions (or quasi-inves and invex) is not necessar- ily invex (Mishra & Giorgi 2008). Then, the above result is helpful to both the invexity and imaging communities, complementing Theorem We present the proof of Theorem in Appendix [H] which uses the fact that any admissible function is point-wise non-increasing when divided by a quadratic mapping. This analysis helps extend the original ADMM algorithm in Section 4.3 A set of map- pings called prox-regular or weakly convex, can be found in the literature (Wang et al. 2019b] Rockafellar & Wets| 2209),, for which Theorem g holds function is prox-regular if program (II) is convex for some A. However, it is crucial to discern a critical disparity between prox- regular functions and admissible mappings. Prox-regularity implies quasi-invexity (proven for the first time in Appendix HH. This highlights the intrinsic advantage of admissible functions in attaining global optima, showcasing their prowess in contrast to prox-regular functions.
4.3 EXTENDED ALTERNATING DIRECTION METHOD OF MULTIPLIERS (ADMM)
The ADMM algorithm has a special place in a range of signal restoration and imaging problems under convex settings with linear equality constraints (Boyd all 2011p. It is particularly favored for handling large-scale problems (Glowinski) 2014) with many constraints given its amenability
for parallelization (Boyd et al. 2011). Some notable applications of ADMM include blind pty-
chography (Li) 2023), phase retrieval (Liagg et al. (017).. computer tomography (Wei et al. 2022
unrolling (iee et al. 2019), and magnetic resonance imaging (Sun et al. 2016).
The ADMM method solves problems such as in (I). where the constraint has the form f(Ae= - y) (c.g. Section 4.1). and program (1] which is rewritten using S E m2mx Pe p2mxp as
minimize
h1 (2:) + h2(z) subject to Sx + P
(12)
#ER" RERP
We provide details on how to cast (1 into program 12 in Appendix [] To solve 12) we form the
scaled augmented Lagrangian as L, z, v) = h + h2(z) + [| St + Pz y + v vlž where
v E RT is the dual variable, and > 0. The optimization of C,(x, 2, v) is summarized as
(++)) = arg min (x)) + f11s + P2()) y + x "0
x =ER"
(1+1) = arg min (h2(2) + P (x+11)) Px y + 01013)
(13)
ZER²
(++1) == (t) Sz + (t+1+)) y.
7

---

Published as a conference paper at ICLR 2024
The convergence of the above algorithm is well-known and established for convex hz hz(x),, and hz(2) (Boyd et al., 2011). However, the convergence properties of ADMM are unknown in the quasi-invex (hence invex) space. Here, we extend the ADMM algorithm that its benefits are available to the signal restoration problems under settings hased on Theorem R as follows.
Theorem 9. Let hy (zr, (2)(2) be admissible functions or quasi-invex constructs in Theorem 3 with
(S) 2 1, and pa, (P) 1. Assume there exists (a=", ,"", ,,") for which Lp(a", 2,, 0) S
Cp(x", x², v²) < Ca 2, v*), for all x, 2, and v. Then, equation (13) satisfies: i) Residual |||(() l2 Sxx(1 + Pz - Ull2 + 0 and el()) + v" as t + oc at a rate of O11/t), where v" is the dual optimal point; ii) ht (xx()) ) + hz(2((t) approaches the optimal value.
Proof of Theorem 9 relies on Theorem g which ensures uniqueness in the estimations of (+++) in equation [13). This uniqueness is not obtained by prox-regular functions which highlights the significance of our proposed family of invex/quasi-invee functions. The proof is pre- sented in Appendix along with applications that satisfy the mild constraints S, S. Appendix !] also provides the proximal solution for the studied invex/ functions along with their com- putational time. These solutions are used perform the numerical experiments in Section 5
5 EXPERIMENTAL EVALUATION
We performed three experiments to evaluate the utility of the proposed invex/quasi-ine -nnvexxxsiiiixxxxx functions against the state-of-ther the-art, and we provide only the key results in Table Additional set of results can be found in Appendix K Across the experiments below we use the following metries: absolute error (AbsError), peak signal-to-noise ratio (PSNR), root mean square error (RMSE), and structural similarity index measure (SSIM). Lower (higher) values for AbsError and RMSE (PSNR and SSIM) imply better signal restoration quality. In addition to AbsError and RMSE, we have also used square root of the AbsError (SqrtError), and log of the RMSE as additional metrics.
Experiment 1: Here, we focus reconstructing dirreeddmessional volumes for Computed Tomog- raphy (CT). CT is a powerful imaging technique with applications to various domains, including medical and material sciences (Ginat & Gupta Jørgensen et al. (021). Mathematically, CT is an inverse program modeled as in (I) where the constraint is given by f(Ax - y) < E with
A E 1m* describing the image formation model of the noisy measurements y E 1 (Xiang
et al. (021). Many model-based recovery algorithms have been proposed to solve (T) using g(ø)
as the fi-norm such as the Iterative Hybrid Gradient algorithm (Kamilov et al.) 2023). However, the performance of the model-based based strategies has been overcome by hybrid methods (i.e., unrolled networks) such as the Fast Iterative Shrinkage Thresholding Network (FISTA-Nee (Xiang et al.) (021). Here, we consider FiSTA-Net as the ofate-of-the-art in CT due to its performance and because FiSTA-Net solves (I) where g(Φ) is the convex & norm and the training loss function f(x) is the fg-norm. Then, the invex/quasi- invex regularizers and losses to be tested accordingly in this experiment are equations 3
SaannkageeThreshoodin Algorithm, ADMM, and Primal-Dual
(T))
and equations 4 4-88 respectively. To train FISTA-Net (with invex/quasi-invex functions above), we follow the experimental setting as in (Xiang et al.) (021). The training image dataset contains 2,378 full-dose CT images from eight of ten material sciences objects]] The reference images were reconstructed by iradon operator in MATLAB using all 720 views (Han & Ye 2018 Jin et al., 2017]. The validation set consists of 409 images and 330 slices for the test set.
Here we focus on image spectral reconstruction (ST), which has profound applica- tions in medical imaging, remote sensing, and material sciences (Wang al. (023). ST focuses on estimating the spectral reflectance of scenes from a set of RGB images captured at different wavelengths (Cai et al. 2022). Mathematically, spectral imaging is an inverse program modeled as in (1) where the constraint is given by f(A y) < with A E Rmxx describing the image formation model of the noisy measurements y E IR' (Arce et al. 2013). Conventional model-based methods adopt hand-crafted priors such as sparsity. total variation, and non-local similarity to reg- ularize the reconstruction procedure Arguello et al. (2023). However, these methods result in poor generalization ability due to convex regularizers like é noorr,, and unsatisfactory reconstruction quality (Cai et al. 2022). Deep learning methods such as the Multi-stage Spectral-wise Transformer (MST++) (Cai al. (2022) has shown to exhibit the best performance because it does not employ regularizer. Originally the fidelity term f(x) of MST++ is the fy-norm. Then, the invex losses to
Experiment 2:
be tested here are equations
To train MST++, we followed the setting in (Cai et al. 2022)
Acquiredd with the ISIS Neutron and Muon Source at the Rutherford Appleton Laboratory
8

---

Published as a conference paper at ICLR 2024
Table 3: Performance Results: Best: green, and the worst: red.
Experiment I (Combination of functions used to train FISTA-Net)
Regularizer
Metrics
equation
equation
equation
equation
equation [
Co-norm
AbsEaror
0,1676
0.1331
0.1543
0.1246
0,1429
0,1834
equation
RMSE
0.0101
D:0075
0.0086
39.68
0.0080
40.78
40.04
0.0093
0.0110
0,9590
0,9608
AbsError
0,3307
0,2909
0,3112
0.2798
0,3079
0,3464
equation 2
PSNR
37.75
38.32
38,00
38.65
38.47
37,50
RMSE:
0,0221
0.0176
0.0156
0.0132
0.0116
0.0286
SSIM
0,9523
0,9551
0.9542
0.9570
0.9561
0.9533
AbsError
0,185y
0.1716
0,1493
0.1401
0.1601
0,2025
squation 10
PSNR
39,00
39,96
39.33
40.71
39.55
38: 68
RMSE
0.0101
0.0125
SSIM
0,9577
0.9590
0,9597
0.9610
0.9603
0,9583
AbsError
0,4106
0,4004
0,3814
0.3726
0,3906
0.4213
Xiang al.
PSNR
37.45
37.94
37.69
38.44
38.18
37.22
RMSE
0.0290
0
0.0222
0.0180
0.0151
0.0420
SSIM
0.9510
0.9540
0,9530
0.9560
0.9550
0,9520
Experiment 2 (Loss functions used to train MST++)
Metrics
equation
equation
equation
equation
equation
fo-norm
AbsEmrr
0.1975
0.2056
0.2241
0.1830
0.1900
0.2145
SentEmrr
1.3752
1.3982
1,4465
1.4219
1.3530
1,3315
RMSE
2.6847
2,9390
2
2,2887
LogRMSE
0,2683
0.2851
0,2943
0.2534
0,2606
0.2765
Experiment
variation liltering using
Metries
TV-2p
Lavex LV-LLp
Quasi-inivx LV-L2z
2y-norm
SSIM
0.6137
0.6320
0.6227
0.6050
MS-SSIM
0,9192
0.9235
0.91449
0.9106
ADMM-residual
10
10
and Appendix K
The image dataset is from (Arad et al. 2022), which contains 1,000 RGB-spectral
pairs. This dataset is split into the train, valid, and test subsets in the ratio of 18:1:1.
This experiment focuses solving (11 As mentioned in Section 4.2 the classical total variation filtering, with global optima guarantees, is defined using the f-norm, then the invex/quasiiinvee invex regularizers to be tested are the TV-regularizer version of L. fp-quasinorm (TV- tp), and equation (3) (Invex TV) equation (10) (Quasi-invex TV.LL,)). We defer the exact details around the ADMM implementation the supplementary material (Section K). We fixed the number iterations of the ADMM method to 1,000 for all tested regularizers, and we included the smallest residual value ||r(()] ||2 across iterations to characterize the accuracy of the restored image. The dataset employed here has 40 images'. These images contain the neutron attenuation properties of the object, which helps analyze the material structure. Here, in addition to SSIM, we also use Multi-scale SSIM (MS-SSIM) and residual error of the ADMM (ADMM-residdua) metrics.
Experiment 3:
6 DISCUSSION, LIMITATIONS AND CONCLUSIONS
In this paper, by defining family of admissible functions with relevant properties, we identified invex/quasi-invex functions support real-world signal processing problems signal restoration. We then provided proof for these functions' invex/quasi-invex behaviors and global optimality be- yond the realm of convexity with their convergence rate(s). Specifically, the breakthrough of Theo- rem compared with current invex literature lies in that the sum of two proposed invex/quasi- invexxxxiiivvxxxxxx functions also invex/quasi-invex. And the results in Theorem / and Appendix show that both ADMM and APGM methods with invex/quasi-invex losses, are as efficient solving the well- established convex optimization problems.
We provided detailed evaluations in Table and Appendix K of these functions in imaging tasks such as computer tomography, spectral imaging, and total-variation regularization. These results show significant benefits of using the proposed family of invex/quasi-inver functions from theo- retical and empirical aspects. Specifically, they overcome the performance of convex mappings in terms of reconstruction quality across several relevant metrics and the studied imaging tasks. The superiority of the tested invex/quasi-invex functions is because they provide a more accurate model of the sparsity and abnormal patterns in the data than convex mappings (Sun & Pong 2022).
While our theoretical analysis to handle signal restoration problems the first of its kind and per- forms well number of tasks, its applications beyond imaging tasks are yet be explored. More specifically, the application of these algorithms around deep learning research yet to be explored, which can pave a way to improve several downstream applications, such as low-rank matrix recov- ery and alike. As pure algorithm. a number of things can be improved, such as decoupling the from (x), improving the efficiency of the algorithm beyond what ADMM and APGM can offer.

---

Published as a conference paper at ICLR 2024
REFERENCES
Cupy library.
Accessed: 2023-09-11.
http:///ssss.o.......//////////r html Ac- cessed: 2023-09-11.
kernel.
Kodak image dataset.
Accessed: 2023-08-08.
Pytorch and cupy.
http:///oos..cuyy..d///////ss//p guw_gguiieeeeeeeee htw.::/www........oooooooooooaa. 2023-09-11.
Accessed:
Boaz Arad, Radu Timofte, Rony Yahel, Nimrod Morag, Amir Bernat, Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang. Yulun Zhang, Hanspeter Pfister, Luc Van Gool, Shuai Liu, Yongqiang Li, Chaoyu Feng, Lei Lei, Jiaojiao Li, Songcheng Du, Chaoxiong Wu, Yihong Leng, Rui Song, Mingwei Zhang, Chongxing Song, Shuyi Zhao, Zhiqiang Lang, Wei Wei, Lei Zhang, Renwei Dian, Tianci Shan, Anjing Guo, Chengguo Feng, Jinyang Liu, Mirko Agarla, Simone Bianco, Marco Buzzelli, Luigi Celona, Raimondo Schettini, Jiang He, Yi Xiao, Jiajun Xiao, Qiangqiang Yuan, Jie Li, Liangpei Zhang, Taesung Kwon, Dohoon Ryu, Hyokyoung Bae, Hao-Hsiang Yang, Hua-En Chang, Zhi-Kai Huang, Wei-Ting Chen, Sy- Yen Kuo, Junyu Chen, Haiwei Li. Song Liu, Sabarinathan, K Uma, B Sathya Bama, and S. Mohamed Mansoor Roomi. NTIRE 2022 spectral recovery challenge and data set. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. $63-881,
June 2022.
Gonzalo R Arce, David Brady, Lawrence Carin, Henry Arguello, and David S Kittle. Compressive coded aperture spectral imaging: An introduction. IEEE Signal Processing Magazine, 31(1): 105-115, 2013.
Henry Arguello, Jorge Bacca, Hasindu Kariyawasam, Edwin Vargas, Miguel Marquez, Ramith Het- tiarachchi, Hans Garcia, Kithmini Herath, Udith Haputhanthri, Balpreet Singh Ahluwalia, et al. Deep optical coding design in computational imaging: data-driven framework. IEEE Signal Processing Magazine, 40(2):75 88, 2023.
Adil Bagirov, Napsu Karmitsa, and Marko M Mäkelä. Introduction to Nonsmooth Optimization: theory, practice and software. Springer, 2014.
Adarsh Barik and Jean Honorio. Fair sparse regression with clustering: An invex relaxation for combinatorial problem. Advances Neural Information Processing Systems, 34, 2021.
Jonathan T Barron. A general and adaptive robust loss function. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4331 4339,, 2019.
Mokhtar S Bazaraa and Chitharanjan M Shetty. Foundations of optimization, volume 122. Springer Science & Business Media, 2012.
Amir Beck. First-order methods in optimization. SIAM, 2017.
Amir Beck and Mare Teboulle. fast iterative shrinkagg-threshooldnng algorithm for linear inverse problems. SIAM journal imaging sciences, 2(1):183-22) 202, 2009.
Radu Ioan Bof, Emö Robert Csetnek, and Szilárd Csaba László. An inertial hrwaaddbbackward algorithm for the minimization of the of two nonconvex functions. EURO Journal on Com- putational Optimization, 4(1):3-25, 2016.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge uni- versity press, 2004.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Pelcato, Jonathan Eckstein, et al. Distributed optimiza- tion and statistical learning via the alternating direction method of multipliers. Foundations and TrendsS in Machine learning, 311) 2011.
Richard I. Burden and Douglas Faires. 2.1 the bisection algorithm. Numerical analysis, 1985.
10

---

Published as a conference paper at ICLR 2024
T Tony Cai and Lie Wang. Orthogonal matching pursuit for sparse signal recovery with noise. IEEE Transactions on Information theory, 57(7):4680-4663 2011.
Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, Radu Timofte, and Luc Van Gool. Mst++; Multi-stage spectral-wise transformer for efficient spectral reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
745-755, 2022.
Ariolfo Camacho, Edwin Vargas, and Henry Arguello. Hyperspectral and multispectral image fusion addressing spectral variability by an augmented linear mixing model. International Journal of Remote Sensing, 43(5):1577-1660 2022.
EJ Candès and Benjamin Recht. The trace norm and nuclear norm of matrix. Communications on Pure and Applied Mathematics, 59(7):849-88) 883, 2006.
Emmanuel J Candès and Michael Wakin. An introduction compressive sampling. IEEE signal processing magazine, 25(2):21-30, 2008.
Emmanuel J Candes, Michael B Wakin, and Stephen P Boyd. Enhancing sparsity by reweighted & minimization. Journal of Fourier analysis and applications, 14(5):877-905, 2008.
Rafael E Carrillo, Kenneth E Barner, and Tuncer C Aysal. Robust sampling and reconstruction methods for sparse signals the presence of impulsive noise. IEEE Journal of Selected Topics in Signal Processing, 4(2):392-400) 2010.
Rafael E Carrillo, Ana B Ramirez, Gonzalo R Arce, Kenneth E Barner, and Brian Sadler. Ro- bust compressive sensing of sparse signals: A review. EURASIP Journal on Advances in Signal Processing, 2016:1-17. 2016.
Antonin Chambolle. An algorithm for total variation minimization and applications. Journal of Mathematical imaging and vision, 20:89-97,, 2004.
Scott Shaobing Chen, David L Donoho, and Michael Saunders. Atomic decomposition by basis pursuit. SIAM review, 43(1):129-159) 2001.
Juan Carlos De los Reyes, C-B Schönlieb, and Tuomo Valkonen. Bilevel parameter learning for higher-order total variation regularisation models. Journal of Mathematical Imaging and Vision, 57(1):1-25, 2017.
Wei Deng, Ming-Jun Lai, Zhimin Peng, and Wotao Yin. Parallel multi-block admm with (11kk) convergence. Journal of Scientific Computing, 71:712-736, 2017.
Jie Ding, Vahid Tarokh, and Yuhong Yang. Model selection techniques: An overview. IEEE Signal Processing Magazine, 35(6):16-34 2018
Matthias Ehrhardt and Marta Betcke. Multicontrast MRI reconstruction with structure-guided
total variation. SIAM Journal on Imaging Sciences, 103)-1004-1166 2016.
Vania Vicira Estrela, Hermes Aguiar Magalhães, and Osamu Saotome. Total variation applications in computer vision. In Handbook of Research Emerging Perspectives in Intelligent Pattern Recognition, Analysis, and Image Processing, pp. 41-64. IGI Global, 2016.
Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association, 96(456): 1348-1360, 2001.
Simon Foucart and Holger Rauhut. A Mathematical Introduction to Compressive Sensing. 2013. 0817649484. doi: 1011007/978-0-8176--55
ISBN
Pierre Frankel, Guillaume Garrigos, and Juan Peypouquet. Splitting methods with variable metric functions and general convergence rates. Journal of Optimization Theory for 165(3)))6665338744447877744 165(3):874-900, 2015. and Applications,
Yun Fu. Low-rank and sparse modeling for visual analysis. Springer, 2014.
11

---

Published as a conference paper at ICLR 2024
Donald Geman and Chengda Yang. Nonlinear image recovery with half half-quadratic regularization. IEEE transactions on Image Processing, 4(7):932 946. 1995.
Daniel Thomas Ginat and Rajiv Gupta. Advances in computed tomography imaging technology.
Annual review of biomedical engineering, 16:431-453, 2014.
Roland Glowinski. On alternating direction methods of multipliers: a historical perspective. Mod-
eling, simulation and optimization for science and technology, pp. -9-82, 2014.
Rémi Gribonval and Morten Nielsen. Highly sparse representations from dictionaries are unique and independent of the sparseness measure. Applied and Computational Harmonic Analysis, 22 (3):335-355, 2007.
Andrés Guerrero, Samuel Pinilla, and Henry Arguello. Phase recovery guarantees from designed coded diffraction patterns in optical imaging. IEEE Transactions on Image Processing, 29:3687- 5697, 2020.
Yoseob Han and Jong Chul Ye. Framing U-Net via deep convolutional framelets: Application to sparse-view CT. IEEE transactions on medical imaging, 1416):141-1443 2018.
Morgan A Hanson. On sufficiency of the Kuhn-Tucker conditions. Journal of Mathematical analysis and applications, 80(2):545- 550, 1981.
Charles R Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array program- ming with numpy. Nature, 585(7825):357-23 2020.
Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5875-5884, 2021.
Yinan Hu and Ivan W Selesnick. Nonconvex Haar-TV TV denoising, Digital Signal Processing, 107: 102855, 2020.
Marian-Daniel lordache, José Bioucas-Dias, and Antonio Plaza. Total variation spatial regulariza- tion for sparse hyperspectral unmixing. IEEE Transactions on Geoscience and Remote Sensing, (111:)444445522 2012.
Andrés Jerez, Samuel Pinilla, and Henry Arguello. Fast target detection via template matching in compressive phase retrieval. IEEE Transactions on Computational Imaging, 6:934-944, 2020.
Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser. Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Processing, 26(9): 4509 4522, 2017.
Jeremy Johnston, Yinchuan Li, Marco Lops, and Xiaodong Wang. ADMM-Net for communication interference removal in sepped-frequenge radar. IEEE Transactions Signal Processing, 69; 2818-2832, 2021.
Jakob S Jørgensen, Evelina Ametova, Genoveva Burca, Gemma Fardell, Evangelos Papoutsellis, Edoardo Pasca, Kris Thielemans, Martin Turner, Ryan Warr, William RB Lionheart, al. Core imaging library-part I: A versatile python framework for tomographic imaging. Philosophical Transactions of the Royal Society A, 379(220))20200192 2021.
Ulugbek Kamilov, Charles A Bouman, Gregery T Buzzard. and Brendt Wohlberg. Plug-and- play methods for integrating physical and learned models in computational imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 40(1):85 97, 2023.
RN Kaul and Surjeet Kaur. Optimality criteria in nonlinear programming involving nonconvex functions. Journal of Mathematical Analysis and Applications. 105(1):104-112, 1985.
Razich Keshavarzian, Ali Aghagolzadeh, and Tohid Yousefi Rezaii. Up norm regularization based group sparse representation for image compressed sensing recovery. Signal Processing: Image Communication, 78:477-493, 2019.
12

---

Published as a conference paper at ICLR 2024
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Dmitry Kovalev and Alexander Gasnikov. The first optimal algorithm for smooth and strongly- convex-strongly-conee minimax optimization. arXiv preprint 0cXiv:2055006655 2022.
Amine Laghrib, Lekbir Afraites, Aissam Hadri, and Mourad Nachaoui. A non-convex PDE- constrained denoising model for impulse and gaussian noise mixture reduction. Inverse Problems and Imaging, 17(1):23-67, 2023.
Huan Li and Zhouchen Lin. Accelerated proximal gradient methods for nonconvex programming. Advances in neural information processing systems, 28, 2015.
Li. Solving blind ptychography effectively via linearized alternating direction method of multi- pliers. Journal of Scientific Computing, 94(1):19, 2023.
Junli Liang, Petre Stoica, Yang Jing, and Jian Li. Phase retrieval via the alternating direction method of multipliers. IEEE Signal Processing Letters, 25(1):5 2017.
Fan Lin, Yingpin Chen, Lingzhi Wang, Yuqun Chen, Wei Zhu, and Fei Yu. An efficient image re- construction framework using total variation regularization with ey-quasinorm and group gradient sparsity. Information, 10(3): 15, 2019.
Xingguo Liu, Yingpin Chen, Zhenming Peng, Juan Wu, and Zhuoran Wang. Infrared image super- resolution reconstruction based on quaternion fractional order total variation with lp quasinorm. Applied Sciences, 8(10):1864, 2018.
Trang C Mai, Hien Quoe Ngo, and Le-Nam Tran. Energy-efficient power allocation in cell-free massive MIMO with zero-forcing: First order methods. Physical Communication, 51:101540, 2022.
Goran Marjanovic and Victor Solo. On l, optimization and matrix completion. IEEE Transactions on signal processing, 60(11):5714-5724, 2012.
Anke Beyer-Bäse, Anke Meyer-Baese and Volker J Schmid. Pattern Recognition and Signal Anal- ysis in Medical Imaging. Academic Press, 2004.
Shashi K Mishra and Giorgio Giorgi. Invexity and optimization, volume 88. Springer Science & Business Media, 2008
Frederick Mosteller and John W Tukey, Data analysis and regression. A second course in statistics. Addison-Wesler series in behavioral science: quantitative methods, 1977.
Giorgos Mountrakis, Jungho Im, and Caesar Ogole. Support vector machines in remote sensing: A review. ISPRS journal of photogrammetry and remote sensing. 66(3):247-259, 2011.
Peter Ochs, Yunjin Chen, Thomas Brox, and Thomas Pock. iPiano: Inertial proximal algorithm for nonconvex optimization. SIAM Journal on Imaging Sciences, 7(2): 2014.
Yuyuan Ouyang, Yunmei Chen, Guanghui Lan, and Eduardo Pasiliao Jr. An accelerated linearized alternating direction method of multipliers. SIAM Journal on Imaging Sciences, 8(1):644-681) 681, 2015.
Diethard Ernst Pallaschke and Stefan Rolewiez. Foundations of mathematical optimization: convex analysis without linearity, volume 388. Springer Science & Business Media, 2013.
Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in optimization, 1(3): 127-239, 2014.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualle Nerfies: Deformable neural radiance fields. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 5865-5874, 2021.
13

---

Published as a conference paper at ICLR 2024
Samuel Pinilla, Tingting Mu, Neil Boure, and Jeyan Thiyagalingam. Improved imag- ing by invex regularizers with global optima guarantees. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 10780-10794. Curran Associates, Inc., 2022a. URL https::/prrceedd neurips cc httpp//ss..cpppeeeeeeeess/aapaee File//5/7927330 http////799112001aaaa11111100010
Paper Conference. Peneeeeeeeeee-nnnneeeeeeeeeeee httprrrrrieeeeeppp///2222
Samuel Pinilla, Seyyed Reza Miri Rostami, Igor Shevkunov, Vladimir Katkovnik, and Karen Egiazarian. Hybrid diffractive opties design via hardware-in-the-loop methodology for achro- matic extended-depth-ooffele imaging. Optics Express, (0018):32333322499 2022b.
Samuel Pinilla, Kumar Vijay Mishra, Igor Shevkunov, Mojtaba Soltanalian, Vladimir Katkovnik, and Karen Egiazarian. Unfolding-aidee aided bootstrapped phase retrieval in optical imaging: Explain- able AI reveals new imaging frontiers. IEEE Signal Processing Magazine, 40(2))466600 2023.
Apostolos F Psaros, Kenji Kawaguchi, and George Em Karniadakis. functions. Journal of Computational Physics, 458: 11121, 2022.
Meta-learning PINN loss
Juan Marcos Ramirez, José Ignacio Martínez-Torea and Henry Arguello. LADMM-Net: An un-
rolled deep network for spectral image fusion from compressive data. Signal Processing, 189: 108239, 2021.
Thomas W Reiland. Nonsmooth invexity. Bulletin of the Australian Mathematical Society, 42(3): 437-446, 1990.
Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009.
Carola-Bibiane Schönlieb, Andrea Bertozzi, Martin Burger, and Lin He. Image inpainting using fourth-order total variation flow. In SAMPTA 09, pp. Special-session, 2009.
Ivan Selesnick, Alessandro Lanza, Serena Morigi, and Fiorella Sgallari. Non-convex total variation regularization for convex denoising of signals. Journal of Mathematical Imaging and Vision, 62 (6-7):825 841, 2020.
Yunquan Song, Zitong Li, and Minglu Fang. Robust variable selection based on penalized composite quantile regression for high-dimensional single-index models. Mathematics, 10(12):2000, 2022.
Jian Sun, Huibin Li, Zonghen Xu, et al. Deep ADMM-Net for compressive sensing MRI. Advances in neural information processing systems, 29, 2016.
Shuqin Sun and Ting Kei Pong. Doubly iteratively reweighted algorithm for constrained compressed sensing models. arXiv preprint w2Xiv:2206.082000 2022.
Yuli Sun, Hao Chen, Jinxu Tao, and Lin Lei. Computed tomography image reconstruction from few views via log-norm total variation minimization. Digital Signal Processing, 18:172-18 2019.
Mujahid Syed, Panos Pardalos, and Jose Principe. Invexity of the minimum error entropy criterion. IEEE Signal Processing Letters, 20(12): 1es:1159-1166 1162, 2013.
Mujahid N Syed, Panos M Pardalos, and Jose C Principe. On the optimization properties of the correntropic loss function in data analysis. Optimization Letters, 8(3):823-39, 2014.
Vladimir Tankovich, Christian Hane, Yinda Zhang, Adarsh Kowdle, Sean Fanello, and Sofien Bouaziz. HITNET: Hierarchical iterative tile refinement network for real-time stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14362-14372. 2021.
Edwin Vargas, Oscar Espitia, Henry Arguello, and Jean- Tourneret. Spectral image fusion from compressive measurements. IEEE Transactions Image Processing, 28(5):2271-228) 2018a.
Edwin Vargas, Samuel Pinilla, Jorge Bacca. and Henry Arguello. Robust formulation for solving underdetermined random linear system of equations via ADMM. 2018 IEEE Statistical Signal Processing Workshop (SSP), pp. 287-287. IEEE, 2018b.
14

---

Published as a conference paper at ICLR 2024
Edwin Vargas, Henry Arguello, and lean-Yves Tourneret. Spectral image fusion from compressive measurements using spectral unmixing and a sparse representation of abundance maps. 1EEE Transactions Geoscience and Remote Sensing, 57(7):5043-500 2019.
Peter Vouras, Kumar Vijay Mishra, Alexandra Artusio-Glimpse, Samuel Pinilla, Angeliki Xenaki, David W Griffith, and Karen Egiazarian. An overview of advances in signal processing techniques for classical and quantum wideband synthetic apertures. arXiv preprint (xXi::2255005622 2022.
Jiaxi Wang, Li Zeng, Chengxiang Wang, and Yumeng Guo. Admm- deep reconstruction for limited-angle CT. Physics in Medicine & Biology, 64(11): 15501, 2019a.
Mengyuan Wang, Wei He, and Hongyan Zhang. A spatial-spectral transformer network with total variation loss for hyperspectral image denoising. IEEE Geoscience and Remote Sensing Letters, 2023.
Rongfang Wang, Yali Qin, Zhenbiao Wang, and Huan Zheng. Group-based sparse representation for compressed sensing image reconstruction with joint regularization. Electronics, 11(2):182, 2022.
Weina Wang and Yunmei Chen. An accelerated smoothing gradient method for nonconvex nons- mooth minimization in image processing. Journal Scientific Computing, 90(1):1-28, 2022.
Yu Wang, Wotao Yin, and Jinshan Zeng. Global convergence of ADMM in nonconvex nonsmooth optimization. Journal of Scientific Computing, 78:29-63. 63, 2019b.
Kaixuan Wei, Angelica Aviles-Rivero, Jingwei Liang, Ying Fu, Hua Huang, and Carola-Bibiane Schönlieb. TFPNP: Tuning-fre free long-and-play proximal algorithms with applications to inverse imaging problems. Journal of Machine Learning Research, 23(16):1-44 1-8 2022.
Fei Wen, Lei Chu, Peilin Liu, and Robert C Qiu. A survey nonconvex regularization-based sparse and low-rank recovery in signal processing, statistics, and machine learning. IEEE Access, 6:69883-69998 69906, 2018.
Joseph Woodworth and Rick Chartrand. Compressed sensing recovery via nonconvex shrinkage penalties. Inverse Problems, 3207):075004, 2016
Rui Wu and Di-Rong Chen. The improved bounds of restricted isometry constant for recovery via fp-minimization. IEEE transactions on information theory, 59(9):6142 6147, 2013.
Jinxi Xiang, Yonggui Dong, and Yunjie Yang. FISTA-net: Learning fast iterative shrinkage thresh- olding network for inverse problems in imaging. IEEE Transactions Medical Imaging, 40(5): 1329-1339, 2021.
Xingyu Xie, Jianlong Wu, Guangean Liu, Zhisheng Zhong, and Zhouchen Lin. Differentiable lin- earized ADMM. In International Conference Machine Learning, pp. 6902-6911. PMLR, 2019.
Constantin Zälinescu. A critical view on invexity. Journal of Optimization Theory and Applications, 162(3):695-704, 2014.
Cun-Hui Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of statistics, 38(2):894 942, 2010.
Zhihua Zhang, James T Kwok, and Yin Yeung. Surrogate maximizatiio/miniiizzaaooo algo- rithms and extensions. Machine Learning, 69(1):1-33, 2007.
Abdelhak M Zoubir, Visa Koivunen, Yacine Chakhchoukh, and Michael Muma. Robust estimation in signal processing: A tutorial-style treatment of fundamental concepts. IEEE Signal Processing Magazine, 29(4):61-80 2012.
15

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM 3
In order to establish a versatile framework for invex functions, we provide proof of invexity for specific class of summation fuucttion. The key result encapsulated in the following lemma, which reveals a fundamental connection: the invexity of the summation function, obtained by ap- plying a unidimensional real function to distinct entries of vector, hinges upon the invexity of this underlying function.
Lemma (Invexity of Function Sum). Let g R + R be a function defined as
((x) Σ r(<li)))
(14)
where r R + R. If (w) is invex function then g(x) is also invex.
Proof. Let R" + R be a function defined as g(w) = Em= r(xl))). Assume r R - is invex. R R such that
Then, from the invexity of r(w) we have that there exists 7/,
r(w)) r(w2) 2 Sug 17 (w1, W2),
(15)
for all W1, We E R, and any Sux E @r(w)). Take x, y E R". Then, we have
r(xli)) (yli)) Tird (x[i). yll),,
(16)
for any == I,..., n, and VGyM E Or(y) From the above inequality we conclude that for any Og(y)
(x]]) - r(ylit)
San (=]l), vli))
g(x) - -gy) 2 Σ Sulal nh (x]1)) vll))
g(0) (y)) 2 <" n)( y),
(17)
such that y) [a=[(r] [1], v(1)) nr (2|n)) ,(m)) and S = [Gwta)..... S. Thus,
from equation (17 the results holds.
Now prove Theorem 3 using the result in Lemma 2
Proof. We proceed by cases.
Let f be an admissible function. Then according Definition 4 we know that f(x) n E ([]=[i]]). for some [0, oo) - [0. oo) with s((w) > 0 where E (0, oo). Since the structure of fits assumption in Lemma 2 then if s(w) invex then f(x) is invex. Take
W1: W2 E (0, oo), and define (0, )2 + R
77(11, (()) = {
0 (ww) - s(wn) Co
if ((w)) > s(w2)
(18)
otherwise.
where is an element in @s(w)) of minimal absolute value which satisfies 5: 1 for all Ç E @w()). The existence ç, is guaranteed because s'(w) > 0 and therefore 0 & @s(w))
(Mishra & Giorgi 2008, Page 64). From the above equation is clear that for all wh, W2 have
s(wi) - s(w2) 2 newi,, Swazy VGwz E as(w2)
(19)
which means is invex. Therefore ((x) is invex.
Let f. be an admissible functions. Considering the result in previous statement, is enough show that h 8f ag is an admissible function for any B, > 0. By definition have that f(0)) = 0, and h h'(w) > addition, since both f. are positive functions it implics that hww)/w² is non-increasing. Thus the result holds.
16

---

Published as a conference paper at ICLR 2024
Let f.g R" R be two admissible functions as in Definition such that f(x) Di=1 $/ ([+[i])), and g(z) = (| Define (x(2)) = Di=1 o sg) [+]]] Then, observe that (sy o 8g)(0) (5))) = s0( O. Additionally, since s (w),, ( > 0, then (sy sy)(w) > 0 (by the chain rule) for all & E (0, oo). Finally, we know that 5/y (w),, 5y (w)
implies sy(w), sy(w)
be strictly increasing. Therefore, for any w1 < W2 we know
(sf (w) < (sy (w2), which implies (8y0s,)(wx) > (e))),)(w>). Thus the result holds. (w2)²
(w1)²
Thus proved the first part of this theorem.
Now we proceed to prove the second part of this theorem.
Proof. Let f R" + R be two admissible functions as in Definition such that f(x)) Ei and sy( 2 Define the set
{w (0, oo) (w) 0}. In conse-
quence, from first statement of this theorem proved above we know that s sy(w) is invex in D for some function TjD. Then, define for any w1, u2 E (0, 0o) the following function
nD W2)
if W1, W2 E otherwise
new, w2)
(20)
Considering the above 17(11, w2) function (which is non-zero), take w1, We E (0, oo), and assume $ (w1) - 8y (w2) S 0.
(21)
If wi; W2 DD then know ()(w) is invex, which implies that
sy sy(w1) (22) 2 n(w1, Swy = nD (w1, w2) Suz 2 7D(11, W2) Swz:
(22)
for all Suz (Bsy((n) (wz), and therefore quasi-invex. Otherwise, if either wi & or wz & D, it clear to conclude that 7 w2) Swz 0, for all Smz E @ Thus, the result holds.
Finally, to establish the validity of the remaining two statements when f is quasi-invex, we follow similar procedure as described above. In the interest of conciseness, we omit the detailed exposition. Thus the results holds.
B PROOF OF THEOREM
We prove Theorem proceeding by cases, exploiting the result in Lemma because equation B is the sum of unidimensional real functions applied different entries of a vector.
EQUATION
Proof. From the definition of function g(x:) in equation 3 is clear that the only aspect we have
prove that log(1 + / is non-increasing for any w > 0, and fixed p (0, 1). Take
r(w) log(1 + w². Observe that the first derivative of h(w) = r(w)/w² is given by h l(ww)
T+ 2 log(1 + w² Since
puu
pwd"
2 log(1 + w < 0, then we have that / N'(w) < 0,
T4
which leads to conclude that r(w)/ //2 w is non-increasing on (0, oo). Then it is clear that rww) /w3 2 is non-increasing on oo).
Now prove the second part of this theorem.
Proof. define the first-order difference matrix D E ((--1)xx as
1
-1
D =
(23)
17

---

Published as a conference paper at ICLR 2024
Observe that it is clear that D is full rank. Then, define (rv g (Dx) where g is admissible function. It is worth noticing that since is point-wise non-increasing when divided by quadratic mapping, then is continuously differentiable. In addition, due to the fact that D is full row rank, we appeal to (Pinilla et al. 2022a Lemma 2) that ensures grv (æ) (z)) is invex. Thus the result holds.
B.1 ADDITIONAL. DISCUSSION THEOREM 4
In this section we discuss the 2D and 3D total-variation-Ile -ike extensions of invex functions.
2D TOTAL-VARIATIO--IIK REGULARIZER
Let be g an admissible function. Then, the 2D iirl-vvriatiin- like regularizer based on is defined as
gTV r = aDa)) + (D,,))
(24)
where matrix DR, and D. model the discrete derivative in the horizontal and vertical directions, respectively. Therefore, D,, is defined as in equation (3) and D, is given
D. =
(25)
Now we prove that equation (24) is quasi-inver inasi-invex follows.
Proof. Recall in the previous section showed that each term ((Dax) and g(D,,x) are invex (since Dr. and D2 are full row-rank). Therefore, there exits m (æ, y) and m/ (2, y) for any a, y e R" such that
g (D, (DDw)) 2 (x.
(26)
for all Sa (D3)) and
g (D.xz) g((.v)) 2 (2.
(27)
for all So E (D().)) Then, assume
grv(x) (y) 0.
(28)
This leads
((Dxz) g((n)) + g
g(D,y) <
</ m. y) + 2 (x, 5
C (x, <
(29)
where C, . <T E Ögrv (y). and Mgry (a, y) = Inf (z, y). (xx y) Thus, from the
above inequality have that grv (æ)) ) is quasi-invex.
3D TOTAL- VARIATION-LIKE REGULARIZER
Let be an admissible function. Then, the 3D total-variation-like regularizer is defined as (x) = (D,xz) + g(D.z) + gDzx),
(30)
where matrix Dn, D.., Dr model the discrete derivative in the horizontal, vertical, and transversal
(third dimension) directions, respectively. Therefore, D₁ is defined in equation (23 and D,, as
in equation 25 and D, is given
1
1
D.
(31)
Now prove that equation
24 is quasi-invex follows.
18

---

Published as a conference paper at ICLR 2024
Proof. Recall in the previous section showed that each term (Dnar), g (D.x), and g(D,a)
invex (since Db. Du, and D, are full row-rank). Therefore, there exits 17h (x, y). To (2. y). and me (2, y) for any æ, y IR" such that
((nx)) - - 2 ch (m y)
(32)
for all Sn (Dg(D)))
9 g - 2 S. (a.
(33)
for all S. E dDg(D. and
gD,x) ((D.y) 2 c. (2 y)
(34)
for all & E Dg(D:y). Then, assume
grv(z) (y) s 0.
(35)
This leads to
-((kv) + -(D))() + g(xx)) - g < 0
C% n. 2) + CT (2, y) (1 5 0
S ( 50
(36)
where S, 15h , E Ogrv(y), and Mgry (x, y) = Inl (æ, y), n/ (ar, y), u).ng (1c, v)1a Thus, from the above inequality we have that (ar) ) is quasi-invex.
C PROOF OF THEOREM 5
In this proof seek to guarantee that the list of functions in Theorem 4 are admissible functions, and we proceed by cases.
EQUATION
Proof. Take r(w) = log(1 + 52 for any w * 0, and fixed R. It is trivial to see that r(0) = O, that r(w) is not identically zero, and non-decreasing on (0, 0o). Then, we just need to show that
"²)
r(w)/ /w2 is non-increasing on (0, oo). Observe that the first derivative of h(w) = r(w)/w² 2 is given
by h'(w) 2(
lo +1)) ) Since g=-wa log(1 + w² ) < 0, then we have that h' (w) < 0, a
w²
5a
which leads to conclude that r(w)/w² (w2 is non-increasing on (0, oo). Then it is clear that r(w)/ui / is non-increasing on (0, 2o).
EQUATION
# for any w * 0, and fixed 8 E IR. It is trivial to see that r(0) = 0, that r(w) it is not identically zero, and non-decreasing on (0, oo). Then, we just need to show that
Proof. Take r(w)
2w
r(w)/ut /w22 is non-increasing on (0, oo). Observe that h(w) = r r(w)/ //2 is given by h(w) = , which leads to conclude that r(w)/ /2 is non-increasing on (0, oo). Then it is clear that r(w) is non-increasing on (0, oo).
EQUATION 66
Take r(w) = exp(-ui exp(-w2/82) for any w + O, and fixed 8 E R. is trivial to see that r(0) = O. that r(w) it is not identically zero, and non-decreasing on (0, oo). Then, just need to show that
r(w) rww)/w2 is non-increasing on (0, oc). For easy of exposition we present in Figure () the plot of r(w) Then it is clear that r r(w)/ui w22 is non-increasing on (0, oo).
19

---

Published as a conference paper at ICLR 2024
0.30
RAW
03
0.
10
Figure 1: Plot of r(w)/w /w for r(w) being (a) equation (6. (b) equation 7 for e 0.1, and (c) equa- tion and check that r(w)/ / non-increasing on (0, oc)
EQUATION
-1) for any w * 0, and fixed a E IR, c > O. It is trivial to see that r(0) = 0, that r(w) ) it not identically zero, and non-decreasing on (0, oo). Then, we just need to show that r(w)/ /w2 is non-increasing (0, oo). For easy of exposition we present in Figure b) the plot of r(w)/w/ Then it clear that r(w)/ /w² 2 is non-increasing on (0, oc).
Take r(w) la-2) (( (w//))
0/2
x- +
EQUATION 8
Take r(w) log 1-+w"+ w²) 201442 for any w # 0. It is trivial to see that r(0) = 0, that r(w) it is not identically zero, and non-decreasing (0, oc). Then, just need to show that r(w)/u( (w) non-increasing on (0, oo). For easy of exposition we present in Figure I the plot of r(w)/w²
Then it is clear that r(w:)/ /w² is non-increasing (0)
D PROOF OF THEOREM
To prove that (x) is quasi-invex, for regularizers in Theorem show the existence of n) Rn R" > IR" satisfying Definition for (xx).
EQUATION
Proof. Let (x)) = EE (i)) for 0 > A (0, 1]. and
re(w)= {
\/w| w2/ (20)
if w < 02
(37)
0x2/2
if w 2 0X
First, will show that if |w| x E D (see Lemma 2). with
AA then rw) is invex. In consequence, go(x)) is invex for any
0X, Vi E D (2 E R" |
Once we prove this, we are able to build 7 R x R + R such that go is quasi-invex for any IR"
Observe that (w) for 0 < |w| < 0A is given by
(w(w)) = x w
w
(38)
The above equation implies that 0 E @r)(w) only when w = 0. Further, since r0( < re(w) for any 0 < |w| < 6A, then 0 is global minimizer of re(w). in the restricted domain, implying its invexity. Therefore, appealing to Lemma there exists n, R" x I² + R such that (x(x) is invex in D. It is worth mentioning that D is open set, because it is the Cartesian product of
20

---

Published as a conference paper at ICLR 2024
open sets. This observation is important because invexity requires compute the subgradient which only possible on open sets. Thus, the invexity of go(x) is well defined. And therefore, go is quasi-invex in D.
In consequence, define the following 12
x
12
R as
{
Thr (2
if a y otherwise
n(. y)
(39)
0
Considering the above na y) function (which is non-zero), take æ, E R2, and assume go(x) go (y) 0.
(40)
æz y E D, then we know go (x) is invex, which implies that
()( go yy) 2 (,,
0 0 n(, y),
(41)
for all Qgo(y), and therefore quasi-invex. Otherwise, if either DD y & D, it is clear to conclude that 5 17(2,)) y) = 0, for all Ç E (g) Thus, the result holds.
EQUATION 10
Proof. Let go(x) >i (a= be the regularizer in equation 10 for 0 0, and ra(a)ii)
min(r(x](i),, 0) where function r(w) takes the form of
r(w) log(1 + ([w) ))),, E (0, 0.
(42)
To prove that equation 10
is quasi-invex we first show that the unidimensional function r(w) is
invex and then we proceed to show the construction of 17 RR"" R - Ra. To prove that function
r(w) is invex observe that if w > 0 then we have that @r(w) (w]++e) 2 ([w]+e) which means
that 0 & Or(w). Conversely, if w < 0 then ar(w) [w]++e]-p-- (-([[[+e] leading to 0 & @r(w). Lets examinate w* 0. Note that + r'w) 2T- and that lim (w() --P4e Additionally, since r(w) is a Lipschitz continuous function, then appealing Theorem we have that Ör(w"
conv 2--4 21-Pte
2- 21-44 This means that 0 ar(())
Further, given the fact that r0)) < r(w) for all w E R. then w" = 0 is a global minimizer of r(w)- Therefore, the function r r(w) is invex.
Now we show the existence of R² x R= - RR" satisfying Definition 3 for go (x). First, observe that function r(w) does not reach the value for any w, then is clear the existence of 7/ : R" x R" + Rd such that go (xx))
Σ r<x(i)) is invex according to Lemma 2 and therefore quasi-invex. On the other hand, if we assume r ()) reaches the value of 8, then there exists w 2
such that r(w) < for any |w| < w". Let D (z R |z|i]]
=² Vi (1,...,,0 n)) is worth mentioning that is an open set, because it is the Cartesian product of n open sets.
This observation is important because invexity requires to compute the subgradient which is only possible on open sets. Thus, the invexity of (x(x) ) in D is well defined. Now, we define the following
y R 1 1 as
= =
No (0, y)
if y E D otherwise
(x y)
(43)
Considering the above 77(x, y) (which non-zero), take x, y E R", and assume
go (xx) - (ÿ) S 0.
(44)
If ar, y E D, then know go(x) is invex, which implies that
(x(x) go(y) <Tn(
0 (,)
(45)
for all Ç E dgo(y). and therefore quasi-invex. Otherwise. if either x & D or y & D, it clear to conclude that 5² 7,x y) 0. for all ç E Ogo(y). Thus, the result holds.
21

---

Published as a conference paper at ICLR 2024
D.1 THIRD PART OF THEOREM 6
Proof. Let g be an admissible function such that ((x) = Di=1 By ([=[]]] and s,(w) > 0. Define
@rV (x) g( (DDz) where D defined in equation (3) Then, according to Theorem 3 have
that g(x)) is quasi-invex for some (z. y), which implies that
grv(x) - 9rv (3) S 0 = (Dx, Dy) 00
(46)
for all Ç E 0g(Dy) We know that matrix is full row-rank which means that (DD*) -1 exits and (DD*) ) In-1 where nn is the identity matrix. Then, from the above inequality
we obtain
gTv(x) - (Tv 0o
<" (DD") na (Dx, Dy) 0 2d
(Do Dy)) (D" (D" no (Dæ, Dy)) (D" n(a,y) y).
(47)
where 712, y) D"DDD" ng (Dx, Dy). Further, since g is continuously differentiable con- cluded from (Bagirov et al. 2014 Theorem 3.20) we know that D Ç Ögrv(y) Thus, from equa- tion 47) we get
grv( - grv 0 = (an y), for all Ç E @grv(y), which implies (rv(()) is quasi-invex. Therefore, the result holds.
(48)
D.2 PROOF SCAD IS QUASI-INVEX
Proof. Let (x)) X Ei=r (x())))), for 0 > 2, X (0, 1}, and if |w| S A if |w| < 02 otherwise
Alu ww
ro(w)=
1-
(49)
20-1))
(0 + 1)A²/2
First, we will show that if |w] x D (see Lemma 2). with
0A then (w is invex. In consequence, gx is invex for any
D= f2 E R" |z|||| < AX, Vi E m!]..
Once we prove this, we able to build 7 R R R such that (ar) is quasi-invex for any 2 E R
Observe that (w) for |w| < 0A is given by
w
if | S
(w(w)=
aAw
(50)
if < |w| < 0X.
The above equation implies that 0 E (r)(w) only when any 0 < |w| < 8\, then w = 0 is global minimizer of (w). in the restricted domain, implying its invexity. Therefore, appealing Lemma 2 there exists 17. F ER" R such that go(x) invex in D. It is worth mentioning that D is an open set, because it is the Cartesian product of open sets. This observation is important because invexity requires to compute the subgradient which is only possible open sets. Thus, the invexity of go(x)) quasi-invex in D.
0. Further, since ro re(w) for
is well defined. And therefore, go ga(x) is
In consequence, define the following
R x R
Rd as
7 (x, y)
i D otherwise
<1 3)
(51)
0
Considering the above y) y) function (which is non-zero), take y E R and assume go(x)) - go (o( 0.
(52)
ar y E D. then we know go (x:) invex, which implies that go(x)) go <T'(0.y)
2 <T (2,
(53)
for all Ç E dgo(y). and therefore quasi-invex. Otherwise. if either x & D or y & D, clear to conclude that 5² 7,x y) 0. for all ç E Ogo(y). Thus, the result holds.
22

---

Published as a conference paper at ICLR 2024
D.3 PROOF CAPPED-E1 QUASI-INVEX
Proof. Let go( = A - ro(a) for 0 > 0, X E (0, 11. and
ro(w) min main[w],
Now we show the existence of 1y R- x l2 + R satisfying Definition for go Let = {z E R" | < 8, Vi E (1,.... n}}. Thus, define R
R R
y otherwise
na. y)
(54)
Considering the above s7(,,)) y) (which non-zero), take æ, y E R", and assume g0(x:) - (o S 0.
(55)
2x, y E D, then we know go (x:) is the & normm, which implies that
go (x) go y 2 c² y) T (a (x-y) y).
(56)
for all Ç E Ogo(y), and therefore quasi-invex. Otherwise, if either & 'D or y & D, is clear to conclude that e² 17, y) = 0, for all Ç E (go(y) Thus, the result holds.
E PROOF OF LEMMA
Proof. Let f.h R R be two invex functions (not necessarily with respect the same m) Define the function (ax) = (f(x() Now we will show that q(x) satisfies Definition
Then, take x. y E IR and assume
(xx) {(v) S O.
(57)
The above inequality implies that
max(f(z), mf h(v)) < 0 max(f(x) h(x)) max h()))))))))))
(58)
Now we proceed by cases.
1. Assume 7(y) max((()),h()))) f then from equation 58
we have
f(x) S max(f(()), (x))) & mf h(y)) = f(y)
(wx) f(y).
(59)
Since f is an invex function, then we know there exits My R x Rm - R such that
equation 59 implies
- ((w)-((3)
> 5t (, y) 0 > 5d 11 (x, y)
(60)
for all Ç E af(y) = ag(yy).
2. Assume q(y) max( h(y)) = hlyy), then from equation (58) we have
h(x) S max (axx(())),
(f))) h(y)
h(x)) < hly))
(61)
Since h is an invex function, then we know there exits Mh equation (61 implies
1 x Rm
R such that
h(x)) h(y) 2 ST mp (,, y)
0 > < m) (x, y)
(62)
for all Ç ah(y)) Aq(r)-
23

---

Published as a conference paper at ICLR 2024
From the above discussed cases, we build 17 R" x R - R²"
( y)
if f(y) h(y) if f(y) h(y)
(12 y) = 0
(63)
7hh y)
otherwise
Thus, from equation 59
equation 61 and the above definition of a,, y), we conclude
(()) - (()) S 0 = ()1 y) S 0,
(64)
Va, E IR", and Vy E Og(y). Therefore the desired result holds.
L
F GLOBAL OPTIMA GUARANTEES OF PROGRAM
To prove the global optima guarantees of optimization problem in T we formulate and prove the following theorem.
Theorem 10. Let f.g: R" -> IR be admissible functions in Definition 4 Assume xx is solution of program (T) for = 0, then x: a global minimizer.
Proof. Assume x² is minimizer of constrained optimization problem in equation (]. and define {æ E [())()) = 0}. This means that a minimizer g(x) in D i.e. x" D. Since, f(x) is an admissible function, then is invex. Therefore, xt must be global minimizer. Thus the result holds.
G PROOF OF THEOREM 7
In order to prove Theorem we introduce an auxiliary definition and lemmata as follows.
Definition 5. (Sparseness measure (Gribonval & Nielsen 2007)) Let g R² - such that g(x) where r j0, oo) + [0, 20) and increasing, If r, not identically zero, with r r(0) 0 such that r(w)/w is non-increasing on (0, oo), then g(x)) is said to sparseness meusure.
Considering the above definitions we obtain the following conclusions.
Lemma 3. (Gribonval Nielsen, 2007 Proposition 1) Let g RR be function satisfying
Definition 5 Then, g satisfies the triangle inequality.
Observe that the above lemma provides practical methodology to verify when the function g(æ) satisfies the triangle inequality.
Lemma 4. Let f : R" + IR be an admissible function. Take æ E R such that || < co.
f(x) < oc, then | c(a) fxx for constant c(x) > 0.
I
Proof. Since f an admissible function, then f(xx)) = Ei= s(x(i)) and s(w)/w² is nonincreasing. Therefore, we have for all i 1
(x
s x oc)
c(xr) > 0.
(65)
2
2
Thus, we obtain
1 c(x)
| 1
f(x).
(66)
Thus the result holds.
With the above definitions and lemmata we proceed to prove Theorem 7 This proof assumes the regularizer g(x) satisfies Definition 5 This assumption proved at the end of this section.
24

---

Published as a conference paper at ICLR 2024
Proof. From hypothesis we assume matrix A satisfies the RIP condition for any k-sparse vector with 62k < v4i
4- Then, taking C {1, nn the k-support of I, from (Foucart & Rauhut 2013
Theorem 6.13) and for constants p E (0, 1).
we have
| 1 - -5 |I + + A+ -yl2.
(67)
where the notation means that æs coincides with æ on the indices in S and is extended zero in its complement 8 Observe that by adding the term p|| i to equation 667 we conclude
|æ S
|
+
T 1 Ax yl12.
(68)
I
Since we assume g(x) satisfies Definition 5 from GGrioonval & Nielsen 2007 Theorem 5) we know that g(ms)
< || [
Thus, combining this inequality with equation (68) we get
(()))
1+ g(x) + T' A| y112 1+ p g(xs) S pg(xx=) + -'IIA'' y1l..
g(xs) - P
(69)
where 7' [(x) 00. Observe that since g(x) holds Lemma then from Woodworth & Chartrand 2016 Proposition 4.4), and equation 669 get
1 1+1
T'
g(x² x) s s + I-p AAx 3/12-
(70)
Now, since f(x) is an admissible function, from Lemma 4 and equation 770 we obtain
g(x² - x)) 2 + g (xg=) + ,, T' = VE///////(m)
(71)
where constant c(1) as in equation
66
It is important remark that the above inequality is only
relevant for g(x) = Di= Sy ([<[i]
quasi-invex when so [1ox S y (which is assumed in this
proof) for 7 min (0, 00 s (w) = 0}. otherwise sy(w) does reflect the behaviour of
æ. This assumption also applies f(n) quasi-invex which implies the constrain s (l)lo))) Y
In addition, for regularizers g(x) satisfying Definition 5 we also know from (Gribonval & Nielsen
2007 Lemma 1) that exists constant b(æz such that [ S b(s) g(x). Then, from equation
we conclude
||00 æ011 S (3),(()) DVeer,a(t))
(72)
where C = 2 1-p: (x)) = g(ane)) b(x)
D 4 1-pp and (z)(( v1/c(n)/((a).. Thus the result
holds.
SATISFYING DEFINITION 5
In this section we show that regularizers g( equation Definition 5
3 equation
9
equation
10 satisfy
EQUATION 3
Proof. Take r(w) = log(1 + (w("') for any w * O, and fixed (0, 1). It is trivial to see that r(0) O. that r(w) it is not identically zero, and non-decreasing on (0, oo). Then, we just need to show that ((w)/w is non-increasing on (0, oc). Observe that the first derivative of h(w) = r(w)) /w
log + w²) < O, then we have that N((w) < O, which leads to conclude that r(w)/u w is non-increasing on (0, oo). Then is clear that r(w)/ / is non-increasing (0, o0).
is given by h'(w)= = -
[+t
pu."
log(1 + =w)) Since pur
T+wp
25

---

Published as a conference paper at ICLR 2024
EQUATION 9
Proof. Take
re(w) {
1/w| w² (20) Ax² /2
if w < 0X
if w 2 0X
for any R and fixed X > 0, 0 2 1. It is trivial to that (0( 0. and that re(w) it is not
identically zero. Then, we just need to show that h(w) = re (w( //w is non-increasing on (0, oo).
Observe that the first derivative of h(w) r(w)// is given by
-1((20) 11 k((w)= -0X (222(22²)))))
if w < 02 if [ 2 0X
Since N' (w) < 0, then r(w)/w is non-increasing on (0, oc). Thus it is clear that r(w)/w is non- increasing on (0, oc).
H PROOF OF THEOREM
Proof. Let g 1 R be admissible function. Then from Lemma
know that g(x)
e||=1 13 for some c > O, and therefore grv 2 <'||= for some c' > 0. Then from (Pallaschke)
& Rolewicz 2013 Proposition 5.2.13) we know there exits constant d' > such that program (T]p
has a unique minimizer for any 5 20 This fact implies that program 11 is invex. It is easy to
verify that the TV-regularizer version of mappings in equation equation and equation (T)) the constant d' is equal to one.
Now for the second part of this theorem, we have that h(x) grV (a) + 21 x u11 for some
E Rn is an invex function (this is also true for h(x)= = g(x) + 2x x 21 with À S 3 then Theorem 2 states that any global minimizer of h satisfies that 0 E ( This condition implies that 0 E dgrv (H) + ( - u). from which we obtain that y E (Adgrv +I) (u). Thus, we have that prox gTV (w) = (X0grv + I) (u) (this is also true for g(x)) from which the result holds. O
H.1 REMARKS ON PROX- REGULAR FUNCTIONS
In this section we prove prox-regular function is quasi-invex. To that end, we introduce the fol- lowing definition first.
Definition 6. Let f R" + R be lower semi-continuous function, and u E R". Then f is said to be prox-regular if f(x) + 2X || - ull? is convex for some X > 0.
Now we proceed with the proof.
Proof. Let f 1 + I be prox-regular function for some X > 0. Then we know that for any x, y E R
1
((x) s(y) > <T y)
| y/1
(73)
2X
for all Ç E @f(y). Define function n)(x,)) y) as
if E 0f(g)
n/(x, y)
| - 2
otherwise
(74)
where Ç* is an element in 0f(y) of minimum norm. Take x, y E RR" and assume
f(x) - f(v) $ 0.
(75)
Observe that 0f(y) then we get T,) y) 0, for all Ç 0f(y). Additionally, 0 & 0f(y), then from equation (73) we obtain
0 >
x y)
x y1
2
T
y
2x
x
1
y ul2
5*)
= - v),
(76)
26

---

Published as a conference paper at ICLR 2024
where the second inequality comes from the fact that Ç* is element in Qf(y) of minimum norm =²C Le. T 2 for all Ç E df(y) (Bazaraa & Shetty 2012 Theorem 2.4.4). From the above
incquality the result holds.
I PROOF OF THEOREM 9
Here prove Theorem 9 following a similar strategy presented in (Parikh & Boyd 2014).
Proof. Since (x", v*) ) is saddle point for Co. we have
Lu(x" 2², v²) I (x ((++1) v*). Using Sx + P=* = the left hand side is h* = (hr + h2(z) 1((+1)= = h1 (x((+1)) + (2(2((+))), this can be written as
(77)
Sa: + Pz = =y) With
h" ("")"" (++1)
(78)
for ((+1) Sx((t+)) + (2(++)) y. Now, by definition, x (((+1) minimizes La(ar 2(0, v(0). From Appendix H and the fact that (S) > 1, popp 2 we know that the (necessary and sufficient)
optimality condition for Cp(2, ((), v(1) is given by
0 (z:((x((()) (1+1) 10, (0) 0h, (x: (c+ (x 1) + v + pS (++1) (Sx((++)) + Pz yy)
(79)
Since (+++1)= v09) + pq (1+), we can plug in (()) v((+1) (gg and rearrange obtain
(t+1) 0 E 0h (x (t+1)) + ST (v(++) - pP( (1))).
(80)
From Appendix H this implies that x (++1) uniquely minimizes
In (xx) + (v(++1) pP(z(t=1) 2(1))) So.
(81)
A similar argument shows that z ((+1) uniquely minimizes 12(22) + (v(++)))z Pz. It follows that hi (x+(1))) + (t+1)) -PP()(14))2 - =1(1)) (00))_(Sa)) Sz
s (xx)) + (x((+)) -pP(2 (2(11) x (1 "f"""""=""
(82)
and that
h2 (2+2) + (1 P2(+1)
S h2(22) + (+(t)) Pz" Pz*
(83)
Adding the two inequalities above, using Sx" + Pz" = y. and rearranging, we obtain
h (1+1) - h² < (( (++1) p( (((( (1+1) (- (1+1) + ((z -²)))
(84)
On the other hand, adding equation 78). and equation through by 2 gives
84 regrouping terms, and multiplying
((1(++1) -0")==""" (1+1) (++)) -2p (++ - ) (1+2)
+ +2p(P(x((+1) (2() (P(2(()))
- z))) S O. (85. and substituting (1+1) = -((() (t+++) it gives
(85)
Now by rewriting the first term in equation
2( )² ( (++1) (1+ + ( (111)
(86)
and substituting (+1) = (1/ (1/p) - ()))) in the first two terms gives
(2 - u²)T (2))(((tt)) (1(tt))_p(0) + (1/p))lo((-))
" pll (
(87)
Since (1+1) (0) = (q(--1) - -q") - () q²"). this can be written as
(1/0)(1)0((+1)) v01l (10 w w²1l)) و|ا (++1) 112
(88)
27

---

Published as a conference paper at ICLR 2024
We now rewrite the remaining terms
plla 21 12 2p( 2p(P(2((+1) - =11))" (0)"")((()))) ((+1)) + 20 (P( -
(P)))(( (89)
where 11 taken from equation (8 Substituting
2(+11) -z" == (z((++) - =1())
(2(1)= -2",,
(90)
in the last term gives
( - P(z - (2))11))2(0))13 + pll p()((())))) - 2(11)113
+ 2p ((x - -2(2)))(P((((
(91)
and substituting
((+1) - z(t) (xx(++]) - -2") -(x(1) -36)-22 -²))
(92)
in the last two terms, we get
p plg(tt1) -(2((+))
(0) + (IP -2a1))==))00 2²) - - -
-P((20))22)1
(93)
With the previous step, this implies that equation 88 can be written as
V - v(+11) 2 pllq (++1 ((z -
(94)
where V(O = (1/D)]lu() 0z 11 pl P(z - ==
Now, we show that the middle term- - ( (P(2((+))))))) 2(1)) of the expanded right hand side of equation (04) is positive. To see this, recall that 2(++1) minimizes h2(2) + (u(t+))TPP Pz, and z(t minimizes h2(z)) + (z( Pz, so we can add
(+12( + (o((+1)) (z((1) S (2(2((0) + (2((+)))T P22(0)
(95)
and
h²(z(0)) + (e(0))TP2') < (2(22(+ + (oll))TPPpl (2((+1)
(96)
to get that
(x(1+1) - 0(0) P(z(1+1) - =(()) S 0.
(97)
Substituting (++1) (t) = (++1) gives the result, since p > 0. Thus, from equation (44 and
equation 997 we obtain
( < v(t) (++1) 1 (()(() - (0))
(98)
which states that (t) decreases in each iteration by an amount that depends on the norm of the residual (()) and on the change in (1)) over one iteration. Then, because (tt)) I v(0), it follows that (1)) and (2( are bounded. Iterating the inequality above gives that
2
P
l1q (++1
22 + ((2(++)) 2(1))113) S V(00)
(99)
which implies that (0)
(t) + (z()) - y + O. and ((2(1+1)_ 2(t) 0 as + co.
Additionally, applying (Deng et al. 2017 Lemma 1.2) on equation p9 we obtain convergence
rate for q ((+1) -2(1) to zero of (1/1). equation ( 199) also implies that the right hand side
in equation 84 goes to zero as f + co, because ((2(+1)
2²) is bounded and both (1+1) and
(1+1) z()) go to zero. The right hand side in equation 78 goes to zero as + co, since q(t)
goes to zero. Thus we have lim htt) = h", i.c., objective convergence. Therefore the result of Theorem holds.
28

---

Published as a conference paper at ICLR 2024
1.1 REMARKS ON STABILITY OF ADMM
In this section we wish to emphasize that the stability and reliability (of the ADMM) can be estab- lished from the following: 1) ADMM decomposes the overall optimization problem into number of simpler subproblems that have a unique solution (as rigorously demonstrated in previous section), aiding convergence and stability. 2) As shown in previous section, the sequences x (t+1), 20++1,, and ((++2), constructed by ADMM algorithm, always converge to global optima irrespective of the initial states (0)), 20),, and v0(0), conferring a steadfast assurance of reliable attainment of optimal solutions, and finally, 3) The effectiveness of the ADMM is well demonstrated in the literature us- ing a range of real- world applications (Boyd et al. 2011; Wang et al. 2019b; Glowinski 2014| Xie et al. 2019), which we believe can reaffirm the reliability (and potentially the stability) of ADMM.
REMARKS ON OPTIMIZATION PROBLEM IN
In this section we show how program is expressed as equation (12). where the non-linear constrain in the form of f(Ar - y) From (Chen al. 2001 Cai & Wang 2011). we know that
minimize g(x) subject to f(Ax y) <,,
can be equivalently rewritten as
minimize g(x:) + Af(Ax y))
for some À 0 that helps to satisfy the condition f(Ax y) Thus, if introducing variable
x Ax y. program can be finally rewritten as
minimize g(x) ( x=) subject to z Ax y.
which is in the optimization form of ADMM pointed out in equation 122 In this way, optimiza- tion problem in ( is converted from non- non-linear constraints into linear ones.
1.2 ADDITIONAL DISCUSSION ON THEOREM 9
In this section, discuss why the assumptions (S) > 1, and pap(P) for > 0 in Theorem
9 are mild, by showing practical imaging examples that satisfy these conditions. We list those applications the following.
Image fusion has been receiving increasing attention the research community to investigate general formal solutions to broad spectrum of applications (Vargas et al. 2019). For example, in the remote sensing field, the increasing availability of spaceborne imaging sen- sors operating in various ground scales and spectral bands undoubtedly provides strong motivations (Camacho al. 2022). Because the trade-off imposed by the physical constraint between spatial and spectral reso resolutions, utions, spatial enhancement of rosouuesolution multispectral (MS) data is desirable
Image Fusion:
(Camacho et al. 2022). Another example is medical imaging, in which the goal of image fusion is suitable for human visual perception (Meyer- Bäse et al.) 2004).
create new images
Mathematically the image fusion problem in spectral imaging can be modeled as equation [22
defining matrices S, and P as (Vargas et al. 2019
0
I
R
M
0
0
0
0
0
I
0
0
O
I
M
0
0
0
S =
B₁
P
0
0
0
I
0
0
0
=I
H.
(100)
G
0
1
0
0
D
0
0
0
0
0
1
0
I
0
0
0
Observe that o. (S) (.)(()))))))))))) 2+ > 1 for > 0 where the smallest singular value of B B, G G+ D'D D. In addition, o, (P) 1++ where is the smallest singular value of H H" +HTH H. Thus, taking 2 max (0.5, 1/(1 {)} the needed conditions in Theorem 9 are satisfied.
Spectral Imaging: Spectral imaging combines two disciplines sppcctrssopy and photography sample image data many wavelength bands. In general, spectral imaging is separated into either
29

---

Published as a conference paper at ICLR 2024
multispectral (i 20 wavelength bands sampled) hyperspectral 20 wavelength bands) (Vargas jt al. 2a). Mathematically the spectral imaging problem can be modeled as in equation 12)
defining matrices S, and P are given by (Vargas et al. 2018a
s = I I V I
P=
(101)
where & is an orthogonal matrix. Observe that o,, (S) K +++1+8 1 for 5 > 0 where is the smallest singular value of L L. In addition, on (P) = Thus, taking p 2 1 |tee needed conditions in Theorem are satisfied.
3. Computer Tomography: Computer tomography refers to a computerized X-ray imaging proce- dure in which a narrow beam of X-rays is aimed at a patient and quickly rotates around the body. producing signals processed by the machine's computer to generate cross-sectional images (Wang et al. 2019a). These slices are called tomographic images and can give a clinician more detailed information than conventional X-rays. Mathematically, the computer tomography problem can be modeled as in equation (12) defining matrices S, and P are given by S = I. and P = - (Vargas) al. 2018a). Thus, taking p Z 1 the needed conditions in Theorem are satisfied.
Magnetic Resonance Imaging: Magnetic Resonance Imaging (MRI) non-invasive imaging technique providing both functional and anatomical information for clinical diagnosis. Imaging speed is fundamental challenge. Fast MRI techniques are essentially demanded accelerate data acquisition while still reconstructing a high-quality image (Sun et al. (016)). Mathematicallyy the magnetic resonance imaging problem can be modeled as in equation (12) defining matrices S, and P are given by S I, and P = V (Vargas et al. (2) for y an orthogonal matrix. Thus, taking p 2 1 the needed conditions in Theorem 2 are satisfied.
Compressive Sensing: Compressive sensing is recent highly applicative approach. It enables efficient data sampling at much lower rate than the requirements indicated by the Nyquist theo- rem. Compressive sensing possesses several advantages, such as the much smaller need for sensory devices, much less memory storage, higher data transmission rate, many times less power consump- tion (Ramirez et al. 2021). Due to all these advantages, compressive sensing has been used a wide range of applications. Mathematically, the compressive sensing problem can be modeled as in equation (12) defining matrices S, and P are given by S = Y and = -I (Ramirez et al.) 2021), for v an orthogonal matrix. Thus, taking p 1 the needed conditions in Theorem satisfied.
6.
Radar: Step-frequency is a radar waveform consisting of a series of sine waves with linearly increasing frequency. The radar measures the phase and amplitude each fre- quency and used an inverse Fourier transform of these data to build a time domain profile (Johnston Radar problem can be modeled as in equa-
al. 2021). Mathematically, the Stepped-Frequene lion (12) defining matrices S, and P are given by S = I, and P = I (Johnston et al. 202 Thus, taking p 2 1 the needed conditions in Theorem 9 are satisfied.
1.3
PROXIMAL SOLUTIONS
In this section present the proximal operator for the studied invex/quasi-invex functions in this work, summarized in the following Table 4
PROXIMAL OF FUNCTION IN EQUATION
Consider lww)= = X log(1 + (w(P) + (w u)
3 for X E (0, 1], + O, and fixed u E RR (0, 1). We note first that we only consider w's s for which sign(w) (w) = sign(u). otherwise h(w) = log(1 + |w[P)+ 22 2 + which is clearly minimized at = 0. Then, since with (w) sign(u) have (w u)² ([w] lul))?, we replace with |ul and take 2 0. As h(w) is differentiable for W. > 0, re-arranging A' d'(w) 0 gives dix (w) A App
I+ + w = |u] Observe that & (w) is always positive then it means that v (w) is monotonically increasing. Thus, the equation eh (x(w) |ul has unique solution i.e. at some point the quality holds. Thus. solving gi (w)) ul is equivalent
w(w - [u])(w P P +1)+
(102)
We solve equation 102) using standard bisection method (Burden & Faires 1985).
30

---

Published as a conference paper at ICLR 2024
Table Proximal operator for regularizers Ep -quasinorm, and equation is a thresholding parameter).
equation
9 (A (0,
Ref
Invex function
Proximal operator
B <
Marjanovie & Solo 2012
gx(w) = Mww.. (0,1), #
Proxg, (w) = fo. ()
[] =
sign(w))y
[t] >
where 18 12 10--1
AppP-1,
h(y) App y
O.
[..
(x Alg((1 (1 +uul)) P (0, 1,
Proxa, (w)
sign (w)) otherwise
[e]){ equation using standaard bisection method.
where 9(3
0. 0. so this
Zhang 2010
MCP
Prox (w) =
dwr sipn/a 10
w| < 0A
w
u 2 8A
Implementation details: In order to efficiently compute the proximal solutions, which is an entry- wise operation, in Table 4 we use the Python library CuPy (Lib). The reason is that this library allows the creation of GPU kernels in the language C++ from Python (quick guide on how create and use these kernels (ker)). Therefore, inside this GPU kernel, we implemented the entry-wise operation from the Proximal Operator column of T We report as an example how to implement the proximal solution for fp-quasinorm. We remark that this implementation is efficient because it runs in parallel and at the GPU speed. Additional features of the CuPy library are the flexibility to connect both TensorFlow and Py Torch libraries for training neural networks (pyt).
invex20 kernel cp. RawKeerell(""" "C"
global_ void invex2DFilter(const float *x, floot "y, float lamb, float 9, int size)
int tid blockDin.x blockIdx.x threadIdx.x;
(tid size){ float float tau = beta (obbaaao
float sign = (indsi 1.0;
yitid] 0.0F;
ix(fabs(x(tid)) > tau){ float = beta (fabs(x(tid))-beta))
for k=0;k 4jk++){ = fabs(x(tid))
1amb#@"powf(2.g-1.00
y[tid] = sign 2;
'invexZDFilter")
Lastly, the running time compute the proximal of invex/quasi-invex functions is also essential to compare with its convex competitor, i.e., -noom. The reason is the desire to improve imaging qual- ity, keeping the same computational complexity obtain it. Therefore, the following Table repport the running time to compute the proximal (in GPU) of equation (3). lp quasiinomm and equation p for image of 2048 x 2048 pixels. Table 5 suggests that computing the proximal of the & norm faster than the proximal of invex regularizers. However, this difference is given in milliseconds, making it negligible in practice.
31

---

Published as a conference paper at ICLR 2024
Table 5: Time to compute the proximal for all invex and convex regularizers, of an image with 2048 x 2048 pixels. The reported time is the averaged over 256 trials. For fp-quusinorm we select
p = 0.5, and c = (f(1 - p))=
equation
Ep --quasinorm
equation (9)
-nrrr
Time
2.8ms
1.06ms
0.86mss
0.66ms
J EXTENDED ACCELERATER PROXIMAL GRADIENT METHOD (APGM)
The accelerated proximal gradient method (APGM) Li & 2013 has been shown to be effective
in solving program by minimizing F(xr) = (ax) + (a) where f(x) 5 (max f(x) c, 0})²
achieving better quality in less iterations than its predecessors (Beck Teboulle, 2009 Frankel
et al.
2015 Bot et al. 2016), and been frequently used by recent imaging works (Wang & Chen
2022
Mai et al. 2022). Note that f is invex with the same di as f (which is also the same of
according to Theorem since huw)) = (max - e, 0))² is an increasing convex function (Mishra
& Giorgi 2008 Theorem 2.) Therefore, F(x:) is invex. The iterative process performed by
APGM IS summarized as
(+ (t)
+ Te-1 (2(1) - (0) + Th-1 r--1- (x) (x((1) - (t-11))
P₁
r².
(++1) prox
(y - -22 ((z(())))
(103)
u ((+1) = ano (xx
an f((z))))) r1,11
2 (1+1) z(++1), if f(z(+11) ((2(++))) & f(b((+1)) + g(v(((t))
b(t+1),
otherwise
for some positive constants 01, 022 and assuming is L-smooth. The reported convergence guaran- tees of (f++1) to global optima of APGM has been only stated for convex losses (Li & Lin 2015).
For non-convex cases, convergence to critical point has been stated (Li & Lin 2015). Here,
extend the general convergence guarantees of APGM to invex/quasi-invex settings in Theorem so that its benefits are available to the signal restoration problems.
Theorem 11. Let f,g The > R be admissible functions as in Definition where / L-smooth. Assume x² a global minimizer of F(x:) = g(x) + f(x) for c = 0, then
F(x - F(x*) v
a' + 1)2 ||x: (0) x x11
(104)
where is number iterations, 01 02 depends f.g.
<
Le+4
x 1 =
> 0, with x a constant that
Proof of Theorem 11 relies on the properties of admissible functions including Theorem g The proof is presented below. It is worth mentioning that the convergence rate for admissible functions as in Theorem is the same as the convex case (see (Li Lin 2015)).
Proof. Step in the APGM is given by
(++1) arg min =ER
(VI x y ())
+
x y 20 + +g()).
(105)
201
From Theorem we know that equation 1055 is unique for a aq
<
1. for some Kg > Then the
optimality condition for equation (105) is given by
1 (2(++)) ())) + ((+1).
0 E V (t)
(106)
0i
Additionally, since g(xr is an admissible function, then from (Pallaschke & Rolewicz 2013 Propo- sition 5.2. we have that
g(x) (=((++))) > (-V a
-(2(141)- (1))), x 2((11)
1
|| 2 (++1)
Kig
(107)
32

---

Published as a conference paper at ICLR 2024
for all æ. From the Lipschitz continuous of f,, we have for all x and Ri < that
f V ()) y01)
1 11.
(108)
KS
In addition of the Lipschitz continuutt of f we have that
F(z(++1)) S g(2((t1) + (() + (f(()(t)) 2,((+)) -1)) + L
1 (t-1) - -000
2
= f0) + (f ), -y ( (Vi (0) 2 - x))
L -1la(tt)) - yy 1 3
(109)
2
Then, plugging equation (07 and equation 108) into equation
109)
we obtain that
F(z(++1)) s ((=((+1)) + ((x) + ( (++1) - xx) + Kg
x y 12 + 2 [| - -y
S FF(x)+ + (v + 1 (++11 () ) x((+1) ) + (V)(l)()) x((+11 - x))
),
a1
1 ||0 - 100 +
-
||0 - x 11 + _ || - 201
Kf
Kig
S F(x)) + 1 (2(0+1) y0) x - 2((+1))
+ _ [0=(0 - -3(")))))""11
I
a1
1/200 |100 - y 20 + || 2 113)
1
(110)
where 1 max 1 }. Observe that the last term in the above equation 110 satisfies
Ky mg
||0 y
2 + || (a+1) 1
[|
2 ( ((())) (0)) x. 2 ((+1))
(111)
Combining equation 110)
and equation
111 we obtain
F(z((+1)) S F(x)+ +
2
(1(++))
y x 2(011)
T, 1
||
- -000012
+
a1 K
2 K
= F(()) +
1
2
(20+1)
-)(1) x - y00))
1
L
3
|| ||
- 2000
a1
k
a1
2
K
(112)
Thus, if a1 < Ixt7
x
then we have
F(2(++1)) S F(x) +
1
2)
(20t11) -00) (2(+))-g(0,00 ())
1
1
||(((+]]
x
2011
k
(113)
Let x = x()) and x,, we have
F(2(+1)) < ()) +
an - 2) (z((t1) (
1
2)
y00) - )00)
1
||=((+1]
||
x
2011
K
(114)
F(2(++1)) S F(x²)+ +
{ 2)
- -20 x y/t)) -(()))
1
||
y
10
a1 K
2011
(115)
1 = --1-- -1- 2 Then, multiplying equation (14 by r. 1 and adding equation 115 we 03
Define
have
r/ ((F(2()11) - (rr 108-11 1))((((0) - FF(x")
< 1 (s(tti) - (0 (r=- 1z) ())) + xx -000)
rt. 1| (++1) - 10
(116)
20
33

---

Published as a conference paper at ICLR 2024
So have
ri ( - -F(xt) (rr -1) (F(z(())) -F(z))
1 a' (2(+t)= - n (r+ -1)(2! -0( + +x" - -10))
Tt
[| - y
(117)
2a
Multiplying both sides by re and using (ri)² Tt (rc-1)2 from APGM we have
(rx)² [((((+))) - F(x*)) 1)² [fxx(t)) - -F(x²))
Σ a' rt ((((+)) y(t) (rr 1) (0))) y(0) + xx" - -y(0)
rr
|l (re(2((+1)) - -
20
a' [t (x((t)) (01) (rr l)x(1) rly + xx²)
1
1
Inne |ne(2(tt2) 111)
2a
20 (I(r== 1)xx(1) ry + x 2=1 ll(r: - 1)xx r1 + +x
(118)
Define
g(++1) (1+2) - 1)xx()) xx
(Fre-
(119)
Let
T(t) = ---12 - (Ft-1 - 1) xx" = =r( - (r: 1)xx()) x²"
(120)
We have
(()) 11-12(1) (r--1 ()x((-11) + (re - 1)x((1)
Ft.
= (1) + (2(11) - zl(t)) + Mh-1 T2-1-1 (ax(1) (ar() (t(111)).
(121)
which is the same in Step 1 of APGM. So we have
(re)² (F(2 - F(x*)) ) - (ri-1)²
(F(x(0) - F(z*)) $ 2a
(1o 110()+111)))
(122)
If ((2((+1)) S F(o((t1),, then ((+1) = 2(1+1)
(r:)² (F(2(+1))) - F(x*)) - (rr--1) (F(a)()) - F(xx')
= (f(x - F(x*))
(rr-1)² (F(x(()) F(x*))
$ 2a' (11000) 1 -110(442)11))
(123)
If (2((+1)) > F(o((+1)) ) then x (1+1) 1)= = ((tt)) So.
(r.)3 (z)x(++1) F(x*)) -rr-1)²² (F( - F(x*))
<(ri)3 (F(z(++1)) - r(x*)) - -(re_1² (F(xl()) - F F(x*))
(107()) 12 1
$ 2a'
(124)
Summing equation
124p over = 1..... T we have
(rT-1) (F(x - F(x"))
(rr 1) (F(x) F(x*)) - (Fo)((F((211)) - F(x*)) ))
20 211 || -10((+))) -
2x | 112
1
|x (0)
(125)
20
From APGM we can easily have that r; 2 1+1
So we have
2
((x()+ - F(xx) v (T + 1)2 |
x =='13
(126)
where // 2 2) and on < Ix+7
x with 1 = max I m4y mog
for ky < t.
34

---

Published as a conference paper at ICLR 2024
REMARKS ON STABILITY OF APGM
The APGM stability and reliability come from the pivotal factors: 1) APGM employs two prox- imal steps that have a unique solution (as rigorously demonstrated the section above), aiding convergence and stability. 2) As shown the section above, the sequence ((++1), constructed by APGM algorithm, always converge to global optima irrespective of the initial state æ;()), conferring a steadfast assurance of reliable attainment of optimal solutions. 3) The remarkable effectiveness
demonstrated across an array of real-world applications (Beck Teboulle 2009 Ochs al. 2014
Bot et al. 2016) reaffirms APGM's 's reliability.
K ADDITIONAL IMPLEMENTATION DETAILS AND RESULTS
In this section we present additional numerical results and details to complement the three experi- ments in Section 5 For Experiments and 2 the constants c, a the adaptive invex loss in equa- tion (7) are implemented as
softplus(Cvar) Cmii
(127)
a = (Omar Ommn )iigmidd ((ppar) + Ommn
(128)
where Cvars and Qvear parameters be learned simultancously with the network weights using the same Adam optimizer instance. In addition, we fix O'mas 1.99, amin = 0.0, and Cmin 10-8
K. EXPERIMENT 1
To train FISTA-Ne for the CT experiment using convex, invex and quasi-invex regularizers, we batch size of 64, optimized using Adam, and with seven hidden layers. Datasets used for train- ing, validation and testing were normalized to the interval of [0, 1]. In addition to the experiments reported Table 3 for Experiment 1, we show the performance invex regularizers reported in (Pinilla et al. 2022a) under FISTA-Net. These results are summarized in Table
Table 6: Performance Comparison: Lower the better. We highlight the best, and the worst results for each metrie. (Green: Best, and Red: Worst).
Experiment
(Combination functions used to train FISTA-NeO)
Regularizer
Metrics
equation
equation
equation
equation
equation
4--norm
AbsError
0.2380
0.2106
0.2212
0.1867
0.1973
0.2556
equation 7
PSNR
38.36
39.09
38.65
39.13
39.08
38.08
Pinilla
2022a
RMSE
0.0150
0.0124
0.0123
0.0105
0.0095
0.0174
SSIM
0.9550
0.9574
0.9566
0.9590
0.9582
0.9558
AbsError
0.2768
0.2501
0.2352
0.2240
0.2586
0.2941
equation 8
PSNR
38,05
38.33
38.72
38.87
38.78
37.79
Pinilla al.
2022
RMSE
0.0179
0.0104
0.0145
0.0137
0.01
0.0217
SSIM
0.9537
0.9554
0.9563
0.9580
0.9571
0.9545
AbsErtor
0,2088
0.1819
0.1700
0.1601
0.1933
0.2259
equation 10
PSNR
38.68
39.40
39.32
39.54
38.99
38.38
Pinilla et al. 2022a
RMSE
0.0129
0.0087
0,0111
0.0108
0.0095
0.0146
SSIM
0.9563
0.9585
0.9578
0.9600
0.9593
0.9571
K.2 EXPERIMENT 2
To train MST+ using the invex loss functions we use batch size of 20, optimized using Adam with parameters 31 0.9, and 83 = 0.999, and the cosine Annealing scheme is adopted for 300 epochs. Datasets used for training, validation and testing were normalized to the interval of [o, 1]. To complement results in Table(3]e present Figure 2 which reports some restored spectral images using MST++ trained with losses equation lequation To evaluate the performance we employ the absolute error.
"code of can he found at httpt///tit///ttttto///o///////a Implemmnntttion he found httptttttt/ggtbbtoooott www.ccoomoaaaaaaooooo.a
yuanhaol 998 wSS'sppppll 198
35

---

Published as a conference paper at ICLR 2024
Original
Eq. (10)
Eq. (11)
Eq. (7)
AbsErro -0.181
AbsErro -0192
AbsErro 1.196
Eq. (8)
E--norm
Eq. (9)
Ab:20rof00.205
=0.205
AbsBrcof -0.113
AhsEEro
Figure 2: Reconstructed spectral images with MST+ trained with losses fz-norm, and the invex functions equation eqqutton
To evaluate the performance we employ the absolute error.
Noisy image
Invex TV-LLpp Map
Quassi-ivvex TV-L.Ip
COMMAA
SS7M-0
SSIM=0.7112
TV-
f--norm
SSIk Map
SSIM Map
SS0M=0.6987
6.647
Figure 3: Restored images using the ADMM iterative process in equation 129 solve pro- gram using votal-variation-like invex/quasiiinvex regularizers of fp-quasinorm, and equa- tion 3 equation (TO). To evaluate the performance we employ the structural similarity index mea- sure (SSIM) by reporting the SSIM map for each image and its averaged value. Recall that SSIM is reported in the range [0,1] where is the best achievable quality and the worst. the SSIM map small values of SSIM appear as dark pixels.
K.3 EXPERIMENT 3
In order implement the ADMM iterative process in equation (3) solve the total variation filtering optimization problem in 11],
linear approximation update (+++1) based Ouyang al. 2015). The reason of this approximation comes from the fact that updating at++1 for the total variation filtering optimization problem in (11) requires the computation of (pD" D++)-1 which is computationally expensive step. Therefore, we implement the following ADMM iteration process to solve program (II) for the total-variation-like invex/quasi-invee regularizers of l,-
quasinorm, and equation equation T0
36

---

Published as a conference paper at ICLR 2024
[++1] (1 -a) ax)
22
2 (++1) = 0 2B
(1t11 (Do2
2()) xl0) + x2 (1+1)
x ((+1) (1 - -a)x(!) (!) + ax((t)) (t+1)
(129)
(0+1) = Prox (tx( b (t)ppp)
(++1)) = ( + p - ((++)))
where D is the discrete spatial derivative (see Section B for more details), the first three equal- ity is the linear approximation to estimate (++1) following (Ouyang et al. 2015). u is the noisy image, and a, p, are positive constants chosen be the best for each analyzed function deter- mined by cross-validation. This iterative procedure is initialized xx u. 20)) Dx and
21 (0) (0) = o0)) = O. Further, the proximal of regularizers g to compute ((++)) are given in Table To complement results in Table we present Figure which reports some restored images obtained by lotal-variation-like invex/quasi-invex regularizers of 6--quasinorm, and equation (3) equation (IO). To numerically evaluate their performance we employ the structural similarity index measure (SSIM) by reporting the SSIM map for each denoised image and its averaged value. Recall that SSIM is reported in the range [0,1] where is the best achievable quality and 0 the worst. In the
SSIM map small values of SSIM appear as dark pixels.
K.4 EXPERIMENTS WITH APGM AND VALIDITY OF THEOREM
In this section we assess the performance of APGM ( Lin 2015) solving compressive sensing problems. Specifically, this section provides two experiments summarized in Tables 7 and The aim of the first experiment is to numerically validate Theorem 7 and the second it is to show the performance of APGM a realistic setting when invex/quasi-invex are employed.
For the first experiment, we generate noisy measurements y = A. + n where A E R follows Gaussian random matrix with & nz-rrmale columns, æ E R is a k-sparse vector, and n R" models the noise. Specifically, the size n and the sparsity k of the unknown underlying signal æ are fixed as = 1024, and = 0.03n. Therefore, appealing to the theoretical result in (Foucart) & Rauhut 2013, Theorem 9.13), we determined from the fixed values of n and h the number of measurements m as m = 0.45n in order to satisfy equation (67) for T = 0.3 and = 0.4. In
consequence, the constant C and D of equation 72p are given by C = 4.6, and D = 2 g(x)) Under this setting, we are ready to validate inequality in equation 72) numerically (i.e. validity of Theorem 7) where the regularizer g(x) in program (I) takes the form of equations 33. p 10 and the data lidelity term f(x) takes the form of equations (). (these were chosen due their performance in Table Further, in order to solve program (T) and estimate the distance ||x - x
in equation (72). consider additive white Gaussian noise in the measurements data vector with three different levels of SNR (Signal-to-Noise Ratio) 20, 30. The number of iterations T of the APGM is fixed to T = 1000. The positive constants a2 of APGM, and c, a of equations (1) are chosen to be the best for each analyzed combination of functions by cross-validation.
APGM (see Appendix where
the solution returned by APGM. We
We summarize the results in Table 7 From these numerical results, we observe that invex/quasi- invex mappings for both regularizer and fidelity term perform better than using the combination of -norm and fy-norm because the upper bound in equation (72) is smaller. Additionally, the results in Table 7 suggest this better lower upper bound is due the regularizer. This is an expected
behavior since this is a compressive sensing test, and the regularizer is the mapping that determines the reconstruction quality (Candès & Wakin 2008).
For the second experiment, we use 24 images of size 256 x 256 (i.e. dak dataset (kod), and we convert them to grayscale. In order to have noisy measurements y in compressive sensing setup, we compute the Hadamard transform over these images x (which is sparse), after we add Cauchy distributed noise with a dispersion o = 1, and we randomly select m 32000 noisy transformed coefficients. Therefore, assuming A is submatrix of size x of the Hadamard transform. then = Ax + n where 7 is the Cauchy distributed noise. Thus, these experiments are intended recover æ from the noisy measurements y using APGM, where the data
65536) from the Ko-
37

---

Published as a conference paper at ICLR 2024
Table 7: Performance Results: Best: green, and the worst: red. Column named "dist" reports | x"1 || where x" is the solution returned by APGM for each combination of g(z)), and f(a) mappings. Additionally, column named "bound" reports g( ,,) in equation 72) for each combination of g(x), and f()) mappings.
Tested f(x) functions for APGM
Eg-norm
(
SNR
dist
bound
dist
bound
dist
bound
dist
bound
20dB
0.401
1.330
0.420
1.330
0.442
1.330
0.467
1.330
30dB
0.218
0.453
0.188
0.453
0.217
0.453
0.260
0.453
20dB
0.588
5,255
0.671
5.255
0.783
5.255
0.941
5.255
30dB
0,374
1.632
0.332
1.632
0.370
1.632
0.426
1.632
20dB
0,477
1.330
0.516
1,330
0.565
1.330
0.624
1.330
10
30dB
0.275
0.453
0.240
0.453
0.273
0.453
0.323
0.453
20dB
0.768
5,431
0.959
5.431
1.278
5.431
1.916
5.431
f--nrmm
30dB
0.585
1.857
0.541
1.857
0.573
1.857
0.627
1.857
fidelity functions ((x are the f2-norm, and equation (4)-equation (8). and the regularizers g(x) are the fi-norm and equation (3). equation (9). equation (T0) We point out that the combination of 2z-norm as data fidelity term and & -norm as regularizer taken as the convex state-of-the-a The number of iterations T of the APGM is fixed to T 1000. The positive constants a1, 02 of APGM, and c, 7 of equations 10) are chosen to be the best for each analyzed combination functions by cross-validation. The performance for each combination of functions is measured by averaging the yeak-signal-to-nois----
(PSNR) in dB over the image set. These results are summarized in Table 8 These results suggest that the best result is obtained employing regularizer equation (3) and fidelity term in equation (7). The intuition behind the superiority combining these two mapping comes from the possibility adjusting the value of p for equation and c, for equation (7) in
data-dependent manner [W & Chen 2013 Barron 2019).
Table 8: Performance Results: Best: green, and the worst: red.
Tested f(x)) functions for APGM
Regularizer
Metric
equation
equation
equation
equation
equation
2--norm
equation
PSNR
20.53
17.48
16.27
22.50
18.88
15.25
equation 9
PSNR
18.03
15.64
14.67
19.52
16.76
13.83
equation
10
PSNR
19.20
16.51
15.43
20.90
17.75
14.50
6 -norm
PSNR
17.00
14.87
13.99
18.32
15.87
13.22
38


Reviews:
[{"review_id": "4SbmmJsgSi", "reviewer": "Reviewer_ta5r", "summary": "This paper studies the global optimality of a constrained nonconvex (particularly, invex/quasi-invex) optimization problem for signal restoration, which was found to provide an improved signal quality over the convex optimization-based signal restoration. The invexity property is interesting as it makes any critical point a global minimizer, while being quite nonconvex. This paper not only identifies the invexity of existing non-convex functions (for signal restoration) but also comes up with new invexity functions, based on a new definition of admissible function, yielding a global optimality guarantee. This paper then shows that the standard ADMM also works for constrained \"invex\" problems. Various experiments are provided to validate the effectiveness of using nonconvex but invex functions for both fidelity and regularization terms.", "questions": "- Are the invex functions considered in this paper prox-friendly? In other words, how is the proximal step of ADMM implemented?\n- Does Theorem 9 assume either invexity or quasi-invexity? I was not able to see where such property is used in the proof. It seems the prox-regularity condition is used, which is only a sufficient condition for quasi-invexity. If I am misunderstanding, I suggest revising this part\nso that everything is easier to follow.\n\n*Minor* \n- abstract: largest \"class of\" optimizable\n- page 1: $f(w) = f(Ax - y)$ should be fixed\n- page 6: use \"of\" Lemma 1\n- page 7: constrain\"t\"\n- page 8: What do you mean by the uniqueness result in Theorem 3?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This identifies various nonconvex but invex constrained signal restoration problems with a global optimality guarantee, theoretically supporting the empirical success of using nonconvex functions in signal restoration.\n- This paper provides a set of admissible functions that are invex/quasi-invex, making it easy to identify or construct invex/quasi-invex functions.", "weaknesses": "- The global optimality of quasi-invex function was mentioned in the beginning, but it is not clearly stated after all.\n- (Pinilla, 2022a) studied unconstrained signal restoration with invex regularizers, and the contributions of this paper over (Pinilla, 2022a), such as quasi-invexity and the assumptions on the extended ADMM (see question below), are not clearly stated. So they do not seem significant. (The fact that this paper considers a constrained case seems a straightforward extension of (Pinilla, 2022a).) I will reconsider my score depending on the clarification of these contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EIhMaT50Tf", "reviewer": "Reviewer_a9wE", "summary": "This paper studies the problem of nonconvex constrained optimization in which the objective and constraints are both assumed to be invex (or quasi-invex) functions. The main results are a list of new invex/quasi-invex regularizers and data fidelity loss functions which are constructed based on the concept of admissible functions. Several applications of the proposed invex/quasi-invex functions to compressive image restoration and total variation filtering are provided, along with an extended ADMM algorithm for global optimization. Numerical experiments on relevant tasks are conducted to evaluate the performance of the proposed models and algorithms.", "questions": "- Q1. Can you explain in more details how the extended ADMM algorithm as developed in Section 4.3 might be used to solve the constrained invex problem (1) with global guarantees? \n\n- Q2. In Section 4.1, what is the exact invex problem formulation designed for compressive image restoration? Why the quantities $\\beta_k^g(x)$ and $v_f(x)$ in Theorem 7 can be regarded as constants while they are obviously functions of $x$? Also can you say a bit more on the scales of these quantities in the considered problem setting?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of using invex functions for both data fidelity and regularization terms for nonconvex image restoration sounds interesting. The proposed family of admissible functions seems useful for constructing invex loss functions with a unified treatment. The numerical results show some benefits of the proposed approach in a bunch of image restoration tasks.", "weaknesses": "The main concerns are in regard with the significance and correctness of technical contribution.\n\n- As highlighted in Section 3, the authors mainly introduced a set of new invex /quasi-invex functions from the perspective of admissible functions. While these invexity-type functions are guaranteed to be globally minimal at any stationary point, a more challenging problem is how to find these points in a computationally efficient way. This challenge, however, is largely left unaddressed in the present study. The authors did mention an ADMM-style algorithm at the end of Section 4, but it is not quite clear how this algorithm could be exactly used for solving the considered problem formulation in Equation (1). \n\n- The statements and proof arguments for some of the results in Section 4 are unclear in many places. For example, the statement of Theorem 7 reads a bit confusing, mainly due to that: 1) the comments on extensions are included at the end of the statement, and 2) the quantities $\\beta_k^g(x)$ and $v_f(x)$, which are functions of $x$, are claimed as universal constants. Moreover, according the last but one line of the proof of Theorem 7, the quantity $v_f(x)$ should also be dependent on the noise vector $\\eta$. In regard with Theorem 8, the first claim that $h$ is convex seems only valid for a proper range of $\\lambda$, which needs to be explicitly clarified in the theorem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YMj0Xn3XcQ", "reviewer": "Reviewer_4Bfb", "summary": "This work uses the notion of invexity and quasi-invexity in non-convex constrained inverse problems to find global optima.  Through application of admissible functions they build some mathematical tools to show invexity or quasi-invexity of some functions mentioned in the paper. These are the functions which are later used as the loss function or the regularizer in equation (1). Multiple applications of their analysis are studied such as compressive image restoration, total variation filtering, and E-ADMM. Experimental results are provided to support their claims.", "questions": "1- Was there any computational difference when replacing $L_1$-norm and $L_2$-norm in your first experiment with the proposed functions? \n\n2- Theorems 5 and 6  prove that some previously known functions satisfy invexity/quasi invexity property. Is it only due to invexity/quasi-invexity that these functions have superior performance in applications listed in table 2? \n\n3- Do you have any experiments confirming the result in Theorem 7?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1- sound paper, \n\n2- well-established connection between invexity and various applications, \n\n3- interesting theoretical results.", "weaknesses": "1- In my opinion, the paper needs major revision in writing, organization, and presentation.\n\n2- In many applications the computational complexity of signal recovery is very high (e.g. in image restoration, application of low-rank matrix recovery in sparse image recovery). This issue was not discussed.\n\n3-The limitations are not mentioned.\n\n4- Some points which might help improving the presentation of the paper\n\n(a) The text is generic in some parts. E.g. in conclusion you mention that your results are very promising, but you do not specify in what sense.\n\n(b) I think Table 3 does not need the second best results highlighted. \n\n(c)There are some\"hard to read\" sentence in the text. One example in the introduction is:\n\n\"The lack of approaches for guaranteeing the global minima in non-convex settings leaves any optimization-based algorithmic solution incomplete, non-unique, and hence cannot be categorically accepted as the best possible solution despite their improved performance over convex functions.\"\n\n(d) The E-ADMM sounds like a new algorithm is proposed. Instead, you showed the convergence of the same ADMM under the assumption of invexity/quasi-invexity. The name \"E-ADMM\" causes confusion.\n\n(e) (11) is referred to as equation. It is an optimization problem.\n\n5- Background on applications in experiments 1,2,3 is not suggested. This is the case at the beginning of every section. Maybe having a separate related work section could help with organizing the paper. \n\n6- I appreciate the vast context of the paper (signal restoation in a general sense). I think in such cases when the topic entails many applications, you can limit your attention to the main pioneer works in the field. This helps the reader to follow references easier. When one faces 5,6 references on similar topic it might be discouraging or hard to choose references efficiently. I believe it is the writer’s task to present the most efficient references related to their work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)