Title: A Unified Framework for Uniform Signal Recovery in Nonlinear Generative Compressed Sensing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Compressed sensing

generative models

nonlinearity

uniform recovery
Abstract: In generative compressed sensing (GCS), we want to recover a signal $\mathbf{x^*}\in\mathbb{R}^n$ from $m$ measurements ($m\ll n$) using a generative prior $\mathbf{x^*}\in G(\mathbb{B}_2^k(r))$, where $G$ is typically an $L$-Lipschitz continuous generative model and $\mathbb{B}_2^k(r)$ represents the radius-$r$ $\ell_2$-ball in $\mathbb{R}^k$. Under nonlinear measurements, most prior results are non-uniform, i.e., they hold with high probability for a fixed $\mathbf{x^*}$ rather than for all $\mathbf{x^*}$ simultaneously. In this paper, we build a unified framework to derive uniform recovery guarantees for nonlinear GCS where the observation model is nonlinear and possibly discontinuous or unknown. Our framework accommodates GCS with 1-bit/uniformly quantized observations and single index model as canonical examples. Specifically, using a single   realization of the sensing ensemble and generalized Lasso,   all $\mathbf{x^*}\in G(\mathbb{B}_2^k(r))$ can be recovered up to an $\ell_2$-error at most $\epsilon$ using roughly $\tilde{O}({k}/{\epsilon^2})$ samples, with omitted logarithmic factors typically being dominated by $\log L$.  Notably, this almost coincides with existing non-uniform guarantees up to logarithmic factors, hence the uniformity costs very little.    As part of our technical contributions, we introduce Lipschitz approximation to handle discontinuous observation models.    We also develop a concentration inequality  that produces tighter bound for product process whose index sets have low metric entropy.  Experimental results are presented to corroborate our theory.

A Unified Framework for Uniform Signal Recovery in Nonlinear Generative Compressed Sensing

Abstract

In generative compressed sensing (GCS), we want to recover a signal x ∗ ∈ R n from m measurements ( m ≪ n ) using a generative prior x ∗ ∈ G ( B k 2 ( r )) , where G is typically an L -Lipschitz continuous generative model and B k 2 ( r ) represents the radiusr ℓ 2 -ball in R k . Under nonlinear measurements, most prior results are non-uniform, i.e., they hold with high probability for a fixed x ∗ rather than for all x ∗ simultaneously. In this paper, we build a unified framework to derive uniform recovery guarantees for nonlinear GCS where the observation model is nonlinear and possibly discontinuous or unknown. Our framework accommodates GCS with 1-bit/uniformly quantized observations and single index models as canonical examples. Specifically, using a single realization of the sensing ensemble and generalized Lasso, all x ∗ ∈ G ( B k 2 ( r )) can be recovered up to an ℓ 2 -error at most ϵ using roughly ˜ O ( k/ϵ 2 ) samples, with omitted logarithmic factors typically being dominated by log L . Notably, this almost coincides with existing non-uniform guarantees up to logarithmic factors, hence the uniformity costs very little. As part of our technical contributions, we introduce the Lipschitz approximation to handle discontinuous observation models. We also develop a concentration inequality that produces tighter bounds for product processes whose index sets have low metric entropy. Experimental results are presented to corroborate our theory.

1 Introduction

In compressed sensing (CS) that concerns the reconstruction of low-complexity signals (typically sparse signals) [5, 6, 15], it is standard to employ a random measurement ensemble, i.e., a random sensing matrix and other randomness that produces the observations. Thus, a recovery guarantee involving a single draw of the measurement ensemble could be non-uniform or uniform -the nonuniform one ensures the accurate recovery of any fixed signal with high probability, while the uniform one states that one realization of the measurements works simultaneously for all structured signals of interest. Uniformity is a highly desired property in CS, since in applications the measurement ensemble is typically fixed and should work for all signals [17]. Besides, the derivation of a uniform guarantee is often significantly harder than a non-uniform one, making uniformity an interesting theoretical problem in its own right.

Jonathan Scarlett

Inspired by the tremendous success of deep generative models in different applications, it was recently proposed to use a generative prior to replace the commonly used sparse prior in CS [2], which led to numerical success such as a significant reduction of the measurement number. This new perspective for CS, which we call generative compressed sensing (GCS), has attracted a large volume of research interest, e.g., nonlinear GCS [29, 33, 45], MRI applications [24, 46], and information-theoretic bounds [27,34], among others. This paper focuses on the uniform recovery problem for nonlinear GCS, which is formally stated below. Our main goal is to build a unified framework that can produce uniform recovery guarantees for various nonlinear measurement models.

Problem: Let B k 2 ( r ) be the ℓ 2 -ball with radius r in R k . Suppose that G : B k 2 ( r ) → R n is an L -Lipschitz continuous generative model, a 1 , ..., a m ∈ R n are the sensing vectors, x ∗ ∈ K := G ( B k 2 ( r )) is the underlying signal, and we have the observations y i = f i ( a ⊤ i x ∗ ) , i = 1 , . . . , m , where f 1 ( · ) , . . . , f m ( · ) are possibly unknown, 2 possibly random non-linearities. Given a single realization of { a i , f i } i m =1 , under what conditions we can uniformly recover all x ∗ ∈ K from the corresponding { a i , y i } i m =1 up to an ℓ 2 -norm error of ϵ ?

1.1 Related Work

We divide the related works into nonlinear CS (based on traditional structures like sparsity) and nonlinear GCS.

Nonlinear CS: Beyond the standard linear CS model where one observes y i = a ⊤ i x ∗ , recent years have witnessed rapidly increasing literature on nonlinear CS. An important nonlinear CS model is 1-bit CS that only retains the sign y i = sign( a ⊤ i x ∗ ) [3,22,41,42]. Subsequent works also considered 1-bit CS with dithering y i = sign( a ⊤ i x ∗ + τ i ) to achieve norm reconstruction under sub-Gaussian sensing vectors [9, 14, 48]. Besides, the benefit of using dithering was found in uniformly quantized CS with observation y i = Q δ ( a ⊤ i x ∗ + τ i ) , where Q δ ( · ) = δ ( ⌊ · δ ⌋ + 1 2 ) is the uniform quantizer with resolution δ [8, 48, 52]. Moreover, the authors of [16, 43, 44] studied the more general single index model (SIM) where the observation y i = f i ( a ⊤ i x ∗ ) involves (possibly) unknown nonlinearity f i .

While the restricted isometry property (RIP) of the sensing matrix A = [ a 1 , ..., a m ] ⊤ leads to uniform recovery in linear CS [4,15,49], this is not true in nonlinear CS. In fact, many existing results are nonuniform [9,16,21,41,43,44,48], and some uniform guarantees can be found in [7,8,14,17,41,42,52]. Most of these uniform guarantees suffer from a slower error rate.

The most relevant work to this paper is the recent work [17] that described a unified approach to uniform signal recovery for nonlinear CS. The authors of [17] showed that in the aforementioned models with k -sparse x ∗ , a uniform ℓ 2 -norm recovery error of ϵ could be achieved via generalized Lasso using roughly k/ϵ 4 measurements [17, Section 4]. In this work, we build a unified framework for uniform signal recovery in nonlinear GCS. To achieve a uniform ℓ 2 -norm error of ϵ in the above models with the generative prior x ∗ ∈ G ( B k 2 ( r )) , our framework only requires a number of samples proportional to k/ϵ 2 . Unlike [17] that used the technical results [36] to bound the product process, we develop a concentration inequality that produces a tighter bound in the setting of generative prior, thus allowing us to derive a sharper uniform error rate.

Nonlinear GCS: Building on the seminal work by Bora et al. [2], numerous works have investigated linear or nonlinear GCS [1, 11, 12, 19, 20, 23, 25, 30, 39, 40, 51], with a recent survey [47] providing a comprehensive overview. Particularly for nonlinear GCS, 1-bit CS with generative models has been studied in [26, 31, 45], and generative priors have been used for SIM in [29, 32, 33]. In addition, score-based generative models have been applied to nonlinear CS in [10,38].

The majority of research for nonlinear GCS focuses on non-uniform recovery, with only a few exceptions [33,45]. Specifically, under a generative prior, [33, Section 5] presented uniform recovery guarantees for SIM where y i = f i ( a ⊤ i x ∗ ) with deterministic Lipschitz f i or f i ( x ) = sign( x ) . Their proof technique is based on the local embedding property developed in [31], which is a geometric property that is often problem-dependent and currently only known for 1-bit measurements and deterministic Lipschitz link functions. In contrast, our proof technique does not rely on such

geometric properties and yields a unified framework with more generality. Furthermore, [33] did not consider dithering, which limits their ability to estimate the norm of the signal.

The authors of [45] derived a uniform guarantee from dithered 1-bit measurements under bias-free ReLU neural network generative models, while we obtain a uniform guarantee with the comparable rate for more general Lipschitz generative models. Additionally, their recovery program differs from the generalized Lasso approach ( cf. Section 2.1) used in our work. Specifically, they minimize an ℓ 2 loss with ∥ x ∥ 2 2 as the quadratic term, while generalized Lasso uses ∥ Ax ∥ 2 2 that depends on the sensing vector. As a result, our approach can be readily generalized to sensing vectors with an unknown covariance matrix [33, Section 4.2], unlike [45] that is restricted to isotropic sensing vectors. Under random dithering, while [45] only considered 1-bit measurements, we also present new results for uniformly quantized measurements (also referred to as multi-bit quantizer in some works [13]).

1.2 Contributions

In this paper, we build a unified framework for uniform signal recovery in nonlinear GCS. We summarize the paper structure and our main contributions as follows:

We present Theorem 1 as our main result in Section 2. Under rather general observation models that can be discontinuous or unknown, Theorem 1 states that the uniform recovery of all x ∗ ∈ G ( B k 2 ( r )) up to an ℓ 2 -norm error of ϵ can be achieved using roughly O ( k log L ϵ 2 ) samples. Specifically, we obtain uniform recovery guarantees for 1-bit GCS, 1-bit GCS with dithering, Lipschitz-continuous SIM, and uniformly quantized GCS with dithering.

We provide a proof sketch in Section 3. Without using the embedding property as in [33], we handle the discontinuous observation model by constructing a Lipschitz approximation. Compared to [17], we develop a new concentration inequality (Theorem 2) to derive tighter bounds for the product processes arising in the proof.

We also perform proof-of-concept experiments on the MNIST [28] and CelebA [35] datasets for various nonlinear models to demonstrate that by using a single realization of { a i , f i } i m =1 , we can obtain reasonably accurate reconstruction for multiple signals. Due to the page limit, the experimental results and detailed proofs are provided in the supplementary material.

1.3 Notation

We use boldface letters to denote vectors and matrices, while regular letters are used for scalars. For a vector x , we let ∥ x ∥ q ( 1 ≤ q ≤ ∞ ) denote its ℓ q -norm. We use B n q ( r ) := { z ∈ R n : ∥ z ∥ q ≤ r } to denote the ℓ q ball in R n , and ( B n q ( r )) c represents its complement. The unit Euclidean sphere is denoted by S n -1 := { x ∈ R n : ∥ x ∥ 2 = 1 } . We use C, C i , c i , c to denote absolute constants whose values may differ from line to line. We write A = O ( B ) or A ≲ B (resp. A = Ω( B ) or A ≳ B ) if A ≤ CB for some C (resp. A ≥ cB for some c ). We write A ≍ B if A = O ( B ) and A = Ω( B ) simultaneously hold. We sometimes use ˜ O ( · ) to further hide logarithmic factors, where the hidden factors are typically dominated by log L in GCS, or log n in CS. We let N ( µ , Σ ) be the Gaussian distribution with mean µ and covariance matrix Σ . Given K 1 , K 2 ⊂ R n , a ∈ R n and some a ∈ R , we define K 1 ±K 2 := { x 1 ± x 2 : x 1 ∈ K 1 , x 2 ∈ K 2 } , a + K 1 := { a } + K 1 , and a K 1 := { a x : x ∈ K 1 } . We also adopt the conventions of a ∧ b = min { a, b } , and a ∨ b = max { a, b } .

2 Main Results

We first give some preliminaries.

Definition 1. For a random variable X , we define the sub-Gaussian norm ∥ X ∥ ψ 2 := inf { t > 0 : ❊ exp( X 2 /t 2 ) ≤ 2 } and the sub-exponential norm ∥ X ∥ ψ 1 := inf { t > 0 : ❊ exp( | X | /t ) ≤ 2 } . X is sub-Gaussian (resp. sub-exponential) if ∥ X ∥ ψ 2 < ∞ (resp. ∥ X ∥ ψ 1 < ∞ ). For a random vector x ∈ R n , we let ∥ x ∥ ψ 2 := sup v ∈ S n -1 ∥ v ⊤ x ∥ ψ 2 .

Definition 2. Let S be a subset of R n . We say that a subset S 0 ⊂ S is an η -net of S if every point in S is at most η distance away from some point in S 0 , i.e., S ⊂ S 0 + B n 2 ( η ) . Given a radius η , we

define the covering number N ( S , η ) as the minimal cardinality of an η -net of S . The metric entropy of S with respect to radius η is defined as H ( S , η ) = log N ( S , η ) .

2.1 Problem Setup

We make the following assumptions on the observation model.

Assumption 1. Let a ∼ N (0 , I n ) and let f be a possibly unknown, possibly random non-linearity that is independent of a . Let ( a i , f i ) i m =1 be i.i.d. copies of ( a , f ) . With a single draw of ( a i , f i ) i m =1 , for x ∗ ∈ K = G ( B k 2 ( r )) , where G : B k 2 ( r ) → R n is an L -Lipschitz generative model, we observe { y i := f i ( a ⊤ i x ∗ ) } m i =1 . We can express the model more compactly as y = f ( Ax ∗ ) , where A = [ a 1 , ..., a m ] ⊤ ∈ R m × n , f = ( f 1 , ..., f m ) ⊤ and y = ( y 1 , ..., y m ) ⊤ ∈ R m .

In this work, we consider the generalized Lasso as the recovery method [16,33,43], whose core idea is to ignore the non-linearity and minimize the regular ℓ 2 loss. In addition, we need to specify a constraint that reflects the low-complexity nature of x ∗ , and specifically, we introduce a problemdependent scaling factor T ∈ R and use the constraint ' x ∈ T K '. Note that this is necessary even if the problem is linear; for example, with observations y = 2 Ax ∗ , one needs to minimize the ℓ 2 loss over ' x ∈ 2 K '. Also, when the generative prior is given by T x ∗ ∈ K = G ( B k 2 ( r )) , we should simply use ' x ∈ K ' as constraint; this is technically equivalent to the treatment adopted in [33] (see more discussions in Remark 5 below). Taken collectively, we consider

Importantly, we want to achieve uniform recovery of all x ∗ ∈ K with a single realization of ( A , f ) .

2.2 Assumptions

Let f be the function that characterizes our nonlinear measurements. We introduce several assumptions on f here, and then verify them for specific models in Section 2.3. We define the set of discontinuities as

We define the notion of jump discontinuity as follows.

Definition 3. (Jump discontinuity) . A function f : R → R has a jump discontinuity at x 0 if both L -:= lim x → x -0 f ( x ) and L + := lim x → x + 0 f ( x ) exist but L -= L + . We simply call the oscillation at x 0 , i.e., | L + -L -| , the jump.

̸

Roughly put, our framework applies to piece-wise Lipschitz continuous f i with (at most) countably infinite jump discontinuities, which have bounded jumps and are well separated. The precise statement is given below.

Assumption 2. For some ( B 0 , L 0 , β 0 ) , the following statement unconditionally holds true for any realization of f (specifically, f 1 , . . . , f m in our observations):

D f is one of the following: ∅ , a finite set, or a countably infinite set;

All discontinuities of f (if any) are jump discontinuities with the jump bounded by B 0 ;

f is L 0 -Lipschitz on any interval ( a, b ) satisfying ( a, b ) ∩ D f = ∅ .

̸

| a -b | ≥ β 0 holds for any a, b ∈ D f , a = b (we set β 0 = ∞ if | D f | ≤ 1 ).

For simplicity, we assume f ( x 0 ) = lim x → x + 0 f ( x ) for x 0 ∈ D f . 3

We note that Assumption 2 is satisfied by L -Lipschitz f with ( B 0 , L 0 , β 0 ) = (0 , L, ∞ ) , 1bit quantized observation f ( · ) = sign( · + τ ) ( τ is the potential dither, similarly below) with ( B 0 , L 0 , β 0 ) = (2 , 0 , ∞ ) , and uniformly quantized observation f ( · ) = δ ( ⌊ · + τ δ ⌋ + 1 2 ) with ( B 0 , L 0 , β 0 ) = ( δ, 0 , δ ) .

̸

Under Asssumption 2, for any β ∈ [0 , β 0 2 ) we construct f i,β as the Lipschitz approximation of f i to deal with the potential discontinuity of f i (i.e., D f i = ∅ ). Specifically, f i,β modifies f i in D f i +[ -β 2 , β 2 ] to be piece-wise linear and Lipschitz continuous; see its precise definition in (3.4).

We develop Theorem 2 to bound certain product processes appearing in the analysis, which produces bounds tighter than [36] when the index sets have low metric entropy. To make Theorem 2 applicable, we further make the following Assumption 3, which can be checked case-by-case by estimating the sub-Gaussian norm and probability tail. Also, U (1) g and U (2) g can even be a bit crude because the measurement number in Theorem 1 depends on them in a logarithmic manner.

Assumption 3. Let a ∼ N (0 , I n ) , under Assumptions 1-2, we define the Lipschitz approximation f i,β as in (3.4). We let

For all β ∈ (0 , β 0 2 ) , we assume the following holds with some parameters ( A (1) g , U (1) g , P (1) 0 ) and ( A (2) g , U (2) g , P (2) 0 ) :

/a80 To build a more complete theory we further introduce two useful quantities. For some x ∈ K , we define the target mismatch ρ ( x ) as in [17, Definition 1]:

ρ ( x ) = ∥ ∥ ❊ [ f i ( a ⊤ i x ) a i ] -T x ∥ ∥ 2 . (2.3) It is easy to see that ❊ [ f i ( a ⊤ i x ) a i ] minimizes the expected ℓ 2 loss ❊ [ ∥ y -Ax ∥ 2 2 ] , thus one can roughly understand ❊ [ f i ( a ⊤ i x ) a i ] as the expectation of ˆ x . Since T x is the desired ground truth, a small ρ ( x ) is intuitively an important ingredient for generalized Lasso to succeed. Fortunately, in many models, ρ ( x ) with a suitably chosen T will vanish (e.g., linear model [2], single index model [33], 1-bit model [31]) or at least be sufficiently small (e.g., 1-bit model with dithering [45]).

As mentioned before, our method to deal with discontinuity of f i is to introduce its approximation f i,β , which differs from f i only in D f i +[ -β 2 , β 2 ] . This will produce some bias because the actual observation is f i ( a ⊤ i x ∗ ) rather than f i,β ( a ⊤ i x ∗ ) . Hence, for some x ∈ K we define the following quantity to measure the bias induced by f i,β :

The following assumption can often be satisfied by choosing suitable T and sufficiently small β 1 .

Assumption 4. Suppose Assumptions 1-3 hold true with parameters B 0 , L 0 , β 0 , A (1) g , A (2) g . For the T used in (2.1), ρ ( x ) defined in (2.3) satisfies

Moreover, there exists some 0 < β 1 < β 0 2 such that

In the proof, the estimation error ∥ ˆ x -T x ∗ ∥ is contributed by a concentration term of scaling ˜ O ( ( A (1) g ∨ A (2) g ) √ k/m ) and some bias terms. The main aim of Assumption 4 is to pull down the bias terms so that the concentration term is dominant.

2.3 Main Theorem and its Implications

We now present our general theorem and apply it to some specific models.

Theorem 1. Under Assumptions 1-4, given any recovery accuracy ϵ ∈ (0 , 1) , if it holds that m ≳ ( A (1) g ∨ A (2) g ) 2 k L ϵ 2 , then with probability at least 1 -m ( P (1) 0 + P (2) 0 ) -m exp( -Ω( n )) -C exp( -Ω( k )) on a single realization of ( A , f ) := ( a i , f i ) i m =1 , we have the uniform signal recovery guarantee ∥ ˆ x -T x ∗ ∥ 2 ≤ ϵ for all x ∗ ∈ K , where ˆ x is the solution to (2.1) with y = f ( Ax ∗ ) , and L = log ˜ P is a logarithmic factor with ˜ P being polynomial in ( L, n ) and other parameters that typically scale as O ( L + n ) . See (C.11) for the precise expression of L .

To illustrate the power of Theorem 1, we specialize it to several models to obtain concrete uniform signal recovery results. Starting with Theorem 1, the remaining work is to select parameters that justify Assumptions 2-4. We summarize the strategy as follows: (i) Determine the parameters in Assumption 2 by the measurement model; (ii) Set T that verifies (2.5) (see Lemmas 8-11 for the following models); (iii) Set the parameters in Assumption 3, for which bounding the norm of Gaussian vector is useful; (iv) Set β 1 to guarantee (2.6) based on some standard probability argument. We only provide suitable parameters for the following concrete models due to space limit, while leaving more details to Appendix E.

(A) 1-bit GCS. Assume that we have the 1-bit observations y i = sign( a ⊤ i x ∗ ) ; then f i ( · ) = f ( · ) = sign( · ) satisfies Assumption 2 with ( B 0 , L 0 , β 0 ) = (2 , 0 , ∞ ) . In this model, it is hopeless to recover the norm of ∥ x ∗ ∥ 2 ; as done in previous work, we assume x ∗ ∈ K ⊂ S n -1 [31, Remark 1]. We set T = √ 2 /π and take the parameters in Assumption 3 as A (1) g ≍ 1 , U (1) g ≍ √ n, P (1) 0 ≍ exp( -Ω( n )) , A (2) g ≍ 1 , U (2) g ≍ 1 , P (2) 0 = 0 . We take β = β 1 ≍ k m to guarantee (2.6). With these choices, Theorem 1 specializes to the following:

Corollary 1. Consider Assumption 1 with f i ( · ) = sign( · ) and K ⊂ S n -1 , let ϵ ∈ (0 , 1) be any given recovery accuracy. If m ≳ k ϵ 2 log ( Lr √ mn ϵ ∧ ( k/m ) ) , 4 then with probability at least 1 -2 m exp( -cn ) -m exp( -Ω( k )) on a single draw of ( a i ) i m =1 , we have the uniform signal recovery guarantee ∥ ∥ ˆ x -√ 2 π x ∗ ∥ ∥ 2 ≤ ϵ for all x ∗ ∈ K , where ˆ x is the solution to (2.1) with y = sign( Ax ∗ ) and T = √ 2 π .

Remark 1. A uniform recovery guarantee for generalized Lasso in 1-bit GCS was obtained in [33, Section 5]. Their proof relies on the local embedding property in [31]. Note that such geometric property is often problem-dependent and highly nontrivial. By contrast, our argument is free of geometric properties of this kind.

Remark 2. For traditional 1-bit CS, [17, Corollary 2] requires m ≳ ˜ O ( k/ϵ 4 ) to achieve uniform ℓ 2 -accuracy of ϵ for all k -sparse signals, which is inferior to our ˜ O ( k/ϵ 2 ) . This is true for all remaining examples. To obtain such a sharper rate, the key technique is to use our Theorem 2 (rather than [36]) to obtain tighter bound for the product processes, as will be discussed in Remark 8.

(B) 1-bit GCS with dithering. Assume that the a ⊤ i x ∗ is quantized to 1-bit with dither 5 τ i iid ∼ U [ -λ, λ ] ) for some λ to be chosen, i.e., we observe y i = sign( a ⊤ i x ∗ + τ i ) . Following [45] we assume K ⊂ B n 2 ( R ) for some R > 0 . Here, using dithering allows the recovery of signal norm ∥ x ∗ ∥ 2 , so we do not need to assume K ⊂ S n -1 as in Corollary 1. We set λ = CR √ log m with sufficiently large C , and T = λ -1 . In Assumption 3, we take A (1) g ≍ 1 , U (1) g ≍ √ n, P (1) 0 ≍ exp( -Ω( n )) , A (2) g ≍ 1 , U (2) g ≍ 1 , and P (2) 0 = 0 . Moreover, we take β = β 1 = λk m to guarantee (2.6). Now we can invoke Theorem 1 to get the following.

Corollary 2. Consider Assumption 1 with f i ( · ) = sign( · + τ i ) , τ i ∼ U [ -λ, λ ] and K ⊂ B n 2 ( R ) , and λ = CR √ log m with sufficiently large C . Let ϵ ∈ (0 , 1) be any given recovery accuracy. If m ≳ k ϵ 2 log ( Lr √ mn λ ( ϵ ∧ ( k/m )) ) , then with probability at least 1 -2 m exp( -cn ) -m exp( -Ω( k )) on a single draw of ( a i , τ i ) i m =1 , we have the uniform signal recovery guarantee ∥ ˆ x -λ -1 x ∗ ∥ 2 ≤ ϵ for all x ∗ ∈ K , where ˆ x is the solution to (2.1) with y = sign( Ax ∗ + τ ) (here, τ = [ τ 1 , ..., τ m ] ⊤ ) and T = λ -1 .

Remark 3. To our knowledge, the only related prior result is in [45, Theorem 3.2]. However, their result is restricted to ReLU networks. By contrast, we deal with the more general Lipschitz generative models; by specializing our result to the ReLU network that is typically ( n Θ( d ) ) -Lipschitz [2] ( d is

the number of layers), our error rate coincides with theirs up to a logarithmic factor. Additionally, as already mentioned in the Introduction Section, our result can be generalized to a sensing vector with an unknown covariance matrix, unlike theirs which is restricted to isotropic sensing vectors. The advantage of their result is in allowing sub-exponential sensing vectors.

(C) Lipschitz-continuous SIM with generative prior. Assume that any realization of f is unconditionally ˆ L -Lipschitz, which implies Assumption 2 with ( B 0 , L 0 , β 0 ) = (0 , ˆ L, ∞ ) . We further assume /a80 ( f (0) ≤ ˆ B ) ≥ 1 -P ′ 0 for some ( ˆ B,P ′ 0 ) . Because the norm of x ∗ is absorbed into the unknown f ( · ) , we assume K ⊂ S n -1 . We set β = 0 so that f i,β = f i . We introduce the quantities µ = ❊ [ f ( g ) g ] , ψ = ∥ f ( g ) ∥ ψ 2 , where g ∼ N (0 , 1) . We choose T = µ and set parameters in Assumption 3 as A (1) g ≍ ψ + µ, U (1) g ≍ ( ˆ L + µ ) √ n + ˆ B, P (1) 0 ≍ P ′ 0 + exp( -Ω( n )) , A (2) g ≍ ψ + µ, U (2) g = 0 , P (2) 0 = 0 . Now we are ready to apply Theorem 1 to this model. We obtain:

Corollary 3. Consider Assumption 1 with ˆ L -Lipschitz f , suppose that /a80 ( f (0) ≤ ˆ B ) ≥ 1 -P ′ 0 , and define the parameters µ = ❊ [ f ( g ) g ] , ψ = ∥ f ( g ) ∥ ψ 2 with g ∼ N (0 , 1) . Let ϵ ∈ (0 , 1) be any given recovery accuracy. If m ≳ ( µ + ψ ) k ϵ 2 log ( Lr √ m [ n ( µ + ϵ )( ˆ L + µ )+ √ nµ ˆ B + ψ ] ( µ + ψ ) ϵ ) , then with probability at least 1 -2 m exp( -cn ) -mP ′ 0 -c 1 exp( -Ω( k )) on a single draw of ( a i , f i ) i m =1 , we have the uniform signal recovery guarantee ∥ ˆ x -µ x ∗ ∥ 2 ≤ ϵ for all x ∗ ∈ K , where ˆ x is the solution to (2.1) with y = f ( Ax ∗ ) and T = µ .

Remark 4. While the main result of [33] is non-uniform, it was noted in [33, Section 5] that a similar uniform error rate can be established for any deterministic 1 -Lipschitz f . Our result here is more general in that the ˆ L -Lipschitz f is possibly random. Note that randomness on f is significant because it provides much more flexibility (e.g., additive random noise).

Remark 5. For SIM with unknown f i it may seem impractical to use (2.1) as it requires µ = ❊ [ f ( g ) g ] where g ∼ N (0 , 1) . However, by assuming µ x ∗ ∈ K = G ( B k 2 ( r )) as in [33], which is natural for sufficiently expressive G ( · ) , we can simply use x ∈ K as constraint in (2.1). Our Corollary 3 remains valid in this case under some inessential changes of log µ factors in the sample complexity.

(D) Uniformly quantized GCS with dithering. The uniform quantizer with resolution δ > 0 is defined as Q δ ( a ) = δ ( ⌊ a δ ⌋ + 1 2 ) for a ∈ R . Using dithering τ i ∼ U [ -δ 2 , δ 2 ] , we suppose that the observations are y i = Q δ ( a ⊤ i x ∗ + τ i ) . This satisfies Assumption 2 with ( B 0 , L 0 , β 0 ) = ( δ, 0 , δ ) . We set T = 1 and take parameters for Assumption 3 as follows: A (1) g , U (1) g , A (2) g , U (2) g ≍ δ , and P (1) 0 = P (2) 0 = 0 . We take β = β 1 ≍ kδ m to confirm (2.6). With these parameters, we obtain the following from Theorem 1.

Corollary 4. Consider Assumption 1 with f ( · ) = Q δ ( · + τ ) , τ ∼ U [ -δ 2 , δ 2 ] for some quantization resolution δ > 0 . Let ϵ > 0 be any given recovery accuracy. If m ≳ δ 2 k ϵ 2 log ( Lr √ mn ϵ ∧ [ kδ/ ( m √ n )] ) , then with probability at least 1 -2 m exp( -cn ) -c 1 exp( -Ω( k )) on a single draw of ( a i , τ i ) i m =1 , we have the uniform recovery guarantee ∥ ˆ x -x ∗ ∥ 2 ≤ ϵ for all x ∗ ∈ K , where ˆ x is the solution to (2.1) with y = Q δ ( Ax + τ ) and T = 1 (here, τ = [ τ 1 , . . . , τ m ] ⊤ ).

Remark 6. While this dithered uniform quantized model has been widely studied in traditional CS (e.g., non-uniform recovery [8,48], uniform recovery [17,52]), it has not been investigated in GCS even for non-uniform recovery. Thus, this is new to the best of our knowledge.

A simple extension to the noisy model y = f ( Ax ∗ ) + η where η ∈ R m has i.i.d. sub-Gaussian entries can be obtained by a fairly straightforward extension of our analysis; see Appendix F.

3 Proof Sketch

To provide a sketch of our proof, we begin with the optimality condition ∥ y -A ˆ x ∥ 2 2 ≤ ∥ y -A ( T x ∗ ) ∥ 2 2 . We expand the square and plug in y = f ( Ax ∗ ) to obtain

For the final goal ∥ ˆ x -T x ∗ ∥ 2 ≤ ϵ , up to rescaling, it is enough to prove ∥ ˆ x -T x ∗ ∥ 2 ≤ 3 ϵ . We assume for convenience that ∥ ˆ x -T x ∗ ∥ 2 > 2 ϵ , without loss of generality. Combined with

c

where the normalized error lives, i.e. ˆ x -T x ∗ ∥ ˆ x -T x ∗ ∥ 2 ∈ ( K -ϵ ) ∗ . Our strategy is to establish a uniform lower bound (resp., upper bound) for the left-hand side (resp., the right-hand side) of (3.1). We emphasize that these bounds must hold uniformly for all x ∗ ∈ K .

It is relatively easy to use set-restricted eigenvalue condition (S-REC) [2] to establish a uniform lower bound for the left-hand side of (3.1), see Appendix B.1 for more details. It is significantly more challenging to derive an upper bound for the right-hand side of (3.1). As the upper bound must hold uniformly for all x ∗ , we first take the supremum over x ∗ and ˆ x and consider bounding the following:

where ( K -ϵ ) ∗ is defined in (3.2). Clearly, R u is the supremum of a product process, whose factors are indexed by x ∈ K and v ∈ ( K -ϵ ) ∗ . It is, in general, challenging to control a product process, and existing results often require both factors to satisfy a certain 'sub-Gaussian increments' condition (e.g., [36, 37]). However, the first factor of R u (i.e., f i ( a ⊤ i x ∗ ) -T a ⊤ i x ∗ ) does not admit such a condition when f i is not continuous (e.g., the 1-bit model f i = sign( · ) ). We will construct the Lipschitz approximation of f i to overcome this difficulty shortly in Section 3.1.

Remark 7. We note that these challenges stem from our pursuit of uniform recovery. In fact, a non-uniform guarantee for SIM was presented in [33, Theorem 1]. In its proof, the key ingredient is [33, Lemma 3] that bounds R u without the supremum on x . This can be done as long as f i ( a ⊤ i x ∗ ) is sub-Gaussian, while the potential discontinuity of f i is totally unproblematic.

3.1 Lipschitz Approximation

For any x 0 ∈ D f i we define the one-sided limits as f -i ( x 0 ) = lim x → x -0 f i ( x ) and f + i ( x 0 ) = lim x → x + 0 f i ( x ) , and write their average as f a i ( x 0 ) = 1 2 ( f -i ( x 0 )+ f + i ( x 0 )) . Given any approximation accuracy β ∈ (0 , β 0 2 ) , we construct the Lipschitz continuous function f i,β as:

We have defined the approximation error ε i,β ( · ) = f i,β ( · ) -f i ( · ) in Assumption 3. An important observation is that both f i,β and | ε i,β | are Lipschitz continuous (see Lemma 1 below). Here, it is crucial to consider | ε i,β | rather than ε i,β as the latter is not continuous; see Figure 1 for an intuitive graphical illustration and more explanations in Appendix B.2.

Lemma 1. With B 0 , L 0 , β 0 given in Assumption 2, for any β ∈ (0 , β 0 2 ) , f i,β is ( L 0 + B 0 β ) -Lipschitz over R , and | ε i,β | is ( 2 L 0 + B 0 β ) -Lipschitz over R .

3.2 Bounding the product process

We now present our technique to bound R u . Recall that ξ i,β ( a ) and ε i,β ( a ) were defined in (2.2). By Lemma 1, ξ i,β is ( L 0 + T + B 0 β ) -Lipschitz. Now we use f i ( a ) -Ta = ξ i,β ( a ) -ε i,β to decompose R u (in the following, we sometimes shorten ' sup x ∈K sup v ∈ ( K -ϵ ) ∗ ' as ' sup x , v '):

-2

-1

0

1

2

-1

0

1

2

3

4

-2

-1

0

1

2

-0.5

0

0.5

It remains to control R u 1 and R u 2 . By the Lipschitz continuity of ξ i,β and | ε i,β | , the factors of R u 1 and R u 2 admit sub-Gaussian increments, so it is natural to first center them and then invoke the concentration inequality for product process due to Mendelson [36, Theorem 1.13], which we restate in Lemma 5 (Appendix A). However, this does not produce a tight bound and would eventually require ˜ O ( k/ϵ 4 ) to achieve a uniform ℓ 2 -error of ϵ , as is the case in [17, Section 4].

In fact, Lemma 5 is based on Gaussian width and hence blind to the fact that K , ( K -ϵ ) ∗ here have low metric entropy (Lemma 6). By characterizing the low intrinsic dimension of index sets via metric entropy, we develop the following concentration inequality that can produce tighter bound for R u 1 and R u 2 . This also allows us to derive uniform error rates sharper than those in [17, Section 4].

Theorem 2. Let g x = g x ( a ) and h v = h v ( a ) be stochastic processes indexed by x ∈ X ⊂ R p 1 , v ∈ V ⊂ R p 2 , both defined with respect to a common random variable a . Assume that:

(A1.) g x ( a ) , h v ( a ) are sub-Gaussian for some ( A g , A h ) and admit sub-Gaussian increments regarding ℓ 2 distance for some ( M g , M h ) :

(A2.) On a single draw of a , for some ( L g , U g , L h , U h ) the following events simultaneously hold with probability at least 1 -P 0 :

Let a 1 , ..., a m be i.i.d. copies of a , and introduce the shorthand S g,h = L g U h + M g A h and T g,h = L h U g + M h A g . If m ≳ H ( X , A g A h √ mS g,h ) + H ( V , A g A h √ mT g,h ) , where H ( · , · ) is the metric entropy defined in Definition 2, then with probability at least 1 -mP 0 -2 exp [ -Ω ( H ( X , A g A h √ mS g,h ) + H ( V , A g A h √ mT g,h ) )] we have I ≲ A g A h √ m √ H ( X , A g A h √ mS g,h ) + H ( V , A g A h √ mT g,h ) , where I := sup x ∈X sup v ∈V ∣ ∣ 1 m ∑ m i =1 ( g x ( a i ) h v ( a i ) -❊ [ g x ( a i ) h v ( a i )] )∣ ∣ is the supremum of a product process.

Remark 8. We use R u 2 as an example to illustrate the advantage of Theorem 2 over Lemma 5. The key step is on bounding the centered process

Let g x ( a i ) = | ε i,β ( a ⊤ i x ) | and h v ( a i ) = | a ⊤ i v | , then one can use Theorem 2 or Lemma 5 to bound R u 2 ,c . Note that ∥ a ⊤ i v ∥ ψ 2 = O (1) justifies the choice A h = O (1) , and both H ( K , η ) and H (( K -ϵ ) ∗ , η ) depend linearly on k but only logarithmically on η (Lemma 6), so Theorem 2 could bound R u 2 ,c by ˜ O ( A g √ k/m ) that depends on M g in a logarithmic manner. However, the bound produced by Lemma 5 depends linearly on M g ; see term M g A h ω ( K ) √ m in (A.1). From (3.6), M g should be proportional to the Lipschitz constant of | ε i,β | , which scales as 1 β (Lemma 1). The issue is that in many cases we need to take extremely small β to guarantee that (2.6) holds true (e.g., we take β ≍ k/m in 1-bit GCS). Thus, Lemma 5 produces a worse bound compared to our Theorem 2.

4 Conclusion

In this work, we built a unified framework for uniform signal recovery in nonlinear generative compressed sensing. We showed that using generalized Lasso, a sample size of ˜ O ( k/ϵ 2 ) suffices to uniformly recover all x ∈ G ( B k 2 ( r )) up to an ℓ 2 -error of ϵ . We specialized our main theorem to 1-bit GCS with/without dithering, single index model, and uniformly quantized GCS, deriving uniform guarantees that are new or exhibit some advantages over existing ones. Unlike [33], our proof is free of any non-trivial embedding property. As part of our technical contributions, we constructed the Lipschitz approximation to handle potential discontinuity in the observation model, and also developed a concentration inequality to derive tighter bound for the product processes arising in the proof, allowing us to obtain a uniform error rate faster than [17]. Possible future directions include extending our framework to handle the adversarial noise and representation error.

Acknowledgment. J. Chen was supported by a Hong Kong PhD Fellowship from the Hong Kong Research Grants Council (RGC). J. Scarlett was supported by the Singapore National Research Foundation (NRF) under grant A-0008064-00-00. M. K. Ng was partially supported by the HKRGC GRF 17201020, 17300021, CRF C7004-21GF and Joint NSFC-RGC N-HKU76921.

References

M. Asim, M. Daniels, O. Leong, A. Ahmed, and P. Hand, 'Invertible generative models for inverse problems: Mitigating representation error and dataset bias,' in International Conference on Machine Learning . PMLR, 2020, pp. 399-409.

A. Bora, A. Jalal, E. Price, and A. G. Dimakis, 'Compressed sensing using generative models,' in International Conference on Machine Learning . PMLR, 2017, pp. 537-546.

P. T. Boufounos and R. G. Baraniuk, '1-bit compressive sensing,' in Annual Conference on Information Sciences and Systems . IEEE, 2008, pp. 16-21.

T. T. Cai and A. Zhang, 'Sparse representation of a polytope and recovery of sparse signals and low-rank matrices,' IEEE Transactions on Information Theory , vol. 60, no. 1, pp. 122-132, 2013.

E. J. Candes and T. Tao, 'Decoding by linear programming,' IEEE Transactions on Information Theory , vol. 51, no. 12, pp. 4203-4215, 2005.

E. J. Candes and T. Tao, 'Near-optimal signal recovery from random projections: Universal encoding strategies?' IEEE Transactions on Information Theory , vol. 52, no. 12, pp. 5406-5425, 2006.

J. Chen and M. K. Ng, 'Uniform exact reconstruction of sparse signals and low-rank matrices from phase-only measurements,' IEEE Transactions on Information Theory , vol. 69, no. 10, pp. 6739-6764, 2023.

J. Chen, M. K. Ng, and D. Wang, 'Quantizing heavy-tailed data in statistical estimation: (Near) Minimax rates, covariate quantization, and uniform recovery,' arXiv preprint arXiv:2212.14562 , 2022.

J. Chen, C.-L. Wang, M. K. Ng, and D. Wang, 'High dimensional statistical estimation under uniformly dithered one-bit quantization,' IEEE Transactions on Information Theory , vol. 69, no. 8, pp. 5151-5187, 2023.

H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, 'Diffusion posterior sampling for general noisy inverse problems,' in International Conference on Learning Representations , 2022.

G. Daras, J. Dean, A. Jalal, and A. Dimakis, 'Intermediate layer optimization for inverse problems using deep generative models,' in International Conference on Machine Learning . PMLR, 2021, pp. 2421-2432.

M. Dhar, A. Grover, and S. Ermon, 'Modeling sparse deviations for compressed sensing using generative models,' in International Conference on Machine Learning . PMLR, 2018, pp. 1214-1223.

S. Dirksen, 'Quantized compressed sensing: a survey,' in Compressed Sensing and Its Applications: Third International MATHEON Conference 2017 . Springer, 2019, pp. 67-95.

S. Dirksen and S. Mendelson, 'Non-Gaussian hyperplane tessellations and robust one-bit compressed sensing,' Journal of the European Mathematical Society , vol. 23, no. 9, pp. 29132947, 2021.

S. Foucart and H. Rauhut, A Mathematical Introduction to Compressive Sensing . Springer, 2013.

M. Genzel, 'High-dimensional estimation of structured signals from non-linear observations with general convex loss functions,' IEEE Transactions on Information Theory , vol. 63, no. 3, pp. 1601-1619, 2016.

M. Genzel and A. Stollenwerk, 'A unified approach to uniform signal recovery from nonlinear observations,' Foundations of Computational Mathematics , pp. 1-74, 2022.

R. M. Gray and T. G. Stockham, 'Dithered quantizers,' IEEE Transactions on Information Theory , vol. 39, no. 3, pp. 805-812, 1993.

P. Hand, O. Leong, and V. Voroninski, 'Phase retrieval under a generative prior,' Advances in Neural Information Processing Systems , vol. 31, 2018.

P. Hand and V. Voroninski, 'Global guarantees for enforcing deep generative priors by empirical risk,' in Conference On Learning Theory . PMLR, 2018, pp. 970-978.

L. Jacques and T. Feuillen, 'The importance of phase in complex compressive sensing,' IEEE Transactions on Information Theory , vol. 67, no. 6, pp. 4150-4161, 2021.

L. Jacques, J. N. Laska, P. T. Boufounos, and R. G. Baraniuk, 'Robust 1-bit compressive sensing via binary stable embeddings of sparse vectors,' IEEE Transactions on Information Theory , vol. 59, no. 4, pp. 2082-2102, 2013.

A. Jalal, S. Karmalkar, A. Dimakis, and E. Price, 'Instance-optimal compressed sensing via posterior sampling,' in International Conference on Machine Learning (ICML) , 2021.

A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. Tamir, 'Robust compressed sensing MRI with deep generative priors,' Advances in Neural Information Processing Systems , vol. 34, pp. 14 938-14 954, 2021.

A. Jalal, L. Liu, A. G. Dimakis, and C. Caramanis, 'Robust compressed sensing using generative models,' Advances in Neural Information Processing Systems , vol. 33, pp. 713-727, 2020.

Y. Jiao, D. Li, M. Liu, X. Lu, and Y. Yang, 'Just least squares: Binary compressive sampling with low generative intrinsic dimension,' Journal of Scientific Computing , vol. 95, no. 1, p. 28, 2023.

A. Kamath, E. Price, and S. Karmalkar, 'On the power of compressed sensing with generative models,' in International Conference on Machine Learning . PMLR, 2020, pp. 5101-5109.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' Proceedings of the IEEE , vol. 86, no. 11, pp. 2278-2324, 1998.

J. Liu and Z. Liu, 'Non-iterative recovery from nonlinear observations using generative models,' in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 233-243.

Z. Liu, S. Ghosh, and J. Scarlett, 'Towards sample-optimal compressive phase retrieval with sparse and generative priors,' Advances in Neural Information Processing Systems , vol. 34, pp. 17 656-17 668, 2021.

Z. Liu, S. Gomes, A. Tiwari, and J. Scarlett, 'Sample complexity bounds for 1-bit compressive sensing and binary stable embeddings with generative priors,' in International Conference on Machine Learning . PMLR, 2020, pp. 6216-6225.

Z. Liu and J. Han, 'Projected gradient descent algorithms for solving nonlinear inverse problems with generative priors,' arXiv preprint arXiv:2209.10093 , 2022.

Z. Liu and J. Scarlett, 'The generalized Lasso with nonlinear observations and generative priors,' Advances in Neural Information Processing Systems , vol. 33, pp. 19 125-19 136, 2020.

Z. Liu and J. Scarlett, 'Information-theoretic lower bounds for compressive sensing with generative models,' IEEE Journal on Selected Areas in Information Theory , vol. 1, no. 1, pp. 292-303, 2020.

Z. Liu, P. Luo, X. Wang, and X. Tang, 'Deep learning face attributes in the wild,' in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 3730-3738.

S. Mendelson, 'Upper bounds on product and multiplier empirical processes,' Stochastic Processes and their Applications , vol. 126, no. 12, pp. 3652-3680, 2016.

S. Mendelson, 'On multiplier processes under weak moment assumptions,' in Geometric Aspects of Functional Analysis: Israel Seminar (GAFA) 2014-2016 . Springer, 2017, pp. 301-318.

X. Meng and Y. Kabashima, 'Quantized compressed sensing with score-based generative models,' in International Conference on Learning Representations , 2023.

A. Naderi and Y. Plan, 'Sparsity-free compressed sensing with applications to generative priors,' IEEE Journal on Selected Areas in Information Theory , 2022.

G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, 'Deep learning techniques for inverse problems in imaging,' IEEE Journal on Selected Areas in Information Theory , vol. 1, no. 1, pp. 39-56, 2020.

Y. Plan and R. Vershynin, 'Robust 1-bit compressed sensing and sparse logistic regression: A convex programming approach,' IEEE Transactions on Information Theory , vol. 59, no. 1, pp. 482-494, 2012.

Y. Plan and R. Vershynin, 'One-bit compressed sensing by linear programming,' Communications on Pure and Applied Mathematics , vol. 66, no. 8, pp. 1275-1297, 2013.

Y. Plan and R. Vershynin, 'The generalized lasso with non-linear observations,' IEEE Transactions on information theory , vol. 62, no. 3, pp. 1528-1537, 2016.

Y. Plan, R. Vershynin, and E. Yudovina, 'High-dimensional estimation with geometric constraints,' Information and Inference: A Journal of the IMA , vol. 6, no. 1, pp. 1-40, 2017.

S. Qiu, X. Wei, and Z. Yang, 'Robust one-bit recovery via ReLU generative networks: Nearoptimal statistical rate and global landscape analysis,' in International Conference on Machine Learning . PMLR, 2020, pp. 7857-7866.

T. M. Quan, T. Nguyen-Duc, and W.-K. Jeong, 'Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss,' IEEE Transactions on Medical Imaging , vol. 37, no. 6, pp. 1488-1497, 2018.

J. Scarlett, R. Heckel, M. R. Rodrigues, P. Hand, and Y. C. Eldar, 'Theoretical perspectives on deep learning methods in inverse problems,' IEEE Journal on Selected Areas in Information Theory , 2022.

C. Thrampoulidis and A. S. Rawat, 'The generalized lasso for sub-gaussian measurements with dithered quantization,' IEEE Transactions on Information Theory , vol. 66, no. 4, pp. 2487-2500, 2020.

Y. Traonmilin and R. Gribonval, 'Stable recovery of low-dimensional cones in Hilbert spaces: One RIP to rule them all,' Applied and Computational Harmonic Analysis , vol. 45, no. 1, pp. 170-205, 2018.

R. Vershynin, High-dimensional probability: An introduction with applications in data science . Cambridge University Press, 2018, vol. 47.

J. Whang, E. Lindgren, and A. Dimakis, 'Composing normalizing flows for inverse problems,' in International Conference on Machine Learning , 2021, pp. 11 158-11 169.

C. Xu and L. Jacques, 'Quantized compressive sensing with RIP matrices: The benefit of dithering,' Information and Inference: A Journal of the IMA , vol. 9, no. 3, pp. 543-586, 2020.

Supplementary Material

A Unified Framework for Uniform Signal Recovery in Nonlinear Generative Compressed Sensing

(NeurIPS 2023)

A Technical Lemmas

Lemma 2. (Lemma 2.7.7, [50]) . Let X,Y be sub-Gaussian, then XY is sub-exponential with ∥ XY ∥ ψ 1 ≤ ∥ X ∥ ψ 2 ∥ Y ∥ ψ 2 .

Lemma 3. (Centering, [50, Exercise 2.7.10]) . For some absolute constant C , ∥ X -❊ X ∥ ψ 1 ≤ C ∥ X ∥ ψ 1 .

Lemma 4. (Bernstein's inequality, [50, Theorem 2.8.1]) . Let X 1 , ..., X N be independent, zero-mean, sub-exponential random variables. Then for every t ≥ 0 , for some absolute constant c we have

Lemma 5. ( [36], statement adapted from [16, Theorem 8]) . Let g x = g x ( a ) and h v = h v ( a ) be stochastic processes indexed by x ∈ X ⊂ R p 1 , v ∈ V ⊂ R p 2 , both defined on some common random variable a . Assume that (A1.) in Theorem 2 holds, and let a 1 , ..., a m be i.i.d. copies of a . Then for any u ≥ 1 , with probability at least 1 -2 exp( -cu 2 ) we have the bound

where ω ( · ) is the Gaussian width defined as ω ( X ) = ❊ sup x ∈X g ⊤ x where g ∼ N (0 , I p 1 ) . The proofs of the remaining lemmas will be provided in Appendix D. (Some simple facts such as Lemma 8 were already used in prior works; while we provide the proofs for completeness.)

Lemma 6. (Metric entropy of some constraint sets) . Assume K = G ( B k 2 ( r )) for some L -Lipschitz generative model G . Let K -= K-K , for some T > 0 , ϵ ∈ (0 , 1) let K -ϵ := ( T K -) ∩ ( B n 2 (2 ϵ ) ) c , and further define ( K -ϵ ) ∗ = { z ∥ z ∥ 2 : z ∈ K -ϵ } . Then for any η ∈ (0 , Lr ) , we have

where H ( · , · ) is the metric entropy defined in Definition 2.

/a80 In the following, Lemmas 8-11 indicate suitable choices of T in the concrete models we consider. These choices can make ρ ( x ) in (2.3) sufficiently small or even zero.

Lemma 7. (Bound the ℓ 2 -norm of Gaussian vector) . If a ∼ N (0 , I n ) , then /a80 ( |∥ a ∥ 2 - √ n | ≥ t ) ≤ 2 exp( -Ct 2 ) . In particular, setting t ≍ √ n yields ( ∥ a ∥ 2 ≥ √ n ) ≤ 2 exp( -Ω( n )) .

Lemma 8. (Choice of T in 1-bit GCS) . If a ∼ N (0 , I n ) , then for any x ∈ S n -1 it holds that [sign( a ⊤ x ) a ] = √ 2 π x .

❊ Lemma 9. (Choice of T in 1-bit GCS with dithering) . If a ∼ N (0 , I n ) and τ ∼ U [ -λ, λ ] are independent, and λ = CR √ log m with sufficiently large C , then for any x ∈ B n 2 ( R ) it holds that ∥ ❊ [sign( a ⊤ x + τ ) a ] -x λ ∥ 2 = O ( m -9 ) .

Lemma 10. (Choice of T in SIM) . If a ∼ N (0 , I n ) , for some function f and any x ∈ S n -1 it holds that [ f ( a ⊤ x ) a ] = µ x for µ = [ f ( g ) g ] with g ∼ N (0 , 1) .

❊ Lemma 12 facilitates our analysis of the uniform quantizer.

❊ ❊ Lemma 11. (Choice of T in uniformly quantized GCS with dithering) . Given any δ > 0 , let τ ∼ U [ -δ 2 , δ 2 ] and Q δ ( · ) = δ ( ⌊ · δ ⌋ + 1 2 ) . Then, for any a ∈ R , it holds that ❊ [ Q δ ( a + τ )] = a . In particular, let a ∈ R n be a random vector satisfying ❊ ( aa ⊤ ) = I n , and τ ∼ U [ -δ 2 , δ 2 ] be independent of a , then we have [ Q δ ( a ⊤ x + τ ) a ] = x .

Lemma 12. Let f i ( · ) = δ ( ⌊ · + τ i δ ⌋ + 1 2 ) for τ i ∼ U [ -δ 2 , δ 2 ] , and f i,β ( · ) be defined in (3.4) for some 0 < β < δ 2 . Moreover, let ξ i,β ( a ) = f i,β ( a ) -a , ε i,β ( a ) = f i,β ( a ) -f i ( a ) , then for any a ∈ R , | ξ i,β ( a ) | ≤ 2 δ , | ε i,β ( a ) | ≤ δ holds deterministically.

More generally, the approximation error | ε i,β ( a ) | can always be bounded as follows.

Lemma 13. Suppose that f i satisfies Assumption 2, and for any β ∈ [0 , β 0 2 ] we construct f i,β as in (3.4). Then, for any a ∈ R , we have | ε i,β ( a ) | ≤ ( 3 L 0 β 2 + B 0 ) ✶ ( a ∈ D f i +[ -β 2 , β 2 ]) . B More Details of the Proof Sketch

B.1 Set-Restricted Eigenvalue Condition

Definition 4. Let S ⊂ R n . For parameters γ, δ > 0 , a matrix A ∈ R m × n is said to satisfy S-REC( S , γ, δ ) if the following holds:

It was proved in [2] that 1 √ m A satisfies the S-REC with high probability if the entries of A are i.i.d. standard Gaussian.

Lemma 14. (Lemma 4.1 in [2]) . Let G : B k 2 ( r ) → R n be L -Lipschitz for some r, L > 0 , and define K = G ( B k 2 ( r )) . For any α ∈ (0 , 1) , if A ∈ R m × n has i.i.d. N (0 , 1) entries, and m = Ω ( k α 2 log Lr δ ) , then 1 √ m A satisfies S-REC( K , 1 -α , δ ) with probability at least 1 -exp( -Ω( α 2 m )) .

B.2 Lipschitz Approximation

The approximation error ε i,β ( · ) can be expanded as:

Although | ε i,β | is Lipschitz continuous, ε i,β is not. In particular, given x 0 ∈ D f i we note that

Thus, it is crucial to include the absolute value for rendering the continuity.

C Proofs of Main Results

C.1 Proof of Theorem 1

Proof. Up to rescaling, we only need to prove that ∥ ˆ x -T x ∗ ∥ 2 ≤ 3 ϵ holds uniformly for all x ∗ ∈ K . We can assume ∥ ˆ x -T x ∗ ∥ 2 ≥ 2 ϵ ; otherwise, the desired bound is immediate.

(1) Lower bounding the left-hand side of (3.1).

We use S-REC to find a lower bound for ∥ ∥ A √ m ( ˆ x -T x ∗ ) ∥ ∥ 2 2 . Specifically, we invoke Lemma 14 with α = 1 2 and δ = ϵ 2 T , which gives that under m = Ω ( k log LTr ϵ ) , with probability at least 1 -exp( -cm ) , the following holds:

Recall that we assume ∥ ˆ x -T x ∗ ∥ 2 ≥ 2 ϵ , T -1 ˆ x ∈ K , and x ∗ ∈ K , so we set x 1 = ˆ x T , x 2 = x ∗ in (C.1) to obtain

Thus, the left-hand side of (3.1) can be lower bounded by Ω ( ∥ ˆ x -T x ∗ ∥ 2 2 ) .

(2) Upper bounding the right-hand side of (3.1).

As analysed in (3.3) and (3.5), the right-hand side of (3.1) is bounded by 2 ∥ ˆ x -T x ∗ ∥ 2 · ( R u 1 + R u 2 ) , so all that remains is to bound R u 1 , R u 2 . In the rest of the proof, we simply write sup x , v := sup x ∈K , v ∈ ( K -ϵ ) ∗ and recall the shorthand ξ i,β ( a ) = f i,β ( a ) -Ta . Thus, the first factor of R u 1 is given by ξ i,β ( a ⊤ i x ) . By centering, we have

and

We will invoke Theorem 2 multiple times to derive the required bounds.

(2.1) Bounding the centered product process R u 1 ,c .

We let g x ( a i ) = ξ i,β ( a ⊤ i x ) and h v ( a i ) = a ⊤ i v , and write

We verify conditions in Theorem 2 as follows:

For any x , x ′ ∈ K , because ξ i,β is ( L 0 + B 0 β + T ) -Lipschitz continuous (Lemma 1), we have

Since a i ∼ N ( 0 , I n ) , by Lemma 7, with probability 1 -2 exp( -Ω( n )) we have ∥ a i ∥ 2 = O ( √ n ) . On this event, we have

Recall ( K -ϵ ) ∗ in (3.2). Since ( K -ϵ ) ∗ ⊂ S n -1 , for any v , v ′ ∈ ( K -ϵ ) ∗ , we have ∥ a ⊤ i v -a ⊤ i v ′ ∥ ψ 2 = O (1) ∥ v -v ′ ∥ 2 , and when ∥ a i ∥ = O ( √ n ) we have | a ⊤ i v -a ⊤ i v ′ | ≤ ∥ a i ∥ 2 ∥ v -v ′ ∥ 2 = O ( √ n ) ∥ v -v ′ ∥ 2 . Moreover, because ( K -ϵ ) ∗ ⊂ B n 2 , for any v ∈ ( K -ϵ ) ∗ we have ∥ a ⊤ i v ∥ ψ 2 = O (1) , | a ⊤ i v | ≤ ∥ a i ∥ 2 = O ( √ n ) .

Combined with Assumption 3 and its parameters ( A (1) g , U (1) g , P (1) 0 ) and ( A (2) g , U (2) g , P (2) 0 ) , R u 1 ,c satisfies the conditions of Theorem 2 with the following parameters

and P 0 = P (1) 0 +2exp( -Ω( n )) . Now suppose that we have

and note that by using Lemma 6, (C.4) can be guaranteed by

Then Theorem 2 yields that the following bound holds with probability at least 1 -mP (1) 0 -C exp( -Ω( k )) -m exp( -Ω( n )) :

(2.2) Bounding the centered product process R u 2 ,c .

We let g x ( a i ) = | ε i,β ( a ⊤ i x ) | and h v ( a i ) = | a ⊤ i v | , and write

We verify the conditions in Theorem 2 as follows:

For any x , x ′ ∈ K , because | ε i,β | is (2 L 0 + B 0 β ) -Lipschitz continuous (Lemma 1), we have

By Lemma 7, with probability at least 1 -2 exp( -Ω( n )) we have ∥ a i ∥ 2 = O ( √ n ) . On this event, we have

For any v , v ′ ∈ ( K -ϵ ) ∗ we have ∥ ∥ | a ⊤ i v | - | a ⊤ i v ′ | ∥ ∥ ψ 2 ≤ ∥ a ⊤ i ( v -v ′ ) ∥ ψ 2 = O (1) ∥ v -v ′ ∥ 2 . Similarly as before, we assume ∥ a i ∥ 2 = O ( √ n ) , which gives | a ⊤ i v -a ⊤ i v ′ | ≤ ∥ a i ∥ 2 ∥ v -v ′ ∥ 2 = O ( √ n ) ∥ v -v ′ ∥ 2 . Moreover, ( K -ϵ ) ∗ ⊂ B n 2 implies ∥ ∥ | a ⊤ i v | ∥ ∥ ψ 2 = O (1) and | a ⊤ i v | ≤ ∥ a i ∥ 2 ∥ v ∥ 2 = O ( √ n ) holds for all v ∈ ( K -ϵ ) ∗ .

Combined with Assumption 3, R u 2 ,c satisfies the conditions of Theorem 2 with

and P 0 = P (2) 0 +2exp( -Ω( n )) . Suppose we have

which can be guaranteed (from Lemma 6) by

Then, we can invoke Theorem 2 to obtain that the following bound holds with probability at least 1 -mP (2) 0 -2 m exp( -Ω( n )) -C exp( -Ω( k )) :

(2.3) Bounding the expectation terms R u 1 ,e , R u 2 ,e .

Recall that ξ i,β ( a ) = f i,β ( a ) -Ta and ε i,β ( a ) = f i,β ( a ) -f i ( a ) , and so ξ i,β ( a ) = ε i,β ( a )+ f i ( a ) -Ta . Hence, by using [ a i a ⊤ i ] = I n and ∥ v ∥ 2 = 1 , we have

where ρ ( x ) is the model mismatch defined in (2.3), and R u 2 ,e is defined in (C.3). It remains to bound R u 2 ,e , for which we first apply Cauchy-Schwarz (with ∥ v ∥ 2 ≤ 1 ) and then use Lemma 13 to obtain

where we use Lemma 13 in the third and fourth line, and µ β ( x ) is defined in (2.4).

(3) Combining everything to conclude the proof.

Recall that in Assumption 4, we assume that

and we take sufficiently small β 1 such that

then by setting β = β 1 , the derived bound of R u 1 ,c + R u 2 ,c (see (C.6) and (C.8)) dominates that of R u 1 ,e + R u 2 ,e (see (C.9) and (C.10)), and so R u 1 + R u 2 ≲ R u 1 ,c + R u 2 ,c .

Recall that (C.6) and (C.8) are guaranteed by the sample size of (C.5) and (C.7), while (C.5) and (C.7) hold as long as

with probability at least 1 -m ( P (1) 0 + P (2) 0 ) -m exp( -Ω( n )) -C exp( -Ω( k )) we have

Therefore, the right-hand side of (3.1) can be uniformly bounded by

Combining with the uniform lower bound for the left-hand side of (3.1), i.e., Ω( ∥ ˆ x -T x ∗ ∥ 2 2 ) , we obtain the following bound uniformly for all x ∗ :

we again obtain ∥ ˆ x -T x ∗ ∥ 2 ≤ 3 ϵ , which completes the proof.

C.2 Proof of Theorem 2

Proof. Step 1. Control the process over finite nets.

Recall that X and V are the index sets of x and v , as stated in Theorem 2. We first establish the desired concentration for a fixed pair ( x , v ) ∈ X × V . By Lemma 2, ∥ g x ( a i ) h v ( a i ) ∥ ψ 1 ≤ ∥ g x ( a i ) ∥ ψ 2 ∥ h v ( a i ) ∥ ψ 2 ≤ A g A h . Furthermore, centering (Lemma 3) gives

Thus, for fixed ( x , v ) ∈ X × V we define

Then, we can invoke Bernstein's inequality (Lemma 4) to obtain for any t ≥ 0 that

Hence, as long as

We construct G 1 as an η 1 -net of X , and G 2 as an η 2 -net of V , with both nets being minimal in that log |G 1 | = H ( X , η 1 ) , log |G 2 | = H ( V , η 2 ) , and where η 1 , η 2 are to be chosen later. Then, we take a union bound of (C.13) over ( x , v ) ∈ G 1 ×G 2 to obtain

Now we set t ≍ A g A h √ H ( X ,η 1 )+ H ( V ,η 2 ) m for a sufficiently large hidden constant. Then, if m ≥ C ( H ( X , η 1 ) + H ( V , η 2 )) for large enough C so that t A g A h ≤ 1 (we assume this now and will confirm it in (C.21) after specifying η 1 , η 2 ), (C.14) gives

Hence, from now on we proceed with the proof on the event

which holds within the promised probability.

Step 2. Control the approximation error of the nets.

We have derived a bound for sup x ∈G 1 , v ∈G 2 | I x , v | , while we want to control I = sup x ∈X , v ∈V | I x , v | , so we further investigate how close these two quantities are. We define the event as

then by assumption (A2.) in the theorem statement, a union bound gives /a80 ( E 1 ) ≥ 1 -mP 0 . In the following, we proceed with the analysis of the event E 1 . Combining with (C.15) we now bound | I x , v | for any given x ∈ X , v ∈ V . Specifically, we pick x ′ ∈ G 1 , v ′ ∈ G 2 such that ∥ x ′ -x ∥ 2 ≤ η 1 , ∥ v ′ -v ∥ 2 ≤ η 2 , and thus we have

Moreover, we have

We bound err 1 using the event E 1 as follows:

On the other hand, we bound err 2 using assumption (A1.) in the theorem statement. Noting that ❊ | X | = O ( ∥ X ∥ ψ 1 ) [50, Proposition 2.7.1(b)], and further applying Lemma 2 we obtain

Note that the bounds (C.18) and (C.19) hold uniformly for all ( x , v ) ∈ X × V , and hence, we can substitute them into (C.16) and (C.17) to obtain

Recall that we use the shorthand S g,h := L g U h + M g A h and T g,h := L h U g + M h A g . We set η 1 ≍ A g A h √ mS g,h , η 2 ≍ A g A h √ mT g,h so that the right-hand side of (C.20) is dominated by the first term. Overall, with a sample size satisfying

we can bound I = sup x ∈X sup v ∈V | I x , v | (defined in the theorem statement) as

with probability at least

This completes the proof.

D Other Omitted Proofs

D.1 Proof of Lemma 1 (Lipschitz continuity of f i,β and ε i,β ).

Proof. It is straightforward to check that f i,β and | ε i,β | are piece-wise continuous functions; hence, it suffices to prove that they are Lipschitz with the claimed Lipschitz constant over each piece. In any interval contained in the part of x / ∈ D f i + [ -β 2 , β 2 ] , f i,β = f i and | ε i,β | = 0 trivially satisfy the claim. In any interval contained in [ x 0 -β 2 , x 0 ] for some x 0 ∈ D f i , f i,β is linear with slope 2 β ( f a i ( x 0 ) -f i ( x 0 -β 2 ) ) , combined with the bound | f a i ( x 0 ) -f i ( x 0 -β 2 ) | ≤ | f a i ( x 0 ) -f -i ( x 0 ) | + | f -i ( x 0 ) -f i ( x 0 -β 2 ) | ≤ | f + i ( x 0 ) -f -i ( x 0 ) | 2 + L 0 β 2 ≤ 1 2 ( B 0 + L 0 β ) , we know that f i,β is ( L 0 + B 0 β ) -Lipschitz. Further, | ε i,β | = | f i,β -f i | , and f i is L 0 -Lipschitz over this interval, so | ε i,β | is ( 2 L 0 + B 0 β ) -Lipschitz continuous. A similar argument applies to an interval contained in [ x 0 , x 0 + β 2 ] .

D.2 Proof of Lemma 6 (Metric entropy of constraint sets).

Proof. Bounding H ( K , η ) .

By [50, Corollary 4.2.13], there exists an ( η Lr ) -net G 1 of B k 2 such that

where we use η ≤ Lr . Note that r G 1 is an ( η L ) -net of B k 2 ( r ) , and because G is L -Lipschitz, G ( r G 1 ) is an η -net of K , thus yielding H ( K , η ) ≤ k log 3 Lr η .

Bounding H ( K -, η ) and H ( K -ϵ , η ) .

We construct G 2 as an ( η 2 ) -net of K satisfying log |G 2 | ≤ k log 6 Lr η . Then, it is easy to see that G 2 -G 2 is an η -net of K -= K-K , showing that

For a given T > 0 , this directly implies H ( T K -, η ) ≤ 2 k log 6 TLr η . Moreover, because K -ϵ ⊂ T K -, by [50, Exercise 4.2.10] (which states that H ( K 1 , r ) ≤ H ( K 2 , r 2 ) holds for any r > 0 if K 1 ⊂ K 2 ) we obtain

Bounding H ( ( K -ϵ ) ∗ , η ) .

We construct G 3 as an ( ϵη ) -net of K -ϵ satisfying log |G 3 | ≤ 2 k log 12 TLr ϵη , then we consider ( G 3 ) ∗ := { z ∥ z ∥ 2 : z ∈ G 3 } . We aim to prove that ( G 3 ) ∗ is an η -net of ( K -ϵ ) ∗ . Note that any x 1 ∈ ( K -ϵ ) ∗ can be written as z 1 ∥ z 1 ∥ 2 for some z 1 ∈ K -ϵ and recall that ∥ z 1 ∥ 2 ≥ 2 ϵ . Moreover, by construction, there exists some z 2 ∈ G 3 such that ∥ z 1 -z 2 ∥ 2 ≤ ϵη . Note that z 2 ∥ z 2 ∥ 2 ∈ ( G 3 ) ∗ , and moreover we have

Hence, we obtain

which completes the proof.

D.3 Proof of Lemma 8 (Choice of T in 1-bit GCS).

Proof. Since x ∈ S n -1 , for some orthogonal matrix P we have Px = e 1 (the first column of I n ). Since ˜ a := Pa = [˜ a i ] has the same distribution as a , we have

D.4 Proof of Lemma 9 (Choice of T in 1-bit GCS with dithering).

Proof. We first note that

We plug this into (D.1), and note that x ⊤ v = [( a ⊤ x )( a ⊤ v )] , which gives

For any x ∈ B n 2 ( R ) and v ∈ S n -1 , we have ∥ a ⊤ x ∥ ψ 2 = O ( R ) and ∥ a ⊤ v ∥ ψ 2 = O (1) . Applying the Cauchy-Schwarz inequality, we obtain

Note that in the third line, we use the probability tail bound of the sub-Gaussian | a ⊤ x | , and in the last line, we use λ = CR √ log m with some sufficiently large C . The proof is completed by substituting this into (D.2).

D.5 Proof of Lemma 10 (Choice of T in SIM).

Proof. This lemma slightly generalizes that of Lemma 8. We again choose an orthogonal matrix P such that Px = e 1 , where e 1 represents the first column of I n . Since a and Pa have the same distribution, we have

D.6 Proof of Lemma 11 (Choice of T in uniformly quantized GCS with dithering).

Proof. In the theorem, the statement before 'In particular' can be found in [18, Theorem 1]. Based on this, we have [ Q δ ( a ⊤ x + τ ) a ] = a τ [ Q δ ( a ⊤ x + τ ) a ] = a ( aa ⊤ x ) = x .

❊ ❊ ❊ ❊ D.7 Proof of Lemma 12. (Bounds on | ξ i,β | and | ε i,β | for the uniform quantizer).

Proof. By the definition of f i,β in (3.4), we have | ε i,β ( a ) | = | f i,β ( a ) -f i ( a ) | ≤ δ . It follows that f i ( · ) = Q δ ( · + τ ) with Q δ ( a ) = δ ( ⌊ a δ ⌋ + 1 2 ) , and |Q δ ( a ) -a | ≤ δ 2 holds for any a ∈ R . Hence, we have | f i ( a ) -a | = |Q δ ( a + τ ) -( a + τ )+ τ | ≤ |Q δ ( a + τ ) -( a + τ ) | + | τ | ≤ δ 2 + δ 2 = δ . To complete the proof, we use the inequalities | ξ i,β ( a ) | ≤ | f i,β ( a ) -f i ( a ) | + | f i ( a ) -a | ≤ δ + δ = 2 δ .

D.8 Proof of Lemma 13. (Bound on the approximation error | ε i,β | )

Proof. For any a / ∈ D f i +[ -β 2 , β 2 ] , by the definition in (3.4) we have ε i,β ( a ) = 0 . If a ∈ [ x 0 -β 2 , x 0 ] for some x 0 ∈ D f i , then we have

where we use Lemma 1 and Assumption 2 in the third line, and use | a -x 0 | ≤ β 2 and f a i ( x 0 ) = 1 2 ( f -i ( x 0 ) + f + i ( x 0 ) ) in the fourth line.

E Parameter Selection for Specific Models

E.1 1-bit GCS

To specialize Theorem 1 to this model, we select the parameters as follows:

Assumption 2. Under the 1-bit observation model y i = sign( a ⊤ i x ∗ ) , the function f i ( · ) = f ( · ) = sign( · ) satisfies Assumption 2 with ( B 0 , L 0 , β 0 ) = (2 , 0 , ∞ ) .

(2.5) in Assumption 4. Recall that K ⊂ S . Under the assumption ∥ x ∥ 2 = 1 , we set T = √ 2 /π so that ρ ( x ) = 0 holds for all x ∈ X (Lemma 8), which provides (2.5).

Assumption 3. By Lemma 7, we have /a80 ( ∥ a ∥ 2 = O ( n )) ≥ 1 -2 exp( -Ω( n )) , and we suppose that this high-probability event holds. Also note that | f i,β | ≤ 1 . Hence, we have

n -1 ∗ √

Because ε i,β = f i,β -f i , we have ∥ ε i,β ( a ⊤ x ) ∥ ψ 2 = O (1) , and | ε i,β ( a ⊤ x ) | ≤ 2 holds deterministically. Hence, regarding the parameters in Assumption 3, we can take

(2.6) in Assumption 4. It remains to pick β 1 that satisfies (2.6). Note that D f i = { 0 } , and for any x ∈ K , a ⊤ x ∼ N (0 , 1) , so we have

Thus, we take β = β 1 ≍ k m to guarantee (2.6).

E.2 1-bit GCS with dithering

To specialize Theorem 1 to this model, we select the parameters as follows:

Assumption 2. The observation function can be written as f ( · ) = sign( · + τ ) with τ ∼ U [ -λ, λ ] , which satisfies Assumption 2 with ( B 0 , L 0 , β 0 ) = (2 , 0 , ∞ ) . √

Assumption 3. By Lemma 7, we have /a80 ( ∥ a ∥ 2 = O ( n )) ≥ 1 -2 exp( -Ω( n )) . Assume this event holds, and note that f i,β is still bounded by 1 , we have

(2.5) in Assumption 4. We set λ = CR log m with C large enough, so that Lemma 9 justifies (2.5). √

Moreover, because ε i,β = f i,β -f i , the following hold deterministically: ∥ ε i,β ( a ⊤ x ) ∥ ψ 2 = O (1) , sup x ∈K | ε i,β ( a ⊤ x ) | = O (1) . Thus, regarding the parameters in Assumption 3 we can take

(2.6) in Assumption 4. It remains to confirm (2.6) for suitable β 1 . For any β , note that D f i +[ -β 2 , β 2 ] = [ τ -β 2 , τ + β 2 ] , and hence for any x ∈ K ⊂ B n 2 ( R ) we have

which can be seen by conditioning on a . Hence, we can take β = β 1 = λk m to guarantee (2.6).

E.3 Lipschitz-continuous SIM with generative prior

To specialize Theorem 1 to this model, we select the parameters as follows:

Assumption 2. Since f is ˆ L -Lipschitz by assumption, it satisfies Assumption 2 with ( B 0 , L 0 , β 0 ) = (0 , ˆ L, ∞ ) .

(2.5) in Assumption 4. Recall that we have defined the quantities µ = ❊ [ f ( g ) g ] , ψ = ∥ f ( g ) ∥ ψ 2 , where g ∼ N (0 , 1) . Then, we choose T = µ so that ρ ( x ) = 0 holds for any x (Lemma 10), thus justifying (2.5).

Assumption 3. Because f i is ˆ L -Lipschitz and does not contain any discontinuity, there is no need to construct the Lipschitz approximation f i,β for some β > 0 , while we simply use β = 0 , which implies f i,β = f i and ε i,β = 0 . Note that ξ i,β ( a ) = f i ( a ) -µa , and so we have

We suppose ∥ a ∥ 2 = O ( √ n ) , which holds with probability at least 1 -2 exp( -Ω( n )) (Lemma 7); we also suppose f i (0) ≤ ˆ B , which holds with probability at least 1 -P ′ 0 by assumption. On these two events, we have

Combined with ε i,β = 0 , we can set the parameters in Assumption 3 as follows:

(2.6) in Assumption 4. Because β = 0 and D f i = ∅ , (2.6) is trivially satisfied.

E.4 Uniformly quantized GCS with dithering

To specialize Theorem 1 to this model, we select the parameters as follows:

Assumption 2. The uniform quantizer with resolution δ > 0 is defined as Q δ ( a ) = δ ( ⌊ a δ ⌋ + 1 2 ) for a ∈ R . We consider this quantizer with dithering τ i ∼ U [ -δ 2 , δ 2 ] . Specifically, we observe y i = Q δ ( a ⊤ i x ∗ + τ i ) , so the observation function is f ( · ) = Q δ ( · + τ ) with τ ∼ U [ -δ 2 , δ 2 ] . Hence, Assumption 2 is satisfied with ( B 0 , L 0 , β 0 ) = ( δ, 0 , δ ) .

(2.5) in Assumption 4. The benefit of dithering is to whiten the quantization noise. With T = 1 , for any x ∈ K , Lemma 11 implies ρ ( x ) = ∥ ❊ [ Q δ ( a ⊤ i x + τ i ) a i ] -x ∥ 2 = 0 , thus justifying (2.5).

Assumption 3. Note that for any β ∈ (0 , δ 2 ) , by Lemma 12, we can take the parameters for Assumption 3 as follows:

(2.6) in Assumption 4. All that remains is to pick β = β 1 that satisfies (2.6). Because D f i = -τ i + δ Z , hence for any x ∈ K we have

/a80 /a80 which can be seen by using the randomness of τ ∼ U [ -δ 2 , δ 2 ] conditionally on x . Hence, we take β = β 1 ≍ kδ m , which provides (2.6).

F Handling Sub-Gaussian Additive Noise

In this appendix, we describe how our results can be extended to the noisy model y = f ( Ax ∗ ) + η , where η ∈ R m is the noise vector that is independent of ( A , f ) and has i.i.d. sub-Gaussian entries η i satisfying ∥ η i ∥ ψ 2 = O (1) . Along similar lines as in (3.1)-(3.3), we find that η gives rise to an additional term 2 m ⟨ η , A ( ˆ x -T x ∗ ) ⟩ to the right-hand side of (3.1), which is bounded by

2 ∥ ˆ x -T x ∗ ∥ 2 · sup v ∈ ( K -ϵ ) ∗ 1 m ⟨ η , Av ⟩ , with the constraint set ( K -ϵ ) ∗ defined in (3.2). Thus, in (3.3), in addition to R u , in the noisy setting we need to bound the additional term

This can be done by the following lemma, which indicates that the sharp (uniform) rate in Theorem 1 can be retained in the presence of noise η .

Lemma15. (Bounding the additional term R ′ u ) . In the noisy setting described above, with probability at least 1 -C 1 exp( -Ω( k log TLr ϵ )) -C 2 exp( -Ω( m )) , we have R ′ u ≲ √ k log TLr ϵ m .

Proof. Conditioning on η , the randomness of a i 's gives 1 m ∑ m i =1 η i a i ∼ N (0 , ∥ η ∥ 2 2 m 2 I n ) , and so ∥ ( 1 m ∑ m i =1 η i a i ) ⊤ v 1 -( 1 m ∑ m i =1 η i a i ) ⊤ v 2 ∥ ψ 2 ≤ C 0 ∥ η ∥ 2 ∥ v 1 -v 2 ∥ 2 m holds for any v 1 , v 2 ∈ R n . Let ω ( · ) be the Gaussian width as defined in Lemma 5. Then, using the randomness of a i 's, Talagrand's comparison inequality [50, Exercise 8.6.5] yields that for any t ≥ 0 , we have

/a80 Next, we bound the Gaussian width ω (( K -ϵ ) ∗ ) . Recall that ( K -ϵ ) ∗ is defined in (3.2), and Lemma 6 bounds its metric entropy as H (( K -ϵ ) ∗ , η ) ≤ 2 k log 12 TLr ϵη . Thus, we can invoke Dudley's integral inequality [50, Theorem 8.1.3] to obtain

Now, we further let t = √ k log TLr ϵ in (F.1) to obtain that R ′ u ≲ ∥ η ∥ 2 √ k log TLr ϵ m holds with probability at least 1 -2 exp( -k log TLr ϵ ) . It remains to deal with the randomness of η and bound ∥ η ∥ 2 . Because η has i.i.d. entries with ∥ η i ∥ ψ 2 = O (1) , by [50, Theorem 3.1.1] we can obtain that ∥ η ∥ 2 ≤ C 3 √ m with probability at least 1 -2 exp( -c 3 m ) . Substituting this bound into R ′ u ≲ ∥ η ∥ 2 √ k log TLr ϵ m , the result follows.

To close this appendix, we briefly state how to adapt the proof of Theorem 1 to explicitly include the additive noise η . Specifically, the left-hand side of (3.1) and its uniform lower bound Ω( ∥ ˆ x -T x ∗ ∥ 2 2 ) remain unchanged, while the right-hand side of (3.1) is now bounded by 2 ∥ ˆ x -T x ∗ ∥ 2 · ( R u 1 + R u 2 + R ′ u ) (with 2 ∥ ˆ x -T x ∗ ∥ 2 R ′ u being the additional term); thus, combining the bound (C.12) on 2 ∥ ˆ x -T x ∗ ∥ 2 · ( R u 1 + R u 2 ) and Lemma 15, we establish a uniform upper bound

for the right-hand side of (3.1). Therefore, to ensure uniform recovery up to the ℓ 2 -norm accuracy of ϵ under the sub-Gaussian noise η , it suffices to have a sample complexity

Since the logarithmic factors L in (C.11) dominates log TLr ϵ , (F.2) indeed coincides with the sample complexity m ≳ ( A (1) g ∨ A (2) g ) 2 k L ϵ 2 in Theorem 1 under the mild condition of A (1) g ∨ A (2) g = Ω(1) .

G Experimental Results for the MNIST dataset

G.1 Details of the Settings

In this section, we conduct experiments on the MNIST dataset [28] to support our theoretical framework. We use various nonlinear measurement models, including 1-bit, dithered 1-bit, ReLU,

Original

CSGM

0/23456780/23456789012345628

q

0723490720/23956262052

345698

9

and uniformly quantized CS with dithering (UQD). We select 30 images from the MNIST testing set, ensuring that there are three images from each of the 10 classes for maximum variability. A single measurement matrix A is generated and used for all 30 test images. All the experiments are repeated for 10 random trials. All the experiments are run using Python 3.10.6 and PyTorch 2.0.0, with an NVIDIA RTX 3060 Laptop 6GB GPU.

We train a variational autoencoder (V AE) on the training set of the MNIST dataset, which has 60,000 images, each of size 784. The decoder of the VAE is a fully connected neural network with ReLU activations, with input dimension k = 20 and output dimension n = 784 , and two hidden layers with 500 neurons each. We train the VAE using the Adam optimizer with a mini-batch size of 100 and a learning rate of 0 . 001 .

Since our contributions are primarily theoretical, we only provide simple proof-of-concept experimental results. In particular, since (2.1) is intractable to solve exactly, to estimate the underlying signal, we choose to use the algorithm proposed in [2] (referred to as CSGM ) to approximate it. CSGM performs a gradient descent algorithm in the latent space in R k with random restarts. In addition, we compare with the Lasso program that is solved by the iterative shrinkage thresholding algorithm.

For CSGM , we follow the setting in [2] and perform 10 random restarts with 1000 gradient descent steps per restart and pick the reconstruction with the best measurement error.

G.2 Experimental Results for Noiseless 1-bit Measurements and Uniformly Quantized Measurements with Dithering

In this subsection, we present the numerical results for 1 -bit measurements and uniformly quantized measurements with dithering, while the results for dithered 1-bit measurements and the Lipschitz SIM where the nonlinear link function is ReLU are similarly provided in Appendix G.3. For 1 -bit measurements, since the underlying signal is assumed to be a unit vector and we aim to recover the direction of the signal, we use cosine similarity that is calculated as ˆ x T x ∗ / ( ∥ ˆ x ∥ 2 · ∥ x ∗ ∥ 2 ) with ˆ x being the estimated vector to measure the reconstruction performance. For uniformly quantized measurements with dithering, we use the relative ℓ 2 -norm distance between the underlying signal and the estimated vector, i.e., ∥ ˆ x -x ∗ ∥ / ∥ x ∗ ∥ 2 , to measure the reconstruction performance.

Since this paper is concerned with uniform recovery performance, in each trial, we record the worstcase reconstruction performance (i.e., the smallest cosine similarity or the largest relative error) over the 30 test images, and the worst-case cosine similarity or relative error is averaged over 10 trials.

Figures 2, 3, and 4 show that for noiseless 1 -bit measurements and uniformly quantized measurements with dithering with δ = 3 , the CSGM approach can produce reasonably accurate reconstruction for all the test images when the number of measurements m is as small as 150 and 100 respectively.

G.3 Experimental Results for ReLU and Dithered 1-bit Measurements

We present the experimental results for the ReLU link function and dithered 1 -bit measurements in Figures 5, 6, and 7. For dithered 1-bit measurements, we set λ = R √ log m with R > 0 being a tuning parameter. For the case of using the ReLU link function, similarly to noiseless 1-bit measurements, we calculate the cosine similarity to measure the reconstruction performance. For dithered 1-bit measurements, similarly to uniformly quantized measurements with dithering, we calculate the relative ℓ 2 -norm distance. We observe that for these two nonlinear measurement models

Original

CSGM

023456780/23567

69012345028

b

27234567290723956789002345678

q

50

75

100 125 150 175 200 225 250

m

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Cosine Similarity

CSGM

50

75

100 125 150 175 200 225 250

m

0.5

0.6

0.7

0.8

0.9

1.0

Relative Error

CSGM

0.0

2.5

5.0

7.5

10.0 12.5 15.0 17.5 20.0

0.5

0.6

0.7

0.8

0.9

1.0

Relative Error

CSGM

(a)

1

-bit with varying

m

(b) UQD with fixed

δ

= 3

(c) UQD with fixed

m

= 200

with a single realization of the random measurement ensemble, CSGM can also lead to reasonably good reconstruction for all the test images when the number of measurements is small compared to the ambient dimension.

H Experimental Results for the CelebA dataset

In this section, we present numerical results for the CelebA dataset [35], which contains more than 200,000 face images for celebrities with an ambient dimension of n = 12288 . We train a deep convolutional generative adversarial network (DCGAN) following the settings in https: //pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html . The latent dimension of the generator is k = 100 and the number of epochs for training is 20 . Since the experiments for CelebA are more time-consuming than those of MNIST, we select 20 images from the test set of CelebA and perform 5 random trials. Other settings are the same as those for the MNIST dataset.

Since we have observed from the numerical results for MNIST that the experiments for the ReLU link function and dithered 1-bit measurements are similar, we only present the results for noiseless 1-bit measurements and uniformly quantized observations with dithering.

From Figures 8 and 10, we observe that for noiseless 1-bit measurements with 1500 samples, a single measurement matrix A can lead to reasonably accurate reconstruction for all the 20 test images. In addition, from Figures 9 and 10, we observe that for uniformly quantized measurements with

Original

CSGM

0/23456780/23

4567

9012345028

q

0/239567240723956289012345678

b

Original

CSGM

07234567840723456769012345978

q

9567901

2

345998

b

50

75

100 125 150 175 200 225 250

m

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Cosine Similarity

CSGM

0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Cosine Similarity

CSGM

150 200 250 300 350 400 450 500 550

m

0.5

0.6

0.7

0.8

0.9

1.0

Relative Error

CSGM

(a) ReLU with fixed

σ

= 0

.

1

(b) ReLU with fixed

m

= 150

(c) Dithered 1-bit with fixed

R

= 5

dithering, a single realization of the measurement matrix and random dither is sufficient for the reasonably accurate recovery of the 20 test images when m = 1000 and δ = 20 .

CSGM Original

1888

CSGM Original

1

0

200

400

600

800 1000 1200 1400

m

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Cosine Similarity

CSGM

0

200

400

600

800 1000 1200 1400

m

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Relative Error

CSGM

0

50

100

150

200

250

300

0.5

0.6

0.7

0.8

0.9

1.0

1.1

1.2

Relative Error

CSGM

(a)

1

-bit with varying

m

(b) UQD with fixed

δ

= 20

(c) UQD with fixed

m

= 500

Peer reviews:
comments: SUMMARY:
In non-linear compressed sensing, one would like to recover x from a
series of observations y_i = f_i(a_ix); and in generative compressed
sensing, x is drawn from a generative model.  The contribution of this
paper is a framework for uniform recovery with general nonlinear
measurements and Lipschitz generative models; in particular, this
framework includes Lipschitz generative models and dithered 1-bit
measurements, which were not previously known to give uniform
recovery.

STRENGTHS:
The result is fairly general, handling dithered 1-bit or otherwise
discretized measurements, as well as noise in the measurements.  The
bound is pretty good, basically ideal except possibly for some terms
inside the log factor.

WEAKNESSES:
The first specific application the authors present for this framework
is:

 (1) uniform recovery of (2) Lipschitz generative models from (3)
 dithered 1-bit measurements

This is pretty specific, and if any one of the terms is relaxed
(nonuniform, a ReLU generator, or non-dithered) then prior work covers
it.  That is to say: the result is nice and general, but prior work
has pretty well covered the most interesting cases covered by this
result.

The writeup could be improved, repeatedly defining things in terms of
terms that are defined pages later.  For example, the main theorem
(Theorem 1) relies on script(L), which isn't defined until the
supplemental material; Assumption 3 uses section 3;

QUESTIONS:
* Can you give a corollary for handling Gaussian noise?

* What would you do for nonlinear sensing that's more nonlinear than discretization, e.g., sinusoidal or quadratic observations?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a unified framework for uniform signal recovery in nonlinear generative compressed sensing, in particular, 1-bit generative compressed sensing (GCS) and single-index models (SIM).   The authors obtain  uniform recovery guarantees for 1-bit  GCS, 1-bit GCS with dithering, Lipschitz-continuous SIM, and uniformly quantized GCS with dithering.  Experimental results are presented to corroborate the theoretical results.\n\nSTRENGTHS:\n1.  A unified framework for uniform signal recovery in nonlinear generative compressed sensing is proposed. \n2.  Uniform recovery guarantees are obtained for 1-bit  GCS, 1-bit GCS with dithering, Lipschitz-continuous SIM, and uniformly quantized GCS with dithering.\n\nWEAKNESSES:\n1. Both main statements and corresponding proofs build on the strong assumption that the target signal exactly belongs to the domain of generative models. Apparently, this is not the case for real-life compressed sensing, both theoretically and empirically. The key idea of generative compressed sensing is to recover an unknown target signal by leveraging the generative prior to capture its intrinsic structure. Nevertheless,  it does not assume that the target signal is exactly generated by the assumed generative models. However, the goal of this paper is to prove the uniform recovery of a signal that is exactly generated by a known generative model, which is apparently not the case of real compressed sensing, or is a different problem.  What if the target signal is not generated by the assumed generative model? \n\n\n2. Given that this paper assumes that the target signal is exactly generated by a known generative model, how can the results be applied to practical generative models. For example, given two algorithms, one using a VAE prior, the other using a GAN prior, how to compare the corresponding reconstruction results? \n\n\n3. Are the main results applicable to diffusion models? This paper assumes an L-Lipschitz continuous generative model and a radius r Ball in R^k. Recently, there are several recent studies on diffusion models for generalized linear inverse problems including 1-bit compressed sensing, for example:\n\n[R1] Meng, Xiangming, and Yoshiyuki Kabashima. \"Quantized Compressed Sensing with Score-Based Generative Models.\" ICLR2023\n[R2] Chung, Hyungjin, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. \"Diffusion posterior sampling for general noisy inverse problems.\" ICLR2023.\n\n[R1] discussed quantized compressed sensing with score-based generative models and [R2] introduces a unified diffusion posterior sampling method for general noisy linear and nonlinear problems. Can the main statements in this paper apply to the above latest studies with diffusion models? Please illustrate the main differences and explain why.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nUnreasonable assumptions and lack of discussions of some latest works.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper discusses a unified framework for uniform signal recovery in nonlinear generative compressed sensing. The authors utilized the use of generalized Lasso and Lipschitz approximation to allow for a lower sample size of measurements.

STRENGTHS:
In what follows the strengths of the paper are given:

1) The paper presents a framework for deriving uniform recovery guarantees for nonlinear generative compressed sensing where the observation model is nonlinear and possibly discontinuous or unknown.
2) Utilizing the generalized Lasso and Lipschitz approximation allowed for a lower sample size of $\tilde{O} \left( \frac{k}{\varepsilon^2} \right)$, which is smaller than the previously best bound of $\tilde{O}\left( \frac{k}{\varepsilon^4}\right)$.
3) The idea of drawing back from using a concentration inequality (Lemma 5) in the appendix and instead leveraging the use of metric entropy to derive a tighter upper bound on $\mathcal{R}_u$ is quite interesting.
4) Finally, the paper is mainly theoretical and the experimentation done in the supplementary is to show confirm the findings of the paper, which are extensively shown.

WEAKNESSES:
The paper is somewhat hard to follow:
1) Some notations are used before they are defined.
2) Some explanations around the mathematical parts are not properly stated, e.g., transitions.

QUESTIONS:
1) Please expand on the transitions between equations.
2) How did you ensure that the assumptions that were presented in the paper hold in your experiments?

LIMITATIONS:
The limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06