Stable and Interpretable Unrolled Dictionary Learning
Authors: Bahareh Tolooshams
Demba E. Ba
Venue: Accepted by TMLR ()

Abstract: The dictionary learning problem, representing data as a combination of a few atoms, has long stood as a popular method for learning representations in statistics and signal processing. The most popular dictionary learning algorithm alternates between sparse coding and dictionary update steps, and a rich literature has studied its theoretical convergence. The success of dictionary learning relies on access to a good initial estimate of the dictionary and the ability of the sparse coding step to provide an unbiased estimate of the code. The growing popularity of unrolled sparse coding networks has led to the empirical finding that backpropagation through such networks performs dictionary learning. We offer the theoretical analysis of these empirical results through PUDLE, a Provable Unrolled Dictionary LEarning method. We provide conditions on the network initialization and data distribution sufficient to recover and preserve the support of the latent code. Additionally, we address two challenges; first, the vanilla unrolled sparse coding computes a biased code estimate, and second, gradients during backpropagated learning can become unstable. We show approaches to reduce the bias of the code estimate in the forward pass, and that of the dictionary estimate in the backward pass. We propose strategies to resolve the learning instability by tuning network parameters and modifying the loss function. Overall, we highlight the impact of loss, unrolling, and backpropagation on convergence. We complement our findings through synthetic and image denoising experiments. Finally, we demonstrate PUDLE's interpretability, a driving factor in designing deep networks based on iterative optimizations, by building a mathematical relation between network weights, its output, and the training set.

Published in Transactions on Machine Learning Research (08/2022)
Stable and Interpretable Unrolled Dictionary Learning
Baharch Tolooshams
biolooshammsdsssss.aaaaaa edu dembo wwweeaeeeessssee.. edu
Demba Ba
School of Engineering and Applied Sciences
Horvard University
Reviewed on OpenReview: https: // openreview. formm? id- e3S0BL2R08
Abstract
The dictionary learning problem, representing data as a combination of few atoms, has long stood popular method for learning representations in statistics and signal processing. The most popular dictionary learning algorithm alternates between sparse coding and dictionary update steps, and rich literature has studied its thcorctical convergence. The success of dictionary learning relies on access to a "good" initial estimale of the dictionary and the ability of the sparse coding step to provide an unbiased estimate the code. The growing popularity of unrolled sparse coding networks has led to the empirical finding that backpropagation through such networks performs dictionary learning. We offer the theoretical analysis of these empirical results through PUDTE,, a Provable Unrolled Dictionary LFarning method. We provide conditions the network initialization and data distribution sufficient to recover and preserve the support of the latent code. Additionally, we address two challenges; first, the vanilla unrolled sparse coding computes a biased code estimate, and second, gradients during backpropagated learning can become unstable. We show approaches to reduce the bias of the code estimate the forward pass, and that of the dictionary estimate in the backward pass. We propose strategies to resolve the learning instability by tuning network parameters and modifying the loss function. Overall, highlight. the impact of loss, unrolling, and backpropagation convergence. We complement our findings through synthetic and image denoising experiments. Finally, demonstrate PUDLE's interpretability, driving factor in designing deep networks based on iterative optimizations, by building a mathematical relation between network weights, its output, and the training set.
1 Introduction
This paper' considers the dictionary learning problem, namely representing data E X R as linear combinations of few atoms from dictionary D E D C Rmxp, Given and D, the problem of recovering the sparse (few non-zero elements) coefficients z E R is referred to sparse coding, and can be solved through the lasso (Tibshirani, 1996) (also known as basis pursuit (Chen et al., 2001)):
(D) == min La(2, + h(2)
(1)
where (z, D) = 1 - Dz 2, and h(2) = A|22|11 Specifically, the problem aims to recover a dictionary D' that generates the data, i.e.
D
(2)
where z" is sparse. Olshausen and Field (Olshausen & Field, 1997) introduced (2) in computational neuroscience as model for how early layers of the visual cortex process natural images. Sparse coding has been widely studied and utilized in the statistics (Hastie et al., 2015) and signal processing communities (Flad, 2010). A few practical examples are denoising (Elad & Aharon, 2006), super-resolution (Yang et al., 2010), text processing (Jenatton et al., 2011), and classification (Mairal al., 2009b), where it enables the extraction
Source code is available https: thub. http://ggghhh.ccccccooooooaaaaae http.///cooooooooe//taeeeeeeeeee
unrolled-dl

---

Published in Transactions on Machine Learning Research (08/2022)
Encoder
Decoder
D"'
+
DDD_D
gac-ls
L Lz(Zy, D)
dec
Al|zz/h
Figure 1: Provable unrolled dictionary learning (PUDLE): Unrolled network architecture with dictionary D.
of sparse high-dimensional features representing data. Moreover, sparse modelling is ubiquitous in many other fields such as seismic signal processing (Nose-Fillu) et al., 2018), radar sensing for target detections (Bajwa et al., 2011), and astrophysics for image reconstruction from interferometric data (Akiyama et al., 2017). Furthermore, Cleary et al. (2017; 2021) use this model to learn a dictionary consisting of gene modules for efficient imaging transcriptomics.
Sparse coding has been utilized to construct neural architectures through approaches such as sparse energy- based models (Ranzato et al., 2007; 2008) or recurrent sparsifying encoders (Gregor & LeCun, 2010). The latter has initiated a growing literature on constructing interpretable deep networks based on an approach referred to as algorithm unrolling (Hershey et al., 2014; Monga et al., 2019). Deep unrolled neural networks have gained popularity as inference maps in recent years due to their computational efficiency and their performance in various domains such as image denoising (Simon & Elad, 2019; Tolooshams et al., 2021a; 2020), super-resolution (Wang al., 2015), medical imaging (Solomon et al., 2020), deblurring (Schuler et al., 2016; Li et al., 2020), radar sensing (Tolooshams et al., 2021b), and speech processing (Hershey al., 2014).
Prior to the advent of unrolled networks, gradient-based dictionary learning relied on analytic gradients com- puted from the lasso given the sparse code. With unrolled networks, automatic differentiation (Baydin et al., 2018), referred to as backpropagation (LeCun et al., 2012) in the reverse-mode, gained attention for parameter estimation (Tolooshams et al., 2018). The automatic gradient is obtained by backpropagation through the algorithm used estimate the code. Automatic differentiation in reverse and forward-mode (Franceschi et al., 2017) is used in other areas, e-g., hyperparameter selection (Feurer & Hutter, 2019), and in more relevant. context, in the seminal work of LISTA (Gregor & LeCun, 2010). Other works demonstrated empirically the convergence of fi-based dictionary learning by backpropagation through unrolled networks (Tolooshams et al., 2021a). Given finite computational power, Tolooshams et al. (2021a) convert sparse coding into an encoder by unrolling T itcrations of ISTA (Daubechics et al., 2004; Blumensath & Davies, 2008), and attach to it a lincar decoder for reconstructing. Uurolled networks obtained in this manner suffer from two important limitations.
First, the sparse coding step in the forward pass computes a biased estimate of the code. This results, in turn, in a biased estimate of the backward gradient and, hence, degradation of dictionary recovery performance. Second, as studied recently (Malézieux et al., 2022), inaccuracies in the early iterations of the unrolled network make backpropagation unstable. We address both of these shortcomings in this paper. Moreover, while Malézieux et al. (2022) analyze the gradient computed by backpropagation through unrolled sparse-coding networks, there is no known thcoretical analysis of how weight updates using this gradient impact the recovery of ground-truth code 2", nor of their convergence to a ground-truth dictionary D*
This paper proposes a Provable Unrolled Dictionary LEarning (PUDLE) (Figure 1). We aim to recover D* by training the network using backpropagation with learning rate of 17. Three different choices affect the gradient: the number of unrolled iterations, the loss, and whether one backpropagates through the decoder only or through both the encoder and decoder. We highlight the impact of such choices on the convergence of the training algorithm. Backpropagation through the decoder results in the analytic gradient gele using the code estimate z, The gradients ge--assoo and g. are computed by backpropagation through the autoencoder using the lasso and least-squares objectives, respectively (Algorithm 2). We compare the gradients with the classical pandienttbased alternating minimization algorithm for dictionary learning (Chatterji & Bartlett, 2017) (i.e., cycling bet ween sparse coding and dictionary update steps using the analytic gradient 9 (Algorithm 1)),
alternatin-mmiiiizzttii
2

---

Published in Transactions on Machine Learning Research (08/2022)
and provide a theoretical analysis of gradient-based recovery of the dictionary D We provide sufficient conditions under which the gradient computation, hence the learning, is stable. Additionally, we show how using the reconstruction loss with backpropagation not only does not suffer from backpropagated instability but also ameliorates the propagation of the forward pass bias into the gradient estimate from the backward pass. Finally, we demonstrate the interpretability of the unrolled network. Our contributions are:
Unrolled sparse coding Unlike prior work (Malézieux et al., 2022) that studies the ability of sparse coding to recover the solution of the lasso (1) given the current estimate of the dictionary (we call this local estimation), we study unrolled sparse coding for recovery of the true generating code in (2) (we call this global estimation). We provide sufficient conditions on the network and data distributions such that the forward pass recovers (Theorem 4.1) and preserves (Theorem 4.2) the correct code support. Assuming support identification, we show the linear convergence of the code estimated through the unrolled iterations to the solution of the lasso (Theorem 4.3). We provide an explicit code expression at unrolled layer and its error with respect the ground-truth code z"; we highlight the biased estimate of the code when the forward pass strictly solves lasso (Theorems 4.4 and 4.5). Moreover, in more general scenario, we show that the error in the code estimate is upper bounded by two terms, ie., one associated with the dictionary error and the other to the bias of the estimate of code amplitude, due to f--based optimization (Theorem 4.6). The latter highlights that vanilla lasso (f1-based) sparse coding computes a biased estimate of codes, and below we discuss strategies to either alleviate this bias in the forward pass or mitigate its propagation into the backward pass for dictionary learning.
Mitigation of coding bias propagation into dictionary learning We study gradient estimation for dictionary learning in PUDLE. We decompose the upper bound on the gradient errors compared to the gradient direction to recover D into terms involving the current dictionary error, the bias of the code estimate, and the lasso loss used to compute the gradient. We show that using only the reconstruction loss while backpropagating (i.e., ge results in the vanishing of the upper bound due to the usage of lasso loss. This means that given fixed X, gā ameliorates the propagation of the forward pass bias into the backward pass. Specifically, we show that gā is a better estimator of the direction to recover D* than gf and gt qe-lasso, Hence, weight updates using g se-ls converges to a closer neighbourhood of D (Theorem 4.10). In supervised image denoising task, we show that the advantage of ge goes beyond dictionary learning; g. ge results in better image denoising compared to g, Furthermore, our network outperforms the sparse coding scheme in NOODL, a state-of-the-arl online dictionary learning algorithm (Rambhatla el al., 2018) (Table 1). Moreover, we show that the bias in the estimate of D vanishes as X = >v (with 0 < P < 1) decays within the forward unrolled layers (Figure 16). This strategy, supported by Theorem 4.4, results in an unbiased estimate of the code x" and recovery of D* (Theorem 4.11).
Stability of unrolled learning Our approach to resolve the instability issue of backpropagation in unrolled networks is two-fold. First, we show that under proper dictionary initialization, the instability of the gradient gt -as-l computation, studied by Malézicux et al. (2022), as T increases is resolved. We give condition under which the code support is identified and recovered after one iteration and, hence, gradient computation stays stable. Second, in the absence of support identification in carly iterations, we propose to use the gradient ge- which resolves the stability issue introduced by lasso loss in the backward pass. We highlight this stability through image denoising training without gradient explosion (Figure 6).
Interpretable sparse codes and dictionary Prior work has discussed algorithm unrolling for designing interpretable deep architectures based on optimization models (Monga et al., 2019), or interprotability of sparse representations in dictionary learning models (Kim et al., 2010). However, there is no known work to mathematically characterize the interpretability of unrolled network architectures. In this regard, first, we construct a mathematical relation between Iearned weights (dictionary) at gradient convergence and the training data (Theorem 5.1). Second, we relate the inferred representation, cttpataattiooorccottrrtttii /reconstruction of test examples to the training data. We highlight several interpretable features of the unrolled dictionary learning network. Specifically, we perform analysis that provide insights into questions such as why am learning a particular feature in the dictionary? or from what part of the training set or an image I am learning that feature? (Figure 8). Moreover, we provide an explanation of the relation between the new test image denoised secoisdd/reconstaan /rcoonstructed through the network and the training datasct. The model provides insights on how braining images are used to reconstruct new lest image (Figure 9) or how the test image picks up treining images that have a similar representation to itself to reconstruct (Figure 10).
3

---

Published in Transactions on Machine Learning Research (08/2022)
2
Related Works
There is vast literature on the theoretical convergence of dictionary learning. Spiclman et al. (2012) proposed
a factorization method to recover the dictionary in the undercomplete setting (i.e., p S m). Barak et al. (2015)
proposed to solve dictionary learning via sum-of-squares semidefinite program. K-SVD (Aharon et al., 2006)
and MOD (Engan et al.; 1999) are popular greedy approaches. have been used extensively in theory and practice (Jain et al., 2013; Agarwal et al., 2014; Arora et al., 2014).
methods
Recent work has incorporated gradient-based updates into alternating minimization (Chatterji & Bartlett, 2017; Arora et al., 2015; Rambhatla et al., 2018). Chatterji & Bartlett (2017) provided a finite sample analysis and convergence guarantees when updating the dictionary using the analytic gradient. Arora et al. (2015) proposed neurally plausible sparse coding approaches with analytic gradients. Another work focused on online dictionary learning (Mairal et al., 2009a) with an unbiased gradient updates (Rambhatla et al., 2018). Arora et al. (2015) discussed methods to reduce the bias of dictionary estimate, and Rambhatla et al. (2018) showod how to reduce bias in code and dictionary estimates. A common feature in the above-mentioned work the use of analytic gradients, i.e., explicitly designing gradient updates independent of the sparse coding step and not utilizing automatic gradients with deep learning optimizers. A theoretical analysis of backpropagation for dictionary learning exists only for shallow autoencoders (Rangamani et al., 2018; Nguyen et al., 2019).
The theorctical analysis of unrolled neural networks has mainly analyzed the convergence speed of variants of LISTA (Gregor & LeCun, 2010), where the focus is on sparse coding (i.e., the encoder) not dictionary learning (Sprechmann et al., 2012; Xin et al., 2016; Moreau & Bruna, 2017: Giryes et al., 2018; Chen et al., 2018; Liu & Chen, 2019; Ablin al., 2019). Moreau & Bruna (2017) showed that upon successful factorization of the Gram matrix of the dictionary within layers, the network achieves accelerated convergence. Giryes et al. (2018) examined the tradeolfs between reconstruction accuracy and convergence speed of LISTA. Moreover, Chen et al. (2018) studied the learning dynamics of the weights and biases of unrolled- ISTTA and proved that it achioves lincar convergence. Follow-up works invostigated the dynamics of stop size in a recursive sparse coding encoder (Liu & Chen, 2019; Ablin et al., 2019). Ablin et al. (2019) minimized the lasso through backpropagation but still assumed the knowledge of the dictionary at the decoder.
Ablin et al. (2020) compared analytic and automatic gradient estimators of min-min optimizations with smooth and differentiable functions. Moreover, Malézieux et al. (2022) studied the stability of gradient approximation in the early regime of unrolling for dictionary learning. Unlike our work, where we evaluate the gradients for model recovery, Ablin et al. (2020) and Malézieux et al. (2022) studied the asymptotic gradient errors locally in each step of an alternating minimization ion and did not provide errors concerning z" or D
3 Preliminaries
Given n independent samples, dictionary learning aims to minimize the empirical risk, i.e.,
minDED (D) with RD) R 2 fa: (D)
(3)
where 100 Rr, (D) = Exex (D)) a.s. To prevent scaling ambiguity between the code z and dictionary
D, it is common to constrain the norm of the dictionary columns. Hence, we define the set of feasible solutions for the dictionary as D A {D E Rmxp s.t. Vj E (1, PP) || < 1j. We can project estimates of D onto the feasible set by performing D, + 1/max(| 1l.. either at every update or at the end of training. We assume certain properties on the data, specifically its domain (Assumption 3.1), energy (Assumption 3.2), code distribution (Assumption 3.3), and generating dictionary (Assumption 3.4).
Assumption 3.1 (Domain signals). X and D are both compact convez sets.
Assumption 3.2 (Bounded signals). 3 M > 0 s.t. ||0||| < M Vo E X.
Assumption 3.3 (Code distribution). The code z" is at most s-sporse with the support S* supp(z"). Each element in S* is chosen from the set [1, uniformly at random without replacement. p== P(i (*) = ( and Psj = p(i,j E S = (s² /y²). Given the support, 2/s is i.i.d, has symmetric probability distribution density function, Eiz are Gabusaass and lower bounded, i.e., for i E St |=يي| 2 Cmin where 0 < Crnin S 1.
= 0 and =( |5 = I. Moreover, the non-zero entries of the code
TT
4

---

Published in Transactions on Machine Learning Research (08/2022)
Assumption 3.4 (Generating dictionary). D* is j-incoherent (see Definition A.1) where μ (log (m)). D is unil-norm columns matris = 1), || 2| (V/p/m), and p (m))
To achieve model recovery using gradient descent, we assume an appropriate dictionary initialization, ie., Assumption 3.5 (Dictionary closeness). The initial dictionary D is (ōv, c to D' (see Definition A.2). The dictionary closeness at every update is denoted by || - Djlla S d; Vj. Furthermore, 6j = =0 l)
Arora et al. (2015) proposed dictionary initialization method offering (6, 2)-close to D for 8 = (1 /lg m). The method is based on pairwise reweighting of samples { from the generative model (2), and does not require access to D In addition, Rambhatla et al. (2018) utilize dictionary closeness assumplions and such dictionary initialization for their theoretical analysis. Moreover, Agarwal et al. (2017) proposed a clustering approach to find a close initial estimate of the dictionary.
Given the p-incoherence of D (Assumption 3.4) and 6--closeness of the dictionary, D is ju-incoherent, i.e.,
Lemma 3.1 (H--incoherent) D is pu-incoherent where μ = μ + 2/moi
The recurrent encoder and decoder, which perform the computations shown in Algorithm 2, use the loss L and proximal operator Po(()) _ sign(v) max([u] - b, 0) for the & norm h: IR + R. The encoder implements ISTA (Daubechies et al., 2004; Blumensath & Davies, 2008) with step size a, assumed y (D). With infinite encoder unrolling, the encoder's output is the solution to the lasso (1), following the optimality condition (Lemma A.3) where we denote [a(z, D) A (z, D) h(z). One immediate observation is that X 2|| || |D >> + {0} E arg min f=(z, D). We assume X < DT We specify in Theorem 4.1 and Theorem 4.2 the conditions on X at every encoder iteration to ensure support recovery and its preservation through the encoder. In case of a constant À across encoder iterations while using D' as the dictionary (i.e., sparse coding using & norm), the network recovers a biased code s* We denote this amplitude error in the code by 8+ - -| - 22 which is small and goes to zero with À decaying through the encoder.
be less than
In addition, we assume the solution to (1) is unique; sufficient conditions for uniqueness in the overcomplete case (i.e., p> m) are extensively studied in the literature (Wainwright, 2009; Candès & Plan, 2009; Tibshirani, 2013). Tibshirani (2013) discussed that the solution is unique with probability one if entrics of D are drawn from a contimuous probability distribution (Tibshirani, 2013) (Assumption 3.6). This assumption implies that DY Ds is full-rank. We argue that as long as the data x E X' are sampled from continuous distribution, this assumption holds for the entire learning process. The preservation of this property is guaranteed at all iterations of the alternating minimization proposed in (Agarwal et al., 2014). Moreover, this assumption has been previously considered in analyses of unrolled sparse coding networks (Ablin et al., 2019; Malézieux et al., 2022) and can be extended to basdd optimization problems (Tibshirani, 2013; Rossct et al., 2004).
Assumption 3.6 (Lasso uniqueness). The entries of the dictionary D are continuously distributed. Hence, the minimizer of (1) is unique, i.e., á min fz(z, D) with probability one.
Lemma 3.2 states the fixed-point property of the encoder recursion (Parikh & Boyd, 2014). Given the definitions for Lipschitz and Lipschilz differentiable functions, (Definitions A.3 and A.4), the loss and function h satisfy following Lipschitz properties.
+ + 2h(2). The Lemma 3.2 (Fixed-poin) property of lasso). Given Assumption 3.6, we have E 12, D) minimizer is a fixed-point of the mapping, i.e., 2 Pax(E a VV, D)) = (()) (Purikh & Boyd, 2014).
Lemma 3.3 (Lipschitz differentiable least squares). Given the loss is Lipschitz differentiable. Let L1 and Le denote the Lipschitz constants of the first derivatives Vi D) and Vzh.(z, D), Ln and L21 the Lipschitz constants of the second derivatives V Cz, D) and v2, (2, (2, D), all w.r.l. z. Let V (z,,(z, D) be Lin Lin-Lipschitz w.r.t D, and we denote the Lipschilz constant of Cz, D) and (2, ()) w.r.t to D by LIID and L21D, respectively.
Dz13, D, and Assumption 3.2,
Lemma 3.4 (Lipschitz proximal). Given h(z) = >||=||2, its prozimal operator has bounded sub- deriivavee,ee i.e., ||0P((> (=) la S Cproz-
5

---

Published in Transactions on Machine Learning Research (08/2022)
Algorithm 1: Classical alternating-mminmizattii
dictionary learning using lasso (1).
Initialize: Samples {0 J X; initial dictionary D
Repeat: / = 0, ...,, number of epochs
Sparse coding step: x²(1) arg min, Cai (z, ( + h(z), (for i [1,
Dictionary update: D(++1) = D ng(() where g()) A Ei= V2Lzr (2 (())
Algorithm 2: PUDLE: Provable unrolled dictionary learning framework.
Initialize: Samples {-') E X, initial dictionary D and Zo 0.
Repeat: 1= 0, 0,.,..., number of epochs
Forward pass: (for E nl)
=11) 4+1 (2 (0) ) = (a) aV (2, (2, D()) (repeat for 7)
<(0)
Encoder:
(6)
Decoder:
tl(() D(U2) itl)
2T
Backward pass:
D(++1) D
19T
(1)
where 9T !)
is either of
9T
(1)
dec
En
Li=1 22L (2T
(zT
{(1) D
9T
laslasso
P E V2L [
i(l)
(()) + JT
(00 ) 0h(z)))
(7)
91
(1) ae-ls
1
n. Ei= V2La (ZT
i[l)
D 1 V2Lzi (
i(!)
See Definition 4.1 for J
4 Unrolled Dictionary Learning
The gradients defined in PUDLE (Algorithm 2) can be compared against the local direction at each update of classical alternating-mininnn minimization (Algorithm 1). Assuming there are infinite samples, i.e.,
Best local direction : lim E V2Cai (2), D) EEEx (V2C(
(4)
where & arg mingeRo (z, D) + h(z). Additionally, to assess the estimators for model recovery, hence dictionary learning, we compare them against gradient pointing towards D", namely
Desired global gradient for D g" A limm- Er V2ha D) EZEX V2Ln(2", D)).
(5)
To see why the above is the desired direction, (z", D*) is critical point of the loss L which reaches zero for data following the model (2). Hence, to reach D* E arg min DED X D)), move towards the direction minimizing the loss in expectation. Specifically, using the gradient (2, D) = =
(D-D")- ">=="* T descent direction, we move from D toward D' modulo the code presence matrix x "T Given these directions, we analyze the error of the gradients gdec, ge-lasso and go assuming infinite samples. In local analysis, we compare the code and gradient estimates to the lasso optimization in each update of the alternating minimization. In global analysis, we evaluate the performance recovery of the ground-truth code s² and the dictionary D In this regard, we first study the forward pass.
4.1 Forward pass
We show convergence results the forward pass for and the Jacobian, i.c., Definition 4.1 (Code Jacobian). Given D, the Jacobian of z, defined as J, 2 Oz with adjoint J
3D
The forward pass analyses give upper bounds the error between 2; and & and the error between Jj and j as function of unrolled iterations f. We define J as following: considering the function z L D) h(z), 2(D) is its minimizer and f ae(D) gradient. estimation errors. Similar to (Chatterji & Bartlett, 2017), the error associated with geee depends
aD We will require these errors in Section 4.2, where we analyze the

---

Published in Transactions on Machine Learning Research (08/2022)
the code convergence. Unlike get,, the convergence of backpropagation with gradient estimates gi - and g relies on the convergence properties of the code and the Jacobian (Ablin et al., 2020). Forward-pass theories are based on studies by Gilbert (1992) on the convergence of variables and their derivatives in an iterative process governed by a smooth operator (Gilbert, 1992). Morcover, Hale et al. (2007) studied the convergence analysis of fixed point iterations for & regularized optimization problems (Hale el. al., 2007).
Support recovery and preservation We re-state a result from (Hale et al., 2007) on support selection.
Proposition 4.1 (Finite-iteration support selection). Given Assumplion 3.6, let arg min (z D) with S a supp(E). There erists B > 0 such that supp(zi) S, Vt > B.
This means the unrolled encoder identifies the support in finite iterations. Support recovery in finite iterations has been studied in the literature for LISTA (Chen et al., 2018), Step-LISTA (Ablin et al., 2019), and shallow autoencoders (Arora et al., 2015; Rangamani et al., 2018; Nguyen et al., 2019; Tolooshams et al., 2020). We show that under proper initialization of the dictionary, the encoder achieves linear convergence. Arora et al. (2015) discussed some appropriate initialization which is used by Rambhatla et al. (2018). Given initial closeness do, the encoder selects and recovers the correct signed support of the code with high probability in one iteration B = 1 (Theorem 4.1), and the iterations preserve the correct support (Theorem 4.2). In spite of slow convergence of ISTA Liang et al. (2014), support recovery after one iteration in unrolled networks is studied in the literature (Arora et al., 2015; Rambhatla et al., 2018; Chen et al., 2018; Nguyen et al., 2019).
Thcorem 4.1 (Forward pass support recovery). Given Assumptions 3.3 and 3.4, suppose D is ö1 0'(1/Vlog) close to D If 8 O'mm//llo
and μ O(log m), then with probabilily of al least rsupp-rec, the choice of Xo = Cmin/4 recovers the support of the code x" in one encoder iteration, i.e., C²
(1) sign (arro ()) = sign(z"), where Esupp-rec = 2pexp o-(6,) (-0
Theorem 4.2 (Forward pass support preservation). Given Assumptions 3.3 and 3.4, suppose D is 6j (1(loog))) close to D' If == 1===00(0 (0m(v/// m), μ Ollog m), and the regularizer and step size are chosen such that X
m and (0) S 11 22 22 (1-4 then with probability || . zella + a, = S2( (
#e vm
of at least 1 - (1)
Cuppp-pres, the support, recovered at the first iteration, is preserved through the encoder iterations. (!) We have a,- = O(V and csapp-pres = Caupp-rec + Cy 2p exp (8))
-iin + 2s exp (o(6))
The support preservation conditions on X and a introduce two insights. First, with an increase of t, the code error decrease, hence the lower bound on d Second, the decay of X as the encoder unrolls increases the upper bound on a. Hence, we suggest a decaying strategy in values of X as t increases.
The utilization of knowledge of the code error, as we do in Theorem 4.2, to set the proper threshold- ing/ ing/bias/reggiie brassarrguaarizttionn parameters (A) constitutes a fairly standard practice. Below we discuss similar results in the literature. For the preservation of correct signed-support (2018) provided proper thresholding value at every iteration as a function of the fn-norm of the code error with respect to ground-truth code; they additionally demonstrated an upper bound on the estimate of the code coefficients as function of dictionary closeness. Moreover, Nguyen et al. (2019) used information the range of ground-truth code to choose proper biases in their neural network to guarantce support recovery. Chen et al. (2018) similarly provided an upper bound on the bias of their unrolled sparse coding network at every layer as a function of fz-norm error between the code estimate at the layer and the ground-truth code. Overall, the error between a code estimate and the ground-truth code appearing in the lower bound on X can further simplified into terms related to termss such as the dictionary closeness d;, code sparsity. For example, Chatterji & Bartlett (2017), for their particular sparse coding algorithm, provided form upper bound as a function of terms such as code sparsity, data dimensionality, code range, and dictionary error.
a sparse coding network, Rambhatla et al.
Code convergence and error Given the support recovery and its preservation, the encoder convergence studied in (Malézieux et al., 2022) can achieve linear convergences after its first itcration. We re-state this result on the rate of convergence of the encoder in Theorem 4.3. We drop the superscript (I) to simplify the notation.
7

---

Published in Transactions on Machine Learning Research (08/2022)
Theorem 4.3 (Local forward pass codc convergence). Given the encoder 21+1 = Φ 2,, D), Assumption 3.6, Lemmas 3.2, A.1 and A.2, then 3 p < 1, B > s.t. || εl12 S (p²)) VI > B, where & is the unique minimizer of lasso (1). Furthermore, given Theorem 4-1 and Theorem 4.2, B 1.
10a
10
10-2
Theorem 4.3 shows that in PUDLE, 2i converges to 2 at a linear rate eventually after a certain number of unrolling (Figure 2). The local linear convergence of ISTA and FISTA (Beck & Teboulle, 2009) (with global rates of O(/s) and O()()) in the neighbourhood of a fixed-point is studied in (Tao et. al., 2016). The speed of convergence depends on when support selection happens (Proposition 4.1) (Bredies & Lorenz, 2008; Zhang et al., 2017b; Liang et al., 2014). We showed in Theorem 4.1 and Theorem 4.2 that under mild assumptions, the support is selected and recovered after one encoder iteration. In addition to local convergence, we focus on recovery of x² and show error on the unrolled code coefficients (!)) increases. In Theorem 4.4, we consider the case where X at layer t is set to according to Theorem 4.2; the bias decreases the code error decreases among the layers and dictionary updates. We provide an upper bound on the coefficients errors as function of code sparsity, dictionary error, and an unrolling error (fuaroll The unrolling error goes to zero for appropriately large t. Morcover, Theorem 4.5 studies the case (,
50 Number of unfolding [t]
100
Figure 2: Code convergence (Theo- rem 4.3). As the network unrolls, 2; converges to &, the solution of lasso.
=.,))) with respect to ground-truth z(ح) as
where the bias is fixed across the layers. In this scenario, we observe an additional term of Afxed in the upper bounds on the code coefficients crror; this term shows that the code error when we strictly perform f--orm based sparse coding does not go zero. We refer this error as an amplitude bias estimate error.
Theorem 4.4 (Global forward pass code error with variable Xt). Given Assumptions 3.3 and 3.4, suppose D is pt- in-incoheeent and 8t = O* (1/ p) close to D If = O² (0(mm/ (///llogm) logm,, p O(log m)) and the regularizer
and step size are chosen such that 11)) I -zell2 ay = 52( 2 and ail) < 21 (1- 4 Cmm
vm
then with probability of at least 1 - Coupp-pree, for j E supp(z"), the code coefficient error is
(1)
1 ((,)) z,,, S 00 00(5 - Dilla + E
Munrolll
(8)
and
2r,() 21 -1/00 (t
(9)
Muurrlll = 2(s 1)ta UL max; (0) 2/0) 6,,t-1 + +120/0) -2 da,t (1- + 2a where e.j
(1)
PL
)1
7m
vm
(699)1 = O(ay) with an = O(Voi), 800 = (D; - D. Dj) S 2
82 and (1)
(1)
(1)
papp--prss 1= Cuppp-re: + En
-aan 2pexp (o.)))) ) + 2s exp (o With appropriately large t, (2,00) -z,ح - Olv s [ID -Djll2).
(1)
Theorem 4.5 (Global forward pass code error with fixed Ne). Given Assumptions 3.3 and 3.4, suppose Dill is mu-incoherent and & (1/log)) close to D* If 8 O O'
P O(log and
the regularizer and step size are chosen such that x(0 = X ffxed vm
mt 1 - zollı + aa = 52 2 and
< 2\ -11- 1 Cmmm then with probability of at least 1- supp- for j E supp(z"), the code coefficient
(
(1)
195
error is
11n1 S (001s((/ $|1 $ 1 - Djlla + Et,
(nurool,ffeed + Rired
(10)
(.))
and
zr.(j) z1) - (1,5111) + (1)
Sr
(11)
where (1) (n)rools,, fined = (s 1)ta Ue {.)
max; (z 200 ffaee + 120,00
1 sfived = (1 - + a 22t
7m
(11) = O(ay + (iied) with an = ((v 800h = (D; - , < 2 and compp-pres == (1)
87
(1]
(supp-rec + 6,
2pexp( (--00)))
-Ca
+ 28 exp ( ). With appropriately large 1 (,)
أاااة = Ol /l/s)D) (ID - DDil2 1/2 + )Rrred).
8

---

Published in Transactions on Machine Learning Research (08/2022)
Aside from code estimation where the network parameters (e.g., regnlarization and step size) are finely tuned according to support recovery and preservation conditions (Theorems 4.1 and 4.2), we provide a general upper bound on the error between the converged code and >=; the bound can be decomposed into two terms of the dictionary error and the biased amplitude estimate of the code.
Theorem 4.6 (Global forward pass code error). Let & be the fized-point of the encoder with iterations *++1 = (z D). Given Assumption 3.6, Lemmas 3.2, A.1 and A.2, we have || S O -2---++))))
where 8 = -| 21112. 2 is the unique minimizer of lasso (1) given the dictionary D, 2 is the unique minimizer of lasso (1) given the dictionary D", and z" is the ground-truth code.
This general decomposition is to emphasize that aside from the current estimate of the dictionary, the code error is a function of the forward pass algorithm used to solve the sparse coding problem. Specifically, the upper bound states that at the best scenario where there is access to the generating dictionary D*, the forward pass solving lasso with fixed À still gives a biased amplitude estimate of z=. Overall, the assumptions to get this bound are mild; the bound is valid independent of successful support recovery or data distribution. With incorporation of data distribution and conditions stated in Theorem 4.1 and Theorem 4.2, the upper bound 8 can be replaced with terms involving X, and reaches at zero as À decays across forward iterations.
Jacobian convergence and error Following properties similar to those used in Theorem 4.3, and assuming Jr is bounded (Assumption 4.1), we show in Theorem 4.7 that, as the PUDLE unrolls, the code Jacobian Jj converges to J, the Jacobian of the solution of the lasso, The convergence of the Jacobian of proximal gradient descent is also studied in (Bertrand et al., 2021) for hyperparameter selection through implicit differentiation (Bengio, 2000), where the Jacobian is taken w.r.l to the hyperparameter as opposed to D.
Assumption 4.1 (Bounded Jacobian). The Jucobian is bounded, i.e., E M, 0, s.l. || S M, Vl.
Thcorem 4.7 (Local forward pass Jacobian convergence). Given the recursion z++1 = ((z,, D), and & the unique månimizer of lasso with Jacobian J, then p 1, 0 s.l. [| 111 O(p'') VI > B. Purthermore, given Theorem 4-1 and Theorem 4.2, B 1.
B
The forward pass code and Jacobian convergences after support selection is similar to the results from (Malézieux et al., 2022). The highlights of our finding are that the order of upper bound conver- gences can be achieved from the first iteration of the encoder. In other words, we specify, in Theorem 4.1 and Theorem 4.2, the dictionary and data conditions such that the support can be recovered with 1. This resolves the instability issue discussed by Malézieux et al. (2022) in computation of the gradient g. ge outside of the support. Finally, we show that the global Jacobian error is in the order of dictionary error.
Theorem 4.8 (Global forward pass Jacobian error). Let & be the fized-point of the encoder with iterations Et+1= = 2(zz; D). Given Assumption 3.6, Lemmas 3.2, A.1 and A.2, we have || JJ-J'lla S (-
where 83 = Jr- 1l,,,, and J, J and J are Jacobians corresponding to 2, 2" and z".
4.2 Backward pass
We show two results for local gradient ğ and global gradient g" convergence. The goal is not to provide a finite sample analysis but to emphasize the relative differences between the gradients in Algorithm 2. The impact of gradient error for parameter es- timation in the convex setting has been studied by Devolder et al. (2013) indicating that the convergence to the parameter's 'ss neigh- bourhood is dictated by the gradient error (Devolder et al., 2013; 2014). As dictionary learning is a bi-convex problem, findings of Devolder et al. (2013) hold as well for better estimation of the local dictionary at every step of alternating minimization. More- over, Arora et al. (2015), provided a detailed analysis of sparse coding and various gradient estimations for dictionary learning. showing that by computing a more accurate gradient at every step of the alternating minimization scheme, the dictionary estimates
10-1.
<50
dss
10
gi a0-la
50 Number of unfolding [t]
100
Figure 3: Convergence rate of gradients (Theorem 4.9).
9

---

Published in Transactions on Machine Learning Research (08/2022)
converge to closer neighbourhood of D Overall, the intuition is that the size of the gradient error dictates the size of the neighbourhood of the dictionary within which one can guarantee convergence. We argue that. the method with lower gradient error recovers the dictionary better.
Local gradient estimations We highlight the effect of finite unrolling on the gradient for parameter estimation (Ablin et al., 2020). Theorem 4.9 shows the convergence rate of gradients to 9, determining the similarity of PUDLE and Algorithm 1.
Theorem 4.9 (Local convergence of gradients). Given the forward poss converyence results (Theorems 4.9 and 4.7), 3 1, B O such that vI > B, the errors of gradients defined in Algorithm w.r.l 9 (4) salisfy
de 0|12 s O(p')
g ae pe-lasso gl12 - (ttp²t)
(12)
lgge - Ql1 S Oltp² + M,Avs).
Moreover, the order of upper bounds is light (see Lemma A.
First, upper bounds on the errors related to geee and gee-lasso go to zero as I increases. Hence, both gradients converge to 9. This means that asymptotically as t increases, training PUDLE with g and gin is oquivalent to classical alternatingminimizaaiin
(Algorithm 1). Second, as increases, gt ge has faster convergence than 9, dee, Lastly, gf biased estimator of 9 (Figure 3). The convergence results on the و| 2 is previously studied by Malézieux et al. (2022).
error ||9| -e-
Given the above convergence results, one may conclude that gieh gee should be used for dictionary recovery. However, we show next that for dictionary recovery, the gradient ghe-lasso, used by Malézieux et al. (2022), is indeed a biased estimator of the global gradient g* for recovery of D We decrease this bias by replacing go with go me-ls and show that gi results in a better recovery of D than g ga--lasso
Global gradient estimations Theorem 4.10 shows the global gradient errors w.r.t g" from (5). We omit the gradient glec as it asymptotically equivalent geelusso We study the errors in the limit to unrolling, i.., as cc. This determines which PUDLE gradients recover D' better (Devolder et al., 2013; 2014).
Theorem 4.10 (Global error of gradients). Given the convergence results from the forward pass, (Theorems 4.6 and 4.8), the errors of gradients defined in Algorithm 2 w.r.t global direction g" (defined in (5)) satisfy
11g x
lasso
g'll2 O D 11 + || D D*1 2 + 8+ + 883 M,M33)
(13)
gg le
21 < O() D D 11 + D - D' + + +89).
Several factors affect the order of upper bounds: the current estimate of the dictionary, code amplitude- bias error due to norm, and the usage of & norm in the loss used for backpropagation. To study the bias in the gradient computation, let consider the scenario where D = D We denote those gradients by superscript D'. If the gradients are not biased, then the upper bounds should goes to zero. The gradient errors are
ao- ge - 0 + 85). 1 1oee aa-lasso, D' - 12 < 0 + +82 + M) and gg
(14)
For gos D" the radius of the error ball is only a function of the amplitude error of the code estimated through lasso compare to the ground-truth code z². However, the error ball for the gradient, go se-lasso, D includes an additional term concerning the usage of lasso loss containing the regularization term x. This implies that the D' neighbourhood at which the gradient " is guaranteed to converge to is smaller than of the Iassos D' (Figure 4a). Implications of such gradient estimation are seen in dictionary learning where ga--ls recovers D' better (Figures 4b and 4c). In Figure 4b, the encoder unrolls for T = 25, hence the phenomenon of implicit acceleration is seen in faster and better dictionary learning performance of go gae-lasso than gdee, In Figure 4c where T' = 100, similar performance g and gas aae-lasso illustrates their asymptotic equivalence as - 20 (See Appendix for additional noisy dictionary learning experiments where the measurements x are corrupted with pero-mean Gaussian noise such that the Signal-to-NNise--aaan Ratio is approximately 12 SNR; in this setting, the aforementioned comparative analysis still holds.)
10

---

Published in Transactions on Machine Learning Research (08/2022)
10
109
10a
10-1
10
10
60
10
100 10-2
10-2
0
50
500 Number of epochs
250
250
500
Number of unfokding [4]
Number of epochss
(a) Convergence for g".
(b) Learning (T' 25).
(c) Learning (T = 100).
Figure 4: Results for PUDLE's global convergence (Theorem 4.10) and dictionary learning,
Towards unbiased estimation As long À is fixed within PUDLE, all defined gradients remain biased estimators of g"; due to the biased estimate of the code z" through l norm. This bias exists while dictionary lcarning is performed strictly using lasso through Algorithm 1. Given the conditions on the regularizer in Theorem 4.2 which we discussed in Section 4.1 and the derived upper bounds in Theorem 4.10, we suggest. the decaying of À across the encoder to reduce the gradient biases and improve dictionary learning. Next, we prove Theorem 4.11 that PUDLE converges to D if X docays across the layers t according to Thcorem 4.4. Moreover, Theorem 4.12 proves that if X stays fixed according Theorem 4.5, then PUDLE only guarantees to converge to close neighbourhood of the dictionary. In these analyses, we focus on g Furthermore, we show in Section 4.3 that by decaying Ả at each unrolled layer, the gradient bias vanishes, and recover D
Dictionary learning Given the network parameters set by Theorem 4.4, Theorem 4.11 proves that using 94x PUDLE recovers the dictionary; the dictionary error contracts at every update. Moreover, Theorem 4.12 proves that as long as À stays fixed across the unrolled layers, PUDLE guarantecs to converge to only D neighbourhood characterized by the regularization parameter A. These analyses requires for (L) to maintain a closeness to D" which we provide a proof for Lemma A.7. Hence, the dictionary closeness assumption (Assumption 3.5) stays valid.
Theorem 4.11 (Dictionary learning with variable Xt). Given Assumptions 3.3 and 3.4, suppose D is μi -iiiccoheeent and (8t, close to D with & (1/ logp). If s O(ym), μ Olog m)) learning rate is " O ((11-6//2)) and the regularizer and step size set according to Theorem 4.4, then for any dictionary (1) update using g/ with probability of at least 1 Caupp-pres:
ID
|
51 DD
(1-b))
-3
(15)
where (1) Cupp-ppres Cappp-rec + 7 epp
(1)
11
Cham ()-(;))
+ 2s cxp (TTT))
Theorem 4.12 (Dictionary learning with fixed Xe). Given Assumptions 3.3 and 3.4, suppose D is pu- incoherent and (o.. close to D' with 8; = O" (1//o///))) If s Olym), P Ollog m), learning rate is = 0 (( ) and the regularizer Xized and step size are set according to Theorem 4.5, then for any dictionary update using gos with probability of least Caupp-pres:
(I)
(D
-D71) < (1 +1)
Dj +
(1)
(16)
where& 6X e = s(1-3(0))
( 2p fred2 Eappp-pres = cou + Eg 2pexp C + 2s exp ( )
(1)
4.3 Experiments
Dictionary learning We focus on the performance of the best-performin gradient estimator ge and compare with NOODL (Rambhatla et al., 2018), a statc-of-the-art online dictionary learning algo- rithm, and SPORCO (Wohlberg, 2017), an alternating-miniizzaate minimization dictionary learning algorithm that uses lasso. NOODL, which uses iterative hard-thresholding (HT) for sparse coding and gradient update
11

---

Published in Transactions on Machine Learning Research (08/2022)
employing the code's sign, has linear convergence upon proper initialization (Rambhatla et al., 2018). We note that the results from go are not shown, as the gradient computation was unstable (Malézieux et al., 2022). We emphasize that our proposed gradient g2n0 does not suffer such instability. We train:
St se-la; À is fixed across iterations.
ae-ks, decay, : docays (i.e., A = Av", with 0 < V < 1) where
10-1
9t
decreases as training progresses.
se-la, HT (v) replaced with HT&(O) _ 10/26-
9t
DDD
||
With HT, the sparse coding step reduces that from NOODL. In this case, highlight the difference between the gradient update of our method (backpropagation) with NOODL. We focus on convergence, as y) across methods is not comparable.
10
NOODL SPORCO
Figure shows the convergence D 1000x 10 1500 to D when the code is 20-sparse (for other sparsity levels and details see Appendix C). A biased estimate of the code amplitudes results in convergence only to a neighbourhood of the die- tionary (Rambhatla et al., 2018). This is observed in the convergence of g. and SPORCO (final error is shown). The convergence of g. to a closer neighbourhood than SPORCO supports Theorem 4.10. Moreover, with decaying X, the code bias vanishes, hence ae-ls, decay le-ls, HT converges to D* similar NOODL. g.-.ls, and g:
10-7
250
500
750
Number of iterations
Figure 5: Dictionary convergences.
g.
26
26.0
24
22
g
dre
25.5
PSNR [dB)
PSNR (B
DC lasso
20
gr
ls
25.0
18
16
24.5
14
21.0
100
200
300
400
0
100
200
300
400
Number epochs
Number epuchs
(a) gi
ao-lasso training is not stable.
(b) pe-la improvement smoother than ge
Figure 6: Networks behaviour (test PSNR) during training as a function epochs.
Image denoising To further highlight the advantage ge over the other gradients, we compare them supervised task of image denoising. In addition to ge--l,, g. ge-llasso and gdec we consider gr ae-ls, HT where the proximal operator is replaced with HT. This is to compare with sparse coding scheme of NOODL. We do not compare against NOODL's dictionary update, as this computation for two-dimensional convolutions not straightforward. Prior works have shown that variants of PUDLE either rival or outperform state-of-the-art architectures (Simon & Elad, 2019; Tolooshams et al., 2020). Thus, we focus the gradients. We trained 432 and tested on 68 images from BSD (Martin et al., 2001). BSD dataset is popular training dataset for denoising (Zhang et al., 2017a; Simon & Elad, 2019; Mohan et al., 2019). We used a convolutional dictionary and corrupted images with zero-mean Gaussian noise of standard deviation of 25 (see Appendix for details). We initialized the dictionary filters by standard Normal distribution; this is to follow the norm in the deep learning literature and to demonstrate the practicality and usefulness of PUDLE in the absence of an initialization method. We evaluate the denoising performance of soft- thresholding using A and HT with b in peak coenaaltt-noise-re (PSNR).
comparative analysis of
12

---

Published in Transactions on Machine Learning Research (08/2022)
First, we highlight the stability of gt-h against Pi g Figure 6a shows the network dynamics in terms of test PSNR as function of epochs when 0.16 for gg gi gi--lasso, getc and b = 0.05 for g: ge-ls, HT We obscrved that gf g uses full backpropagation and stays stable. However, the training with g.o-lasso is not stable and unstable to perform demoising where the noisy PSNR is approximately 20 dB (Malézieux et al., 2022). Second, Figure 6b shows that compared to g,tt the backpropagated gradients result in a smoother improvement during training. Moreover, Table 1 shows that the advantage of g over geee is not limited to dictionary learning and is seen in denoising. We have excluded the results for g. g. from Table 1 as the network failed to denoise (see Figure Ga). Additionally, the superior performance of gt ge--ss compared to ae-ls, HT highlights the benefits of PUDLE (i.c., ba-based unrolling) against HT used in NOODL. gt.
Table 1: Denoising of BSD68. Reported numbers are mean (std) PSNR given three independent trials.
METHOD
PSNR [dB]
0.08
0.12
0.16
0.2
glee
24.21 (0.12)
24.93 (0.14)
25. 25.25 (0.06)
24.88 (0.00)
g ge0-ls
24.79 (0.03)
25.43 (0.03)
25.63 (0.04)
25.46 (0.05)
b
0.02
0.05
0.08
0.1
9t
an-ls, HT
22.92 (0.07) 25.26 (0.1)
24.76 (0.06)
23.94 (0.13)
5 Interpretable Sparse Codes and Dictionary
One motivation behind using algorithm unrolling to design deep architectures is interpretability (Monga et al., 2019); they argue that the designed networks are interpretable they capture domain knowledge via an optimization model. For example, Tolooshams et al. (2021a) takes advantage of the interpretability of learned weights in an unrolled dictionary learning network to solve spike sorting, an unsupervised source separation problem in computational neuroscience. Moreover, Kim et al. (2010) uses sparse coding to learn interpretable reprosentations of human motions. However, none of the cxisting methods in the literature provide interpretability results that open the black-box network through building a mathematical This section
relation between the learned dictionary, training data, and test representatoonrreonsttttee analyzes the interpretability of the unrolled sparse coding method in this context. We note that such mathematical relation and interpretability results also hold for dictionary learning. However, it is missing in the literature, irrespective of whether one uses an unrolling worw We provide the following theorem.
Theorem 5.1 (Interpretable unrolled network). Consider the dic- tionary learning optimization of the form minz,D 1| - 1IXIIIII 13 + A| |1 + 4/2| D where X = xe",.,"...,,"" E Rmxn and Z =
3
/
4
3
3
2
4
2
3
3
2
3
2
2
[=',>² ,,,,".,,,""" E RRx", Let 2 be the given converged sparse codes,
0
4
O
3
V
then stationary points of the problem w.r.l the network weights (diclio- nary) follows D XG" where we denote G = (Z Z + ll
0
O
6
3
S
The dictionary interpolates the training data Given Theo- rem 5.1, each learned atom interpolates the training data, i.e.,
D, - X w))
Σ (G-'w,)aat
(17)
k=1
where w = [ 27. 27] E RT is a vector containing the training code activity for dictionary atom j. Specifically, the importance of training image = in learning dictionary atom j is captured by the term (G-lw;)-- This proves the dictionary lives in the spans of the
Figure 7: Fraction of dictionary atoms learned from 1, 3, 4) MNIST.
13

---

Published in Transactions on Machine Learning Research (08/2022)
training set. Given the small number of atoms compared to the training size, (17) shows that the dictionary summarizes the training examples. We trained the network digits of (0, 1, 2, 4) MNIST (Figure 7 shows a fraction of the most used learned atoms). Figure 8 visualizes dictionary atoms along with training images with the highest contribution (green) and the lowest contribution (red). In addition, we used (17) on the partial training data to reconstruct learned atoms (shown as Estimate). Next, we interpret the relation between a new data to the training data using representer point selection, similar to (Yeh et al., 2018).
Leered 000031
o 07461
0,00000
0000000
Lermel
011222
D 01100
D,00000
00300
Leomed 0,00055
01018811
0,0000
0,0000
Leamed 0 006141 0.8017 0.00000 O 0.000C0
O O
3
2
4
4
4
2
3
3 33
3
1
2
Eatimatto
0,00000
0.00000
Extiatt
0.00000
Extmntt
0.05743
0,00000
3
4 4
4
2
4
3 3
3
2
(a) looking like atom.
(b) looking like atom.
(c) looking like atom.
(d) looking like atom.
Figure 8: Training image contributions to learning the dictionary.
0.08151
0.02106
0.00215
0.00000
0.08973
032237
000007
0.00003
Imnger
0.03057
0.03677
0.00003
0.00007
0.03473
0.03222
0.00008
0,000CO
4
/
/
2
4
3
3
3
L
4
O 0,00026
Rece
5.00016
O 0,028LL
0,033
0.00220
Jice
0.03220
0.03444
0.00000
0.00013
fiee
0.03036
0.03015
0.00047
0,00001
Boe
0.03148
0.03145
0.20026
/
3
4
3
3
3
2
2
4
4
4
2
Ballmatte
0.00862
0,03361
0.00246
0.00253
0.03509
0.00022
0.00033
0.00113
0.00120
Exiiass
0.03091
0.03087
0.00035
0,00036
[
L
/
4
O
3
3
3
(
3
4
4
4
[
(a) 0 test image.
(b) 1 test image.
(c) 3 test image.
(d) 4 test image.
Figure 9: Interpolation of training data reconstruct new image. Contribution of training images are shown in green (high contribution) and red (low contribution). 8 is normalized over the used examples.
Relation between new test image and training data For representation of data, observe that the reconstruction of a example x! is lincar combination of all the training examples, i.c.
e = - X = Σ x
(18)
where a denotes reconstruction, 2 is the code estimate, G-1TTT E R", and = Gid 21). We observe that the contribution of image k into the reconstruction of the test image is a function of 8% and the energy of By itself depends the whole training set, and G-1. (18) shows how each image is reconstructed interpolation of the training images. Figure 9 shows this results, where images with high (green) By contribution are similar to the test image and those with low (red) B% contribution are different. In addition, we can evaluate the overall quality of the reconstruction by looking into By in (18). For example, we observed that for lest MNIST, unnormalized B% corresponding to high contributing training images is above 1. However, for resized-CTFAR unnormalized B% of high contributing training images are often half or an order of magnitude lower than the MNIST case. This informs us of a bad representaation/rcconsst rewoostrroottiiinnnnnnn of CIFAR image by the trained network. From another perspective, we can write the new image as
x = DeC
De(E k=1
(19)
i.c., the contribution of cach training image for reconstruction is function of their code similarity to the new image and properties of the Gran matrix of training set code similarities. Specifically, the relation rules the contribution of transformed image & (i.e., (XG-1))) into reconstruction the test image as a function of its code similarity (** 2²). In other words, (19) shows that training images with the highest code similarity the representation of the new image have the highest contribution to its reconstruction. This interpretation is retonstraaed in Figure 10. The training images with the highest code similarity (green) and the lowest.
14

---

Published in Transactions on Machine Learning Research (08/2022)
999961
59A03
20010
20913
Insgo
097380
0.95077
0,20100
2,2000
2
2
2
4
I-digt
ddigt
Odigh
2-digh
Boo
0.99673
0,88449
0,20100
022225
Res
0.08555
0.38611
0,24417
0,20055
/
/
/
4
2
2
2
Stadacdo Sealany Sankaty Sardaste Saalett
Stalastt Stadade Sandante Sandletty
Saslanty
(a) Digit 1 test image.
(b) Digit test image.
Figure 10: Contribution of images with code similarity into reconstruction of a new test image along with the histograms of the similarity of the lest code to training codes from each class.
similarity (red) are shown. In addition, the figure demonstrates the histogram of the code similarity between the test image and the training set, grouped by their class digit. For example, for digit I test image, its code similarity to train images from class are bimodal. This corresponds to 1 digits that are lilted to the left (low similarity) and right (high similarity). Moreover, for digit 2 test image, we observe that the histogram of images corresponding digit 2 are shifted the most to the right (highost similarity) than the other classes.
6 Conclusions
This paper studied dictionary learning and analyzed the dynamics of unrolled sparse coding networks through a provable unrolled dictionary learning (PUDLE) framework. First, we provided a theoretical analysis of the forward pass for code recovery and dictionary learning. We discussed the bias introduced by f-baaed sparse coding in the forward pass, and how this affects the dictionary estimate in the backward pass. Second, we showed strategies to mitigate the propagation of this code bias into the backward pass; this is achieved by modification of the training loss function. We demonstrated that this bias could be further reduced and eliminated by decaying the regularization parameter within the unrolled layers. Additionally, we provided sufficient conditions on the data distribution and network to guarantee stability of backpropagated gradient computations. In the absence of such conditions, we proposed modification to the loss function that resolves the gradient explosion and allows stable learning. In an image denoising task, we showed PUDLE outperforms the NOODL sparse coding scheme (Rambhatla et al., 2018). Motivated by interpretability as a popular feature for unrolled networks, we derived a mathematical relation between the network weights (dictionary) and the training set. We proved that the network weights live in the span of the training set, and constructed a relation between predictions of new input examples and the training set. The latter allows the user to extract. images from the training set that are similar ddisimi/ssiiillar to the input image in representaton//reonsttuee
Acknowledgments
Demba Ba and Bahareh Tolooshams acknowledge the partial support of the ARO under grant number W911NF-16-1-0388 This is part of a collaboration between US DOD, UK MOD and UK Engineering and Physical Research Council (EPSRC) under the Multidisciplinary University Research Initiative. The authors would also like to thank the reviewers for their feedback, led the improvement of the manuscript.
References
Pierre Ablin, Thomas Morcau, Mathurin Massias, and Alexandre Gramfort. Learning step sizes for unfolded sparse coding, In Proceedings of Advances in Newral Information Processing Systems, volune 32, pp. 1 11, 2019.
Pierre Ablin, Gabriel Peyré, and Thomas Moreau. Super-efficiency of automatic differentiation for functions defined as a minimum. In Proceedings of International Conference on Machine Learning, pp. 32 41. PMLR, 2020.
Alekh Agarwal, Animashree Anandkumar, Prateek Jain, Praneeth Netrapalli, and Rashish Tandon. Learning sparsely used overcomplete dictionarics. In Maria Florina Balcan, Vitaly Feldman, and Csaba Szepesvári (eds.), Proc the 27th Conference on Learning Theory, volume 35 of Proceedings of Machine Learning Research, pp. 123 137, Barcelona, Spain, 13 15 Jun 2014. PMLR.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Alekh Agarwal, Animashree Anandkumar, and Pranceth Netrapalli. A clustering approach to learning sparsely used overcomplete dictionaries. TEEE Transactions on Information Theory, 63(1):575-555) 592, 2017. doi: 2110///T....11.1114414.
M. Aharon, M. Elad, and A. Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11):4311 4322, 2006.
Kazunori Akiyama, Kazuki Kuramochi, Shiro Ikoda, Vincent L Fish, Fumic Tazaki, Marcki Honma, Sheperd S Doeleman, Avery E Broderick, Jason Dexter, Monika Mošeibrodzka, et al. Imaging the schwarzschild- radius-scale structure of m87 with the event horizon telescope using sparse modeling. The Astrophysical Journal, 83)(1):1, 2017.
Sanjeev Arora, Rong Ge, and Ankur Moitra. New algorithms for learning incoherent and overcomplete dictionaries. In Maria Florina Balcan, Vitaly Feldman, and Csaba Szepesvári (eds.), Proceedings of the 27th Conference on Learning Theory, volume 35 of Proceedings of Machine Learning Research, pp. 779-8066 Barcelona, Spain, 13-15 Jun 2014. PMLR.
Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. In Peter Grünwald, Elad Hazan, and Satyen Kale (eds.), Proceedings of Conference Learning Theory, volume 40 of Proceedings of Machine Learning Research, pp. 113 149, Paris, France, 03 06 Jul 2015. PMLR.
Hedy Attouch and Jérôme Bolte. On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Mothematical Programming, 116(1):5 16, 2009.
Waheed U. Bajwa, Kfrr Gedalyahu, and Yonina C. Eldar. Identification of paramctric underspread lincar systems and super-resolution radar. IEEE Transactions on Signal Processing, 59(6):2548 2561, 2011.
Boaz Barak, Jonathan A. Kelner, and David Steurer. Dictionary learning and tensor decomposition via the sum-of-squares method. In Proceedings of Ansuual ACM Symposium on Theory of Compuling, STOC '15, pp. 143-151, 2015.
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic diffcrentiation in machine lcarning: survey. Journal of machine learning research, 18, 2018.
Amir Bock and Mare Teboulle. A fast iterative shrinkage-threshoddigg algorithm for linear inverse problems. SIAM journal imaging sciences, 2(1):113-20) 2022 2009.
Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12( 1900, 2000.
Quentin Bertrand, Quentin Klopfenstein, Mathurin Massias, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, and Joseph Salmon. Implicit differentiation for fast hyperparameter selection in non-smooth convex learning. arXiv:2105.014 2021.
Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fowrier analysis and Applications, 14(5-6):629 654, 2008.
Kristian Bredies and Dirk A Lorenz. Linear convergence of iterative soft-thresholding. Journal of Fourier Analysis and Applications, 14(5-6):813-88)) 2008.
Emmanuel Candès and Yaniv Plan. Near-ideal model selection by & minimization. The Annals of Statistics, 37(5A):2144 2177, 2009.
Niladri S Chatterji and Peter L Bartlett. Alternating minimization for dictionary learning: Local convergence guarantees. arXiv: 1116633,,, pp. 1 26, 2017.
Scott Shaobing Chen, David Donoho, and Michael Saunders. Atomic decomposition by basis pursuit. STAM revieu, 43(1):122-158) 2001.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ista and its practical weights and thresholds. In Proceedings of Advances in Neural Information Processing Systems, volume 31, pp. 1 -1, 2018.
Brian Cleary, Le Cong, Anthea Cheung, Fric S. Lander, and Aviv Regev. Efficient generation of transcriptomic profiles by random composite measurements. Cell, 171(6))1424-143..c11 2017. ISSN 0092-8674.
Brian Cleary, Brooke Simonton, Jon Bezney, Evan Murray, Shahul Alam, Anubhav Sinha, Ehsan Habibi, Jamie Marshall, Eric S Lander, Fei Chen, al. Compressed sensing for highly efficient imaging transcriptomics. Nalure Biolechnology, pp- 1 1-7, 2021.
I. Daubechies, M. Defrise, and C. De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics, 57(11):1413- 1457, 2004.
Olivier Devolder, François Glineur, Yurii Nesterov, et al. First-order methods with inexact oracle: the strggyy convex case. Technical report, UniversitA@ catholique de Louvain, Center for Operations Rescarch and 2013.
Olivier Devolder, François Glincur, and Yurii Nestcrov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 146(1):37-75, 2014.
Michael Elad. Spurse and redundant representations: from theory to applications in signal and image processing. Springer Science & Business Media, 2010.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15(12):3736-3777 3745, 2006.
K. Engan, S.O. Aase, and J. Hakon Husoy. Method of optimal directions for frame design. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 5, pp. 444--4446 vol.5, 1999.
Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated Machine Learning, pp. 3 33. Springer, Cham, 2019.
Luca Franceschi, Michole Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient- based hyperparameter optimization. In Proceedings of International Conference on Machine Learning, pp. 1165 1173, 2017.
Jean Charles Gilbert. Automatic differentiation and iterative processes. Optimization methods and software, (1):13321,, 1992.
Raja Giryes, Yonina C. Eldar, Alex M. Bronstein, and Guillermo Sapiro. Tradeoffs between convergence speed and reconstruction accuracy in inverse problems. IEEE Transactions on Signal Processing, 66(7): 1676-1690, 2018.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of international conference international conference machine learning, pp. 399 0999-06,,4 2010.
Elaine Hale, Wotao Yin, and Yin Zhang. A fixed-point continnation method for 11-regularized minimization with applications to compressod sensing. CAAM TR07-07, Rice University, 43:44, 2007.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sporsity; the lasso and generalizations. CRC press, 2015.
John R. Hershey, Jonathan Le Roux, and Felix Weninger. Deep unfolding: Model-based inspiration of novel deep architectures. arXic:1409.2577, pp. 1 27, 2014.
Prateck Jain, Pranccth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of Annual ACM Symposium on Theory of Computing, pp. 665-674, 2013.
ISBN 4781440030090.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, and Francis Bach. Proximal methods for hierarchical sparse coding. The Journal of Machine Learning Research, 12:2297-2338 2011.
Tachwan Kim, Gregory Shakhnarovich, and Raquel Urtasun. Sparse coding for learning interpretable spatio-temporal primitives. In Lafferty, C. Williams, Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Proceedings of Advances Neural Information Processing Systems, volume 23, 2010.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 2014.
Yann A LoCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Newral networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Yuelong Li, Mohammad Tofighi, Junyi Geng, Vishal Monga, and Yonina C. Eldar. Efficient and interpretable deep blind image deblurring via algorithm unrolling. IEEE Transactions Computational Imaging, 6: 666-681, 2020.
Jingwei Liang, Jalal Fadili, and Gabriel Peyré. Local lincar convergence of forward-backer backward under partial smoothnees. Proceedings of Advances in neural information processing systems, 27, 2014.
Jialin Liu and Xiaohan Chen. Alista: Analytic weights are good as learned weights in lista. In Proceedings of International Conference on Learning Representations, 2019.
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. Proceedings of Annual International Conference on Machine Learning, pp. 689 696, 2009a.
Julien Mairal, Jean Ponce, Guillermo Sapiro, Andrew Zisscrman, and Francis Bach. Supervised dictionary learning. In Proceedings of Advances in Neural Information Processing Systems, volume 21, pp. 1-8, 2009b.
Benoîl Malézieux, Thomas Moreau, and Matthieu Kowalski. Understanding approximate and unrolled dictio- nary learning for pattern recovery. In Proceedings of International Conference on Learning Representations, 2022.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithus and measuring ecological statistics. In Proceedings of IEEE International Conference on Computer Vision, volume 2, pp. 416-423, 2001.
Graandd Robust and interpretable Sreyas Mohan, Zahra Kadkhodaie, Eero P Simoncelli, and Carlos Fernandez- Granda. blind image denoising via bias-free convolutional neural networks. In Proceedings of International Conference on Learning Representations, 2019.
Vishal Monga, Yuclong Li, and Yonina Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. arXio:1912.1053: 10557, pp. 1-27, 2019.
Thomas Moreau and Joan Bruna. Understanding trainable sparse coding via matrix factorization. In Proceedings of 5th International Conference on Learning Representations, pp. 1 13, 2017.
Thanh V Nguyen, Raymond KW Wong, and Chinmay Hegde. On the dynamics of gradient descent for autocncoders. In Proceedings of International Conference on Artificial Intelligence and Statistics, pp. 2858 2867. PMLR, 2019.
Kenji Noso-Filho, Andre Kazuo Takahata, Renato Lopes, and Joao Marcos Travassos Romano. Improving sparse multichannel blind deconvolution with correlated seismic data: Foundations and further results. IEFE Signal Processing Magazine, (522):11-50, 2018.
Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, (1123)333113333 1997.
Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in optimization, 1(3):127-233) 2014.
18

---

Published in Transactions on Machine Learning Research (08/2022)
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Sirisha Rambhatla, Xingguo Li, and Jarvis Haupt. Noodl: Provable online dictionary learning and sparse
coding. In Proceedings of International Conference on Learning Representations, pp. 1-11, 2018.
Akshay Rangamani, Anirbit Mukherjee, Amitabh Basu, Ashish Arora, Tejaswini Ganapathi, Sang Chin, and Trac D. Tran. Sparse coding and autocncoders. In Proceedings of IEEE International Symposium on Information Theory (ISIT), pp. 36 36-40, 2018.
Mare aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann Cun. Efficient learning of sparse representations with energy-based model. In Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007.
Marc aurelio Ranzato, Y-lan Boureau, and Yann Cun. Sparse feature learning for deep belief networks. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Proceedings of Advances in Neural Information Processing Systems, volume 20, 2008.
Saharon Rosset, Ji Zhu, and Trevor Hastie. Boosting as regularized path to a maximum margin classifier. The Journal of Machine Learning Research, 1-941-973 2004.
Christian J. Schuler, Michael Hirsch, Stefan Harmeling, and Bernhard Schölkopf. Learning to deblur. IEFF Transactions on Pattern Analysis and Machine Intelligence, 38(7):1439 1451, 2016.
Dror Simon and Michael Elad. Rethinking the csc model for natural images. In Proceedings of Advances in Neural Information Processing Systems, volume 32, pp. 1-11, 2019.
Oren Solomon, Regev Cohen, Yi Zhang, Yi Yang, Qiong He, Jianwen Luo, Ruud G. van Sloun, and Yonina C. Eldar. Deep unfolded robust pea with application clutter suppression in ultrasound. IEEE Transactions Medical Imaging, 39(4):1051-1063, 2020.
Daniel A. Spielman, Huan Wang, and John Wright. Exact recovery of sparsely-used dictionaries, In Proceedings of Annual Conference on Learning Theory, volume 23 of PMRL, pp. 37.1-37.18, 2012.
Pablo Sprechmann, Alex Bronstein, and Guillermo Sapiro. Learning efficient structured sparse models. In Proceedings of International Coference International Conference on Machine Learning, pp. 219-2201 2012.
Shaozhe Tao, Daniel Boley, and Shuzhong Zhang. Local linear convergence of ista and fista on the lasso problem. SIAM Journal on Optimization, 26(1):313-336) 2016.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1):267-288) 1996. ISSN 00359246.
Ryan J. Tibshirani. The lasso problem and uniqueness. Electronic Journal of Statistics, 7(none):1456 1490, 2013.
Bahareh Tolooshams, Sourav Dey, and Demba Ba. Scalable convolutional dictionary learning with constrained recurrent sparse auto-encoders. In 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6, 2018.
Bahareh Tolooshams, Andrew Song, Simona Temereanca, and Demba Ba. Convolutional dictionary learning based auto-encoders for natural exponential-family distributions. In Hal Daumé III and Aarti Singh (eds.), ) Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9493-9503. PMLR, 13-18 Jul 2020.
Baharch Tolooshams, Sourav Dey, and Demba. Ba. Deep residual autoencoders for expectation maximization- inspired dictionary learning. IEEE Transactions on Neural Networks and Learning Systems, 22(6):2415-2 24299, 2021a.
19

---

Published in Transactions on Machine Learning Research (08/2022)
Baharch Tolooshams, Satish Mulleti, Demba Ba, and Yonina C Eldar. Unfolding neural networks for compressive multichannel blind deconvolution. In JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2890 2894. LEEE, 2021b.
Martin J Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using & f1-constrained quadratic programming (lasso). IEEE transactions on information theory, 55(5):2183 2202, 2009.
Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, and Thomas Huang. Deep networks for image super- resolution with sparse prior. In Proceedings of IEEE International Conference on Computer Vision, pp.
300-378,22 2015.
Brendt Wohlberg, Sporco: A python package for standard and convolutional sparse representations. In Proceedings of the 15th Python in Science Conference, Austin, TX, USA, pp. 1-8, 2017.
Bo Xin, Yizhou Wang, Wen Gao, David Wipf, and Baoyuan Wang. Maximal sparsity with deep networks? In Proceedings of Advances in Neural Information Processing Systems, volume 29, pp. 1-9, 2016.
Jianchao Yang, John Wright, Thomas S. Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, (9(11):28112277) 2010.
Chih-Kuan Yeh, Joon Sik Kim, Ian EH Yen, and Pradeep Ravikumar. Representer point selection for explaining deep neural networks. In Proceedings of the 32nd International Conference on Neural Informalion Processing Systems, pp. 9311 9321, 2018.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cun for image denoising, IEEE transactions on image processing, 26(7):3142 3155, 2017a.
Lufang Zhang, Yaohua Hu, Chong Li, and Jen-Chih Yao. A new linear convergence result for the iterative soft thresholding algorithm. Oplimizalion, 66(7):1177 1189, 2017b.
A Appendix proofs
A.1 Notation
Bold-lower-case and upper-case letters refer to vectors d and matrices D. We use d to denote the jth element of the vector d, and Dj the jth column of the matrix D. We denote the code estimate unrolled layer t by 22. À 0 is the regularization (sparsity-enforeing) parameter. m is the maximum singular value of D. When taking the derivatives or norms w.r.t the matrix D, we assume that D is vectorized. V Cz, D) and V2Ca (z, D) are the first derivatives of the loss w.r.t z and D, respectively. Vi Cz, (z, is the second derivative of the loss w.r.l z. V Ln, D) is the derivative of Vn D) w.r.t D. The support of supp(z) & [j: z()) # 0}.
A.2 Basic definitions and Lemmas
We list four definitions used throughout the paper below.
Definition A.1 (u-incoherencee. D is p-incoherent, i.e., for every pair (i,j) of columns, I(D,, S Vm. Definition A.2 ((ō, clossnnsss.. Dictionary D is 8-close to D", i.e., there is permutation and sign flip operator such that Vi (((()) D;ll2 5. Additionally, |D D' D2 D1Il2
Definition A.3 (Lipschitz function). A function f: T RP L-psschiil w.r.l norm | if L 0 s.t. (If(a) f(6)] < Ll(a- bll Va, b "
Definition A.4 (Lipschitz differentiable function). A tuiic differentiable function f: m + R L-Lipschitz differentiable w.r.t norm iff 3 L > 0 s.t. [[V s L Va E m
Definition A.5 (Strong convexity). twice differentiable function f: R. - R is strongly conver if A p>>0 0 s. (a) ž nI.
20

---

Published in Transactions on Machine Learning Research (08/2022)
Definition A.6 (Norm of subgradient). For norms involving subgradents, define [[0h(z)]
max) ||=||-
In the proof of the theorems, we use the strong convexity of the reconstruction loss after support selection and the bounded property of the Lipschitz mapping stated below.
Lemma A.1 (Strong convexity of reconstruction loss). Given the support selection (Proposition 4.1), D Ds is full-rank. Thus, Vt B, (2,, D) = L (21,s, Ds) is strongly conver (Definition A.5) in 2. Lemma A.2 (Lipschitz mapping). Given the recursion Zt-1 $(z)) Pas(ze a aViLa (2,, D)), from Lemma A.1, there exist B such that loss L Loa D) is j-strongly conver 1I > B. Hence, us- ing Lemma 3.4,
(2, D)ll2 II(I - Co (1)0Pxxill)
(20)
where p ê Cpros(1 - ap) <1 1.
One key term, used in the proofs, is that 0 E (2, D) + ah(2) which is followed by the lasso optimality, i..,
Lemma A.3 (Lasso optimality). Lasso Karush-Kuui Tucker (KKT) optimality conditions
(sign(2())
2 +
& E arg min (z, D) + D= D2) e A0I and 0] ZERP
Vj (1,2..... .P). if f = 0
1-1,1]
(21)
A.3 Forward pass proof details
Given the p-incoherence of D*, and current dictionary closeness of 51, re-stale Lemma 3.1 and proof it below. It shows that the current dictionary is pr-close to D
Lemma 3.1 (p=-incoherent). ( is pı-incoherent where μr + 2/mo
Proof.
(D(", D. = DJ) (D; - (D Dj) (D.") D; - D.
(22)
(D D / S u/vm + | D. | Dj 2 |D || Dj D S u/Vm vm 28:
We ro-state and proof the forward pass support recovery (Thcorem 4.1). This shows that given proper initialization and under mild conditions, the support of the true code 2* is recovered with high probability in one iteration of the encoder.
Theorem 4.1 (Forward pass support recovery). Given Assumptions 3.3 and 3.4, suppose D is òj O'(1/ /logp) close to D Tf = coupp-rec 2pexp (- (
m), and = Ollog m), then with probability of at least rupp-rrec: the choice of An = Cmin/4 /4 recovers the support of the code x* in one encoder iteration, i.e., sign(S.... (aD sign(2"), where (1)
(1)
C2
Proof. The code estimate after one iteration is 21 = Pax (a x) = (DT ( D D(T D'=1|- \a)). We focus on the positive entries. The analysis for negative entries is similar. Writting the relation for i-th entry,
z1,() = sign ( =) Σ (D!
D)}= -Ao)
jeS.
(23)
Re Di) +
Σ (D.) (P).) Ao))
JES+N(S)
We focus on the term inside ReLU and discard a, shared by all terms. We shows that under proper choice of An (D.") D))2%) is greater than Ag and v; Ejesnro (D()) Di)= is small with respect to Ao hence
21

---

Published in Transactions on Machine Learning Research (08/2022)
getting cancelled by ReLU. The small value of Vi, compared to (D ziy results in (ign(D()) be equal to the sign(Dilve D²z") which is equal to the sign of z*.
Given the current dictionary distance follows
| Dill2 I 61, we can find a lower bound on (D.") 2))= as
(D DT) - 1 2 2
+ ! 1 1 - Dj B = 1 - 2 IDD - 1
2
(24)
(D)
(1) D 2 6/
Hence, for i E 5"
KD D()= 2 (1 (1-88/2) 02
(25)
otherwise, it is O. Given, var()(s) 1 for E S*, we find an upper bound on the variance os of as follows
var(u,)- = Σ (D
D Σ (D), Dj) + (D - D;.
JES-149
jES-
Σ 2((D; D3y² (D. (1) D Dj)) Σ
/2) + 2D( D;) D Sv l1
(26)
jES-M9
jES
s (2spp /m) + 21!( D P 11 5 2ssx² /m + +477 = (6
where we used the Gershgorin Circle Theorem for the bound D ll S 2. With the sub-Gaussian assumption on the coefficients 2", we get the following using Chernoff bound concerning U..
P( 2 Cmin S exp (-
C 4sp2/mm + 165?
O
= eexp (- (- (62))
(27)
4
Taking a union bound over all indices E [1,p] will result in
Cmin 4
C2 min S 2p exp (-O- (b))))
P(max [0s] 2
(1) suppp-rec
(28)
Hence, we can set Ao = Cmin/2.
We re-state and prove the forward pass support preservation (Theorem 4.2).
Theorem 4.2 (Forward pass support preservation). Given Assumptions 3.3 and 3.4, suppose D(1) is & = =''00/1log7 close to D* If 8 = O (Vm/p/ log m), = Ollog m)), and the regularizer and step size are chosen such that ( 1= - zelli + ay 52( (llogmm) and (1) < 1- -1----- then with probability
x
of at least (1) supp-press the support, recovered the first iteration, is preserved through the encoder iterations. Ca + 2s exp ( (
We have a,, O( ssbi) and esupp-pres = Csupp-rac + d =2pexp exp (oP)
(1)
(1)
Proof. Given current dictionary D in each iteration of the forward pass, we have 2++1 Pai + aDT(D""** -Dz). We focus on the entires that are non-negative. Then procedure for negative code entries is similar. We follow similar steps as in (Rambhatla et al., 2018). We get
*++1,) = ReLU((I (DT
(T ( (1
)a + a ( (T D" D') -a2 22
ReLUU(I (I-- 11. 2t + ta( - D " 2,,)" + a(D D') -a.
ReLU((1 - a)z(,)) Σ (D) D. 11 ) + a(( ( - Dj), Di)z
(29)
itj
(DD (1) - Dj. DJ) Di)= + +zzl) + Σ (Dj. Di 92
itj
=ReLU((1 a)z(,)) +a(1 //0) )²ú))
!
!\!
+mmp
22

---

Published in Transactions on Machine Learning Research (08/2022)
where =D; Dj), and Th. - 0 EDf D. )x,0) + ((D))) Dj,
With
IID 1- - -Dil s 51, 030h can be bounded as follows
8 = (D; - - Dj) - /
(30)
where we used the relation ||D]00 - = 2(1 -(D) Dj)). We re-write Thi below
(I)
m = - L(D D 0) 100 + +(D - -D D + (D) Di))=))
*#1
-E (D))) D (1, +
<( 9 D Di) + (Dj. i1)) + (D th (DD" -
i#1
iti
(+)
i#1
EDD] ()
(1) (210) - -2((0)
Σ ((D) (1 -Dj. D3) (Dj. Df) (D D.
i#j
ij
E 10) (2 - z0,0)+ + EDD),
(9 , D. - -
*fj
itj
EDD (=(0) - z.10)) +7p
(1)
(31)
where 7/ = Enpg (D) D.- - D200 Given the sub-Gaussian entrics of the code x", we provide a bound
on the variance of 7i below:
(1)
var ())() = EDD L1 /D S soe
-
(32)
ifj
Now, using Chernoff bound on the mub-Gaussian code entries, we get
-a²
P(17) (1) a) S 2exp 2007
(33)
To bound all the terms in the support, for j E S*, we have
P(max
1 (0,xx11,)) 1,,01 > an) S c9
(34)
where 20 = 2s exp 2x077 Let a, = 0( then (1) = 2s exp ( (oTh)) The above analysis states that with probability of least Gy 17,5 S a, = (VVs))). Next, we write the recursion for when the support is identified (see Theorem 4.1). For the code at iteration T, we have
T
T
Er.0) = (1-a)" 20,0) + 20)
Σ
al1- - (1 - T-
a)) +
1 A1T1,)) - T-
1=1
1=1
T
= -1-op (1-a)T 20,11) + 2tt (1 - -1-1100000 (1- )) + Σ
(1 -
a(m a(1-111
- 11 (1 - T)
(35)
1=1
T
=1 ) + (1- a)) -1 -1(01) + Σ
a a(T--11 X (1 -a)
t=1
21y
(1) + 50
- (1-111110
where 1/0 (1- identified at iteration t - 1, we show that the support is preserved at itcration t. With ||2" -zall1 = O(s), for each j E S*, we have
2 (1- (1- 1 alii"11--- -
11 (1--))-++ With the support correctly
||"" zell2 + a, =0(=log 00000 m
mh = DDD D (= 21(9)) + 72 S
(
=x,0)+
141 vm
vm
(36)
if1
23

---

Published in Transactions on Machine Learning Research (08/2022)
We make sure the regularizer is chosen such that
A
1| Vm
zelli + ay
(37)
We see that the larger the code error and coherence between the columns of the current dictionary, the larger X should be. This is to suppress the noise component in the code recursion and make sure no false support is introduced. Furthermore, we want d to be lower than half of the signal component, i.c.,
1
ad
2,,
+ (1 ")
2
(38)
1 ad S
a of
2 4 2 2 Cmin
where zithn = minj 21,0)- We further shrink the upper bound, given the code from previous iteration & (i.e., ad-1 S >. Hence, we want the regularizer to follow
1 - a
87
ad 2
ad--1
Cmiu
2
2
2
(39)
1 0
1
57
4 5 2
Λ +
)CCin
2
This condition is to make sure the identified supports are not killed the recursion. We use the condition to set the step size a. We get
6 )mmi
a < 1 24: (1
(40)
A--1
Hence, A 52( log and should be chosen sufficiently small such that the condition above is met. We
vmm
denote Gsupp-pres = reupp-rec + eq = exp (1)
+ 2s exp (otdy). Hence, with probability of at least prep: the support, recovered at the first iteration, is preserved through the encoder ilerations.
(9)
Giin (87)
Theorem 4.1 and Theorem 4.2 allow to achieve linear convergence in the forward pass right after the first encoder iteration, i.e., B = 1. With support recovery at first iteration and its preservation, we now re-state the forward pass code convergence (Theorem 4.3).
Theorem 4.3 (Local forward pass code convergence). Göven the encoder 21-1 = Φ(2,, D), Assump- tion 3.6, Lemmas 3.2, A.1 and A.2, then 3 < B, > 0 s.t. ||z - E112 I O(p') Vt > B, where & is the unique minimizer of lasso (1). Furthermore, given Theorem 4.1 and Theorem 4.2, B = 1.
Proof. Given the support selection at iteration B, from Lemma A.1, we have V Co D) 2 pl restricted
to the support for & B. Then, from Lemma A.2, we get.
|V (2,, D)ll2 (| Vi D)) sp
where p A Cpraxe (1 oqp) < 1. Hence, using fixed-point property (Lemma 3.2)
3 B > 0, s.l. ||= 2||2 [},() 2 - م -212 1I > B
where & = aig fa(z, D). Unrolling the recursion,
|| $ S pp- [ 2112-
Theorem 4.4 (Global forward pass code error with variable Xr). Given Assumptions 3.3 and 3.4, suppose (
is inncoheeennt and 5n O' (1 (1/logp) close to D If s O* O"mm//// hog//logmm m), P Ollog m), and the regularizer
24

---

Published in Transactions on Machine Learning Research (08/2022)
and slep size are chosen such that 11) e
1|2 =ell2 ay = 52( (2( vm
m
and a <
22,0)
-(1-
yCmin
vm
then with probability of at least 1 Esupp-pres, for j E supp(z"), the code coefficient error is
(1)
(1) 2ا,, - ( -Dill + Ctj ((.)
Muurolll
(8)
and
2T,(1) 21) -10)) + C2'
(9)
where (Uunvoll = 2(s 1)ta ايا et.)
max, (0) 2/) s + -12065) 21)18a,1 da, (1 + 20x
(1)
ym
m
(57)) = O(ay) with an = O(vsol), 3 = (D; - D DD) I
6 and Esupp-pres == Esupy-rec + Eg
(1)
2
epexp (6+)
Ch + 2s cxp (otb)) ). With appropriately large t, (x
(,))
أا) = Olv 8 ID D2
Proof. We define (1 = Eins ( 11=00 -2,(0) + t ( and upper bound it as
Th
Σ Ees +7
(41)
where Eeg = [2()) - -2 Given (21), we re-write the code recursion
--+1,(1 = P a2 u) ((1 - a), a(1 a(l- ) + amm1)) (0)
E (1 - (1-0)2(,(0) + a(1 - 0/1- + a(1 (9) - -200 =+1,00)))
(42)
Est1, = (2+1,0)) - أ(,1 - (1-a) a + aa( x
Opening up the recursion, we get
1+1 [+1
H+1
[+1
E411 S Eog 11 a1) 1/01200)
- a)
Σ a=1
a(
(faa-11
(1)
+
II (1-a) a)
(43)
q=0
a=1
g=a
g=a
where we define [T=o (1 - a) =. Using the upper bound from (41) and
X from Theorem 4.2, we get
H+
H+
E411 S U141,
14
Σ>
E
1,
II a)
(44)
mm
q=a
where Utt1, == Eo,; 11 o1 - a) + (18)
|اا + 27,, (1)
Ea+1 a I[ - a). We now derive the general upper
bound on Est1, as follows
2a 14
E1, S U1,i
Ea,ia
(45)
Vm
For E2, we have
E2,1, U2,1 + 2a
μ4
E
+ Σ E1 a))
(46)
Vm
$2
$2
Substituting E1,517
E2n S U2,1 + 2a
Σ t2fi1
(Vi
+
2ix
14
Σ
Eo,is)+ + Σ
Eo,ig (1 a))
(47)
Vm
m
For Eg we have
Eas
U3.1
20 m
14
En Ea-1,ig - a)3-a
(48)
25

---

Published in Transactions on Machine Learning Research (08/2022)
Unrolling the recursion,
Pa.
S
V3,11
Vm
M1
22 + 2ex
vm
(1 (1-a)
Σ
01
2a
vm m
14
Σ
01,i3
12
12
(49)
2a
14
(1-a)
Σ
Ea,s, + - (2)
11
Σ
Eo,is
+ (2a
14 )2
E
Eu,i
Vm
Vm
vm
127/11
Given above, we can write up the relation as
2 (ss - 1)02, + 01, a)(s -13 + 2a
M1 (s -2])
Eg,
0a,
2a Vm
vm
+ 2a
14 Ea,i
(s (s- 1)(1 az)² + +1(1 - a2( 14
(s 2) (2a
14 mm
)² -3)
Vm
\m
(50)
03,0
(s 1)2a
Mi
+1,,((1 (11 U2 +
a) 2a
M1
vm
vm
+ (ss 1)a-
M1
Fa,i
(1- 2(1 a2 14
+ (2a 14
)2
vm
vm
vm
Following similar steps for Ea,1y, we get,
Ea,i, 04,i + (s-1)22
1
U3,1 + U2 - 2a
14
01 a)2 + 2(1 a)(
14
) + (2xx 14 )2)
vm
mm
mm
2(s 1)a Mi
Eo,i
(1- + 3(1 (11-a)++3(1- 3 a)
" )2 3(1 a)2 ki
)+ + (2a
He
)3
vm
m
vm
vm
(51)
This leads to the term
Fa.i, I 14.1, + (s - 1)2a
14
U3,1 + 12
+ 2a vm
) + 01 (1 + 20
14 )2
Vm
v m
(52)
+ 2(s - 1)a Vm
14
Eo. (1 + 2a
H1
)
mm
Hence, the general term for code error layer is
E S U441, + 2(s l)a
Uax (1 - a + 20 f-a + 2(s - 1)xx
MI
Ea (1 - a + 2a
)
Vm
Vm
a=l
(53)
where for j in the support, we define the upper bounds May S Va,max and Fo, S ma. Next, we define (1 (1-a) a)' & da,t == (1 (1-a 2a in ) and use to find an upper bound on the expression El= va, a + 2a (nx We have
+ (1)0 2 + +2200)
Σ k=1
a(1 -)
(54)
We bound the expression
Va (1 + 2a
EB, a)" |= + 27,
(0)
a(1 Ea(l
)(1
+ 2a
M
v m
Vm
1=1
a=1
k=
< Eagldo.t + +359) | + 22,(0) Σ
2a t-o Σ a(1 a) k-
m
(55)
26

---

Published in Transactions on Machine Learning Research (08/2022)
1- 1-11-a22 )
Using sum of geometric scries, we write Ex (1 2a Mi ) a 2a
Hence,
vm
a(1- )
using (30) and (34), with probability of at least 1 E7, we have
0 maax
ma, (1 20
Pl Vm
)'u S Eo,
+ 22y)
(56)
my a(1 2
mm
Hence, we bound the code error on the coefficients as following
Essia S V441 + 2(s 1) Vm
1
0f
maxx + 2a,)) 2(1 1)(s 1)a Eo,maxdant
M1 Vm
(57)
2 vm
2
Next, we further simplify the first term
H+1
87 > + 20.
U4415 = Eo, (1) + (8)) -1) + 22,
(1)
Σ
aal--a)=R+1 s E0 +
(58)
2
k=
Substituting the above upper bound into the upper bound for Est1,g we get
82 Exs1. S Eo,5on,t11 + (1 + 2(s 1)ma)(
Fmax + 2uy) + 2(1 + 1)(8 1)x
di Vm
(59)
where Ki == Hi
Given 8 = p/ m), we have (8- (1-1)k) < 1. Hence, with probability of at.
/m )
least 1 - -7 we have
Etg & O(a,) + 2(s 1)ta H4
Eo Eo + Eogda,t
m
(60)
(1)
أازاج s0lv $|| D Dillz 2 [t)
(Qunroll)
((()
(Munroll 2(s - 1)ta #y max; where et,j (Murroll = 0. Henco, for the code error on non-zero coefficients, we get layer 6, Ct,
vm 1 ( 18 (0.00)
(1)
(1)
10, With appropriately large unrolled
(() o 51ID - -Dill))
(61)
for large enough t. Now, we try to prove the relation for zT,(). For shrinkage, we re-write (35)
2T,0) 2 3300
CT
(
(62)
where GJj decays very fast as 7' increases. Hence, bound the second term. susutttte n1 (D,) D. ( (2(0) 21,(1) + 7/ in GT
(1) (1) = "TJ +T= a(T -M, (z -1 Tr and my. = (1-0) - (1 - -g(00)) KT.
T
E
TT
+
amm a11
a)
t=1
(
+
D (1) D. (270) 2,-1((0) + 2,
121 -1--
t=1
(63)
< SKy + 27,
(1)
(1)
a(1 - a) T-t + + 2x
14
E (1-))
m
1=1
-=1 igf
T Σ
< KTg + 27,
(1)
(1)
+ 2(s - 1)a
14
E, 11 a) T-
Vm
t=1
27

---

Published in Transactions on Machine Learning Research (08/2022)
Given above, we find an upper bound Et - T- below. From analysis in Theorem 4.4, have
14
Mt at-a-2
B- Ut-1 + 2(s - 1)a
Ua,max (1 - a + 2a
Vm
aml
141
Mi )r-
+ 2(s 1)x
max (1 - + 2nx
mm
vm
(64)
E1-1,(11 - a²*² 2 U1-1, a)² + 2(s - 1)a
H1
U ma, (1 a a 2a
H1 )r- 1 a)
m
vm
a=1
+ 2 - 1)a
Fo (1 - + 2a
14
2 a0
vm
m
Re-write the first term,
Ut-1, (-11(1-a)T-1 a)T- -(',s -Loo,(L-a) a) (8 -
a(1 (1-a)
(65)
1-1
= Eo,s - a)² + = +27,00
Σ
a(1 - a) T-k-
Similarly,
a
mamaa
Eo,max (1 a) +(3 maxx 2-
a(1 a))
(66)
We write
2
Va.max (1 + 2(x 1-a-2
Ea (1 (1 - + 2a
)
M1 )
V m
vm
m
(g(t) maxl + +2 Σ a(1 a(1-a)=-k(
14 1t--2
2a
v m
(67)
Fo,max (1 + 20
141 1-2
m
2-2
(800) mnxx
x + (1)
(1 a 2a
1 t-a-2
mmxx
m
0=1
Hence,
(1 a) T-t ma (1 a + 2a
M1 at-a-2
Eo,max
2a
1-- 1-2 T-
mm
Vm
1-2
+ (g(1) max
=
2
a + 20
I11
aa
m
< 2) Eo, (1
2a
14 T-2
mm
(1-a) T-
+ (g(1)
max =maaxl
a - 2 m1
(68)
28

---

Published in Transactions on Machine Learning Research (08/2022)
Combining all terms, get
2-2
(1-1((1a)) a)²- S 11-1 -1) 2(s 1)a
Pt
2 Va,max (1 -a
2a
M1 j a)=
m
m
2(s 1)a
11 mTI
Eo,max (1 a + 2a
Mt
| -a)
vm
(1 r-)-11_ + 2(s 1)a M1
(1 1) Fo,
vm
(69)
(8
maxl +2
Σ a(1 a)T=k-1 2(s 2(5--1)1111 (1))11))-
a)T-
MI (t
(1 + 2(s 1)a
mm
[-1
(B(1) /mmxl + +2 [I)
Ea(1 a(1 a) T-k-1 2(s 1)k((1 a)T-1
14
where K
V
Moreover, we bound
21-2
T
a(L -
S
Σ (1-a)
T--
T-t
1
(70)
a
a
Finally, we are ready to write the bound for (1)
1 - b +2/7, + 2(s - 1) 141 (3(1))
= | +22200x (1 + 2(s - 1))x)
mm max
(71)
+ Σ 2(s l)a 111 ((1 2(s - 1)a
14 (t - 1)) Eo.
14
vm
Given h = a, with probability of
i where a, /sōi and O¹ (Vmm//log m., we will have
| < DoV>sIDDf - Dille)
(72)
Theorem 4.5 (Global forward pass code error with fixed Xt). Given Assumptions 3.3 and 3.4, suppose Dilli is un-incoherent and (1/ log close D 1f s O (' hogm/logmm the regularizer and step size are chosen such that X fixed py
logml m), μ. Ollg m), and
x zoll1 + a, = -22 (022ogmm and Vm
(1)) < 1- 2A.9 -1-4)0mm Cmm then with probability of at least 1 prupp-pres, for supplz"), the code coefficient
(1)
error is
21n1 =5n1 OlV D2ll2
(0)aroll,fieed + RReed etj
(10)
(.1))
and
2r,() = =1 (1))) + (1]
(11)
where (1) un)ro,,,,, fined = (s 1)ta ML 7 ct)
max; (z z(0) 210)00 1ffeed 1=0,0)
g 8fived (1 + a 1
(11) = Ola + (liee) with an = O(vsoi)) 3800 = (D; - , < 2 and compp-pres == (1)
D))
87
Csupp-rec + 6,
!1]
2pexp (--1014)
-Ca
+ 28 exp (oT))) ). With appropriately large I, |z (1)
(,()
أر,ا = O 8 (DD -Dille 2|22 + )Rrrdd
29

---

Published in Transactions on Machine Learning Research (08/2022)
Proof. We denote the regularization used in all layers X X =.... X2 Afxee We assume that there exists such Aixed that meets the lower bounds of regularization and also allow to pick an >> 0 according to Theorem 4.2. We define Th. Cots1(D) D D M| ze,(5) + + and upper bound it
(!())
E + 7j
(73)
where Fe, = 12(0 - -2,(- Given (21), we re-write the code recursion
2++1,(1 = P ax (1-1(aaa.00p)) a(1 ((- =1)) + am)))
E E - (1-a))z,()) + a(1 - - + aa(m) - -10 z+11)))
(74)
B = (++1,)) 2 (1- Et, a:3 a + 10
Opening up the recursion, get
E S Eogs II
(1- - a) 03.0 ( Σ aa (1
a) + Σ
a(
aa + 1 2-15))
(1)
a1 (1-aa
(75)
q=0
4=1
y=a
a=1
q=a
where we define [ 1==a
a) = Using the upper bounds from (73) and X from Theorem 4.2, we get
E S Uit1l + T241.
a1-a)
(76)
where Bt41, = Eo, (1- 1 | (1) Σ a I
12)
(1 (1-a)
a) and Mtti = Ett a1
e-1ي 11
a). Following similar steps in Theorem 4.4, we now derive the general upper bound on Es+1,5 as follows
14 m 12
Σ Eo,is
E1s, S U1,1 + trin +a
(77)
For E2g we have
Eg,5, S U2,1
12,1 +
mm
14
Σ
E1,1
Σ
Ea,i (1 a))
(78)
12
f2
Substituting E1,11
E2,1 U2,5 + -2,5
Σ mm t2f
HL
(U
+
tr, + a
Vm
Σ Eoms)
Σ
Ea,is (1 - a))
(79)
is
12311
For Egir, we have
3
Es,s, S U3.6 + T3,6 + a 1
Σ
E Ea- (1-11)))))) -a)3
(80)
a=1
127
Unrolling the recursion,
Ea, 5 U3.6 Ta,i +
Σ
(U2,22 + +22,2) + Hi
(1-a) Σ
(U1,1z + +r1,2))
Mi
Σ
(Viss + r1,6s)
m
Vm
vm
12
#i Vm
HI (1 a1- a)2 Σ Eo.i2 + 2(1 a)(a
Σ
Ea,ia (a M4
Σ
Eo,is
Vm
V
ig
(81)
30

---

Published in Transactions on Machine Learning Research (08/2022)
Given above, we can write up the relation
E3,i, I U3,51 + r3,i,
+a
14 vm
(s 1) + r2,1)
+ (V1,1 + (1,)((1 a)) - 1) +a
pr
(s - -2))
vm
M Ev,i (s - 1)(1 - aa²² + + - a)(a 1 14
[s 2) + (a 14 )² 3)
+ aa Vm
vm
mm
(82)
12,1 + 12 + (Ui,i + (1 - a)) + a I1
03,1 + r3.i + (s - 1)a
vm
mm
+ (s 1)a
14 Eo,i
(1 a1=a)²+ + 2(1 - a) 14 ) + (a
14
)²
vm
Vm
vm"
We denote 1,+1, = 1,+1, + 1,,1, and following similar steps for Eajiy we get,
Ea,i, S U4.6 + (s - 1)a
14
U3, + u2 - +
14 u1,
a)² 2 2(1 ai)
Mt
+ (a
M )²)
vm
vm
vm
+ (s 1)a H1l
Eo,t
(1-a)² + 3(1 () _ 1² + 3(1 (a
Mr
+ (a
Mr
)3
m
vm
m
(83)
This leads to the term
Fa,i, I us,i + (s - 1)a
u3,1 + u2 - a a )
+ u1,( a +
)
vm
vm
vm
(84)
+ (s 1)a 141
M
E0,i(1 - a a
Vm
mm"
Hence, the general term for code error at layer is
E+t1 S uutis + (s - 1)a
Σ Vm a=1
fi
(Va,max + [1 + t-a + +(s (s- 1)a Vm
Eo (1 a + a Vm
Hi
Vm
(85)
where for j in the support, we define the upper bounds Va.j S Ua,max Tajj S Vamaxx and Eg.- S Ea, Next, we define (1 a)) < 8fixed = (1 a + a "n) and use it to find an upper bound on the two expressions
El=1 max (1 a ) )- and Eiz Fa,max + vm ) The following bound can be achieved similar to the steps in Thcorem 4.4
87 =max
M y & Eo.maxto fffxed Vm
ma a
+ an)
(86)
a<1
mi 7m
Hence, we focus on Ee= Fa, (1 - a + a ) next. First, we rewrite
1+1
Tt41
g Σ
as
(1-
(87)
0=1
We replace all X with a fixed one A fixed and write
Tas(1 +
)
A fixed
a(1 aaa-k+1]((
+ b-o Vm
mm
k=1
(88)
A fixed
(1 a +
1I 1-- v m
a(1 a a-a-kt
1- (1- 1-(1-ata
Using sum of geometric series, we write Σ (1 a
1-a
Hence, we get
ym
a-a vm
a(1 1
Ta (1 a a a
14
1-- <
fixed
(89)
Vm
a
pi /
31

---

Published in Transactions on Machine Learning Research (08/2022)
Hence, we bound the code error the coefficients following
E1+1, S "41, + (1-1)
14t mm
maxl + ay + ((xod)) + +
1)a
14
F Fo d
(90)
Jrr v
2
Next, we further simplify the first term.
From before, we have
[+1
8 + mmax + ayy
Ut+ij = Eo, a) + 10 + +2,
Σ
a(1 - a) a-)-t111--
S o
ffixed
(91)
2
1=1
and
(+1
r442 = ad fixed
(1-a)) 1 fixed
(92)
k=1
Substituting the above upper bound into the upper bound for Est1, we get
E+t1 S Fort + (1+ + (s 1) ( of
mmaxl + ay + ((xed) (t 1)(s 1)a 14
(93)
vm
where xi
xfixed
14
Given O* (Vmm (og//logmm) m), we have (s - 1)nffixed <
1. Hence, with probability
7m (1 vm
of at least
e we have
(1)
Eij S O(ay + +fixod) (s 1)ta
Mi Fo, d
Foo gfxxed
mm
(94)
(21,6) L1)1 = / -Dill++ Aixed + Et.
(1)
Munrroll, fixed,
MJurolll fixed = (s l) Li where (L
max; (0.0)
i 10 +
(0.0) وز}) [tixed, With appropriately large
(1)
vm
unrolled layer t, Eg
(fumroll, fixed
0. Hence, for the code error on non-zero coefficients, we get
! (())
O( (ID) DD 1|2 + A (Bxed)
(95)
for large enough t. Now, provide the relation for 2T,())- We re-write (35)
21.0) (1) (1- - + CT.
(96)
11) where 10 = + -I - 01 and n = (1-a))) -2 (1 (1,(111)))) (1)
KTS
decays very last as increeses. Hence, we bound the second term. Wesubstitute (1)
ni ni
Eits D. (.')) (=7
=x,) + 7/ in c
T
( TT E RTJ
(1)
+
a
(1)
D ) z.-1(()) - -1 TT
E KT.
t=1
(97)
< <KT + (7, + > fixed Σ a111-00 - x+ + a
E, (1 - a-
1=1
T
< Kig + +7g
(1) + \ fixd
+ (s- - 1)a
Σ
V
32

---

Published in Transactions on Machine Learning Research (08/2022)
Given above, we find an upper bound Et - T- below. From analysis in Theorem 4.4, have
EI-1 S V-1 + (s - 1)a
14
1 Va (1
+ )m-a-
Vm a=1
vm
+ (s - 1)a 14
Fo, Fo,max
a a
1 )t= Vm
Vm
(98)
t-2
(1-1)(1-a a))-t S 11-11
a))= + -2(s-11a
14
2 Ve,max (1 a + a
a-aa
14
) aa)
m
Vm
0=1
+ (s- - 1)a
14
Bo,max a a
2 (1 - a) T-
ViI
Re-write the first term,
- -)=-==(-=x (Eo,s (--1a(1a)"~+ a)) + 18,% |اأزا| + 7 (0 fixed Σ a(1 a) a))=-
(99)
= -o, -
أ 1z) 7 1)) + x fixed
a(1 a) T-K-1
k=1
Similarly,
Ca,max Fo, (1 - aa) + max mmax + 7 + fixed
((11)
Σ a:(1 a) a-
(100)
k=1
We write
max (1 + H1 1 1-a-2 Vm
Fo,max (1 (1 a a
H4 a--a-
Vm
(g(1) max mmae + (1) mmax
+ fixed
Σ
a a)
+ a di 1-a-2
k=1
Eo, max (1 - a )r
Vm
(8(1) max F
+ +100)
A fixed)
(1 + 1
mm
(101)
Hence,
(1 a) T-t 2-2
Ua max (1 a + a
M1 f-a-2
Ea (1 a + a
ML
(1- T-
m
Vm
t-2
(9(1) M m + 7
X fixed
a + a
1
mm
S (t Eo. a
14
T-2
vm
(1 (2) T-
+ (1911
max l
+ 1 + fixed
a(1
11 mm
(102)
33

---

Published in Transactions on Machine Learning Research (08/2022)
Combining all terms, get
(1- s 11 - -x)T= (8 1)a Pr
Ua,max (1 1x
pi a
211 - xx
/m
vm
+ (s 1)a 141 Eo, (1 a + a
14
/ (1 - x-
vm
Vm
En, -x) (8 (1--)4 11a 11
(1
1) Eg, m
Vm
[-1
(103)
+ (9 max Finaxl + (00) + fixed
Σ
a(1 a + (1-11)a -)) -7)--(
Me M (t 1)) T0 $ (1 + (s 1)a
mm
+ (3 mmax + max + X fixcdd
Σ aa(1-a)"-k-14 (s 1)m((1 - a)x-1
21
where iy
xftxed_
vm
Moreover, we bound
vm
a(1 - a) a(1 --
T
1
Σ (1 a)
T-t
(104)
a
a
Finally, are ready to write the bound for CT.1
(1)
S KT. + +17,01 + A fixed + (s - 1) 14 (3(0)
(1)
(1)
(s--1
vm max Fmax + 7 mmax + A fixd )1 + (8 1)x(fxood)
(105)
+ Σ (s l)a
11
((1 (s 1)a
H1 (t 1)) ma)
v Vm
vm
(1) Given 1750 = a, with probability of Eg where a, = sod and O² =0mmmm/ploogm) we will have
/|ID 1 Dilla ffxod
(106)
We now re-state the forward pass Jacobian (Theorem 4.7) convergence.
Theorem 4.7 (Local forward pass Jacobian convergence). Given the recursion Z+41 = =(z,, D), and & the unique ménimizer of lasso with Jacobian J, then p < B s.t. || S O(tp') Vt > B. Furthermore, given Theorem 4.1 and Theorem 4.2, B 1.
Proof. Differentiating the recursion,
J V, V)", V24(2,, D..
Similarly,
j- v ()) V2
where 2 is a minimizer of lasso and fixed-point of the mapping (sec Lemma 3.2). Subtract the terms
Jen1 j f= (2, (J f) (V, D) Vi (V20(z,, D) V20(2, D
34

---

Published in Transactions on Machine Learning Research (08/2022)
Given the Lipschitz properties of L and h, further get the upper bounds VV Vi D)ll2 < Te,, ||b - all2 and [2 D) (b D)ll2 I Læz b all2. Hence, with upper bound on the norm of Jacobian (Assumption 4.1), there exists B > 0 such that Vt > B
J141 - J112 < IVV D) f | (2,, D) ())221
V2 2,, D) V2 D2
- el J + c| =212
where A MrLa, + Laz. Hence,
|1++1 I112 S pllJe -112+ O(p').
Unrolling the recursion,
|1++1- - -112 s0((++1)pp)
Theorem 4.6 (Global forward pass code error). Let & be the fized-poind of the encoder with ilerations Zt+1 = ((z D) Given Assumption 9.6, Lemmas 3.2, A.1 and A.2, we have 1|2 S OO] D* +
where 8- -| 12, & is the unéque minimizer of lasso (1) given the dictionary D, 2 is the unique méndmizer of lasso (1) given the dictionary D and x² is the ground trutt code.
Proof. We first find the error between and 2 which is the unique minimizer of lasso (1) given the true dictionary D Using fixed-point property (Lemma 3.2), get
2 - 21 [ D) -1 D2) S [1 D) D(2", D)ll2 1|0(2", D) -( D') (107)
Using the u-strongly convexity of Lu (2,, D) on the support, and L21 Lipschitz constants of Lz, D), we upper bound the term follows:
_ 20122 S - + 01 D - D*1l2
(108)
Where p - Cprox app) 1. Denote g - aCprozLi21 step size
which be made to be small with proper choice of 1-p
2 20 S gl D2
(109)
Hence, we get the following code error
|| 22112 < || 21 1| - 22 s D D' + S0 SO(ID-D"'l D'
(110)
Theorem 4.8 (Global forward pass Jacobian error). Let & the fiaed-pord point of the encoder with ilerations 2++1 = ((z,, D)- Given Assumption 3.6, Lemmas 3.2, A.1 and A.2, we have | JJ S D-D'22 where 83 = 1| 12 and J, J and J are Jucobians corvesponding to 2, 2" and z².
Proof. First, define Jr. For z", we define the mapping function + Lz D), where 2D( is its minimizer evaluated at D i.., ViLa(z", D*)
x) 0 given the generative model D- D.. From implicit function theorem, we get 8D
(x = D'z²). Hence, we define the Jacobian J 8z'(D)
J V D*) (= D*) =
which is later used in the global backward pass analysis. Alternatively, if La,La(z", (") is invertible, then can compule J++ as follows:
J+ -
DD))1
35

---

Published in Transactions on Machine Learning Research (08/2022)
The Jacobian w.r. row i of the dictionary is
Jay DD. 2 (D x: pp)s-
on the support S* of x*. Outside of the support, it is zero. Now, given the recursion z++1 (2 D), we differentiate the recursion,
J D
Hence, we have
j =
J = ( D') "20(3",D")
where Jz is the Jacobian of 2". Then, following similar step to Theorem 4.7, we can write
J- = V,
JT) (V,0 - V,0(2" D*)) (V20(2, D) - V20(22" D*))T
With respect to D, we denote the Lipschitz constants of V (2, D) and V2 D) with LoiD and Lond respectively. Then,
| - j- S pll. - J c|i ش l|2 + cp D D
where e f MyLa, + La, and eD - M1 + LooD Given the global forward pass code error, we get
| 1 9: 2 1/2 + + - D' S (gD + 9=9))l D
(111)
A c where 0= I-p: qD I-p
A cp. Hence, we get
[ 12 v |1 12112 + +111 J'll2 s O(I|D - D 8t)
(112)
where we denote 83 = j J2
A.4 Local backward pass proof details
In each update of the dictionary, we bound the gradient approximations as function of unrolling t (Thcorem 4.9). This shows that gle-lasso converges laster than dee and gā- and the latter a biased estimator 9. This is followed by Theorem 4.9 showing the order magnitude of the bounds is indeed tight.
Theorem 4.9 (Local convergence gradients). Given the forward pass convergence results (Theorems 4.3 and 4.7), 3 p < 1, B > O such that Vt B, the errors of gradients defined in Algorithm w.r.t 9 (4) satisfy
dec
øll2 S 0(p')
[ lasso 0l12 < 0(4p2))
(12)
lgg ac ~ll2 S O(tp² MJAV3).
Morcover, the order of upper bounds is tight (see Lemma A.4).
Proof. For gdec with the infinite fresh samples, we have lim n E V2Lzi D) =
Exxx [V2L (zz, D)] a.s. Based on Lemina 3.3, we get
|9 doe /||2 = ||Eeex (V26m D)] Emex V22 D)]
(113)
< EmEx 2a D) (2, D) < Bnex [Tal - -2122] =Op').
Similarly, for gt ao-lasso and gf n0-ls we replace the sample mean for gradient computations with expectation in their limit. We re-write the gradient estimation error as following
g ee-sas
ğ EEEx [)) - -=]] ETEX [Q) + EzEx [J.Q]ass0-11 (2))
(114)
g. 9 ExEx [Q(2, J:) 2)] + ExEx (2) + ExEX [J.Q). (2))
s0-ls
36

---

Published in Transactions on Machine Learning Research (08/2022)
where
(2) (z) - V2Co(z), D) - V2Co(z, D) - (z, D) - x)
Q 11 (2) - ViLa(z D) ah(2)) Vi D) =)
(115)
Ql (z) - (z, D) V Lz D) =) x)
Q(z,J) - V+ Co D) + Ln, D).
We provide bounds the above in Lemma A.5. Hence, it suffices to bound the terms the r.h.s as follows:
g de 2112 S Emex [LI]] J 1221 2|12 + (4m//)]| - و + Ms(Ln/22]lzz - 2112)
(116)
Using the convergence errors from the forward pass (Theorems 4.3 and 4.7),
g pe-lasso 112 L,0(1p²t) (L21/2 M) Ott2²t).
(117)
Similarly,
11g -gl2 S Beex [Lil]s, - -1122 - 2112 (4m1/2 2112 M,((Ln/2)llzz 2113 (10n())1l2) (118)
Using the convergence errors from the forward pass (Theorerss 4.3 and 4.7),
| 0112 s (tp(tt²)) + (22//2 + Mohru/2) O(p²)) Ms]OM( Octp + M,AV))
(119)
Lemma A.4 (Tight local bound). The order magnitude of the upper bounds in Theorem 4.9 is light.
Proof. It is sufficient to show that there exist an example such that its forward pass code and Jacobian convergences are Opp') and Ott''), respectively. The following example confirms this. Without loss of generality, let x* be 1-sparse and non-negative,
= D and D, = 0 for j * The loss function is = D2 l12 + \/z()l- Given the support: recovery after first iteration, the encoder forward pass implements (t) 2,) aaD 21 + A) (1 () a(*)) A). Hence, the forward pass convergences are
(t) (1- (1= Σ a(1 a)) 1)= (1-a)2z +1 a))) X)
(120)
k=1
=1)(1) 21) = 02200 z(i) A) =0(p')
and
Jr, = J.-1,()) - -a(J,-1,00) +2D;2,,0) -(
= pJ,-1,10 + O(0') + J
(121)
J1,0) -J() p'Ja, Σ O(p') =O(tp')
h=1
where p 1 a, 2(3) =0 X, and J a: (2D(((()) Di2(0)
Lemma A.5 (Local bounds). From local gradient errors in Theorem 4.9, the following satisfied
(e)]l < (zm//)|lz=- - 211 1Q/asso-11 (2)]12 < (Ln/2)]| - =212 12)2 < Li]]J, -
(122)
1- 11 (2)1l2 < (4m/2)l|=e - |0h(2)]22
Proof. For (2), given convexity of VI D) and its domain (Assumption 3.1) and Lemma 3.3, we achieve the quadratic upper bound. For (1a)s we add and subtract. V, D), and then use quadratic
37

---

Published in Transactions on Machine Learning Research (08/2022)
upper bound. At line four, given Lemma A.3, we use O E VICu(2, + 0h(2) and assume that 2, recovers the sign entries of 2.
||Qr 11Q
Lo,, (K)(() - 2||l
11
1|
V (z, D) VIL. V) Vi D) 0h(z,) Cz, D) 2)]l
(123)
< (k1s/2)|zz -21ī [ D)ll2
S (bz/2)||2z - -218 + (10h(z)) - (3))) < (41/2)]1
-
Similarly,
11 ||2 = VV D) ()) - =Ill
= ( D) (2) + -(i S (1m/2))|=== - 2 (2, (In/2)]lZz 2112 + 110h(2)ll2
2)ll
(124)
For Q(2, Jt), from implicit function theorem, Q(2, J) 0 under the support S of & that is identified by 2
To prove this, consider the minimizer (D). We have 0 E V f D), hence we get 0 E i D) + f D). Given the support recovery, the relation ( (á, D) o 1s-) + V2, + 1g = 0 also holds which is equivalent to Q(2, Ji under the support. To show this, given the recursion 2++1 = (2,, D),
differentiate it and get JI+1 V (z,, D)J, V20(z,, D). Given the support recovery and fixed-point. property, can write
-= 3 ( aVi D)
- D)
f - 1s @ J = -aV @1s a, 01s
(125)
j (Vi, D) V2 01s
If the term (Zi, D) @ 1s) is invertible, then we can write
j+ = - v Cn, D) O Is Is(Vi, [(2, D) o 1s)
(126)
For the Jacobian corresponding to row i of the dictionary, we get
2 f(,)) DD! (D,,, + (D] (65-x;),,1 1p)s
(127)
on the support. Outside of the support S, the Jacobian is zero. This proof is similarly provided by Malézieux et al. (2022). Hence, we can use La D) j Vi, C2, D) under the support S in the following
1Q) || D) Ca, D) (2) j Ca, D)
(128)
$ -] J (8, 12 - IJ - J122
A.5 Global backward pass proof details
We re-state and proof Theorem 4.10 as follows:
Theorem 4.10 (Global error of gradients). Given the convergence results from the forward pass, (Theorems 4.6 and 4.8), the errors of gradients defined in Algorithm 2 w.r.t global direction g" (defined in (5)) sotisfy
||g le-sosso - g" g2 < O D D' + ||D D² + + + 83 + MsAv/5)
(13)
aa g" 0'112 < O() D D 212 + D D'
8 + +8)).
Proof. For gt dec we compute the gradient in their limit assuming infinite fresh samples
1im n
Ex V2Lai (2/, D) = E (V1La(z1, D)] a.s.. Similar to Theorem 4.9, we re-write the la-susso
of gradients get lasso and g gh following
g ge-lasso g" ExEx 1Q(x", J.) - =²)]] + Exex (x*)(= + ExEx [JiQ)
(129)
se-ls
g" = -Eaex 1Q(x", J.) - -2")] + Emex !Q (2*)) + ExEx 1-1
11
gt
38

---

Published in Transactions on Machine Learning Research (08/2022)
where (z), Q()))))(()) (z), and Q(z, J) are defined as in Theorem 4.9. Given Assumption 4.1 and
Lemma A.6, we find an upper bound on the r.h.s of the gradient errors as follows:
9 10l-lasso
92ll2 s BEEX [L1]]] -
D 12 22112 (la1/2)| -x =11
+ ExEX
LiDI DD
- BREX
[11 - - + D D (l2) - -2112 + || x= =²112]]
+ ExEx [(4m/22 2112 + 1 - =11 -
+ ExEx
[Ma (M] =112 || - ==1)² (2))0h(z))laa
(130)
Similarly,
[9 - g² 122 S BRE
- J' + Malnn/anll1 - 12)1)) - 2"112 (L1//2]lzz - ==
ExEx [M)(Ln/2)]|= ==13 InD| [
S ExEx
[L J112 +1 - MMLLinn D D2 8112 + 1| - ==112)]
+ ExEx ((^m\/2((] 211 + + =11
ExEx (M/ 213 || - =B1 LID D D
(131)
Using the convergence errors from the forward pass (Theorems 4.3 and 4.7),
[1 ne-lasso ="ll2 s Otp + (I
D2 + p'( D + 88)
-DDlaal + D' ())) + - - D'112(0"=
(132)
+ (La/2+M)lu/33) M/hu/2) O0' + + - D' O(I D2 + M=1(0h(z)1l2)
Hence,
gg x0
la-sa
-g" - D D' 1|2 - 81) + + M
(133)
= D' _/ D D D D"
1- 85) 8 + M,Ve
Similarly,
9''ll2
<O D ID D' -I 1' + 8 +83)
(134)
Lemma A.6 (Global bounds). From global gradient errors in Theorem 4.10, the following are satisfied
(0 (2 122 < (fm//2]| - 2
|1Q 10ssoll (x") < (f1/2)]|2z == =² LID D D 112 [0h(2)))2
(135)
(Q"))1()"") < (44/2)] ==' =
Lin D - D*
J.)ll < - MyLunhainlD- D'
Proof. For (z*), we achieve the quadratic bound using convexity V Cz,z D) and domain (Assump- tion 3.1) and Lemma 3.3. For Qlasso-1(2"") we add and subtract Vi D), and use quadratic upper bound similar to Lemma A.5. At line four, we use ViLn(z', D*) (Lemma A.3) and assume that z;
39

---

Published in Transactions on Machine Learning Research (08/2022)
recovers the sign entries of x" (see Theorem 4.1 and Theorem 4.2).
QQ (2 = VV D) ah(z,) Vi ()(; - -20
= (z D) - V=C,( D) (="=(=", D) + dh(zz) - , - =x"]ll
< (/1/2)]lzz- z' 11 +
(=<=(=' D)
(136)
2 (11/2)]lzz 21112 + + D) ,
s (1m/2)]lzz =1113 + LID D' 12 - |0h(2z)]l2-
Similarly,
(2*)(((>)11 (2 D) V=," (2: 22)112
VV (2, D) - Viha(2" D) + Vz ( D) C=",, D) 21)122
I (111/2)122 - z²133 Li (z", D)ll $ (1/1/2]l22 211 + VVLL D) - ViLa(2", D²)
- D
(137)
For 1(==, JJ), from implicit function theorem, Q(=",J") = 0 for D evaluated at D Hence, we can use V
",0==,D) - D*) in the following
1,)=1, =IJ, Ca D) + Cn D) Cz V") D') |J Ca D) J Vitiii D') [V D) V21 D')
|J ( - J fo D')
+ fo - J V' (2", D²) + T2nD | D D2
-
MMyLnn D D 11
D*) - J Pi D') + LnDD D'
$ II(JT - J't' D)112 + M, Ma D'I12
s L-lJa-- J2 + MyLnnpaa - D
(138)
Theorem 4.11 (Dictionary learning with variable Xt). Given Assumptions 3.3 and 3.4, suppose D is 14 in-incoherent and (&, 2)-close to D with & = (1/ p) IS O(ym), l Ollog m)) learning rale is n 0( P ((11--/7/2)) and the regularizer and step size are set according to Theorem 4.4, then for any dictionary
(I) update using g with probability of at least Campp-pres;
| - -2 < (1- (1-
(15)
(1) where Csupp-pres expp + eg epexp (ا-())) 2s exp (
(1)
(1)
Proof. In this proof, we study geg and for case of notation we drop the superscript.
TT
(
= E
2r 20 (T, (T xz) = E 1 T (D xx)] +
(139)
where 7 (( 1sy و #0 3r, (t)) 27 x)). We have the event 1 10 1
happening with probability of 1 - euupp-pres; and psupp-pres decreases with decrease in 6,. Hence, 7 gets smaller. We write
(1)
(1)
gr
E #0 #02T.00 (DU) x:) +
(140)
where B is an diagonal matrix with 3 for j E as entries. For j $ S, 12; #0 = 0, which results in (1)
gT,3 0. Hence, only focus j E S where I 70 1. We condition on the support and re-write the
40

---

Published in Transactions on Machine Learning Research (08/2022)
gradient as
(1) IT.
E( D (1) 2T x)] +y
|| D
(1) (I B )=(s) + D3 5 STS - (32(s) |] +
(1)
E[DS (I BI- B - [8] B
[ +
-1 -|-
||]
(1- - +7
+ + (I- ) [T] -
[]] + (1- + + |S]
(141)
where in the last line, we use the fact that E 1 E S 0 and E (ss) |S] Computing the expectation, we get
11) IT. = (1- -,,"0"1--000---0000000000-00110 p -8 ((1 1 D -D;) + U +7
(142)
where U = E (I- BI -
S5) + E _ B (1 /3 + ST STS ST].
Given this gradient, now find bound on Um.
U TJ = EED (I- F(= 2 1 S] - E 40)
[]] + BIDSTT(SSt - 130) + + 178 S]]
(143)
First, we bound E
Biz
ST] as following.
(1))))) STyj
T a(1-a)
2 1[+ HI Σ a(1 a)= T-
E|
S]] &
-
sisgg (z(s) 11)1, [s]
1=1
a(1 - a) T Σ
Eigi-n,siggu(// - 21 =iig, (2), S] + +
m
(144)
where
(1) = = ( 77.
" Similar
docay very fast as increases. Hence, we bound the other
terms. We have
S 5
if
Bhy 2())
||]
(145)
=0 0
if =i=
< E111, =
E(E--inssgnn((e) 2z-1,(N) |5]
(146)
ik
and
$ E if
E[E, (iiiu(is 2=-1114))8i00((2200)).) S]
(147)
=0
if
Hence,
S E--1, if
Σ E[ (.signii)) Z1-1,())) |S] ド
(148)
=0
ifj=i
and
E iii
Σ E F. 1 1 sign(270) (=-1,)))sign((2,000))) [)
(149)
ifi=i
41

---

Published in Transactions on Machine Learning Research (08/2022)
Hence, for j fi, i, we can write
T
E []] - 26 +
A
aa(1-on - + +XT
(150)
vm
t=1
where from (103), we have
E4-1,1 - aa) - (1 + 1)a
141 mm
(t 1)) aa
(151)
(3 max /maax 2
Σ Ea) 2(s 1)m/(1 - a))=+
Henco, given the sparsity level, the term below is bounded by a, with probability of - (0)
T
Σ 11-1, -1-11-))- a( = [1+ 2(s 1)a
la mm
(t 1)) 0.T
T t-1
T
(152)
+ (3(
max xmaxl
+ 2%
Σ
a(1 - a)
T-k-1
+ Σ
2(s 1)xi(1 - a)--1)
k=1
1=1
s (3 maxx mmax
+2aa!")
ski) Oaa!))
Finally, we get
26, + di
((a (1) ) + Kig.
if
(
STg
[S]]
5
(1)
vm
jf = i
(153)
For appropriately large T, KT. can be make small; Hence, in this case, we get
(
10% la O(VPPGō) 112)
(154)
Now, we can re-write the gradient as
g = -1 3D) DJ) + p, (
U
1 (7 -(D) -DD +
(155)
Pj
Pj
where T == pi -= ) and 0 = p- D
UT
(1)
7). We can bound the norm of 0 as follows:
|01| S 3/ DD (1) 1|2 + | T lla +7
(156)
Given |D 1|2 and = =D; D. D)) = D Di we modify the upper bound
|00|| - I (1) - D2 + ( (1)) +
(157)
We assume dictionary closeness during training, i.c., | D* 12 - 2 D 2l, which we prove in Lemma A.7.
Given this closeness, we have
D 122 (D
D² + | 12 2 0( p
(158)
Moreover, with y dropping with 6,, and for = O(vm), it is reduced to
||0||2 < P 9 D2i
(159)
We get
9 112 S (, -/(118/\\110" 12 + p, ( -DiIl
(160)
1g 13 < 1/ 3 Di
(1)
42

---

Published in Transactions on Machine Learning Research (08/2022)
Using this bound, we can find lower bound on the correlation between the gradient direction and the desired direction as follows
1 11) 122 - (1,(1 - - Di12 + 1011 2p, (1 - 88 DT) -Dj)
200 D Dj) -py(1 )D D
1
|1
(1) 11
pj(1 (60)) 11012
1
Pj - (()))
(161)
2(0T3 D DJ) = p5 | D
|19T 11
11)
P3 / 11012
Pj(1 )
2 (p, - ) - Pj (1,
1 Djil2 +
1
1 112
1 (1)
P,(1 - //0))
Hence, using the descent property of Theorem 6 from (Arora et al., 2015), sctting the learning rate to 1 max, p/(1- and &i (1 (1 - ) Pj (- S 1
(1
|ID -
- Dj113 S (1-
(1-b) , -
(162)
Theorem 4.12 (Dictionary learning with fixed Xτ). Given Assumptions 3.3 and 9.4, suppose D is p4- incoherent and (o,, 2)-close to D with &t = O* (1/ logp). If = O(ym), P Ollog learning rate is
" = O 51175))) ) and the regularizer Xized and step size are set according to Theorem 4.5, then for any
dictionary update using g with probability of least Coupp-pres
(I)
- Dj13 s (1 - DDI - Dj + e0
(1-))|
(16)
where 11 = s(1 (
2p fwed2 csupp-pres == supp--ro + e = =ppexp (5 + 2s exp (Thh))
(1)
Proof. Following steps similar Theorem 4.11, we write the gradient as
(1)
(1) D - p, - + U.g - )(²) - (1 -D;) + +UT +7
gT. -
(163)
where U1h = (I- ( ||]
(I-B
(1)
E1D3E (1) S| 5 E (( (1 850h) + TT TTS ST].
Given this gradient, we now find bound on U2} First, we bound Biz ST. S]] as following.
T Σ
Bi (
S]] S
a(1-a))-
1 + Σ a(1 - fixed
|S]
t=1
(164)
T
P
aall-a) T-
Σ
(z()) - =1-1,(1))= |S] + R
where we set all (
1 = fixe and Kig
(1) (1) = E El T. Similar to KiT KTg
decay very fast as T increases.
Hence, we bound the other terms. We have
S 5 if
B 2
||]
(165)
=0 0
if
= fixed
if = i
E[A fixed sign(2+-1.05)) 2(11) |s|
(166)
=0
if j +
< E,-1,
if =
=1-1,()))2(0)
|S]
(167)
0
i
43

---

Published in Transactions on Machine Learning Research (08/2022)
Hence,
S E1 ifi
Σ Lign-1a - zc-1(())2(0) |S]
(168)
0
i=
Hence, for j # i, we can write
BI 5 + mm Σ a(1 a)² (Bo,; + Bx- + KTg
(0)
Si
(169)
We have
Fa-1, -²) S (1 2(s 1)a- mm
(t 1j) Fo, Pa
(170)
+ (91)) |maxl + 2
Σ Ea(1 -)) + 1)) - ax)T
Hence, given the sparsity level, the term below is bounded by a, with probability of 1 1
T
T
Σ -1-
Σ 21+2(8
1)a
1a1 mm
(t
+ (1(()
=max + 22
Σ
T t-1
a(1 - a)
T-k-1
+ Σ
T
2(s 1)x((1 - a)T-1)
(171)
max
1=1 k=1
1=1
< (1 maxl smaxl 22a (ki) 0(a!!)
Finally, we get
6) th 0aa (1) + KT.
i if = i
E Ei
(1)
[S]
(172)
A + R
For appropriately large T, @ can be make small; Hence, in this case, we gel
| - (( 1|2 + pfxfixod
(173)
Now, we re-write the gradient
( Ir. p3 - (D,)) - +3)+,,(-8)) D
11
(())
+
1 ->) - Dj) +
(174)
Pj
Dj
where T = pp ),0), and 0 = p,(- D LU())
We can bound the norm of as follows:
|101| < p30 D (1
1 + | +y
(175)
Given |DD 112 and 8 (D;- 0h
Dj) -
= ! D D we modify the upper bound
|0|| < 1; (1) + 0(
D (1) ||2 + p3 fixed) +
(176)
We assume dictionary closeness during training, i.e. | D* 1|2 - 2 D' ||,, which prove in Lemma A.7. Given this closeness, we have
D (D D" + |D' 1 Olv
L m
(177)
Moreover, with y dropping with 6,, and for O(ym), it is reduced to
||01|2 D Dj || + +> ABxed)
(178)
44

---

Published in Transactions on Machine Learning Research (08/2022)
We get
l Lo S p,(1 - (1) Djll2 + p)(() -Dill + \(xed)
(179)
| = 22 ) DJD Djll fixed2
Using this bound, we can find lower bound the correlation between the gradient direction and the desired direction as follows
19 113 = (p,(1 - - - 101
2p, (1
(.) D3)
2(0, D -D)) -p,(1 0. ||I|
Dÿll
1
119
(1)
11
1
1013
Py )(0)
ps (1-))))
(1) 2g D Dj) p1 M|
Dj11
11
!1)
11013
)
)
P5
Pj(1
2 (p,(1 - )/ 2p
1
(0)
Djl
1
11g 11
2p,
fxxe22
1
8
(
P3
(
(1
(180)
Hence, using the descent property of Theorem 6 from (Arora et al., 2015), setting the learning rate to 7 = max, p)(1 and vb npp (1 ) 2p; 1-g00)
| (++ Dj1 3 S (1 - b - -231 +
(181)
where eΛ == 11
2n, X f
Lemma A.7 (Dictionary maintains closeness). Suppose D has (6). 2)-clssnness to D where & (1)
O (1(1/ lo)m), then with probability of Esupp-pres, we have || (((+1) D2 s 2 D2 when using of and the network porameters set by Theorem 4.4.
Proof- Given the dictionary update
Dj (1+1] - -D Dj
ngrn
(
(182)
(1)
Then, with probability at least 1 Espp--pres, we have the gradient
1 9T. (1 (1(1-15)) (D + DTD
1 1(0) _n)
1
(183)
Pj
Pj
which we substitute in the dictionary update below
DJ D++ - Dj -1- (1-n(p,(11 -3))(11)))0))0)))) D2) + m D -nu(1 -mm)
(184)
writing the update in matrix form
D((+1) - D' = (D()) D* D')diag(1 - 1pps(1 -1- 300 + D
- F DD H - -17)) (185)
where Frg) = P.y - ((0)2² ST3
S] Piy (1, (1- + C 1/9 $). and H(w)) = Bi ST ST.
(1)
Given, the bound [1 ||2 < 0 D (1)) from before, we get | p (pp:) and |||| O (p). Hence,
2
DUF+D'H < D 0 121 D 1 = ( = O
1|2
(186)
pog m
Using the maintained closeness at update I, we bound the terms in the dictionary update one by one below
(ID - D' D')diag(1 nlp =")diag(1=1oo01-
|)) 9|) (1 min np
-D'l - 2(1 ((ns/>)) (187)
45

---

Published in Transactions on Machine Learning Research (08/2022)
| )llg < max ps of ( D D S <<ss/))|
(188)
J
Given the bounds above, the dictionary update can be bounded following
D - DD 2(1 D(ns/p))] o(ns/p) D2 + O r7s
|| + my s 2/
(189)
poog m
A.6 Interpretability
Theorem 5.1 (Interpretable unrolled network). Consider the dictionary learning optimization of the form
minz,D 1 X D >|| +/2|| Dlir, where X , ..... x=) E lmmx and Z = =1= ,²,, ,² ,.,,,""""" z² E Texx. Let Z be the given converged sparse codes, then stationary points of the problem w.r.t the network weights (dictionary) follows D XG- ZT, where we denote G = (2 T + wl)
Proof. For all stationary points, the objective gradient is 0 with respect to the dictionary, i.e.,
(X 2 + wD
(190)
where D is the learned dictionary at convergence. Re-aranging the terms, we get
D = x2TT (22 +wl)
(191)
Using the relation - (AT A wI) TA we re-write the solution as
D DG
(192)
where we denote G (ŻT7 wI).
B Appendix future works and limitations
Beyond dictionary learning Our results are founded on three main properties: Lipschitz differentiability of the loss, proximal gradient descent, and strong convexity in finite-iteration. The findings can be applied to other min-min optimization problems, e.g., ridge regression and logistic regression, following such properties. For example, our analysis generalizes to the unrolled network in (Tolooshams et al., 2020) for learning dictionarics using data from the natural exponential family. In this case, the lcast-squares loss is replacod with negative log-likelihood, and the dictionary models the data expectation.
Limitations Finite-itcration support selection (Proposition 4.1) (Hale et al., 2007) and strong convexity may seem stringent going beyond dictionary learning. Ablin et al. (2020) discuss generalization of local gradient convergence by relaxing strong convexity to the p-Lojasiewicz property (Ablin et al., 2020; Attouch & Bolte, 2009). We considered the noiscless setting and conjecture that the relative comparison of the gradients in the presence of noise still holds, where the upper bounds will involve an additional noise term. We focused on infinite sample convergence to highlight the relative differences between the gradients. We leave for future work the derivation of finite-sample bounds, a step similar to (Chatterji Bartlett, 2017; Arora al., 2015).
C Appendix details of experiments
PUDLE is developed using PyTorch (Paszke et al., 2017). We used one GeForce GTX 1080 Ti GPU.
C.1 Numerical experiments for theories
46

---

Published in Transactions on Machine Learning Research (08/2022)
Dataset We generated n= 10,000 samples following (2). We sampled D' E 50x 100 from zero-mean Gaussian distribution, and normalized the columns. The codes are 5-sparse with their support uniformly chosen at random and their amplitudes are sampled from Uniform(1,2)
1.5
Code z
1.0
Training We let T 200, A 0.2, and a 0.2. The dictionary is initialized to D D + TB with B (O I) For Figures 2, 3 and 4a, we set тB 2 0.55/log Figure 11 shows the sparse code estimates from one example given this initialized dictionary; this is to highlight that a) the initial dictionary is not very close to the ground- truth dictionary, and b) our algorithm able to successfully perform dictionary learning and recover the support by the end of training in spite of a failed exact recovery of the support.
0.5
50
100
Signal length
Figure 11: Example of code esti- mates with the initialized dictionary.
For Figures 4b and 4c, we chose much larger noise level, tn RJ 2.8/ogg.. The network is trained for 600 epochs with full-batch gradient descent using Adam optimizer (Kingma & Ba, 2014) with learning rate of 10 and c 10- The learned dictionary is evaluated based on the error D- D= D'/ The results and conclusion were consistent across various realizations of the dat aset and across various optimizers. Hence, in the main paper, the figures visualize results of one realization.
Noisy measurements We repeated the dictionary learning experiments shown in Figures 4b and 4e where the measurements corrupted by zero-mean Gaussian noise such that the SNR is approximately 12 dB. Accordingly, we set X = 0.3. Figure 12 shows the results for both noisy and noiseless scenarios.
10"
10
DD-D"
10
10
10
10
10
250
500
500
250
250
500
Number of epochs
Number of epochs
Number epochs
Number of epochs
(a) Noisy (T = 25).
(b) Noiseless (T 25).
(c) Noisy (T = 100).
(d) Noiseless (T = 100).
Figure 12: Dictionary learning in noisy and noiseless scenarios.
Stochastic Dictionary Learning In addition to the full-batch gradient descent results in the main paper, we repeated the experiments Figure 4c using batch size of 4, 16 and 64. We observed (Figure 13) that in all scenarios PUDLE is able to learn a good estimate of the ground-truth dictionary, and ge is superior to the other two. We note that for lower batch-size, there will be more gradient updates in one epoch, hence, the algorithm converges in lower number of epochs.
10
10
10a
10
10
10
10
10 2
10
Number epochs
Number of epochs
Number of epochs
(a) Batch size (T 25).
(b) Batch size (T 100).
(c) Batch size 64 (T 100).
Figure 13: Dictionary learning using various batch sizes.
Effect of learning rate We performed the experiments in Figure 4e for various learning rate of 10 10-3 and 10-2 This shows the robustness of the gradient-based dictionary learning against. learning rate. Figure 14
47

---

Published in Transactions on Machine Learning Research (08/2022)
demonstrates such results where PUDLE successfully converges to the neighbourhood of the ground-truth dictionary; Regardless of the learning rate, gi converges to closer neighbourhood than the other two gradients. Overall, smaller the learning rate, more epochs is needed to reach convergence.
10a
10
10a
D-D'll
DD- D'Ila
DD D'Ila
10"
10
10
10-2
10-2
10-2
100
250
500
2000
4000
Number of epochs
Number of epochs
Number of epochs
(a) Learning rate 10-2.
(b) Learning rate 10-3
(c) Learning rate 10-1.
Figure 14: Dictionary learning for various learning rates when T = 100.
Dictionary initialization We conducted similar experiments to Figures 4b and 4c. We let n 10,000, m 100, and 100. We generated an orthogonal D The sparse codes x" are 5-sparse and their amplitudes are drawn from sub-Gaussian (O0, 1). We set X = 0.05, and a = 0.2. We used the pairwise method proposed by Arora et al. (2015) to initialize the dictionary. This close initialization resulted in a dictionary that provides support recovery prior training. Figure 15 shows successful dictionary learning where ge converges to closer neighbourhood of D than the other two gradients. We used linear assignment optimization (i.e., scipy. cettyy..ptimize... linear linear_sum, __m___sum____ assignment) to find the correct column permutations before computing the dictionary distance error.
10a
108
10
10
ID D' 1
10
10-2
10-3
10
50
100
150
200
250
0
50
100
150
200
250
Number of epochs
Number of epochs
(a) T 25.
(b) T 100.
Figure 15: Dictionary learning when is initialized using the pairwise method Arora et al. (2015).
C.2
Dictionary learning
Dataset We generated n= 50,000 samples following (2). We let m= 1000 and p= p= 1500, and sample D* from zero-mean Gaussian distribution, and then normalized the columns. The sparse codes z are 10, 20, 40-sparse, where their supports are chosen uniformly at random and amplitudes are sampled from Uniform(1,2),
Training The ionary is initialized to D= = D + with B ~ N' I) where nn R 1/log m. We let = 0.2, and a = 0.2, and T 100. The network trained for 1,000 iterative updates with batch-size of using Adam (Kingma Ba, 2014) with learning rate of 10-3 and <= 10-3 3 For decay method, is decreased in value by 0.005 every 100 update iterations. Each filter is normalized after every update. The learned dictionary is evaluated based the relative error D DD-D"lla/ppr 1a/111"
48

---

Published in Transactions on Machine Learning Research (08/2022)
10
10
10
D-
10-
D-D'll
10
D-D'lls
10-3
mm
R
K gookke
10-5
10-3
kill
10-3
&
SPORCO
SPORCO
SPORCO
10
10
10-7
250
500
750
250
500
750
250
500
750
Number iterations
Number iterations
Number of iterations
(a) 10-sparse code.
(b) 20- sparse code.
(c) 40- 40-sparse code.
Figure 16: Dictionary learning convergence using ge compared to NOODL and SPORCO.
C.3 Image denoising
Training We trained PUDLE where the dictionary is convolutional with 64 filters of size x 9 and strides of 4. The encoder unrolls for T = 15, and the step size is set to = 0.1. Unlike the theoretical analysis where full-batch gradient descent is studied, the network is trained stochastically with Adam optimizer (Kingma & Ba, 2014) with a learning rate of 10-4 and = 10-3 for 250 epochs. At every training iteration, a random 129 x 129 patch cropped and zero-mean Gaussian noise with a standard deviation of 25 added. We utilize random horizontal and vertical flip for augmentation. We report results in terms of the peak signal-to-noiss ratio (PSNR). The standard deviation of the test PSNR across multiple noise realizations was lower than 0.02 dB for all the methods. Hence, only reported the mean PSNR of the test set.
C.4 Interpretable sparse coding and dictionary learning
We focused on digits of {0, 1, 2, 3, 3, 4} of MNIST. We set T' 15, x= 0.7, and The dictionary dimensions are 784 and p - 500. We trained the network for 200 epochs using Adam optimizer with learning rate of 10 and batch size of 32. For construction of G, w is set to 0.001. For Figure contributions using 6,000 randomly chosen training images. The Gram matrix used in Figure 9, is constructed by 6,000 training examples, and the reconstruction is from the 200 most contributed training images.
computed the image
49


Reviews:
[{"review_id": "jYwzjKxhWu", "reviewer": "Reviewer_i5t1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------\n\nThis paper proposes a new gradient estimator for the dictionary update. This gradient update seems promising when the observed data are actually a noiseless sparse combination of atoms.\n\nWeaknesses\n-----------------\n\n**Theoretical analysis is lacking**\n-  Theorem 4.1 and 4.2 show that under very restrictive conditions (among which, the fact that the current estimate of the dictionary $D$ and the true dictionary $D^*$ are close), then one iteration of ISTA is enough for support recovery. This is in starck contrast with what is usually observed in practice, and the well documented slow convergence and slow support identification of ISTA, see e.g. [1]. Also, these theorems are really local convergence theorems, since in practice one cannot get the guarantee that at initialisation, $D$ and $D^*$ are close \n- Theorem 4.6 is very confusing: what is the Jacobian of $z^*$ w.r.t. $D$? This is critical, and should be defined. I don't understand how $z^*$, the ground truth code, depends on the dictionary $D$. Also, *this theorem is never proven* in appendix !\n- The authors mention that their method solves some stability issues observed in [Malezieux et al. 2022], but in fact the authors just provide some very restrictive conditions upon which ISTA leads to support recovery and stable linear convergence, bypassing the problems raised by [Malezieux et al. 2022]. To me, the phrasing in most of the paper made it sound as if the authors proposed a new method to prevent this problem, not a simple setting in which these problems do not arise.\n\n- Assumption 3.5 assumes that after each dictionary update, the dictionary $D$ stays close enough from $D^*$. This is a strong assumption; one could imagine that drifting happens and that even though $D$ starts close from  $D^*$, it goes away from it during training. Such case is not investigated.\n\n- Lemma 3.1 looks suspicious to me: in the best scenario where $D^l = D^*$, then $D^l$ is only $\\mu$-incoherent since $D^*$ is $\\mu$-incoherent by definition. However, Lemma 3.1 would show that $D^l$ is rather $\\mu / \\sqrt{m}$-incoherent, which is stronger than $\\mu$-incoherence since $m \\geq 1$.\n\n**I found the paper quite confusing and hard to read**\n\n As examples:\n- the used gradients $g^{dec}, g^{ae-lasso}$ and $g^{ae- ls}$ are mentioned many times in the introduction but only properly defined on page 6.\n- In assumptions 3.4  / 3.5, it would be better to write the meaning of incoherence and closeness in the main text, since these definitions are short and easy to understand.\n- In assumption 3.5, $\\delta_0$ and the sequence $\\delta_l$ are not defined... They will be defined in hindsight in the next theorems.\n- P.5, the loss function was already defined\n- In algorithms 1 adn 2, there seem to be some projection step missing\n- In eq.7, $\\partial h(z)$ is a set, so how is it chosen when $z$ is 0?\n- In prop 4.1, thm 4.3 and thm 4.5, why even introduce $B$ if in the end the conditions in thm 4.1/4.2 allow to take $B=1$?\n- In eq.9, there are many higher order terms that can be safely removed, for instance $O(\\hat{\\delta}^2  + \\hat{\\delta}) = O(\\hat{\\delta})$.\n\n\n**Experiments are not conclusive** \n\n- The code, as it is shipped now, is impossible to run, since the datasets are not provided. Running e.g. ``python train.py with cfg.dictionarylearning_g_aelasso_T25`` gives  the error ``FileNotFoundError: [Errno 2] No such file or directory: '../data/simulated_data.pt'`` . \n- In the experiments, the dictionary $D$ is initialized as a small perturbation of the true dictionary $D^*$: what happens when the initialization is random (or when we don't know in advance $D^*$? -- which is the case on real data). Also, the authors only choose one learning rate for the experiments, while a grid-search would have been much more convincing to highlight the merits of the proposed method.\n- It would be preferable to have more experiments where the benefits of the proposed method are displayed.\n- In the denoising experiment, the authors mention \"$g^{ae-ls}$ uses full backpropagation and stays stable. This\nhighlights stability of the gradient computation as opposed to $g^{ae-lasso}$.\", but this instability is never demonstrated or shown here.\n- The denoising experiment is done only on a small dataset of 432 images. It would be more convincing if it where done on several datasets. Also, the authors should indicate the error bars in table 1, using different initializations for the dictionaries. \n- The authors only compare to two other DL methods, while there are numerous methods readily available (e.g. online DL in scikit-learn).\n- Section 5 is independent from the rest of the paper, and is a standard analysis of the learned atoms on the MNIST dataset. It is not connected to the results in the previous sections.\n\n[1] : Liang, Jingwei, Jalal Fadili, and Gabriel Peyré. \"Local linear convergence of Forward--Backward under partial smoothness.\" Advances in neural information processing systems 27 (2014).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces puddle, which is a new way of updating the dictionary in dictionary learning. After the sparse coding phase, the gradient of the model w.r.t. the dictionary is estimated by doing backprop through the least-squares loss (this gradient is called $g^{ae-ls}$), and not through the standard lasso cost function of dictionary learning (the gradient is called $g^{ae-lasso}$), or the standard gradient of the loss with respect to the dictionary (called $g^{dec}$). \n\nThe authors also provide a theoretical framework around dictionary learning, where it is shown that under sufficient conditions, one step of ISTA with appropriate choice of soft thresholding level gives perfect support recovery, and then go on to show linear convergence of the method. The authors then analyse the error made using the different gradient estimators compared to the gradient at the optimum of the lasso $\\hat{z}$ and the gradient obtained at $z^*$, the true sparse code. The authors show that the proposed gradient $g^{ae-ls}$ is, under suitable hypothesis, a good estimator of the gradient obtained at $z^*$, while the other gradients are not. This justifies the approach of the paper.\n\nThese findings are demonstrated on some synthetic experiments, and on an image denoising task. They also provide a study of the learned atoms on MNIST.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- In my view, a proposition that shows that assumption 3.5 is somewhat realistic would greatly strengthen the paper : is it true that if $D$ is close to $D^*$ at initialization, then it stays close to $D^*$ during training?\n\n- Section 5 seems very disconnected from the rest of the paper, I think the paper would be strengthened if it were removed and instead replaced by more convincing and extensive experiments demonstrating the merits of the proposed method.", "broader_impact_concerns": "In my view, there is no broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CjeGblbQ6w", "reviewer": "Reviewer_JMW1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThey attempt to find the right recipe for unrolled dictionary learning. Their final inference on the gradient g-ae-ls  indicates that it is better to relegate the need for sparse regularization to the sparse coding steps rather and that upon the computation of a sparse code z_T, the quality of a current dictionary iterate is better captured by the least squares objective. \n\nWeaknesses:\n1) Although this is a theoretical paper, the requirement of infinite data and infinite unrolled iterations feels quite strong. Infinite data in essence removes the stochasticity in the learning task, and it is unclear what a back-propagated gradient is under infinite unrolled iterations… \n2) Theorem 4.1 makes high probability statements on the recover of the true code z*’s support. But the failure probability as stated is a fixed constant once the parameters - C_min , delta_l , p are fixed from the assumptions. A high probability bound requires a free parameter that can be varied such that the failure probability tends to 0 and in this case it is not so. Theorem 4.2 makes a similar probabilistic argument but here to set lambda_t appropriately requires knowledge of the true sparse code! Dependence of step-size or regularization hyperparameters on Lipschitz constants or diameters is relatively common but a direct dependence on the optimal sparse code is quite strong. This raises the following thought - does the procedure only give me convergence in high probability when the optimal sparse code is already known or should I treat this as an existence of unknowable optimal value for the parameter lambda_t ? \n3) The analysis is tailored to the lasso problem for sparse coding and they infer that l-1 regularization biases the recovery of the true sparse code. But couldn’t this be rectified by simply considering the l-0 regularized problem and IHT for sparse coding? \n4) Section 5 switches track to study the interpretability of learnt dictionaries. As a reader, this section appears to be completely independent of the remaining 90% of the paper. Theorem 5 for eg. is a straightforward inference from the stationary points. It would be great if the authors can clarify which part of Theorem 5 or their discussion thereafter is different for standard vs unrolled dictionary learning. \n5) Additionally for the statement (on page 11, left of Figure 6) - “this proves the dictionary span the training set”, did the authors perhaps mean to say “this proves the dictionary *lives in* the span of the training set” as indicated by Equation 11? Further, in Equation 12, is x^j an actual test image? Or is the reconstructed version of the test image using a learnt dictionary?  The latter is more understandable and in essence just says that dictionary representation of an image is a linear combination of dictionary atoms each of which depend on the actual data. In this case, it is unclear what the interpretation is on direct test images. For eg - Even for a test image from a different subspace, I can produce reconstructed image using the sparse codes and quantify the contribution of different training images towards this reconstruction. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper seeks to establish theoretical guarantees for unrolled dictionary learning. In unrolled dictionary learning, the gradients are back-propagated from the loss of the decoded sparse code through the algorithm that generated the sparse code. The task is global estimation, i.e. recovery of a ground truth dictionary D* and corresponding sparse code z*. \nThis paper has a few contributions. First, they establish sufficient conditions on the data distributions and the sparse coding parameters so that the “forward pass” i.e. sparse coding recovers the true code support.  Understandably success in recovering true sparse code relies on the proximity of the dictionary initialization to the optimum and a “bias” stemming from l-1 regularization\nSecond, they study three dictionary gradient variants using the final sparse code z_T- \n1) g-dec : a gradient computed using only the loss incurred by decoding z_T,  \n2) g-ae-lasso : a gradient computed by back-propagating the lasso loss through the unrolled layers. \n3) g-ae-ls : a gradient computed by back-propagating the least squares through the unrolled layers. \nThis paper analyzes the effectiveness of each of these gradients under a strong assumption - the existence of infinite sample data and the sparse code computed from infinite unrolled iterations. They conclude that the gradient g-ae-ls (in this infinite limit) is closer to the optimal gradient pointing directly to D*. \n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1) According to Definition A1. \\mu-incoherence implies inner-products are bounded by \\mu/\\sqrt{m}.\n2)  In Lemma 3.1, the inner-products of D^{l} are directly bounded by \\mu_l rather than \\mu_l/\\sqrt{m}. Please correct and adjust subsequent calculations. \n3) In equations 17, the index 1 on the LHS is in the superscript for z_1 and sub-script for z*. Please modify the notation to be consistent. \n4) Please differentiate the notation of the three gradients for the following two cases 1) finite sample 2) infinite samples. \n5) In equation 7, the indices on the gradient variants on the LHS should be T rather than t. \n6) Typo in Proposition 4.1 “supp(z_B)”  \n7) Is C_min strictly required to be greater than 0 ?\n8) Assumption 3.6 - for a unique minimizer the singleton set is exactly the argmin ? ", "broader_impact_concerns": "The paper does not have a broader impact statement. However, I don’t foresee any negative impact. \n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JwUAIuUSJD", "reviewer": "Reviewer_YjJw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors proposed the so-called PUDLE framework (a Provable Unrolled Dictionary LEarning). Under this framework, they provide conditions on network initialization and data distribution sufficient to recover and preserve the support of the latent code. They also provide theoretical analysis on the forward pass to characterize the behaviors of the code Jacobian, which are later used for the backward analysis. In the backward analysis, the authors analyzed the convergence of two alternative gradients using back-propagation along with the standard decoder gradient used in classic dictionary learning methods in the context of the unrolled dictionary, and provide results of convergence and biases towards the local gradient and global gradient.\n\nThe authors conduct experiments on synthesized data and real-image denoising data to verify their results and corroborated the effectiveness of their proposed PUDLE method with properly modified loss. The proposed method also enables them to interpret how each atom in the learned dictionary is related to samples in the training set and how to relate new testing samples with training data.\n\nThe math proof looks like no problem to me.\n\nOverall, I think this is a very nice piece of work that contains complete and deep analyses of the forward and backward passes in an unrolled dictionary learning framework, and that provides how the choices on the loss function, unrolling, and back-propagation influence the convergence of the learned dictionary. However, there are still a few issues that need to be fixed before acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript is focused on applying unrolled sparse coding to dictionary learning and investigates the bias of the sparse coding step and the gradient stability when using auto differentiation (back-propagation) to calculate the dictionary gradient. The motivation of this work is to address two limitations in the previous work: (1) the bias introduced by the sparse coding step with constant sparsity regularization; (2) the unstable gradient issue during back-propagation when updating the dictionary.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "In terms of issues, I think the major issue of this manuscript is the lack of discussions of many important assumptions for the theorems, especially on if and how they are (approximately, at least) satisfied in real-world problems.\n- The dictionary closeness assumption (Assumption 3.5) is an important one for the theorems proposed later. But without access to information about D^*, it could be hard to have such good initialization. In Appendix C.1 and C.2, the initialization of D is done by adding noises to the ground truth D^*. In the image denoising experiments (Appendix C.3), the detail of D initialization is dismissed. This needs to be discussed.\n- The decaying scheme of lambda_t in Theorem 4.2 also depends on the access to the ground-truth code z^*, which is not practical in real optimization. I understand that in practice maybe we can just decay lambda_t with a constant decaying ratio, as the authors have done in the experiments. But some discussions are needed.\n\nOverall, this work has done a great job providing clear and confusion-free definitions and notations. However, there are still some places that confuse me:\n- The biggest confusion to me is the definition of Code Jacobian (Def 4.1), where the Jacobian of z_t, J_t, is defined as \\partial{z_t} over \\partial{D}. It is natural to understand in terms of z_t because the calculation of z_6 depends on D according to the forward pass in eqn (6). But how should I understand \\hat{J} and J^* in Theorem 4.6, when \\hat{z} and z^* are just two objects that could be totally independent of D? At least, it does not make sense to me what they denote without specifying an algorithm to obtain them.\n- In the formulation of gradients during the backward pass in eqn (7), why the gradient of D depends on the lowercase t but not the uppercase T?\n\n\nMinor issues:\n- Page 1, 5th line to the last, \"... is ubiquitous is many other fields...\", the second \"is\" --> \"in \"\n- Page 7, last line of Theorem 4.1 about the sign consistency. Why a ReLU is used there? With ReLU, the signs on the left side of the equation are either 0 or +1, but on the other side, the signs of the ground-truth code z^* can definitely take -1, according to the code distribution assumption 3.3.\n- Page 10, Fig.5, the lines of NOODL and SPORCO are not quite distinguishable. I would suggest add markers to the lines for better visualization.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on applying unrolled sparse coding to dictionary learning and studies theoretical guarantees for unrolled dictionary learning, where the gradient with respect to the dictionary is estimated by doing back-propagation. The authors establish sufficient conditions on data distribution and dictionary so that the unrolled sparse coding recovers the true code support, and then shows dictionary error contracts at every back-propagation update. The work also proposes approaches to address (1) the bias introduced by the vanilla unrolled sparse coding and (2) the unstable gradient issue during back-propagation when updating the dictionary.

All the reviewers acknowledged the contribution of the analyses of the forward and backward passes in an unrolled dictionary learning framework. The reviewers pointed out comments on theoretical analysis (such as dictionary closeness and data assumption) as well as experiments. The authors submitted a revised version that addresses these comments and derives new theoretical results. All the reviews either lean toward or recommend accepting the paper. I agree with the reviewers to recommend accepting. 

There are additional minor comments recently posted by one reviewer, which can be addressed in the final version. I may agree with the reviewer that it makes more sense to require C_{min} nonnegative. Aside from the reason for making failure probability small, at the end of the proof of Theorem 4.1, the regularizer parameter \lambda_0 is chosen as C_{min}/2 which also implies that nonnegative C_{min}.
Decision: Accept (poster)