Title: Global Identifiability of  $\ell_1$-based Dictionary Learning via Matrix Volume Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: dictionary learning

matrix volume

nonconvex optimization
Abstract: We propose a novel formulation for dictionary learning that minimizes the determinant of the dictionary matrix, also known as its volume, subject to the constraint that each row of the sparse coefficient matrix has unit $\ell_1$ norm. The main motivation for the proposed formulation is that it provides global identifiability guarantee of the groundtruth dictionary and sparse coefficient matrices, up to the inherent and inconsequential permutation and scaling ambiguity, if a set of vectors obtained from the coefficient matrix lies inside the $\ell_\infty$ norm ball but contains the $\ell_2$ norm ball in their convex hull. Unlike existing work on identifiability of dictionary learning, our result is global, meaning that a globally optimal solution to our proposed formulation has to be a permuted and rescaled version of the groundtruth factors. Another major improvement in our result is that there is no additional assumption on the dictionary matrix other than it is nonsingular, unlike most other work that require the atoms of the dictionary to be mutually incoherent. We also provide a probabilistic analysis and show that if the sparse coefficient matrix is generated from the widely adopted Bernoulli-Gaussian model, then it is globally identifiable if the sample size is bigger than a constant times $k\log k$, where $k$ is the number atoms in the dictionary, with overwhelming probability. The bound is essentially the same as those local identifiability results, but we show that it is also global. Finally, we propose algorithms to solve the new proposed formulation, specifically one based on the linearized-ADMM with efficient per-iteration updates. The proposed algorithms exhibit surprisingly effective performance in correctly and efficiently recovering the dictionary, as demonstrated in the numerical experiments.

Global Identifiability of ℓ 1 -based Dictionary Learning via Matrix Volume Optimization

Abstract

We propose a novel formulation for dictionary learning that minimizes the determinant of the dictionary matrix, also known as its volume, subject to the constraint that each row of the sparse coefficient matrix has unit ℓ 1 norm. The main motivation for the proposed formulation is that it provides global identifiability guarantee of the groundtruth dictionary and sparse coefficient matrices, up to the inherent and inconsequential permutation and scaling ambiguity, if a set of vectors obtained from the coefficient matrix lies inside the ℓ ∞ norm ball but contains the ℓ 2 norm ball in their convex hull. Unlike existing work on identifiability of dictionary learning, our result is global, meaning that a globally optimal solution to our proposed formulation has to be a permuted and rescaled version of the groundtruth factors. Another major improvement in our result is that there is no additional assumption on the dictionary matrix other than it is nonsingular, unlike most other works that require the atoms of the dictionary to be mutually incoherent. We also provide a probabilistic analysis and show that if the sparse coefficient matrix is generated from the widely adopted Bernoulli-Gaussian model, then it is globally identifiable if the sample size is bigger than a constant times 𝑘 log 𝑘 , where 𝑘 is the number of atoms in the dictionary, with overwhelming probability. The bound is essentially the same as those local identifiability results, but we show that it is also global. Finally, we propose algorithms to solve the new proposed formulation, specifically one based on the linearized-ADMM with efficient per-iteration updates. The proposed algorithms exhibit surprisingly effective performance in correctly and efficiently recovering the dictionary, as demonstrated in the numerical experiments.

1 Introduction

Dictionary learning is a famous unsupervised learning problem [Toˇ si´ c and Frossard, 2011] that also goes by the name sparse coding [Olshausen and Field, 1997] or sparse component analysis [Georgiev et al., 2005]. It stems from the intuition that samples from a data set can all be approximately represented as a sparse combination of atoms from a complete or overcomplete dictionary, or 𝒙 𝑖 = 𝑨𝒔 𝑖 for 𝑖 = 1 , . . . , 𝑛 , where 𝒙 𝑖 ∈ R 𝑚 are the given data points, columns of 𝑨 ∈ R 𝑚 × 𝑘 represents atoms of a dictionary, and 𝒔 𝑖 ∈ R 𝑘 are the sparse representations of 𝒙 𝑖 , indicating that 𝒙 𝑖 is linearly dependent on only a few atoms from the dictionary 𝑨 with coefficients given in the nonzeros of 𝒔 𝑖 . Staking 𝒙 𝑖 as columns of 𝑿 ∈ R 𝑚 × 𝑛 and 𝒔 𝑖 as columns of 𝑺 ∈ R 𝑘 × 𝑛 , it can be written as a matrix factorization model 𝑿 = 𝑨𝑺 . Although in some cases there exist known dictionaries that admit good-quality sparse representations, such as the DCT or wavelet basis for image compression, sometimes it is better to learn a sparse-inducing dictionary from data in an unsupervised fashion, hence the task of dictionary learning that tries to estimate 𝑨 and 𝑺 jointly. In most cases, people

assume 𝑚 ≤ 𝑘 : the dictionary is complete if 𝑚 = 𝑘 , and overcomplete if 𝑚 < 𝑘 . In this paper we focus on the complete dictionary case when 𝑨 is square and nonsingular, which is common in a lot of prior works that this work is built upon [Gribonval and Schnass, 2010, Sun et al., 2016a,b, Wu and Yu, 2017, Wang et al., 2020]

Dictionary learning has found numerous applications in signal denoising [Elad and Aharon, 2006], audio coding [Plumbley et al., 2009], and medical imaging [Toˇ si´ c et al., 2010], to name just a few. On the theory side, most of the existing works have focused on algorithm design. Famous algorithms include 𝑘 -SVD [Aharon et al., 2006a] and online dictionary learning [Mairal et al., 2009], among numerous other algorithms based on generic nonconvex algorithm design with guarantee of convergence to a stationary point. More recently, there have appeared a line of research that attempts to show global optimality for dictionary learning under more restrictive assumptions, such as [Spielman et al., 2012, Agarwal et al., 2016, Arora et al., 2014, 2015, Sun et al., 2016a,b, Rambhatla et al., 2019, Bai et al., 2019, Zhai et al., 2020a,b, Shen et al., 2020, Tolooshams and Ba, 2022].

1.1 Model identifiability

The dictionary learning problem can be considered an instance of the broader matrix factorization model 𝑿 = 𝑨𝑺 subject to the opaque requirement that 𝑺 is sparse. It is well-known that matrix factorization without any additional assumptions on the latent factors is not unique, since we can always 'insert' an invertible matrix 𝑸 and 𝑸 -1 as 𝑿 = e 𝑨 e 𝑺 where e 𝑨 = 𝑨𝑸 and e 𝑺 = 𝑸 -1 𝑺 , and one cannot distinguish whether 𝑺 or e 𝑺 are the groundtruth sources. Such rotation ambiguity cannot be resolved by the well-known principal component analysis (PCA) [Jolliffe, 2002].

On the other hand, most matrix factorization models allow scaling and permutation ambiguity, i.e., a permutation matrix 𝜫 and a diagonal matrix 𝑫 such that the recovered mixing matrix is 𝑨𝑫𝜫 and the recovered sources are 𝜫 ⊤ 𝑫 -1 𝑺 . In addition to inherent constraints that fit the application, it is sometimes helpful to have an optimization criterion in search for the 'correct' factorization (up to scaling and permutation). In light of this, we formally define the identifiability of any matrix factorization model as follows:

Definition 1.1 (Identifiability) . Consider the generative model 𝑿 = 𝑨 ♮ 𝑺 ♮ , where 𝑨 ♮ and 𝑺 ♮ are the groundtruth latent factors. Let ( 𝑨 ★ , 𝑺 ★ ) be optimal for an identification criterion 𝑞

where A and S are some constraint set that 𝑨 and 𝑺 should belong to, respectively. If 𝑨 ♮ and/or 𝑺 ♮ satisfy some condition such that for any ( 𝑨 ★ , 𝑺 ★ ) , there exist a permutation matrix 𝜫 and a diagonal matrix 𝑫 such that 𝑨 ♮ = 𝑨 ★ 𝑫𝜫 and 𝑺 ♮ = 𝜫 ⊤ 𝑫 -1 𝑺 ★ , then we say that the matrix factorization model is essentially identifiable, up to permutation and scaling, under that condition.

For dictionary learning in specific, where the essential constraint on the latent factors is sparsity on 𝑺 , there have been some existing works on its identifiability issue. It has been shown that simply by putting a hard sparsity constraint on 𝑺 is enough to guarantee a unique factorization, provided that the sample size 𝑛 is large enough [Aharon et al., 2006b, Hillar and Sommer, 2015, Cohen and Gillis, 2019]. The main drawback is that the required sample size 𝑛 is usually too large to be practical-results from [Aharon et al., 2006b] and [Hillar and Sommer, 2015] both require 𝑛 to be 𝑂 (( 𝑘 + 1 ) GLYPH<0> 𝑘 𝑠 GLYPH<1> ) , where 𝑠 is the number of nonzeros that each column of 𝑺 is allowed to have; it has been relaxed to 𝑂 ( 𝑘 3 /( 𝑘 -𝑠 ) 2 ) by Cohen and Gillis [2019], but still very large. The existence of 𝑠 in the bound also shows that an outlier sample that does not admit a sparse representation from the dictionary is absolutely not allowed in the data set, which is also restrictive. Inspired by the success of compressive sensing and sparse recovery, it is perhaps more common to use the ℓ 1 norm to approximately sparsify 𝑺 . Identifiability results based on the ℓ 1 norm formulation has been predominantly local, meaning the model is identifiable within a neighborhood of the groundtruth factors ( 𝑨 ♮ , 𝑺 ♮ ) , but on the bright side the sample size requirement is typically down to 𝑂 ( 𝑘 log 𝑘 ) and allows the existence of dense outliers [Gribonval and Schnass, 2010, Wu and Yu, 2017, Wang et al., 2020].

1.2 This paper

In this paper, we intend to bridge the gap by providing global identifiability guarantee of complete dictionary recovery using a ℓ 1 norm based formulation, given as follows:

To the best of our knowledge, this is a novel formulation that has never appeared in the literature; however, as we will show in the next section, the intuition stems from the more commonly used ℓ 1 -based formulations. We stress that the formulation is ℓ 1 -based, as the ℓ 1 norm is the most widely used term to promote sparsity of the solution. Although in the context of dictionary learning, using the ℓ 1 norm as the convex surrogate for sparsity is not enough to make the overall problem tractable, there are still benefits of using it instead of other ones such as hard sparsity constraints, for example in terms of sample complexity as introduced in the previous subsection.

After some gentle derivation to motivate the proposed new formulation (1) for dictionary learning, we provide the following contributions:

We give a deterministic characterization of the identifiability condition on the sparse coefficient matrix 𝑺 under which the dictionary (as well as the sparse coefficients) can be uniquely determined, up to the inherent and inconsequential permutation and scaling ambiguity, via solving (1). Unlike existing work that shows local identifiability, our result is global, meaning that the groundtruth factors have to be a global minimizer to (1). Furthermore, perhaps surprisingly, our result only require the complete dictionary to be nonsingular-there are no additional assumptions that require the atoms to be mutually incoherent whatsoever.

The result shows that ℓ 1 -based dictionary learning is globally identifiable if a certain set of points, which are by definition inside the hypercube [-1 , 1 ] 𝑘 , contains the unit ball in their convex hull. This geometric assumption is called 'sufficiently scattered', with similar ideas appearing in pertinent unsupervised learning models. We provide intuition why this assumption generally leads to a sparse 𝑺 . We further show that if the sparse coefficient follows the widely used Bernoulli-Gaussian generative model, then it is identifiable with very high probability, provided the sample size 𝑛 is bigger than a constant times 𝑘 log ( 𝑘 ) . This is the same sample complexity bound obtained in some of the local identifiability analysis, but we show that it is also global, using the proposed new formulation (1).

We propose algorithms to solve (1), specifically one based on linearized-ADMM in the main paper; in the supplementary we also present one based on the Frank-Wolfe framework and the other based on block coordinate descent. Even though the proposed formulation (1) is nonconvex (just like any other dictionary learning formulation), numerical experiments on synthetic data show that they achieve global optimality in almost all cases, provided that the groundtruth latent factors satisfy the identifiability assumptions given before.

2 Problem Formulation

We start by proposing a novel formulation for dictionary learning. The purpose is to show that the seemingly different formulation does not come out of thin air, but is highly related, if not equivalent, to existing ones that have been studied extensively. Consider the following formulation adopted in [Gribonval and Schnass, 2010, Geng and Wright, 2014, Wu and Yu, 2017, Wang et al., 2020]:

We use the noiseless model 𝑿 = 𝑨𝑺 since the main focus of this work is identifiability, and a lot of algorithms are designed by replacing this term with a loss function that measures the residual 𝑿 -𝑨𝑺 , e.g. [Gribonval et al., 2015]. The purpose of the ℓ 1 norm of 𝑺 is to encourage it to be sparse, but since it is also dependent on the scaling of 𝑺 , the constraint ∥ 𝑨 : ,𝑗 ∥ ≤ 1 is imposed onto the columns of 𝑨 . Such constraints are without loss of generality due to the scaling ambiguity 𝑨𝑺 = 𝑨𝑫𝑫 -1 𝑺 , so for any admissible factorization, we can rescale the columns of 𝑨 to unit norm, while absorbing the scaling into the rows of 𝑺 , without affecting the matrix product.

Due to the Lagrangian multiplier theorem, there exist positive scalars 𝛽 1 , . . . , 𝛽 𝑘 and 𝛾 1 , . . . , 𝛾 𝑘 such that any solution of the following problem is also optimal for (2)

since they would lead to exactly the same Lagrangian function with appropriate choices of 𝛾𝑗 and 𝛽𝑗 . Again, due to the inherent scaling ambiguity, we can assume without loss of generality that 𝛽 1 = · · · = 𝛽𝑘 = 1 , leaving the unknown parameters to just 𝛾𝑗 's.

If we replace the summation in the objective of (3) with product, or equivalently replacing the arithmetic mean with the geometric mean, the term ˛ 𝑗 𝛾𝑗 ∥ 𝑨 : ,𝑗 ∥ would be intimately related to | det 𝑨 | if 𝑨 is square. Indeed, there must exist some unknown matrix 𝑮 such that | det 𝑮𝑨 | = ˛ 𝑗 𝛾𝑗 ∥ 𝑨 : ,𝑗 ∥ . Furthermore, for the purpose of optimization it is completely unnecessary to know 𝑮 since | det 𝑮𝑨 | = | det 𝑮 | | det 𝑨 | , which is just a positive scaling of | det 𝑨 | that does not affect the optimization solution. As a result, our proposed novel formulation of dictionary learning takes the form of (1).

Although our paper focuses on the case of complete dictionaries, i.e., when 𝑨 is square and nonsingular, they can be effortlessly extended to the overdetermined case when 𝑨 is tall, and the only modification is to change the objective to det 𝑨 ⊤ 𝑨 . This is sometimes called the volume of the matrix [Ben-Israel, 1992] with applications in change-of-variable integration [Ben-Israel, 1999] and probability [Ben-Israel, 2000]. It has been used as the identification criterion for several variants of nonnegative matrix factorization with an interesting geometric interpretation of finding the minimum volume enclosing simplex of a set of points [Lin et al., 2015, Huang et al., 2016, 2018, Fu et al., 2018, Huang and Fu, 2019]. The most related work is the polytopic matrix factorization framework proposed by Tatli and Erdogan [2021], which includes a case that constraints the columns of 𝑺 to have bounded ℓ 1 norm. Our proposed formulation constrains the rows of 𝑺 , which seems to be a minor difference but is in fact a great leap forward since row scaling of 𝑺 is inherent for any matrix factorization, while column scaling is not. It also turns out that the identifiability analysis would be drastically different, albeit the formulations look similar.

3 Global identifiability

In this section we study the conditions under which the proposed dictionary learning formulation (1) is identifiable, i.e., any solution to (1) equals to the groundtruth dictionary 𝑨 ♮ and sparse coefficients 𝑺 ♮ up to permutation and scaling, as per Definition 1.1. The groundtruth 𝑺 ♮ can have arbitrary scaling, and it is without loss of generality to assume that its rows have unit ℓ 1 norms; we denote such rescaled 𝑺 ♮ to be e 𝑺 ♮ , i.e., ∥ e 𝑺 ♮ 𝑗 , : ∥ 1 ≤ 1 , 𝑗 = 1 , . . . , 𝑘 ; we absorb the counterscaling into the columns of 𝑨 ♮ to generate e 𝑨 ♮ , so that 𝑿 = 𝑨 ♮ 𝑺 ♮ = e 𝑨 ♮ e 𝑺 ♮ .

Since we consider only complete dictionaries, for any feasible ( 𝑨 , 𝑺 ) for (1), there must exist a nonsingular matrix 𝑾 such that 𝑺 = 𝑾 e 𝑺 ♮ and 𝑨 = e 𝑨 ♮ 𝑾 -1 , therefore Problem (1) is, conceptually, equivalent to maximize | det 𝑾 | subject to ∥ 𝒘 ⊤ 𝑗 e 𝑺 ♮ ∥ 1 ≤ 1 , 𝑗 = 1 , . . . , 𝑘 , (4)

where 𝒘 ⊤ 𝑗 denotes the 𝑗 th row of 𝑾 . Since 𝑾 = 𝑨 -1 e 𝑨 ♮ and e 𝑨 ♮ is feasible, the optimal value of (4) is at least 1. On the other hand, if the model is identifiable, then the optimal value of (4) should be exactly equal to 1, and the optimal solution to (4) must be a permutation matrix times a diagonal ± 1 matrix. The rest of this section is dedicated to characterizing when such condition holds for the sparse coefficients 𝑺 . Notice that all we need for the dictionary 𝑨 is that its columns are independent, so that the transform from (1) to (4) is possible-there is no additional assumptions such as 'incoherence' for the rest of the arguments to hold, which is a great improvement to the existing works such as [Gribonval and Schnass, 2010, Sun et al., 2016a,b, Wu and Yu, 2017, Wang et al., 2020].

3.1 Deterministic characterization

The intuition is simple: the Hadamard inequality states that | det 𝑾 | ≤ ˛ 𝑗 ∥ 𝒘 𝑗 ∥ , so a sufficient condition for identifiability is that the set of solutions to the following (nonconvex) quadratic programming

is {± 𝒆 1 , . . . , ± 𝒆 𝑘 } ; in other words, a maximizer must be an indicator vector or its negative.

To proceed, we introduce the following notion:

Assumption 3.1 (Sufficiently scattered in the hypercube) . Let B denote the Euclidean ball B = { 𝒙 ∈ R 𝑘 | ∥ 𝒙 ∥ ≤ 1 } and C denote the hypercube C = { 𝒙 ∈ R 𝑘 | ∥ 𝒙 ∥∞ ≤ 1 } . A set S is sufficiently scattered in the hypercube if:

B ⊆ S ⊆ C ;

𝜕 B ∩ 𝜕 S = {± 𝒆 1 , . . . , ± 𝒆 𝑘 } , where 𝜕 denotes the boundary of the set.

A geometric illustration of a set that is sufficiently scattered in the hypercube is shown in Figure 1, which in this case is a polytope with vertices depicted as dots. As we can see, B is a subset of the hypercube [-1 , 1 ] 𝑘 , but touches the boundary of [-1 , 1 ] 𝑘 at points ± 𝒆 1 , . . . , ± 𝒆 𝑘 . If a set S is sufficiently scattered in [-1 , 1 ] 𝑘 , S contains B as a subset and, as a second requirement, B touches the boundary of S only at those points too.

The term 'sufficiently scattered' was first coined by Huang et al. [2016] to characterize the identifiability condition for nonnegative matrix factorization that has already appeared in [Huang et al., 2013]. The difference is that in [Huang et al., 2013, 2016, Fu et al., 2018], the condition is defined over the conic hull of a set of points in the nonnegative orthant containing a specific hyperbolic cone. It has also been defined over the convex hull of a set of points in the probability

x

x

x

x

x

x

x

x

x

x

x

x

simplex [Fu et al., 2015, Lin et al., 2015]. The most related case is the sufficiently scattered condition defined for the hypercube ( ℓ ∞ -norm ball) [Tatli and Erdogan, 2021], as well as the standard orthoplex ( ℓ 1 -norm ball).

To see how the sufficiently scattered condition could be useful for dictionary learning, we first rewrite the single constraint in (5) as, equivalently, a set of 2 𝑛 linear inequalities 𝒘 ⊤ e 𝑺 ♮ 𝒗 ≤ 1 , ∀ 𝒗 ∈ {± 1 } 𝑛 . Since the rows of e 𝑺 ♮ have been rescaled to have unit ℓ 1 norm, we must have that ∥ e 𝑺 ♮ 𝒗 ∥∞ ≤ 1 , meaning that all vectors e 𝑺 ♮ 𝒗 should lie in the hypercube [-1 , 1 ] 𝑘 , and we further assume the convex hull of all such e 𝑺 ♮ 𝒗 is sufficiently scattered. Define the 𝑛 × 2 𝑛 matrix 𝑽 that contains all elements of {± 1 } 𝑛 as its columns, we have the following deterministic characterization of the identifiability of (1):

Theorem 3.2. Consider the generative model 𝑿 = 𝑨 ♮ 𝑺 ♮ , where 𝑨 ♮ ∈ R 𝑘 × 𝑘 is the groundtruth dictionary and 𝑺 is the groundtruth sparse coefficients. If rank ( 𝑨 ♮ ) = 𝑘 and conv ( 𝑺 ♮ 𝑽 ) is sufficiently scattered in the hypercube as in Assumption 3.1, then for any solution of (1) , denoted as ( 𝑨 ★ , 𝑺 ★ ) , there exist a permutation matrix 𝜫 and a diagonal matrix 𝑫 such that 𝑨 ♮ = 𝑨 ★ 𝑫𝜫 and 𝑺 ♮ = 𝜫 ⊤ 𝑫 -1 𝑺 ★ . In other words, dictionary learning is identifiable if the groundtruth 𝑨 ♮ has full column rank and 𝑺 ♮ 𝑽 is sufficiently scattered.

The full proof is relegated to Appendix A. Here we provide a concise sketch of the proof: if e 𝑺 ♮ 𝑽 is sufficiently scattered, a solution to (5) must be ± 𝒆 𝑗 ; as a result, | det 𝑾 | ≤ ˛ 𝑗 ∥ 𝒘 𝑗 ∥ ≤ 1 , and equality holds only if 𝑾 is a permutation matrix with some sign flips of its rows.

How is this assumption relevant to the sparsity of 𝑺 ? Notice that if 𝑺𝑽 is sufficiently scattered, then for any vector 𝒖 such that ∥ 𝒖 ∥ ≤ 1 , there must exist a convex coefficient 𝜽 , i.e., 𝜽 ≥ 0 and 1 ⊤ 𝜽 = 1 , such that 𝒖 = 𝑺𝑽 𝜽 . Due to the definition of 𝑽 , we know that the vector 𝒚 = 𝑽 𝜽 must satisfy ∥ 𝒚 ∥∞ ≤ 1 . Let us instantiate it at 𝒖 = 𝒆 𝑗 . This means the 𝑗 th row of 𝑺 inner product with 𝒚 equals to 1. But the 𝑗 th row of 𝑺 has unit ℓ 1 norm and ∥ 𝒚 ∥∞ ≤ 1 , so their inner product is 1 only if 𝒚 ⊤ = sign ( 𝑺 𝑗 , : ) , which is uniquely defined if 𝑺 𝑗 , : is completely dense. Now 𝒆 𝑗 = 𝑺𝒚 also implies the inner products between 𝒚 and the rest of the rows of 𝑺 all equal to zero, which is almost impossible if 𝒚 ⊤ = sign ( 𝑺 𝑗 , : ) is the only candidate. On the other hand, if some of the entries in 𝑺 𝑗 , : are zeros, then the corresponding entries in 𝒚 are free to choose any value in [-1 , 1 ] ; to make the rest of the 𝑘 -1 inner products equal to zero, we generically need at least 𝑘 -1 free variables in 𝒚 , meaning 𝑺 𝑗 , : contains at least 𝑘 -1 zeros-it is still not guaranteed that ∥ 𝒚 ∥∞ ≤ 1 , but it intuitively makes sense that the more zeros each row of 𝑺 contains, the more likely that 𝒆 𝑗 = 𝑺𝒚 is feasible, which is necessary for 𝑺𝑽 to be sufficiently scattered.

3.2 Probabilistic characterization

The analysis given in §3.1 gives an exact characterization of when dictionary learning is identifiable using the proposed formulation (1) with a matrix volume identification criterion. However, given some sparse coefficient matrix 𝑺 , checking whether it satisfies the identifiability condition in Theorem 3.2 amounts to solving a nonconvex quadratic programming (5), which is NP-hard in general. In this section, we assume that the sparse coefficient matrix 𝑺 is generated from a probabilistic generative model, specifically the Bernoulli-Gaussian model that has been widely considered in the literature, such as [Gribonval and Schnass, 2010, Sun et al., 2016a], and show that it would satisfy guarantee identifiability with high probability, provided that the number of data points 𝑛 is 𝑂 ( 𝑘 log 𝑘 ) .

Assumption 3.3 (Bernoulli-Gaussian model) . The matrix 𝑺 ∈ R 𝑘 × 𝑛 is generated from a BernoulliGaussian model with parameter 𝑝 ∈ ( 0 , 1 ) , denoted as 𝑺 ∼ BG( 𝑝 ) , if its elements are i.i.d. with 𝑆𝑖𝑗 = 𝑏𝑖𝑗 𝑔 𝑖 𝑗 , where 𝑏𝑖𝑗 ∈ { 0 , 1 } are i.i.d. Bernoulli random variables with Pr [ 𝑏𝑖𝑗 = 1 ] = 𝑝 (thus Pr [ 𝑏𝑖𝑗 = 0 ] = 1 -𝑝 ) and 𝑔𝑖𝑗 ∼ N( 0 , 1 ) are i.i.d. standard normal random variables.

Note that due to the scaling ambiguity of matrix factorization the assumption that the normal random variable has unit variance is without loss of generality-its rows will be rescaled to have unit ℓ 1 norms nonetheless. What we are essentially assuming is that 1) a significant portion of 𝑺 are zero; 2) nonzeros of 𝑺 have zero mean and approximately the same variance.

Theorem 3.4. Suppose 𝑺 ∈ R 𝑘 × 𝑛 is generated from the Bernoulli-Gaussian model BG( 𝑝 ) , and e 𝑺 is obtained by scaling its rows to have unit ℓ 1 norm. Then

Corollary 3.5. Consider the generative model 𝑿 = 𝑨 ♮ 𝑺 ♮ , where 𝑨 ♮ ∈ R 𝑘 × 𝑘 is the groundtruth dictionary and 𝑺 is the groundtruth sparse coefficients. If rank ( 𝑨 ♮ ) = 𝑘 and the matrix 𝑺 ♮ ∈ R 𝑘 × 𝑛 is generated from the Bernoulli-Gaussian model BG( 𝑝 ) , then ( 𝑨 ♮ , 𝑺 ♮ ) are globally identifiable via optimizing (1) with probability at least 1 -4 exp ( 𝑘 log ( 𝑘 ) -𝑛𝑝 ( 1 -𝑝 )) .

The proof of Theorem 3.4 is relegated to Appendix B. It is appealing to see that the presented result looks much cleaner than prior work such as [Gribonval and Schnass, 2010, Wu and Yu, 2017, Wang et al., 2020], on top of the fact that such identifiability result is global , and does not require any incoherence assumption on the dictionary, as long as it is nonsingular. An interesting observation is that most of the foundational results have appeared in the seminal work [Gribonval and Schnass, 2010]; the difference is that they are used in the improved formulation (1), and in return leads to more significant result in Corollary 3.5. From the above result, we observe the following:

The right-hand-side of (6) would be a valid probability ( < 1 ) if 𝑛 is bigger than a constant times 𝑘 log ( 𝑘 )/ 𝑝 ( 1 -𝑝 ) , and very quickly approaches zero as it grows bigger. This agrees with the prior work on local identifiability that the sample complexity is of the same order [Gribonval and Schnass, 2010, Geng and Wright, 2014, Wu and Yu, 2017, Wang et al., 2020].

The Bernoulli parameter 𝑝 cannot be exactly equal to 1 or 0 , which affirms the obvious fact that 𝑺 can neither be completely dense nor entirely zero. The more interesting implication is that DL is identifiable (with high probability) as long as there is some nonzero probability of having zeros in 𝑺 , no matter how small that probability is, provided the sample size is large enough.

4 Proposed algorithm

In this section, we propose two algorithms for the volume-optimization based formulation (1). We will reformulate (1) as a determinant maximization problem subject to linear constraints, and then apply the Frank-Wolfe algorithm or block coordinate descent, which are both guaranteed to converge to a stationary point.

Since we assume 𝑨 is nonsingular, we can define 𝑷 = 𝑨 -1 and apply a change of variable to problem (1): the objective would become minimizing 1 /| det 𝑷 | , which we apply the log function and make it -log det | 𝑷 | , and in the constraints we can now eliminate the 𝑺 variables by simply requiring ∥ 𝑷𝑿 𝑗 , : ∥ 1 ≤ 1 , 𝑗 = 1 , . . . , 𝑘 . This leads to the following reformulation

Problem (7) now has a convex, or more specifically linear, constraint set, although the objective is still not convex. In Appendix 4, we propose two algorithms to directly tackle this formulation, one based on the Frank-Wolfe framework, and the other is an instance of block coordinate descent. Both algorithms guarantee convergence to a stationary point, which is the best one can get for a generic nonconvex problem. However, both algorithms require solving a linear programming problem in each iteration, making the overall algorithm double-looped, and the time efficiency is greatly affected by the specific subroutine to solve linear programming. In the rest of this section, we propose an algorithm based on the linearized alternating direction method of multipliers (L-ADMM) with well-defined and low-complexity iterations.

4.1 Linearized alternating direction method of multipliers (L-ADMM)

We first modify the formulation (7) by introducing an auxiliary variable 𝑺 :

Formulation (8) consists of two sets of variables ( 𝑷 and 𝑺 ) over two separable functions (one of them being an indicator function of 𝑺 that the ℓ 1 norms of its rows are no bigger than 1) and linear equality constraints. It is easy to derive the alternating direction method of multipliers (ADMM) [Boyd et al., 2011] for (8):



The second step (9b) is well-defined, as it projects each row of the matrix 𝑷 ( 𝑡 + 1 ) 𝑿 + 𝑼 ( 𝑡 ) to the ℓ 1 norm ball, which can be efficiently computed with linear complexity [Duchi et al., 2008]. In our implementation we use the bisection method described in [Parikh and Boyd, 2014]. The first step (9a), however, is not clear how to compute. One popular method to mitigate this issue, which is prevalent in ADMM, is to take a linear approximation of the loss function of 𝑷 at the previous iterate 𝑷 ( 𝑡 ) [Lu et al., 2021]. The gradient of log | det 𝑷 | is 𝑷 -⊤ , therefore the linear approximation of -log | det 𝑷 | at 𝑷 ( 𝑡 ) is -log | det 𝑷 ( 𝑡 ) | -Tr 𝑷 -1 ( 𝑡 ) ( 𝑷 -𝑷 ( 𝑡 ) ) . The overall update of 𝑷 ( 𝑡 + 1 ) becomes minimizing a convex quadratic function, which can be done in closed-form. The derived linearized ADMM(L-ADMM) iterates are



Finally, we notice that formulation (8) and the derived L-ADMM algorithm are invariant under linear transformation of columns of 𝑿 , meaning if we replace 𝑿 with e 𝑿 = 𝑮𝑿 in (8), where 𝑮 is a 𝑘 × 𝑘 invertible matrix, then every iterate 𝑷 ( 𝑡 ) is uniquely mapped to 𝑷 ( 𝑡 ) 𝑮 -1 (while 𝑺 ( 𝑡 ) and 𝑼 ( 𝑡 ) are exactly the same), and the objective value has a constant difference -log | det 𝑷 ( 𝑡 ) 𝑮 -1 | = -log | det 𝑷 ( 𝑡 ) | + log | det 𝑮 | . This observation allows us to preprocess the data matrix 𝑿 by orthogonalizing its rows, so that the update for 𝑷 in (10a) can be further simplified. The overall algorithm is summarized in Algorithm 1. We empirically found that setting 𝜌 = 𝑛𝑘 works very well in practice.

Algorithm 1 Solving (7) with L-ADMM

1: take the QR factorization of 𝑿 ⊤ = 𝑸𝑹

2: set 𝜌 = 𝑛𝑘 and initialize 𝑷 ( 0 )

3: for 𝑡 = 0 , 1 , 2 , . . . until convergence do

6:

𝑼

(

𝑡

+

1

𝑼

(

𝑡

) +

7:

8: return 𝑷 ( 𝑡 ) 𝑹 -⊤

end for

𝑷

(

𝑡

+

1

)

𝑸

⊤ -

𝑺

(

𝑡

+

1

)

5 Experiments

We now provide some numerical experiments to showcase the effectiveness of the proposed volume optimization formulation using the two algorithms described in §4. All the experiments are conducted in MATLAB.

5.1 Optimization performance

We synthetically generate random problems, and show the surprisingly effective performance of the two algorithms proposed in §4. For 𝑘 = 20 and 𝑛 = 1000 , we randomly generate the groundtruth sparse coefficient matrix 𝑺 ♮ according to the Bernoulli-Gaussian model with 𝑝 = 0 . 5 , and the groundtruth dictionary matrix 𝑨 ♮ completely random, and generate the data matrix 𝑿 = 𝑨 ♮ 𝑺 ♮ . Due to Theorem 3.4, we know it is identifiable with very high probability, despite 𝑛 being not that big compared to the number of atoms 𝑘 . Matrix 𝑿 is used as input to the L-ADMM algorithm as described in Alg. 1. Although Problem (7) is nonconvex, as long as it is identifiable, we know the global optimum is attained at 𝑷 ★ = 𝑫 ( 𝑨 ♮ ) -1 , where 𝑫 is a diagonal matrix with the 𝑖 th diagonal equal to the inverse of the ℓ 1 norm of the 𝑖 th row of 𝑺 ♮ . As a result, -log | det 𝑫 ( 𝑨 ♮ ) -1 | is the optimal value for Problem (7) as long as the model is identifiable, and we shall see whether the proposed algorithm is able to attain that optimal value. On the other hand, since L-ADMM directly tackles formulation 8, it is not guaranteed that 𝑷 ( 𝑡 ) is feasible in every iteration, which makes little sense to check the difference -log | det 𝑷 ( 𝑡 ) | + log | det 𝑷 ★ | . We instead check the optimality gap of the Lagrangian function values, using the optimal dual variable 𝜦 , since we have

for any 𝑷 and a feasible 𝑺 . Obviously, the gap equals to zero when 𝑷 = 𝑷 ★ and 𝑺 = 𝑺 ★ , in which case 𝑷 ★ 𝑿 -𝑺 ★ = 0 . Furthermore, it is easy to show that an optimal 𝜦 is ( e 𝑺 ♮ ) † . In this simulation with known groundtruth factors, we will use this to measure the optimality gap.

The convergence behavior of 100 random trials of the L-ADMM Algorithm 1 are shown in Figure 2. The somewhat surprising observation is that it achieves essentially zero optimality gap in all instances, which is somewhat surprising as we are, after all, trying to solve NP-hard problems. Among the 100 random trials, L-ADMM usually requires approximately 500-1000 iterations to converge; together with the low-complexity iterations described in Algorithm 1, it usually takes just a few seconds to execute a trial. For a dictionary learning problem of this size, our proposed algorithm takes a lot less time than most existing state-of-the-arts. Some more detailed comparisons can be found in the supplementary material.

5.2 Dictionary recovery

Next we fix the sample size 𝑛 = 1000 and vary the dictionary size as well as the sparsity parameter 𝑝 in the Bernoulli-Gaussian model and check how often do we get exact recovery of the groundtruth dictionary 𝑨 ♮ up to column permutation and scaling. In the previous experiment we simply check the optimality gap between the algorithm output and -log | det 𝑫 ( 𝑨 ♮ ) -1 | , which is also a pretty good indicator of exact recovery since the only ambiguity is an orthogonal rotation, which is very unlikely to exist. Nevertheless, to ensure definitively that the dictionary is recovered as 𝑷 -1 , we will normalize the column and use the Hungarian algorithm [Kuhn, 1955] to find the best column matching, and then calculate the estimation error. We declare success if the estimation error is smaller than 1e-5 . The results are shown in Figure 3, which agrees with the bound in Theorem 3.4.

0

200

400

600

800

1000

1200

iteration

10 -8

10 -6

10 -4

10 -2

10 0

10 2

optimality gap

k=5

k=10

k=15

k=30

k=35

k=40

k=45

p=0.9

p=0.8

p=0.7

p=0.6

p=0.5

p=0.4

p=0.3

p=0.2

p=0.1

0

0

0

0

0.1

0

0

0

0

0.1

0

0

0

0

0

0

0.2

0

0

0

0

0

0

0

0

0.7

1

1

1

1

1

1

1

1

0.9

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

0.8

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

We would like to emphasize that compared to similar figures that have appeared in the literature, the groundtruth dictionaries that we try to identify are far from being orthogonal, nor do we care to check whether they satisfy the incoherence assumption; moreover, the recovery is of the entire dictionary, not one of the atoms. More comparisons with some existing algorithms can be found in the supplementary.

5.3 Real data experiment

We conclude this section with a classical application of dictionary learning on image compression. For a given image, it is first divided into 8 × 8 non-overlapping patches, reshaped into a vector in R 𝑘 with 𝑘 = 64 , and stacked as columns of the data matrix 𝑿 . After feeding it into the L-ADMM Algorithm 1, the output is inverted to obtain an estimate of the dictionary 𝑨 . Each column of 𝑨 is then reshaped back into a 8 × 8 patch to be shown as an atom for the dictionary. For a monochrome image with 512 × 512 pixels, it would be reshaped into a data matrix 𝑿 of size 64 × 4096 . Thanks to the high-efficiency of the proposed L-ADMM algorithm, learning a dictionary on a data set of this size takes only a few minutes from a random initialization. The resulting dictionary learned from a natural image is shown in Figure 4. The result agrees with most other dictionary learning algorithms, with some atoms representing edges and bright spots, while some others resemble a few DCT bases.

6 Conclusion

In this paper we aimed at answering the question of when dictionary learning is identifiable, with a specific requirement that the formulation uses the ℓ 1 norm to promote sparsity. With a novel formulation that minimizes the volume of the dictionary subject to ℓ 1 constraints, we close the case with an affirmative yes, complete dictionary learning is indeed globally identifiable under mild conditions. Compared to similar works, our result is global, requires no conditions such as incoherence on the complete dictionary, and at the same time achieves the same sample complexity bound when the sparse coefficients satisfy the Bernoulli-Gaussian model. Finally, we propose algorithms to solve the volume optimization problem, specifically one based on linearized ADMM in the main paper, and demonstrate their exceptional performances on both synthetic and real data.

Acknowledgments and Disclosure of Funding

This work is supported in part by NSF ECCS-2237640.

References

Alekh Agarwal, Animashree Anandkumar, Prateek Jain, and Praneeth Netrapalli. Learning sparsely used overcomplete dictionaries via alternating minimization. SIAM Journal on Optimization , 26 (4):2775-2799, 2016.

Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing , 54(11): 4311-4322, 2006a.

Michal Aharon, Michael Elad, and Alfred M Bruckstein. On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them. Linear algebra and its applications , 416(1): 48-67, 2006b.

Sanjeev Arora, Rong Ge, and Ankur Moitra. New algorithms for learning incoherent and overcomplete dictionaries. In Conference on Learning Theory , pages 779-806. PMLR, 2014.

Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. In Conference on Learning Theory , pages 113-149. PMLR, 2015.

Yu Bai, Qijia Jiang, and Ju Sun. Subgradient descent learns orthogonal dictionaries. In International Conference on Learning Representations , 2019.

Adi Ben-Israel. A volume associated with mxn matrices. Linear algebra and its applications , 167: 87-111, 1992.

Adi Ben-Israel. The change-of-variables formula using matrix volume. SIAM Journal on Matrix Analysis and Applications , 21(1):300-312, 1999.

Adi Ben-Israel. An application of the matrix volume in probability. Linear Algebra and its Applications , 321(1-3):9-25, 2000.

George Bennett. Probability inequalities for the sum of independent random variables. Journal of the American Statistical Association , 57(297):33-45, 1962.

Dimitri P. Bertsekas. Nonlinear Programming . Athena Scientific, 2 edition, 1999.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122, 2011.

Jeremy E Cohen and Nicolas Gillis. Identifiability of complete dictionary learning. SIAM Journal on Mathematics of Data Science , 1(3):518-536, 2019.

John Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the ℓ 1 -ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning , pages 272-279, 2008.

Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing , 15(12):3736-3745, 2006.

Xiao Fu, Wing-Kin Ma, Kejun Huang, and Nicholas D Sidiropoulos. Blind separation of quasistationary sources: Exploiting convex geometry in covariance domain. IEEE Transactions on Signal Processing , 63(9):2306-2320, 2015.

Xiao Fu, Kejun Huang, and Nicholas D Sidiropoulos. On identifiability of nonnegative matrix factorization. IEEE Signal Processing Letters , 25(3):328-332, 2018.

Quan Geng and John Wright. On the local correctness of l1-minimization for dictionary learning. In 2014 IEEE International Symposium on Information Theory , pages 3180-3184. IEEE, 2014.

Pando Georgiev, Fabian Theis, and Andrzej Cichocki. Sparse component analysis and blind source separation of underdetermined mixtures. IEEE transactions on neural networks , 16(4):992-996, 2005.

R´ emi Gribonval and Karin Schnass. Dictionary identification--sparse matrix-factorization via ℓ 1 -minimization. IEEE Transactions on Information Theory , 56(7):3523-3539, 2010.

R´ emi Gribonval, Rodolphe Jenatton, Francis Bach, Martin Kleinsteuber, and Matthias Seibert. Sample complexity of dictionary learning and other matrix factorizations. IEEE Transactions on Information Theory , 61(6):3469-3486, 2015.

Christopher J Hillar and Friedrich T Sommer. When can dictionary learning uniquely recover sparse data from subsamples? IEEE Transactions on Information Theory , 61(11):6290-6297, 2015.

Kejun Huang and Xiao Fu. Detecting overlapping and correlated communities without pure nodes: Identifiability and algorithm. In International Conference on Machine Learning , pages 2859-2868, 2019.

Kejun Huang, Nicholas D Sidiropoulos, and Ananthram Swami. Non-negative matrix factorization revisited: Uniqueness and algorithm for symmetric decomposition. IEEE Transactions on Signal Processing , 62(1):211-224, 2013.

Kejun Huang, Xiao Fu, and Nikolaos D Sidiropoulos. Anchor-free correlated topic modeling: Identifiability and algorithm. Advances in Neural Information Processing Systems , 29, 2016.

Kejun Huang, Xiao Fu, and Nicholas D. Sidiropoulos. Learning hidden Markov models from pairwise co-occurrences with application to topic modeling. In International Conference on Machine Learning (ICML) , pages 2073-2082, 2018.

Ian T Jolliffe. Principal Component Analysis . Springer, 2 edition, 2002.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

Chia-Hsiang Lin, Wing-Kin Ma, Wei-Chiang Li, Chong-Yung Chi, and ArulMurugan Ambikapathi. Identifiability of the simplex volume minimization criterion for blind hyperspectral unmixing: The no-pure-pixel case. IEEE Transactions on Geoscience and Remote Sensing , 53(10):5530-5546, 2015.

Songtao Lu, Jason D Lee, Meisam Razaviyayn, and Mingyi Hong. Linearized admm converges to second-order stationary points for non-convex problems. IEEE Transactions on Signal Processing , 69:4859-4874, 2021.

Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In Proceedings of the 26th annual international conference on machine learning , pages 689-696, 2009.

Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research , 37(23):3311-3325, 1997.

Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends® in Optimization , 1 (3):127-239, 2014.

Gilles Pisier. The volume of convex bodies and Banach space geometry , volume 94. Cambridge University Press, 1999.

Mark D Plumbley, Thomas Blumensath, Laurent Daudet, R´ emi Gribonval, and Mike E Davies. Sparse representations in audio and music: from coding to source separation. Proceedings of the IEEE , 98 (6):995-1005, 2009.

Sirisha Rambhatla, Xingguo Li, and Jarvis Haupt. Noodl: Provable online dictionary learning and sparse coding. In International Conference on Learning Representations , 2019.

R Tyrrell Rockafellar. Convex Analysis , volume 18. Princeton University Press, 1970.

Yifei Shen, Ye Xue, Jun Zhang, Khaled Letaief, and Vincent Lau. Complete dictionary learning via lp-norm maximization. In Conference on Uncertainty in Artificial Intelligence , pages 280-289. PMLR, 2020.

Daniel A. Spielman, Huan Wang, and John Wright. Exact recovery of sparsely-used dictionaries. In Shie Mannor, Nathan Srebro, and Robert C. Williamson, editors, Proceedings of the 25th Annual Conference on Learning Theory , volume 23 of Proceedings of Machine Learning Research , pages 37.1-37.18, Edinburgh, Scotland, 25-27 Jun 2012. PMLR.

Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere i: Overview and the geometric picture. IEEE Transactions on Information Theory , 63(2):853-884, 2016a.

Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere ii: Recovery by riemannian trust-region method. IEEE Transactions on Information Theory , 63(2):885-914, 2016b.

Gokcan Tatli and Alper T Erdogan. Polytopic matrix factorization: Determinant maximization based criterion and identifiability. IEEE Transactions on Signal Processing , 69:5431-5447, 2021.

Bahareh Tolooshams and Demba E Ba. Stable and interpretable unrolled dictionary learning. Transactions on Machine Learning Research , 2022.

Ivana Toˇ si´ c and Pascal Frossard. Dictionary learning. IEEE Signal Processing Magazine , 28(2): 27-38, 2011.

Ivana Toˇ si´ c, Ivana Jovanovi´ c, Pascal Frossard, Martin Vetterli, and Neb Duri´ c. Ultrasound tomography with learned dictionaries. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 5502-5505. IEEE, 2010.

Yu Wang, Siqi Wu, and Bin Yu. Unique sharp local minimum in l1-minimization complete dictionary learning. Journal of Machine Learning Research , 21(63):1-52, 2020.

Siqi Wu and Bin Yu. Local identifiability of l1-minimization dictionary learning: a sufficient and almost necessary condition. The Journal of Machine Learning Research , 18(1):6121-6176, 2017.

Yuexiang Zhai, Hermish Mehta, Zhengyuan Zhou, and Yi Ma. Understanding l4-based dictionary learning: Interpretation, stability, and robustness. In International Conference on Learning Representations , 2020a.

Yuexiang Zhai, Zitong Yang, Zhenyu Liao, John Wright, and Yi Ma. Complete dictionary learning via l4-norm maximization over the orthogonal group. Journal of Machine Learning Research , 21 (165):1-68, 2020b.

A Proof of Theorem 3.2

The proof uses the notion of polar sets from convex analysis [Rockafellar, 1970]:

Definition A.1 (Polar set) . Given a set C that contains the origin, its polar set, denoted as C ◦ , is defined as

As an example, the polar of the unit ball B = { 𝒙 | ∥ 𝒙 ∥ ≤ 1 } is itself, i.e., 𝑩 ◦ = 𝑩 , using the CauchySchwartz inequality. Due to the definition of dual norm, we also have immediately that the polar set of any norm ball is the norm ball defined by its dual norm, e.g., the ℓ ∞ norm ball [-1 , 1 ] 𝑛 is the polar of the ℓ 1 norm ball { 𝒙 | ∥ 𝒙 ∥ 1 ≤ 1 } . Given a polyhedron in the form of conv ( 𝑯 ) , i.e., the convex hull of a set of vectors given as the columns of 𝑯 , then its polar is a polyhedron in the form of linear inequalities:

The important property of the polar sets is that if two sets satisfy A ⊆ B , then B ◦ ⊆ A ◦ , which will be crucial in the sequel. The proof is not hard and can be found in standard textbooks such as [Rockafellar, 1970].

Before proving Theorem 3.2, we show the following lemma as intermediate step:

Lemma A.2. If conv ( e 𝑺 ♮ 𝑽 ) is sufficiently scattered in the hypercube as described in Assumption 3.1, then the solution set of problem (5) is {± 𝒆 1 , . . . , ± 𝒆 𝑘 } .

Proof. As explained in the main paper, the single constraint ∥ 𝒘 ⊤ e 𝑺 ♮ ∥ 1 ≤ 1 is equivalent to a set of 2 𝑛 linear inequalities

where 𝑽 ∈ {± 1 } 𝑛 × 2 𝑛 contains all 2 𝑛 vectors {± 1 } 𝑛 as its columns. This means 𝒘 ∈ conv ( e 𝑺 ♮ 𝑽 ) ◦ , i.e., 𝒘 belongs to the polar set of conv ( e 𝑺 ♮ 𝑽 ) . The sufficiently scattered condition first assumes that B ⊆ conv ( e 𝑺 ♮ 𝑽 ) , so conv ( e 𝑺 ♮ 𝑽 ) ◦ ⊆ B since B ◦ = B . We immediately have that any feasible 𝒘 for (5) must satisfy ∥ 𝒘 ∥ ≤ 1 . Obviously, letting 𝒘 = ± 𝒆 𝑗 is feasible and attains the maximum.

The second assumption is that the intersection of 𝜕 B and 𝜕 conv ( e 𝑺 ♮ 𝑽 ) , the boundary of the two sets, is the finite set {± 𝒆 1 , . . . , ± 𝒆 𝑘 } . This, together with the first assumption, implies that for any direction 𝒚 not along the coordinates, there exists a point along that direction with norm strictly bigger than

1 that still lie in conv ( e 𝑺 ♮ 𝑽 ) , i.e., for any 𝒚 ≠ ± 𝒆 𝑗 and ∥ 𝒚 ∥ = 1 + 𝜖 with sufficiently small 𝜖 > 0 , there exists 𝜽 ∈ Δ 𝑛 such that 𝒚 ∈ conv ( e 𝑺 ♮ 𝑽 ) . Then 𝒘 must also satisfy 𝒘 ⊤ 𝒚 ≤ 1 . Now consider 𝒘 = 𝒚 /∥ 𝒚 ∥ , we would have that ∥ 𝒘 ∥ = 1 but it's infeasible since 𝒘 ⊤ 𝒚 = ∥ 𝒚 ∥ > 1 . This excludes any vector with norm 1 that is not in {± 𝒆 1 , . . . , ± 𝒆 𝑘 } to be optimal for (5). □

Proof of Theorem 3.2. Consider the rescaled groundtruth factors ( e 𝑨 ♮ , e 𝑺 ♮ ) and an optimal solution to (1) ( 𝑨 ★ , 𝑺 ★ ) . Obviously, both of them are feasible:

Since 𝑨 ★ is invertible, we have that 𝑺 ★ = ( 𝑨 ★ ) -1 𝑿 = ( 𝑨 ★ ) -1 e 𝑨 ♮ e 𝑺 ♮ . Now define 𝑾 = ( 𝑨 ★ ) -1 e 𝑨 ♮ . Since e 𝑨 ★ is optimal for (1), we have that

On the other hand, 𝑺 ★ = 𝑾 e 𝑺 ♮ being feasible means

i.e., each row of 𝑾 satisfies the constraint set of (5). Since e 𝑺 ♮ 𝑽 satisfies the sufficiently scattered condition, Lemma A.2 implies that

As a result, using the Hadamard inequality we have

and the supremum is attained when each row of 𝑾 is ± 𝒆 𝑗 . Combining (11) and (12) shows that | det 𝑾 | = 1 and each row of 𝑾 is ± 𝒆 𝑗 . Furthermore, | det 𝑾 | is nonzero only if none of the rows is collinear to each other, therefore 𝑾 must be a permutation matrix times a diagonal matrix with ± 1 on its diagonals. This guarantees identifiability of e 𝑺 ♮ (and e 𝑨 ♮ ) up to permutation and sign ambiguity, which is equivalent to identifiability of 𝑨 ♮ and 𝑺 ♮ up to permutation and scaling. □

B Proof of Theorem 3.4

We assume that 𝑺 ∼ BG( 𝑝 ) , and the first thing we do is rescaling its rows to have unit ℓ 1 norms and use it for the problem 5. To simplify the analysis, we can instead directly maximize ∥ 𝒘 ∥ 2 subject to ∥ 𝒘 ⊤ 𝑺 ∥ 1 ≤ 1 , and compare it with the largest ℓ 1 norm of the rows of 𝑺 . The complement of the intended probability can be bounded as

with arbitrary choice of 𝛼 . Conversely,

where the second inequality is obtained from the union bound. The rest of this section is dedicated to bounding the above two terms. Both of these results rely on the following version of the Bernstein inequality [Bennett, 1962]:

Theorem B.1 (Bernstein's inequality) . Let 𝑍 1 , . . . , 𝑍 𝑛 be independent random variables with E [ 𝑍 2 𝑖 ] ≤ 𝑣 2 and there exists some constant 𝑐 such that for all integer 𝑑 > 2

Then

Lemma B.2 (Bounding the second term in (13)) . Suppose 𝑺 ∈ R 𝑘 × 𝑛 is generated from the BernoulliGaussian model BG( 𝑝 ) . Then

Proof. Let 𝒔 = ( 𝑠 1 , . . . , 𝑠 𝑛 ) be generated from BG( 𝑝 ) , i.e., each 𝑠 𝑖 = 𝑏𝑖 𝑔 𝑖 with 𝑏𝑖 Bernoulli with probability 𝑝 and 𝑔𝑖 ∼ N( 0 , 1 ) , we will use Bernstein's inequality with 𝑍𝑖 = 𝑏𝑖 | 𝑔𝑖 | . The moments of 𝑏𝑖 all equal to 𝑝 . The random variable | 𝑔𝑖 | follows a Chi-distribution of degree 1 so its moments are

For 𝑑 = 1 and 2 we have E [ 𝑍𝑖 ] = 𝑝 √︁ 2 / 𝜋 and E [ 𝑍 2 𝑖 ] = 𝑝 . Using the recurrence relation for the Gamma function Γ ( 𝑡 + 1 ) = 𝑡 Γ ( 𝑡 ) and √ 2 / Γ ( 1 / 2 ) = √︁ 2 / 𝜋 < 1 we can bound the rest of the moments with 𝑑 > 2 as

Therefore the moments satisfy (14) with 𝑐 = 1 / √ 2 , results in

Finally, using the union bound

We now proceed to bound the first term in (13). First we note the following equivalence:

We are also going to use the following notion of 𝛿 -cover from convex geometry [Pisier, 1999] that holds for all ℓ 𝑝 norm balls, but we are only to instantiate the Eucleadian ball:

Lemma B.3 ( 𝛿 -cover) . A finite 𝛿 -cover of the unit sphere in R 𝑘 is a finite set C 𝛿 of points with unit ℓ 2 norm such that any point on the unit sphere is within 𝜖 away from an element in C 𝛿 , i.e.

For 𝛿 ∈ ( 0 , 1 ) there always exists an 𝛿 -cover C 𝜖 with cardinality |C 𝛿 | < ( 3 / 𝛿 ) 𝑘 .

Lemma B.4. Let C 𝛿 = { 𝒘 𝑖 } be an 𝛿 -cover for the sphere in R 𝑘 . Assume that we have both the lowerbound and the upperbound

Then

Proof. By definition of the 𝛿 -cover, for all 𝒘 with unit norm we can find 𝒘 𝑖 ∈ C 𝛿 with ∥ 𝒘 -𝒘 𝑖 ∥ < 𝛿 . Therefore

□

Lemma B.5 (Bounding the first term in (13)) . Suppose 𝑺 ∈ R 𝑘 × 𝑛 is generated from the BernoulliGaussian model BG( 𝑝 ) . Then

where 𝛿 ∈ ( 0 , 1 ) represents any choice of 𝛿 -cover for the unit sphere.

Proof. Following Lemma B.4, we have

where C 𝛿 is a 𝛿 -cover of the unit sphere with cardinality |C 𝛿 | < ( 3 / 𝛿 ) 𝑘 according to Lemma B.3. The bound to the first term in (18) is almost identical to Lemma B.2. Dropping the subscript of 𝒘 𝑖 , we write

Without the absolute value, 𝑍𝑖 is normally distributed with zero mean and variance 𝜎 2 = ˝ 𝑗 𝑚 2 𝑗 𝑏𝑖𝑗 , so

where 𝐺 is a standard normal random variable. Since 𝜎 2 = ˝ 𝑗 𝑚 2 𝑗 𝑏𝑖𝑗 ≤ ˝ 𝑗 𝑚 2 𝑗 = 1 , we have for 𝑑 ≥ 2





where for 𝑑 = 1 we have

Furthermore, | 𝐺 | is chi-distributed with degree 1, whose moments are given in (15), so the moments of | 𝑍𝑖 | again follows (16). Again, the moments satisfy (14) with 𝑐 = 1 / √ 2 , results in

To bound ∥ 𝑺 ⊤ ∥ 1 , we recall that this is the ℓ 1 induced norm for matrix 𝑺 ⊤ , which is shown to be the maximum of the ℓ 1 norms of the columns of 𝑺 ⊤ . This means we can use similar arguments used in Lemma B.2 (but applied to the other direction) to have

where we pick an arbitrary 𝑗 ∈ [ 𝑘 ] in the second line since this event implies that the maximum ℓ 1 norm of the rows are lowerbounded, and in the third line each 𝑍𝑖 = 𝑏𝑖 | 𝑔𝑖 | satisfies (16). The proof is complete by combining (18), (19), and (20) with 𝛽 = 𝑛𝑝 ( √︁ 2 / 𝜋 -𝜖 ) and 𝛾 = 𝑛𝑝 ( √︁ 2 / 𝜋 + 𝜖 ) . □

Proof of Theorem 3.4. We first instantiate Lemma B.5 with

which obviously satisfies 𝛿 < 1 and would also satisfies 𝛿 > 0 if

Then we have

Combining (13), (17), and Lemma B.2 with 𝛼 = 𝑛𝑝 ( √︁ 2 / 𝜋 -𝜖 ) , we obtain

Further, with

where the right hand side is obviously positive, we have

We can further relax (21) to

where in the last inequality we simply picked a small-enough 𝜖 so that

This completes the proof.

C Additional Algorithms

In this section we introduce two algorithms that directly tackles the formulation (7), repeated here:

C.1 Frank-Wolfe algorithm

Our first attempt is to apply the Frank-Wolfe algorithm to approximately solve (7). The Frank-Wolfe algorithm, also known as the conditional gradient method for constrained optimization [Bertsekas, 1999], iteratively minimizes a linear objective, defined by the gradient at the current iterate, under the same constraint set to determine the search direction and obtain the next iterate via some line search approach along the search direction. For the log-determinant objective in (7), we have that the gradient is -( 𝑷 -1 ) ⊤ . As a result, the Frank-Wolfe algorithm for (7) is given in Algorithm 2.

Regarding the line search step, we propose to use the backtracking line search (Armijo rule) [Bertsekas, 1999] to guarantee sufficient decrease of the objective function. Since the constraint set of (7) is convex, as long as 𝑷 ( 𝑡 ) is feasible, then 𝑷 ( 𝑡 + 1 ) is also feasible since it is a convex combination of 𝑷 ( 𝑡 ) and 𝑷 𝑑 , which are by definition feasible. Therefore, the only nontrivial part is to find a feasible initialization 𝑷 ( 0 ) . This can be done by optimizing an arbitrary linear objective subject to the same

□

Algorithm 2 Solving (7) with Frank-Wolfe

constraint as (7) (which means one should not apply line search at this step). If 𝑨 is square, there is a simple initialization that works really well in our experience, by setting 𝑷 ( 0 ) as a diagonal matrix with

i.e., rescaling each row of 𝑿 to have unit ℓ 1 norm.

In terms of complexity, each iteration is dominated by the linear programming with 𝑘 2 variables. Without exploiting any structure, the per-iteration complexity could be as high as 𝑂 ( 𝑘 6 ) . However, the linear programming to be solved in Algorithm 2 is blessed with structures to be exploited to greatly reduce the complexity. Denote 𝒑 𝑖 as the 𝑖 th row of 𝑷 , then the linear programming in each iteration of Algorithm 2 is in fact 𝑘 independent problems, each involving only one row of 𝑷 ; let 𝒇 𝑖 denote the 𝑖 th column of 𝑷 -1 ( 𝑡 ) , then we should solve the following problem with 𝑖 = 1 , . . . , 𝑘

Each of these problems involves 𝑘 variables, which can be solved with 𝑂 ( 𝑘 3 ) flops. This important observation brings the per-iteration complexity of Algorithm 2 down to 𝑂 ( 𝑘 4 ) .

Note that the Frank-Wolfe algorithm is also able to handle the overdetermined case when the dictionary matrix 𝑨 is tall, although this case is rarely considered in the context of dictionary learning: as long as 𝑨 has full column rank, we can make the change-of-variable 𝑷 = 𝑨 † , and the objective of (7) can be replaced by ( 1 / 2 ) log ( 𝑷𝑷 ⊤ ) , with its gradient equals to -( 𝑷 † ) ⊤ . Everything else follows easily.

We implement the algorithm in MATLAB and use the built-in linprog function in MATLAB to solve each of the linear programming sub-problems (22). Dropping the subscripts in (22), the ℓ 1 norm constraint can be transformed as linear ones using the standard trick:

Using the same setting as in §5.1 but with smaller problem dimension 𝑛 = 200 and 𝑘 = 10 , the convergence of 50 random trials are shown in Figure 5.

C.2 Cyclic row update

Inspired by the row-separable structure of the constraint set in (7), we also attempt to solve (7) via a block coordinate descent-type algorithm that cyclically updates each row. However, we note that this algorithm only works when 𝑷 is square, unlike the Frank-Wolfe algorithm, which can be extended to the rare overdetermined case when 𝑨 is tall. We will drop the log term in the objective function of (7) when designing the cyclic row update algorithm in this subsection.

Using the Laplace's formula, we know that det 𝑷 is a linear function with respect to the 𝑖 th row of 𝑷 using the co-factor expansion

5

10

15

20

25

30

iteration

-2

0

2

4

6

8

10

12

optimality gap

where the matrix 𝑷 𝑖 𝑗 is obtained by deleting the 𝑖 th row and 𝑗 th column of 𝑷 . Then to apply the block coordinate descent algorithm [Bertsekas, 1999], one would minimize -| 𝒇 ⊤ 𝑖 𝒑 𝑖 | subject to ∥ 𝒑 ⊤ 𝑖 𝑿 ∥ 1 ≤ 1 , where 𝒇 𝑖 ∈ R 𝑘 is a vector defined according to the co-factor expansion formula.

It looks as if we need to solve two linear programs for each row update, one to maximize 𝒇 ⊤ 𝑖 𝒑 𝑖 and one to minimize 𝒇 ⊤ 𝑖 𝒑 𝑖 , but notice that the constraint ∥ 𝒑 ⊤ 𝑖 𝑿 ∥ 1 ≤ 1 is invariant under a sign flip, therefore the only difference between the solutions of the two linear programs would be a sign flip, which is both acceptable due to the inherent sign ambiguity in matrix factorization and dictionary learning in specific. Furthermore, recall that the Cramer's rule shows that

meaning we can simply define 𝒇 𝑖 as the 𝑖 th column of 𝑷 -1 , and it would not affect the row updates. This makes the definition of 𝒇 𝑖 exactly the same way as in (22).

Algorithm 3 Solving (7) with cyclic row update

The two proposed algorithms 2 and 3 shows striking similarities, especially considering they both fundamentally solve linear programs in the form of (22) in each iteration. For nonconvex optimization, both algorithms guarantees that every limit point is a stationary point with some additional assumptions, which is reassuring to know. The differences are as follows:

Block coordinate descent is guaranteed to monotonically improve the objective function by design, so there is no need for line search as in Frank-Wolfe. This could save some computation as calculating det 𝑷 may not be cheap when 𝑘 is large.

On the other hand, each iteration of Frank-Wolfe only need to calculate 𝑷 -1 once for the update of all its 𝑘 rows, whereas cyclic row update requires to recalculate 𝑷 -1 for each of its row updates. Overall, the per-iteration complexity is almost identical.

An interesting observation is that solving one subproblem (22) resembles the problem formulations in works such as [Spielman et al., 2012, Geng and Wright, 2014, Sun et al., 2016a, Zhai et al., 2020b, Shen et al., 2020] that attempt to find one atom of the dictionary at a time in a deflation manner. The difference is that they are still trying to solve nonconvex problems, and they typically require multiple random initializations due to the risk of finding the same atoms multiple times. From a different perspective, we see that the benefit of using the proposed volume optimization framework provides a

smartly chosen search direction given by rows of 𝑷 -1 , which is by definition the current iterate of the dictionary.

This algorithm is again implemented in MATLAB using the built-in linprog function. Using the same setting as before with the Frank-Wolfe algorithm, the convergence of 50 random trials are shown in Figure 6. Comparing the two algorithms specifically, we see that Frank-Wolfe succeeds in all 50 trials, while CRU has one failure, although CRU takes much fewer number of iterations to converge.

1

2

3

4

5

6

7

8

9

10

iteration

-0.5

0

0.5

1

1.5

2

2.5

3

optimality gap

D Additional Experiments

Here we provide some additional experiments to showcase the performance of the proposed volumeoptimization formulation as well as the three proposed algorithms. We start by demonstrating the time-efficiency of the three proposed algorithms to motivate the use of L-ADMM as the go-to method. Then we compare with some state-of-the-art dictionary learning algorithms with identifiability guarantees to showcase the improved performance of dictionary recovery by incorporating the volume criterion.

D.1 Comparing the three proposed algorithms

In Figures 2, 5, and 6, we showed the per-iteration convergence of L-ADMM, Frank-Wolfe, and cyclic row update (i.e., block coordinate descent [BCD]) as described in Algorithm 1, 2, and 3. While all three algorithms manage to obtain essentially zero optimality gap, L-ADMM takes significantly more number of iterations; on the other hand, it is obvious that the cost of L-ADMM to perform one iteration is much cheaper than those of Frank-Wolfe or BCD. In this subsection we show that, thanks to the simple and efficient per-iteration update of L-ADMM, it is significantly faster than the other two proposed algorithms. We generate synthetic data sets like before, with 𝑛 = 1000 and the sparsity level 𝑝 = 0 . 5 , and set 𝑘 = 20 in Figure 7a and 𝑘 = 50 in Figure 7b. Notice that both the vertical and horizontal axes are in log scale, indicating that L-ADMM is several magnitude faster than the other alternatives. The difference would only become bigger as we increase 𝑛 and 𝑘 .

D.2 Dictionary recovery

We compare the dictionary recovery performance of the proposed volume-optimization formulation (using the L-ADMM algorithm in Algorithm 1) with some state-of-the-arts. The most related work are [Gribonval and Schnass, 2010, Wu and Yu, 2017, Wang et al., 2020], which focus on the local identifiability of formulation (2) without proposing new algorithms. For algorithms with claimed identifiability guarantees, the closest we find are the alternating direction method (ADM) by Sun et al. [2016a,b], ℓ 4 -norm maximization ( ℓ 4 -max) by Zhai et al. [2020b], and ℓ 3 -norm maximization ( ℓ 3 -max) by Shen et al. [2020]. We note that ADM is not the main algorithm that is analyzed in Sun et al. [2016a,b], but it works better and has the benefit of recovering the entire dictionary simultaneously; the analyzed manifold optimization algorithm in [Sun et al., 2016a,b] recovers the

10 -2

10 -1

10 0

10 1

10 2

10 3

time / seconds

10 -6

10 -4

10 -2

10 0

10 2

optimality gap

L-ADMM

BCD

(a)

𝑘

=

20

10 -2

10 0

10 2

10 4

time / seconds

10 -6

10 -4

10 -2

10 0

10 2

optimality gap

L-ADMM

BCD

(b)

𝑘

=

50

dictionary one atom at a time, which requires multiple initializations to guarantee complete recovery, and such procedures are not provided by the authors on their Github page.

Note that all three baseline algorithms, ADM, ℓ 4 -max, and ℓ 3 -max, require that the groundtruth dictionary to be orthogonal . They claim that this is, to some extend, without loss of generality by pre-whitening the data ( 𝑿𝑿 ⊤ ) -1 / 2 𝑿 . We discover that this claim is only true if the sparse coefficients matrix 𝑺 follows strictly from the Bernoulli-Gaussian generative model, and that the sample size tends to infinity, in which case ( 1 / 𝑛 ) 𝑺𝑺 ⊤ → 𝑝 𝑰 ; if this assumption does not hold, such a pre-whitening step is not able to make an arbitrary dictionary orthogonal. As a result, when we use the same experimental setting as in §5.2, none of these algorithms are able to exactly recover the true dictionary in any circumstance.

If we restrict ourselves to orthogonal dictionary recovery, then we obtain some sensible results from the three baseline algorithms, although they still perform significantly worse than our proposed work. What is more, even for exactly orthogonal dictionaries they are not able to recover them with very high accuracy. Here we test them in two cases: In the first case shown in Tables 1 and 2, we fix 𝑛 = 1000 and 𝑘 = 20 , and vary the sparsity level 𝑝 from 0 . 1 to 0 . 9 ; in the other case shown in Tables 3 and 4, we fix 𝑛 = 1000 and 𝑝 = 0 . 5 , and vary the dictionary size 𝑘 from 5 to 50 . The recovered dictionary is again permuted using the Hungarian algorithm, and then we declare a successful recovery if the error is less than a threshold. In each case we present two tables, one with the threshold to be 10 -5 (same as the threshold used in §5.2), and the other with threshold 10 -2 . We see that the three baseline algorithms can provide reasonable result with the more loose threshold, especially ℓ 3 -max. Regardless, the proposed volume-optimization formulation with the L-ADMM, performs significantly better than all of them.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a new method for dictionary learning. The proposed optimisation problem minimises the determinant (volume) is derived from the l1 formulation of dictionary learning. Global identification results are proved under a general assumption of being 'sufficiently scattered'. The case of a Bernoulli-Gaussian model is also studied and recovery with high probability is guaranteed.\n\nOn the algorithmic side, the authors describe an algorithm for solving the proposed optimisation problem based on ADMM, and several numerical examples (both synthetic and real data) are presented.\n\nSTRENGTHS:\nThe theoretical results are impressive, as the authors establish global identifiability results without the use of the incoherence condition.\n\nWEAKNESSES:\nThe main concrete example is the Bernoulli-Gaussian setting, which is already covered by previous works, albeit with local guarantees -- It would be good to see more concrete examples on settings that the new framework can handle, but previous results cannot. \n\nThere are no comparisons with previous methods in the experiments section of the main paper -- it would be instructive to see comparisons with previous formulations, e.g. (2) to understand if the proposed method does have better recoverability properties, or is simply easier to analyse.\n\nQUESTIONS:\nThe equivalence between the different optimisation problems is unclear to me. The main optimisation problem is (1), but the analysis consider (4) and (5). Can you clarify on why these problems are equivalent? You also use phrases such as \"conceptually, equivalent\" which I find confusing (as I'm not sure whether you actually mean equivalent). To add to this confusion, (4) and (5) seem to assume knowledge of the true factor $S^\\sharp$.\n\nIs there a relationship between the 'sufficiently scattered' condition and the incoherence conditions? It would be helpful to see a discussion on why one condition is 'better' than the other.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studied the classic problem of dictionary learning. A new method is proposed based on optimizing the determinant of the dictionary matrix. Deterministic and probabilistic analysis are conducted for global recovery of the dictionary and coefficients.  Experiments are provided to help demonstrate the effectiveness of the method.\n\nSTRENGTHS:\nThe formulation utilizes determinant, which is not common in the current dictionary learning literature.\nA global analysis is provided, which does not require the incoherent of the dictionary matrix.\n\nWEAKNESSES:\n## Novelty\nA crucial related work is Tatli and Erdogan [2021] as the authors pointed out, which studied (1) except that they bound the norm of columns of S while this paper did so for rows of S. What differs in the theoretical analysis? What differs in empirical performance? Please clarify.\n\n## Significance\n1. Lack of comparison with other dictionary learning methods\n    1. I agree that the proposed method guarantees the correctness of the global solution, which is attractive - but we do not have an algorithm provably converging to the global solution, since the problem is non-convex. On the other hand, Sun et al 2016a/b showed global convergence to a correct solution. Regardless, it would be great if one could compare theoretical results with the literature.\n    1. Moreover, it is interesting to see in synthetic experiments how the proposed method compares to baselines.\n    1. In particular, since the dictionary is only needed to be invertible rather than incoherent as per Theorem 3.2, why don’t we see some experiments where the dictionary is invertible but coherent?\n\nQUESTIONS:\n## Presentation\n1. It appears that linearized ADMM is somewhat standard, so I would suggest maybe some of section 4 could be postponed, and the authors may focus more on the comments above.\n1. Multiple optimization problems (1)-(5), (7), (8) have been introduced for different purposes, some of them are equivalent, and some of them are equivalent only under certain conditions. It would be great if the authors could summarize the problems, and clarify the equivalences and purposes.\n1. In the remark below Corollary 3.5, it was said the bound works best when p is close to 0.5. Yet, Figure 3 shows that as long as p is <= 0.5. Could you comment on the cause of this apparent discrepancy? Also, you said “this agrees with empirical results in numerous works in dictionary learning”, could you cite which ones?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel formulation for dictionary learning by minimizing the determinant of the dictionary matrix, subject to the constraint that each row of the sparse coefficient matrix has unit $\\ell_1$ norm. The proposed method provides global identifiability guarantees for the ground truth dictionary and sparse coefficient matrices, as long as certain conditions are met. The authors also develop an algorithm based on linearized-ADMM for complete dictionary recovery and provide a probabilistic analysis for the case where the sparse coefficient matrix is generated from the Bernoulli-Gaussian model.\n\nSTRENGTHS:\n1. The paper establishes an identifiability condition on the sparse coefficient matrix and proves that $\\ell_1$-based dictionary learning is globally identifiable. This result is a significant contribution, as it extends existing work on identifiability and provides a more general understanding of the problem.\n\n2. The authors introduce an algorithm based on linearized-ADMM for complete dictionary recovery. This algorithm is an important addition to the paper, as it demonstrates a practical approach to solving the new proposed formulation.\n\nWEAKNESSES:\n1. A notable limitation of the paper is that the identifiability condition relies on the global minimizers of problem (1), yet the introduced algorithm cannot guarantee obtaining these global minimizers. This concern raises questions about the practical applicability of the theoretical findings, as it may not consistently converge to the desired solution.\n\n2. The paper lacks empirical evidence to support the theoretical results concerning the identifiability condition for dictionary learning. Including experimental observations that corroborate the theoretical findings would strengthen the paper and provide a more convincing argument.\n\nQUESTIONS:\nPlease address the two points mentioned in the Weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}