SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Global Identifiability of  $\ell_1$ -based Dictionary Learning via Matrix Volume Optimization
Jingzhou Hu Kejun Huang
Department of Computer and Information Science and Engineering
Gainesville, FL 32611
(jingzhouhu,kejun.huang)@ufl.edu
We propose a novel formulation for dictionary learning that minimizes the determinant of the dictionary matrix, also known as its volume, subject to the constraint that each row of the sparse coefficient matrix has unit  $\ell_1$  norm. The main motivation for the proposed formulation is that it provides global identifiability guarantee of the groundtruth dictionary and sparse coefficient matrices, up to the inherent and inconsequential permutation and scaling ambiguity, if a set of vectors obtained from the coefficient matrix lies inside the  $\ell_{\infty}$  norm ball but contains the  $\ell_2$  norm ball in their convex hull. Unlike existing work on identifiability of dictionary learning, our result is global, meaning that a globally optimal solution to our proposed formulation has to be a permuted and rescaled version of the groundtruth factors. Another major improvement in our result is that there is no additional assumption on the dictionary matrix other than it is nonsingular, unlike most other works that require the atoms of the dictionary to be mutually incoherent. We also provide a probabilistic analysis and show that if the sparse coefficient matrix is generated from the widely adopted Bernoulli-Gaussian model, then it is globally identifiable if the sample size is bigger than a constant times  $k\log k$ , where  $k$  is the number of atoms in the dictionary, with overwhelming probability. The bound is essentially the same as those local identifiability results, but we show that it is also global. Finally, we propose algorithms to solve the new proposed formulation, specifically one based on the linearized-ADMM with efficient per-iteration updates. The proposed algorithms exhibit surprisingly effective performance in correctly and efficiently recovering the dictionary, as demonstrated in the numerical experiments.
Dictionary learning is a famous unsupervised learning problem [Tošić and Frossard, 2011] that also goes by the name sparse coding [Olshausen and Field, 1997] or sparse component analysis [Georgiev et al., 2005]. It stems from the intuition that samples from a data set can all be approximately represented as a sparse combination of atoms from a complete or overcomplete dictionary, or  $\boldsymbol{x}_i = \boldsymbol{A}\boldsymbol{s}_i$  for  $i = 1,\dots ,n$ , where  $\boldsymbol{x}_i\in \mathbb{R}^m$  are the given data points, columns of  $A\in \mathbb{R}^{m\times k}$  represents atoms of a dictionary, and  $\boldsymbol{s}_i\in \mathbb{R}^k$  are the sparse representations of  $\boldsymbol{x}_i$ , indicating that  $\boldsymbol{x}_i$  is linearly dependent on only a few atoms from the dictionary  $A$  with coefficients given in the nonzeros of  $\boldsymbol{s}_i$ . Staking  $\boldsymbol{x}_i$  as columns of  $X\in \mathbb{R}^{m\times n}$  and  $\boldsymbol{s}_i$  as columns of  $S\in \mathbb{R}^{k\times n}$ , it can be written as a matrix factorization model  $X = AS$ . Although in some cases there exist known dictionaries that admit good-quality sparse representations, such as the DCT or wavelet basis for image compression, sometimes it is better to learn a sparse-inducing dictionary from data in an unsupervised fashion, hence the task of dictionary learning that tries to estimate  $A$  and  $S$  jointly. In most cases, people
assume  $m \leq k$ : the dictionary is complete if  $m = k$ , and overcomplete if  $m < k$ . In this paper we focus on the complete dictionary case when  $A$  is square and nonsingular, which is common in a lot of prior works that this work is built upon [Gribonval and Schnass, 2010, Sun et al., 2016a,b, Wu and Yu, 2017, Wang et al., 2020]
Dictionary learning has found numerous applications in signal denoising [Elad and Aharon, 2006], audio coding [Plumbley et al., 2009], and medical imaging [Tošić et al., 2010], to name just a few. On the theory side, most of the existing works have focused on algorithm design. Famous algorithms include  $k$ -SVD [Aharon et al., 2006a] and online dictionary learning [Mairal et al., 2009], among numerous other algorithms based on generic nonconvex algorithm design with guarantee of convergence to a stationary point. More recently, there have appeared a line of research that attempts to show global optimality for dictionary learning under more restrictive assumptions, such as [Spielman et al., 2012, Agarwal et al., 2016, Arora et al., 2014, 2015, Sun et al., 2016a,b, Rambhatla et al., 2019, Bai et al., 2019, Zhai et al., 2020a,b, Shen et al., 2020, Tolooshams and Ba, 2022].
1.1 Model identifiability
The dictionary learning problem can be considered an instance of the broader matrix factorization model  $X = \mathbf{A}S$  subject to the opaque requirement that  $S$  is sparse. It is well-known that matrix factorization without any additional assumptions on the latent factors is not unique, since we can always "insert" an invertible matrix  $Q$  and  $Q^{-1}$  as  $X = \widetilde{A}\widetilde{S}$  where  $\widetilde{A} = AQ$  and  $\widetilde{S} = Q^{-1}S$ , and one cannot distinguish whether  $S$  or  $\widetilde{S}$  are the groundtruth sources. Such rotation ambiguity cannot be resolved by the well-known principal component analysis (PCA) [Jolliffe, 2002].
On the other hand, most matrix factorization models allow scaling and permutation ambiguity, i.e., a permutation matrix  $\pmb{\Pi}$  and a diagonal matrix  $\pmb{D}$  such that the recovered mixing matrix is  $AD\pmb{\Pi}$  and the recovered sources are  $\pmb{\Pi}^{\top}\pmb{D}^{-1}\pmb{S}$ . In addition to inherent constraints that fit the application, it is sometimes helpful to have an optimization criterion in search for the "correct" factorization (up to scaling and permutation). In light of this, we formally define the identifiability of any matrix factorization model as follows:
Definition 1.1 (Identifiability). Consider the generative model  $\pmb{X} = \pmb{A}^{\natural}\pmb{S}^{\natural}$ , where  $A^{\natural}$  and  $S^{\natural}$  are the groundtruth latent factors. Let  $(\pmb{A}^{\star},\pmb{S}^{\star})$  be optimal for an identification criterion  $q$
$$
(\boldsymbol{A}^{\star},\boldsymbol{S}^{\star}) = \operatorname *{arg  min}_{\substack{\boldsymbol {X} = \boldsymbol {A}\boldsymbol {S}\\ \boldsymbol {A}\in \mathcal{A},\boldsymbol {S}\in \mathcal{S}}}q(\boldsymbol {A},\boldsymbol {S}),
$$
where  $\mathcal{A}$  and  $S$  are some constraint set that  $\pmb{A}$  and  $\pmb{S}$  should belong to, respectively. If  $A^{\natural}$  and/or  $S^{\natural}$  satisfy some condition such that for any  $(A^{\star}, S^{\star})$ , there exist a permutation matrix  $\pmb{\Pi}$  and a diagonal matrix  $\pmb{D}$  such that  $A^{\natural} = A^{\star} D \pmb{\Pi}$  and  $S^{\natural} = \Pi^{\top} D^{-1} S^{\star}$ , then we say that the matrix factorization model is essentially identifiable, up to permutation and scaling, under that condition.
For dictionary learning in specific, where the essential constraint on the latent factors is sparsity on  $S$ , there have been some existing works on its identifiability issue. It has been shown that simply by putting a hard sparsity constraint on  $S$  is enough to guarantee a unique factorization, provided that the sample size  $n$  is large enough [Aharon et al., 2006b, Hillar and Sommer, 2015, Cohen and Gillis, 2019]. The main drawback is that the required sample size  $n$  is usually too large to be practical—results from [Aharon et al., 2006b] and [Hillar and Sommer, 2015] both require  $n$  to be  $O((k + 1)\binom{k}{s})$ , where  $s$  is the number of nonzeros that each column of  $S$  is allowed to have; it has been relaxed to  $O(k^3 / (k - s)^2)$  by Cohen and Gillis [2019], but still very large. The existence of  $s$  in the bound also shows that an outlier sample that does not admit a sparse representation from the dictionary is absolutely not allowed in the data set, which is also restrictive. Inspired by the success of compressive sensing and sparse recovery, it is perhaps more common to use the  $\ell_1$  norm to approximately sparsify  $S$ . Identifiability results based on the  $\ell_1$  norm formulation has been predominantly local, meaning the model is identifiable within a neighborhood of the groundtruth factors  $(A^\natural, S^\natural)$ , but on the bright side the sample size requirement is typically down to  $O(k\log k)$  and allows the existence of dense outliers [Gribonval and Schnass, 2010, Wu and Yu, 2017, Wang et al., 2020].
1.2 This paper
In this paper, we intend to bridge the gap by providing global identifiability guarantee of complete dictionary recovery using a  $\ell_1$  norm based formulation, given as follows:
$$
\underset {A, S} {\text {m i n i m i z e}} | \det A | \quad \text {s u b j e c t t o} X = A S, \| S _ {j:} \| _ {1} \leq 1, j = 1, \dots , k, \tag {1}
$$
To the best of our knowledge, this is a novel formulation that has never appeared in the literature; however, as we will show in the next section, the intuition stems from the more commonly used  $\ell_1$ -based formulations. We stress that the formulation is  $\ell_1$ -based, as the  $\ell_1$  norm is the most widely used term to promote sparsity of the solution. Although in the context of dictionary learning, using the  $\ell_1$  norm as the convex surrogate for sparsity is not enough to make the overall problem tractable, there are still benefits of using it instead of other ones such as hard sparsity constraints, for example in terms of sample complexity as introduced in the previous subsection.
After some gentle derivation to motivate the proposed new formulation (1) for dictionary learning, we provide the following contributions:
1. We give a deterministic characterization of the identifiability condition on the sparse coefficient matrix  $S$  under which the dictionary (as well as the sparse coefficients) can be uniquely determined, up to the inherent and inconsequential permutation and scaling ambiguity, via solving (1). Unlike existing work that shows local identifiability, our result is global, meaning that the groundtruth factors have to be a global minimizer to (1). Furthermore, perhaps surprisingly, our result only requires the complete dictionary to be nonsingular—there are no additional assumptions that require the atoms to be mutually incoherent whatsoever.
2. The result shows that  $\ell_1$ -based dictionary learning is globally identifiable if a certain set of points, which are by definition inside the hypercube  $[-1, 1]^k$ , contains the unit ball in their convex hull. This geometric assumption is called "sufficiently scattered", with similar ideas appearing in pertinent unsupervised learning models. We provide intuition why this assumption generally leads to a sparse  $S$ . We further show that if the sparse coefficient follows the widely used Bernoulli-Gaussian generative model, then it is identifiable with very high probability, provided the sample size  $n$  is bigger than a constant times  $k \log(k)$ . This is the same sample complexity bound obtained in some of the local identifiability analysis, but we show that it is also global, using the proposed new formulation (1).
3. We propose algorithms to solve (1), specifically one based on linearized-ADMM in the main paper; in the supplementary we also present one based on the Frank-Wolfe framework and the other based on block coordinate descent. Even though the proposed formulation (1) is nonconvex (just like any other dictionary learning formulation), numerical experiments on synthetic data show that they achieve global optimality in almost all cases, provided that the groundtruth latent factors satisfy the identifiability assumptions given before.
We start by proposing a novel formulation for dictionary learning. The purpose is to show that the seemingly different formulation does not come out of thin air, but is highly related, if not equivalent, to existing ones that have been studied extensively. Consider the following formulation adopted in [Gribonval and Schnass, 2010, Geng and Wright, 2014, Wu and Yu, 2017, Wang et al., 2020]:
$$
\underset {A, S} {\text {m i n i m i z e}} \| \mathbf {S} \| _ {1} \quad \text {s u b j e c t t o} \quad \mathbf {X} = A \mathbf {S}, \| A _ {., j} \| \leq 1, j = 1, \dots , k. \tag {2}
$$
We use the noiseless model  $\mathbf{X} = \mathbf{A}\mathbf{S}$  since the main focus of this work is identifiability, and a lot of algorithms are designed by replacing this term with a loss function that measures the residual  $\mathbf{X} - \mathbf{A}\mathbf{S}$ , e.g. [Gribonval et al., 2015]. The purpose of the  $\ell_1$  norm of  $\mathbf{S}$  is to encourage it to be sparse, but since it is also dependent on the scaling of  $\mathbf{S}$ , the constraint  $\| A_{\cdot ,j}\| \leq 1$  is imposed onto the columns of  $\mathbf{A}$ . Such constraints are without loss of generality due to the scaling ambiguity  $\mathbf{A}\mathbf{S} = \mathbf{A}\mathbf{D}\mathbf{D}^{-1}\mathbf{S}$ , so for any admissible factorization, we can rescale the columns of  $\mathbf{A}$  to unit norm, while absorbing the scaling into the rows of  $\mathbf{S}$ , without affecting the matrix product.
Due to the Lagrangian multiplier theorem, there exist positive scalars  $\beta_{1},\ldots ,\beta_{k}$  and  $\gamma_1,\dots ,\gamma_k$  such that any solution of the following problem is also optimal for (2)
$$
\underset {\boldsymbol {A}, \boldsymbol {S}} {\text {m i n i m i z e}} \quad \sum_ {j = 1} ^ {k} \gamma_ {j} \| \boldsymbol {A}. _ {, j} \| \quad \text {s u b j e c t t o} \quad \boldsymbol {X} = \boldsymbol {A} \boldsymbol {S}, \| \boldsymbol {S} _ {j;} \| _ {1} \leq \beta_ {j}, j = 1, \dots , k, \tag {3}
$$
since they would lead to exactly the same Lagrangian function with appropriate choices of  $\gamma_{j}$  and  $\beta_{j}$ . Again, due to the inherent scaling ambiguity, we can assume without loss of generality that  $\beta_{1} = \dots = \beta_{k} = 1$ , leaving the unknown parameters to just  $\gamma_{j}$ 's.
If we replace the summation in the objective of (3) with product, or equivalently replacing the arithmetic mean with the geometric mean, the term  $\prod_{j} \gamma_{j} \| A_{:j} \|$  would be intimately related to  $|\det A|$  if  $A$  is square. Indeed, there must exist some unknown matrix  $G$  such that  $|\det GA| = \prod_{j} \gamma_{j} \| A_{:j} \|$ . Furthermore, for the purpose of optimization it is completely unnecessary to know  $G$  since  $|\det GA| = |\det G| |\det A|$ , which is just a positive scaling of  $|\det A|$  that does not affect the optimization solution. As a result, our proposed novel formulation of dictionary learning takes the form of (1).
Although our paper focuses on the case of complete dictionaries, i.e., when  $\mathbf{A}$  is square and nonsingular, they can be effortlessly extended to the overdetermined case when  $\mathbf{A}$  is tall, and the only modification is to change the objective to  $\operatorname*{det}\mathbf{A}^{\top}\mathbf{A}$ . This is sometimes called the volume of the matrix [Ben-Israel, 1992] with applications in change-of-variable integration [Ben-Israel, 1999] and probability [Ben-Israel, 2000]. It has been used as the identification criterion for several variants of nonnegative matrix factorization with an interesting geometric interpretation of finding the minimum volume enclosing simplex of a set of points [Lin et al., 2015, Huang et al., 2016, 2018, Fu et al., 2018, Huang and Fu, 2019]. The most related work is the polytopic matrix factorization framework proposed by Tatli and Erdogan [2021], which includes a case that constrains the columns of  $S$  to have bounded  $\ell_1$  norm. Our proposed formulation constrains the rows of  $S$ , which seems to be a minor difference but is in fact a great leap forward since row scaling of  $S$  is inherent for any matrix factorization, while column scaling is not. It also turns out that the identifiability analysis would be drastically different, albeit the formulations look similar.
3 Global identifiability
In this section we study the conditions under which the proposed dictionary learning formulation (1) is identifiable, i.e., any solution to (1) equals to the groundtruth dictionary  $A^{\natural}$  and sparse coefficients  $S^{\natural}$  up to permutation and scaling, as per Definition 1.1. The groundtruth  $S^{\natural}$  can have arbitrary scaling, and it is without loss of generality to assume that its rows have unit  $\ell_{1}$  norms; we denote such rescaled  $S^{\natural}$  to be  $\widetilde{S}^{\natural}$ , i.e.,  $\| \widetilde{S}_{j,:}^{\natural} \|_1 \leq 1, j = 1, \ldots, k$ ; we absorb the counterscaling into the columns of  $A^{\natural}$  to generate  $\widetilde{A}^{\natural}$ , so that  $X = A^{\natural} S^{\natural} = \widetilde{A}^{\natural} \widetilde{S}^{\natural}$ .
Since we consider only complete dictionaries, for any feasible  $(A,S)$  for (1), there must exist a nonsingular matrix  $W$  such that  $S = W\widetilde{S}^{\natural}$  and  $A = \widetilde{A}^{\natural}W^{-1}$ , therefore Problem (1) is, conceptually, equivalent to  $\max_{W}|\det W|$  subject to  $\| \pmb{w}_j^\top \widetilde{\pmb{S}}^\natural \| _1\leq 1,j = 1,\ldots ,k,$  (4)
where  $\pmb{w}_j^\top$  denotes the  $j$ th row of  $\pmb{W}$ . Since  $\pmb{W} = \pmb{A}^{-1}\widetilde{\pmb{A}}^\natural$  and  $\widetilde{\pmb{A}}^\natural$  is feasible, the optimal value of (4) is at least 1. On the other hand, if the model is identifiable, then the optimal value of (4) should be exactly equal to 1, and the optimal solution to (4) must be a permutation matrix times a diagonal  $\pm 1$  matrix. The rest of this section is dedicated to characterizing when such condition holds for the sparse coefficients  $\mathbf{S}$ . Notice that all we need for the dictionary  $\pmb{A}$  is that its columns are independent, so that the transform from (1) to (4) is possible—there is no additional assumptions such as "incoherence" for the rest of the arguments to hold, which is a great improvement to the existing works such as [Gribonval and Schnass, 2010, Sun et al., 2016a,b, Wu and Yu, 2017, Wang et al., 2020].
3.1 Deterministic characterization
The intuition is simple: the Hadamard inequality states that  $|\operatorname{det} \boldsymbol{W}| \leq \prod_{j} \| \boldsymbol{w}_{j} \|$ , so a sufficient condition for identifiability is that the set of solutions to the following (nonconvex) quadratic programming
$$
\underset {\boldsymbol {w}} {\text {m a x i m i z e}} \| \boldsymbol {w} \| ^ {2} \quad \text {s u b j e c t t o} \| \boldsymbol {w} ^ {\top} \widetilde {\boldsymbol {S}} ^ {\natural} \| _ {1} \leq 1 \tag {5}
$$
is  $\{\pm e_1,\ldots ,\pm e_k\}$ ; in other words, a maximizer must be an indicator vector or its negative.
To proceed, we introduce the following notion:
Assumption 3.1 (Sufficiently scattered in the hypercube). Let  $\mathcal{B}$  denote the Euclidean ball  $\mathcal{B} = \{\pmb{x} \in \mathbb{R}^k \mid \| \pmb{x} \| \leq 1\}$  and  $C$  denote the hypercube  $C = \{\pmb{x} \in \mathbb{R}^k \mid \| \pmb{x} \|_{\infty} \leq 1\}$ . A set  $S$  is sufficiently scattered in the hypercube if:
1.  $\mathcal{B} \subseteq S \subseteq C$ ;
2.  $\partial \mathcal{B}\cap \partial S = \{\pm e_1,\dots ,\pm e_k\}$  , where  $\partial$  denotes the boundary of the set.
A geometric illustration of a set that is sufficiently scattered in the hypercube is shown in Figure 1, which in this case is a polytope with vertices depicted as dots. As we can see,  $\mathcal{B}$  is a subset of the hypercube  $[-1,1]^k$ , but touches the boundary of  $[-1,1]^k$  at points  $\pm \pmb{e}_1, \ldots, \pm \pmb{e}_k$ . If a set  $S$  is sufficiently scattered in  $[-1,1]^k$ ,  $S$  contains  $\mathcal{B}$  as a subset and, as a second requirement,  $\mathcal{B}$  touches the boundary of  $S$  only at those points too.
The term "sufficiently scattered" was first coined by Huang et al. [2016] to characterize the identifiability condition for nonnegative matrix factorization that has already appeared in [Huang et al., 2013]. The difference is that in [Huang et al., 2013, 2016, Fu et al., 2018], the condition is defined over the conic hull of a set of points in the nonnegative orthant containing a specific hyperbolic cone. It has also been defined over the convex hull of a set of points in the probability
simplex [Fu et al., 2015, Lin et al., 2015]. The most related case is the sufficiently scattered condition defined for the hypercube  $(\ell_{\infty}$ -norm ball) [Tatli and Erdogan, 2021], as well as the standard orthoplex  $(\ell_{1}$ -norm ball).
Figure 1: An example of sufficiently scattered in 3D.
To see how the sufficiently scattered condition could be useful for dictionary learning, we first rewrite the single constraint in (5) as, equivalently, a set of  $2^n$  linear inequalities  $\pmb{w}^{\top}\widetilde{\pmb{S}}^{\natural}\pmb{v} \leq 1$ ,  $\forall \pmb{v} \in \{\pm 1\}^n$ . Since the rows of  $\widetilde{\pmb{S}}^{\natural}$  have been rescaled to have unit  $\ell_1$  norm, we must have that  $\| \widetilde{\pmb{S}}^{\natural}\pmb{v} \|_{\infty} \leq 1$ , meaning that all vectors  $\widetilde{\pmb{S}}^{\natural}\pmb{v}$  should lie in the hypercube  $[-1,1]^k$ , and we further assume the convex hull of all such  $\widetilde{\pmb{S}}^{\natural}\pmb{v}$  is sufficiently scattered. Define the  $n \times 2^n$  matrix  $\pmb{V}$  that contains all elements of  $\{\pm 1\}^n$  as its columns, we have the following deterministic characterization of the identifiability of (1):
Theorem 3.2. Consider the generative model  $X = A^{\natural}S^{\natural}$ , where  $A^{\natural} \in \mathbb{R}^{k\times k}$  is the groundtruth dictionary and  $S$  is the groundtruth sparse coefficients. If  $\operatorname{rank}(A^{\natural}) = k$  and  $\operatorname{conv}(S^{\natural}V)$  is sufficiently scattered in the hypercube as in Assumption 3.1, then for any solution of (1), denoted as  $(A^{\star}, S^{\star})$ , there exist a permutation matrix  $\pmb{\Pi}$  and a diagonal matrix  $D$  such that  $A^{\natural} = A^{\star}D\pmb{\Pi}$  and  $S^{\natural} = \pmb{\Pi}^{\top}\pmb{D}^{-1}\pmb{S}^{\star}$ . In other words, dictionary learning is identifiable if the groundtruth  $A^{\natural}$  has full column rank and  $S^{\natural}V$  is sufficiently scattered.
The full proof is relegated to Appendix A. Here we provide a concise sketch of the proof: if  $\widetilde{\pmb{S}}^{\sharp}\pmb{V}$  is sufficiently scattered, a solution to (5) must be  $\pm \pmb{e}_j$ ; as a result,  $|\operatorname*{det}\pmb {W}|\leq \prod_j\| \pmb {w}_j\| \leq 1$ , and equality holds only if  $\pmb{W}$  is a permutation matrix with some sign flips of its rows.