SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The second assumption is that the intersection of  $\partial \mathcal{B}$  and  $\partial \operatorname{conv}(\widetilde{\mathbf{S}}^{\sharp} \mathbf{V})$ , the boundary of the two sets, is the finite set  $\{\pm \pmb{e}_1, \dots, \pm \pmb{e}_k\}$ . This, together with the first assumption, implies that for any direction  $\pmb{y}$  not along the coordinates, there exists a point along that direction with norm strictly bigger than 1 that still lie in  $\operatorname{conv}(\widetilde{\mathbf{S}}^{\sharp} \mathbf{V})$ , i.e., for any  $\pmb{y} \neq \pm \pmb{e}_j$  and  $\| \pmb{y} \| = 1 + \epsilon$  with sufficiently small  $\epsilon > 0$ , there exists  $\pmb{\theta} \in \Delta^n$  such that  $\pmb{y} \in \operatorname{conv}(\widetilde{\mathbf{S}}^{\sharp} \mathbf{V})$ . Then  $\pmb{w}$  must also satisfy  $\pmb{w}^{\top} \pmb{y} \leq 1$ . Now consider  $\pmb{w} = \pmb{y} / \| \pmb{y} \|$ , we would have that  $\| \pmb{w} \| = 1$  but it's infeasible since  $\pmb{w}^{\top} \pmb{y} = \| \pmb{y} \| > 1$ . This excludes any vector with norm 1 that is not in  $\{\pm \pmb{e}_1, \dots, \pm \pmb{e}_k\}$  to be optimal for (5).
Proof of Theorem 3.2. Consider the rescaled groundtruth factors  $(\widetilde{A}^{\natural},\widetilde{S}^{\natural})$  and an optimal solution to (1)  $(A^{\star},S^{\star})$ . Obviously, both of them are feasible:
$$
\boldsymbol {X} = \widetilde {\boldsymbol {A}} ^ {\natural} \widetilde {\boldsymbol {S}} ^ {\natural} = \boldsymbol {A} ^ {\star} \boldsymbol {S} ^ {\star}, \quad \| \widetilde {\boldsymbol {S}} _ {j,:} ^ {\natural} \| _ {1} \leq 1, \| \boldsymbol {S} _ {j,:} ^ {\star} \| _ {1} \leq 1, \quad j = 1, \dots , k.
$$
Since  $A^{\star}$  is invertible, we have that  $\pmb {S}^{\star} = (A^{\star})^{-1}\pmb {X} = (A^{\star})^{-1}\widetilde{\pmb{A}}^{\natural}\widetilde{\pmb{S}}^{\natural}$ . Now define  $\pmb {W} = (A^{\star})^{-1}\widetilde{\pmb{A}}^{\natural}$ . Since  $\widetilde{A}^{\star}$  is optimal for (1), we have that
$$
\left| \det A ^ {\star} \right| \leq \left| \det \widetilde {A} ^ {\natural} \right| \Rightarrow \left| \det \left(A ^ {\star}\right) ^ {- 1} \widetilde {A} ^ {\natural} \right| \geq 1 \Rightarrow \left| \det W \right| \geq 1. \tag {11}
$$
On the other hand,  $S^{\star} = W\bar{S}^{\natural}$  being feasible means
$$
\| \boldsymbol {w} _ {j} ^ {\top} \widetilde {\boldsymbol {S}} ^ {\natural} \| _ {1} \leq 1, \quad j = 1, \dots , k,
$$
i.e., each row of  $\mathbf{W}$  satisfies the constraint set of (5). Since  $\widetilde{\mathbf{S}}^{\natural}\mathbf{V}$  satisfies the sufficiently scattered condition, Lemma A.2 implies that
$$
\left\| \boldsymbol {w} _ {j} \right\| \leq 1, \quad j = 1, \dots , k.
$$
As a result, using the Hadamard inequality we have
$$
| \det  \boldsymbol {W} | \leq \prod_ {j = 1} ^ {k} \| \boldsymbol {w} _ {j} \| \leq 1, \tag {12}
$$
and the supremum is attained when each row of  $\mathbf{W}$  is  $\pm \mathbf{e}_j$ . Combining (11) and (12) shows that  $|\det \mathbf{W}| = 1$  and each row of  $\mathbf{W}$  is  $\pm \mathbf{e}_j$ . Furthermore,  $|\det \mathbf{W}|$  is nonzero only if none of the rows is collinear to each other, therefore  $\mathbf{W}$  must be a permutation matrix times a diagonal matrix with  $\pm 1$  on its diagonals. This guarantees identifiability of  $\widetilde{\mathbf{S}}^{\natural}$  (and  $\widetilde{\mathbf{A}}^{\natural}$ ) up to permutation and sign ambiguity, which is equivalent to identifiability of  $\mathbf{A}^{\natural}$  and  $\mathbf{S}^{\natural}$  up to permutation and scaling.
We assume that  $S \sim \mathcal{BG}(p)$ , and the first thing we do is rescaling its rows to have unit  $\ell_1$  norms and use it for the problem 5. To simplify the analysis, we can instead directly maximize  $\| \boldsymbol{w} \|^2$  subject to  $\| \boldsymbol{w}^\top \boldsymbol{S} \|_1 \leq 1$ , and compare it with the largest  $\ell_1$  norm of the rows of  $S$ . The complement of the intended probability can be bounded as
$$
\Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\top} \tilde {\boldsymbol {S}} \| _ {1} \leq 1} \| \boldsymbol {w} \| \leq 1 \right] \geq \Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\top} \boldsymbol {S} \| _ {1} \leq 1} \| \boldsymbol {w} \| \leq \alpha \cap \max  _ {j} \| \boldsymbol {S} _ {j,:} \| _ {1} \geq \alpha \right],
$$
with arbitrary choice of  $\alpha$ . Conversely,
$$
\begin{array}{l} \Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\top} \tilde {\mathbf {S}} \| _ {1} \leq 1} \| \boldsymbol {w} \| > 1 \right] \leq \Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\top} \mathbf {S} \| _ {1} \leq 1} \| \boldsymbol {w} \| > \alpha \cup \max  _ {j} \| \mathbf {S} _ {j,:} \| _ {1} <   \alpha \right] \\ \leq \Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\top} \boldsymbol {S} \| _ {1} \leq 1} \| \boldsymbol {w} \| > \alpha \right] + \Pr \left[ \max  _ {j} \| \boldsymbol {S} _ {j,:} \| _ {1} <   \alpha \right] \tag {13} \\ \end{array}
$$
where the second inequality is obtained from the union bound. The rest of this section is dedicated to bounding the above two terms. Both of these results rely on the following version of the Bernstein inequality [Bennett, 1962]:
Theorem B.1 (Bernstein's inequality). Let  $Z_{1},\ldots,Z_{n}$  be independent random variables with  $\operatorname{E}[Z_i^2] \leq v^2$  and there exists some constant  $c$  such that for all integer  $d > 2$
$$
\mathrm {E} \left[ \left| Z _ {i} \right| ^ {d} \right] \leq \frac {1}{2} d! v ^ {2} c ^ {d - 2}. \tag {14}
$$
$$
\Pr \left[ \left| \sum_ {i = 1} ^ {n} \left(Z _ {i} - \mathrm {E} \left[ Z _ {i} \right]\right) \right| > \epsilon \right] \leq 2 \exp \left(- \frac {\epsilon^ {2}}{2 (n v ^ {2} + c \epsilon)}\right)
$$
Lemma B.2 (Bounding the second term in (13)). Suppose  $S \in \mathbb{R}^{k \times n}$  is generated from the Bernoulli-Gaussian model  $\mathcal{BG}(p)$ . Then
$$
\Pr \left[ \max  _ {j} \| \mathbf {S} _ {j,:} \| _ {1} <   n p (\sqrt {2 / \pi} - \epsilon) \right] \leq 2 k \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right)
$$
Proof. Let  $s = (s_1, \ldots, s_n)$  be generated from  $\mathcal{BG}(p)$ , i.e., each  $s_i = b_i g_i$  with  $b_i$  Bernoulli with probability  $p$  and  $g_i \sim \mathcal{N}(0, 1)$ , we will use Bernstein's inequality with  $Z_i = b_i |g_i|$ . The moments of  $b_i$  all equal to  $p$ . The random variable  $|g_i|$  follows a Chi-distribution of degree 1 so its moments are
$$
\mathrm {E} \left[ \left| g _ {i} \right| ^ {d} \right] = 2 ^ {d / 2} \frac {\Gamma \left(\frac {d + 1}{2}\right)}{\Gamma \left(\frac {1}{2}\right)}. \tag {15}
$$
For  $d = 1$  and 2 we have  $\operatorname{E}[Z_i] = p\sqrt{2 / \pi}$  and  $\operatorname{E}[Z_i^2] = p$ . Using the recurrence relation for the Gamma function  $\Gamma(t + 1) = t\Gamma(t)$  and  $\sqrt{2} / \Gamma(1/2) = \sqrt{2 / \pi} < 1$  we can bound the rest of the moments with  $d > 2$  as
$$
\mathrm {E} \left[ \left| Z _ {i} \right| ^ {d} \right] \leq p \frac {d !}{2 ^ {d / 2}}. \tag {16}
$$
Therefore the moments satisfy (14) with  $c = 1 / \sqrt{2}$ , results in
$$
\begin{array}{l} \Pr \left[ \| \boldsymbol {s} \| _ {1} <   n p (\sqrt {2 / \pi} - \epsilon) \right] \leq \Pr \left[ \left| \sum_ {i = 1} ^ {n} \left(Z _ {i} - \mathrm {E} [ Z _ {i} ]\right) \right| > n p \epsilon \right] \\ \leq 2 \exp \left(- \frac {n ^ {2} p ^ {2} \epsilon^ {2}}{2 (n p + n p \epsilon / \sqrt {2})}\right) \\ = 2 \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right). \\ \end{array}
$$
Finally, using the union bound
$$
\begin{array}{l} \Pr \left[ \max  _ {j} \| \mathbf {S} _ {j,:} \| _ {1} <   n p (\sqrt {2 / \pi} - \epsilon) \right] \leq k \Pr \left[ \| \mathbf {S} _ {j,:} \| _ {1} <   n p (\sqrt {2 / \pi} - \epsilon) \right] \\ \leq 2 k \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right). \\ \end{array}
$$
We now proceed to bound the first term in (13). First we note the following equivalence:
$$
\Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\top} \boldsymbol {S} \| _ {1} \leq 1} \| \boldsymbol {w} \| > \alpha \right] = \Pr \left[ \inf  _ {\| \boldsymbol {w} \| = 1} \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} <   1 / \alpha \right] \tag {17}
$$
We are also going to use the following notion of  $\delta$ -cover from convex geometry [Pisier, 1999] that holds for all  $\ell_p$  norm balls, but we are only to instantiate the Euclidean ball:
Lemma B.3 ( $\delta$ -cover). A finite  $\delta$ -cover of the unit sphere in  $\mathbb{R}^k$  is a finite set  $C_\delta$  of points with unit  $\ell_2$  norm such that any point on the unit sphere is within  $\epsilon$  away from an element in  $C_\delta$ , i.e.
$$
\min  _ {\boldsymbol {w} _ {i} \in C _ {\delta}} \| \boldsymbol {w} - \boldsymbol {w} _ {i} \| <   \delta , \forall \| \boldsymbol {w} \| = 1.
$$
For  $\delta \in (0,1)$  there always exists an  $\delta$ -cover  $C_{\epsilon}$  with cardinality  $|\mathcal{C}_{\delta}| < (3 / \delta)^k$ .
Lemma B.4. Let  $C_{\delta} = \{\pmb{w}_i\}$  be an  $\delta$ -cover for the sphere in  $\mathbb{R}^k$ . Assume that we have both the lowerbound
$$
\| \boldsymbol {S} ^ {\top} \boldsymbol {w} _ {i} \| _ {1} \geq \beta , \forall \boldsymbol {w} _ {i} \in C _ {\delta}
$$
and the upperbound
$$
\| \boldsymbol {S} ^ {\top} \| _ {1} = \sup  _ {\| \boldsymbol {w} \| _ {1} \leq 1} \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} \leq \gamma .
$$
$$
\inf  _ {\| \boldsymbol {w} \| \leq 1} \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} \geq \beta - \gamma \delta \sqrt {k}
$$
Proof. By definition of the  $\delta$ -cover, for all  $\pmb{w}$  with unit norm we can find  $\pmb{w}_i \in C_\delta$  with  $\| \pmb{w} - \pmb{w}_i \| < \delta$ . Therefore
$$
\begin{array}{l} \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} \geq \| \boldsymbol {S} ^ {\top} \boldsymbol {w} _ {i} \| _ {1} - \| \boldsymbol {S} ^ {\top} (\boldsymbol {w} - \boldsymbol {w} _ {i}) \| _ {1} \geq \beta - \| \boldsymbol {S} ^ {\top} \| _ {1} \| \boldsymbol {w} - \boldsymbol {w} _ {i} \| _ {1} \\ \geq \beta - \| \boldsymbol {S} ^ {\top} \| _ {1} \| \boldsymbol {w} - \boldsymbol {w} _ {i} \| \sqrt {k} \geq \beta - \gamma \delta \sqrt {k}. \\ \end{array}
$$
Lemma B.5 (Bounding the first term in (13)). Suppose  $S \in \mathbb{R}^{k \times n}$  is generated from the Bernoulli-Gaussian model  $\mathcal{BG}(p)$ . Then
$$
\Pr \left[ \inf  _ {\| \boldsymbol {w} \| = 1} \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} <   n p (\sqrt {2 / \pi} - \epsilon) - \delta \sqrt {k} n p (\sqrt {2 / \pi} + \epsilon) \right] \leq \left(\left(\frac {3}{\delta}\right) ^ {k} + 1\right) 2 \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right),
$$
where  $\delta \in (0,1)$  represents any choice of  $\delta$ -cover for the unit sphere.
Proof. Following Lemma B.4, we have
$$
\Pr \left[ \inf  _ {\| \boldsymbol {w} \| \leq 1} \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} <   \beta - \gamma \delta \sqrt {k} \right] \leq \sum_ {\boldsymbol {w} _ {i} \in C _ {\delta}} \Pr \left[ \| \boldsymbol {S} ^ {\top} \boldsymbol {w} _ {i} \| _ {1} <   \beta \right] + \Pr \left[ \| \boldsymbol {S} ^ {\top} \| _ {1} > \gamma \right], \tag {18}
$$
where  $C_{\delta}$  is a  $\delta$ -cover of the unit sphere with cardinality  $|C_{\delta}| < (3 / \delta)^{k}$  according to Lemma B.3.
The bound to the first term in (18) is almost identical to Lemma B.2. Dropping the subscript of  $\boldsymbol{w}_i$ , we write
$$
\| \boldsymbol {\mathcal {S}} ^ {\top} \boldsymbol {w} \| _ {1} = \sum_ {i = 1} ^ {n} \left| \sum_ {j = 1} ^ {k} b _ {i j} g _ {i j} m _ {j} \right| := \sum_ {i = 1} ^ {n} | Z _ {i} |.
$$
Without the absolute value,  $Z_{i}$  is normally distributed with zero mean and variance  $\sigma^2 = \sum_j m_j^2 b_{ij}$ , so
$$
\operatorname {E} \left[ \left| Z _ {i} \right| ^ {d} \right] = \operatorname {E} \left[ \left| \sigma G \right| ^ {d} \right] = \operatorname {E} \left[ \left| \sigma \right| ^ {d} \right] \operatorname {E} \left[ \left| G \right| ^ {d} \right],
$$
where  $G$  is a standard normal random variable. Since  $\sigma^2 = \sum_j m_j^2 b_{ij} \leq \sum_j m_j^2 = 1$ , we have for  $d \geq 2$
$$
\operatorname {E} \left[ | \sigma | ^ {d} \right] \leq \operatorname {E} \left[ | \sigma | ^ {2} \right] = \operatorname {E} \left[ \sum_ {j} m _ {j} ^ {2} b _ {i j} \right] = p,
$$
where for  $d = 1$  we have
$$
\operatorname {E} \left[ | \sigma | \right] \geq \operatorname {E} \left[ | \sigma | ^ {2} \right] = \operatorname {E} \left[ \sum_ {j} m _ {j} ^ {2} b _ {i j} \right] = p.
$$
Furthermore,  $|G|$  is chi-distributed with degree 1, whose moments are given in (15), so the moments of  $|Z_i|$  again follows (16). Again, the moments satisfy (14) with  $c = 1 / \sqrt{2}$ , results in
$$
\begin{array}{l} \Pr \left[ \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} <   n p (\sqrt {2 / \pi} - \epsilon) \right] \leq \Pr \left[ \left| \sum_ {i = 1} ^ {n} \left(Z _ {i} - \mathrm {E} [ Z _ {i} ]\right) \right| > n p \epsilon \right] \\ \leq 2 \exp \left(- \frac {n ^ {2} p ^ {2} \epsilon^ {2}}{2 (n p + n p \epsilon / \sqrt {2})}\right) \\ = 2 \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right). \tag {19} \\ \end{array}
$$
To bound  $\| S^{\top}\|_{1}$ , we recall that this is the  $\ell_1$  induced norm for matrix  $S^{\top}$ , which is shown to be the maximum of the  $\ell_1$  norms of the columns of  $S^{\top}$ . This means we can use similar arguments used in Lemma B.2 (but applied to the other direction) to have
$$
\begin{array}{l} \Pr \left[ \| \boldsymbol {S} ^ {\top} \| _ {1} > n p (\sqrt {2 / \pi} + \epsilon) \right] = \Pr \left[ \max  _ {j} \| \boldsymbol {S} _ {j,:} \| _ {1} > n p (\sqrt {2 / \pi} + \epsilon) \right] \\ \leq \Pr \left[ \| S _ {j,:} \| _ {1} > n p (\sqrt {2 / \pi} + \epsilon) \right] \\ \leq \Pr \left[ \left| \sum_ {i = 1} ^ {n} \left(Z _ {i} - \mathrm {E} \left[ Z _ {i} \right]\right) \right| > n p \epsilon \right] \leq 2 \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right), \tag {20} \\ \end{array}
$$
where we pick an arbitrary  $j \in [k]$  in the second line since this event implies that the maximum  $\ell_1$  norm of the rows are lowerbounded, and in the third line each  $Z_i = b_i | g_i|$  satisfies (16). The proof is complete by combining (18), (19), and (20) with  $\beta = np(\sqrt{2/\pi} - \epsilon)$  and  $\gamma = np(\sqrt{2/\pi} + \epsilon)$ .
Proof of Theorem 3.4. We first instantiate Lemma B.5 with
$$
\delta = \frac {n ^ {2} p ^ {2} (\sqrt {2 / \pi} - \epsilon) ^ {2} - 1}{n ^ {2} p ^ {2} (\sqrt {2 / \pi} - \epsilon) (\sqrt {2 / \pi} + \epsilon) \sqrt {k}},
$$
which obviously satisfies  $\delta < 1$  and would also satisfy  $\delta > 0$  if
$$
\epsilon <   \sqrt {\frac {2}{\pi}} - \frac {1}{n p}.
$$
$$
\Pr \left[ \inf  _ {\| \boldsymbol {w} \| = 1} \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} <   1 / n p (\sqrt {2 / \pi} - \epsilon) \right] \leq \left(\left(\frac {3}{\delta}\right) ^ {k} + 1\right) 2 \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right),
$$
Combining (13), (17), and Lemma B.2 with  $\alpha = np(\sqrt{2 / \pi} -\epsilon)$  , we obtain
$$
\begin{array}{l} \Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\top} \tilde {\mathbf {S}} \| _ {1} \leq 1} \| \boldsymbol {w} \| > 1 \right] \leq \Pr \left[ \inf  _ {\| \boldsymbol {w} \| = 1} \| \boldsymbol {S} ^ {\top} \boldsymbol {w} \| _ {1} <   1 / \alpha \right] + \Pr \left[ \max  _ {j} \| \boldsymbol {S} _ {j,:} \| _ {1} <   \alpha \right] \\ \leq 2 \left(k + \left(\frac {3}{\delta}\right) ^ {k} + 1\right) \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right). \tag {21} \\ \end{array}
$$
Further, with
$$
\epsilon <   \frac {(\sqrt {3} k ^ {1 / 4} - 1) \sqrt {2 / \pi}}{\sqrt {3} k ^ {1 / 4} + 1},
$$
where the right hand side is obviously positive, we have
$$
\delta = \frac {n ^ {2} p ^ {2} (\sqrt {2 / \pi} - \epsilon) ^ {2} - 1}{n ^ {2} p ^ {2} (\sqrt {2 / \pi} - \epsilon) (\sqrt {2 / \pi} + \epsilon) \sqrt {k}} > \frac {n ^ {2} p ^ {2} (\sqrt {2 / \pi} - \epsilon) ^ {2}}{n ^ {2} p ^ {2} (\sqrt {2 / \pi} + \epsilon) ^ {2} \sqrt {k}} > \frac {3}{k}.
$$
We can further relax (21) to
$$
\begin{array}{l} \Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\top} \tilde {\mathbf {S}} \| _ {1} \leq 1} \| \boldsymbol {w} \| > 1 \right] \leq 2 (k + (k) ^ {k} + 1) \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right) \leq 4 (k) ^ {k} \exp \left(- \frac {n p \epsilon^ {2}}{2 + \sqrt {2} \epsilon}\right) \\ \leq 4 \exp (k \log (k) - n p (1 - p)), \\ \end{array}
$$
where in the last inequality we simply picked a small-enough  $\epsilon$  so that
$$
\frac {\epsilon^ {2}}{2 + \sqrt {2} \epsilon} <   1 - p.
$$
C Additional Algorithms
In this section we introduce two algorithms that directly tackle the formulation (7), repeated here:
$$
\underset {\boldsymbol {P}} {\text {m i n i m i z e}} \quad - \log | \det  \boldsymbol {P} | \qquad \text {s u b j e c t t o} \quad \| [ \boldsymbol {P} \boldsymbol {X} ] _ {j,:} \| _ {1} \leq 1, j = 1, \ldots , k.
$$
C.1 Frank-Wolfe algorithm
Our first attempt is to apply the Frank-Wolfe algorithm to approximately solve (7). The Frank-Wolfe algorithm, also known as the conditional gradient method for constrained optimization [Bertsekas, 1999], iteratively minimizes a linear objective, defined by the gradient at the current iterate, under the same constraint set to determine the search direction and obtain the next iterate via some line search approach along the search direction. For the log-determinant objective in (7), we have that the gradient is  $-\left(\boldsymbol{P}^{-1}\right)^{\top}$ . As a result, the Frank-Wolfe algorithm for (7) is given in Algorithm 2.
Regarding the line search step, we propose to use the backtracking line search (Armijo rule) [Bertsekas, 1999] to guarantee sufficient decrease of the objective function. Since the constraint set of (7) is convex, as long as  $\pmb{P}_{(t)}$  is feasible, then  $\pmb{P}_{(t + 1)}$  is also feasible since it is a convex combination of  $\pmb{P}_{(t)}$  and  $\pmb{P}_d$ , which are by definition feasible. Therefore, the only nontrivial part is to find a feasible initialization  $\pmb{P}_{(0)}$ . This can be done by optimizing an arbitrary linear objective subject to the same
Algorithm 2 Solving (7) with Frank-Wolfe
initialize  $\pmb{P}_{(0)}$    
for  $t = 0,1,2,\ldots$  until convergence do   
 $\pmb {P}_d = \arg \min_{\pmb{P}} - \mathrm{Tr}(\pmb{P}_{(t)}^{-1}\pmb {P})$  subject to  $\| [PX]_{j,:}\| _1\leq 1,j = 1,\dots ,k$ $\alpha \gets 1$    
while  $-\log |\operatorname *{det}(\pmb {P}_{(t)} + \alpha_t(\pmb {P}_d - \pmb {P}_{(t)}))|>$ $-\log |\operatorname *{det}\pmb {P}_{(t)}| + (\alpha /2)\operatorname *{Tr}(\pmb{P}_{(t)}^{-1}(\pmb {P}_d - \pmb {P}_{(t)}))$  do   
 $\alpha \leftarrow \alpha /2$    
end while   
 $\pmb{P}_{(t + 1)} = \pmb{P}_{(t)} + \alpha (\pmb{P}_{d} - \pmb{P}_{(t)})$    
end for
constraint as (7) (which means one should not apply line search at this step). If  $\mathbf{A}$  is square, there is a simple initialization that works really well in our experience, by setting  $\mathbf{P}_{(0)}$  as a diagonal matrix with
$$
\boldsymbol {P} _ {(0)} (c, c) = 1 / \left\| \boldsymbol {X} _ {j,:} \right\| _ {1},
$$
i.e., rescaling each row of  $X$  to have unit  $\ell_1$  norm.
In terms of complexity, each iteration is dominated by the linear programming with  $k^2$  variables. Without exploiting any structure, the per-iteration complexity could be as high as  $O(k^6)$ . However, the linear programming to be solved in Algorithm 2 is blessed with structures to be exploited to greatly reduce the complexity. Denote  $\pmb{p}_i$  as the  $i$ th row of  $\pmb{P}$ , then the linear programming in each iteration of Algorithm 2 is in fact  $k$  independent problems, each involving only one row of  $\pmb{P}$ ; let  $\pmb{f}_i$  denote the  $i$ th column of  $\pmb{P}_{(t)}^{-1}$ , then we should solve the following problem with  $i = 1, \dots, k$
$$
\underset {\boldsymbol {p} _ {i}} {\text {m i n i m i z e}} - \boldsymbol {f} _ {i} ^ {\top} \boldsymbol {p} _ {i} \quad \text {s u b j e c t t o} \| \boldsymbol {p} _ {i} ^ {\top} \boldsymbol {X} \| _ {1} \leq 1. \tag {22}
$$
Each of these problems involves  $k$  variables, which can be solved with  $O(k^3)$  flops. This important observation brings the per-iteration complexity of Algorithm 2 down to  $O(k^4)$ .
Note that the Frank-Wolfe algorithm is also able to handle the overdetermined case when the dictionary matrix  $\mathbf{A}$  is tall, although this case is rarely considered in the context of dictionary learning: as long as  $\mathbf{A}$  has full column rank, we can make the change-of-variable  $\mathbf{P} = \mathbf{A}^{\dagger}$ , and the objective of (7) can be replaced by  $(1/2)\log(\mathbf{PP}^{\top})$ , with its gradient equals to  $-(\mathbf{P}^{\dagger})^{\top}$ . Everything else follows easily.
We implement the algorithm in MATLAB and use the built-in linprog function in MATLAB to solve each of the linear programming sub-problems (22). Dropping the subscripts in (22), the  $\ell_1$  norm constraint can be transformed as linear ones using the standard trick:
$$
\underset {\boldsymbol {p}, \boldsymbol {t}} {\text {m i n i m i z e}} - \boldsymbol {f} ^ {\top} \boldsymbol {p} \quad \text {s u b j e c t t o} - \boldsymbol {t} \leq \boldsymbol {p} _ {i} ^ {\top} X \leq \boldsymbol {t}, I ^ {\top} \boldsymbol {t} \leq 1.
$$
Using the same setting as in §5.1 but with smaller problem dimension  $n = 200$  and  $k = 10$ , the convergence of 50 random trials are shown in Figure 5.
C.2 Cyclic row update
Inspired by the row-separable structure of the constraint set in (7), we also attempt to solve (7) via a block coordinate descent-type algorithm that cyclically updates each row. However, we note that this algorithm only works when  $\pmb{P}$  is square, unlike the Frank-Wolfe algorithm, which can be extended to the rare overdetermined case when  $\pmb{A}$  is tall. We will drop the log term in the objective function of (7) when designing the cyclic row update algorithm in this subsection.
Using the Laplace's formula, we know that  $\operatorname{det} P$  is a linear function with respect to the  $i$ th row of  $P$  using the co-factor expansion
$$
\det  \boldsymbol {P} = \sum_ {j = 1} ^ {k} (- 1) ^ {i + j} P _ {i j} \det  \boldsymbol {P} _ {i j},
$$
Figure 5: 50 random trials of the Frank-Wolfe Algorithm 2.
where the matrix  $\pmb{P}_{ij}$  is obtained by deleting the  $i$ th row and  $j$ th column of  $\pmb{P}$ . Then to apply the block coordinate descent algorithm [Bertsekas, 1999], one would minimize  $-|\pmb{f}_i^\top \pmb{p}_i|$  subject to  $\| \pmb{p}_i^\top \pmb{X} \|_1 \leq 1$ , where  $\pmb{f}_i \in \mathbb{R}^k$  is a vector defined according to the co-factor expansion formula.
It looks as if we need to solve two linear programs for each row update, one to maximize  $\pmb{f}_i^\top \pmb{p}_i$  and one to minimize  $\pmb{f}_i^\top \pmb{p}_i$ , but notice that the constraint  $\| \pmb{p}_i^\top \pmb{X} \|_1 \leq 1$  is invariant under a sign flip, therefore the only difference between the solutions of the two linear programs would be a sign flip, which is both acceptable due to the inherent sign ambiguity in matrix factorization and dictionary learning in specific. Furthermore, recall that the Cramer's rule shows that
$$
\left[ \boldsymbol {P} ^ {- 1} \right] _ {j i} = (- 1) ^ {i + j} \det \boldsymbol {P} _ {i j} \Bigg / \det \boldsymbol {P},
$$
meaning we can simply define  $\pmb{f}_i$  as the  $i$ th column of  $P^{-1}$ , and it would not affect the row updates. This makes the definition of  $\pmb{f}_i$  exactly the same way as in (22).
Algorithm 3 Solving (7) with cyclic row update
initialize  $\pmb{P}_{(0)}$    
repeat for  $i = 1,\dots ,k$  do  $\pmb {f} = \pmb{P}^{-1}\pmb {e}_i$ $\pmb {p} = \arg \min_{\pmb{p}} - \pmb{f}^{\top}\pmb{p}$  subject to  $\| p^{\top}X\| _1\leq 1$  replace ith row of  $\pmb{P}$  with  $\pmb{p}$    
end for   
until convergence
The two proposed algorithms 2 and 3 shows striking similarities, especially considering they both fundamentally solve linear programs in the form of (22) in each iteration. For nonconvex optimization, both algorithms guarantees that every limit point is a stationary point with some additional assumptions, which is reassuring to know. The differences are as follows: