SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Block coordinate descent is guaranteed to monotonically improve the objective function by design, so there is no need for line search as in Frank-Wolfe. This could save some computation as calculating  $\operatorname{det} P$  may not be cheap when  $k$  is large.
2. On the other hand, each iteration of Frank-Wolfe only need to calculate  $P^{-1}$  once for the update of all its  $k$  rows, whereas cyclic row update requires to recalculate  $P^{-1}$  for each of its row updates. Overall, the per-iteration complexity is almost identical.
An interesting observation is that solving one subproblem (22) resembles the problem formulations in works such as [Spielman et al., 2012, Geng and Wright, 2014, Sun et al., 2016a, Zhai et al., 2020b, Shen et al., 2020] that attempt to find one atom of the dictionary at a time in a deflation manner. The difference is that they are still trying to solve nonconvex problems, and they typically require multiple random initializations due to the risk of finding the same atoms multiple times. From a different perspective, we see that the benefit of using the proposed volume optimization framework provides a
smartly chosen search direction given by rows of  $P^{-1}$ , which is by definition the current iterate of the dictionary.
This algorithm is again implemented in MATLAB using the built-in 1 inprog function. Using the same setting as before with the Frank-Wolfe algorithm, the convergence of 50 random trials are shown in Figure 6. Comparing the two algorithms specifically, we see that Frank-Wolfe succeeds in all 50 trials, while CRU has one failure, although CRU takes much fewer number of iterations to converge.
Figure 6: 50 random trials of the cyclic row update Algorithm 3.
Here we provide some additional experiments to showcase the performance of the proposed volume-optimization formulation as well as the three proposed algorithms. We start by demonstrating the time-efficiency of the three proposed algorithms to motivate the use of L-ADMM as the go-to method. Then we compare with some state-of-the-art dictionary learning algorithms with identifiability guarantees to showcase the improved performance of dictionary recovery by incorporating the volume criterion.
D.1 Comparing the three proposed algorithms
In Figures 2, 5, and 6, we showed the per-iteration convergence of L-ADMM, Frank-Wolfe, and cyclic row update (i.e., block coordinate descent [BCD]) as described in Algorithm 1, 2, and 3. While all three algorithms manage to obtain essentially zero optimality gap, L-ADMM takes significantly more number of iterations; on the other hand, it is obvious that the cost of L-ADMM to perform one iteration is much cheaper than those of Frank-Wolfe or BCD. In this subsection we show that, thanks to the simple and efficient per-iteration update of L-ADMM, it is significantly faster than the other two proposed algorithms. We generate synthetic data sets like before, with  $n = 1000$  and the sparsity level  $p = 0.5$ , and set  $k = 20$  in Figure 7a and  $k = 50$  in Figure 7b. Notice that both the vertical and horizontal axes are in log scale, indicating that L-ADMM is several magnitude faster than the other alternatives. The difference would only become bigger as we increase  $n$  and  $k$ .
D.2 Dictionary recovery
We compare the dictionary recovery performance of the proposed volume-optimization formulation (using the L-ADMM algorithm in Algorithm 1) with some state-of-the-arts. The most related work are [Gribonval and Schnass, 2010, Wu and Yu, 2017, Wang et al., 2020], which focus on the local identifiability of formulation (2) without proposing new algorithms. For algorithms with claimed identifiability guarantees, the closest we find are the alternating direction method (ADM) by Sun et al. [2016a,b],  $\ell_4$ -norm maximization ( $\ell_4$ -max) by Zhai et al. [2020b], and  $\ell_3$ -norm maximization ( $\ell_3$ -max) by Shen et al. [2020]. We note that ADM is not the main algorithm that is analyzed in Sun et al. [2016a,b], but it works better and has the benefit of recovering the entire dictionary simultaneously; the analyzed manifold optimization algorithm in [Sun et al., 2016a,b] recovers the
(a)  $k = 20$
(b)  $k = 50$
Figure 7: Convergence (with respect to time in seconds) of the three proposed algorithms. In both cases  $n = 1000$  and  $p = 0.5$ , and we vary the value of  $k$ . It is clear from both figures that L-ADMM is significantly more efficient than Frank-Wolfe or BCD.
dictionary one atom at a time, which requires multiple initializations to guarantee complete recovery, and such procedures are not provided by the authors on their Github page.
Note that all three baseline algorithms, ADM,  $\ell_4$ -max, and  $\ell_3$ -max, require that the groundtruth dictionary to be orthogonal. They claim that this is, to some extent, without loss of generality by pre-whitening the data  $(\boldsymbol{X}\boldsymbol{X}^{\top})^{-1/2}\boldsymbol{X}$ . We discover that this claim is only true if the sparse coefficients matrix  $\boldsymbol{S}$  follows strictly from the Bernoulli-Gaussian generative model, and that the sample size tends to infinity, in which case  $(1/n)\boldsymbol{S}\boldsymbol{S}^{\top} \to p\boldsymbol{I}$ ; if this assumption does not hold, such a pre-whitening step is not able to make an arbitrary dictionary orthogonal. As a result, when we use the same experimental setting as in §5.2, none of these algorithms are able to exactly recover the true dictionary in any circumstance.
If we restrict ourselves to orthogonal dictionary recovery, then we obtain some sensible results from the three baseline algorithms, although they still perform significantly worse than our proposed work. What is more, even for exactly orthogonal dictionaries they are not able to recover them with very high accuracy. Here we test them in two cases: In the first case shown in Tables 1 and 2, we fix  $n = 1000$  and  $k = 20$ , and vary the sparsity level  $p$  from 0.1 to 0.9; in the other case shown in Tables 3 and 4, we fix  $n = 1000$  and  $p = 0.5$ , and vary the dictionary size  $k$  from 5 to 50. The recovered dictionary is again permuted using the Hungarian algorithm, and then we declare a successful recovery if the error is less than a threshold. In each case we present two tables, one with the threshold to be  $10^{-5}$  (same as the threshold used in §5.2), and the other with threshold  $10^{-2}$ . We see that the three baseline algorithms can provide reasonable results with the more loose threshold, especially  $\ell_3$ -max. Regardless, the proposed volume-optimization formulation with the L-ADMM, performs significantly better than all of them.
Table 1: Probability of success when fixing  $n = {1000}$  and  $k = {20}$  ,threshold  ${10}^{-5}$
p=0.1	p=0.2	p=0.3	p=0.4	p=0.5	p=0.6	p=0.7	p=0.8	p=0.9
This work	1	1	1	1	0.9	1	0.9	0	0
ADM [Sun et al., 2016a,b]	1	1	0.9	0.1	0	0	0	0	0
\( \ell_4-max \) [Zhai et al., 2020b]	0	0	0	0	0	0	0	0	0
\( \ell_3-max \) [Shen et al., 2020]	0	0	0	0	0	0	0	0	0
Table 2: Probability of success when fixing  $n = {1000}$  and  $k = {20}$  ,threshold  ${10}^{-2}$
p=0.1	p=0.2	p=0.3	p=0.4	p=0.5	p=0.6	p=0.7	p=0.8	p=0.9
This work	1	1	1	1	1	1	1	0.2	0
ADM [Sun et al., 2016a,b]	1	1	0.9	0.2	0	0	0	0	0
\( \ell_4\text{-max} \) [Zhai et al., 2020b]	0	0	0	0	0	0	0	0	0
\( \ell_3\text{-max} \) [Shen et al., 2020]	1	1	0.9	0.3	0	0	0	0	0
Table 3: Probability of success when fixing  $n = {1000}$  and  $p = {0.5}$  ,threshold  ${10}^{-5}$
k=5	k=10	k=15	k=20	k=25	k=30	k=35	k=40	k=45	k=50
This work	1	1	1	1	0.9	1	1	1	1	0.8
ADM [Sun et al., 2016a,b]	0.9	0.6	0.2	0	0	0	0	0	0	0
\( \ell_4 \)-max [Zhai et al., 2020b]	0	0	0	0	0	0	0	0	0	0
\( \ell_3 \)-max [Shen et al., 2020]	0	0	0	0	0	0	0	0	0	0
Table 4: Probability of success when fixing  $n = {1000}$  and  $p = {0.5}$  ,threshold  ${10}^{-2}$
k=5	k=10	k=15	k=20	k=25	k=30	k=35	k=40	k=45	k=50
This work	1	1	1	1	1	1	1	1	1	0.8
ADM [Sun et al., 2016a,b]	1	0.6	0.2	0	0	0	0	0	0	0
\( \ell_4 \)-max [Zhai et al., 2020b]	0.7	0	0	0	0	0	0	0	0	0
\( \ell_3 \)-max [Shen et al., 2020]	1	0.9	0	0	0	0	0	0	0	0