SlowGuess/NeurIPS_2023_OCR huggingface dataset:
How is this assumption relevant to the sparsity of  $S$ ? Notice that if  $SV$  is sufficiently scattered, then for any vector  $\mathbf{u}$  such that  $\| \mathbf{u} \| \leq 1$ , there must exist a convex coefficient  $\theta$ , i.e.,  $\theta \geq 0$  and  $I^{\top} \theta = 1$ , such that  $\mathbf{u} = SV \theta$ . Due to the definition of  $V$ , we know that the vector  $\mathbf{y} = V \theta$  must satisfy  $\| \mathbf{y} \|_{\infty} \leq 1$ . Let us instantiate it at  $\mathbf{u} = \mathbf{e}_j$ . This means the  $j$ th row of  $S$  inner product with  $\mathbf{y}$  equals to 1. But the  $j$ th row of  $S$  has unit  $\ell_1$  norm and  $\| \mathbf{y} \|_{\infty} \leq 1$ , so their inner product is 1 only if  $\mathbf{y}^{\top} = \mathrm{sign}(S_{j,:})$ , which is uniquely defined if  $S_{j,:}$  is completely dense. Now  $\mathbf{e}_j = S \mathbf{y}$  also implies the inner products between  $\mathbf{y}$  and the rest of the rows of  $S$  all equal to zero, which is almost impossible if  $\mathbf{y}^{\top} = \mathrm{sign}(S_{j,:})$  is the only candidate. On the other hand, if some of the entries in  $S_{j,:}$  are zeros, then the corresponding entries in  $\mathbf{y}$  are free to choose any value in  $[-1, 1]$ ; to make the rest of the  $k - 1$  inner products equal to zero, we generically need at least  $k - 1$  free variables in  $\mathbf{y}$ , meaning  $S_{j,:}$  contains at least  $k - 1$  zeros—it is still not guaranteed that  $\| \mathbf{y} \|_{\infty} \leq 1$ , but it intuitively makes sense that the more zeros each row of  $S$  contains, the more likely that  $\mathbf{e}_j = S \mathbf{y}$  is feasible, which is necessary for  $SV$  to be sufficiently scattered.
3.2 Probabilistic characterization
The analysis given in §3.1 gives an exact characterization of when dictionary learning is identifiable using the proposed formulation (1) with a matrix volume identification criterion. However, given some sparse coefficient matrix  $S$ , checking whether it satisfies the identifiability condition in Theorem 3.2 amounts to solving a nonconvex quadratic programming (5), which is NP-hard in general. In this section, we assume that the sparse coefficient matrix  $S$  is generated from a probabilistic generative model, specifically the Bernoulli-Gaussian model that has been widely considered in the literature, such as [Gribonval and Schnass, 2010, Sun et al., 2016a], and show that it would satisfy guarantee identifiability with high probability, provided that the number of data points  $n$  is  $O(k\log k)$ .
Assumption 3.3 (Bernoulli-Gaussian model). The matrix  $S \in \mathbb{R}^{k \times n}$  is generated from a Bernoulli-Gaussian model with parameter  $p \in (0,1)$ , denoted as  $S \sim \mathcal{B}\mathcal{G}(p)$ , if its elements are i.i.d. with  $S_{ij} = b_{ij}g_{ij}$ , where  $b_{ij} \in \{0,1\}$  are i.i.d. Bernoulli random variables with  $\operatorname*{Pr}[b_{ij} = 1] = p$  (thus  $\operatorname*{Pr}[b_{ij} = 0] = 1 - p$ ) and  $g_{ij} \sim \mathcal{N}(0,1)$  are i.i.d. standard normal random variables.
Note that due to the scaling ambiguity of matrix factorization the assumption that the normal random variable has unit variance is without loss of generality—its rows will be rescaled to have unit  $\ell_1$  norms nonetheless. What we are essentially assuming is that 1) a significant portion of  $S$  are zero; 2) nonzeros of  $S$  have zero mean and approximately the same variance.
Theorem 3.4. Suppose  $S \in \mathbb{R}^{k \times n}$  is generated from the Bernoulli-Gaussian model  $\mathcal{BG}(p)$ , and  $\widetilde{S}$  is obtained by scaling its rows to have unit  $\ell_1$  norm. Then
$$
\Pr \left[ \sup  _ {\| \boldsymbol {w} ^ {\intercal} \tilde {\boldsymbol {S}} \| _ {1} \leq 1} \| \boldsymbol {w} \| > 1 \right] \leq 4 \exp (k \log (k) - n p (1 - p)). \tag {6}
$$
Corollary 3.5. Consider the generative model  $X = A^{\natural}S^{\natural}$ , where  $A^{\natural} \in \mathbb{R}^{k\times k}$  is the groundtruth dictionary and  $S$  is the groundtruth sparse coefficients. If  $\mathrm{rank}(A^{\natural}) = k$  and the matrix  $S^{\natural} \in \mathbb{R}^{k\times n}$  is generated from the Bernoulli-Gaussian model  $\mathcal{BG}(p)$ , then  $(A^{\natural}, S^{\natural})$  are globally identifiable via optimizing (1) with probability at least  $1 - 4\exp (k\log (k) - np(1 - p))$ .
The proof of Theorem 3.4 is relegated to Appendix B. It is appealing to see that the presented result looks much cleaner than prior work such as [Gribonval and Schnass, 2010, Wu and Yu, 2017, Wang et al., 2020], on top of the fact that such identifiability result is global, and does not require any incoherence assumption on the dictionary, as long as it is nonsingular. An interesting observation is that most of the foundational results have appeared in the seminal work [Gribonval and Schnass, 2010]; the difference is that they are used in the improved formulation (1), and in return leads to more significant result in Corollary 3.5. From the above result, we observe the following:
1. The right-hand-side of (6) would be a valid probability  $(< 1)$  if  $n$  is bigger than a constant times  $k\log (k) / p(1 - p)$ , and very quickly approaches zero as it grows bigger. This agrees with the prior work on local identifiability that the sample complexity is of the same order [Gribonval and Schnass, 2010, Geng and Wright, 2014, Wu and Yu, 2017, Wang et al., 2020].
2. The Bernoulli parameter  $p$  cannot be exactly equal to 1 or 0, which affirms the obvious fact that  $S$  can neither be completely dense nor entirely zero. The more interesting implication is that DL is identifiable (with high probability) as long as there is some nonzero probability of having zeros in  $S$ , no matter how small that probability is, provided the sample size is large enough.
4 Proposed algorithm
In this section, we propose two algorithms for the volume-optimization based formulation (1). We will reformulate (1) as a determinant maximization problem subject to linear constraints, and then apply the Frank-Wolfe algorithm or block coordinate descent, which are both guaranteed to converge to a stationary point.
Since we assume  $\mathbf{A}$  is nonsingular, we can define  $\mathbf{P} = \mathbf{A}^{-1}$  and apply a change of variable to problem (1): the objective would become minimizing  $1 / |\det \mathbf{P}|$ , which we apply the log function and make it - log det  $|\mathbf{P}|$ , and in the constraints we can now eliminate the  $S$  variables by simply requiring  $\| \mathbf{P}\mathbf{X}_{j:}\|_{1}\leq 1,j = 1,\ldots ,k$ . This leads to the following reformulation
$$
\underset {\boldsymbol {P}} {\text {m i n i m i z e}} - \log | \det  \boldsymbol {P} | \quad \text {s u b j e c t t o} \| [ \boldsymbol {P} \boldsymbol {X} ] _ {j,} \| _ {1} \leq 1, j = 1, \dots , k. \tag {7}
$$
Problem (7) now has a convex, or more specifically linear, constraint set, although the objective is still not convex. In Appendix 4, we propose two algorithms to directly tackle this formulation, one based on the Frank-Wolfe framework, and the other is an instance of block coordinate descent. Both algorithms guarantee convergence to a stationary point, which is the best one can get for a generic nonconvex problem. However, both algorithms require solving a linear programming problem in each iteration, making the overall algorithm double-looped, and the time efficiency is greatly affected by the specific subroutine to solve linear programming. In the rest of this section, we propose an algorithm based on the linearized alternating direction method of multipliers (L-ADMM) with well-defined and low-complexity iterations.
4.1 Linearized alternating direction method of multipliers (L-ADMM)
We first modify the formulation (7) by introducing an auxiliary variable  $S$ :
$$
\underset {\boldsymbol {P}, \boldsymbol {S}} {\text {m i n i m i z e}} - \log | \det  \boldsymbol {P} | \quad \text {s u b j e c t t o} \quad \boldsymbol {P} \boldsymbol {X} = \boldsymbol {S}, \| \boldsymbol {S} _ {j,:} \| _ {1} \leq 1, j = 1, \dots , k. \tag {8}
$$
Formulation (8) consists of two sets of variables  $(\pmb{P}$  and  $\pmb{S}$ ) over two separable functions (one of them being an indicator function of  $\pmb{S}$  that the  $\ell_1$  norms of its rows are no bigger than 1) and linear equality constraints. It is easy to derive the alternating direction method of multipliers (ADMM) [Boyd et al., 2011] for (8):
$$
\left\{ \begin{array}{l} \boldsymbol {P} _ {(t + 1)} \leftarrow \arg \min  _ {\boldsymbol {P}} - \log | \det  \boldsymbol {P} | + (\rho / 2) \| \boldsymbol {P} \boldsymbol {X} - \boldsymbol {S} _ {(t)} + \boldsymbol {U} _ {(t)} \| ^ {2}, \\ \boldsymbol {S} _ {(t + 1)} \leftarrow \arg \min  _ {\boldsymbol {S}} \mathbb {1} _ {\| \cdot \| _ {1} \leq 1} (\boldsymbol {S}) + (\rho / 2) \| \boldsymbol {P} _ {(t + 1)} \boldsymbol {X} - \boldsymbol {S} + \boldsymbol {U} _ {(t)} \| ^ {2}, \\ \boldsymbol {U} _ {(t + 1)} \leftarrow \boldsymbol {U} _ {(t)} + \boldsymbol {P} _ {(t + 1)} \boldsymbol {X} - \boldsymbol {S} _ {(t + 1)}. \end{array} \right. \tag {9a}
$$
The second step (9b) is well-defined, as it projects each row of the matrix  $\mathbf{P}_{(t + 1)}\mathbf{X} + \mathbf{U}_{(t)}$  to the  $\ell_1$  norm ball, which can be efficiently computed with linear complexity [Duchi et al., 2008]. In our implementation we use the bisection method described in [Parikh and Boyd, 2014]. The first step (9a), however, is not clear how to compute. One popular method to mitigate this issue, which is prevalent in ADMM, is to take a linear approximation of the loss function of  $\mathbf{P}$  at the previous iterate  $\mathbf{P}_{(t)}$  [Lu et al., 2021]. The gradient of  $\log |\det \mathbf{P}|$  is  $\mathbf{P}^{-\top}$ , therefore the linear approximation of  $-\log |\det \mathbf{P}|$  at  $\mathbf{P}_{(t)}$  is  $-\log |\det \mathbf{P}_{(t)}| - \operatorname{Tr}\mathbf{P}_{(t)}^{-1}(\mathbf{P} - \mathbf{P}_{(t)})$ . The overall update of  $\mathbf{P}_{(t + 1)}$  becomes minimizing a convex quadratic function, which can be done in closed-form. The derived linearized ADMM (L-ADMM) iterates are
$$
\left\{ \begin{array}{l} \boldsymbol {P} _ {(t + 1)} \leftarrow \left(\left(\boldsymbol {S} _ {(t)} - \boldsymbol {U} _ {(t)}\right) \boldsymbol {X} ^ {\top} + \left(1 / \rho\right) \boldsymbol {P} _ {(t)} ^ {- \top}\right) \left(\boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {- 1}, \\ \boldsymbol {S} _ {(t + 1)} \leftarrow \operatorname {P r o j} _ {\| \cdot \| _ {1} \leq 1} \left(\boldsymbol {P} _ {(t + 1)} \boldsymbol {X} + \boldsymbol {U} _ {(t)}\right), \\ \boldsymbol {U} _ {(t + 1)} \leftarrow \boldsymbol {U} _ {(t)} + \boldsymbol {P} _ {(t + 1)} \boldsymbol {X} - \boldsymbol {S} _ {(t + 1)}. \end{array} \right. \tag {10a}
$$
Finally, we notice that formulation (8) and the derived L-ADMM algorithm are invariant under linear transformation of columns of  $X$ , meaning if we replace  $X$  with  $\widetilde{X} = GX$  in (8), where  $G$  is a  $k\times k$  invertible matrix, then every iterate  $P_{(t)}$  is uniquely mapped to  $P_{(t)}G^{-1}$  (while  $S_{(t)}$  and  $U_{(t)}$  are exactly the same), and the objective value has a constant difference  $-\log |\det P_{(t)}G^{-1}| = -\log |\det P_{(t)}| + \log |\det G|$ . This observation allows us to preprocess the data matrix  $X$  by orthogonalizing its rows, so that the update for  $P$  in (10a) can be further simplified. The overall algorithm is summarized in Algorithm 1. We empirically found that setting  $\rho = nk$  works very well in practice.
Algorithm 1 Solving (7) with L-ADMM
1:	take the QR factorization of X^T = QR
2:	set ρ = nk and initialize P(0)
3:	for t = 0, 1, 2, ... until convergence do
4:	P(t+1) ← (S(t) - U(t))Q + (1/ρ)P^T(t)
5:	S(t+1) ← Proj||·||1≤1(P(t+1)Q^T + U(t))
6:	U(t+1) ← U(t) + P(t+1)Q^T - S(t+1)
7:	end for
8:	return P(t)R^-T
We now provide some numerical experiments to showcase the effectiveness of the proposed volume optimization formulation using the two algorithms described in §4. All the experiments are conducted in MATLAB.
5.1 Optimization performance
We synthetically generate random problems, and show the surprisingly effective performance of the two algorithms proposed in §4. For  $k = 20$  and  $n = 1000$ , we randomly generate the groundtruth sparse coefficient matrix  $S^{\natural}$  according to the Bernoulli-Gaussian model with  $p = 0.5$ , and the groundtruth dictionary matrix  $A^{\natural}$  completely random, and generate the data matrix  $X = A^{\natural}S^{\natural}$ . Due to Theorem 3.4, we know it is identifiable with very high probability, despite  $n$  being not that big compared to the number of atoms  $k$ . Matrix  $X$  is used as input to the L-ADMM algorithm as described in Alg. 1. Although Problem (7) is nonconvex, as long as it is identifiable, we know the global optimum is attained at  $P^{\star} = D(A^{\natural})^{-1}$ , where  $D$  is a diagonal matrix with the  $i$ th diagonal equal to the inverse of the  $\ell_1$  norm of the  $i$ th row of  $S^{\natural}$ . As a result,  $-\log |\det D(A^{\natural})^{-1}|$  is the optimal value for Problem (7) as long as the model is identifiable, and we shall see whether the proposed algorithm is able to attain that optimal value. On the other hand, since L-ADMM directly tackles formulation 8, it is not guaranteed that  $P_{(t)}$  is feasible in every iteration, which makes little sense to check the difference  $-\log |\det P_{(t)}| + \log |\det P^{\star}|$ . We instead check the optimality gap of the Lagrangian function values, using the optimal dual variable  $\Lambda$ , since we have
$-\log |\operatorname{det}P| + \operatorname{Tr}(PX - S)\Lambda^{\star}\geq -\log |\operatorname{det}P^{\star}|,$  for any  $P$  and a feasible  $S$ . Obviously, the gap equals to zero when  $P = P^{\star}$  and  $S = S^{\star}$ , in which case  $P^{\star}X - S^{\star} = 0$ . Furthermore, it is easy to show that an optimal  $\varLambda$  is  $(\widetilde{S}^{\natural})^{\dagger}$ . In this simulation with known groundtruth factors, we will use this to measure the optimality gap.
The convergence behavior of 100 random trials of the L-ADMM Algorithm 1 are shown in Figure 2. The somewhat surprising observation is that it achieves essentially zero optimality gap in all instances, which is somewhat surprising as we are, after all, trying to solve NP-hard problems. Among the 100 random trials, L-ADMM usually requires approximately 500-1000 iterations to converge; together with the low-complexity iterations described in Algorithm 1, it usually takes just a few seconds to execute a trial. For a dictionary learning problem of this size, our proposed algorithm takes a lot less time than most existing state-of-the-arts. Some more detailed comparisons can be found in the supplementary material.
Figure 2: 100 random trials of the proposed L-ADMM for dictionary learning as in Algorithm 1. Since we know the groundtruth dictionary is the (essentially) unique minimizer, we can use it to calculate the optimality gap. We see that the optimality gap goes to zero in all random instances.
5.2 Dictionary recovery
Next we fix the sample size  $n = 1000$  and vary the dictionary size as well as the sparsity parameter  $p$  in the Bernoulli-Gaussian model and check how often do we get exact recovery of the groundtruth dictionary  $A^{\natural}$  up to column permutation and scaling. In the previous experiment we simply check the optimality gap between the algorithm output and  $-\log |\det D(A^{\natural})^{-1}|$ , which is also a pretty good indicator of exact recovery since the only ambiguity is an orthogonal rotation, which is very unlikely to exist. Nevertheless, to ensure definitively that the dictionary is recovered as  $P^{-1}$ , we will normalize the column and use the Hungarian algorithm [Kuhn, 1955] to find the best column matching, and then calculate the estimation error. We declare success if the estimation error is smaller than 1e-5. The results are shown in Figure 3, which agrees with the bound in Theorem 3.4.
Figure 3: The probability of exactly recovering the dictionary for various dictionary size  $k$  and sparsity  $p$  (probability of nonzeros in  $S^{\natural}$ ). The sample size is fixed with  $n = 1000$ .
Figure 4: Learned dictionary from a natural image.
We would like to emphasize that compared to similar figures that have appeared in the literature, the groundtruth dictionaries that we try to identify are far from being orthogonal, nor do we care to check whether they satisfy the incoherence assumption; moreover, the recovery is of the entire dictionary, not one of the atoms. More comparisons with some existing algorithms can be found in the supplementary.
5.3 Real data experiment
We conclude this section with a classical application of dictionary learning on image compression. For a given image, it is first divided into  $8 \times 8$  non-overlapping patches, reshaped into a vector in  $\mathbb{R}^k$  with  $k = 64$ , and stacked as columns of the data matrix  $\mathbf{X}$ . After feeding it into the L-ADMM Algorithm 1, the output is inverted to obtain an estimate of the dictionary  $\mathbf{A}$ . Each column of  $\mathbf{A}$  is then reshaped back into a  $8 \times 8$  patch to be shown as an atom for the dictionary. For a monochrome image with  $512 \times 512$  pixels, it would be reshaped into a data matrix  $\mathbf{X}$  of size  $64 \times 4096$ . Thanks to the high-efficiency of the proposed L-ADMM algorithm, learning a dictionary on a data set of this size takes only a few minutes from a random initialization. The resulting dictionary learned from a natural image is shown in Figure 4. The result agrees with most other dictionary learning algorithms, with some atoms representing edges and bright spots, while some others resemble a few DCT bases.
In this paper we aimed at answering the question of when dictionary learning is identifiable, with a specific requirement that the formulation uses the  $\ell_1$  norm to promote sparsity. With a novel formulation that minimizes the volume of the dictionary subject to  $\ell_1$  constraints, we close the case with an affirmative yes, complete dictionary learning is indeed globally identifiable under mild conditions. Compared to similar works, our result is global, requires no conditions such as incoherence on the complete dictionary, and at the same time achieves the same sample complexity bound when the sparse coefficients satisfy the Bernoulli-Gaussian model. Finally, we propose algorithms to solve the volume optimization problem, specifically one based on linearized ADMM in the main paper, and demonstrate their exceptional performances on both synthetic and real data.
This work is supported in part by NSF ECCS-2237640.
Daniel A. Spielman, Huan Wang, and John Wright. Exact recovery of sparsely-used dictionaries. In Shie Mannor, Nathan Srebro, and Robert C. Williamson, editors, Proceedings of the 25th Annual Conference on Learning Theory, volume 23 of Proceedings of Machine Learning Research, pages 37.1-37.18, Edinburgh, Scotland, 25-27 Jun 2012. PMLR.
Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere i: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2):853-884, 2016a.
Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere ii: Recovery by riemannian trust-region method. IEEE Transactions on Information Theory, 63(2):885-914, 2016b.
Gokcan Tatli and Alper T Erdogan. Polytopic matrix factorization: Determinant maximization based criterion and identifiability. IEEE Transactions on Signal Processing, 69:5431-5447, 2021.
Bahareh Tolooshams and Demba E Ba. Stable and interpretable unrolled dictionary learning. Transactions on Machine Learning Research, 2022.
Ivana Tošić and Pascal Frossard. Dictionary learning. IEEE Signal Processing Magazine, 28(2): 27-38, 2011.
Ivana Tošić, Ivana Jovanović, Pascal Frossard, Martin Vetterli, and Neb Durić. Ultrasound tomography with learned dictionaries. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5502-5505. IEEE, 2010.
Yu Wang, Siqi Wu, and Bin Yu. Unique sharp local minimum in 11-minimization complete dictionary learning. Journal of Machine Learning Research, 21(63):1-52, 2020.
Siqi Wu and Bin Yu. Local identifiability of 11-minimization dictionary learning: a sufficient and almost necessary condition. The Journal of Machine Learning Research, 18(1):6121-6176, 2017.
Yuexiang Zhai, Hermish Mehta, Zhengyuan Zhou, and Yi Ma. Understanding 14-based dictionary learning: Interpretation, stability, and robustness. In International Conference on Learning Representations, 2020a.
Yuexiang Zhai, Zitong Yang, Zhenyu Liao, John Wright, and Yi Ma. Complete dictionary learning via l4-norm maximization over the orthogonal group. Journal of Machine Learning Research, 21 (165):1-68, 2020b.
The proof uses the notion of polar sets from convex analysis [Rockafellar, 1970]:
Definition A.1 (Polar set). Given a set  $C$  that contains the origin, its polar set, denoted as  $C^\circ$ , is defined as
$$
\mathcal {C} ^ {\circ} = \{\boldsymbol {x} \mid \boldsymbol {x} ^ {\top} \boldsymbol {y} \leq 1, \forall \boldsymbol {y} \in \mathcal {C} \}.
$$
As an example, the polar of the unit ball  $\mathcal{B} = \{\pmb{x}||\pmb{x}||\leq 1\}$  is itself, i.e.,  $\pmb{B}^{\circ} = \pmb{B}$ , using the Cauchy-Schwartz inequality. Due to the definition of dual norm, we also have immediately that the polar set of any norm ball is the norm ball defined by its dual norm, e.g., the  $\ell_{\infty}$  norm ball  $[-1,1]^n$  is the polar of the  $\ell_1$  norm ball  $\{\pmb{x}||\pmb{x}||_1\leq 1\}$ . Given a polyhedron in the form of  $\mathrm{conv}(\pmb{H})$ , i.e., the convex hull of a set of vectors given as the columns of  $\pmb{H}$ , then its polar is a polyhedron in the form of linear inequalities:
$$
\operatorname {c o n v} (\boldsymbol {H}) ^ {\circ} = \left\{\boldsymbol {x} \mid \boldsymbol {H} ^ {\top} \boldsymbol {x} \leq 1 \right\}.
$$
The important property of the polar sets is that if two sets satisfy  $\mathcal{A} \subseteq \mathcal{B}$ , then  $\mathcal{B}^{\circ} \subseteq \mathcal{A}^{\circ}$ , which will be crucial in the sequel. The proof is not hard and can be found in standard textbooks such as [Rockafellar, 1970].
Before proving Theorem 3.2, we show the following lemma as intermediate step:
Lemma A.2. If  $\operatorname{conv}(\widetilde{\mathbf{S}}^{\sharp}\mathbf{V})$  is sufficiently scattered in the hypercube as described in Assumption 3.1, then the solution set of problem (5) is  $\{\pm \pmb{e}_1,\dots ,\pm \pmb {e}_k\}$ .
Proof. As explained in the main paper, the single constraint  $\| \pmb{w}^{\top}\widetilde{\pmb{S}}^{\sharp}\|_{1}\leq 1$  is equivalent to a set of  $2^{n}$  linear inequalities
$$
\boldsymbol {w} ^ {\top} \tilde {\boldsymbol {S}} ^ {\natural} \boldsymbol {V} \leq \boldsymbol {I} ^ {\top},
$$
where  $\pmb{V} \in \{\pm 1\}^{n \times 2^n}$  contains all  $2^n$  vectors  $\{\pm 1\}^n$  as its columns. This means  $\pmb{w} \in \mathrm{conv}(\widetilde{\pmb{S}}^\natural \pmb{V})^\circ$ , i.e.,  $\pmb{w}$  belongs to the polar set of  $\mathrm{conv}(\widetilde{\pmb{S}}^\natural \pmb{V})$ . The sufficiently scattered condition first assumes that  $\mathcal{B} \subseteq \mathrm{conv}(\widetilde{\pmb{S}}^\natural \pmb{V})$ , so  $\mathrm{conv}(\widetilde{\pmb{S}}^\natural \pmb{V})^\circ \subseteq \mathcal{B}$  since  $\mathcal{B}^\circ = \mathcal{B}$ . We immediately have that any feasible  $\pmb{w}$  for (5) must satisfy  $\| \pmb{w} \| \leq 1$ . Obviously, letting  $\pmb{w} = \pm e_j$  is feasible and attains the maximum.