SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Geyu Liang
lianggy@umich.edu
Naichen Shi
naichens@umich.edu
Raed Al Kontar
alkontar@umich.edu
Salar Fattahi
fattahi@umich.edu
We introduce a relevant yet challenging problem named Personalized Dictionary Learning (PerDL), where the goal is to learn sparse linear representations from heterogeneous datasets that share some commonality. In PerDL, we model each dataset's shared and unique features as global and local dictionaries. Challenges for PerDL not only are inherited from classical dictionary learning (DL), but also arise due to the unknown nature of the shared and unique features. In this paper, we rigorously formulate this problem and provide conditions under which the global and local dictionaries can be provably disentangled. Under these conditions, we provide a meta-algorithm called Personalized Matching and Averaging (PerMA) that can recover both global and local dictionaries from heterogeneous datasets. PerMA is highly efficient; it converges to the ground truth at a linear rate under suitable conditions. Moreover, it automatically borrows strength from strong learners to improve the prediction of weak learners. As a general framework for extracting global and local dictionaries, we show the application of PerDL in different learning tasks, such as training with imbalanced datasets and video surveillance.
Given a set of  $n$  signals  $\mathbf{Y} = [\mathbf{y}_1, \dots, \mathbf{y}_n] \in \mathbb{R}^{d \times n}$ , dictionary learning (DL) aims to find a dictionary  $\mathbf{D} \in \mathbb{R}^{d \times r}$  and a corresponding code  $\mathbf{X} = [\mathbf{x}_1, \dots, \mathbf{x}_n] \in \mathbb{R}^{r \times n}$  such that: (1) each data sample  $\mathbf{y}_i$  can be written as  $\mathbf{y}_i = \mathbf{D}\mathbf{x}_i$  for  $1 \leq i \leq n$ , and (2) the code  $\mathbf{X}$  has as few nonzero elements as possible. The columns of the dictionary  $\mathbf{D}$ , also known as atoms, encode the "common features" whose linear combinations form the data samples. A typical approach to solve DL is via the following optimization problem:
$$
\min  _ {\mathbf {X}, \mathbf {D}} \| \mathbf {Y} - \mathbf {D X} \| _ {F} ^ {2} + \lambda \| \mathbf {X} \| _ {\ell_ {q}}, \tag {DL}
$$
Here  $\| \cdot \|_{\ell_q}$  is often modeled as a  $\ell_1$ -norm (Arora et al., 2015; Agarwal et al., 2016) or  $\ell_0$ -(pseudo-)norm (Spielman et al., 2012; Liang et al., 2022) and has the role of promoting sparsity in the estimated sparse code. Due to its effective feature extraction and representation, DL has found immense applications in data analytics, with applications ranging from clustering and classification (Ramirez et al. (2010); Tošić and Frossard (2011)), to image denoising (Li et al. (2011)), to document detection (Kasiviswanathan et al. (2012)), to medical imaging (Zhao et al. (2021)), and to many others.
However, the existing formulations of DL hinge on a critical assumption: the homogeneity of the data. It is assumed that the samples share the same set of features (atoms) collected in a single dictionary  $D$ . This assumption, however, is challenged in practice as the data is typically collected and processed in heterogeneous edge devices (clients). These clients (for instance, smartphones and wearable devices) operate in different conditions (Kontar et al., 2017) while sharing some congruity. Accordingly, the collected datasets are naturally endowed with heterogeneous features while potentially sharing common ones. In such a setting, the classical formulation of DL faces a major dilemma: on the one hand, a reasonably-sized dictionary (with a moderate number of atoms  $r$ ) may overlook the unique features specific to different clients. On the other hand, collecting both shared and unique features in a single enlarged dictionary (with a large number of atoms  $r$ ) may lead to computational, privacy, and identifiability issues. In addition, both approaches fail to provide information about "what is shared and unique" which may offer standalone intrinsic value and can potentially be exploited for improved clustering, classification and anomaly detection, amongst others.
With the goal of addressing data heterogeneity in dictionary learning, in this paper, we propose personalized dictionary learning (PerDL); a framework that can untangle and recover global and local (unique) dictionaries from heterogeneous datasets. The global dictionary, which collects atoms that are shared among all clients, represents the common patterns among datasets and serves as a conduit of collaboration in our framework. The local dictionaries, on the other hand, provide the necessary flexibility for our model to accommodate data heterogeneity.
- Identifiability of local and global atoms: We provide conditions under which the local and global dictionaries can be provably identified and separated by solving a nonconvex optimization problem. At a high level, our identifiability conditions entail that the true dictionaries are column-wise incoherent, and the local atoms do not have a significant alignment along any nonzero vector.
- Federated meta-algorithm: We present a fully federated meta-algorithm, called PerMA (Algorithm 1), for solving PerDL. PerMA only requires communicating the estimated dictionaries among the clients, thereby circumventing the need for sharing any raw data. A key property of PerMA is its ability to untangle global and local dictionaries by casting it as a series of shortest path problems over a directed acyclic graph (DAG).
- Theoretical guarantees: We prove that, under moderate conditions on the generative model and the clients, PerMA enjoys a linear convergence to the ground truth up to a statistical error. Additionally, PerMA borrows strength from strong clients to improve the performance of the weak ones. More concretely, through collaboration, our framework provides weak clients with the extra benefits of averaged initial condition, convergence rate, and final statistical error.
- Practical performance: We showcase the performance of PerMA on a synthetic dataset, as well as different realistic learning tasks, such as training with imbalanced datasets and video surveillance. These experiments highlight that our method can effectively extract shared global features while preserving unique local ones, ultimately improving performance through collaboration.
Dictionary Learning  
Spielman et al. (2012); Liang et al. (2022) provide conditions under which DL can be provably solved, provided that the dictionary is a square matrix (also known as complete DL). For the more complex case of overcomplete DL with  $r > d$ , Arora et al. (2014, 2015); Agarwal et al. (2016) show that alternating minimization achieves desirable statistical and convergence guarantees. Inspired by recent results on the benign landscape of matrix factorization (Ge et al., 2017; Fattahi and Sojoudi, 2020), Sun et al. (2016) show that a smoothed variant of DL is devoid of spurious local solutions. In contrast, distributed or federated variants of DL are far less explored. Huang et al. (2022); Gkillas et al. (2022) study DL in the federated setting. However, they do not provide any provable guarantees on their proposed method.
Federated Learning & Personalization Recent years have seen explosive interest in federated learning (FL) following the seminal paper on federated averaging (McMahan et al., 2017). Literature
along this line has primarily focused on predictive modeling using deep neural networks (DNN), be it through enabling faster convergence (Karimireddy et al., 2020), improving aggregation schemes at a central server (Wang et al., 2020), promoting fairness across all clients (Yue et al., 2022) or protecting against potential adversaries (Bhagoji et al., 2019). A comprehensive survey of existing methodology can be found in (Kontar et al., 2021). More recently, the focus has been placed on tackling heterogeneity across client datasets through personalization. The key idea is to allow each client to retain their own tailored model instead of learning one model that fits everyone. Approaches along this line either split weights of a DNN into shared and unique ones and collaborate to learn the shared weights (Liang et al., 2020; T Dinh et al., 2020), or follow a train-then-personalize approach where a global model is learned and fine-tuned locally, often iteratively (Li et al., 2021). Again such models have mainly focused on predictive models. Whilst this literature abounds, personalization that aims to identify what is shared and unique across datasets is very limited. Very recently, personalized PCA (Shi and Kontar, 2022) was proposed to address this challenge through identifiable extracting shared and unique principal components using distributed Riemannian gradient descent. However, PCA cannot accommodate sparsity in representation and requires orthogonality constraints that may limit its application. In contrast, our work considers a broader setting via sparse dictionary learning.
Notation. For a matrix  $\mathbf{A}$ , we use  $\| \mathbf{A} \|_2$ ,  $\| \mathbf{A} \|_F$ ,  $\| \mathbf{A} \|_{1,2}$ , and  $\| \mathbf{A} \|_1$  to denote its spectral norm, Frobenius norm, the maximum of its column-wise 2-norm, and the element-wise 1-norm of  $\mathbf{A}$ , respectively. We use  $\mathbf{A}_i$  to indicate that it belongs to client  $i$ . Moreover, we use  $\mathbf{A}_{(i)}$  to denote the  $i$ -th column of  $\mathbf{A}$ . We use  $\mathcal{P}(n)$  to denote the set of  $n \times n$  signed permutation matrices. We define  $[n] = \{1, 2, \dots, n\}$ .
2 PerDL: Personalized Dictionary Learning
In PerDL, we are given  $N$  clients, each with  $n_i$  samples collected in  $\mathbf{Y}_i\in \mathbb{R}^{d\times n_i}$  and generated as a sparse linear combination of  $r^g$  global atoms and  $r_i^l$  local atoms:
$$
\mathbf {Y} _ {i} = \mathbf {D} _ {i} ^ {*} \mathbf {X} _ {i} ^ {*}, \quad \text {w h e r e} \quad \mathbf {D} _ {i} ^ {*} = \left[ \begin{array}{l l} \mathbf {D} ^ {g *} & \mathbf {D} _ {i} ^ {l *} \end{array} \right], \quad \text {f o r} \quad 1 = 1, \dots , N. \tag {1}
$$
Here  $\mathbf{D}^{g*} \in \mathbb{R}^{d \times r^g}$  denotes a global dictionary that captures the common features shared among all clients, whereas  $\mathbf{D}_i^{l*} \in \mathbb{R}^{d \times r_i^l}$  denotes the local dictionary specific to each client. Let  $r_i = r^g + r_i^l$  denote the total number of atoms in  $\mathbf{D}_i^*$ . Without loss of generality, we assume the columns of  $\mathbf{D}_i^*$  have unit  $\ell_2$ -norm. $^1$  The goal in PerDL is to recover  $\mathbf{D}^{g*}$  and  $\{\mathbf{D}_i^{l*}\}_{i=1}^N$ , as well as the sparse codes  $\{\mathbf{X}_i^*\}_{i=1}^N$ , given the datasets  $\{\mathbf{Y}_i\}_{i=1}^N$ . Before presenting our approach for solving PerDL, we first consider the following fundamental question: under what conditions is the recovery of the dictionaries  $\mathbf{D}^{g*}$ ,  $\{\mathbf{D}_i^{l*}\}_{i=1}^N$  and sparse codes  $\{\mathbf{X}_i^*\}_{i=1}^N$  well-posed?
To answer this question, we first note that it is only possible to recover the dictionaries and sparse codes up to a signed permutation: given any signed permutation matrix  $\Pi \in \mathcal{P}(r_i)$ , the dictionary-code pairs  $(\mathbf{D}_i, \mathbf{X}_i)$  and  $(\mathbf{D}_i \Pi_i, \Pi_i^\top \mathbf{X}_i)$  are equivalent. This invariance with respect to signed permutation gives rise to an equivalent class of true solutions with a size that grows exponentially with the dimension. To guarantee the recovery of a solution from this equivalent class, we need the  $\mu$ -incoherency of the true dictionaries.
Assumption 1 ( $\mu$ -incoherency). For each client  $1 \leq i \leq N$ , the dictionary  $\mathbf{D}_i^*$  is  $\mu$ -incoherent for some constant  $\mu > 0$ , that is,
$$
\left. \max  _ {j, k} \left| \left\langle \left(\mathbf {D} _ {i} ^ {*}\right) _ {(j)}, \left(\mathbf {D} _ {i} ^ {*}\right) _ {(k)} \right\rangle \right| \leq \frac {\mu}{\sqrt {d}}. \right. \tag {2}
$$
Assumption 1 is standard in dictionary learning (Agarwal et al. (2016); Arora et al. (2015); Chatterji and Bartlett (2017)) and was independently introduced by Fuchs (2005); Tropp (2006) in signal processing and Zhao and Yu (2006); Meinshausen and Buhlmann (2006) in statistics. Intuitively, it requires the atoms in each dictionary to be approximately orthogonal. To see the necessity of this assumption, consider a scenario where two atoms of  $\mathbf{D}_i^*$  are perfectly aligned (i.e.,  $\mu = \sqrt{d}$ ). In this case, using either of these two atoms achieve a similar effect in reconstructing  $\mathbf{Y}_i$ , contributing to the ill-posedness of the problem.
This assumption is without loss of generality since, for any dictionary-code pair  $(\mathbf{D}_i,\mathbf{X}_i)$ , the columns of  $\mathbf{D}_i$  can be normalized to have unit norm by re-weighting the corresponding rows of  $\mathbf{X}_i$ .
Our next assumption guarantees the separation of local dictionaries from the global one in PerDL. First, we introduce several signed permutation-invariant distance metrics for dictionaries, which will be useful for later analysis.
Definition 1. For two dictionaries  $\mathbf{D}_1, \mathbf{D}_2 \in \mathbb{R}^{d \times r}$ , we define their signed permutation-invariant  $\ell_{1,2}$ -distance and  $\ell_2$ -distance as follows:
$$
d _ {1, 2} \left(\mathbf {D} _ {1}, \mathbf {D} _ {2}\right) := \min  _ {\boldsymbol {\Pi} \in \mathcal {P} (r)} \| \mathbf {D} _ {1} \boldsymbol {\Pi} - \mathbf {D} _ {2} \| _ {1, 2}, \tag {3}
$$
$$
d _ {2} \left(\mathbf {D} _ {1}, \mathbf {D} _ {2}\right) := \min  _ {\boldsymbol {\Pi} \in \mathcal {P} (r)} \| \mathbf {D} _ {1} \boldsymbol {\Pi} - \mathbf {D} _ {2} \| _ {2}. \tag {4}
$$
Furthermore, suppose  $\Pi^{*} = \arg \min_{\mathbf{II}\in \mathcal{P}(r)}\| \mathbf{D}_{1}\Pi -\mathbf{D}_{2}\|_{1,2}$ . For any  $1\leq j\leq r$ , we define
$$
d _ {2, (j)} \left(\mathbf {D} _ {1}, \mathbf {D} _ {2}\right) := \left\| \left(\mathbf {D} _ {1} \boldsymbol {\Pi} ^ {*} - \mathbf {D} _ {2}\right) _ {(j)} \right\| _ {2}. \tag {5}
$$
Assumption 2 ( $\beta$ -identifiability). The local dictionaries  $\{\mathbf{D}_i^{l*}\}_{i=1}^N$  are  $\beta$ -identifiable for some constant  $0 < \beta < 1$ , that is, there exists no vector  $\mathbf{v} \in \mathbb{R}^d$  with  $\|\mathbf{v}\|_2 = 1$  such that
$$
\max  _ {1 \leq i \leq N} \min  _ {1 \leq j \leq r _ {l}} d _ {2} \left(\left(\mathbf {D} _ {i} ^ {l ^ {*}}\right) _ {(j)}, \mathbf {v}\right) \leq \beta . \tag {6}
$$
Suppose there exists a unit-norm  $\mathbf{v}$  satisfying (6) for some small  $\beta > 0$ . This implies that  $\mathbf{v}$  is sufficiently close to at least one atom from each local dictionary. Indeed, one may treat this atom as part of the global dictionary, thereby violating the identifiability of local and global dictionaries. On the other hand, the infeasibility of (6) for large  $\beta > 0$  implies that the local dictionaries are sufficiently dispersed, which in turn facilitates their identification.
With the above assumptions in place, we are ready to present our proposed optimization problem for solving PerDL:
$$
\min  _ {\mathbf {D} ^ {g}, \left\{\mathbf {D} _ {i} \right\}, \left\{\mathbf {X} _ {i} \right\}} \sum_ {i = 1} ^ {N} \| \mathbf {Y} _ {i} - \mathbf {D} _ {i} \mathbf {X} _ {i} \| _ {F} ^ {2} + \lambda \sum_ {i = 1} ^ {N} \| \mathbf {X} _ {i} \| _ {\ell_ {q}}, \quad \text {s . t .} \left(\mathbf {D} _ {i}\right) _ {(1: r ^ {g})} = \mathbf {D} ^ {g} \quad \text {f o r} \quad 1 \leq i \leq N.
$$
For each client  $i$ , PerDL-NCVX aims to recover a dictionary-code pair  $(\mathbf{D}_i, \mathbf{X}_i)$  that match  $\mathbf{Y}_i$  under the constraint that dictionaries for individual clients share the same global components.
3 Meta-algorithm of Solving PerDL
In this section, we introduce our meta-algorithm (Algorithm 1) for solving PerDL-NCVX, which we call Personalized Matching and Averaging (PerMA). In what follows, we explain the steps of PerMA:
Local initialization (Step 3): PerMA starts with a warm-start step where each client runs their own initialization scheme to obtain  $\mathbf{D}_i^{(0)}$ . This step is necessary even for the classical DL to put the initial point inside a basin of attraction of the ground truth. Several spectral methods were proposed to provide a theoretically good initialization (Arora et al., 2015; Agarwal et al., 2016), while in practice, it is reported that random initialization followed by a few iterations of alternating minimization approach will suffice (Ravishankar et al., 2020; Liang et al., 2022).
Global matching scheme (Step 6) Given the clients' initial dictionaries, our global matching scheme separates the global and local parts of each dictionary by solving a series of shortest path problems on an auxiliary graph. Then, it obtains a refined estimate of the global dictionary via simple averaging. A detailed explanation of this step is provided in the next section.
Dictionary update at each client (Step 10) During each communication round, the clients refine their own dictionary based on the available data, the aggregated global dictionary, and the previous estimate of their local dictionary. A detailed explanation of this step is provided in the next section.
Algorithm 1 PerMA: Federated Matching and Averaging
1: Input:  $\{\mathbf{Y}_i\}_{i=1}^N$ .  
2: for client  $i = 1, \dots, N$  do  
3: Client: Obtain  $\mathbf{D}_i^{(0)}$  based on  $\mathbf{Y}_i$ . // Initialization step  
4: Client: Send  $\mathbf{D}_i^{(0)}$  to the server.  
5: end for  
6: Server:  $\left(\mathbf{D}^{g,(0)}, \{\mathbf{D}_i^{l,(0)}\}_{i=1}^N\right) = \text{global_MATCHing}\left(\{\mathbf{D}_i^{(0)}\}_{i=1}^N\right)$  // Separating local from global dictionaries  
7: Server: Broadcast  $\left(\mathbf{D}^{g,(0)}, \{\mathbf{D}_i^{l,(0)}\}_{i=1}^N\right)$   
8: for  $t = 0, 1, \dots, T$  do  
9: for client  $i = 1, \dots, N$  do  
10: Client:  $\left(\mathbf{D}_i^{g,(t+1)}, \mathbf{D}_i^{l,(t+1)}\right) = \text{local_update}\left(\mathbf{Y}_i, \mathbf{D}^{g,(t)}, \mathbf{D}_i^{l,(t)}\right)$  // Updating the local and global dictionaries for each client  
11: Client: Send  $\mathbf{D}_i^{g,(t+1)}$  to the server.  
12: end for  
13: Server: Calculate  $\mathbf{D}^{g,(t+1)} = \frac{1}{N} \sum_{i=1}^N \mathbf{D}_i^{g,(t+1)}$ . // Averaging global dictionaries  
14: Server: Broadcast  $\mathbf{D}^{g,(t+1)}$ .  
15: end for  
16: return  $\left(\mathbf{D}^{g,(T)}, \{\mathbf{D}_i^{l,(T)}\}_{i=1}^N\right)$ .
Global aggregation (Step 13) At the end of each round, the server updates the clients' estimate of the global dictionary by computing their average.
A distinguishing property of PerMA is that it only requires the clients to communicate their dictionaries and not their sparse codes. In fact, after the global matching step on the initial dictionaries, the clients only need to communicate their global dictionaries, keeping their local dictionaries private.
3.1 Global Matching and Local Updates
In this section, we provide detailed implementations of global_MATCHing and local_update subroutines in PerMA (Algorithm 1).
Given the initial approximations of the clients' dictionaries  $\{\mathbf{D}_i^{(0)}\}_{i = 1}^N$ , global_MATCHing seeks to identify and aggregate the global dictionary by extracting the similarities among the atoms of  $\{\mathbf{D}_i^{(0)}\}$ . To identify the global components, one approach is to solve the following optimization problem
$$
\left. \min  _ {\boldsymbol {\Pi} _ {i}} \sum_ {i = 1} ^ {N - 1} \left\| \left(\mathbf {D} _ {i} ^ {(0)} \boldsymbol {\Pi} _ {i}\right) _ {(1: r ^ {g})} - \left(\mathbf {D} _ {i + 1} ^ {(0)} \boldsymbol {\Pi} _ {i + 1}\right) _ {(1: r ^ {g})} \right\| _ {2} \quad \text {s . t .} \quad \boldsymbol {\Pi} _ {i} \in \mathcal {P} (r _ {i}) \quad \text {f o r} \quad 1 \leq i \leq N. \right. \tag {7}
$$
The above optimization aims to obtain the appropriate signed permutation matrices  $\{\Pi_i\}_{i = 1}^N$  that align the first  $r^g$  atoms of the permuted dictionaries. In the ideal regime where  $\mathbf{D}_i^{(0)} = \mathbf{D}_i^*,1\leq i\leq N$  the optimal solution  $\{\Pi_i^*\}_{i = 1}^N$  yields a zero objective value and satisfies  $\left(\mathbf{D}_i^{(0)}\Pi_i^*\right)_{(1:r^g)} = \mathbf{D}^{g*}$ . However, there are two main challenges with the above optimization. First, it is a nonconvex, combinatorial problem over the discrete sets  $\{\mathcal{P}(r_i)\}$ . Second, the initial dictionaries may not
Figure 1: A schematic diagram for global_MATCHing (Algorithm 2). At each iteration, we find the shortest path from  $s$  to  $t$  (highlighted with red), estimate one atom of  $\mathbf{D}^{g*}$  using all passed vertices and remove the path (including the vertices) from  $\mathcal{G}$ .
coincide with their true counterparts. To address the first challenge, we show that the optimal solution to the optimization (7) can be efficiently obtained by solving a series of shortest path problems defined over an auxiliary graph. To alleviate the second challenge, we show that our proposed algorithm is robust against possible errors in the initial dictionaries.
Consider a weighted  $N$ -layered directed acyclic graph (DAG)  $\mathcal{G}$  with  $r_i$  nodes in layer  $i$  representing the  $r_i$  atoms in  $\mathbf{D}_i$ . We connect any node  $a$  from layer  $i$  to any node  $b$  from layer  $i + 1$  with a directed edge with weight  $w(a, b) = d_2((\mathbf{D}_i)_a, (\mathbf{D}_{i + 1})_b)$ . We add a source node  $s$  and connect it to all nodes in layer 1 with weight 0. Similarly, we include a terminal node  $t$  and connect all nodes in layer  $N$  to  $t$  with weight 0. A schematic construction of this graph is presented in Figure 1. Given the constructed graph, Algorithm 2 aims to solve (7) by running  $r^g$  rounds of the shortest path problem: at each round, the algorithm identifies the most aligned atoms in the initial dictionaries by obtaining the shortest path from  $s$  to  $t$ . Then it removes the used nodes in the path for the next round. The correctness and robustness of the proposed algorithm are established in the next theorem.
Algorithm 2 global_MATCHing
1: Input:  $\left\{\mathbf{D}_i^{(0)}\right\}_{i = 1}^N$  and  $r^g$
2: Construct the weighted  $N$ -layer DAG  $\mathcal{G}$  described in Section 3.1.
3: Initialize  $\{\mathrm{index}_i\}_{i = 1}^N$  as empty sets and  $\mathbf{D}^{g,(0)}$  as an empty matrix.
4: for  $j = 1, \dots, r^g$  do
5: Find the shortest path  $\mathcal{P} = \left(s, (\mathbf{D}_1^{(0)})_{(\alpha_1)}, (\mathbf{D}_1^{(0)})_{(\alpha_2)}, \dots, (\mathbf{D}_N^{(0)})_{(\alpha_N)}, t\right)$ .
6: Add  $\frac{1}{N}\sum_{i=1}^{N}$  sign  $\left(\left\langle (\mathbf{D}_i^{(0)})_{(\alpha_i)}, (\mathbf{D}_1^{(0)})_{(\alpha_1)} \right\rangle\right)\left(\mathbf{D}_i^{(0)}\right)_{(\alpha_i)}$  as a new column of  $\mathbf{D}^{g,(0)}$ .
7: Add  $\alpha_{i}$  to index  $i$  for every  $i = 1,\ldots ,N$
8: Remove  $\mathcal{P}$  from  $\mathcal{G}$ .
9: end for
10: Set  $\mathbf{D}_i^{l,(0)} = (\mathbf{D}_i^{(0)})_{([r_i]\backslash \mathrm{index}_i)}$  for every  $i = 1,\ldots ,N$
11: return  $\left(\mathbf{D}^{g,(0)},\left\{\mathbf{D}_i^{l,(0)}\right\}_{i = 1}^N\right)$ .
Theorem 1 (Correctness and robustness of global_MATCHing). Suppose  $\{\mathbf{D}_i^*\}_{i=1}^N$  are  $\mu$ -incoherent (Assumption 1) and  $\beta$ -identifiable (Assumption 2). Suppose the initial dictionaries  $\left\{\mathbf{D}_i^{(0)}\right\}_{i=1}^N$  satisfy  $d_{1,2}\left(\mathbf{D}_i^{(0)}, \mathbf{D}_i^*\right) \leq \epsilon_i$  with  $4\sum_{i=1}^N \epsilon_i \leq \min \left\{\sqrt{2 - 2\frac{\mu}{\sqrt{d}}}, \beta\right\}$ . Then, the output of Algorithm 2 satisfies:
$$
d _ {1, 2} \left(\mathbf {D} ^ {g, (0)}, \mathbf {D} ^ {g *}\right) \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i}, \quad a n d \quad d _ {1, 2} \left(\mathbf {D} _ {i} ^ {l, (0)}, \mathbf {D} _ {i} ^ {l *}\right) \leq \epsilon_ {i}, \quad \text {f o r} \quad 1 \leq i \leq N. \tag {8}
$$