SlowGuess/NeurIPS_2023_OCR huggingface dataset:
One notable difference between this experiment and the previous one is in the choice of the DL algorithm  $\mathcal{A}_i$ . To provide more flexibility, we relax the orthogonality condition for the dictionary. Therefore, we use the alternating minimization algorithm introduced in Arora et al. (2015) for each client (see Algorithm 5). The main difference between this algorithm and Algorithm 4 is that the polar decomposition step in Algorithm 4 is replaced by a single iteration of the gradient descent applied to the loss function  $\mathcal{L}(\mathbf{D},\mathbf{X}) = \| \mathbf{DX} - \mathbf{Y}\|_F^2$ .
Algorithm 5 Alternating minimization for general dictionary learning (Arora et al. (2015))
1: Input:  $\mathbf{Y}_i, \mathbf{D}_i^{(t)}$
2: Set  $\mathbf{X}_i^{(t)} = \mathrm{HT}_{\zeta}\left(\mathbf{D}_i^{(t)\top}\mathbf{Y}_i\right)$
3: Set  $\mathbf{D}_i^{(t + 1)} = \mathbf{D}_i^{(t)} - 2\eta \left(\mathbf{D}_i^{(t)}\mathbf{X}_i^{(t)} - \mathbf{Y}_i\right)\mathbf{X}_i^{(t)^\top}$
4: return  $\mathbf{D}_i^{(t + 1)}$
Even with the computational saving brought up by Algorithm 5, the runtime significantly slows down for PerMA due to large  $N$ ,  $d$ , and  $p$ . Here we report a practical trick to speed up PerMA, which is a local refinement procedure (Algorithm 6) added immediately before local_update (Step 10 of Algorithm 1). At a high level, localdictionary_refinement first finds the local residual data matrix  $\mathbf{Y}_i^l$  by removing the contribution of the global dictionary. Then it iteratively refines the local dictionary with respect to  $\mathbf{Y}_i^l$ . We observed that localdictionary_refinement significantly improves the local reconstruction quality. We leave its theoretical analysis as a possible direction for future work.
Algorithm 6 local_dictary_refinement
1: Input:  $\mathbf{D}_i^{(t)} = \left[ \begin{array}{cc}\mathbf{D}^{g,(t)} & \mathbf{D}_i^{l,(t)} \end{array} \right],\mathbf{Y}_i$
2: Find  $\left[ \begin{array}{l}\mathbf{X}_i^g\\ \mathbf{X}_i^l \end{array} \right]$  such that  $\mathbf{Y}_i\approx \left[\mathbf{D}^{g,(t)}\quad \mathbf{D}_i^{l,(t)}\right]\left[\begin{array}{l}\mathbf{X}_i^g\\ \mathbf{X}_i^l \end{array} \right]$
// Solving a sparse coding problem
3: Set  $\mathbf{Y}_i^l = \mathbf{Y}_i - \mathbf{D}^{g,(t)}\mathbf{X}_i^g$
4: Set  $\mathbf{D}_i^{\mathrm{refine},(0)} = \mathbf{D}_i^{l,(t)}$ .
5: for  $\tau = 0,1,\dots,T^{\mathrm{refine}} - 1$  do
6: Set  $\mathbf{D}_i^{\mathrm{refine},(\tau +1)} = \mathcal{A}_i\left(\mathbf{Y}_i^l,\mathbf{D}_i^{\mathrm{refine},(\tau)}\right)$
7: end for
8: return  $\mathbf{D}_i^{\mathrm{refine},(T^{\mathrm{refine}})}$  as refined  $\mathbf{D}_i^{l,(t)}$
In this section, we present the results of some additional experiments to better showcase the efficiency of PerMA compared with the existing methods and its potential to adapt to a parameter-free algorithm.
B.1 Comparison with Existing Methods
To the best of our knowledge, the problem of personalized dictionary learning (PerDL) has not been previously explored and formally defined. While existing methods in federated learning bear some resemblance to PerDL, they lack provable guarantees in recovering the global and local dictionaries. To clarify these distinctions, we present a detailed comparison between our work and the most closely related papers by Huang et al. (2022) and Gkillas et al. (2022). We compare our method with these methods under the same setting as in Section 5.1. The results can be seen in Figure 5, which shows that PerMA consistently outperforms methods proposed by Gkillas et al. (2022) and Huang et al. (2022).
Figure 5: Comparisons of different methods on synthetic datasets. In the first row, clients are provided with heterogeneous datasets with similar sizes; in the second row, we consider the special cases in which one of the clients has an insufficient sample size and evaluate the performance of the dictionary learned by that specific client. The first column corresponds to final errors with varying  $d$ ; the second column corresponds to total running times with varying  $d$ ; the third column corresponds to final errors with varying  $N$ ; and the forth column corresponds to final errors with varying sparsity level. All the results are averaged over 3 independent trials.
Next, in the context of Section 5.2, we compare the quality of the reconstructed images using dictionaries learned from different methods under three metrics: MSE, PSNR and SSIM. A smaller MSE, a larger PSNR, and an SSIM closer to 1 indicate better image reconstruction quality. In Table 1,  $k$  denotes the number of atoms used to reconstruct the image. As can be seen in the table, PerMA achieves the best result in all sections except for the training time.
Finally, in this paper we use PerMA on the surveillance video datasets, with the goal of separating common elements shared by all clients (the background) and unique elements (different cars). Such a task cannot be accomplished by Gkillas et al. (2022) and Huang et al. (2022) due to their lack of personalization. On the other hand, a recently proposed method based on personalized PCA (PerPCA) has been shown to be effective in separating common and unique elements (Shi and Kontar, 2022). As a result, we run PerPCA on the same dataset to compare its performance with our method. According to Figure 6, PerDL outperforms PerPCA by achieving better separation and higher resolution.
B.2 Auto-tuning of  $r_g$
One can easily see from Section 3 that  $r_g$  is an important hyper-parameter of our algorithm. A larger  $r_g$  means more global atoms are sent between central servers and clients, which leads to a stronger collaboration between them. In the synthetic experiment, we assume to know the value of  $r_g$ , while in
Figure 6: Objects and background separation by PerPCA.
real-life applications, one needs to fine-tune this parameter. An interesting observation we made is that the proposed PerMA algorithm can be augmented by a simple detection mechanism for identifying the correct choice of  $r_g$ . Specifically, during the Global Matching step, where we iteratively remove shortest paths, we can monitor the length of the obtained shortest path. By terminating the removal of paths (i.e., adding global atoms) when the length of the path experiences a significant increase beyond a predefined threshold, we can effectively identify the appropriate value of  $r_g$  without requiring prior knowledge. This detection mechanism alleviates the burden of fine-tuning  $r_g$  and allows for a more practical and robust implementation of the algorithm.
To validate the efficacy of this approach, we conducted a series of experiments, the results of which are presented in Figure 7. We use different  $r_g = 4, 6, 8$  with  $r = 10$  and monitor the lengths of paths. As evident from the outcomes, a clear and drastic increase in the length of the  $r_g + 1$ -th shortest path is observed, signifying the correct value of  $r_g$ .
Figure 7: Increase of path length for different  $r_g$ . Here the  $x$ -axis denotes the number of iterations for Algorithm 2 and the  $x$ -axis denotes the distance of the  $\mathcal{P}$  for each iteration.
Methods	MSE	PSNR	SSIM	Running Time
k=10	k=20	k=10	k=20	k=10	k=20	(seconds)
PerMA	0.0319	0.0207	15.3795	17.3771	0.6286	0.7074	40.8
Local DL	0.0448	0.0330	14.1936	15.7203	0.5538	0.6162	4.9
Gkillas et al.	0.1069	0.1084	9.8930	9.8360	0.2736	0.2659	3600+
Huang et al.	0.1062	0.1062	9.9203	9.9212	0.2737	0.2628	3600+
PerPCA	0.0837	0.0734	11.0118	11.6426	0.3520	0.3959	59.3
Table 1: Comparisons of different methods on MNIST datasets. A smaller MSE, a larger PSNR, and an SSIM closer to 1 indicate better image reconstruction quality. Here  $k$  denotes the number of atoms used to reconstruct the image.
C Further Discussion on Linearly Convergent Algorithms
In this section, we discuss a linearly convergent DL algorithm that satisfies the conditions of our Theorem 2. In particular, the next theorem is adapted from (Arora et al., 2015, Theorem 12) and shows that a modified variant of Algorithm 5 introduced in (Arora et al., 2015, Algorithm 5) is indeed linearly-convergent.
Theorem 3 (Linear convergence of Algorithm 5 in Arora et al. (2015)). Suppose that the data matrix satisfies  $\mathbf{Y} = \mathbf{D}^{*}\mathbf{X}^{*}$ , where  $\mathbf{D}^*$  is an  $\mu$ -incoherent dictionary and the sparse code  $\mathbf{X}^*$  satisfies the generative model introduced in Section 1.2 and Section 4.1 of Arora et al. (2015). For any initial dictionary  $\| \mathbf{D}^{(0)}\|_2 \leq 1$ , Algorithm 5 in Arora et al. (2015) is  $(\delta, \rho, \psi)$ -linearly convergent with  $\delta = O(1 / \log d)$ ,  $\rho \in (1/2, 1)$ , and  $\psi = O(d^{-\omega(1)})$ .
Algorithm 5 in Arora et al. (2015) is a refinement of Algorithm 5, where the error is further reduced by projecting out the components along the column currently being updated. For brevity, we do not discuss the exact implementation of the algorithm; an interested reader may refer to Arora et al. (2015) for more details. Indeed, we have observed in our experiments that the additional projection step does not provide a significant benefit over Algorithm 5.
To begin with, we establish a triangular inequality for  $d_{1,2}(\cdot ,\cdot)$ , which will be important in our subsequent arguments:
Lemma 1 (Triangular inequality for  $d_{1,2}(\cdot ,\cdot)$ ). For any dictionary  $\mathbf{D}_1,\mathbf{D}_2,\mathbf{D}_3\in \mathbb{R}^{d\times r}$ , we have
$$
d _ {1, 2} \left(\mathbf {D} _ {1}, \mathbf {D} _ {2}\right) \leq d _ {1, 2} \left(\mathbf {D} _ {1}, \mathbf {D} _ {3}\right) + d _ {1, 2} \left(\mathbf {D} _ {3}, \mathbf {D} _ {2}\right) \tag {13}
$$
Proof. Suppose  $\Pi_{1,3}$  and  $\Pi_{3,2}$  satisfy  $d_{1,2}(\mathbf{D}_1,\mathbf{D}_3) = \| \mathbf{D}_1\boldsymbol{\Pi}_{1,3} - \mathbf{D}_3\|_{1,2}$  and  $d_{1,2}(\mathbf{D}_3,\mathbf{D}_2) = \| \mathbf{D}_3 - \mathbf{D}_2\boldsymbol{\Pi}_{3,2}\|_{1,2}$ . Then we have
$$
\begin{array}{l} d _ {1, 2} \left(\mathbf {D} _ {1}, \mathbf {D} _ {3}\right) + d _ {1, 2} \left(\mathbf {D} _ {3}, \mathbf {D} _ {2}\right) = \left\| \mathbf {D} _ {1} \boldsymbol {\Pi} _ {1, 3} - \mathbf {D} _ {3} \right\| _ {1, 2} + \left\| \mathbf {D} _ {3} - \mathbf {D} _ {2} \boldsymbol {\Pi} _ {3, 2} \right\| _ {1, 2} \\ \geq \left\| \mathbf {D} _ {1} \boldsymbol {\Pi} _ {1, 3} - \mathbf {D} _ {2} \boldsymbol {\Pi} _ {3, 2} \right\| _ {1, 2} \tag {14} \\ \geq d _ {1, 2} \left(\mathbf {D} _ {1}, \mathbf {D} _ {2}\right). \\ \end{array}
$$
Given how the directed graph  $\mathcal{G}$  is constructed and modified, any directed path from  $s$  to  $t$  will be of the form  $\mathcal{P} = s\rightarrow (\mathbf{D}_1^{(0)})_{\alpha (1)}\rightarrow (\mathbf{D}_2^{(0)})_{\alpha (2)}\rightarrow \dots \rightarrow (\mathbf{D}_N^{(0)})_{\alpha (N)}\rightarrow t$ . Specifically, each layer (or client) contributes exactly one node (or atom), and the path is determined by  $\alpha (\cdot):[N]\to [r]$ . Recall that  $\mathbf{D}_i^* = \left[\mathbf{D}^{g*}\quad \mathbf{D}_i^{l*}\right]$  for every  $1\leq i\leq N$ . Assume, without loss of generality, that for every client  $1\leq i\leq N$
$$
\mathbf {I} _ {r _ {i} \times r _ {i}} = \arg \min  _ {\boldsymbol {\Pi} \in \mathcal {P} (r _ {i})} \left\| \mathbf {D} _ {i} ^ {*} \boldsymbol {\Pi} - \mathbf {D} _ {i} ^ {(0)} \right\| _ {1, 2}. \tag {15}
$$
In other words, the first  $r^g$  atoms in the initial dictionaries  $\{D_i^{(0)}\}_{i = 1}^N$  are aligned with the global dictionary. Now consider the special path  $\mathcal{P}_j^*$  for  $1\leq j\leq r^g$  defined as
$$
\mathcal {P} _ {j} ^ {*} = s \rightarrow \left(\mathbf {D} _ {1} ^ {(0)}\right) _ {j} \rightarrow \left(\mathbf {D} _ {2} ^ {(0)}\right) _ {j} \rightarrow \dots \rightarrow \left(\mathbf {D} _ {N} ^ {(0)}\right) _ {j} \rightarrow t. \tag {16}
$$
To prove that Algorithm 2 correctly selects and aligns global atoms from clients, it suffices to show that  $\{\mathcal{P}_j^*\}_{j=1}^{r^g}$  are the top- $r^g$  shortest paths from  $s$  to  $t$  in  $\mathcal{G}$ . The length of the path  $\mathcal{P}_j^*$  can be bounded as
$$
\begin{array}{l} \mathcal {L} \left(\mathcal {P} _ {j} ^ {*}\right) = \sum_ {i = 1} ^ {N - 1} d _ {2} \left(\left(\mathbf {D} _ {i} ^ {(0)}\right) _ {j}, \left(\mathbf {D} _ {i + 1} ^ {(0)}\right) _ {j}\right) \\ = \sum_ {i = 1} ^ {N - 1} \min  \left\{\| (\mathbf {D} _ {i} ^ {(0)}) _ {j} - (\mathbf {D} _ {i + 1} ^ {(0)}) _ {j} \| _ {2}, \| (\mathbf {D} _ {i} ^ {(0)}) _ {j} + (\mathbf {D} _ {i + 1} ^ {(0)}) _ {j} \| _ {2} \right\} \\ \leq \sum_ {\substack {i = 1 \\ N - 1}} ^ {N - 1} \| \left(\mathbf {D} _ {i} ^ {(0)}\right) _ {j} - \left(\mathbf {D} _ {i + 1} ^ {(0)}\right) _ {j} \| _ {2} \tag{17} \\ \leq \sum_ {i = 1} ^ {N - 1} \| (\mathbf {D} _ {i} ^ {(0)}) _ {j} - (\mathbf {D} ^ {g *}) _ {j} \| _ {2} + \| (\mathbf {D} _ {i + 1} ^ {(0)}) _ {j} - (\mathbf {D} ^ {g *}) _ {j} \| _ {2} \\ \leq \sum_ {i = 1} ^ {N - 1} \left(\epsilon_ {i} + \epsilon_ {i + 1}\right) \\ \leq 2 \sum_ {i = 1} ^ {N} \epsilon_ {i}. \\ \end{array}
$$
We move on to prove that all the other paths from  $s$  to  $t$  will have a distance longer than  $2\sum_{i=1}^{N}\epsilon_i$ . Consider a general directed path  $\mathcal{P} = s \to (\mathbf{D}_1^{(0)})_{\alpha(1)} \to (\mathbf{D}_2^{(0)})_{\alpha(2)} \to \dots \to (\mathbf{D}_N^{(0)})_{\alpha(N)} \to t$  that is not in  $\{\mathcal{P}_j^*\}_{j=1}^{r^g}$ . Based on whether or not  $\mathcal{P}$  contains atoms that align with the true global ground atoms, there are two situations:
Case 1: Suppose there exists  $1 \leq i \leq N$  such that  $\alpha(i) \leq r^g$ . Given Model 1 and the assumed equality (15), we know that for layer  $i$ ,  $\mathcal{P}$  contains a global atom. Since  $\mathcal{P}$  is not in  $\{\mathcal{P}_j^*\}_{j=1}^{r^g}$ , there must exist  $k \neq i$  such that  $\alpha(k) \neq \alpha(i)$ . As a result, we have
$$
\mathcal {L} (\mathcal {P}) \stackrel {(a)} {\geq} d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {(0)}\right) _ {\alpha (i)}, \left(\mathbf {D} _ {k} ^ {(0)}\right) _ {\alpha (k)}\right)
$$
$$
\begin{array}{l} \stackrel {(b)} {\geq} \min  \left\{\| (\mathbf {D} _ {i} ^ {*}) _ {\alpha (i)} - (\mathbf {D} _ {k} ^ {*}) _ {\alpha (k)} \| _ {2}, (\mathbf {D} _ {i} ^ {*}) _ {\alpha (i)} + (\mathbf {D} _ {k} ^ {*}) _ {\alpha (k)} \| _ {2} \right\} \\ - \left\| \left(\mathbf {D} _ {i} ^ {*}\right) _ {\alpha (i)} - \left(\mathbf {D} _ {i} ^ {(0)}\right) _ {\alpha (i)} \right\| _ {2} - \left\| \left(\mathbf {D} _ {k} ^ {*}\right) _ {\alpha (k)} - \left(\mathbf {D} _ {k} ^ {(0)}\right) _ {\alpha (k)} \right\| _ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(c)} {\geq} \sqrt {2 - 2 \left| \left\langle \left(\mathbf {D} _ {k} ^ {*}\right) _ {\alpha (i)}, \left(\mathbf {D} _ {k} ^ {*}\right) _ {\alpha (k)} \right\rangle \right|} - 2 \max  _ {1 \leq i \leq N} \epsilon_ {i} \tag {18} \\ \stackrel {(d)} {\geq} \sqrt {2 - 2 \frac {\mu}{\sqrt {d}}} - 2 \max _ {1 \leq i \leq N} \epsilon_ {i} \\ \stackrel {(e)} {\geq} 2 \sum_ {i = 1} ^ {N} \epsilon_ {i} ^ {g} \\ \end{array}
$$
Here  $(a)$  and  $(b)$  are due to Lemma 1,  $(c)$  is due to assumed equality (15),  $(d)$  is due to the  $\mu$ -incoherency of  $\mathbf{D}_k^*$ , and finally  $(e)$  is given by the assumption of Theorem 1.
Case 2: Suppose  $\alpha(i) > r^g$  for all  $1 \leq i \leq N$ , which means that the path  $\mathcal{P}$  only uses approximations to local atoms. Consider the ground truth of these approximations,  $(\mathbf{D}_1^*)_{\alpha(1)}, (\mathbf{D}_2^*)_{\alpha(2)}, \ldots, (\mathbf{D}_N^*)_{\alpha(N)}$ . There must exist  $1 \leq i, j \leq N$  such that  $d_{1,2}((\mathbf{D}_i^*)_{\alpha(i)}, (\mathbf{D}_j^*)_{\alpha(j)}) \geq \beta$ . Otherwise, it is easy to see that  $\{\mathbf{D}_i^{l*}\}_{i=1}^N$  would not be  $\beta$ -identifiable
because any  $(\mathbf{D}_i^*)_{\alpha (i)}$  will satisfy (6). As a result, we have the following:
$$
\begin{array}{l} \mathcal {L} (\mathcal {P}) \geq d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {(0)}\right) _ {\alpha (i)}, \left(\mathbf {D} _ {j} ^ {(0)}\right) _ {\alpha (j)}\right) \\ \geq d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {*}\right) _ {\alpha (i)}, \left(\mathbf {D} _ {j} ^ {*}\right) _ {\alpha (j)}\right) - \left\| \left(\mathbf {D} _ {i} ^ {*}\right) _ {\alpha (i)} - \left(\mathbf {D} _ {i} ^ {(0)}\right) _ {\alpha (i)} \right\| _ {2} - \left\| \left(\mathbf {D} _ {j} ^ {*}\right) _ {\alpha (j)} - \left(\mathbf {D} _ {j} ^ {(0)}\right) _ {\alpha (j)} \right\| _ {2} \\ \geq \beta -2\max_{i}\epsilon_{i} \\ \geq 2 \sum_ {i = 1} ^ {N} \epsilon_ {i} \tag {19} \\ \end{array}
$$
So we have shown that  $\{\mathcal{P}_j^*\}_{j=1}^{r^g}$  are the top- $r^g$  shortest paths from  $s$  to  $t$  in  $\mathcal{G}$ . Moreover, it is easy to show that  $\text{sign}\left(\left\langle (\mathbf{D}_1^{(0)})_j, (\mathbf{D}_i^{(0)})_j \right\rangle\right) = 1$  for small enough  $\{\epsilon_i\}_{i=1}^N$ . Therefore, the proposed algorithm correctly recovers the global dictionaries (with the correct identity permutation). Finally, we have  $\mathbf{D}^{g,(0)} = \frac{1}{N} \sum_{i=1}^N (\mathbf{D}_i^{(0)})_{1:r^g}$ , which leads to:
$$
\begin{array}{l} d _ {1, 2} \left(\mathbf {D} ^ {g, (0)}, \mathbf {D} ^ {g *}\right) \leq \max  _ {1 \leq j \leq r ^ {g}} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\mathbf {D} _ {i} ^ {(0)}\right) _ {j} - (\mathbf {D} ^ {g *}) _ {j} \right\| _ {2} \\ \leq \max  _ {1 \leq j \leq r ^ {g}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \left(\mathbf {D} _ {i} ^ {(0)}\right) _ {j} - \left(\mathbf {D} ^ {g *}\right) _ {j} \right\| _ {2} \tag {20} \\ \leq \max  _ {1 \leq j \leq r ^ {g}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \\ = \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i}. \\ \end{array}
$$
This completes the proof of Theorem 1.
Throughout this section, we define:
$$
\bar {\rho} := \frac {1}{N} \sum_ {i = 1} ^ {N} \rho_ {i}, \quad \bar {\psi} := \frac {1}{N} \sum_ {i = 1} ^ {N} \psi_ {i}. \tag {21}
$$
We will prove the convergence of the global dictionary in Theorem 2 by proving the following induction: at each  $t \geq 1$ , we have
$$
d _ {1, 2} \left(\mathbf {D} ^ {g, (t + 1)}, \mathbf {D} ^ {g *}\right) \leq \bar {\rho} d _ {1, 2} \left(\mathbf {D} ^ {g, (t)}, \mathbf {D} ^ {g *}\right) + \bar {\psi}. \tag {22}
$$
At the beginning of communication round  $t$ , each client  $i$  performs local_update to get  $\mathbf{D}_i^{(t+1)}$  given  $\left[ \begin{array}{cc} \mathbf{D}^{g,(t)} & \mathbf{D}_i^{l,(t)} \end{array} \right]$ . Without loss of generality, we assume
$$
\mathbf {I} _ {r _ {i} \times r _ {i}} = \arg \min  _ {\boldsymbol {\Pi} \in \mathcal {P} (r _ {i})} \left\| \mathbf {D} _ {i} ^ {*} \boldsymbol {\Pi} - \left[ \begin{array}{l l} \mathbf {D} ^ {g, (t)} & \mathbf {D} _ {i} ^ {l, (t)} \end{array} \right] \right\| _ {1, 2}, \tag {23}
$$
$$
\mathbf {I} _ {r _ {i} \times r _ {i}} = \arg \min  _ {\boldsymbol {\Pi} \in \mathcal {P} (r _ {i})} \left\| \mathbf {D} _ {i} ^ {*} \boldsymbol {\Pi} - \mathbf {D} _ {i} ^ {(t + 1)} \right\| _ {1, 2}. \tag {24}
$$
Assumed equalities (23) and (24) imply that the permutation matrix that aligns the input and the output of  $\mathcal{A}_i$  is also  $\mathbf{I}_{r_i\times r_i}$ . Specifically, the linear convergence property of  $\mathcal{A}_i$  and Theorem 1 thus suggest:
$$
\left\| \left(\mathbf {D} _ {i} ^ {(t + 1)}\right) _ {j} - \left(\mathbf {D} _ {i} ^ {*}\right) _ {j} \right\| _ {2} \leq \rho_ {i} \left\| \left(\mathbf {D} ^ {g, (t)}\right) _ {j} - \left(\mathbf {D} _ {i} ^ {*}\right) _ {j} \right\| _ {2} + \psi_ {i} \quad \forall 1 \leq j \leq r ^ {g}, 1 \leq i \leq N. \tag {25}
$$
However, our algorithm is unaware of this trivial alignment. We will next show the remaining steps in local_update correctly recovers the identity permutation. The proof is very similar to the proof of Theorem 1 since we are essentially running Algorithm 2 on a two-layer  $\mathcal{G}$ . For every  $1 \leq i \leq N$ ,  $1 \leq j \leq r^g$ , we have
$$
\begin{array}{l} d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {(t + 1)}\right) _ {j}, \left(\mathbf {D} ^ {g, (t)}\right) _ {j}\right) \leq d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {(t + 1)}\right) _ {j}, \left(\mathbf {D} _ {i} ^ {*}\right) _ {j}\right) + d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {*}\right) _ {j}, \left(\mathbf {D} ^ {g, (t)}\right) _ {j}\right) \tag {26} \\ \leq 2 \delta_ {i}. \\ \end{array}
$$
Meanwhile for  $k \neq j$ ,
$$
\begin{array}{l} d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {(t + 1)}\right) _ {k}, \left(\mathbf {D} ^ {g, (t)}\right) _ {j}\right) \\ \geq d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {*}\right) _ {k}, \left(\mathbf {D} _ {i} ^ {*}\right) _ {j}\right) - d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {(t + 1)}\right) _ {k}, \left(\mathbf {D} _ {i} ^ {*}\right) _ {k}\right) - d _ {1, 2} \left(\left(\mathbf {D} _ {i} ^ {*}\right) _ {j}, \left(\mathbf {D} ^ {g, (t)}\right) _ {j}\right) \tag {27} \\ \geq \sqrt {2 - \frac {2 \mu}{\sqrt {d}}} - 2 \delta_ {i}. \\ \geq 2 \delta_ {i}. \\ \end{array}
$$
As a result, we successfully recover the identity permutation, which implies
$$
\left\| \left(\mathbf {D} _ {i} ^ {g, (t + 1)}\right) _ {j} - \left(\mathbf {D} _ {i} ^ {g *}\right) _ {j} \right\| _ {2} \leq \rho_ {i} \left\| \left(\mathbf {D} ^ {g, (t)}\right) _ {j} - \left(\mathbf {D} _ {i} ^ {g *}\right) _ {j} \right\| _ {2} + \psi_ {i} \quad \forall 1 \leq j \leq r ^ {g}, 1 \leq i \leq N. \tag {28}
$$
Finally, the aggregation step (Step 13 in Algorithm 1) gives:
$$
\begin{array}{l} d _ {1, 2} \left(\mathbf {D} ^ {g, (t + 1)}, \mathbf {D} ^ {g *}\right) \leq \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbf {D} _ {i} ^ {g, (t + 1)} - \mathbf {D} ^ {g *} \right\| _ {1, 2} \\ = \max  _ {1 \leq j \leq r ^ {g}} \left\| \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \mathbf {D} _ {i} ^ {g, (t + 1)}\right) _ {j} - (\mathbf {D} ^ {g *}) _ {j} \right\| \\ \leq \max  _ {1 \leq j \leq r ^ {g}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \left(\mathbf {D} _ {i} ^ {g, (t + 1)}\right) _ {j} - \left(\mathbf {D} _ {i} ^ {g *}\right) _ {j} \right\| _ {2} \tag {29} \\ \leq \max  _ {1 \leq j \leq r ^ {g}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\rho_ {i} \left\| \left(\mathbf {D} ^ {g, (t)}\right) _ {j} - \left(\mathbf {D} _ {i} ^ {g *}\right) _ {j} \right\| _ {2} + \psi_ {i}\right) \\ \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\rho_ {i} d _ {1, 2} \left(\mathbf {D} ^ {g, (t)}, \mathbf {D} ^ {g *}\right) + \psi_ {i}\right) \\ = \bar {\rho} d _ {1, 2} \left(\mathbf {D} ^ {g, (t)}, \mathbf {D} ^ {g *}\right) + \bar {\psi}. \\ \end{array}
$$
As a result, we prove the induction (22) for all  $0 \leq t \leq T - 1$ . Inequality (12) is a by-product of the accurate separation of global and local atoms and can be proved by similar arguments. The proof is hence complete.