## 4 Dictionary Pair Learning Algorithm
### Sparse coding phase
5: compute every \(\
u_{q}+l(\hat{\mu}_{i})\) where \(\hat{\mu}_{i}\) is the point closest to \(\lambda_{i}\) in \(\xi_{j-1}\). 6:\(\hat{\mu}_{i}=\operatorname*{arg\,min}_{\mu_{i}}\
u_{q}+l(\mu_{i})\). 7: Compute \(\lambda_{i}\) via Problem (31). 0:End for

### Dictionary Pair Update Phase

To obtain \(\mathbf{\Psi}\), we solve the following problem with all other variables fixed:

\[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]1

Such problem is a highly nonlinear optimization due to the definition of \(\mathcal{S}_{\lambda}\). Here we solve \(\mathbf{\Psi}\) columnwisely by updating each column of \(\mathbf{\Psi}\). For each \(\mathbf{\psi}_{i}\), we solve the following subproblem:

\[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]2

We denote \(\mathcal{J}_{i}\) and \(\hat{\mathcal{J}}_{i}\) as the indices set as before. Set the elements of \(\mathbf{\overline{y}}_{i}\) corresponding to the indices \(\hat{\mathcal{J}}_{i}\) to be zeros and denote the new vector as \(\mathbf{\overline{z}}_{i}\). This operation leads to a consequence that \(\mathbf{\psi}_{i}^{T}\mathbf{X}^{\hat{\mathcal{J}}_{i}}\approx 0\). Then we solve the following quadratic optimization problem that is easy to solve with least squares. \[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]3

The optimization problem to pursue \(\mathbf{\Phi}\) is formulated as

\[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]4 \[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]5

where the frame operator \(\mathbf{F}\) is given by \(\mathbf{\Phi}\mathbf{\Phi}^{T}\) and \(\mathbf{F}^{-1}\) is defined as Equation (22). The target function then becomes

\[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]6

which is denoted by \(h(\mathbf{\Phi})\). We apply the gradient descent method to unconstraint version of Problem (35) and then project the solution to the feasible space. The gradient is given by a very complicated form as follows

\[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]7

In order to reduce the complexity, the gradient can also be computed with the fixed \(\mathbf{F}\) calculated in the previous step of the ADM. Then at the \(k\)-th iteration, the gradient can be written as

\[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]8

where \(\mathbf{F}=\mathbf{\Phi}^{k-1}\mathbf{\Phi}^{(k-1)}^{T}\). The descent step length can be obtained by optimizing the problem \(\min_{\theta}h(\mathbf{\Phi}+\theta\
abla h(\mathbf{\Phi}))\) with fixed \(\mathbf{F}\), which is given by

\[\hat{\mathbf{\lambda}}=\operatorname*{arg\,min}_{\mathbf{\lambda}}\|\mathbf{Y}-\mathcal{S }_{\mathbf{\lambda}}(\mathbf{\Psi}^{T}\mathbf{X})\|_{F}^{2} \tag{25}\]9

where \(\mathbf{a}=\
abla h(\mathbf{\Phi})\mathbf{Y},\mathbf{b}=\mathbf{X}-\mathbf{\Phi}\mathbf{Y},\mathbf{c}=\mathbf{F}^{ -1}\
abla h(\mathbf{\Phi}),\mathbf{d}=\mathbf{\Psi}-\mathbf{F}^{-1}\mathbf{\Phi}\). For the frame condition \(\sqrt{A}\leq\delta_{\Phi}\leq\sqrt{B}\), we apply a SVD decomposition \(\mathbf{\Phi}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^{T}\) and map the singular values to the interval \([\sqrt{A},\sqrt{B}]\) linearly.