B.1 Proof of Proposition 1
Since S is symmetric,

\operatorname{vec}[Z^{(l+1)}]=\operatorname{vec}[X]+\gamma\operatorname{vec}[g(F)Z^{(l)}S]=\gamma[S\otimes g(F)]\operatorname{vec}[Z^{(l)}]+\operatorname{vec}[X].

By repeatedly apply this to Z^{(l)} ,

\operatorname{vec}[Z^{(H)}]=\sum_{k=0}^{H}\gamma^{k}[S\otimes g(F)]^{k}\operatorname{vec}[X]=\sum_{k=0}^{H}\gamma^{k}[S^{k}\otimes g(F)^{k}]\operatorname{vec}[X].

Let s_{H}=\sum_{k=0}^{H}\gamma^{k}[S^{k}\otimes g(F)^{k}] . Then for H>l ,
by using the triangle inequality of a norm and the submultiplicativity of an induced matrix norm,

\|s_{H}-s_{l}\|_{2}=\left\|\sum_{k=l+1}^{H}\gamma^{k}[S^{k}\otimes g(F)^{k}]\right\|_{2} \leq\sum_{k=l+1}^{H}\left\|\gamma^{k}[S^{k}\otimes g(F)^{k}]\right\|_{2} \leq\sum_{k=l+1}^{H}\prod_{i=1}^{k}\left\|\gamma[S\otimes g(F)]\right\|_{2} =\sum_{k=l+1}^{H}\prod_{i=1}^{k}\gamma\|S\|_{2}\|g(F)\|_{\mathrm{F}}

where the last line follows
from the property of the Kronecker product. Since S is symmetric and normalized,

\|S\|_{2}=|\lambda_{\max}(S)|\leq 1.

Since \epsilon_{F}>0 ,

\|g(F)\|_{2}\leq\|g(F)\|_{\mathrm{F}}=\left\|\frac{1}{\|F^{\top}F\|_{\mathrm{F}}+\epsilon_{F}}F^{\top}F\right\|_{\mathrm{F}}=\frac{\|F^{\top}F\|_{\mathrm{F}}}{\|F^{\top}F\|_{\mathrm{F}}+\epsilon_{F}}<1.

In other words, there exists \delta\in[0,1) such that \|g(F)\|_{2}\leq\delta .
Combining,

\|s_{H}-s_{l}\|_{2}\leq\sum_{k=l+1}^{H}\prod_{i=1}^{k}\gamma\|S\|_{2}\|g(F)\|_{\mathrm{F}}\leq\sum_{k=l+1}^{H}\prod_{i=1}^{k}\gamma\delta=\sum_{k=l+1}^{H}(\gamma\delta)^{k}.

Here, we have \gamma\delta\in[0,1) since \gamma\in(0,1] and \delta\in[0,1) . Thus, we have \sum_{k=0}^{H}(\gamma\delta)^{k}=\frac{1-(\gamma\delta)^{H+1}}{1-\gamma\delta} . Therefore,

\sum_{k=l+1}^{H}(\gamma\delta)^{k}=\sum_{k=0}^{H}(\gamma\delta)^{k}-\sum_{k=0}^{l}(\gamma\delta)^{k}=\frac{(\gamma\delta)^{l+1}-(\gamma\delta)^{H+1}}{1-\gamma\delta}\leq\frac{(\gamma\delta)^{l+1}}{1-\gamma\delta}.

Combining

\|s_{H}-s_{l}\|_{2}\leq\sum_{k=l+1}^{H}(\gamma\delta)^{k}\leq\frac{(\gamma\delta)^{l+1}}{1-\gamma\delta}.

Since \|s_{H}-s_{l}\|_{2}\leq\frac{(\gamma\delta)^{l+1}}{1-\gamma\delta} with \gamma\delta<1 , we have that for any \epsilon>0 , there exists an integer \bar{l} such that \|s_{H}-s_{l}\|_{2}<\epsilon for any H,l\geq\bar{l} .
Thus, the sequence (s_{l})_{l}=(\sum_{k=0}^{l}\gamma^{k}[S^{k}\otimes g(F)^{k}])_{l} is a Cauchy sequence and converges.

Since it converges, by recalling the fact that for an matrix M ,

(1-M)\left(\sum_{k=0}^{H}M^{k}\right)=I-M^{H+1},

we have that

\left(I-\gamma[S\otimes g(F)]\right)\left(\lim_{H\rightarrow\infty}\sum_{k=0}^{H}\gamma^{k}[S^{k}\otimes g(F)^{k}]\right) =\lim_{H\rightarrow\infty}\left(I-\gamma[S\otimes g(F)]\right)\left(\sum_{k=0}^{H}\gamma^{k}[S^{k}\otimes g(F)^{k}]\right) =\lim_{H\rightarrow\infty}(I-\gamma^{H+1}[S^{H+1}\otimes g(F)^{H+1}]) =I

where the last line follows from the fact that

\|\gamma^{H+1}[S^{H+1}\otimes g(F)^{H+1}]\|_{2}\leq\prod_{i=1}^{H}\left\|\gamma[S\otimes g(F)]\right\|_{2}=\prod_{i=1}^{H}\gamma\|S\|_{2}\|g(F)\|_{\mathrm{F}}\leq(\gamma\delta)^{H}\rightarrow 0\ \ \ as H\rightarrow\infty

where the inequalities follows from the above equations used to bound \|s_{H}-s_{l}\|_{2} .

This implies that

\left(\lim_{H\rightarrow\infty}\sum_{k=0}^{H}\gamma^{k}[S^{k}\otimes g(F)^{k}]\right)=\left(I-\gamma[S\otimes g(F)]\right)^{-1},

if the inverse \left(I-\gamma[S\otimes g(F)]\right)^{-1} exists. This inverse exists, which can be seen by the following facts. From the above equations used to bound \|s_{H}-s_{l}\|_{2} (and the fact that S and g(F) are symmetric and hence their singular values are the absolute values of their eigenvalues), we have \lambda_{\max}(\gamma[S\otimes g(F)])=\gamma\lambda_{\max}(S)\lambda_{\max}(g(F))<1 . Also, since (I-\gamma[S\otimes g(F)]) is real symmetric (as [S\otimes g(F)]^{\top}=[S^{\top}\otimes g(F)^{\top}]=[S\otimes g(F)] ), all the eigenvalues of the matrix (I-\gamma[S\otimes g(F)]) are real numbers. Combining these facts, all the eigenvalues of (I-\gamma[S\otimes g(F)]) are strictly positive (since the eigenvalues of the identity matrix I are ones). Therefore, (I-\gamma[S\otimes g(F)]) is positive definite and invertible.

Using these results, we have

\lim_{H\rightarrow\infty}\operatorname{vec}[Z^{(H)}]=\left(I-\gamma[S\otimes g(F)]\right)^{-1}\operatorname{vec}[X].

\operatorname{vec}[f(X,F,B)]=\operatorname{vec}\left[B\left(\lim_{H\rightarrow\infty}Z^{(H)}\right)\right]=[I_{n}\otimes B]\operatorname{vec}\left[\left(\lim_{H\rightarrow\infty}Z^{(H)}\right)\right]=[I_{n}\otimes B]\left(I-\gamma[S\otimes g(F)]\right)^{-1}\operatorname{vec}[X].

∎