B.1 Proof of Theorem 1
First, we note that the Jacobian of the dynamical system for BPTT update (Eq. 7 ) is simply the loss Hessian scaled by -\eta . This implies that

|\lambda^{J}_{1}| =\eta_{B}|\lambda^{H}_{1}|, (23)

where we remind the reader that \lambda^{J}_{1}\in\mathbb{R} (resp. \widehat{\lambda_{1}^{J}}\in\mathbb{R} ) is the leading eigenvalue for BPTT (resp. an approximate rule) Jacobian; \lambda^{H}_{1}\in\mathbb{R} is the leading eigenvalue of the loss’ Hessian matrix.

Second, with the assumption of a single output \hat{y} , loss presented only at the last time step T and stochastic gradient descent (updates on a single data example as opposed to batch updates), least squares loss in Eq. 3 can be simplified to:

L(W_{h})=\frac{1}{2}(\hat{y}-y)^{2}, (24)

resulting in the following difference equations for discrete dynamical systems defined by BPTT (Eq. 7 ) and an approximate rule (Eq. 8 ):

\left.\Delta W_{h}\right|_{BPTT} =F(W_{h})=-\eta_{B}(\hat{y}-y) abla\hat{y}, (25) \left.\Delta W_{h}\right|_{approximate} =\hat{F}(W_{h})=-\eta_{e}(\hat{y}-y)\tilde{ abla}\hat{y}, (26)

where we remind the reader that \tilde{\nabla} is the notation for an approximate gradient.

We then compute the Jacobian of the difference equations above:

 J =-\eta_{B}\left( abla\hat{y} abla\hat{y}^{T}+(\hat{y}-y) abla^{2}\hat{y}\right)\quad(for BPTT) (27) \widehat{J} =-\eta_{e}\left( abla\hat{y}\tilde{ abla}\hat{y}^{T}+(\hat{y}-y) abla\tilde{ abla}\hat{y}\right)\quad(for an approximate rule)

In the limit of zero error (y-\hat{y})=0 , i.e. close to an optimum, the term involving (y-\hat{y}) becomes negligible. That simplifies the Jacobian to

 J \approx-\eta_{B} abla\hat{y} abla\hat{y}^{T} (29) \widehat{J} \approx-\eta_{e} abla\hat{y}\tilde{ abla}\hat{y}^{T}

In this case, J and \widehat{J} are rank-1 matrices. A rank-1 square matrix has only one nonzero eigenvalue, and by inspection, that one eigenvalue is

For $J$:|\lambda^{J}_{1}(W^{*}_{B})|\quad =\left.\eta_{B} abla\hat{y}^{T} abla\hat{y}\right|_{W^{*}_{B}} (31) \rightarrow|\lambda^{H}| \overset{\eqref{eqn:Jlam_bptt}}{=}|\lambda^{J}_{1}|/\eta_{B}= abla\hat{y}^{T} abla\hat{y} (32) For $\widehat{J$}:\quad|\widehat{\lambda^{J}_{1}}(W^{*}_{e})| =\left.\eta_{e}|\tilde{ abla}\hat{y}^{T} abla\hat{y}|\right|_{W^{*}_{e}} (33) \overset{(a)}{=}\left.|\eta_{e}\rho abla\hat{y}^{T} abla\hat{y}|\right|_{W^{*}_{e}} (34) \overset{~{}\eqref{eqn:simplifiedH}}{=}|\rho\eta_{e}\lambda^{H}_{1}(W^{*}_{e})|, (35)

where equality (a) is explained as follows. We first remind the reader that \rho is defined such that \vec{\hat{g}}=\rho\vec{g}+\vec{e} (Eq. 6 ). For the case of a scalar output \hat{y} , \vec{\hat{g}}=\frac{\partial L}{\partial\hat{y}}\tilde{\nabla}\hat{y} and \vec{g}=\frac{\partial L}{\partial\hat{y}}\nabla\hat{y} . So if we divide both sides of \vec{\hat{g}}=\rho\vec{g}+\vec{e} by \frac{\partial L}{\partial\hat{y}} we get \tilde{\nabla}\hat{y}=\rho\nabla\hat{y}+\vec{e}/\frac{\partial L}{\partial\hat{y}} . Since we have \vec{e}\perp\vec{g} by definition, then \vec{e}^{\top}\nabla\hat{y}=0 because \vec{g} is just a scaled \nabla\hat{y} when the output is a scalar. This leads to \tilde{\nabla}\hat{y}^{T}\nabla\hat{y}=(\rho\nabla\hat{y}+\vec{e}/\frac{\partial L}{\partial\hat{y}})^{\top}\nabla\hat{y}=\rho\nabla\hat{y}^{T}\nabla\hat{y} .

Since we assume the gradient descent dynamical system converges to an optimum, this corresponds to an asymptotic stable fixed point. Hence, |\lambda^{J}_{1}|<1 and |\widehat{\lambda^{J}_{1}}|<1 , which implies:

|\lambda^{J}_{1}(W^{*}_{B})|<1 \overset{\eqref{eqn:Jlam_bptt}}{\rightarrow}\eta_{B}|\lambda^{H}_{1}(W^{*}_{B})|<1\rightarrow|\lambda^{H}_{1}(W^{*}_{B})|<\frac{1}{\eta_{B}} (36) |\widehat{\lambda^{J}_{1}}(W^{*}_{e})|<1 \overset{\eqref{eqn:Jlam_ratio}}{\rightarrow}|\rho\eta_{e}\lambda^{H}_{1}(W^{*}_{e})|<1\rightarrow|\lambda^{H}_{1}(W^{*}_{e})|<\frac{1}{|\rho|\eta_{e}}

∎