# Forecasting Economy-Related Data Utilizing Weight-Constrained Recurrent Neural Networks
## 3 Weight-Constrained Recurrent Neural Network Training Algorithm  
In this section, we present the Weight-Constrained Recurrent Neural Network (WCRNN) training algorithm, while a high level description is presented in Algorithm 1 for completeness. The original BFGS method requires the storage and manipulation of an \(n\times n\) matrix. Nevertheless, for large-scale problems such as neural network training, this is unwieldy. The limited-memory BFGS method attempts to alleviate this handicap by storing only a (usually) small number of \(m\) curvature pairs. Let \(\hat{m}=\min\{k,m-1\}\), then give the set of correction vector pairs \((s_{i},y_{i})_{i=k-1}^{k-\hat{m}}\) satisfying \(s_{i}^{T}y_{i}>0\). At each iteration, the algorithm approximates the error function \(E(w)\) at a point \(w_{k}\), utilizing a Hessian approximation \(B_{k}\) by a quadratic model \(m_{k}(w)\), namely:  
\[m_{k}(w)=E_{k}+g_{k}^{T}(w-w_{k})+\frac{1}{2}(w-w_{k})^{T}B_{k}(w-w_{k}), \tag{3}\]  
where \(E_{k}=E(w_{k})\), \(g_{k}=\
abla E(w_{k})\). The Hessian approximation \(B_{k}\) is defined (in compact form) in terms of the correction matrices \(S_{k}\) and \(Y_{k}\) as \(n\times m\) matrices:  
\[S_{k}=[s_{k-\hat{m}},\ldots,s_{k-1}]\qquad\text{and}\qquad Y_{k}=[y_{k-\hat{m} },\ldots,y_{k-1}].\]  
More specifically, the limited memory matrix \(B_{k}\) is obtained from \(\hat{m}\) updates to the basic matrix \(B_{0}^{(k)}=\theta_{k}I\) by:  
\[B_{k}=\theta_{k}I-W_{k}M_{k}^{-1}W_{k}, \tag{4}\]  
where:  
\[W_{k} = [\theta_{k}S_{k}\quad Y_{k}]\,,\] \[M_{k} = \left[ \begin{array}{cc}\theta_{k}S_{k}^{T}S_{k}&L_{k}\\ L_{k}^{T}&D_{k}\end{array} \right],\]  
\(\theta_{k}\) is a positive scalar, and \(D_{k}\) and \(L_{k}\) are the matrices:  
\[D_{k}=diag\left[s_{k-\hat{m}}^{T}y_{k-\hat{m}},\ldots,s_{k-1}^{T}y_{k-1} \right].\]  
and:  
\[(L_{k})_{ij}=\left\{ \begin{array}{ll}(s_{k-\hat{m}-1+i})^{T}(y_{k-\hat{m}-1+ i}),&\text{if $i>j$;}\\ 0,&\text{otherwise.}\end{array} \right.\]  
It is worth noting that the computation of \(B_{k}\) is performed via a computationally-efficient recursive technique presented by [PERSON] et al. (2018), which requires only vector inner products with complexity \(\mathcal{O}(m^{2}n)\). In the sequel, the algorithm will perform a minimization procedure of the approximation model \(m_{k}(w)\) to compute the new vector of weights, which consists of three stages: the generalized Cauchy point, the subspace minimization, and the line search. _Stage I: Cauchy point computation_. The basic aim of this stage is to minimize the model \(m_{k}(w)\) approximately subject to the feasible domain:  
\[D=\{w\in\mathbb{R}^{n}\,|\,l\leq w\leq u\},\]  
Therefore, the gradient projection method is utilized in order to compute the generalized Cauchy point \(w^{C}\) and eventually find a set of active bounds \(\mathcal{A}(w^{C})\). More specifically, let \(w_{k}\) be the current iterate and the path \(w(t)\) defined by:  
\[w(t)=P(w_{k}-t\
abla E_{k};l;u).\]where \(P\) denotes the projection of the steepest descent direction on the feasible domain \(D\). The generalized Cauchy point \(w^{C}\) is computed as the local minimum quadratic approximation of the error function on the path defined by \(w(t)\). Next, the active set \(\mathcal{A}(w^{C})\) consists of the indices of the variables whose values at \(w^{C}\) are at the lower or upper bound; thus, these variables are held fixed.