## 3 Proposed BICORN-RNN
### Random Mapping Input Space  
We assume that the network is built up properly, so as to obtain suitable dynamics dependencies. At each time-step, the network stores the previous outputs as an additional predictive sequence. We combine these predictions with the current input and feed it into the network, as illustrated in Figure 3.  
This predictive information can provide more representative features and help extract more correlations within the input series. Note that the merger is in virtue of a saturating nonlinear function \(\sigma(\cdot)\), such as logistic sigmoid function, squashing the two parts of the inputs into (0,1) nonlinearly, to avoid degeneration into a linear combination. So, at the \(t\)-th time-step, there is,  
\[x^{\prime}(t)=\sigma\Bigg{(}x(t)+\sum\limits_{i=1}^{t-1}\beta_{i}\mathbf{P}_{ i}y(i)\Bigg{)} \tag{10}\]  
where \(x^{\prime}(t)\) is the new input. It consists of two parts: one part is the original input value at the \(t\)-th time-step, that is \(x(t)\); the other part is the prediction information \(y(i)\) (\(i=1\),..., \(t\)-1) at each time before. And we use a random matrix \(\mathbf{P}_{i}\) to project \(y(i)\) onto a high-dimensional space, and then \(\sum\limits_{i=1}^{t-1}\beta_{i}\mathbf{P}_{i}y(i)\) is weighted cumulative of \(\mathbf{P}_{i}y(i)\). \(\beta_{i}\) denotes the weight parameter which determines the proportion of the predicted output. We make \(\sum\limits_{i=1}^{t-1}\beta_{i}=1\) to prevent immoderate predictions from visiting or over-diluting \(x(t)\). Notably, a significant operation is random mapping, i.e., the previous predictions would be randomly projected onto a high-dimensional space by a random matrix \(\mathbf{P}_{i}\in\mathbb{R}^{\text{ T}\times\text{K}}\), \(i\in[1,t-1]\), whose elements are normally distributed in (0,1). It modifies the input data examples separately, and does not casually change the characteristics of the data itself. In addition, such non-linearity projections, akin to the kernel functions of support vector machines, have desirable properties in terms of nonlinear classification and regression problems. Intuitively, random mapping can extend data from different labels to different directions, making the modified data more distinguishable. As the quasi-orthogonality property of a high-dimensional space proved in [33], when the dimension of the feature space T is relatively large, the column vectors of \(\mathbf{P}_{i}\) are much likely to be approximately orthogonal. It is a mathematical certainty that each column vector corresponds to the bias of per-class applied to the original sample \(x(t)\), if the real output \(y(t)\) is equal to the expected output \(\hat{y}(t)\). So, the random projection of previous predictions pushes the input examples apart from one another, with high probability [34]. The following proposition provides further evidence that, with accurate predictions of the target class in advance, it is possible to move the data manifolds apart by adding an offset to  
Figure 3: Illustration of an unfolded (RaNdom Recurrent Neural Networks) RN-RNN in time, where the input receives the random projection of output predictions of all the previous time-steps. per-class examples, so as to guarantee an improvement in classification accuracy by using the RN-RNNs classifier. Let \(\xi\) be a set of data examples \((x(t),d_{k})\) at the \(t\)-th time-step, where \(x(t)\in\mathbb{R}^{\text{ D}\times 1}\) is the input example and \(d_{k}\in\{1,\cdots,\text{K}\}\) is the target class label of \(x(t)\). Let \(\theta\) be the corresponding parameters of RN-RNNs classifier with the loss function \(L(\xi,\theta)\). Then, there exists \(w_{d_{k}}\in\mathbb{R}^{\text{ D}\times 1}\) such that the translated set \(\xi^{\prime}\) defined as \(\big{(}x(t)+w_{d_{k}},d_{k}\big{)}\) results in a better optimum \(L(\xi^{\prime},\theta)\leq L(\xi,\theta)\). Assume that \(f\) and \(g\) are the element-wise and monotonically increasing activation function and output function of the RN-RNNs classifier.