# A Topological Approach to Enhancing Consistency in Machine Learning via Recurrent Neural Networks
## 3 Methodology
### Consistent RNN Cell (ConsRNN)  
Some specialized RNN transition functions have been developed to address specific issues or improve upon standard RNNs like LSTMs and GRUs [20, 21, 32, 33, 34, 35, 36, 37, 38, 39]. However, only [20, 21] explicitly focused on ensuring consistency in predictions. Our treatment here will closely follow these two references, and we will observe that by using a very simple variant of the transition function in cited work, accurate and consistent results can be obtained in applications. To explain consistency, we use the work of [PERSON] and [PERSON] [20] and as an example, the continuous input path in Figure 2A. The shown path is represented by input values \(\{x_{1},x_{2},\ldots,x_{n}\}\) in a finite dimensional Euclidean space. Further, we can also construct a vector of corresponding increments \(\Delta x_{i}=x_{i}-x_{i-1}\) as shown in Figure 2A. Accordingto [20; 21], when the number of increments exceeds a certain threshold for the continuous input curve, the incremental differences between time step inputs can be assumed to be linear, as shown in the inset of Figure 2A and more explicitly in Figure 2B. Consequently, the incremental differences \(\Delta x\) can be used in the training of specialized RNN Cell instead of the time step input values \(x\). Keeping this in mind, and letting \(v\) be the Euclidean norm of \(\{\Delta x_{1},\Delta x_{2},\ldots,\Delta x_{n}\}\), the given increment is decomposed as  
\[\Delta x_{t}=\alpha v+(\beta-\alpha)v. \tag{1}\]  
where \(\beta=v^{-1}\Delta x_{t}\), and \(\alpha<\beta\) are scalar values, as shown in Figure 2B. Based on the assumption of the linearization of the continuous input curves, [PERSON] and [PERSON] [20] defined the consistency condition of the RNN transition function as  
\[f_{W}(h_{t-1},\Delta x_{t})=f_{W}(h_{t-1},\beta v)=f_{W}(f_{W}(h_{t-1},\alpha v ),(\beta-\alpha)v), \tag{2}\]  
where \(h_{t}\) is the state variable corresponding to the input variable \(\Delta x_{t}\). The conditional statement in Equation (2) is required to achieve consistent predictions from the RNN transition function. Therefore, any given transition formula of the RNN that satisfies the condition in Equation (2) would ensure consistent estimates. In our approach, we first observe that for the function \(f_{W}(a,b)=a+b\), the consistency condition in Equation (2) can be easily verified. Considering the state and increment as the inputs, this in particular reads as in Equation (3). \[f_{W}(h_{t-1},\Delta x_{t})=f_{W}(h_{t-1},\beta v)=h_{t-1}+\beta v. \tag{3}\]  
Considering the formula presented in Equation (3), we introduced the transition function in our work as follows:  
\[f_{W}(h_{t-1},\Delta x_{t})=h_{t-1}+\Big{(}g_{W}\bigg{(}h_{t-1},\frac{\Delta x _{t}}{v}\bigg{)}-h_{t-1}\bigg{)}v. \tag{4}\]  
Figure 2: Representation of an incremental update during the learning process of recurrent neural networks (RNNs) in 2D, shown in (**A**). The condition in Equation (2) must hold on the RNN transition function to provide consistent predictions. The \(\alpha\) and \(\beta\) values in Equation (2) is illustrated through one incremental update. As shown in (**B**), an incremental difference \(\Delta x_{t}\) is divided by the norm of the input vector, expressed as \(\frac{\Delta x_{t}}{v}\). At the introduced transition formula in Equation (4), while the \(g_{W}(h_{t-1},\frac{\Delta x_{t}}{v})-h_{t-1}\) learns the update effect of \(\beta\) on the previous state vector, multiplying this difference with the norm as \(v\) gives the total update. Here, \(g_{W}\) represents any given update transition function. The heuristic intuition and the main assumption in the introduced transition formula in Equation (4) is that the update function \(g_{W}(h_{t-1},\frac{\Delta x_{t}}{v})\) learns how much update occurs in the state vector based on the incremental difference input divided by the norm, which is \(\frac{\Delta x_{t}}{v}\). Multiplying the difference \(g_{W}(h_{t-1},\frac{\Delta x_{t}}{v})-h_{t-1}\) with the norm \(v\) will give the total essential update.