## 2 Materials and Methods
### Image Classification Methods
#### 2.3.3 Liquid Time-Constant Networks  
Although an LSTM can effectively solve the vanishing or exploding gradient problem, RNN models also face another problem, which is that they operate in discrete time steps, which can be inefficient for modeling continuous-time processes or data with irregular time intervals. Another limitation of traditional RNNs lies in their memory efficiency. Traditional RNNs suffer from a fixed-size memory bottleneck, where they need to store the entire history of past inputs and activations to compute the current hidden state. This can lead to memory constraints, especially when dealing with long sequences of data. To address these limitations, researchers have introduced a novel framework called Neural Ordinary Differential Equations (Neural ODEs) [33], which seamlessly integrates ordinary differential equations (ODEs) with neural networks. In a Neural ODE-based RNN, instead of explicitly updating the hidden state at each time step, the hidden state evolves continuously over time according to an ODE. The ODE is parameterized by a neural network that takes the current hidden state, the current time, and possibly other variables as input. The solution to this ODE provides the hidden state trajectory over time. Mathematically, the dynamics of the hidden state in a Neural ODE-based RNN can be described as follows:  
\[\frac{dx(t)}{dt}=f(x(t),t;\theta), \tag{3}\]  
where  
* \(x(t)\) is the hidden state at time \(t\);
* \(f\) is a neural network function with parameters \(\theta\) that determines the rate of change of \(x(t)\) with respect to time;
* \(t\) represents time. During training, the parameters of the neural network \(\theta\) are learned by optimizing a loss function using gradient-based optimization techniques. This is typically performed by solving the ODE using numerical methods such as Euler's method or Runge-Kutta methods and then applying backpropagation through the ODE solver to compute gradients with respect to the parameters and optimize the model based on those gradients. Applying Neural ODEs to an RNN offers several significant benefits. Firstly, Neural ODEs provide a more flexible and expressive framework for modeling sequential data, allowing for the effective capture of complex temporal dynamics and long-range dependencies. By seamlessly integrating ordinary differential equations with neural networks, a Neural ODE-based RNN can adaptively adjust the rate of change of the hidden state over time, enabling more efficient and memory-effective processing of sequential data. Additionally, Neural ODEs offer a continuous-time formulation, which is particularly advantageous for modeling continuous-time processes or data with irregular time intervals. This continuous-time approach not only improves the scalability and performance of RNNs but also enhances their ability to handle diverse types of sequential data in various applications such as time series prediction, sequence generation, and dynamic system modeling. To enhance the stability of the model, Ref. [34] introduced an additional term to the differential equation, \(-\frac{x(t)}{\tau}\), forming a new equation to calculate the hidden state:  
\[\frac{dx(t)}{dt}=-\frac{x(t)}{\tau}+f(x(t),I(t),t;\theta), \tag{4}\]where  
* The term \(-\frac{x(t)}{\tau}\) represents a decay term where \(\tau\) is a time-constant;
* \(x(t)\) is the hidden state at time \(t\);
* \(I(t)\) is the input;
* \(f\) is a neural network function with parameter \(\theta\) that determines the rate of change of \(x(t)\) with respect to time;
* \(t\) represents time. This modification facilitates the creation of a continuous-time recurrent neural network (CT-RNN) that exhibits increased stability [34]. This decay term \(-\frac{x(t)}{\tau}\), characterized by the time constant \(\tau\), plays a crucial role in guiding the system towards equilibrium, thus enhancing its ability to effectively capture and represent temporal dynamics inherent in sequential data.