## 3 Model Development and Training
### Backpropagation and Backpropagation Through Time  
The universal approximation theorem states that a neural network with sufficient neurons and appropriate activation functions can approximate any continuous function [25]. The neural network is a set of interconnected layers containing a vector of neurons. The vision of machine learning is to approximate desired functions by learning the best weights that map the input to the output. The main challenge is training the model and updating the weights to optimize the model's mathematical function. The most popular technique for training is backpropagation. Typically, when data is fed into the neural network, it flows sequentially from the input layer to the hidden layer and finally to the output layer. The output is then compared to the ground truth (the true data). The loss is a function that mirrors the difference between our model expectation and the true data and is computed.  
The backpropagation adjusts the weight of the whole network starting from the output layer and propagates back into the input layer to reduce the loss function (reduce the error). As the training goes back through each layer and reaches the recurrent layer with a self-loop, the recurrent layer should be unfolded [26]. Thus, the backpropagation method must go through every step to train the inner weights. To achieve this, the G-CTRNN uses the Backpropagation Through Time (BPTT) concept to adjust the internal weights and the time constant. In the continuous-time domain, BPTT unfolds the recurrent neural network over time by discretizing it into small time steps and computing gradients through each time step to update weights using the chain rule. We chose BPTT because it is the standard approach for training continuous-time recurrent models where hidden states depend on past temporal dynamics, which is central to G-CTRNN operation.  
To train the G-CTRNN using the BPTT, it is necessary to solve the associated ordinary differential equations (ODEs). One commonly used numerical method for solving ODEs is the Euler method [27], derived from the Taylor series. The G-CTRNN Euler representation was implemented using the TensorFlow library 2.8. TensorFlow is esteemed for its modern design, reliability, and adaptability [28]. In this implementation, the TensorFlow engine converts the network to a static computation graph before training, allowing full GPU acceleration and parallelism compatibility [29]. Figure 4 shows the steps to implement the  
Figure 3: Model architecture: The input consists of 3 acceleration features: X, Y, and Z. The interconnected recurrent layer contains 16 cells (256 interconnections), followed by a dense layer with 10 cells, and an output layer with 6 cells.  
forward and backward passes in TensorFlow for the G-CTRNN. These passes are tightly coupled in TensorFlow, but we separated them in the figure to illustrate the implementation details more clearly. In these steps, the auto-differentiation engine in TensorFlow traces the computational graph over all unrolled time steps. Using the chain rule, gradients are computed for both synaptic weights and time constants. The built-in ADAM optimizer updates parameters, with modifications to accommodate time-constant tuning. The ADAM Optimizer employs adaptive learning rates to accelerate the convergence of the model. During the initial learning phase of the model, it commences with a random initialization [30]. The ADAM optimizer leverages a movement's momentum to accelerate the progression consistently, leading to smooth convergences.