# Enhanced Localisation and Handwritten Digit Recognition Using ConvCARU
## 3 Proposed Method
### Feature Extraction and Adaptive Fusion  
Taking into account the spatiotemporal characteristics of videos, the challenge of maintaining spatial and temporal coherence is addressed by replacing traditional linear operations in RNN units with convolutional mechanisms. This optimises parameter efficiency and accelerates convergence, ensuring more effective feature retention. ConvCARU enhances this approach further by incorporating convolutional operations within 'cell' units instead of FC layers, thereby maintaining crucial spatial information that evolves over time. At its core, ConvCARU processes current-time-step data and the hidden state from the previous time step using a feature extraction framework comprising distinct modules. Combining ConvCARU-based video feature extraction with Conv2D networks for spatial prediction improves the system's ability to interpret dynamic sequences overall. Moreover, context-adaptive fusion is realised through a multi-level attention gate, which seamlessly integrates spatial and temporal elements. The primary components include the following:  
\[x_{(t)} =\textbf{W}e_{(t)}+\textbf{B} \tag{6a}\] \[n_{(t)} =\text{tanh}\Big{(}\textbf{W}e_{(t-1)}+\textbf{B}+x_{(t)}\Big{)}\] (6b) \[z_{(t)} =\sigma\Big{(}\textbf{W}e_{(t-1)}+\textbf{B}+\textbf{W}e_{(t)}+ \textbf{B}\Big{)}\] (6c) \[l_{(t)} =\sigma\Big{(}x_{(t)}\Big{)}\otimes z_{(t)}\] (6d) \[e_{(t)} =\Big{(}1-l_{(t)}\Big{)}\otimes e_{(t-1)}+l_{(t)}\otimes n_{(t)} \tag{6e}\]The received feature is encoded from Equations (6a)-(6e), with the bold variables \(W\) and \(B\) indicating trainable parameters. tanh and \(\sigma\)--as the activation functions--tend to push the input values to either end of the curve (0 or 1). The symbol \(\otimes\) denotes the tensor product, which combines two algebraic objects into a higher-dimensional structure while preserving their linear properties. This procedure facilitates the integration of both images and content via content-adaptive gates and involves the following steps:  
* The input undergoes processing through a linear layer. The resulting output determines the next hidden state, which is passed to the content-adaptive gate. If the previous hidden state \(e_{(t-1)}\) is unavailable, the output directly becomes \(e_{(t)}\).
* The linear layer combines the previous hidden state \(h_{t-1}\) with the output of (6a), and the sum is processed by the tanh activation function to extract integrated information.
* The CARU's update gate(s) facilitate hidden state transitions by combining the current input with the previous hidden state. This step uncovers relationships over content but introduces challenges in handling long-term dependencies.
* To address long-term dependency issues, this step evaluates the features of the current input, which are able to dynamically adjust through \(z_{(t)}\) conduction, boosting or reducing the input for accurate predictions during RNN decoding.
* The final output is derived through linear interpolation.  
ConvCARU is designed to extend CARU's ability to capture long-term dependencies within sequential data. It incorporates early fusion strategies to provide a more comprehensive understanding of appearance and motion patterns. Unlike conventional techniques, this model enhances spatial feature extraction from high-level representations during video prediction. Despite its structural similarities to basic ConvCARU architectures. As presented in Figure 2, it has several key internal differences. Rather than using conventional FC layers, convolutional operations within 'cell' units govern the processing of input signals, resulting in improved feature extraction fidelity. The underlying mathematical components that are central to the design of ConvCARU contribute to its effectiveness in modelling complex video sequences.