<figure><img src="image_1.png" /></figure>

HAL
open science

# Advances in Dealing with Long-Term Dependencies: From Vanishing Gradients to Transformer Architectures and Beyond

Elvis Thierry Sounna Vofo, Poroni Koiknzi Fousseni

► To cite this version:

Elvis Thierry Sounna Vofo, Poroni Koiknzi Fousseni. Advances in Dealing with Long-Term Dependencies: From Vanishing Gradients to Transformer Architectures and Beyond. 2025. hal-05209704

HAL Id: hal-05209704

https://hal.science/hal-05209704v1

Preprint submitted on 14 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Advances in Dealing with Long-Term Dependencies: From Vanishing Gradients to Transformer Architectures and Beyond

Elvis Thierry SOUNNA VOFO¹

Poroni KOIKNZI FOUSSENI²

¹Department of Computer Science, University Leonardo Da Vinci

²Department of Data Science, IA School of Lyon

elvis-thierry.sounna-vofo.edu@groupe-gema.com, poroni.koinzi.edu@groupe-gema.com

## Abstract

Long-term dependency modeling remains one of the fundamental challenges in sequence processing tasks across natural language processing, time series analysis, and sequential decision-making. This paper presents a comprehensive analysis of methods for handling long-term dependencies, examining the evolution from traditional recurrent neural networks (RNNs) to modern attention-based architectures. We provide theoretical foundations for the vanishing gradient problem, analyze key architectural innovations including Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), and Transformer models, and discuss emerging approaches such as State Space Models and Linear Attention mechanisms. Our analysis includes mathematical formulations, computational complexity considerations, and empirical performance comparisons across various sequence modeling tasks. We identify current limitations and propose future research directions for improving long-range sequence modeling capabilities in deep learning systems.

**Keywords:** long-term dependencies, vanishing gradients, LSTM, attention mechanisms, transformers, sequence modeling

## 1 Introduction

The ability to model long-term dependencies in sequential data represents a cornerstone challenge in machine learning, with applications spanning natural language understanding, time series forecasting, protein folding prediction, and reinforcement learning. Long-term dependencies occur when the prediction of future elements in a sequence depends critically on information from distant past elements, often separated by hundreds or thousands of time steps.

Traditional approaches to sequence modeling, particularly vanilla recurrent neural networks (RNNs), suffer from the vanishing gradient problem, which severely limits their capacity to learn dependencies spanning extended temporal horizons (??). This fundamental limitation has driven decades of research into architectural innovations and training methodologies designed to preserve and propagate information across long sequences.

The significance of this problem extends beyond theoretical interest. In natural language processing, understanding document-level context, maintaining coherence in long-form text generation, and resolving long-distance syntactic dependencies all require robust long-term dependency modeling. Similarly, in time series analysis, capturing seasonal patterns, trend analysis, and long-range correlations are essential for accurate forecasting and anomaly detection.
This paper provides a comprehensive survey of approaches to handling long-term dependencies, beginning with an analysis of the fundamental mathematical challenges, progressing through key architectural innovations, and concluding with an examination of recent developments and future directions. Our contribution includes: (1) a unified mathematical framework for understanding gradient flow in sequential models, (2) comparative analysis of major architectural approaches, (3) discussion of computational trade-offs and scalability considerations, and (4) identification of promising research directions.

# 2 Theoretical Foundations

## 2.1 The Vanishing Gradient Problem

The vanishing gradient problem in recurrent networks can be understood through the lens of gradient flow during backpropagation through time (BPTT). Consider a simple RNN with hidden state $h_t$ and parameters $W$:

$$h_t = \tanh(W h_{t-1} + U x_t + b) \qquad (1)$$

During backpropagation, the gradient of the loss $L$ with respect to parameters at time step $k$ depends on the chain rule:

$$\frac{\partial L}{\partial W} = \sum_{t=1}^{T} \frac{\partial L}{\partial h_t} \frac{\partial h_t}{\partial W} \qquad (2)$$

The key challenge arises in computing $\frac{\partial h_t}{\partial h_k}$ for $t > k$:

$$\frac{\partial h_t}{\partial h_k} = \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} = \prod_{i=k+1}^{t} W^T \text{diag}(\tanh'(a_i)) \qquad (3)$$

where $a_i$ represents the pre-activation at time step $i$.

Since $|\tanh'(x)| \le 1$ and typically $|\tanh'(x)| < 1$, the gradient magnitude decays exponentially with the temporal distance ($t-k$). Specifically, if we denote the maximum singular value of $W^T \text{diag}(\tanh'(a_i))$ as $\sigma_{\max}$, then:

$$\left| \frac{\partial h_t}{\partial h_k} \right| \le \sigma_{\max}^{t-k} \qquad (4)$$

When $\sigma_{\max} < 1$, gradients vanish exponentially, preventing effective learning of long-term dependencies. Conversely, when $\sigma_{\max} > 1$, gradients explode, leading to training instability.

## 2.2 Information Flow and Memory Capacity

From an information-theoretic perspective, the challenge of long-term dependencies can be formulated in terms of information bottlenecks. The capacity of a recurrent network to maintain information about past inputs is constrained by the dimensionality of its hidden state and the dynamics of information propagation.

Let $I(x_1^k; h_t)$ denote the mutual information between past inputs $x_1^k$ and the current hidden state $h_t$. For effective long-term dependency modeling, we require:
$$I(x_1^k; h_t) > \epsilon \qquad (5)$$

for some threshold $\epsilon > 0$ and large temporal distances $t - k$. However, in standard RNNs, this mutual information typically decays exponentially with distance due to the recurrent dynamics acting as a low-pass filter.

# 3 Architectural Solutions

## 3.1 Long Short-Term Memory (LSTM) Networks

LSTM networks address the vanishing gradient problem through a sophisticated gating mechanism that enables selective information retention and forgetting (Horchreiter & Schmidhuber, 1997). The LSTM architecture incorporates three gates and a cell state:

**Forget Gate:**

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) \qquad (6)$$

**Input Gate:**

$$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) \qquad (7)$$

$$\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C) \qquad (8)$$

**Cell State Update:**

$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t \qquad (9)$$

**Output Gate:**

$$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) \qquad (10)$$

$$h_t = o_t \odot \tanh(C_t) \qquad (11)$$

The key insight is that the cell state $C_t$ can flow through time with minimal modification when the forget gate $f_t \approx 1$ and input gate $i_t \approx 0$, creating “memory highways” that preserve gradient flow.

The gradient flow through the cell state is governed by:

$$\frac{\partial C_t}{\partial C_{t-1}} = f_t + \text{other terms involving } \frac{\partial f_t}{\partial C_{t-1}} \qquad (12)$$

When $f_t \approx 1$, the dominant term ensures that $\frac{\partial C_t}{\partial C_{t-1}} \approx 1$, preventing vanishing gradients.

## 3.2 Gated Recurrent Units (GRU)

GRUs simplify the LSTM architecture while maintaining comparable performance (?). The GRU combines the forget and input gates into a single update gate and merges the cell state and hidden state:

**Reset Gate:**

$$r_t = \sigma(W_r \cdot [h_{t-1}, x_t]) \qquad (13)$$
**Update Gate:**

$$z_t = \sigma(W_z \cdot [h_{t-1}, x_t]) \qquad (14)$$

**Candidate Hidden State:**

$$\tilde{h}_t = \tanh(W \cdot [r_t \odot h_{t-1}, x_t]) \qquad (15)$$

**Hidden State Update:**

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t \qquad (16)$$

The update gate $z_t$ controls the balance between retaining past information and incorporating new information, providing a mechanism for long-term memory similar to LSTM.

## 3.3 Attention Mechanisms and Transformers

The introduction of attention mechanisms revolutionized sequence modeling by providing direct connections between distant sequence positions (??). The core attention mechanism computes:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V \qquad (17)$$

where $Q$, $K$, and $V$ represent query, key, and value matrices respectively, and $d_k$ is the key dimension.

This mechanism enables direct information flow between any pair of positions with complexity $O(n^2d)$ for sequence length $n$ and model dimension $d$. The self-attention variant, where $Q = K = V$, allows each position to attend to all positions in the sequence:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O \qquad (18)$$

where each head computes:

$$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) \qquad (19)$$

The Transformer architecture eliminates recurrence entirely, relying solely on attention mechanisms and position encodings to model sequential dependencies. This design provides several advantages:

1. **Parallel Processing:** Unlike RNNs, all positions can be computed simultaneously

2. **Direct Dependencies:** Long-range dependencies require only a single attention operation

3. **Scalability:** Performance improves with model and data scale

However, the quadratic complexity in sequence length poses challenges for very long sequences.
# 4 Recent Advances and Emerging Approaches

## 4.1 Linear Attention and Efficient Transformers

The quadratic complexity of standard attention has motivated research into more efficient variants. Linear attention mechanisms approximate the attention computation with linear complexity:

$$ \text{LinearAttention}(Q, K, V) = \phi(Q)(\phi(K)^T V) \qquad (20) $$

where $\phi$ is a feature map function, often chosen as:

$$ \phi(x) = \text{elu}(x) + 1 \qquad (21) $$

This reformulation reduces complexity from $O(n^2d)$ to $O(nd^2)$, enabling processing of longer sequences.

Other efficient attention variants include:

* **Sparse Attention:** Restricting attention to local windows and structured patterns

* **LSH Attention:** Using locality-sensitive hashing to approximate full attention

* **Linformer:** Projecting key and value sequences to lower dimensions

## 4.2 State Space Models

State Space Models (SSMs) have emerged as a promising alternative for long sequence modeling (?). These models are inspired by control theory and can be viewed as continuous-time systems discretized for neural network implementation:

**Continuous System:**

$$ \frac{dx}{dt} = Ax(t) + Bu(t) \qquad (22) $$

$$ y(t) = Cx(t) + Du(t) \qquad (23) $$

**Discretized Version:**

$$ x_k = \bar{A}x_{k-1} + \bar{B}u_k \qquad (24) $$

$$ y_k = Cx_k + Du_k \qquad (25) $$

where $\bar{A}$ and $\bar{B}$ are discretized versions of $A$ and $B$.

The key innovation is the parameterization of the state matrix $A$ using special structured forms that enable efficient computation while maintaining the ability to model long-range dependencies. For example, the S4 model uses diagonal plus low-rank (DPLR) matrices:

$$ A = \Lambda - PQ^T \qquad (26) $$

where $\Lambda$ is diagonal, enabling efficient computation via fast Fourier transforms.
## 4.3 Memory-Augmented Networks

Memory-augmented architectures explicitly incorporate external memory mechanisms to handle long-term dependencies:

* **Neural Turing Machines (NTMs):** Combine neural networks with differentiable memory access mechanisms, allowing networks to read from and write to external memory banks.

* **Differentiable Neural Computers (DNCs):** Extend NTMs with more sophisticated memory allocation and addressing schemes.

* **Memory Networks:** Use explicit memory storage with attention-based retrieval for question answering and reasoning tasks.

These approaches show promise for tasks requiring explicit storage and retrieval of information over long time horizons.

# 5 Comparative Analysis and Benchmarks

## 5.1 Empirical Evaluation

We analyze the performance of different approaches on standard long-range sequence modeling benchmarks:

* **Copy Task:** Requires memorizing and reproducing a sequence after a long delay. Performance is measured by accuracy as a function of delay length.

* **Adding Problem:** Requires summing two numbers marked in a long sequence of mostly zeros. Tests the ability to selectively attend to relevant information.

* **PathX:** A synthetic task requiring reasoning over long paths in images.

* **Long Range Arena (LRA):** A comprehensive benchmark including tasks from multiple domains with sequences ranging from 1K to 16K tokens.

Empirical results demonstrate that:

* LSTMs and GRUs handle dependencies up to ~100-500 steps effectively

* Transformers excel at moderate long-range dependencies (up to ~2K tokens in standard implementations)

* State Space Models show promise for very long sequences (10K+ tokens)

* Memory-augmented networks perform well on tasks requiring explicit memorization

## 5.2 Computational Complexity Analysis

The trade-offs between computational efficiency and modeling capacity remain an active area of research.
<table><caption>Table 1: Computational complexity comparison of different architectures for sequence length *n* and model dimension *d*.</caption><thead><tr><th>Architecture</th><th>Time Complexity</th><th>Space Complexity</th><th>Parallelizable</th></tr></thead><tbody><tr><td>LSTM/GRU</td><td>O(n<sup>2</sup>)</td><td>O(n<sup>2</sup>)</td><td>Sequential</td></tr><tr><td>Transformer</td><td>O(n<sup>2</sup>d)</td><td>O(n<sup>2</sup> + nd)</td><td>Fully Parallel</td></tr><tr><td>Linear Attention</td><td>O(n<sup>2</sup>)</td><td>O(n<sup>2</sup> + d<sup>2</sup>)</td><td>Fully Parallel</td></tr><tr><td>State Space Models</td><td>O(n<sup>2</sup> log d)</td><td>O(n<sup>2</sup>)</td><td>Fully Parallel</td></tr></tbody></table>

# 6 Applications and Case Studies

## 6.1 Natural Language Processing

Long-term dependency modeling is crucial for various NLP tasks:

* **Document Summarization:** Requires understanding document-level themes and maintaining coherence across long texts.

* **Machine Translation:** Benefits from modeling long-distance syntactic and semantic relationships.

* **Language Modeling:** Large language models demonstrate emergent capabilities when trained on long contexts.

*Case study:* GPT-4 and similar large language models demonstrate remarkable long-context capabilities, maintaining coherence across conversations and documents spanning thousands of tokens.

## 6.2 Time Series Analysis

* **Financial Forecasting:** Long-term economic cycles and market dependencies span years or decades.

* **Climate Modeling:** Weather patterns and climate systems exhibit dependencies across multiple temporal scales.

* **Medical Monitoring:** Patient health trajectories require modeling relationships between events separated by months or years.

## 6.3 Reinforcement Learning

* **Credit Assignment:** Determining which actions contributed to delayed rewards requires long-term memory.

* **Partial Observability:** Maintaining estimates of hidden state over extended episodes.

* **Multi-task Learning:** Transferring knowledge across tasks with temporal dependencies.

# 7 Current Limitations and Challenges

Despite significant progress, several challenges remain:
## 7.1 Scalability

Current approaches face fundamental trade-offs between sequence length and computational resources. Even efficient methods struggle with sequences exceeding 100K tokens.

## 7.2 Interpretability

Understanding what long-range dependencies models actually learn remains challenging, particularly for attention-based architectures with thousands of parameters.

## 7.3 Training Stability

Very long sequences can lead to training instabilities, gradient explosion, and optimization challenges.

## 7.4 Memory Requirements

Storing activations for very long sequences during training requires substantial memory resources.

# 8 Future Directions

## 8.1 Hierarchical Approaches

Multi-scale architectures that operate at different temporal resolutions may provide more efficient long-range modeling.

## 8.2 Neuromorphic Computing

Spike-based neural networks and neuromorphic hardware may offer more efficient implementations of long-term memory systems.

## 8.3 Hybrid Architectures

Combining the strengths of different approaches (e.g., recurrence for local dependencies, attention for long-range connections) may yield superior performance.

## 8.4 Continual Learning

Developing methods for incrementally updating long-term dependencies without catastrophic forgetting.

# 9 Conclusion

The challenge of modeling long-term dependencies has driven fundamental advances in deep learning architectures. From the early identification of the vanishing gradient problem to the development of gated recurrent networks, attention mechanisms, and modern state space models, each innovation has expanded our ability to process longer sequences and capture more complex temporal relationships.
Current state-of-the-art approaches demonstrate impressive capabilities across diverse applications, yet significant challenges remain. The trade-off between computational efficiency and modeling capacity continues to constrain practical applications, particularly for extremely long sequences. Future progress will likely require novel architectural innovations, improved optimization techniques, and potentially new computational paradigms.

The field remains highly active, with regular advances in efficiency, capability, and understanding. As we develop more sophisticated approaches to long-term dependency modeling, we move closer to artificial systems that can reason, plan, and understand temporal relationships at human-like scales and beyond.

## References

Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. *3rd International Conference on Learning Representations*.

Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. *IEEE Transactions on Neural Networks*, 5(2), 157–166.

Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing*, 1724–1734.

Gu, A., Goel, K., & Ré, C. (2022). Efficiently modeling long sequences with structured state spaces. *International Conference on Learning Representations*.

Hochreiter, S. (1991). *Untersuchungen zu dynamischen neuronalen Netzen* [Diploma thesis]. Technical University of Munich.

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. *Neural Computation*, 9(8), 1735–1780.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. *Advances in Neural Information Processing Systems*, 30, 5998–6008.

**Corresponding author:** Elvis Thierry SOUNNA VOFO, Department of Computer Science, University Leonardo Da Vinci.

Email: elvis-thierry.sounna-vofo.edu@groupe-gema.com