Before revision: Vanilla RNN with ReLU activation have a simple structure that lends itself to systematic dynamical systems analysis and interpretation, but they suffer from the exploding vs. vanishing gradients problem.
After revision: A main theoretical interest in biology and physics is to identify the nonlinear dynamical system (DS) that generated observed time series. Recurrent Neural Networks (RNNs) are, in principle, powerful enough to approximate any underlying DS, but in their vanilla form suffer from the exploding vs. vanishing gradients problem.