Yes, the insights from this work on linear models can be extended to understand the memory properties of more complex neural network models used in practical applications, albeit with caution. The foundational concepts of memory capacity, controllability, and the Krylov structure provide a framework that can be beneficial for analyzing more intricate architectures, including deep recurrent neural networks and hybrid models that combine recurrent and convolutional layers.
For instance, while linear models offer a clear and mathematically tractable understanding of memory capacity, the principles of controllability and the relationship between inputs and states can still be relevant in nonlinear and deep architectures. Researchers can investigate how the rank of the controllability matrix and the structure of the state dynamics influence memory properties in these more complex systems.
However, it is essential to recognize that nonlinearities and the increased dimensionality in deep networks introduce additional challenges. The interactions between layers, the choice of activation functions, and the training dynamics can all affect memory capacity in ways that are not present in linear models. Therefore, while the theoretical insights provide a valuable starting point, empirical validation and further theoretical development will be necessary to fully understand and optimize memory properties in complex neural network architectures used in real-world applications.