The proposed numerical methods for estimating memory capacity, particularly the orthogonalized subspace and averaged orthogonalized subspace methods, are primarily designed for linear echo state networks (LESNs). However, their underlying principles can be adapted to nonlinear recurrent neural networks (RNNs) with some modifications.
In nonlinear RNNs, the dynamics are more complex due to the presence of nonlinear activation functions, which can affect the memory capacity. The key insight from the linear case is the Krylov structure of the controllability matrix, which captures the relationship between the input and the state dynamics. For nonlinear RNNs, one can still leverage the concept of controllability, but the analysis would need to account for the nonlinear transformations applied to the states.
To generalize the numerical methods, one could explore the use of Taylor expansions or other approximation techniques to linearize the nonlinear dynamics around a given operating point. This would allow for the application of similar Krylov-based techniques to estimate memory capacity in a local sense. Additionally, the neutrality of memory capacity to the input mask, as established in the linear case, may not hold in the same way for nonlinear architectures, necessitating a more nuanced approach to input design and its impact on memory capacity.