Package: keras-lmu
KerasLMU: Recurrent neural networks using Legendre Memory UnitsPaperThis is a Keras-based implementation of the
Legendre Memory Unit (LMU). The LMU is a novel memory cell for recurrent neural
networks that dynamically maintains information across long windows of time using
relatively few resources. It has been shown to perform as well as standard LSTM or
other RNN-based models in a variety of tasks, generally with fewer internal parameters
(seethis paperfor more details). For the Permuted Sequential MNIST (psMNIST) task in particular, it has been demonstrated to outperform the current state-of-the-art results. See the note below for instructions on how to get access to this model.The LMU is mathematically derived to orthogonalize its continuous-time history – doing
so by solvingdcoupled ordinary differential equations (ODEs), whose phase space
linearly maps onto sliding windows of time via the Legendre polynomials up to degreed− 1 (the example ford= 12 is shown below).A single LMU cell expresses the following computational graph, which takes in an input
signal,x, and couples a optimal linear memory,m, with a nonlinear hidden
state,h. By default, this coupling is trained via backpropagation, while the
dynamics of the memory remain fixed.The discretizedAandBmatrices are initialized according to the LMU’s
mathematical derivation with respect to some chosen window length,θ.
Backpropagation can be used to learn this time-scale, or fine-tuneAandB,
if necessary.Both the kernels,W, and the encoders,e, are learned. Intuitively, the kernels
learn to compute nonlinear functions across the memory, while the encoders learn to
project the relevant information into the memory (seepaperfor details).NoteThepaperbranch in thelmuGitHub repository includes a pre-trained
Keras/TensorFlow model, located atmodels/psMNIST-standard.hdf5, which obtains
a psMNIST result of97.15%. Note that the network is using fewer internal
state-variables and neurons than there are pixels in the input sequence.
To reproduce the results fromthis paper,
run the notebooks in theexperimentsdirectory within thepaperbranch.Nengo ExamplesLMUs in Nengo (with online learning)Spiking LMUs in Nengo Loihi (with online learning)LMUs in NengoDL (reproducing SotA on psMNIST)Citation@inproceedings{voelker2019lmu,
  title={Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks},
  author={Aaron R. Voelker and Ivana Kaji\'c and Chris Eliasmith},
  booktitle={Advances in Neural Information Processing Systems},
  pages={15544--15553},
  year={2019}
}Release history0.7.0 (July 20, 2023)Compatible with TensorFlow 2.4 - 2.13ChangedMinimum supported Python version is now 3.8 (3.7 reached end of life in June 2023).
(#54)0.6.0 (May 5, 2023)Compatible with TensorFlow 2.4 - 2.11ChangedLMUFeedforwardcan now be used with unknown sequence lengths, andLMUwill
useLMUFeedforwardfor unknown sequence lengths (as long as the other conditions
are met, as before). (#52)Allowinput_to_hidden=Truewithhidden_cell=None. This will act as a skip
connection. (#52)Changed order of LMU states so that the LMU memory state always comes first, and
any states from the hidden cell come afterwards. (#52)FixedFixed errors when setting non-default dtype on LMU layers. (#52)0.5.0 (January 26, 2023)Compatible with TensorFlow 2.4 - 2.11AddedLayers are registered with the Keras serialization system (no longer need to
be passed ascustom_objects). (#49)0.4.2 (May 17, 2022)Compatible with TensorFlow 2.1 - 2.9AddedAdded support for TensorFlow 2.9. (#48)0.4.1 (February 10, 2022)Compatible with TensorFlow 2.1 - 2.8AddedAdded support for TensorFlow 2.8. (#46)Allow for optional bias on the memory component with theuse_biasflag. (#44)Added regularizer support for kernel, recurrent kernel, and bias. (#44)0.4.0 (August 16, 2021)Compatible with TensorFlow 2.1 - 2.7AddedSettingkernel_initializer=Nonenow removes the dense input kernel. (#40)Thekeras_lmu.LMUFFTlayer now supportsmemory_d > 1.keras_lmu.LMUnow
uses this implementation for all values ofmemory_dwhen feedforward conditions
are satisfied (no hidden-to-memory or memory-to-memory connections,
and the sequence length is notNone). (#40)Addedtrainable_thetaoption, which will allow thethetaparameter to be
learned during training. (#41)Addeddiscretizeroption, which controls the method used to solve for theAandBLMU matrices. This is mainly useful in combination withtrainable_theta=True, where settingdiscretizer="euler"may improve the
training speed (possibly at the cost of some accuracy). (#41)Thekeras_lmu.LMUFFTlayer can now use raw convolution internally (as opposed to
FFT-based convolution). The newconv_modeoption exposes this. The newtruncate_iroption allows truncating the impulse response when running with a
raw convolution mode, for efficiency. Whether FFT-based or raw convolution is faster
depends on the specific model, hardware, and amount of truncation. (#42)ChangedTheAandBmatrices are now stored as constants instead of non-trainable
variables. This can improve the training/inference speed, but it means that saved
weights from previous versions will be incompatible. (#41)Renamedkeras_lmu.LMUFFTtokeras_lmu.LMUFeedforward. (#42)FixedFixed dropout support in TensorFlow 2.6. (#42)0.3.1 (November 16, 2020)ChangedRaise a validation error ifhidden_to_memoryorinput_to_hiddenare True
whenhidden_cell=None. (#26)FixedFixed a bug with the autoswapping inkeras_lmu.LMUduring training. (#28)Fixed a bug where dropout mask was not being reset properly in the hidden cell.
(#29)0.3.0 (November 6, 2020)ChangedRenamed module fromlmutokeras_lmu(so it will now be imported viaimport keras_lmu), renamed package fromlmutokeras-lmu(so it will now be installed viapip installkeras-lmu), and
changed any references to “NengoLMU” to “KerasLMU” (since this implementation is
based in the Keras framework rather than Nengo). In the future thelmunamespace
will be used as a meta-package to encapsulate LMU implementations in different
frameworks. (#24)0.2.0 (November 2, 2020)AddedAdded documentation for package description, installation, usage, API, examples,
and project information. (#20)Added LMU FFT cell variant and auto-switching LMU class. (#21)LMUs can now be used with any Keras RNN cell (e.g. LSTMs or GRUs) through thehidden_cellparameter. This can take an RNN cell (liketf.keras.layers.SimpleRNNCellortf.keras.layers.LSTMCell) or a feedforward
layer (liketf.keras.layers.Dense) orNone(to create a memory-only LMU).
The output of the LMU memory component will be fed to thehidden_cell.
(#22)Addedhidden_to_memory,memory_to_memory, andinput_to_hiddenparameters
toLMUCell, which can be used to enable/disable connections between components
of the LMU. They default to disabled. (#22)LMUs can now be used with multi-dimensional memory components. This is controlled
through a newmemory_dparameter ofLMUCell. (#22)Addeddropoutparameter toLMUCell(which applies dropout to the input)
andrecurrent_dropout(which applies dropout to thememory_to_memoryconnection, if it is enabled). Note that dropout can be added in the hidden
component through thehidden_cellobject. (#22)ChangedRenamedlmu.lmumodule tolmu.layers. (#22)Combined the*_encoders_initializer``parametersof ``LMUCellinto a singlekernel_initializerparameter. (#22)Combined the*_kernel_initializerparameters ofLMUCellinto a singlerecurrent_kernel_initializerparameter. (#22)RemovedRemovedLegendre,InputScaled,LMUCellODE, andLMUCellGatingclasses. (#22)Removed themethod,realizer, andfactoryarguments fromLMUCell(they will take on the same default values as before, they just cannot be changed).
(#22)Removed thetrainable_*arguments fromLMUCell. This functionality is
largely redundant with the new functionality added for enabling/disabling internal
LMU connections. These were primarily used previously for e.g. setting a connection to
zero and then disabling learning, which can now be done more efficiently by
disabling the connection entirely. (#22)Removed theunitsandhidden_activationparameters ofLMUCell(these are
now specified directly in thehidden_cell. (#22)Removed the dependency onnengolib. (#22)Dropped support for Python 3.5, which reached its end of life in September 2020.
(#22)0.1.0 (June 22, 2020)Initial release of KerasLMU 0.1.0! Supports Python 3.5+.The API is considered unstable; parts are likely to change in the future.Thanks to all of the contributors for making this possible!