Package: lava-dl
Lava DLlava-dlis a library of deep learning tools within Lava that support  offline training, online training and inference methods for various Deep Event-Based Networks.There are two main strategies for training Deep Event-Based Networks:direct trainingandANN to SNN converison.Directly training the network utilizes the information of precise timing of events. Direct training is very accurate and results in efficient networks. However, directly training networks take a lot of time and resources.On the other hand, ANN to SNN conversion is especially suitable for rate coded SNNs where we can leverage fast training of ANNs. These converted SNNs, however, typically require increased latency compared to directly trained SNNs.Lava-DL provides an improved version ofSLAYERfor direct training of deep event based networks and a new ANN-SNN accelerated training approach calledBootstrapto mitigate high latency issue of conventional ANN-SNN methods for training Deep Event-Based Networks.The lava-dl training libraries are independent of the core lava library since Lava Processes cannot be trained directly at this point. Instead, lava-dl is first used to train the model which can then be converted to a network of Lava processes using the netx library using platform independent hdf5 network description.The library presently consists oflava.lib.dl.slayerfor natively training Deep Event-Based Networks.lava.lib.dl.bootstrapfor training rate coded SNNs.lava.lib.dl.netxfor training and deployment of event-based deep neural networks on traditional as well as neuromorphic backends.Lava-dl also has the following external, fully compatible, plugin.lava.lib.dl.decollefor training Deep SNNs with local learning and surrogate gradients. This extension is an implementation ofDECOLLElearning repo to be fully compatible to lava-dl training tools. Referherefor the detailed description of the extension, examples and tutorials.J. Kaiser, H. Mostafa, and E. Neftci,Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE).pp 424,  Frontiers in Neuroscience 2020.More tools will be added in the future.Lava-DL WorkflowTypical Lava-DL workflow:Training:usinglava.lib.dl.{slayer/bootstrap/decolle}which results in ahdf5 network description. Training usually follows an iterative cycle of architecture design, hyperparameter tuning, and backpropagation training.Inference:usinglava.lib.dl.netxwhich generates lava proces from the hdf5 network description of the trained network and enables inference on different backends.InstallationCloning Lava-DL and Running from SourceNote:The instructions to follow will setup a virtual environement and install lava-dl and all dependencies in that virtual environment. Pleasesetup git-lfsto ensure large files are pulled during thegit clone.[Linux/MacOS]cd$HOMEgitclonegit@github.com:lava-nc/lava-dl.gitcdlava-dl
curl-sSLhttps://install.python-poetry.org|python3-
poetryconfigvirtualenvs.in-projecttruepoetryinstallsource.venv/bin/activate
pytest[Windows]# Commands using PowerShellcd$HOMEgitclonegit@github.com:lava-nc/lava-dl.gitcdlava-dlpython3-mvenv.venv.venv\Scripts\activatecurl-sSLhttps://install.python-poetry.org|python3-pipinstall-Upippoetryconfigvirtualenvs.in-projecttruepoetryinstallpytestYou should expect the following output after running the unit tests:$ pytest
============================= test session starts ==============================
platform linux -- Python 3.9.10, pytest-7.0.1, pluggy-1.0.0
rootdir: /home/user/lava-dl, configfile: pyproject.toml, testpaths: tests
plugins: cov-3.0.0
collected 86 items

tests/lava/lib/dl/netx/test_blocks.py ...                                [  3%]
tests/lava/lib/dl/netx/test_hdf5.py ...                                  [  6%]
tests/lava/lib/dl/slayer/neuron/test_adrf.py .......                     [ 15%]

...... pytest output ...

tests/lava/lib/dl/slayer/neuron/dynamics/test_adaptive_threshold.py .... [ 80%]
.                                                                        [ 81%]
tests/lava/lib/dl/slayer/neuron/dynamics/test_leaky_integrator.py .....  [ 87%]
tests/lava/lib/dl/slayer/neuron/dynamics/test_resonator.py .....         [ 93%]
tests/lava/lib/dl/slayer/utils/filter/test_conv_filter.py ..             [ 95%]
tests/lava/lib/dl/slayer/utils/time/test_replicate.py .                  [ 96%]
tests/lava/lib/dl/slayer/utils/time/test_shift.py ...                    [100%]

=============================== warnings summary ===============================

...... pytest output ...

src/lava/lib/dl/slayer/utils/time/__init__.py                      4      0   100%
src/lava/lib/dl/slayer/utils/time/replicate.py                     6      0   100%
src/lava/lib/dl/slayer/utils/time/shift.py                        59     16    73%   22-43, 50, 55, 75, 121, 128, 135, 139
src/lava/lib/dl/slayer/utils/utils.py                             13      8    38%   14, 35-45
--------------------------------------------------------------------------------------------
TOTAL                                                           4782   2535    47%

Required test coverage of 45.0% reached. Total coverage: 46.99%
======================= 86 passed, 3 warnings in 46.56s ========================Note:If you see errors regarding*.npfiles or errors similar to "ValueError: Cannot load file containing pickled data when allow_pickle=False" please ensure git-lfs is installed. If you installed git-lfs after cloning the repository please fetch and pull,git lfs fetch --all; git lfs pulland try the tests again.[Alternative] Installing Lava via CondaIf you use the Conda package manager, you can simply install the Lava package
via:condainstalllava-dl-cconda-forgeAlternatively with intel numpy and scipy:condacreate-nlava-dlpython=3.9-cintel
condaactivatelava-dl
condainstall-nlava-dl-cintelnumpyscipy
condainstall-nlava-dl-cconda-forgelava-dl--freeze-installed[Alternative] Installing Lava from BinariesIf you only need the lava-dl package in your python environment, we will publish
Lava releases viaGitHub Releases. Please download
the package and install it.Open a python terminal and run:[Windows/MacOS/Linux]$python3-mvenvpython3_venv
$pipinstall-Upip
$pipinstalllava-dl-0.2.0.tar.gzGetting StartedEnd to end training tutorialsOxford spike train regressionXOR regressionMNIST digit classificationNMNIST digit classificationPilotNet steering angle predictionVideo object detection with sigma-detla YOLOInference on GPUDeep dive training tutorialsDynamics and NeuronsInference tutorialsOxford InferencePilotNet LIF InferencePilotNet LIF BenchmarkingPilotNet SDNN InferencePilotNet SDNN BenchmarkingYOLO-KP SDNN Inferencelava.lib.dl.slayerlava.lib.dl.slayeris an enhanced version ofSLAYER. Most noteworthy enhancements are: support forrecurrent network structures, a wider variety ofneuron modelsandsynaptic connections(a complete list of features ishere). This version of SLAYER is built on top of thePyTorchdeep learning framework, similar to its predecessor. For smooth integration with Lava,lava.lib.dl.slayersupports exporting trained models using the platform independenthdf5 network exchangeformat.In future versions, SLAYER will get completely integrated into Lava to train Lava Processes directly. This will eliminate the need for explicitly exporting and importing the trained networks.Example CodeImport modulesimportlava.lib.dl.slayerasslayerNetwork Description# like any standard pyTorch networkclassNetwork(torch.nn.Module):def__init__(self):...self.blocks=torch.nn.ModuleList([# sequential network blocksslayer.block.sigma_delta.Input(sdnn_params),slayer.block.sigma_delta.Conv(sdnn_params,3,24,3),slayer.block.sigma_delta.Conv(sdnn_params,24,36,3),slayer.block.rf_iz.Conv(rf_params,36,64,3,delay=True),slayer.block.rf_iz.Conv(sdnn_cnn_params,64,64,3,delay=True),slayer.block.rf_iz.Flatten(),slayer.block.alif.Dense(alif_params,64*40,100,delay=True),slayer.block.cuba.Recurrent(cuba_params,100,50),slayer.block.cuba.KWTA(cuba_params,50,50,num_winners=5)])defforward(self,x):forblockinself.blocks:# forward computation is as simple as calling the blocks in a loopx=block(x)returnxdefexport_hdf5(self,filename):# network export to hdf5 formath=h5py.File(filename,'w')layer=h.create_group('layer')fori,binenumerate(self.blocks):b.export_hdf5(layer.create_group(f'{i}'))Trainingnet=Network()assistant=slayer.utils.Assistant(net,error,optimizer,stats)...forepochinrange(epochs):fori,(input,ground_truth)inenumerate(train_loader):output=assistant.train(input,ground_truth)...fori,(input,ground_truth)inenumerate(test_loader):output=assistant.test(input,ground_truth)...Export the networknet.export_hdf5('network.net')lava.lib.dl.bootstrapIn general ANN-SNN conversion methods for rate based SNN result in high latency of the network during inference. This is because the rate interpretation of a spiking neuron using ReLU acitvation unit breaks down for short inference times. As a result, the network requires many time steps per sample to achieve adequate inference results.lava.lib.dl.bootstrapenables rapid training of rate based SNNs by translating them to an equivalent dynamic ANN representation which leads to SNN performance close to the equivalent ANN and low latency inference. More detailshere. It also supportshybrid trainingwith a mixed ANN-SNN network to minimize the ANN to SNN performance gap. This method is independent of the SNN model being used.It has similar API aslava.lib.dl.slayerand supports exporting trained models using the platform independenthdf5 network exchangeformat.Example CodeImport modulesimportlava.lib.dl.bootstrapasbootstrapNetwork Description# like any standard pyTorch networkclassNetwork(torch.nn.Module):def__init__(self):...self.blocks=torch.nn.ModuleList([# sequential network blocksbootstrap.block.cuba.Input(sdnn_params),bootstrap.block.cuba.Conv(sdnn_params,3,24,3),bootstrap.block.cuba.Conv(sdnn_params,24,36,3),bootstrap.block.cuba.Conv(rf_params,36,64,3),bootstrap.block.cuba.Conv(sdnn_cnn_params,64,64,3),bootstrap.block.cuba.Flatten(),bootstrap.block.cuba.Dense(alif_params,64*40,100),bootstrap.block.cuba.Dense(cuba_params,100,10),])defforward(self,x,mode):...forblock,minzip(self.blocks,mode):x=block(x,mode=m)returnxdefexport_hdf5(self,filename):# network export to hdf5 formath=h5py.File(filename,'w')layer=h.create_group('layer')fori,binenumerate(self.blocks):b.export_hdf5(layer.create_group(f'{i}'))Trainingnet=Network()scheduler=bootstrap.routine.Scheduler()...forepochinrange(epochs):fori,(input,ground_truth)inenumerate(train_loader):mode=scheduler.mode(epoch,i,net.training)output=net.forward(input,mode)...loss.backward()fori,(input,ground_truth)inenumerate(test_loader):mode=scheduler.mode(epoch,i,net.training)output=net.forward(input,mode)...Export the networknet.export_hdf5('network.net')lava.lib.dl.netxFor inference using Lava,lava.lib.dl.netxprovides an automated API for loading SLAYER-trained models as Lava Processes, which can be directly run on a desired backend.lava.lib.dl.netximports models saved via SLAYER using the hdf5 network exchange format. The details of hdf5 network description specification can be foundhere.Example CodeImport modulesfromlava.lib.dl.netximporthdf5Load the trained network# Import the model as a Lava Processnet=hdf5.Network(net_config='network.net')Attach Processes for Input-Output interactionfromlava.procimportio# Instantiate the processesdataloader=io.dataloader.SpikeDataloader(dataset=test_set)output_logger=io.sink.RingBuffer(shape=net.out_layer.shape,buffer=num_steps)gt_logger=io.sink.RingBuffer(shape=(1,),buffer=num_steps)# Connect the input to the network:dataloader.ground_truth.connect(gt_logger.a_in)dataloader.s_out.connect(net.in_layer.neuron.a_in)# Connect network-output to the output processnet.out_layer.out.connect(output_logger.a_in)Run the networkfromlava.magmaimportrun_configsasrcfgfromlava.magmaimportrun_conditionsasrcndnet.run(condition=rcnd.RunSteps(total_run_time),run_cfg=rcfg.Loihi1SimCfg())output=output_logger.data.get()gts=gt_logger.data.get()net.stop()