Package: brc-pytorch
brc_pytorchPytorch implementation of bistable recurrent cell with baseline comparisons.This repository contains the Pytorch implementation of the paper"A bio-inspired bistable recurrent cell allows for long-lasting memory". The originaltensorflowimplementation by the author Nicolas Vecoven can be foundhere.Another important feature of this repository is the implementation of a base class that returns a recurrent neural network for a given recurrent cell. Based on the hyperparameters provided, the network can have multiple layers, be bidirectional and the input can either have batch first or not. The outputs from the network mimic that returned by GRU/LSTM networks developed by PyTorch, with an additional option of returning only the hidden states from the last layer and last time step.Package setupbrc_pytorchispypiinstallable:pipinstallbrc_pytorchDevelopment setupCreate avenv, activate it, install dependencies and package in editable mode.python-mvenvvenvsourcevenv/bin/activate
pipinstall-rrequirements.txt
pipinstall-e.Usage (example)frombrc_pytorch.layersimportBistableRecurrentCell,NeuromodulatedBistableRecurrentCellfrombrc_pytorch.layersimportMultiLayerBase# Create a 3-layer nBRC (behaves like a nn.GRU)input_size=32hidden_size=16num_layers=3bidirectional=Truebatch_first=Truereturn_sequences=Falsenum_directions=2ifbidirectionalelse1# Behaves like a nn.GRUCellnbrc=NeuromodulatedBistableRecurrentCell(input_size,hidden_size)# Append cells for subsequent layers keeping in mindfor_inrange(num_layers-1):nbrc.append(NeuromodulatedBistableRecurrentCell(inner_input_dimensions,hidden_size))three_layer_nbrc=rnn=MultiLayerBase("nBRC",nbrc,hidden_size,batch_first,bidirectional,return_sequences,)Validation studiesFirst, the implementations of both the BRC and nBRC are validated on the
Copy-First-Input task (Benchmark 1 from the original paper). Moreover, it is well known
that standard RNNs have difficulties indiscrete counting, especially for
longer sequences (seeNeurIPS 2015 paper).
To this end, we here identifyBinary Additionas another
task for which the nBRC is superior to LSTM & GRU which begs implications for a
set of tasks involving more explicit memorization. For both tasks, the
performances of BRC and nBRC are compared with that of the LSTM and GRU cells.Copy-First-InputThe goal of this task is to correctly predict the number at the start of a sequence of a certain length.This task is reproduced from the paper - 2 layer model with 100 units each, trained on datasets with increasing sequence lengths - 5, 100, 300. The plot is obtained by taking a moving average of the training loss per gradient iteration with window size = 100 for lengths 100 and 300, and window size 20 for length 5.The results from Copy-First-Input task show trends similar to that in the paper, thus confirming their findings. It should, however, be noted that the absolute losses are higher than reported in the paper. This is mostly due to the training and testing sizes being much smaller, and no hyperparameter tuning being done.To reproduce this task do:Change directory to thescriptsfolder. From the terminal, run the following commands:# The following command creates a directory with subdirectories in the scripts folder to save the models and results.mkdir-ptest_benchmark1/{models,results}# Run the training script with your python executable. The following is an example for Anaconda./opt/anaconda3/envs/venv/bin/pythonbrc_benchmark1.pytest_benchmark1/models/test_benchmark1/results/Or, if training takes a very long time, run the script cell-wise, i.e, specify cell name as an additional argument and run multiple jobs in parallell - one for each cell./opt/anaconda3/envs/venv/bin/pythonbrc_benchmark1_cell.pynBRCtest_benchmark1/models/test_benchmark1/results/Calculate the moving average for eachTrainLoss_AllE_*.npyfile from test_benchmark1/results/ and plot.Binary AdditionAdditional testing on Binary Addition was done to test the capabilities of these cells. The goal of this task is to correctly predict the sum of two binary numbers (in integer form).Both single layer and 2 layer models, with constant hidden units 100, are evaluated based on the accuracy of their predictions.The results from this task prove the usefulness of both the nBRC and BRC layers which consistently perform better than both the LSTM and GRU. Moreover, it is interesting to note the potential of nBRC in the binary addition task which is consistent around near perfect accuracy upto sequence length 60. The plots are obtained by averaging the results over 5 runs of the experiment and highlighting the standard error of the average.While the Copy-First-Input task highlights the performance superiority of these cells over the conventional LSTM and GRU, the Binary Addition task, which requires counting, is witness to their usefulness beyond just long-lasting memory.To reproduce this task do:Change directory to thescriptsfolder. From the terminal, run the following command:# The following command creates a directory with subdirectories in the scripts folder to save the models and results.mkdir-ptest_binary_addition/{models,results}/{test1,test2,test3,test4,test5}Create and run the following python script from the same directory. Make sure the python executable file is correct.importosimportsysimportsubprocessdir_models='test_binary_addition/models/'dir_results='test_binary_addition/results/'modelpaths=[os.path.join(dir_models,f'test{i}')foriinrange(1,6)]resultpaths=[os.path.join(dir_results,f'test{i}')foriinrange(1,6)]procs=[]foriinrange(5):proc=subprocess.Popen([sys.executable,'binary_addition_train.py',modelpaths[i],resultpaths[i]])procs.append(proc)forprocinprocs:proc.wait()Calculate the mean and standard error of mean over the different tests for eachtest_acc_*.npyfile and plot.For the 2 layer implementation, simply add another 100 to thehidden_sizesvariable in the training file, and repeat the steps.