Package: torch-reproducible-block
Torch-reproducible-blockRandom number generation is hard to control when experimenting with neural networks. Setting a random seed only get us so far. Each random operation affects the random number generator state.Changes to the model hyper-parameters or architecture can affect how each layer is initialised, the regularisation techniques, how the data is presented to the network during training and more.This package aims to reducevariabilityby limitingside effectscaused byrandom number generation. The main goal is tolimit the changesto the rest of the network whenexperimentingwith different hyper-parameters.What is the problem ?The weight initialisation of a layer constitute a random operation. The initialisation order therefore have an impact on subsequent layers.In this small toy model, theinitial weightsof the fully connected layers will bedifferentif we have a different number of convolutive layers. The initialisation of apre-trained feature extractormight also comprises random operation which willaffect the rest of the network. A different random state will also affect thedataloadingprocess since it also rely on random operations to select random examples when creating batches.SolutionWe isolate different parts of the network by wrapping them inside aReproducible_BlockHow does it work ?Reproducible_Block.set_seed(SEED_VALUE)must be calledbeforeany random operation. This will set thepython,numpyandtorchseeds and will save a copy of theinitial random number generator state.When entering aReproducible_Blockthe random number generator state is reset to theinitial state. The state is then mutated according to theBlock Seedvalue ensuring that each block have a different state. To mutate the state, we simply runXrandom operations whereXisBlock Seed.Feel free to take at look at the code, it's only about 100 lines.InstallationThe package can be installed via pip :pipinstalltorch-reproducible-blockThe following packages are required :torch
numpyUsagefromreproducible_blockimportReproducible_BlockclassMy_model(nn.Module):def__init__(self):withReproducible_Block(block_seed=64):self.conv1=nn.Conv2d(in_channels=3,out_channels=32,kernel_size=[2,2])self.conv2=nn.Conv2d(in_channels=32,out_channels=64,kernel_size=[2,2])withReproducible_Block(block_seed=44):self.fc=nn.Linear(64,128)self.out=nn.Linear(128,10)defforward(self,batch):...if__name__=="__main__":# Will set seed and save the initial random stateReproducible_Block.set_seed(42)model=My_model()# Data loading and other configurations which might do random operations....# Ensure that we always have the same random state when starting trainingwithReproducible_Block(block_seed=128):train_model(model,data,...)Reproducible block can also be used as a function decorator  :@Reproducible_Block(block_seed=128)deftrain_model(model,dataloader,...):...RemarksUsing a differentinitial seed(ViaReproducible_Block.set_seed()), will result in a different random state for eachReproducible_Block.Theblock seedis "part of the code". You should not attempt to tweak it the way we do with "normal seed". Changing theinitial seedis what you want to do in order to create different initial conditions for your training.Using the sameblock seedfor differentReproducible_Blockwill result in the same random state.
Make sure that you are using a differentblock seedfor each block.Was tested onUbuntu 18.04withpython 3.6. Should not have any problems running on other platforms. Fill up an issue if you have any problems.Other sources of randomnessThis package won't make your research 100% reproducible. It simply aim to isolate part of your program to side effects.Setting thePYTHONHASHSEEDenvironment variable is always a good idea.The way python read files from directories (ex :os.listdir())  can vary when ran on different OS (Linux vs Windows)Different version ofpython,Cuda,Cudnnand libraries can affect reproducibilityContributingThe code is pretty simple, have a look at it and if you have improvements ideas feel free to submit a pull request !