Package: tinytorchtest
Tiny TorchtestA Tiny Test Suite for pytorch based Machine Learning models, inspired bymltest.
Chase Roberts lists out 4 basic tests in hismedium
postabout mltest. tinytorchtest is mostly a pytorch port of mltest (which was
written for tensorflow).Forked fromBrainPughwho
forked the repo fromsuriyadeepan.Notable changes:Support for models to have multiple positional arguments.Support for unsupervised learning.Object orientated implementation.Easily reproducible tests - thanks to the object orientated implementation!Fewer requirements (due to streamlining testing).More comprehensive internal unit tests.This repository is still active. I've created anissueto
double check but it looks like the original maintainer is no longer
actioning pull requests.Installationpipinstall--upgradetinytorchtestUsage# imports for examplesimporttorchimporttorch.nnasnnVariables Changefromtinytorchtestimporttinytorchtestasttt# We'll be using a simple linear modelmodel=nn.Linear(20,2)# For this example, we'll pretend we have a classification problem# and create some random inputs and outputs.inputs=torch.randn(20,20)targets=torch.randint(0,2,(20,)).long()batch=[inputs,targets]# Next we'll need a loss functionloss_fn=nn.functional.cross_entropy()# ... and an optimisation functionoptim=torch.optim.Adam(model.parameters())# Lets set up the test objecttest=ttt.TinyTorchTest(model,loss_fn,optim,batch)# Now we've got our tiny test object, lets run some tests!# What are the variables?print('Our list of parameters',[np[0]fornpinmodel.named_parameters()])# Do they change after a training step?#  Let's run a train step and seetest.test(test_vars_change=True)""" FAILURE """# Let's try to break this, so the test failsparams_to_train=[np[1]fornpinmodel.named_parameters()ifnp[0]isnot'bias']# Run test nowtest.test(test_vars_change=True)# YES! bias did not changeVariables Don't Change# What if bias is not supposed to change, by design?#  Let's test to see if bias remains the same after trainingtest.test(non_train_vars=[('bias',model.bias)])# It does! Good. Now, let's move on.Output Range# NOTE : bias is fixed (not trainable)test.test(output_range=(-2,2),test_output_range=True)# Seems to work...""" FAILURE """#  Let's tweak the model to fail the test.model.bias=nn.Parameter(2+torch.randn(2,))# We'll still use the same loss function, optimiser and batch# from earlier; however this time we've tweaked the bias of the model.# As it's a new model, we'll need a new tiny test object.test=ttt.TinyTorchTest(model,loss_fn,optim,batch)test.test(output_range=(-1,1),test_output_range=True)# As expected, it fails; yay!NaN Tensors""" FAILURE """# Again, keeping everything the same but tweaking the modelmodel.bias=nn.Parameter(float('NaN')*torch.randn(2,))test=ttt.TinyTorchTest(model,loss_fn,optim,batch)test.test(test_nan_vals=True)# This test should fail as we've got 'NaN' values in the outputs.Inf Tensors""" FAILURE """model.bias=nn.Parameter(float('Inf')*torch.randn(2,))test=ttt.TinyTorchTest(model,loss_fn,optim,batch)test.test(test_inf_vals=True)# Again, this will fail as we've now got 'Inf' values in our model outputs.Multi-argument models# Everything we've done works for models with multi-arguments# Let's define a network that takes some input features along# with a 3D spacial coordinate and predicts a single value.# Sure, we could perform the concatenation before we pass# our inputs to the model but let's say that it's much easier to# do it this way. Maybe as you're working tightly with other codes# and you want to match your inputs with the other code.classMutliArgModel(torch.nn.Module):def__init__(self):self.layers=torch.nn.Linear(8,1)deffoward(self,data,x,y,z):inputs=torch.cat((data,x,y,z),dim=1)returnself.layers(nn_input)model=MultiArgModel()# This looks a bit more like a regression problem so we'll redefine our loss# function to be something more appropriate.loss_fn=torch.nn.MSELoss()# We'll stick with the Adam optimiser but for completeness lets redefine it belowoptim=Adam(model.parameters())# We'll also need some new data for this modelinputs=(torch.rand(10,5),# datatorch.rand(10,1),# xtorch.rand(10,1),# ytorch.rand(10,1),# z)outputs=torch.rand(10,1)batch=[inputs,outputs]# Next we initialise our tiny test objecttest=ttt.TinyTorchTest(model,loss_fn,optim,batch)# Now lets run some teststest.test(train_vars=list(model.named_parameters()),test_vars_change=True,test_inf_vals=True,test_nan_vals=True,)# Great! Everything works as before but with models that take multiple inputs.Models with tuple or list outputs# Now what about models that output a tuple or list of tensors?# This could be for something like a variation auto-encoder# or anything where it's more convienient to separate# internal networks.# Lets define a modelclassMultiOutputModel(nn.Module):def__init__(self,in_size,hidden_size,out_size,num_outputs):super().__init__()# This network is common for all predictions.nets=[nn.Linear(in_size,hidden_size)]# These networks operate separately (in parallel)for_inrange(num_outputs):nets.append(nn.Linear(hidden_size,out_size))self.nets=nn.ModuleList(nets)defforward(self,x):# Passes through the first networkx=self.nets[0](x)# Returns a list of the seprate network predictionsreturn[net(x)fornetinself.nets[1:]]# 10 features, 5 hidden nodes, 1 output node, 3 output modelsmodel=MultiOutputModel(10,5,1,3)# Creates a batch with 100 samples.batch=[torch.rand(100,10),torch.rand(100,1)]# Optimiser...optim=torch.optim.Adam([pforpinmodel.parameters()ifp.requires_grad])# Here we'll have to define a custom loss function to deal with the multiple outputs# For now, I'll use something trivial (and quite meaningless).def_loss(outputs,target):loss_list=[torch.abs(output-target)**pforp,outputinenumerate(outputs)]returntorch.mean(torch.tensor(loss_list))# Setup test suitetest=ttt.TinyTorchTest(model,_loss,optim,batch,supervised=True)# Run the tests we want to run!test.test(train_vars=list(model.named_parameters()),test_vars_change=True,test_inf_vals=True,test_nan_vals=True,)# Great! Everything works as before but with model with tuple or list outputs.Unsupervised learning# We've looked a lot at supervised learning examples# but what about unsupervised learning?# Lets define a simple modelmodel=nn.Linear(20,2)# Now our inputs - notice there are no labels so we just have inputs in our batchbatch=torch.randn(20,20)# Here we'll write a very basic loss function that represents a reconstruction loss.# This is actually a mean absolute distance loss function.# This would typically be used for something like an auto-encoder.# The important thing to note is tinytorchtest expects the loss to be loss(outputs, inputs).defloss_fn(output,input):returntorch.mean(torch.abs(output-input))# We set supervised to false, to let the test suite# know that there aren't any targets or correct labels.test=ttt.TinyTorchTest(model,loss_fn,optim,batch,supervised=False)# Now lets run some teststest.test(train_vars=list(model.named_parameters()),test_vars_change=True,test_inf_vals=True,test_nan_vals=True,)# Great! Everything works as before but with unsupervised models.Testing the GPU# Some models really need GPU availability.# We can get our test suite to fail when the GPU isn't available.# Sticking with the unsupervised exampletest=ttt.TinyTorchTest(model,loss_fn,optim,batch,supervised=False)# Now lets make sure the GPU is available.test.test(test_gpu_available=True)# This test will fail if the GPU isn't available. Your CPU can thank you later.# We can also explitly ask that our model and tensors be moved to the GPUtest=ttt.TinyTorchTest(model,loss_fn,optim,batch,supervised=False,device='cuda:0')# Now all future tests will be run on the GPUReproducible tests# When unit testing our models it's good practice to have reproducable results.# For this, we can spefiy a seed when getting our tiny test object.test=ttt.TinyTorchTest(model,loss_fn,optim,batch,seed=42)# This seed will be called before running each test so the results should always be the same# regardless of the order they are called.Debuggingtorchtest\torchtest.py", line 151, in _var_change_helperassert not torch.equal(p0, p1)RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'other'When you are making use of a GPU, you should explicitly specifydevice=cuda:0. By defaultdeviceis set tocpu. Seeissue
#1for more
information.test=ttt.TinyTorchTest(model,loss_fn,optim,batch,device='cuda:0')Citation@misc{abdrysdale2022
  author ={Alex Drysdale},
  title ={tinytorchtest},
  year ={2022},
  publisher ={GitHub},
  journal ={GitHub repository},
  howpublished ={\url{https://github.com/abdrysdale/tinytorchtest}},
  commit ={4c39c52f27aad1fe9bcc7fbb2525fe1292db81b7}}@misc{Ram2019,
  author ={Suriyadeepan Ramamoorthy},
  title ={torchtest},
  year ={2019},
  publisher ={GitHub},
  journal ={GitHub repository},
  howpublished ={\url{https://github.com/suriyadeepan/torchtest}},
  commit ={42ba442e54e5117de80f761a796fba3589f9b223}}