Package: torchfunc-nightly
Improve and analyse performance of your neural network (e.g. Tensor Cores compatibility)Record/analyse internal state oftorch.nn.Moduleas data passes through itDo the above based on external conditions (using singleCallableto specify it)Day-to-day neural network related duties (model size, seeding, time measurements etc.)Get information about your host operating system,torch.nn.Moduledevice, CUDA
capabilities etc.VersionDocsTestsCoverageStylePyPIPythonPyTorchDockerRoadmap:bulb: ExamplesCheck documentation here:https://szymonmaszke.github.io/torchfunc1. Getting performance tipsGet instant performance tips about your module. All problems described by comments
will be shown bytorchfunc.performance.tips:classModel(torch.nn.Module):def__init__(self):super().__init__()self.convolution=torch.nn.Sequential(torch.nn.Conv2d(1,32,3),torch.nn.ReLU(inplace=True),# Inplace may harm kernel fusiontorch.nn.Conv2d(32,128,3,groups=32),# Depthwise is slower in PyTorchtorch.nn.ReLU(inplace=True),# Same as beforetorch.nn.Conv2d(128,250,3),# Wrong output size for TensorCores)self.classifier=torch.nn.Sequential(torch.nn.Linear(250,64),# Wrong input size for TensorCorestorch.nn.ReLU(),# Fine, no info about this layertorch.nn.Linear(64,10),# Wrong output size for TensorCores)defforward(self,inputs):convolved=torch.nn.AdaptiveAvgPool2d(1)(self.convolution(inputs)).flatten()returnself.classifier(convolved)# All you have to doprint(torchfunc.performance.tips(Model()))2. Seeding, weight freezing and othersSeed globaly (includingnumpyandcuda), freeze weights, check inference time and model size:# Inb4 MNIST, you can use any module with those functionsmodel=torch.nn.Linear(784,10)torchfunc.seed(0)frozen=torchfunc.module.freeze(model,bias=False)withtorchfunc.Timer()astimer:frozen(torch.randn(32,784)print(timer.checkpoint())# Time since the beginningfrozen(torch.randn(128,784)print(timer.checkpoint())# Since last checkpointprint(f"Overall time{timer}; Model size:{torchfunc.sizeof(frozen)}")3. Recordtorch.nn.Moduleinternal stateRecord and sum per-layer activation statistics as data passes through network:# Still MNIST but any module can be put in it's placemodel=torch.nn.Sequential(torch.nn.Linear(784,100),torch.nn.ReLU(),torch.nn.Linear(100,50),torch.nn.ReLU(),torch.nn.Linear(50,10),)# Recorder which sums all inputs to layersrecorder=torchfunc.hooks.recorders.ForwardPre(reduction=lambdax,y:x+y)# Record only for torch.nn.Linearrecorder.children(model,types=(torch.nn.Linear,))# Train your network normally (or pass data through it)...# Activations of all neurons of first layer!print(recorder[1])# You can also post-process this data easily with applyFor other examples (and how to use condition), seedocumentation:wrench: Installation:snake:pipLatest release:pipinstall--usertorchfuncNightly:pipinstall--usertorchfunc-nightly:whale2:DockerCPU standaloneand various versions ofGPU enabledimages are available
atdockerhub.For CPU quickstart, issue:dockerpullszymonmaszke/torchfunc:18.04Nightly builds are also available, just prefix tag withnightly_. If you are going forGPUimage make sure you havenvidia/dockerinstalled and it's runtime set.:question: ContributingIf you find any issue or you think some functionality may be useful to others and fits this library, pleaseopen new Issueorcreate Pull Request.To get an overview of things one can do to help this project, seeRoadmap.