Package: torchtraining-nightly
So you want to train neural nets with PyTorch? Here are your options:plain PyTorch- a lot of tedious work like writingmetricsorforloopsexternal frameworks- more automated in exchange for less freedom,less flexibility,lots of esoteric functionsandstuff under the hoodEntertorchtraining- we try to get what's best from both worlds while adding:
explicitness, functional approach, easy extensions and freedom to structure your code!All of that using single**piping operator!VersionDocsTestsCoverageStylePyPIPythonPyTorchDockerLOCTutorialsSee tutorials to get a grasp of what's the fuss is all about:Introduction- quick tour around functionalities with CIFAR100 classification
andtensorboard.GAN training- more advanced example and creating you own pipeline components.InstallationSeedocumentationfor full list of extras (e.g. installation with integrations likehorovod).To just start you can install viapip:pipinstall--usertorchtrainingWhytorchtraining?There are a lot of training libraries around for a lot of frameworks. Why would
you choose this one?torchtrainingfits you, not the other way aroundWe think it's impossible to squeeze user's code in an overly strict API.
Weare nottrying tofiteverything into a single....fit()method (orTrainergod class,
see40!arguments in PyTorch-Lightning trainer).
This approach has shown time and time again it does not work for more complicated
use cases as one cannot foresee the endless possibilities
of training neural network and data generation user might require.torchtraininggives you building blocks to calculate metrics, log results,
distribute training instead.Implement singleforwardinstead of 40 methodsImplementingforwardwithdataargument isallyou willeverneed (okay,accumulatorsalso needcalculate,
but that's it), we add thin__call__.
Compare that toPyTorch-Lightning'sLightningModule(source codehere)training_steptraining_step_endtraining_epoch_end(repeat all the above forvalidationandtest)validation_end,test_endconfigure_sync_batchnormconfigure_ddpinit_ddp_connectionconfigure_apexconfigure_optimizersoptimizer_stepoptimizer_zero_gradtbptt_split_batch(?)prepare_datatrain_dataloadertng_dataloadertest_dataloaderval_dataloaderThis list could go on (and probably will grow even bigger as time passes).
We believe in functional approach and using only what you need (a lot of decoupled building blocks instead
of gigantic god classestryingto do everything). Once again: we can't foresee
future andwon'tsquash everything into singleclass.ExplicitnessYou are offered building blocks and it's up to you what you want to use.
Still, you are explicit abouteverythinggoing on in your code, for example:when, where and what to log totensorboardwhen and how often to run optimizationwhatneural network(s)go into what stepwhat data you choose to accumulate and how oftenwhich component of your pipeline should log vialoguruand how to log (e.g. tostdoutandfileor maybe over the web?)See introduction tutorial to see how it's doneNeural network != trainingWe don't think your neural network source code should be polluted with training.
We think it's better to havedatapreparation indata.pymodule,optimizersinoptimizers.pyand so on. Withtorchtrainingyou don't have to
crunch any functionalities into single godclass.Nothing under the hood (almost)~3000lines of code (includingcomet-ml,neptuneandhorovodintegration)
and short functions/classes allow you to quickly dig
into the source if you find something odd/not working. It's leverages what exists
instead of reinventing the wheel.PyTorch firstWe don't force you to jump into and fromnumpyas most of the tasks can already be
done inPyTorch. We arepytorchfirst.
Unless we have to integrate third party tool... In that caseyou don't pay for
this feature if you don't use it!Easy integration with other toolsIf we don't provide an integration out of the box, you can request it viaissuesor make your ownPR. Any code you want can almost always be integrated via following steps:make a new module (sayamazing.py)create new classes inheriting fromtorchtraining.Operationimplementforwardfor each operation which takes single argumentdatawhich can be anything (Tuple,List,torch.Tensor,str, whatever really)process this data inforwardand return resultsyou have your own operator compatible with**!Other tools integrate components by trying to squash them into their predefined APIs
and/or trying to be smart and guess what the user does (which often fails).
Here's how we do:Example of integration ofneptuneimage logging:importtorchtrainingasttclassImage(tt.Operation):def__init__(self,experiment,log_name:str,image_name:str=None,description:str=None,timestamp=None,experiment=None,):super().__init__()self.experiment=experimentself.log_name=log_nameself.image_name=image_nameself.description=descriptionself.timestamp=timestamp# Always forward some data so it can be reuseddefforward(self,data):self.experiment.log_image(self.log_name,data,self.image_name,self.description,self.timestamp)returndataContributingThis project is currently in it's infancy andwe would love to get some help from you!You can find current ideas insideissuestagged by[DISCUSSION](seehere).accelerators.pymodule for distributed trainingcallbacks.pythird party integrations (experiment handlers likecomet-mlorneptune)Also feel free to make your own feature requests and give us your thoughts inissues!Remember: It's only0.0.1version, direction is there but you can be sure
to encounter a lot of bugs along the way at the momentWhy**as an operator?Indeed, operators like|,>>or>would be way more intuitive,but:Those are left associative and would require users to explicitly uses
parentheses around pipes>cannot be piped as easilyWay morecomplicated code on our side to handle>>or|Currently**seems like a reasonable trade-off, still it may be subject to
change in future.