Package: torch-lighter
LighterLightweight extension for torch to speed up project prototyping and enable dependency injection of object instances.
This framework includes a setup script to get started with a pre-defined project structure and the documentation offers examples and an overview of the initial usage.
For more examples follow thelink to the GitHub repository.Install$>pipinstalltorch-lighterQuickstartCreate a JSON fileconfig.json:{"num_of_epochs":50}Create a python file and use the decorators to inject the config instance:# creates the application contextContext.create()classDemo:@config(path='config.json',property='config')def__init__(self):passdefrun(self):print(self.config.num_of_epochs)# creates the object instance and injects the configsDemo().run()This will print the following output:[]:50Lighter can be simply added to any existing project or one can create a new lighter template project as shown in the next section.Create a projectInitialize projectType the following in your command line / bash:$>lighter-init<project-name>Structure overviewThe above command will create the following project structure:my_project:
  * configs
    - modules.config.json
  * criterions
    - __init__.py
    - defaults.config.json
    - defaults.py
  * data_builders
    - __init__.py
    - defaults.config.json
    - defaults.py
  * datasets
    - __init__.py
    - defaults.config.json
    - defaults.py
  * experiments
    - __init__.py
    - defaults.config.json
    - defaults.py
  * models
    - __init__.py
    - defaults.config.json
    - defaults.py
  * optimizers
    - __init__.py
    - defaults.config.json
    - defaults.py
  * tests
    - __init__.py
    - test_experiment.py
  * transforms
    - __init__.py
    - defaults.py
  * __init__.py
  * README.md
  * LICENSE
  * setup.pyOf course this is only a preset example and the projects can be structured at will.Modules descriptioncollectible: are used to gather metrics over multiple steps / epochscriterion: torch criterion classesdata_builder: data builder classes creating the data loadersdataset: dataset used by the data builderexperiment: experiment logic files containing the mainrun(),train()andeval()methodsmetric: metrics used to measure performancemodule: references all available modulesoptimizer: optimizers used for performing the steptest: contains the main testing sources for the project and mark the main entrance pointtransform: data transforms are used by the datasets to reshape the data structureswriter: tensorboard writersTODOsEach directory containing a python file may also contain a config file outsourcing properties for testing.
Modules that require further code insertion are marked withTODOflags.Run an experimentAfter creating a demo project and inserting code into theTODOplaceholders simply run an experiment by calling:$>pythontests/test_experiment.pyIt might be necessary to set yourPYTHONPATHbefore running the experiment file:$>PYTHONPATH=.pythontests/test_experiment.pyOne can also simply change the global device setting or initialize an experiment from command line by specifying theses arguments:$>pythontests/test_experiment.py--devicecuda:1--config<path-to-config>AdvancedInheritance from base classOne can simply extend from any base class within the lighter framework:classSimpleExperiment(DefaultExperiment):@config(path='experiments/defaults.config.json',property='experiments')@strategy(config='configs/modules.config.json')@referencesdef__init__(self):super(SimpleExperiment,self).__init__(epochs=self.config.experiments.num_epochs)defeval(self):...deftrain(self):...preds=self.model(inputs)loss=self.metric(preds,targets)...In this example we subclass theDefaultExperimentclass and create a new experiment with some customeval()andtrain()behaviour.
Therun()work-flow is still used as defined in the base classDefaultExperiment.
The@referencesdecorator injects by default some properties into the currentSimpleExperimentobject instance.
If not other specified the@referencetries to inject the properties['dataset', 'data_builder', 'criterion', 'model', 'writer', 'optimizer', 'transfom', 'metric', 'collectible']which in this case are declared in theconfigs/modules.defaults.config.jsonconfig.
Sincemodelandmetricare injected via@referencesthey can be directly accessed via theselfinstance.The@configdecorator injects additional configs for the current experiment instance by referencingdefaults.config.jsonwith anexperimentsalias.
All configs are by default created as a sub-property ofself.configand are now referencable through theexperimentsalias.
One can then access the properties as followsself.config.experiments.<property>.DecoratorsIn general it is also possible to simply use the required decorators without base class sub-classing.There are currently the following decorators available:@config- injects the default config instance and allows to load new configs@context- injects the application context which holds config and registry instances@transform- injects the default data transform instance@dataset- injects the default dataset instance@model- injects the default model instance@metric- injects the default metric instance@reference- injects a singe defined instance@references- injects a set of pre-set or defined instance(s)@strategy- defines and loads a training strategy from a specified config path and injects the default object instances@register- allows to quickly register a new type to the context registry and inject the instance into the current instance@hook- allows to hook and overwrite an existing method to change the execution logic@inject- allows to inject single instances into different context options (registry, types, instances, configs)@search- allows to register hyper-parameters to parse a config schedule for parallelled executionsBy default the templates for a new project are creating a config file in theconfigs/modules.config.jsonpath, which uses the following default names for the dependency injection['dataset', 'data_builder', 'criterion', 'model', 'writer', 'optimizer', 'transfom', 'metric', 'collectible'].
An experiment gets these instances automatically instantiated and injected at runtime.{"strategy":{"transform":"type::transforms.defaults.Transform","dataset":"type::datasets.defaults.Dataset","data_builder":"type::data_builders.defaults.DataBuilder","criterion":"type::criterions.defaults.Criterion","model":"type::models.defaults.Model","optimizer":"type::optimizers.defaults.Optimizer","metric":"type::lighter.metric.BaseMetric","collectible":"type::lighter.collectible.BaseCollectible","writer":"type::lighter.writer.BaseWriter"}}The above example groups the types in thestrategyalias name to avoid config name collisions if other classes import configs using the same general names (model, metric, etc.) across the application.It is also simple to deviate from this naming convention or even inject custom types by exchanging names within the referencing configs:{..."other_model":"type::models.other.Network","other_metric":"type::metrics.other.Metric",...}The defined nameother_model_nameis now uniquely assigned in the instance context and can be accessed via@references(names=['other_model', 'other_metric'])It is also possible to register types on the fly within the code at the class or method level:classDemo:@register(type='envs.defaults.Environment',property='env')@reference(name='env')def__init__(self):...obs=self.env.reset()...Here the@referencedecorator requires a name argument to match the instance.Config templatesAll configs are based onjsonfiles. Yet, when importing the configslighterparses some escape sequences allowing for more modularization of your project structure:config::<config-file-path>- imports another config into the current json config instanceimport::<config-file-path>- imports all properties of the referenced config file into the current config instance (attention may override existing properties)type::<type-file-path>- imports a python type which is registered to the application contextregistry.typesand can be used to instantiate new objectsParameter search parserOne can simply use lighter for searching hyper-parameters or different setting using the@searchdecorator, which registers parameters referenced by configs to the global context.
This generates a permutation of different settings using the parameter parser, which then allows for parallel execution either with the default built in scheduler or any other specialized framework for distributed optimization, such asray.classParameterSearchRegistration:@search(group='sgd',params=[('lr',GridParameter(ref='optimizer.lr',min=0.001,max=0.005,step=0.001)),('weight_decay',ListParameter(ref='optimizer.weight_decay',options=[0.0,0.9])),('freeze_pretrained',BinaryParameter(ref='model.freeze_pretrained')),('hidden_units',GridParameter(ref='model.hidden_units',min=100,max=200,step=100)),('optimizer',SetParameter(ref='strategy.optimizer',option="type::optimizers.defaults.Optimizer")),('model_output',SetParameter(ref='model.output',option=1)),('model_pretrained',SetParameter(ref='model.pretrained',option=True)),('strategy',StrategyParameter(ref='strategy',options=['searches/coco_looking.config.json'])),('model',ListParameter(ref='strategy.model',options=['type::models.alexnet.AlexNetFeatureExtractionModel','type::models.resnet.ResNetFeatureExtractionModel']))])def__init__(self):passHeregroupmarks the grouping alias under which the upcoming parameters are registered in the context.
Theparamsargument takes in a list of two-tuples(<name: str>, <param: Parameter>), wherenamemarks the parameter alias within the group andparamthe type of search parameter used.Currently lighter offers the following parameter types:GridParameter: Search a range of values (min, max, steps) with fixed step sizeListParameter: Commit a list of parameters to lighter which is sequentially executedCallableGridParameter: Search a range of values (min, max, callable) but with a callable lambda function which allows for non-linear step behaviourAnnealParameter: Same asCallableGridParameterbut for annealing a parameterBinaryParameter: ReturnTrueandFalsevalues for a parameterStrategyParameter: Allows to define different training strategies and switch between themSetParameter: Represents a simple setter of properties that is used to register settings, which don't change, but need to be importedTo trigger the parameter config parsing run:fromsearches.defaultsimportParameterSearchRegistrationfromlighter.parserimportConfigParserfromlighter.contextimportContextcontext=Context.create(auto_instantiate_types=False)psr=ParameterSearchRegistration()cp=ConfigParser(experiment=psr)cp.parse()By default this saves all permuted configs to theruns/search/<gen-name>directory.
Theauto_instantiate_typesparameter prevents that the context instantiates the registered types when loading in configs.Parameter search schedulerLighter comes with a built in config scheduler that can execute experiments in parallel for single machine experiments.
For many prototyping cases this is sufficient and helps during development, but naturally flexibility is highest priority and it is possible and recommended to use 3rd party frameworks to schedule decentralized parallel executions on clusters of machines.For the simple case,importtorchfromlighter.schedulerimportScheduler# required to execute torch in parallel processestorch.multiprocessing.set_start_method('spawn',force=True)num_devices=torch.cuda.device_count()# sets the path to the list of configsscheduler=Scheduler(path='runs/search/witty-poodle',experiment='experiments.defaults.Experiment',device_name='cuda',num_workers=num_devices)scheduler.run()The scheduler in the code snipped above starts as many processes as there are GPUs available and schedules the list of configs across these devices.
Since we defined a simple writer, one can simply monitor the progress using tensorboard.