Package: ignite-simple
Ignite SimpleThis module provides the necessary functionality for rapidly prototyping
machine learning models on conventional datasets.This also contains ignite_simple.gen_sweep for sweeping across other
parameters. See corresponding readme at ignite_simple/gen_sweep/README.mdUsageYou must explicitly specify the model, training set, validation set, and the
loss. Then a preset is applied for how to tune hyperparameters and the amount
of information to gather and export.Although not required, it is recommended usetorchluentto simplify the model
creation process. This package accepts a single model with tensor outputs or
a tuple of two models where the first is the stripped model (returns only
a single tensor) and the second model has the same underlying parameters but
return values are of the form (tensor, list of tensors), where the list of
tensors contains relevants snapshots of the data as it was transformed by the
network.This package supports repeating trials with the final selected hyperparameters,
either by specifying the number of repeats withtrialsor by simply
settingis_continuationtoTrueand callingtrainwith the same output
directory, or both.API Referenceimporttorchluentimportignite_simpleimporttorchimporttorchvisionimportlogging.configdefmodel():return(torchluent.FluentModule((1,28,28)).wrap(True).conv2d(32,5).maxpool2d(3).operator('LeakyReLU').save_state().flatten().dense(64).operator('Tanh').save_state().dense(10).save_state().build())defdataset():transform=torchvision.transforms.ToTensor()train_set=torchvision.datasets.MNIST('datasets/mnist',download=True,transform=transform)val_set=torchvision.datasets.MNIST('datasets/mnist',train=False,download=True,transform=transform)returntrain_set,val_setloss=torch.nn.CrossEntropyLossaccuracy_style='classification'defmain():# a reasonable logging.conf is in this repository to get you started.# you won't see any stdout without a logging config!logging.config.fileConfig('logging.conf')ignite_simple.train((__name__,'model',tuple(),dict()),(__name__,'dataset',tuple(),dict()),(__name__,'loss',tuple(),dict()),folder='out',hyperparameters='fast',analysis='images',allow_later_analysis_up_to='video',accuracy_style=accuracy_style,trials=1,is_continuation=False,history_folder='history',cores='all')if__name__=='__main__':main()This involves some boilerplate, especially when you want to include optionally
reanalyzing under different settings and configuring the output folder via
command line arguments. Theignite_simple.helpermodule does this for you,
reducing the amount of repeated code and allowing one to train models quickly
and robustly. In the previous example, everything after'accuracy_style'can
be replaced withif__name__=='__main__':ignite_simple.helper.handle(__name__)which will result in the following command-line arguments:usage: helper.py [-h] [--folder FOLDER] [--hparams HPARAMS]
                 [--analysis ANALYSIS] [--analysis_up_to ANALYSIS_UP_TO]
                 [--trials TRIALS] [--not_continuation] [--cores CORES]
                 [--reanalyze] [--module MODULE] [--loggercfg LOGGERCFG]Usepython -m ignite_simple.helper --helpand the module documentation for
details.Continuations and trialsIn the above example, by changingis_continuationtoTrue, the file may be
invoked multiple times. Withis_continuation=True, the hyperparameters will
be reused from the first run and the model and statistics will be saved
alongside (not overwriting) the existing ones. Furthermore, additional plots
(averaged accuracy, averaged loss, etc) will be available. Withis_continuation=Falsethe output folder will be archived and moved into
the history folder with the current timestamp as its name prior to starting
the run.Note that trials is treated as theminimumnumber of trials to perform.
This will attempt to use all available cores (i.e., the number specified in
cores), which may mean multiple trials can be run in parallel without any
significant difference in runtime. This can be suppressed with the parametertrials_strict=True.Validation setsFor automatic dataset splitting into training and validiation, one can useignite_simple.utils.splitas follows:importignite_simple.utilsimporttorch.utils.dataasdatafull:data.Dataset# dataset to splitval_perc:float=0.1# perc in the validation settrain_set,val_set=ignite_simple.utils.split(full,val_perc,filen='mydataset/train_val_split')The split is random and stored in the given file (extensionless is recommended,
in which the appropriate extension will be added). If the file already exists,
this returns the split stored in the specified file. This makes it easier to
verify the training and validation accuracy after the fact and simplifies
comparisons of models on the same dataset.Accuracy styleValid values areclassification,multiclass, andinv-loss. Classification
is for MNIST-style labels (labels are one-hot and the output of the network
is a one-hot encoding of the label). Multi-class is for when the labels are
one-hot encoded class labels extended to potentially multiple ones.inv-lossuses 1/(loss+1) as the performance metric instead of accuracy.Note: both classification and multiclass support more an arbitrary number
of classes of images, but classification says that each image has exactly
one class and multiclass says that each image may have more than one class.
Multiclass uses a >=0.5 thresholding on the output, classification uses
argmax. This does not effect training or model selection, only output.In both cases, the output of the network should be (batch, num labels)
and the targets should be (batch, num labels).Automatic hyperparameter tuningValid presets arefastest,fast,slow, andslowest.The learning rate and batch size are automatically tuned since they can
dramatically effect model performance. The methodology is inspired byCyclical Learning Rates for Training Neural Networks, 2017.For tuning the learning rate, a reasonable cycle size is used and the process
is analagous to that described by the paper.The batch size is found in a similar way - vary the batch size upward linearly
over a few epochs. The range of batch sizes where the accuracy increased is
found, and then batch sizes are randomly drawn from that range and tested for
a few epochs. The batch size with the highest accuracy at the end of the short
test is used.The hyperparameter presets correspond roughly to:
- How many trials to average tests over (i.e., for learning rate it can be
beneficial to find the range via an average of several runs rather than the
very noisy output of a single run)
- How many batch sizes are attempted within the reasonable range
- How many times to repeat the process for checking for batch size and
learning rate interactions.It is intended that the preset 'fast' is used while making rapid adjustments
and then 'slow' used occassionally as a sanity check. Then 'slowest' can be
used if you don't require any additional features and want to use this package
for a final model.Automatic analysisValid presets arenone,text,images,images-min,animations-draft,animations,video-draft, andvideo.  Seeignite_simple.analaramsfor
details.If unsure, chooseimages-minand then upgrade toimagesorvideofor
final analysis or for additional information as necessary.This package is capable of producing some explanation about how the model was
trained and some information about it's solution. The analysis includes:An explanation of the learning rulesAll relevant hyperparameters for the network and brief explanations of eachWhere relevant, how hyperparameters were selectedNetwork representation in 3d principal-component-spaceThe analysis can be provided in text form, all the previous
plus image references, all the previous plus animation references, or all
the previous plus a video guide. The-draftsettings produce lower-quality
(FPS and resolution) versions that are somewhat faster to generate.Analysis can be performed after-the-fact assuming that sufficient data was
collected (which is specified in theallow_later_analysis_up_toparameter).
The following snippet performs video analysis on the first example without
repeating training, assuming its in the same file:defreanalyze():ignite_simple.analyze((__name__,'dataset',tuple(),dict()),(__name__,'loss',tuple(),dict()),folder='out',settings='video',accuracy_style='classification',cores='all')This can be done automatically with the--reanalyzeoption in the helper
module.Note that reanalysis does not reproduce unless it believes the result would be
different from that which exists on the file system. The analysis output is
in theanalysissubdirectory of the output folder, and then in a folder
indexed by the number of models trained in the current hyperparameter settings.
To ensure you get the most up-to-date analysis you can delete the analysis
folder before calling analyze.Implementation detailsThis package trains with SGD without momentum on a linear cyclical learning
rate rule for a fixed number of epochs. The batch size is fixed throughout
training, and the validation dataset is not used during hyperparameter
selection nor model training, however it is measured and reported for analysis.
In the output folder,analysis/html/index.htmlis produced which explains the
details of training to a reasonable degree. Further details can be found by
checking theignite_simple.tunerandignite_simple.model_managerdocumentation and source code.