Package: adas
Adas: Adaptive Scheduling of Stochastic GradientsStatusTable of ContentsAdas: Adaptive Scheduling of Stochastic GradientsStatusIntroductionLicenseCiting AdasEmpirical Classification Results on CIFAR10 and CIFAR100QC MetricsRequirementsSoftware/HardwareComputational OverheadInstallationUsageCommon Issues (running list)TODOPytestIntroductionAdasis an adaptive optimizer for scheduling the learning rate in training Convolutional Neural Networks (CNN)Adas exhibits the rapid minimization characteristics that adaptive optimizers likeAdaMare favoured forAdas exhibitsgeneralization(low testing loss) characteristics on par with SGD based optimizers, improving on the poorgeneralizationcharacteristics of adaptive optimizersAdas introduces no computational overhead over adaptive optimizers (seeexperimental results)In addition to optimization, Adas introduces new probing metrics for CNN layer evaulation (quality metrics)This repository contains aPyTorchimplementation of the Adas learning rate scheduler algorithm as well as the Knowledge Gain and Mapping Condition metrics.Visit thepaperbranch to see the paper-related code. You can use that code to replicate experiments from the paper.LicenseAdas is released under the MIT License (refer to theLICENSEfile for more information)PermissionsConditionsLimitationsCommerical useLicense and Copyright NoticeLiabilityDistributionWarrantyModificationPrivate UseCiting Adas@article{hosseini2020adas,
  title={Adas: Adaptive Scheduling of Stochastic Gradients},
  author={Hosseini, Mahdi S and Plataniotis, Konstantinos N},
  journal={arXiv preprint arXiv:2006.06587},
  year={2020}
}Empirical Classification Results onCIFAR10,CIFAR100andTiny-ImageNet-200Figure 1: Training performance using different optimizers across three datasets and two CNNsTable 1: Image classification performance (test accuracy) with fixed budget epoch of ResNet34 trainingQC MetricsPlease refer toQC on Wikifor more information on two metrics of knowledge gain and mapping condition for monitoring training quality of CNNsRequirementsSoftware/HardwareWe usePython 3.7.Please refer toRequirements on Wikifor complete guideline.Computational OverheadAdas introduces no overhead (very minimal) over adaptive optimizers e.g. all mSGD+StepLR, mSGD+Adas, AdaM consume 40~43 sec/epoch to train ResNet34/CIFAR10 using the same PC/GPU platformInstallationYou can install Adas directly from PyPi using `pip install adas', or clone this repository and install from source.You can also download the files insrc/adasinto your local code base and use them directly. Note that you will probably need to modify the imports to be consistent with however you perform imports in your codebase.All source code can be found insrc/adasFor more information, also refer toInstallation on WikiUsageThe use Adas, simply import theAdas(torch.optim.optimier.Optimizer)class and use it as follows:fromadasimportAdasoptimizer=Adas(params=list(model.parameters()),lr:float=???,beta:float=0.8step_size:int=None,gamma:float=1,momentum:float=0,dampening:float=0,weight_decay:float=0,nesterov:bool=False):...forepochinepochs:forbatchintrain_dataset:...loss.backward()optimizer.step()optimizer.epoch_step(epoch)Note,optipmizer.epoch_step()is just to be called at the end of each epoch.Common Issues (running list)None :)TODOAdd medical imaging datasets (e.g. digital pathology, xray, and ct scans)Extension of Adas to Deep Neural NetworksPytestNote the following:Our Pytests write/download data/files etc. to/tmp, so if you don't have a/tmpfolder (i.e. you're on Windows), then correct this if you wish to run the tests yourself