Package: activations
AReLU: Attention-based-Rectified-Linear-UnitActivation function player with PyTorch on supervised/transfer/meta learning.ContentIntroductionInstallRunExploreMeta Learning1. IntroductionThis repository is the implementation of paperAReLU: Attention-based-Rectified-Linear-Unit.This project is friendly to newcomers of PyTorch.You can design and compare different activation functions under different learning tasks.2. InstallInstall activations as packagepipinstallactivationscondacreate-nAFPpython=3.7-y
condaactivateAFP
pipinstall-rrequirements.txtNOTE: PAU is only CUDA supported. You have to compile it first:pipinstallairspeed==0.5.14cdactivations/pau/cuda
pythonsetup.pyinstallThe code of PAU is directly token fromPAU, if you occur any problems while compiling, please refer to the original repository.If you just want to have a quick start, and do not want to compile with PAU, just comment out the following lines inactivations/__init__.py:try:from.pau.utilsimportPAU__class_dict__["PAU"]=PAUexceptException:raiseNotImplementedError("")3. Runpython-mvisdom.server&# start visdompythonmain.py# run with default parametersClickhereto check your training process.NOTE: The program will download and save dataset under args.data_root automatically.Run with specified parameterspythonmain_mnist.py-husage:main_mnist.py[-h][--batch_sizeBATCH_SIZE][--lrLR][--lr_auxLR_AUX][--epochsEPOCHS][--epochs_auxEPOCHS_AUX][--timesTIMES][--data_rootDATA_ROOT][--dataset{MNIST,SVHN,EMNIST,KMNIST,QMNIST,FashionMNIST}][--dataset_aux{MNIST,SVHN,EMNIST,KMNIST,QMNIST,FashionMNIST}][--num_workersNUM_WORKERS][--net{BaseModel,ConvMNIST,LinearMNIST}][--resumeRESUME][--af{APL,AReLU,GELU,Maxout,Mixture,SLAF,Swish,ReLU,ReLU6,Sigmoid,LeakyReLU,ELU,PReLU,SELU,Tanh,RReLU,CELU,Softplus,PAU,all}][--optim{SGD,Adam}][--cpu][--exname{AFS,TransferLearning}][--silent]ActivationFunctionPlayerwithPyTorch.optionalarguments:-h,--helpshowthishelpmessageandexit--batch_sizeBATCH_SIZEbatchsizefortraining--lrLRlearningrate--lr_auxLR_AUXlearningrateoffinetune.onlyusedwhiletransferlearning.--epochsEPOCHStrainingepochs--epochs_auxEPOCHS_AUXtrainingepochs.onlyusedwhiletransferlearning.--timesTIMESrepeatruningtimes--data_rootDATA_ROOTthepathtodataset--dataset{MNIST,SVHN,EMNIST,KMNIST,QMNIST,FashionMNIST}thedatasettoplaywith.--dataset_aux{MNIST,SVHN,EMNIST,KMNIST,QMNIST,FashionMNIST}thedatasettoplaywith.onlyusedwhiletransferlearning.--num_workersNUM_WORKERSnumberofworkerstoloaddata--net{BaseModel,ConvMNIST,LinearMNIST}networkarchitectureforexperiments.youcanaddnewmodelsin./models.--resumeRESUMEpretrainedpathtoresume--af{APL,AReLU,GELU,Maxout,Mixture,SLAF,Swish,ReLU,ReLU6,Sigmoid,LeakyReLU,ELU,PReLU,SELU,Tanh,RReLU,CELU,Softplus,PAU,all}theactivationfunctionusedinexperiments.youcanspecifyanactivationfunctionbyname,ortrywithallactivationfunctionsby`all`--optim{SGD,Adam}optimizerusedintraining.--cpuwithcudatraining.thiswouldbemuchfaster.--exname{AFS,TransferLearning}experimentnameofvisdom.--silentifTrue,shutdownthevisdomvisualizer.Full Experimentnohup./main_mnist.sh>main_mnist.log&4. ExploreNew activation functionsWrite a python script file underactivations, such asnew_activation_functions.py, where contains the implementation of new activation function.Import new activation functions inactivations/__init__.py.New network structureWrite a python script file undermodels, such asnew_network_structure.py, where contains the definition of new network structure. New defined network structure should be a subclass ofBaseModel, which defined inmodels/models.py.Import new network structure inmodels/__init__.py.NOTE: New activation functions and network sctructures will be automatically added into argparse. So, it is not necessary to modifymain.py.5. Meta LearningSetuppipinstalllearn2learnRunpythonmeta_mnist.py--helpusage:meta_mnist.py[-h][--waysN][--shotsN][-tpsN][-fasN][--iterationsN][--lrLR][--maml-lrLR][--no-cuda][--seedS][--download-locationS][--afs{APL,AReLU,GELU,Maxout,Mixture,SLAF,Swish,ReLU,ReLU6,Sigmoid,LeakyReLU,ELU,PReLU,SELU,Tanh,RReLU,CELU,Softplus,PAU}]Learn2LearnMNISTExampleoptionalarguments:-h,--helpshowthishelpmessageandexit--waysNnumberofways(default:5)--shotsNnumberofshots(default:1)-tpsN,--tasks-per-stepNtasksperstep(default:32)-fasN,--fast-adaption-stepsNstepsperfastadaption(default:5)--iterationsNnumberofiterations(default:1000)--lrLRlearningrate(default:0.005)--maml-lrLRlearningrateforMAML(default:0.01)--no-cudadisablesCUDAtraining--seedSrandomseed(default:1)--download-locationSdownloadlocationfortraindata(default:data--afs{APL,AReLU,GELU,Maxout,Mixture,SLAF,Swish,ReLU,ReLU6,Sigmoid,LeakyReLU,ELU,PReLU,SELU,Tanh,RReLU,CELU,Softplus,PAU}activationfunctionusedtometalearning.Run allnohup ./meta_mnist.sh > meta_mnist.log &CitationIf you use this code, please cite the following paper:@misc{AReLU,
Author = {Dengsheng Chen and Kai Xu},
Title = {AReLU: Attention-based Rectified Linear Unit},
Year = {2020},
Eprint = {arXiv:2006.13858},
}