Package: keras-adamw
Keras AdamWKeras/TF implementation ofAdamW,SGDW,NadamW, andWarm Restarts, based on paperDecoupled Weight Decay Regularization- plusLearning Rate MultipliersFeaturesWeight decay fix: decoupling L2 penalty from gradient.Why use?Weight decay via L2 penalty yieldsworse generalization, due to decay not working properlyWeight decay via L2 penalty leads to ahyperparameter couplingwithlr, complicating searchWarm restarts (WR): cosine annealing learning rate schedule.Why use?Better generalizationandfaster convergencewas shown by authors for various data and model sizesLR multipliers:per-layerlearning rate multipliers.Why use?Pretraining; if adding new layers to pretrained layers, using a globallris prone to overfittingInstallationpip install keras-adamwor clone repositoryUsageIf using tensorflow.keras imports, setimport os; os.environ["TF_KERAS"]='1'.Weight decayAdamW(model=model)Three methods to setweight_decays = {<weight matrix name>:<weight decay value>,}:# 1. AutomaticallyJustpassin`model`(`AdamW(model=model)`),anddecayswillbeautomaticallyextracted.Loss-basedpenalties(l1,l2,l1_l2)willbezeroedbydefault,butcanbekeptvia`zero_penalties=False`(NOTrecommended,seeUseguidelines).# 2. Use keras_adamw.utils.pyDense(..,kernel_regularizer=l2(0))# set weight decays in layers as usual, but to ZEROwd_dict=get_weight_decays(model)# print(wd_dict) to see returned matrix names, note their order# specify values as (l1, l2) tuples, both for l1_l2 decayordered_values=[(0,1e-3),(1e-4,2e-4),..]weight_decays=fill_dict_in_order(wd_dict,ordered_values)# 3. Fill manuallymodel.layers[1].kernel.name# get name of kernel weight matrix of layer indexed 1weight_decays.update({'conv1d_0/kernel:0':(1e-4,0)})# exampleWarm restartsAdamW(.., use_cosine_annealing=True, total_iterations=200)- refer toUse guidelinesbelowLR multipliersAdamW(.., lr_multipliers=lr_multipliers)- to get,{<layer name>:<multiplier value>,}:(a) Name every layer to be modified(recommended), e.g.Dense(.., name='dense_1')- OR(b) Get every layer name, note which to modify:[print(idx,layer.name) for idx,layer in enumerate(model.layers)](a)lr_multipliers = {'conv1d_0':0.1} # target layer by full name- OR(b)lr_multipliers = {'conv1d':0.1}   # target all layers w/ name substring 'conv1d'Exampleimportnumpyasnpfromkeras.layersimportInput,Dense,LSTMfromkeras.modelsimportModelfromkeras.regularizersimportl1,l2,l1_l2fromkeras_adamwimportAdamWipt=Input(shape=(120,4))x=LSTM(60,activation='relu',name='lstm_1',kernel_regularizer=l1(1e-4),recurrent_regularizer=l2(2e-4))(ipt)out=Dense(1,activation='sigmoid',kernel_regularizer=l1_l2(1e-4,2e-4))(x)model=Model(ipt,out)lr_multipliers={'lstm_1':0.5}optimizer=AdamW(lr=1e-4,model=model,lr_multipliers=lr_multipliers,use_cosine_annealing=True,total_iterations=24)model.compile(optimizer,loss='binary_crossentropy')forepochinrange(3):foriterationinrange(24):x=np.random.rand(10,120,4)# dummy datay=np.random.randint(0,2,(10,1))# dummy labelsloss=model.train_on_batch(x,y)print("Iter{}loss:{}".format(iteration+1,"%.3f"%loss))print("EPOCH{}COMPLETED\n".format(epoch+1))(Full example + plot code, and explanation oflr_tvs.lr:example.py)Use guidelinesWeight decaySet L2 penalty to ZEROif regularizing a weight viaweight_decays- else the purpose of the 'fix' is largely defeated, and weights will be over-decayed --My recommendationlambda = lambda_norm * sqrt(1/total_iterations)-->can be changed; the intent is to scale λ todecoupleit from other hyperparams - including (butnot limited to), # of epochs & batch size. --Authors(Appendix, pg.1) (A-1)total_iterations_wd--> set to normalize overall epochs(or other interval!= total_iterations) instead of per-WR when using WR; maysometimesyield better results --My noteWarm restartsDone automatically withautorestart=True, which is the default ifuse_cosine_annealing=True; internally setst_cur=0aftertotal_iterationsiterations.Manually: sett_cur = -1to restart schedule multiplier (seeExample). Can be done at compilation or during training. Non--1is also valid, and will starteta_tat another point on the cosine curve. Details in A-2,3t_curshould be set atiter == total_iterations - 2; explanationhereSettotal_iterationsto the # of expected weight updatesfor the given restart--Authors(A-1,2)eta_min=0, eta_max=1are tunable hyperparameters; e.g., an exponential schedule can be used foreta_max. If unsure, the defaults were shown to work well in the paper. --AuthorsSave/loadoptimizer state; WR relies on using the optimizer's update history for effective transitions --Authors(A-2)# 'total_iterations' general purpose exampledefget_total_iterations(restart_idx,num_epochs,iterations_per_epoch):returnnum_epochs[restart_idx]*iterations_per_epoch[restart_idx]get_total_iterations(0,num_epochs=[1,3,5,8],iterations_per_epoch=[240,120,60,30])Learning rate multipliersBest used for pretrained layers - e.g. greedy layer-wise pretraining, or pretraining a feature extractor to a classifier network. Can be a better alternative to freezing layer weights. --My recommendationIt's often best not to pretrain layers fully (till convergence, or even best obtainable validation score) - as it may inhibit their ability to adapt to newly-added layers.  --My recommendationThe more the layers are pretrained, the lower their fraction of new layers'lrshould be. --My recommendation