Package: keras-importance-sampling
This python package provides a library that accelerates the training of
arbitrary neural networks created withKerasusingimportance sampling.# Keras importsfromimportance_sampling.trainingimportImportanceTrainingx_train,y_train,x_val,y_val=load_data()model=create_keras_model()model.compile(optimizer="adam",loss="categorical_crossentropy",metrics=["accuracy"])ImportanceTraining(model).fit(x_train,y_train,batch_size=32,epochs=10,verbose=1,validation_data=(x_val,y_val))model.evaluate(x_val,y_val)Importance sampling for Deep Learning is an active research field and this
library is undergoing development so your mileage may vary.Relevant ResearchOursNot All Samples Are Created Equal: Deep Learning with Importance Sampling [preprint]Biased Importance Sampling for Deep Neural Network Training [preprint]By othersStochastic optimization with importance sampling for regularized loss
minimization [pdf]Variance reduction in SGD by distributed importance sampling [pdf]Dependencies & InstallationNormally if you already have a functional Keras installation you just need topip installkeras-importance-sampling.Keras> 2A Keras backend amongTensorflow,TheanoandCNTKblinkernumpymatplotlib,seaborn,scikit-learnare optional (used by the plot
scripts)DocumentationThe module has a dedicateddocumentation sitebut you can also read thesource codeand theexamplesto get an
idea of how the library should be used and extended.ExamplesIn theexamplesfolder you can find some Keras examples that have been edited
to use importance sampling.Code examplesIn this section we will showcase part of the API that can be used to train
neural networks with importance sampling.# Import what is needed to build the Keras modelfromkerasimportbackendasKfromkeras.layersimportDense,Activation,Flattenfromkeras.modelsimportSequential# Import a toy dataset and the importance trainingfromimportance_sampling.datasetsimportMNISTfromimportance_sampling.trainingimportImportanceTrainingdefcreate_nn():"""Build a simple fully connected NN"""model=Sequential([Flatten(input_shape=(28,28,1)),Dense(40,activation="tanh"),Dense(40,activation="tanh"),Dense(10),Activation("softmax")# Needs to be separate to automatically# get the preactivation outputs])model.compile(optimizer="adam",loss="categorical_crossentropy",metrics=["accuracy"])returnmodelif__name__=="__main__":# Load the datadataset=MNIST()x_train,y_train=dataset.train_data[:]x_test,y_test=dataset.test_data[:]# Create the NN and keep the initial weightsmodel=create_nn()weights=model.get_weights()# Train with uniform samplingK.set_value(model.optimizer.lr,0.01)model.fit(x_train,y_train,batch_size=64,epochs=10,validation_data=(x_test,y_test))# Train with importance samplingmodel.set_weights(weights)K.set_value(model.optimizer.lr,0.01)ImportanceTraining(model).fit(x_train,y_train,batch_size=64,epochs=2,validation_data=(x_test,y_test))Using the scriptThe following terminal commands train a small VGG-like network to ~0.65% error
on MNIST (the numbers are from a CPU).
.. code:$ # Train a small cnn with mnist for 500 mini-batches using importance
$ # sampling with bias to achieve ~ 0.65% error (on the CPU).
$ time ./importance_sampling.py \
>   small_cnn \
>   oracle-gnorm \
>   model \
>   predicted \
>   mnist \
>   /tmp/is \
>   --hyperparams 'batch_size=i128;lr=f0.003;lr_reductions=I10000' \
>   --train_for 500 --validate_every 500
real    1m41.985s
user    8m14.400s
sys     0m35.900s
$
$ # And with uniform sampling to achieve ~ 0.9% error.
$ time ./importance_sampling.py \
>   small_cnn \
>   oracle-loss \
>   uniform \
>   unweighted \
>   mnist \
>   /tmp/uniform \
>   --hyperparams 'batch_size=i128;lr=f0.003;lr_reductions=I10000' \
>   --train_for 3000 --validate_every 3000
real    9m23.971s
user    47m32.600s
sys     3m4.188s