Package: kormos
KormosThekormospackage provides an interface betweenscipy.optimize.minimizeandKerasfor training models with deterministic minimization algorithms like L-BFGS.It providesKerasusers with:keras.Modelsubclasses that may be optimized without changes in the API—a model may be trained usingeitherthe built-in stochastic mini-batch algorithmsorthe deterministic batch algorithms fromscipy.optimizeOut-of-the-box interoperability with the usualKerasutilities and workflows; e.g.:fit()still returns a the history object with optimization metadata and validation metrics from each iteration of the solver and is usable byKerasTunerSupport for distributed training strategies (at least in principle—this has admittedly not been integration tested)The ability to usesecond orderoptimization methods fromscipyby evaluating Hessian-vector-products if you have a very specific need for this (spoiler: you almost certainly do not)MotivationWhy would anyone want to go full batch in this day and age?Kerasis a powerful tool for developing predictive models and optimizing their parameters in a high-level language.
While its primary use case is large-scale deep learning, Keras’s auto-differentiation utilities (fromTensorflow) that enable rapid prototyping and optimization with gradient-based minimization algorithms are great for other use cases in mathematical modelling and numerical optimization.If you are working with models or datasets in which training data can reasonably fit in memory on a single machine, you may have situations in which deterministic algorithms likeL-BFGSorNewton-CGare complementary or viable alternatives to thestochasticoptimizersavailable inKeras, since:deterministic algorithms don’t require additional hyperparamter tuning to ensure convergence; if you’re just prototyping something small and having trouble tuning learning rates, you may just want to crank out L-BFGS for a few minutes as a sanity check that your model can in fact be optimizedthese algorithms may have faster convergence to accurate solutions of the optimization problem if the dataset is small enough that full batch gradient and Hessian-vector-product computation times aren’t prohibitiveSo TL;DR: because Luddites exist even in the field of numerical optimization.Why The Name Kormos?BecauseKerasis a powerful and useful tool, and is named after the Greek wordκέρας, which meanshorn.Kormosis related toKeras, but it’s not very powerful or useful.
It’s named after the Greek wordκορμός, which meansstump.LicenseThis project is released under the MIT license, and contains adaptations of other codes released under the Apache and MIT licenses.
Please see the header in each source file for the applicable license and copyright notices therein.SetupRequirementsThekormospackage is built for:Python 3+Tensorflow 2+ (and the respectivetensorflow.kerasmodule with it)Scipy 0.1+ (any version really, since thescipy.optimize.minimizesignature is stable)InstallationInstall via the PyPI package [kormos](https://pypi.org/project/kormos/) usingpip:pip3installkormosAlternatively, if you like your releases bloody rare you may install fromgitdirectly:pip3installgit+https://github.com/mbhynes/kormosUsage ExamplesAkormosmodel is drag-and-drop replaceable with anyKerasmodel.
Below we provide some toy code examples, including Collaborative Filtering and MNIST classification examples adapted from theCode Examplessection ofkeras.io.Example: Linear Regression with Sequential APIimportnumpyasnpfromtensorflowimportkerasimportkormosrank=50# Define the model using the keras.model.Model Sequential APImodel=kormos.models.BatchOptimizedSequentialModel()model.add(keras.layers.Dense(units=1,input_shape=(rank,),activation=None,use_bias=False,kernel_regularizer=keras.regularizers.L2(1e-3),kernel_initializer="ones",))loss=keras.losses.MeanSquaredError()model.compile(loss=loss,optimizer='l-bfgs-b',metrics=['mean_absolute_error'])# Generate samples of normally distributed random datanp.random.seed(1)w=np.random.normal(size=rank)X=np.random.normal(size=(1000,rank))y=np.expand_dims(X.dot(w),axis=1)Xval=np.random.normal(size=(1000,rank))yval=np.expand_dims(Xval.dot(w),axis=1)# Fit the modelhistory=model.fit(x=X,y=y,epochs=10,validation_data=(Xval,yval),options={"maxcors":3},# can pass options payload if so desired)best_fit_weights=np.reshape(model.trainable_weights[0].numpy(),(1,-1))assertnp.allclose(best_fit_weights,w,1e-2)We can now inspect the optimization metris traced in thehistoryobject returned fromfit().
The training metrics captured bykormosinclude the:training loss function value (including regularization terms)2-norm of the batch gradientnumber of evaluations of the loss/gradient function (equivalent to anepochfor a stochastic optimizer)number of evaluations of the Hessian-vector-product function, if applicable (equivalent to anepochfor a stochastic optimizer)>>>importpandasaspd;pd.DataFrame(history.history)lossgradfg_evalshessp_evalsval_lossval_mean_absolute_error079.12197217.9462332078.4181217.13786010.1920050.713242300.2321640.34465720.0564290.186013400.0591400.08870030.0473970.042760500.0473480.01553140.0470060.008019600.0470060.00640150.0469910.001854700.0469940.00584660.0469900.000350800.0469920.00567570.0469900.000073900.0469920.00564280.0469900.0000511100.0469920.005642We can now alsorecompilethe model to use a stochastic optimizer; let’s refit the model using ADAM:# Recompile the model to use a different optimizer (this doesn't change its weights)model.compile(loss=model.loss,optimizer='adam',metrics=['mean_absolute_error'])# Reset the weightsmodel.set_weights([np.random.random(size=(rank,1))])# Fit the model using ADAMhistory=model.fit(x=X,y=y,epochs=150,validation_data=(Xval,yval),)This is a somewhat contrived example in modern machine learning (small dataset and simple model with very few parameters), but it’s the kind of classical use case in which a deterministic algorithm will converge faster than a stochastic algorithm. If you were interested inKerasprimarily for the nicetensorflowAPI and autodifferentiation routines, but had unsexy, non-deep modelling goals, this bud’s for you:>>>importpandasaspd;pd.DataFrame(history.history)lossmean_absolute_errorval_lossval_mean_absolute_error059.7513696.21811152.5185665.756832150.0428125.68821845.3445895.346300243.6741565.30886940.3688325.043641339.0742805.02130436.4925274.795147435.3899124.78166633.4237104.588754..............1450.0470310.0089660.0470310.0090471460.0470230.0086060.0470250.0087181470.0470170.0082680.0470190.0083441480.0470120.0079340.0470130.0079771490.0470080.0076550.0470090.007717[150rowsx4columns]Example: Linear Regression using the Functional APIThe same linear regression model as above may be expressed equivalently by the functional API.
Here we specify a differentscipysolver, the Newton-CG algorithm that uses Hessian-vector-products:# Define the model using the keras.model.Model functional APImodel_input=keras.Input(shape=(rank,),name="input")model_output=keras.layers.Dense(units=1,input_shape=(rank,),activation=None,use_bias=False,kernel_regularizer=keras.regularizers.L2(1e-3),kernel_initializer="ones",)(model_input)model=kormos.models.BatchOptimizedModel(inputs=model_input,outputs=model_output,)loss=keras.losses.MeanSquaredError()model.compile(loss=loss,optimizer='newton-cg',metrics=['mean_absolute_error'])# Fit the model on the same data as previouslyhistory=model.fit(x=X,y=y,epochs=10,validation_data=(Xval,yval),)best_fit_weights=np.reshape(model.trainable_weights[0].numpy(),(1,-1))assertnp.allclose(best_fit_weights,w,1e-2)The Newton-CG algorithm has second order convergence, so we should find that the gradient norm has decreased by several orders of magnitude more than with the L-BFGS-B algorithm.
(Of course, practically speaking this is a moot point in the world of approximate parameter estimation due to the limitations of both imperfect models and sampling bias that exists in training datasets: the numerical error in the solution is orders of magnitude smaller than other errors…)Example: Collaborative Filtering for Item RecommendationWe present a simple linear matrix factorization model for building a recommender system using the MovieLens dataset, and use the same preprocessing steps as in theKerasexample,Collaborative Filtering for Movie Recommendations.Define the ModelWe define a simple matrix factorization model for factorizing the ratings matrix into the product of 2 latent feature matrices, represented byuseranditemembeddings:importtensorflowastffromtensorflowimportkerasimportkormosdefbuild_model(rank,num_users,num_items,**kwargs):inputs=[keras.Input(shape=(1,),name="user",dtype=tf.int32),keras.Input(shape=(1,),name="item",dtype=tf.int32),]user_embedding=keras.layers.Embedding(input_dim=(num_users+1),output_dim=rank,mask_zero=True,embeddings_initializer="normal",embeddings_regularizer=keras.regularizers.L2(1e-5),name="user_embedding",)item_embedding=keras.layers.Embedding(input_dim=(num_items+1),output_dim=rank,mask_zero=True,embeddings_initializer="normal",embeddings_regularizer=keras.regularizers.L2(1e-5),name="item_embedding",)features=[user_embedding(inputs[0]),item_embedding(inputs[1]),]output=keras.layers.Dot(axes=2,normalize=False)(features)model=kormos.models.BatchOptimizedModel(inputs=inputs,outputs=output,**kwargs)returnmodelPrepare the DataWe run the same pre-processing steps as in theKerasexample above.
(Please be aware that there are methodological errors in these steps that we have left unchanged: (1) it is not correct to split the training and testing data uniformly randomly, since some movies have only 1 rating and hence should not be members of the testing set, and (2) it is not possible to construct a factorization model that represents each user/item by a vector of rankkifkisgreaterthan the number of observations (ratings) that that user/item has in the training data—such a system isoverdetermined).importpandasaspdimportnumpyasnpfromzipfileimportZipFileimporttensorflowastffromtensorflowimportkerasfromtensorflow.kerasimportlayersfrompathlibimportPath# Download the data from http://files.grouplens.org/datasets/movielens/ml-latest-small.zip"# Use the ratings.csv filemovielens_data_file_url=("http://files.grouplens.org/datasets/movielens/ml-latest-small.zip")movielens_zipped_file=keras.utils.get_file("ml-latest-small.zip",movielens_data_file_url,extract=False)keras_datasets_path=Path(movielens_zipped_file).parents[0]movielens_dir=keras_datasets_path/"ml-latest-small"# Only extract the data the first time the script is run.ifnotmovielens_dir.exists():withZipFile(movielens_zipped_file,"r")aszip:# Extract filesprint("Extracting all the files now...")zip.extractall(path=keras_datasets_path)print("Done!")ratings_file=movielens_dir/"ratings.csv"df=pd.read_csv(ratings_file)user_ids=df["userId"].unique().tolist()user2user_encoded={x:ifori,xinenumerate(user_ids)}userencoded2user={i:xfori,xinenumerate(user_ids)}movie_ids=df["movieId"].unique().tolist()movie2movie_encoded={x:ifori,xinenumerate(movie_ids)}movie_encoded2movie={i:xfori,xinenumerate(movie_ids)}df["user"]=df["userId"].map(user2user_encoded)df["movie"]=df["movieId"].map(movie2movie_encoded)num_users=len(user2user_encoded)num_movies=len(movie_encoded2movie)df["rating"]=df["rating"].values.astype(np.float32)# min and max ratings will be used to normalize the ratings latermin_rating=min(df["rating"])max_rating=max(df["rating"])print("Number of users:{}, Number of Movies:{}, Min rating:{}, Max rating:{}".format(num_users,num_movies,min_rating,max_rating))df=df.sample(frac=1,random_state=42)x=df[["user","movie"]].values# Normalize the targets between 0 and 1. Makes it easy to train.y=df["rating"].apply(lambdax:(x-min_rating)/(max_rating-min_rating)).values# Assuming training on 90% of the data and validating on 10%.train_indices=int(0.9*df.shape[0])x_train,x_val,y_train,y_val=(x[:train_indices],x[train_indices:],y[:train_indices],y[train_indices:],)Train the ModelWe may now train our factorization model:rank=5model=build_model(rank,num_users,num_movies)model.compile(loss=tf.keras.losses.MeanSquaredError(),optimizer="l-bfgs-b",)history=model.fit(x=(x_train[:,0],x_train[:,1]),y=y_train,batch_size=2**14,epochs=10,verbose=1validation_data=((x_val[:,0],x_val[:,1]),y_val),)>>>importpandasaspd;pd.DataFrame(history.history)lossgradfg_evalshessp_evalsval_loss00.4994310.001055200.49742410.4920910.010318500.49674920.4910670.015367700.49912730.4611400.012731900.47277240.2710200.0175151200.32717350.2286580.0215851400.29812060.1564810.0126981600.22634970.1253500.0078331700.19314580.1014110.0079571800.16951390.0933750.0132331900.162208100.0828760.0053072000.152423110.0777890.0047172100.149731120.0728670.0044202200.144979130.0669270.0064632300.137852140.0638500.0049832400.136306150.0618970.0023532500.133633160.0605140.0018672600.132471170.0586290.0022112700.131402180.0574080.0037102800.130704190.0561110.0014842900.129850Example: MNIST convnetAs a more realistic example of usingkormoson a canonical dataset, we adapt the sample classification problem from theMNIST convnetexample.
Please note that this convolutional network model has a large number of highly correlated parameters to optimize, and stochastic algorithms like ADAM will generally perform better and provide better results.
However we provide it as an example of how both stochastic and deterministic algorithms may be combined byrecompilingakormosmodel.Prepare the Dataimportnumpyasnpfromtensorflowimportkerasfromkerasimportlayers# Model / data parametersnum_classes=10input_shape=(28,28,1)# Load the data and split it between train and test sets(x_train,y_train),(x_test,y_test)=keras.datasets.mnist.load_data()# Scale images to the [0, 1] rangex_train=x_train.astype("float32")/255x_test=x_test.astype("float32")/255# Make sure images have shape (28, 28, 1)x_train=np.expand_dims(x_train,-1)x_test=np.expand_dims(x_test,-1)print("x_train shape:",x_train.shape)print(x_train.shape[0],"train samples")print(x_test.shape[0],"test samples")# convert class vectors to binary class matricesy_train=keras.utils.to_categorical(y_train,num_classes)y_test=keras.utils.to_categorical(y_test,num_classes)Build the Modelfromkormos.modelsimportBatchOptimizedSequentialModeldefbuild_model():model=BatchOptimizedSequentialModel([keras.Input(shape=input_shape),layers.Conv2D(32,kernel_size=(3,3),activation="relu"),layers.MaxPooling2D(pool_size=(2,2)),layers.Conv2D(64,kernel_size=(3,3),activation="relu"),layers.MaxPooling2D(pool_size=(2,2)),layers.Flatten(),layers.Dropout(0.5),layers.Dense(num_classes,activation="softmax"),])returnmodelmodel=build_model()model.summary()Model: "batch_optimized_sequential_model"
_________________________________________________________________
 Layer (type)                   Output Shape              Param #
=================================================================
 conv2d (Conv2D)                (None, 26, 26, 32)        320

 max_pooling2d (MaxPooling2D)   (None, 13, 13, 32)        0

 conv2d_1 (Conv2D)              (None, 11, 11, 64)        18496

 max_pooling2d_1 (MaxPooling2D) (None, 5, 5, 64)          0

 flatten (Flatten)              (None, 1600)              0

 dropout (Dropout)              (None, 1600)              0

 dense (Dense)                  (None, 10)                16010

=================================================================
Total params: 34,826
Trainable params: 34,826
Non-trainable params: 0
_________________________________________________________________Train the ModelWe use this example train the model by running a combination of different algorithms.
We start by running ADAM for 1 epoch, and then using this solution as a warm start initial guess for a batch solver byrecompilingthe model:loss=keras.losses.CategoricalCrossentropy()# Train a model with ADAMmodel=build_model()model.compile(loss=loss,optimizer="adam",metrics=["accuracy"])hist1=model.fit(x_train,y_train,batch_size=2**5,epochs=1,validation_data=(x_test,y_test))# Continue training the model with a batch algorithm.# We can instantiate the optimizer as well instead of a string identifieroptimizer=kormos.optimizers.ScipyBatchOptimizer()model.compile(loss=loss,optimizer=optimizer,metrics=["accuracy"])# We can specify the method and any options for it in fit as keyword wargshist2=model.fit(x_train,y_train,batch_size=2**14,# this is much larger than for stochastic solvers!epochs=3,validation_data=(x_test,y_test),method='bfgs',)Implementation DetailsThekormospackage implements an interface for batch optimization and wrapsscipy.optimize.minimizein that interface in the following steps:We create a subclass ofkeras.Model,BatchOptimizedModel(andBatchOptimizedSequentialModelto extend theSequentialAPI).The subclass provides afit_batch()method with nearly identical signature to the parentfit(), but does not perform stochastic mini-batch optimization. Instead, this method offloads all optimization to the the model’soptimizerattribute, which must implement the methodminimize()to perform training by minimizing the the loss function provided during model compilation.When aBatchOptimizedModelis compiled with aBatchOptimzer(or string identifier for one) as itsoptimizerargument, thefit()method inherited fromkeras.Modelis overriden with a pointer tofit_batch()(such that aBatchOptimizedModelmay be trained with either stochastic or deterministic solvers, depending on how it’s compiled).TheScipyBatchOptimizerclass extends theBatchOptimizerinterface and uses thescipy.optimize.minimizeroutine to fit the model.At first face this is more complicated than therecommendedway of extendingKerasto perform custom training (i.e. by overridingkeras.Model.train_step()such as in the articleCustomizing what happens in fit()).
However, unfortunately we found extendingtrain_step()to be awkward or infeasible for implementing a batch optimization algorithm while still making use of the standardKerasutilities for computingvalidation metricsat each iteration end (epoch).
Overriding the modeltrain_step()(and putting the call toscipy.optimize.minimizeinside it) would mean that from theKerasmodel’s perspective only a singleepochwould be performed, such that validation metrics would only be computed at the very end of the optimzation routine.Acknowledgements & Related WorkThis package has adapted code from the following sources:Pi-Yueh Chuang’sMIT-licensedscipy.optimize.minimize_lbfgswrapper ongithub here.Allen Lavoie’sHessian-vector-product routines fromtensorflow, available in thefollowing commitunder the Apache License version 2.There is also a related projectkeras-optwith the same goal but different implementation and API.
Thekormospackage is recommended overkeras-optbecause its implementation is faster and more robust when training models with large memory requirements, it exposes all of the arguments toscipy.optimize.minimizeif you wish to solve a constrained optimization problem, and is a little bit more seemless to use as part of the nativeKerasworkflow.