How to reuse your Python models without retraining them | by Tom Waterman | Towards Data Science

Note: In computer science, the process of saving an object to be re-used later is called serialization, while loading a saved object is called deserialization.
As a Data Scientist, it’s likely that you’ve wanted to reuse one of your models without having to re-train it.
Avoiding the need to re-train your models is especially useful because it allows “deploying” the model, ie transferring your trained model to a public server which can be queried for predictions by anyone with an internet connection.
For example, suppose you had created a simple scikit-learn model that you wanted to save and re-use later.
>>> from sklearn.datasets import load_iris>>> from sklearn.tree import DecisionTreeClassifier>>> model = DecisionTreeClassifier()>>> X, y = load_iris(return_X_y=True)>>> clf = model.fit(X,y)
In this case, our trained model is stored in the Python object clf.
To save the clf object to be reused later, we’ll use the built-in pickle library.
>>> import pickle>>> with open('clf.pickle', 'wb') as f:...     pickle.dump(clf, f)
We opened a file named clf.pickle with the "write bytes mode” by specifying wb, and we’ve stored a reference to that file in the variable f.
Then, we used the pickle.dump method to save the object clf in the file clf.pickle. Behind the scenes, Python actually converts the clf object into a series of bytes that it can later convert back into the original clf object.
You can check that the contents of the working directory now contain a file named clf.pickle.
>>> import os>>> os.listdir()['clf.pickle']
If you exit the current Python session by typing exit(), and then start a new Python prompt, you can then reload the clf object to recover the trained model.
>>> import pickle>>> with open('clf.pickle', 'rb') as f:...     clf = pickle.load(f)>>> type(clf)sklearn.tree._classes.DecisionTreeClassifier
Note how we opened the file clf.pickle with the “read bytes” mode by specifying rb, and then stored the reference to that file in the variable f.
Then, the pickle.load was able to read the file clf.pickle and return a new object that is the exact same as the clf object we originally passed into pickle.dump.
Any Python object can be saved to a file using pickle.dump, and the exact same object can be loaded from the file using pickle.load
So we can use the new clf object loaded from the file to make predictions in the exact same way that we could use the original clf object to make predictions.
>>> clf.predict([...     [1,1,0,0],...     [1,0,1,1]... ])[0,0]
Of course, saving a trained model to a file in a local directory means that other people won’t be able to reuse the model. It’s more convenient to save the trained model to a database that other programs can access.
The pickle module allows converting objects to in-memory bytes, which we can then use to save the trained model in a database.
>>> pickled_clf = pickle.dumps(clf)>>> type(pickled_clf)bytes
Note how we used the pickle.dumps method, ie “dump string,” rather than the pickle.dump method.
The dumps method returned a bytes object pickled_clf directly, rather than writing it to a file. (The naming of the dumps method was chosen to be consistent with other serialization libraries, and it does not return a str type object).
We can now save the pickled_clf bytes object to a database directly.
For example, suppose we had an sqlite3 database named db.sqlite3 that contains a table named models with a BLOB (ie bytes) field.
sqlite3 db.sqlite3 "create table models(name TEXT, data BLOB);"
To save the pickled model to that table, we can use the sqlite3 module.
>>> import sqlite3>>> conn = sqlite3.connect('db.sqlite3')>>> query = 'insert into models values (?, ?)'>>> conn.execute(query, ['clf', pickled_clf])>>> conn.commit()
We can then reload the pickled object from the database, and then convert that bytes object back into the original model.
>>> cursor = conn.execute('select * from models')>>> name, pickled_clf = cursor.fetchone()>>> clf = pickle.loads(pickled_clf)
Note how we’ve loaded the classifier using the pickle.loads method. Just as we used pickle.load to recover an object that was saved with pickle.dump, we can use pickle.loads to recover an object that was saved with pickle.dumps.
You can now reuse the model without needing to re-train it.
>>> clf.predict([...     [1,1,0,0],...     [1,0,1,1]... ])[0,0]
Thanks for reading! If you found this tutorial useful, I write about Python and Data Science tricks on Medium, so you can follow me to get more articles like this one.