How to Split a Pickled Model File to Bypass Upload Limits on PythonAnywhere | by Jacob Tadesse | Towards Data Science

In my last post, “Building a Convolutional Neural Network to Recognize Shaved vs UnShaved Faces”, I ended the article sharing the method I used to save my final trained model with Pickle.
“Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream (from a binary file or bytes-like object) is converted back into an object hierarchy. — Source code: Lib/pickle.py
As a refresher, here is the line of code to pickle your final model, saving all the weights, without the hassle of saving the structure as .json file, and weights separately as an .h5 file.
import picklepickle.dump(model, open(“Your_Saved_Model.p”, ‘wb’))
Since I had a model, my goal was to build a Flask Application and deploy it for the public to use. I chose PythonAnywhere to host my app because I needed access to the Bash console and also needed to setup a virtual environment, but realized there was a limit on file upload, and my model was three times bigger than allowed.
I did some research and found a very helpful article, in understanding how to split and join the files with Python. Using the os library and the following function, I was able to split the pickled model file into three smaller files:
def split(source, dest_folder, write_size):    # Make a destination folder if it doesn't exist yet    if not os.path.exists(dest_folder):        os.mkdir(dest_folder)    else:        # Otherwise clean out all files in the destination folder        for file in os.listdir(dest_folder):            os.remove(os.path.join(dest_folder, file))    partnum = 0        # Open the source file in binary mode    input_file = open(source, 'rb')    while True:        # Read a portion of the input file        chunk = input_file.read(write_size)                # End the loop if we have hit EOF        if not chunk:            break                # Increment partnum        partnum += 1                # Create a new file name        filename = '/Model_Files/final_model' + str(partnum)                # Create a destination file        dest_file = open(filename, 'wb')                # Write to this portion of the destination file        dest_file.write(chunk)        # Explicitly close         dest_file.close()    # Explicitly close    input_file.close()    # Return the number of files created by the split    return partnum
The function requires the source, write_size, and dest_folder. The source is where you saved the entire model, the write_size is the amount of bytes for each file, and the dest_folder is the destination folder to save each split file.
Here is an example:
split(source='Your_Saved_Model.p', write_size=20000000, dest_folder='/Model_Files/')
Since I now had three acceptable files for upload, I needed to update my flask application to handle joining the files on the server, once the application was started in the browser. Here is a link to my GitHub repo to view the entire code before deploying to PythonAnywhere.
I used the following function to join the files and save the combined model on the applications’ server:
def join(source_dir, dest_file, read_size):    # Create a new destination file    output_file = open(dest_file, 'wb')         # Get a list of the file parts    parts = ['final_model1','final_model2','final_model3']     # Go through each portion one by one    for file in parts:                 # Assemble the full path to the file        path = file                 # Open the part        input_file = open(path, 'rb')                 while True:            # Read all bytes of the part            bytes = input_file.read(read_size)                         # Break out of loop if we are at end of file            if not bytes:                break                             # Write the bytes to the output file            output_file.write(bytes)                     # Close the input file        input_file.close()             # Close the output file    output_file.close()join(source_dir='', dest_file="Combined_Model.p", read_size = 50000000)
This function joins the individual file bytes into one single pickled object. To see how I used jQuery to allow users an interactive way to upload images and make predictions through POST requests to the backend, using the new combined model to serve the predictions. Check out my GitHub repo for app.