Notify with Python. Make life easier with Python built... | by James Briggs | Towards Data Science

Working in Python, I often run data processing, transfer, and model training scripts. Now, with any reasonable degree of complexity and/or big data, this can take some time.
Although often enough we all have some other work to be done whilst waiting for these processing to complete, occasionally, we don’t.
For this purpose, I put together a set of Python scripts built for this exact problem. I use these scripts to send process updates, visualizations and completion notifications to my phone.
So, when we do occasionally have those moments of freedom. We can enjoy them without being worried about model progress.
Okay so the first thing we need to ask is — what do we need to know?
Now of-course, this really depends on the work you are doing. For me I have three main processing tasks that have the potential to take up time:
Model training
Data processing and/or transfer
Financial modelling
For each of these, there are of-course different pieces of information that we need to stay informed about. Let’s take a look at an example of each.
Update every n epochs, must include key metrics. For example, loss and accuracy for training and validation sets.
Notification of completion (of-course). For this I like to include:
prediction outputs, for text generation, the generated text (or a sample of it) — for image generation, a (hopefully) cool visualization.
visualization of key metrics during training (again, loss and accuracy for both training and validation sets)
other, less essential but still useful information such as local model directories, training time, model architecture etc
Let’s take the example of training a neural network to reproduce a given artistic style.
For this, we want to see; generated images from the model, loss and accuracy plots, and current training time, and a model name.
In this scenario, every 100 epochs, an email containing all of the above will be sent. Here is one of those emails:
This one is slightly less glamorous, but in terms of time consumed, is number one by a long-shot.
We will use the example of bulk data upload to SQL Server using Python (for those of us without BULK INSERT).
At the end of the upload script, we include a simple message notifying us of upload completion.
If errors are occasionally thrown, we could also add a try-except clause to catch the error, and add it to a list to include in our update and/or completion email.
In the case of financial modelling, everything I run is actually pretty quick, so I can only provide you with an ‘example’ use-case here.
We will use the example of a cash-flow modelling tool. In-reality, this process take no more than a 10–20 seconds, but for now let’s assume we’re hot-shot Wall Street quants processing a few million (rather than hundred) loans.
With this email, we may want to include a high-level summary of the analysed portfolio. We can randomly select a few loans and visualize key values over the given time period — giving us a small sample to cross-check model performance is as expected.
All of the functionality above filters from a single script called notify.py.
We will use Outlook in our example code. However, translating this to other providers is incredibly easy, which we will also cover quickly at the end.
There are two Python libraries we need here, email and smtplib.
email — For managing email messages. With this we will setup the email message itself, including subject, body, and attachments.
smtplib — Handles the SMTP connection. The simple mail transfer protocol (SMTP) is the protocol used by the majority of email systems, allowing mail to be sent over the internet.’
The message itself is built using a MIMEMultipart object from the email module. We also use three MIME sub-classes, which we attach to the MIMEMultipart object:
MIMEText — This will contain the email ‘payload’, meaning the text within the email body.
MIMEImage — Reasonably easy to guess, this is used to contain images within our email.
MIMEApplication — Used for MIME message application objects. For us, these are file attachments.
In addition to these sub-classes, there are also other parameters, such as the Subject value in MimeMultipart. All of these together gives us the following structure.
Let’s take a look at putting these all together.
This script, for the most part, is reasonably straightforward. At the top, we have our imports — which are the MIME parts we covered before, and Python’s os library.
Following this define a function called message. This allows us to call the function with different parameters and build an email message object with ease. For example, we can write an email with multiple images and attachments like so:
email_msg = message(    text="Model processing complete, please see attached data.",    img=['accuracy.png', 'loss.png'],    attachments=['data_in.csv', 'data_out.csv'])
First we initialize the MIMEMultipart object, assigning it to msg.
We then set the email subject using the 'Subject' key.
The attach method allows us to add different MIME sub-classes to our MIMEMultipart object. With this we can add the email body, using the MIMEText sub-class.
For both images img and attachments attachment, we can pass either nothing, a single file-path, or a list of file-paths.
This is handled by first checking if the parameters are None, if they are, we pass. Otherwise, we check the data-type given, is it is not a list, we make it one — this allows us to use the following for loop to iterate through our items.
At this point we use the MIMEImage and MIMEApplication sub-classes to attach our images and files respectively. For both we use os.basename to retrieve the filename from the given file-path, which we include as the attachment name.
Now that we have built our email message object, we need to send it.
This is where the smtplib module comes in. The code is again, pretty straight-forward, with one exception.
As we are dealing directly with different email provider’s, and their respective servers, we need different SMTP addresses for each. Fortunately, this is really easy to find.
Type “outlook smtp” into Google. Without even clicking on a page, we are given the server address smtp-mail.outlook.com, and port number 587.
We use both of these when initalizing the SMTP object with smtplib.SMTP — near the beginning of the send function.
smtp.ehlo() and smtp.starttls() are both SMTP commands. ehlo (Extended Hello) essentially greets the server. starttls informs the server we will be communicating using an encrypted transport level security (TLS) connection. You can learn more about SMTP commands here.
After this we simply read in our email and password from file, storing both in email and pwd respectively.
We then login to the SMTP server with smtp.login, and send the email with smtp.sendmail.
I always send the notifications to myself, but in the case of automated reporting (or for any other reason), you may want to send the email elsewhere. To do this, change the destination_address: smtp.sendmail(email, destination_address, msg.as_string).
Finally, we terminate the session and close the connection with smtp.quit.
All of this is placed within a try-except statement. In the case of momentary network connection loss, we will be unable to connect to the server. Resulting in a socket.gaierror.
Implementing this try-except statement prevents the program from breaking in the case of a lapse in network connection. How you deal with this may differ, depending on how important it is for the email to be sent.
For me, I use this for ML model training updates and data transfer completion. If an email does not get sent, it doesn’t really matter. So this simple, passive handling of connection loss is suitable.
Now we have written both parts of our code, we can send emails with just:
# build a message objectmsg = message(text="See attached!", img='important.png',              attachment='data.csv')send(msg)  # send the email (defaults to Outlook)
Jason Farlette pointed out that those of you using Gmail may need to ‘allow access for less secure apps’. Steps for doing so can be found in this Stack Overflow question.
That is all for email notification and/or automation using Python. Thanks to the email and smptlib libraries this is an incredibly easy process to setup.
For any processing or training tasks that consume a lot of time, progress updates and notification on completion, is often truly liberating.
I hope this article has been useful to a few of you out there, as always, let me know if you have any questions or suggestions!
Thanks for reading!
I’ve also written other ‘quick fix’ articles for making our lives easier with Python, check out this one for Excel and Python integration: