Automatic Email Notifications for HiveQL/SQL Script Completion | by Andrew Young | Towards Data Science

If you find this article helpful in any way, please comment or click the applause button at the left to give me free virtual encouragement!
This bash script is very useful for executing HiveQL files (.hql)or SQL files (.sql). For instance, rather than having to periodically check when a CREATE TABLE or JOINhas finished, this script provides email notifications as well as a record for the clock time needed for the Hive/SQL operation(s) involved. This is especially useful for completing a series of Hive/SQL JOINs overnight for analysis the next day.
A s a practicing data scientist, I dislike overexplained fluff pieces when I’m just looking for code to recycle and modify for my purposes so please find the script below. If you would like more explanation of the script beyond the code comments, please see the appendix at the end. A sufficient explanation of bash , its transformations at the bit-level, HiveQL/SQL and database engines is beyond the scope of this article. As it is written, the script below will only work for HiveQL when copy-pasted. You will have to make slight modifications for it to work with your SQL distribution. For example, changing hive -f to mysql source if you are using MySQL. It is also possible to have the script send emails to more than one email address.
#!/bin/bash## By Andrew Young ## Contact: andrew.wong@team.neustar## Last modified date: 13 Dec 2019################## DIRECTIONS #################### Run the following <commands> without the "<" or ">".## <chmod u+x scriptname> to make the script executable. (where scriptname is the name of your script)## To run this script, use the following command:## ./<yourfilename>.sh################################### Ask user for Hive file name ###################################echo Hello. Please enter the HiveQL filename with the file extension. For example, 'script.hql'. To cancel, press 'Control+C'. For your convenience, here is a list of files in the present directory\:ls -lread -p 'Enter the HiveQL filename: ' HIVE_FILE_NAME#read HIVE_FILE_NAMEecho You specified: $HIVE_FILE_NAMEecho Executing...######################## Define variables ########################start="$(date)"starttime=$(date +%s)####################### Run Hive script #######################hive -f "$HIVE_FILE_NAME"################################### Human readable elapsed time ###################################secs_to_human() {    if [[ -z ${1} || ${1} -lt 60 ]] ;then        min=0 ; secs="${1}"    else        time_mins=$(echo "scale=2; ${1}/60" | bc)        min=$(echo ${time_mins} | cut -d'.' -f1)        secs="0.$(echo ${time_mins} | cut -d'.' -f2)"        secs=$(echo ${secs}*60|bc|awk '{print int($1+0.5)}')    fi    timeelapsed="Clock Time Elapsed : ${min} minutes and ${secs} seconds."}################## Send email ##################end="$(date)"endtime=$(date +%s)secs_to_human $(($(date +%s) - ${starttime}))subject="$HIVE_FILE_NAME started @ $start || finished @ $end"## message="Note that $start and $end use server time. \n $timeelapsed"## working version:mail -s "$subject" youremail@wherever.com <<< "$(printf "Server start time: $start \nServer   end time: $end   \n$timeelapsed")"################### FUTURE WORK #################### 1. Add a diagnostic report. The report will calculate:#    a. number of rows,#    b. number of distinct observations for each column,#    c. an option for a cutoff threshold for observations that ensures the diagnostic report is only produced for tables of a certain size. this can help prevent computationally expensive diagnostics for a large table## 2. Option to save output to a text file for inclusion in email notification.#################
H ere I will explain the code for each section of the script.
################## DIRECTIONS #################### Run the following <commands> without the "<" or ">".## <chmod u+x scriptname> to make the script executable. (where scriptname is the name of your script)## To run this script, use the following command:## ./<yourfilename>.sh
chmod is a bash command that stands for “change mode.” I am using it to change the permissions of the file. It ensures that you, the owner, is allowed to execute your own file on the node/machine you are using.
################################### Ask user for Hive file name ###################################echo Hello. Please enter the HiveQL filename with the file extension. For example, 'script.hql'. To cancel, press 'Control+C'. For your convenience, here is a list of files in the present directory\:ls -lread -p 'Enter the HiveQL filename: ' HIVE_FILE_NAME#read HIVE_FILE_NAMEecho You specified: $HIVE_FILE_NAMEecho Executing...
In this section, I use the command ls -l to list all files in the same directory (i.e. “folder”) as the .sh file you saved my script to. The read -p is used to save the user input, i.e. the name of the .hql or .sql file you want to run, to a variable that is used later on in the script.
######################## Define variables ########################start="$(date)"starttime=$(date +%s)
This is where the system’s clock time is recorded. We save two versions to two different variables: startand starttime.
start="$(date)" saves the system time and date. starttime=$(date +%s) saves the system time and date in number seconds offset from 1900. The first version is later used in the email to show the timestamp and date of when the HiveQL/SQL script began execution. The second is used to calculate the number of seconds and minutes that have elapsed. This provides a handy record for the amount of time it took to build the table(s) in the HiveQL/SQL script you supplied.
################################### Human readable elapsed time ###################################secs_to_human() {    if [[ -z ${1} || ${1} -lt 60 ]] ;then        min=0 ; secs="${1}"    else        time_mins=$(echo "scale=2; ${1}/60" | bc)        min=$(echo ${time_mins} | cut -d'.' -f1)        secs="0.$(echo ${time_mins} | cut -d'.' -f2)"        secs=$(echo ${secs}*60|bc|awk '{print int($1+0.5)}')    fi    timeelapsed="Clock Time Elapsed : ${min} minutes and ${secs} seconds."}
This is a complicated section to decode. Basically, it is a lot of code that does something very simple: finding the difference between start and end time, both in seconds, then converting that elapsed time in seconds to minutes and seconds. For example, if the job took 605 seconds, we transform that to 10 minutes 5 seconds and save it to a variable called timeelapsed to be used in the email we have sent to ourselves and/or various stakeholders.
################## Send email ##################end="$(date)"endtime=$(date +%s)secs_to_human $(($(date +%s) - ${starttime}))subject="$HIVE_FILE_NAME started @ $start || finished @ $end"## message="Note that $start and $end use server time. \n $timeelapsed"## working version:mail -s "$subject" youremail@wherever.com <<< "$(printf "Server start time: $start \nServer   end time: $end   \n$timeelapsed")"
In this section, I again record system time in two different formats as two different variables. I actually only use one of these variables. I decided to keep the unused variable, endtime for potential future extensions to this script. The variable $end is used in the email notification to report the system time and date for when the HiveQL/SQL file completed.
I define a variable subject that, unsurprisingly, becomes the subject line of the email. I commented out the message variable because I couldn’t get the printf command to correctly replace \n with new lines in the message body of the email. I left it in place because I wanted to leave you with the option of editing it for your own purposes.
mail is a program that sends emails. You will probably have it or something like it. There are other options to mail, like mailx, sendmail, smtp-cli, ssmtp, Swaks and a number of others. Some of these programs are subsets or derivatives of each other.
I n the spirit of collaboration and furthering this effort, here are some ideas for future work:
Allow a user to specify the directory of HiveQL/SQL files in which to look for the target file to be run. An example use-case is where a user might have organized their HiveQL/SQL code into multiple directories based on project/time.Improve the formatting of the email notification.Add robustness to the script by automatically detecting whether a HiveQL or SQL script was input, and if the latter, an option to specify the distribution (i.e. OracleDB, MySQL, etc.).Add the option for the user to specify whether the output of the query should be sent to the email(s) specified. Use-case: running a multitude of SELECT statements for exploratory data analysis (EDA). For example, counting number of distinct values of each field of a table, finding distinct levels of a categorical field, finding counts by group, numerical summary statistics like mean, median, standard deviation, etc.Add the option for the user to specify one or more email addresses at the command line.Add an option for the user to specify multiple .hql and/or .sql files to be run in sequence or perhaps in parallel.Add a parallelization option. Use-case: you have a tight deadline and need not concern yourself with resource utilization or co-worker etiquette. I need my results!Add an option to schedule code execution. Likely in the form of a sleep specified by the user. Use-case: you want to be courteous and avoid runs during production code runs and when cluster usage is high during the work day.Make one or more of the options above accessible via flags. This would be really cool, but also a lot of work just to add sprinkles on top.More.
Allow a user to specify the directory of HiveQL/SQL files in which to look for the target file to be run. An example use-case is where a user might have organized their HiveQL/SQL code into multiple directories based on project/time.
Improve the formatting of the email notification.
Add robustness to the script by automatically detecting whether a HiveQL or SQL script was input, and if the latter, an option to specify the distribution (i.e. OracleDB, MySQL, etc.).
Add the option for the user to specify whether the output of the query should be sent to the email(s) specified. Use-case: running a multitude of SELECT statements for exploratory data analysis (EDA). For example, counting number of distinct values of each field of a table, finding distinct levels of a categorical field, finding counts by group, numerical summary statistics like mean, median, standard deviation, etc.
Add the option for the user to specify one or more email addresses at the command line.
Add an option for the user to specify multiple .hql and/or .sql files to be run in sequence or perhaps in parallel.
Add a parallelization option. Use-case: you have a tight deadline and need not concern yourself with resource utilization or co-worker etiquette. I need my results!
Add an option to schedule code execution. Likely in the form of a sleep specified by the user. Use-case: you want to be courteous and avoid runs during production code runs and when cluster usage is high during the work day.
Make one or more of the options above accessible via flags. This would be really cool, but also a lot of work just to add sprinkles on top.
More.
Open whatever command line terminal interface you prefer. Examples include Terminal on MacOS and Cygwin on Windows. MacOS has Terminal already installed. On Windows, you can download Cygwin here.Navigate to the directory with your .hql files. For example, type cd anyoung/hqlscripts/ to change your current working directory to that folder. This is the conceptual equivalent of double-clicking on a folder to open it, except here we are using a text-based command.nano <whateverfilename>.shnano is a command that opens a program of the same name and asks it to create a new file called <whateverfilename>.sh To revise this file, utilize the same command.Copy my script and paste it into this new .sh file. Change the recipient email address in my script from youremail@wherever.com to yours.bash <whateverfilename>.sh Bash uses files of the extension type .sh. This command asks the program, bash to run your “shell” file denoted with the .sh extension. Upon running, the script will output a list of all files in the same directory as the shell script. This list output is something I programmed for convenience.Marvel at your productivity gains!
Open whatever command line terminal interface you prefer. Examples include Terminal on MacOS and Cygwin on Windows. MacOS has Terminal already installed. On Windows, you can download Cygwin here.
Navigate to the directory with your .hql files. For example, type cd anyoung/hqlscripts/ to change your current working directory to that folder. This is the conceptual equivalent of double-clicking on a folder to open it, except here we are using a text-based command.
nano <whateverfilename>.shnano is a command that opens a program of the same name and asks it to create a new file called <whateverfilename>.sh To revise this file, utilize the same command.
Copy my script and paste it into this new .sh file. Change the recipient email address in my script from youremail@wherever.com to yours.
bash <whateverfilename>.sh Bash uses files of the extension type .sh. This command asks the program, bash to run your “shell” file denoted with the .sh extension. Upon running, the script will output a list of all files in the same directory as the shell script. This list output is something I programmed for convenience.
Marvel at your productivity gains!
Andrew Young is an R&D Data Scientist Manager at Neustar. For context, Neustar is an information services company that ingests structured and unstructured text and picture data from hundreds of companies in the domains of aviation, banking, government, marketing, social media and telecommunications to name several. Neustar combines these data ingredients then sells a finished dish with added value to enterprise clients for purposes like consulting, cyber security, fraud detection and marketing. In this context, Mr. Young is a hands-on lead architect on a small R&D data science team that builds the system feeding all products and services responsible for $1+ billion in annual revenue for Neustar.
A pre-requisite to running the shell script above is to have a HiveQL or SQL script for it to run. Here is an example HiveQL script, example.hql:
CREATE TABLE db.tb1 STORED AS ORC ASSELECT *FROM db.tb2WHERE a >= 5;CREATE TABLE db.tb3 STORED AS ORC ASSELECT *FROM db.tb4 aINNER JOIN db.tb5 bON a.col2 = b.col5WHERE dt >= 20191201 AND DOB != 01-01-1900;
Note that you can have one or more CREATE TABLE commands in a single HiveQL/SQL script. They will be executed in sequence. If you have access to a cluster of nodes, you can also benefit from parallelization of your queries. I may explain that in another article.