Jupyter Magics with SQL. Jupyter/IPython notebooks can be used... | by Sayat Satybaldiyev | Towards Data Science

Jupyter/IPython notebooks can be used for an interactive data analysis with SQL on a relational database. This fuses together the advantages of using Jupyter, a well-established platform for data analysis, with the ease of use of SQL and the performance of SQL engines.
Jupyter magic functions allow you to replace a boilerplate code snippets with more concise one. Let’s explore Jupyter SQL magic that allows us to interact with Presto or any other relational databases.
Magic functions are pre-defined functions(“magics”) in Jupyter kernel that executes supplied commands. There are two kinds of magics line-oriented and cell-oriented prefaced with % and %% respectively.
To see the difference we start comparing code examples using magics functions and without. Ultimately, two statements achieves the same result.
Let’s start creating connection with SQLAchemy to fetch last executed query and put it to data-frame.
import pandas as pd
from sqlalchemy.engine import create_engine

# Presto
engine = create_engine('presto://localhost:8080/system/runtime') 

#Read Presto Data query into a DataFrame
df = pd.read_sql('select * from queries limit 1', engine)
df.head()

This is how would we do it in case we have enabled SQL magic functions.
pd = %sql select * from runtime.queries limit 1
pd.DataFrame().head()

 * presto://user@localhost:8080/system
Done.

The examples use Prestodb as a SQL engine. However, the approach can be extended to any engine that compatible with Python SQLAlchemy and has driver implementing Python DB 2.0 Specification. To enable the magic we need an ipython-sql library. The examples further are mostly adopted from the ipython-sql official repository.
pip install pandaspip install sqlalchemy # ORM for databasespip install ipython-sql # SQL magic function
To work with Prestodb we will need to have PyHive library. SQLAlchemy under the hood will use the library to make a connection and submit SQL queries. For other engines, you need to install a proper driver i.e. PostgresSQL, MySQL.
pip install pyhive[presto] # DB driver library
The ipython-sql library is loaded using the %load_ext iPython extension syntax and is pointed to the connection object as follows:
%load_ext sql%config SqlMagic.autocommit=False # for engines that do not support autommit
Please note that for Presto, Impala and some other engines you need to disable autocommit feature. This is done via SqlMagic config property. To list all options of config you can run in the cell: %config SqlMagic
To connect to the database you need to pass connection string in SQLAlchemy format to the %sql function.
%sql presto://user@localhost:8080/system
If connection string is not provided and connection has not been made yet, ipython-sql tries to get connection information from DATABASE_URLenvironment variable. You can export environment variable DATABASE_URL in ~/.bashrc in case your connection information is static. %env is another magic function that sets environment variables.
%env DATABASE_URL=presto://user@localhost:8080/system%sql SELECT 1 as "Test"
In case of multiple SQL engines, and you want to combine data from them you can pass connection string with each query of the magic function in cell-mode.
%%sql user@jmxSHOW SCHEMAS
Parameter substitution is a handy feature that allows defining SQL query parameters at query run-time. It makes code less fragile and expressive. The parameter needs to be defined in the local scope and prefixed with colon i.e. :parameter
state='FINISHED'
%sql SELECT :state as "bind_variable"

Done.

Ordinary IPython assignment works for single-line %sql queries:
result = %sql select query_id, state from runtime.queries limit 1
For multi-line query you need to use << syntax.
%%sql result_set << SELECT query_id, state, query FROM runtime.queriesLIMIT 2
SQL magic has a nice integration with pandas library. Result from SQL query can be converted to regular pandas data frame via DataFrame call.
You should also consider reading about build-in magic functions that allows you to achieve more and type less! You can also take a look at the full notebook with the examples from the post.