Supercharging MS SQL Server with Python | by James Briggs | Towards Data Science

At work, I use SQL a lot. It is not without annoying nuances and limitations, but in the end, it is the foundation of all data professions. For that reason, it is an absolute necessity for anyone working in the land of data. Being competent in SQL is incredibly important.
Although SQL is great, why settle for great? Why don’t we supercharge SQL?
The limitations of SQL stem from it being a declarative language, meaning we tell SQL what we want and SQL will fetch that from a specified database. For many data extraction or simple data manipulation tasks, this is all that is needed.
But what if we want more?
That is exactly what I will show you how to do in this article.
This code is the foundation to supercharging MS SQL server. Once this is built we can connect to SQL from Python with just:
sql = Sql('database123')
Easy right? There are a few things going on, so let’s dissect the code.
class Sql:
The first thing to note, we are containing this within a class. I have found this approach logical as in this format we add or remove processes for this specific database. This will make more sense once seeing how this works.
We initialise our class:
def __init__(self, database, server="XXVIR00012,55000"):
Myself and my colleagues almost always connect to the same server. So I set this common server name as the default argument for server.
Your server name can be found in the ‘Connect to Server’ dialog box or at the top of your window in MS SQL Server Management Studio:
Next, we establish our connection to SQL:
self.cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"                           "Server="+self.server+";"                           "Database="+self.database+";"                           "Trusted_Connection=yes;")
This is made incredibly easy thanks to the pyodbc module. We simply pass a connection string to the pyodbc.connect(...) function, more details on this can be found here.
Finally, I like to create a query string within the Sql class which is updated with every query passed to the class:
self.query = "-- {}\n\n-- Made in Python".format(datetime.now()                                              .strftime("%d/%m/%Y"))
This allows us to keep a log of our code and also doubles as a more readable output to colleagues that may be more comfortable reading SQL code.
Note that I will exclude the update to self.query sections of the code in the following code snippets, please check the full code in the links provided if needed.
There are a few essential functions that I find incredibly useful and use almost daily. These all focus on transferring data in or out of a database.
Let's start with the following directory:
For our current project, we need to:
import these files to SQL server
union them into a single table
dynamically create multiple tables based on categories within a column
After fleshing out our SQL class a little more this will be as easy as:
Let's start at the top.
This function is contained inside our SQL class. It allows us to easily push a Pandas dataframe to the SQL database.
This is particularly useful when needing to upload a lot of files. However, the true power in allowing Python to import data to SQL comes with Python’s flexibility.
Importing a specific tab across a dozen Excel workbooks into SQL can be a nightmare. But with Python it’s easy. Now we have built a function that allows us to use Python to read those tabs and import them into SQL for us.
This function is actually used in the union and drop functions. It simply allows us to execute SQL code as simply as possible.
The response argument gives us the option to extract the output of our query to a DataFrame. Allowing us to extract all unique values from colX in our generic_jan table using just:
sets = list(sql.manual("SELECT colX AS 'category' FROM generic_jan GROUP BY colX", response=True)['category'])
Now we have built the manual function, creating the union function is easy:
This simply loops through a list of table names we provide via table_list, building a UNION query for all table names given. This is then executed with self.manual(query).
We have the ability to upload a very large number of tables to SQL Server. This is great but can quickly overpopulate our database. To remedy this, we will create a drop function:
Again, this function is incredibly simple thanks to the manual function. This allows us to either drop a single table by providing a string to tables or to drop multiple tables by providing a list of table names to tables.
When combined these incredibly simple functions allow us to massively expand the capabilities of SQL Server by leveraging the strengths of Python.
If you are interested in seeing more please check out the GitHub repo for pysqlplus. This is a small project and is simply built to enhance interaction with MS SQL Server for my own workplace.
Nonetheless, I hope it may help some other users find ways to incorporate some Python into their SQL routines. I personally use this almost daily, it is so simple yet incredibly powerful.
Thanks for reading!
If you are interested in learning more about Python and SQL in the cloud, consider my review on IBMs Fundamentals of Scalable Data Science course available on Coursera here: