Use Python and Bulk Insert to Quickly Load Data from CSV Files into SQL Server Tables | by Randy Runtsch | Towards Data Science

Data acquisition is a large part of many data analytics projects and system development life cycles. This article will show you how to write a simple Python program that uses the BULK INSERT utility to rapidly insert data from a CSV file into a SQL Server database table.
There are many ways to load data from a CSV file into a SQL Server table. Here a few methods:
Run the BULK INSERT utility from the command line.
Run the BULK INSERT utility from SQL Server Management Studio (SSMS).
Use the SQL Server Management Studio (SSMS) Import Flat File wizard.
Write a program that opens the CSV file, reads its records one-by-one, and calls a SQL INSERT statement to insert the rows into a database table.
If there are so many ways to get data from a CSV into a SQL Server database, why write a Python program that calls the BULK INSERT utility to load it into a table? Here are some reasons why this might be a helpful approach:
The Python program could perform process steps before it executes BULK INSERT.
The Python program could ensure that it does not write duplicate data to the destination table.
The program could cleanse or transform the data following the BULK INSERT.
It could perform error-handling functions.
It could send notifications, by email or other methods, about its actions.
I use a Windows 10 environment and a variety of Microsoft’s tools and others for my software development, database, and data analytics work. Here is the list of software that I used for this article:
Windows 10 Home — While I use Windows, some of these tools may work in other operating systems.
Notepad — Notepad is helpful to view CSV file data in its raw form.
Microsoft Excel — Excel is helpful to view data from CSV files formatted in columns and rows.
Microsoft SQL Server 2019 Express — There are many editions of SQL Server. Express and SQL Server Developer are free versions.
Microsoft SQL Server Management Studio (SSMS) — Use this front-end tool to manage and query SQL Server databases. It is free and works well.
Microsoft Visual Studio Community Edition — I have tried the Microsoft Visual Studio Code (VS Code) integrated development environment (IDE). Still, I prefer to code in Visual Studio since I have used it for years, and I find it easy to configure and use, except for its multitude of deep menu selections.
Python 3.9.2 — This is the most recent Python release as of this writing, but any recent version should work.
For the example presented in this article, I generated a CSV file with 10,000 records of fabricated data from the free Online Data Generator. The file is called c:\test_data_folder\person.csv. It contains the columns ID, Job Title, Email Address, and FirstName LastName. A Python program will execute a SQL Server BULK INSERT statement to load data from the file into a table.
Microsoft introduced the ability to use BULK INSERT to insert data from CSV files in SQL Server 2017. So, you will need that version or newer to use this capability.
Here are the steps used to create a database called HR (for Human Resources):
Connect to SQL Server
Launch SSMS.
Connect to the database server. In this case, I used Windows authentication to connect to the locally-installed instance of SQL Server Express.
Create the HR Database
Expand the [+ Databases] node in Object Explorer. Right-click on [+ Database] and click on [New Database...].
In the New Database dialog box, enter “HR” into the Database name textbox. Leave all settings as is. Click on [OK] to create the database.
Verify that the HR database appears in Object Explorer. If not, right-click on Databases and click on [Refresh]. It should come into view.
At this point, the HR database will not contain any tables or other objects, such as stored procedures. While BULK INSERT can create tables when it runs, I have found that creating tables ahead of time offers advantages. For example, I can specify the table’s key column(s) and each column’s type and length. Looking again at the sample data in the CSV file, let’s create columns with the same names and with these data types:
ID — INT
Job Title — NCHAR(60)
Email Address — NCHAR(120)
FirstName LastName — NCHAR(80)
Since all rows will have values for all columns, set each column to NOT NULL. Also, since ID is a unique identifier for each row, select it as the key.
Follow these steps to create the table:
In Object Explorer, click on [+ HR] to view the selection within the database.
Right-click on [+ Tables] and click on [New] and then click on [Table...].
Enter the data as shown below. After the Column Name, Data Type, and Allow Nulls values have been entered, right-click on the ID column name and click on [Set Primary Key]. Setting ID as a key will ensure that only one row in the table can contain any ID value.
Click on the Save icon in the ribbon menu, and in the Choose Name dialog box, enter the name “Person.” Click on [OK] to save the table.
In Object Explorer, click on [+ Tables] to expand the node. Then, right-click on [- Tables] and click on [Refresh]. The Person table should now be in view.
Click on [+ dbo.Person] and then on [+ Columns] to examine the table’s structure. It should look similar to the following image.
Now that the HR database and Person table exist let’s examine a simple Python program that uses the BULK INSERT utility. It simply inserts all records from the CSV file into the Person table.
This Python program consists of two modules or files:
c_bulk_insert.py contains the c_bulk_insert class. It includes functions to connect to the database and build and execute a BULK INSERT statement to insert data from a CSV file into a database table.
sql_server_bulk_insert.py simply instantiates the c_bulk_insert class and calls it with the information needed to do its work.
When the program instantiates class c_bulk_insert, it performs these steps:
Connect to the SQL Server database.Construct the BULK INSERT query with the destination table’s name, input CSV file, and some settings.Open a database cursor.Execute the query.Clean up: Commit the BULK INSERT transactions, close the cursor, and close the database connection.
Connect to the SQL Server database.
Construct the BULK INSERT query with the destination table’s name, input CSV file, and some settings.
Open a database cursor.
Execute the query.
Clean up: Commit the BULK INSERT transactions, close the cursor, and close the database connection.
The Python class c_bulk_insert in module c_bulk_insert.py performs the logic described in the Code Logic section above.
"""     Name:           c_bulk_insert.py    Author:         Randy Runtsch    Date:           March 17, 2021    Description:    This module contains the c_bulk_insert class that connect to a SQL Server database                    and executes the BULK INSERT utility to insert data from a CSV file into a table.    Prerequisites:  1. Create the database data table.                    2. Create the database update_csv_log table."""import pyodbcclass c_bulk_insert:def __init__(self, csv_file_nm, sql_server_nm, db_nm, db_table_nm):# Connect to the database, perform the insert, and update the log table.conn = self.connect_db(sql_server_nm, db_nm)        self.insert_data(conn, csv_file_nm, db_table_nm)        conn.closedef connect_db(self, sql_server_nm, db_nm):# Connect to the server and database with Windows authentication.conn_string = 'DRIVER={ODBC Driver 17 for SQL Server};SERVER=' + sql_server_nm + ';DATABASE=' + db_nm + ';Trusted_Connection=yes;'        conn = pyodbc.connect(conn_string)return conndef insert_data(self, conn, csv_file_nm, db_table_nm):# Insert the data from the CSV file into the database table.# Assemble the BULK INSERT query. Be sure to skip the header row by specifying FIRSTROW = 2.qry = "BULK INSERT " + db_table_nm + " FROM '" + csv_file_nm + "' WITH (FORMAT = 'CSV', FIRSTROW = 2)"# Execute the querycursor = conn.cursor()        success = cursor.execute(qry)        conn.commit()        cursor.close
The module sql_server_bulk_insert.py instantiates c_bulk_insert. It calls it with:
CSV file name
SQL Server instance engine name
Database name
Destination table name
"""     Name:           sql_server_bulk_insert.py    Author:         Randy Runtsch    Date:           March 17, 2021    Description:    This program is the controller that uses the Microsoft Transact-SQL BULK INSERT                    statement to quickly insert the rows from a CSV file into                    a SQL Server table.    Prerequisites:  1. Create the database data table.                    2. Create the database update_csv_log table."""from c_bulk_insert import c_bulk_insertbulk_insert = c_bulk_insert(r'c:\\test_data\\person.csv', 'xxxxx-DESKTOP-\\SQLEXPRESS', 'HR', 'Person')
After the program runs, executing a SELECT query in SSMS shows that it wrote the records from the CSV file to the Person table.
There may be many reasons and ways to enhance the program. Here are a few ideas:
Add error handling to the database connection, query execution, and other parts of the program. The program could use error handling, for example, to gracefully shut down, retry a set number of times, and notify the appropriate parties by email.
Automate the program to insert data into the database when needed. For example, schedule the program to run periodically with Windows Task Scheduler to insert person records from new CSV files.
Create and write to a log table to capture the program start and finish times and other important events. Add other messages, such as error details, to identify issues to troubleshoot.
As you can see, using Python to call BULK INSERT is a way to automate part of a workflow to quickly insert data from a CSV file into a SQL Server database table. It might prove to be a handy technique to add to your data analytics or software development toolkit.
Randy Runtsch is a data analyst, software developer, writer, photographer, cyclist, and adventurer. He and his wife live in southeastern Minnesota, U.S.A.
Watch for Randy’s upcoming articles on public datasets to drive data analytics solutions, programming, data analytics, photography, bicycle touring, beer, and more.