An Easy Beginners Guide to SQLite in Python and Pandas | by Julia Kho | Towards Data Science

Welcome to an easy beginners guide to SQLite. In this article, you’ll learn about what SQLite is, how to connect to databases, create tables, insert and query for data, and access SQLite in Pandas. I am assuming that you have a basic understanding of SQL and my focus in the article will be on how to use SQLite in Python.
SQLite is a software library that provides us with an open-source relational data management system where we can store large amounts of data.
It is easy to set up, it’s self-contained (requires minimal support), and does not require a server. A SQLite database is literally just a plain file, which makes it easily accessible and portable. Given the simplicity of SQLite, it is the most widely deployed database engine.
SQLite is great for many reasons, but in practice, you shouldn’t use SQLite if you are handling a gigantic amount of data. The database is actually limited to 281 terabytes.
If you would to follow along, you can download a copy of my Jupyter Notebook here.
We start by importing the SQLite library inside our python file. This library does not have to be downloaded as it should already be included in the standard library if you are using Python 2.5 and above.
import sqlite3 as db
Next, we can either connect to an existing database or create a new one if it doesn’t exist.
conn = db.connect('my_database.db')
Note that the SQLite engine maintains the database as a file with the extension .db. So the line of code above will connect to a file called my_database.db IF IT EXISTS in the folder. If it does not exist, a file called my_database.db will automatically be created on your computer as shown in the snippet above.
In order to interact with our database, we now have to create a cursor. This is what we will use to issue commands that will allow us to query or modify our database.
c = conn.cursor()
Now that we have our database connection and cursor created, we can now start interacting. Let’s create our first table in the database.
We use the method execute on the cursor c we created earlier and we pass in our SQL statement.
c.execute("CREATE TABLE employees (empid INTEGER PRIMARY KEY, firstname NVARCHAR(20), lastname NVARCHAR(20))")
The statement above will create a table called employees with three columns: empid, firstname, and lastname.
You can check that the table was successfully created with the following statements.
c.execute("SELECT name FROM sqlite_master WHERE type='table';")print(c.fetchall())
The select query above will show you all tables that are in your database. Note that the results of the query are not outputted if you only have the first statement. You have to use c.fetchall() in order for the results to show.
To populate the table, we have several options. If only inserting a few rows, you can use the following insert statements to input two rows into the employee table.
c.execute("INSERT INTO employees VALUES (12986,'Michael','Scott')")c.execute("INSERT INTO employees VALUES (12987,'Dwight','Schrute')")
If you have many rows to input into the table, consider making a list of tuples and using theexecutemany()command instead.
new_employees = [(12987, 'Jim', 'Halpert'), (12988, 'Pam', 'Beesly'), (12989, 'Andy', 'Bernard'), (12990, 'Kevin', 'Malone'), (12991, 'Toby', 'Flenderson'), (12992, 'Angela', 'Martin'), (12993, 'Stanley', 'Hudson')]c.executemany('INSERT INTO employees VALUES (?, ?, ?)', new_employees)
Notice that we’ve replaced our insert statement from earlier with (?, ?, ?). The question marks here represent placeholders for the three items that we are inputting into the table.
Let’s take a look at what’s in our table now. We can use the select statement inside the execute command similar to what we did previously.
From the output, we can see all our employees have been correctly inputted into the table. Hurray!
We’ve made a couple changes to our database, but they are not saved to the database just yet! These are just temporary modifications, so we have to make sure to commit those changes with the following command.
conn.commit()
If we don’t, the changes will be lost once we close the connection.
Lastly, it is always good practice for us to close both the cursor and connection once we are done. We can do so with the following commands.
c.close()conn.close()
One cool thing you can do is use both SQLite and Pandas. Pandas has a read_sql_query method that will allow you to return the data as a Pandas dataframe. From there, you can more easily manipulate the data in Pandas. Personally, I prefer bringing the data inside Pandas to work with.
We again have to first establish a connection to our database. Then we can use pd.read_sql_queryand save the output as a dataframe calleddf_employees.
#import libraryimport pandas as pdcon = db.connect('my_database.db')df_employees = pd.read_sql_query('select * from employees', con)
Now that it is a Pandas dataframe, you can use your usual pandas function to manipulate the data as you wish.
For example, let’s say that Stanley (last person in the table) was never meant to be hired and so we need to get rid of him from the table.
df_new = df_employees[:-1]
We can write our new dataframe back to SQLite and replace our original employee table. To write to SQLite, we use the method to_sql() on the new dataframe.
df_new.to_sql("employees", con, if_exists="replace")
We provide three parameters inside this method:
Name of the SQL tableThe connection to the databaseHow to behave if the table already exists. “Replace” will drop the original table. “Fail” will raise a value error in Python. “Append” will insert the data as new rows into the table
Name of the SQL table
The connection to the database
How to behave if the table already exists. “Replace” will drop the original table. “Fail” will raise a value error in Python. “Append” will insert the data as new rows into the table
To check that it worked as we expected, I can query the table again to see if Stanley has been removed. It has!
pd.read_sql_query ('select * from employees', con)
And just like before, once we’re done, let’s close the connection.
con.close()
That’s it! SQLite is pretty simple to use. If you are looking for a good database to practice SQL and/or SQLite, you can download the Chinook database here. More information about the data inside the Chinook database can be found here.
Thank you for reading! Let me know in the comments if you have questions or want to tell a SQL joke.