Generating Random Data into a Database Using Python | by Jordan Williams | Towards Data Science

In this article we’re going to demonstrate how to generate dummy data and input for a MySQL database. Majority of the times we need to create a database to test some software that we’ve built. To do this, we can use Structured Query Language (SQL) to create a mock database system. However, problems may arise if we want to test hundreds or even thousands of records. There is definitely no way we could write that many insert statements into our SQL scripts to meeting our testing deadlines for product deployment, that would be too tedious and time consuming.
To help with this problem, our trusty friend Python comes to assist !
Python to the rescue !
As you may or may not know python is a scripting language that is well known for its plethora (abundance) of libraries. Python is also known for its ability to automate task using its very simple syntax.
Pre-requisite Knowledge :
Installation of python packages using pip.
Basic understanding of what databases are and how they use SQL.
Even if you don’t have what is required above, this article will ensure that you understand its content to help you further your Python career.
So let’s get into it !
The python libraries that we’ll be used for this project are:
Faker — This is a package that can generate dummy data for you. We will use this to generate our dummy data.
Pandas — This is a data analysis tool. This will be used to package our dummy data and convert it to tables in a database system.
SQLAlchemy — This is an Object Relational Mapper(ORM) for SQL. ORM will be used as our database engine as it will provide our connection to our database.
Other tools that we’ll be using includes:
XAMPP — This is a distribution containing MariaDB, PHP, and Perl database management systems. We’ll be using MariaDB which is our SQL database.
Note: Different types of databases can be used however, for this article will focus on the MySQL connections. So, for any other database such as PostgreSQL, Microsoft SQL Server, Oracle, etc. an alternate configuration will be required.
Let’s Begin !
First, we need those python libraries we spoke about earlier. Pip — Pip Installs Packages(PIP for short). PIP is the package manager for python and also what we’ll be using to install our python packages.
Note: Conda can also be used for those anaconda distribution lovers.
pip install pandas Faker sqlalchemy mysqlclient
We now have all our relevant packages installed from using the code above.
First, we need to import our relevant libraries.
import pandas as pdfrom faker import Fakerfrom collections import defaultdictfrom sqlalchemy import create_engine
The defaultdict (default dictionary) is from the collections package. This will act as our dictionary as it will provide additional functionality over a normal dictionary as we will see later on in this article.
To generate our dummy data, we will first initialize our Faker instance that we’ll be using to get our dummy data.
fake = Faker()
We’ll usefake_data to create our dictionary. defaultdict(list) will create a dictionary that will create key-value pairs that are not currently stored within the dictionary when accessed. Essentially, you do not need to define any keys within your dictionary.
fake_data = defaultdict(list)
Now, let’s decide on the kind of data that we’re going to take from the Faker instance to be stored in the fake variable. An example of this data includes:
First NameLast NameOccupationDate of Birth(DOB)Country
First Name
Last Name
Occupation
Date of Birth(DOB)
Country
We can access this data through different methods within the fake data instance.
The methods we’ll be accessing are first_name() , last_name(), job(), date_of_birth()andcountry().
We’ll loop a thousand times to access these methods and then add them to our dictionary.
for _ in range(1000):    fake_data["first_name"].append( fake.first_name() )    fake_data["last_name"].append( fake.last_name() )    fake_data["occupation"].append( fake.job() )    fake_data["dob"].append( fake.date_of_birth() )    fake_data["country"].append( fake.country() )
Note: Because defaultdict(list)was used, whenever we accessed a key we did not create, it will automatically be added with an empty list [] as its value.
Now, since we have all our random data within our dictionary fake_data. We need to package this data into our pandas dataframe.
df_fake_data = pd.DataFrame(fake_data)
The pandas dataframe provides many features for analyzing and manipulating data. For the purpose of this project we’ll be manipulating this dataframe as a database entry.
First, let’s set-up our database connection ! We’ll be using XAMPP as previously mentioned to start our instance of SQL .
We’ve successfully started our database. Now we will create the database that will house the table that will be generated by our python script using the code shown below.
mysql -u rootcreate database testdb;
Look ! We started our MySQL database(MariaDB in this case) and created our testdbdatabase.
Note: Root is my username. No password was configured (created) for my account however, there might be a prompt after mysql -u root where you will be asked to enter your password, if you had created one.
As you can see, I am using the command line version of MySQL. The GUI version is also an alternative option.
Now, our next step is to connect the database we created to our python code. To do this, we’ll use the engine from thesqlalchemy library.
engine = create_engine('mysql://root:@localhost/testdb', echo=False)
The string in the create_engine follows the structure of the driver://username:password@host:port/database . As you can see, in my case I had no password, so it was left blank. The port can be specified but if you are using the default configurations of MySQL it’s not necessary.
Now that we have created our connection, we can now save the pandas dataframe directly to our database testdb.
df_fake_data.to_sql('user', con=engine,index=False)
When the code above is executed, a user table will be created in our database with a thousand records being added to it.
As you can see by looking at the data and the row number at the bottom left of our screenshot, we’ve just generated a thousand records !
The full code for this project is located in the gist below:
A task that can be done is to increase the number of fake records generated by increasing the number of iterations in the loop.
Note. We first need to delete our user table that was previous created by our script by using the query drop table user;
for _ in range(100000):    fake_data["first_name"].append( fake.first_name() )    fake_data["last_name"].append( fake.last_name() )    fake_data["occupation"].append( fake.job() )    fake_data["dob"].append( fake.date_of_birth() )    fake_data["country"].append( fake.country() )
Due to the output being too many records for my terminal, I will just illustrate the row count:
The timestamp below shows how long it took to generate one hundred thousand records on my machine:
Not a bad result right !
We created 100,000 records in a database in simply 31 seconds. 31 seconds! Can you believe that, crazy right !
We were able to create a database populated by common user data such as their first name, last name, date of birth, occupation and country. Not only did we create it, but we created it in a rapid timeframe. One of the benefits of this solution is that it is faster than many other solutions that I have come across. Please feel free to play around with the faker library. There are many other types of data that it can be used to generate.
If you liked this article, sign up to receive emails whenever we post a new article. Use this link enter your email, it’s completely free !