Generating fake data with Python, Numpy, and Faker | Towards Data Science

The first step in data analysis is finding data to analyze.
All too often, this crucial first step is next to impossible. Data that meets our needs may be proprietary, expensive, hard to collect, or simply may not exist.
Finding a suitable dataset is the most common problem I face when wanting to try out a new library or technique — or beginning to write a new article.
Here we solve this problem, once and for all, by creating our own dataset!
Here we will create a dataset for an imaginary widget factory.
Our widget factory has employees whose only job is to make widgets. The factory monitors widget making productivity by counting the number of widgets made and how fast the workers can make them. Making a widget consists of three steps, all of which are timed by the widget monitoring system.
Let’s get started making our fake widget factory dataset!
We can use the amazing package, Faker to get started. Faker is self described as “a Python package that generates fake data for you.”
Faker is available on PYPI and is easily installable with pip install faker.
Let’s initialize a faker generator and start making some data:
# initialize a generatorfake = Faker()#create some fake dataprint(fake.name())print(fake.date_between(start_date='-30y', end_date='today'))print(fake.color_name())>>> Bruce Clark>>> 1996-08-24>>> LimeGreen
Faker also has a method to quickly generate a fake profile!
print(fake.profile()>>> { 'address': '56393 Steve Lock\nNew Paul, FL 14937',>>>  'birthdate': datetime.date(1968, 1, 2),>>>  'blood_group': 'O+',>>>  'company': 'Hernandez, Keller and Montes',>>>  'current_location': (Decimal('66.6508355'),>>>      Decimal('54.569691')),>>>  'job': 'Museum education officer',>>>  'mail': 'levyjoan@yahoo.com',>>>  'name': 'Kim Stephens',>>>  'residence': '0248 Patricia Street >>>       Apt. 507\nEast Dustin, WV 65366',>>>  'sex': 'F',>>>  'ssn': '248-78-5827',>>>  'username': 'ruizdavid',>>>  'website': ['https://www.garcia-garza.org/',>>>      'https://www.williamson.info/']}
While this is very cool, it's not super helpful for our widget factory just yet.
We need to create multiple examples of each element. We can do this with a list comprehension:
# create a list of color namescolors = [fake.color_name() for x in range(4)]print(colors)>>> ['LightCoral', 'Yellow', 'WhiteSmoke', 'LightGray']
Now we’re getting somewhere! The profile method gives a little bit more information for each worker than what we need — Our widget factory keeps minimal records on each worker!
Let’s start creating a dataframe of workers, with only the information we’re interested in. We’ll use a list comprehension to generate a dictionary of worker information with the worker’s name and hire date:
fake_workers = [    {'Worker Name':fake.name(),      'Hire Date':fake.date_between(start_date='-30y', end_date='today')    } for x in range(10)]                 print(fake_workers)
This is a great start! But, what if we want to assign each worker to a team? If we just use Faker we would have a huge number of teams with potentially only one worker per team.
Let’s say we want to create four teams and evenly divide the workers between them. We can use NumPy’s random sampling for this task.
Numpy’s random sampling module contains many methods for generating pseudo random numbers. Here we’ll explore just a few of the available options.
We can use numpy.random.choice to randomly select a color from our colors list we created using Faker above:
# numpys random choice to select a color from our colors listnp.random.choice(colors)>>> Yellow
Alone, this is not super useful, as we only have one randomly selected color. Let's\ use a list comprehension to randomly select several colors:
# generate an array of colors with a list comprehension[np.random.choice(colors) for x in range(5)]>>> ['WhiteSmoke',>>>   'WhiteSmoke', >>>   'Yellow', >>>   'WhiteSmoke', >>>   'LightGray']
Now we’re getting somewhere! What if we want to select colors from the colors list with differing probabilities?
numpy.random.choice takes an optional argument p that allows us to specify the probability of each item. (When p is not specified a uniform distribution is assumed.)
Let’s specify probabilities for our colors:
# generate an array of colors with differing probabilities[np.random.choice(colors, p=[0.1, 0.6, 0.1, 0.2]) for x in range(10)]>>>['LightGray',>>> 'LightCoral',>>> 'LightGray',>>> 'LightGray',>>> 'LightCoral',>>> 'WhiteSmoke',>>> 'Yellow',>>> 'Yellow',>>> 'Yellow',>>> 'Yellow']
We can see that yellow and light gray are more common in the output! Due to the small sample size the distributions are not exactly as specified. We would need to increase the sample size to approach our specified probabilities.
We can also generate data based on a specified distribution. Here we randomly generate several distributions and visualize with a histogram:
# generate normal distributionnormal_dist = np.random.normal(loc=5, scale=1, size=1000)# generate gamma distributiongamma_dist = np.random.gamma(shape=3, scale=1, size=1000)# generate exponential distributionexp_dist = np.random.exponential(scale=4, size=1000)# histograms to visualize the distributionsfig, [ax1, ax2, ax3] = plt.subplots(1,3)ax1.hist(normal_dist)ax1.set_title('Normal Distribution')ax2.hist(gamma_dist)ax2.set_title('Gamma Distribution')ax3.hist(exp_dist)ax3.set_title('Exponential Distribution')plt.show()
These are looking good! Let’s put everything together and create our fake widget factory dataset!
We’ll create two dataframes: One for worker data and one for widget data.
We will create an employee roster dataset that contains employee id, name, hire date, status, and team using Faker, Numpy, and list comprehensions.
First, we create a function, make_workers(). This function will take one argument, num, which will represent the number of employees to generate.
We use np.random.choice to assign each employee to a team, using the default uniform distribution, and to assign each employee to a status of full time, part time, or per diem by specifying probabilities.
This looks great! Now let’s create some widget data.
We’ll begin by randomly assigning each widget an item number using the built in id() method.
From the documentation, id() “return[s] the ‘identity’ of an object. This is an integer which is guaranteed to be unique and constant for this object during its lifetime”. Because we’re iterating through the same range each time our make_widget_data() function is called, we will create duplicate item numbers. To solve this problem we’ll append the worker id to the item number to guarantee uniqueness.
Next, we’ll assume in widget creation there are three steps. For each step, we’ll generate a different random distribution that will represent the amount of time the step took to complete.
Next, we assume that employees do not all work at the same rate — and that full time workers will complete more widgets than part time workers.
To reflect this in the data, we iterate through the worker_df, created above and randomly select a number of widgets created by each worker based on their status.
We’ll create an empty list to store dataframes for each worker’s widgets, and then finally concatenate the list of dataframes into one final widget dataframe:
The widget factory workers were busy! They created 2817285 widgets!
Finally, we can save our datasets to be used for future analyses:
worker_df.to_csv('data/workers.csv', index=False)widget_df.to_csv('data/widgets.csv', index=False)
Here we explored the Python package Faker and Numpy’s Random Sampling module.
Faker makes it easy to generate a wide variety of data, including names, addresses, and dates.
Numpy’s random sampling module allows us to generate specific distributions and pseudo randomly select items from lists.
We were able to generate fake data for an imaginary widget factory. We created two datasets, one with worker details, and the second with widget creation data.
In future articles, we’ll explore this fake data, perform exploratory analysis, and generate reports for the widget managers.