Creating a Dataset from Scratch. Using web scraping, API calls, and... | by Jesse Neumann | Towards Data Science

This blog walks through the entire process of creating a dataset for use in a larger data science project.
The data science project involves modeling the success of video games, discovering the importance of individual features, and then creating an interactive web app.
The web app will allow a game developer to input and modify a game design in order to maximize the predicted success given the resources available to the developer.
One of the most important aspects of any project, and one of the most overlooked is project planning.
Having a plan will ensure you consistently move forward toward your goal, and your resulting project will be much better organized and will be completed much sooner (if you’re like me you are easily distracted by new feature ideas).
The best way I have found so far is to use the planning section of your github’s project repository. I use this section to set up a automated Kanban style project board that will complete steps automatically after you merge a pull request for the specific issue.
Setting up the board is simple:
Go to your project repo and click on ‘Projects’ tab.Fill out the form and click ‘Create Project’
Go to your project repo and click on ‘Projects’ tab.
Fill out the form and click ‘Create Project’
Now that we have our project board set up, go ahead and delete the default informative cards that appear as part of the template.
After that, begin adding cards for each of the main steps necessary for successfully completing the project:
Identify features that will be strong predictors for a games success, and find a source where they can be obtainedCreate a data dictionary with the feature name, a short description, the type, and the sourceDecide on which database to use and set it upImplement a class for interacting with the databaseImplement a class for scraping the data from SteamImplement a program for obtaining the data from RAWG APICombine the two programs to create a script that will populate the databaseUpload the completed dataset to Kaggle
Identify features that will be strong predictors for a games success, and find a source where they can be obtained
Create a data dictionary with the feature name, a short description, the type, and the source
Decide on which database to use and set it up
Implement a class for interacting with the database
Implement a class for scraping the data from Steam
Implement a program for obtaining the data from RAWG API
Combine the two programs to create a script that will populate the database
Upload the completed dataset to Kaggle
Here is our project board now, with the cards moved around a little to illustrate how you can track your workflow:
Now as you begin working on the project you can select the card you are working on currently, and click on the ‘...’ and convert it to an issue. This will enable you to submit a pull request that will automatically close the issue and move it into the ‘Done’ column.
Here is what the almost completed project board looks like:
Now that we have an outline and a project board set up, we can ensure that anytime we are working on a card we are moving toward the end goal. No more getting sidetracked by adding cool (unnecessary?) features we think up on the fly.
Now that we have the planning out of the way, it’s time to start implementing each of our cards. Not all of the cards require coding or a pull request, especially in the early phase when we are developing the architecture.
The first place to start will be the Steam store, which is the largest online marketplace for PC games, and one of the most active gaming communities online.
Googling steam store will give us this link: https://store.steampowered.com/
Clicking on the link will take us to the homepage:
Clicking on a random game will show us what data is available and we can decide whether or not that piece of data will be a valuable predictor or not.
Heres a random game page:
Right away we can see some data points that will be useful for our dataset.
Let’s keep these data points in mind as we start looking for complementary data from an API source.
After googling “game developer apis” and clicking on the first result that is not an ad, we arrive at https://rapidapi.com/blog/top-video-game-apis/.
Several of the API’s are dedicated to specific games such as Minecraft and League of Legends, and will not be useful for our dataset.
The very first API on the list, however, is the RAWG Video Games Database API, which sounds promising even though I don’t know what ‘RAWG’ stands for.
We can check out the different response schema’s and identify any features that sound relevant for our dataset:
Once again we can see several features that could be useful for our dataset. Let’s keep these two resources handy and come up with a complete list of features we want our dataset to contain.
Not only is it best-practice to create a high-quality data dictionary which explains each variable, it is also extremely useful for several of the tasks we will complete for our project.
Creating a table using markdown is quick and easy, and provides us with a clean looking overview of all the features:
We now have a nice looking data-dictionary which will be our source of truth for the dataset.
Any changes to the dataset should first be recorded in the data-dictionary before being implemented by the individual classes.
One of the worst feelings ever is when your scraping program throws an error after 8 hours and all the data gathered up to that point is erased.
In order to avoid such bad feelings, we will use a simple database to persist the data while we are running the scraping program.
All we need our database to do is hold our data and let us know if a game has already been saved or not, so we can avoid redundancy.
There are several database options that would work for this simple project, but we will be using sqlite3. This will give us a super simple database on our local system that we can interact with using SQL.
To organize our code better and to keep the SQL all in one place we will create a class to manage and interact with our database. One important method we do need to add however is the ability to save our database as a ‘csv’ file.
In order to keep this post from taking 200 hours to read I will not be diving into much detail for the classes.
I plan on creating a few shorter blogs about how to create each individual class, but for now you can check out the full code repository here:
https://github.com/Jesse989/game-oracle
Also, if you have any comments, questions, or recommendations, please reach out to me. I really appreciate the chance to discuss these things.
This is basically a utility class for managing our database and handling the SQL so we don’t have to re-write the queries multiple times.
In addition, I have given it a ‘.to_csv()’ method that will create a csv file with the entire contents of our ‘games’ table.
Now that we have a way to persist the data, let’s focus on finding the data we want to save. Since we are dealing with two sources for our data, it makes sense to create two separate classes, one for each source.
First will be to come up with a way to obtain the data from the Steam store. To do this we will write a class that will use some basic python scraping libraries; Requests, BeautifulSoup, and the python library re (regular expressions).
This object will be responsible for requesting the individual game url, and then parsing the desired information using a combination of BeautifulSoup searches and regex.
As you can see, the class consists mostly of two public methods: ‘get_game_html’ and ‘strip_features’.
The first method fetches the page from the specified url, and then the second method is called on the resulting html.
The ‘strip_features’ method is where the entire results object is built, and consists mainly of composing the resulting game dict by using the private helper methods.
Now that we have completed the first half of obtaining the data, let’s move on and create the class that will interact with the RAWG API.
Dealing with API’s is definitely the easier of the two tasks, and as you can see this class is almost half as many lines of code.
The goal for this class is to have an interface we can use to call the API and parse the response into the correct format for our database.
This class exposes one public method, ‘get_game’ which expects a string representing a game title.
It then makes an API request to search for games that have the same title as the string we passed in. We assume the first game in the response list is the correct one, definitely not the best strategy, though it works in our case.
We then finish up by sending one more request with the newly acquired RAWG ID, which sends us back the full game information we are looking for.
The rest of the private helper methods parse and shape the response data into the correct format for our database.
Now that we have the tools for obtaining the data and saving it in our database, we just need to create a class that will orchestrate the procedures.
The main purpose of this class is to obtain the urls for the games we want in our dataset, and then to iterate through that list and use our previously defined classes to obtain and save the resulting data of each game.
This class introduces another useful tool, Selenium. Selenium was created to automate web browsing for testing, though it has become a standard in the data scientists web scraping toolkit.
We use Selenium over Requests here because the results page loads incrementally as we scroll down. Using Selenium we will automate the scrolling action and save the entire page-source once we have reached the bottom.
We can then load the html into BeautifulSoup and strip out all the individual game urls.
One of the reasons I chose create the url list this way, is that the results are sorted by popularity. That means our database index will inherently represent the popularity of a game, which we can use in our analysis and modeling.
To run the crawler:
# instantiate our crawlersc = SteamCrawl()# start the processsc.crawl()
The crawler will then extract all the individual urls from the saved html, and will begin to iterate through each individual game url. First the crawler will check and make sure the game has not been saved already. If it has been then we move onto the next url, if it has not then we fetch the info using our Game Parser and RAWG API, and insert the new info into the database.
Disclaimer: The program did throw an error on the last game in the list. Thankfully the games were already persisted in the database so I could just call ‘.to_csv’ on the database client.
# Using the crawlers database client 'to_csv' method we createdsc.dbc.to_csv('games.csv')
Now that we have a csv file for our database, we can inspect it and see how well our program did at obtaining as much of the data as possible.
games_df = pd.read_csv('games.csv')games_df.info()
From the info above, we can see that there are some issues (‘Publisher’ was not collected for any game, definitely an oversight in the code), but overall the dataset appears to be fairly close to what we were expecting.
After cleaning and imputing we should have a pretty solid dataset we can use to answer our original questions.
games_df.head()
Not bad for writing a few scripts and then crossing our fingers for 11 hours.
Now that we have done this much work it only makes sense that we share it with the data science community. The best way to do this is to upload it to Kaggle.com, where it will be available to any data scientist who could benefit from it.
We navigate to https://www.kaggle.com/datasets, and click the ‘New Dataset’ button. This will open up a popup that will ask us to drag or upload our desired file. As usual, Kaggle makes it very easy for us to create our dataset.
After it uploads, we will be redirected to the landing page for our dataset. To make our dataset more available and give it a higher usability score, we make sure to go through and fill in as much information as possible.
Here is the link for the newly created dataset:
https://www.kaggle.com/jesneuman/pc-games
In this article, we walked through the process of creating a dataset from scratch using web scraping and API calls. These are two of the most common methods for gathering data, and having these skills will greatly increase your ability to gather insights and create recommendations.
Thanks so much for taking the time to read my article. I hope it was informative and concise. If not please let me know how I could improve it!
Please leave comments, questions, suggestions, or any feedback, as I really appreciate the opportunity to engage with others and to grow as a data scientist.
Disclaimer: Although I do attempt to write high-quality best-practices conforming code, I am in no way an expert and this code may not represent the ‘best’ way to implement this.
If you have any recommendations concerning any of the code, procedures, writing, etc. I would love to hear those as well.