Access Free Google Cloud Public Dataset with Python | by Joe T. Santhanavanich | Towards Data Science

Good news for Data Scientists! Google is making a hosted repository of public datasets, like Johns Hopkins Center for Systems Science and Engineering (JHU CSSE), the Global Health Data from the World Bank, and OpenStreetMap data, free to access and query through their COVID-19 Public Dataset Program to aid researchers, data scientists, and analysts in the effort to combat COVID-19. These datasets remove barriers and provide access to critical information quickly and easily, eliminating the need to search for and onboard large data files. Researchers can access the datasets from within the Google Cloud Console, along with a description of the data and sample queries to advance research. All data we include in the program will be public and freely available. *The program will remain in effect until September 15, 2020. [1]
Storing, managing, updating data myself with the database is sometimes boring and pricey. With Google Public Dataset, they are taking care of these boring parts for me. And for pricing, before the COVID-19 pandemic, it has a free-tier limit of 1 TB of querying per month (Well, really a lot of free data already!) To support the pandemic situation, there is no limit anymore!! So I could happily concentrate on the FUN parts: data analysis, sexy data visualization, developing web applications, etc.
This article will share with you step-by-step guides as follows:
Enable Google Cloud BigQuery API.Create Service Account for BigQuery API to grant permission to your Python script.Explore Dataset with BigQuery Web Interface.Access Dataset with Python and import it to the Pandas dataframe.
Enable Google Cloud BigQuery API.
Create Service Account for BigQuery API to grant permission to your Python script.
Explore Dataset with BigQuery Web Interface.
Access Dataset with Python and import it to the Pandas dataframe.
So, let’s get started!
To be able to access the dataset, you should register for the Google Cloud account. After you have registered or if you already have an account, please go to your Google Cloud console and then activate your Google Cloud BigQuery API (link). After you click on “Activate” the API, then you can see the confirmed “API enabled” as the figure below.
In order to give BigQuery API access to your machine with Python or other programming languages later, you need a new Google service account key by the following steps. (If you only need to access through the web interface, you may skip these steps)
Go to your Google API Credentials page to create the API key for this BigQuery API (link).
Select Service Account permission role as “BigQuery Admin”
Then, click on Create Key and select the JSON type. Please, make sure that you keep this file securely or someone might have access to your API account too. We will use this file again in the last step.
Users can retrieve or query the Google Public dataset using SQL Query syntax. At the starting point, you may try observing what dataset you have access to and test the SQL query through the Google BigQuery Web interface with three following steps:
Explore available data resources. Select one of them.Type your query syntax in the “Query Editor” window. In this step, you gonna need a basic knowledge of SQL to query (search) the dataset. If you have never used it before, please check the online document about it. Several of them are online such as W3.Explore your Query Results.
Explore available data resources. Select one of them.
Type your query syntax in the “Query Editor” window. In this step, you gonna need a basic knowledge of SQL to query (search) the dataset. If you have never used it before, please check the online document about it. Several of them are online such as W3.
Explore your Query Results.
To access the BigQuery API with Python, install the library with the following command:
pip install --upgrade google-cloud-bigquery
Create your project folder and put the service account JSON file in the folder. Then, create a Python file and edit it with the editor you like. The first thing you need is to set the OS environment of Google Application point to your JSON key directory which you can do by using os.environ["xxx"]="yyy"
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="Your json directory"
Then, you can start your BigQuery client and start a query with this command:
client = bigquery.Client()          # Start the BigQuery ClientQUERY = ('SELECT ...')              # Input Query Syntaxquery_job = client.query(QUERY)     # Start Query API Requestquery_result = query_job.result()   # Get Query Resultdf = query_result.to_dataframe()    # Save the Query to Dataframe
For example, you can follow the full Python script below to query the top 20 COVID-19 confirmed and death cases in USA order by the later date and the higher confirmed cases from the New York Times COVID-19 database.
So, that’s about it! Congratulation, now you access the dataset, have fun with your new dataframe!! ✌🎉🎉
This article covers basic steps on how to use and access the dataset with Google Cloud BigQuery service with WebUI and Python. Please note that this article covers only the beginner steps and there are many more features and interfaces to access and play around with this service. You may check the full BigQuery document here.
I hope you like this article and found it useful for your daily work or projects. Feel free to leave me a message if you have any questions or comments.
About me & Check out all my blog contents: Link
Be Safe and Healthy! 💪
Thank you for Reading. 📚
[1] Chad W. Jennings, COVID-19 public dataset program: Making data freely accessible for better public outcomes (Mar, 30 2020), Google Cloud Data Analytics