# How to work with Johns Hopkins University COVID-19 Data in MongoDB Atlas

## TL;DR

Our MongoDB Cluster is running in version 7.0.3.

You can connect to it using MongoDB Compass, the Mongo Shell, SQL or any MongoDB driver supporting at least MongoDB 7.0
with the following URI:

``` none
mongodb+srv://readonly:readonly@covid-19.hip2i.mongodb.net/covid19
```

> `readonly` is the username and the password, they are not meant to be replaced.

## News

### November 15th, 2023

- John Hopkins University (JHU) has stopped collecting data as of March 10th, 2023.
- Here is JHU's GitHub repository.
- First data entry is 2020-01-22, last one is 2023-03-09.
- Cluster now running on 7.0.3
- Removed the database `covid19jhu` with the raw data. Use the much better database `covid19`.
- BI Tools access is now disable.

### December 10th, 2020

- Upgraded the cluster to 4.4.
- Improved the python data import script to calculate the daily values using the existing cumulative values with
  an Aggregation Pipeline.
    - confirmed_daily.
    - deaths_daily.
    - recovered_daily.

### May 13th, 2020

- Renamed the field "city" to "county" and "cities" to "counties" where appropriate. They contain the data from the
  column "Admin2" in JHU CSVs.

### May 6th, 2020

- The `covid19` database now has 5 collections. More details in
  our README.md.
- The `covid19.statistics` collection is renamed `covid19.global_and_us` for more clarity.
- Maxime's Charts are now
  using the `covid19.global_and_us` collection.
- The dataset is updated hourly so any commit done by JHU will be reflected at most one hour later in our cluster.

## Table of Contents

- Introduction
- The MongoDB Dataset
- Get Started
    - Explore the Dataset with MongoDB Charts
    - Explore the Dataset with MongoDB Compass
    - Explore the Dataset with the MongoDB Shell
    - Accessing the Data with Java
    - Accessing the Data with Node.js
    - Accessing the Data with Python
    - Accessing the Data with Golang
    - Accessing the Data with Google Colaboratory
    - Accessing the Data with Business Intelligence Tools
    - Accessing the Data with any SQL tool
    - Take a copy of the data
- Wrap up
- Sources

## Introduction

As the COVID-19 pandemic has swept the globe, the work of JHU (Johns Hopkins University) and
its COVID-19 dashboard has become vitally important in keeping people informed
about the progress of the virus in their communities, in their countries, and in the world.

JHU not only publishes their dashboard,
but they make the data powering it freely available for anyone to use.
However, their data is delivered as flat CSV files which you need to download each time to then query. We've set out to
make that up-to-date data more accessible so people could build other analyses and applications directly on top of the
data set.

We are now hosting a service with a frequently updated copy of the JHU data in MongoDB Atlas, our database in the cloud.
This data is free for anyone to query using the MongoDB Query language and/or SQL. We also support
a variety of BI tools directly, so you can query the data with Tableau,
Qlik and Excel.

With the MongoDB COVID-19 dataset there will be no more manual downloads and no more frequent format changes. With this
data set, this service will deliver a consistent JSON and SQL view every day with no
downstream ETL required.

None of the actual data is modified. It is simply structured to make it easier to query by placing it within
a MongoDB Atlas cluster and by creating some convenient APIs.

## The MongoDB Dataset

All the data we use to create the MongoDB COVID-19 dataset comes from the JHU dataset. In their
turn, here are the sources they are using:

- the World Health Organization,
- the National Health Commission of the People's Republic of China,
- the United States Centre for Disease Control,
- the Australia Government Department of Health,
- the European Centre for Disease Prevention and Control,
- and many others.

You can read the full list on their GitHub repository.

Using the CSV files they provide, we are producing two different databases in our cluster.

- `covid19jhu` contains the raw CSV files imported with
  the mongoimport tool,
- `covid19` contains the same dataset but with a clean MongoDB schema design with all the good practices we are
  recommending.

Here is an example of a document in the `covid19` database:

``` javascript
{
  "_id" : ObjectId("5e957bfcbd78b2f11ba349bf"),
  "uid" : 312,
  "country_iso2" : "GP",
  "country_iso3" : "GLP",
  "country_code" : 312,
  "state" : "Guadeloupe",
  "country" : "France",
  "combined_name" : "Guadeloupe, France",
  "population" : 400127,
  "loc" : {
    "type" : "Point",
    "coordinates" :  -61.551, 16.265 ]
  },
  "date" : ISODate("2020-04-13T00:00:00Z"),
  "confirmed" : 143,
  "deaths" : 8,
  "recovered" : 67
}
```

The document above was obtained by joining together the file `UID_ISO_FIPS_LookUp_Table.csv` and the CSV files time
series you can find
in [this folder.

Some fields might not exist in all the documents because they are not relevant or are just not provided
by JHU. If you want more details, run a schema analysis
with MongoDB Compass on the different collections available.

If you prefer to host the data yourself, the scripts required to download and transform the JHU data are
open-source. You
can view them and instructions for how to use them on our GitHub repository.

In the `covid19` database, you will find 5 collections which are detailed in
our GitHub repository README.md file.

- metadata
- global (the data from the time series global files)
- us_only (the data from the time series US files)
- global_and_us (the most complete one)
- countries_summary (same as global but countries are grouped in a single doc for each date)

## Get Started

You can begin exploring the data right away without any MongoDB or programming experience
using MongoDB Charts
or MongoDB Compass.

In the following sections, we will also show you how to consume this dataset using the Java, Node.js and Python drivers.

We will show you how to perform the following queries in each language:

- Retrieve the last 5 days of data for a given place,
- Retrieve all the data for the last day,
- Make a geospatial query to retrieve data within a certain distance of a given place.

### Explore the Dataset with MongoDB Charts

With Charts, you can create visualisations of the data using any of the
pre-built graphs and charts. You can
then arrange this into a unique dashboard,
or embed the charts in your pages or blogs.

:charts]{url=https://charts.mongodb.com/charts-open-data-covid-19-zddgb id=60da4f45-f168-4266-8264-d37ce88ff9fa
theme=light autorefresh=3600}

> If you want to create your own MongoDB Charts dashboard, you will need to set up your
> own [Free MongoDB Atlas cluster and import the dataset in your cluster using
> the import scripts or
> use `mongoexport & mongoimport` or `mongodump & mongorestore`. See this section for more
> details: Take a copy of the data.

### Explore the Dataset with MongoDB Compass

Compass allows you to dig deeper into the data using
the MongoDB Query Language or via
the Aggregation Pipeline visual editor. Perform a range of
operations on the
data, including mathematical, comparison and groupings.
Create documents that provide unique insights and interpretations. You can use the output from your pipelines
as data-sources for your Charts.

For MongoDB Compass or your driver, you can use this connection string.

``` 
mongodb+srv://readonly:readonly@covid-19.hip2i.mongodb.net/covid19
```

### Explore the Dataset with the MongoDB Shell

Because we store the data in MongoDB, you can also access it via
the MongoDB Shell or
using any of our drivers. We've limited access to these collections to 'read-only'.
You can find the connection strings for the shell and Compass below, as well as driver examples
for Java, Node.js,
and Python to get you started.

``` shell
mongo "mongodb+srv://covid-19.hip2i.mongodb.net/covid19"  --username readonly --password readonly
```

### Accessing the Data with Java

Our Java examples are available in
our Github Repository Java folder.

You need the three POJOs from
the Java Github folder
to make this work.

### Accessing the Data with Node.js

Our Node.js examples are available in
our Github Repository Node.js folder.

### Accessing the Data with Python

Our Python examples are available in
our Github Repository Python folder.

### Accessing the Data with Golang

Our Golang examples are available in
our Github Repository Golang folder.

### Accessing the Data with Google Colaboratory

If you have a Google account, a great way to get started is with
our Google Colab Notebook.

The sample code shows how to install pymongo and use it to connect to the MongoDB COVID-19 dataset. There are some
example queries which show how to query the data and display it in the notebook, and the last example demonstrates how
to display a chart using Pandas & Matplotlib!

If you want to modify the notebook, you can take a copy by selecting "Save a copy in Drive ..." from the "File" menu,
and then you'll be free to edit the copy.

### Accessing the Data with Business Intelligence Tools

You can get lots of value from the dataset without any programming at all. We've enabled
the Atlas BI Connector (not anymore, see News section), which exposes
an SQL interface to MongoDB's document structure. This means you can use data analysis and dashboarding tools
like Tableau, Qlik Sense,
and even MySQL Workbench to analyze, visualise and extract understanding
from the data.

Here's an example of a visualisation produced in a few clicks with Tableau:

Tableau is a powerful data visualisation and dashboard tool, and can be connected to our COVID-19 data in a few steps.
We've written a short tutorial
to get you up and running.

### Accessing the Data with any SQL tool

As mentioned above, the Atlas BI Connector is activated (not anymore, see News section), so you can
connect any SQL tool to this cluster using the following connection information:

- Server: covid-19-biconnector.hip2i.mongodb.net,
- Port: 27015,
- Database: covid19,
- Username: readonly or readonly?source=admin,
- Password: readonly.

### Take a copy of the data

Accessing *our* copy of this data in a read-only database is useful, but it won't be enough if you want to integrate it
with other data within a single MongoDB cluster. You can obtain a copy of the database, either to use offline using a
different tool outside of MongoDB, or to load into your own MongoDB instance. `mongoexport` is a command-line tool that
produces a JSONL or CSV export of data stored in a MongoDB instance. First, follow
these instructions to install the MongoDB Database Tools.

Now you can run the following in your console to download the metadata and global_and_us collections as jsonl files in
your current directory:

``` bash
mongoexport --collection='global_and_us' --out='global_and_us.jsonl' --uri="mongodb+srv://readonly:readonly@covid-19.hip2i.mongodb.net/covid19"
mongoexport --collection='metadata' --out='metadata.jsonl' --uri="mongodb+srv://readonly:readonly@covid-19.hip2i.mongodb.net/covid19"
```

> Use the `--jsonArray` option if you prefer to work with a JSON array rather than a JSONL file.

Documentation for all the features of `mongoexport` is available on
the MongoDB website and with the command `mongoexport --help`.

Once you have the data on your computer, you can use it directly with local tools, or load it into your own MongoDB
instance using mongoimport.

``` bash
mongoimport --collection='global_and_us' --uri="mongodb+srv://:@.mongodb.net/covid19" global_and_us.jsonl
mongoimport --collection='metadata' --uri="mongodb+srv://:@.mongodb.net/covid19" metadata.jsonl
```

> Note that you cannot run these commands against our cluster because the user we gave you (`readonly:readonly`) doesn't
> have write permission on this cluster.
> Read our Getting Your Free MongoDB Atlas Cluster blog post if you want to know more.

Another smart way to duplicate the dataset in your own cluster would be to use `mongodump` and `mongorestore`. Apart
from being more efficient, it will also grab the indexes definition along with the data.

``` bash
mongodump --uri="mongodb+srv://readonly:readonly@covid-19.hip2i.mongodb.net/covid19"
mongorestore --drop --uri=""
```

## Wrap up

We see the value and importance of making this data as readily available to everyone as possible, so we're not stopping
here. Over the coming days, we'll be adding a GraphQL and REST API, as well as making the data available within Excel
and Google Sheets.

We've also launched an Atlas credits program for
anyone working on detecting, understanding, and stopping the spread of COVID-19.

If you are having any problems accessing the data or have other data sets you would like to host please contact us
on the MongoDB community. We would also love to showcase any services you build on top
of this data set. Finally please send in PRs for any code changes you would like to make to the examples.

You can also reach out to the authors
directly (Aaron Bassett, Joe Karlsson, Mark Smith,
and Maxime Beugnet) on Twitter.

## Sources

- MongoDB Open Data COVID-19 GitHub repository
- JHU Dataset on GitHub repository