Package: covid-data
COVID Data LoaderThis project is part of theCore Data COVIDproject. Here are the tools needed for:Create the databasePopulate the databaseQuery the dataTable of contentsInstallation 📥Library 📖CLI 🤖Configuration ⚙Usage as CLI 🎛️loadcsvscrapPopulate DB from backupContributing ♥Add new scrappersAdd new commandsAs libraryLicenseInstallation 📥Library 📖To use this project as a library, you have to install it withpip:pip install covid-dataCLI 🤖To use it as CLI, you still have to install it withpip. After that, you can run the commands from a terminal:covid-data --helpConfiguration ⚙The project looks for the following environment variables to configure several parts:POSTGRES_USER: Postgres usernamePOSTGRES_PASS: Postgres passwordPOSTGRES_HOST: Postgres hostPOSTGRES_PORT: Postgres portPOSTGRES_DB: Postgres databaseCAGEDATA_API_KEY: OpenCageData API KeyYou can create the OpenCage data ley from theirwebsite. The package uses it to enrich and normalize places.Usage as CLI 🎛️To populate the database, you may use the commands the library provides.The first step is to create the database with the schema availablehere. The database structure is the following one:Once the package finishes creating the database, you can populate it from a CSV with the command:❯covid-dataloadcsv/path/to/confirmed.csv,/path/to/dead.csv,/path/to/recovered.csv-tf-oFor getting help on how to use the command, run:❯covid-data--help
Usage:covid-data[OPTIONS]COMMAND[ARGS]...

Options:--helpShowthismessageandexit.

Commands:loadsLoadsFILESasCSVdata.scrapScrapcasesofchosenCOUNTRY.loadcsv❯covid-dataloadcsv--help
Usage:covid-dataloadcsv[OPTIONS]FILESLoadsFILESasCSVdata.Ifyouwanttoloadseveralfiles,eachfileshouldbeseparatedbyacomma

Options:-t,--typeTEXTTypeofcasescontainedineachfile,separatedbycomma.Leaveblankifusing--type-in-file-tf,--type-in-fileSetthistotrueifthefilenamesare<case_type>.csvBeing<case_type>oneofconfirmed,recoveredordead-o,--optimizeSettotruetoskiplinesforplacesthathasmorecasesthancolumnsontheCSV--helpShowthismessageandexit.This command allows you to populate the database with one or more CSV files. Each file may only contain one case type.You have to provide the paths separated by commas.As well as with the file names, each file case type has to be separated by commas. Pass the type list under the argument--typeandin the same orderas the files.Another option is not using the--typeargument. In that case, you have to name the files according to the case type each of them contains (e.g.,recovered.csv). If you choose this alternative, you have to set the flag--type-in-file.The last flag is--optimize. If set, for each row, the CLI is going to get the country from that row. Then check the number of cases already saved for that country (with the same type as the file). If the country has the same or more number of cases as columns are in the CSV, it will skip that row.An example of how to run the command for loading all CSVs:❯covid-dataloadcsv/path/to/confirmed.csv,/path/to/dead.csv,/path/to/recovered.csv-tf-oIn that example, I'm using the file name to specify the case type contained. I'm also setting the optimization flag.scrapWith this command, you can scrap the cases of a specific country, allowing you to extend the data from that country.The command usage is as follows:❯covid-datascrap--help
Usage:covid-datascrap[OPTIONS]COUNTRYScrapcasesofchosenCOUNTRY.Tocheckavailablecountriestoscrapuse--check

Options:--checktoUsethistocheckavailablecountries--start-dateTEXTDatetostartscrapingcasesfrom,informatDD/MM/YYYY--helpShowthismessageandexit.There is one required argument, the country to scrap. If you want to know which countries are available, you can pass whatever value toCOUNTRYand set the flag--check:❯covid-datascrapXXX--check
Availablecountriesare:FranceSpainOnce you know which country to scrap, you can launch the command:❯covid-datascrapSpain
Fetchingcasesforprovince1/20
Fetchingcasesforprovince2/20
Fetchingcasesforprovince3/20
...You can set the start date with the argument--start-date. The date format required isDD/MM/YYYY❯covid-datascrapFrance--start-date01/08/2021Populate DB from backupYou may find attached to eachrelease, acovid-data.sqlfile. That script allows you to load into an SQL database the most recent data.If you run that script on thecovid-datadatabase, it will populate it with the necessary data for running the app.Contributing ♥You can become part of this project by proposing (and even implementing) enhancements, new commands, or new scrappers.Add new scrappersAdding a new scrapper is as simple as creating a new file named<country_name>.pyin the foldercovid_data/scrappersand make that file expose a function with the following signature:defscrap(db_engine:psycopg2._psycopg.connection,start_date:datetime.datetime)That function has to be in charge of scrapping, processing, formatting as well as saving new cases for the requested country.Add new commandsAdding a new command is also a simple process. Place a file named<command_name>.pyin the foldercovid_data/commandsand expose a functionmainwith the neededclickdecorators.A mandatory decorator is@click.command("command-name"), to avoid overwriting commands as all functions are namedmain.As libraryThe potential of this library is offered by thedbsubmodule, which you can find onCODE.LicenseThe Unlicense