Project: YugabyteDB (App Definition and Development / Database)
File: stage-the-csv-files.md

title: Copy the .csv files to staging tables linkTitle: Copy the .csv files to staging tables headerTitle: Copy each of the COVIDcast data .csv files to a dedicated staging table description: Copy each of the COVIDcast data .csv files to a dedicated staging table for analysis using YSQL functions for linear regression analysis menu: v2.18: identifier: stage-the-csv-files parent: ingest-the-covidcast-data weight: 20 type: docs It isn't essential to do this. It's very unlikely that the names of the objects that are created for your COVIDcast case study project will collide with those of any objects that you might already have (and might care about). But the scripts that create them follow usual practice and start by dropping them. So there is a riskat least in principle. Anyway, it makes sense to create a dedicated ordinary user for this project for all the usual reasons. For example, when all the project objects are owned by a user that owns no other objects (and are all in the same schema), it's easy to list all of the objects (and only these). Each of the `.csv` files that you downloaded has its unique name; each will be imported, using the `\copy` meta-command "as is" into a dedicated staging table whose name reflects the source `.csv` file; and each has its own value in the column. Because these names will be needed in various places within the overall set of scripts, they are defined in the "covidcastnames"_ table, thus: ```plpgsql drop table if exists covidcast_names cascade; create table covidcastnames(csvfile text primary key, staging_table text not null, signal text not null); -- Each of these files contains 50 days of observations. insert into covidcastnames(csvfile, staging_table, signal) values ('csv-files/covidcast-fb-survey-smoothed'|| 'wearingmask' ||'-2020-09-13-to-2020-11-01.csv', 'maskwearers', 'smoothedwearing_mask'), ('csv-files/covidcast-fb-survey-smoothed'|| 'cli' ||'-2020-09-13-to-2020-11-01.csv', 'symptoms', 'smoothedcli'), ('csv-files/covidcast-fb-survey-smoothed'|| 'hhcmntycli' ||'-2020-09-13-to-2020-11-01.csv', 'cmntysymptoms', 'smoothedhhcmnty_cli'); create unique index covidcastnamesstagingtableunq on covidcastnames(stagingtable); create unique index covidcastnamessignalunq on covidcastnames(signal); ``` This code is included directly in so you don't need to save it to a file. Because the three `.csv` files have the same format as each other (see the section ), each of the three staging tables has the same column names and data types. Dynamic SQL is used to create these tables to avoid repetition of code (i.e. to keep the code size as small as possible), and to bring optimal "single point of definition" maintainability. This code is implemented by the procedure crstagingtables() created by the script. After creating the procedure, run it like this: ```plpgsql call crstagingtables(); ``` This invocation is included in . Now take an inventory of your tables with the `\d`

You should see something like this, with the names of the user and schema that you created in place of "u1": ``` Schema | Name | Type | Owner --+--+-+- u1 | cmnty_symptoms | table | u1 u1 | covidcast_names | table | u1 u1 | mask_wearers | table | u1 u1 | symptoms | table | u1 ``` The SQL statement is designed to ingest data from a file "as is". However, its simple use requires that the to-be-read (or to-be-written) file resides server-side on the local filesystem of the YB-TServer that you connect to. If you specify `stdin` as the argument of `COPY FROM`, then these input and output channels are defined client-side in the environment of the client where you run `ysqlsh`. This sounds promising. But the snag is that you must include the text of the `COPY FROM` statement at the start of the file that contains the data that you intend to ingest. This is described in the section of the documentation for the `COPY` statement. The preferred option for the present case study, because is chosen for running all the SQL statements, is to use the meta-command. Because the three `.csv` files all have the same format, as do their three dedicated staging tables, the `copy` command will have the same form for each of its invocations, thus: ``` \copy <staging table> from <csv file> with (format 'csv', header true) ``` This is another case where a procedure that reads the names from the "covidcastnames" table avoids repetition of code and brings optimal "single point of definition" maintainability. However, there's particular design dilemma to confront. The `\copy` meta-command cannot be run from a stored procedure written in PL/pgSQL. Therefore a function_ is used that will return the text of the `\copy` meta-command. You execute the appropriately parameterized function, spool its output to a `.sql` script and start that script. Create the function with the script. Test it manually like this: ```plpgsql \t on select crcopyfrom_scripts(1); \t off ``` This is the result: ``` \copy symptoms from 'csv-files/covidcast-fb-survey-smoothed_cli-2020-09-13-to-2020-11-01.csv' with (format 'csv', header true); ``` Execute it for each to-be-ingested `.csv` file. ```plpgsql \t on \o copyfromcsv.sql select crcopyfrom_scripts(1); \o \i copyfromcsv.sql \o copyfromcsv.sql select crcopyfrom_scripts(2); \o \i copyfromcsv.sql \o copyfromcsv.sql select crcopyfrom_scripts(3); \o \i copyfromcsv.sql \t off ``` This code is included directly in so you don't need to save it to a file. This patternusing a PL/pgSQL function to create a `sql` script whose details reflect the present content of database tables and then executing it from `ysqlsh` is a useful generic technique. Finally, check visually that each staging table has the same number of rows: ```plpgsql select (select count(*) from maskwearers) as "maskwearers count", (select count(*) from symptoms) as "symptoms count", (select count(*) from cmntysymptoms) as "cmntysymptoms count"; ``` This is the result: ``` maskwearers count | symptoms count | cmntysymptoms count --+-+- 2550 | 2550 | 2550 ``` The value 2,550 is the product of the 51 states (including DC) and the 50 days on which survey data were collected.