Project: YugabyteDB (App Definition and Development / Database)
File: join-the-staged-data.md

title: Join the staged COVIDcast data into a single table linkTitle: Join the staged data into a single table headerTitle: Join the staged COVIDcast data into the "covidcastfbsurvey_results" table menu: v2.18: identifier: join-the-staged-data parent: ingest-the-covidcast-data weight: 40 type: docs The section concluded by showing that the three staging tables, one for each `.csv` file, have the same primary key whose columns have the meaning "(surveydate, state)". This means that the data in all of the tables can be joined using this primary key into a single table. It showed, further, that the three "payload" columns in each of the staging tables, which there have the same names"value", "stderr", and "samplesize"can be carried over by the join into columns whose names reflect their origin and therefore their ultimate meaning, thus: ``` VALUE # Will go to MASKWEARINGPCT, SYMPTOMSPCT, or CMNTYSYMPTOMS_PCT. STDERR # Will go to MASKWEARINGSTDERR, SYMPTOMSSTDERR, or CMNTYSYMPTOMS_STDERR. SAMPLESIZE # Will go to MASKWEARINGSAMPLESIZE, SYMPTOMSSAMPLESIZE, or CMNTYSYMPTOMSSAMPLE_SIZE. ``` Here is the statement that creates the table: ```plpgsql drop table if exists covidcastfbsurvey_results cascade; create table covidcastfbsurvey_results( survey_date date not null, state text not null, maskwearingpct numeric not null, maskwearingstderr numeric not null, maskwearingsample_size int not null, symptoms_pct numeric not null, symptoms_stderr numeric not null, symptomssamplesize int not null, cmntysymptomspct numeric not null, cmntysymptomsstderr numeric not null, cmntysymptomssample_size int not null, constraint covidcastfbsurveyresultspk primary key (state, survey_date), constraint covidcastfbsurveyresultschkmaskwearingpct check(maskwearing_pct between 0 and 100), constraint covidcastfbsurveyresultschksymptomspct check(symptoms_pct between 0 and 100), constraint covidcastfbsurveyresultschkcmntysymptomspct check(cmntysymptoms_pct between 0 and 100), constraint covidcastfbsurveyresultschkmaskwearingstderr check(maskwearing_stderr > 0), constraint covidcastfbsurveyresultschksymptomsstderr check(symptoms_stderr > 0), constraint covidcastfbsurveyresultschkcmntysymptomsstderr check(cmntysymptoms_stderr > 0), constraint covidcastfbsurveyresultschkmaskwearingsamplesize check(maskwearingsample_size > 0), constraint covidcastfbsurveyresultschksymptomssamplesize check(symptomssample_size > 0), constraint covidcastfbsurveyresultschkcmntysymptomssamplesize check(cmntysymptomssample_size > 0) ); ``` The constraints honor the same principle that the use of the procedure does: to check, explicitly, that all of the documented rules about the source data hold so that the results of applying the linear regression analysis can be trusted. The procedure creates and populates the "covidcastfbsurveyresults" table. It uses dynamic SQL so that it can use the names for the staging tables that are defined in the "covidcastnames" table. This is the logic: ``` declare maskwearersname text not null := (select stagingtable from covidcastnames where stagingtable = 'maskwearers'); symptomsname text not null := (select stagingtable from covidcastnames where stagingtable = 'symptoms'); cmntysymptomsname text not null := (select stagingtable from covidcastnames where stagingtable = 'cmntysymptoms'); stmt text not null := ' insert into covidcastfbsurvey_results( survey_date, state, maskwearingpct, maskwearingstderr, maskwearingsample_size, symptomspct, symptomsstderr, symptomssamplesize, cmntysymptomspct, cmntysymptomsstderr, cmntysymptomssample_size) select timevalue, geovalue, m.value, m.stderr, round(m.sample_size), s.value, s.stderr, round(s.sample_size), c.value, c.stderr, round(c.sample_size) from ?1 as m inner join ?2 as s using (timevalue, geovalue) inner join ?3 as c using (timevalue, geovalue)'; begin execute replace(replace(replace(stmt, '?1', maskwearersname), '?2', symptoms_name), '?3', cmntysymptomsname); ... end; ``` Invoke the join into the final table manually, thus: ```plpgsql call xformtocovidcastfbsurvey_results(); ``` The script creates the procedure "xformtocovidcastfbsurveyresults()"_ and then invokes it like this: ```plpgsql do $body$ begin call assertassumptionsok( startsurveydate => to_date('2020-09-13', 'yyyy-mm-dd'), endsurveydate => to_date('2020-11-01', 'yyyy-mm-dd')); call xformtocovidcastfbsurvey_results(); end; $body$; ``` Notice that if aborts with an assert failure, then "crcovidcastfbsurveyresults()" will not be called.