Project: YugabyteDB (App Definition and Development / Database)
File: check-data-conforms-to-the-rules.md

title: Check the ingested COVIDcast data conforms to the rules linkTitle: Check staged data conforms to the rules headerTitle: Check that the values from the .csv files do indeed conform to the stated rules menu: v2.18: identifier: check-data-conforms-to-the-rules parent: ingest-the-covidcast-data weight: 30 type: docs The rules to which the values copied in from the `.csv` files must conform are described in the section . The form of each rule is the same for each of the three files. But sometimes, specific values that the rules govern are specific to the file. Each of the rules is tested, using the `assert` construct, in a single PL/pgSQL stored procedure, created by the script . The tests share these common declarations: ```plpgsql -- Each survey date (i.e. "timevalue") has exactly the same number of states (i.e. "geovalue"). -- Each state has the same number, of survey date values. expected_states constant text[] not null := array[ 'ak', 'al', 'ar', 'az', 'ca', 'co', 'ct', 'dc', 'de', 'fl', 'ga', 'hi', 'ia', 'id', 'il', 'in', 'ks', 'ky', 'la', 'ma', 'md', 'me', 'mi', 'mn', 'mo', 'ms', 'mt', 'nc', 'nd', 'ne', 'nh', 'nj', 'nm', 'nv', 'ny', 'oh', 'ok', 'or', 'pa', 'ri', 'sc', 'sd', 'tn', 'tx', 'ut', 'va', 'vt', 'wa', 'wi', 'wv', 'wy' ]; expectedstatecount constant int := cardinality(expected_states); actualstatesqry constant text not null := 'select arrayagg(distinct geovalue order by geo_value) from ?'; actual_states text[] not null := '{}'; expecteddates date[] not null := array[startsurvey_date]; actualdatesqry constant text not null := 'select arrayagg(distinct timevalue order by time_value) from ?'; actual_dates date[] not null := '{}'; expecteddatecount int not null := 0; names constant covidcast_names[] not null := ( select arrayagg((csvfile, stagingtable, signal)::covidcastnames) from covidcast_names); expectedtotalcount int not null := 0; r covidcast_names not null := ('', '', ''); d date not null := startsurveydate; t text not null := ''; n int not null := 0; b boolean not null := false; ``` The first several tests are applied, in turn, to each staging table. It is therefore convenient to implement them using dynamic SQL in a loop that iterates over the elements in the "names" array, thus: ```plpgsql foreach r in array names loop ... end loop; ``` Check that every value in the column is the same and that it is equal to the value that is specific to the source `csv` file's name. ``` execute replace('select distinct signal from ?', '?', r.staging_table) into t; assert t = r.signal, 'signal from '||r.staging_table||' <> "'||r.signal||'"'; ``` Check that every value in the column is the same and that it is equal to 'state': ``` execute 'select distinct geotype from

into t; assert t = 'state', 'geotype from '||r.stagingtable||' <> "state"'; ``` Check that every value in the column is the same and that it is equal to 'fb-survey': ``` execute 'select distinct datasource from '||r.stagingtable into t; assert t = 'fb-survey', 'datasource from '||r.stagingtable||' <> "fb-survey"'; ``` Check that every value in the columns `NULL`: ``` execute $$select distinct coalesce(direction, '<null>') from $$||r.staging_table into t; assert t = '<null>', 'direction from '||r.staging_table||' <> "<null>"'; ``` Check that the total number of rows is equal to "(number of distinct states)*(number of distinct survey dates)": ``` execute 'select count(*) from '||r.staging_table into n; assert n = expectedtotalcount, 'count from '||r.stagingtable||' <> expectedtotal_count'; ``` Check that there are 51 distinct values of , corresponding to the 51 actual states (including DC): ``` execute replace(actualstatesqry, '?', r.stagingtable) into actualstates; assert actualstates = expectedstates, 'actualstates <> expectedstates'; execute 'select count(distinct geovalue) from '||r.stagingtable into n; assert n = expectedstatecount, 'distinct state count per survey date from '||r.stagingtable||' <> expectedstate_count'; ``` Check that there are 50 distinct values of , corresponding to the values in the dense sequence that the download range, "2020-09-13 - 2020-11-01", specified: ``` execute replace(actualdatesqry, '?', r.stagingtable) into actualdates; assert actualdates = expecteddates, 'actualdates <> expecteddates'; execute 'select count(distinct timevalue) from '||r.stagingtable into n; assert n = expecteddatecount, 'distinct survey date count per state from '||r.stagingtable||' <> expecteddate_count'; ``` Because this is a `SELECT DISTINCT ... INTO` query, it will cause an error if the number of states per survey date isn't the same for each date. ``` execute ' with r as ( select timevalue, count(timevalue) as n from '||r.staging_table||' group by time_value) select distinct n from r' into n; assert n = expectedstatecount, 'distinct state count from '||r.stagingtable||' <> expectedstate_count'; ``` Because this is a `SELECT DISTINCT ... INTO` query, it will cause an error if the number of survey dates per state isn't the same for each date. ``` execute ' with r as ( select geovalue, count(geovalue) as n from '||r.staging_table||' group by geo_value) select distinct n from r' into n; assert n = expectedsurveydatecount, 'distinct state count from '||r.stagingtable||' <> expectedsurveydate_count'; ``` Check that the column is a number in the range 0..100. ``` execute ' select max(value) between 0 and 100 and min(value) between 0 and 100 from '||r.staging_table into b; assert b, 'max(value), min(value) from '||r.staging_table||' both < 100 FALSE'; ``` These tests need to check that sets of values in each of the three staging tables are the

They therefore use the `UNION ALL` of `EXCEPT` queries pattern to perform the test in a single `SELECT` that is expected to produce no rows. Each state has a unique numeric "code". The following `SELECT` confirms that this is the case. The "code" column can therefore be safely left out of the final "covidcastfbsurveyresults"_ table. ``` declare chkcodeandgeovalues constant text := $$ with a1 as ( select tochar(code, '90')||' '||geovalue as v from ?1), v1 as ( select v, count(v) as n from a1 group by v), a2 as ( select tochar(code, '90')||' '||geovalue as v from ?2), v2 as ( select v, count(v) as n from a2 group by v), a3 as ( select tochar(code, '90')||' '||geovalue as v from ?3), v3 as ( select v, count(v) as n from a3 group by v), v4 as (select v, n from v1 except select v, n from v2), v5 as (select v, n from v2 except select v, n from v1), v6 as (select v, n from v1 except select v, n from v3), v7 as (select v, n from v3 except select v, n from v1), r as ( select v, n from v4 union all select v, n from v5 union all select v, n from v6 union all select v, n from v6) select count(*) from r$$; begin execute replace(replace(replace(chkcodeandgeovalues, '?1', names[1].staging_table), '?2', names[2].staging_table), '?3', names[3].staging_table ) into n; assert n = 0, '(code, geo_value) tuples from the three staging tables disagree'; end; ``` ``` declare chkputativepks constant text := ' with v1 as ( select geovalue, timevalue from ?1 except select geovalue, timevalue from ?2), v2 as ( select geovalue, timevalue from ?2 except select geovalue, timevalue from ?1), v3 as ( select geovalue, timevalue from ?1 except select geovalue, timevalue from ?3), v4 as ( select geovalue, timevalue from ?3 except select geovalue, timevalue from ?1), v5 as ( select geovalue, timevalue from v1 union all select geovalue, timevalue from v2 union all select geovalue, timevalue from v3 union all select geovalue, timevalue from v4) select count(*) from v5'; begin execute replace(replace(replace(chkputativepks, '?1', names[1].staging_table), '?2', names[2].staging_table), '?3', names[3].staging_table) into n; assert n = 0, 'pk values from ' || replace(replace(replace('?1, ?2, ?3', '?1', names[1].staging_table), '?2', names[2].staging_table), '?3', names[3].staging_table) || ' do not line up'; end; ``` Copy the code and paste it into `ysqlsh`. Then do this: ```plpgsql call assertassumptionsok( startsurveydate => to_date('2020-09-13', 'yyyy-mm-dd'), endsurveydate => to_date('2020-11-01', 'yyyy-mm-dd')); ``` It will finish silently. You can provoke and `assert` failure by invoking it with, say: ``` endsurveydate => to_date('2020-11-02', 'yyyy-mm-dd') ``` The error is reported thus: ``` count from cmntysymptoms <> expectedtotal_count ``` The invocation is included in .