# Using the Data Quality Dashboard to Improve the EHDEN Network
## 2 Materials and Methods
### Data Quality Dashboard Comparisons  
At the end of the entire data conversion process, when a DP achieved an OMOP CDM-formatted database they and the taskforce were confident in, both the first and last DQD reports were collected. The data quality checks comprising these reports were then organized into four outcome categories: a _pass_ indicated that the percentage of failing rows for a given quality check was below the threshold specified in the DQD tool, a _fail_ indicated that the percentage of failing rows for a given quality check was above the threshold  
Figure 2: The Extract, Transform, & Load (ETL) to Common Data Model (CDM) to Data Quality Dashboard (DQD) Feedback Loop used in the European Health Data & Evidence Network (EHDEN) COVID-19 Rapid Collaboration Call.  
specified, an _error_ indicated that there was an error in the SQL and no result for the given quality check was returned, and _no data_ indicated that there was no data in the database that satisfied the criteria for a given quality check.  
\[\frac{\text{\# of passed checks}}{\text{(\# of passed checks}+\text{\# of failed checks}+\text{\# of check errors)}}\times 100 \tag{1}\]  
After organizing the data quality checks by outcome, the results from the first DQD report were compared with the results from the last DQD report. Initially, a simple check pass percentage (e.g., 95% of the DQD checks passed) was obtained by DP and time (first or last) using Equation (1) to determine if the percent of data quality checks passed in the last run of the DQD was higher than the percent of data quality checks passed in the first run of the DQD. It was then observed that almost all DPs ran a different number of quality checks at first versus at last. Upon inspection it was revealed that this phenomenon was due to a few factors: the iterative ETL process, the automatic nature of the tool, and the fact that DPs removed checks with no data to support them from the calculation. For example, many DPs entered the Data Conversion to Analysis Pipeline beginning with the information they believed was easiest to standardize, typically patient demographics and patient diagnoses. Once they achieved the first standardized database containing this information, many ran the DQD with only a subset of the total checks turned on. These tended to be checks that evaluated conformance to the model and completeness of standard vocabulary mapping. As they iterated through the process, the DPs continued to turn on more checks, usually those evaluating plausibility. To account for this, and to enable more appropriate comparisons, we conducted a second analysis limiting to only those data quality checks that were run both during the first DQD and the last DQD.  
As mentioned above, the DQD determines a pass or a fail by comparing the percentage of rows that fail a check to a pre-specified threshold. While this gives an immediate idea of how well a quality check performed in a database, it only provides part of the story. The binary nature of assigning a pass or fail label overlooks the notion of improvement. Since we were interested in how well the DPs improved the quality of their database, we quantified, by DP, the number of checks that decreased the percent failing rows from first to last DQD run in addition to the passes and fails.  
To gain insight into which types of checks failed most often in the beginning compared to the end of the data conversion process we also stratified the percent of passing checks by check type and time (first or last).