Making room for implicit error: a visualization approach to managing data discrepancy
By: Nina Mccurdy
Tags: Global Health, Data Visualization, Uncertainty

How do we resolve this? How can we capture and incorporate expert knowledge about discrepancies in data in order to increase data quality, synchronize knowledge across experts, and perhaps even begin to analyze the discrepancies themselves?

We believe visualization can help!

Background

This research was based in a visualization collaboration with global health experts who were working to combat the Zika virus in countries across Latin America and the Caribbean. The examples above are an attempt to convey the revelation that snuck up on us, the visualization researchers, in the early stages of the project. Over and over again, we’d show our collaborators a choropleth of their data, and they’d say, yeah, but this country does X and so it really shouldn’t look like that.

As we dug deeper, we came to understand that the data we were visualizing was laced with discrepancies, rendering visualization effectively useless for analysis. It didn’t matter what clever new visualization techniques we came up with, it wasn’t going to help our collaborators with their decision-making.

The exciting piece was: 1) that our domain collaborators had intimate and extensive knowledge of these data discrepancies, and 2) that showing them choropleths of the original data was helping to trigger and organize this knowledge.

So while visualization couldn’t support their analysis, it could support the externalization and incorporation of expert knowledge about data discrepancies.

Based on this revelation, we pivoted the focus of our research to better understanding expert knowledge about discrepancies — which led to our notion of implicit error— and to understanding how to leverage visualization to capture implicit error and incorporate it back into the visual analysis pipeline.

Implicit Error

Ok, so what do we mean by implicit error?

The concept of measurement error is fundamental to experimental and observational science — it is the estimated difference between a measured value, and the true value as it exists in the world.

Grounded in our experience working with global health experts, we use the term implicit error to describe measurement error that is believed to be inherent to a dataset, assumed to be present and prevalent, but is not explicitly defined or accounted for. Implicit error exists as tacit knowledge in the minds of domain experts, it is perceived as effectively unquantifiable, and it is accounted for subjectively during expert interpretation of the data.

implicit error: measurement error that is believed to be inherent to a dataset, assumed to be present and prevalent, but is not explicitly defined or accounted for.

Where does implicit error come from?

In the context of the Zika outbreak, official data is generated through a sequence of stages, from detection to reporting.

Zika outbreak data generation pipeline.

Each country has its own pipeline, and each pipeline is shaped by the country’s political, economic, geographic, and demographic context, resulting in discrepancies like this:

The union in region X goes on strike often, and doesn’t report Zika case data.

and this:

country Y overhauled its surveillance system, leading to an increase in detected cases.

What you end with are subtle variations in the way these data are generated and processed — the vast majority of which goes undocumented and unaccounted for. So while implicit error is perceived and subjective, it is grounded in a general expectation of variation across distributed, heterogeneous, data generation pipelines.

We speculate that this extends much more broadly beyond Zika and global health.

What does implicit error look like?

Based on qualitative analysis of a sample set of implicit errors collected from our global health collaborators, we found that we could capture the important elements of implicit error, as perceived by experts, via a set of 6 characterizing traits: source, type, direction, magnitude, confidence and extent.

In the context of Zika outbreak data, sources of implicit error range from inconsistencies across data generation pipelines — e.g. country X reports all confirmed and suspected cases as confirmed cases, to retrospective adjustments to previously reported data — e.g. after retrospective review, laboratory-confirmed cases were adjusted by X’s Ministry of Health as of 25 August 2016 and X number of confirmed cases were reclassified as suspected.

The source of an implicit error is often critical for speculating about error type, which can be either systematic or random. For example, the inconsistency above — country X reports all confirmed and suspected cases as confirmed cases — is likely a systematic error, whereas the retrospective adjustment — after retrospective review, laboratory-confirmed cases were adjusted by X’s Ministry of Health as of 25 August 2016 and X number of confirmed cases were reclassified as suspected — is likely a random error. Characterizing the type of an implicit error has important implications for implicit error mitigation, as systematic error can often be reduced via downstream modeling or adjustments to the data generation pipeline itself.

Implicit errors can also be characterized by their direction, describing the sign of the difference between the reported value and the true value (positive/negative/unknown), and by the magnitude of the difference, which in our work with Zika outbreak data, was most often expressed qualitatively —e.g. reported confirmed cases really just shows the tip of the iceberg. The confidence trait describes the domain expert’s confidence in their knowledge or understanding of the error. Finally, the extent trait describes the data that are perceived to be impacted by the error.

We think of these characterizing traits as the data of implicit error, as they allow for visual exploration and analysis. It is interesting to consider how implied error, which is perceived by experts, relates to the more formal statistical definitions of forms of error. Implicit error could be construed as the starting point for potential downstream modeling of the error.

In addition to the set of characterizing traits, we found that including a contextual description of an expert’s knowledge of the implicit error was critical both for evaluating the underlying error in the data and transferring knowledge of the implicit error across experts. We think of these contextual descriptions as the information of implicit error.

To illustrate these concepts, let’s take a look at the following characterization of an implicit error:

source: inconsistency

type: systematic

direction: negative

magnitude: unknown

confidence: very certain

indicator extent: number of cases of Zika in pregnant women

geographic extent: country X

temporal extent: all weekly reports

From these traits we can see that we are dealing with an error in the cases of Zika detected in pregnant women, it’s perceived to be a systematic error, the true value is thought to likely be higher and it’s perceived as definitely happening.

The additional contextual description — country X only reports cases of Zika in pregnant women detected within the first trimester — provides critical insight into the nature of the error, why the error may exist, and the impact that it likely has on reported values. For example if, hypothetically, an expert knew that more and more cases of the Zika virus were presenting in the second and third trimesters, she would know that this was a significant discrepancy.

Where visualization fails and where it can help

As mentioned earlier, a high prevalence of implicit error — as was the case with the Zika outbreak data — precludes visualization from being used for any real analysis.

In early discussions with our collaborators we pitched the idea of allowing experts to manually adjust the colors in a choropleth to reflect their knowledge of discrepancies, similar to what we showed in the opening activity, to which one collaborator responded, “well, I can just do that in my head.”

What this approach doesn’t take into account is the cognitive load of storing many different mental adjustments, and then on top of that, trying to make comparisons and run other visual analysis tasks. Furthermore, how could we ever go about synchronizing these mental adjustments across experts, validating them, and compiling them in order to build institutional knowledge and to begin studying the errors themselves?

This is where visualization comes in!

Visualization has long been recognized as a powerful mechanism for eliciting knowledge and generating new knowledge. Work within the visual analytics community has looked deeply at the role that visualization plays in externalizing expert knowledge and incorporating it back into the visual analysis pipeline. This body of work, however, focuses on knowledge that can be represented and interpreted computationally. Our work looks to extend and adapt existing models in order to support the externalization and incorporation of implicit error — a more qualitative and subjective form of knowledge, which cannot be fully interpreted computationally, but which can provide tremendous insight to experts.

Our adapted model can be described in three stages: the identify stage, the externalize stage, and the analyze stage.