Unless tangential data is collected within a concerted quality control effort, it should probably not be included in a data warehouse. Special data consists of data collected by special data collection operations such as surveys and questionnaires. It also includes imported governmental and commercial data sources. Data quality of special data varies considerably and should be carefully assessed before it is used in critical or pilot applications. Additional properties of data that should be considered in the data assay are quantity and commonality.
Quantity refers to the relative frequency of occurrence.
Levels include universal, common, and occasional data. Such data, if accurate, should be included but flagged for appropriate use to avoid generating unreasonable expectations. Commonality refers to whether this data is in a form that other companies are likely to collect. While not critical for the kind of pilot that we recommend, this issue becomes important when combining data from multiple sources. Unique data should be stored directly in the Data Mart.