**DATA VALIDATION AND VERIFICATION**

**ASSESSMENT MATRIX**

**U.S. DEPARTMENT OF THE INTERIOR**

**December 2002**

LeRon E. Bielak, Office of Planning and Performance Management

Phone: (202) 208-1818

email: leron_bielak@ios.doi.gov

**[Data Verification and Validation (Data V&V):]{.underline}**

-   **What is the Data V&V "Matrix?"**

> It is a basic set of criteria that addresses two aspects of data
> integrity or credibility namely data validation and data verification.
> Validation is basically the first of six hurdles that must be
> successfully jumped. The validation criteria consists of a set of
> questions or sub-criteria to consider to determine if data that are
> being used are appropriate for the outcome that is being pursued.
> Verification is a little more complex. There are five different
> verification criteria to be satisfied, each with a suite of questions
> or sub-criteria to help determine if each criteria has been met. The
> distinction between validation and verification is based on a 1999 GAO
> report and reinforced by practices in a number of agencies with high
> marks for GPRA and/or technical data credibility.

-   **When should the matrix be used?**

> This matrix is a means of assessing whether the GPRA data provided by
> DOI to the Congress, the President, and the public is reliable and a
> valuable decision making tool. The matrix should be used to set up a
> process for validating and/or verifying GPRA data before it is
> collected, consolidated, and reported and can be used to evaluate GPRA
> data before it is used or published. The matrix can also be used to
> find gaps or weaknesses in an existing Data V&V process.

-   **Who should use the matrix?**

> All Interior offices that are engaged in the process of collecting,
> assembling, or reporting GPRA data should establish a Data V&V system
> that as a minimum employs the principles of the Data matrix for
> assessing whether they are adequately ensuring the completeness and
> reliability of their GPRA data.

-   **Why is Data V&V needed?**

> Basically, bad information begets poor decisions. If the information
> being used is not credible, there is no point in using it. Decisions
> based on inaccurate or unreliable data can adversely affect the
> decision making process. The importance of data validity and
> verification has been expressed through the Government Performance and
> Results Act (Results Act/GPRA) itself (Section 1115), through GAO
> investigations and reports (see Reference section), and through the
> requests and findings of Congressional Committees; e.g., report on
> management challenges by the Senate Governmental Affairs Committee and
> the recent House Governmental Affairs Committee request for an
> assessment of data reliability for ten key Departmental GPRA measures.
> The importance of data verification and validation is fundamental to
> all business and scientific practices and has been the subject of
> conferences and discussion groups, including those of the National
> Academy for Public Administration.
>
> Congressional attention is rapidly turning away from rudimentary
> concerns with GPRA reporting format and results of initial reports, to
> more substantive issues including the suitability and accuracy of the
> data being used as a barometer of Federal agency performance. Many
> agencies have recognized data issues and moved to address them and
> point out data limitations. The Department of the Interior must ensure
> that its performance information is reliable and that it is conducting
> internal evaluations to assess the state of its GPRA data. This
> proposal offers an approach to the process of addressing data
> verification and validation in a systematic way across the Department
> and, in so doing, is providing a basis for insuring the reliability of
> performance data Department-wide.
>
> Although the Results Act became law in 1993, its implementation and
> acceptance is still in a relatively early stage. Performance reports
> were not required under GPRA until FY 1999, and questions about the
> efficacy and longevity of the GPRA approach to performance management
> are finally fading. The emphasis of the new Administration on
> "results" has helped solidify GPRA performance evaluation as a
> non-partisan and permanent management tool. Many businesses and
> governmental agencies are focusing more attention on performance
> measurement and using it as the vehicle for informing the public on
> how effectively they are accomplishing their missions and utilizing
> resources. However, imposing a major new reporting burden for
> determining data integrity could be counterproductive to achieving
> broader GPRA acceptance and use. This approach was developed with two
> objectives in mind --- First, to capture the basic process for
> producing credible data, and, second, to apply that approach across an
> organization without a massive "paperwork" burden.

**Working Definition:** Validation is assessing whether data collected
and measured are a true reflection of the performance being measured and
having a clear relationship to the mission of the organization.

+-----------------------------------------------------------------------+
| **Validation**                                                        |
|                                                                       |
| -Criteria-                                                            |
+-----------------------------------------------------------------------+
| -   **Goal/Measure is appropriate to the identified mission of the    |
|     organization**                                                    |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Performance measured has direct bearing (relationship to) on the  |
|     goal in question                                                  |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Goal/Measure is realistic and measurable**                      |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Goal is achievable in the time frame established                  |
|                                                                       |
| -   Goal is neither too aggressive in its expectations or set too low |
|     for easy achievement                                              |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Goal/Measure is understandable to users**                       |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Terms in goal statement are unambiguous and/or terminology is     |
|     defined                                                           |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Goal/Measure is used in decision making**                       |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Decision makers are identified and their judgement on continued   |
|     use of the goal in decision making is periodically evaluated      |
+-----------------------------------------------------------------------+

Validation applies at several levels. First, it is important to
establish that the goals that have been selected to measure the
performance of the organization have a direct connection and relevance
to the mission and desired outcomes that the organization is pursuing.
Second, if that relationship is positively established, the next
question to ask is whether the information that is collected clearly
relates to the targets that have been set.

***[Illustration:]{.underline}***

If for example, the mission of an organization is to reduce the
incidence of a certain disease, it is too indirect to measure the number
of brochures that it has distributed to the public about the disease.
While the brochures may be educational, their distribution is not a
direct indicator of strategy being pursued. It will not inform decision
makers of progress in disease eradication, and it may in itself be
difficult to measure. For example, while brochures may be distributed to
2,000 centers for distribution, there is no gauge for determining who,
if anyone, is taking and reading the information.

**Working definition:** Assessing data accuracy, completeness,
consistency, availability and internal control practices that serve to
determine the overall reliability of the data collected.

+-----------------------------------------------------------------------+
| **Verification**                                                      |
|                                                                       |
| -Criteria-                                                            |
+-----------------------------------------------------------------------+
| **[STANDARDS & PROCEDURES]{.underline}**:                             |
+-----------------------------------------------------------------------+
|  **Source data are well defined, documented; definitions are         |
| available and used**                                                  |
|                                                                       |
| -   Data definitions are well documented and distributed\* to all     |
|     responsible for specific data collection                          |
|                                                                       |
| -   Responsible offices can document adherence to data definitions    |
|                                                                       |
| -   Definitions and standards are used in a consistent manner for all |
|     parties involved in specific data collection                      |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Collection standards are documented/available/used**            |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Protocols and methodology for data collection are documented,     |
|     distributed\* to all responsible for data collection, and         |
|     adherence to the protocols is required and can be verified        |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Review and edit procedures are defined and documented             |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Data reporting schedules are documented/distributed/followed**  |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   GPRA and other data reporting schedules linked to decisionmaking  |
|     are issued\* to all parties responsible for data collection;      |
|     timely data collection and reporting is mandated and enforced     |
|                                                                       |
| ** Performance data supporting documentation is maintained and       |
| readily available**                                                   |
|                                                                       |
| -   **Collection staff are skilled/trained in proper procedures**     |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Those employees or contractors responsible for either collecting  |
|     or assembling data are trained for the job. (For data entry       |
|     responsibilities, see next page)                                  |
|                                                                       |
| \* distribution can be electronic or hard copy                        |
+-----------------------------------------------------------------------+

***[Illustration]{.underline}*:**

Standards and procedures refer to establishing the ground rules that
should be applied to all data collection efforts for a specific measure.
The question is whether the rules are consistently and uniformly applied
and clearly communicated to those who are responsible for grass roots
data collection. If procedures vary from locale to locale or among
individual collectors, results will not be comparable and may not be
legitimate. For example, having no clear definition of the data to be
collected, or mechanisms by which data are collected will inevitably
lead to problems in interpreting results or trusting the accuracy of the
information. If data definitions are clear, but individuals are not
well-trained for the collection effort, which may be complex, additional
sources of error may be introduced. Requirements may very well differ
from goal to goal, but for a single measure, the same standard should
always apply.

+-----------------------------------------------------------------------+
| **Verification**                                                      |
|                                                                       |
| -Criteria-                                                            |
+-----------------------------------------------------------------------+
| **[DATA ENTRY AND TRANSFER]{.underline}**                             |
+-----------------------------------------------------------------------+
| -   **Data entry methodology is documented and followed**             |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Documentation of data entry procedures/protocols is available,    |
|     understood by, and used by data entry personnel                   |
|                                                                       |
| -   Data sources are identified and assumptions about sources are     |
|     documented                                                        |
|                                                                       |
| -   Methods used are comparable for all data entry locations          |
|                                                                       |
| -   Data is entered one time only and corrected by the original       |
|     entrant (exceptions must be documented by a record trail of       |
|     users)                                                            |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Data are verified**                                             |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Calculations are checked                                          |
|                                                                       |
| -   Data is checked for obvious inaccuracies (e.g. Feb. 31) and       |
|     against business rules and other edits                            |
|                                                                       |
| -   Data consistency checks are employed e.g. electronic editing when |
|     available                                                         |
|                                                                       |
| -   Data is re-checked against source information (folders, case      |
|     files, etc.)                                                      |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Procedures for making changes to previously entered data are    |
|     documented and followed**                                         |
|                                                                       |
| -   **Data are available when needed for GPRA reporting and other     |
|     critical decision making cycles**                                 |
|                                                                       |
| -   **Data entry staff are skilled and trained** **in proper          |
|     procedures**                                                      |
+-----------------------------------------------------------------------+

***[Illustration:]{.underline}***

Despite the fact that efforts may have been taken to standardize data
collection methodology, errors can be introduced when data are entered,
transcribed, or transferred during the reporting process. Whether
information is being entered into a computer database or being typed up
in a report from handwritten notes, errors are possible. The question is
whether there is any system in place for detecting these inadvertent
errors. Has an office established protocols for checking and approving
data that are transcribed in any way? Is there a procedure for
addressing the problem of missing data and ensuring that calculations
are correct? Does an office employ computer editing systems, when
appropriate and feasible, to help identify data entry problems? The use
of computer technology to capture data has afforded analytical tools and
power that save considerable labor; however, the issue of the accuracy
of data being analyzed can be too easily ignored. Identifying data entry
or transfer errors is often a very tedious and unrewarding process, but
the importance of follow through in this area is nevertheless high.

+-----------------------------------------------------------------------+
| **Verification**                                                      |
|                                                                       |
| -Criteria-                                                            |
+-----------------------------------------------------------------------+
| **[DATA SECURITY AND INTEGRITY]{.underline}**                         |
+-----------------------------------------------------------------------+
| -   **Duplicate copies or back-up system for data exists**            |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Procedures, including frequency of back up system use, is         |
|     documented and followed                                           |
|                                                                       |
| -   Disaster recovery plan in place                                   |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Data security protocols are in place and effective**            |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Firewalls/password protection, access levels, etc. are            |
|     established                                                       |
|                                                                       |
| -   Security authority must be established for those responsible for  |
|     data entry                                                        |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Equipment and program reliability cannot compromise data        |
|     accuracy**                                                        |
|                                                                       |
| -   **Accountability for data integrity clearly rests with the person |
|     entering the data, and the responsible program specialist and     |
|     manager.**                                                        |
+-----------------------------------------------------------------------+

***[Illustration:]{.underline}***

This area pertains to precautionary measures that must be taken in the
event that computer malfunctions, natural disasters, or human error or
actions occur that could affect collected data. Organizations must
ensure, whether systems being used are hand-entered records or powerful
relational database records, that data are not compromised by lack of
attention to security of the data or to the reliability of systems or
methods being employed to handle or house data. This means having
duplicate records or back up files and ensuring that equipment being
employed does the job it was set up or purchased to do. While some
problems may be a rare situation, they do occur. For example, some
mathematical processes with certain Pentium computers were found to
introduce error due to a faulty processor a few years ago. As another
example, back-up files, if stored within the same CPU unit on which they
were produced, do not offer any additional protection to a malfunction
of the hard drive or a fire in that office. These are more indirect
considerations for the issue of data accuracy, but cannot be totally
disregarded.

Another aspect of data security and integrity is the major question of
unauthorized use of data. This could include both external and internal
access issues from database "hacking" by external parties, to
unauthorized use, including data manipulation, by parties who are not
authorized users. A properly designed system will protect internal
database systems against unauthorized external use, as well as establish
password protection and a clearance process for database changes within
the organization.

+-----------------------------------------------------------------------+
| **Verification**                                                      |
|                                                                       |
| -Criteria-                                                            |
+-----------------------------------------------------------------------+
| **[DATA QUALITY AND LIMITATIONS]{.underline}**                        |
+-----------------------------------------------------------------------+
| -   **Accuracy limits of all data are defined**                       |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Estimated data are identified; methodology for estimation is      |
|     documented and is supportable; use of estimates are minimized     |
|                                                                       |
| -   Data with margins of error due to accuracy of instrumentation or  |
|     interpretive leeway, are identified and margin of error (e.g. +/- |
|     1%) is reported.                                                  |
|                                                                       |
| -   Incomplete data are identified as such and extent of missing data |
|     is reported                                                       |
|                                                                       |
| -   Preliminary data are identified and qualifications on data are    |
|     described; timetable for delivery of final data should be         |
|     documented                                                        |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Any other data limitations are explained and documented**       |
|                                                                       |
| -   **Method for handling anomalous data is established and used**    |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Data that appears to be incongruous compared to most other data   |
|     obtained is analyzed and explained                                |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **3^rd^ party evaluations are conducted**                         |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Objective internal and/or external parties are periodically used  |
|     to verify accuracy/quality of data                                |
|                                                                       |
| -   Use of other cross-checks on data quality such as comparison to   |
|     similar databases are employed and documented                     |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Use of externally controlled data is documented**               |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Need to use external data is established                          |
|                                                                       |
| -   External is identified as such                                    |
|                                                                       |
| -   Degree of completeness and limitations of external data are       |
|     documented                                                        |
+-----------------------------------------------------------------------+

***[Illustration:]{.underline}***

While every action may be taken to ensure that data are accurately
entered, transcribed, reported or otherwise reproduced, there is an
underlying question of whether the data itself is accurate or has some
inherent limitations. For example, do reports clearly specify that
performance may be based on partial data or estimates or that the source
of the data is a third party? Do we have any control over such third
party data or know how whether the data is accurate or has certain
limitations? Are data reported with a confidence interval if that is
applicable? Is there a track record for any changes that may have been
made over time to the information and why that change was made? In some
cases, confidence in data may be bolstered by employing third parties to
evaluate the data by peer review, under contract, through an auditing
process, or other options. Qualifications on reported data are important
pieces of information to decision makers within an agency and in
Congress. Recent evaluations of agency performance reports have
commended agencies who have explicitly addressed the question of data
limitations.

+-----------------------------------------------------------------------+
| **Verification**                                                      |
|                                                                       |
| -Criteria-                                                            |
+-----------------------------------------------------------------------+
| **[OVERSIGHT AND CERTIFICATION]{.underline}**                         |
+-----------------------------------------------------------------------+
| -   **Accountability for data accuracy exists in performance          |
|     standards**                                                       |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Accountability resides with all employees responsible for         |
|     accuracy of data                                                  |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Responsible officials certify that procedures were followed     |
|     each reporting period**                                           |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Signed certifications are filed                                   |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   **Responsible officials certify that data accuracy has been       |
|     checked each reporting period.**                                  |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Signed certifications are filed                                   |
+-----------------------------------------------------------------------+

***[Illustration:]{.underline}***

The underlying purpose of GPRA is to establish accountability. From the
Congressional viewpoint, this means establishing a clear connection
between an agency's mission, the work it sets out to do, and what it
accomplishes for the funds that have been authorized and appropriated
for those purposes. Within each level of an agency, accountability must
rest with individuals and officials who are delegated the authority and
responsibility for achieving certain goals and striving for specific
outcomes. This essentially means that a system of checks and balances
are employed to encourage an integrated effort to achieve desired
results. Practically speaking, it may often be difficult for management
to determine whether information collected or generated by employees is
accurate and complete. If they have had extensive prior experience in
the area, management may have considerable insight into the processes
involved and how to evaluate results. Regardless of the depth of
knowledge, staff and management must both be accountable for GPRA data
reported. While incorporating accountability into performance standards
or employing certifications or attestations to data accuracy will not
guarantee that GPRA data is valid and verifiable, such measures will
reinforce the importance of accountability and responsibility for
performance measurement data and tend to improve the odds that decision
makers are dealing with bona fide and reliable information.

**[Selected References:]{.underline}**

Bielak, L.E., and J. Zippin. *Data Validation and Verification Matrix*.
U.S. Department of the Interior; 2000

DeMaio, Carl. *Improving the Quality of Performance Information in
Government: A Maturity Model for Assessing Performance Data* Systems;
September 2000.

GAO Report to the Senate Committee on Governmental Affairs: *Performance
Plans: Selected Approaches for Verification and Validation of Agency
Performance Information*; GAO 99-139, July 1999.

Joseph, Ed. *Putting the Pieces Together: Developing a Comprehensive
Data Audit Plan*. September, 2000

Peak, Lois, *Standards for Evaluating the Quality of Program Performance
Data,* U.S. Department of Education, 2000

Performance Consortium Discussion Forum: *Challenges of Data -
Collection, Verification, Validation and Use*; November 3, 1999;
Washington, D.C.

Wellman, Martha. *Auditing Performance Information: A View from
Florida*; September 2000

**[Acknowledgments:]{.underline}**

This data evaluation process was developed in the Office of Planning and
Performance Management of the U.S. Department of the Interior. This
approach incorporates or adapts data verification and validation
approaches that are used in offices that have been referenced above. The
following individuals have contributed to this product through
discussions and reviews and/or by virtue of products they have authored:

[U.S. Department of the Interior]{.underline}:

Norma Campbell, Jeffrey Zippin (Office of Planning and Performance
Management)

Mary Adler, Pam Boteler, Gloria Denmark (Office of the Inspector
General)

Daryl Koncer (Bureau of Land Management)

[U.S. Department of Education:]{.underline} Lois Peak, Hugh Walkup

[U.S. Department of Transportation]{.underline}: Jim McEntire

[General Accounting Office:]{.underline} Cliff Fowler, Stan Divorski

(Bielak\-- P:\\GPRA\\Data V&V\\DataV&V Guide Nov 02)