OPM's Central Personnel Data File: Data Appear Sufficiently Reliable to Meet Most Customer Needs
Released: 1998-09-30T00:00:00

Letter

Introduction

According to the Office of Personnel Management’s (OPM) Guide to the Central Personnel Data File (CPDF), the CPDF is the federal government’s central personnel automated database that contains statistically accurate demographic information on about 1.9 million federal civilian employees. The CPDF’s primary objective is to provide a readily accessible database for meeting the workforce information needs of the White House, Congress, OPM, other federal agencies, researchers, and the public. A second objective is to relieve agencies that submit personnel data to the CPDF of the need to provide separate data or reports to meet a variety of reporting requirements. Data that agencies submit to the CPDF represent their official workforce statistics.

OPM’s Office of Workforce Information (OWI) is responsible for accepting and entering data into the CPDF and processes the data using the Central Personnel Data System. OWI also prepares reports using CPDF data and distributes CPDF data to both OPM and non-OPM users.

In order to safeguard the privacy of federal civilian employees as required under the Privacy Act of 1974, OPM must protect CPDF data from unauthorized disclosure. For example, at OPM access to agencies’ CPDF submissions is limited to OPM staff responsible for determining if the data meet OPM’s guidelines for acceptance into the CPDF. When disseminating CPDF data, OPM is to protect the privacy of individuals. For example, OPM is not to provide employees’ names, Social Security numbers, or birth dates to requesters or to make this information available to the federal agenciesthat are allowed to access the CPDF via OPM’s electronic User Simple and Efficient Retrieval (USER) system to retrieve personnel data to do their work.

Background

The CPDF contains personnel data for most of the executive branch departments and agencies as well as a few agencies in the legislative branch. Included are all of the cabinet departments (e.g., State, Treasury, Justice); the independent agencies (e.g., Environmental Protection Agency, Small Business Administration, National Aeronautics and Space Administration); commissions, councils, and boards (e.g., National Council on the Handicapped); and selected legislative branch agencies, such as the Government Printing Office.

The CPDF does not contain employee data for the Central Intelligence Agency, Defense Intelligence Agency, the Board of Governors of the Federal Reserve System, National Security Agency, Office of the Vice President, Postal Rate Commission, Tennessee Valley Authority, U.S. Postal Service, or the White House Office. The CPDF also excludes from coverage non-U.S. citizens working for federal agencies in foreign countries; most nonappropriated fund personnel; commissioned officers in the Department of Commerce, Department of Health and Human Services (HHS), and the Environmental Protection Agency; and all employees of the judicial branch.

The History of the CPDF

The Civil Service Commission, OPM’s predecessor, decided it would install a type of central personnel database—the CPDF—in 1972 to provide a source that was capable of (1) satisfying minimum essential statistical data needs for central management agencies and the public; (2) meeting reporting requirements, such as periodic surveys of affirmative employment programs and semiannual turnover reports; and (3) alleviating the need for agencies to individually report similar information separately to requesters. The CPDF also expanded and replaced the Federal Personnel Statistics Program Sample File, which was established in 1962. The File contained a continuous work history on each federal employee whose Social Security account number ended in the digit “5,” a population that constituted a 10-percent sample of the federal workforce.

How the CPDF Operates

OPM builds six files from agency-submitted data. These are the longitudinal history (a record of personnel actions arranged by date within Social Security number), organizational component (a listing of the codes used by each agency to identify its various work units, e.g., regions, divisions, branches); personnel office identifier (contains the mailing address and telephone number for personnel offices that report to the CPDF); name (a cross-reference listing of names, Social Security numbers, accession dates, and applicable separation dates of employees reported to the CPDF); status; and dynamics files. Of the six, this report focuses on the status and dynamics files. They are the source of the demographic information used by OWI to write reports and to respond to data requests by users of CPDF data.

The status file consists of data elements describing each employee as of the date of the file. Agencies are required to submit these files on a quarterly basis, with the submissions due at OPM no later than the 22nd of the month following the end of the quarter (e.g., input for the quarter ending December 31 must be submitted by January 22). All of the employees covered by the CPDF are to be included in each file. The data elements include information on the type of work; the employee’s pay; and personal information, such as gender and birth date.

The dynamics file consists of data elements describing each personnel action taken by an agency during the period covered by the file. Personnel actions are the official records of employees’ careers, such as hires, promotions, reassignments, pay changes, resignations, and retirements. The file includes information about the action taken, the agency/subelement, the position, pay, and the individual employee. The normal reporting period is a calendar month but may end as of the last full biweekly pay period of the month. Submissions are due at OPM as soon as possible following completion of agency processing but no later than 22 days following the end of a monthly reporting period. As of February 1998, the CPDF consisted of 95 separate data elements. Of this number, 68 are to be reported by agencies in their monthly and quarterly dynamics and status file submissions.

OPM relies on agencies to ensure that the data they submit are timely, accurate, complete, and edited in accordance with OPM standards. OPM provides agencies with guidance, the Guide to the CPDF, which says agencies are to test the data they provide to the CPDF to ensure that the data are accurate and complete. To help agencies ensure the quality of their data, OPM provides them with the CPDF Edit Manual, which prescribes the data values to which agencies’ data are to conform before they are submitted. To test the values of their data, agencies are to use OPM’s CPDF edits. These edits are computer instructions that are to check the validity of individual data elements as well as the proper relationship of values among associated data elements. For example, the edit for the sex data element checks that the character used to define the data element is either “M” for male or “F” for female; the edit identifies other characters as errors. OPM expects agencies to incorporate these CPDF edits into their internal personnel data systems. These edits constitute the minimum level of quality control OPM expects the agencies to employ. Agencies have the option of incorporating additional quality controls, such as testing a sample of the data for accuracy before submitting it, in addition to applying the CPDF edits.

The CPDF edits cannot detect all types of errors. For example, an edit for the sex data element would not be able to detect if the character “M” was incorrectly used to identify a female employee. According to OWI officials, although they provide agencies with the edits, errors still occur in submissions, which OPM strives to identify through OWI’s quality review process. The officials also said that errors in pay-related data elements often occur at the beginning of the year because agencies make their beginning-of-the-year submissions before they install edits that reflect annual cost-of-living pay increases. The Guide to the CPDF also informs agencies about what data elements should be included in their CPDF data submissions and the frequency and timing of the submissions. As mentioned earlier, frequency and timing requirements differ for the status and dynamics data files.

After agencies submit the personnel data, OPM puts the submissions through an acceptance process before the data can be entered into the CPDF. This process includes putting the data through the same CPDF edits the agencies were to use before submitting the data as well as other analyses. OWI manages the process. Its staff are to provide agencies with feedback on their submissions, requesting, as needed, corrections to submissions that fail edit checks or other analyses and preventing data that are not within the acceptable range of data values from being entered into the CPDF. OWI is to make the final decision about what data are entered into the CPDF. At the time of our review, the Central Personnel Data System was operated by OPM’s Office of Information Technology (OIT) for OWI. The CPDF Quality Control team that monitored agencies’ data submissions was part of OIT. However, operation of the System was transferred to OPM’s Retirement and Insurance Service in 1997, and the Quality Control team were reassigned to OWI.

OPM Has Authority to Request Agency Data for the CPDF

OPM may require agencies under 5 C.F.R. section 7.2 to report “in such manner and at such times as OPM may prescribe, such personnel information as it may request.” On the basis of this authority, OPM is able to direct agencies to submit selected personnel data to the CPDF. However, although the OPM Director can request data, she cannot ensure that agencies provide accurate information in a timely manner. The responsibility for providing timely, accurate information remains with the head of the agency providing the information. OPM officials rely on federal agencies to voluntarily comply with CPDF guidelines and correct problem submissions.

Objectives, Scope, and Methodology

For this review, we had three objectives: (1) determine the extent to which selected CPDF data elements are accurate, including the data elements used by OPM’s Office of the Actuaries for estimating the government’s liability for future payments of federal retirement programs; (2) determine whether selected users of CPDF data believed CPDF products met their needs, including whether the products were current, accurate, and complete and whether the cautions OPM provided to them on the limitations associated with using the data were sufficient for them to present the CPDF data correctly; and (3) determine whether OPM has documented changes to the System and verified the System’s acceptance of those changes, as recommended in applicable federal guidance, and whether the System would implement CPDF edits as intended.

Objective 1

To determine the extent to which selected CPDF data elements are accurate, including the data elements used by OPM’s Office of the Actuaries for estimating the government’s liability for future payments of federal retirement programs, we (1) designed and sent questionnaires to a random sample of federal employees to have them verify some of their CPDF data and (2) compared CPDF data with information in randomly selected official personnel folders and in other agency records at selected personnel offices. Table 1.1 presents a list of the CPDF data elements we used in our employee questionnaire and comparison of CPDF data with information in official personnel folders.

We used two approaches, i.e., a questionnaire (see app. V) and a comparison of data in official personnel folders and agency records to CPDF data, to measure the accuracy of CPDF data. We sent the questionnaire to a sample of employees, because OPM studies show that official personnel files or agency records may be in error. We compared the results of both approaches to develop our findings.

We also reviewed past OPM accuracy measurements, examined CPDF data for missing and unusable information, and interviewed an official of OPM’s Office of the Actuaries to discuss the accuracy of the data elements the Office uses for estimating the government’s liability for future retirement payments. These steps are more fully described in the following sections.

Questionnaire

As part of our evaluation of the accuracy of CPDF data, we selected a stratified random sample of 565 federal employees and attempted to send each a questionnaire containing 20 data elements about themselves obtained from the CPDF (see app. V for a copy of our questionnaire). The data elements that we included in the questionnaire were among those we most frequently use to do our work, those OPM analysts use most frequently in preparing CPDF reports, and those used by OPM’s Office of the Actuaries to estimate the government’s liability for future payments of federal retirement programs. We selected those data elements that we believed employees would be able to verify. We included in each individual’s questionnaire data elements from the September 1996 CPDF status file about that individual. The elements consisted of (1) Social Security number, (2) employing agency/subelement, (3) adjusted basic pay (including locality pay), (4) month and year of birth, (5) duty station, (6) pay plan, (7) grade, (8) handicap, (9) occupation, (10) race or national origin, (11) service computation date, (12) sex, (13) veterans preference, (14) veterans status, (15) work schedule, (16) education level, (17) rating of record, (18) retirement plan, (19) annuitant indicator, and (20) employee name. We asked the respondents to verify the accuracy of each data element, indicating whether it was correct or incorrect as of September 30, 1996. When a respondent indicated that a data element was incorrect, we asked the respondent to enter the correct information.

We pretested the questionnaire to assure ourselves that respondents could interpret the questions correctly and could provide the information requested. We modified question wording and questionnaire format on the basis of what we learned from five pretests.

The random sample of 565 was drawn from 7 strata to represent a study population of 1,905,787 non-Federal Bureau of Investigation (FBI) federal employees whose names were contained in the CPDF database as of September 30, 1996. Random samples of 30 selections each were drawn from 6 smaller strata, each of which comprised a single personnel office. These six personnel offices were among the eight largest personnel offices in the federal government. These offices were the Social Security Administration (SSA), Baltimore, MD; Department of the Army, Fort Benning, GA; U.S. Customs Service, Washington, D.C.; National Institutes of Health, Bethesda, MD; Department of State, Washington, D.C.; and Department of the Navy, Pensacola, FL. We selected these personnel offices because of their size and because our work at the offices could then be representative of a relatively large portion of records contained in the CPDF. The 6 personnel offices were among 1,425 in the government and served over 8 percent of the employees whose data were contained in the CPDF as of September 30, 1996. We used the selections from these six personnel offices for both the employee questionnaire and a review of official personnel folders. The remainder of the sample—385 selections—was randomly drawn to represent the remaining stratum of 1,746,592 employees from all other personnel offices. The total sample of 565 was designed to ensure that it approximately mirrored the population distribution with respect to type of appointment (career or noncareer), work schedule (full-time or non-full-time), type of service (competitive or excepted), and location (stationed in or outside the United States).

Because the CPDF does not contain mailing addresses for employees, we mailed most of our questionnaires to personnel officers whom were identified in the CPDF as serving the employees in our sample. In all, 562 of the 565 sampled employees were covered by 280 personnel officers. We were not able to identify the personnel officers for three of the sampled employees. We asked the personnel officers to whom we sent questionnaires to forward them to the sampled employees. In addition, we asked them to provide us with the direct mailing address of each sampled employee so that we would be able to mail follow-up questionnaires directly to sampled employees who did not return a questionnaire to us within 45 days. We also asked the personnel officers to furnish us with reasons why any of the questionnaires could not be forwarded to the sampled employees. After an initial and a follow-up mailing, we received 407 usable questionnaires out of 565, for a 72 percent response rate. Table 1.2 presents a breakdown of the number of sampled federal employees responding to our questionnaire as well as the various reasons why some sampled employees did not respond.

We edited the questionnaires received from respondents to identify data elements marked as incorrect. In cases where a respondent indicated that a data element from the CPDF was incorrect, the editor then made an effort to determine if the correction entered onto the questionnaire by the respondent was logical. For example, a number of respondents indicated that the annual pay amount shown on the questionnaire was incorrect. However, in researching the “correct” amount entered by the respondent, it was determined that the amount entered was his or her current annual pay, not the annual pay as of September 30, 1996, as indicated in the question. In these cases, the response was changed from incorrect to correct by the editor.

The 407 returned questionnaires from the 7 strata were weighted to represent the population of 1,905,787 federal employees for all results presented in this report. Sampling errors have been calculated to take into account the different weights assigned to each stratum. Unless otherwise noted, the 95 percent confidence intervals around all reported results are plus or minus 5 percentage points or less.

In addition to sampling errors, the practical difficulties of administering any questionnaire may introduce other types of errors, commonly referred to as nonsampling errors. For example, differences in how a particular question is interpreted by the questionnaire respondents could introduce unwanted variability in the questionnaire’s results. We took steps in the development of the questionnaire, the data collection, and the data editing and analysis to minimize nonsampling errors.

Comparison of Official Personnel Folders and Agency Records With CPDF Data

We also compared data contained in official personnel folders and other agency records with data in the CPDF for the same period at the six selected personnel offices. For each of the 6 personnel offices we selected, we chose 30 employees at random from the September 1996 CPDF status file. The employees were those who were reported by the CPDF as being served by the six respective personnel offices. At each of the 6 personnel offices, we asked for official personnel folders for the 30 employees. We also asked for information from the personnel offices’ automated files on ratings, handicap, and race or national origin because such information is not necessarily contained in personnel folders. We then selected 20 employees at random from those whose official personnel folders were available. We over-sampled by 10 employees in our initial sample for each personnel office because we anticipated that some folders would be unavailable because of employee departures or other reasons. At SSA, we reviewed official personnel folders and other agency records for only 13 employees because the official personnel folders for 17 of the 30 employees we chose at random were located in offices throughout the country and not in a central location as we initially expected. In total, we reviewed folders and other agency records for 113 employees for the 6 personnel offices.

For each of the 113 employees in our sample, we obtained information from the September 1996 CPDF status and dynamics files. The information we obtained consisted of the 20 data elements we used for our questionnaire and the data elements that we most frequently use to do our work, including key status and dynamics data elements. The eight data elements that were in addition to the data elements used for the questionnaire were current appointment authority, effective date of action, legal authority code, nature of action code, pay rate determinant, personnel office identifier, position occupied, and tenure. We reviewed a total of 28 data elements: 23 data elements common to both the status and dynamics files, 1 element found only in the status file, 3 elements found only in the dynamics file, and employee name (see table 1.1 for the CPDF data elements we reviewed and their file locations).

For each employee, we compared the CPDF data with relevant documents, such as Standard Forms 50 (notification of personnel action) and employment applications, in official personnel folders. We also compared the CPDF data with automated files on those employees’ ratings, handicap, and race or national origin. We discussed any mismatches we found with personnel officials in an attempt to determine how differences can occur between the CPDF and agency documentation.

Past OPM Accuracy Measurements

OPM conducts periodic measurements of CPDF accuracy by comparing data in the official personnel folders of separated employees with data in the CPDF. We reviewed the six measurements of CPDF accuracy OPM did from April 1984 to July 1996 and compared the results of our evaluation of CPDF accuracy with the results of OPM’s last two measurements, which were issued in January 1992 and July 1996.

Data Used by OPM’s Office of the Actuaries

To determine if the CPDF data used by OPM’s Office of the Actuaries to estimate the government’s liability for future retirement payments are sufficiently accurate for use by the Office, we first met with the actuary responsible for calculating this liability to determine the CPDF data elements used in the estimate. After our analysis of the employee questionnaire and our comparison of personnel folders and other agency records to CPDF data, we again interviewed the actuary to discuss the results of our two approaches and the impact of errors on the estimate.

Additional Methodological Characteristics

The results of our employee questionnaire are generalizable to the universe of 1,905,787 employees included in the CPDF’s September 1996 status file. Table 2.1 shows the generalized results as a percentage of records in the September 1996 status file. The results of our comparison of employees’ official personnel folders and other agency records to CPDF data are not generalizable to the CPDF as a whole, although they may be indicative of the personnel offices at which we performed our work.

The CPDF data elements measured for accuracy generally were among those identified by OPM as key to the accuracy of its recurring reports. We cannot determine from the work we did the accuracy of data elements we did not review. We did not independently verify educational levels reported by employees or any of the responses of employees.

Our accuracy findings are for CPDF data in the September 30, 1996, status file and the fiscal year 1996 dynamics file. The accuracy might differ for previous and future CPDF files, especially when agency procedures or information processing technology change.

Our accuracy measurement was not designed to evaluate the reliability of CPDF data from individual agencies or specific subsets of employees, such as those on leave without pay. OPM reports on the percentage of data elements in agency submissions that do not pass standard CPDF edits show considerable variation across agencies.

Objective 2

To determine whether selected users of CPDF data believed CPDF products met their needs including whether the products were current, accurate, and complete and whether the cautions OPM provided to them on the limitations associated with using the data were sufficient for them to present the CPDF data correctly, we designed, with advice from OPM, a CPDF customer questionnaire (see app. VI for a copy of our questionnaire). We mailed the questionnaires to 247 individuals identified by OPM’s OWI as representing all the requesters of CPDF products in fiscal year 1996 who obtained data directly from OPM.

We mailed the customer questionnaires in May 1997 to the return addresses on letters in OWI’s fiscal year 1996 correspondence files that had requested CPDF products and to recipients of recurring CPDF-based reports in 1996. We followed up our initial mailing with a second one in June and a third one in July. We did not include in our analysis any questionnaires received after August 6, 1997.

After August 6, 1997, we made follow-up telephone calls to all nonrespondents and determined that 40 of the original 247 individuals we sent the questionnaire to were either not CPDF users or had left their organizations. Of the remaining 207 individuals who were CPDF users, 140 (or 68 percent) responded to the mail questionnaire, and an additional 21 responded to an abbreviated version of the mail questionnaire we used in follow-up telephone calls to nonrespondents. The combined response rate for the mail-out questionnaire and the telephone follow-up was 78 percent.

After we received the questionnaires from the respondents, we edited them for completeness and consistency. All of the data from the questionnaires were double-keyed and verified during data entry. In addition, a random sample of these data was verified back to the source questionnaires.

Additional Methodological Characteristics

The results of our customer questionnaire are not generalizable to the universe of users of CPDF data and products for 1996 because we could not define the universe of users necessary to draw a representative sample. The distribution of CPDF products, such as recurring reports, is not controlled. These products are available through various outlets, such as libraries, that do not track customers. Therefore, we relied on OWI to identify those customers who corresponded with it in 1996 to request CPDF data and sent our questionnaire to this defined but nonrepresentative subset of the 1996 universe of CPDF users.

Objective 3

To determine whether OPM has documented changes to the Central Personnel Data System and verified the System’s acceptance of those changes, as recommended in applicable federal guidance, and whether the System would implement CPDF edits as intended, we first reviewed federal guidance on managing automated information systems. To determine the extent to which OPM’s OIT followed the guidance in managing the development of the System, we conducted interviews at OIT, which was responsible for operating the System, and OWI, which is the System’s owner, about their basis for determining the System’s reliability. From these officials, we requested available documentation relating to modifications and upgrades of software used by the System to process CPDF data and documentation relating to verification that these modifications and upgrades worked as planned. We also reviewed available documentation on OPM’s current Information Technology Strategy to determine whether it includes procedures for managing the System in the future. To determine whether the System would implement CPDF edits OPM uses to screen the 68 data elements reported by agencies to OPM as intended, we reviewed 18 of the 63 validity and all 700 of the call-relational edits the System uses to screen agencies’ data submissions.

Additional Methodological Characteristics

We judgmentally selected only the 18 validity edits OWI uses to screen the data elements it considers critical; therefore, the findings of our review of these 18 edits cannot be generalized to all 63 validity edits. Because we did not actually put test data through the System or otherwise test the reliability of the System’s hardware and software under operating conditions, we cannot verify the reliability of the System. We did not assess the likelihood that the CPDF would be Year 2000 compliant by December 31, 1999.

We conducted our work between November 1996 and June 1998, in accordance with generally accepted government auditing standards. The employee CPDF data verification questionnaire and CPDF customer survey were administered between May 1997 and September 1997; thus, the data are as of those dates.

We requested comments on a draft of this report from the Director of OPM. OPM provided written comments on a draft of this report (see app. VII) that are discussed at the end of chapters 2, 3, and 4.

CPDF Data Reviewed Appear to Be Mostly Accurate in the Aggregate

The accuracy of the data the CPDF contains depends on the accuracy of the data that agencies submit. Errors in those data can occur at various stages of the personnel process, such as when agency personnel clerks enter data for newly hired employees or when they code information on personnel actions (e.g., performance appraisals). OPM does not have an official accuracy standard for agencies’ submissions. On a periodic basis, however, OPM draws a governmentwide sample of CPDF records and measures CPDF data accuracy by comparing selected data in former federal employees’ official personnel folders to data in the CPDF for the same period. OPM generally makes the results of its measurements of CPDF accuracy available to OPM users of CPDF data but not to non-OPM users. In spite of the important uses of CPDF data, no independent evaluation of the accuracy of the data has been done. Our work showed that most of the CPDF data elements we reviewed were 99 percent or more accurate on a governmentwide basis. The rating of record and education level data elements had the highest error rates, at about 5 and 16 percent for rating of record, and 23 and 27 percent for education level based on our questionnaire and comparison, respectively. Our overall findings are broadly similar to what OPM found when it measured historical accuracyin 1996 by comparing 1994 data in former employees’ official personnel folders with the data in the CPDF. We shared the results of our work with the actuary responsible for calculating the federal government’s liability for future retirement payments to retired federal employees and their survivors, and he said that the CPDF data elements were sufficiently accurate for making this estimate.

OPM Measures Historical Accuracy of CPDF, but Does Not Report Results of Its Accuracy Measurements to Non-OPM Users

OPM periodically measures the accuracy of selected data that are in the CPDF. As we said earlier, OPM relies on agency data passing CPDF edits to eliminate errors that would result in inaccurate data being entered in the CPDF. For example, the edits are to identify a salary amount that is too high for a particular pay plan or grade. However, the edits are not able to identify an error in salary that is within the range of that pay plan or grade. Thus, inaccurate data can get into the CPDF.

To measure the historical accuracy of CPDF data, OPM periodically compares certain data found in a sample of former federal employees’ official personnel folders to data in the CPDF for the same period. From April 1984 to July 1996, OPM conducted six such measurements. OPM analysts used a sample of former employees and compared certain data elements in their official personnel folders to information in the CPDF’s status and dynamics files. For example, the latest measurement, which was released in 1996 for fiscal year 1994 data, used a sample of 135 former employees and compared 35 status file and 40 dynamics file data elements to information in the official personnel folders. An error was defined as a value found in the CPDF that was not the same as that found in the employee’s official personnel folder. OPM officials told us—and OPM’s accuracy surveys state—that the surveys were designed to measure the accuracy of governmentwide data only and not the accuracy of data from individual agencies. OPM generally makes the results of its measurements of CPDF data accuracy available to CPDF data users within OPM but not to non-OPM users.

In five of the historical accuracy measurements, OPM found that most CPDF data were generally accurate, and in most cases the selected data elements matched the corresponding official personnel folder entries 99 percent or more of the time. However, OPM did not make that statement for its December 1990 measurement of 1988 CPDF data. Instead, it advised OPM users of CPDF data to review the results of the accuracy measurement and determine for themselves whether the data were sufficiently accurate for their use. OPM officials said that OPM does not routinely inform non-OPM users of the results of its measurements of historical accuracy.

OPM has not promulgated a standard for the accuracy of CPDF data. To our knowledge, no federal agency has promulgated accuracy standards that are generally applicable to federal databases. In general, the level of accuracy for data must be balanced against what the data are to be used for and the cost of obtaining a greater level of accuracy.

Most CPDF Data Tested Were Accurate and Agreed With Agencies’ Personnel Records

To measure the accuracy of the CPDF, we (1) sent a questionnaire to a random sample of federal employees to gather information about the accuracy of 20 of the 68 CPDF data elements reported by agencies and (2) compared data for 28 data elements in the CPDF with the data contained in the official personnel folders and other agency records for 113 randomly selected employees at 6 of the largest federal personnel offices. We found that most CPDF data elements we tested were accurate and agreed with information in employees’ official personnel folders and other agency personnel records. Although our methodology differed from the one OPM uses in its measurements of historical accuracy, the results of our review were broadly similar to OPM’s results.

Questionnaire Results and Comparison of Selected CPDF Data to Employee Records Showed Most Data Were Accurate and Agreed With Agencies’ Personnel Records

To determine the accuracy of 20 selected CPDF data elements, we sent a questionnaire to a random sample of federal employees that was representative of federal employees governmentwide (see ch. 1 for a description of our sampling methodology). We asked them to review information about themselves that we obtained from the September 1996 CPDF. The data elements we asked about were those about which we believed employees would be most familiar, including employee name,birth date, and Social Security number. The results of our questionnaire showed that 14 of the 20 data elements, or 70 percent, matched data in the CPDF in 99 percent or more of the cases (see table 2.1). There were no inaccuracies for seven of these data elements and the other seven data elements had error rates of less than 1 percent. The remaining six data elements had error rates greater than 1 percent (see table 2.1).

The two most error-prone data elements were education level and rating of record. Education level had a 26.7 percent error rate and rating of record had a 4.7 percent error rate. The education level data element is intended to reflect the highest education level that a federal employee achieved. The rating of record data element indicates an employee’s most recent rating or performance appraisal. The results of our employee questionnaire are generalizable to the universe of 1,905,787 employees included in the CPDF’s September 1996 status file. Table 2.1 shows the generalized results as a percentage of records in the September 1996 status file.

We also compared data in employees’ personnel folders or other agency records with data in the CPDF for 113 randomly selected employees at 6 of the largest federal personnel offices (see the Objectives, Scope, and Methodology section in ch. 1 for a discussion of our selection process). For this comparison, we reviewed a total of 28 data elements: 23 data elements common to both the status and dynamics files, 1 element found only in the status file, 3 elements found only in the dynamics file, and the employee name data element found in the CPDF name file. (See table 1.1 in the Objectives, Scope, and Methodology section in ch. 1 for the CPDF data elements we reviewed and their file locations.)

In our review of official personnel folders and agency records, we found no inconsistencies among the 23 data elements we included in our comparison that were common to both the status and dynamics files. For example, if the status file data element showed an erroneous education level for a given employee, the dynamics file element showed the same erroneous code. Our review of official personnel folders showed that personnel actions reflected in the CPDF dynamics file appeared to be generally complete. There were no inaccuracies for 12 of the data elements. For another five data elements, our comparison showed error rates of less than 1 percent. The remaining nine data elements had error rates greater than 1 percent. For the legal authority code data element, we could not determine the error rate because some employees had no transactions for fiscal year 1996. Table 2.2 shows the results of our comparison.

Concerning the most error-prone data elements, our review of employees’ official personnel folders and agency records showed results similar to those of our questionnaire—education level and rating of record were the most error-prone data elements. (See app. III for a more detailed discussion of the data elements that contained the highest rates of error.) However, the results of our comparison between the data in the official personnel folders and the CPDF differ somewhat from those of our questionnaire. For example, the results of the questionnaire showed education level to have a 26.7 percent error rate and rating of record to have a 4.7 percent error rate. The results of the comparison showed education level to have a 23.0 percent error rate and rating of record to have a 15.9 percent error rate. Although we did not try to determine the reason for these differences, two reasons appear most likely. First, the results of the questionnaire are generalizable governmentwide, although the results of the comparison are not because the sample of the comparison is not generalizable. Second, the information in the employees’ official personnel folders might not be current. In particular, employees may not have informed their personnel offices of additional education completed, so this information may not be in the official personnel folder. Thus, the information in the official personnel folder might match the CPDF, but neither would be current.

The Results of Our Review Were Broadly Consistent With Those of OPM’s Historical Accuracy Measurements

In its measurements of historical accuracy of CPDF data, OPM has reported results broadly consistent with ours. That is, OPM has found that most data elements it reviewed were 99 percent or more accurate but has found high error rates for rating of record and education level. Table 2.3 groups by percent of errors the error rates identified by our two methods for measuring CPDF status and dynamics file data accuracy and OPM’s measurement of the historical accuracy of fiscal year 1994 CPDF status file data. The table shows that between the two methods we used to measure CPDF data accuracy (although variation existed in the accuracy of some data elements) at least 63 percent of CPDF data elements were 99 percent or more accurate.

Although OPM’s and our results were broadly consistent, there are important differences between OPM’s methodology and ours. First, we sent our questionnaire to a generalizable sample of current federal employees and reviewed a random sample of official personnel files of current federal employees. In contrast, OPM reviewed centrally located records of former employees. Second, OPM’s methodology in comparing CPDF data with those in employees’ official personnel folders differed from ours. We often relied on agency records (e.g., records maintained separately from official personnel folders for race, national origin, and handicap) in cases where data were not in official personnel folders, but OPM generally limited its review to documents that were in personnel folders. Third, the way we determined errors differed in part from OPM’s. OPM did not determine if the official personnel folder data element itself was correct, but we did so by researching available agency personnel records.

The Accuracy of CPDF Data Varied by Agency

Our review of employees’ official personnel folders and other agency records was intended to evaluate CPDF accuracy in general, not to compare CPDF data accuracy among individual agencies. Such a review would have required a much larger sample to represent each agency. But during our review, we did find circumstances that demonstrated how accuracy varied by agency and why. For example, although the five other agencies we reviewed were routinely providing information on employee performance ratings to the CPDF, SSA had not updated rating information in the CPDF for over 2 years at the time of our review. SSA officials told us this lapse occurred because temporary procedures that had been established to correct SSA’s difficulty in providing appraisal data to HHS proved to be cumbersome; as a result, SSA did not provide its appraisal data to HHS for HHS to submit the data to the CPDF for 1995. According to SSA officials, SSA continued to capture these data in its human resource management information system, but HHS did not ask for the data, and SSA was not aware that it was to report them to the CPDF.

The importance of a data element to an agency can affect the level of effort that the agency gives to ensuring the data element’s accuracy. For example, some personnelists in the offices we visited said that the accuracy of education level information was “of little concern” to them. In contrast, two other personnel offices reported taking steps to improve the accuracy of this information. Officials in one of these offices (the Pensacola Naval Air Station) told us they had updated the education level information on their employees as part of an overall records review. An official in the other office (State) told us that promotions for certain of their employees are based, in part, on education levels. Therefore, the official said that employees are asked to review such information maintained by the agency and report needed changes.

We also observed in previous work that agency-specific CPDF data could be inaccurate. For example, in 1997 the House Committee on International Relations asked us to examine a discrepancy between the number (17) of Schedule C political appointees reported to Congress by the Agency for International Development (AID) and the number (0) that appeared in the CPDF for the period January 19, 1993, through November 14, 1995. Through our analysis of the CPDF data, we determined that AID used the wrong legal authority when coding the appointing authority for these individuals. As a result, the information in the CPDF (0) did not correctly identify any of the 17 individuals as political appointees.

Inaccuracies in specific agencies’ CPDF data, such as SSA not submitting current rating of record data for 2 years and AID using the wrong legal authority code for Schedule C political appointees, can distort users’ analyses, findings, and conclusions and result in OPM’s reporting on federal agencies that misinforms policymakers and the public. These examples also show that errors in agency-specific data may go unnoticed for several years and that the accuracy of a particular data element can vary from year to year for a particular agency.

OPM officials told us that they believe that the periodic accuracy measurements that OPM does are a good indicator of problematic data elements governmentwide. For example, OPM’s measurement of historical accuracy for fiscal year 1994 discusses why errors occurred and gives error rates for status and dynamics file data elements governmentwide. However, as we said earlier, OPM does not provide the results of these measurements to non-OPM users of CPDF data. Therefore, non-OPM users of CPDF data are most likely not aware of the findings of OPM’s accuracy measurements. In addition, OPM officials said that their periodic accuracy measurements are not useful for identifying errors in CPDF data elements at individual agencies. OPM officials said they sometimes become aware of agency-specific inaccuracies in the CPDF when non-OPM users of the data, such as us or the agencies affected, contact OPM about the inaccuracies. For example, OPM said that after it discovered that AID Schedule C appointees were not identified in the CPDF, it began working with AID to improve the future reporting on political appointees.

Awareness of inaccuracies in specific data elements and variation in data accuracy among agencies is important because OPM and non-OPM users rely on CPDF data to monitor and report on individual agencies’ demographics, compliance with government policies, or other characteristics. For example:

OPM’s Office of Merit Systems Oversight and Effectiveness uses CPDF data to monitor and report on individual agencies’ compliance with selected Merit Systems Principles set out in title 5 of the United States Code;the National Performance Review used CPDF data in a 1993 report on Transforming Organizational Structures to compare the numbers of federal personnel by occupation; the Equal Employment Opportunity Commission used CPDF data in its fiscal year 1991 report to the President and Congress on affirmative employment programs for minorities and women and for hiring, placement, and advancement of people with disabilities in the federal government; and we use the data in some of our reports to Congress.

According to these officials, OPM’s current approach for measuring CPDF data accuracy is not designed to include representative samples for individual agencies, and such a sample would be significantly larger than the 135 official personnel folders OPM examined to do its latest measurement for fiscal year 1994 data. OPM officials recognize that the results of rigorous measurements of CPDF data accuracy, i.e., measurements designed to test the accuracy of individual agencies’ data, could help users of CPDF data determine if the data are sufficiently accurate for their purposes. However, OPM officials believe the cost of doing such measurements would be prohibitive and would not guarantee that users would consider the measurements when working with CPDF data or that agencies would use the results of the measurements to improve the accuracy of their CPDF data submissions.

OPM’s Office of the Actuaries Reported That CPDF Data Are Sufficiently Accurate for Estimating the Government’s Liability for Future Retirement Payments

OPM’s Office of the Actuaries uses CPDF data to help estimate the federal government’s liability for future payments of federal retirement programs. According to the actuary responsible for calculating the federal government’s liability for future retirement payments to federal employees and their survivors, the office uses CPDF data on adjusted basic pay, sex, birth date, retirement plan, and service computation date in calculating the estimate of this liability.

We discussed with the actuary the error rates we found for these data elements both as measured in our employee questionnaire and in comparison to official personnel folders and records. Except for adjusted base pay, which was about 94-percent accurate in our nongeneralizable comparison of official personnel folders and CPDF data, we found all of these data to be 99 percent or more accurate. We shared these results with the actuary, and he told us that the CPDF data elements were sufficiently accurate for making the liability estimate. The actuary also told us that erroneous national economic assumptions were much more likely to affect his estimate than inaccuracies in the CPDF data. For instance, the actuary said that slight variances in estimated future interest rates or rates of return on investment could have a significant impact on the government’s estimated liability for future payments. Furthermore, the actuary said that the CPDF is not the only source of information for certain information the office uses for its estimate. For example, the actuary told us that he makes independent calculations of salaries by using data on contributions to pension plans. In addition, OPM received an unqualified opinion on its retirement program financial statements for fiscal year 1997.

Conclusion

Most of the 28 data elements we reviewed were 99 percent or more accurate in the aggregate. A minority of data elements we reviewed, especially education level and rating of record, was much less accurate. OPM has found broadly similar results in its accuracy measurements but has not informed non-OPM users of CPDF data of these results even though the lower level of accuracy for some data elements could affect the validity of analyses relying on those data elements. Further, the accuracy levels that both OPM and we have found are generalizable only governmentwide. Anecdotal evidence from this review and our prior work illustrates that the accuracy of CPDF data elements can vary significantly among agencies. Nevertheless, OPM and non-OPM analysts rely on CPDF data to monitor and report on individual agencies’ demographics, compliance with government policies, and other characteristics. OPM officials said that gauging the accuracy of individual data elements by agency would require a significantly larger measurement sample and thus increase its measurement costs. Informing users of CPDF data of the governmentwide accuracy results and a specific caution that individual agencies’ results may vary significantly could nevertheless be useful. This would allow analysts and those using CPDF products to make better informed judgments before using agency-specific CPDF data and perhaps to seek information to corroborate the CPDF data.

Recommendation to the Director of OPM

We recommend that the Director of OPM make the results of OPM’s measurements of historical accuracy available to all users. To make this information available OPM could post the results of its accuracy measurements on its Internet web site including cautionary language indicating that the accuracy of CPDF data elements may vary by agency. OPM could also inform users of the availability of this information whenever it distributes CPDF data or reports.

Agency Comments and Our Evaluation

In a letter dated September 11, 1998, (see app. VII), the OPM Director said our findings are consistent with OPM’s internal quality measures. In particular, the OPM Director cited our draft report’s findings that CPDF data, including the data used by OPM’s Office of the Actuaries to estimate the government’s liability for future retirement payments, were accurate.

The OPM Director also said that although our findings were positive, she believed many of the report’s headings tended to obscure rather than clarify the findings. In addition, she said that the Results in Brief discussion of CPDF accuracy standards and error rates in education level data is so limited that it presents only our view of CPDF limitations. According to the OPM Director, for “complete and accurate information that provides a more balanced rationale for CPDF specifications, one must look beyond the Results in Brief” to the body of the report.

We believe the view presented in the Results in Brief is balanced. For example, in the first paragraph, we report that about two-thirds of the selected CPDF data elements it reviewed were at least 99-percent accurate. We also disagree that the report’s headings tend to obscure rather than clarify the findings. The report’s title, chapter titles, and main captions note the positive findings of our review. We believe, as the OPM Director acknowledged, that our report clearly states that most of the CPDF data we reviewed were accurate.

The OPM Director did not specifically refer to our recommendation that she make the results of OPM’s historical measurements of the CPDF’s accuracy available to all users. However, she said that OPM will make available appropriate explanatory material to all CPDF users. As stated in this chapter, we believe that this explanatory material should include the accuracy measurements.

USERs Generally Reported CPDF Products Met Their Needs, but Further Awareness of Cautions on CPDF Data Could Affect Use of the Data

We used a questionnaire to determine the extent to which selected CPDF users believed (1) the CPDF data they used met their needs, including whether the products were current, accurate, and complete; and (2) they received sufficient cautions about the limitations of CPDF data to use or present the CPDF data correctly. OPM officials identified 247 CPDF users as representing all of the requesters of CPDF data products who corresponded directly with OPM in 1996. We surveyed those 247, and 40 said they did not use CPDF products. Of the remaining 207, 161 responded to our questionnaire as users of the CPDF. The results of our CPDF customer questionnaire showed that the majority of CPDF users responding believed that CPDF products met their needs, including being sufficiently current, accurate, and complete. However, 29 of the 71 CPDF users said knowing about cautions they were not made aware of would have affected the way they used or presented CPDF data. OPM officials said, and respondents’ answers to our questionnaire indicated, that the extent to which OPM provided users cautions about the general limitations of the CPDF varied. OPM officials said they were considering creating a CPDF web site that would allow OPM to make CPDF data more widely available and to “bundle” or link specific cautions on limitations associated with specific sets of data.

USERs Generally Reported That CPDF Data Met Their Needs Including Being Current, Accurate, and Complete

OPM distributes a variety of CPDF-based products, including data extracts that consist of selected data elements, e.g., “service computation date” or “duty station,” which are provided on tape or diskette to users; recurring reports, such as the Demographic Profile of the Federal Workforce; ad hoc reports containing specific information from the CPDF, such as results of matching CPDF data with other data; and the User Simple and Efficient Retrieval (USER) system, which is an information retrieval system that provides electronic access to the CPDF’s status and dynamics files. The majority of the respondents to our questionnaire reported that the data in the CPDF products they used met their needs, including being current, accurate, and complete.

For example, when asked about the extent to which CPDF products that they used over the past 2 years (i.e., data extracts, recurring reports, ad hoc reports, and the USER system) met their needs, depending on the type of product, 67 to 81 percent of respondents rated CPDF products as meeting their needs to a great or very great extent. When asked about the extent that these products were current enough to meet their needs, the majority of CPDF users responding to this question reported that the CPDF was, to a great or very great extent, current enough to meet their needs. Seventy to 73 percent of the users who answered this question rated the data products we asked about as current enough for their needs to a great or very great extent.

When asked about the extent to which they believed the CPDF products that they used over the past 2 years were accurate, the majority (65 to 87 percent) of users responding to this question rated the products we asked about as accurate to a great or very great extent. Similarly, the majority (71 to 89 percent) of the users responding to our question about the completeness of CPDF data said they believe the products listed were complete to a great or very great extent. Of those users of CPDF products who reported that specific products met their needs to a great or very great extent, a large majority also reported that those products were accurate and complete.

In addition to the data products that we asked about, 15 respondents to our questionnaire reported they used the Installation Level Data Retrieval System (ILDRS)—a database system that uses CPDF data to provide a “snap shot” of a federal agency’s personnel. When asked about the extent to which ILDRS was current enough to meet their needs over the past 2 years, unlike the response we got from most users about the currency of CPDF products, only 4 of these 15 respondents rated it as being current enough to meet their needs to a great or very great extent. Eight of the 15 respondents rated ILDRS as being accurate to a great or very great extent, and 9 of the 15 rated ILDRS as being complete to a great or very great extent.

Most CPDF USERs Said Cautions OPM Provides on Data Limitations Were Sufficient, but Some Said Further Awareness of Cautions Could Affect Use of Data

OWI does not provide users of CPDF products with a uniform set of cautions about the limitations of the data elements contained in the CPDF. The extent of the cautions OPM provides about the limitations of CPDF data to users of CPDF-based products varies because, according to OPM officials, the cautions are tailored to the CPDF product being requested. Users responding to our questionnaire demonstrated a wide range of awareness of caution statements about the CPDF data’s limitations. The majority of users responding to our questionnaire reported that they were aware of the limitations of the data they received and that the caution statements on limitations provided by OPM were sufficient for them to correctly use the data.

OPM Does Not Disclose to Users All the Cautions About the CPDF’s Limitations

Although OPM’s CPDF-based governmentwide and ad hoc reports contained some cautions on limitations, none of the reports we reviewed disclosed all of the cautions on the CPDF. We observed that CPDF products, such as ad hoc reports, that OPM prepares to respond to requests for specific information do not fully disclose all 28 cautions about the limitations of the CPDF that OPM officials identified for us. For example, OPM’s response to a state’s request for CPDF data that were to be used in a data match to identify federal employees by selected data elements, such as pay grade, who graduated from state education and training programs cautioned the requester that the CPDF contains records of personnel only in executive branch agencies. OPM did not warn the requester that OPM’s quality assurance procedures cannot detect agency miscoding of certain data elements, such as pay grade (e.g., submission of grade 11 when the grade is actually 12). In contrast, the recurring reports that are widely distributed and that contain governmentwide statistics, such as OPM’s Biennial Report of Employment by Geographic Area, contained quality measurements of the data in the reports and error rates (i.e., estimated percentage of data elements that failed edit checks) for each of the data elements reported.

OWI analysts routinely monitor and report to agencies submitting data about the quality of their own submissions, that is, the degree to which their data submissions fall within OWI’s acceptable range of data values, or edit standards. This information is also made available within OPM and to certain non-OPM users. For example, information on the percentage of data elements not passing CPDF edits and the quality of the CPDF status and dynamics files is currently available through OPM’s USER system. According to OPM, we, the Equal Employment Opportunity Commission, the Merit Systems Protection Board, the Department of Agriculture, Department of Labor, Environmental Protection Agency, National Guard, National Security Agency, Congressional Budget Office, and the Office of Management and Budget were trained and given access to this system by them. OPM officials reported that they do not know to what extent these agencies use the quality reports available through the USER system.

Although OPM does not make information about the quality of individual agencies’ CPDF submissions directly available to nonfederal and most federal users, it bases some caution statements to users about the limitations of CPDF data on this information. For example, in its Demographic Profile of the Federal Workforce as of September 30, 1996, OPM informed users that about 0.4 percent of the total CPDF records available for the report were rejected because they failed edits on key data elements. OPM also cautions users in correspondence responding to requests for information and in its recurring CPDF-based reports, such as OPM’s Biennial Report of Employment by Geographic Area, about certain general limitations of the data, such as the exclusion of certain agencies’ employees from the CPDF’s population coverage. However, OPM does not caution users about other limitations, such as that OPM may change submitted values that are missing or known to be in error.

Most CPDF Users Said CPDF Products Met Their Needs, but Some Said Further Awareness of Cautions on CPDF Data Could Affect Use of Data

In our questionnaire to CPDF customers, we asked them to indicate how many of the 28 cautions about the CPDF OPM made them aware of. The CPDF users responding to our questionnaire showed a wide range of awareness of the cautions. For example, more than 95 percent of those answering our question about CPDF cautions said they were cautioned by OPM that certain agencies are exempt from reporting to the CPDF. However, only about 34 percent of those answering the question said they were made aware that OPM may change submitted values that are missing or known to be in error by matching records to older files or making values consistent with statistical assumptions. According to OPM officials, these changes rarely happen; and, when they do, they affect only one or two agencies once every four quarterly status files. Overall, from 72 to 86 percent of the users reported that the caution statements on limitations provided by OPM were sufficient for them to correctly use or present the data contained in the various CPDF products they used to a great or very great extent. However, 29 of the 71 CPDF users said knowing about cautions they were not made aware of would have affected the way they used or presented CPDF data.

Of the 28 caution statements about limitations of the CPDF listed on our questionnaire, the 5 that respondents were least aware of were the following: (1) a small number (0.2 percent) of employees have more than 1 record in a CPDF status file; (2) the FBI does not report duty station location for employees outside of the District of Columbia; (3) OPM may change submitted values that are missing or known to be in error by matching records to older files or making values consistent with statistical assumptions; (4) there is no CPDF standard format for submitting employee names; and (5) CPDF status files are generally considered to reflect employment at the end of the quarter, but they might actually reflect employment at the end of the pay period just prior to the end of the quarter.

OWI officials reported that OPM provides information about the specific limitations of a data product to requesters but does not provide information about other limitations, such as the list of 28 caution statements about CPDF data, to all requesters. OPM officials said that making caution statements about CPDF data limitations more widely available might be useful to some users of the data. However, OWI officials believe this alone would not prevent the possible misinterpretation of a specific set of data by a third-party user, i.e., someone who does not receive CPDF data directly from OPM reports or OPM. Because OPM officials are not always aware of the intended use of data requested by users, these officials may not be aware of which of the 28 caution statements would be most beneficial to those users. For example, if a user intended to derive the average education level of the employees of a particular agency but only requested status file data as of a particular date from OPM, OPM officials might not provide the user with the caution statement that some data were collected at the time of appointment, e.g., education level data, but not routinely updated. Therefore, the average education level derived for the agency would not be current and most likely be understated. OWI officials reported they have been considering creating a CPDF web site that would allow OPM to make CPDF data more widely available and allow OPM to bundle specific caution statements on limitations with the sets of data.

Most CPDF USERs Surveyed Rated the Overall Quality of CPDF Products as Excellent or Very Good

OPM sends customer feedback questionnaires to its CPDF users to determine if it is meeting their needs and to solicit suggestions for improvement. We reviewed 149 OPM customer feedback questionnaires for the period covering March 27, 1990, through February 28, 1994, and determined that 140 of the 149 (94 percent) of the CPDF users responding rated the overall quality of the CPDF products they received as very good or excellent. The majority (from about 72 to 84 percent) of the CPDF users responding to our questionnaire also rated the overall quality of the specific CPDF products they used as very good or excellent.

Conclusions

Most of the users of CPDF data we surveyed reported that they believed that the data in those CPDF-based products they used met their needs, including being current, accurate, and complete. The majority of users we sent questionnaires to reported they had received sufficient cautions about the CPDF’s limitations to use or present the data correctly. However, although OPM highlighted cautions about CPDF data that are most likely to be applicable to the interests of a particular requester of those data, it did not make all 28 caution statements available to each of those requesters. Some users reported that knowing about cautions they were not made aware of would have affected the way they used or presented CPDF data. In addition, users who obtain CPDF data regularly without a specific request to OPM may not be cautioned about the limitations associated with using the data.

Recommendation to the Director of OPM

We recommend that the Director of OPM ensure that OPM make all 28 caution statements about limitations associated with CPDF data available to all users. In addition, it may be useful for OPM to continue its practice of highlighting cautions on the data limitations of the CPDF that are most likely to be applicable to the interests of a particular requester of CPDF data. To make this information available to all users, OPM could (1) post, on its Internet web site, a complete listing of the 28 caution statements about limitations associated with CPDF data, (2) apprise all recipients of CPDF data of the availability of the caution statements, and (3) implement its proposal to bundle specific cautions on limitations associated with specific sets of data.

Agency Comments and Our Evaluation

In a letter dated September 11, 1998, (see app. VII), the OPM Director said our findings were consistent with OPM’s internal quality measures. The OPM Director cited our draft report’s findings that most of the users of CPDF data we surveyed rated the overall quality of the data excellent to very good and believed they received explanatory material that enabled them to use the data correctly.

The OPM Director also said that although our findings were positive, she believed the Results in Brief section was too skimpy and that many of the report’s headings tended to obscure rather than clarify the findings. We believe the view presented in the Results in Brief is balanced. We also disagree that the report’s headings tend to obscure rather than clarify the findings. The report’s title, chapter titles, and main captions note the positive findings of our review. We believe, as the OPM Director acknowledged, that our report clearly states that most CPDF users’ needs were met.

The OPM Director did not specifically refer to our recommendation that she make all 28 caution statements about limitations associated with CPDF data available to all users. However, she said that OPM will make available appropriate explanatory material to all CPDF users. As stated in this chapter, we believe that this explanatory material should include all 28 caution statements about limitations associated with CPDF data. In addition, the OPM Director identified additional agencies that have access to OPM’s USER system, which we added to the report where appropriate.

System Software Development Not Documented According to Applicable Federal Guidance, but Software Appears to Implement Edits as Intended

From 1976 to 1995 applicable federal guidance recommended that agencies use a structured approach for operating and maintaining automated information systems, such as the Central Personnel Data System. The guidance suggested that agencies document the life cycle of an automated information system from its initiation through installation and operation. Although the guidance was issued before OPM’s major redesign of the System software in 1986, OPM’s OIT did not document changes that were made to the System or have independent testing done to ensure that changes to the software would perform as intended. OIT officials said that to their knowledge the System has not had problems processing data reliably and that the System’s owner, OPM’s OWI, concurred. Our review of 718 of the 763 computer instructions used by the CPDF showed that the System uses instructions that should implement CPDF edits as intended. OIT officials said that for OPM to accomplish its future information technology (IT) goals it will have to follow a structured approach for computer application development. Toward this end, OPM has adopted a software development goal that would require such an approach no later than fiscal year 2002.

OPM Did Not Document an Upgrade of the System’s Software as Recommended in Federal Guidance

From 1976 to 1995, federal guidance issued by the National Bureau of Standards and other federal agencies said that sufficient planning and documentation are needed for cost-effective operation and maintenance of information systems. This guidance described the need for organizations to adopt a structured, or System Development Life Cycle (SDLC), approach. An SDLC approach requires organizations to document the phases of the development life cycle for automated information systems and their software applications, including any changes that are made to the systems or their software. Although federal guidance recommending that agencies follow best practices for automated information systems’ best practices were issued before OPM’s major redesign of the System’s software in 1986, OIT did not document changes that were made to the System. OIT officials said that to their knowledge there was no effect on the System from their not having used the SDLC approach because they believe the System was still reliable without it.

Federal Guidance Recommended Using a Structured Approach to System’s Software Development

From 1976 to 1995, federal guidance existed to assist agencies as they developed computer software applications and made changes in their automated information systems from initiation through operation. For example, on February 15, 1976, the Department of Commerce’s National Bureau of Standards issued the Federal Information Processing Standards (FIPS) Publication 38, which provided basic guidance for the preparation of 10 document types that agencies were to use in the development of computer software. FIPS Publication 64, which was issued on August 1, 1979, provided guidance for determining the content and extent of documentation needed for the initiation phase of the software life cycle. In 1995 the Secretary of Commerce approved the withdrawal of nine such guidelines, including FIPS Publications 38 and 64. However, agencies that find these guidelines useful may continue to use them.

The National Bureau of Standards’ 1988 Guide to Auditing for Controls and Security: A System Development Life Cycle Approach, was to be used as an audit program for auditing automated information systems under development. It included many guidelines that were published from 1976 through 1984 that described the SDLC approach and its requirements, including documentation. This guide also referenced other federal sources that required documentation, including federal information resource management reports and OMB Circular A-130.

The federal government does not follow a single SDLC approach, but an SDLC approach generally includes the following phases: (1) initiation (the recognition of a problem and the identification of a need); (2) definition (the specification of functional requirements and the start of detailed planning); (3) system design (specification of the problem solution); (4) programming and training (the start of testing, evaluation, certification, and installation of programs); (5) evaluation and acceptance (the integration and testing of the system or software); and (6) installation and operation (the implementation and operation of the system or software, the budgeting for it, and the controlling of all changes and the maintenance and modification of the system during its life).

SDLC documentation is important because it provides a basis for (1) systematically making decisions while moving through a system’s life-cycle phases and establishing a baseline for future changes to the system and (2) auditing systems that are under development. According to federal guidance, software acceptance testing, like other testing of the automated information system, must be documented carefully, with traceability of test cases to the system requirements and the acceptance criteria. Without acceptance testing, changes to an automated information system cannot be verified as working as intended.

Ensuring an information system’s reliability is not the only reason for following an SDLC approach. The National Bureau of Standards’ Guide to Auditing for Controls and Security: A System Development Life Cycle Approach states that if agencies use a structured approach to systems development, the probability increases for a well-defined life cycle and compliance to such a cycle. According to the Guide, an unstructured approach leads to free-form system development that may result in serious omissions. Without a structured approach to software applications development, no assurance exists that adequate testing, verification, validation, and certification will be done; resources will be appropriately expended; the anticipated return on investment will be achieved; or user requirements will be met. In addition, without documentation, the history of system changes can be lost if staff changes occur, thus making future system modifications or problem corrections more time-consuming and costly.

During the evaluation and acceptance phase, the computer instructions that have been written or modified undergo testing to verify that they will perform according to user specifications. Although federal guidance said that some changes to the SDLC may be appropriate “if the subject to be addressed is a major modification to a system rather than the development of a new one,” it also said that “the need to continually assess the user’s needs (validation) and to ensure the conceptual integrity of the design (verification) are not arguable.” Thus, evaluation and acceptance testing is a phase that no agency should leave out of an SDLC. As we have described in guidance for the Year 2000 computing challenge, acceptance testing should be done by an independent reviewer. An independent review helps to ensure that internal controls and security are adequate to produce consistently reliable results.

OPM Did Not Document Upgrade of the System’s Software as Recommended in Federal Guidance

According to OPM officials, since the System’s development in 1972, it has gone through only one major software upgrade, which was done in conjunction with the replacement of the System’s hardware. According to an OIT official, in 1985, OPM replaced its existing Honeywell computer with an IBM computer and converted CPDF application programs to run on the new hardware. He also reported that at about the same time, OPM decided to upgrade CPDF capabilities by procuring several commercial software packages as well as designing customized software. According to OIT managers, the software upgrade was done in 1986 to improve the timeliness and accuracy of the CPDF because it was not working efficiently. The OIT managers who were responsible for the System at that time told us that OPM did not document the phases of this major system software modification as recommended in applicable federal guidance under an SDLC approach. Other OIT officials also told us that OPM did not follow an SDLC approach for these 1986 CPDF changes or have documentation that would show that acceptance testing was done. In addition, the testing that was done was not done by an independent reviewer. OPM officials said that because of time constraints, OIT staff who designed the software modifications also did the acceptance testing and did not document it.

Although OIT did not follow an SDLC approach and did not have documentation to show that the 1986 software upgrade passed acceptance tests or that subsequent modifications to the System’s software applications worked as intended, its managers said that they believe the System is reliable. They said that they base their beliefs on the fact that OPM’s OWI, the System’s owner, has not complained that the System is not meeting its needs.

The System Appears to Implement CPDF Data Edits Reliably

Because OIT did not document software upgrades and modifications to the System, we could not review this type of documentation as a basis for independently evaluating the extent to which the System is operating as intended. As an alternative, of the 763 total edits (700 call-relational and 63 validity) that the system used at the time we did our work, we reviewed the computer instructions written to implement the 700 (470 dynamic file and 230 status file) call-relational edits and 18 of the 63 validity edits that together check the key status and dynamics data fields. This approach allowed us to indirectly determine if the System would reliably implement CPDF data edits, the computer instructions that are to check the validity of individual data elements.

Putting test data through the System or otherwise testing the reliability of the System’s hardware and software under operating conditions would have allowed us to directly test the reliability of the System. However, we did not attempt to directly test the System’s reliability. OPM officials raised a concern about the possible adverse effects of putting test data through the System. They were concerned that putting test data through the System could disrupt its production schedule and introduce “bad” data that could have unforeseeable consequences on the System’s operations. Because of the lack of any indications that routine System operations to process agencies’ data submissions had caused data errors and the concern raised by OPM, we decided to limit our test of the System’s reliability to a review of the computer instructions the System uses to implement edits.

Through our review, we determined that the computer instructions the System uses would implement as intended the selected CPDF call-relational edits and the validity edits used to identify data inconsistencies in the data elements submitted by agencies. We found only one true error. The computer instructions for a dynamics file call-relational edit that is 1 of 20 subprograms used to edit the prior basic pay data element was written in 1995 but was not applied to agencies’ dynamics file submissions. CPDF programmers attributed this error to a mistake and oversight on their part and not to a lack of documentation.

OPM Has Implicitly Committed to Adopt an SDLC Approach

In January 1997, OPM initiated a project to develop and implement an Information Technology Architecture Vision, which describes the hardware, software, network, and systems management components of the technical infrastructure required to support OPM business applications and data. This project was initiated in response to various federal government initiatives intended to help ensure that government agencies achieve their missions by changing management practices concerning IT investment and operational decisions.

The first phase of this project was the development of an OPM IT architecture vision, which is intended to provide the framework within which OPM can make IT decisions. OPM published its IT architecture vision in December 1997 and has as one of its components a description of the technology infrastructure that will be needed to support OPM’s data and application needs. Under this technology infrastructure component, OPM is to adopt standards for application development and plans to provide training to staff with the goal of reaching a specified software development level of process maturity as described in the Capability Maturity ModelSM (CMM ). was developed by the Software Engineering Institute, which is a federally funded research and development center operated by Carnegie Mellon University. It has as a major purpose guiding process improvement efforts in a software organization. CMM levels—(1) initial, (2) repeatable, (3) defined, (4) managed, and (5) optimizing—to represent evolutionary plateaus on the road to a high level of software process capability. Each maturity level after the first defines several key process areas—groups of related software practices—all of which must be satisfied for an organization to attain that level. An OIT official reported that OPM’s IT is at level 1 and has a goal under its IT architecture vision of reaching level 2 or higher by fiscal year 2002. recommends that an organization use specific software development practices, tools, and methodologies. It does not stipulate how the organization must perform software development or management activities.

For level 2 and higher, CMM requires an agency to define and document an SDLC approach that is to be used in the development, modification, and management of automated information systems and their software applications. Therefore, by adopting level 2 as a goal, OPM also is committing to follow an SDLC approach by fiscal year 2002. Because an applies to the development, modification, and management of all significant systems, once OPM has adopted an SDLC approach, it would need to make changes to the CPDF that would conform to an SDLC approach.

Successfully adopting an SDLC approach would be a significant change for OPM because it said in its IT architecture vision that OPM’s application development style has been situational, with few common approaches to system development. The lack of an SDLC was a repeat material weakness reported in independent audits of the financial statements for fiscal years 1996 and 1997 of the retirement program that was administered by OPM’s Retirement and Insurance Service.

OIT officials told us that they recognize the importance of having an SDLC approach for accomplishing the applications development goals OPM’s IT architecture vision and in its strategic plan for fiscal years 1997 to 2002. In the strategic plan, OPM includes a strategy for ensuring that OPM’s mission-critical computer systems, of which the CPDF is one, are Year 2000 compliant in time to ensure that services to customers are not interrupted. This strategy includes detailed tracking of progress on renovation and testing of each IT system and validating and testing that software changes are working as intended. These steps generally conform to SDLC requirements.

Other than efforts for making its information systems Year 2000 compliant, it is not clear whether OPM would follow an SDLC approach when modifying any other systems, including the CPDF, before fiscal year 2002. Neither the IT architecture vision nor the strategic plan specifically identifies when OPM plans to adopt an agencywide SDLC approach.

Conclusions

OPM has not followed an SDLC approach to software development that includes documenting the phases of such development as recommended in applicable federal guidance. OPM also has not documented the testing of changes to software to verify that those changes worked as intended or had such changes tested by an independent reviewer. Nevertheless, although we did not directly test the System’s hardware and software under operating conditions, our review of the computer instructions the System uses to implement CPDF call-relational and validity edits shows that the System should implement these edits reliably.

OPM has adopted a goal of achieving at least a CMM level 2 by 2002, and doing so would require OPM to define and document an agencywide SDLC approach. OPM’s current significant modification to CPDF and other mission-critical systems to be Year 2000 compliant is following a structured approach like an SDLC, but it is unclear when OPM might adopt an SDLC approach for other future system changes.

Documentation of system changes in part helps agencies make any future system modifications more quickly and cost effectively, and independent review of system or software changes helps ensure that they will work as intended. Therefore, following these procedures for any changes to the CPDF before OPM adopts an agencywide SDLC could be beneficial.

Recommendation to the Director of OPM

We recommend that the Director of OPM document any changes to the CPDF before OPM adopts an agencywide SDLC approach as specified in CMM guidelines and that such changes be independently verified to ensure that they will work as intended.

Agency Comments and Our Evaluation

In a letter dated September 11, 1998, (see app. VII), the OPM Director said our findings are consistent with OPM’s internal quality measures. The OPM Director cited our draft report’s findings that the CPDF edit programs should function well.

The OPM Director also said that although our findings were positive, she believed many of the report’s headings tended to obscure rather than clarify the findings. According to the OPM Director, for “complete and accurate information that provides a more balanced rationale for CPDF specifications, one must look beyond the Results in Brief” to the body of the report.

We believe the view presented in the Results in Brief is balanced. We also disagree that the report’s headings tend to obscure rather than clarify the findings. The report’s title, chapter titles, and main captions note the positive findings of our review. We believe, as the OPM Director acknowledged, that our report clearly states that the System’s edit programs should operate as intended.

The OPM Director agrees with our recommendation that OPM document all future computer system and software changes and perform independent verification that the changes function as intended. She said that OPM is committed to adopting a formal SDLC methodology and is currently in the process of implementing interim measures to ensure that the System is fully documented and continues to function reliably. As an enclosure to her comments, the Director provided OPM’s plans for implementing an SDLC methodology.