Military Personnel: DOD and the Services Need to Take Additional Steps to Improve Mobilization Data for the Reserve Components
Released: 2006-09-20T00:00:00

Letter

Background

In 1975, DOD implemented the Reserve Components Common Personnel Data System (RCCPDS) to collect information on current and past members of the six reserve components—Army National Guard, Air National Guard, Army Reserve, Navy Reserve, Marine Corps Reserve, and Air Force Reserve. This information included data on reservists’ personal characteristics, such as name, Social Security number, date of birth, gender, home address, and education, as well as data on their military characteristics, such as service, reserve component, prior service status, and date of initial entry into the reserve forces. According to the director of DMDC, the services send daily, weekly, and monthly updated data submissions to DMDC in accordance with applicable guidance.

After the first Gulf War, in a May 15, 1991, memorandum, DOD identified 16 recommendations requiring action by many offices within DOD regarding Desert Storm personnel data issues. For example, the memorandum said that DOD should consistently report on who participated in the operations and cites examples of key terms, such as in theater, that were being interpreted differently by DMDC, the Office of the Secretary of Defense, and the services. In December 1991, DOD reported on how DMDC provided information about operations Desert Shield and Desert Storm. This report cited areas for improvement. For example, the report indicated that DMDC created makeshift procedures to establish and maintain the new data sources and to accommodate varied data requests. The report cited that these procedures sometimes resulted in inconsistent or incomplete data being provided in response to a request. On May 2, 2001, DOD updated guidance to the military services, among others, to maintain a centralized database of active duty personnel. In this guidance, DOD requires the services to report personnel information about all active duty military servicemembers as well as reservists who are ordered to active duty. While this instruction called for the services to report information about servicemembers on active duty in support of a contingency, the requirements for reporting contingency data were not specific.

On October 4, 2001, the Under Secretary of Defense, Personnel and Readiness (USD (P&R)), issued a memorandum that required the services to report personnel information to DMDC on all active and reserve component personnel mobilized or deployed in support of GWOT, in accordance with DOD guidance. The purpose of GWOT data was, among other things, to establish eligibility for benefits and entitlements as a result of participation in the named contingencies. The information is critical because it provides a historical database with which to assess the impact of policies and processes, events, and exposures on the health of deployed reserve component servicemembers. DMDC was tasked with providing reporting guidance to the services for these data submissions. DMDC sent this guidance to the services on October 12, 2001.

DMDC is a civilian-led agency with a mission to deliver timely and quality support to its customers, and to ensure that data received from different sources are consistent, accurate, and appropriate when used to respond to inquiries. DMDC reports to the Deputy Under Secretary of Defense for Program Integration, who is in the Office of the USD (P&R) (see fig. 1).

In February 2002, USD (P&R) reminded the services in another memorandum of its earlier requirement for reporting personnel data to DMDC and informed the services that they had 2 weeks to provide plans to DMDC on how they were going to correct any personnel data reporting problems. On August 6, 2004, DOD updated prior guidance regarding RCCPDS to include an enclosure that set out specific requirements for the services to report personnel information for all reserve component servicemembers supporting a named contingency, unlike previous guidance. The purpose of the new enclosure was to ensure more accurate reporting on a named contingency, such as GWOT missions, as well as to establish eligibility for benefits and entitlements, and to develop a registry of participants for tracking in support of research and evaluation of DOD programs and policies. According to DOD officials, the services, in general, were still reporting data according to previous guidance for a few years after the new guidance was issued.

In August 2004, DMDC began operation of its CTS database to address DOD’s reporting requirements, including those in the new enclosure (that is, enclosure 11). The CTS database is DOD’s repository for collecting activation, mobilization, and deployment data for reservists who have served and continue to serve in support of GWOT. The CTS database contains both an activation file, which contains mobilization data, and a deployment file. Both files are updated monthly by service submissions and cover GWOT from September 11, 2001, to the present. The purpose of the activation file is to account for and provide medical and educational benefits for all reservists called to active duty in support of GWOT contingencies, and it allows DOD to provide data on the number of reservists who have been mobilized in support of GWOT. The purpose of the CTS deployment file is to account for a deployed servicemember’s deployment date and location during each deployment event in support of deployment health surveillance and DOD guidance. The database is also used to track and report the number of reservists who have been deployed in support of GWOT since September 11, 2001.

DOD Data Show Demographic and Deployment Characteristics of Hundreds of Thousands of Reservists Deployed in Support of GWOT

Our analysis of DOD data indicates that more than 531,000 reservists have been mobilized in support of GWOT and more than 378,000 reservists, or about 71 percent of the number mobilized, have been deployed in support of GWOT through June 30, 2006 (see fig. 2). The Army National Guard deployed the greatest number of reservists in support of GWOT from September 2001 through June 30, 2006, and, of those, the majority were deployed once. The data also indicate that the vast majority of reservists who deployed in support of GWOT were U.S. citizens, White, and male. Further, the data indicate that most of the reservists spent 1 year or less deployed.

DOD guidance requires the services to report timely, accurate, and complete activation, mobilization, and deployment data. DOD guidance also requires DMDC to collect and maintain mobilization and deployment data from the services about the reservists. DOD is required by policy to report personnel data about reservists, such as service, service component, reserve component category, race, ethnicity, gender, citizenship status, occupation, unit, and volunteer status regarding a current mobilization. In addition, DOD is required by policy to capture deployment information such as the location a reservist is deployed to and the dates the reservist was deployed to that location.

DOD Data on Selected Demographic and Deployment Variables

Number of Reservists Mobilized and Deployed

Our analysis of DOD data indicates that more than 531,000 reservists have been mobilized in support of GWOT and more than 378,000 reservists, or 71 percent of the number mobilized, have been deployed in support of GWOT through June 30, 2006 (see fig. 2). The number of mobilizations and deployments peaked in fiscal year 2003 with about 206,000 reservists mobilized and about 127,000 reservists deployed (see figs. 3 and 4). Since fiscal year 2003, the total number of mobilizations has declined, while the number of deployments remained stable through fiscal year 2005. The Army National Guard has mobilized and deployed the greatest number of reservists—more than 230,000 mobilized and more than 163,000 deployed. The Navy Reserve had the least number of reservists mobilized—with about 29,000—while the Marine Corps Reserve had the fewest number deployed with about 19,000 reservists (see fig. 2). The percentage of the total reservists mobilized or deployed varies across the fiscal years (see figs. 3 and 4). For example, looking at the percentage of mobilizations by component each year, Navy Reserve, Air Force Reserve, and Air National Guard mobilizations occurred early in GWOT and have generally declined over time. Conversely, the percentage of Army National Guard and Army Reserve mobilizations has generally increased over time. The greatest number of Army National Guard deployments—more than 60,000— occurred in fiscal year 2005 (see table 5 totals in app. II), while also in fiscal year 2005, the Army National Guard represented the largest deploying component, with 52 percent of deployments belonging to it (see fig. 4).

Although reservists usually deployed only once, some experienced multiple deployments (see fig. 5). For example, compared to the other reserve components, the Air National Guard and the Air Force Reserve had nearly half of their reservists deploying two and three or more times, but they tend to have shorter deployment cycles according to the Air Expeditionary Force cycle. Under this cycle, reservists deploy for about 120 days in a 20-month cycle. However, servicemembers assigned to stressed specialties deploy for longer periods of time and in greater frequency. At the unit level, some deployment rules have been modified to increase volunteerism or to add stability to key missions. The Army National Guard and the Marine Corps Reserve had the lowest percentage of reservists deploying two and three or more times, but they tend to have longer deployment cycles. In general, DOD policy stipulates that Army units spend 1 year “boots on the ground” in theater. This policy also states that Marine Corps units below the regimental or group level deploy for 7 months while regimental and group headquarters units and above deploy for 12 months. This policy also states that the Chief of Naval Operations’ goal is for servicemembers to have a 6-month deployment with 12 months in a nondeployed status.

Our analysis of DOD data indicates that across the services, the majority of reservists have been deployed once, and of those deployed in support of GWOT, most—about 307,000 reservists, or 81 percent—have spent a year or less deployed. Alternatively, more than 65,000 reservists, or 17 percent, have spent more than 1 year but less than 2 years deployed, and about 6,000 reservists, or fewer than 2 percent, have spent more than 2 years deployed. The data also indicate that the Marine Corps Reserve had the highest percentage of reservists serving more than 2 years. In addition, the data also indicate that very few—less than 1 percent—of Air National Guard reservists served more than 2 years (see fig. 6).

Our analysis of DOD data indicates that most reservists who have deployed in support of GWOT through June 30, 2006, were members of the Selected Reserve (see fig. 7 and table 5 in app. II). The majority of units and individuals in each reserve component are part of the Selected Reserve. These units and individuals have been designated as so essential to the initial wartime mission that they have priority for training, equipment, and personnel over all categories of reservists. Congress authorizes end strength for Selected Reserve personnel each year. The authorized end strength for the Army National Guard has been about 350,000 for the past several years. For fiscal year 2005, data provided by the services to DMDC indicate that the Army National Guard deployed more than 60,000 Selected Reserve servicemembers, which represents the highest number of Selected Reserve servicemembers deployed in a single fiscal year by a single reserve component since GWOT began.

Although the services are authorized a maximum number of selected reservists, the actual number of reservists will fluctuate when additional reservists are recruited or others leave the reserve component. In addition, reservists such as those in the Individual Ready Reserve, are also available for deployment. In general, reservists are trained to have specific skills and specialties and may not be suited to deploy for a specific mission until additional training is provided. In addition, some reservists may not be available for deployment because they are in training, on medical leave, or awaiting training.

Citizenship Status of Reservists at the Time of Their Most Current Deployment in Support of GWOT

Our analysis of DOD data indicates that almost 98 percent of reservists who have deployed in support of GWOT through June 30, 2006, were U.S. citizens at the time of their most current deployment (see fig. 8). The data indicate that about 1 percent of reservists were non-U.S. citizens or non- nationals at the time of their most current deployment. The citizenship status of more than 1,400 reservists was unknown. DOD data also indicate that 168 reservists’ citizenship status changed. Table 1 shows the citizenship status of reservists by reserve component by fiscal year.

Our analysis of DOD data indicates that about 78 percent of those deployed for GWOT were White; about 14 percent were Black or African American; about 2 percent were Asian, Native Hawaiian, or Other Pacific Islander; and about 1 percent were American Indian or Alaskan Native (see table 2). Overall, about 5 percent of the deployed reservists declined to indicate their race. The Army National Guard, the Air National Guard, and the Air Force Reserve had the highest percentages of the reservists who identified themselves as White. Further, about 90 percent of those who responded identified themselves as non-Hispanic and 8 percent as Hispanic (see table 3).

Our analysis of DOD data indicates that about 338,000 reservists, or about 89 percent of the number deployed, were male (see table 4). About 11 percent of those deployed in support of GWOT were female. Of the approximately 163,500 Army National Guard servicemembers who have been deployed through June 30, 2006, more than 92 percent were male. Almost 98 percent of those deployed in support of GWOT through June 30, 2006, for the Marine Corps Reserve were male, representing the highest percentage of males compared with females for all of the reserve components.

Our analysis of DOD data indicates that California, Texas, Pennsylvania, and Florida had the highest numbers of reservists who have deployed in support of GWOT through June 2006 (see table 6 in app. II for the number of reservists deployed by state of residence by reserve component by fiscal year). The 4 states combined had more than 76,000 reservists in residence at the time of their deployments. Eleven states deployed more than 10,000 reservists each, accounting for more than 160,000 reservist deployments. Of those deployed, about 39 percent came from states in the southern United States, about 23 percent from the midwest, about 18 percent from states in the western United States, and about 15 percent came from states in the northeast part of the country. More than 20,000 reservists indicated California or Texas as their state of residence at the time they were deployed (see fig. 9). Nineteen states and 5 territories had fewer than 5,000 reservists in residence at the time of their deployment and 20 states and 1 territory had from 5,000 to 9,999 reservists in residence at the time of their deployment.

Our analysis of DOD data indicates that since GWOT began, the occupational areas of enlisted reservists deployed in support of GWOT have stayed somewhat consistent across all services. For example, the Army National Guard, the Air Force Reserve, and the Marine Corps Reserve have deployed reservists mostly in infantry occupational areas including such groups as infantry, air crew, and combat engineering. All six reserve components have deployed electrical and mechanical equipment repairers, such as automotive, aircraft, and armament and munitions. Three of the six reserve components—the Army National Guard, the Army Reserve, and the Marine Corps Reserve—have deployed reservists who are service and supply handlers, such as law enforcement and motor transport.

Since GWOT began, the occupational areas most deployed for reserve component officers have varied, but all reserve components primarily deployed tactical operations officers, to include ground and naval arms, helicopter pilots, and operations staff subgroups. The Army National Guard, the Air National Guard, and the Navy Reserve have deployed engineering and maintenance officers, such as the communications and radar and aviation maintenance occupational subgroups. The Air National Guard, the Army Reserve, and the Air Force Reserve have deployed reservists in the health care officer occupational areas, including physicians and nurses. The Army Reserve and the Marine Corps Reserve have deployed supply and procurement occupational areas that include transportation, general logistics, and supply occupational subgroups. The Air Force Reserve has also deployed intelligence officers in occupational subgroups such as general intelligence and counterintelligence.

Data for the Volunteer Status, Location Deployed, and Unit Deployed Variables Were Either Not Available or Not Reliable

We were unable to analyze the volunteer status variable because the data do not exist for all of the reserve components. Similarly, we were unable to analyze the deployment location and deployment unit variables because we determined, in agreement with DMDC officials, that the data in these fields were not reliable. This issue is discussed further below.

DOD and the Services Have Taken Steps to Improve the Reliability of Reserve Component Data, but More Needs to Be Done

While we found selected deployment and mobilization data to be sufficiently reliable for our purposes (that is, providing descriptive data), some of the data were not reliable enough for us to report, even for descriptive purposes. DMDC and the services, as required by DOD policy, have taken steps to improve the reliability of the mobilization data; however, more action is needed to improve the reliability of CTS data and DMDC’s analyses of those data. For example, (1) the rebaselining effort resulted in substantial changes being made to the mobilization data, and the Army—which has mobilized and deployed the largest number of reservists for GWOT—has not completed this rebaselining effort, which the Joint Staff tasked DMDC and the services to do in November 2005; (2) we identified data issues that DOD has not addressed that could further improve the reliability of the data, such as standardizing the use of key terms like deployment; and (3) DMDC does not have effective controls for ensuring the accuracy of its data analyses used to produce reports as required by federal government internal control standards. Although DMDC and DOD have undertaken a major data cleaning—or rebaselining—effort to improve the reliability of mobilization data, the effort does not address some fundamental data quality issues. While we recognize that such a large-scale effort, although replete with challenges, is a positive step toward better quality data, if data reporting requirements and definitions are not uniform, and if there are no quality reviews of DMDC’s analyses, some data elements and DMDC’s analyses of those data may continue to be unreliable. A senior DMDC official stated that it emphasizes getting data to customers in a timely manner rather than documenting the internal control procedures needed to improve the reliability of the data and the data analyses produced. However, with proper internal controls, DMDC could potentially achieve both timeliness and accuracy. Without reliable data and analyses, DOD cannot make sound data-driven decisions about reserve force availability. Moreover, DOD may not be able to link reservists’ locations with exposure to medical hazards.

DMDC and the Services Are Updating the Mobilization Data in CTS, but Concerns Remain

We have found the deployment and mobilization data we used to be sufficiently reliable for our purposes (that is, providing descriptive data), and DMDC and the services have recently taken steps to improve the reliability of mobilization data. However, additional steps are needed to make mobilization data more reliable. As previously noted, DOD guidance requires the services to report timely, accurate, and complete activation, mobilization, and deployment data. DMDC officials responsible for overseeing the CTS database stated that a rebaseline of the deployment data was not necessary because the deployment data matched the data in the Defense Finance and Accounting Service’s (DFAS) systems by more than 98 percent. Although DMDC and the services rebaselining of the mobilization data in CTS has resulted in improvements, the Army, which has mobilized the greatest number of reservists for GWOT, has not completed its rebaselining effort. A senior-level DMDC official responsible for overseeing the CTS database said that the mobilization data in the CTS database prior to the rebaselining effort were less than 80 percent accurate for the Army, the Navy, and the Air Force, but that the Marine Corps’ data were generally considered to be accurate prior to the rebaselining effort. The official also stated that DMDC expects that the mobilization data within the CTS database will be 90 percent accurate because of this rebaselining effort, which was still ongoing through August 2006. While we recognize that this is a considerable undertaking, to date, only the Navy and the Air Force have validated or certified their mobilization data files. Navy officials said that the Navy has validated its personnel records and established a common baseline of data with DMDC. Air Force Reserve officials said that their data within CTS are now 99 to 100 percent accurate. The Chief of the Personnel Data Systems Division for the Air National Guard certified that although file discrepancies are still being reconciled, the data that were processed by DMDC on June 11, 2006, were the most accurate activation data and that data accuracy will improve with each future file sent to DMDC. The DMDC official said that the Marine Corps had only partially completed its rebaselining effort and would not be finished until the Marine Corps provided its August 2006 data file in September 2006. The Army National Guard and Army Reserve are still working to rebaseline their mobilization data, and the Army has not provided a time frame for completing the effort.

However, we still have concerns regarding the reliability of the mobilization data, because the scope of the rebaselining effort changed and the data changed substantially as a result of the rebaselining. At the beginning of our review, DMDC and the services referred to the rebaselining effort as a “reconciliation,” which, according to a DMDC official and a Reserve Affairs official, would have resulted in all data (current and past) being reviewed and corrected as needed. We acknowledge that some degree of change is expected in any data cleaning effort, especially with large-scale, multisource collection methods such as DMDC’s data collection process. However, our experience has shown that cleaning efforts that result in a large degree of change would suggest systematic error. Such error raises concerns about the reliability of both the original data and the “cleaned” data. If both the source data and the cleaned data are populated with the same assumptions and information, any reconciliation of data points should result in relatively small change that correct simply for random error, such as from keypunch or data source errors. However, for some variables, the data changed substantially as a result of DMDC and the services’ rebaselining or data cleaning effort. Our analysis shows that data from the period of September 2001 through December 2005 have changed by about 4 percent to as much as 20 percent. For example: The number of reservists mobilized for GWOT through December 2005 went from about 478,000 to about 506,000—an increase of more than 27,000 reservists or a change of more than 5 percent.

The Army Reserve data sustained the greatest change during this time with a more than 19 percent increase in the number of reservists mobilized. The number of mobilized Army National Guard reservists increased more than 7 percent. According to a senior DMDC official, the Army data are expected to continue to change, perhaps substantially enough to require the rebaselining of the data again in the future.

The number of Air National Guard reservists mobilized decreased by more than 13 percent.

The Navy Reserve, the Marine Corps Reserve, and the Air Force Reserve data all changed about 5 percent.

DOD officials stated that the rebaselining effort occurred because the Joint Staff tasked DMDC and the services with ensuring that the data the Joint Staff’s Manpower and Personnel office was using in CTS were the same data as the services were using to determine reserve force availability. According to a senior-level DMDC official responsible for overseeing CTS, the rebaselining effort’s scope changed because all of the services agreed that starting over and replacing all of the data would make more sense than trying to correct transactions already in CTS, because the services found errors in the CTS files initially used for the reconciliation. Service officials said that some of the data discrepancies developed because of a DMDC quality check procedure that sometimes resulted in DMDC replacing the service-submitted data with data from other sources. DMDC officials said that they did this because the services were unable to report some of the required CTS data. According to DMDC officials, service submissions have become more complete over time, resulting in DMDC now using the quality check procedures only to check the data rather than to populate the CTS database. This DMDC official stated that DMDC expected the data to change substantially based on the issues identified with service data during the initial reconciliation effort and the subsequent rebaselining effort.

Because the rebaselining effort is not complete and the Army—which has mobilized and deployed the largest number of reservists for GWOT—has not finished the rebaselining, we do not know how much the data will continue to change as DMDC and the services work to finish this effort. DOD data on reservists’ mobilizations and deployments are important because decision makers at DOD and in Congress need the data to make sound decisions about personnel issues and for planning and budgeting purposes.

DOD Has Not Fully Addressed Data Issues That Could Improve Data Reliability

Prior to the rebaselining effort, some services recognized that there were data issues that needed to be addressed and took steps to do so, as DOD guidance requires the services to report accurate and complete mobilization and deployment data. However, some data issues that would ensure more accurate, complete, and consistent mobilization and deployment data across the services in the future have not been fully addressed by DOD. Some examples of data issues being addressed include the following: The Air Force and the Navy were having difficulty tracking mobilizations based on reservists’ mobilization orders, which has resulted in both services independently working to develop and implement systems that write reservists’ orders.

The Army Reserve recently began to modify its mobilization systems, which Army officials expect will improve the collection of reservists’ mobilization data.

The Air Force identified problems with the way in which the Defense Eligibility Enrollment Reporting System (DEERS) processed end dates for reservists’ mobilizations, which resulted in some reservists not receiving appropriate benefits (for example, dental benefits). Air Force officials worked with officials from the Office of the Secretary of Defense and DMDC to identify and address the data processing logic issues.

Despite these positive steps, service process improvements are not all complete, and further, there has been no comprehensive review across DOD to identify data issues that if addressed, could result in more complete, accurate, and consistent mobilization and deployment data across and within the services. Reserve Affairs officials in the office of Reserve Systems Integration said that a more sustainable fix to the processes of collecting data is needed to ensure that data captured in the future are accurate and more efficiently collected. We agree and have identified some issues that may continue to affect data reliability, such as the following: The use of terms, such as activated, mobilized, and deployed, has not been standardized across the services. Although the department has defined these terms in the Department of Defense Dictionary of Military and Associated Terms, the terms are used differently by the individual services. In the Air Force, “activation” can refer to the time when a reservist either volunteers or is involuntarily mobilized; however, the term “mobilized” refers only to someone who is not a volunteer. Even within a single service, these words can have different meanings. For example, an Army National Guard official who participated in the rebaselining effort said that Army National Guard servicemembers who backfill active duty servicemembers are not considered deployed since they have not left the United States. However, according to this official, some staff in the Army National Guard use “deployed” to include reservists who are mobilized within the United States.

There is no single data entry process that would minimize the potential for contradictory data about reservists in multiple systems. Currently, data about reservists are entered separately into multiple systems.

There is no mechanism for DMDC to ensure that the services are addressing the data inconsistencies DMDC identifies during its ongoing, monthly validation process, such as Social Security numbers that are duplicated in two reserve components.

DOD has taken an ad hoc, episodic approach to identifying data reporting requirements and to addressing data issues. DOD has periodically issued policies regarding its need to collect and report specific data, such as volunteer status and location deployed, about active duty servicemembers and reservists. As a result of changing requirements, many of these policies have addendums that include these additional data requirements, which are not immediately supported by the services’ existing systems that are used to collect the data. Over time, this has led to disjointed policies that overlap and that require the services to modify their existing systems and processes, which can take months to complete.

There are incomplete data submissions across the services. Specifically, data for volunteer status was not available in CTS for all service components, and the location deployed and deploying unit data were not reliable enough for the purposes of this report. Only three of the six reserve components—the Air National Guard, the Marine Corps Reserve, and the Air Force Reserve—provide information on a reservist’s volunteer status, which neither we nor DMDC report because it is not available for all six components. Further, DMDC officials said that they consider CTS location data incomplete although the data are improving with each fiscal year. DMDC officials said that most unit information is based on the unit a reservist is assigned to and may not represent the unit the reservist is currently deployed with in theater. For this reason, we did not consider these data reliable enough to report.

A DMDC official stated that DMDC does not have the authority to direct the services to correct data errors or inconsistencies or to address data issues. DMDC does, however, work with the services and tries to identify and address data challenges. According to some service officials, the department plans to implement a new, integrated payroll and personnel system—Defense Integrated Military Human Resource System (DIMHRS)—and that the services have been diverting resources needed to modify their existing systems and relevant processes to support DIMHRS. However, our past work has shown that DOD has encountered a number of challenges with DIMHRS, which is behind schedule, and the current schedule has it available no sooner than April 2008, when the Army is scheduled to begin implementing the system.

In general, service officials said that they are working to collect data on volunteer status, location deployed, and deploying unit; however, Air Force officials stated that they do collect data on location deployed and deploying unit and that these data are accurate and are being provided to DMDC. Army Reserve officials stated that they currently do not have plans to collect data on volunteer status.

DMDC Does Not Have Effective Controls for Ensuring the Accuracy of Its Data Analyses Used to Produce Reports

DMDC has not documented (1) its procedures for verifying that the data analyses it performs are correct and (2) the procedures for monthly validation of service data or the procedures used to perform analyses of data. Either of these issues could, if documented as part of DMDC’s verification process, address some of our concerns about internal controls. DMDC is required by policy to develop and produce reports about mobilization data and respond to requests for information about deployed personnel. DOD policy requires DMDC and the reserve components to ensure the accuracy of files and the resulting reports. Federal government internal control standards require that data control activities, such as edit checks, verifications, and reconciliations, be conducted and documented to help provide reasonable assurance that agency objectives are being met.

DMDC officials said that they have internal verification procedures that require supervisors to review all data analyses used to generate reports, although these procedures are not documented. Specifically, the supervisors are to review (1) the statistical programming code used to generate the data analyses to ensure that the code includes the customer’s data analyses parameters (that is, the assumptions used to produce the analyses) and (2) the “totals” generated to ensure that these totals match the control totals that show the number of reservists currently or ever mobilized or deployed in support of GWOT. DMDC officials acknowledge the importance of verifying the accuracy of the data analyses prior to providing the reports to customers, and they stated that they had verified the accuracy of the analyses provided to us. However, we found numerous errors in the initial and subsequent analyses we received of the GWOT data through May 2006, causing us to question whether DMDC verified the data analyses it provided to us and, if it did, whether the current process is adequate. For example, we found that DMDC had done the following: Counted reservists with more than one deployment during GWOT also among those who deployed only once during GWOT, which resulted in overcounting the number of reservists’ deployments.

Used ethnicity responses to identify race despite having told us that the internal policy was changed in 2006 and that this was no longer an acceptable practice.

Counted reservists whose ethnicity was “unknown” as “non-Hispanic” although “unknown” does not necessarily mean someone’s ethnicity is “non-Hispanic” and there was a category for unknowns.

Repeatedly categorized data based on a reservist’s first deployment (when there was more than one) despite agreeing to modify this analytical assumption so that we could present data by the reservist’s most current deployment.

Reported thousands of reservists as having changed citizenship status during GWOT although, in our analyses, we found that only 168 reservists had changed status.

Analyzed data by reserve component categories (for example, Selected Reserve and Individual Ready Reserve) rather than by reserve component as we had asked. By analyzing the number of days a reservist was deployed by reserve component category, a reservist could be counted multiple times within one component if he or she changed category. This error affected the way in which the total number of days a reservist was deployed was calculated. For example, if the same reservist served 350 days as an Army National Guard Selected Reserve member and an additional 350 days as an Army National Guard Individual Ready Reserve member, he or she would be counted as two reservists who were each deployed for less than a year. However, our intent was to report that the same individual had been deployed for a total of 700 days. In our analysis, all of a reservist’s days deployed were totaled and counted once for each reserve component, regardless of which category he or she belonged to when deployed.

Miscoded the end date for the analysis of how many days reservists were deployed for GWOT. This resulted in up to an additional 90 days of deployment being counted for reservists who were still deployed at the time the data were submitted to DMDC.

In our discussions with DMDC officials, they readily acknowledged that errors had been made, although they stated that the analyses had undergone supervisory review prior to our receiving them. During these discussions, we also discovered that many of these errors occurred because DMDC had not used all of our data analyses parameters, although these officials had stated that this was one of the verification process steps followed. Although we were able to work with DMDC officials and identify the analytical assumptions they were going to use to complete our analyses, without documented analytical procedures, it is unclear to what degree the analyses DMDC provides to other users of the data also contain errors since many may not similarly verify the analyses provided to them by DMDC.

In addition, DMDC officials have not documented additional processes that would further support a verification process, such as (1) the ongoing, monthly validation process of service-provided data and (2) the procedures to perform analyses and generate reports, including the assumptions DMDC uses when producing periodic and special reports for customers. In the past, according to the services, the ongoing, monthly validation process DMDC used resulted in two sets of data—one set of service data and one set of DMDC data—that may not have been the same. For example, we were told by the Air Force that, in some cases, service data were replaced with default values because of a business rule that DMDC applied to the data and that this change resulted in errors to the service-provided data. These inconsistent data caused the Joint Staff to request that the services and DMDC reconcile the data. As stated above, there were errors in the analyses performed to generate the reports DMDC provided to us, including DMDC’s not using many of the assumptions we agreed to for the analyses. DMDC also made errors that contradicted its own undocumented policy.

A senior DMDC official said DMDC has not documented these procedures because the organization emphasizes getting data and reports to its customers in a timely manner rather than preparing this documentation. This official said that documentation is not a top priority because situations change rapidly, and it would be hard to keep these documents up-to-date. The official also said that the errors made in the analyses provided to us were caused by human error and the need to provide data quickly. Further, the DMDC official said that while there are standard data requests that are generated frequently, GAO’s request was an ad hoc request, and the procedures for addressing such requests, in practice, are not as well defined. While we agree that our requests met DMDC’s definition of an ad hoc request, we disagree that sufficient time was not allowed for DMDC to prepare the analyses. For the initial request, we worked with DMDC over the course of about 5 business days to define the analytical assumptions that would be used during the analysis. DMDC then took about 8 business days to complete the analysis and provide it to us.

DOD data analyses are important because decision makers at DOD and in Congress need the data to make sound decisions about reserve force availability, medical surveillance, and planning and budgeting. In the absence of documented procedures and the necessary controls to ensure that they are implemented, it is difficult for an organization to ensure that it has established a robust process that is being consistently applied and that accurate results are being achieved. Joint Staff and Reserve Affairs officials are emphasizing the need to use one data source for most analyses to further reduce the inconsistencies in data analyses because service-produced analyses and DMDC-produced analyses could differ if both are not using the same set of data and assumptions. Otherwise, it is possible that the data analyses provided to decision makers at DOD or in Congress will be incomplete and inconsistent. If the data analyses are incorrect, users could draw erroneous conclusions based on the data, which could lead to policies that affect reservists in unanticipated ways.

Conclusions

DOD recognizes the need for accurate, complete, and consistent data and data analyses, and it has taken some preliminary, ad hoc steps to improve its data, including undertaking a considerable effort to rebaseline its mobilization data. It has not, however, addressed some of the inconsistencies in data and data analyses departmentwide, such as when terms are used differently from one service to the next. Further, service officials stated that it is anticipated that a lot of these problems will be addressed when DIMHRS is implemented. However, the schedule for DIMHRS continues to slip, so it is unclear when this solution will be available. We recognize that the need for accurate, complete, and consistent data and data analyses about reservist mobilization and deployment is always important, and even more so during higher levels of mobilization and deployment, such as is the case now with GWOT. This is especially true since, in general, there are restrictions on the maximum length of time a reservist can be involuntarily activated. Thus, having accurate and complete data on a reservist’s status is critical for determining availability for future deployments. This is especially true of the CTS data since the Manpower and Personnel office in the Joint Staff and the Office of the Assistant Secretary of Defense for Reserve Affairs mostly use the data found in CTS. These data also help DOD and Congress to understand the potential impacts of policy decisions as they relate to reservists who are eligible for TRICARE Reserve Select and educational benefits based on the number of days a reservist is deployed. DOD has not provided guidance to the services to better define and standardize the use of key terms. DOD also has not collected and maintained all essential data nor has it established a process for ensuring that data inconsistencies are resolved. Further, DOD has not documented key procedures and processes for verifying the data analyses it provides to its customers, thus compromising its ability to ensure the accuracy, completeness, and consistency of these analyses. Until decision makers in DOD and Congress have accurate, complete, and consistent data and analyses, they will not be in the best position to make informed decisions about the myriad of reserve deployment matters.

Recommendations for Executive Action

We recommend that the Secretary of Defense take the following four actions: Direct the Under Secretary of Defense, Personnel and Readiness, to provide guidance to the services to better define and standardize the use of key terms, like activation, mobilization, and deployment, to promote the completeness, accuracy, and consistency of the data within CTS.

Direct the service secretaries to (1) take the steps necessary to provide all required data to DMDC, such as volunteer status and location deployed, and (2) have the services address data inconsistencies identified by DMDC.

Direct the service secretaries to establish the needed protocols to have the services report data consistent with the guidance above.

Direct the Under Secretary of Defense, Personnel and Readiness, to require DMDC to document its internal procedures and processes, including the assumptions it uses in data analyses. In doing this, the Under Secretary of Defense, Personnel and Readiness, should collaborate on the reasonableness of the assumptions established and used by DMDC in its data analyses with the Office of the Assistant Secretary of Defense for Reserve Affairs and the Joint Staff.

Agency Comments and Our Evaluation

The Under Secretary of Defense, Personnel and Readiness, provided written comments on a draft of this report and stated that we changed one of our original audit objectives and did not inform the department of this change. We disagree. While the scope of our audit did change after our initial notification letter of June 17, 2005, was sent to DOD, we notified the proper officials of this change in a December 2, 2005, email to the agency- designated liaison within the DOD Inspector General’s office. In this email, we specifically said that we would be contacting DMDC and that we would be focusing on data for reserve component activation, mobilization, and deployment for GWOT. In accordance with generally accepted government auditing standards (GAGAS), GAO analysts are expected, as appropriate, to review an agency’s internal controls as they relate to the scope of the performance audit. Specifically, we are required by GAGAS to review the reliability of the data and the data analyses provided to us. To assess the reliability of data and data analyses, we often review an agency’s internal controls that are put in place to ensure the accuracy of the data and analyses. As we discuss in our report, we found the data to be sufficiently reliable for our purposes. However, over the course of the work, the analyses of the data DMDC provided to us continued to have errors. This raised concerns about the adequacy of DMDC’s internal controls for preparing and verifying these analyses, which DMDC stated were not documented. In accordance with GAGAS, when reporting on the results of their work, auditors are responsible for disclosing all material or significant facts known to them which, if not disclosed, could mislead knowledgeable users or misrepresent the results. Consistent errors in DMDC’s analyses led us to include an audit objective on the reliability of the data and the data analyses.

In its written comments, DOD generally concurred with three of our recommendations and did not concur with one of our recommendations. DOD also provided technical comments, which we have incorporated in the report, as appropriate. Regarding our recommendation that DOD provide guidance to the services to better define and standardize the use of key terms, DOD stated that this requirement has already been addressed because these terms are defined. We acknowledged in our draft report that these key terms are defined in the Department of Defense Dictionary of Military and Associated Terms. However, as we state in our report, our audit work indicates that the services are not operationalizing the use of the terms in a consistent manner. The intent of our recommendation is to have DOD standardize the use of the key terms across the services.

DOD generally concurred with our recommendation that the services provide all required data to DMDC and address data inconsistencies, and stated that the services have been directed to provide all necessary data and are working to address data inconsistencies. While we agree that the services are working with DMDC to address data inconsistencies with regard to the rebaselining of mobilization data, we also identified other data inconsistencies that DOD has not addressed, such as Social Security numbers that are duplicated in more than one reserve component. We agree with DOD that some requirements cannot be immediately supported by service data systems and modifications to them can take time to complete. However, as our report notes, some service officials stated that resources are being diverted from these efforts to the DIMHRS program, which we reported is behind schedule. We continue to observe the need for the services to provide all necessary data, to address these data inconsistencies, and to establish needed protocols to have the services report data consistent with DOD guidance, especially since the data are used to determine reserve force availability and for medical surveillance.

DOD also generally concurred with our recommendation that DMDC document its internal procedures and processes, including the assumptions it uses in data analyses. In its written comments, DOD stated that DMDC is in the process of developing documentation on its internal procedures and processes and has a draft that addresses the processes used from receipt of the data from the service components to the final quality control of the consolidated file. DOD also stated that DMDC has a draft product regarding many of the data analyses procedures used. During this engagement, we asked if these procedures and processes were documented. As we say in the report, DMDC stated that they were undocumented and that documenting them was not a priority. Although DOD stated that it is in the process of drafting these procedures and processes, we were never provided a draft of these documents. DOD also stated that while DMDC attempts to document the assumptions made in resulting report titles and footnotes, the disclosure of assumptions used in data analyses remain the responsibility of the requester of the data analyses. Although we agree that the requesters of the data bear responsibility to disclose the analytical assumptions used in the data analyses, our audit work indicates that there are basic assumptions that DMDC establishes and uses that, if documented and discussed with those who request data analyses, would allow the users to understand how the information can be used, as well as the limitations of the data analyses. For example, during a discussion with a Reserve Affairs official, who uses the data analyses provided by DMDC to provide information to senior DOD officials, we stated that DMDC defaults to using a servicemember’s first deployment rather than the most current deployment when preparing data analyses. This official was unaware that DMDC used this assumption and stated that the expectation was that DMDC was using the most current deployment to generate the analyses. This official planned to discuss this issue with DMDC in the future.

In its written comments, DOD did not concur with what it characterized as our fourth recommendation. Specifically, DOD separated a single recommendation into two recommendations. In the draft report we sent to DOD, the recommendation read: “We recommend that the Secretary of Defense direct the Under Secretary of Defense, Personnel and Readiness, to require DMDC to document its internal procedures and processes, including the assumptions it uses in data analyses. In doing this, the Under Secretary of Defense, Personnel and Readiness, should collaborate on the reasonableness of the assumptions used by DMDC in its data analyses with the Office of the Assistant Secretary of Defense for Reserve Affairs and the Joint Staff.” DOD stated that DMDC is a support organization that generates reports for a multitude of organizations and that each organization that requests reports provides the assumptions that DMDC uses to develop the reports. However, our audit work showed that DMDC has established and uses some basic assumptions in analyzing data and that DMDC may not always discuss these assumptions with other DOD offices, such as Reserve Affairs. As a result, we continue to emphasize the need for DMDC to document these assumptions and to collaborate with these offices to ensure a common understanding of these assumptions. Although DOD organizations can request data analyses using multiple assumptions, without written documentation other organizations may not be fully aware of the analytical assumptions used by DMDC and this may lead to miscommunication and, ultimately, the data analyses may not be valid in that it does not report what the user intended. We continue to believe that the assumptions used need to be documented and discussed with other DOD offices as we recommended. Based on DOD’s comments, we modified this recommendation to clarify our intent.

We are sending copies of this report to the Secretary of Defense; the Secretaries of the Army, the Navy, and the Air Force; the Commandant of the Marine Corps; the Under Secretary of Defense, Personnel and Readiness; and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions on this report, please contact me at (202) 512-5559 or stewartd@gao. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

Our objectives were to determine (1) what Department of Defense (DOD) data indicate are the number of reservists mobilized and deployed in support of the Global War on Terrorism (GWOT), and the selected demographic and deployment characteristics of those deployed and (2) whether DOD’s reserve deployment and mobilization data and analyses are reliable.

We identified, based on congressional interest and our knowledge of DOD issues, selected demographic and deployment variables to review. We then worked with the Defense Manpower Data Center (DMDC) to identify the data fields within DMDC’s Contingency Tracking System (CTS) that best provided information about the selected demographic and deployment variables we wanted to analyze. Although we wanted to analyze the locations to which reservists were deployed and the units with which reservists were deployed, DMDC officials said, and we agreed based on our review of the data, that the data were not reliable enough for those purposes. Our selected variables included the number of deployed reservists who volunteered for at least one deployment; the number of deployed reservists who have served one, two, or three or more deployments; the race and ethnicity of the deployed reservists; the gender of the deployed reservists; the state of residence of the deployed reservists; the number of deployed reservists who were Selected Reserve, Individual Ready Reserve, Standby Reserve, or Retired Reserve; the number of deployed reservists who were citizens at the time of their deployment; the number of days the reservists were deployed; and the top occupational areas for reservists deployed in support of GWOT.

To address objective 1, we obtained and analyzed data for September 2001 through June 2006 from DMDC’s CTS. CTS consists of two files—the activation file, which tracks activations and mobilizations, and the deployment file, which tracks deployments. Using CTS data from both files, we analyzed the number of National Guard and Reserve servicemembers mobilized and deployed in support of GWOT, as well as selected demographic and deployment variables, using statistical analysis software.

To address objective 2, we performed a data reliability assessment on the data provided by DMDC from CTS’ activation and deployment files. We requested DMDC reports that replicated our analyses and then compared those report results to our analyses, and we reviewed the programming code DMDC used to generate those reports. To assess the reliability of CTS data, we obtained an understanding of the data, the file structure, the sources of the data, and relevant DOD guidance. Specifically, we (1) performed electronic testing of the data files for completeness (that is, missing data), out-of-range values, and dates outside of valid time frames; (2) assessed the relationships among data elements (for example, determining whether deployment dates were overlapping since each record in the deployment file is intended to represent one deployment); (3) reviewed existing information about the data and the systems that produced them; (4) interviewed department officials to identify known problems or limitations in the data, as well as to understand the relationship between the two files and how data are received from the services, cleaned (“rebaselined”), and processed by DMDC; and (5) compared “prerebaselined” mobilization data to “postrebaselined” mobilization data to determine the extent to which the data changed as a result of the cleaning effort. When we found discrepancies (for example, overlapping deployment dates), we worked with DMDC to understand the discrepancies.

In our interviews with DMDC officials, we discussed the purpose and uses of CTS, the service data rebaselining effort and the internal controls for verifying data analyses, monthly validation of data, and performing data analyses. Similarly, we discussed data collection, processing, and reliability issues as well as service-specific data issues and the rebaselining effort with officials from the Office of the Assistant Secretary of Defense for Reserve Affairs and from each of the reserve components, including the U.S. Army National Guard, the U.S. Air National Guard, the U.S. Army Reserve, the U.S. Navy Reserve, the U.S. Marine Corps Reserve, and the U.S. Air Force Reserve. We also discussed the reliability of the services’ data, the rebaselining effort, and the results of a previous Joint Staff review of the quality of service data within CTS with officials in the Joint Chiefs of Staff Manpower and Personnel office. Finally, we interviewed officials from the Deployment Health Surveillance Directorate and the Army Medical Surveillance Activity about the quality of the deployment data and how they use the data.

In the course of our review, we determined that some data fields were highly unreliable. For example, electronic testing indicated that data on location and reservist unit information were missing in many cases. Based on our conversations with DMDC and our understanding of the data system, we decided not to conduct lower level analyses (for example, analyses of reservists’ assigned units) because the results would be less reliable than aggregate level analyses. Although we are reasonably confident in the reliability of most CTS data fields at the aggregate level, because we could not compare source documentation from each of the services to a sample of DMDC data, we could not estimate precise margins of error. Consequently, we used the data for descriptive purposes, and we did not base any recommendations on the results of our analyses. In addition, we presented only higher level, aggregate data from fields that we determined were sufficiently reliable for our reporting purposes. For these purposes, and presented in this way, the CTS data we use are sufficiently reliable with the following caveat: The Army had not completed its rebaselining effort for mobilization data before the completion of our review, and we could not, therefore, assess the reliability of Army mobilization data to the same extent as those of the other services. However, based on our electronic testing, data comparisons, and interviews with officials, we believe that the data are sufficiently reliable to present as descriptive information.

To assess the reliability of DMDC’s reports (that is, its own analyses) of CTS data, we compared our independent analyses of National Guard and Reserve servicemembers’ mobilization and deployment statistics with results that DMDC provided from its own analyses of the same data. To pinpoint differences in analytical assumptions, we reviewed the statistical code DMDC used to produce its reports and compared it with our programming code. Through an iterative process, we noted errors in DMDC’s programs and requested changes and reruns of the data. We worked with DMDC to ensure that discrepancies were not caused by differences in our analytical assumptions. Where there were discrepancies, we reached the following consensus on how to address them: Removed the Coast Guard entries from our analyses of the CTS database since, as we state in this report, the Coast Guard Reserve is under the day- to-day control of the Department of Homeland Security rather than DOD.

Combined a reservist’s Social Security number with his or her reserve component to create a unique identifier. DMDC officials said they do this because they are unsure where the source of the error is when they find that a Social Security number corresponds with two reserve components for a deployment during approximately the same time period. DOD’s policy, when there is a duplicate Social Security number for more than one reserve component, is to count both transactions. However, the use of duplicate Social Security numbers results in overcounting. Specifically, the June 2006 file had 38 reservists with overlapping mobilizations, 20 reservists with overlapping deployments, and more than 800 deployed reservists who appeared to have legitimately changed components. To compensate for the 58 “errors” where DMDC did not know which mobilization or deployment to count, it double-counted all 58 reservists. Likewise, the 800 deployed reservists who changed reserve components during GWOT were also double-counted.

Removed reservists from all analyses when their reserve component category is unknown, so that the numeric totals across analyses would be consistent. DMDC officials said that this is an undocumented standard operating procedure.

Utilized the reservists’ information for most recent deployment to provide the most current information possible in cases where a reservist deployed more than once.

Calculated the length of a reservist’s deployment by including both the day the deployment began and the day on which the deployment ended. Thus, the number of days deployed is inclusive of the beginning and end dates.

Combined the race categories for Asian, Native Hawaiian, and Other Pacific Islander because, prior to 2003, the distinction between these two groups was not captured in the data.

After clarifying and agreeing on the analytical assumptions, we again reviewed DMDC’s code and compared its results with our own to determine whether and why there were remaining discrepancies. We also requested written documentation of DMDC’s internal control procedures for the CTS data and, when no documentation was available, interviewed knowledgeable officials about existing internal control procedures. Using the framework of standards for internal control for the federal government, we compared the information from those documents and interviews with our numerous, iterative reviews of DMDC’s statistical programs used to generate comparative reports to assess the reliability of DMDC-generated reports from CTS. We determined that the reports DMDC generated for our review were not sufficiently reliable for our reporting purpose. Thus, we completed our own data analyses.

We performed our work from December 2005 through August 2006 in accordance with generally accepted government auditing standards.

Appendix II: Data on Reservists Deployed in Support of GWOT through June 30, 2006

Our analysis of DOD data indicates that most reservists who deployed in support of GWOT through June 30, 2006, were part of the Selected Reserve (see table 5). In addition, California, Texas, Pennsylvania, and Florida had the highest numbers of reservists who have deployed in support of GWOT through June 30, 2006 (see table 6).

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, Cynthia Jackson, Assistant Director; Crystal Bernard; Tina Kirschbaum; Marie A. Mak; Ricardo Marquez; Julie Matta; Lynn Milan; Rebecca Shea; and Cheryl Weissman made key contributions to this report.