Military Personnel: Actions Needed to Improve Evaluation and Oversight of Reserve Officers' Training Corps Programs
Released: 2013-11-13T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: ROTC is the largest source of newly commissioned officers for DOD. GAO's analysis of DOD data identified more than 9,000 officers commissioned from ROTC in fiscal year 2012. ROTC is critical for producing officers from the nation's colleges and universities to meet the leadership and readiness needs of the armed forces. The National Defense Authorization Act for Fiscal Year 2013 mandated GAO to review the services' ROTC programs. This report assesses the extent that ROTC programs (1) met goals and minimum annual requirements for producing officers over a 5-year period, (2) have established performance measures and conducted evaluations for managing ROTC programs, and (3) are subject to oversight and have effective processes for communicating with key stakeholders. GAO analyzed ROTC production data from fiscal years 2008 through 2012; reviewed relevant legislation and DOD and service policies and guidance for ROTC; and reviewed and discussed assessments of ROTC efficiency and effectiveness, and oversight of ROTC with officials from OSD and the services.

section_title: What GAO Found
paragraphs: From fiscal years 2008 through 2012, each military service met at least 91 percent of their overall Reserve Officers' Training Corps (ROTC) goals for producing the number of officers needed to meet service end strength authorizations, but each has reported challenges in commissioning officers for some certain occupational specialties, such as engineers and nurses. Further, GAO's analysis found that half of the Department of Defense's (DOD) ROTC units did not meet DOD's minimum average annual production requirement over the 5-year period reviewed. Further, cost per commissioned officer varies greatly depending on unit production. For example, excluding tuition costs, the average cost per officer produced across all units was about $68,000, compared to an average cost of about $95,000 per officer for units that produced fewer than 15 officers on average annually from fiscal years 2008 through 2012.

DOD's instruction specifies factors to consider before closing ROTC units, but these factors do not constitute clearly defined performance measures that provide a comprehensive understanding of the effectiveness and efficiency of ROTC programs, and service evaluations of ROTC programs are ad hoc. DOD's instruction specifies that in assessing units for closure, the services are to consider the quality of officers produced by a unit but it does not clearly define characteristics that comprise quality, which has led to the inconsistent application of the measure by the services. The services conducted 11 evaluations over the past decade to assess performance and identify units for consolidation or closure. However, the evaluations have largely occurred on an ad hoc basis because the services have not established a systematic process to routinely evaluate ROTC program performance. Key attributes of successful performance measures include clearly, defined measures. Moreover, results-oriented program management practices include routine program evaluations that determine how well a program is working. Without clearly defined performance measures and routinely conducted evaluations, it will be difficult for the services to accurately determine if ROTC programs are effectively and efficiently operating.

The Office of the Secretary of Defense (OSD) conducts some oversight functions, such as setting policy for the ROTC program. However, although specified in guidance, OSD does not review the services' methodologies for closing ROTC units because OSD officials believe this is a service responsibility. Further, the services do not consistently communicate with key stakeholders, such as members of Congress and schools, about performance of ROTC programs, except when closure decisions are being considered. This may have contributed to the difficulty the services have experienced in gaining political support for such closures. GAO has noted that regular communication with stakeholders helps build trusting relationships to gain buy-in. Without clearly delineated responsibilities for oversight of ROTC programs and a formal strategy to communicate with key stakeholders on ROTC program performance, DOD will find it difficult to obtain the support that is needed to improve the efficiency and effectiveness of ROTC programs.

section_title: What GAO Recommends
paragraphs: GAO is making six recommendations to DOD, to include establishing clearly defined performance measures and conducting routine program evaluations; reexamining oversight roles and responsibilities; and developing a strategy for communicating with Congress and other key stakeholders on program performance. DOD concurred with each of GAO's recommendations.

section_title: Letter
paragraphs: The Reserve Officers’ Training Corps (ROTC) is the largest source of  commissioned officers for the Department of Defense (DOD), and hence  plays a critical role in developing future leaders for and ensuring the  readiness of the armed forces. In fiscal year 2012, the military services’  ROTC programs collectively produced over 9,000 newly commissioned  officersproduced by DOD’s three primary sources of commissions that year. In  addition to its contribution to military readiness, the 479 ROTC units— which have a presence at nearly 1,800 colleges and universities  throughout the United States and its territories—help the military to  maintain a critical link with the civilian population. In particular, ROTC’s  geographically diverse presence is credited with helping to preserve the  citizen-soldier tradition of producing officers that reflect the communities  they serve, and provides the possibility of a military career to students  who otherwise would not have considered it to be an option. The size and  significance of ROTC programs make it a complex mission that requires a  combination of deliberate planning and the strategic employment of  limited resources, especially in light of the current and long-term fiscal  pressures facing DOD. —accounting for nearly half of all new active duty officers  We have published several reports regarding the military services’ ROTC  programs, each of which identified challenges. Specifically, in 1991, we  highlighted issues regarding the services’ retention of underproductive  ROTC units, among other issues. Further, in 1992, we issued a report  on officer commissioning programs including ROTC, in which we found  that the services were not systematically assessing these programs or the  quality of officers they produced and that the lack of coordinated  management and oversight hampered efforts to make officer production  cost-effective. Most recently, in 2007, we recommended that the Army  develop and implement a dedicated strategic plan to address current and  projected officer accession and retention shortfalls. The Office of the  Secretary of Defense (OSD) and the military services concurred or  partially concurred with our prior recommendations related to closures of  underproductive units and the need to develop a comprehensive strategy  that helps to ensure ROTC resources are used in the most efficient  manner possible. However, these recommendations have largely not  been addressed.

Section 554 of The National Defense Authorization Act for Fiscal Year  2013 mandated that we assess the productivity, structure, and oversight  of the military services’ ROTC programs, among other things. To respond to this mandate, in this report we examine the extent to which the  military services’ ROTC programs (1) met goals and minimum annual  requirements for producing officers over a 5-year period, (2) have  established performance measures and conducted evaluations to help  manage ROTC programs, and (3) are subject to oversight and have  effective processes for communicating with key stakeholders.

To assess the extent to which the military services’ ROTC programs met  goals and minimum requirements for producing officers over a 5-year  period, we obtained the programs’ goals for producing commissioned  officers program-wide for fiscal years 2008 through 2012 for each military  service, and we compared those goals to data on actual officer production  during the same period. We also compared the average annual production of individual units to DOD’s minimum production requirement  for unit viability during the same time period. We obtained data on  program and unit level costs and analyzed these data by comparing the  cost of an ROTC unit to its actual officer production, and comparing the  general cost of establishing and maintaining different types of ROTC  units. We assessed the reliability of these data by conducting electronic  testing of these data and discussing any data limitations with appropriate  service officials. We determined that these data were sufficiently reliable  for determining the extent to which services’ ROTC programs are meeting  production goals and requirements, and the general cost to establish and  maintain an ROTC unit.

To assess the extent to which the services have established performance  measures and conducted evaluations to help manage ROTC programs,  we obtained and analyzed information from OSD and the services to  determine the extent to which ROTC program performance measures  demonstrate attributes of effective performance measures. We also obtained and analyzed information from the services to identify how  frequently each service conducted evaluations and how each had used  existing performance measures to identify units for potential consolidation  or closure. We also reviewed relevant legislation and OSD and military  service guidance on ROTC programs to identify guidelines and  requirements pertaining to the efficient and effective management of  ROTC programs and we compared it to the administration of ROTC  programs to determine the extent to which such legislation and guidance  has been implemented. To assess the extent to which ROTC programs  are subject to oversight and have effective processes for communicating  with key stakeholders, we reviewed OSD and military service guidance on  ROTC programs to identify guidelines and requirements for conducting  oversight of ROTC programs and communicating with key stakeholders.  To supplement the analyses, we interviewed knowledgeable officials from  OSD and the military services on the administration of and responsibility  for ROTC programs and their approaches to communicating with  stakeholders about program performance.

We conducted this performance audit from February 2013 to November  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. A more in depth  discussion of our scope and methodology appears in appendix I of this  report.
subsections: section_title: Background
paragraphs: ROTC is the largest of the three major officer commissioning programs  for DOD. The other two major commissioning officer sources include the  military service academies and Officer Candidate Schools/Officer Training  School. As shown in figure 1, in fiscal year 2012, the ROTC program  produced nearly half of all the newly commissioned active duty officers  from the three major commissioning sources.

ROTC programs prepare college and university students to serve as  officers in the military services. The program was established by the  National Defense Act of 1916 to supplement the military academies in  preparing selected students for commissioned military service and to  provide officers for the reserve forces. The services organize their ROTC  programs into administrative units and assign a command structure,  including a commanding officer, to each unit. In addition to a commanding  officer, each ROTC unit has additional staff assigned to the unit, including  enlisted and officer personnel, and for Army and Navy, civilian personnel.  Table 1 below provides specific information about military and civilian  personnel assigned to ROTC units for each service.

The ROTC units are spread across the country, as represented in figure  2. Most states have a unit from each service located within the state and  all states have at least one unit located within the state. Each unit may  support students at more than one school. Typically, in these cases, the  unit’s command staff is assigned to the school that has been designated  as the host, and students travel from their affiliate school to the host unit  for their ROTC activities.

The Navy maintains 12 ROTC units in which the command structure and ROTC  activities are divided between more than one host school. Naval ROTC refers to these as  consortium units.

This typically happens no later than the junior year, prior to the service’s advanced  ROTC training. regarding ROTC. The services’ headquarters determine the number of  newly commissioned officers needed annually to support the services’  congressionally mandated end strength requirements for military  personnel by considering beginning strength, losses, and transfers.  They also provide guidance on the number of officers expected from each  officer commissioning program, including ROTC.  To meet the  requirements set by service headquarters, the military services typically  operate ROTC programs using a 5-year production cycle, beginning the  year prior to school entry and culminating in the graduation year. ROTC’s  relevance extends beyond its quantitative production capability. ROTC  supports the concept of a citizen-soldier and the all-volunteer force with  its potential to provide officers from a variety of backgrounds and  experiences available within the student population on a college or  university campus. ROTC also provides a method of military outreach and  contact with the public. By statute, at least one ROTC unit must be  established and maintained in each state if certain conditions are met.

section_title: ROTC Programs Met Some, but Not All, Production Goals to Achieve Officer End Strength, and Retain Some Units That Do Not Meet Minimum Production Requirements
paragraphs: The military services’ ROTC programs each came within a few  percentage points of meeting or exceeding the annual officer production  goals that they established to meet their respective authorized end  strengths. However, each service reported that various factors may affect  their ability to meet these goals, and that they have faced challenges  commissioning officers for certain occupational specialties. In addition,  half of all ROTC units did not produce the minimum average annual  production of officers over a 5-year period required by DOD to justify  investment in an ROTC unit.
subsections: section_title: The Services Met Some, but Not All, of Their Overall ROTC Officer Production Goals for Newly Commissioned Officers
paragraphs: In fiscal years 2008 through 2012, each service came within a few  percentage points of meeting or exceeding its overall ROTC officer  production goals for newly commissioned officers.  The military services  annually issue guidance that specifies goals for the number and types of  officers that their respective ROTC programs need to produce to meet  their current authorized end strength as well as future needs that are  projected over a 5-year period. For the 5 years we reviewed, fiscal years  2008 through 2012, the Army’s ROTC program goal for newly  commissioned officers ranged from 4,500 to 5,350; the Navy’s goal,  which includes officers for the Marine Corps, ranged from 1,163 to 1,387;  and the Air Force’s goal ranged from 1,788 to 1,940. The services  generally met these overall production goals for newly commissioned  officers. Specifically, as shown in table 2 below, the Army met at least 95  percent of its goal, the Navy met at least 91 percent of its goal, and the  Air Force met at least 96 percent of its goal during this period.  Furthermore, the Army and the Air Force each exceeded their production  goals for 3 of the 5 years we studied. Table 2 shows the production goal,  actual production, and the percent of the goal met by each service for  fiscal years 2008 through 2012.

Service officials told us that they manage ROTC enrollments, contract  offers, and scholarship awards to help ensure that they reach but do not  significantly exceed their officer production goals. In the event that an  over- or underproduction of officers is anticipated, each service  correspondingly adjusts the number of people they plan to admit into  future classes at their respective Officer Candidate School or, in the case  of the Air Force, Officer Training School.

section_title: DOD Does Not Have Clearly Defined and Comprehensive Performance Measures to Evaluate ROTC Programs, and Recent Evaluations Have Been Ad Hoc
paragraphs: Neither OSD nor the services have established performance measures  that provide a comprehensive understanding of the overall effectiveness  and efficiency of ROTC programs. Specifically, current measures do not  take into account a service’s investment in a particular ROTC unit, and  are not clearly defined to help ensure that they are objectively applied. In  addition, over the past decade, the services have conducted a total of 11  evaluations to identify units for consolidation or closure. However, the  evaluations have largely occurred on an ad hoc basis and are narrowly  focused on assessing the productivity of individual ROTC units, rather  than the overall performance of the program. Furthermore, although the  services’ evaluations of ROTC units yielded recommendations for  consolidation or closures, no closures have occurred as a result.
subsections: section_title: DOD Has Not Established Clearly Defined, Comprehensive Performance Measures
paragraphs: DOD’s management of ROTC programs is hampered by not having  performance measures that provide a comprehensive understanding of  the overall cost-effectiveness and efficiency of ROTC programs. Key  attributes of successful performance measures include clearly defined,  comprehensive measures that enable an organization to evaluate  accomplishments, make decisions, and balance competing priorities.

DOD’s instruction specifies five factors for the military services to  consider before closing a unit, including, the (1) quality of the officers  produced by the unit, (2) operations and maintenance cost of maintaining  the unit, (3) numbers of officers produced to meet service commissioning  goals from categories that are difficult to achieve, (4) number of officers  produced by the unit, and (5) number and location of units in an area or  state where the unit being considered for closure resides. However,  several of these factors, which officials told us are the measures they  currently use to assess the programs, are not clearly defined. Specifically,  DOD’s instruction directs the services to consider the quality of officers  produced by a unit that is being evaluated for closure. However, the  instruction does not include guidance on how the services should define  and assess quality, which could lead to the inconsistent application of the  measure by the services. Another example is that DOD’s instruction  directs the services to consider the geographic representation of ROTC  units. However, the instruction does not describe how concentrated the  ROTC presence in an area or state should be, which could lead the  services to under- or over-emphasize the relative importance of an ROTC  unit to a particular area.

Furthermore, DOD’s instruction requires officer production from each unit  to be adequate to justify DOD investment and generally requires ROTC  units to produce a 5-year average of 15 officers annually to remain  viable. However, DOD’s production requirement does not take into  account a service’s investment in a particular ROTC unit because it does  not account for cost-related factors, such as the differences in the cost of  tuition for public and private colleges or for in-state and out-of-state  students. Moreover, the tuition cost difference is not insignificant, with  college tuition for some schools costing six times more than the annual  average in-state tuition of about $7,100 for a 4-year college or  university. Despite the significant variance in tuition, all ROTC units are  subject to the same numerical production requirement regardless of the  actual cost each service incurs at a particular college or university.  Because such cost data are not consistently accounted for, it is not clear  how the production requirement can be used to justify DOD’s investment.

The absence of clearly defined measures also affects the objectivity of  DOD’s unit closure consideration factors because it requires the services  to subjectively determine how each factor should be defined and applied.  Further, these determinations can change over time without explanation.  For example, we reviewed the methodologies used by each service to  identify units for potential closure and found that, in addition to the five  factors to consider before closing a unit, two of the services incorporated  additional factors into the closure consideration process. For example, in  fiscal year 2012, the Navy included the U.S. News and World Report  ranking of the schools associated with the units as 1 of 12 factors in its  closure considerations, and the Air Force included “strategic partnerships”  as an important consideration. However, neither service identified  specifically how these factors correlate to the factors in DOD’s instruction,  defined how these factors contribute to a service-specific need, or  explained why these factors were added to the 2012 analysis. In  comments on a draft of this report, Navy officials pointed out that the U.S.  News and World report ranking correlates to the quality of applicants to  the Navy Nuclear Propulsion Program, a specific Navy officer accession  requirement. However it is not clear specifically how the Navy made that  correlation, nor why the Navy added this ranking for its 2012 review. We  recognize that the services need some flexibility to prioritize factors on the  basis of their individual needs for newly commissioned officers, but  inconsistent measurements limit the services’ ability to compare program  progress, results, and efficiency over time.

We reviewed service policies for ROTC programs and likewise found no  additional guidance on how to define or apply the closure consideration  factors in DOD’s instruction.  Although the Army regulation specifies a  process for categorizing potential closures, Army officials told us that the  disestablishment categories specified in the regulation have not been  used for years because, according to officials, the process was too time  consuming. The variability of the factors used in considering ROTC  units for closure hinders the services’ efforts to compare evaluation  results over time, even within a single service. Until clearly defined and  comprehensive performance measures are established to evaluate  overall program performance, the services will have limited information  available about their ROTC programs’ overall progress toward their goals.

section_title: Ad Hoc Evaluations to Determine Potential Unit Closures Do Not Provide Needed Performance Information
paragraphs: In the absence of fully developed performance measures, the military  services have evaluated aspects of their respective ROTC programs.  However, the evaluations are not routine and systematic and have not  yielded closures that service officials believe are needed. Our prior work  has shown that results-oriented program management practices include  routine program evaluations, which are a key source of information about  how well a program is working and whether it is achieving its intended  results. Further, these practices indicate that a body of evidence is more  valuable to decision makers than a single study; multiple studies with  similar results strengthen confidence in their conclusions.

The military services’ ROTC programs have, over the past decade,  conducted a total of 11 evaluations—largely driven by the need to  produce cost-savings to help offset DOD’s near-term budget shortfalls— to identify ROTC units that could potentially be consolidated or closed.  These evaluations have provided the services with some useful  information on the performance of ROTC programs, but they have largely  occurred on an ad hoc basis because the services have not established a  systematic process to routinely evaluate ROTC program performance  using comprehensive performance measures. For example, service  officials told us that the Army conducted 3 evaluations over 6 years; the  Navy conducted 4 evaluations over 3 years; and the Air Force conducted  4 evaluations over 8 years on production of officers by individual ROTC  units. These evaluations provided information about individual ROTC unit  performance, with a focus on officer production. However, the evaluations  did not assess the overall ROTC program against strategic goals and  objectives. Our prior work has shown that leading organizations that have  progressed toward results-oriented management use performance  information to identify gaps in performance, to improve organizational  processes, and to improve their performance by aligning measures with  the goals and objectives at each level of the organization.  Further, the commitment of managers to results-oriented management is critical to the  increased use of performance information that could be measured against  ROTC program goals and objectives to influence policy and program  decisions. In addition, this commitment could be demonstrated by their  willingness and ability to make decisions and manage programs on the  basis of results, and to inspire others to embrace such a model. Without  routine evaluations that measure progress toward strategic goals and  objectives, decision makers cannot determine the cost-effectiveness or  efficiency of ROTC programs over time and cannot reliably identify trends  in program performance.

Furthermore, without DOD guidance for comprehensive evaluations of  ROTC programs, each service’s program managers developed a  service’s own methodology to evaluate the productivity of ROTC units.  Each service’s methodology varies based on their chosen combination of  factors from DOD’s instruction, and how they are prioritized. As we  previously noted, DOD Instruction 1215.08 requires the military services  to consider the operation and maintenance cost of maintaining ROTC  units that are being evaluated for productivity and possibly considered for  closure. However, we reviewed the methodologies used by each service  to evaluate unit productivity and found that the operations and  maintenance costs of ROTC units were not routinely considered in their  evaluations. Specifically, in recent evaluations of unit productivity, the  Army and the Air Force did not consider any ROTC costs, and the Navy  considered only the cost of tuition. As a result, the services may be using  results of incomplete evaluations to draw conclusions about unit  performance. Table 5 further details the methodologies used by each  service in recent assessments of unit productivity.

Moreover, although the services’ evaluations of ROTC units yielded  recommendations for consolidation or closures, no closures have  occurred as a result. DOD Instruction 1215.08 specifies that the  Secretaries of the military departments shall, among other things,  establish, operate, and disestablish ROTC units, and that the decision to  disestablish a ROTC unit is the prerogative of the Secretary of the military  department concerned. Although DOD guidance specifies service  Secretary responsibilities for and the authority to close ROTC units,  service officials told us that recommended closures and consolidations  have not been acted on because some senior service officials have  concluded that it is easier to maintain underproductive ROTC units than  attempt to secure the congressional support needed for their closure.  Importantly, DOD’s instruction gives service Secretaries discretion on  which units to close. However, until the services conduct routine  evaluations of ROTC programs that are based on a comprehensive set of  performance measures, the programs risk having more units than needed  or units that are not most effectively located to meet service goals for their  officers.

section_title: Oversight of ROTC Program and Communication with Key Stakeholders Is Limited
paragraphs: OSD and the military services are generally assigned responsibility for  conducting oversight of ROTC programs. Although the Under Secretary  of Defense for Personnel and Readiness conducts some oversight  functions such as setting policy, DOD’s Instruction 1215.08 does not  specify department-level review of performance measures. Further,  services do not consistently communicate with key stakeholders, such as  members of Congress and schools, about performance of ROTC  programs, except when closure decisions are being considered. This has  contributed to difficulty gaining political support for such closures.
subsections: section_title: DOD and Service Guidance Provides Limited Information on Oversight Responsibilities
paragraphs: ROTC programs are governed by both DOD and service-level guidance.  Key attributes of results-oriented program management include the  cascading of goals and objectives throughout an organization and  aligning performance measures with the objectives from the executive  level down to the operational level. In addition, internal control  standards require that management sets and monitors these objectives  and that these responsibilities should be appropriately documented.

At the department level, the purpose of DOD Instruction 1215.08 is to  establish policy, assign responsibilities, and prescribe procedures for  DOD oversight of ROTC. The instruction assigns responsibility for these  actions to the Under Secretary of Defense for Personnel and Readiness  and the Secretaries of the military services. Specifically, DOD Instruction  1215.08 tasks the Under Secretary of Defense for Personnel and  Readiness with establishing policy and providing implementing guidance,  resolving matters of conflict among the services that are operating ROTC  units, approving methodologies for closure in coordination with the Under  Secretary of Defense (Comptroller), and receiving annual reports on costs  and budget for service ROTC programs.does not specify responsibilities for the Under Secretary to conduct  oversight related to monitoring performance of ROTC programs against  goals and objectives using performance measures.

However, DOD’s instruction  Further, we found that the Under Secretary of Defense for Personnel and  Readiness is not fully implementing oversight responsibilities that exist in  guidance. While the Under Secretary of Defense for Personnel and  Readiness has established DOD’s policy related to ROTC in DOD  Instruction 1215.08, and officials from that office told us that they resolve  conflicts among the services related to ROTC, the officials acknowledged  that they do not conduct the other oversight activities specified in  guidance.  First, OSD officials told us that they do not review and approve  the services’ methodologies for proposed unit closures because they see  it as encroaching on the services’ authority to man, train, and equip their  respective forces. Given that DOD Instruction 1215.08 requires this  approval, which is to occur when a military department proposes closure  of a particular unit it is difficult to understand how this methodology  approval function, in connection with a proposed decision of a military  service, could be viewed as an encroachment.  Moreover, DOD officials  did not further elaborate on or provide additional support for their position  that approving a methodology, as set forth in the DOD instruction, is not  an appropriate oversight function to be performed at the department level.   In light of the ad hoc reviews by the services noted earlier in this report,  such oversight by the Under Secretary of Defense for Personnel and  Readiness that is clearly delineated in guidance could support efforts to  improve the consistency of the services’ application of performance  measures and evaluations.

Second, OSD officials told us that they stopped enforcing a requirement  for annual reports from the services identifying the cost and budget for  their respective ROTC programs—even though the officials viewed those  reports as useful in facilitating oversight—in response to the department’s  efficiency initiatives, which removed reporting requirements related to  internally generated oversight reports. To regain some of the information  on program visibility that it no longer obtains through the reports, OSD  officials told us that they instead use data from service budget  submissions to estimate ROTC program cost data that is less specific  than the data that were previously obtained through the services’  automatic submission of these reports. Moreover, because they receive  the data after it is submitted for the budget, there is little opportunity for  the office to make changes or affect future management decisions.  In  addition, as noted previously, the DOD and service-level guidance  provide performance measures that do not provide a comprehensive  understanding of the overall cost-effectiveness and efficiency of ROTC  programs. With (1) the absence of OSD following some requirements in  existing guidance, and (2) weaknesses we identified in performance  measures in both OSD and service guidance, it is unclear how OSD and  the services will each conduct effective oversight of ROTC programs.

section_title: OSD and the Military Services Do Not Regularly Communicate with Key Stakeholders on ROTC Program Performance
paragraphs: The military services have had difficulty gaining buy-in for  recommendations to close dozens of underproductive ROTC units  collectively across the services, in part because they do not have an  effective strategy for communicating with stakeholders such as Congress  or school administrators about ROTC program performance. Stakeholder  involvement is an important strategy that we have identified in our past  work to facilitate agencies’ use of evaluations in program management  and policy making. Stakeholder involvement is best facilitated when  organizations reach out early to gain buy-in, engage in regular  communication, and build trusting relationships—even when faced with  the possibility that political or ideological concerns may override  evaluation findings in decision making. In addition, our past work has  shown that key elements of a results-oriented oversight framework  include communicating results and using performance information to  make decisions for improvement.

As previously noted, the military services collectively have conducted 11  assessments of unit productivity in the past 10 years—all of which have  produced proposals recommending selected ROTC units for  consolidation or closure. However, no closures have occurred as a result.  The military services have been unsuccessful in completing any of these   politically sensitive consolidations or closures in part because  and prior the services do not proactively and periodically engage congressional  stakeholders on ROTC program performance. Specifically, service  officials told us that DOD Instruction 1215.08 does not require  congressional notification about ROTC program performance and that the  decision to close units is the prerogative of the Secretary of the military  department concerned. Nonetheless, as a courtesy, the services have, in  the past, provided Congress with a list of units that are being  recommended for consolidation or closure. Furthermore, OSD officials  stated that the lack of closures in successive rounds of assessments of  unit viability is producing a growing backlog of underproductive units. As a  result, service officials acknowledged that this increasing number of  underproductive ROTC units may make congressional acceptance of all  the closures at one time difficult to achieve.

We have previously reported on the department’s lack of reporting and communication  of ROTC program performance. For example, in our 1973 and 1977 reports we found that  the services’ reporting to DOD and Congress were inadequate to make necessary  judgments on ROTC program effectiveness. Additionally, in follow-on reports on ROTC  and officer commissioning programs in 1991 and 1993, we found that DOD and the  services had not made any significant progress in improving their ability to report ROTC  program performance. established at a college or university, school administrators have to agree  to various requirements, to include granting the academic rank of  professor to the senior commissioned officer assigned to the program at  that institution and adopting the military coursework for ROTC into its  curriculum. Further,  DOD Instruction 1215.08 requires the military  services—prior to recommending a unit for consolidation or closure—to  advise schools whose ROTC units are not producing enough officers to  justify investment, and to work closely with those schools to restore officer  production to acceptable levels. However, the military services do not  consistently adhere to this requirement, and thus may be missing  important opportunities to build awareness about the ROTC program and  to leverage the support of schools in meeting annual production goals.  For example, Army and Navy officials told us that they currently do not  regularly communicate with school administrators about the level of  officer production at their respective college or university.  In contrast, Air  Force officials told us that they provide written notification annually to  participating colleges and universities on their level of officer production.  Until the military services establish a formal strategy to communicate with  key stakeholders on ROTC program performance, the military services  will find it difficult to obtain the support that is needed to make the  necessary changes to improve the efficiency and effectiveness of ROTC  programs.

section_title: Conclusions
paragraphs: While ROTC programs continue to be the largest and most important  source of newly commissioned officers for DOD, they do so with a  structure where half of the units do not meet DOD’s production  requirements. Further, service efforts to revise this inefficient structure are  hampered by performance measures that are not clearly defined, allow  for inconsistent application of factors for closure, and do not provide a  comprehensive understanding of programs’ efficiency or cost- effectiveness. Moreover, the department’s management approach has  not required routine program evaluations and this has led to ad hoc  analyses that present challenges in supporting unit closure  recommendations. Clearly defined and comprehensive performance  measures and routine evaluations using these measures against strategic  goals and objectives would give OSD and the services tools to identify,  collect, and evaluate a range of relevant performance information for their  ROTC programs. This information, particularly when available over a  period of time, would allow OSD and the services to recognize the  programs’ accomplishments, balance competing priorities, and address  the retention or disestablishment of underproductive units.

Effective oversight and regular communication with stakeholders are  critical to the program’s effectiveness and continued success. Clear lines  of responsibility when combined with performance measures provide  leadership and decision makers with the transparency and authority  required to make difficult decisions about the program. In addition,  communication with key stakeholders, such as Congress and school  administrators, about ROTC performance is important to raising  awareness about the program and ensuring that program-related  decisions can be implemented effectively.

section_title: Recommendations for Executive Action
paragraphs: To help ensure that OSD, the military services, and congressional  decision makers have a comprehensive understanding of whether ROTC  programs are achieving desired results in a cost-effective and efficient  manner, we recommend that the Secretary of Defense direct the Under  Secretary of Defense for Personnel and Readiness, in conjunction with  the Secretaries of the military services, to establish a systematic process  to routinely evaluate ROTC program performance that includes taking the  following three actions: establish performance measures that are clearly defined and  require routine evaluations of ROTC programs that measure  progress against the strategic goals and objectives of ROTC  programs; and use the performance information resulting from ROTC program  evaluations to assess and document the need for the existing  number of units,  To help improve the oversight and accountability of the military services’  ROTC programs, we recommend that the Secretary of Defense direct the  Under Secretary of Defense for Personnel and Readiness to take the  following three actions:   reexamine and clarify DOD Instruction 1215.08 to clearly delineate  roles and responsibilities for oversight of ROTC programs; coordinate with the military services to ensure that service ROTC  guidance aligns with the updated DOD instruction; and develop and implement, in conjunction with the Secretaries of the  military services, a strategy to periodically communicate with  Congress and other key stakeholders on ROTC program  performance.

section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD concurred with our six  recommendations to make management, oversight, and accountability  improvements to ROTC programs. DOD also described steps that it  planned to take in response to our recommendations. DOD’s comments  are reprinted in appendix IV. DOD also provided technical comments on  the draft report, which we incorporated as appropriate.

Regarding our first recommendation to establish performance measures  that are clearly defined and include cost components, DOD concurred  with our recommendation but stated that our report contained a “minor  discrepancy” because it did not include the Navy’s explanation of how a  particular unit closure consideration factor—outside of those specified in  DOD Directive 1215.08—contributed to a service-specific need.  Specifically, our draft report noted that, in fiscal year 2012, the Navy  included the U.S. News and World Report ranking of the schools affiliated  with a Naval ROTC unit as one of twelve factors included in its unit  closure consideration process, but that it did not define how its use of this  factor correlated to its need for a specific type of officer. We updated the  draft to include the Navy’s comment that U.S. News and World Report  ranking correlates to the quality of applicants to the Navy Nuclear  Propulsion Program, a specific Navy officer accession requirement. We  also clarified that because the services are required to subjectively  determine how factors should be applied, these factors can change over  time without justification. We therefore continue to believe that these  inconsistent measurements limit the services’ ability to compare program  progress, results and efficiencies over time.

For our other five recommendations, DOD concurred with each of them  and estimated actions would be completed by July 2015. We encourage  DOD to begin taking actions now to facilitate implementation of these  recommendations as soon as possible.

We are sending copies of this report to the Secretary of Defense, the  Under Secretary of Defense for Personnel and Readiness, the  Secretaries of the Army, the Navy, and the Air Force, and appropriate  congressional committees. In addition, this report will also be available at  no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions regarding this report, please  contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.

section_title: Appendix I: Scope and Methodology
paragraphs: To assess the extent to which the military services’ ROTC programs met  goals and minimum requirements for producing officers over a 5-year  period, we obtained and analyzed ROTC unit production data from each  of the military services for fiscal years 2008 through 2012 and fiscal year  2012 enrollment data. We chose to assess the officer production from  each service’s ROTC program for fiscal years 2008 through 2012  because it is the most recently completed 5-year period at the time of our  review. We determined the overall productivity of the services’ ROTC  programs by comparing the number of ROTC officers produced to the  ROTC officer goals as outlined in service-specific guidance for fiscal  years 2008 through 2012. Using fiscal year 2012 data found in fiscal year  2014 budget documents and other service documents, we compared  overall officer production with other major commissioning sources. We  determined the productivity of ROTC units by comparing the average  annual production of individual units from fiscal year 2008 through 2012  to the Department of Defense’s (DOD) minimum production requirement  in the DOD Instruction for Senior ROTC (DOD Instruction 1215.08)—a  minimum of 15 officers a year averaged over 5 years for 4-year schools  and a minimum of 7 officers a year averaged over 5 years for 2-year  schools. We also analyzed the data provided to determine the degree to  which the units are not meeting minimum production requirements and  the degree to which the units met minimum production requirements. In  particular, we analyzed ROTC officer production data from fiscal years  2008 through 2012 to determine (1) the number and percentage of ROTC  units that met and did not meet the DOD minimum production  requirement for ROTC officer production and (2) the number and  percentage of ROTC units that produced officers in specific ranges of  officer production (e.g., fewer than 5, 5 to fewer than 10, 10 to fewer than  15, 15 to fewer than 20, 20 to fewer than 30, etc. to over 50). We also  reviewed fiscal year 2012 staff data of military personnel located at each  unit provided to us by the military services and DOD’s fiscal year 2012  composite rates to determine the cost of military personnel per each  ROTC unit in fiscal year 2012. In addition, we reviewed the fiscal year  2014 budget documentation to identify the average operation and  maintenance cost (excluding scholarships) per ROTC unit in fiscal year  2012. We used these data to determine the fiscal year 2012 cost of  ROTC units that met and did not meet DOD minimum production  requirements for fiscal years 2008 through 2012 .To assess the reliability  of the production and cost data, we discussed these data with officials  from the military services to gain an understanding of the processes and  databases used to collect and record data and to understand existing  data quality control procedures and known limitations of the data. When  we found inconsistencies in the data, we followed up with service officials  to attempt to reconcile these differences. We determined that these data  were sufficiently reliable for determining the extent to which the military  services are meeting ROTC production goals and unit level requirements  as well as the cost per unit.

To assess the extent to which the services have established performance  measures and evaluations for determining the structure of ROTC  programs, we reviewed relevant legislation, specifically sections 2101- 2111b of Title 10 of the U.S. Code, and department and service guidance  for ROTC programs, to identify what, if any, guidelines for cost-effective  program management are identified. In addition, we reviewed prior GAO  reports on ROTC and officer accessions programs to determine what  recommendations and actions have previously been taken in regard to  improving the efficiency and effectiveness of ROTC programs. We also  obtained and analyzed information from each of the services to determine  the frequency with which the services evaluated unit productivity to  identify units for potential consolidation or closure. Further, we identified  the measures used by each service to conduct these evaluations and  compared them with key attributes of effective performance measures  that our prior work has shown are needed to successfully evaluate  program performance. We also interviewed knowledgeable officials from  the Office of the Secretary of Defense (OSD) and the military services  who are responsible for officer accessions and service-specific ROTC  programs. We discussed with these officials the extent to which the  services were meeting ROTC requirements for producing newly  commissioned officers, ROTC guidance and policies, roles and  responsibilities, program performance and costs, procedures for the  establishment and disestablishment of ROTC units, assessments of  ROTC unit productivity, and program oversight.

To assess the extent to which the military services’ ROTC programs are  subject to oversight and have effective processes for communicating with  key stakeholders, we reviewed and analyzed relevant OSD and military  service guidance to identify guidelines and responsibilities for conducting  oversight of ROTC programs and communicating with key stakeholders  about ROTC program performance. We obtained and analyzed  documentation on military service correspondence with schools about  ROTC program performance and compared them with provisions in DOD  Instruction 1215.08 pertaining to communication with schools. We also  interviewed knowledgeable officials from OSD and the military services  on the activities and mechanisms used to conduct oversight of, and to  communicate with stakeholders on, ROTC programs.

We visited or contacted the following organizations during our review:

Deputy Assistant Secretary of Defense for Military Personnel Policy,  Directorate of Accession Policy, Arlington, Virginia

Deputy Chief of Staff, Military Personnel Management Directorate,  Officer Division, Arlington, Virginia

Army Cadet Command, Fort Knox, Kentucky

Deputy Chief of Naval Operations, Manpower Personnel Education  and Training, Arlington, Virginia

Naval Service Training Command, Great Lakes, Illinois  Department of the Air Force

Deputy Chief of Staff for Manpower Personnel and Services, Air Staff,

Jeanne M. Holm Center for Officer Accessions and Citizen  Development, Montgomery, Alabama

Marine Corps Recruiting Command, Quantico, Virginia  We conducted this performance audit from February 2013 to November  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Fiscal Year 2012 Officer Production by ROTC Unit and by Service
paragraphs: In fiscal year 2012, the Reserve Officers’ Training Corps (ROTC) program  produced over 9,000 active duty, guard, and reserve officers. Table 6  provides a listing of the number of officers produced in fiscal year 2012 by  unit, school location, and service.

section_title: Appendix III: Enrollment by Year of Study Appendix III: Enrollment by Year of Study and by Military Service and by Military Service
paragraphs: This appendix contains enrollment information on the number of students  in each of the military services’ Reserve Officers’ Training Corps (ROTC)  programs across all years of study by unit and by the schools within the  unit. Table 7 presents the Army’s enrollment information as of  November 2012, table 8 presents the Navy’s enrollment information as of  November 2012, and table 9 presents the Air Force’s enrollment  information as of December 2012. The enrollment information is  alphabetized by state or U.S. territory and then by unit and then by school  within the unit.enrollment data, we identified that the services’ ROTC programs had  about 53,000 students enrolled across all years of study in 2012—the  Army had over 33,000 students enrolled, the Navy had over 6,000  enrolled, and the Air Force had over 14,000 students enrolled.

section_title: Appendix IV: Comments from the Department of Defense

section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Margaret Best (Assistant  Director), Emily Biskup, Rebekah J. Boone, Richard Burkard, Timothy  Carr, Susan Ditto, Cynthia Grant, Kimberly Mayo, Michael S. Pose,  Stephanie Santoso, and John Wren made major contributions to this  report.

section_title: Related GAO Products
paragraphs: Military Personnel: DMDC Data on Officers’ Commissioning Programs is  Insufficiently Reliable and Needs to be Corrected. GAO-07-372R.  Washington, D.C.: March 8, 2007.

Military Personnel: Strategic Plan Needed to Address Army’s Emerging  Officer Accession and Retention Challenges. GAO-07-224. Washington,  D.C.: January 19, 2007.

Reserve Officers’ Training Corps: Questions Related to Organizational  Restructuring. GAO/NSIAD-96-56. Washington, D.C.: February 6, 1996.

Officer Commissioning Programs: More Oversight and Coordination  Needed. GAO/NSIAD-93-37. Washington, D.C.: November 6, 1992.

Reserve Officers’ Training Corps: Less Need for Officers Provides  Opportunity for Significant Savings. GAO/NSIAD-91-102. Washington,  D.C.: May 6, 1991.


Question-summary annotations: questions: question: What have been the results of the ROTC goals for each military service?
answer: From fiscal years 2008 through 2012, each military service met at least 91 percent of their overall Reserve Officers' Training Corps (ROTC) goals for producing the number of officers needed to meet service end strength authorizations, but each has reported challenges in commissioning officers for some certain occupational specialties, such as engineers and nurses.
child_questions: question: What did GAO analysis find about ROTC units meeting DOD minimum average annual production requirement?
answer: Further, GAO's analysis found that half of the Department of Defense's (DOD) ROTC units did not meet DOD's minimum average annual production requirement over the 5-year period reviewed.

question: How does cost per commissioned officer vary?
answer: Further, cost per commissioned officer varies greatly depending on unit production.
child_questions: question: What is an example of cost per commissioned officer variation?
answer: For example, excluding tuition costs, the average cost per officer produced across all units was about $68,000, compared to an average cost of about $95,000 per officer for units that produced fewer than 15 officers on average annually from fiscal years 2008 through 2012.

questions: question: What do DOD instruction specified factors not do?
answer: DOD's instruction specifies factors to consider before closing ROTC units, but these factors do not constitute clearly defined performance measures that provide a comprehensive understanding of the effectiveness and efficiency of ROTC programs, and service evaluations of ROTC programs are ad hoc.

question: What do DOD instructions specify?
answer: DOD's instruction specifies that in assessing units for closure, the services are to consider the quality of officers produced by a unit but it does not clearly define characteristics that comprise quality, which has led to the inconsistent application of the measure by the services.

question: What have the services done to assess performance?
answer: The services conducted 11 evaluations over the past decade to assess performance and identify units for consolidation or closure.
child_questions: question: How have these evaluations occurred?
answer: However, the evaluations have largely occurred on an ad hoc basis because the services have not established a systematic process to routinely evaluate ROTC program performance.

question: What do results-oriented program management practices include?
answer: Moreover, results-oriented program management practices include routine program evaluations that determine how well a program is working.
child_questions: question: What happens without clearly defined performance measures?
answer: Without clearly defined performance measures and routinely conducted evaluations, it will be difficult for the services to accurately determine if ROTC programs are effectively and efficiently operating.

questions: question: What does the office of the Secretary of Defense conduct?
answer: The Office of the Secretary of Defense (OSD) conducts some oversight functions, such as setting policy for the ROTC program.
child_questions: question: What does the OSD not review?
answer: However, although specified in guidance, OSD does not review the services' methodologies for closing ROTC units because OSD officials believe this is a service responsibility.

question: What do the services not consistently communicate with key stakeholders about?
answer: Further, the services do not consistently communicate with key stakeholders, such as members of Congress and schools, about performance of ROTC programs, except when closure decisions are being considered.

question: What may this have contributed to?
answer: This may have contributed to the difficulty the services have experienced in gaining political support for such closures.

question: What has GAO noted about regular communication with stakeholders?
answer: GAO has noted that regular communication with stakeholders helps build trusting relationships to gain buy-in.

question: What happens without clearly delineated responsibilities for oversight?
answer: Without clearly delineated responsibilities for oversight of ROTC programs and a formal strategy to communicate with key stakeholders on ROTC program performance, DOD will find it difficult to obtain the support that is needed to improve the efficiency and effectiveness of ROTC programs.

questions: question: What does ROTC do for DOD?
answer: ROTC is the largest source of newly commissioned officers for DOD.

question: What did GAO analysis of DOD data identify?
answer: GAO's analysis of DOD data identified more than 9,000 officers commissioned from ROTC in fiscal year 2012.

question: What is ROTC critical for?
answer: ROTC is critical for producing officers from the nation's colleges and universities to meet the leadership and readiness needs of the armed forces.

question: What did the National Defense Authorization Act mandate?
answer: The National Defense Authorization Act for Fiscal Year 2013 mandated GAO to review the services' ROTC programs.
child_questions: question: What does this report assess?
answer: This report assesses the extent that ROTC programs (1) met goals and minimum annual requirements for producing officers over a 5-year period, (2) have established performance measures and conducted evaluations for managing ROTC programs, and (3) are subject to oversight and have effective processes for communicating with key stakeholders.

question: What did GAO analyze and review?
answer: GAO analyzed ROTC production data from fiscal years 2008 through 2012; reviewed relevant legislation and DOD and service policies and guidance for ROTC; and reviewed and discussed assessments of ROTC efficiency and effectiveness, and oversight of ROTC with officials from OSD and the services.