Military Personnel: DOD Needs to Establish Performance Measures for the Armed Forces Sports Program
Released: 2017-06-08T00:00:00

Letter

Background

Roles and Responsibilities for the Armed Forces Sports Program

DOD instruction 1330.04 outlines the following roles and responsibilities regarding the Armed Forces Sports Program:

Principal Deputy Under Secretary of Defense for Personnel and Readiness: Provides guidance and oversight concerning the participation of servicemembers in Armed Forces, national, and international amateur sports competitions.

Senior Military Sports Advisor: Serves as the Service Personnel Chief who is responsible for the management and operation of the program and reports to the Principal Deputy Under Secretary of Defense for Personnel and Readiness.

Armed Forces Sports Council: Serves as the governing body of the program, and is composed of the Morale, Welfare, and Recreation representatives from each service or their designated representatives.

Armed Forces Sports Council Secretariat: Serves as the executive office for the council and serves as the U.S. liaison to the International Military Sports Council.

Armed Forces Sports Council Working Group: Serves as the staffing body of the Armed Forces Sports Council, which is composed of Morale, Welfare, and Recreation representatives from each service.

Secretaries of the Military Departments: Develop sports programs based on specific needs and mission requirements that provide the opportunity for servicemembers to prepare for and compete in national and international amateur sports competitions on a voluntary basis.

According to Sports Council Secretariat officials and the policies for managing servicemembers’ participation in national and international amateur sports competitions, the Sports Council Secretariat and the service sports offices each have responsibilities for managing the Armed Forces Sports Program. Table 1 further describes the responsibilities of the Armed Forces Sports Council Secretariat’s and the service sports offices for the Armed Forces Sports Program.

Number of Staff Members Working for the Armed Forces Sports Program and Team and Individual Sports Offered

The number of staff members working in the Armed Forces Sports Council Secretariat and the service sports offices and the percentage of time staff members spend working for the Armed Forces Sports Program varies. For example, the Navy Sports Office has two staff members who work on the program nearly full time, while the Army Sports Office has four staff members who work on the program part time. In addition, the staff members working for the Armed Forces Sports Program include both civilians and active-duty servicemembers. Table 2 provides further details on the number of staff members and the estimated percentage of time they spend working for the Armed Forces Sports Program.

DOD Has Participation Data, but Has Not Developed Performance Measures or Clarified Roles and Responsibilities to Help Ensure Effective Program Implementation

DOD has data on participation in and costs of the Armed Forces Sports Program, but has not taken steps, including developing performance measures and clarifying roles and responsibilities, that are needed to help ensure that the program is implemented effectively.

DOD Has Data on Participation in and Costs of the Armed Forces Sports Program

Sports Council Secretariat officials provided us with data for fiscal years 2012-2016 on servicemember participation in the program, including on the number of days servicemembers are away from their unit participating in the program and on civilians supporting the program, and data for fiscal years 2014-2016 on program costs.

Servicemember Participation

In analyzing the number of servicemembers participating in the program, we found that servicemember participation changed from 968 servicemembers in fiscal year 2012 to 848 servicemembers in fiscal year 2016. Table 3 provides further details about the number of servicemembers who participated in or supported the Armed Forces Sports Program in fiscal years 2012-2016.

We also found that servicemember participation ranged from an average of 6.8 days per event in fiscal year 2013 to 13.2 days per event in fiscal year 2016. Sports Council Secretariat and service officials stated that the servicemembers who participate in the program are in peak physical shape and that they were unaware of any additional recovery time that a participant has needed after competing. Table 4 breaks out these data for each year from fiscal years 2012 through 2016.

According to officials, DOD civilians provide various types of support to the Armed Forces Sports Program and may include employees who work for the program on a full- or part-time basis, as well as those who serve in a volunteer capacity. Civilians who support the program as volunteers may serve in a variety roles, as coaches or as staff, for example, which include athletic trainers, service representatives, or medical staff. Table 5 provides further details on the number of civilians who supported the Armed Forces Sports Program in fiscal years 2012 through 2016.

Sports Council Secretariat officials stated the program covers the cost of servicemembers participating and units do not have to provide any funding. Program costs ranged from about $2.1 million to about $2.8 million from fiscal years 2014 through 2016. Table 6 provides additional details about these costs.

Armed Forces Sports Championships are hosted by one of the services and must include at least three of the services in competition for all team sports and most individual sports. Higher level competitions are attended by the most competent athletes from the Armed Forces Sports Championships or athletes selected based on other qualifying events or criteria and may include U.S. national, International Military Sports Council, or other international events. In table 7 we break out the costs for participation in events from table 6 associated with Armed Forces Sports Championships and higher level competitions for fiscal years 2014 through 2016.

DOD Has Not Developed Performance Measures to Demonstrate the Effectiveness of the Armed Forces Sports Program

While DOD has data on program participation and cost, these data are outputs and not outcomes and therefore do not exhibit important attributes of successful performance measures that are necessary to demonstrate that the Armed Forces Sports Program is being implemented effectively. Federal internal control standards state, among other things, that managers should establish activities to monitor performance measures. Furthermore, our prior work on performance measurement identified ten key attributes of performance measures, such as clarity, objectivity, having a measurable target, and having baseline and trend data in order to identify, monitor, and report changes in performance and to help ensure that performance is viewed in context. Table 8 identifies each attribute of effective performance measures along with its definition.

Sports Council Secretariat officials stated that they use data on the number of servicemembers and services annually participating in each sport and competition to measure the performance and effectiveness of the Armed Forces Sports Program. While these data provide important context about the program’s size and reach, they are outputs and do not constitute performance measures because they do not exhibit several of the key attributes previously discussed.

First, we found that the Sports Council Secretariat’s use of participation data does not exhibit the attribute of linkage in that there is not clear alignment between the number of participants and how it affects the program’s ability to achieve its goals and mission. For example, while DOD Instruction 1330.04 does not specify goals or a mission, the Armed Forces Sports Council’s standard operating procedures identify that the five objectives of the program are to: (1) promote goodwill among the Armed Services through sports, (2) promote a positive image of the Armed Forces through sports, (3) provide the incentive and encourage physical fitness by promoting a highly competitive sports program, (4) provide a venue for military athletes to participate in national and international competitions, and (5) engage in valuable military-to-military opportunities with International Military Sports Council member nations through sports.

However, Sports Council Secretariat officials have not established a link between the participant data that they stated are used to measure program performance and the achievement of these objectives.

Further, our prior work has shown that linkages between goals and measures are most effective when they are clearly communicated and create a “line of sight” so that everyone understands what an organization is trying to achieve and the goals it seeks to reach. During meetings with the Sports Council Secretariat, officials stated that they use data, such as servicemember participation in the Armed Forces Sports Championships, International Military Sports Council Championships, U.S. Nationals, and the Olympic and Paralympic Games to measure the performance and effectiveness of the Armed Forces Sports Program, and that they have created performance measures on an as-needed basis when it has been necessary to prioritize the allocation of funds for individual sports. However, none of the documents we were provided on the program identify participation or any other data as a performance measure, and these efforts do not exhibit a deliberate, four-stage performance measurement process that involves (1) identifying goals, (2) developing performance measures, (3) collecting data, and (4) analyzing data and reporting results. Further, servicemember participation in the Olympic and Paralympic Games is not a valid performance measure because, according to officials from the Office of the Secretary of Defense, the Sports Council Secretariat, and the services, the Armed Forces Sports Program does not have responsibility for these games.

Second, participation data do not exhibit the measurable target attribute because they represent a summary of the program’s activity and are not associated with numerical goals, which are needed to gauge program progress and results. Our prior work has shown that numerical targets or other measurable values facilitate future assessments of whether overall goals and objectives are achieved because comparisons can be easily made between projected performance and actual results. While the Sports Council Secretariat’s data included the “actual” number of program participants, they did not identify projected performance targets that would enable program officials to determine how far they have progressed toward a desired outcome or end state. In response to our analysis, Sports Council Secretariat officials stated that they consider the list of 24 sports and total number of competitions that servicemembers may participate in to be the target—the attainment of which is based on variables such as available funding and the extent to which each service agrees to provide teams to participate in the competitions. However, this is not a valid demonstration of this attribute because neither the target in this sense nor the variables affecting participation (e.g., funding and service branch involvement) demonstrate how well the Armed Forces Sports Program performs or carries out its mission.

In addition, officials from the Sports Council Secretariat and the services stated that the program directly benefits the services’ readiness, recruitment, and retention efforts. Specifically, officials cited the program’s emphasis on a higher level of physical fitness than is otherwise required by the services as contributing to individual servicemember readiness, and involvement in national and international sports championships as aiding recruiting efforts because it showcases some of the unique opportunities open to those in the services. Further, officials stated that the opportunity to participate in higher level competitions through the program helps retention because it provides an incentive for some servicemembers to stay in the services. However, outside of participation and cost data and some anecdotal examples, officials did not have specific measures for or data on the Armed Forces Sports Program’s contribution to the services’ readiness, recruiting, and retention efforts.

Third, while DOD has program participation data, it does not track baseline and trend data in order to assess the program’s performance and progress over time. Our prior work has demonstrated that by tracking and developing a performance baseline for all measures—including those that demonstrate the effectiveness of a program—agencies can better evaluate progress made and whether or not goals are being achieved. Further, identifying and reporting deviations from the baseline as a program proceeds provides valuable information for oversight by identifying areas of program risk and their causes for decision makers. According to Sports Council Secretariat officials, many of the program’s benefits—such as helping with readiness, recruitment, and retention—are not measured, and commanding officers are responsible for determining and managing the program’s effect on the readiness of their units. Thus, given the relatively small number of program participants and participation being contingent on obtaining commanding officer approval, Sports Council Secretariat officials stated that they do not believe that the services’ readiness is negatively affected by servicemembers participating in the Armed Forces Sports Program.

We acknowledge that the measurement of the program’s performance may be difficult, but DOD’s participation data do not include targets allowing program performance to be measured and do not assess the intended benefits of the program. Without effective performance measures that demonstrate linkage with the program’s goals or mission, have measurable targets, and an established baseline of data, DOD will be unable to effectively demonstrate the benefits of the program and will not have the information needed to ensure that the department is allocating resources to its highest priority efforts.

DOD’s Implementation of Armed Forces Sports Program’s Roles and Responsibilities Differs from Those Currently Defined in Policy

The roles and responsibilities that are currently being implemented for the Armed Forces Sports Program differ from the program’s roles and responsibilities specified in DOD policy. DOD Instruction 1330.04 and the Armed Forces Sports Council’s standard operating procedures specify that the Armed Forces Sports Program includes training or national qualifying events in preparation for participation in International Military Sports Council events, the Pan American Games, the Olympic Games, the Paralympic Games, and other international competitions. While this is how the program is defined in key program documents, the Office of the Secretary of Defense, Sports Council Secretariat, and service officials stated that all responsibilities, including costs, associated with servicemember participation in the Pan American, Olympic, and Paralympic Games are, in practice, handled by the services.

According to these officials, the program’s primary objective when it was established was to support the Olympic movement by providing servicemembers the opportunity to compete in the 1948 London Olympic Games. Further, DOD Instruction 1330.04 specifies that the Armed Forces Sports Program includes, among other things, training or national qualifying events in preparation for participation in the Pan American Games, the Olympic Games, and the Paralympic Games. However, officials stated that over time, the services assumed responsibility for their respective servicemembers’ participation in the Pan American, Olympic, and Paralympic Games.

The Office of the Secretary of Defense and Sports Council Secretariat officials stated they plan to review DOD Instruction 1330.04 and make necessary updates but did not indicate what specific changes would be made to clarify the program’s roles and responsibilities. Further, these officials stated that they were not sure whether they would remove the Pan American, Olympic, and Paralympic Games from the Armed Forces Sports Council’s standard operating procedures because of the potential for responsibilities to shift again in the future.

Conclusions

The Armed Forces Sports Program provides a means by which servicemember athletes can participate in national and international competitions while representing the Armed Forces. However, the program currently does not have performance measures with linkage, measurable targets, or a baseline. Without measures that address the desired outcomes and include these attributes, it will be difficult for DOD and Congress to determine whether the program is meeting the desired goals or benefiting readiness, recruitment, and retention.

Recommendations for Executive Action

To improve the management of the Armed Forces Sports Program and better determine whether the program is achieving its desired results, we recommend that the Secretary of Defense direct the Under Secretary of Defense for Personnel and Readiness to develop and implement performance measures for the Armed Forces Sports Program that measure the desired outcomes for the program and, at a minimum, demonstrate linkage to the program’s goals or mission, have a measurable target, and include a baseline that can be used to demonstrate program performance.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD and the Department of Homeland Security (DHS) for review and comment. In its comments on a draft of this report, DOD concurred with our recommendation and their comments are reprinted in their entirety in appendix II. DOD and DHS also provided technical comments, which we incorporated into the report as appropriate.

DOD concurred with our recommendation to develop and implement performance measures for the Armed Forces Sports Program that measure the desired outcomes for the program and, at a minimum, demonstrate linkage to the program’s goals or mission, have a measurable target, and include a baseline that can be used to demonstrate program performance but also noted potential limitations on establishing measures. Specifically, DOD said that it will explore the development and implementation of performance outcome measures for the Armed Forces Sports Program and that it will review Department of Defense Instruction 1330.04 for potential opportunities to incorporate appropriate guidance regarding performance measures for the Armed Forces Sports Program. However, DOD stated that there are limitations on establishing metrics for several of the program’s objectives, such as goodwill and positive image, which are challenging to measure. Further, DOD said that quantifying outcomes for some objectives, such as the “spirit” of the program, also will be challenging, but that the lack of a performance measurement does not negate the importance of pursuing objectives that contribute to demonstrating the program’s overall effectiveness. In our report, we acknowledged that measurement of the program’s performance may be difficult but also necessary to produce the evidence-based support that is needed to objectively demonstrate how the specific activities that comprise a program are contributing to its effectiveness. Exploring the development and implementation of performance measures and reviewing DOD guidance regarding performance measures are positive steps, but we continue to believe that DOD needs to develop and implement performance measures in order to demonstrate if the Armed Forces Sports Program is being implemented effectively. While it may be challenging to develop performance measures, our prior work has demonstrated that even for highly complex areas such as DOD’s reform of its medical health system and prevention of sexual assault, developing and implementing performance measures can be done, and if implemented correctly, can enhance decision-making. Until DOD does develop and implement performance measures, it will be unable to effectively demonstrate the benefits of the program and will not have the data needed to monitor the program, make decisions about program management and ensure that the department is allocating resources to its highest priority efforts.

We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Secretary of Homeland Security; the Secretaries of the Army, the Navy, and the Air Force; the Commandants of the Marine Corps and the Coast Guard; and the Under Secretary of Defense for Personnel and Readiness. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

To assess the effectiveness of the Department of Defense’s (DOD) implementation of the Armed Forces Sports Program, we reviewed DOD and service (including the Coast Guard) policies and procedures related to the administration of and participation in the program. We interviewed officials from the Office of the Under Secretary of Defense for Personnel and Readiness, the Armed Forces Sports Council Secretariat (“Sports Council Secretariat”), and each service about these policies and procedures.

We also discussed the extent to which any performance measures had been established to assess the program’s effectiveness, including any effects of program participation on the services’ readiness. We obtained and analyzed data from DOD on the number of active-duty servicemembers, by service, who had participated in the Armed Forces Sports Program in fiscal years 2012 through 2016 as well as on the number of days servicemembers had spent away from their respective units participating in the program during the same time frame. We also obtained and analyzed data from DOD on the number of DOD and Coast Guard civilians who had supported the Armed Forces Sports Program in fiscal years 2012 through 2016. Further, we obtained and analyzed data from DOD on program costs for fiscal years 2014 through 2016, including the administrative, travel, and salary costs incurred by the Armed Forces Sports Council Secretariat, program-related travel and salary costs for each service, and participation costs of travel participants, which according to program officials include transportation and lodging costs. The time frame of the participant and cost data that we obtained differs because DOD officials stated that fiscal year 2014 was the most recent year that cost data were available from all the services. Based on responses from the Armed Forces Sports Program office to data reliability questionnaires, we determined that the data we obtained were sufficiently reliable for the purposes of this review. We compared DOD’s policy for the program against the federal standards for internal control that state, among other things, that managers should establish activities to monitor performance measures. Additionally, we compared DOD’s participant data—the department’s measure for demonstrating the effectiveness of the Armed Forces Sports Program—with our prior work on performance measurement to determine the extent to which these data exhibit the ten key attributes of successful performance measures.

To obtain servicemembers’ perspectives on the Armed Forces Sports Program and its effect on individual readiness, we interviewed 13 randomly selected servicemembers who had participated in the program in calendar year 2015 since, at that time, this was the most recent year for which the program had a complete set of participant data. To understand any effect that a servicemembers’ participation may have had on unit readiness, we also interviewed ten commanding officers who had approved one of the randomly selected servicemembers’ requests to participate in the Armed Forces Sports Program. While the information that we obtained was nongeneralizable, it provided perspectives from individuals with first-hand experience with the Armed Forces Sports Program.

We also reviewed DOD and service policies and procedures to identify roles and responsibilities associated with implementing the Armed Forces Sports Program. Further, we interviewed officials within each organization to discuss how designated roles and responsibilities were being implemented.

We conducted this performance audit from August 2016 to June 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Kimberly A. Mayo, Assistant Director; Christopher H. Conrad; Mae Frances Jones; Stephanie Moriarty; Shahrzad Nikoo; Shane T. Spencer; Andrew Stavisky; and John W. Van Schaik made key contributions to this report.