Military Recruiting: DOD Needs to Establish Objectives and Measures to Better Evaluate Advertising's Effectiveness
Published: 2003-09-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) must convince more than 200,000 people each year to join the military. To assist in recruiting, the military services advertise on television, on radio, and in print and participate in other promotional activities. In the late 1990s, some of the services missed their overall recruiting goals. In response, DOD added recruiting resources by increasing its advertising, number of recruiters, and financial incentives. By fiscal year 2003, DOD's total recruiting budget was approaching $4 billion annually. At the request of Congress, GAO determined the changes in DOD's advertising programs and funding trends since the late 1990s and assessed the adequacy of measures used by DOD to evaluate the effectiveness of its advertising. GAO recommends that DOD set clear, measurable advertising
section_title: What GAO Found
paragraphs: Since the late 1990s, DOD has revamped its recruiting advertising programs and nearly doubled the funding for recruiting advertising. The military services have revised many of their advertising campaigns and focused on complementing traditional advertising, such as by increasing the use of the Internet, and participating in more promotional activities, such as sports car racing events. DOD's total advertising funding increased 98 percent in constant dollars from fiscal year 1998 through fiscal year 2003--from $299 million to $592 million. The advertising cost per enlisted recruit has nearly tripled and is now almost $1,900. The military services agree that the revised strategies and increased investments have energized their advertising campaigns and better positioned them to recruit in an increasingly competitive marketplace. Today, almost all of the active and reserve components are meeting their overall recruiting goals in terms of the quality and quantity of new recruits. DOD does not have clear program objectives and adequate outcome measures to evaluate the effectiveness of its advertising as part of its overall recruiting effort. Thus, DOD cannot show that its increased advertising efforts have been a key reason for its overall recruiting success. Isolating the impact of advertising on recruiting efforts is inherently difficult because joining the military is a profound life decision. Moreover, DOD has not consistently tracked key information, such as public awareness of military recruiting advertising and the willingness of young adults to join the military. Such data could be used to help evaluate the effectiveness of advertising. Without sufficient information on advertising's effectiveness, DOD cannot determine the return on its advertising funding or make fact-based choices on how its overall recruiting investments should be allocated.

section_title: Letter
paragraphs: To meet its human capital needs, the Department of Defense (DOD) must  convince about 200,000 people each year—the majority of them recent  high school graduates—to join the military. To assist in this recruiting  effort, the military services advertise on television, on radio, in print, and  on the Internet; sponsor sports teams; and participate in other  promotional activities. Such advertising is designed primarily to raise  awareness of the military as a career option and help recruiters meet their  goals for new recruits. During the exceptionally strong U.S. economy of  the late 1990s, most of the services missed their overall recruiting goals. In  response, DOD put additional resources into recruiting by increasing  advertising, the number of recruiters, and various incentives, such as  enlistment bonuses. By fiscal year 2003, DOD’s total recruiting budget was  approaching $4 billion annually.
The Senate Committee on Armed Services directed that we examine  DOD’s growing investments in military recruitment advertising. As agreed  with your committees, the objectives of this report were to (1) determine  the changes in DOD’s advertising programs and funding trends since the  late 1990s and (2) assess the adequacy of the measures used by DOD to  evaluate the effectiveness of its advertising. In March 2003, we provided  your committees with an interim briefing that described the trends in  advertising funding requests since fiscal year 2000 and DOD’s justifications  for those requests.
This report updates the information discussed in the interim briefing and  provides our analysis of the other issues in your request. To determine the  changes in DOD’s advertising programs and funding trends since the late  1990s, we reviewed the changes in the services’ advertising programs,  DOD’s and the services’ congressional justification books, and DOD  funding data. To assess the adequacy of DOD’s outcome measures, we  used established management guidance provided in the Government  Performance and Results Act (GPRA) and in Office of Management and  Budget guidance. Our scope included DOD’s active duty services, reserve  components, and joint advertising program. We conducted our review  from October 2002 through July 2003 in accordance with generally  accepted government auditing standards. (See appendix I for more  detailed information on our scope and methodology.)
subsections: section_title: Background
paragraphs: Most of the military services’ active and reserve components faced  recruiting difficulties during the strong economic climate of the late 1990s.  As a result, the services stepped up their recruiting to ensure that they  would have enough recruits to fill their ranks. Recruiting efforts focus on  three initiatives. First, a “sales force” of more than 15,000 recruiters, who  are mostly located in the United States, recruit from the local population.  Second, these recruiters have financial and other incentives that they can  use to convince young adults to consider a military career. Such incentives  include enlistment bonuses and college benefits. Finally, the services use  advertising to raise the public’s awareness of the military and help the  sales force of recruiters reach the target recruiting population and  generate potential leads for recruiters. This advertising can include  television and radio commercials, Internet and printed advertisements,  and special events.
DOD believes that advertising is increasingly critical to its recruiting effort  because convincing young adults to join the military is becoming more  difficult. In 2001, over 70 percent of polled young adults said that they  probably or definitely would not join the military, compared with 57  percent in 1976. The number of veterans is declining, which means that  fewer young adults have influencers—a relative, coach, or teacher—who  have past military experience. Compounding these difficulties,  proportionally more high school graduates are attending college. Finally,  the perception that service in the military is arduous—and possibly  dangerous—can inhibit recruiting efforts. DOD believes that these factors  together make the military an increasingly harder sell as a career choice  and life-style option for young adults.
The Office of the Secretary of Defense is responsible for establishing  policy and providing oversight for the military recruiting and advertising  programs of the active and reserve components. Within the Office of the  Secretary of Defense, the Under Secretary for Personnel and Readiness is  responsible for developing, reviewing, and analyzing recruiting policy,  plans, and resource levels. The office provides policy oversight for  advertising programs and coordinates them through the Joint Marketing  and Advertising Committee. DOD’s strategic plan for military personnel  human resources emphasizes the need to recruit, motivate, and retain  adequate and diverse numbers of quality recruits.
DOD’s recruiting and advertising programs are not centrally managed. All  of the active components and some of the reserve components manage  their separate advertising programs and work closely with their own  contracted advertising agencies. DOD and the services believe that this  decentralized approach better differentiates between the service “brands”  (i.e., Army, Navy, Air Force, Marines). The Joint Advertising, Market  Research, and Studies program, which is funded separately by DOD, exists  to address common DOD requirements, such as conducting market  research and obtaining and distributing lists of potential leads. The joint  program has developed a DOD-wide advertising campaign to target the  adult influencers of potential recruits, but this program had not been fully  implemented at the time of our review.
section_title: DOD Has Revised Advertising Programs and Increased Funding
paragraphs: After most of the services experienced recruiting shortfalls in the late  1990s, DOD reviewed its advertising programs and identified opportunities  for improvement. The services, except the Marine Corps, substantially  revised their advertising campaigns and slogans and contracted with new  advertising agencies. The services told us that their revised campaigns  place them in a better position to recruit today’s young adults. Currently,  almost all of the services and reserve components are achieving their  recruiting goals, and advertising funding has almost doubled since fiscal  year 1998. The increases in funding have not been used to buy more  national media, such as television commercials. Rather, the funding  increases are being directed to other types of advertising, such as special  events marketing and the Internet, that are intended to better reach  today’s young adults. Advertising funding for DOD increased from $299  million in fiscal year 1998 to $592 million in fiscal year 2003, an increase of  98 percent.
subsections: section_title: Military Services Have Revised Their Advertising Campaigns to Better Attract Today’s Young Adults
paragraphs: Recruiting shortfalls in the late 1990s led to an examination and revision of  DOD’s advertising programs. The Army, Navy, and Air Force missed their  recruiting quantity goals, while some of the reserve components fell short  of both their quantity and quality goals. Following these recruiting  shortfalls, Congress asked the Secretary of Defense to review DOD’s  advertising programs and make recommendations for improvements.  DOD has revamped its advertising programs. The active-duty services,  except for the Marine Corps, substantially revised their advertising  campaigns and selected new advertising agencies as their contractors.  They produced new advertising strategies and campaigns, complete with  new slogans and revised television, print, and radio advertisements, along  with new brand images defined by distinct logos, colors, and music. The  services, in conjunction with their advertising agencies, conducted new  research on young adults—their primary target market. During this period,  the joint program developed an advertising campaign to target influencers  of prospective recruits, as recommended in DOD’s review.
In addition to their overall campaigns, all of the services have specialized  campaigns to target diverse segments of the young adult population. For  instance, the Navy created a Web site, called El Navy, which is designed to  better communicate with the Hispanic market, and the Army has  specifically tailored radio advertisements to reach the African American  market. The services also incorporated a greater variety of public relations  and promotional activities, such as participating in job fairs and  sponsoring sports car racing teams, as an integral part of their advertising  programs. As shown in table 1, there are essentially nine advertising  programs that are managed separately by the military services, reserve  components, and the Office of the Secretary of Defense.
The active services told us that they are pleased with their new advertising  campaigns and agencies, and they believe that the revised and better- funded campaigns have placed them in a more competitive position to  recruit young adults. The sluggish U.S. economy has also narrowed  employment options and is considered to be an important factor in easing  the recruiting challenge. Today, all of the active services are meeting or  exceeding their overall recruiting goals. Most of the reserve components  are also achieving their recruiting goals. As of June 2003, the Army  National Guard was falling short of its recruiting goals because of  extensive overseas deployments and the implementation of stop loss  (restrictions on leaving the military). Army National Guard officials stated  that they expect to meet their goals by the end of fiscal year 2003. Some  reserve officers expressed concerns about the negative impact of the  recent high deployment rates on future recruiting. The services, especially  the reserve components, continue to face challenges in recruiting  individuals with some types of specific training or skills, such as medical,  legal, and construction, and they have developed some specialized  advertising campaigns targeted to recruit them.
Since fiscal year 1998, the services have changed how they allocate  advertising funding, according to the figures provided by DOD. Grouped  into three broad categories, advertising funding includes: (1) events  marketing, public affairs and public relations, Internet, and other; (2)  national media; and (3) direct mail and miscellaneous recruiting support.  One of the categories—events marketing, public affairs and public  relations, Internet, and other—has shown the greatest increase as a  percentage of the total budget, nearly tripling from around 10 percent in  fiscal year 1998 to 29 percent in fiscal year 2003. This increase was used  partly to create and produce new advertising campaigns and strategies.  Service officials told us that event marketing and public relations activities  provide recruiters with greater opportunities to interact with potential  recruits and supplement their national media campaigns and other  methods of advertising. One example is the Army’s sponsorship of a sports  racing car.  (See fig. 1.) Internet and Web-site recruiting have also  increased significantly from fiscal year 1998 through fiscal year 2003. All of  the active military services have increased the amount of advertising on  the Internet and have used interactive Web sites to complement their  traditional recruiting and advertising methods.
The expenditures for the national media category, which includes paid  television, radio, and magazine advertisements, have remained relatively  constant. This means that this category’s proportion of the growing total  advertising budgets has actually declined. Specifically, expenditures for  the national media in fiscal year 1998 were more than half of the  advertising budget; currently, it represents about 40 percent. Television  advertising—which offers tremendous reach to target audiences— dominates this category. Television advertising has remained the single  largest advertising expenditure: paid television is still about a quarter of  the total advertising budget for all of the military components.
section_title: DOD Has Significantly Increased Funding for Advertising
paragraphs: DOD’s advertising funding has nearly doubled in the years since 1998 and  most of these increases occurred in the earlier years. (See fig. 2.) Total  advertising funding for all of the services increased 98 percent, from $299  million in fiscal year 1998 to $592 million in fiscal year 2003. The total  DOD advertising budget request to Congress for fiscal year 2004 was  $592.8 million.
Since fiscal year 1998, DOD’s advertising funding, which is included in  DOD’s operation and maintenance appropriations, has increased at a  significantly higher rate than the total of all of DOD’s operation and  maintenance funding. DOD officials cite media inflation as one reason for  increased advertising funding. Inflation for some types of media, especially  for television commercials, has been higher than general inflation.  However, this is not the reason for all of the increases in advertising  funding during this period because not all of the advertising funding is  used for media advertising. For example, only about a quarter of  advertising funds are currently spent to buy time to run television  commercials.
Growing advertising costs are only part of a rapidly increasing total  investment in recruiting. The rising advertising and overall recruiting costs  can be seen in the investment per enlisted recruit—an important bottom- line measure that shows the amount of money spent to enlist each recruit.  Today, the services are spending almost three times as much on  advertising per recruit than in fiscal year 1990. We examined data  collected by DOD from the services, and it showed that the total  advertising investment per enlisted recruit rose from approximately $640  to $1,900 between fiscal year 1990 and fiscal year 2003. As a proportion of  the total recruiting investment, advertising has increased from 8 percent in  fiscal year 1990 to 14 percent in fiscal year 2003. Bonuses and incentives to  enlist have also increased substantially during this same period. The total  recruiting investment per recruit increased almost 65 percent, from  approximately $8,100 in fiscal year 1990 to $13,300 in fiscal year 2003. Very  steep growth occurred between fiscal year 1998 and fiscal year 2002. This  is shown in figure 3.
The increases are not evenly distributed across the services’ advertising  programs. (See table 2.) The Army has the largest advertising budget, and  the Army active and reserve components account for nearly half (about  $295 million) of the total advertising funding. The Marine Corps, at just  under $50 million, has the smallest advertising budget. The Air Force has  experienced the most significant increase in funding, in part owing to the  creation of its first national television campaign. The Navy’s advertising  funding has also increased, but this is primarily due to the addition of  costs related to the Blue Angels and a program to test recruiting kiosks at  public locations.
DOD’s Joint Advertising, Market Research, and Studies Program is  responsible for (1) providing market research and studies for recruiting  and (2) developing an advertising campaign to target adult influencers,  such as parents, coaches, and career counselors. Currently, the joint  program is conducting market research and studies and providing other  support for the services’ advertising programs, such as purchasing lists of  high school students and recent graduates for use in mailing  advertisements. In addition, the program is implementing a limited print  advertising campaign targeting influencers in fiscal year 2003.
The joint advertising campaign has not had consistent funding. Program  managers told us that the current funding level is insufficient to fully  implement the influencer advertising campaign they have developed. In  past years, DOD cut funding for the joint advertising program because of  concerns that the program office was not adequately executing its  advertising budget. For fiscal year 2003, Congress provided the joint  advertising program with less funding than DOD requested, and DOD  subsequently reallocated part of the remaining joint advertising funding to  a program that it considered a higher priority.
section_title: DOD Does Not Adequately Measure Advertising’s Effectiveness
paragraphs: DOD does not have adequate outcome measures to evaluate the  effectiveness of its advertising as part of its overall recruiting effort.  Effective program management requires the establishment of clear  objectives and outcome measures to evaluate the program, and DOD has  established neither. This has been a long-standing problem for DOD  primarily because measuring the impact of advertising is inherently  difficult, especially for a major life decision such as joining the military.  Owing to the absence of established advertising objectives and outcome  measures, DOD has not consistently collected and disseminated key  information that would allow it to better assess advertising’s contribution  to achieving recruiting goals. This information would include public  awareness of military recruiting advertising and the willingness of young  adults to join the military. Rather, the services report outcome measures  that focus on achieving overall recruiting goals instead of isolating the  specific contribution of advertising. Without adequate information and  outcome measures, the Office of the Secretary of Defense cannot  satisfactorily review the services’ advertising budget justifications nor can  it determine the return on their advertising dollars as part of their overall  recruiting investment.
subsections: section_title: Effective Management Requires Clear Program Objectives and Outcome Measures
paragraphs: The Secretary of Defense is required by law to enhance the effectiveness  of DOD’s recruitment programs through an aggressive program of  advertising and market research targeted at prospective recruits and those  who may influence them. DOD guidance requires the services, by active  and reserve components, to report their resource inputs—how much they  are spending on advertising. DOD guidance also requires the services to  report on overall recruiting outcomes—their recruit quantity and quality.  However, the guidance does not require active and reserve components to  report information specifically about the advertising programs’ recruiting  effectiveness.
Effective program management requires the establishment of clearly  defined objectives and outcome measures to evaluate programs. The  Government Performance and Results Act was intended to help federal  program managers enhance the effectiveness of their programs. It  requires agencies to establish strategic plans for program activities that  include, among other things, a mission statement covering major functions  and operations, outcome-related goals and objectives, and a description of  how these goals and objectives are to be achieved. GPRA shifted the focus  of accountability for federal programs from inputs, such as staffing and  resource levels, to outcomes. This requires agencies to measure the  outcomes of their programs and to summarize the findings of program  evaluations in their performance reports. The Office of Management and  Budget’s guidance implementing GPRA requires agencies to establish  meaningful program objectives and identify outcome measures that  compare actual program results with established program objectives.
section_title: Lack of Adequate Outcome Measures Limits DOD’s Ability to Effectively Allocate Its Recruiting Investments
paragraphs: DOD does not have adequate information to measure the effectiveness of  its advertising as part of the overall recruiting effort. Measuring  advertising’s effectiveness has been a long-standing problem, partly  because it is inherently difficult to measure the impact that advertising has  on recruiting. DOD has not established advertising program objectives and  it lacks adequate outcome measures of the impact that advertising  programs have on recruiting. Outcome measures are used to evaluate how  closely a program’s achievements are aligned with program objectives, and  to assess whether advertising is achieving its intended outcome. DOD  currently requires the services and reserve components to report on inputs  and outcomes related to overall recruiting. These measures are important  in assessing DOD’s overall recruiting success; however, they do not assess  advertising’s contribution to the recruiting process.
subsections: section_title: Measuring Advertising’s Effectiveness Is a Long- standing and Difficult Problem
paragraphs: In our 2000 report, we noted that the services do not know which of their  recruiting initiatives—advertising, recruiters, or bonuses—work best.  This prevented DOD from being able to effectively allocate its recruiting  investment among the multiple recruiting resources. We recommended  that DOD and the services assess the relative success of their recruiting  strategies, including how the services can create the most cost-effective  mix of recruiters, enlistment bonuses, college incentives, advertising, and  other recruiting tools. In comments on that report, DOD stated that it  intended to develop a joint-service model that would allow trade-off  analyses to determine the relative cost-effectiveness of the various  recruiting resources. This has not been done, and the current DOD cost  performance trade-off model does not support analyses across the  services’ budgets.
Similarly, a 2002 Office of Management and Budget assessment, known as  the Program Assessment Rating Tool, found that DOD’s recruiting  program had met its goal of enlisting adequate numbers of recruits;  however, since there were no measures of program efficiency, the overall  rating for the recruiting program was only “moderately effective.” In its  assessment, the Office of Management and Budget noted the inability of  the recruiting program to assess the impact of individual resources, such  as advertising and recruiters. The services continually adjust the mix of  funding between advertising and other recruiting resources to accomplish  their program goals. They have generally increased spending on  advertising, added recruiters, and increased or added bonuses at the same  time, making it impossible to determine the relative value of each of these  initiatives. Other studies have reached similar conclusions. In 2000, a  review of DOD’s advertising programs resulted in a recommendation that  they be evaluated for program effectiveness. More recently, the National  Academy of Sciences also cited the need to evaluate advertising’s direct  influence on actual enlistments. The academy is now doing additional  work on evaluating DOD’s advertising and recruiting.
The lack of adequate information is partly attributable to the inherent  difficulty in measuring advertising’s affect on recruiting. Measuring  advertising’s effectiveness is a challenge for all businesses, according to  advertising experts. Private-sector organizations cannot attribute  increases in sales directly to advertising because there are many other  factors influencing the sale of products, such as quality, price, and the  availability of similar products. Many factors impact recruiting as well,  such as employment and educational opportunities, making it especially  difficult to isolate and measure the effectiveness of advertising. Enlisting  in a military service is a profound life decision. Typically, an enlistment is  at least a 4-year commitment and can be the start of a long military career.
Another complicating factor in measuring advertising’s effectiveness is  that it consists of different types and is employed differently throughout  the recruiting process to attract and enlist potential recruits. Figure 4  displays the recruiting process and demonstrates the role of advertising  while a young adult may be considering enlisting in the military.
As the figure shows, the use of multiple types of advertising at various  stages in the recruiting continuum makes it difficult to assess the  effectiveness of specific types of advertising. A single recruit may be  exposed to some or all of these advertising types. Traditional advertising  in the national media, such as television and magazines, is intended to  disseminate information designed to influence consumer activity in the  marketplace. The services typically use such national media to make  young people aware of a military service, the career options available in a  service, and other opportunities the services have to offer them. Direct  mail, special events, and the services’ Web sites are utilized to provide  more detailed information about the services and the opportunities  available for persons who enlist. These marketing resources give people  the opportunity to let a recruiter know they are possibly interested in  enlisting in a service.
section_title: DOD Lacks Guidance Establishing Advertising Programs’ Objectives and Outcome Measures
paragraphs: Another contributing factor to the absence of advertising objectives and  outcome measures is the lack of DOD-wide guidance. Officials from the  Office of the Secretary of Defense view their role as overseeing the  decentralized programs managed by the individual services and reserve  components. They scrutinize the quality and quantity of recruits and  gather data about the uses of advertising funds. However, they told us they  were reluctant to be more prescriptive because of a concern about  appearing to micromanage the successful recruiting programs of the active  and reserve components. On the basis of our work, their sensitivity is  warranted. The active and reserve components tend to guard their  independence, seeking to maintain their “brand” and arguing that the  current decentralized structure allows them to be more responsive to their  individual needs. The Office of the Secretary of Defense seeks to  coordinate the active and reserve components’ activities through joint  committees and to centralize research that can be utilized by all.
Defining exactly what to measure may be difficult, but it is not impossible.  DOD and the services, as well as their contracted advertising agencies,  generally agree that there are at least two key advertising outcomes that  should be measured: (1) the awareness of recruiting advertising and (2)  the willingness or “propensity” to consider joining the military. However,  this is not clearly stated in any program guidance. Current DOD guidance  requires only that the services provide information on funding for  advertising, the quality and quantity of recruits, and the allocation of  resources to the various advertising categories. Although this information  is valuable—in fact, critical—it is not sufficient to evaluate and isolate the  effectiveness of the services’ advertising programs.
DOD’s efforts thus far to measure the awareness of recruiting advertising  and willingness to join the military have met with problems. Inconsistent  funding for the Joint Advertising, Market Research, and Studies program  has hampered consistent collection of this information. DOD has  sponsored an advertising tracking study designed to monitor the  awareness of individual service campaigns since 2001. However, officials  from the Army, Navy, and Marine Corps told us that they do not regularly  use the research provided by this study. According to program officials,  there were numerous problems with the advertising tracking study. DOD  is implementing changes to the study that are intended to improve its  usefulness to all of the active and reserve components. In the absence of  reliable and timely advertising tracking, the Army implemented its own  tracking study, and the Air Force is currently planning an experimental  study to assess the effectiveness of its national television advertising  campaign, according to program managers. To monitor the willingness to  join the military, DOD sponsors youth and adult polls, which are designed  to track changes in attitudes and young adults’ aspirations. These polls  replaced the Youth Attitude Tracking Survey, which had been in place for  a number of years and provided long-term trend data about the propensity  of young adults to consider the military. The services expressed concern  that the current polls ask questions that are significantly different from  those asked in the prior survey, which makes the analysis of trends  difficult.
DOD officials also pointed to research indicating that advertising is a cost- effective recruiting investment when compared with other recruiting  initiatives. For example, a report that was done for DOD found that it was  less expensive to enlist a recruit through increased investments in  advertising than through increased investments in military pay for new  recruits in the Army and Navy. Similarly, a study for DOD analyzed the  marginal cost of different recruiting initiatives and concluded that, under  certain conditions, it was more cost-effective to invest additional funds in  advertising than in military pay for recruits or recruiters. DOD officials  told us that these reports, which used data from the 1980s and early 1990s,  provide the best research available on the topic. However, the situation  has changed dramatically in recent years. DOD has altered its advertising  and recruiting strategies and is spending much more on advertising.  Advertising itself is also changing and is more fragmented with an  expanding array of television channels and other media. Finally, media  inflation, which has increased faster than general inflation even in the  sluggish economy, has lessened buying power.
section_title: Conclusions
paragraphs: Funding devoted to advertising has increased considerably since fiscal  year 1998. Although the military services are now generally meeting their  overall recruiting goals, the question of whether the significant increases  in advertising budgets were a main contributor to the services’ recruiting  successes remains open. During the same period, DOD also greatly  increased funding for bonuses and other incentives to enlist recruits. At  the same time, the U.S. economy slowed dramatically, narrowing the other  employment options available to young people. These factors make it  difficult to disentangle the effects of the internal DOD investments made  in recruiting from the changes in the external recruiting environment.  Even though the effect of advertising is inherently difficult to measure, this  issue needs to be addressed. This is crucial because DOD is now spending  nearly $592 million annually on recruiting advertising, or about $1,900 per  enlisted recruit. In addition, the total funding for all of DOD’s recruiting  efforts is now almost $4 billion.
DOD needs better advertising outcome measures to allow it to oversee and  manage the advertising investment as part of its overall recruiting effort.  DOD and the services have an understandable focus on the most  important program outcome—to ensure that the military has enough  quality recruits to fill its ranks. Judged by this short-term measure, the  recruiting programs are successful. But now that DOD is meeting its  recruiting goals, should it reduce advertising funding or continue at its  current funding levels? DOD believes that continued investments in  advertising are critical to keeping awareness up in the young adult  population and combating the declining propensity among today’s young  adults to join the military. However, DOD has neither stated these goals  clearly in its guidance, nor consistently gathered information to ensure  that these objectives are being met. Now that it is meeting its recruiting  goals, DOD needs to turn its attention to program effectiveness and  efficiency to ensure that the active and reserve components are getting the  best return on their recruiting and advertising investments.
section_title: Recommendations for Executive Action
paragraphs: To improve DOD’s ability to adequately measure the impact of its  advertising programs on its recruiting mission, we recommend that the  Secretary of Defense direct the Under Secretary of Defense for Personnel  and Readiness to issue guidance that would (1) set clear, measurable  objectives for DOD’s advertising programs; (2) develop outcome measures  for each of DOD’s advertising programs that clearly link advertising  program performance with these objectives; and (3) use these outcome  measures to monitor the advertising programs’ performance and make  fact-based choices about advertising funding as part of the overall  recruiting investment in the future.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD concurred with all of our recommendations. In commenting on this  report, DOD stated that the Office of the Under Secretary of Defense for  Personnel and Readiness, in concert with the services, will develop an  advertising strategic framework to provide overall direction for DOD’s  advertising programs. The framework, with associated outcome measures,  would allow the office to monitor advertising results regularly and make  fact-based decisions at a strategic level. It would provide an overarching  structure within which each service would develop its own advertising  program strategy, program objectives, and outcome measures. The  framework would also direct the activities of the DOD joint program to  ensure support to the services. DOD also commented that current  research has not advanced to the point where models exist that adequately  account for the many factors that affect recruiting as well as for the  differences in the services. DOD stated that it will address this research  gap through several initiatives intended to advance the measurement of  the performance of recruiting and advertising. The National Academy of  Sciences is currently developing an evaluation framework for recruiting  and advertising and expects to publish a report in early 2004.
DOD’s comments are provided in their entirety in appendix II. DOD  officials also provided technical comments that we have incorporated as  appropriate.
We are sending copies of this report to interested congressional  committees; the Secretaries of Defense, the Army, the Navy, and the Air  Force; and the Commandant of the Marine Corps. We will send copies to  other interested parties upon request. In addition, the report will be  available at no charge on the GAO Web site at http://www.gao.gov.
Please contact me at (202) 512-5559 if you or your staffs have any  questions regarding this report. Key contributors to this report were John  Pendleton, Lori Atkinson, Nancy Benco, Kurt Burgeson, Alan Byroade,  Chris Currie, LaTonya Gist, Jim McGaughey, Charles Perdue, Barry  Shillito, and John Smale.
section_title: Appendix I: Scope and Methodology
paragraphs: To describe the changes in the Department of Defense’s (DOD) advertising  programs and advertising funding trends since the late 1990s, we reviewed  advertising exhibits in the operation and maintenance congressional  justification books as well as budget information provided by the Office of  the Secretary of Defense. Since our objective was to look at broad funding  trends, we did not reconcile these requested amounts with actual  obligations or expenditures by the active and reserve components. We  interviewed active and reserve component officials to understand program  changes since the late 1990s. We obtained recruiting mission goals and  actual accessions back to fiscal year 1990 from the Office of the Secretary  of Defense and the services. We obtained information on the quality of  accessions of each of the active and reserve components back to fiscal  year 1990, as well as the investment per active enlisted accession back to  fiscal year 1990. We reviewed information from the Defense Human  Resources Activity and the Joint Marketing and Advertising Committee for  discussions of advertising programs. The services provided additional  information regarding the types of advertising media they use.
To assess the adequacy of the measures used by DOD to evaluate the  effectiveness of advertising, we reviewed information on outcome  measures used to evaluate the effectiveness of advertising provided by  each of the active and reserve components; the advertising agencies that  are their contractors; and the DOD Joint Advertising, Market Research,  and Studies program. We spoke with the advertising contractors to learn  what measures of effectiveness they are aware of and use. We also  reviewed the requirements for establishing program objectives and  outcome measures in the Government Performance and Results Act and in  Office of Management and Budget guidance.
We interviewed DOD and advertising officials from each of the active and  reserve components, as well as representatives from the services’  advertising agencies. We also reviewed their programs, procedures, and  oversight activities. These interviews were conducted with officials in the  Office of the Under Secretary of Defense for Personnel and Readiness;  Office of the Under Secretary of Defense (Comptroller/Chief Financial  Officer); Defense Human Resources Activity, Joint Advertising, Market  Research, and Studies Office; Army Accessions Command, Fort Knox,  Kentucky; Air Force Recruiting Service, Randolph Air Force Base, Texas;  Navy Recruiting Command, Millington, Tennessee; Marine Corps  Recruiting Command, Quantico Marine Corps Base, Virginia; Army  National Guard Recruiting and Retention Command, Arlington, Virginia;  Naval Reserve Command, New Orleans, Louisiana; Air Force Reserve  Command, Robins Air Force Base, Georgia; and the Air National Guard  Office of Recruiting and Retention, Arlington, Virginia. We also  interviewed officials at the contracted advertising agencies for the joint  program, the Army, the Navy, the Marine Corps, and the Air Force. We  reviewed reports on recruiting and advertising from DOD, the  Congressional Research Service, the private sector, and others. We  obtained recruiting advertising budget and funding data for types of  advertising from the Office of the Secretary of Defense. We reviewed, but  did not verify, the accuracy of the data provided by DOD.
We conducted our review from October 2002 through July 2003 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
section_title: Related GAO Products
paragraphs: Program Evaluation: Strategies for Assessing How Information  Dissemination Contributes to Agency Goals. GAO-02-923. Washington,  D.C.: September 30, 2002.
Military Personnel: Services Need to Assess Efforts to Meet Recruiting  Goals and Cut Attrition. GAO/NSIAD-00-146. Washington, D.C.: June 23,  2000.
Military Personnel: First-Term Recruiting and Attrition Continue to  Require Focused Attention. GAO/T-NSIAD-00-102. Washington, D.C.:  February 24, 2000.
Military Recruiting: DOD Could Improve Its Recruiter Selection and  Incentive Systems. GAO/NSIAD-98-58. Washington, D.C.: January 30, 1998.
Military Personnel: High Aggregate Personnel Levels Maintained  Throughout Drawdown. GAO/NSIAD-95-97. Washington, D.C.: June 2,  1995.
Military Recruiting: More Innovative Approaches Needed. GAO/NSIAD- 95-22. Washington, D.C.: December 22, 1994.
Military Downsizing: Balancing Accessions and Losses Is Key to  Shaping the Future Force. GAO/NSIAD-93-241. Washington, D.C.:  September 30, 1993.