DOD Advertising: Better Coordination, Performance Measurement, and Oversight Needed to Help Meet Recruitment Goals
Released: 2016-05-12T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: As part of its efforts to meet yearly recruitment goals for the military, DOD requested almost $575 million in fiscal year 2017 to conduct advertising intended to increase awareness of military service and ultimately generate leads for potential recruits.

Senate Report 114-49 included a provision for GAO to assess DOD's advertising activities. This report examines the extent to which DOD (1) has coordinated advertising activities among the military service components, (2) has followed commercial best practices to assess the effectiveness of advertising activities, and (3) has oversight of its components' advertising activities.

GAO identified best practices for assessing the effectiveness of advertising in consultation with advertising industry experts and reviewed DOD and service policies and the most recent version of each military service components' advertising plans, from fiscal years 2014-16.

section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has taken steps to coordinate some advertising activities among the military service components, but it has not developed a formal process for coordination. DOD conducts joint market research and service officials responsible for advertising at times share some information about lessons learned. However, there is no formal process for addressing inefficiencies and to ensure information sharing among the services. GAO found examples of possible unnecessary duplication, overlap, and fragmentation that may result from the absence of coordination. For example, the Air Force has three advertising programs that contract with three advertising agencies, but officials could not provide a rationale for requiring separate programs. In the absence of a formal process for coordination, the services may be missing opportunities to effectively leverage advertising resources.

While DOD has generally followed commercial best practices GAO identified to assess the effectiveness of advertising, DOD's components vary in their ability to determine whether their activities are generating leads for potential recruits. For example, while the Marine Corps has developed a framework to assess the effectiveness of its advertising including leads generated from advertising activities at the local level, Army officials stated they do not have reliable data to evaluate whether locally executed advertising activities are generating leads, and the Army National Guard does not require state units to report on the performance of their advertising activities. Without fully measuring advertising performance, especially at the local levels, DOD may be unable to ensure advertising dollars are used efficiently to help meet recruiting goals.

DOD does not have comprehensive oversight of the components' advertising activities; instead, DOD's components oversee their own programs. However, examples identified by GAO and others of some components paying sport teams to provide recognition ceremonies for servicemembers—a practice later deemed unacceptable by DOD—suggest that the absence of DOD oversight may have contributed to some activities of questionable appropriateness. Further, GAO observed discrepancies in how recent sports advertising guidance was being interpreted and in service officials' understanding of regulations that direct digital advertising. Without a department-wide policy that clearly defines its oversight role, DOD lacks reasonable assurance that advertising is carried out in an appropriate manner.

section_title: What GAO Recommends
paragraphs: GAO recommends that DOD develop a formal process for coordination among its components on crosscutting advertising issues; ensure that the components fully measure the performance of advertising activities; and issue department-wide policy that, among other things, defines DOD's oversight role in advertising. DOD generally agreed with GAO's recommendations.

section_title: Letter
paragraphs: Each year, the Department of Defense (DOD) seeks to meet recruitment  goals targeting the small percentage of the U.S. population between the  ages of 17 and 24 that, according to DOD officials, is able to meet the  education and physical fitness standards required for military service.  According to DOD officials, advertising is one of several tools, which also  includes recruiters, that the department uses to influence individuals to  consider military service. DOD requested almost $575 million for fiscal  year 2017 for its advertising programs. Advertising is the placement of  messages intended to inform or persuade an audience through various  types of media, such as television, radio, digital media, direct mail, and  others. Sports-related advertising is among the strategies that the military  services have increasingly used to reach their target market for  advertising. Each military service carries out advertising activities to raise  the public’s awareness of the military and help recruiters generate leads  of potential recruits. Public and congressional interest in DOD advertising  increased after 2015 media reports that an Army National Guard unit had  paid a professional football team to conduct ceremonies that recognized  or honored servicemembers, a practice that the media has referred to as  “paid patriotism” as these ceremonies appeared to voluntarily honor  military service but were paid for by the military as part of an advertising  contract.

In our 2003 review of DOD’s advertising activities, we found that DOD did  not have clear program objectives and adequate outcome measures in  place to evaluate the effectiveness of advertising as part of its overall  recruiting effort. We therefore recommended that DOD set such  objectives and develop measures to monitor advertising’s performance  and make choices about advertising funding as part of the overall  recruiting investment. DOD implemented this recommendation and  sponsored research intended to determine how best to evaluate the  effectiveness of military advertising.

Senate Report 114-49 accompanying a bill for the National Defense  Authorization Act for Fiscal Year 2016 included a provision that we  assess DOD’s sponsorships, sports marketing, and advertising activities  for the active duty, reserve, and guard military service components. This  report examines the extent to which DOD (1) has coordinated its  advertising activities among the military service components, (2) has  followed commercial best practices to assess the effectiveness of  advertising activities, and (3) has oversight of the components’  advertising activities.

To determine the extent to which DOD has coordinated its advertising  activities among the military service components, we reviewed  department- and service-level guidance pertaining to roles and  responsibilities for advertising. We interviewed officials from the Office of  the Undersecretary of Defense for Personnel and Readiness, the Joint  Advertising and Marketing Research Studies (JAMRS) office, and each of  the military service components’ advertising programs to discuss any  coordination of advertising activities that occurs. We compared the  guidance and information obtained from these interviews against a best  practice identified by private sector advertising experts, which calls for an  effective use of coordination in advertising to improve efficiencies, and  our duplication, overlap, and fragmentation evaluation and management  guide. This guide describes how to identify and evaluate instances of  fragmentation (more than one agency involved in the same broad area),  overlap (multiple agencies or programs with similar goals, activities, or  beneficiaries), and duplication (two or more agencies or programs  engaged in the same activities or services to the same beneficiaries)  among programs.

To determine the extent to which DOD has followed commercial best  practices to assess the effectiveness of advertising activities, we took  several steps. First, we interviewed officials from the Office of the Under  Secretary of Defense for Personnel and Readiness and each of the  military service components responsible for advertising to determine what  types of advertising they conduct and how its effectiveness is determined.  Second, to identify the best practices for assessing the effectiveness of  advertising in the commercial sector, we conducted a preliminary search,  including a literature review, and determined that a well-defined and  widely accepted list of best practices had not been established.  Therefore, we selected a nongeneralizable sample of advertising  companies and professional organizations using a “snowball sampling  methodology,” which consisted of interviewing advertising industry  experts from an initial set of organizations and requesting those experts  to refer additional contacts whom we then interviewed. During these  interviews, we asked each expert or organization representative to  identify best practices for assessing the effectiveness of advertising. We  then compiled these practices into a list and provided the list to the  organizations for review and comment. Although the perspectives of the  organizations included in our sample are not generalizable to the  advertising industry as a whole, we found a sufficient degree of  consensus among the sample for the purpose of developing criteria for  this review. Third, we reviewed the most recent version of each military  service component’s annual marketing and advertising plan, which  ranged from fiscal year 2014 through fiscal year 2016, depending on the  service component. We compared each plan and associated processes  with the best practices we identified for assessing the effectiveness of  advertising activities and the Standards for Internal Control in the Federal  Government, which calls for an organization to have relevant, reliable,  and timely information and communication of that information throughout  the agency in order to achieve all of its objectives. Lastly, we interviewed  officials from the headquarters of each of the military service components’  advertising programs and representatives from their contracted  advertising agencies, and reviewed each component’s marketing and  advertising plan to determine the extent to which the services followed the  identified best practices.

To determine the extent to which DOD has oversight of the components’  advertising activities, we reviewed federal internal control standards,  which calls for planning, directing, and controlling activities to ensure that  goals and objectives can be achieved. Further, these internal control  standards call for an agency’s activities to be directed through policies.  We compared these standards with (1) any department-level guidance for  advertising or recruiting and (2) advertising activities and service-level  guidance on the review and approval processes related to the services’  and components’ advertising programs. In addition, to determine how the  components provide direction to and oversight of advertising, we  interviewed officials from the headquarters of each of the military services  components’ advertising programs and a judgmental, nonprobability  sample of two lower level recruiting units from each military service  component with advertising responsibilities selected to obtain varying  perspectives. Specifically, we met with units responsible for recruiting and  advertising that represent different regions or areas of the United States,  such as a recruiting district, squadron, or brigade depending on the  service. The subordinate level units were selected to obtain distinct views  within each component, based on variations in the size of budgets,  recruiting goals, or the propensity of youth to serve in the military within  the geographical area of the unit. Lastly, we reviewed contracting data  from fiscal years 2014 through 2015 to identify examples of variations in  the types of activities included in military service component advertising  contracts. Through our discussions with relevant experts and our review  of past work, we determined the data to be sufficiently reliable for the  purpose of corroborating the types of activities included in advertising  contracts.

We conducted this performance audit from June 2015 to May 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. We provide additional  information about our scope and methodology in appendix I.
subsections: section_title: Background
subsections: section_title: Roles and Responsibilities for DOD Advertising Programs
paragraphs: Under Title 10 of the United States Code, the Secretary of each Military  Department (the Army, the Navy, and the Air Force) has the responsibility  to recruit personnel, subject to the authority, direction, and control of the  Secretary of Defense. As such, each Secretary has the authority to  organize and delegate responsibility for advertising efforts within each  military service or components within each military service and, as a  result, the organizational structure of advertising programs and  associated recruiting organizational structure for each of the military  departments differs.

While advertising is carried out by the military services, some roles and  responsibilities for advertising reside with the Office of the Under  Secretary of Defense for Personnel and Readiness. Within that office, the  Accessions Policy office within Manpower and Reserve Affairs has  responsibility for the (1) policy, planning, and program review of active  and reserve personnel, procurement, and processing and (2)  development, review, and analysis of policies, resource levels, and  budgets for recruiting enlisted personnel and for officer commissioning  programs. One function for recruiting enlisted personnel and officer  commissioning programs is advertising. Further, JAMRS reports to the  Director of DOD’s Defense Manpower and Data Center and is  responsible for joint marketing communications and market research and  studies. JAMRS is to conduct research about the perceptions, beliefs,  and attitudes of American youth as they relate to joining the military,  which is often referred to as the propensity to serve. JAMRS officials  stated that understanding these factors helps ensure recruiting efforts are  directed in the most efficient and beneficial manner. JAMRS has in the  past carried out joint advertising aimed at “influencers,” or the adults that  can influence or impact a potential recruit’s decisions regarding military  service. JAMRS also maintains a database that is used across the  military services to begin their outreach to potential recruits.

section_title: Types of Advertising Conducted by the Military Service Components
paragraphs: The military service components conduct advertising in support of their  recruitment missions. Consistent with the private sector, the components’  advertising programs follow a strategy that considers the phases of an  individual’s decision-making process, sometimes referred to as the  consumer journey. The decision to enlist in the military is a significant  commitment and can be affected by numerous factors, such as other  employment or educational opportunities available to an individual who is  considering a career in the military. According to military service officials,  these phases are generally characterized by the components as  awareness, engagement, and lead generation, as illustrated in figure 1.  The goal of military advertising is to move a potential recruit through each  phase and, ultimately, to a decision to enlist. Further, each military  service conducts advertising throughout each phase that according to  military service officials is intended to communicate and reinforce a  certain brand or image among potential recruits, leading potential recruits  to determine that a particular military service is the best fit for their  individual interests, beliefs, or goals.

Awareness. The military service components conduct general  awareness advertising to inform members of an audience about the  opportunity to serve in the military and the distinct characteristics of  each military service. The components typically pursue awareness  through traditional advertising formats such as television commercials,  print advertisements, and banners at events or signs within a  community.

Engagement. Advertising focused on building engagement targets  individuals who are aware of the military as a career option and have  begun to consider the possibility of enlisting. During this phase, the  components seek to provide recruits with additional information to aid  in their decision-making process. Often this phase of advertising takes  place in the digital environment, as components seek to provide  informative social media posts and use banner advertisements to  attract individuals to visit their websites for more information. Figure 2  describes a variety of digital advertising activities conducted by the  components.

Lead Generation. Lead generation advertising targets individuals  who have considered military service and are ready to discuss the  possibility of enlistment. As such, lead generation activities seek to  encourage these individuals to provide their contact information in  order to schedule an opportunity to meet with a recruiter. Lead  generation is often conducted in person, such as through recruiters’  presence at events like career fairs or sports games. Lead generation  may also be conducted through other means—such as direct mail and  online or print classified advertisements—as long as the  advertisement features a “call to action” intended to prompt viewers to  provide their contact information. Further, the military services often  employ the use of “mobile assets,” such as large trucks and trailers  fitted with equipment and activities intended to draw crowds and  encourage and facilitate public interaction with a recruiter at an event  in order to generate leads.

Figure 3 shows some examples of various types of military service  advertising used for recruiting purposes, such as mobile assets used at  recruiting events to advertise a specific military service, digital advertising  on social media, and print brochures.

section_title: DOD Has Coordinated Some Advertising Activities among Its Components, but Has Not Developed a Formal Process for Coordination
subsections: section_title: DOD Has Taken Steps to Coordinate Some Advertising Activities
paragraphs: DOD has taken steps to coordinate some advertising activities among the  military service components. Within the military departments, there are  seven military service component advertising programs (see app. II for  more details) that compete to attract recruits from a relatively small pool  of individuals that are eligible for military service. Private sector  advertising industry experts we spoke to emphasized the importance of  maintaining a unique brand and strategy for each of the service  components when there is competition for a target audience. According to  DOD officials, each component works to develop a unique brand that  differentiates the military services in order to compete for potential  recruits. While industry experts stated that competition is inherent to  advertising, these experts also stated that coordination can sometimes be  beneficial to increase efficiencies and effectiveness, and that DOD could  pursue greater coordination in some instances to help address any  inefficiencies.

Despite the competition that exists among the military components’  advertising programs, DOD has coordinated certain advertising activities.  For example, DOD established the JAMRS office in 2002 to create a  centralized program for joint market research and communication.  JAMRS provides the military components with information that informs  their advertising programs from surveys of U.S. youth attitudes toward  joining the military, which change and evolve considerably over time.  Officials across the military components stated that they heavily relied on  the market research conducted by JAMRS and reported using the  information to tailor their advertising and recruiting activities to address  the interests of U.S. youth.

JAMRS used to provide joint advertising campaigns directed at adults  who might influence an individual’s decision to join the military, such as  parents or coaches, known as influencers. According to JAMRS officials,  joint advertising campaigns of any type have not been carried out in  recent years due to budget constraints. Military service component  officials stated that JAMRS’ joint advertising campaigns had been  important for building awareness among influencers and assisting in  promoting a positive image of military service to the U.S. public. Further,  these officials stated that joint advertising for influencers provided by  JAMRS was beneficial for all the military services and that due to the  resources needed to conduct service-specific advertising, it is not feasible  for their components to conduct additional advertising focused on  influencers.

Additionally, the Office for Accessions Policy within the Under Secretary  of Defense for Personnel and Readiness has taken specific steps to  coordinate among the components to attempt to increase the  effectiveness of DOD’s advertising or to address shared challenges in  advertising that are crosscutting in nature and can affect all of the military  service components. For example, the Office coordinated a response to  media and congressional interest in recent activities conducted as part of  sports advertising that were questioned as being inappropriate. Further,  the Under Secretary responded to this crosscutting challenge in  September 2015 by issuing interim guidance for sports marketing that  applied to all service components to prevent inappropriate activities from  occurring as part of sports advertising. In addition, DOD officials stated  that, in an effort to improve the effectiveness of DOD’s advertising, in  2008 the Office for Accessions Policy convened a cross-service working  group of military service component advertising officials to develop a set  of consistent performance measures that each military service component  could use to assess performance. However, DOD officials stated that  consensus was not reached, and thus the measures were not developed.

Officials from the military service components stated that they meet about  quarterly with their counterparts at meetings held by JAMRS to obtain  their respective results from joint market research conducted by JAMRS,  and at these meetings they will on occasion discuss crosscutting issues  that affect the services’ advertising programs. Military service component  advertising officials stated that these discussions have at times provided  an opportunity to engage in discussions on effective advertising practices  and lessons learned across service programs. Further, military service  officials stated that they have long-standing working relationships with  their counterparts among the services and that they do on occasion share  some information during these discussions. However, the officials stated  that they may not share comprehensive details of best practices or  lessons learned because of the competition for recruits among the  components and because they are not required to share information or  coordinate.

section_title: DOD Does Not Have a Formal Process for Coordination among the Services, Which Can Result in Possible Unnecessary Duplication, Overlap, and Fragmentation
paragraphs: The private sector executives and professional association  representatives we spoke with stated that while competition is an inherent  aspect of advertising, increasing coordination should be considered in  order to reduce inefficiencies and leverage resources effectively. Further,  we have found in prior work that mechanisms to coordinate programs that  address crosscutting issues may reduce the instances of potentially  duplicative, overlapping, and fragmented efforts. While DOD has taken  some steps to coordinate, the department has not established a formal  process to ensure that service component advertising officials have a  forum to share information and address crosscutting issues  systematically, and that these discussions are consistently occurring, to  help ensure an efficient use of resources in the competitive environment  for recruitment.

In the absence of a formal process for coordination, we found examples  of possible unnecessary duplication, overlap, and fragmentation within  DOD’s advertising activities.

Risk of Increased Cost of Advertising Media Purchases. Industry  experts and various military service officials stated that as each  military service component purchases advertising space in the same  media market, such as the time to air an advertisement on radio or  television, competition among the services could increase the prices  of these purchases. Further, they stated that working together could  increase the buying power for certain activities. As such, from a  department-wide perspective, DOD is at risk of making duplicative  purchases in the same market with no coordination regarding the cost  and effectiveness of those purchases. Officials from one military  service component acknowledged that the competition for media  purchases most likely results in a more expensive manner of doing  business for the entire military.

Multiple Contracts for Similar Functions. While each service  component’s advertising agency works with the component to develop  a unique brand that differentiates the military services, the  components each contract for some functions that are not brand  specific. For example, service officials reported that each component  has advertising contracts that include services such as call centers  and website chat-function support that respond to requests for  information generated by advertising, and that screen potential  recruits against general criteria for military service. As such, there are  at least seven call center functions that the department uses to screen  potential recruits. Most of these call centers are staffed by contractors  from the general public and do not provide a service-specific function.  Thus the department is paying several different companies to provide  a similar service. Advertising industry experts pointed to functions that  are not brand specific as possible opportunities for obtaining  efficiencies through coordination or consolidation. However, military  service officials disagreed with this assessment and stated that  benefits that might result from consolidation of these types of  functions are unclear and that call centers may provide brand-specific  information. For example, officials from the Army National Guard  stated that trained individuals, sometimes Army National Guard  members, can offer state- or territory guard- specific information when  responding to calls from their contracted call center.

As discussed below, we also found examples where better coordination  within and among the military service components could increase  efficiencies or effectiveness by addressing the fragmentation of these  advertising programs.

Three Advertising Programs within the Air Force. The Air Force  has three components that each contract with a different advertising  agency to develop and implement three separate advertising  programs. Maintaining three separate programs can lead to  inefficiencies. For example, officials from the Air Force reserve and  guard components stated that they do not have the resources to do  marketing mix modeling. Marketing mix modeling is a best practice  employed by agencies to determine the most efficient and effective  allocation of a client’s budget toward media buys including print,  television, and digital advertising. However, if the Air Force  components coordinated on similar advertising functions that are not  part of their unique branding, such as marketing mix modeling, they  could potentially afford to jointly contract for such functions. In  response to our questions about why they had not coordinated these  programs, officials from the components stated that both the reserve  component and Air National Guard need to focus on the geographic  location of potential recruits to fill vacancies, whereas this is not a  concern for the active duty components. However, they further stated  that there had been discussion of consolidating some components in  the past, possibly the guard and reserve components, but that this  was not pursued. Officials could not provide any further rationale for  requiring separate programs or for why further efforts for consolidation  were not pursued.

Two Army Advertising Programs. There are two Army advertising  programs, one for the Army active duty and reserve components and  another for the Army National Guard. Better coordination between  these two programs could result in more effective use of resources for  a common purpose. Army National Guard officials stated that they do  not have the resources to fund some needed services that the active  Army could support. For example, Army National Guard officials  stated that they do not procure warehouse space for the storage of  advertising materials for guard units throughout the country or mobile  assets to deploy for National Guard campaigns, which the active Army  could potentially support. Army National Guard officials stated regular  coordination does not exist between these two programs and to  address these shortcomings they have begun to engage in  discussions with the active Army component to further explore  coordination for these activities. A January 2016 report from the  National Commission on the Future of the Army stated that separation  of the Army’s recruiting programs and associated advertising is  inefficient and unproductive. The report recommended the  establishment of pilot programs that align the recruiting efforts of the  active duty Army, Army National Guard, and Army Reserve as  consolidating the administration and budgeting of recruiting,  advertising, and branding for all components will yield increased  effectiveness and efficiency.

Army National and Local Advertising. We also found fragmentation  in the Army’s active and reserve advertising programs resulting in  coordination and communication challenges. Within the active and  reserve Army components’ advertising program, the Army has two  organizations that share some degree of responsibility for carrying out  advertising activities: its Army Marketing and Research Group  advertising office which directs the Army’s national level advertising  and U.S. Army Recruiting Command which directs local advertising  for recruiting. Senior officials from both organizations stated their  organization is responsible for lead generation, and recruiters  generate leads at both national and local events.

While we acknowledge the importance of maintaining a unique brand for  the service components, a formal process for coordination could allow the  military service components to more effectively share best practices. As  part of this process, a review of DOD’s existing programs could potentially  identify opportunities to obtain efficiencies by reducing unnecessary  duplication, overlap, and fragmentation that may exist within and among  military service components’ advertising programs. As numerous  department officials cited a lack of needed resources to appropriately  carry out advertising in order to best execute the difficult task of recruiting,  identifying and reducing unnecessary duplication, overlap, and  fragmentation could potentially free up additional resources. In the  absence of a formal process for coordination of the department’s several  advertising programs, DOD may not be positioned to best leverage its  advertising funds.

section_title: DOD Has Generally Followed Commercial Best Practices for Assessing the Effectiveness of Advertising, but Components Vary in Their Ability to Determine Whether Their Activities Are Generating Recruitment Leads
paragraphs: With some exceptions, DOD—through its military service components  and together with their contracted advertising agencies—generally follows  commercial best practices that we identified for assessing the  effectiveness of advertising, shown in table 1. We determined that a well- defined and widely accepted list of best practices had not been  established; therefore, we asked a nongeneralizable sample of  advertising experts and professional associations to identify best  practices. We compiled and condensed the identified best practices into a  list that we then validated with those same industry experts. However,  when we compared the military service components advertising programs  against the best practices that we identified, we found variations among  the components in establishing measurable goals. Further, differences  across the components in their processes for collecting and reviewing  performance data have resulted in varying abilities to measure the  effectiveness of advertising on generating leads, especially at the local  level.
subsections: section_title: Components Have Structured Their Advertising Programs’ Organization to Safeguard against Bias in Performance Measurement and Coordinate Vendors
paragraphs: We found that the military service components have largely established  roles and responsibilities to address the need for unbiased performance  information, and ensured that coordinator roles were assigned when  multiple vendors were used. Industry experts we spoke with stated that  advertising decisionmakers should consider whether to distribute  responsibilities for various aspects of advertising—including creative  development, media buying, and performance analysis—across different  vendors, but when doing so, should ensure that a coordinator role is  assigned to maintain a cohesive advertising strategy. While the majority  of the military service components rely heavily on their contracted  advertising agencies to assess the effectiveness of their advertising  activities, they also adhered to these best practices, as described in the  examples below.

Consistent with this commercial best practice that we identified, for  example, we found the following:

A Marine Corps official responsible for overseeing the component’s  advertising program stated that four Marine Corps project officers are  designated to work closely with its advertising agency in monitoring  and analyzing performance across key areas.

Officials from the Army, which has the largest advertising budget of  the components, stated that the Army had additionally contracted with  third-party research firms to further assess the effectiveness of its  advertising.

All of the components independently review quantitative performance  data, which they may access through their own data systems or  reports provided by the advertising agencies. Military service officials  stated that monitoring such data can help them to assess the  performance of both their advertising activities and contracted  advertising agencies.

Each of the components also cited research conducted by DOD’s  JAMRS program as a source of unbiased performance data. For  example, JAMRS conducts a quarterly advertising tracking survey,  evaluating the target audience’s recall and reactions to the  components’ television advertising campaigns, which service officials  stated can help the components assess the effectiveness of that  specific type of advertising.

As each component contracts with a lead advertising agency, the  components typically adhere to this best practice as the agency fulfils  the coordinator role in conjunction with the component.

section_title: Components Vary in the Extent to Which They Meet Commercial Best Practices on Planning That We Identified
paragraphs: While all of the military service components develop evaluation  frameworks that identify a target audience in accordance with the  commercial best practice, the goals set by the components vary in  measurability. In our prior work, we have also found that high-performing  organizations have goals that are aligned with performance  management. We found that almost all of the service components  typically develop an annual advertising plan, including information about  their target market and goals to be met by advertising that year. All of  the plans we reviewed demonstrated a detailed understanding of the  target market. For example, the Marine Corps’ fiscal year 2015 plan  recognizes the varying levels of the propensity to join the military among  its target market and identifies an opportunity to focus on “the movable  middle”—the portion of prospective recruits who are not currently inclined  to join the military but might be willing to consider it. Likewise, reflecting  its organizational structure of 54 state and territorial units, the Air National  Guard’s 2015 annual advertising plan emphasizes targeting advertising to  audiences located near units where career opportunities are available.

However, the commercial best practice on planning that we identified also  calls for advertising plans to include measurable goals. In our review of  the most recent version of each component’s annual advertising plan, we  found that the components’ plans varied in the extent to which their goals  can be measured. For example, the Army plan we reviewed identifies a  series of marketing objectives supported by measurable, numeric goals  related to the public perception of the Army. In contrast, the Marine Corps  and active duty Air Force plans state goals that relate to emphasizing  positive aspects of the components, such as diversity and core values,  but these goals are not numeric and measurable with related  performance measures. For example, the most recent annual advertising  plan of the Marine Corps includes a goal to weave diversity into all  advertising efforts, while the Air Force annual plan contains a goal that all  advertising tells the Air Force a story in a way that highlights Air Force  core values. Neither plan includes information on how these goals would  be measured. We also found that the annual plans of the Air Force  National Guard and Air Force reserve had a mixture of the types of goals  identified, some of which could be measured. In addition, Army National  Guard officials stated that while it requires each state unit to create an  annual plan that includes goals, the Army National Guard headquarters  does not establish annual goals at a national level. Army National Guard  officials stated that there is not a national level annual advertising plan  with associated goals because each state unit has a unique recruiting  mission, including the types of positions that need to be filled and the  demographic makeup of the target audience. The absence of measurable  goals at the national level may limit Army National Guard headquarters’  ability to determine the success of any national level advertising efforts or  to distribute advertising funds strategically and efficiently among the state  units. Military service officials stated that the annual advertising  campaigns and associated advertising plans for each component can  change considerably from year to year. Ensuring that future iterations of  each component’s annual advertising plan contain measurable goals  could enhance their ability to demonstrate the success of their advertising  programs.

The components also vary in their use of sophisticated modeling to  determine how to distribute available advertising funds across different  types of advertising (e.g., television, print, Internet, etc.), which was a key  planning best practice cited by industry experts. Specifically, industry  experts stated it is a best practice to use some form of modeling, such as  marketing mix modeling, to determine the optimum distribution of  advertising funds. Officials from the active duty Army, Marine Corps, and  Air Force active duty components stated that they use such modeling,  provided by their contracted advertising agencies, to determine how to  spend their advertising funds. However, the remaining components do not  currently leverage marketing mix modeling, and Air Force Reserve and  Army National Guard officials cited its high cost as a barrier. Industry  experts acknowledged that, while marketing mix modeling is a best  practice, it can present a significant expense, and other methods of  modeling may be more appropriate in some situations. As such, Army  National Guard officials stated that marketing mix modeling might not be  best suited for modeling when mission requirements fluctuate year to year  or when geographic location of vacancies are of a primary concern.

section_title: Components Have Taken Some Steps to Measure Performance, but Some Have Insufficient Data to Measure Whether Outcomes Are Attributable to Advertising
subsections: section_title: Components Successfully Applied Industry Standard Measures for Two of the Three Purposes of Advertising Activities
paragraphs: We found that the military service components successfully applied  industry standard measures appropriate for two of the purposes of  advertising activities—awareness and engagement— but varied in their  ability to assess the effectiveness of advertising related to the third  activity, lead generation. For example, to measure effectiveness of  awareness related advertising, such as television commercials or national  print advertisements, the components typically measure the number of  times an advertisement is viewed by a member of the audience, numbers  known as impressions. To measure engagement, the components use a  range of real-time digital analytics, including click-through rates and  social media “likes,” among others, when they conduct digital advertising.  According to military officials and representatives of their advertising  agencies, it is an industry standard to directly link the results  demonstrated by these analytics to the purchase of advertising for  awareness and engagement. The components rely on their advertising  agencies to negotiate these purchases with the intent of achieving a level  of performance that is consistent with industry standards for these forms  of advertising.

In contrast, we found that the components varied in their ability to use  industry standard performance measures to assess the effectiveness of  advertising activities focused on lead generation. Service officials stated  that such activities can include recruiter booths at events, direct mail, and  certain types of print or digital advertising, for which performance is  measured on the basis of how many leads are generated by the activity.  Whereas military service officials stated that advertising focused on  generating awareness and engagement is generally executed at the  national level, lead generation activities are often carried out at the local  level and can depend on recruiters’ familiarity and knowledge of their  local markets. For example, service officials stated that recruiters may  select high school populations to receive direct mailings or interact with  potential recruits at local events such as career fairs or sports games and  may also be responsible for identifying the number of leads generated by  such activities. As a result, the responsibility for measuring the  performance of locally executed lead generation advertising activities is  carried out, in part, at the local level, and some components do not obtain  or measure the performance data needed to assess the effectiveness of  these activities.

section_title: Shortcomings in Measuring Lead Generation Contribute to Difficulties in Understanding How Outcomes Can Be Attributed to Advertising
paragraphs: Insufficient data to measure the performance of local lead generating  activities has diminished some of the components’ ability to understand  how to attribute outcomes to specific advertising activities. According to  industry experts and our prior work, determining the precise impact of  advertising on outcomes, such as recruitment, is inherently challenging, in  part due to the concurrent effects of external factors, such as the  influence of family support and the availability of other career or  educational opportunities. In addition, the length and complexity of a  decision to enlist in the military necessitates the use of multiple types of  advertising throughout the recruiting process, and an individual may  ultimately decide to enlist months or years after first being exposed to  advertising by the military. Although industry experts acknowledged that  understanding how outcomes are impacted by advertising is a challenge  in both the private sector in general and for the military in particular, they  stated it is nonetheless important for advertising managers to develop an  understanding of how outcomes can be attributed to advertising.

With the exception of certain new requirements specific to sports  advertising contracts, DOD does not require that the service components  measure the performance of their advertising activities, and the extent to  which the components can currently measure the effectiveness of their  advertising activities varies. As such, measurement and monitoring  performance is at the discretion of the military service components, and  there is variation in the components’ processes for collecting and  reviewing advertising activity performance data. According to service  officials, we found that some components either do not collect local  advertising performance data related to lead generation in their systems  or cited issues with the reliability of such data. Specifically, in terms of  collecting this type of performance data, service component officials  stated the following:

The Marine Corps’ leads tracking system allows the Marine Corps to  link multiple exposures to advertising—such as a direct mailing or  interaction with a recruiter at an event—to a prospective recruit’s lead  record, enhancing the ability to analyze performance in lead  generation.

While the Navy requires that performance data for local advertising be  entered into its data systems, currently it does not have the capability  to analyze performance in the same manner and cannot attribute  potential leads to multiple exposures to advertising.

Air Force active duty recruiting squadrons are not required to report to  their headquarters on the performance of their advertising activities,  including the performance of local advertising activities in lead  generation. However, Air Force officials stated they conduct and  assess the performance of the majority of their advertising at the  headquarters level, and a comparatively small portion of advertising  funds are distributed to the squadrons for local advertising activities.

Both the Air Force Reserve and Air National Guard require that  recruiters submit performance data, such as attendance and leads  generated for any local advertising activity and that headquarters  officials review these data in their lead tracking data systems.

The Army National Guard does not routinely require state units to  provide headquarters with performance data related to advertising,  including lead generation.

The Army requires in its policy that subheadquarters units submit  advertising performance data for headquarters review, but Army  advertising officials cited concerns with the reliability of these data for  lead generation.

Regarding the Army’s concerns regarding unreliable data, Army officials  stated that when leads are collected by recruiters at advertising events, in  many cases those leads are coded in their data system as “recruiter  generated,” rather than being attributed to the appropriate advertising  activity. As a result, Army officials stated that although they believe their  locally executed advertising activities are a good investment, they do not  have sufficient evidence to demonstrate their effectiveness. For example,  while Army Recruiting Command officials cited “register to win” giveaways  for promotional items at local events as an effective lead generator, Army  Marketing and Research Group officials stated they do not have data to  support the effectiveness of this advertising activity and questioned the  quality of the leads produced. Army officials stated that they are currently  working to address this issue to ensure leads are properly coded and thus  improve the reliability of this data. We did not find the proper coding of  lead generation to be an issue for other service components; for example,  according to service officials, recruiters from the Marine Corps and active  Air Force that attend an event typically record a potential lead’s  information into an electronic tablet that automatically assigns the proper  event code for the lead based on the recruiter’s location and the date and  time of the lead’s entry.

Federal internal controls standards state that program managers need  appropriate data to determine whether they are meeting stated goals and  achieving an effective and efficient use of resources. Without access to  the necessary performance data, such as reliable leads collection and  attribution data, a military service component may be limited in its ability  to measure the performance of its advertising activities against stated  goals. Without processes in place to facilitate the measurement and  monitoring of advertising performance across all levels—especially at the  local level—the military service components may be unable to ensure  advertising dollars are used efficiently to help meet stated recruiting goals  could result in inefficient use of advertising dollars.

section_title: In the Absence of Policy, DOD Does Not Have Comprehensive Oversight of Its Components’ Advertising Activities
paragraphs: DOD does not have comprehensive oversight of the military service  components’ advertising activities, as it does not have a policy that  defines its oversight role as well as procedures to guide the components’  respective advertising activities. Federal internal control standards for an  agency’s organizational structure call for planning, directing, and  controlling activities to ensure goals and objectives can be achieved.  Further, these standards require that an agency’s activities be directed by  policies and that management should provide appropriate oversight of  activities. When DOD issues policy, such as directives and instructions,  the department requires that the policy establishes roles and  responsibilities and defines the procedures that are to be followed by all  defense offices and organizations that are involved with the activity or  program the policy directs.
subsections: section_title: DOD’s Components Oversee Their Own Programs, Which May Have Led to Negative Effects in Some Instances
paragraphs: As there is no department-wide policy that defines DOD’s role in  overseeing advertising activities or the procedures that should be  followed when the components carry out their advertising activities, the  department’s advertising activities are overseen at the military service  level and in some cases, within a service’s individual active, reserve, or  guard components. Each military service component has component- level policy and guidance that defines the overall objectives of its  advertising program and sets forth the roles and responsibilities for the  component’s advertising program. The high-level policy is in many  cases supplemented by yearly guidance that communicates the recruiting  goals and the priorities of the commanding officer or senior leader  responsible for a component’s advertising. However, the high-level  policies issued by each service component for its advertising program  vary considerably in the level of oversight and direction provided, as the  procedures the components are to follow as they carry out their  respective advertising activities have not been defined by DOD policy. For  example, the Navy’s guidance includes financial thresholds for  headquarters’ level review of certain expenditures and the types of cost- effectiveness reviews that must be performed, while other components’  guidance does not address the review of expenditures. The Marine Corps  guidance specifies some advertising activities that are prohibited,  whereas other components’ guidance—including the Air Force—does not  include this type of detail.

The variation in the oversight and direction provided to each service  component’s advertising program has allowed for an inconsistent  understanding of digital advertising rules and regulations and for  negatively perceived activities to occur in some instances.

Digital Advertising. We found differences in the understanding of  rules and regulations that apply to digital advertising activities. The  government is restricted in tracking the digital behavior of those that  visit government websites. However, we observed differing opinions  and understanding among military service officials regarding what  types of digital tracking was permissible. Officials from the Air Force  active duty component’s advertising program and representatives  from their contracted advertising agency stated that persistent  tracking would be used in their new website development and that  these plans adhere to relevant DOD guidance. Further, they stated  that subsequent advertising could be sent to individuals after visiting  the Air Force website. Officials from the Office of the Under Secretary  of Defense for Personnel and Readiness stated that they had  reviewed the plans of the Air Force regarding digital tracking and  subsequent advertising and agreed that these actions would be  permissible and in accordance with DOD guidance. However, officials  from other military services we spoke with described these same  digital advertising strategies as not allowed per DOD guidance and  stated that they could not pursue digital tracking similar to that of the  private sector. The other services could be at a disadvantage as a  result of these different interpretations of regulations.

Sports-Related Advertising. While contracting with sports teams  and events for advertising is widely practiced among the military  service components, media reports from 2015 and congressional  attention revealed that some components’ contracts with professional  sports teams included provisions to conduct ceremonies that honored  servicemembers and provide items such as tickets to games, which  were perceived to be inappropriate and came at a cost to the federal  government. In our review of contracts from fiscal years 2013 and  2014, we found several contracts with a marketing firm that connects  brands to collegiate sports teams that included costs for honorary or  swearing in ceremonies and items that are personal in nature. For  example, we identified a contract spanning fiscal years 2013 and  2014 in which a major public university received $20,000 from a state  Army National Guard unit for two swearing in ceremonies of recruits  conducted at sporting events, and another state Army National Guard  unit contracted in fiscal year 2014 for a “VIP experience” at university  sporting events costing approximately $8,700.

section_title: DOD Has Taken Some Steps to Improve Oversight but Does Not Have a Department-Wide Policy to Guide the Components’ Advertising Programs
paragraphs: Subsequent to the negative media reports in 2015 and resulting DOD  reviews in September 2015, the acting Under Secretary for Personnel  and Readiness issued interim guidance that provided more focused  direction and strengthened oversight of sports marketing and advertising  contracts. The interim guidance acknowledged the inappropriateness of  paying for recognition or swearing in ceremonies during sports events or  the inclusion of items that are personal in nature, typically sports tickets or  parking, for which the receipt of those items is not clear or controlled.  Specifically, this guidance (1) requires that a senior military component  reviewing official approve sports marketing contracts, (2) prohibits paying  for recognition ceremonies, (3) restricts items that are personal in nature,  and (4) requires reporting and analysis on the returns generated by larger  sports partnerships.

While the issuance of the interim DOD-level sports marketing guidance is  a positive step, a senior DOD official acknowledged that the interim  guidance was vague regarding how some of these new requirements are  to be implemented, did not include some monetary thresholds that it could  have identified, and applied only to sports marketing and advertising  contracts. As such, in our discussions with service officials responsible for  carrying out local level advertising, we observed discrepancies in how the  guidance was being interpreted. For example, there were differences in  opinions among local level advertising officials about the inclusion of  honorary or swearing in ceremonies in contracts with sports teams. One  local level official stated that the recent interim guidance now restricts any  mention of an honorary ceremony in a contract, while other officials stated  that such activities could still be included in a contract as long as they  were listed with a cost of $0.00 and considered “added value.” Further, as  the interim guidance only applies to sports advertising, it does not  address the procedures that the components should follow when carrying  out other types of advertising. For example, the components may contract  to carry out advertising at music concerts or festivals. These events are  similar in nature to sporting events as they may also allow for the  inclusion of tickets other premium items in advertising contracts, but  currently there is no DOD-wide advertising guidance that prevents the  inclusion of such items in contracts.

DOD officials stated that they are currently in the beginning stages of  developing guidance for advertising that is more comprehensive to  replace and build upon the interim guidance that was focused solely on  sports-related advertising, but the details of what this proposed expanded  guidance will cover are not clear. These officials stated that the  department’s process for development, coordination, approval, and  publication of new DOD guidance can be a lengthy one. Further, they  stated that they planned to review the implementation of the interim  guidance and incorporate any lessons learned from the issuance of the  September 2015 guidance, as well as incorporate the findings of our  report before moving forward to create a DOD-wide policy. Because DOD  officials were not able to provide us a draft of this proposed expansion of  guidance, and they stated that they are in the earliest stage of a lengthy  process, it is unclear when DOD will issue department-wide guidance for  advertising. Further, it is unclear whether the guidance will clearly define  DOD’s role in oversight, clarify all remaining issues related to sports  advertising, and provide direction to other types of advertising such as  digital advertising and concerts or other types of event advertising.  Without a policy that clearly defines DOD’s role in oversight of the  advertising activities of the military service components and outlines the  procedures the components should follow for all types of advertising  activities, DOD may not be able to ensure that the components are  carrying out other types of advertising in a manner that it considers  appropriate. More broadly, while it may be appropriate for the service  components to have variations in advertising policies given that they  operate separate programs in some instances and we are encouraged  that DOD officials stated that they will address current weaknesses such  as those we identify in this report, without a policy that defines procedures  for all types of advertising activities, DOD risks abuses or inappropriate  activities in other advertising in the future.

section_title: Conclusions
paragraphs: Advertising activities provide information and seek to influence the beliefs  and understanding of potential recruits about each military service, and  the services conduct advertising to help meet their recruitment goals. The  unique branding of each service plays a role in the decision of an  individual to become a soldier, sailor, airman, or marine and, as such, the  department relies on the military services to carry out their own  advertising programs. However, despite the competition for potential  recruits, a formal process for coordination of advertising activities among  the military service components could improve the department’s ability to  leverage resources and thus improve the efficiency of DOD’s advertising  activities. Further, while the components generally follow commercial best  practices we identified for evaluating advertising, DOD has not addressed  variations in measurable goals among the components or insufficient data  that has prevented some components from being able to assess the  effectiveness of their advertising activities in generating leads. Lastly, the  absence of a department-wide policy that clearly defines DOD oversight  of and procedures to guide the advertising activities has allowed for  activities of questionable appropriateness to occur in some instances.  Without such a policy, DOD cannot ensure—through comprehensive  oversight—that each service or component is carrying out advertising that  meets departmental standards and that appropriately invests taxpayer  dollars.

section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense take the following three  actions:

Direct the Under Secretary of Defense for Personnel and Readiness, in consultation with officials from the military service components and the JAMRS office, to develop a formal process for coordination on crosscutting issues to facilitate better leveraging of resources. As part of this process, DOD could review existing advertising programs to identify opportunities to reduce unnecessary duplication, overlap, and fragmentation and obtain potential efficiencies.

Direct the Secretaries of the Departments of the Army, the Navy, and the Air Force to ensure that each military service component fully measure advertising performance. This should include both the identification of measurable goals in future versions of the service components’ advertising plans and assurance that the service components have access to the necessary performance data to determine the effectiveness of their advertising activities for lead generation activities.

Direct the Under Secretary of Defense for Personnel and Readiness  to ensure, as the department undertakes its effort to issue a  department-wide policy for advertising, that this policy (1) clearly  defines DOD’s role in overseeing the advertising activities of military  service components; (2) clarifies issues related to sports-related  advertising; and (3) outlines procedures that should guide the  components’ advertising activities for other types of advertising, such  as music concerts, other event advertising, and digital advertising.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD for comment. In written  comments, DOD generally concurred with our three recommendations.  Specifically, DOD concurred with our recommendations aimed at  improving coordination and providing more direction to military service  advertising. DOD partially concurred with our recommendation regarding  the need for better performance measurement. DOD stated in its written  comments that it will work with the military services to develop guidance  that addresses our recommendations, which will be provided in the form  of a DOD issuance. DOD’s comments are reprinted in their entirety in  appendix IV. DOD also provided technical comments, which we  incorporated into the report as appropriate.

DOD concurred with our first recommendation that the department  develop a formal process for coordination on crosscutting issues to  facilitate better leveraging of resources. In its written comments, DOD  stated that it is developing a DOD instruction for marketing and that this  guidance will formalize coordination among the military services, which it  states should facilitate better leveraging of resources.

DOD partially concurred with our second recommendation that the  military departments fully measure advertising performance. We  recommended that this performance measurement should include both  the identification of measurable goals in future versions of the service  components’ advertising plans and assurance that the service  components have access to the necessary performance data to  determine the effectiveness of their advertising activities for lead  generation activities. In its written comments, DOD stated that it agrees  with our recommendation in broad terms. DOD highlighted actions  already underway, and stated that as part of the development of its  instruction for marketing, it will further clarify and codify guidance related  to performance measurement. However, DOD further states that not all  goals and measures relate to lead generation and that other goals and  objectives can be used to measure success. We acknowledge in this  report that the goal of some forms of advertising is to improve awareness  or engagement, and not solely lead generation. Therefore, we do not  believe that all measurement and performance data should be tied back  to lead generation if that is not the goal of the advertising activity in  question. However, as we state in our report, we found that not all military  services collect the necessary performance data to determine if activities  intended to generate leads are performing as intended, and DOD did not  address this issue of data collection in its written comments. We believe  that DOD and GAO largely agree on this issue and are encouraged that  DOD plans to issue an instruction that will clarify the need for better  performance measurement of advertising activities. While its September  2015 interim guidance was an important step that required assessment of  performance of sports related advertising, we reiterate that the draft DOD  instruction in development should clarify that performance measurement  is important for all types of advertising, not only sports related advertising  activities, and that the appropriate goals, performance data, and  performance measures be used to assess the performance of all  advertising activities.

DOD concurred with our third recommendation that the department issue  policy for advertising. In its written comments, DOD stated that it is  developing a DOD instruction for marketing and that this instruction will  clearly define DOD’s role in overseeing the advertising activities of  military service components; clarify issues related to sports-related  advertising; and outline procedures that should guide the components’  advertising activities for other types of advertising.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense; the Under Secretary of Defense  for Personnel and Readiness; the Director of JAMRS; and the Secretaries  of the Army, the Navy, and the Air Force. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  Andrew Von Ah at (213) 830-1011 or vonaha@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix V.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of our review were to examine the extent to which the  Department of Defense (DOD) (1) has coordinated its advertising  activities among the military service components, (2) has followed  commercial best practices to assess the effectiveness of advertising  activities, and (3) has oversight of the components ‘advertising activities.

To determine the extent to which DOD has coordinated its advertising  activities among the military service components, we reviewed  department- and service-level guidance pertaining to roles and  responsibilities for advertising. We interviewed officials from the  Accessions Policy office within the Office of the Under Secretary of  Defense for Personnel and Readiness, the Joint Advertising and  Marketing Research Service (JAMRS), and each of the military service  components’ advertising programs to discuss the ways in which DOD has  taken steps to coordinate advertising activities as well as ways in which  the military service components coordinate with each other to achieve  efficiencies in advertising. Further, we discussed any methods used to  share best practices or lessons learned for advertising, either formally or  informally, among the military service components. We also conducted  interviews in Fort Knox, Kentucky, with officials from the U.S. Army  Recruiting Command, the U.S. Army Cadet Command, and the U.S.  Army Accessions Support Brigade, organizations that play a role in  advertising for the active and reserve Army components. We selected  these Army organizations for this objective because the Army is the only  service component that has an office responsible for advertising  functions—the Army Marketing and Research Group—that is separate  from the recruiting function, carried out by the U.S. Army Recruiting  Command. We compared identified policies and practices that describe  any coordination, as well as any instances of coordination described by  officials, against a best practice identified by private sector advertising  experts during the course of this review and our duplication, overlap, and  fragmentation evaluation and management guide. Specifically, the  private sector advertising experts we met with during the course of this  review stated that it is a best practice to consider an effective use of  coordination to increase efficiencies in advertising. As there are seven  advertising programs carried out by the military services as well as  JAMRS, we reviewed these programs against the evaluation and  management guide as it describes how to identify and evaluate instances  of fragmentation (more than one agency involved in the same broad  area), overlap (multiple agencies or programs with similar goals,  activities, or beneficiaries), and duplication (two or more agencies or  programs engaged in the same activities or services for the same  beneficiaries) among programs. Further, the guide can help identify  options to reduce or better manage the negative effects of fragmentation,  overlap, and duplication, and evaluate the potential trade-offs and  unintended consequences of these options.

We took four major steps to determine the extent to which DOD has  followed commercial best practices to assess the effectiveness of  advertising activities. First, we interviewed officials from the Office of the  Under Secretary of Defense for Personnel and Readiness and each of  the advertising programs of the military service components to determine  the types of advertising they conduct and how its effectiveness is  determined. Second, to identify existing best practices for assessing  effectiveness of advertising, we conducted a preliminary search, including  a literature review, and determined that a well-defined and widely  accepted list of best practices had not been established. Therefore, we  selected a nongeneralizable sample of advertising companies and  professional organizations using a “snowball sampling methodology,”  which consisted of interviewing advertising industry experts from an initial  set of organizations and requesting those experts to refer additional  contacts to participate in the review. Based on this approach, we  identified and interviewed the following organizations:

Companies: Ad Council, Agent, WideOpen, Widmeyer

Professional organizations: Association of National Advertisers,  An executive from another private sector advertising company preferred  that we not include the name of the organization that currently employs  the individual, stating that the input provided during the course of our  review reflected experience obtained from numerous positions held at  various advertising companies.

During the interviews, we asked officials to describe their knowledge of  industry best practices for assessing the effectiveness of advertising.  Based on the interviews, we identified key practices reported by each  expert or organization, including organizational structures to safeguard  against bias in performance evaluation, processes for planning and goal- setting in advance of advertising, and standard performance measures  used to assess advertising. We compiled these practices into a list and  provided the list to the organizations for review and comment. Although  the perspectives of the organizations included in our sample are not  generalizable to the advertising industry as a whole, we found a sufficient  degree of consensus among the sample for the purpose of developing  criteria for this review.

Third, following the development of the best practices, we interviewed  officials from each of the military service components responsible for  advertising, as well as the components’ contracted advertising agencies,  to determine the extent to which the components follow the practices we  identified. We compared the most recent annual advertising plan of each  of the military service components (which ranged from fiscal year 2014  through 2016, depending on the component) and other department- and  service-level guidance pertaining to advertising with the Standards for  Internal Control in the Federal Government, which requires that an  organization have relevant, reliable, and timely information and that there  is communication of that information throughout the agency in order to  achieve its objectives. We also considered regulatory requirements or  other potential barriers to government agencies following the best  practices we identified.

Finally, to determine how performance data related to advertising  activities is collected and reviewed below the headquarters level, we  developed and administered a structured questionnaire to lower level  recruiting officials with responsibility for local advertising from two lower  level units per service component. Based on our discussions with the  components’ headquarters-level advertising officials, we determined the  appropriate officials to participate in the semistructured interview were  located at the brigade level of the Army, district level of the Navy and  Marine Corps, squadron level of the Air Force active and reserve  components, and state level of the Army and Air National Guards. Given  the total population of 177 units, we chose to use a judgmental,  nonprobability sampling approach to select 2 subordinate units from each  of the seven components, for a total 14 interviews. We selected the units,  in consultation with service component headquarters officials, using  criteria intended to obtain a variation of perspectives within each of the  components. Due to the degree of differences among the components in  command structure and methods for allocating advertising funds across  subordinate units, we tailored the criteria to the characteristics of each  component. For several components, we selected units based on the size  of the budget received for local advertising, choosing a unit with a larger- than-average advertising budget and a unit with a smaller-than-average  advertising budget. If a component provided roughly equal local  advertising funding to all subordinate units, we used another selection  criterion to obtain varying perspectives, such as selecting two units that  represented regions with different propensities to serve or two units with  different recruiting missions such as an enlisted unit and a health  professionals unit. We relied on the headquarters officials from each of  the service components to identify states or units that matched our  selection criteria. Because we interviewed a nongeneralizable sample,  the results cannot be used to make inferences about the population as a  whole.

To determine the extent to which DOD has oversight of its advertising  activities, we reviewed (1) department-level guidance on advertising and  recruiting and (2) service-level guidance on the review and approval  processes related to the services components’ advertising programs. We  interviewed officials from the Under Secretary of Defense for Personnel  and Readiness about the oversight provided to service advertising  programs and the role of JAMRS. In addition, to determine how the  military service components provide direction to and oversight of their  respective advertising programs, we interviewed officials from the  headquarters of each of the military services components’ advertising  programs and we developed and administered a structured questionnaire  to lower level recruiting officials with responsibility for local advertising  from two lower level units per service component. We met with officials  from a judgmental, nonprobability sample of lower level units that  represent a geographic region or area of the United States, such as a  recruiting district, squadron, or brigade depending on the service. The  subordinate level units were selected to obtain distinct views within each  component, based on variations in the size of budgets, recruiting goals, or  the propensity of youth to serve in the military within the geographical  area of the unit. We compared any departmental and service guidance,  as well as the information obtained during our interviews, against  Standards for Internal Control in the Federal Government, which states  that an agency’s organizational structure should feature planning,  directing, and controlling activities to ensure that goals and objectives of  the agency can be achieved. Further, internal controls require that  policies be in place to direct an agency’s activities. Lastly, we reviewed  contracting data from fiscal year 2014 and some of fiscal year 2015 to  identify variations in the types of activities included in military service  component advertising contracts. We obtained contracting data from  fiscal year 2014 and some of 2015 because this time frame included  information before and after the issuance of interim guidance related to  sports advertising, including the most recently available data as of March  2016. Through our discussions with relevant experts and our review of  past work, we determined the data to be sufficiently reliable for the  purpose of corroborating the types of activities included in advertising  contracts.

We conducted this performance audit from June 2015 to May 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Military Service Component and Joint Advertising Organizations
paragraphs: Within the military departments, there are seven military service  component advertising programs and the organizational structure of each  program differs. Further, there is a joint advertising program—Joint  Advertising and Marketing Research Service—that is to provide some  advertising functions for the department.

Within all of the military service components, except for the Army Active  Duty and Reserve Component Advertising Program, the remaining six  programs’ advertising function is part of the recruiting function of the  respective military service component. Each of the military services has  its own recruiting structures and organizations, which are responsible for  the military service’s recruiting mission and functions. The role of a  military service’s recruiting command is to provide support to the  recruiting force and guidance for the recruitment and enlistment process.  In addition, a recruiting command plays a role in developing the recruiting  goals. The commands are structured similarly across the military services  with some variation in organizational structure. The recruiting command is  the recruiting headquarters for each military service, with subordinate  commands between the headquarters level and recruiting stations or  substations where frontline recruiters work to reach out to prospective  applicants and discuss the benefits of joining the military. Table 2  describes each of the seven advertising programs within the military  departments.

section_title: Appendix III: Advertising Program Budgets and Recruitment and Retention Goals
paragraphs: Each military service component receives an annual appropriation to  carry out operations and maintenance activities, including advertising and  marketing. Table 3 shows the DOD reported amounts allotted to the  military service components’ respective advertising and marketing  activities from each component’s annual operations and maintenance  appropriations for fiscal years 2015 through 2017. The Army has received  the highest allotments for its advertising programs. Specifically, the Army  has received about at least two to three times the amount that the other  active-duty components have received.

In addition, each military service component sets annual goals for  recruitment and retention, for both enlisted servicemembers and officers,  in order to meet defined endstrength requirements. Recruits must meet  numerous standards before they are accessed into the military,  specifically physical, educational, and other standards (e.g., an  acceptable record of behavior). For the reserve and guard components,  the geographic location of vacancies is also important for understanding  and addressing recruitment and retention goals because these  components recruit from a geographic area to fill vacancies in that area  and recruits typically continue to live and work in that area. Conversely,  active duty components recruit from anywhere within the United States  and its territories to meet recruitment goals and recruits are assigned  subsequently to duty stations. Table 4 shows the recruitment goals by  fiscal year from fiscal year 2014 through fiscal year 2016, as of March 1,  2016. The Army and the Army National Guard had the largest recruiting  missions over this time period.

section_title: Appendix IV: Comments from the Department of Defense

section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributors to this report  were Margaret Best (Assistant Director), Serena Epstein, Mae Jones,  William Lamping, Felicia Lopez, Suzanne Perkins, Carol Petersen,  Ophelia Robinson, Andrew Stavisky, and Amie Lesser.


Question-summary annotations: questions: question: What has DOD done to coordinate advertising activities?
answer: The Department of Defense (DOD) has taken steps to coordinate some advertising activities among the military service components, but it has not developed a formal process for coordination.
child_questions: question: What coordination already exists in DOD advertising?
answer: DOD conducts joint market research and service officials responsible for advertising at times share some information about lessons learned.

question: What limitations exist in its system?
answer: However, there is no formal process for addressing inefficiencies and to ensure information sharing among the services. GAO found examples of possible unnecessary duplication, overlap, and fragmentation that may result from the absence of coordination. For example, the Air Force has three advertising programs that contract with three advertising agencies, but officials could not provide a rationale for requiring separate programs.

question: What is the effect of poor coordination processes?
answer: In the absence of a formal process for coordination, the services may be missing opportunities to effectively leverage advertising resources.

questions: question: To what extent is DOD able to determine whether their advertisements are generating leads?
answer: While DOD has generally followed commercial best practices GAO identified to assess the effectiveness of advertising, DOD's components vary in their ability to determine whether their activities are generating leads for potential recruits.
child_questions: question: How does this vary across the different components?
answer: For example, while the Marine Corps has developed a framework to assess the effectiveness of its advertising including leads generated from advertising activities at the local level, Army officials stated they do not have reliable data to evaluate whether locally executed advertising activities are generating leads, and the Army National Guard does not require state units to report on the performance of their advertising activities.

question: Why is it important to assess lead generation?
answer: Without fully measuring advertising performance, especially at the local levels, DOD may be unable to ensure advertising dollars are used efficiently to help meet recruiting goals.

questions: question: How are DOD advertising activities overseen?
answer: DOD does not have comprehensive oversight of the components' advertising activities; instead, DOD's components oversee their own programs.
child_questions: question: What has the lack of centralized oversight produced?
answer: However, examples identified by GAO and others of some components paying sport teams to provide recognition ceremonies for servicemembers—a practice later deemed unacceptable by DOD—suggest that the absence of DOD oversight may have contributed to some activities of questionable appropriateness.

question: How may the lack of oversight lead to ambiguity?
answer: Further, GAO observed discrepancies in how recent sports advertising guidance was being interpreted and in service officials' understanding of regulations that direct digital advertising.

question: Why is a department-wide DOD advertising policy important?
answer: Without a department-wide policy that clearly defines its oversight role, DOD lacks reasonable assurance that advertising is carried out in an appropriate manner.